CaseVBench: AI Benchmark for Architectural Drawings

Yes, to a degree. Models detect many cabinets, but they can merge adjacent units into one object or miss units entirely. Counting accuracy depends heavily on how the drawing is processed before the model sees it.

The most common reasons are image resolution limits, small and dense symbols, and objects that look similar to the surrounding linework. Much of the miscounting starts before inference, when the drawing is compressed to fit the model’s input size.

Yes. In our September 2026 run GPT-6 Astra was the most accurate model we tested — 92% overall, and the only one that finds countertops and callouts about as reliably as it finds floor plans. The rest of the GPT line sits well below it on the same drawings: GPT-5.6 Sol at 65%, GPT-5.6 Terra at 56%, GPT-6 Sol at 49% and GPT-6 Luna at 23%.

GPT-6 Astra, in our September 2026 run: 92% overall against fifteen models measured on the same drawings with the same rules. Claude Opus 5.5 is second at 79%, and at a median of $0.04 a page it costs under a quarter of what GPT-6 Astra does. The spread between models is real, but it is narrowest on floor plans and elevations, which every model reads well, and widest on countertops and callouts.

Not as it stands. Finding an object on a sheet is one step. A takeoff needs the right product code, split units merged into one, counts tied to callouts and the finish schedule, and output your estimating system can read — none of which is what these scores measure. At $0.19 a page it is also not cheap across a 500-page bid set, once per revision.

That is the work we do — custom systems trained on a shop’s own drawings, built to produce a usable takeoff rather than a list of detections.

Not on the basis of object recognition alone. It can automate parts of quantity takeoff, but estimating also involves scope, specifications, and judgment. See the section above.