Yes, to a degree. Models detect many cabinets, but they can merge adjacent units into one object or miss units entirely. Counting accuracy depends heavily on how the drawing is processed before the model sees it.
CaseVBench: how accurately does AI read architectural drawings?
In partnership
We evaluate how accurately AI models can interpret architectural drawings and identify cabinets, countertops, floor plans, elevations, and callouts.
CaseVBench: object detection accuracy across top AI models
Each model was given the same set of architectural drawings and asked to identify cabinets, countertops, floor plans, elevations, and callouts directly from the sheets. The results below compare 11 models using the same evaluation criteria. A higher score means the model found more of what is on the sheet without inventing objects that aren’t there.
Top performing models
| Model | Overall F1 | Cabinets | Countertops | Floor plans | Elevations | Callouts |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 92% | 82% | 84% | 99% | 94% | 96% |
| Gemini 3.8 Flash | 70% | 59% | 20% | 97% | 86% | 68% |
| Claude Fable 5.1 | 70% | 61% | 34% | 98% | 87% | 64% |
| GPT-5.6 Sol | 65% | 53% | 24% | 98% | 88% | 57% |
| Qwen3.8-Max | 62% | 60% | 32% | 98% | 90% | 47% |
Additional model results
| Model | Overall F1 | Cabinets | Countertops | Floor plans | Elevations | Callouts |
|---|---|---|---|---|---|---|
| GPT-5.6 Terra | 56% | 51% | 23% | 94% | 87% | 38% |
| Gemini 3.5 Flash | 53% | 45% | 16% | 96% | 90% | 33% |
| Gemini 3.1 Pro Preview | 51% | 39% | 10% | 89% | 78% | 37% |
| Claude Opus 5 | 40% | 51% | 19% | 96% | 89% | 15% |
| Grok 4.6 | 30% | 5% | 7% | 91% | 69% | 0% |
| Claude Sonnet 5 | 21% | 10% | 0% | 73% | 47% | 4% |
Every score is an F1 score: the balance between objects the model found and objects it invented. A model that misses half the cabinets scores badly — so does one that finds them all by drawing boxes over everything.
Run September 2026, on ~120 sheets with ~1,430 objects marked up by hand. Overall is that same measure pooled across all five object types, not the average of the five columns.
Version history
[2]
-
v2 — 5 Sep 2026
LatestFive models added: GPT-6 Astra, Gemini 3.8 Flash, Claude Fable 5.1, GPT-5.6 Sol and GPT-5.6 Terra. GPT-6 Astra changes the picture — 92% overall, and the first model to read countertops (84%) and callouts (96%) about as well as it reads floor plans, which is the v1 finding it overturns. The six models from v1 were re-scored on the same drawings under the same rules and did not move.
-
v1 — 26 Aug 2026
We tested 6 models: Gemini 3.5 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 5, Claude Opus 5, Qwen3.8-Max, Grok 4.6. Best overall accuracy: Qwen3.8-Max (62%), then Gemini 3.5 Flash (53%) and Gemini 3.1 Pro Preview (51%). Lowest: Claude Sonnet 5 (21%) and Grok 4.6 (30%). Every model reads floor plans and elevations well (73–98%), but no model found more than a third of countertops or half of callouts.
Version history — v1 — 26 Aug 2026 Model Overall F1 Cabinets Countertops Floor plans Elevations Callouts Qwen3.8-Max 62% 60% 32% 98% 90% 47% Gemini 3.5 Flash 53% 45% 16% 96% 90% 33% Gemini 3.1 Pro Preview 51% 39% 10% 89% 78% 37% Claude Opus 5 40% 51% 19% 96% 89% 15% Grok 4.6 30% 5% 7% 91% 69% 0% Claude Sonnet 5 21% 10% 0% 73% 47% 4%
How exactly is a detection counted?
Each model places a box around the objects it identifies on a drawing. We compare these boxes with the expert annotations. A detection is counted as correct when the model’s box overlaps the corresponding annotated object by enough to meet the evaluation threshold.
Boxes that do not correspond to an annotated object are counted as false detections. The resulting score therefore considers both correctly identified objects and incorrect detections.
-
IoU 0.34
missed at threshold 0.50
-
IoU 0.54
matched at threshold 0.50
-
IoU 0.84
matched at threshold 0.50
- ground-truth box
- model’s box
- overlap
The IoU threshold is how closely a model’s box has to match the expert’s box to count as found; we set it at 50% overlap. Boxes that match, boxes that fall short, and boxes drawn where nothing is marked are then combined into one number, the F1 score, used in the tables above.
Elevations & Countertops //Claude Fable 5.1
Elevations & Countertops //Gemini 3.8 Flash
Floor plans & callouts //GPT-6 Astra
Elevations, floor plans & callouts //Claude Fable 5.1
Cost and processing time
Median cost and processing time per drawing page, based on 119 pages evaluated for each model. We use the median because page density varies substantially, from pages with a single object to pages with more than sixty.
| Model | Cost per page, relative to the dearest model | ||
|---|---|---|---|
| Gemini 3.5 Flash | $0.026 | 15.5s | |
| Gemini 3.1 Pro Preview | $0.033 | 19.2s | |
| Qwen3.8-Max | $0.055 | 154.9s | |
| GPT-5.6 Terra | $0.056 | 32.0s | |
| Claude Sonnet 5 | $0.072 | 60.8s | |
| Claude Opus 5 | $0.112 | 37.2s | |
| GPT-5.6 Sol | $0.114 | 45.2s | |
| Grok 4.6 | $0.134 | 331.9s | |
| Gemini 3.8 Flash | $0.142 | 100.1s | |
| Claude Fable 5.1 | $0.148 | 22.3s | |
| GPT-6 Astra | $0.186 | 28.2s |
Follow the benchmark
We update the benchmark when new models are added and publish the results from the same evaluation process. Subscribe by email to receive updates when new results are available.
Frequently Asked Questions
-
-
The most common reasons are image resolution limits, small and dense symbols, and objects that look similar to the surrounding linework. Much of the miscounting starts before inference, when the drawing is compressed to fit the model’s input size.
-
Yes. In our September 2026 run GPT-6 Astra was the most accurate model we tested — 92% overall, and the only one that finds countertops and callouts about as reliably as it finds floor plans. Earlier GPT releases sit mid-field on the same drawings: GPT-5.6 Sol at 65% and GPT-5.6 Terra at 56%.
-
GPT-6 Astra, in our September 2026 run: 92% overall against eleven models measured on the same drawings with the same rules. The spread between models is real, but it is narrowest on floor plans and elevations, which every model reads well, and widest on countertops and callouts.
-
Not as it stands. Finding an object on a sheet is one step. A takeoff needs the right product code, split units merged into one, counts tied to callouts and the finish schedule, and output your estimating system can read — none of which is what these scores measure. At $0.19 a page it is also not cheap across a 500-page bid set, once per revision.
That is the work we do — custom systems trained on a shop’s own drawings, built to produce a usable takeoff rather than a list of detections.
-
Not on the basis of object recognition alone. It can automate parts of quantity takeoff, but estimating also involves scope, specifications, and judgment. See the section above.