CaseVBench now covers 18 models. This issue sums up what has changed over the past month, because progress has been moving quickly.
When we published our first results on August 26, we tested six models. The top performer, Qwen3.8 Max, correctly identified 62% of objects. No model correctly identified more than a third of the countertops.
Only six weeks later, the results look very different. GPT-6 Astra now correctly identifies 92% of all objects, 84% of countertops and 96% of callouts.
The 4 key trends we see
1. GPT-6 Astra is still far ahead, while Claude Opus 5.5 is the new runner-up
Out of every 100 objects, Astra gets about 92 right. Opus 5.5 gets 79.
Four models were added in late September: Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna and Grok 4.7. Two of them landed near the bottom.
2. Two of four newer model versions scored lower than the ones they replaced
Each line compares an older model with its replacement. Claude’s upgrade from Opus 5 to Opus 5.5 is the biggest improvement we’ve measured.
In contrast, GPT-6 Sol and Grok 4.7 are both worse than their predecessors at interpreting drawings.
3. Floor plans are easy for everyone, but the small objects decide the ranking
Each dot represents one model.
For floor plans, even the worst-performing model scores 73%. Callouts, the small circles that link a plan to a detailed view, are much harder, with scores ranging from 0% to 96%. Countertops are the toughest of all: only one model identifies more than half of them correctly.
4. The second-best model costs about one-fifth as much as the best one
This chart shows the cost of processing a 200-page drawing set with each of the six most accurate models. Astra delivers 13 percentage points higher accuracy but costs about $35 extra per project.
All results
New case study: Arxion Takeoff Tool Audit
The AI audit focused on a takeoff tool, which reads architectural drawings and counts the items needed to price a construction job. The goal was to measure its accuracy and identify ways to improve it.
The team’s original accuracy scores came from drawings the model had already been trained on. We tested it on drawings it had never seen before and created an evaluation plan to track its performance going forward.
The team also had 69 features planned. After the audit, we retained 29: four that work with the current system and 25 that need reworking.
In October: see you in Florida!
I’m heading to the AWI Annual Convention and will be in Florida for three weeks. If you’re around, let me know! I’d also love any recommendations for places to visit or try while I’m there.
Subscribe for new updates
We’ll re-run CaseVBench whenever an interesting new model comes out. If you’d like to see the numbers as they change, subscribe on the benchmark page.
This issue first went out to subscribers of our LinkedIn newsletter, AI Era Development Stories, on October 12, 2026.
