# COXIT — full site content > Engineer-led AI/ML development studio building custom AI for document- and drawing-heavy workflows — computer vision, LLMs, and AI agents. Production systems shipping for organizations such as Internet Archive and Stevens Industries. Founded 2018. Offices in Casper, Wyoming (USA) and Lviv (Ukraine). Founders: Iryna Mykytyn (Founder, CEO & CTO) and Volodymyr Hresko (Co-Founder & COO). Contact: volodymyr@coxit.co Index version of this document: https://coxit.co/llms.txt Generated: 2026-10-06 ## Services --- ### AI Audit URL: https://coxit.co/service/ai-audit/ Independent technical review of an existing AI system or build plan. You get accuracy numbers, a prioritised defect list, and a fix plan with time estimates — from engineers who ship production AI, not analysts. Frequently asked questions Q: What do we actually get? A: A written report covering: data quality assessment with specific examples, reproducible model evaluation with accuracy numbers, infrastructure and scaling findings, security review, and a prioritized task list with time estimates. Quick wins separated from architectural fixes – so you know what to do this week vs what to budget for next quarter. Q: We don't have AI in production yet — is the audit still for us? A: Yes. Many audits happen before any AI is built – checking whether your data is good enough, whether your architecture can handle what you’re planning, and whether AI is actually the right answer for the workflow. Sometimes the recommendation is “don’t build AI for this – here’s a simpler approach.” Better to know before you spend. Q: What kind of problems do you typically find? A: Common patterns: data labeling errors that no amount of GPU compute can fix, prototypes built fast with no scaling plan, missing security basics (exposed API keys, no rate limits, no prompt injection protection), inefficient model choices that quietly drive up monthly bills, and architecture decisions that look fine at low load and break at scale. The audit catches issues that look small until they hit production. Q: What about NDA and data security? A: Standard NDA before any access. Data stays in your environment whenever possible; when we need samples, we use your secure transfer of choice. We don’t reuse client code or data for anything outside your engagement. Same protections we use on long-term builds for Stevens Industries and Internet Archive. Q: Won't you just try to upsell us on fixing what you find? A: The audit report is yours – take it to any vendor or hand it to your in-house team. Many clients do exactly that for at least some of the fixes. We’ll also tell you upfront if a different team is better suited for the build (e.g., niche domain expertise we don’t have). Independence is the point of an audit. --- ### AI/ML Software Development URL: https://coxit.co/service/machine-learning/ Custom AI systems built end to end: computer vision, LLMs, and multi-agent automation for document- and drawing-heavy workflows. Senior engineers, and a production system you own. Frequently asked questions Q: How quickly can you start? A: The free technical review takes about 5 working days once we have everything we asked for. PoC kickoff is roughly 10 days after you green-light. Q: What's the smallest engagement? A: $50k total budget, or one full-time engineer for three months – whichever is larger. If you’re not sure, start with a free technical review. Q: Who owns the code and IP? A: You do, from day one. Full IP transfer is our default. All code stays in your repositories – we keep only general methodology and domain knowledge. Q: What if we're not sure AI will work on our data? A: That’s what the PoC is for. We train on a slice of your real data and show measured accuracy against your own standard before you commit to a full build. Q: Do you work in our time zone? A: At least four hours of overlap with both US Eastern and Pacific. Q: What happens after launch? A: Ongoing support – retraining on new data, new object and document types, monitoring, and fixes. The team that built it runs it. --- ### Proof of Concept URL: https://coxit.co/service/poc/ 8-12 weeks with our senior team. You leave with a working prototype on your real data - measured against your own accuracy standard- plus a technical plan and a budget for the full build. Frequently asked questions Q: What do you actually walk away with? A: A working prototype trained on your real data, plus a written report covering accuracy metrics (recall, precision, F1), architecture decisions and why we made them, infrastructure cost estimates for production, edge cases we found, and a prioritized MVP plan with budget and timeline. The prototype isn’t a slide deck demo – it’s a runnable system you can show internal stakeholders and use to validate the business case before committing to a full MVP build. Q: Who owns the prototype, code, and trained models? A: You do. Default is full IP transfer – all code, models, and training data stay yours. We retain methodology and general domain knowledge only, nothing client-specific. This matters if you want to hand the PoC to another team for the MVP, or take what we built in-house. It’s written into the contract before any work starts. Q: What data do you need, and how is it protected? A: For the free technical review: a small sample, usually 5–20 documents. For the full PoC: a larger labeled or labelable dataset – exact volume depends on the problem and gets scoped in the discovery call. NDA before any access. Data stays in your environment when possible; when we need transfers, we use your secure channel of choice. Same protections we use on long-term builds for Stevens Industries and Internet Archive. Q: What if the PoC shows AI won't work for our case? A: That’s still a successful PoC. Better to find out in 6 weeks than 6 months into an MVP. Roughly a third of these engagements end with “don’t build this” or “build a simpler version” – and clients tell us that answer alone paid for the work. You still walk away with the accuracy report, the edge cases we documented, and a clear understanding of what would need to change for AI to work – useful even if you go a different direction. Q: Why $20-60k? What drives the range? A: Three factors: data complexity (clean structured PDFs sit at the low end, mixed-format legacy documents at the high end), how much annotation we need (existing labeled data shortens the timeline), and whether we’re testing one model approach or comparing several. Q: What happens after the PoC? A: The report gives you a scope, budget, and timeline for production. From there: a short proposal, contract, and we either continue with the same team or scale up for the MVP build (typically 3-6 months, $80–150k depending on integrations). We’ve done this trajectory end-to-end with Stevens Industries – PoC in late 2023, MVP launched in 2024, system now in daily production use. You’re not obligated to continue with us though; the PoC deliverables are vendor-neutral by design. Q: A slice of real data A: Ideally examples you've already processed, so we can measure the AI against known-good answers. Q: Your accuracy standard A: The threshold you already hold your team to. We measure against yours, not ours. Q: A domain expert's time A: a few hours a week for the first month, to walk us through the manual workflow and settle edge cases. The first couple of weeks go to understanding how the work is done today, before we write model code. --- ### Team Extension URL: https://coxit.co/service/team-extension/ Senior AI/ML, Python, and C++ engineers embedded on your team in 2–4 weeks at an all-inclusive rate. Screened by engineers, not recruiters. ## Industries --- ### Automotive URL: https://coxit.co/industry/automotive/ Embedded and backend engineering for automotive and in-car technology, including multi-year team extension for Xperi. --- ### FinTech URL: https://coxit.co/industry/fintech/ AI and data engineering for fintech, including a multi-agent LLM platform for investment research and coverage management. --- ### IT Services URL: https://coxit.co/industry/it-services/ Engineering teams extended into IT services and product companies, covering AI/ML, Python, C++, and backend development. --- ### Manufacturing URL: https://coxit.co/industry/manufacturing/ AI that reads your drawings and spec books — estimating, takeoffs, submittals, and revision tracking for casework and millwork manufacturers. Stevens Industries cut estimating from about 6 hours to 10 minutes. Frequently asked questions Q: What does a Free Technical Review involve? A: Send us 2–3 of your recent project PDFs. Within 3–5 days we send back an annotated drawing showing what our AI detects, a structured spreadsheet of items found, and an honest read on whether AI is the right fit for your workflow. No commitment, no sales pressure. If it’s not a fit, we’ll tell you. Q: How do I know AI can read my drawings accurately? A: The honest answer: not until we test it on yours. Off-the-shelf tools fail on production drawings because they’re trained on synthetic data or rely on text search for AWI codes. We trained custom systems with real drawings and hit 91% accuracy. Your drawings are different — different title blocks, different conventions, different cabinetry mix. That’s why every engagement starts with a Free Technical Review on your actual files before anyone signs anything. Q: Where does our data live? Is it secure? A: Your drawings and specs stay in isolated environments — not used to train models for other clients, not shared, not pooled. Deployments run on AWS or GCP under your account if you prefer, or under ours with full audit logs. We sign NDAs before any file moves. For shops with stricter requirements, on-prem deployment is possible. Q: Who owns the IP? A: You own what we build. The system, the trained model on your data, the code – all yours, in your repository, from day one. We retain methodology and knowledge, not your system. Q: What does it cost? A: Free Technical Review: free. Discovery + Proof of Concept: $20K–$60K, 8-12 weeks. Production build: $80K–$200K, 3–6 months. Ongoing support: $5K-$10K/month. Most shops start with the PoC to validate accuracy on their drawings before committing to a full build. Q: Do you integrate with Cabinet Vision, Microvellum, or our ERP? A: Yes. We export structured data (CSV, JSON, or direct API) that drops into Cabinet Vision, Microvellum, Innergy, or custom ERPs. Output formats are defined during discovery so the spreadsheet matches your quoting workflow line-for-line. No retyping, no copy-paste. Q: How long until we see results? A: First annotated output from Free Technical Review: 3–5 days. PoC running on your data with accuracy metrics: 4–8 weeks. Production system in daily use: 3–6 months. Stevens went from manual review to AI-assisted estimation in under 6 months. --- ### Non-profit URL: https://coxit.co/industry/non-profit/ Engineering support for non-profit and public-interest technology, including large-scale book digitisation at Internet Archive. ## Case studies --- ### Multi-Agent Investment Research Platform for BRDGE Insights URL: https://coxit.co/cases/strem/ Client: BRDGE Insights Published: 2026-05-22 | Updated: 2026-09-18 A multi-agent LLM system built for BRDGE Insights that automates investment research and coverage management for fintech analysts. --- ### Building the Foundation for an AI-Powered Investment Research Platform URL: https://coxit.co/cases/strem-discovery/ Client: BRDGE Insights Published: 2026-05-20 | Updated: 2026-06-04 Discovery and architecture work laying the foundation for the BRDGE Insights AI-powered investment research platform. --- ### Production Takeoff Tool for Stevens Industries URL: https://coxit.co/cases/estimation-stevens-part3/ Client: Stevens Industries Published: 2025-08-09 | Updated: 2026-06-12 How COXIT built Archiscan, a production takeoff tool that cut cabinet estimating from about 6 hours to 10 minutes at Stevens Industries — 90% object detection accuracy on construction sets up to 470 sheets. Frequently asked questions Q: How did the collaboration with COXIT begin? A: The collaboration started after Curtis Garrard identified the need for a tailored solution. He drafted a statement of requirements and sought external development partners, ultimately selecting COXIT for their understanding of the technical scope. Q: What metrics were used to measure the success of the solution? A: Success was measured through metrics such as the number of projects processed, user engagement levels, feedback on usability, and the overall increase in efficiency and accuracy in estimates. Q: What advice would you give to companies looking to implement a similar solution? A: Clearly define the business problem and requirements, involve end users in the development process, and be open to iterative feedback. Q: How can companies ensure effective user adoption of new technology? A: Providing education and training, emphasizing the benefits of the new tool, offering ongoing support, and creating a gradual transition plan. Q: How much faster is estimating now? A: Drawing review that took 2–6 hours now takes about 10 minutes, with the estimator reviewing the draft. The goal the tool is built around: one estimator processing around twenty drawings a day instead of two to four. Q: Have you tried using off-the-shelf tools for your needs? A: Yes, several. Most did not adequately address the specific requirements of cabinetry estimation, which led to pursuing a custom solution. Q: Why can't off-the-shelf AI read these drawings? A: Because there is no standard. The same wall cabinet is drawn differently by every architect: shelves shown or not, the handle as a stick, a rounded rectangle, or a square one; top views on the same page or a separate one. Three files from three projects can mean three different drawing styles for the same cabinet type. Only the concept stays constant — cabinets above, an elevation label below — so the model has to learn the concept, not the picture. --- ### Stevens Industries POC 2 URL: https://coxit.co/cases/poc-stevens-part2/ Client: Stevens Industries Published: 2025-08-09 | Updated: 2026-06-08 A proof of concept validating the machine-learning estimation approach for Stevens Industries before committing to a custom production build. --- ### Spec Book AI Agent for Stevens Industries URL: https://coxit.co/cases/ai-powered-document-agent/ Client: Stevens Industries Published: 2025-07-03 | Updated: 2026-06-08 An AI agent built for Stevens Industries that reads architectural spec books and extracts project requirements from drawing-heavy documents. --- ### Stevens Industries POC URL: https://coxit.co/cases/discovery-stevens-part1/ Client: Stevens Industries Published: 2025-06-03 | Updated: 2026-06-08 The discovery phase for machine-learning-powered estimation software at Stevens Industries: validating feasibility and mapping a technical path before development began. --- ### Xperi URL: https://coxit.co/cases/xperi/ Client: Xperi Published: 2024-08-07 | Updated: 2026-06-08 COXIT screened 426 candidates and staffed 15 developers to Xperi across three years of embedded and backend engineering collaboration. --- ### Internet Archive URL: https://coxit.co/cases/archive-org/ Client: Internet Archive Published: 2024-08-06 | Updated: 2026-05-28 Python engineering supporting large-scale book digitisation at Internet Archive, the world's largest online library. Archive.org --- ### Tempest URL: https://coxit.co/cases/tempest/ Client: Tempest Published: 2024-08-06 | Updated: 2026-06-08 Backend engineers improving data parsers for Tempest, a privacy-focused browser. Tempest ## Articles --- ### Understanding F1 Scores in AI Drawing Benchmarks URL: https://coxit.co/blog/understanding-f1-scores-ai-drawing-benchmarks/ Author: Volodymyr Hresko Published: 2026-09-15 | Updated: 2026-09-15 Category: AI for Drawings When we publish CaseVBench results, one of the main numbers we report is called F1. CaseVBench is a benchmark for measuring how accurately AI models can locate objects on architectural drawings, such as cabinets, countertops, floor plans, elevations and callouts. What is F1? F1 is a single score that combines two questions about a model's output: how much of what it found was correct (precision), and how much of what exists it found (recall). It always lands closer to the weaker of the two, so a model can't score well by being good at only one. F1 is a standard metric in machine learning, but the name does not tell you much about what it measures. Precision tells you how many of the model's predictions are correct Imagine you hand someone a kitchen drawing and ask them to circle every cabinet. There are 6 cabinets on the page. They circle 5 things. When you check, 4 of those are cabinets. The fifth is a fridge they mistook for a cabinet. Of the 5 things they identified, 4 were correct. That gives a precision of 80%. Recall tells you how many of the actual objects the model found There are 6 cabinets on the drawing and the person found 4 of them. That gives a recall of 67%. The two measurements describe different types of errors. Precision is affected by incorrect predictions. Recall is affected by objects that were missed. F1 combines precision and recall so that both types of error affect the score Each measurement can look good on its own. A cautious person might circle one cabinet they are completely sure about. Precision would be 100%, but recall would be only 17%. Another person might circle everything that looks even slightly like a cabinet, say 12 things. All 6 cabinets are in there, so recall is 100%, but half of what they circled is wrong, so precision is 50%. F1 combines precision and recall into one number, and it always lands closer to the weaker of the two. This is what stops a lopsided score from hiding behind a good one. For the person with 80% precision and 67% recall, F1 is approximately 73%, about the same as a plain average. For the person with 100% recall and 50% precision, it is 67%, where a plain average would give 75%. The formula is F1 = 2 × (precision × recall) / (precision + recall). The gap is widest when precision and recall are far apart. With 100% precision and 17% recall, for example, F1 is approximately 29%, rather than the 58% that a simple average would produce. A 70% F1 doesn't mean 70% of the drawing is correct An F1 score of 70% does not mean that the model understands 70% of the drawing. In CaseVBench, a model is asked to find specific objects and draw a rectangle around each one. That rectangle is called a bounding box. Each prediction is compared with a rectangle a person drew around the same object, and the two count as a match only if they overlap enough. F1 measures detection performance under a specific evaluation method. It is not a general measure of how well an AI model understands an architectural drawing. How close the box has to be to count Finding the right cabinet is only half of a match. The model also has to draw its box in roughly the right place. The rule for "roughly" is called IoU. What is IoU? Lay the model's box and the person's box on top of each other. IoU is the area they share, divided by the total area the two boxes cover together. Boxes sitting exactly on top of each other score 1; boxes that don't touch score 0. The name stands for Intersection over Union, which is the maths way of saying overlap over combined area. CaseVBench counts a prediction as a match when IoU is at least 0.50, so at least half of the combined area has to be shared. A box that is a little off still passes. A box that slid halfway off the cabinet, or one drawn twice too big, does not, even though the model clearly saw the cabinet. The threshold changes F1 too. The same set of predictions scores higher at IoU 0.20 and lower at 0.70, which is why every F1 in the results says which threshold it was measured at. The primary CaseVBench results use 0.50. F1 can be very different for different types of objects The same model can perform well on one type of architectural object and poorly on another. In the initial CaseVBench results, Qwen3.8-Max scored 98% F1 on floor plans, 90% on elevations, 60% on cabinets, 47% on callouts and 32% on countertops. Its overall F1 was 62%. Large objects such as floor plans and elevations are easier to locate than small callouts. Countertops can also be difficult because they may appear as thin lines within an elevation rather than as clearly defined shapes. This is why an overall F1 score needs to be read alongside the object type being measured. Two models with the same F1 can make different kinds of mistakes F1 does not show whether a model's errors mainly come from missed objects or incorrect predictions. One model might have relatively high precision but lower recall. It finds fewer objects, but most of its predictions are correct. Another might have higher recall but lower precision. It finds more objects, but also produces more incorrect detections. The two models could have similar F1 scores while requiring different amounts of human review. That is why precision and recall are useful alongside F1 when comparing models. An F1 score needs to be interpreted in the context of the task There is no universal F1 score that defines whether a model is good enough for architectural drawings. The useful level of performance depends on the task, the object type and how the output will be used. When reading a CaseVBench result, look at F1, precision, recall, IoU threshold and object type together. F1 gives you the combined result, while the other measurements provide the context needed to understand it. --- ### September 2026: Introducing CaseVBench, an Accuracy Benchmark for AI on Architectural Drawings URL: https://coxit.co/blog/introducing-casevbench-accuracy-benchmark-ai-architectural-drawings/ Author: Volodymyr Hresko Published: 2026-09-14 | Updated: 2026-09-15 Category: COXIT Newsletter A few weeks ago, we launched our own benchmark, CaseVBench. We ran 11 models across 119 real drawing pages from different sets, and asked each model to find 5 types of objects: cabinets, countertops, floor plans, elevations and callouts. The scoring rules Before running the models, our data annotator went through all 119 pages and marked every object by hand. There were 1,430 objects in total. We then gave the same pages to each model with one simple prompt: find these objects and draw a box around each one. We could then compare each model's boxes against the annotations. Models make two typical errors: they can miss an object, or they can draw a box where there isn't one. The second mistake shows up a lot in the results. A model could find a callout by drawing boxes all over the page. It would probably catch the real callout eventually, but the result wouldn't be useful. Here, for example, there was one callout marked on the sheet, and the model drew 67 boxes. That's why we set F1 as the main score. It combines two things: how many of the annotated objects the model found, and how many of its boxes were correct. A model that finds every object but draws 67 boxes to do it scores low, and so does a model that draws only correct boxes but misses half the objects. 100% means every box matched the annotation and nothing was missed. The CaseVBench results GPT-6 Astra is the clear winner at 92% As you can see, GPT-6 Astra was the clear winner at 92%, followed by Claude Fable 5.1 and Gemini 3.8 Flash at 70%. Floor plans and elevations were the easiest objects for the models to find. That makes sense: they're large, usually separated from everything else, and often take up a big part of the sheet. This page is a good example: the model found all 12 objects. On the other hand, a callout might be only a few pixels wide once the sheet is reduced. A countertop can be a thin line sitting directly on top of a run of cabinets. Those are much harder for the models to distinguish. Here's a page where the model found 9 of 16 cabinets: Learn more about the research On the benchmark page you can dig into more of our findings, like how speed compares with cost per model run. There is also a deeper article about the benchmark that highlights some insights and what you can do with them. So… is AI takeoff here? Could you just upload your drawings to Astra and get a complete takeoff? 🫣 Not really. CaseVBench measures just finding objects on a page. A professional takeoff also requires reading dimensions and schedules, distinguishing base cabinets from wall cabinets, measuring linear feet, connecting information across sheets, and making judgements when the drawing isn't clear. And even at this first step, only one model is close to being reliable. It still gets roughly one box in ten wrong. So the general model isn't the product. What we're finding works much better is building a system around the model. You can crop the elevations, read them at full resolution, ask the model to identify the objects, and then check what it returned. That's what we're building for clients, and we'll share those numbers in the next benchmark. IWF in Atlanta I was also at the International Woodworking Fair in Atlanta recently, and had a lot of good conversations there. I was glad to spend some time offline with Curtis Garrard and to receive the Wood Industry 40 Under 40 award. What positively surprised me is that many people are already building something on Claude Code. Shop owners are trying to build tools that optimize their own workload. But at the same time, most are still using AI mainly to summarize text and write things. So I left very motivated to produce even more educational materials. Subscribe for new updates We'll re-run CaseVBench whenever an interesting new model comes out. If you'd like to see the numbers as they change, subscribe on the benchmark page. This issue first went out to subscribers of our LinkedIn newsletter, AI Era Development Stories, on September 14, 2026. --- ### GPT-6 Astra Scores 92% on Architectural Drawings: What Does It Mean for AI Takeoffs? URL: https://coxit.co/blog/gpt-6-astra-architectural-drawings-benchmark/ Author: Volodymyr Hresko Published: 2026-09-09 | Updated: 2026-09-09 Category: AI for Drawings CaseVBench: a benchmark of 11 multimodal AI models on 119 architectural drawing pages, September 2026. Models are re-run as new versions become available. GPT-6 Astra scores 92%, outperforming all other models We tested eleven multimodal AI models on 119 pages of architectural drawings. Each model received the same task: find every cabinet, countertop, floor plan, interior elevation, and callout on the page and draw a box around each one. The reference set contained 1,430 objects marked by a human annotator, giving us a consistent way to compare the model results. GPT-6 Astra scored 92% overall. Claude Fable 5.1 and Gemini 3.8 Flash followed at 70%. No other model scored above 65%. That is a substantial change from our August test, when the highest score among the models we tested was 62%. Large drawing elements are easier for most models Floor plans and interior elevations were the easiest categories for most models. Astra scored 99% on floor plans and 94% on elevations. Most of the other models also performed well on these categories, with floor-plan scores of at least 89% for all but one model and elevation scores of 86% or higher for eight of the eleven models. These elements are generally larger and occupy more of the drawing sheet, which makes them easier to identify. The larger differences appear with smaller objects. Countertops Astra scored 84% on countertops. The next highest model scored 34%. Countertops can appear as thin lines along cabinet runs, making them more difficult to identify and outline on a full drawing sheet. Callouts Astra scored 96%, compared with 68% for the next highest model. Callouts can be very small when an entire drawing sheet is provided to a model as a single image. At lower image resolutions, a callout may occupy only a few pixels. Cabinets Astra scored 82%. The next highest model scored 61%. These results show that the difference between models is not simply about whether they can understand a drawing. Large drawing elements are handled reasonably well by many of the models in this test. The bigger differences appear when the objects are smaller or require more precise identification. How the models compare ModelCabinetsCountertopsFloor plansElevationsCalloutsAll objectsGPT-6 Astra82%84%99%94%96%92%Claude Fable 5.161%34%98%87%64%70%Gemini 3.8 Flash59%20%97%86%68%70%GPT-5.6 Sol53%24%98%88%57%65%Qwen3.8-Max60%32%98%90%47%62%GPT-5.6 Terra51%23%94%87%38%56%Gemini 3.5 Flash45%16%96%90%33%53%Gemini 3.1 Pro Preview39%10%89%78%37%51%Claude Opus 551%19%96%89%15%40%Grok 4.65%7%91%69%0%30%Claude Sonnet 510%0%73%47%4%21%F1 score at IoU 0.50, based on 119 pages from nine publicly available projects. Direct mode was used: the complete page was provided as one image and the model returned a list of bounding boxes. All models received the same prompt. Here we see two patterns stand out: First, the models are generally stronger at finding large drawing elements than smaller ones. The rankings are largely determined by performance on cabinets, countertops, and callouts. Second, newer models within the same family generally performed better than earlier versions in this test. Gemini 3.8 Flash scored 17 points higher than Gemini 3.5 Flash. Claude Fable 5.1 scored 30 points higher than Claude Opus 5, with much of the difference coming from callouts: 64% compared with 15%. GPT-6 Astra scored 27 points higher than GPT-5.6 Sol. Fable shows a large improvement over Opus 5 Claude Fable 5.1 scored 70% overall, compared with 40% for Opus 5. Most of the improvement came from smaller objects: callouts increased from 15% to 64%, countertops from 19% to 34%, and cabinets from 51% to 61%. Fable also produced fewer incorrect extra boxes. Opus often labelled circles, tags, and other symbols as callouts, while Fable was better at distinguishing them. Fable produced about 440 extra boxes across the test, compared with about 1,500 for Opus. The broader pattern is similar across the models: large drawing elements are already handled well, while newer generations are making more progress on small symbols, thin lines, and other details on crowded pages. Gemini 3.8 Flash improves mainly on callouts Gemini 3.8 Flash also scored 70% overall, but its results differ from Fable's. It found slightly fewer objects, while a higher share of its reported objects were correct. In practice, this means less deleting of incorrect boxes but more checking for objects it missed. Most of its improvement over Gemini 3.5 Flash came from callouts, which increased from 33% to 68%. The trade-off is processing time: Gemini 3.5 Flash took about 20 seconds per page, while version 3.8 took about 104 seconds. Cost and speed Accuracy is only one consideration when choosing a model. We also recorded processing time and cost for each page. ModelCost per pageTime per pageAll objectsGemini 3.5 Flash$0.03520 s53%Gemini 3.1 Pro Preview$0.04023 s51%GPT-5.6 Terra$0.05632 s56%Qwen3.8-Max$0.057161 s62%Claude Sonnet 5$0.07360 s21%GPT-5.6 Sol$0.12049 s65%Claude Opus 5$0.13448 s40%Grok 4.6$0.145362 s30%Gemini 3.8 Flash$0.150104 s70%Claude Fable 5.1$0.16426 s70%GPT-6 Astra$0.22039 s92%Average results from the 119-page run through OpenRouter, with pages processed sequentially. Astra was the most expensive model in the test at $0.22 per page. It also had the highest overall score. Cost and accuracy did not follow the same pattern across the other models. Some of the more expensive models scored below less expensive alternatives. Processing speed also varied independently of cost and accuracy. Astra processed a page in about 39 seconds. Fable was the fastest of the three highest-scoring models at 26 seconds per page. Gemini 3.8 Flash was less expensive than both but took about 104 seconds per page. Does this mean AI can do a takeoff? This is where the benchmark needs some context. The test measures one part of a takeoff: finding predefined objects on a drawing page and drawing a box around each one. A complete takeoff involves several additional steps. An estimator may need to read dimensions and schedules, distinguish between different cabinet types, match elevations to the rooms they belong to, calculate linear feet of countertop rather than simply count countertop views, and make decisions based on information that may be implied rather than explicitly shown. None of those tasks are measured here. What the results show is that AI has become more capable at locating objects within architectural drawings. With Astra, 92% of the reference objects were correctly identified at the benchmark's main scoring threshold. It scored 96% on callouts and 94% on elevations. That makes its output useful as a starting point for navigating a drawing set and locating objects. But the remaining errors still require review, and performance varies between projects. Astra also produced more additional boxes than missed objects. In practical terms, this means a reviewer may spend more time removing incorrect boxes than searching for objects the model failed to find. That can make the output easier to review, but it does not remove the need for review. For the other ten models, the results suggest a greater level of checking is needed. At 70% overall or below, the output should be treated as an initial reference rather than a finished result. Countertops and callouts remain particularly difficult. AI can assist with takeoffs, but object detection is only one step AI can help locate and organize information in architectural drawings, but these results do not show that AI can complete a construction takeoff without human review. Object detection is becoming more capable, particularly for large drawing elements and, with the best-performing model in this test, for smaller elements as well. A takeoff, however, requires more than locating objects. We re-run this benchmark as new models ship; the running results live on CaseVBench. --- ### How Accurate Is AI at Reading Architectural Drawings? Benchmark Results for 6 AI Models. Part 2 URL: https://coxit.co/blog/ai-architectural-drawings-benchmark-part-2/ Author: Volodymyr Hresko Published: 2026-08-28 | Updated: 2026-08-28 Category: AI for Drawings Quick answer: In our August 2026 benchmark of six multimodal AI models across 119 architectural drawing pages and 1,430 hand-labelled objects, Qwen3.8-Max achieved the highest overall F1 score at 62%, followed by Gemini 3.5 Flash at 53%. AI was much better at locating large objects such as floor plans and interior elevations than small or thin objects such as callouts and countertops. In direct full-page processing, current general-purpose AI models are useful for navigating drawing sets and generating starting-point object counts, but their output still requires human verification for estimating work. These are initial results from August 2026. More models and newer versions will be added as the benchmark grows. Part 1 recap In Part 1 we introduced the problem. Estimators working in casework and millwork spend a significant part of their day finding objects on architectural drawings: cabinets, countertops, interior elevations, and the small callout bubbles that connect one view to another. Multimodal AI models can now take a drawing page as an image and answer questions about what is on it. We wanted to measure how reliably they can locate specific objects on architectural drawings, and where they break down. This is harder than it sounds. Architectural drawing pages are dense. When a full sheet is compressed to the resolution accepted by an AI model, a callout can become only a few pixels wide. Object size, image resolution, drawing density, scan quality, and the way the task is posed can all affect the result. Part 2 focuses on the first of these questions: given the same full architectural drawing page and the same task, how accurately can different AI models locate the same objects? In this benchmark, "reading" a drawing means locating predefined objects on a page and returning bounding boxes around them. We are not testing whether a model can interpret a whole project or produce a takeoff (more on that in the FAQ below). The AI architectural drawing benchmark: 119 pages, 1,430 objects, 6 models We collected drawings from nine publicly available projects, covering 119 pages in total. The projects come from different offices and different decades of drafting habits, so drawing quality and page density vary. Some pages contain only one labelled object; the busiest pages contain more than 60. An annotator manually marked every object on every page. The benchmark contains approximately: 560 callouts 320 cabinets 300 elevations 170 floor plans 90 countertops That gives us approximately 1,430 hand-labelled objects against which to compare AI output. Expert markup on one sheet. Every box is drawn by hand. Which AI models were tested? Six multimodal AI models were run on every page using the same direct-mode approach: Qwen3.8-Max Gemini 3.5 Flash Gemini 3.1 Pro Preview Claude Opus 5 Grok 4.6 Claude Sonnet 5 In direct mode, the complete drawing page is provided to the model and the model returns a list of bounding boxes with coordinates. Other models will be added as the benchmark grows. How accuracy is scored A predicted box counts as correct when it overlaps the human-labelled box by at least half of their combined area (IoU 0.50). From the correct, stray, and missed boxes we calculate an F1 score, which balances objects found against objects invented. A loose box that misses the threshold counts as both a stray prediction and a missed object. The FAQ explains IoU and F1 in more detail. Three model boxes for the same object. Only the middle and right ones count at the 0.50 threshold. Which AI model is best at reading architectural drawings? In this first benchmark, Qwen3.8-Max achieved the highest overall score, with an F1 score of 62%. Gemini 3.5 Flash was second at 53%, followed by Gemini 3.1 Pro Preview at 51%. However, there is no single model that performs equally well on every type of architectural drawing object. AI modelCabinetsCountertopsFloor plansElevationsCalloutsAll objectsQwen3.8-Max60%32%98%90%47%62%Gemini 3.5 Flash45%16%96%90%33%53%Gemini 3.1 Pro Preview39%10%89%78%37%51%Claude Opus 551%19%96%89%15%40%Grok 4.65%7%91%69%0%30%Claude Sonnet 510%0%73%47%4%21%F1 score by AI model and object type, at IoU 0.50, over 119 architectural drawing pages (August 2026). The model ranking is less informative than the difference between object types. The same AI model can be very good at finding a floor plan and very poor at finding a countertop on the same type of drawing. That means there is no meaningful answer to "How accurate is AI at reading architectural drawings?" without also asking: accurate at finding what? How accurate is AI at finding different objects on architectural drawings? Floor plans: AI performs very well Floor plans are the easiest object category in this benchmark. The best result was 98% F1 from Qwen3.8-Max, and every model except Claude Sonnet 5 scored at least 89%. Floor plans are relatively large visual objects and often occupy a substantial portion of a drawing sheet. Their size makes them easier for a multimodal model to identify than small symbols or thin lines. Interior elevations: AI also performs well Interior elevations are the second-strongest category. The best models reached 90% F1, and three of the six scored 89% or higher. Like floor plans, elevations are relatively large views, often occupying a quarter of a page or more. Their visual scale gives the AI model considerably more information to work with than a tiny callout or thin countertop line. Cabinets: AI is usable, but not reliable enough to skip checking Cabinets are smaller objects embedded within elevations and other views. The strongest cabinet result was 60% F1 from Qwen3.8-Max; the top four models scored between 39% and 60%. For estimating workflows, cabinet detection can provide a useful starting point, but the benchmark does not support treating raw AI output as a final cabinet count without verification. Callouts: AI struggles with small symbols Callouts are small circles containing numbers or other identifiers. The models performed very differently here. Qwen3.8-Max achieved 47%, while Gemini 3.1 Pro Preview and Gemini 3.5 Flash achieved 37% and 33%. Other models performed poorly. Grok 4.6 returned 317 callout boxes across the dataset, but only two matched the human-labelled callouts at the 0.50 IoU threshold. The problem is partly geometric. A callout may be only a few pixels across in a full-page image, so a small positional error makes a large difference to its IoU score. Countertops: the hardest category Countertops produced the lowest scores. The best result was only 32% F1 from Qwen3.8-Max. Countertops are not necessarily the smallest objects on a sheet. The problem is their visual representation. On an elevation, a countertop can appear as a thin horizontal line sitting above a cabinet run. Unlike a cabinet, it may not have a clear enclosing shape that an AI model can use to determine the object's boundaries. No model scored above one third on countertops, and Claude Sonnet 5 found none at the primary IoU threshold. Object size predicts AI accuracy better than model choice For every model, the broad pattern is similar: floor plans, then elevations, then cabinets, then callouts, then countertops. The best model scores 98% on floor plans but only 32% on countertops. That is a 66-point difference within one model. By comparison, the gap between the best and worst model is much smaller for several individual object categories. This suggests that improving the way an architectural drawing is presented to a multimodal AI model may be at least as important as switching between models. AI models can count objects better than they can locate them Gemini 3.5 Flash returned 314 cabinet boxes across the dataset. The human markup contains 317 cabinets. At first glance, that sounds extremely accurate. But the cabinet F1 score is only 45%. Knowing that there are approximately 317 cabinets is different from drawing a sufficiently accurate bounding box around each individual cabinet. Gemini Flash's project-level cabinet counts were off by 17% on average, but many of its boxes were shifted or oversized and therefore failed the IoU 0.50 requirement. What happens if we relax the localization requirement? At an IoU threshold of 0.20 instead of 0.50: Gemini 3.5 Flash increases from 53% to 76% overall Claude Opus 5 increases from 40% to 59% Claude Sonnet 5 increases from 21% to 42% The same six models at IoU 0.20 instead of 0.50, overall and by object type. Callouts gain the most: Gemini 3.5 Flash goes from 0.33 to 0.73. Callouts show the largest improvement. Their small size means that even a few pixels of positional drift can substantially reduce IoU. This distinction matters for AI-assisted estimating. If the task is "How many cabinets are on this page?", the models may be more useful than the headline F1 score suggests. If the task is "Show me exactly where every cabinet is so an estimator can trust the result without checking the sheet", the current results are much weaker. We use IoU 0.50 as our primary benchmark because a loose bounding box has limited practical value when a person must verify the result on the drawing. How do different AI models fail on architectural drawings? The overall F1 score hides an important difference: models can fail by missing objects or by inventing objects. Those failures create different amounts of work for a human estimator. Claude Opus tends to over-detect Across the dataset, Claude Opus produced 1,405 callout boxes for 557 actual callouts. At a lower overlap threshold, it covers approximately three quarters of the true callouts, but only around one in ten of its predicted boxes is a callout. The remaining predictions include other circles, tags, and bubbles on the sheet. For a human reviewer, an Opus-style result can require significant cleanup. Gemini Pro tends to skip objects Gemini 3.1 Pro Preview has a different failure pattern. On elevations, when it draws a box, it is right 94% of the time, but it only draws boxes for 67% of the elevations that exist. The output is relatively precise but incomplete. For an estimator, the work is going back to the drawing to find what the AI missed. Grok misses many objects Grok 4.6 finds only 26% of all objects in this benchmark. Its performance is particularly weak on callouts, cabinets and countertops. Qwen finds more, at the cost of extra boxes Qwen3.8-Max finds approximately 71% of objects at 55% precision. It sees more of the drawing than the other models, but it also generates more boxes that require checking. Two AI models can have similar headline accuracy but create very different workloads for an estimator. One model might produce "Here are 20 objects. I am confident about most of them." Another might produce "Here are 40 possible objects. You need to delete the wrong ones." The best model for an estimating workflow therefore depends on F1 and on how the model's errors affect human review time. Drawing complexity matters as much as the AI model AI accuracy also changes substantially from project to project. Qwen3.8-Max ranges from 44% on the densest project to 88% on two lighter projects. Gemini 3.5 Flash ranges from 26% to 85%. The densest project contains 323 objects and 103 elevations. Every model performs worst on that project. Conversely, every model performs well on the project containing only floor plans and elevations. An AI model's benchmark score should not be interpreted as a universal accuracy number. A model's performance depends on the drawing set, object types, page density, image quality and task definition. What is the fastest and cheapest AI model for architectural drawings? We also measured model response time and cost per page over the 119-page run. AI modelCost per pageTime per pageGemini 3.5 Flash$0.03520 sGemini 3.1 Pro Preview$0.04023 sQwen3.8-Max$0.057161 sClaude Sonnet 5$0.07360 sClaude Opus 5$0.13448 sGrok 4.6$0.145362 sCosts and response times are averages over the 119-page run via OpenRouter. Pages were processed sequentially. Cost varies by a factor of approximately four, from $0.035 to $0.145 per page. Response time varies by a factor of approximately eighteen. The two most expensive models, Grok and Opus, rank fourth and fifth in overall accuracy. Qwen3.8-Max, the highest-scoring model, sits in the middle on cost but is the second slowest at approximately 2.7 minutes per page. For a 100-page drawing set, that means approximately: Gemini: $3–4 Qwen: $6 Opus or Grok: $13–14 Qwen's processing time also adds up: approximately four and a half hours for 100 pages when pages are run one after another. Gemini 3.5 Flash is an interesting alternative if cost and speed matter more than the highest benchmark score. It scores 53% overall versus Qwen's 62%. The bottom line The first run gives a fairly clear picture of where multimodal AI stands on architectural drawings in August 2026. Qwen3.8-Max is the strongest overall model in this benchmark at 62% F1. But object type matters more than model choice. AI performs very well on large visual objects such as floor plans and elevations. Performance falls significantly for cabinets and becomes poor for small callouts and thin countertops. The models also show a distinction between counting an object and locating it precisely. A model can produce a cabinet count that is close to the human count while still failing to place sufficiently accurate boxes around individual cabinets. And the drawing itself matters. Dense pages reduce performance across models. For an estimator, the current practical use of general-purpose multimodal AI is closer to "use AI to navigate the drawing set, identify relevant views, and create a starting point that a human estimator verifies" than to "upload the drawing set and let AI do the entire takeoff". That is already useful, and it is a different thing from autonomous estimating. It is also the gap a purpose-built system closes: at the largest U.S. casework factory, that work went from six hours to ten minutes. What comes next: structured processing Everything measured above uses a simple workflow: one full drawing page in, one answer out. Part 1 described a second approach, structured processing, where the task is broken into steps: Find the interior elevations. Crop the relevant elevation. Analyze the cropped elevation at higher effective resolution. Identify the cabinets, countertops and other objects inside it. Our early runs using this approach perform substantially better than the direct full-page results reported here. The likely reason is resolution: a cropped elevation can reach the model at several times the effective resolution of the same elevation embedded in a complete sheet. We will publish those results once the structured-processing benchmark has been verified across the full dataset. The dataset is also growing, and models will be re-run as new versions become available. If you want the method behind the scoring rather than the results, our LLM evaluation work covers how these ground-truth datasets and automated runs are built. --- ### How Accurate Is AI at Reading Architectural Drawings? A Benchmark of 6 AI Models. Part 1 URL: https://coxit.co/blog/ai-architectural-drawings-benchmark-part-1/ Author: Volodymyr Hresko Published: 2026-08-20 | Updated: 2026-08-20 Category: AI for Drawings Current multimodal AI models can identify many woodwork-related objects on architectural drawings, but accuracy varies substantially by object type, drawing resolution, and workflow. Large, clearly defined objects such as elevation views are easier to detect than small symbols and callouts. Our early testing also suggests that the way the drawing is processed and prompted can matter as much as the choice of model itself. Can AI Read Architectural Drawings? What We Tested Can AI detect objects correctly on architectural and construction drawings? If you follow any discussions on Reddit, you may have noticed that most public answers to this question fall into two camps. Vendors say the technology is ready and everything works. Skeptics point out that models miscount objects on a drawing and conclude the technology is not there yet. So we decided to do the research ourselves. Our team has experience training models to recognise and count casework for millwork takeoff, so we had a practical reason to know the answer. In this study, woodwork-related objects refers to architectural casework: cabinets, countertops, interior elevations, and elevation callouts. These are the objects an estimator needs to find and count when preparing a casework estimate. The benchmark is not intended to rank every AI model on the market. We selected multimodal models that are widely available and relevant to image and document analysis at the time of writing: GPT Claude Fable Claude Opus Gemini Pro Gemini Flash Grok Why it's hard for AI to work with drawings Accuracy on drawings is not one number. It depends on at least three things. First, the type of object. Large objects like elevation views are relatively easy to find. Small reference markers, like elevation callouts, are much harder. A model can be good at one and poor at the other on the same page. Second, the drawing itself. Resolution matters more than we expected. Construction drawings are dense, and their meaning lives in thin lines. A sheet designed for 24 x 36 inch printing may contain details that shrink to a few pixels when the whole page is converted into a single image for a model. When that happens, the model is not failing to understand architecture. The information has effectively disappeared before the model ever sees it. Different models have very different input limits here, and this alone changes the results. Third, and this was the biggest finding of our early experiments, the way you use the model. If you send a full page and ask the model to find everything, you get one result. If you break the task into steps, for example first locating the elevation views and then searching for objects within each one, you get a very different result. Same model, same drawing, different approach. The gap between these two results is large enough that any benchmark that ignores it tells only half of the story. How we approach our research We collect drawings from real projects of different types and scales. The dataset keeps growing, and we will publish its exact size, including projects, pages, and annotated objects, together with the results in part 2. An expert annotator on our team marks every object on these drawings by hand. Cabinets, countertops, elevations, and callouts. These expert annotations become the ground truth. Ground truth means the reference answer we compare every model against: a complete, human-verified map of every object on the page. Then we run the same drawings through each AI model and compare its answers against the ground truth. A model gets credit when its answer overlaps with what the expert marked. We track accuracy separately for each object type, and we record the cost and speed of every run, because in practice these matter as much as accuracy. We test models in two modes. Direct, where the model receives a full page and a question. Structured, where the same model works inside a pipeline. A structured pipeline breaks the task into steps: first locate the elevation views, then search for objects within each one. This is the pipeline that has performed best in our testing so far, though we are still experimenting with the exact steps and the final structured mode may evolve before part 2. Comparing these two modes is the part we find most interesting, because it separates what models can do on their own from what they can do when applied carefully. The early findings The dataset is still growing, so we are not publishing final numbers yet. But a few things are already clear from the first runs. Resolution limits are a serious constraint. Some models accept detailed images, others compress them heavily. On photos this is a minor detail. On drawings it decides whether the model sees the lines at all. Small objects are the hardest part. Finding an elevation view is a task most models handle. Finding a small callout marker among hundreds of similar-looking symbols is a different level of difficulty. And the approach matters more than the model. The difference between sending a full page and using a structured pipeline is larger than the difference between competing models. To us, this is the most useful insight so far. The question "which model is best" turns out to be less important than "how do you apply it". Can AI replace estimators? Based on our early results, we don't think object recognition alone is enough to replace an estimator. Detecting 18 cabinets on a drawing is only one part of estimating. A quantity takeoff workflow also requires interpreting specifications, resolving ambiguities, understanding scope, reconciling different drawing views, and deciding what should be included in the estimate. Our research therefore asks a narrower question first: how reliably can AI extract structured information from drawings? If that layer becomes dependable enough, it can automate parts of casework estimating without replacing the estimator's judgment. What comes next In part 2, we will publish the numbers. A comparison table with accuracy by model, object type, and mode, together with cost and speed per page, and a detailed methodology page. When a new model is released, we will run it through the benchmark and update the results. We are also exploring ways to let you see these results on your own drawings, with detected objects marked directly on the page. More on that in part 2. We will publish part 2 with the numbers soon. Subscribe to get it first. --- ### July 2026: Woodworking Innovations in Italy, a 40 Under 40 Award, and a New Client URL: https://coxit.co/blog/woodworking-innovations-italy-40-under-40-major-new-client/ Author: Volodymyr Hresko Published: 2026-07-15 | Updated: 2026-08-14 Category: COXIT Newsletter My first time at XYLEXPO I've heard plenty of good things about European woodworking machinery over the years, and many of our US clients travel to the EU specifically to find new equipment. So when XYLEXPO came around, I decided it was time to see it for myself. It is hard to overstate how much of the serious manufacturing hardware comes from Europe, especially Germany and Italy. Germany brings brands like Homag, Weinig, and Festool that everyone in the industry knows by name. Italy has Biesse and SCM Group, which dominate in CNC routers and full production lines. If you work in millwork or joinery and want to see where the industry is heading, this is the place to be. Seeing these massive machines run live, instead of just reading about them, is a completely different experience. Out of everything there, the SCM Group booth was the one that really stuck with me. I also discovered a new exhibition I hadn't heard of before: Intermob Istanbul, a furniture and wood tech fair in Turkey happening this September. Perhaps someone from our team will make the trip there. The legacy desktop software challenge One thing that surprised me at XYLEXPO was how much of the software around these machines still lives on desktop. Part of that is fair — machine control has to run locally. No shop floor should stop cutting because the internet dropped. But the business software around the machines is a different story. Estimating, quoting, project data, reporting: a lot of it is still locked inside desktop apps on legacy Windows machines. That's the part that blocks everything modern — no AI on top, no integrations, no access from outside the office. We've helped clients through this exact transition, rebuilding a legacy on-prem system in the cloud for one, and turning a desktop-era estimating workflow into a cloud web app for a US casework manufacturer. The mistake is treating it as one big rewrite. Staged, each step ships something the team uses right away, the old system keeps running underneath, and payback starts well before the migration is done. Wood Industry 40 Under 40 I'll admit I had to read the email twice — I've been named one of this year's Wood Industry 40 Under 40. A huge thank you to Curtis Garrard for the nomination. For a bit of context, the Wood Industry 40 Under 40 is an annual award by the Woodworking Network. It honours young professionals in North American wood products manufacturing who are making a real impact on their teams, their communities, and the industry as a whole. This year marks its 11th anniversary. See you in Atlanta at the International Woodworking Fair, where I look forward to meeting the other nominees. New client: a leading multi-family flooring contractor I am thrilled to welcome a new client: a leading US multi-family flooring contractor. I wish I could tell you more, but we appreciate our client's confidentiality. Nonetheless, it has been fascinating learning how their team operates. They specialise in make-ready flooring for multi-family housing, which means they handle the fast turnaround required to prepare an apartment between tenants. When someone moves out, the unit needs to be ready for the next resident immediately. We are currently running a PoC to see how AI and machine learning can speed up flooring quotes directly from floor plans. It is a task that made our AI engineers' eyes light up, because it's a bit of a challenge. In this sector, professional architectural drawings are often missing entirely, so we are training the system to work with the marketing floor plans found on apartment listing sites — often 3D renderings with no visible measurements. I will keep you posted on our progress with this one. Meanwhile, in real life I took my first actual days off this year. It is funny how AI was supposed to take everyone's jobs, yet my workload has only multiplied. I started the trip in Switzerland, and a few days of quiet in the mountains turned out to be the perfect reset. Stepping away into that calm helps you see the big picture instead of just the immediate tasks. While there, I also held a few productive meetings with partners before heading into the energy of Milan for XYLEXPO. It was a brief but incredibly rewarding trip. This issue first went out to subscribers of our LinkedIn newsletter, AI Era Development Stories, on July 15, 2026. --- ### 6 Questions About AI Takeoffs for Casework Shops URL: https://coxit.co/blog/6-questions-about-ai-takeoffs-for-casework-shops/ Author: Volodymyr Hresko Published: 2026-06-22 | Updated: 2026-08-19 Category: AI for Drawings You're about to spend significant money and management attention on an AI system. Whether it succeeds or becomes expensive technical debt depends almost entirely on what you nail down in the first few meetings. Here are six questions that expose which vendors have built systems designed to work over time and which ones are hoping you won't look too closely. The questions below are designed to guide those early meetings. They are not traps to catch anyone out. Instead, they are benchmarks you want both sides to agree on before any work begins. 1. What accuracy will you guarantee? This question filters out half the vendors in five minutes. An honest answer requires specifics (like 85% or 90%) measured against a defined test set. Anything vague is a red flag. Watch out for phrases like "very accurate," "industry-leading," or even "around 90%." None of those mean anything without a clear measurement method. "Around 90%" is particularly risky because without a baseline, it is just a number generated for the meeting. An experienced vendor will give you a range, anchor it to a specific measurement method, and explain how that number is validated. Look for commitments like this: "We aim for 85–90% accuracy on your specific data, measured against your test set, with the live dashboard ready by week eight." That is a team you can sign a contract with. 2. How will we measure accuracy? Committing to a number is only the first step. You also need to know how that number tracks once the project goes live. If a vendor says, "you'll see it works," keep looking. That is a feeling, not a metric. When something goes wrong six months down the line, you will have no data to pinpoint where the system is failing. A reliable measurement framework relies on three separate elements: A "ground truth" baseline: A file of real examples paired with their correct results to test the AI against regularly. Error categorization: A system that sorts mistakes by type (such as missed data, misidentifications, or mixed-up categories) rather than just labeling them "wrong." Historical tracking: A continuous record to show whether accuracy is steady, climbing, or slipping over time. Without these three pieces, you are managing by guesswork. 3. What's included in your evaluation pipeline? This is the question most vendors hope you skip. The evaluation pipeline is simply the system built to check, track, and improve the AI after launch. Without it, a model might perform perfectly on day one but quietly degrade a few months later without anyone noticing. If a vendor treats this setup as optional or an expensive upsell, they likely plan to launch and walk away. That works for a demo, but not for a system your business will rely on for years. A robust pipeline always covers four core areas: the ground truth file, a live health dashboard, a feedback loop for users to flag mistakes, and safety checks to ensure new updates do not break features that already work. If a vendor cannot describe these four pieces concretely, they probably have not built them. 4. How do we improve accuracy after launch? You are not just buying an AI model as it performs on day one. You are investing in a system that needs to adapt over the next two years. Whether it improves or quietly degrades depends on the post-launch plan. Be cautious if the answer sounds like: "Submit a support ticket and we'll take a look." That is just a service contract. Every single tweak will turn into a billable scope discussion. Eventually, you will stop asking, the system will plateau, and you will have to settle for mediocre results. Instead, look for a built-in feedback cycle. Users flag errors directly in the software. Those flagged cases automatically feed back into the training data. The model retrains, and automated tests verify the change fixed the issue without causing new bugs. That is a system designed to scale efficiently. 5. Can we see accuracy metrics in real time? You need direct access to these metrics. This isn't about checking the numbers every morning; it is about transparency. The moment you lose independent visibility, you only know what the vendor chooses to tell you. AI accuracy fluctuates. New clients might upload documents in an unexpected format, or new product lines might shift the data baseline. If you only discover a drop in accuracy during a quarterly review, that issue has already cost you money for three months. Insist on a shared dashboard from the start. Ask to see a sample version before signing. If it does not exist yet, secure a clear timeline for when it will be delivered and exactly what metrics it will track. 6. What happens when accuracy drops? Here is the reality: accuracy always drops eventually. A client will use non-standard templates, an edge case will slip through, or real-world conditions will shift. The goal is to ensure the vendor has anticipated this breakdown. "That shouldn't happen" is a hope, not a strategy. A mature response balances three operational steps: automated alerts that trigger the moment accuracy dips below your agreed threshold, a clear diagnostic process to find the root cause, and a pre-budgeted retraining workflow. With these three safeguards, a drop in accuracy is a minor adjustment. Without them, it is a business crisis. The pattern behind the questions All six questions point to a single core requirement: measurable accountability. A vendor who provides clear, structured answers across these areas might not be the cheapest or the fastest option, but they are the partner whose system will still be driving value a year after launch. COXIT builds these evaluation systems as part of every AI project, and reviews existing ones through an independent AI audit. See how the questions play out in practice in our AI for manufacturing work. --- ### COXIT Co-Founder Volodymyr Hresko Named to Woodworking Network’s 2026 Wood Industry 40 Under 40 URL: https://coxit.co/blog/coxit-co-founder-volodymyr-hresko-named-to-woodworking-networks-2026-wood-industry-40-under-40/ Author: Iryna Mykytyn Published: 2026-06-16 | Updated: 2026-08-14 Category: COXIT Newsletter The custom AI developer agency is recognized for bringing computer vision and AI agents to document- and drawing-heavy manufacturing workflows in the wood products industry. COXIT, a custom artificial intelligence and machine learning developer agency for manufacturers, announced today that Co-Founder and Chief Operating Officer Volodymyr Hresko has been named to Woodworking Network's Wood Industry 40 Under 40 Class of 2026. The annual program honors leaders under the age of 40 who are shaping the future of North American wood products manufacturing through innovation, leadership, and industry impact. Now in its 11th year, the Wood Industry 40 Under 40 received close to 100 submissions across cabinetry, furniture, closets and home organization, architectural woodwork, retail fixtures, wood components, and educational institutions. Honorees are selected by Woodworking Network's team of editors, and the 2026 class joins more than 400 alumni recognized over the past decade. Why Volodymyr Hresko was recognized COXIT builds custom AI systems that combine computer vision, large language models, and AI agents to process the complex documents and technical drawings that manufacturing runs on. In woodworking and architectural millwork, those workflows include estimating, takeoffs, and reading dense specification books, tasks that have historically required hours of manual interpretation by skilled staff. As Co-Founder and COO, Hresko has helped lead COXIT's work with manufacturers in the sector. The company developed a Production Takeoff Tool and a Spec Book AI Agent for Stevens Industries, a wood products manufacturer, automating parts of the estimating and document-review process that previously slowed production. COXIT is an Amazon Web Services partner and a supplier member of the Architectural Woodwork Institute (AWI), and it has published research on AI adoption in the woodworking industry. Industry impact The recognition points to a broader shift in a traditionally hands-on industry. By applying AI to the paperwork and drawings that surround every manufactured product, COXIT helps shops reduce manual effort, shorten quoting cycles, and free experienced employees to focus on higher-value work. Hresko's selection reflects how technology partners, not only manufacturers themselves, are helping modernize wood products manufacturing across North America. What's next Volodymyr Hresko and the rest of the 2026 class will be recognized at the Woodworking Network Leadership Reception on August 25, 2026, from 5:30 to 7:30 p.m. at the Georgia World Congress Center in Atlanta, following the opening day of the International Woodworking Fair (IWF). Honorees will also be featured in the October Leadership issue of FDMC magazine. The 40 Under 40 Class of 2026 is sponsored by PaintLine and Pennsylvania Lumbermens Mutual Insurance Co. (PLM). About COXIT COXIT is a custom AI and machine learning development company serving manufacturers and technology organizations. Its engineer-led team builds production systems using computer vision, large language models, and AI agents for document- and drawing-heavy workflows, with clients including Stevens Industries, Internet Archive, and others. COXIT operates from Casper, Wyoming, and Lviv, Ukraine. Learn more at coxit.co. --- ### From 6 Hours to 10 Minutes: How AI Transformed Estimating at the Largest U.S. Casework Factory URL: https://coxit.co/blog/ai-cabinet-estimating-takeoff/ Author: Volodymyr Hresko Published: 2026-06-08 | Updated: 2026-08-19 Category: AI for Drawings One of  largest U.S. manufacturer of commercial cabinetry operates a highly automated factory: CNC machines, automated material handling, and robotics. One stage of the workflow was slowing down the overall process - estimating. Each estimate took the team two to six hours, most of it spent manually counting cabinets on construction drawings. Given that the company handles hundreds of thousands of cabinets annually, the team wanted to speed up and automate this step. Here's how the new system works. Their estimating team spent two to six hours on every quote. With hundreds of thousands of cabinets shipped each year, that adds up fast.  Here's how we got it down to under ten minutes. The slowest step in a fast factory Quote requests came in as large architectual drawings. The estimator's job sounds simple: read the drawings, count the cabinets by type, enter the numbers into the quoting system, send the bid. In practice, every step was manual. Cabinet symbols varied between architects. Drawings had to be reviewed page by page, with relevant sections highlighted by hand. Counts were tallied manually. Then the data was entered into the quoting system. Why this needed custom AI? They had already tried off-the-shelf takeoff software, but none of it fit, because the work is really three jobs: reading drawings, searching spec documents, and feeding clean data into the ERP. So the project needed a custom build. Three AI types, working as one system 1. Computer Vision for reading the drawings We trained a custom vision model on more than 10,000 labeled cabinet examples. It detects each cabinet, classifies it by type, counts them, and marks up the drawing with annotations the estimator can review at a glance. Different architects draw cabinets in subtly different ways. A model trained on real, varied examples picks up those patterns instead of breaking on them. 2. LLM for searching the spec documents Each project comes with a 200-page spec document detailing the architect's requirements for materials, hardware, finishes, and dozens of other details. When an estimator needed a specific answer from it, they used to spend 30+ minutes searching. We built a tool that lets them ask a question and get the answer in under 30 seconds, with the exact page reference. Once the AI counted the cabinets and pulled the answers from the specs, the results still had to land in the right place: the manufacturer's internal ordering system, in their format, using their internal product codes.  3. Custom integration for fitting their workflow We connected the AI directly to their system, so the estimator's spreadsheet fills itself. How accuracy improved over time The AI's accuracy didn't grow smoothly. It moved in jumps, and each jump came from fixing a specific thing. We started at 76%.  Looking at the data, we realized the problem wasn't the AI. Three different people had labeled our training examples slightly differently, so the AI was learning the inconsistency, not the cabinets. We re-labeled everything with one expert and clear definitions.  Accuracy jumped to 82%. Next, we started tracking exactly what kinds of mistakes the AI was making. The most common one: confusing elevation views with plan views. We added more training examples specifically for that case. Accuracy went to 87%. By weeks 7–10, we had a proper system in place for spotting errors and improving the model systematically. Accuracy reached 90%. Could we have pushed for 97%? Yes. It would have taken another six to eight weeks for diminishing returns. At 94%, the estimator reviews the AI's output in 10 minutes instead of doing the work from scratch in four hours. That math wins. Going further would have cost weeks for marginal gain. The results, in numbers Time per estimate: 2–6 hours → under 10 minutes Accuracy: 90%, measured against a ground-truth test set ROI: break-even in under one month Headcount: no layoffs. The estimating team didn't get smaller — they got 10–20x more output per person Scalability: the system keeps improving as estimators flag edge cases. When a new architect started using non-standard symbols, the estimator flagged 20 examples, the system retrained overnight, and it now handles that architect's drawings automatically What this means for your operation The headline numbers: 2–6 hours per quote down to under 10 minutes. Accuracy at 94%, measured against ground truth. Break-even in under a month. The estimators didn't shrink in number — their role shifted from counting symbols to reviewing the system's output and making judgment calls. If your team is doing significant document work by hand ( drawings, specs, quote prep) there's almost certainly an AI-assisted version of that workflow that's already feasible. The hard part isn't the AI. It's mapping your workflow precisely enough that the AI can be built around it. The same approach is described in more depth on our AI for manufacturing page. --- ### June 2026: How We Built a Multi-Agent Investment Research Platform URL: https://coxit.co/blog/how-we-built-multi-agent-investment-research-platform-strem/ Author: Volodymyr Hresko Published: 2026-06-04 | Updated: 2026-08-14 Category: COXIT Newsletter A few months ago, two people reached out to me on Upwork with a pretty bold idea. Torrence Jennette, who's a pro in the investment world, and Mallory Musante, an expert in marketing analytics, noticed something that most people in finance just accept. They realised that the way research is done hasn't really changed in years, and it's holding people back. Right now, analysts at mid-sized funds are so buried in paperwork that they often miss great opportunities. There just aren't enough hours in the day to manually dig through every filing and find those hidden gems. Two months invested in audit and research One thing I'm really proud of is that we didn't rush to code anything right away. Honestly, the biggest risk in tech isn't that you can't build something — it's building something nobody actually needs. We spent two months in the audit phase specifically to make sure the development budget was invested where it creates value for the client. By understanding how funds operate, we realised that for an analyst to win, they don't need a generic AI. They need a system that mimics a high-performing research team. Luckily, Torrence and Mallory could share this exact knowledge, and our task was to make a context-aware AI system out of it. That's when we decided to build a team of specialised AI agents: one for news, one for financials, one for valuations, all managed by an orchestrator. We worked in two-week cycles, providing regular demos and gathering feedback. We linked four data sources, covering over 3,000 US companies — more than any analyst could cover manually. Next, we made natural language queries return real answers, added alerts, and built the rest of the functionality that matters to users. I believe the key feature we worked on is coverage. Rather than processing isolated queries, the platform acts as a context-aware intelligence system. Analysts don't start from scratch each time; they build on the knowledge they've already accumulated. What Strem looks like today Fast forward to now: Strem by BRDGE Insights is live, monitoring 3,000+ US securities 24/7. While an analyst might spend weeks digging through dry filings to find one hidden gem, the platform now handles that heavy lifting. It filters the noise so the investment team can focus straight on high-value insights. The core mission behind Strem is to provide institutional-grade firepower. It allows mid-sized funds to compete with giants without the need for a massive research staff. With the foundation built, the project is already expanding into global markets and alternative data to catch market shifts before they even hit the news. The case studies We've written up the development process in two parts. The first covers the audit phase and the creation of the initial prototype. The second walks through the key features and explains how the platform accumulates and uses context. Something not about AI This year I joined a community of people passionate about business development, called Sales Label. I saw that Anton Fedulov, one of the founders, is actively developing this space and constantly shares insights that actually have value. Since we were both living in Lisbon and on the same wavelength, I reached out to grab a coffee. We met up, and then for our next meeting we ended up doing a hike in Sintra. We spent the whole time talking about the housing market and what it's like living in Portugal — basically the standard expat starter pack of topics. It's cool how a simple message can turn into a genuine connection and a great Saturday afternoon. Catch you in the next one, Volodymyr This issue first went out to subscribers of our LinkedIn newsletter, AI Era Development Stories, on June 4, 2026. --- ### March 2026: AI Audit Launching, Swiss Market Research, Canaries Workation URL: https://coxit.co/blog/2026-ai-audit-swiss-market-research-canaries-workation/ Author: Volodymyr Hresko Published: 2026-03-11 | Updated: 2026-09-18 Category: COXIT Newsletter Hey there, 2025 flew by. We grew from 35 to nearly 45 people and took on some technically complex projects, like a financial analytics system that took six months just to get the data models working right. We learned a lot about what we can actually build. So here's what happened last year and where we're heading now. Our biggest clients in 2025 This year we kept building with some great companies. Three stand out. Xperi Inc. — a 3,000-person automotive and media company. We have 11 engineers working with them, and the partnership keeps growing. Stevens Industries, Inc. — a 70-year-old manufacturing company with over 1,000 employees. 13 of our team members are helping them build ML and AI-powered estimation tools that turned hours of manual work into minutes. Internet Archive — a digital library archiving websites, books, and media. We've got 3 engineers supporting them. We're grateful for these partnerships, and we're planning to grow alongside them. AI in woodworking: what we learned We surveyed 40 woodworking professionals about AI adoption in their industry. The results? Half use digital tools: software for design, estimating, and project management. But only 1 in 5 have actually started using AI. There's a big gap between "we're digitised" and "we're using AI to automate work." The top challenge remains the same: estimating and takeoffs — figuring out how much material a project needs and what it'll cost. Over a third of respondents said it's their biggest time drain. And when we asked what they want to learn about AI, 60% said: can AI actually read architectural drawings and do accurate takeoffs? The answer is yes. We've built systems — like the one for Stevens Industries — that take estimation time from 2 to 6 hours down to under 10 minutes. One shop from our survey is already doing this: they're using AI to read architectural prints and pull out specific information, saving 30 to 60 minutes per estimate. It works. But they still won't use it on their biggest projects. Why? They're not confident it's accurate enough to trust with a $500K bid. If you want to dig deeper, you can read the full survey results here. Heading to Switzerland in 2026 Four projects came from Switzerland in 2025. Two things stood out. First, all of them wanted AI integrated into existing systems, not new products built from scratch. Companies with established CRMs and ERP platforms realise their products feel outdated without AI features, but their teams lack the expertise to build those integrations. We added AI engineers to their teams to bridge that gap. Second, one project involved training a model specifically for Swiss German. General LLMs handle Swiss German poorly: it's not a dialect they were heavily trained on. So we're training a custom model on Swiss German datasets for language recognition. General purpose LLMs won't get to these local dialects quickly, so specialised models still matter. So I'm booking a business trip to Switzerland. I'll research the market, meet existing and potential partners, and attend CloudFest, one of the largest cloud events in Europe, held not far from Switzerland in Germany's Europa-Park. Also, I've been told that the chocolate in Switzerland is pretty good. If you're going to CloudFest — I'll be there for the third time — let's talk. AI Audit: a new service in 2026 Vibecoding is having a moment, and for good reason. You can go from idea to working prototype incredibly fast. Perfect for testing concepts and getting buy-in. But prototypes and production systems are built differently. A demo handles 10 users perfectly. Production needs to handle 10,000, and that's when you discover your database can't scale, your prompts are inconsistent, and nobody documented how anything works. That's when we usually hear from them: "How do we make this production ready?" "Should we rebuild or can we scale what we have?" So we're launching a new service: AI Audit. Our AI architect and head of delivery review what's been built, identify what's salvageable, and create a roadmap for turning a prototype into a real system. Top request in 2026: AI integration Hard to miss how much demand there is right now for integrating AI into existing systems. Not building from scratch, just adding AI features where users already expect them. Your customers are using ChatGPT. They're using Claude. They expect software to have AI capabilities now. If your product doesn't, it feels outdated. We're helping companies bridge that gap, bringing AI into legacy systems without ripping everything out and starting over. Workation in the Canaries Most of my time I work from my home office in Lisbon. There's so much work that I rarely make it out of the house during the week. One way I deal with burnout and that stuck feeling: workations. A few times a year, when I realise I'm hitting a wall, I book a ticket and work from somewhere else. In February, it was the Canaries. I spent the week working and thinking through our direction for 2026 — planned conference visits, started mapping out a US trip for the autumn. I was there four years ago with friends but never drove around the whole island. This time I did, took the whole day, stopped at different viewpoints. Working from a different place doesn't solve everything. But three decisions I'd been putting off for months became obvious that week. What's next We're growing the team and shifting focus: less outstaff, more outsource. Less "here's an engineer for your team," more "here's the complete solution you actually need." Discovery phases, AI/ML solutions, and helping companies move from "we think we need AI" to "here's exactly what we're building and why" — that's where we're doubling down. That's 2025: three incredible clients, 40 woodworking shops surveyed, four Swiss projects, one new service launching, and a lot of conversations about whether vibecoded prototypes can actually scale. Spoiler: they can't. That's the year. Now let's see what 2026 actually brings. See you around, Volodymyr This issue first went out to subscribers of our LinkedIn newsletter, AI Era Development Stories, on March 11, 2026. --- ### October 2025: AI Engineers Who Don’t Accept Offers, Document Extraction, and LLMs vs SLMs URL: https://coxit.co/blog/ai-engineers-who-dont-accept-offers-document-extraction-llms-vs-slms/ Author: Volodymyr Hresko Published: 2025-10-13 | Updated: 2026-08-14 Category: COXIT Newsletter Hey there, I don't know about your September, but mine flew by. We had a ton of work. It feels like everyone decided to build something at the same time. And you know what comes with a lot of work — failures. Because the only way to avoid failing is to do nothing at all. So yeah, failures happen. I want to tell you about one of ours. I met a lot of AI engineers who don't accept offers Let me start with our biggest challenge this month. We had an open position for an AI Engineer. Over 200 candidates were sourced, and the process looked like this: 10% invited to screening (20 candidates) 80% passed internal screening (16) 50% selected for client interviews (8) 2 offers made 0 accepted Zero. Not a single one. Both offers fell through — one candidate accepted another opportunity, and another suffered an injury right after the interview and couldn't commit to a start date. Our client needed this position filled urgently and wanted to ensure future candidates wouldn't back out after receiving an offer. We couldn't afford to lose more time, so we changed the process. How our process works At COXIT, every outstaffing engagement follows a transparent flow: Recruiter screening. Each candidate goes through a structured interview. We collect background, expectations, and motivation, and record a Loom video so the client can review quickly and make informed decisions. Notion workspace. All profiles, videos, and evaluations are shared in Notion, allowing the client to review, rate, and comment in real time. Client interview → offer → test period. Once approved, we handle all contracting, onboarding, and test period coordination. Normally this results in smooth placements and high acceptance rates. But given the urgency and back-to-back drop-offs, we reinforced the system to make it fail-proof. What we changed Senior Delivery Manager involvement. A Delivery Manager now joins after screening to double-check motivation, confirm timing, and communicate project context directly. Re-verification before every key step. Before each interview and before the offer, we confirm candidate availability, competing offers, and commitment. Real-time feedback loops. Instead of asynchronous email updates, we now hold short alignment calls with the client to adjust the search direction immediately. Backup shortlist. For urgent positions, we maintain a vetted reserve of candidates ready to start if an offer falls through. On the bright side: the team is growing On a much more positive note, we onboarded a fractional CTO, Serhii Herasymov. Serhii has over 15 years in software development, most of it as a Solution Architect and CTO. He's scaled engineering teams from zero to 300+ people and helped companies move from small dev shops to serious technology businesses. His focus is on strategic planning, organizational structure, and making sure business goals actually align with what's technically possible — so we can grow and take on more ambitious projects. We're also close to full AWS certification. I personally earned two AWS certificates this month. It's proof that we're serious about cloud infrastructure and know what we're doing. The questions clients keep asking September was busy partly because we had a lot of discovery calls. Two of them really stood out because they show a pattern we're seeing more and more. First, I talked with a company that participates in government and corporate tenders. Every time they want to bid, they receive massive documents — 50 to 200 pages of requirements, product lists, specifications. Someone on their team has to sit down and manually go through everything to extract what products are needed, how many of each, what sizes, what materials, all the technical details. What they can do instead: use LLMs to read through Word and PDF files automatically and extract all the relevant information into structured tables. Second, a legal team working with contracts and agreements. When contracts get revised, changes are marked — deleted text in red, new clauses highlighted, modifications underlined. Their lawyers have to read through 30, 50, sometimes 100-page documents to find every single change and make sure nothing important slipped through. What they can do: use technology that can read the document and analyse its formatting, detecting colours, highlights, and other visual markers. The system identifies modifications and generates a list of all changes. Different industries, different documents, but the same core problem: people spending hours extracting structured information from messy files. This is where most real AI projects actually happen — solving repetitive problems that waste smart people's time. Yes, LLMs are typically the answer to these questions, but how you use them very much depends on your specific business needs. Which brings me to the article I want to share with you. "Faith in God-like large language models is waning" The Economist recently published an article with that title, and as a business owner myself I completely agree with the reality check. The article makes a good comparison: the early days of ChatGPT felt like the iPhone launch in 2007. Revolutionary. But now? AI advances feel more like incremental phone upgrades. GPT-5 got about the same reaction as the new iPhone 17 — a collective shrug. Two points from the article resonated. One, companies are shifting from "spend whatever it takes" to focusing on actual ROI. As one VC in the article put it: you may need a Boeing 777 to fly from San Francisco to Beijing, but not from San Francisco to Los Angeles. Two, small language models are often more practical than massive LLMs. Your HR chatbot doesn't need to know advanced physics. Many businesses are realising they don't need God-like intelligence. They need something tailored to their specific problem that actually works and doesn't cost a fortune to run. This lines up with what we're seeing in real projects. The hype phase is over. Now it's about building things that make business sense. I also asked our Lead AI Engineer for his perspective on big LLMs: The Economist is right — the "God-like LLM" era is fading. Businesses no longer want magic; they want measurable ROI. Big models got us here, but small, focused ones are what will keep delivering, automating real tasks like document parsing, QA, and data extraction. The shift isn't about smarter models; it's about smarter integration. From my side, I see history repeating itself. Recently, a huge part of IT budgets went into code modernisation — cleaning up legacy systems from the dot-com era when software design wasn't optimal. The same thing is happening now with GenAI. Companies are investing heavily, and while these large LLMs bring real value, they're often overkill for narrow business problems. In a few years, we'll be modernising again, replacing God-like LLMs with smaller, specialised models that do one thing well and fit naturally into business pipelines. That's September: two failed offers, one new CTO, and a dozen companies asking if AI can read their documents. (It can.) Catch you in October, Volodymyr This issue first went out to subscribers of our LinkedIn newsletter, AI Era Development Stories, on October 13, 2025. --- ### Evaluating RAG using LLM-as-a-Judge Independent Evaluation, Automatic Insights Collection URL: https://coxit.co/blog/evaluating-rag-using-llm-as-a-judge-independent-evaluation-automatic-insights-collection/ Author: Valerii Verezhynskyi Published: 2025-06-19 | Updated: 2026-08-19 Category: ML Engineering The entire approach is open-sourced and is accessible as a GitHub repository. Evaluation? Why RAG output still needs checking In my latest article, "Chatting with Large PDFs," I discussed the technical aspects of implementing a RAG for your PDFs to ensure the model has all the necessary knowledge to answer questions about those PDFs, regardless of their size.  From what our experts could tell, RAG performed well; it retrieved the necessary context and generated visually appealing answers, referencing specific paragraphs of the PDFs and quoting and referring to them.  But wait, how well? How would we know which part of the RAG pipeline is struggling? How do we iterate over RAG improvements? How can we determine if one RAG iteration is statistically superior? Evaluation! The one thing that answers those questions I daresay it’s impossible to further develop RAG without answering all the questions above. And only one thing could help with it— its name is “Evaluation.” Nowadays, it’s relatively easy to implement RAG for yourself; perhaps you could even pick a pre-baked MCP for that or develop one using some fancy-schmancy agentic libraries everyone talks about. Whichever approach you choose, and your RAG is up and running, face it: there are no shortcuts for Evaluation.  Yes, I’ve heard of Ragas, a RAG evaluation framework, but I’ve also heard of its low quality and unheard-of greed for tokens. One Reddit user reported that Ragas consumed $500 worth of tokens on a single pass, although he didn’t share how many documents, questions, etc., his eval dataset consisted of; $500 is a lot, nonetheless. Spoiler alert: the approach presented in the article costs approximately $3, assuming the dataset consists of 10 PDFs, each with 10 pages and 19 questions per file. The cost could also be adjusted on demand by choosing different LLMs. With all things said, chances are, there’s a need to develop Evaluation, as said, from scratch. Evaluation of RAG is always case-centric, aligned to what’s expected from RAG, and then comparing “expected” to “real.” Subtract one from another, and you will get a metric. What is our case at COXIT? Imagine a person, not really a computer savant, who was previously instructed how to use AI Chats and is eager to try them on daily tasks. Now, that person uploaded a PDF, a small one, a large one, or a really large one; RAG consumed that PDF, digested it, vectorized it, and indexed it in nearly ~1 minute, depending on document size. Here comes a question from the user: a composite with numerous technical details about processes, measurements, technologies, and manufacturers, all in one single query.  As it’s our case, not somebody else’s, we can assume that an LLM of our choice, given a whole document as a context, will be able to produce a perfect response—the one that the user expects and will be happy with. We assume that because we know the average file in the dataset is 10-30 pages long, it is not large enough to cause the model to forget context. That perfect response—in other words, the “Golden” response—will be our target. The response answers all questions in the user’s query and explicitly states that some other questions cannot be answered—as they aren’t mentioned in the document. Once again, subtracting the Golden Response from the RAG response—comes metric. Evaluation, how? LLM-as-a-Judge explained That “subtraction” produces a target metric that can be achieved in many ways; LLM-as-a-Judge (explored in “Judging LLM-as-a-Judge” and “A Survey on LLM-as-a-Judge”) is among those, and I chose it specifically to evaluate RAG in our case. LLM-as-a-Judge—is a fully automated evaluation technique that uses LLMs to evaluate other LLMs’ responses. Two primary approaches of LLM-as-a-Judge are Comparative Evaluation (Golden Answer vs Predicted Answer) and Independent Evaluation, where both Golden Answers and Independent Answers are evaluated independently. In this article, I will refer to independent evaluation; however, both independent and comparative evaluations are necessary for a well-balanced LLM-as-a-Judge application. Figure 1: LLM-as-Judge variations Golden Answers, Predicted Answers, where do they all come from?  Both of them are derived from questions. And questions are a tricky part of this story. You can have them beforehand, as we did, or you could generate a set of questions using LLM. As for Comparative evaluation and Independent Evaluation, both real-world user-composed and machine-generated questions are needed for one fair and balanced evaluation. However, once again, machine-generated questions won’t be covered or further mentioned in this article, as they require additional significant time spent.  For a combination (Golden Answers, User-composed Questions) within a Single document that could be freely fit into the model’s context, golden answers are swiftly ejected. Simply feed the model of the said document, ask a question, and embed it into a prompt of your choice. You will then receive the said golden answer for each question in your dataset. Token spending is significant in this case; however, you could consider generating golden answers only once per dataset, in which case it won’t be a big deal. Figure 2: Example conversation with Golden Answer generation Interestingly enough, for Predicted Answers—for RAG Answers—things are more straightforward; we should just let the Chat chat with itself, emulating what happens when a real person sends a question in a chat. Along with the system and user message—our question—the chat receives a list of tools it can use to retrieve context. Limited by chat_turns_left, chat can retrieve context freely until it’s ready to answer user questions, but it's out of turns. If out of turns—we just forbid chat from calling any more tools. Rinse and repeat for each file and all the questions. Figure 3: Example conversation with Predicted Answer generation Ultimately, we end up with a set of (Golden Answer, Predicted Answer) for each combination of (Question, PDF) in your dataset. Those are ready to be evaluated! Independent evaluation, step by step In a previous section, I mentioned that although both Comparative and Independent evaluations are suggested for well-rounded LLM-as-a-Judge evaluation, we will only cover Independent evaluation in this article. Remember when I mentioned earlier in a text that in our use case, users provide not only simple questions but also queries that contain multiple questions in their text? For evaluation, we are to evaluate questions that are split from this query. Let’s call those—Split Questions— derivatives of user queries. For the Independent evaluation of each pair of (Golden Answer, Predicted Answer), each one will be evaluated independently: an LLM call with a specialized system prompt receives question text, answer text, and is asked to assign 4 boolean values: is_question_answered, requires_additional_information, is_speculative, is_confident. I believe that asking the model to assign a numeric value from the scale of 0 to 10 is useless, and boolean values will be much more honest. Considering that we operate Split Questions in evaluation, for each user query in the initial evaluation dataset, we receive N evaluation results—N arrays of 4 boolean values. Those 4 boolean values are then transformed into a target boolean value—comprehensive_answer, when the question is answered, doesn’t require additional information, is not speculative, and is confident.  Resulting sets of boolean comprehensive_answer arrays must have the exact dimensions for both Golden and Predicted Answers. Figure 4: System prompt to evaluate Golden / Predicted Answer Evaluation metrics: accuracy, precision, recall Even though we passed the Golden and Predicted answers generation and the LLM-as-a-Judge evaluation, it’s not enough to call it a day. The next step is to collect metrics and decide if they are statistically significant. In our case, I choose the following metrics: Accuracy: Percentage of correct predictions overall Precision: Percentage of true positives among all positive predictions Recall: Percentage of true positives captured from all actual positives F1: Harmonic mean of precision and recall Cohen’s Cappa: Agreement between predictions and gold standard beyond random chance Specifically, because we didn’t have a lot of questions per file—19 user queries that were later transformed into ~67 Split Questions—I needed to add Confidence Intervals for each metric. Confidence intervals are even more critical when looking at metrics on file or question levels instead of an overall level. Said confidence intervals are a fairly easy-to-interpret metric—it means that 95% of values of a metric lie in the following interval: e.g., from 0.2 to 0.6. The first value of that interval also shows a minimal certain performance of an approach. If the lower value of the interval is 0.7 accuracy, it means the approach almost never produces results of lower accuracy than 0.7. Confidence intervals are truly indispensable when comparing one experimental RAG approach to another. They allow us to compare not only one value of another but also the distribution of those values. In other words, if the accuracy of experiment 1 is greater than the accuracy of experiment 2, it wouldn’t suffice to call experiment 1–more successful because if CI shows significant overlap e1: [0.5, 0.6], e2: [0.52, 0.65], it gets clear, that difference in results is rather caused by a random variation then in actual difference in performance.  In the same way, Confidence intervals express when a metric's results are rather random than confident: when the span between lower and upper values of the interval is too big. For accuracy, 0.4 would be enough to “ignore” the results of an experiment. At this point, when metrics are collected—as a main “deliverable” of an experiment, it’s worth mentioning that across all previous steps where LLM calls were present, we were constantly collecting logs of Chat Messages and Responses. Instead of “just” being an addition to debugging, I also found logs to have greater usage. To extract something valuable from logs, you’d need an engineer who needs to dedicate some time. But what if that could be automated? In short, yes, it could and should be automated. Evaluation insights: reading the logs automatically With metrics collected and logs sitting in their pile after the experiment was conducted, it’s all peaceful there. Someone should go check those logs, but machines should work, and humans should rest. I tried feeding my well-structured logs, specifically the logs of the LLM-as-a-Judge evaluator, when it, as you recall, took question and answer text and applied those 4 binary labels into an LLM, grouped by file, and put it into a single model call along with Metrics collected from previous steps and some extra logs. It turns out I shouldn’t check the depth of logs unless I see something unexpected in the Insights output. Without human intervention, without me having to check the logs, the model already handles producing insights into the experiment quite well. So, when my evaluation run ends, instead of going to metrics and then to logs, I look up the Insights summary produced by the LLM and only then check the logs for specific things or cases that the model has pointed out. Those File-level insights are then once again summarized into a human-readable markdown format, collected, and put into a single prompt to deliver high-level, human-comprehendible notes from LLM.  Figure 5: Example Analysis Report, Model Generated Conclusion and Future Thoughts You just witnessed my specifically tailored implementation of LLM-as-a-Judge evaluation. Even though it already delivers positive values, I want to address improvements and advancements to be considered in the approach: introduce Comparative Evaluation alongside Independent Evaluations; in that way, the model would get to compare Golden and Predicted Results directly, which could deliver robust metrics and insights; introduce machine-generated, human-answered questions which would bring more positive diversity to the question set; current independent evaluation does not handle cases positively when question just cannot be answered: Golden Answers are certain, as model seen all the document, while Predicted Answers are way less confident as they think that they failed to extract correct information. When the quirk above is adjusted, and Comparative Evaluation is implemented, that approach becomes solid and will bring positive business value. If you need to know how accurate an LLM system actually is, this is what our LLM evaluation work measures — and what an AI audit reports on. --- ### Chatting with Large PDFs (100–500 Pages): Using RAG with OpenAI Embeddings (Local vs. API) URL: https://coxit.co/blog/chatting-with-large-pdfs-100-500-pages-using-rag-with-openai-embeddings-local-vs-api/ Author: Valerii Verezhynskyi Published: 2025-05-31 | Updated: 2026-08-19 Category: ML Engineering At COXIT, in 2025, we observe a trend when more and more potential customers ask us to implement a way to chat with massive documents or even sets of documents, search information in them, and perform an analysis dynamically. Those documents happen to be in various domains: technical specification, documentation, and legal contracts. How to read this article This article contains a wealth of information, including both practically validated results and insights for further considerations and research. While it’s dense with numbers and domain-specific terms that probably may make it challenging to read on its own. However, its true value emerges when explored alongside with LLM (Large Language Model). So, if you’d like to obtain specific information, create summaries, or better understand technical details, please, do not hesitate to use LLM for that. Motivation and Purpose Having a request from the market, COXIT decided to invest resources to evaluate research and develop a prototype that could’ve been integrated in existing and future projects. COXIT wanted to validate and then evaluate approaches to consciously embed large documents (100–500 pages) into LLM’s context, how to make LLM operate, refer to, quote information from those PDFs, what document extraction and processing pipelines should look like. Hence, I, the author of this article, Valerii, AI Developer, ex-Founding Engineer with a Code Generation Background, was summoned to COXIT to design and implement prototypes that use large language models in their chassis. App’s Architecture Before I began implementing the prototype this article is about, Iryna Mykytyn (CEO of COXIT), Vladyslav (Lead Engineer) and I decided to use the following stack of technologies:  RAG (Retrieval Augmented Generation), MCP (Model Context Protocol) and Function Calling.  All three became merely a standard in AI Chat Applications. Function Calling, for example, is supported not only by bigger models such as GPT-4o, Claude-3.7-Sonnet but also by much smaller models with open weights that could be run locally. If I was to explain to a person who hasn’t had much prior experience working with LLMs, I would define those technologies as: RAG:  Combines generation with a search to provide more accurate, up-to-date, hallucinations-robust answers. MCP: Keeps access to external functions or APIs in a dedicated server with standardized API. Function Calling: Lets the model decide to call external functions or APIs through structured outputs. On a lower level, I choose to use Python in most of the services, which is de-facto a standard language for AI-based applications with minimal technological overhead and dependencies. I keep intentionally avoiding Agentic Frameworks like LangChain, as they show no benefit from a long-term perspective, as they are generally poorly optimized and overbloated. Speaking about App’s Architecture (Figure 1): I designed it to consist of  4 microservices: ChatUI—React-powered WebUI. I declare that the React code is notoriously badly written, as it was mainly gpt-4o and claude-3.7-sonnet that coded this service (Figure 2). Chat Tools Service—Handles completion requests, implements message history limiting, local tools execution and model validation, and files handling, including file stream uploads. Chat Proxy Streamer— re-routes incoming chat completion requests to various LLM Providers e.g. OpenAI, Anthropic, Google using a standardized OpenAI Format, provides authentication, usage stats collecting. Document Search MCP—a dedicated service that is only used to perform operations with documents: extraction, processing, storage, and vector search. Figure 1: App’s Architecture Figure 2: ChatUI’s UI Why do we use RAG? In reality, many of the documents could’ve fit into LLM’s context directly, especially using Llama 4 that is mentioned to have 10M Context Window, or Gemini-2.0-Flash which has 1M Context Window. Which also means, we could fit ~2000 pages of text into 1M Context Window. At first sight, giving an entire document at once seems beneficial over giving a model only small chunks of text, in case of RAG.  Not to mention the price of a request to LLM, its performance: TTFT (time to first token), TPS (tokens per second), and more importantly, accuracy, depend on how many tokens model received on the input.  When a model receives a big continuous corpus of text, or a lot of messages in message history, it tends to become distracted, unfocused, making it prone to miss vital details in a document. This specifically–missing details–we endeavor to avoid. Thus, RAG, its Retrieval, that produces only relevant to query text chunks–is specifically what we needed. Keeping context as low as possible, filled with only what’s needed–is a right way to get a high-quality response. Another valid point of using RAG is to give an ability to chat with documents not only to models with hundreds of billions parameters and hundreds of thousands of tokens in context window, but also to much smaller self-hosted models such as Qwen2.5-72B-Instruct with GPTQ-Int8 quantization, that could easily be run on somebody’s premise or in the cloud using, for example, 2x A6000 GPUs. Having seen a high-level App architecture, let’s descend into the domain of the documents and how they are used to providing additional context for LLMs. Retrieval of Relevant Information from Documents For a Retrieval, which is ‘R’ in RAG, we perform vector search within all chunks from a single document, with an ability to include prefilter arguments to search not in a whole document but in a chapter or a section of a document. That pre-filter can only be accessed by a model in a case when it decides that a user would like to search only in specific parts of the document.  That Retrieval, in our case, produces up to 10 most relevant search results for a given query (Figure 3). Figure 3: Typical message history when RAG tool is called Alas, with no reranking and post-filtration added, often, not all top-10 results would be relevant to a search, top-1, top-3 might be, others might be not.  Therefore, those top-N results should better be post-filtered by an additional not expensive LLM (e.g., Gemini-2.0-Flash) or simply adding a threshold value to a proximity metric. The thing one should consider when embedding RAG in the chat is limiting the number of messages in a message history, as a number of messages could grow indefinitely. Indeed, up-to-date LLMs rapidly increase their token windows; those 128-200k token windows we have now feel really refreshing after having worked with 2-4-8k context windows in the past.  Yet, it does not really mean we have to utilize all of those without having a compelling reason for that. There could be different strategies to limit messages in message history, e.g., prioritizing dropping old Retrieval context messages over dropping other messages–is a good practice.  For limiting messages, I prefer not to use tokenizers to count the number of tokens in each message; in cases like that, I would simply assume that 1 token = 4 chars to maximize performance when accuracy doesn’t really matter. Now, where did those chunks come from?  Extracting Text from Documents When a document is shared with Document Search MCP, it gets extracted: Using regexp and library pymupdf in python, from each page set of paragraphs: blocks of text surrounded by line breaks are extracted. Along with paragraphs comes metadata.  That metadata would normally include a page number, chapters, sections, and coordinates. Those coordinates: 2 pairs of numeric values represent a location of an imaginary box that wraps a paragraph. A box could be highlighted by UI, if needed, to visually communicate to users where those answers came from. Additionally, paragraphs are split into chunks of text 250–1000 tokens (using a linebreak-based strategy) which are passed for vectorization. Embeddings from vectorization are to be used in vector search. However, this approach imposes a serious limitation of being unable to work with raster PDFs e.g., PDFs where text is embedded into an image – scanned documents. Also, coordinate extraction is a weak part of this approach as it relies on regexps and heuristics, so it would only work well for the specific limited set of PDFs it was tested on.  In a real-world production-ready application, it’s vital to use a CV model that would detect boxes of paragraphs, schemas, images, tables and to use an OCR model to extract text values from text-type boxes and technical summarizations from image-type boxes. The proposed pipeline will enable extractor working with any text format, raster PDF or not, simply converting any page of a document into an image, and having a single pipeline for them all. Figure 4: Document’s Lifetime Approaches to work with extracted data from Documents As OpenAI’s Vector Storage API was released shortly after I began developing this prototype, we decided to test it as well. It was interesting to try, as I would rather shove extra complexity on OpenAI’s side, then implement storing, indexing and vector search locally by myself. And theoretically, OpenAI’s API should be able to handle dozens of thousands of files, so there’d be no need to implement a complex architecture locally.  After paragraphs, metadata, and text chunks are extracted from a document, they enter into the Processing Phase, performed by a Thread-Processor.  At the moment, Document Search MCP  utilizes 2 different approaches: Approach 1: Storing, Vectorizing, Searching using OpenAI’s  Files and OpenAI’s Vector Storage APIs. Approach 2: Vectorizing is performed using any 3-rd party Embeddings Model e.g., OpenAI’s text-embedding-3-small, and a vector search is performed locally. Approaches were tested on the same 100-page documents, which contains: ~600 parsed paragraphs or ~44k tokens. Approach 1: OpenAI-based vector search As supposed, this approach brought us merely no extra architectural complexity. Although, that benefit went along with a pretty disappointing, yet expected, processing performance.  For each text-chunk or a paragraph (in case you are OK with OpenAI’s chunking strategies), you’d need to perform 2 requests to upload a paragraph as a file, that’d be 1200 requests for the document above. I achieved sending all 1200 requests in ~140s of real time*, with 10 concurrent requests at a time and 5s of waiting until timeout.  *–10 concurrent requests at a time and 5s of awaiting response until timeout. Enlarged concurrent requests’ bandwidth resulted in an increased number of requests resulting in timeout errors, so I select 10 as an optimal amount. Even if a time span of 140s doesn’t seem too big for a 100-page document, it actually appears to be that on a second thought. This is a service that should be accessible by many users at once, or even multiple apps. A user that uploads a document will happen to have to wait not just 140s for their document to be processed, but also additional 140s for each user who uploaded a file before. In other words, when users A, B, C uploaded 100-page documents in that order at the same time, user C would wait for their document for 140s × 3 = 7 minutes.  Approach 2: local vector search and storage, external embedding requests Funny enough, an approach that is based on storing vectors locally and, consequently, performing vector search also locally, uses only 6* requests to fully process a given document. That's a 200x improvement over the previous approach. Total waiting time for processing a document resulted in 11s – that’s 14x improvement. *– For 100-page document embeddings requests with batch-size = 128 were sent. For documents of other sizes, batch-size should also be adjusted to keep a minimal response time from an external embedding model. E.g., for 400-page document you might try 128 × 4 = 512 batch-size. Time of processing alone had me convinced to declare supremacy of locally-based approach over OpenAI-based approach. Therefore, I would hereinafter suggest that approach over other cloud-based solutions, where a significant architectural overhead is acceptable. For storing and vector search there is an endless list of solutions: pgvector, milvus, qdrant, or milvus-lite for in-memory vector search. Approaches’ Pricing Comparison Pricing of Approach 1: OpenAI-based vector search has 3 components: File Storage: $0.1/Gb per day, 1st Gb is free. Vector Search Calls: $2.5 / 1000 calls Embedding model: $0.02 / 1M tokens for text-embed-3-small. 1Gb of text really is a lot, it’s difficult to exceed, as a 100-page document is ~171kb. That means, there could be ~6132 100-page documents. Thus, pricing would seem like a sweet deal if performance wasn’t that inefficient in our case.  In a case of Approach 2: Local-based vector search, you would just need an embedding model from that list above. Which means, the price of processing a 100-page document would be ~$0.0009. Infrastructure cost, however, could vary widely depending on the project's needs. RAG Results Validation For validation out of all domains we choose a technical specification as we already have a specifically designed set of questions of a group of documents that LLM with RAG should be able to handle on a daily basis. For a set of documents (nearly a dozen) we made LLM compose answers using RAG, after which results were carefully validated by a person.   During a validation phase, we also noticed that our extracting wasn’t working on some documents properly, which disabled the model to produce valid results on a whole set of questions on said documents. For a large majority of documents which didn’t have issues with extracting, the model responded with accurate results in ~80% of all cases. When this prototype is used in real-life projects, we’d have to set up a more precise validation pipeline, taking into account documents’ domain. That pipeline should be centered about the same idea: measure the difference between model response in 2 cases: when the model has all context needed to answer a question e.g., provide a complete document and a 2nd case: when the model will have to use RAG calls to retrieve that context.  Having a set of ground truth answers and set of predicted answers, we could use different metrics to evaluate how close we get with RAG-used answers: e.g., BLEU/ ROUGE metrics or an embedding-based similarity. Human analysis metric could also be important when there’s a person with domain knowledge. In areas where a set of questions will be difficult / unreasonable to create, we could use a model-generated set of questions instead. That set of questions could be as diverse, thematic-rich, large as needed. Additionally, that set of questions could be post-filtered by a technical specialist. Results This article shows step-by-step implementation and concrete engineering insights of an app that would allow chatting with no-matter how big PDFs. The app is designed to follow distributed architecture guidelines to become scalable and resilient in production usage. Along with validated engineering approaches, the article provides approaches, including hardware, ML models, and practical advices, how to embed this application into a production-ready project. We build document-heavy RAG and retrieval systems like this one as custom AI development projects. --- ### How to set up monitoring of CPU and memory usage for C++ multithreaded application with Prometheus, Grafana, and Process Exporter URL: https://coxit.co/blog/how-to-set-up-monitoring-of-cpu-and-memory-usage-for-c-multithreaded-application-with-prometheus-grafana-and-process-exporter/ Author: Iryna Mykytyn Published: 2025-05-23 | Updated: 2026-08-19 Category: ML Engineering Have you ever wondered how to know how much CPU and memory resources is your app taking during a week? Or how to collect and visualize this data to get some useful insights? In this guide, I will show you how to do it with Grafana, Prometheus, and possibly minor updates to your C++ app. 💡 This was tested on Ubuntu 20.04 on x86_64 and aarch64 architectures. Tools Setup Install Prometheus Following Prometheus official setup guide: https://prometheus.io/docs/introduction/first_steps/. wget https://github.com/prometheus/prometheus/releases/download/v2.33.1/prometheus-2.33.1.linux-amd64.tar.gz tar xvfz prometheus-2.33.1.linux-amd64.tar.gz cd prometheus-2.33.1.linux-amd64 ./prometheus You should be able to load Prometheus UI at http://localhost:9090. Install Process Exporter Process-exporter - is a Prometheus exporter that mines /proc to report on selected processes. wget https://github.com/ncabatoff/process-exporter/releases/download/v0.7.10/process-exporter-0.7.10.linux-amd64.tar.gz tar xvzf process-exporter-0.7.10.linux-amd64.tar.gz cd process-exporter-0.7.10 Run process exporter specifying your process name ./process-exporter-0.7.10 -procnames You will see the following printout where you could also have a look at the metrics for your process via http://localhost:9256/metrics. Configure Prometheus to digest metrics from process-exporter Open prometheus.yml in ./prometheus-... root folder. Add process-exporter to scrape configs: global: scrape_interval: 15s scrape_configs: - job_name: "process-exporter" static_configs: - targets: ["localhost:9256"] Restart Prometheus: ./prometehus --config.file="./prometheus.yml" Install Grafana Follow Grafana Install Guide for more information. Or you can also use Grafana Cloud solution, please go here for instructions. sudo apt-get update sudo apt-get install grafana-enterprise sudo systemctl start grafana-server Check out http://localhost:3000. By default credentials are admin:admin credentials. Configure Grafana to use Prometheus as a data source Go to the Data sources in Configuration and add Prometheus as a data source. Set URL for this data source to http://localhost:9090. Watch this guide about Getting started with Grafana for more details. Example of usage Imagine we have a simple C++ application. The app creates 4 threads and printout messages in while loops with some random sleeps. We wanna visualize how much CPU each thread is taking, and how much memory the whole process need for execution over an hour. sample_app.cpp #include #include #include #include void print(int n, const std::string &str){ unsigned run_times = 1000000; while(run_times) { std::string msg = std::to_string(n) + " : " + str; std::cout << msg << std::endl; std::this_thread::sleep_for(std::chrono::microseconds(run_times/100)); --run_times; } } int main(){ std::vector s = { "message 0", "message 1", "message 2", "message 3" }; std::vector threads; for (int i = 0; i < s.size(); i++){ threads.push_back(std::thread(print, i, s[i])); } for(auto &th : threads){ th.join(); } return 0; } Compiled and executed it. We should be able to see namedprocess_namegroup_thread_cpu_seconds_total in Prometheus prompt: Wondering why there is only one thread name, while we were expected to see 4 threads, besides the main process thread? That’s because we need to give a proper name to the threads. Let’s check it out with htop: htop -p $(pgrep -d',' -f ./sample_app) In order to be able to distinguish threads, we need to give them names. I used prctl system function for this. #include void print(int n, const std::string &str){ unsigned run_times = 1000000; std::string thread_name = "thread-" + std::to_string(n); prctl(PR_SET_NAME, thread_name.c_str(), 0, 0, 0); while(run_times) {...} } ... Now it’s better: The next step is to visualize these metrics in Grafana and PromQL (read about PromQL here and here). Explore Go to Explore section in Grafana and run the query rate(namedprocess_namegroup_thread_cpu_seconds_total{mode="system", groupname="sample_app"}[5m]). Observing separate threads may be useful if you wanna see optimisation gups. Visualize other useful data about your process Use predefined Grafana dashboard to visualize CPU, memory usage, read/write bytes of your process: https://grafana.net/dashboards/249. Just import it to Grafana. Don’t know what Resident and Virtual memory mean on this dashboard? In my next article, I will talk about Linux memory, follow me so don’t miss this and other blog posts. COXIT places senior C++ and Python engineers on client teams through team extension. --- ### How To Integrate Valgrind into GitHub Actions? URL: https://coxit.co/blog/how-to-integrate-valgrind-into-github-actions/ Author: Iryna Mykytyn Published: 2025-05-14 | Updated: 2026-08-19 Category: ML Engineering Have you ever got into the situation that after you fixed memory leaks with Valgrind you see Definitely lost again after a while? Then this CI setup may be useful for you. You can do it in 3 steps: Install Valgrind in your CI environment. Run tests with Valgrind and output results to XML. Add script to parse XML and report errors. You may also want to control when to run it. Running with Valgrind may be time-consuming, so you may want to run it only on Pull Requests to the master branch. Or you may want to run it only for some specific tests. These steps are common for every possible CI. Let’s see it on GitHub Actions example. 💡 If you did not work with GitHub Actions before, see Quickstart for GitHub Actions. Add Valgrind install step to yml file - name: Install Valgrind run: | sudo apt install valgrind echo "Valgrind installed" You can run into the issue that Valgrind version is not compatible with your CI system. Then you will need to build and install another version of Valgrind: here is an example of how to do it. Update test’s run script to run with Valgrind - name: Install Valgrind run: | sudo apt install valgrind echo "Valgrind installed" --suppressions=./path/to/suppresion/my_app.supp - this option is needed only if you have suppression file. You can generate suppressions for the errors for the libraries, but it’s not a good practice. It's hard to know if a memory error in the library is caused by a problem in your code or not. But if it is not possible to fix a leak, and you are sure it’s not your leak, then 1) suppress errors from the library (short-term solution); 2) upgrade the library or connect the community who supports it and ask for a fix. Example of the first option. Ignore Leak errors in libcrypto only, you could put a suppression like: { ignore_libcrypto_conditional_jump_errors Memcheck:Leak ... obj:*/libcrypto.so.* } into a file and pass it to valgrind with --suppressions=./path/to/suppresion/my_app.supp. You can also let Valgrind generate suppression file for you. Use -gen-suppressions=yes from Core Command-line Options. --xml=yes --xml-file=unit_tests_valgrind.xml - enable Valgrind’s to report in XML format and save it in unit_tests_valgrind.xml. Parse XML output You can write your own using ValgrindCI tool or write a custom XML parser. ValgrindCI tool - name: Valgrind Memory check run: | pip install ValgrindCI echo "Summary report of errors" valgrind-ci /path/to/output_file.xml --summary valgrind-ci /path/to/output_file.xml --abort-on-errors # abort on errors Custom XML parser in Bash The output XML format is specified in the file docs/internals/xml-output-protocol4.txt in the source tree for Valgrind 3.5.0 or later. It has errorcounts or error tags to report errors: ERROR definition -- common structure ------------------------------------ ERROR defines an error, and is the most complex nonterminal. For all of the tools, the structure is common, and always conforms to the following: HEX64 INT NAME if set KIND ... optionally: SUPPRESSION * Each error contains a unique, arbitrary 64-bit hex number. This is used to refer to the error in ERRORCOUNTS nonterminals (see above). It’s enough to check if /ERROR tag exists to know if at least one error was reported. But it’s more secure to parse all tags and return the exact number of errors and short summary. # Check if the xml has any "" attributes. If it has, memory errors exists. sed 's/"/ /g' < $xml_file | grep '' &> /dev/null if [ $? == 0 ]; then printf "%s" "$boldred" echo "[ FAILED ] Some memory checks failed for $xml_file." error=-1 exit $error; else echo "[ SUCCESS ]" fi yml step: - name: Valgrind Memory check run: | echo "Checking Valgrind xmls ..." ./check-memory.sh It is good to use a custom script if you need to parse a few XML’s and report separately on each. Example of how to control Valgrind run with environment variable RUN_MEMORY_CHECK - environment variable to control when to run Valgrind memory check. In the example below, it will run memory check steps only for main branch or if RUN_MEMORY_CHECK is added to GitHub Actions Secrets and is set to true name: "Build and run tests with memory check" on: pull_request: push: branches: - main - dev env: RUN_MEMORY_CHECK: ${{ (github.base_ref == 'main') || ((secrets.RUN_MEMORY_CHECK == 'true') }} jobs: build: name: "Build" runs-on: ubuntu-latest strategy: fail-fast: false matrix: include: - target: x86_64 steps: - uses: actions/checkout@v2 - name: Set up environment run: | sudo apt update sudo apt install -y libev-dev - name: Install Valgrind if: ${{ env.RUN_MEMORY_CHECK == 'true' }} run: | sudo apt install valgrind echo "Valgrind installed" - name: Build Tests run: | # add build script execution here - name: Run Tests with Valgrind run: | valgrind --leak-check=full --suppressions=./path/to/suppresion/my_app.supp --xml=yes --xml-file=unit_tests_valgrind.xml ./unit_tests_binary - name: Valgrind Memory check if: ${{ env.RUN_MEMORY_CHECK == 'true' }} run: | echo "Checking Valgrind xml ..." ./check-memory.sh CI and memory-safety work like this is typical of the C++ engineers COXIT embeds through team extension. --- ### Differences in Prompting Techniques: Claude vs. GPT URL: https://coxit.co/blog/differences-in-prompting-techniques-claude-vs-gpt/ Author: Yaroslav Biziuk Published: 2025-04-08 | Updated: 2026-08-19 Category: ML Engineering Introduction When working with language models (LLMs) like Claude and GPT, the effectiveness of prompts can vary significantly based on the model's architecture and capabilities. This article explores key differences in prompting strategies for Claude and GPT, using advanced techniques such as meta-prompting, XML tagging, and the Chain of Thoughts (CoT) method. By understanding these differences, users can optimize their prompts to enhance the accuracy and reliability of LLM outputs. Context Window Size and Information Processing One of the most significant differences between Claude and GPT lies in their context window size—the amount of text they can handle in a single prompt. Claude (anthropic/claude-3.5-sonnet) can process up to 200,000 tokens making it ideal for tasks requiring the analysis of large documents or aggregating data from multiple sources. This feature is especially valuable in fields like business and academic research, where Claude can process entire reports or research papers in one go. Because Claude has such a large capacity, it's essential to be clear and explicit in prompts to help the model focus on the relevant parts of the input. GPT-4 (gpt-4o-2024-08-06) supports up to 128,000 tokens, which, while smaller than Claude’s window, still represents a significant improvement from previous models. This makes GPT ideal for tasks involving moderate-length documents or specific, well-defined queries. Our Claude prompts in average have 9000 tokens per input and 600 tokens per output. For GPT it’s 9000 tokens per input and 650 tokens per out, because of different token embeddings  In practice, Claude’s larger context window means prompt engineering requires attention to chunking and summarizing within the input to make the most of its potential, while GPT might be better suited for more focused and precise tasks. Role of Examples and Instructions Both models work better when provided with examples and clear instructions: In Claude, examples play a central role in shaping the model’s response. For complex tasks, offering several examples that showcase the expected response pattern can significantly improve the output. It’s important to ensure that these examples are concise and relevant, avoiding unnecessary complexity. Importantly, examples have to be diverse in order not to tend LLM to the basis. Example prompt with One Shot. We provided our prompt with an example of correctly resolving our Testcase-2 (1100-C).  OUTPUT: Result: LLM chose IPB Core Compliant and Sq Flat Edge, although previously she easily solved this test case (1200-NAUF), which means that one test case can form the basis to which the LLM will tend when making its answer. Therefore, according to the Claude documentation, examples should be diverse. Example prompt with Few Shots. We provided our prompt with an example of correctly resolving our second test case (1100-C) and first (1200-N).  OUTPUT: We asked Claude to evaluate our examples on relevance, diversity and clarity. OUTPUT: Certainly, I'd be happy to evaluate the examples for relevance, diversity, and clarity. Let me analyze each example : Example 1: This example is highly relevant as it demonstrates the process of analyzing a specification and selecting the appropriate Series option. It shows a clear step-by-step thought process, which is excellent for illustrating how to approach the task. The example covers various aspects of cabinet construction, including core material, front and end materials, thickness, and edging. This diversity in the specification details helps to showcase how different elements are considered in the decision-making process. The thinking process is clearly laid out, making it easy to follow the reasoning behind the final selection. The format, with separate and tags, is clear and consistent with the instructions. Example 2: This example is also relevant and provides a different set of specifications to analyze. It adds diversity to the set of examples by presenting a case where No Added Urea Formaldehyde (NAUF) core is specified, which wasn't a factor in the first example. The specification in this example is more detailed and covers additional elements like hardware and accessories, countertops, and solid surface materials. This breadth of information tests the ability to focus on the relevant details for cabinet construction while filtering out extraneous information. The thinking process is again clearly laid out, following the same format as the first example. This consistency aids in clarity and understanding. Overall evaluation: 1. Relevance: Both examples are highly relevant to the task of analyzing specifications and selecting the appropriate Series option for cabinet construction. 2. Diversity: The examples provide good diversity in terms of the specifications presented. They cover different materials, construction details, and environmental considerations (NAUF vs. non-NAUF). This diversity helps to illustrate how the selection process can vary based on different requirements. 3. Clarity: Both examples are clear in their presentation. The step-by-step thinking process is well-articulated, making it easy to follow the reasoning. The consistent use of the and tags adds to the clarity. Suggestion for improvement: While these examples are strong, you could consider adding a third example that presents a more challenging or ambiguous case. This could help illustrate how to handle situations where the specification is less clear-cut or where there might be conflicting information to resolve. This would add another layer of complexity and further demonstrate the critical thinking skills required for this task. GPT performs well when given examples to demonstrate expected output, especially in creative or technical contexts. Its ability to generalize from a few examples (i.e., few-shot learning) allows it to produce accurate outputs even from a limited context. Conclusion: We can add as many solved test cases to the prompt as long as it all fits into the context window of LLM, and as research shows it will increase accuracy, but unfortunately it is not practical. Although incorporating nine chain-of-thought resolved results into a prompt can improve accuracy in solving tasks, the cost of 35 cents per call makes this method impractical for production use. These nine test cases, comprising around 100,000 tokens, would fit into the context windows of both GPT (128,000 tokens) and Claude (200,000 tokens). An alternative approach to this problem could be implementing Retrieval-Augmented Generation (RAG) with a few-shot chain of thought. This would allow us to insert only the three most relevant resolved test cases into the prompt based on vector search matching the input specification, optimizing both accuracy and relevance. Unfortunately, implementing RAG would require at least 25 test cases to function effectively, which we currently lack. Additionally, RAG would be time-consuming to implement and challenging to test, despite its potential accuracy benefits. Meta-Prompting Meta-prompting is a technique used to refine and enhance the performance of a prompt. For Claude LLM, meta-prompting can significantly improve prompt effectiveness. The Claude documentation recommends using the  "Generate a Prompt" feature that allows users to quickly create the first prompt or enhance their existing prompts. This functionality supports iterative improvements, helping to fine-tune prompts based on specific needs. In contrast, meta-prompting for GPT has not shown substantial improvements in performance in our tests. Although this may be case-specific, it highlights a fundamental difference in how these models handle prompt enhancements. Claude playground Formatting Prompts The way prompts are formatted and segmented can affect the performance of LLMs. Claude’s documentation recommends using XML tags to separate parts of the prompt template or large prompts. This method has proven effective in practice, improving the results significantly. For GPT, it is advised to specify the delimiter being used, such as “You will be provided with a pair of articles (delimited with XML tags).” This indicates a fundamental difference in how Claude and GPT handle prompt formatting.м Results of correct formatting for Claude are described here, where meta-prompting made correct formatting. Example for Claude:    [OPTIONS]    [INSTRUCTION]    [INPUT_TEXT] Example for GPT: OPTIONS: {OPTIONS} INSTRUCTION: {INSTRUCTION} SPECIFICATION: {INPUT_TEXT} OR You will be provided with a instruction, options, specification (delimited with XML tags):    [OPTIONS]    [INSTRUCTION]    [INPUT_TEXT] System and User Input Separation GPT and Claude’s performance benefits from separating system prompts and user inputs, allowing the model to process information more effectively. For instance, the system prompt can be set up with specific instructions, while the user input follows separately. In contrast, separating system and user inputs for GPT often results in diminished performance. GPT tends to handle tasks more efficiently when all instructions are provided in a single system message. Example for Claude: Example for Claude: messages=[     {         "role": "system",         "content": prompt_template.format(             OPTIONS=options, INSTRUCTION=instruction,         ),     },     {         "role": "user",         "content": input_text,     }, ], Example for GPT: messages=[     {         "role": "system",         "content": prompt_template.format(             OPTIONS=options, INSTRUCTION=instruction, INPUT_TEXT=input_text         ),     }, ] Additionally, it was observed that when XML tags were used to separate user input for Claude, it actually led to worse results. Specifically, tagging inputs separately (e.g., encapsulating user input within XML tags) made it harder for Claude to generate meaningful responses, likely due to how the model parses structured inputs compared to plain text. This finding highlights the importance of avoiding unnecessary formatting structures like XML tags when separating inputs for Claude. Example Prompt for Claude: You are a Casework expert tasked with reviewing a specification and selecting the correct Series option that represents cabinet materials and thickness. Follow these instructions carefully: 1. Review the available Series options: {OPTIONS} 2. Carefully read and understand the following instructions for selection: {INSTRUCTION} 3. Examine the provided specification: SPECIFICATION_USER_INPUT 4. Think through your decision-making process step by step. Consider the following:    a. Identify key information in the specification related to cabinet materials and thickness.    b. Compare this information to the criteria outlined in the instruction.    c. Match the identified information with the available Series options.    d. Eliminate options that do not meet the criteria.    e. Select the most appropriate Series option based on your analysis. 5. After your analysis, provide your reasoning and selection in the following format:        [Include your step-by-step reasoning here]            RESULT: [Insert selected Series option here]     Remember to base your decision solely on the information provided in the specification, instruction, and available options. Do not make assumptions or introduce external information. Worse results: messages=[ { "role": "system", "content": prompt_template.format( OPTIONS=options, INSTRUCTION=instruction, ), }, { "role": "user", "content": f"{input_text}", }, ], Better results: messages=[ { "role": "system", "content": prompt_template.format( OPTIONS=options, INSTRUCTION=instruction, ), }, { "role": "user", "content": input_text, }, ], Chain of Thoughts (CoT) Technique The Chain of Thoughts (CoT) technique is essential for guiding LLMs through complex instructions by encouraging them to think step-by-step. For Claude, it is recommended to use XML tags to designate where the model should include its reasoning. This structured approach helps Claude organize its thoughts and produce coherent outputs. For GPT, a simpler approach of instructing the model to "think step by step" or "take your time" is often sufficient. This technique encourages GPT to consider each step in the process but does not require the specific tagging used in Claude. Example for Claude:    [Include your step-by-step reasoning here] Output: Let's analyze the specification step-by-step: 1. Core Material Selection:    - The specification mentions "Particleboard: ANSI A208.1, Grade M-2 M-2-Exterior Glue (at wet sink areas)."    - There's no mention of NAUF or urea formaldehyde restrictions.    - Default to Industrial Particleboard CARB Compliant. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  Based on these selections, we can narrow down our options to series that have: - Industrial Particleboard CARB Compliant core - HPL fronts and ends - Square Flat Edge The series that best matches these criteria is: Example for GPT: You think step by step. Take your time to answer. Output: ### Step-by-Step Analysis: #### 1. Material Requirements: - **Core Material**: The specification mentions that adhesives should not contain urea formaldehyde. Therefore, we should select NAUF (No Added Urea Formaldehyde) Particleboard. - **Fronts Material**: The specification requires high-pressure decorative laminate (HPL) for exposed surfaces. - **Fronts Thickness**: The specification mentions GP-50 (0.050-inch nominal thickness) for horizontal surfaces and GP-28 (0.028-inch nominal thickness) for vertical surfaces. However, for the fronts, we will consider the standard thickness options provided in the series. - **Ends Material**: The ends should match the fronts, which is HPL. - **Fronts Edging**: The specification does not allow PVC edging, so we should look for options without PVC edging. - **Cabinet Edgebanding**: The specification does not allow PVC edging, so we should look for options without PVC edging.  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  #### 7. Cabinet Edgebanding Selection: - **Selected Cabinet Edgebanding**: Since PVC edging is not allowed, we should look for options without PVC edging. ### Final Review: - **Core Material**: NAUF PB Core. - **Fronts Material**: 3mm High Pressure Laminate (HPL). - **Ends Material**: 3mm High Pressure Laminate (HPL). - **Fronts Edging**: 3mm Radius Edge. - **Cabinet Edgebanding**: Options without PVC edging. Decomposing Complex Tasks in GPT: A Recommended Approach One of the key recommendations in the GPT documentation is to decompose complex tasks into simpler, more manageable steps. This method significantly enhances performance and reduces the likelihood of the model making mistakes by handling more digestible components of a task one by one. GPT excels when provided with specific instructions in a sequence, allowing it to focus on a single objective at a time. For example, when working on determining the correct Series option that represents cabinet materials and thickness from a PDF document, a sequence of chain-of-calls (or sequential prompts) could simplify the task. This approach reduces cognitive overload for the model, ensuring more accurate and reliable outputs. Here’s how this decomposition would look: First prompt: Extract all relevant information about cabinet materials and thickness from the provided document. This step isolates data extraction from interpretation, ensuring GPT focuses solely on identifying and retrieving the relevant details. Second prompt: Based on the extracted information, determine the correct Series option that represents the cabinet materials and thickness using the provided instructions. Here, the model is now tasked with processing the extracted data, interpreting it according to specific criteria, and generating a decision. By breaking down the task into these two steps, we reduce the complexity for the model, minimizing the chances of misinterpretation or errors. This method also makes it easier to troubleshoot and refine individual steps, should the need arise. Example on Claude: Example GPT: In general, this strategy of decomposing tasks into smaller, sequential prompts is especially useful in GPT’s chain of thought framework, where the model benefits from clear and distinct steps. When a PDF file is loaded into GPT for processing, the pricing is calculated based on the total word count of the entire PDF, which can become expensive, especially with large documents. But in my opinion, this method is one of the best ways to minimize hallucinations, and make the model's responses more deterministic by extracting relevant sections before submitting them to another prompt to GPT. This strategy allows for a focused query, ensuring the model processes only the necessary information, leading to more reliable answers. However, in our specific test case, it might be more cost-efficient to develop an algorithm that automatically identifies and extracts only "Part 2 products," instead of processing the entire document with GPT, making it a cheaper and more targeted solution. Roleplay Models vs. GPT: Following Instructions and Logical Compliance Roleplay models like Claude are more rigid in following instructions, making them ideal for tasks requiring strict adherence to step-by-step guidelines. In contrast, GPT models often rely on their own internal logic, which can lead to creative and effective problem-solving but may deviate from precise instructions. This flexibility is beneficial for open-ended tasks but requires additional review when strict accuracy is needed. From our experience with Claude and GPT on similar prompts, we observe that Claude tends to follow a more structured, step-by-step approach to instructions, while GPT’s outputs often involve more improvisation, making it necessary to monitor tasks for precise compliance. TESTCASE-3 (2800-C 3mm Frts & TF Color Throughout IPB CARB Compliant Core) GPT OUTPUT: Let's follow the steps outlined in the instructions to determine the correct series option. ### Step 1: Material Requirements - The specification mentions that the basis for the specification is Stevens Advantage 2800 Designer Series. ### Step 2: Core Material Selection - The core material should be Industrial Particleboard CARB Compliant unless adhesives with urea formaldehyde are prohibited. - There is no mention of prohibiting adhesives with urea formaldehyde, so we will use Industrial Particleboard CARB Compliant. ### Step 3: Fronts Material Selection - The specification states that door and drawer fronts must be matching high pressure laminate (HPL) on both sides. - Therefore, the fronts material should be High Pressure Laminate (HPL). ### Step 4: Fronts Thickness Determination - The specification states that door and drawer fronts shall be 3/4" thick. - The standard thickness for HPL fronts is 3mm or Square Flat Edge. ### Step 5: Ends Material Selection - The specification allows for either color matching thermofused or matching HPL for the rest of the exposed/semi-exposed cabinet components. - Since the basis for the specification is Stevens Advantage 2800 Designer Series, which uses Thermally Fused Laminate (TFL) for color matching, we will select TFL for the ends. ### Step 6: Fronts Edging Selection - The specification states that door and drawer fronts shall have 3mm radius extrusion banding. - Therefore, the edging style should be 3mm Radius Edge. ### Step 7: Cabinet Edgebanding Selection - The specification states that cabinet sides, top, bottom, adjustable shelves, and other interior components shall be edged with (.020") flat edge extrusion. - Therefore, the cabinet edgebanding style should be Flat (.020") PVC Cabinet Edgebanding. ### Step 8: Final Review - Ensure all selections are compatible with each other. - Verify that all requirements from the original specification have been addressed. Based on the above steps, the correct series option is: **RESULT: 2800-C - 3mm Frts & TF Color Throughout IPB CARB Compliant Core** Claude OUTPUT: (link here) Let's go through this step-by-step: 1. Material Requirements:    The specification mentions color matching thermofused or high pressure laminate for exposed/semi-exposed surfaces. Door and drawer fronts must be high pressure laminate on both sides. 2. Core Material Selection:    The specification mentions industrial grade particleboard, but doesn't explicitly state NAUF. Therefore, we'll default to Industrial Particleboard CARB Compliant. 3. Fronts Material Selection:    Door and drawer fronts are specified as high pressure laminate (HPL) on both sides. 4. Fronts Thickness Determination:    The specification states that door and drawer fronts shall be 3/4" thick. 5. Ends Material Selection:    Exposed/semi-exposed surfaces can be either color matching thermofused or high pressure laminate. 6. Fronts Edging Selection:    The specification states "Edges shall have 3mm radius extrusion banding." 7. Cabinet Edgebanding Selection:    Cabinet edges are specified as (.020") flat edge extrusion. 8. Interior Material Selection:    Non-exposed interior surfaces shall be thermofused melamine laminate. 9. Final Review:    Based on the above, we need a series with:    - Industrial Particleboard CARB Compliant core    - HPL fronts    - 3mm radius edge on fronts    - Either HPL or TFL ends (TFL is more cost-effective)    - 3/4" thickness The series that best matches these requirements is: RESULT: 1200-C - 3mm HPL Frts & HPL Ends IPB CARB Compliant Core This series provides: - High Pressure Laminate fronts and ends - Industrial Particleboard CARB Compliant core - 3mm Radius Edge style - Flat (.020") PVC Cabinet Edgebanding While it doesn't explicitly mention the 3/4" thickness, this is standard for this type of construction. The HPL ends meet the requirement, though TFL ends would also have been acceptable if available in a matching series. Here GPT’s outputs involved more improvisation, looking through documentations, it incorrectly determined the HPL for fronts material, but chose 2800-C, because found the sentence “the basis for the specification is Stevens Advantage 2800 Designer Series”. Claude adheres to the instructions more strictly and rarely uses such tricks. It determined that fronts is HPL and draws conclusions based solely on the instructions we gave it. Conclusions The differences between prompt engineering for Claude and GPT are nuanced and center around formatting and step-by-step guidance, structuring, rather than the overall content of the prompts. Claude, with its ability to handle larger context windows, excels in tasks requiring the processing of extensive documents. It benefits from the use of separated system and user inputs, as well as XML-based formatting, which can help structure the conversation effectively. GPT, on the other hand, is more streamlined and performs optimally with a single, well-structured prompt, without the need for separated formatting. Even though GPT and Claude have some structural differences, we found that the same core instructions worked for both models with only minor adjustments. What initially seemed like a significant difference in prompt construction eventually converged, as identical instructions worked effectively for both. In conclusion, while the formatting and handling of step-by-step instructions may vary between Claude and GPT, the fundamental principles of prompt engineering remain largely consistent across both models. By understanding and leveraging each model's unique strengths, it is possible to achieve high-quality results regardless of the choice between Claude and GPT. References Claude Prompting Guide GPT Prompting Guide Prompting decisions like these are part of the custom AI systems COXIT builds for clients. --- ### March 2025: How COXIT Transformed Estimation at Stevens Industries URL: https://coxit.co/blog/how-coxit-transformed-estimation-at-stevens-industries/ Author: Volodymyr Hresko Published: 2025-03-18 | Updated: 2026-09-18 Category: COXIT Newsletter Our work with Stevens Industries, Inc. started in 2023 with one question: could AI actually read architectural drawings? After a deep dive into their processes, we built a proof of concept — and it worked. That PoC turned into our biggest development project to date, and we are still coming up with new ideas alongside Stevens' Continuous Improvement Manager, Ausby Curtis Garrard, and his team. Who are Stevens Industries? Stevens Industries is the largest manufacturer of commercial casework and architectural millwork in the United States. In 2015 the company was recognized as one of the fastest growing wood manufacturers in the country. Last summer I got to visit their factory in person. Seeing their production in action — machines, materials, and people working together — was incredible. It made me appreciate just how much precision and craftsmanship goes into their work. So let's hear it from Curtis himself: how AI changed their estimation process, what they have learned, and what is next for the industry. Meet Curtis Garrard, Continuous Improvement Manager at Stevens Curtis is always driving innovation and trying to solve complicated business challenges. We have been working alongside him since 2023, and I can say he is one of the best professionals I know — always looking forward and planning big. Today I am giving him this space to share his AI development story. Before implementing AI/ML, what challenges did you face with your estimation process? Curtis: Like many trades and crafts within our industry, the role of an estimator is one that is built on knowledge and experience of what it takes to do the work they are responsible for estimating. Because of this experience, the learning curve is quite long for people just entering the role. Additionally, in the case of our specific products where there is no straightforward way to automate all of the rules and logic, that experience is heavily relied on to produce custom level quotations for our customers. Because of these facts, we face the constant challenge of supply and demand to grow our business. In order to grow our business, we must estimate more, and this is hard to do when the talent required to supply the needed demand is not easy to find. What made you decide it was time for an AI-powered solution? Curtis: The goals of the business began to transform where we realized that if we wanted to estimate more to support a goal of greater sales, we had to figure out how to streamline workflows and look for outside the box solutions that might help us. Thinking outside of the box is where an AI solution began to come into mind. After researching what was already available to the market, we found that there were tools for other crafts that already existed, but nothing that quite fit our use case. We knew then that the technology could help solve our problem, we just needed to find a way to create it since it wasn't readily available. How did the transition from manual to AI-powered estimation work? Curtis: It transitioned slowly, and is still an ongoing transition. Changing the day-to-day processes of experienced team members is not an easy one, especially when they may not have the same technical understanding of what is possible with an AI/ML solution. It has taken a lot of teaching and showing to help establish the truth that utilizing an ML driven solution leads to a more efficient solution. What results have you seen since adopting AI/ML? Curtis: There have been several results that I would highlight. Most notably, we began to fix our issue where we can now get through our estimating process more efficiently and therefore can support goals for higher sales with our existing staff. In some cases where the pre-estimating process could take between 2–3 hours, we now see it being completed in about 10 minutes by utilizing our new technology. Indirectly, we've begun to benefit from a more technologically knowledgeable team as well. Because of the teaching required to help our team members see the vision we were working towards, they've begun to understand what capabilities actually exist and are now actually suggesting improvements to make our new solution even better. How has this impacted the business overall? Curtis: I would say we are still in the early adoption phases of the new technology, so the actual impact we've seen so far is still a small sample size. But overall, we've seen an increase in our pre-estimating and estimating processing time. As we continue through the adoption phase and actually improve the ML side of our tool by continued use, this should help us to be able to increase our overall estimating capacity without having to hire additional resources. This impacts our business by allowing us to serve more customers without having to add cost into the business to do so. What advice would you give to other manufacturers considering AI/ML? Curtis: We are in an age of technology where if you can think it, you can do it. Don't be afraid to challenge your own assumptions about how you do work. This will help you explore thought processes like: "Is this the best way to do this?" It all starts with the ideas that are generated from that thought experiment. If the idea is good enough, execution of the idea can be planned. But before jumping into the execution of the idea, involve your stakeholders early and often to help scope out your requirements and ideas as much as possible. This does several things: establishes buy-in to the idea before actually starting, helps fill in possible missing gaps in the current idea that should be considered, and helps lead to a more creative and complete scope of work for your project. The output to this will be a well thought out idea that can be directly transferred to an executable path. As Thomas Edison once said, "Having a vision for what you want is not enough. Ideas without execution are hallucinations." The work behind this The Stevens engagement is written up in three parts: the proof of concept that answered the "can AI read a drawing?" question, the Production Takeoff Tool that came out of it, and the Spec Book AI Agent that reads the specification books around each job. I wouldn't find a better way to finish this edition. As you know, I am always open to discussing new ideas and finding a way AI can optimize your business operations. Just get in touch and we can set up a quick meeting. All the best, Volodymyr This issue first went out to subscribers of our LinkedIn newsletter, AI Era Development Stories, on March 18, 2025. --- ### Experiments with different LMMs URL: https://coxit.co/blog/experiments-with-different-lmms/ Author: Yaroslav Biziuk Published: 2024-12-26 | Updated: 2026-08-19 Category: ML Engineering MythoMax 13B (nitro) Code: gryphe/mythomax-l2-13b:nitro Description: One of the highest performing and most popular fine-tunes of Llama 2 13B, with rich descriptions and roleplay.  GPT prompt output: Claude prompt output: Result: Tried to answer in few sentence, without Chain of Thoughts. MythoMax has very bad results.  Reflection 70B (free) Code: mattshumer/reflection-70b:free Description: Reflection Llama-3.1 70B is trained with a new technique called Reflection-Tuning that teaches a LLM to detect mistakes in its reasoning and correct course. The model was trained on synthetic data. GPT prompt output: TESTCASE - 3 OUTPUT: To solve this problem, I'll follow these steps: Identify key information in the specification related to cabinet materials and thickness. Compare this information to the criteria outlined in the instruction. Match the identified information with the available Series options. Eliminate options that do not meet the criteria. Select the most appropriate Series option based on the analysis. Let's go through this process: Key information from the specification: Cabinet ends: 3/4" thick Cabinet backs: 1/4" prefinished Medium Density Fiberboard (MDF) Adjustable shelves: 3/4" thick Drawer members: 3/4" thick Drawer bottoms: 1/4" MDF Comparing to the instruction criteria: The instruction asks for a Series option that represents cabinet materials and thickness. We need to focus on the thickness of the cabinet ends and the material of the cabinet backs. Matching with available Series options: Looking at the options, we can see that most of them have: Ends: High Pressure Laminate (HPL) or Thermally Fused Laminate (TF) Core: Industrial Particleboard (IPB) or No Added Urea Formaldehyde Particleboard (NAUF PB) Edgebanding: Flat (.020") PVC Cabinet Edgebanding None of the options explicitly mention 3/4" thickness for ends or 1/4" MDF for backs. However, we need to find the closest match. Eliminating options: We can eliminate options that don't match the general criteria: Series 1600-C and 1600-N can be eliminated as they have wood veneer fronts, which is not mentioned in the specification. Series 1100-C, 1100-N, 1150-C, and 1150-N can be eliminated as they have square flat edge HPL fronts, which doesn't match the 3mm radius edge mentioned in the specification. Selecting the most appropriate option: After elimination, we're left with Series 1200-C, 1200-N, 1250-C, and 1250-N. These all have 3mm HPL fronts and 3mm radius edges, which seems to be the closest match to the specification. Among these, Series 1200-C and 1200-N have HPL ends, while Series 1250-C and 1250-N have TF ends. The specification doesn't explicitly mention the type of ends, so we can't make a distinction based on this. The main difference between these options is the core material: Series 1200-C and 1250-C have Industrial Particleboard (IPB) CARB Compliant Core Series 1200-N and 1250-N have No Added Urea Formaldehyde Particleboard (NAUF PB) Core The specification doesn't mention CARB compliance or NAUF, so we can't make a decision based on this. Upon reflection, I realize that I may have been too quick to eliminate the 1600 series options. While they do have wood veneer fronts, the specification doesn't explicitly rule out wood veneer. Additionally, these options have 3mm wood edges, which could be considered similar to the 3mm radius edge mentioned in the specification. Let's reconsider these options. Reconsidering the 1600 series options: Series 1600-C and 1600-N have wood veneer fronts with 3mm wood edges, which could be seen as similar to the 3mm radius edge mentioned in the specification. They also have HPL ends, which is consistent with the specification's mention of 3/4" thick ends (though the exact thickness isn't specified in the options). The core material is IPB CARB Compliant Core for 1600-C and NAUF PB Core for 1600-N, which is similar to the other options. Given this new consideration, Series 1600-C and 1600-N seem to be strong contenders. They match the 3mm edge requirement and have HPL ends, which is closer to the 3/4" thickness mentioned in the specification than the other options. After this reflection, I believe Series 1600-C or 1600-N would be the best choice. They match the 3mm edge requirement and have HPL ends, which is closer to the 3/4" thickness mentioned in the specification than the other options. The wood veneer fronts could be seen as a minor deviation from the exact specification, but it's the closest match overall. Based on the analysis of the specification and the available Series options, the most appropriate Series option that represents cabinet materials and thickness is either Series 1600-C or Series 1600-N. These options stand out for several reasons: Edge style: Both Series 1600-C and 1600-N have a 3mm radius wood edge, which closely matches the 3mm radius edge mentioned in the specification. End material: They have HPL (High Pressure Laminate) ends, which is consistent with the specification's mention of 3/4" thick ends. While the exact thickness isn't specified in the options, HPL ends are a closer match to the 3/4" thickness requirement than the other options. Core material: Both options have a particleboard core, with Series 1600-C having an Industrial Particleboard (IPB) CARB Compliant Core and Series 1600-N having a No Added Urea Formaldehyde Particleboard (NAUF PB) Core. This is similar to the core materials mentioned in the other options. Overall construction: The wood veneer fronts with 3mm wood edges in these options could be seen as a minor deviation from the exact specification, but they provide a strong and durable construction that aligns with the overall quality implied in the specification. While the other options (Series 1200-C, 1200-N, 1250-C, and 1250-N) have 3mm HPL fronts and 3mm radius edges, they don't match the end thickness requirement as closely as the 1600 series options. The 1100 and 1150 series options can be eliminated due to their square flat edge HPL fronts, which don't match the 3mm radius edge mentioned in the specification. Therefore, between Series 1600-C and 1600-N, the choice would depend on whether CARB compliance (1600-C) or NAUF (1600-N) is preferred or required for the project. Both options represent the cabinet materials and thickness most closely based on the provided specification and available Series options. Expected RESPONSE: 2800-C - 3mm Frts & TF Color Throughout IPB CARB Compliant Core Result:  Reflection LLMs aim to explain each step of the instructions comprehensively but face significant challenges in executing nearly every step. This difficulty might stem from the specific synthetic data on which the model was trained. Despite its structured approach, the model often fails to accurately address key details outlined in the instructions. Notably, the model exhibits high determinism, consistently providing the same result across multiple prompts with zero temperature.  Claude prompt output: TESTCASE - 1 OUTPUT: To solve this problem, I'll follow these steps: Analyze the specification to identify key information about cabinet materials and thickness. Compare this information with the criteria in the instruction. Match the identified information with the available Series options. Eliminate options that don't meet the criteria. Select the most appropriate Series option. Let's go through this process: Analyzing the specification: Cabinet construction: Particleboard, plywood, hardwood lumber, or high-density fiber board. Thickness: Minimum 1/2-inch for sides, back, and sub-front of drawers. Bottoms: Minimum 1/4-inch tempered hardboard. Drawer bottoms: Minimum 1/2-inch thick particleboard. Comparing with the instruction criteria: The instruction asks for a Series option that represents cabinet materials and thickness. We need to find an option that matches the materials and thicknesses specified. Matching with available Series options: The options mention various materials like HPL (High Pressure Laminate), TF (Thermally Fused Laminate), and wood veneer. Thicknesses mentioned include 3mm (which is approximately 1/8 inch) and no specific thickness for wood veneer. Eliminating options: Options with 3mm thickness (1100-C, 1100-N, 1150-C, 1150-N, 1200-C, 1200-N, 1250-C, 1250-N) can be eliminated as they don't meet the minimum 1/2-inch thickness requirement. Options with HPL or TF for all parts (1100-C, 1100-N, 1150-C, 1150-N, 1200-C, 1200-N, 1250-C, 1250-N) can be eliminated as they don't match the variety of materials specified in the instruction. Selecting the most appropriate option: The only option that remains is 1600-C or 1600-N, as they are the only ones that mention wood veneer and have a 3mm wood edge, which could potentially meet the thickness requirement. Upon reflection, I realize that I may have been too hasty in eliminating the 3mm options. The specification mentions a minimum 1/2-inch thickness for some parts, but not all. The 3mm thickness could potentially apply to certain components. However, the main issue is that none of the 3mm options mention the variety of materials specified in the instruction. The 1600-C and 1600-N options are still the most likely candidates as they mention wood veneer and have a 3mm wood edge, which could meet the thickness requirement for some parts. After careful consideration, the most appropriate Series option appears to be either 1600-C or 1600-N. Both of these options mention wood veneer with a 3mm wood edge, which could potentially meet the thickness requirement for some parts of the cabinet. They also mention HPL for the ends, which aligns with the variety of materials specified in the instruction. Based on the analysis of the specification and the available Series options, the most appropriate Series option that represents cabinet materials and thickness is either 1600-C or 1600-N. These options are the best match for several reasons: Material variety: They mention wood veneer for the fronts, which aligns with the specification's mention of various materials including wood veneer. Thickness: While the specification mentions a minimum 1/2-inch thickness for some parts, the 3mm wood edge mentioned in these options could potentially meet this requirement for certain components. Core material: Both options mention Industrial Particleboard CARB Compliant Core (1600-C) or No Added Urea Formaldehyde Particleboard (1600-N), which aligns with the specification's mention of particleboard as a possible material. Edge style: The 3mm Radius Wood Edge mentioned in these options could potentially meet the thickness requirement for some parts of the cabinet. Overall construction: These options seem to offer a more comprehensive construction method that could accommodate the various materials and thicknesses specified in the instruction. It's important to note that while these options are the best match, they don't perfectly align with all aspects of the specification. The specification mentions a minimum 1/2-inch thickness for some parts, which is not explicitly stated in these options. However, given the available choices, 1600-C or 1600-N appear to be the most suitable options for representing cabinet materials and thickness as specified in the instruction. Expected RESPONSE: 1200-N - 3mm HPL Frts & HPL Ends NAUF PB Core Result:  Reflection LLM follows the prompt structure closely, striving to execute every step of the provided instructions and guidelines. However, it often struggles with selecting the right keywords and sentences from specification and has difficulty identifying the key elements to focus on when analyzing the cabinet specification. Meta: Llama 3.1 8B Instruct (free) Code: meta-llama/llama-3.1-8b-instruct:free Description: Meta's latest class of model (Llama 3.1) launched with a variety of sizes & flavors. This 8B instruct-tuned version is fast and efficient. It has demonstrated strong performance compared to leading closed-source models in human evaluations. GPT prompt output: Claude prompt output: TESTCASE - 8 OUTPUT: To follow the instructions and pick the correct series option that represents the cabinets' materials and thickness, I need to identify the key characteristics mentioned in the specification. From section "A. 3/4-inch panel construction" and "B. Cabinet Frame Construction", I understand that the cabinets have: 3/4-inch thickness for base cabinets, including ends, bottoms, and top and back frames Dowel pin construction for stability No separate attached bases Integrated design with back groove and precision CNC drill pattern for accurate location of fixed members, hardware, and shelf supports From section "C. Back Procurement of Materials", I see that the specific details mentioned include: Backs shall be an integrated system of a 1/4-inch prefinished Medium Density Fiberboard (MDF) captured in side and horizontal grooves Unit back to be further integrated with attachment to 3/4-inch doweled-in mounting frames Fixed backs are mechanically fastened into grooves and sealed with hot melt adhesive Considering the specification requirements, I will look for the series that matches these conditions. The options that best match these specifications are: Series 1200-C - 3mm HPL Frts & HPL Ends IPB CARB Compliant Core (does not match the required 3/4-inch thickness) Series 1600-C - Wood Veneer Frts 3mm Wood Edge & HPL Ends IPB CARB Compliant Core (does not match the required 3/4-inch thickness) After analyzing the options and understanding that series 1100 and 1150 are plain 3/4-inch Flat Edge (does not match the acceptance of dowel pin construction) and series 1200 and 1250 provide 3mm thickness (does not match the required 3/4-inch thickness), so the series must be at least 3/4 inch However, upon closer inspection of the specification for backs, an important aspect is that the backs have a prefinished 1/4-inch MDF. However, there is also an important aspect of dowel pin construction that requires the integral design (series:*. The 1600 series provides a 3mm Wood Veneer and integrated back design. However, from the specified options none match fully. series "1150-C" option shares the integral back feature and the dowell pin construction and 3/4 thickness. ANSWER: series "1150-C" Expected RESPONSE: 1250-C - 3mm HPL Frts & TF Ends IPB CARB Compliant Core Result:  The LLaMA 3.1 8B model tends to provide very brief and short responses and struggles with effectively utilizing the "chain of thought" technique, often making non-transparent decisions that can be difficult to follow. Also LlaMA 3.1 8B has very low determinism and very often writes different solutions for the same test case. Meta: Llama 3.1 405B Instruct Description: The highly anticipated 400B class of Llama3 is here! Clocking in at 128k context with impressive eval scores, the Meta AI team continues to push the frontier of open-source LLMs. Meta's latest class of model (Llama 3.1) launched with a variety of sizes & flavors. This 405B instruct-tuned version is optimized for high quality dialogue use cases. Code: meta-llama/llama-3.1-405b-instruct GPT prompt output: Claude prompt output: Result:  The Meta: LLaMA 3.1 405B Instruct model exhibited a well-structured approach to problem-solving and achieved a decent percentage of correctly executed test cases. However, its success rate was notably lower compared to models like GPT and Claude, which outperformed LLaMA in terms of accuracy and handling complex, multi-step tasks. Despite this, LLaMA still demonstrated promise in maintaining logical consistency and structure in its responses. Anthropic: Claude 3.5 Sonnet (self-moderated) Code: anthropic/claude-3.5-sonnet:beta Claude prompt output: TESTCASE - 8 OUTPUT of Claude-3-opus: 1. Core Material Selection: The specification mentions particleboard core materials with a minimum density of 45-48 lb, tested under ANSI A 208.1 and/or ASTM D-1037-91A standards. It does not explicitly prohibit adhesives containing urea formaldehyde. Therefore, I will select Industrial Particleboard CARB Compliant as the core material. 2. Fronts Material Selection: The specification states that door and drawer fronts must be matching high pressure laminate on both sides. Therefore, I will select 3mm High Pressure Laminate as the fronts material. 3. Fronts Thickness Determination: The specification mentions that door and drawer fronts shall be 3/4" thick. This aligns with the standard thickness for 3mm High Pressure Laminate fronts. 4. Ends Material Selection: The specification does not explicitly mention the material for cabinet ends. However, it states that all exposed/semi-exposed surfaces shall be color matching thermofused or high pressure laminate. Since the fronts are specified as HPL, I will select High Pressure Laminate as the ends material to maintain consistency. 5. Fronts Edging Selection: The specification clearly states that door and drawer front edges shall have 3mm radius extrusion banding. Therefore, I will select 3mm Radius Edge as the fronts edging style. 6. Cabinet Edgebanding Selection: The specification mentions that cabinet sides, top, bottom, adjustable shelves, and other interior components shall be edged with (.020") flat edge extrusion. This matches the default option of Flat (.020") PVC Cabinet Edgebanding. Based on the provided specification and the selection criteria, the most appropriate Series option is: 5120-C - 3mm HPL Frts & Ends IPB CARB Compliant Core RESULT: 5120-C - 3mm HPL Frts & Ends IPB CARB Compliant Core Pricing for claude-3.5-sonnet (self-moderated the same) Cost for 9 test cases: 0.32 $ Average cost for 1 testcase: 0.036 $ For example pricing for gpt-4o-2024-05-13 Cost for 9 test cases: 0.41 $ Average cost for 1 testcase: 0.046 $ Result: Models such as anthropic/claude-3-opus and anthropic/claude-3-sonnet(self-moderated) also delivered highly accurate results, demonstrating a high level of determinism and a structured approach to executing multi-step complex tasks. Their responses aligned consistently with the expected output format, particularly when generating structured responses typical of sonnet output. This consistency in following intricate steps showcases their capability to handle detailed instructions with precision and reliability. Qwen 2 72B Instruct Code: qwen/qwen-2-72b-instruct GPT prompt output: Claude prompt output: TESTCASE - 3 GPT PROMPT: Justification: 1. Core Material Selection: The specification mentions particleboard core materials with a minimum density of 45-48 lb. Since there's no mention of adhesives containing urea formaldehyde, we default to ""Industrial Particleboard CARB Compliant."" 2. Fronts Material Selection: The specification requires door and drawer fronts to be finished entirely in matching high pressure laminate (HPL) materials on both the fronts and backs. The rest of the exposed/semi-exposed cabinet components can be either color matching thermofused (TFL) or matching HPL. Since the fronts must be HPL, we select ""3mm High Pressure Laminate."" 3. Fronts Thickness Determination: The specification indicates that door and drawer fronts shall be 3/4"" thick. This corresponds to the ""3mm"" thickness option for fronts. 4. Ends Material Selection: The specification allows for either color matching TFL or matching HPL for the rest of the exposed/semi-exposed cabinet components. Since we have already selected HPL for the fronts, we default to HPL for the ends. 5. Fronts Edging Selection: The specification requires 3mm radius extrusion banding for door and drawer fronts. This corresponds to the ""3mm Radius Edge"" style. 6. Cabinet Edgebanding Selection: The specification requires (.020"") flat edge extrusion for cabinet sides, top, bottom, adjustable shelves, and other interior components. This corresponds to the ""Flat (.020"") PVC Cabinet Edgebanding"" option. The ""2800-C - 3mm Frts & TF Color Throughout IPB CARB Compliant Core"" series matches all the requirements specified in the input text. It features a 3mm High Pressure Laminate for fronts, matching High Pressure Laminate for ends, an Industrial Particleboard CARB Compliant core, and a 3mm Radius Edge style for the fronts, with Flat (.020"") PVC Cabinet Edgebanding for the cabinet edges. Pricing for  qwen/qwen-2-72b-instruct Cost for 9 test cases: 0.04 $ Average cost for 1 testcase: 0.0045 $ RESULT:  Performance of Qwen 2 72B Instruct in Task Execution The Qwen 2 72B Instruct model demonstrated impressive results on prompts designed for GPT, successfully completing 8 out of 9 tasks. However, this success was achieved when the outputs were manually reviewed rather than programmatically analyzed, due to the model's poorly formatted responses. Despite the formatting issues, the answers were often correct, which is remarkable. It also exhibited a moderate level of determinism, and the high task completion rate suggests it is well-suited for executing such complex tasks. With a cost of $0.35/M input tokens and $0.4/M output tokens (0.04 $ for 9 test cases), Qwen-2-72B  presents itself as a viable option for further testing, when: gpt-4o-2024-08-06 $2.5/M input and $10/M output (0.21 $ for 9 test cases) gpt-4o-2024-05-13 $5/M input and $10/M output (0.40 $ for 9 test cases) claude-3.5-sonnet $3/M input and $15/M output (0.32 $ for 9 test cases) Model selection like this is part of how we scope custom AI projects. --- ### Minimizing Randomness in GPT Responses: A Guide to Using Seeds, Top_p, and Monitoring System Fingerprints URL: https://coxit.co/blog/minimizing-randomness-in-gpt-responses-a-guide-to-using-seeds-top_p-and-monitoring-system-fingerprints/ Author: Yaroslav Biziuk Published: 2024-12-26 | Updated: 2026-08-19 Category: ML Engineering When working with large language models (LLMs) like GPT, a major challenge is ensuring consistency in responses. These models are inherently stochastic, meaning randomness is embedded in their behavior. However, in certain applications, such as testing or content generation where consistency is crucial, it becomes essential to minimize this variability. In this article, we'll explore how to control randomness using parameters like seeds and top_p, and why monitoring the system_fingerprint parameter is critical. We'll also discuss why achieving full determinism in LLM responses is ultimately impossible. Why Full Determinism Is Impossible While setting a seed and controlling parameters like top_p can help reduce randomness, true determinism in LLM responses is impossible to achieve for several reasons: Non-deterministic Model Behavior: Even when top_p is set to an extremely low value (e.g., top_p = 0.00000001), which allows only the highest-ranking token to be selected, the model can still produce different outputs over time. This is due to underlying non-deterministic faults within the model. On longer text generations, even with such strict top_p settings, the highest-ranked token may switch occasionally due to the complex vector math that happens before token sampling. This behavior comes from the probabilistic nature of neural networks, meaning that token choice can vary despite tight constraints. System Fingerprint Variability: The system_fingerprint parameter provides a unique identifier for the backend system serving the model. However, this fingerprint can change every time you call the model. When the fingerprint changes, it can indicate that the underlying API backend or model architecture has been updated. As a result, even with identical seeds and settings, slight changes in the model's internal mechanics can lead to different outputs. Thus, even with strong controls over randomness, true determinism remains out of reach. Using a Seed to Control Randomness One of the most effective ways to reduce variability is by setting a seed. This ensures that, for a given input, the model produces the same output each time. While this helps create repeatability, it’s important to remember that due to the non-deterministic nature of LLMs, even this won’t guarantee perfect consistency across different API calls or versions of the model (especially if the system_fingerprint changes). Lets test this parameters on our dataset. Call structure for GPT (gpt-4o-2024-05-13): completion = await client.chat.completions.create( model=llm_model, temperature=0, seed=111, messages=[ { "role": "system", "content": prompt_template.format( OPTIONS=options, INSTRUCTION=instruction, INPUT_TEXT=input_text ), }, First call Second call Third call The same call to Claude(claude-3.5-sonnet-2024-06-20) gave three identical response for 3 calls: Another call structure: completion = await client.chat.completions.create( model=llm_model, temperature=0, seed=999, messages=[ { "role": "system", "content": prompt_template.format( OPTIONS=options, INSTRUCTION=instruction )}, { "role": "user", "content": input_text, } First call Second and Third calls Conclusion: As demonstrated by the testing results above, setting the temperature or top_p to zero and utilizing a fixed seed do not guarantee determinism in model responses. The key factor influencing the outcome lies in the vector mathematics employed during the token selection process (espesially in Third Layer of token generation : Beam Search with Randomness). This process of selecting tokens can be compared to Dijkstra’s algorithm. While traditional algorithms like Dijkstra’s evaluate all possible paths to ensure an optimal solution, LLMs do not exhaustively explore every potential token combination. This would indeed lead to significantly slower response times. Instead, LLMs prioritize computational efficiency by selecting highly probable tokens based on contextual patterns, which sometimes results in variability across responses. LLM doesn’t check all possible solutions to find the best one but generates outputs that are likely correct, because of that, we can see different result series to the same input cabinet specification. As a result, the model may produce slightly different answers with each invocation, driven by its focus on optimizing computational resources while balancing accuracy. Controlling Output Diversity with Top_p The top_p parameter controls the probability distribution from which the model selects the next token. Setting top_p to a lower value restricts the model to a smaller pool of possible token choices, narrowing down the output variability. However, lowering top_p too much can lead to unintended consequences: Chain of Thought Resolution Issues: For complex tasks that require multi-step reasoning or explanations (e.g., chain of thought processes), setting top_p too low (e.g., top_p = 0.0001) can drastically degrade the model's ability to think through problems. The lack of diversity in token selection limits the model’s ability to explore nuanced responses, leading to overly deterministic but shallow outputs. Performance Deterioration: Lowering top_p to extreme levels can result in outputs that are either repetitive or overly simplistic, as the model is forced to choose from a very narrow set of tokens. But the top_p parameter, even when set alongside a constant seed for deterministic outputs, does not guarantee consistent results across multiple runs. Below you can see proof of this: Call to Claude(claude-3.5-sonnet-2024-06-20) with this structure: completion = await client.chat.completions.create( model=llm_model, top_p=0.00001, seed=111, messages=[ { "role": "system", "content": prompt_template.format( OPTIONS=options, INSTRUCTION=instruction ), }, { "role": "user", "content": input_text, } First call Second call Third call Temperature and Top_p: Avoid Using Together Both temperature and top_p control the randomness of the model’s responses, but in different ways: Temperature: This parameter controls the randomness in token selection by scaling the probability distribution. Lower values (e.g., temperature = 0.1) make the model more deterministic, while higher values (e.g., temperature = 1.0) increase diversity. Top_p: This parameter limits the set of tokens considered for selection based on their cumulative probability. According to the documentation of both GPT and Claude, using temperature and top_p together can lead to unpredictable behavior and poor performance. When both are set simultaneously, the combined effect may confuse the model, leading to degraded reasoning abilities and lower-quality outputs. For example, using both parameters might significantly hamper tasks that require detailed reasoning, like multi-step logic or chain of thought processes. Additional Information on Temperature and "n" Parameter When you set the parameter "n" and the GPT-powered chat generated three responses for a single input, the behavior was as follows: With the temperature parameter set to 0, all three responses were identical. This is because the GPT model generated all three responses in the same system state with a constant system fingerprint. However, when you restarted the LLM call, the response would change due to a different fingerprint. On the other hand, when you set the temperature parameter to 0.15 and the "n" parameter to 3, the three generated responses for the same input were different. This indicates that with a constant system fingerprint, the temperature or top_p parameter is the only factor affecting the diversity of the generated responses. Example with temp = 0, n = 3: Example with temp = 0.15, n=3: Conclusion Minimizing randomness in GPT responses is crucial in scenarios where consistency is needed, but true determinism remains elusive due to the non-deterministic nature of the model and variability in the system_fingerprint. By carefully controlling temperature and seeds and top_p settings, you can reduce randomness, though you must be cautious about over-constraining the model, as it may degrade performance in complex tasks. Monitoring system fingerprint changes is also key to distinguishing random variation from API backend updates, helping you manage LLM behavior more effectively. In our view, improving prompts and instructions is main essential step in reducing randomness.  Another solution might be simplifying the input specification, by using additional LLM calls to extract only relevant data and removing unnecessary information, it would help to reduce hallucinations and prevent GPT from becoming confused. Unlike GPT, this issue is less critical in Claude. Since Claude manages large datasets effectively and consistently produces stable output, it makes sense to retain the current results. As for now, the main method to improve Claude’s accuracy is inserting into a prompt one or a few shotsvexamples selected through vector search based on the highest similarity. That can effectively improve accuracy without increasing complexity. References The “seed” option for GPT does not increase the determinism level ChatCompletions are not deterministic even with seed set, temperature=0, top_p=0, n=1 Determinism I: Understanding Randomness Within LLMs Controlling model variance matters most in production systems — the kind COXIT ships as custom AI development. --- ### Prompt Engineering through Structured Instructions and Advanced Techniques URL: https://coxit.co/blog/prompt-engineering-through-structured-instructions-and-advanced-techniques/ Author: Yaroslav Biziuk Published: 2024-12-26 | Updated: 2026-08-19 Category: ML Engineering Introduction Language models (LLMs) are powerful tools for a variety of tasks, but their effectiveness is highly dependent on the design of prompts. This article examines advanced techniques in prompt engineering, focusing on the impact of instruction order, the "Ask Before Answer" technique, and the "Chain of Thoughts" (CoT) method, etc. By optimizing these factors, we can significantly enhance the accuracy and reliability of LLM outputs. The Importance of Instruction Order Instruction order plays a crucial role in prompt engineering. Altering the sequence of instructions or actions can drastically change the outcome produced by the LLM. For instance, when we previously placed an instruction about not considering “semi-exposed surfaces” as the eleventh step, the LLM would still process these surfaces as it followed each step sequentially, reaching the exclusion instruction too late to apply it effectively. However, when this instruction was moved to precede all other steps, the LLM correctly disregarded “semi-exposed” surfaces. This demonstrates the necessity of positioning general concepts or definitions above the specific step-by-step instructions, ensuring they are applied throughout the process. Example: The "Ask Before Answer" Technique The "Ask Before Answer" technique is particularly effective when optimizing prompts to reduce the occurrence of hallucinations. By prompting the LLM to seek clarification before resolving a task, we can preempt misunderstandings that might lead to incorrect answers. Example Prompt: You are a Casework expert. You review the specification and follow the instruction to pick the correct Series option which represents cabinet materials and thickness: OPTIONS: {OPTIONS} INSTRUCTION: {INSTRUCTION} SPECIFICATION: {INPUT_TEXT} If you have questions or misunderstandings, ask about it before resolving the task. Before proceeding with the itinerary, please ask for any clarifications or additional details. I will give more info if you need. Result: When applying this technique, we ask the LLM to identify specific areas where it may be uncertain or confused in resolving a test case. By doing so, we can pinpoint where hallucinations occur, understand why the LLM struggles with certain choices, and refine the prompt in those areas where the model tends to get confused. This method is highly effective in improving the quality of the instructions provided in the prompt. The Chain of Thoughts (CoT) Technique Result without CoT: One of the most critical steps in creating an effective prompt with complex instructions is the use of the Chain of Thoughts (CoT) technique. By including phrases like "You think step by step," "Take your time," or "Explain every step," the LLM is given time to reflect and process all input data. With the last  This approach significantly improves the results, making them more logical and coherent. But you need to be careful when using “Explain every step”. Because often the LLM can find the most likely things that are correct, but it can not always explain why it chooses this, which often leads to hallucination Result with CoT: Meta-Prompting: An Advanced Strategy in Prompt Engineering Meta-prompting is an advanced technique in prompt engineering that goes beyond merely guiding a language model (LLM) through a task. It involves crafting prompts that instruct the model on how to think or approach a problem before the primary task begins. This strategic layer of prompting enhances the LLM's ability to navigate complex instructions by embedding a meta-level understanding of the task at hand. For example, instead of directly asking the LLM to solve a problem, a meta-prompt might instruct the model to first assess whether it fully understands the task, identify potential ambiguities, and request clarification if necessary. This proactive approach reduces the likelihood of errors and hallucinations by encouraging the model to consider its own limitations and seek additional information before proceeding. When applied in Claude LLM, meta-prompting proved to be more effective than in GPT models. It significantly improved test case outcomes by making instructions simpler and clearer for the model to understand. Claude's advanced processing capabilities allowed it to better interpret and act on the meta-prompts, leading to more accurate and consistent results. By guiding the LLM to reflect and seek clarity at key stages, meta-prompting not only enhanced the accuracy of the results but also optimized structure and the clarity of instructions, making them more intuitive and easier for the model to follow. Example how meta-prompting optimized Claude outputs: However, in our specific case, meta-prompting did not lead to the exceptional results we had hoped for. While it is a valuable technique that has the potential to improve LLM performance, its effectiveness can vary depending on the complexity of the task and the model's inherent capabilities. Despite this, meta-prompting remains a powerful tool in the prompt engineering toolkit, particularly for refining and simplifying instructions for better LLM comprehension. Explanatory Instructions One key insight in optimizing prompt engineering is the importance of providing explanations for why certain choices should be made. Adding reasoning behind instructions helps LLMs make better-informed decisions, thereby improving their overall performance. For instance, consider the following instructions: Worse: "If both HPL and TFL can be used, choose TFL." Better: "If both HPL and TFL can be used, choose TFL as the more cost-effective option." In the "better" example, the LLM is not only told which option to choose but also why that choice is preferable. This additional context helps the model understand the underlying logic and apply it more consistently across different scenarios. Conversely, the "worse" example simply dictates a choice without explanation, which may lead to less reliable or inconsistent outcomes. Incorporating explanations within prompts also reduces the likelihood of errors or hallucinations by giving the LLM a clearer framework for decision-making. This approach enhances the model's ability to follow complex instructions more accurately, resulting in better task execution and improved performance. By emphasizing the "why" behind decisions, prompt engineers can create more robust prompts that align better with the intended goals, ultimately leading to more effective and reliable use of language models. Simplify your instructions When writing instructions for LLMs, it’s crucial to keep them clear and simple. If people find it hard to read and understand the instructions, the model will struggle even more. Use plain language and short sentences, as if you’re explaining things to a child. If the instructions are complex, break them into a few sentences to make them easier to follow. For instance, consider the following instructions: Worse: "If front materials are not specified or not matched with TFL surfaces, HPL, or A-Tech Surface, choose HPL as the default option." Better: "If fronts are not specified, default to HPL unless TFL or A-Tech Surface is mentioned." In the "better" example, the instructions are straightforward and easier to understand. This clarity helps the model grasp what’s needed more effectively and reduces the probability of hallucinations. The simpler the instructions, the more accurately and reliably the model will perform. Breaking instructions into a few shorter sentences can make them more digestible and prevent confusion. Imagine You're Training a Colleague When crafting prompts, it's essential to approach the task as if you are training a child or a new colleague at work to complete a specific task. To achieve this, you need to provide the individual with sufficient context and detailed, clear instructions. The prompt should include all necessary information to ensure precise task execution. This method helps to ensure that the instructions are comprehensive and easily understood, thereby improving the likelihood of successful task completion. The Role in Prompting An important aspect of creating effective prompts is assigning a specific role to the language model (LLM). This role definition helps limit the data the LLM will use to solve the task and sets the appropriate tone and format for the response. Role prompting can also enhance the LLM’s performance in various tasks, such as math or logic problems. For example: "You are a Casework expert tasked with reviewing a specification and selecting the correct Series option that represents cabinet materials and thickness." Worse: "You are a Casework expert" Better: "You are a Casework expert tasked with reviewing a specification and selecting the correct Series option that represents cabinet materials and thickness." This role-based approach provides the LLM with a clear context and specific guidelines, which positively influences the accuracy and relevance of the model's responses. Conclusion In prompt engineering, the careful structuring of instructions, along with techniques like "Meta-prompting" and "Chain of Thoughts," “Role Prompting”, “Ask Before Answer”, and others can dramatically enhance the performance of language models. These methods help in reducing hallucinations, improving clarity, and ensuring more accurate outcomes. Future research could explore further optimization strategies and their effects on various types of LLM tasks. References Prompt Engineering for ChatGPT: A Comprehensive Guide The Prompt Report: A Systematic Survey of Prompting Techniques Structured prompting of this kind underpins the AI systems we build for document- and drawing-heavy workflows.