Right or Close Enough: How We Audit Our AI Field by Field
I do not condone violence. Not against anyone, including the American College of Obstetricians and Gynecologists.
This is a guide to making agentic AI systems less stupid, so they work in the real world. That's all it is. Only use guns where it's legal and safe.
The short version: my AI workflow classified the American College of Obstetricians and Gynecologists as a shooting range. Two state-of-the-art models, from two different AI companies, checked it. Both approved it. Then, because it was approved, it went live on USA Arms. So we audited our current workflow field by field: 2,484 real facts it accepted, each checked against the page it came from. The video and the free guide with every step are below.
How a medical association got classified as a shooting range
If you're into guns, you may recognize the ACOG rifle scope. A state-of-the-art model definitely knows ACOG stands for both the rifle scope and the American College of Obstetricians and Gynecologists.
The agent was given a lead for a shooting range with ACOG in its name. It searched for just "ACOG", and chose the most authoritative website: the College's own. Then the citations snowballed. Each one made sense on its own, even their page on gun violence prevention. And it passed every check we had:
- Is this about firearms?
- Is this the authoritative source for the lead?
- Does it talk about firearms?
- A citation check.
- A "does everything look right" check.
That last check even asked something close to whether it belonged on USA Arms as a shooting range. The model still said yes.
Confidently wrong, with an avalanche of evidence.It's the most common failure mode I've seen in production.
That was June, and luckily I caught it before anyone saw it but me. I thoroughly checked the logs, and thank goodness, I was the only visitor to that page on the website. We've rewritten the whole workflow five times since then, and the one before it got rewritten four times too. With each rewrite we've been ramping up our accuracy, and each step is a lot harder than the last.
Right, or close enough?
Every AI workflow faces this same problem. How much effort are you willing to put in to make sure it's right, versus just good enough?
Close enough is a great deal when something catches the misses: brainstorming names, drafting a first reply, finding leads someone will look at. It has to be right when the result goes straight to someone outside your business. A listing on USA Arms has to be right.
Every version of our harness gave the model more decisions and put up more fences to kick out the wrong answers. But ACOG was the perfect mutant. It got past every one, and showed us the weaknesses in our system. Today we have many more checks. I call them fences instead of walls, because holding too much back is its own sort of problem. Even in our seventh version, we were still holding back up to 80% of listings for ambiguity. That isn't right either.
Local models fail differently
We're tuning our entire workload to run on local models, and most of it already does. We've learned a ton, because local models fail differently than frontier models. A frontier model is confidently wrong. A local model:
- repeats its own broken answer, word for word,
- runs out of room before it checks its work,
- or gives up on the hard jobs a frontier model finishes.
By our metrics, the site is getting better every day. But it's hit a plateau, and it still isn't at the product level I expect. That plateau is us getting local models to do as well as the frontier models did, even as we hand them more decisions.
In my last video, we couldn't take four-bit's speed without also taking a lot more wrong answers. So how do we know when a quicker model is close enough?
How we audited it, field by field
We already had goldens, answers we've proven right, for whole listings. That's how we graded the jobs in my last video. Now we're building goldens for every field, so we can test smaller, faster models, even four-bit ones, as auditors, and see if the speed is worth it.
So we audited our current workflow, field by field:
- We started with 25 of our 95 fields.
- We took 2,484 real facts the production system accepted over nine days this month, about five percent of the new ones: prices, hours, range rules, transfer fees.
- We checked every one against the page it came from.
- Two frontier models each reviewed them on their own, to find the edge cases only frontier models are currently catching, so we know where the high-water mark is.
- Every disagreement got worked through case by case, and every quote was checked against the page.
- Where the rules didn't say, the question came to me. That was 40 questions.
Here's what we found:
- Most of it was right: 1,552 facts, 62.5%.
- About one in ten broke a rule (266), often by leaving something out, like a club's price list missing its one-dollar target fee.
- About one in twelve we couldn't prove either way (213). Those are becoming ambiguity goldens, to see if a model falls for a false positive.
- Almost one in five we set aside for now (453), mostly waiting on a rule we haven't settled.
Put together, more than one in four had no clean answer. That's how much ambiguity there is in the real world. These are working results from a slice of production, from an audit we're still doing, not a final score.
Not all wrong is the same
A price list missing a one-dollar fee is a small miss. Classifying a medical association as a shooting range is a completely wrong thing. So we grade adaptively, by how severe a wrong answer in each field would be. Our harness rewrite is about stopping the completely wrong things first, without holding back huge numbers of listings like earlier versions did.
The rules we'd never written down
The biggest lesson wasn't the score. The goldens showed us where there was true ambiguity, and the rules we'd never written down:
- A holster that fits a Glock isn't proof they sell Glocks.
- A shop menu saying "Ammo" isn't proof that they sell ammo.
Little things, until more than 20,000 listings depend on them. And we still have another hundred decisions or more to wade through.
If you're building agentic workflows, you need to embrace the reality that the real world is messier than you'd expect. This month, holster makers started showing up as gun stores. For some products, that would be fine, and in the future we may list holster makers. But they aren't gun stores, even though, like gun stores, they hold federal firearms licenses.
What we're building
This audit is almost a week in, and my guess is it will take at least a month to get robust goldens for every single field. Once we do, we'll have a solid foundation, so we don't have to do this sort of work on every iteration, and we can deploy local models with even more confidence.
We're making it modular and self-improving. Every miss becomes a rule, and every rule becomes a check that runs on the next version. True ambiguity gets flagged for me to decide. Then the new rubric gets applied against everything we've decided before, trimming the goldens it touches, and ideally finding the listings that have the old ambiguity built in.
Here are four parts of it I'm excited about, and testing over the next few weeks:
- Adaptive validation: checks based on how severe a wrong answer would be.
- Small classifiers that read each claim and decide if it's supported, or if it needs a state-of-the-art model or a bigger local model on our Macs.
- Code that detects where that heavy model call will be worth it.
- A state-of-the-art model that finds where our rubric isn't enough, so that becomes a rule.
Let me know which you'd like to see become its own video.
Watch the video
The whole story in about 8 minutes, with the audit on screen.
Get the guide
I had our agents compile every step we used into a free 7-page guide: how to put your workflows on the right-or-close-enough scale, trace a bad result back to the check that passed it, audit a workflow field by field, write down the rules it surfaces, and grade wrong by how bad it is.
The smart move is to hand the Markdown version to your AI agent and let it walk you through it, one step at a time. The guide starts with this prompt:
A prompt to start with
Walk me through this guide for my own business, one step at a time.
For each step: ask me for one real workflow or result, wait for my answer, then push back where I assume a check that doesn't exist. Help me fill in the audit sheet, the reviewer prompt, the rule template and the severity sheet with my own cases.
Finish with the completely wrong failures I should stop first.
- Every step from the video
- The reviewer prompt, rule template and severity sheet
- An audit sheet to copy for your own workflow
The guide is an AI compilation of our audit. It's what worked for us, not professional advice, so test it on your own work before you trust it.
What's next
Our prior harnesses had proto self-improvement loops. But in our latest harness, I believe we've finally figured out the foundation of how to build a truly self-improving business harness. My findings, and a detailed case study of our harness, are in my next video. The few people who've seen it early called it a master class in harnesses.
On the home lab side: the more agents I run, the more my servers get bogged down, and CI/CD becomes the bottleneck. A small "lander" system saved me from having to move to GitHub's managed merge queues, and I can share our lander if other people want it. Comment "merge queue" on the video if you're interested.
And tell me in the comments: what's the most confidently wrong thing an AI has done in your business? I'll be reading every one, and compiling the best ones for a future video.
Always on your team,