Sadyr, Adroit's revenue agent, doesn't guess. If its confidence score on a retrieved document falls below a threshold, it abstains rather than inventing an answer. That gate is central to the product's promise, and it broke in a way that let every test pass.
The bug divided Sadyr's retrieval confidence score by the number of words in the question. Against the mock retrieval adapter — which returned a raw count of matching words instead of a normalized score — that looked correct. Every test passed. With real OpenAI embeddings, real cosine similarity scores land in the range of about 0.4 to 0.6. Divided by a multi-word question's token count, those scores fell to roughly 0.03 to 0.05, well under the gate's threshold of 0.62. Every multi-word question abstained.
I found it on the first live run against real embeddings, on 14 September 2026. I fixed it the same day.
What Sadyr was doing
Sadyr answers pre-sales questions on a company's website. It checks its sources, qualifies the visitor, and hands a sales rep a briefed lead. The confidence gate sits between retrieval and the reply: Sadyr retrieves the best-matching document from its knowledge base, checks the score, and either answers or declines cleanly.
The retrieval adapter, the mock that stands in for it during tests, and the runtime logic that decides whether to answer are all pieces of the harness around the model.
How the mock hid the problem
Sadyr's test suite runs against a mock retrieval adapter by default, so the platform can verify its behavior without calling out to OpenAI on every test run. The mock faithfully reproduced the bug: it returned a raw count of matching words rather than a normalized score between 0 and 1, and the runtime divided that count by the question's word count before comparing it to the threshold.
Against the mock, that division looked correct in every test.
The whole suite — over a thousand automated tests — passed the entire time the bug was live.
What real embeddings showed
The demo build I was preparing on 14 September was for an insurance provider, testing Sadyr against sample content. The first live questions were normal, multi-word questions a visitor would ask.
Sadyr abstained.
Real cosine similarity scores from OpenAI's embeddings don't look like word counts. Real scores land around 0.4 to 0.6. Divided by the number of words in a multi-word question, those scores landed around 0.03 to 0.05, nowhere near the 0.62 threshold the gate was checking. Every question longer than one word failed.
The fix
The fix landed in commit 69d40bc the same day. Every adapter — the mock included — now returns a normalized score between 0 and 1. The runtime compares that score as-is, with no division. The contract for what a "score" means was written down in the codebase so the mock and the real adapters could not drift apart again.
The diffstat: 3 files changed, 15 insertions, 6 deletions.
What I took from it
When Claude Code or another agent writes both the code and the tests, a passing test suite verifies that the code agrees with itself. It does not verify that the code agrees with reality. The gap only shows up when the real system runs, with someone watching.
My habit since: the tests are where I start, not where I stop. Instead of checking the system-generated mocks, I experiment with it on the front end and make my own attempts at breaking it. The mock is a hypothesis about what the real thing does. The real thing is the test.
Corrected 2026-09-29: the gate "broke in production" and passed tests "for weeks", built in "March and April 2025" → found on its first live run for a demo on 14 September 2026 and fixed the same day; my habit since was "check the mock's behavior" → I start with the tests, then try to break the real thing on the front end; and other details that did not match the record.