Sadyr, Adroit's revenue agent, doesn't guess. If its confidence score on a retrieved document falls below a threshold, it abstains rather than inventing an answer. That gate is central to the product's promise, and it broke in production in a way that let every test pass for weeks.
The bug divided Sadyr's retrieval confidence score by the number of words in the question. Against the mock retrieval adapter — which returned a raw count of matching words instead of a normalized score — that looked correct. Every test passed. With real OpenAI embeddings, real cosine similarity scores land in the range of about 0.4 to 0.6. Divided by a multi-word question's token count, those scores fell to roughly 0.03 to 0.05, well under the gate's threshold of 0.62. Every multi-word question abstained.
I found it on the first live run against real embeddings, on 14 September 2026. I fixed it the same day.
What Sadyr was doing
Sadyr answers pre-sales questions on a company's website. It checks its sources, qualifies the visitor, and hands a sales rep a briefed lead. The confidence gate sits between retrieval and the reply: Sadyr retrieves the best-matching document from its knowledge base, checks the score, and either answers or declines cleanly.
About the same time I was building that gate, in March and April 2025, OpenAI, Google Cloud and others were converging publicly on orchestration as its own architectural layer — what would later be called an agent harness. The retrieval adapter, the mock that stands in for it during tests, and the runtime logic that decides whether to answer are all pieces of that harness.
How the mock hid the problem
Sadyr's test suite runs against a mock retrieval adapter by default, so the platform can verify its behavior without calling out to OpenAI on every test run. The mock faithfully reproduced the bug: it returned a raw count of matching words rather than a normalized score between 0 and 1, and the runtime divided that count by the question's word count before comparing it to the threshold.
When the question was "price" and the mock found three matching words, the score was 3 ÷ 1 = 3.00, well above the 0.62 threshold. When the question was "What is the price?" the mock found the same three words, the score was 3 ÷ 5 = 0.60, and Sadyr abstained. That looked like reasonable behavior during development: longer, vaguer questions should be harder to answer confidently.
The whole suite — 2,816 automated tests, recorded from Sadyr's README and not re-run for this article — passed the entire time the bug was live.
What real embeddings showed
The demo build I was preparing on 14 September was for an insurance provider, testing Sadyr's ability to explain program details from sample content. The first live question I typed was something like "What plans do you offer?" — a normal, multi-word question a visitor would ask.
Sadyr abstained.
Real cosine similarity scores from OpenAI's embeddings don't look like word counts. A strong match might score 0.55; a weak one, 0.42. Divided by the four or five words in a typical question, those scores landed between 0.03 and 0.10, nowhere near the 0.62 threshold the gate was checking. Every question longer than one word failed.
The fix
The fix landed in commit 69d40bc the same day. Every adapter — the mock included — now returns a normalized score between 0 and 1. The runtime compares that score as-is, with no division. The contract for what a "score" means was written down in the codebase so the mock and the real adapters could not drift apart again.
The diffstat: 3 files changed, 15 insertions, 6 deletions.
The test suite still passed after the fix, because the mock now behaved the same way the real adapter did.
What I took from it
When Claude Code or another agent writes both the code and the tests, a passing test suite verifies that the code agrees with itself. It does not verify that the code agrees with reality. The gap only shows up when the real system runs, with someone watching.
My habit since: run the real adapter early, and check the mock's behavior before trusting any test suite an agent has written. The mock is a hypothesis about what the real thing does. The real thing is the test.