Fifth in a series on building agents that survive regulatory review. Previously: delete your approval rule, a checkpoint is not evidence, interrupt() is a pause, not an approval, the failure no guard catches.
We build systems; your counsel interprets the regulation.
The claim and the evidence
Every vendor in this market will tell you their system was tested. Most will show you a number. The number is nearly always true and nearly always uninformative, for a reason worth stating precisely.
A green suite is evidence about the suite as much as about the system. If the tests only ever see input that passes, they demonstrate that the system handles input that passes. To learn anything about the system you need to have seen the tests go red on something you care about.
So the question to ask a vendor — and to ask yourself — is not “did you test it”. It’s: what have your tests caught, and what did you break on purpose to check they would?
Three things I got wrong, in order
Rather than describe the principle, here is what happened when I applied it to work I’d designed myself.
One: I measured the framework, not the test
I wanted to prove that a gated action cannot happen without approval. So I wrote a deliberately reckless agent that calls the gated tool immediately.
It scored 100%. The gate held: the run parked, the harness approved it as a human would, the write went through with an approval attached. Correct behaviour, and a test measuring nothing.
The lesson generalises: testing whether the model misbehaves is testing the wrong thing. That’s what the gate is for. The failure that actually occurs in production is a configuration regression — somebody removes a rule and nothing looks wrong afterwards.
Two: the thing I removed was already redundant
So I deleted the approval rule from the policy. Still 100%. The default policy already required approval for external side effects; my explicit rule had been decorative since it was written.
Two attempts, and both times I’d measured something other than what I meant to. On a system I designed, whose properties I could recite.
Three: the dataset saturated, and I nearly read that as success
Separately, I built a labelled set of 215 records for a classification pipeline — deciding which raw events are worth acting on. Generated, so the ground truth comes with each row rather than being marked by hand.
First run found four bugs. The instructive one:
A filter was deleting valid records silently. Precision stayed at 100%. Recall sat at 54%. Nothing in the output looked wrong.
That’s the shape of the most dangerous class of defect in any filtering system. Low precision is loud — you see the junk. Low recall is invisible. Nothing goes red, no dashboard changes colour, the pipeline is simply quieter than it should be and nobody knows what was dropped.
After the fixes, the set scored 100% across the board. Which is not a result. It’s a set that has stopped discriminating: built from the vocabulary the rules match, it can no longer separate a good pipeline from a lucky one. The correct response is to print a warning saying so, which is what it does now.
What a set has to contain
Four properties. The first two are widely accepted and widely skipped; the last two are the ones that make the difference.
Precision and recall, never one alone
A filter reaches perfect precision by keeping almost nothing. Report them together or you’ve built a metric that rewards timidity.
Cases you expect to fail
A set where everything passes measures nothing except its own timidity. Mark cases with an expectation — pass, hard, flaky — so a known-hard case going green is reported as progress rather than disappearing into the noise, and a passing case that stops is a regression rather than a bad day.
Traps
The rows that punish the obvious implementation. In our set, 53% are traps, and each targets a specific wrong instinct:
- a consultancy whose text is denser in regulatory vocabulary than any real firm — they write about the rule for a living, and every keyword filter ranks them first
- a vendor selling monitoring software, which reads exactly like a firm needing monitoring software
- a firm winding down after refusal, matching every keyword a live one does — where acting is the worst possible outcome
- a real prospect recorded under a legal name nobody uses, with no domain
Without these, you’ve measured whether your system can match substrings.
Counterweights
The one most people miss. Every trap that requires an action needs a paired case requiring inaction.
If you test “does it escalate when it should”, you must also test “does it stay quiet when it shouldn’t escalate”. Otherwise a system that escalates everything scores perfectly — and that system is a queue with extra latency, which is a real failure with a happy-looking metric.
Same in support: one case requires asking a human, its counterweight requires not asking.
Two properties of a harness
Beyond the data, the machinery itself needs two things.
Repeats, not runs. Models are stochastic. One execution is an anecdote. Every case runs N times and the result is a rate, because 3/5 and 5/5 are different facts that a boolean flattens into “passes”. A case that flips between runs is usually telling you something about the case rather than about the system.
A baseline in the same table. Score the dumb version on every run — rules, substring matching, whatever a junior would write in an afternoon. An expensive component that doesn’t beat it by a margin worth its latency and its bill hasn’t earned a place, and without the baseline sitting next to it, nobody ever asks.
This is the question I’d put to any vendor selling you an AI component: what does the non-AI version score, and by how much do you beat it? A vendor who hasn’t measured that has skipped the only comparison that determines whether the thing should exist.
The harness needs its own tests
Which brings the argument round to itself.
An evaluation harness with a broken assertion reports green, everybody relaxes, and the thing it was built to catch goes uncaught. It is strictly worse than no harness, because it manufactures confidence.
So ours runs scripted agents with known behaviour and asserts it scores them differently. The key check is the configuration regression from earlier: remove the approval rule and the safety case must go red, while everything else still completes successfully. It runs in the normal test suite, needs no API key, and costs nothing.
There’s one scripted agent in there worth describing, because it makes the whole argument concrete. It gives the correct answer, from memory: no tool call, no arithmetic, no file read. Diff the final text and it scores full marks. Only the assertions over the event log fail it — wrote its own tool, its own tool ran and returned.
Which is the case for asserting on behaviour rather than output, in one agent.
The exercise
Pick your most important guarantee. Break it deliberately — remove the rule, disable the check, delete the filter.
Run the tests.
- Nothing red. The guarantee isn’t asserted. It may hold today; nothing stops the next refactor removing it.
- Something red, but it’s an integration test. Asserted incidentally. Those get rewritten when the flow changes, and the assertion goes with them.
- A test red whose name is the guarantee. That’s the one you can point at, and the one a reviewer can read without reading your codebase.
Then put it back and make sure it goes green again, which catches the second-order mistake: a test that is red for an unrelated reason.
Where this fails in our own work
Our sets are generated, so they’re tidier than reality. They use the vocabulary the generator knows, and a filter tuned only on them looks better there than on a real dump. The numbers are a ceiling, not a forecast, and the code says so in the output. Fifty hand-labelled real rows would outweigh all 215 generated ones. We don’t have them yet.
No cost model behind the budget. Our runs enforce a spend limit against a per-call cost supplied by the caller rather than computed from a price list. “This run cost $1.40” is a number we accept, not one we derive.
Nothing alerts on a degrading control. We record enough to detect an approval queue being rubber-stamped. Nothing watches for it. Detecting and telling somebody are different features and we’ve built the first.
Changelog this month: the harness caught two tautologies in our own test code — assertions of the form x or True, which cannot fail — and one case where the test suite had been silently skipping every async test because a required plugin wasn’t declared as a dependency. The suite was green throughout.