We build systems; your counsel interprets the regulation. So this is about engineering evidence, not about what any article requires — the obligations below are the ones your compliance team will already have on their list.
Two failed attempts to prove a control worked
We have an agent framework where side effects pass through a policy gate: the model states what it wants to do, and code decides whether it may. The claim is that a gated action cannot happen without a named human approving it.
Claims are cheap. I wanted a test that would go red if the claim stopped being true.
First attempt. I wrote a deliberately reckless agent — one that calls the gated tool immediately, with no regard for policy. Expected: a red test.
It scored 100%.
Not because the agent behaved. Because the gate worked. The call hit the policy, the run parked awaiting approval, my test harness approved it as a human would, and the write went through with an approval attached. Which is the framework doing its job, and my test measuring nothing at all.
Second attempt. The failure worth catching isn’t a misbehaving model — a model can’t bypass a gate it never sees. The failure is a configuration regression: somebody removes a rule and nothing looks wrong afterwards. So I deleted the approval rule from the policy.
Still 100%.
The default policy already required approval for anything flagged as an external side effect. My explicit rule had been decorative since the day it was written. To actually open the gate I had to remove the rule and clear the default.
Then, finally, red.
What this cost, and what it bought
Two wasted hours, and one fact that no amount of code review would have produced: I did not know which of my guarantees were real. One was stronger than I thought. One was redundant. Both looked identical from the outside — green tests, working system, a claim in the documentation.
The second discovery is the useful one for anybody else’s stack. If removing a control changes nothing, one of two things is true. Either you have a second control you’d forgotten about, or you never had one. You can’t tell which by reading, and you can’t tell which under review.
The exercise
On your own system, pick the action you’d least like an agent to take unsupervised. Payment initiation. Account closure. Anything that writes to a customer record.
Then:
- Remove the rule that requires approval for it.
- Run your test suite.
- Count the red tests.
Three outcomes, and all three are informative.
Nothing went red. You have no test that asserts the control exists. The behaviour may still be correct today, but nothing stops the next refactor from removing it silently — and “we’ve always done it that way” is not an answer to “show me”.
Something went red, but it was an integration test. Better. Your control is asserted somewhere, but only as a side effect of a larger flow. Those tests get rewritten when the flow changes, and the assertion tends to go with them.
A test went red whose name is the control. test_the_gated_action_never_runs_without_approval. That’s the one you can point at. It’s also the one a reviewer can read without reading your codebase.
Control versus habit
The distinction is the whole point, so let me put it plainly.
A habit is behaviour that is currently correct. Your agent asks for approval because the prompt says to, or because that’s how the flow happens to be wired, or because nobody has changed it. It works. It demonstrates nothing.
A control is behaviour that cannot silently stop being correct. Something fails loudly when it does. The failure is named after the guarantee, so the name itself is the documentation.
Under normal engineering pressure the two are indistinguishable — both produce a working system. They diverge under exactly two conditions: a refactor six months from now, and a supervisor asking how you know.
The obligations your compliance team tracks — human oversight, record-keeping, third-party risk — are all written in the second language. They ask what the system cannot do and how you know. A habit has no answer to that question, because the answer is “we looked and it seemed fine”.
What to do instead
Three things, in order of how much they buy per hour spent.
Write the negative tests. Positive tests say the feature works. Only a negative one says nothing else happens: never called without approval, never ran after cancellation, never wrote outside the permitted scope. Name them after the guarantee.
Make the failure mode a configuration change, not a model behaviour. Testing whether a model misbehaves is testing the wrong thing — that’s what the gate is for. Test whether removing the gate is detectable. That’s the regression that actually happens, and it happens quietly.
Default closed. If the way to grant a permission is to add a rule, an omission fails safe. If the way to deny one is to add a rule, an omission fails open — and omissions are what happen under deadline. This is the one that saved me above: my redundant rule was redundant because the framework was already closed.
The uncomfortable part
I ran this on a system I wrote, whose design I could recite, having thought about this exact property for weeks. I still got it wrong twice.
Which is an argument for the exercise rather than against it. Nobody’s mental model of their own system is accurate enough to skip the experiment. The question isn’t whether you’re a careful engineer. It’s whether the careful engineering left something a third party can verify.