Last in a series on building agents that survive regulatory review. Previously: delete your approval rule, a checkpoint is not evidence, interrupt() is a pause, not an approval, the failure no guard catches, a test suite nobody has watched fail, what remains when the vendor disappears.
We build systems; your counsel interprets the regulation. Each question below names the obligation family it belongs to so you can hand it to the right person — the engineering claim is that the answer has to be built, not inherited.
The one idea
Six posts, compressed:
Durability, developer velocity and auditability are different properties. Frameworks optimise the first two. The third is yours to build, and most of it cannot be added retroactively.
That last clause is why this matters now rather than at renewal. A system built to continue can always be taught to continue better. A system built to continue cannot be taught to answer after the fact, because the information was never written.
1. Reconstruct one decision
Take a decision your system made last month. Without opening a terminal: what exactly did the model have in front of it at that moment?
Obligation family: record-keeping.
Good: the messages sent to the model are in the record, verbatim, at that step. You can read what it saw.
Weak: you have the state, and could reconstruct the prompt by re-running the template against it — assuming the template hasn’t changed.
Failing: “we’d need the developer.”
Why it’s first: every other question depends on it. If you can’t reconstruct the input, you cannot evaluate the output, and nor can anybody else.
2. Who signed, and what proves their standing
Who approved the last gated action your system took? What evidence exists that they were entitled to approve it, and that they saw enough to decide?
Obligation family: human oversight.
Good: an identity in the decision record, the rendered summary they were shown, elapsed time, and a denial path that structurally terminates the action.
Weak: an identity correlated from your web layer’s access log, with its own retention policy.
Failing: a boolean. resume=True records that something sent true.
The uncomfortable follow-up: what’s your approval rate? If it’s near 100%, either the gate is mis-scoped or nobody is reading. Oversight degrades through volume, not negligence, and it’s visible only in data you had to decide to record.
3. Break it and see
Delete the rule requiring approval for your riskiest action. Run the tests. How many went red?
Obligation family: control effectiveness.
Good: a red test named after the guarantee — one a reviewer can read without reading your codebase.
Weak: an integration test went red. Asserted incidentally; it’ll be rewritten when the flow changes.
Failing: nothing red. You have a habit, not a control. A habit is behaviour that is currently correct; a control is behaviour that cannot silently stop being correct.
I failed this twice on my own system. First I measured the framework working instead of my test working. Then I removed a rule that turned out to have been redundant since the day it was written. Nobody’s mental model of their own system is accurate enough to skip the experiment.
4. What happens when it doesn’t know
Find the last time your agent was uncertain. What did it do?
Obligation family: human oversight, again — the part about knowing when to intervene.
Good: two independent channels. The agent can escalate, and detectors fire on observable symptoms — repeated tool failures, empty retrievals, the customer repeating themselves. Both are recorded, and they’re distinguishable in the log.
Weak: the agent can escalate if it decides to.
Failing: you can’t find an instance, because uncertainty isn’t recorded as anything.
Why one channel isn’t enough: an agent that is confused is frequently confused in a way it cannot perceive. Asking a stuck model whether it’s stuck is unreliable exactly when it matters.
5. The export
Your vendor terminates tomorrow. What file arrives, and can you open it with nothing but a standard library?
Obligation family: third-party risk, exit readiness.
Good: append-only, one row per event, human-legible, answers “what was decided on 14 March and why” with no vendor code present.
Weak: a database dump. Readable, but tells you where the system was rather than what it decided.
Failing: checkpoints in a framework’s serialisation format. That’s a copy of your dependency.
The second-order question people skip: who operated it? A perfect export of a system you cannot run leaves you with evidence and no capability.
6. The right question
How do you know your agent answered the question that was asked, rather than an adjacent one?
Obligation family: accuracy, and the accountability that rests on it.
Good: an acceptance check with wrong_question as a verdict distinct from incomplete, run against the original request by something that didn’t do the work.
Weak: schema validation. Confirms shape, not subject.
Failing: nothing checks. Which is the default, because every standard guard watches process — budget, retries, loops, permissions, cancellation — and none of them ask whether the right question was answered.
The one that costs most: a fluent, well-sourced answer to the adjacent question survives review by a busy person. In document Q&A the failure is almost never hallucination; it’s faithful quotation from the superseded version.
7. What your reviewer can read
Can your risk reviewer read the thing that makes decisions — or only the thing that calls it?
Obligation family: transparency, and the practical reality of vendor review.
Good: the decision logic is a small amount of code they can read in an afternoon, with few dependencies. A dependency is a thing your reviewer also has to read.
Weak: a framework they must first understand — its execution model, its state semantics — before they can understand your audit trail.
Failing: the logic is distributed across prompts, graph topology and library internals, and the honest answer is “you’d have to trust us”.
Why it’s last: every other answer is delivered through this one. An excellent control nobody can verify is an excellent control you cannot evidence.
Our own answers
If this list only pointed outward, it’d be marketing. Here’s how our work scores.
Passes. Decision reconstruction — the log holds the exact messages. Approval records — identity, rendered summary, elapsed time, denial terminates. Breakability — the safety cases go red when the rule is removed, verified by a self-test that runs without an API key. Two escalation channels. Acceptance checking with a distinct wrong_question verdict. Readability — around 5,700 lines, no runtime dependencies.
Fails, three of them.
Retention and redaction. The log keeps everything verbatim, for ever. Correct default for evidence, wrong one for personal data. Field-level redaction at write time with replay intact is in progress; today it’s a decision at ingestion, which is weaker. This is the one I’d raise first if I were reviewing us.
Authority is asserted, not verified. We record who approved. We do not verify they were entitled to. Binding approvals to a role model in your IAM is the right design and is an integration you’d write. No second-signature requirement either — for high-value actions where two independent approvers are wanted, we can’t express it.
Nothing alerts on a degrading control. We record enough to detect rubber-stamping. Nothing watches for it. Detecting and telling somebody are different features and we’ve built the first.
And three known limits that aren’t failures, but you should know. The sandbox is defence in depth, not a security boundary — a real boundary is a VM with a threat model, which is an infrastructure decision. SQLite is a single-node ceiling. Budgets are enforced against a cost the caller supplies rather than one computed from a price list.
What “we improve continuously” should look like
Every vendor says it. It’s undemonstrable, which is why it’s worthless. The version that means something is a changelog with dates.
From the weeks spent writing this series, caught by tests rather than by review:
- a filter deleting valid records silently — precision 100%, recall 54%, nothing in the output looked wrong
- two tautological assertions in our own test code, of the form
x or True, which cannot fail - a test suite silently skipping every async test, because a required plugin wasn’t declared as a dependency — green throughout
- a demo path broken for two weeks, because the tests covered it and the demos weren’t in the build
- an approval gate whose explicit rule turned out redundant against an undocumented default
Five findings, five named tests. That’s what improvement looks like when it’s true: a record, not an intention.
What to do with this
Run the seven against your own system this week. Score yourself honestly — the failures are the useful part, and the exercise is worth more than any vendor’s answer, including ours.
Then run them against whoever builds your AI. Compare the deltas.
If the exercise turns up gaps you’d rather not fix yourself, we do this for authorised firms operating under MiCA, DORA and the EU AI Act. If it turns up gaps you’d rather fix in-house, the list is the same list, and it’s yours.