A proof you can check without asking us

The published run cost US$ 3.408. Its acceptance verdict is Outcome.INCOMPLETE. Both numbers are on the record, in out/run.json and out/audit-trail-t-009.txt. This post is about why we published the second one rather than the polished one, and about the three commands that let a reader answer the vendor's oldest question — how do I know? — without asking us.

We build systems; your counsel interprets the regulation.

The retrieval-accuracy problem

Until this month our own site carried a block with numbers like 0.96 retrieval accuracy and 0.03 hallucination rate. The numbers were real, from a real production system, from a real prior life. A Head of Compliance reading them had exactly one recourse: believe us. There was no document they could open, no command they could run, no hash they could compare. A number without a check next to it is not a claim; it is a colour on a slide.

The page those numbers were on now reads differently. Same URL, different argument. The argument is: here are three specific EU documents; here are their SHA-256 hashes; here is our field list against them; here is a run that produced that field list; here is the code that scored it, the log that recorded it, and the counters that verify one from the other. Check any of it. None of the four claims depends on us being trustworthy.

Three claims, each verifiable separately

The artifact makes three assertions. Each is a different kind of thing, and each has its own falsification path.

One: we pinned three real EU documents. Not a summary, not a paraphrase, not a screenshot. The XHTML that came out of Cellar, byte-for-byte, hashed. The manifest.json records the CELEX identifier, the retrieval date, the byte length and the hash. If any of those bytes have drifted — because we edited them, because they never matched, because the EU has since republished — make proof-verify goes red. The Cellar URLs are in the manifest; anyone can download the same document and run shasum -a 256 against it. This is the claim with the shortest verification path: two shell commands, one comparison.

Two: the deterministic validator scores 100% precision and 100% recall on twenty-five records. The records live at proof/data/, generated by code. Their ground truth is not marked by hand after the fact; it is written in the same file, at the same time, by the same person. That is a caveat, not a boast — the paragraph below in the LIMITS section spells it out. What it lets a reader do is run python -m proof.validate against the same data and get the same scores, and open out/validation-report-32023R1113.md to see the finding-by-finding trace.

Three: every quote in every rule module is literally present in the pinned text. A rule module contains fragments like “a transfer of an amount exceeding EUR 1 000” as a Python string. The test suite — uv run python -m pytest tests/proof/ — opens the pinned XHTML and searches for that exact string. If a rule drifts from its citation, the test goes red; there is no way to sneak in an invented citation without the CI catching it. This is the claim that turns “we read the regulation” into “here is where we read it, and we cannot silently move.”

And a fourth, weaker but load-bearing: the audit trail is derived from the log. out/verify.txt shows steploop verify <run_id> recomputing four counters — llm_calls, tool_calls, input_tokens, cost_usd — from events.jsonl and diffing them against the cached row. It reads projection is consistent. This is the second run to publish a clean verify; the previous one drifted, the bug is on the record, the fix (28e5095) is on the record, and this run is the evidence the fix works on live data.

The twenty-five records, and their traps

The three datasets are shaped to punish a validator that flattens a conditional article into a flat field list. The trap in each dataset is the record that a naive implementation would flag and a competent one would not.

• Travel Rule (nine transfers). A non-DLT transfer with an account number where a distributed ledger address would be required — 14(1)(b) does not apply. A transfer using date and place of birth in lieu of address — 14(1)(d) alternative. A self-hosted transfer of exactly EUR 1 000.00; the article reads “exceeding”, so strict inequality, so no ownership-assessment duty. And a record the run has to qualify rather than decide: LEI absent, message format unknowable from the data.
• DORA register (seven entries). A register maintained internally rather than reported. B_01.01.0050 (competent authority) and 0060 (reporting date) do not apply, and a flat “every column must be populated” rule flags a compliant row. Maintenance-vs-reporting mode is an external fact, so the validator asks rather than guesses.
• MiCAR record-keeping (nine orders). An order where the investment decision was made by the client, not by a person or algorithm within the CASP. Annex Table 2 field 2 does not apply, and a flat reading flags it. A genuine defect, m-008, sits alongside so the compliant case is not just an accidental miss.

Their being expected to produce nothing is the assertion. A run that returns a finding on any trap fails ground-truth diff by construction. That is the property we care about — a system that flags a compliant row is a queue with extra latency, which is a real failure with a happy-looking metric.

We published the INCOMPLETE run

The obvious move is to publish the run whose acceptance verdict is Outcome.DONE. We didn't. The run we published exits 5, Outcome.INCOMPLETE, because the acceptance check — the machinery that reads the final answer and decides whether it actually did the task — found real gaps. The coverage claim rested on deference to an external file rather than being explicit in the answer, and a list_files tool call that happened was not evidenced in the answer text.

Both gaps are quoted verbatim in audit-trail-t-009.txt. The report and the answer are on disk; the reader can judge whether the acceptance check's objections stand.

Publishing this run rather than a polished retry is the whole argument. If the artifact only ever shipped Outcome.DONE runs, the acceptance check would be decorative — there would be no evidence it ever catches anything, and a reader would have to take on faith that it does. An INCOMPLETE run in public is evidence that the check is a control, not a habit. It stopped being correct on something that mattered, and the machinery said so, and we published what it said.

What this does not prove

The full list is in LIMITS.md, and reading it before the report is the argument, not a disclaimer. In brief:

• The data is synthetic. Twenty-five fabricated records. The obligations they are checked against come from the pinned CELEX texts, not from us.
• The volumes are demo-scale. Twenty-five records is enough to carry a trap and a counterweight per document. It is not enough to say anything about throughput.
• No supervisor has seen this. Not ESMA, not the Central Bank of Ireland, not any competent authority. Validating records against a published regulation is not regulatory approval.
• Coverage is partial — and named. Article 14(1)–(2) plus the second sentence of (5). Template B_01.01 of the fifteen DORA templates. Sections A–C, fields 1–5 of Annex Table 2. Every unmodelled clause is declared with a real citation.
• The precision/recall figures. 100% measures whether the validator implements our reading. It does not measure whether our reading matches the document. That is what the “readings open to challenge” section of each report exists to expose.
• The run is dated. One model, one date, both recorded. Model behaviour changes between calls to the same version. A run that asked one clean clarifying question on this date is an observation about that run, not a guarantee about the next.

The three commands

The offline half of the argument runs without an API key. Two commands and a wait.

The live half — re-running the agent against the pinned documents — needs ANTHROPIC_API_KEY and a budget cap around US$ 4. The Makefile target is make proof.

Why a compliance buyer needs this

Every RegTech vendor pitching under MiCA, DORA and the EU AI Act meets the same reflex from the same reader: prove it. There are three answers on the market. Most vendors give the first: a customer logo grid, sometimes a testimonial, no artifact. Some give the second: a demo, run by them, on their laptop. A handful give the third: a case study under an NDA, redacted enough to be uncheckable.

None of the three survives the how do I know? question in the way a Head of Compliance needs it to, which is why the market defaults to reference calls with a competitor they trust. This artifact is a fourth answer. Not a customer, not a demo, not an NDA. A run, in public, against documents we do not control, with a validator anyone can rerun and a log anyone can verify. The reader does not have to trust us because they do not have to ask us anything.

That is not the whole argument for hiring us. It is the part of the argument that survives them not knowing us yet.