A document Q&A system where every answer is a quote — and every quote points to a page.
A Canadian investment manager needed precise answers from fund prospectuses, KIIDs, investment policy statements, board policies, and regulatory filings. The wrong number on a fund question is a compliance event; the wrong policy reference on an IPS question is a client lawsuit. Generic, embedding-only retrieval was not going to clear the bar. We built a document Q&A system that answers with exact page citations and refuses to answer when the evidence is not there. Production thresholds: 0.96 retrieval accuracy, 0.92 faithfulness, near-zero hallucination.
Client
A Canadian investment management firm (under NDA). Delivered for the analyst teams responsible for fund research, investment policy compliance, and client-facing question handling — across fund prospectuses, KIIDs, IPS, board policies, and regulatory filings.
Engagement
Discovery → architecture → build → go live. One Solution Architect accountable for the result, with AI engineers, a business analyst, and QA. The system runs in the client's own cloud estate, against the firm's existing document repositories.
"Close enough" is the wrong answer when the document is a prospectus.
Investment documents are a hostile corpus for generic retrieval. A single prospectus runs to hundreds of pages and cross-references itself, the supplementary prospectus, the KIID, and the underlying fund's holdings — each at specific page anchors that an analyst is expected to cite verbatim. An IPS is a contract; the wrong policy reference in an answer is not "almost right", it is a problem the firm has to write to the client about. KIIDs are regulator-defined, with a fixed structure where the same field name carries different obligations depending on which section it appears in. And the regulator changes the documents on a schedule that the firm does not control.
The analyst's question is not "what does this fund roughly do" — it is "what does the prospectus say, on which page, about the fund's permitted derivatives exposure in adverse market conditions". A generic, embedding-only pipeline produces a confident summary that is mostly correct, occasionally wrong, and never cites the page. None of those failure modes are acceptable. The product the firm needed was a system that cites the page on every answer, refuses to answer when the evidence is not in the corpus, and is trustworthy enough that the analyst can paste the citation into a client letter.
Three constraints shaped the build. Citations are the product, not a feature. Every answer points to a page. Faithfulness is gated, not measured. The pipeline blocks an answer that is not grounded in the retrieved evidence. The evaluation set is the contract. Production thresholds are agreed up front, and prompt changes that drop below them do not ship.
Structure-preserving ingestion. Hybrid retrieval. Mandatory page citations. Faithfulness as a gate.
A document Q&A pipeline where the citation is the contract — not a hopeful link at the end of a paragraph.
Structure-preserving ingestion. Prospectuses, KIIDs, IPS, board policies, and regulatory filings are parsed with structure preserved — page anchors, section hierarchy, tables, footnotes, and cross-references — into a chunked representation that the rest of the pipeline can reason over. A naïve chunker that shreds tables and loses page anchors makes the rest of the pipeline impossible; we did not start there.
Per-document metadata, including effective dates. Each document carries its document type (prospectus / KIID / IPS / board policy / regulatory filing), the fund or product it relates to, its effective date, and its supersession history. The retrieval stage filters on these before similarity ever runs — a question about the current fund's policy never retrieves last year's prospectus.
Hybrid retrieval, dense plus sparse plus rerank. Dense vectors handle paraphrase; a sparse BM25 layer handles the exact-token failures that financial jargon produces ("derivatives for hedging purposes only" is not a paraphrase of itself). The fusion stage merges; a cross-encoder reranks the merged set. Each stage is feature-flagged so the evaluation harness can attribute gains per stage.
Document grading before generation. A grader labels each retrieved candidate as highly relevant, partially relevant, or not relevant. Partially-relevant retrievals trigger a query rewrite and a second pass. If after a bounded number of passes the evidence is not there, the system refuses — the analyst gets "the corpus does not contain this answer at this effective date", not a confident hallucination.
Schema-typed answers with mandatory citations. The model's output is a typed structure with two fields: the answer text with inline citation markers, and the references array of (document, section, page). A response that does not include at least one valid citation against the retrieved evidence does not leave the layer. The UI renders citations as clickable links into the source document at the specific page.
Faithfulness checker before the user sees the answer. After generation, every claim in the answer is cross-referenced against the retrieved evidence by a separate faithfulness model. Claims that are not grounded are flagged. Below the production threshold (0.92), the answer is regenerated or refused — not shown.
Evaluation as the production contract. A golden evaluation set, curated with the analyst team, runs in CI on every prompt or pipeline change. Retrieval accuracy ≥ 0.96 and faithfulness ≥ 0.92 are the production thresholds the system gates against. Changes that drop below either threshold do not ship — regardless of how good the spot-check looks.
Pipeline
Page-cited answers, gated thresholds, and a refusal contract that the analyst trusts.
Retrieval accuracy
Measured on the golden evaluation set, gated in CI. Prompt or pipeline changes that drop below this threshold do not ship.
Faithfulness
Per-claim grounding against the retrieved evidence. Below threshold the answer is regenerated or refused — not shown to the analyst.
Hallucination rate
The faithfulness gate plus the refusal contract removes the failure mode where a confident answer is invented. The system would rather say "the corpus does not contain this".
Every answer
Citations are the product. A response without at least one valid citation does not leave the layer. The UI links straight to the source page.
Filters before similarity
A current-fund question never retrieves a superseded prospectus. Effective-date metadata gates retrieval before the model sees a candidate.
Is the contract
The golden set is curated with the analyst team and runs in CI. Production thresholds are an agreement, not a marketing claim.
In investment documents, the citation is the product. An analyst cannot paste a paragraph of plausible-sounding summary into a client letter. They paste the quote and the page. We built the pipeline around that — citations are mandatory, faithfulness is gated, and the system refuses when the evidence is not there. That is what made the analysts actually use it.
Five decisions that came from treating the citation as the product.
1. Page citations are the product, not a feature. A response without a valid citation does not leave the layer. The analyst's workflow requires the page; the system requires it too.
2. Faithfulness is a gate, not a score. Per-claim grounding runs before the user sees the answer. Below threshold, the answer is regenerated or refused — not shown with a caveat.
3. Hybrid retrieval beats embedding-only on financial language. Sparse BM25 handles the exact-token cases where the prospectus uses one specific phrase and a paraphrase is not the same answer. Dense plus sparse plus rerank is the production default.
4. The evaluation set is the contract with the client. The golden set was curated with the analyst team. Production thresholds are an agreement. Prompt changes that miss them do not ship.
5. The refusal contract is what made the system trustworthy. An honest "the corpus does not contain this at this effective date" is what made the analyst paste the citation into a client letter. A confidently wrong answer would have killed the project.
Discovery → architecture → build → go live.
- Discovery
Real questions, real documents, real failure modes
Worked with the analyst team to characterise the actual questions, the document inventory, the effective-date model, the supersession history, and the golden evaluation set the system would be gated against.
- Architecture
Ingestion, retrieval, citation, and gate design
Designed structure-preserving ingestion, per-document metadata including effective dates, hybrid retrieval, the document grader, the typed response schema with mandatory citations, and the faithfulness gate.
- Build
Pipeline, evaluation harness, and refusal contract
Built the pipeline in the client's own cloud estate. Stood up the evaluation harness in CI with per-feature flags. Implemented the refusal contract and exposed it in the UI so the analyst sees "the corpus does not contain this" rather than a hallucination.
- Go live
Production with gated thresholds
Launched to the analyst team. Production thresholds (retrieval ≥ 0.96, faithfulness ≥ 0.92) gated in CI on every change. Monitoring on hallucination-grader drift and refusal-rate drift in operation.
Same engineering DNA, different problems.
We have already built the pipeline that gates faithfulness, not measures it.
Bring us your document inventory, your effective-date model, and the questions your analyst team is asked. We will come back with the architecture, the golden evaluation set we would gate against, and what it costs to ship and run.