A knowledge assistant over policies and contracts: the retrieval design a professional-services firm can defend

Ask a vector index “which of our licensing agreements have expired under California law?” and it will hand you a handful of well-written clauses about expiry and California. Some will belong to agreements that are still in force. Some will be about New York.

AWS published a worked example of exactly this in August 2026, on the CUAD contract dataset. With plain semantic search over a pool of 55 candidate chunks, the right agreement turned up but its renewal terms were incomplete. Contract questions are mostly filter problems wearing a language problem’s clothes.

This post designs the whole assistant for a professional-services firm, from business case to delivery plan, with retrieval as the lens. It builds on Monday’s retrieval post (hybrid search, contextual chunks, reranking) and Tuesday’s study of what composes (stacked gains overlap, so measure the whole pipeline).

The business case

The organisation is a mid-sized professional-services firm (an assumed profile): about 6,000 staff across 14 offices, in advisory, tax and legal-adjacent work. The knowledge it sells lives in three places: client contracts and engagement letters, internal policies (independence, conflicts, data handling, travel, procurement), and past work product.

The problem: people ask the same questions all day. “Can we accept this liability cap?” “What does our retention policy say about client working papers?” “Which clients have audit rights over subcontractors?” Answers come from a handful of risk and legal specialists, with a median wait that the firm measures in days.

Users are consultants and managers (the askers) and about 40 risk and legal reviewers (the experts the assistant protects). The KPI we pick, as an assumption for this design: 70% of policy questions answered with a cited source in under 20 seconds, with fewer than 2% of cited answers judged wrong by a reviewer sample.

Constraints:

• Budget: a pilot under USD 150,000 all-in, then a run cost the firm can allocate per head.
• Regulation: client confidentiality (engagement letters often forbid sharing across clients), GDPR for EU staff data, and local rules on data residency for Gulf and EU clients.
• Existing systems: a document management system (DMS), a contract repository, SharePoint-style policy sites, and one identity provider.
• Team: five engineers, one ML engineer, no search specialist.

Stakeholders who sign off: the General Counsel (accuracy and privilege), the CISO (access control, residency), the head of risk (conflicts and independence rules), the CFO (run cost), and the practice leaders whose clients’ documents sit in the index.

Do we need AI here?

The simplest option is keyword search over the DMS plus a tidy intranet page of policies. For half the questions it works, and we should say so. “Where is the travel policy?” is a search problem.

It falls short on three things. Questions that span documents (“which agreements with this client allow subcontracting?”) need an answer, not ten PDFs. Questions phrased in the user’s words rarely match the policy’s words (“can I take a client dinner” versus “gifts and hospitality”). And the specialists’ queue is the actual cost.

So the design keeps keyword search as one leg of retrieval and uses the model only to compose a cited answer, never to decide a legal position.

Requirements

Functional:

• Answer natural-language questions over policies and contracts, with a quote and a link to the exact clause and version.
• Respect each user’s access rights, matter walls (ethical walls) and client confidentiality.
• Say “I can’t find this” and route to a human reviewer, with the retrieved evidence attached.
• Support filters users actually think in: client, contract type, governing law, effective and expiry dates, jurisdiction, policy version.
• Keep the index fresh: a changed policy is searchable within an hour; a superseded one stops being returned.

Non-functional requirements (assumptions for this design):

High-level design

The diagram has four areas. Solid arrows are the answer path; the ingestion path runs on the left of the data area.

flowchart TD
    subgraph AI[AI engineering]
        ORCH[Orchestrator: query rewrite and filter extraction]
        RET[Retriever: filter, hybrid search, rerank]
        GEN[Answer model: cited answer or refusal]
        EVAL[Eval harness and judge]
    end
    subgraph DATA[Data engineering]
        SRC[DMS, contract repo, policy sites]
        ING[Ingestion: parse, chunk, contextualise, tag metadata]
        IDX[(Hybrid index: vectors, BM25, metadata, ACLs)]
        LIN[Lineage and version registry]
    end
    subgraph PLAT[Platform and deployment]
        API[API gateway and chat UI]
        OBS[Tracing, cost and quality dashboards]
        CICD[CI/CD: index builds, eval gates]
    end
    subgraph SEC[Security and governance]
        IDP[Identity provider and group sync]
        ACL[ACL and matter-wall service]
        GR[Input and output guardrails]
        AUD[Audit log]
    end
    API --> IDP
    API --> ORCH
    ORCH --> RET
    RET --> IDX
    RET --> ACL
    RET --> GEN
    GEN --> GR
    GR --> API
    SRC --> ING --> IDX
    ING --> LIN
    ACL --> IDX
    API --> AUD
    EVAL --> RET
    CICD --> ING
    OBS --> API

If you need it smaller for a deck, split it: answer path on one slide, ingestion and governance on another.

AI engineering. The orchestrator turns a question into filters plus a keyword query and a semantic query. The retriever runs the filtered hybrid search and the reranker. The answer model sees only the surviving chunks, treated as data. The eval harness replays the labelled set against any change to chunking, models or prompts, and blocks the release if recall or faithfulness drops.

Data engineering. Ingestion parses layout, splits by clause and heading, generates a short context line per chunk, and extracts document-level metadata. A version registry records which document version each chunk came from, so a superseded policy can be withdrawn in one operation and every citation can name its version.

Platform. One API gateway fronts the chat UI and any other client. Index builds and prompt changes ship through CI with the eval set as a gate. Dashboards show spend, latency per stage and retrieval health.

Security and governance. The identity provider is the only source of who a user is. The ACL and matter-wall service resolves a user to groups and is called on every query. Guardrails screen inputs and outputs, and the audit log records who asked what and which documents were cited.

Current state and fit. Today, staff search the DMS (keyword only) and email specialists. The new system sits beside both. It never copies documents out of the DMS as the source of truth: it indexes text, metadata and ACL snapshots, and every answer links back to the DMS. A nightly job reconciles the ACL snapshot with the identity provider, and the retriever re-checks permissions at query time (the deep dive explains why both).

Components and alternatives

The choice readers most often get wrong is the retrieval store. Hybrid search is not available in every managed vector option: on Bedrock Knowledge Bases, the documentation says hybrid search is supported for Amazon RDS, OpenSearch Serverless and MongoDB stores that have a filterable text field, and other stores fall back to semantic search (docs read 28 October 2026). Pick the store for the retrieval features you need, not for the one your cloud’s wizard offers first.

Deep dive: retrieval you can defend

Concept recap. Retrieval-augmented generation fetches evidence from your corpus and makes the model answer from it. In a contract and policy corpus the hard part is not finding text similar to the question. It is finding text from the right document, the right version, that this user may see. Dense search finds similar wording across thousands of near-identical agreements. Filters, keyword matches on defined terms, and a reranker turn “similar” into “relevant”. Monday’s post covers the building blocks; this section applies them.

Data model

Each chunk carries:

{
  "chunk_id": "c-9f31",
  "doc_id": "msa-acme-2024-03",
  "doc_version": 3,
  "status": "in_force",
  "doc_type": "master_services_agreement",
  "client_id": "acme",
  "governing_law": "CA-US",
  "effective_date": "2024-03-01",
  "expiry_date": "2027-02-28",
  "section_path": "12 > 12.3 Limitation of liability",
  "acl_groups": ["matter-4471", "risk-reviewers"],
  "context": "This clause is from the Acme master services agreement ...",
  "text": "...",
  "embedding": [0.01, "..."]
}

Two design points. First, context is a short generated prefix situating the chunk in its document, the technique Anthropic calls contextual retrieval; it is embedded and BM25-indexed with the chunk. Second, document-level metadata is stored once per document and joined at query time where the store allows it, because repeating it on every chunk costs index size and tokens (AIDA’s post makes the same point about a 10 KB metadata file per document).

Query contract

The API is deliberately boring:

POST /v1/ask
{ "question": "Which Acme agreements allow subcontracting?",
  "filters": { "client_id": "acme" },
  "mode": "answer" }

The response returns answer, an array of citations (each with doc_id, doc_version, section_path, a quoted span and a DMS link), a confidence band, and trace_id. mode can be search_only, which returns ranked passages with no model call. That mode is your fallback when the model is down and a feature in its own right: a lot of lawyers want the clauses, not the paragraph.

sequenceDiagram
    participant U as User
    participant G as API gateway
    participant O as Orchestrator
    participant R as Retriever
    participant X as Index
    participant M as Answer model
    U->>G: question (token)
    G->>G: validate token, load groups
    G->>O: question, user id, groups
    O->>R: keyword query, semantic query, filters
    R->>X: filtered hybrid search + group filter
    X-->>R: top 100 candidates
    R->>R: rerank, keep 8 to 10
    R-->>O: chunks with versions and quotes
    O->>M: question + chunks (data, not instructions)
    M-->>O: cited answer or refusal
    O->>O: check every quote exists in a retrieved chunk
    O-->>U: answer, citations, trace_id

Retrieval, in the order it runs

1. Filter first. Client, document type, status, jurisdiction and dates are structured. Extract them from the question with a small model call (Bedrock Knowledge Bases offers this as implicit metadata filtering for Anthropic models; the AIDA post found it works best combined with explicit application-level filters). Where a filter came from a model guess, show it to the user as a chip they can remove.
2. Hybrid search over the survivors. BM25 catches defined terms and clause numbers (“Schedule 3”, “Permitted Subcontractor”); vectors catch paraphrase. Fuse ranks with reciprocal rank fusion (the common default is k = 60) because scores from the two methods aren’t comparable.
3. Rerank. Anthropic’s published test used the top 150 candidates and kept the top 20. A cross-encoder reads question and chunk together, which is why it fixes near-duplicates that embeddings tie.
4. Expand to the parent. Retrieve small chunks; hand the model the clause plus its heading path, because a limitation-of-liability sentence means little without its carve-outs.
5. Verify. Before returning, check that each quoted span exists verbatim in a retrieved chunk. A citation that points at nothing is the failure users remember.

What plain RAG will not do

Three question shapes need something else, and the design says so up front:

• Aggregation (“how many contracts auto-renew?”) is a query over extracted fields, not a retrieval. Extract those fields at ingest into a table and answer with SQL.
• Multi-hop (“does the parent agreement’s cap apply to this SOW?”) needs two retrievals chained. Allow a bounded second hop in the orchestrator.
• Freshness: a superseded policy must disappear. That is status and doc_version in the filter, plus an index update path measured in minutes, not a nightly rebuild.

Failure handling and observability

Retrieval returns nothing: say so and route to a reviewer with the query attached. Reranker times out: serve fused results unreranked and flag the trace. Model call fails: return search_only results. Trace every request with the filters applied, candidate counts before and after each stage, rerank scores and the final chunks. When an answer is wrong, the first question is which stage lost the right chunk, and the trace should answer it in one look.

Threat model and failure modes

The system fails safe in one direction on purpose: when evidence is thin, it returns passages and a referral, not a confident paragraph.

A recent example of the first three rows. In March 2026 the security firm CodeWall published how its autonomous agent got into McKinsey’s internal AI platform, Lilli. Their account: the API documentation listed over 200 endpoints, 22 needed no authentication, and one wrote user search queries to the database with values parameterised but JSON keys concatenated into SQL. Reachable data included, by their count, 46.5 million chat messages, 3.68 million RAG document chunks and 95 model configurations including system prompts, which they say were writable. McKinsey patched the unauthenticated endpoints within a day of disclosure, per CodeWall. These are CodeWall’s figures, and McKinsey’s own account may differ. The lesson for this design is ordinary: the model was not the weak point, the plumbing around it was. Treat the retrieval stack as an application that holds your most sensitive documents, because it does.

Key decisions

ADR 1: Filter by metadata before searching, and extract it at ingest

• Decision: Every document gets structured metadata (type, client, status, governing law, dates) at ingest; queries apply filters inside the search call.
• Options: (a) rely on semantic search and a stronger model; (b) filter by metadata; (c) train a classifier per query.
• Why this one: AIDA’s test is the evidence. With top-k 15 on a question about expired California licensing agreements, baseline RAG scored 27.3% coverage over 55 candidates, explicit filters alone 37.5% over 40 (and misclassified one agreement as expired), and implicit plus explicit filters 75% over 20. Filters shrink the haystack before anything is ranked.
• What would change it: a corpus with little structure (free-form memos), where metadata extraction costs more than it saves, or extraction accuracy below your tolerance on the eval set.

ADR 2: Hybrid search with a reranker, adopted only if the labelled set says so

• Decision: BM25 plus vectors with RRF, then a reranker on the top 50 to 150.
• Options: vectors only; hybrid; hybrid plus reranker; long context with no retrieval.
• Why this one: Anthropic’s own test reports failed top-20 retrievals falling from 5.7% to 2.9% with contextual embeddings plus contextual BM25, and to 1.9% with reranking. Tuesday’s study is the caution: gains from stacked components overlapped, and the best pipeline changed with corpus size. So each stage must earn its latency on our labelled set.
• What would change it: under about 200,000 tokens of total knowledge (Anthropic’s guidance), skip retrieval and put the corpus in a cached prompt. For us the corpus is far larger, so retrieval stays.

ADR 3: Enforce access in the query, and re-check on the way out

• Decision: Group filters are part of the index query; cited documents are re-authorised against the DMS before the answer is shown.
• Options: filter in the prompt (“only use documents the user can see”); post-filter results; pre-filter in the index plus a final check.
• Why this one: a prompt instruction is not an access control. Post-filtering after ranking leaks through counts, rank positions and truncated result sets. Pre-filtering plus a final check costs one extra permission call per cited document.
• What would change it: none for the principle. The mechanism changes with the store: some stores make per-document filters cheap, others make them slow at scale, which is a reason to test before choosing.

ADR 4: Build vs buy: managed search layer, self-built orchestration

• Decision: Use a managed search service for the index and reranker; write the orchestrator (about 600 lines) ourselves.
• Options: all managed knowledge-base product; all self-built on open source; the split above.
• Why this one: index operations are undifferentiated work for a team of six. The orchestration is where the firm’s rules live (matter walls, refusal behaviour), and a managed wizard hides exactly the settings we need to test.
• Lock-in and exit plan: the index is rebuildable from source in under a day at this scale (assumption to test in the pilot). Keep embeddings, chunk text and metadata exportable, and keep the orchestrator behind a retrieval interface with two implementations from week one. Exit cost is a re-embedding run, not a rewrite.
• What would change it: a search specialist on the team and a corpus past tens of millions of chunks, where self-managed tuning pays.

How we know it works

Offline set. Build 300 questions from three sources: real questions from the specialists’ inbox (the best source), questions written from policies by reviewers, and adversarial ones (superseded versions, near-duplicate clients, questions with no answer in the corpus). For each, a reviewer marks the passages that answer it. About 15% should be unanswerable on purpose.

Retrieval metrics. Recall at 10 and nDCG at 10, reported separately for filtered and unfiltered queries. A stage that doesn’t move these doesn’t ship. Re-run the ablation whenever another stage changes, because of the overlap Tuesday’s study found.

Answer metrics. Faithfulness (every claim supported by a cited chunk), citation precision, correct refusals on unanswerable questions, and a reviewer score on a weekly sample of 50. Calibrate any LLM judge against reviewer labels before you trust it; next week’s post covers how.

Online. Thumbs, escalation rate, “answer opened in DMS” rate, and the share of answers with a removed filter chip.

Go-live threshold (proposal). On the offline set: recall at 10 of at least 0.85 on answerable questions, faithfulness of at least 0.95, correct refusal on at least 90% of unanswerable ones, and zero cross-client leaks in a red-team run of 200 attempts. Set the numbers with the risk lead; the point is that they are written down before the pilot.

Sizing, cost and value

The calculator uses list prices checked on 28 October 2026: Claude Sonnet 5.5 at USD 2 per million input tokens and USD 10 per million output tokens (Claude pricing), and a hosted Cohere Rerank 3.5 at USD 2.00 per 1,000 queries on Amazon Bedrock (Bedrock pricing). Search infrastructure, parsing and engineering time are not included, and your negotiated prices will differ. Token counts and the value inputs are assumptions.

At the defaults, model and rerank fees come to about USD 0.019 per answer (USD 0.017 model, USD 0.002 rerank), around USD 1,250 a month, against roughly USD 594,000 of staff time a month if every answer really saves six minutes, which is an optimistic assumption: halve it and the case still holds. Reviewer fixes at a 5% error share cost about USD 74,000 a month in the same units. The ratio is the point: model fees are small next to the value, and next to the cost of a wrong answer. Spend the effort on quality, not on shaving tokens.

One-time ingestion matters too. Anthropic estimates contextual chunking at USD 1.02 per million document tokens, one time, with prompt caching (as of their post; prices have moved since, so recompute from the current pricing page). At 100 million document tokens, that is roughly USD 100 plus embeddings, so it’s cheap enough to redo when you change the chunking.

From design to delivery

Proof of concept (4 weeks). One practice area, 2,000 documents, no ACL complexity. Build the labelled set first (week 1), then the pipeline, then ablate each stage. Exit criterion: the offline numbers above on that slice.

Pilot (8 weeks). 300 users in two offices, real ACLs and matter walls, search-only mode on by default with answers behind a button. Run the red-team. Weekly reviewer sampling.

Production (rolling, 3 months). Remaining offices, residency-specific indexes, the aggregation path for extracted fields, on-call.

Team and roles. A product owner from risk, a tech lead, two backend engineers, one ML engineer for evals and retrieval, a security engineer part time, and three reviewers from the legal team for about four hours a week each. Reviewers are the scarce role; schedule them before you schedule the code.

Top risks.

1. Document quality: scanned or badly OCR’d contracts poison retrieval. Mitigation: measure parse quality per document, quarantine low scorers, show a “text may be incomplete” badge.
2. Permissions drift: DMS groups change faster than the ACL snapshot. Mitigation: query-time re-check, nightly reconciliation, alert on mismatches.
3. Adoption by the wrong users: juniors treating answers as final. Mitigation: confidence bands, mandatory reviewer route for contract positions, training that shows failures.

Operating model. SLOs: 99.5% availability in business hours, p95 latency under 20 seconds, correct refusal rate on the weekly sample. Alerts on daily spend, empty-retrieval rate and rerank timeouts. Index builds go through CI with the eval set as the gate. Users are brought on with a 20-minute session built around three failed examples, because people who’ve seen it refuse once trust it more.

How others built it

Three reference points, labelled by how much they tell you.

AWS and AIDA (reference architecture, August 2026). AWS’s contract search post describes AIDA, a solution for entertainment and media companies with thousands of contracts across jurisdictions: Bedrock Knowledge Bases over OpenSearch or S3 Vectors, metadata files of up to 10 KB per document (type, governing law, dates, parties, confidentiality level), implicit and explicit filters, Bedrock Guardrails against prompt injection and PII leaks, IAM and CloudWatch for access and audit. The numbers in ADR 1 come from it. It is AWS’s own reference solution on a public dataset, not a customer deployment with production results, and it says outputs need review by legal professionals.

McKinsey’s Lilli (named organisation, security write-up, March 2026). CodeWall’s disclosure is a third party’s account of a professional-services firm’s internal assistant: tens of thousands of users, hundreds of thousands of files, millions of RAG chunks. It tells you scale and what went wrong, not the retrieval design. McKinsey’s own account of how Lilli retrieves is not described here: this post relies only on CodeWall’s write-up, and the firm has not published a retrieval-stack write-up that we have read.

Anthropic’s contextual retrieval (primary, 2024). The post is a vendor’s own test across several datasets; older than twelve months, but still the most-cited ablation of the stack used here. Treat its percentages as a starting hypothesis for your corpus.

No firm in this sector has published production numbers (accuracy, adoption, cost) that we could verify. The vendor case studies we found report review-time reductions of 80% and 85% without methodology, so we don’t use them.

Explain it in 3 sentences

To the CTO: we index policies and contracts with structured metadata, search with filters, keywords and vectors, rerank, and let a model write a cited answer only from what survived, with access checked inside the query and again on the way out. We prove it on a 300-question reviewer-labelled set before anyone outside the pilot sees it. Model fees are around a cent or two per answer on our assumptions, so the spend that matters is reviewer time and security work.

To the business owner: staff get a sourced answer in seconds instead of a specialist’s reply in days, and when the system isn’t sure it says so and hands over the evidence.

Exercise: find the flaws

Here is a draft design a team presented. Spend 30 minutes finding what is wrong with it.

We chunk every PDF into 1,000-token pieces and embed them with a general-purpose model. A user’s question goes straight to vector search (top 5). The system prompt tells the model: “Only use documents the user has permission to see. Never reveal confidential client information.” The top 5 chunks go into the prompt and the model answers. We log every question and answer in plaintext for debugging. Policies are re-indexed every Sunday. We will measure success by user thumbs-up. The orchestration framework is a single autonomous agent with web search enabled, so it can fill gaps.

Template: risk register

Copy this and fill it in for your own retrieval system; keep it next to the eval set. Score likelihood and impact from 1 to 5, give every risk one named owner, and review the register at each release, not once a year. The rows below are this design’s starting set: delete the ones that don’t apply and add what your own traces show.

| ID | Risk | Likelihood (1-5) | Impact (1-5) | Early signal | Mitigation | Owner | Review date |
|----|------|------------------|--------------|--------------|------------|-------|-------------|
| R1 | Cross-client leakage via stale ACL snapshot | | | ACL mismatch alerts | Query-time group filter; nightly reconcile | CISO | |
| R2 | Superseded document returned as current | | | Reviewer flags version errors | Status filter; version shown on citations | Data lead | |
| R3 | Injection through a retrieved document | | | Output-quote check failures | Data-not-instructions prompt; no write tools | Tech lead | |
| R4 | Parse failures on scans | | | Low parse-quality scores | Quarantine and badge | Data lead | |
| R5 | Cost spike from retries or long inputs | | | Daily spend alarm | Rate limits; context caps | Platform | |
| R6 | Reviewers overloaded | | | Queue age | Search-only default; sampling | Product | |

Evidence

Ship the small version first. A search-only mode that people trust beats an answer box they double-check, and a week you skipped is simply a week the corpus got older.