Build: a retrieval harness that tells you what each choice is worth, or a contract-aware chunker

Ask ten RAG teams what their chunk size is worth and you’ll get ten confident answers and almost no measurements. The advice is easy to repeat. The number on your corpus is harder to come by.

This week’s posts gave the argument. Monday laid out the stack (structure-aware chunks, hybrid search, a reranker) and Tuesday’s paper showed gains that don’t add up when stacked. Today you build the thing that settles it for your own documents: a small harness that scores each choice.

flowchart TD
    G[Generate contracts + labelled questions] --> C[4 chunkings]
    C --> B[BM25]
    C --> D[Dense: tf-idf + SVD]
    B --> F[RRF k=60]
    D --> F
    F --> M[hit@5, nDCG@10 per query type]
    M --> T[Table: what each choice is worth]

Do it

Pick one option; the harness comes first because it also tests the chunker.

Option A: the retrieval harness

Goal

A script that takes a corpus, a set of chunkers, retrievers and a labelled question set, and prints the results table. The lab is the reference version.

Why it’s worth building

Every stage in the pipeline is a claim. A context header helps, hybrid search helps, a reranker helps. Tuesday’s study found that stacked gains overlap, and that the best pipeline changed with corpus size. You can only know your own numbers by measuring them.

Requirements

• A labelled set where each question points at a specific passage, not a document.
• Relevance judged by containment: a chunk counts only if it holds the whole answer sentence, so a window that cuts the sentence in half scores zero.
• Results broken out by query type (identifier lookup, party-named, paraphrase), never one blended average.
• A scale knob and a seed knob, so you can see results move.

Architecture

The lab’s single file holds the generator, four chunkers, a from-scratch BM25, a dense retriever (tf-idf over words and character trigrams, rank-48 SVD) standing in for an embedding model, RRF, and the scorer. The dense class is the one thing to replace with your real embedder.

Stack

Python 3.11+ and numpy 2.4.6. No GPU, no API key, no download.

Milestones

1. Run the reference (30 minutes).

cd labs/build/2026-10-31
pip install -r requirements.txt
python3 run.py --results results.json
python3 run.py --docs 144 --results r.json   # 4x corpus, about 40 seconds

2. Read the table, then predict (30 minutes). Before looking at the second run, write down what you expect hybrid to do at 4x. Then run it.
3. Swap in your embedder (45 minutes). Replace Dense with a real model’s vectors. Keep the interface (scores(query) returns one score per chunk).
4. Point it at your documents (45 minutes). Write 30 to 50 questions by hand, each with the passage that answers it. Thirty is enough to see a 0.4 gap; it isn’t enough to rank two close configurations.
5. Add a reranker stage (30 minutes). Retrieve 50, rerank to 5, and add a row. Check the latency per query next to the hit rate.

Stretch goals

• Add a context header written by a model instead of the template and price it with the calculator below.
• Add a second query type that cites a clause by number but not the party, and see which chunking survives.
• Bootstrap the question set to put a confidence interval on each cell.

Option B: a contract-aware chunker

Goal

A chunker that splits agreements on clause boundaries, keeps the heading path, and prepends a context line (parties, document ID, date). Option B is a spec: the lab’s make_chunks shows the clause and context logic.

Requirements and milestones

• Split on numbered headings; fall back to paragraphs when none exist.
• Keep tables as one chunk each, with the header row repeated.
• Emit {doc, heading_path, text, char_span} so a citation can point back to the source.
• Test: no chunk should cut an answer sentence in half. Count how many your chunker cuts.

Do the harness first if you can only do one: it is also how you’d test this chunker.

Expected results

Our run (Python 3.11, numpy 2.4.6, seed 7, 36 documents, 3.0 seconds) from run.json:

hit@5, by chunking and query type (hybrid):

Three things the numbers say, with the usual caveat that the corpus is synthetic and clean:

• The context line did most of the work. Without it, 36 near-identical agreements are indistinguishable at clause level. Fixed windows with context reached only 0.49 because they cut sentences; the chunker mattered once the header was there.
• Fusion isn’t free. At 36 documents hybrid sat between its two inputs. At 144 documents the dense stand-in fell to 0.10 on paraphrases and dragged hybrid to 0.14, below BM25’s 0.69. A real embedder should behave better; the harness is how you’d find out. Another seed (2) gave 0.44 for hybrid against 0.56 for BM25.
• Some questions aren’t retrieval questions. One query found the right clause for 4 of 12 chained questions (“the agreement that replaced MSA-xxxx”); reading the bridge ID from the first hit and querying again got all 12 (at 4x scale, 48 questions: 0.96 two-step against 0.08 one-shot). The auto-renewal count is a query over a table: the top 10 chunks held 10 of 28.

Threat model and failure modes

• Poisoned or hostile chunks. Retrieved text is untrusted input. A clause that says “ignore previous instructions” gets the same weight as any other once it’s in the prompt. Keep retrieved text in a data channel, strip or flag instruction-like text, and don’t let retrieval output trigger tools without a separate check.
• Cross-tenant leakage through context lines and caches. The header names parties. If chunks from different clients share an index, a missing filter returns another client’s contract. Filter on tenant and document ACL at query time, not after generation, and never share a semantic cache across tenants.
• Silent wrong answers from a clean-looking score. A hit@5 of 1.00 on identifier questions says nothing about paraphrase, and a stale eval set misses drift. Break results out by query type and rerun the set on every index rebuild.
• Fails safe or fails open. If retrieval returns nothing relevant, the system should abstain, not answer from memory. A harness can’t enforce that, but it can measure the rate at which the top result is wrong and the model answers anyway.
• Runaway cost. Model-written context lines multiply with corpus size and rebuild frequency. Cache by chunk hash and budget the re-index.

Alternatives and when to switch

Build your own

Copy this and fill it in. If you can’t fill in the last line, the project is too vague.

Goal (one sentence):
Which of this week's ideas it uses (chunking / context header / hybrid fusion / reranking / multi-hop):
Inputs and outputs:
Must-have (3 bullets max):
Nice-to-have:
Milestone 1 (done by hour 1):
Milestone 2 (done by hour 2):
Milestone 3 (done by hour 3):
How I'll know it works (a number or a test, not a feeling):

What to check before you trust any of this

• The corpus is generated from ten templates, so exact-ID lookups are unrealistically easy once context is present. Treat the paraphrase column as the more honest signal, and the whole table as a shape, not a benchmark.
• The dense retriever is tf-idf plus SVD. Conclusions about “dense” apply to it, not to a neural embedder.
• The multi-hop and aggregation sets are small (12 and 1 questions at default scale). Use the 144-document run for the multi-hop rate, and treat both as demonstrations.
• Anthropic’s numbers are from September 2024 on its own data. The Tuesday paper (arXiv 2610.29722, 21 October 2026) is the reason to expect your stack to compose worse than the table suggests.
• The full program and our recorded run are in the lab folder.

Evidence

A first table with real numbers beats a perfect harness you meant to write. If you only get as far as the first run, that’s a good Saturday, and the rest will keep for next weekend.