Principal/Staff Architect, project deep-dive: your retrieval stopped improving. Walk the panel through why

Here is a number that should make you uncomfortable. In Meta’s CRAG benchmark, GPT-4 Turbo answered 33.5% of questions correctly with no retrieval at all. Give it straightforward RAG with up to 50 web pages and that rises to 43.6%. Truthfulness, which is accuracy minus hallucination, goes the other way: 20.0 down to 13.4.

Retrieval made the system more often right and, by that score, worse. A project deep-dive is where a panel finds out whether you saw that coming.

This one is written in the style of what a Meta loop might ask a Principal or Staff Architect. It isn’t a reported question, and I haven’t checked any company’s current interview format. It builds on this week’s retrieval building block, Tuesday’s composition study and Wednesday’s knowledge-assistant design.

The question

“Tell me about a retrieval-augmented system you owned that stopped improving. Walk us through the architecture, how you measured it, why it plateaued, and what you’d change. Then tell us what you would have removed.”

Project deep-dives are about ownership. You’d answer from your own history. The model answer below is a composite built from published numbers, so it has no personal anecdote in it, and yours shouldn’t borrow one.

Pause: answer out loud for 10 minutes first

Structure it as: system and metric, diagnosis, the change you’d make, and what you’d cut.

A framework that works here

1. Name the metric you were actually optimising. Panels at this level listen for whether “quality” meant recall@k, answer accuracy or something users trusted. If the headline number is retrieval-only, say so and say what it doesn’t cover.

2. Triage every wrong answer into one of three bins.

flowchart TD
  W[Wrong answer] --> Q{Was the needed fact<br/>in the context shown to the model?}
  Q -- No, and it exists in the corpus --> A[Retrieval miss<br/>chunking, query, candidate pool]
  Q -- No, and it is not in the corpus --> B[Insufficient context<br/>should have abstained]
  Q -- Yes --> C[Model misuse<br/>distraction, conflict, truncation]

The bins matter because the fixes don’t overlap. Reranker tuning does nothing for a question the corpus can’t answer.

3. Show that the middle bin is large. Google’s Sufficient Context paper labelled whether retrieved context contained enough to answer. On Musique, 55.4% of contexts were insufficient. With insufficient context on a human-labelled set, models still hallucinated often: Gemini 1.5 Pro 40.4%, Claude 3.5 Sonnet 36.5%, GPT-4o 15.4%, while abstaining 50.0%, 53.8% and 61.5% of the time. Those are 2024 models and a curated set of 45.2% insufficient cases, so treat them as an illustration of the pattern, not a ranking.

4. Count the cost of wrong answers. CRAG scores an answer as accurate (+1), missing (0) or hallucinated (-1), so a system that abstains beats one that guesses at the same accuracy. Try it on your own figures:

5. Don’t stack components on faith. Dell’s engineers published a deployment study of RAG Fusion, which rewrites a query, retrieves for both and merges with reciprocal rank fusion. Tested at a fixed budget of 10 chunks on 115 synthetic queries, Hit@10 fell from 51.3% to 47.8% for the first prompt. Top-3 moved by up to 4.35 points, which is three to five queries, and none of the differences was statistically significant after correction. Their explanation is that extra candidates are cut by reranking and truncation, and many are near-duplicates. Tuesday’s study found the same shape at larger scale: stacked gains overlap.

A model answer, compressed

What top candidates add at this level

• A number against their own design. They say what they’d remove and why, with a figure, instead of defending every component.
• Metric hygiene. They separate retrieval metrics (recall@k, nDCG) from answer metrics, and say the correlation between them has to be measured, not assumed.
• Cost of a wrong answer. They ask what a confident error costs the business and set the abstain threshold from that, not from a benchmark.
• Staleness. They name when the ablation expires: corpus growth, model swap, embedding change.
• Ownership. They talk about who gets paged when answer quality drops, and which dashboard shows it first.

One tip

Pick one thing from your own project to cut and say it early. Panels at this level trust a candidate who volunteers a mistake more than one who has a defence for everything.

Threat model and failure modes

• Prompt injection through retrieved chunks. A document containing instructions can steer the answer or fake its own sufficiency. Treat chunks as data in a delimited block, give the retrieval stage no write tools, and test with planted instructions in the eval set.
• Confident answers on insufficient context. The measured failure above. Fail safe: abstain with the sources searched, route to a human, and track the abstain rate so it doesn’t quietly fall to zero.
• Permission leaks through shared indexes. If tenants or roles share an index, a chunk from another tenant can enter the context and the abstain logic can’t help. Filter at query time and test the filter with a canary document.
• Eval-set data exposure. Labelled sets built from real traffic carry PII and answers to sensitive questions. Redact before storing and restrict access like production.
• Cost and latency creep. Multi-query steps multiply retrieval calls. In Dell’s run, retrieval for Q1 plus Q2 averaged 66.0 s against 54.6 s for Q1 alone, in their setup. Cap query fan-out and fall back to the single-query path on timeout.

Alternatives and when to switch

Numbers

The sources are mostly from 2024 and 2025. Newer work on the theme is in Tuesday’s post (21 October 2026), and the 2024 models in these tables would behave differently today, so re-measure rather than quote them.

Ten-minute practice

Take 20 recent wrong answers from any RAG system you know. Put each in one of the three bins and write the count. If the “not in the corpus” bin is the biggest, your next sprint is an abstain path, not a better retriever.

Evidence

Pick one idea from this and try it on something you already run. If it slips to next week, the question will keep.