ReCAP: compacting agent memory from attention you already paid for

Here’s a number from the paper’s introduction that should bother you. In about 4,300 Claude Code and Codex sessions (a measurement the authors cite from other work), user think time was 92% of session time, and requests after hour-long gaps almost always missed the serving cache.

So the typical coding-agent turn doesn’t resume a warm session. It resumes a cold one. You’re re-encoding the history from scratch, and if the history is too long, you run a summarization call first and wait for it, before the model can even start on the user’s request.

A paper from 30 September, Persistent Context Graphs for Efficient Memory Compaction in LLM Agents (UCSB and Microsoft), asks a neat question about that situation. Every forward pass already computes attention over the history. We throw it away. What if we kept it?

The problem with the usual compaction

There are two common ways to shrink agent history. Summarize it with the model (what Codex’s default compaction does), or evict entries from the KV cache.

Both have a timing problem. Compaction runs before the next request exists, so it has to guess what the next request will need. Query-aware KV eviction fixes the guess but only works while the cache is alive, and it drops entries permanently. Summarization survives cache expiry, but it costs a model call, loses detail, and can’t bring anything back.

The authors frame it as three linked challenges: what to keep depends on a request you haven’t seen, predicting that costs extra compute, and cache expiry makes reconstruction expensive.

The idea

ReCAP keeps a persistent context graph next to the full, untouched history archive.

• Nodes are atomic blocks: a user message, an assistant message with its tool calls and results (kept together so the sequence stays valid), or a plain assistant message. Each carries an importance score in [0, 1].
• Edges point from a supporting block to a dependent one, built from attention mass the dependent block puts on earlier blocks.

Both come from attention sampled during normal inference, updated on CPU in the background. When a new request arrives, selection is graph traversal plus string matching. No model call, no GPU.

Why it should work: three observations

The authors measured before building, and I like that. On coding-agent sessions:

1. Attention is concentrated. Ranking blocks by attention per token, a 10% token budget captured 65% of the attention mass in their example session.
2. Past attention predicts reuse. Over 64 turn transitions from eight sessions, ranking by the previous turn’s attention reached a median identifier coverage of about 0.68, against 0.71 for an oracle that peeks at the next turn.
3. The request redirects attention. Changing the target file moved 10.1% of attention on average, versus 5.3% for a paraphrase.

So history importance is a stable prior, and the new request is a cheap correction on top. The design follows from that.

The one equation worth keeping

Importance is an exponential moving average of how much attention each block receives per turn (normalised by the turn’s max). At request time, each block gets:

s(v) = z(u(v)) + λ·z(ℓ(v)) + ε·ξ·[v never observed] − κ·[v stale]

u is stored importance. ℓ is an IDF-weighted overlap between identifiers in the request (file names, function names, error symbols) and the block. The other two terms give unmeasured blocks an exploration bonus and penalise blocks that a later, more-attended block seems to supersede.

Then: start from a protected set (all user messages, plus the latest edit per modified file), fill the budget in score order, and add one-hop predecessors through dependency edges. That last step is where a diagnosis pulls in the search result it relied on, even if that result scored low alone.

A detail I’d copy regardless: the selected prefix stays fixed across tool steps within a turn, so prefix caching still works.

Results, and what to read carefully

Setup: Qwen3-Coder-30B-A3B and gpt-oss-20b on SGLang; 106 SWE-Together tasks (GPT-5.4 plays the user) and 100 Lost-in-Conversation code instances. Retention budget fixed at 10% of history. Six compaction baselines including Codex-style summarization and KV eviction.

• Latency: compaction plus cold restoration estimated about 95% lower than summarization on Qwen3-Coder (94.5–95.6% across the methods compared there). On gpt-oss the reductions were 76.2–94.9%.
• SWE-Together, Qwen3-Coder: 28.5 reward at 15,344 mean context tokens, against 28.7 for summarization at 5,239 and 24.4 for full history at 28,504. So: roughly half the context of full history, quality comparable to summarization, not better than it. On gpt-oss, scores were low overall (8.2–9.0), so the gaps there are small.
• Lost-in-Conversation: 85.4 and 67.6 accuracy, versus 65.6 and 26.4 for full history (the 19.8 and 41.2-point gains in the abstract), and 75.4 and 36.8 for summarization.
• State: the graph is about 19 KB per session in the largest case.
• Ablation: removing the attention graph dropped LiC accuracy by 8.8 points.
• Revival: on SWE-Together with Qwen3-Coder, 70% of turn transitions brought back at least one previously omitted block.

What I’d check before believing the headline:

• Latencies are estimated (measured components on one H100, combined), not end-to-end production traces.
• LiC contexts are tiny (a few hundred tokens), so those gains tell you that full history hurts there, not how this scales to a 200k-token session.
• Everything is one family of coding tasks with two open-weight models. A hosted API doesn’t expose attention, but §5.6 of the paper shows a workaround: a small open model (Qwen3-0.6B) replays the context and supplies the attention, scoring within about half a point of the agent’s own attention, even for a different model family.

What it means if you design agent platforms

The practical catch is that last point. ReCAP is simplest where you control the serving stack (vLLM, SGLang). With a hosted API you pay for a second, small model that replays each context to produce the attention, which adds a little latency and infrastructure. If you only call hosted APIs, the transferable ideas are still useful: persist usage and dependency metadata outside the cache, keep the archive lossless, select per request rather than per compaction, and pair tool calls with results.

The bigger design shift is treating compaction as a query-time read over a lossless store instead of a write-time rewrite. Summaries destroy information on a guess. A graph lets the guess be wrong and recoverable.

Two things I’d watch in production: adversarial or odd inputs (attention is a proxy for importance, and the authors assume it holds), and identifier-overlap relevance, which suits code and will be weaker for prose-heavy tasks.

Links

• Paper: arXiv 2609.40118
• Code: the paper points to UCSB-NLP-Chang/ReCAP. As of this post, the repository was empty, so check back before planning around it.

38-minute exercise: measure your own cold-restart tax

No GPU needed.

1. Pull 20 sessions from your agent logs. For each user turn, record the gap since the previous turn.
2. Mark turns with a gap longer than your provider’s cache TTL as cold. What fraction are cold?
3. For cold turns, estimate prefill tokens and your current compaction cost (summary call tokens plus latency).
4. Prototype the cheap half of ReCAP: split history into tool-call-plus-result blocks, score each by IDF overlap with the new request, protect user messages and the latest edit per file, fill 10% of budget, add one-hop predecessors by shared identifiers. Compare against your current summary on five replayed turns.

If fewer than a few percent of your turns are cold, you’ve learned something useful: this problem isn’t yours yet.

Ship the imperfect version of the experiment; a half-finished log analysis still tells you more than none, and skipping a day costs nothing.

ReCAP: compacting agent memory from attention you already paid for

A 30 September paper keeps the attention an agent computes anyway as a tiny graph, then picks context for the next request with no model call, cutting compaction latency by about 95% against Codex-style summarization.

• A new paper stores per-block attention scores and dependency links in a ~19 KB graph, then selects context for the next user request with zero extra model calls.
• Worth knowing if you run long-lived coding or ops agents where the KV cache expires between turns.

Full post and sources:
https://thevoyager.dev/deep-dives/research/2026-10-06/

#AI #Agents #ContextEngineering #Memory