Skip to content

Deep DivesResearch

Your multi-agent system beat a single call. Did it beat five single calls for the same tokens?

A meta-analysis of 516 multi-agent configurations from nine studies finds that against baselines matched on samples or budget, multi-agent accuracy averages 4.2 points lower, and it proposes a six-part resource vector and a baseline ladder to report comparisons honestly.

10 min read · 35 min hands-onWeek 7 · Ingestion and document pipelines
On this page
  1. What the paper actually is
  2. What they found
  3. What the paper claims vs what we checked
  4. Limits worth knowing
  5. What to do with it
  6. Docs vs code
  7. Threat model and failure modes
  8. Alternatives and when to switch
  9. Numbers
  10. Try it in 10 minutes
  11. Evidence

Every multi-agent paper has the same figure: a bar for “single agent”, a taller bar for “our multi-agent system”. The single-agent bar is usually one greedy pass. The multi-agent bar usually used five to fifteen times the tokens, several samples and a voting step.

So what did the architecture buy you? The coordination, or the compute? A paper submitted on 13 November 2026 asks that across nine published studies and gets an uncomfortable answer. It’s this week’s landmark pick (rule 1): it changes how multi-agent systems get justified, and it has a code repository. It’s off this week’s ingestion theme (Monday’s post), but the habit it teaches applies to every pipeline decision, including whether a model should parse your documents at all.

The paper is More Agents, More Samples, or More Thinking? by Seyed Mahmoud Sajjadi Mohammadabadi of the University of Nevada, Reno. One author, one preprint, 43 pages, not yet peer reviewed.

What the paper actually is

Not a new method and not a new experiment. It’s a meta-analysis: the author pulled published results tables from nine studies (516 unique multi-agent configurations) and re-organised them. The contribution is a reporting protocol called PACE, Parity-Aware Compute Evaluation.

It has two parts:

  • A resource vector with six dimensions: reasoning tokens, total tokens (including echoed histories and system prompts), cost, latency, independent samples, and tool or environment calls. Each gets a status: strictly controlled, approximately matched, unmatched, or unreported.
  • A baseline ladder. A is one greedy pass. B is k independent samples with a vote, no communication. C is a baseline matched on a declared resource. D is a Pareto frontier over budgets.

The claim is simple: a comparison against rung A can’t separate coordination from extra computation.

What they found

-0.042mean accuracy difference, multi-agent minus baseline, 474 sample- or budget-matched contrasts
74%of those contrasts where multi-agent ties or loses
8 of 9studies with a nonpositive mean difference
-0.038shift when the baseline moves from greedy to sampled

The headline table, from the paper’s synthesis (mean accuracy difference, multi-agent minus baseline):

Multi-agent minus baseline, by baseline rung (accuracy points)

-5.1pts-3.6pts-2.1pts-0.5pts1ptsMean difference, A: one greedy pass: -3.2ptsMean difference, B: sampled: -5.1ptsMean difference, C: resource-matched: -1.2ptsMean difference, B and C combined: -4.2ptsA: one greedy passB: sampledC: resource-matchedB and C combinedBaseline categoryAccuracy difference
Baseline categoryMean difference
A: one greedy pass-3.2pts
B: sampled-5.1pts
C: resource-matched-1.2pts
B and C combined-4.2pts
Source: Sajjadi Mohammadabadi, arXiv 2611.17610, 13 November 2026. 358, 360, 114 and 474 contrasts. Unweighted contrast means.

Even against one greedy pass, the average is negative (-0.032), and it’s near zero (-0.005) when you drop Zhang et al., who contribute 180 of the 516 configurations. In the means, the multi-agent win mostly isn’t there.

The sharpest single table is from Jwalapuram et al., 107 contrasts with the same benchmark set against a 5-sample chain-of-thought vote:

BenchmarkMulti-agentCoT5-sample voteCost vs single callCost vs vote
GPQA-Diamond0.7280.7180.7387.8x1.1x
SWE-bench Lite0.2460.3100.3577.6x0.9x
BrowseComp-Plus0.6820.7370.75910.9x1.9x
All five0.4480.4550.4898.1x1.6x

The paper’s own caveat applies: the vote row is the best score observed across the multi-agent entries, so it’s an upper-bound baseline. Read it as “a good vote is hard to beat”, not as a measured vote.

Why would communication not help? The paper’s formal argument (stylised, with stated assumptions) is that if all agents see the same evidence, talking adds no information, and sharing answers can pull a correct agent toward a wrong consensus. Positive returns need one of three things: partitioned evidence, an external verifier, or error diversity across different models. The positive examples it lists, such as an expert-built system with external tools reaching 96.51% on one benchmark at $555, are exactly that third kind of case, and the paper labels them untested hypotheses.

What the paper claims vs what we checked

ClaimPaper saysWe checkedReproducible?
Multi-agent trails matched baselines-0.042 mean over 474 B/C contrasts; ahead in 26%Read the abstract, intro and tables on the HTML version; the repo README repeats the figuresData tables in the repo; we did not re-extract them from the nine source papers
Pooled effect is negativeLog-odds estimate -0.195, CR2 interval -0.323 to -0.067Read Table 5 sensitivity panels: the whole-study correlation check gives -0.182 with interval -0.366 to +0.002Code but no data re-run by us
Sampling gap is real-0.038 across 5 studies reporting both baselinesRead the per-study table and the ledger (64 of 72 baseline cells negative)Not tried
Cost of multi-agent vs one call8.1x in Table 7 (all five benchmarks); the intro states a 27.1x meanRead Table 7; the two figures are reported differently, so we use 8.1xNot tried
Controlled experimentllama3.1:8b, 250 MMLU-Pro items: debate and mixture-of-agents don’t clear the equal-token vote frontierRead the repo README; the experiment code is in pace_lab/Yes in principle, single 12 GB GPU; not run by us

Limits worth knowing

It’s exploratory, and the author says so. Nine studies, five found “during preliminary exploration” and seven more screened, not a systematic search. The synthesis was not preregistered. The means are unweighted. Some sample sizes are derived or placeholders, and the interval crosses zero under the most conservative block-correlation assumption. The favourable-to-multi-agent cases (partitioned evidence, verifiers, diverse models) are the ones most real enterprise systems claim, and the paper hasn’t tested them.

So don’t read it as “multi-agent doesn’t work”. Read it as: the burden of proof sits with the architecture, and the proof is a matched-budget baseline.

What to do with it

  1. Add a baseline rung to every agent-architecture eval. Same model, same total tokens, k samples plus a vote. If you can’t run it, say which rung you compared against.
  2. Report the vector. Tokens (with echoed context), cost, latency, samples and tool calls for both systems. Echoed history grows fast: the paper’s accounting has scaffold tokens growing quadratically in agents.
  3. Check the evidence structure. Do your agents see different information, call a real verifier, or use different models? If not, expect little from the extra conversation.

This is the same discipline as the judge-calibration paper from two weeks ago: don’t trust a headline number until the baseline is fair.

Docs vs code

The repo is mahmoudsajjadi/pace, Apache-2.0 code and CC-BY-4.0 data, a single commit dated 16 November 2026. The paper’s appendix describes code/pace/ with ResourceTracker and EvaluationAuditor classes. The repository has a pace/ directory instead and no code/ directory. Its pace/audit.py:63 defines audit(), which imports mae_eval.auditor and mae_eval.tracker at pace/audit.py:64-70, yet no mae_eval directory exists anywhere in the tree, even though pyproject.toml lists mae_eval* in its package includes. We read this code but did not run it, so we can’t say whether pace audit works after a normal install. Check before you rely on it. The badge logic itself, at pace/audit.py:1-14, is simple and readable: parity holds if the token ratio is within 1 ± 0.05 and the baseline rung is C or D.

Threat model and failure modes

  • Baseline gaming. A weak baseline is the oldest trick in benchmarking, and it’s easy to do unintentionally. Fix the baseline rung in the eval spec before you see results, and have someone who didn’t build the system pick it.
  • Cost runaway. A system that costs 8x per request and loses to a vote fails on cost before it fails on accuracy. Put a token ceiling on every agent loop and alert on tokens per task, not only on errors.
  • Prompt injection amplification. More agents mean more tool results and more channels a poisoned document can enter by, and agents that read each other’s output carry it further. Treat inter-agent messages as untrusted input.
  • Conformity failures. Agents that see each other’s answers can talk a correct answer into a wrong one. The paper models this and sets a threshold. Keep independent votes available as a fallback path and measure agreement against ground truth, not agreement itself.
  • Fails safe how? When agents disagree or time out, the safe default is to return the vote result or route to a human, not the last speaker’s answer.

Alternatives and when to switch

OptionWhat it isWhen to switch
Single model, k samples, voteSelf-consistency at matched tokensDefault. Switch only when your matched-budget eval shows a consistent gap
Multi-agent with partitioned evidenceAgents own different documents, tools or data slicesThe task can’t fit one context or needs separate permissions per source
Verifier or tool loopOne generator plus an external checker (tests, schema, retrieval check)Answers are checkable, as in code, extraction or SQL; verification beats debate

Numbers

  • 516 multi-agent configurations from 9 studies; 474 B/C contrasts; mean difference -0.042; multi-agent ahead in 26% of them (arXiv 2611.17610, 13 November 2026).
  • Sampling gap -0.038 across 5 studies reporting both baselines (same paper, 13 November 2026).
  • One-study cost ratios: 8.1x a single call, 1.6x a 5-sample vote (Table 7, 13 November 2026).
  • Controlled run: debate frontier gap +0.014 and mixture-of-agents -0.043 on 250 MMLU-Pro items, neither clearing Holm correction; split-evidence debate +0.076 on 200 items (repo README, 16 November 2026).

Try it in 10 minutes

Log token counts per item for your multi-agent system and for a single-agent baseline, then ask whether they’re comparable. Prompt, completion and thinking tokens all count.

import json
import sys


def totals(path):
    """JSONL, one line per call: system, item, prompt_tokens, completion_tokens, thinking_tokens."""
    sums = {}
    for line in open(path):
        r = json.loads(line)
        used = r["prompt_tokens"] + r["completion_tokens"] + r.get("thinking_tokens", 0)
        sums[r["system"]] = sums.get(r["system"], 0) + used
    return sums


t = totals(sys.argv[1])
ratio = t["mas"] / t["sas"]
print(f"MAS uses {ratio:.1f}x the tokens of the baseline")
if abs(ratio - 1) > 0.05:
    print("Not token-matched: rerun the baseline as k samples plus a vote until it is.")

If the ratio is far from 1, your comparison is rung A or B. Rerun the baseline as k samples plus a vote, where k is about your ratio, and see what’s left.

Check yourself

1 Why is a comparison against a single greedy pass a weak test of a multi-agent architecture?
It confounds coordination with extra samples and tokens. Moving to a sampled baseline lowered the multi-agent advantage by 0.038 on average across 5 studies.
2 Under what conditions does the paper expect communication between agents to add value?
Partitioned evidence, an external verifier, or error diversity across different models. With identical evidence, communication adds no information.
3 Name two reasons not to treat the paper's pooled result as settled.
Nine studies chosen without a systematic search, unweighted means, and an interval that crosses zero under the whole-study correlation sensitivity check.

Evidence

SourceTypeDateWhat we checked
arXiv 2611.17610, abstract pageprimary13 November 2026Authors, Comments field (43 pages, 11 figures, 13 tables), abstract, licence
arXiv 2611.17610, HTML textprimary13 November 2026Abstract, introduction and appendices read in full; the Table 1 to 4 body sections and the limitations list were read through an extractor, and the PDF was not read
mahmoudsajjadi/pace repositoryprimary16 November 2026Cloned; read the README, licence, layout and pace/audit.py; did not run the code
Judge-calibration postsecondary3 November 2026Linked for the shared baseline habit

Pick one agent comparison this week and give it a fair baseline. If the week gets away from you, it’ll still be there on Monday.

Researched and drafted with AI, checked against the linked sources. How we use AI →