Your multi-agent system beat a single call. Did it beat five single calls for the same tokens?
A meta-analysis of 516 multi-agent configurations from nine studies finds that against baselines matched on samples or budget, multi-agent accuracy averages 4.2 points lower, and it proposes a six-part resource vector and a baseline ladder to report comparisons honestly.
On this page
Every multi-agent paper has the same figure: a bar for “single agent”, a taller bar for “our multi-agent system”. The single-agent bar is usually one greedy pass. The multi-agent bar usually used five to fifteen times the tokens, several samples and a voting step.
So what did the architecture buy you? The coordination, or the compute? A paper submitted on 13 November 2026 asks that across nine published studies and gets an uncomfortable answer. It’s this week’s landmark pick (rule 1): it changes how multi-agent systems get justified, and it has a code repository. It’s off this week’s ingestion theme (Monday’s post), but the habit it teaches applies to every pipeline decision, including whether a model should parse your documents at all.
The paper is More Agents, More Samples, or More Thinking? by Seyed Mahmoud Sajjadi Mohammadabadi of the University of Nevada, Reno. One author, one preprint, 43 pages, not yet peer reviewed.
What the paper actually is
Not a new method and not a new experiment. It’s a meta-analysis: the author pulled published results tables from nine studies (516 unique multi-agent configurations) and re-organised them. The contribution is a reporting protocol called PACE, Parity-Aware Compute Evaluation.
It has two parts:
- A resource vector with six dimensions: reasoning tokens, total tokens (including echoed histories and system prompts), cost, latency, independent samples, and tool or environment calls. Each gets a status: strictly controlled, approximately matched, unmatched, or unreported.
- A baseline ladder. A is one greedy pass. B is k independent samples with a vote, no communication. C is a baseline matched on a declared resource. D is a Pareto frontier over budgets.
The claim is simple: a comparison against rung A can’t separate coordination from extra computation.
What they found
The headline table, from the paper’s synthesis (mean accuracy difference, multi-agent minus baseline):
Even against one greedy pass, the average is negative (-0.032), and it’s near zero (-0.005) when you drop Zhang et al., who contribute 180 of the 516 configurations. In the means, the multi-agent win mostly isn’t there.
The sharpest single table is from Jwalapuram et al., 107 contrasts with the same benchmark set against a 5-sample chain-of-thought vote:
| Benchmark | Multi-agent | CoT | 5-sample vote | Cost vs single call | Cost vs vote |
|---|---|---|---|---|---|
| GPQA-Diamond | 0.728 | 0.718 | 0.738 | 7.8x | 1.1x |
| SWE-bench Lite | 0.246 | 0.310 | 0.357 | 7.6x | 0.9x |
| BrowseComp-Plus | 0.682 | 0.737 | 0.759 | 10.9x | 1.9x |
| All five | 0.448 | 0.455 | 0.489 | 8.1x | 1.6x |
The paper’s own caveat applies: the vote row is the best score observed across the multi-agent entries, so it’s an upper-bound baseline. Read it as “a good vote is hard to beat”, not as a measured vote.
Why would communication not help? The paper’s formal argument (stylised, with stated assumptions) is that if all agents see the same evidence, talking adds no information, and sharing answers can pull a correct agent toward a wrong consensus. Positive returns need one of three things: partitioned evidence, an external verifier, or error diversity across different models. The positive examples it lists, such as an expert-built system with external tools reaching 96.51% on one benchmark at $555, are exactly that third kind of case, and the paper labels them untested hypotheses.
What the paper claims vs what we checked
| Claim | Paper says | We checked | Reproducible? |
|---|---|---|---|
| Multi-agent trails matched baselines | -0.042 mean over 474 B/C contrasts; ahead in 26% | Read the abstract, intro and tables on the HTML version; the repo README repeats the figures | Data tables in the repo; we did not re-extract them from the nine source papers |
| Pooled effect is negative | Log-odds estimate -0.195, CR2 interval -0.323 to -0.067 | Read Table 5 sensitivity panels: the whole-study correlation check gives -0.182 with interval -0.366 to +0.002 | Code but no data re-run by us |
| Sampling gap is real | -0.038 across 5 studies reporting both baselines | Read the per-study table and the ledger (64 of 72 baseline cells negative) | Not tried |
| Cost of multi-agent vs one call | 8.1x in Table 7 (all five benchmarks); the intro states a 27.1x mean | Read Table 7; the two figures are reported differently, so we use 8.1x | Not tried |
| Controlled experiment | llama3.1:8b, 250 MMLU-Pro items: debate and mixture-of-agents don’t clear the equal-token vote frontier | Read the repo README; the experiment code is in pace_lab/ | Yes in principle, single 12 GB GPU; not run by us |
Limits worth knowing
It’s exploratory, and the author says so. Nine studies, five found “during preliminary exploration” and seven more screened, not a systematic search. The synthesis was not preregistered. The means are unweighted. Some sample sizes are derived or placeholders, and the interval crosses zero under the most conservative block-correlation assumption. The favourable-to-multi-agent cases (partitioned evidence, verifiers, diverse models) are the ones most real enterprise systems claim, and the paper hasn’t tested them.
So don’t read it as “multi-agent doesn’t work”. Read it as: the burden of proof sits with the architecture, and the proof is a matched-budget baseline.
What to do with it
- Add a baseline rung to every agent-architecture eval. Same model, same total tokens, k samples plus a vote. If you can’t run it, say which rung you compared against.
- Report the vector. Tokens (with echoed context), cost, latency, samples and tool calls for both systems. Echoed history grows fast: the paper’s accounting has scaffold tokens growing quadratically in agents.
- Check the evidence structure. Do your agents see different information, call a real verifier, or use different models? If not, expect little from the extra conversation.
This is the same discipline as the judge-calibration paper from two weeks ago: don’t trust a headline number until the baseline is fair.
Docs vs code
The repo is mahmoudsajjadi/pace, Apache-2.0 code and CC-BY-4.0 data, a single commit dated 16 November 2026. The paper’s appendix describes code/pace/ with ResourceTracker and EvaluationAuditor classes. The repository has a pace/ directory instead and no code/ directory. Its pace/audit.py:63 defines audit(), which imports mae_eval.auditor and mae_eval.tracker at pace/audit.py:64-70, yet no mae_eval directory exists anywhere in the tree, even though pyproject.toml lists mae_eval* in its package includes. We read this code but did not run it, so we can’t say whether pace audit works after a normal install. Check before you rely on it. The badge logic itself, at pace/audit.py:1-14, is simple and readable: parity holds if the token ratio is within 1 ± 0.05 and the baseline rung is C or D.
Threat model and failure modes
- Baseline gaming. A weak baseline is the oldest trick in benchmarking, and it’s easy to do unintentionally. Fix the baseline rung in the eval spec before you see results, and have someone who didn’t build the system pick it.
- Cost runaway. A system that costs 8x per request and loses to a vote fails on cost before it fails on accuracy. Put a token ceiling on every agent loop and alert on tokens per task, not only on errors.
- Prompt injection amplification. More agents mean more tool results and more channels a poisoned document can enter by, and agents that read each other’s output carry it further. Treat inter-agent messages as untrusted input.
- Conformity failures. Agents that see each other’s answers can talk a correct answer into a wrong one. The paper models this and sets a threshold. Keep independent votes available as a fallback path and measure agreement against ground truth, not agreement itself.
- Fails safe how? When agents disagree or time out, the safe default is to return the vote result or route to a human, not the last speaker’s answer.
Alternatives and when to switch
| Option | What it is | When to switch |
|---|---|---|
| Single model, k samples, vote | Self-consistency at matched tokens | Default. Switch only when your matched-budget eval shows a consistent gap |
| Multi-agent with partitioned evidence | Agents own different documents, tools or data slices | The task can’t fit one context or needs separate permissions per source |
| Verifier or tool loop | One generator plus an external checker (tests, schema, retrieval check) | Answers are checkable, as in code, extraction or SQL; verification beats debate |
Numbers
- 516 multi-agent configurations from 9 studies; 474 B/C contrasts; mean difference -0.042; multi-agent ahead in 26% of them (arXiv 2611.17610, 13 November 2026).
- Sampling gap -0.038 across 5 studies reporting both baselines (same paper, 13 November 2026).
- One-study cost ratios: 8.1x a single call, 1.6x a 5-sample vote (Table 7, 13 November 2026).
- Controlled run: debate frontier gap +0.014 and mixture-of-agents -0.043 on 250 MMLU-Pro items, neither clearing Holm correction; split-evidence debate +0.076 on 200 items (repo README, 16 November 2026).
Try it in 10 minutes
Log token counts per item for your multi-agent system and for a single-agent baseline, then ask whether they’re comparable. Prompt, completion and thinking tokens all count.
import json
import sys
def totals(path):
"""JSONL, one line per call: system, item, prompt_tokens, completion_tokens, thinking_tokens."""
sums = {}
for line in open(path):
r = json.loads(line)
used = r["prompt_tokens"] + r["completion_tokens"] + r.get("thinking_tokens", 0)
sums[r["system"]] = sums.get(r["system"], 0) + used
return sums
t = totals(sys.argv[1])
ratio = t["mas"] / t["sas"]
print(f"MAS uses {ratio:.1f}x the tokens of the baseline")
if abs(ratio - 1) > 0.05:
print("Not token-matched: rerun the baseline as k samples plus a vote until it is.")
If the ratio is far from 1, your comparison is rung A or B. Rerun the baseline as k samples plus a vote, where k is about your ratio, and see what’s left.
Evidence
| Source | Type | Date | What we checked |
|---|---|---|---|
| arXiv 2611.17610, abstract page | primary | 13 November 2026 | Authors, Comments field (43 pages, 11 figures, 13 tables), abstract, licence |
| arXiv 2611.17610, HTML text | primary | 13 November 2026 | Abstract, introduction and appendices read in full; the Table 1 to 4 body sections and the limitations list were read through an extractor, and the PDF was not read |
| mahmoudsajjadi/pace repository | primary | 16 November 2026 | Cloned; read the README, licence, layout and pace/audit.py; did not run the code |
| Judge-calibration post | secondary | 3 November 2026 | Linked for the shared baseline habit |
Pick one agent comparison this week and give it a fair baseline. If the week gets away from you, it’ll still be there on Monday.
Researched and drafted with AI, checked against the linked sources. How we use AI →