Your judge says 90% sure. A new audit of 1,176 transfers says don't ship that calibration map
A 13-judge audit finds that moving a judge's confidence calibration from one domain to another often hurts (Platt scaling made 34% of transfers worse), while 16 labels from the target domain beat it.
On this page
An LLM judge returns “pass, confidence 0.9”. Teams use that number to route cases to human review, to set release gates and to decide which traces get flagged. The number comes from a model that was never trained to be a probability, so someone fits a small map that turns raw confidence into a calibrated one, usually on whichever labelled set exists.
Then the judge moves to a new domain. Does the map still work? A paper submitted on 23 August 2026 tests that directly, and the answer is a tidy “no, and you can’t predict when”. This is the freshest paper I found on this week’s theme, evals and judge calibration (rule 2: a recent paper on the theme). The paper is When Accuracy Gaps Fail to Certify, by Fariya Afrin and Ibne Farabi Shihab.
I’d call it a careful negative result on a small test bed. The authors say so themselves, and the practical advice is worth having anyway.
What they did
The question is narrow. Take a judge that gives a verbalised confidence in [0, 1] that an answer is correct. Fit a scalar recalibration map on a source domain, apply it to a target domain, and ask whether the judge’s accuracy gap between the two domains can warn you when the transfer will hurt.
The setup, from the paper:
- 13 judges. Eight small open models (Phi-3-mini, TinyLlama-1.1B, Zephyr-7B, MiniCPM-2B, StableLM-Zephyr-3B, Qwen2.5-3B, Gemma-2-2b-it, OLMo-2-7B), four 7B to 9B replication models, and one frontier judge (Claude Sonnet 4.5) reported descriptively only.
- 2 generators whose answers get judged: Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct.
- 8 domains, 40 items each: GSM8K, MMLU, SQuAD, BoolQ, SAMSum, HumanEval, TruthfulQA and StrategyQA. Labels for SAMSum and TruthfulQA are proxies.
- 1,176 predeclared transfers, scored by Brier score (lower is better), with log loss, ECE and AUROC as secondary metrics. “Harm rate” is the share of transfers where the calibrated score is worse than raw.
Malformed judge outputs get a neutral 0.5, and the paper reports parse failures as high as 20.6% for Gemma-2 in the primary run.
What they found
Table 2 of the paper, primary transfers (change in Brier against raw, negative is better):
| Method | Δ Brier | Harm rate |
|---|---|---|
| Temperature (source-fitted) | −0.147 | 13% |
| Platt, a ≥ 0 (source-fitted) | −0.052 | 34% |
| Label-shift EM correction | +0.023 | 49% |
| Temperature, 16 target labels | −0.171 | 9% |
| Platt, 16 target labels | −0.136 | 12% |
Three things stand out.
Raw confidence is bad to begin with. Raw ECE is 0.393 and AUROC 0.611, a long way from “useful probability”. Every sensible method helps on average. The trouble is the tail: a source-fitted Platt map made one transfer in three worse, and the label-shift correction made half worse.
The accuracy gap is a weak alarm. The intuitive check is “the judge was 85% accurate on the source and 60% on the target, so don’t transfer”. The leak-free correlation between that gap and harm is r = 0.25, with a cluster confidence interval of −0.09 to 0.55, and 0.09 on the second generator. A version of the gap that shares labels with the outcome looks stronger (0.44), and the authors attribute roughly 40% of that association to the leakage. Identical gaps produced opposite outcomes.
A formal certificate exists, but won’t fire. The paper gives a finite-sample lower bound that can reject a transfer. It keeps its nominal coverage, but in synthetic populations its power is 0 at 32 and 128 target labels, and 0.128 at 1,024. On the real benchmark budgets, it rejected nothing in any of the 2,688 audit cells. Useful as theory, not as a gate.
The practical result is the last row of the table: a few labels from the target domain beat a map fitted on lots of source data. Sixteen labels cost an afternoon.
One more finding that matters for gating. At a fixed threshold (accept if confidence is at least 0.5), always-on Platt scaling raised expected decision cost from 0.507 to 0.728 and cut coverage from 0.74 to 0.37. Calibration changes what your threshold means, so re-pick the threshold after you recalibrate.
What the paper claims vs what we checked
| Claim | Paper says | We checked | Reproducible? |
|---|---|---|---|
| Source-fitted Platt hurts often | 34% harm rate, Δ Brier −0.052 (CI −0.149 to +0.005) | Read Table 2 on the HTML version. The improvement’s confidence interval crosses zero | Analysis scripts are named in the text; the abstract page links no repository (as of 3 November 2026) |
| Target labels beat source maps | Temperature with 16 labels: Δ Brier −0.171, harm 9%; second generator −0.155, harm 5% | Read Table 2 and Table 7 | Code not linked, not tried |
| Accuracy gap is a weak signal | r = 0.25 [−0.09, 0.55]; 0.09 on the second generator | Read the text and the seed appendix: the primary correlation ranges from 0.21 to 0.45 over ten split seeds | Not tried |
| Certificate has low power | 0.128 at 1,024 labels; zero rejections on real budgets | Read Table 5 | Not tried |
| Result holds for frontier judges | One judge (Claude Sonnet 4.5), 56 transfers, descriptive | Read the setup; no conclusion is drawn from this arm | Not tried |
Limits worth knowing
The authors list them, and they matter. Eight primary judges are small open models, and the generators are 1B to 1.5B parameters. Each domain has only 40 items, split 24 for fitting and 16 for audit. Confidence is verbalised, English only, and the effective sample is 8 judges × 8 domains. The theory covers binary correctness and scalar confidence only.
So read the percentages as direction, not as rates you’ll see in production. A 2026 frontier judge on your own support tickets could behave differently. The paper’s contribution is the habit it forces: measure transfer harm instead of assuming it is zero.
What I’d take from it
If you gate releases or route reviews on a judge’s score:
- Calibrate per domain, with a few labels. About 16 labelled cases per domain, a single temperature parameter, and no source-domain map as a default.
- Prefer fewer parameters. Temperature (one parameter) beat Platt (two) on harm in this study. With tiny label budgets, the simple map is the safer one.
- Re-pick thresholds after you recalibrate, and report coverage next to accuracy.
- Don’t trust an accuracy gap as a safety check. Re-label.
A minimal temperature fit on a small labelled set looks like this:
import math
def logit(p):
p = min(max(p, 1e-3), 1 - 1e-3)
return math.log(p / (1 - p))
def sigmoid(x):
return 1 / (1 + math.exp(-x))
def fit_temperature(confs, labels):
"""Grid-search T that minimises Brier on a small labelled set."""
def brier(t):
ps = [sigmoid(logit(c) / t) for c in confs]
return sum((p - y) ** 2 for p, y in zip(ps, labels)) / len(labels)
return min((x / 10 for x in range(5, 80)), key=brier)
If the grid hits its upper edge (7.9 here), the judge’s confidence carries almost no signal in that domain. Treat that as a finding about the judge, not a tuning problem.
How it’s done now
In 2023 most teams reported a judge’s raw agreement with humans once, on a general chat benchmark, and assumed it carried over. The MT-Bench paper reported over 80% agreement with human preferences for GPT-4, on chat tasks, with older models. Most teams now keep a small human-labelled set for each judged task, report agreement beyond chance, and treat the judge as a measuring instrument that needs a calibration record, as in Monday’s building-block section. A related paper, How to Correctly Report LLM-as-a-Judge Evaluations (ICML 2026), corrects judge bias from sensitivity and specificity and gives confidence intervals that include calibration-set uncertainty. Use it when your question is “what is the true pass rate”, and use this audit’s advice when your question is “how much should I trust one confidence score”.
Enterprise adoption: this paper cites no named production deployment, and I found no engineering post documenting per-domain judge calibration at a named company as of 3 November 2026. The practice is still mostly in-house.
Threat model and failure modes
- Judge injection. Candidate answers can contain text that steers a grader. A raised confidence from an injected answer looks like a well-calibrated “pass”. Keep answers as quoted data, and sample outputs for a code grader or human check.
- Calibration data leakage. Labelled eval sets built from real traffic contain PII and customer text. Redact them, restrict access, and keep per-tenant sets separate when tenants differ in domain.
- Silent transfer. A map fitted on one domain keeps running after the product moves to another. It fails unsafe by default: scores look reasonable and gates pass. Fail safe: tag every calibration map with its domain and block use outside it.
- Judge or generator change. The audit’s second generator changed gap correlations from 0.25 to 0.09. A provider updating the judge or a team swapping the generator invalidates the map. Pin versions and re-fit on change.
- Parse failures. Up to 20.6% of one judge’s outputs were malformed in the primary run. A fallback that defaults to 0.5 hides this. Count unparseable outputs, alert on the rate, and record “unscored”, never “pass”.
Alternatives and when to switch
| Option | What it is | When to switch |
|---|---|---|
| Per-domain temperature scaling (16+ labels) | One parameter fitted on target-domain labels | Default for gating on judge confidence |
| Bias-corrected pass rate with confidence intervals (Lee et al., ICML 2026) | Correct the estimated pass rate using judge sensitivity and specificity | You need an unbiased aggregate score, not a per-case probability |
| Deterministic checks and human review | Code graders, tests, sampled expert labels | The domain has a checkable answer, or the judge’s AUROC stays near raw’s 0.61 after calibration |
Numbers
- 34% of 448 primary transfers harmed by source-fitted Platt scaling; 13% by source-fitted temperature; 9% by temperature with 16 target labels (Table 2, 23 August 2026).
- Raw ECE 0.393; AUROC 0.611 raw (paper, 23 August 2026).
- Accuracy-gap correlation with harm: r = 0.25 (second generator 0.09) (paper, 23 August 2026).
- Certificate power: 0.128 at 1,024 target labels, 0 at 32 and 128 (Table 5, 23 August 2026).
- Decision cost at a 0.5 threshold: 0.507 raw versus 0.728 with always-on Platt; coverage 0.74 versus 0.37 (paper, 23 August 2026).
- GPT-4 judge versus human agreement over 80% (Zheng et al., 2023, older models).
Try it in 10 minutes
Take the 30 labelled cases from Monday’s try-it and ask your judge for a confidence on each. Split them by topic. Fit one temperature on all of them and one on each topic’s half (use fit_temperature above), then compare Brier on the held-out half. If the per-topic temperatures differ a lot, or one lands on the grid’s edge, your judge isn’t one instrument across your domains.
Evidence
| Source | Type | Date | What we checked |
|---|---|---|---|
| arXiv 2609.27954, abstract page | primary | 23 August 2026 | Authors, date, licence (CC BY 4.0); no repository linked as of 3 November 2026 |
| arXiv 2609.27954, HTML text | primary | 23 August 2026 | Setup, Tables 2, 4, 5 and 7, limitations, appendices H to K. Several figure assets are missing in the HTML |
| arXiv 2511.21140, abstract page | primary | 31 May 2026 (v4) | Read the abstract and venue; the full text was not read |
| MT-Bench paper | primary | 2023 | Cited for the 80% agreement figure, as quoted in Monday’s post |
| Temperature-fit snippet | our run | 3 November 2026 | Ran it on synthetic data; it runs, and it hits the grid edge when the signal is weak |
A calibrated judge is a small habit, not a project; if this week gets away from you, the 16 labels will still be there next week.
Researched and drafted with AI, checked against the linked sources. How we use AI →