Skip to content

Deep DivesResearch

Your judge says 90% sure. A new audit of 1,176 transfers says don't ship that calibration map

A 13-judge audit finds that moving a judge's confidence calibration from one domain to another often hurts (Platt scaling made 34% of transfers worse), while 16 labels from the target domain beat it.

10 min read · 35 min hands-onWeek 5 · Evals and LLM observability
On this page
  1. What they did
  2. What they found
  3. What the paper claims vs what we checked
  4. Limits worth knowing
  5. What I’d take from it
  6. How it’s done now
  7. Threat model and failure modes
  8. Alternatives and when to switch
  9. Numbers
  10. Try it in 10 minutes
  11. Evidence

An LLM judge returns “pass, confidence 0.9”. Teams use that number to route cases to human review, to set release gates and to decide which traces get flagged. The number comes from a model that was never trained to be a probability, so someone fits a small map that turns raw confidence into a calibrated one, usually on whichever labelled set exists.

Then the judge moves to a new domain. Does the map still work? A paper submitted on 23 August 2026 tests that directly, and the answer is a tidy “no, and you can’t predict when”. This is the freshest paper I found on this week’s theme, evals and judge calibration (rule 2: a recent paper on the theme). The paper is When Accuracy Gaps Fail to Certify, by Fariya Afrin and Ibne Farabi Shihab.

I’d call it a careful negative result on a small test bed. The authors say so themselves, and the practical advice is worth having anyway.

What they did

The question is narrow. Take a judge that gives a verbalised confidence in [0, 1] that an answer is correct. Fit a scalar recalibration map on a source domain, apply it to a target domain, and ask whether the judge’s accuracy gap between the two domains can warn you when the transfer will hurt.

The setup, from the paper:

  • 13 judges. Eight small open models (Phi-3-mini, TinyLlama-1.1B, Zephyr-7B, MiniCPM-2B, StableLM-Zephyr-3B, Qwen2.5-3B, Gemma-2-2b-it, OLMo-2-7B), four 7B to 9B replication models, and one frontier judge (Claude Sonnet 4.5) reported descriptively only.
  • 2 generators whose answers get judged: Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct.
  • 8 domains, 40 items each: GSM8K, MMLU, SQuAD, BoolQ, SAMSum, HumanEval, TruthfulQA and StrategyQA. Labels for SAMSum and TruthfulQA are proxies.
  • 1,176 predeclared transfers, scored by Brier score (lower is better), with log loss, ECE and AUROC as secondary metrics. “Harm rate” is the share of transfers where the calibrated score is worse than raw.

Malformed judge outputs get a neutral 0.5, and the paper reports parse failures as high as 20.6% for Gemma-2 in the primary run.

What they found

34%of 448 primary transfers made worse by source-fitted Platt scaling
9%made worse by temperature scaling fitted on 16 target labels
0.25correlation between accuracy gap and transfer harm
13%power of the paper's failure certificate at 1,024 labels

Table 2 of the paper, primary transfers (change in Brier against raw, negative is better):

MethodΔ BrierHarm rate
Temperature (source-fitted)−0.14713%
Platt, a ≥ 0 (source-fitted)−0.05234%
Label-shift EM correction+0.02349%
Temperature, 16 target labels−0.1719%
Platt, 16 target labels−0.13612%

Share of transfers where calibration made the judge worse

0%12.5%25%37.5%50%Harm rate, Temperature, source: 13%Harm rate, Platt, source: 34%Harm rate, Label-shift EM: 49%Harm rate, Temperature, 16 target labels: 9%Harm rate, Platt, 16 target labels: 12%Temperature, sourcePlatt, sourceLabel-shift EMTemperature, 16 target labelsPlatt, 16 target labelsCalibration methodTransfers made worse than raw
Calibration methodHarm rate
Temperature, source13%
Platt, source34%
Label-shift EM49%
Temperature, 16 target labels9%
Platt, 16 target labels12%
Source: Afrin and Shihab, Table 2, 448 primary transfers (arXiv 2609.27954, 23 August 2026).

Three things stand out.

Raw confidence is bad to begin with. Raw ECE is 0.393 and AUROC 0.611, a long way from “useful probability”. Every sensible method helps on average. The trouble is the tail: a source-fitted Platt map made one transfer in three worse, and the label-shift correction made half worse.

The accuracy gap is a weak alarm. The intuitive check is “the judge was 85% accurate on the source and 60% on the target, so don’t transfer”. The leak-free correlation between that gap and harm is r = 0.25, with a cluster confidence interval of −0.09 to 0.55, and 0.09 on the second generator. A version of the gap that shares labels with the outcome looks stronger (0.44), and the authors attribute roughly 40% of that association to the leakage. Identical gaps produced opposite outcomes.

A formal certificate exists, but won’t fire. The paper gives a finite-sample lower bound that can reject a transfer. It keeps its nominal coverage, but in synthetic populations its power is 0 at 32 and 128 target labels, and 0.128 at 1,024. On the real benchmark budgets, it rejected nothing in any of the 2,688 audit cells. Useful as theory, not as a gate.

The practical result is the last row of the table: a few labels from the target domain beat a map fitted on lots of source data. Sixteen labels cost an afternoon.

One more finding that matters for gating. At a fixed threshold (accept if confidence is at least 0.5), always-on Platt scaling raised expected decision cost from 0.507 to 0.728 and cut coverage from 0.74 to 0.37. Calibration changes what your threshold means, so re-pick the threshold after you recalibrate.

What the paper claims vs what we checked

ClaimPaper saysWe checkedReproducible?
Source-fitted Platt hurts often34% harm rate, Δ Brier −0.052 (CI −0.149 to +0.005)Read Table 2 on the HTML version. The improvement’s confidence interval crosses zeroAnalysis scripts are named in the text; the abstract page links no repository (as of 3 November 2026)
Target labels beat source mapsTemperature with 16 labels: Δ Brier −0.171, harm 9%; second generator −0.155, harm 5%Read Table 2 and Table 7Code not linked, not tried
Accuracy gap is a weak signalr = 0.25 [−0.09, 0.55]; 0.09 on the second generatorRead the text and the seed appendix: the primary correlation ranges from 0.21 to 0.45 over ten split seedsNot tried
Certificate has low power0.128 at 1,024 labels; zero rejections on real budgetsRead Table 5Not tried
Result holds for frontier judgesOne judge (Claude Sonnet 4.5), 56 transfers, descriptiveRead the setup; no conclusion is drawn from this armNot tried

Limits worth knowing

The authors list them, and they matter. Eight primary judges are small open models, and the generators are 1B to 1.5B parameters. Each domain has only 40 items, split 24 for fitting and 16 for audit. Confidence is verbalised, English only, and the effective sample is 8 judges × 8 domains. The theory covers binary correctness and scalar confidence only.

So read the percentages as direction, not as rates you’ll see in production. A 2026 frontier judge on your own support tickets could behave differently. The paper’s contribution is the habit it forces: measure transfer harm instead of assuming it is zero.

What I’d take from it

If you gate releases or route reviews on a judge’s score:

  1. Calibrate per domain, with a few labels. About 16 labelled cases per domain, a single temperature parameter, and no source-domain map as a default.
  2. Prefer fewer parameters. Temperature (one parameter) beat Platt (two) on harm in this study. With tiny label budgets, the simple map is the safer one.
  3. Re-pick thresholds after you recalibrate, and report coverage next to accuracy.
  4. Don’t trust an accuracy gap as a safety check. Re-label.

A minimal temperature fit on a small labelled set looks like this:

import math

def logit(p):
    p = min(max(p, 1e-3), 1 - 1e-3)
    return math.log(p / (1 - p))

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def fit_temperature(confs, labels):
    """Grid-search T that minimises Brier on a small labelled set."""
    def brier(t):
        ps = [sigmoid(logit(c) / t) for c in confs]
        return sum((p - y) ** 2 for p, y in zip(ps, labels)) / len(labels)
    return min((x / 10 for x in range(5, 80)), key=brier)

If the grid hits its upper edge (7.9 here), the judge’s confidence carries almost no signal in that domain. Treat that as a finding about the judge, not a tuning problem.

How it’s done now

In 2023 most teams reported a judge’s raw agreement with humans once, on a general chat benchmark, and assumed it carried over. The MT-Bench paper reported over 80% agreement with human preferences for GPT-4, on chat tasks, with older models. Most teams now keep a small human-labelled set for each judged task, report agreement beyond chance, and treat the judge as a measuring instrument that needs a calibration record, as in Monday’s building-block section. A related paper, How to Correctly Report LLM-as-a-Judge Evaluations (ICML 2026), corrects judge bias from sensitivity and specificity and gives confidence intervals that include calibration-set uncertainty. Use it when your question is “what is the true pass rate”, and use this audit’s advice when your question is “how much should I trust one confidence score”.

Enterprise adoption: this paper cites no named production deployment, and I found no engineering post documenting per-domain judge calibration at a named company as of 3 November 2026. The practice is still mostly in-house.

Threat model and failure modes

  • Judge injection. Candidate answers can contain text that steers a grader. A raised confidence from an injected answer looks like a well-calibrated “pass”. Keep answers as quoted data, and sample outputs for a code grader or human check.
  • Calibration data leakage. Labelled eval sets built from real traffic contain PII and customer text. Redact them, restrict access, and keep per-tenant sets separate when tenants differ in domain.
  • Silent transfer. A map fitted on one domain keeps running after the product moves to another. It fails unsafe by default: scores look reasonable and gates pass. Fail safe: tag every calibration map with its domain and block use outside it.
  • Judge or generator change. The audit’s second generator changed gap correlations from 0.25 to 0.09. A provider updating the judge or a team swapping the generator invalidates the map. Pin versions and re-fit on change.
  • Parse failures. Up to 20.6% of one judge’s outputs were malformed in the primary run. A fallback that defaults to 0.5 hides this. Count unparseable outputs, alert on the rate, and record “unscored”, never “pass”.

Alternatives and when to switch

OptionWhat it isWhen to switch
Per-domain temperature scaling (16+ labels)One parameter fitted on target-domain labelsDefault for gating on judge confidence
Bias-corrected pass rate with confidence intervals (Lee et al., ICML 2026)Correct the estimated pass rate using judge sensitivity and specificityYou need an unbiased aggregate score, not a per-case probability
Deterministic checks and human reviewCode graders, tests, sampled expert labelsThe domain has a checkable answer, or the judge’s AUROC stays near raw’s 0.61 after calibration

Numbers

  • 34% of 448 primary transfers harmed by source-fitted Platt scaling; 13% by source-fitted temperature; 9% by temperature with 16 target labels (Table 2, 23 August 2026).
  • Raw ECE 0.393; AUROC 0.611 raw (paper, 23 August 2026).
  • Accuracy-gap correlation with harm: r = 0.25 (second generator 0.09) (paper, 23 August 2026).
  • Certificate power: 0.128 at 1,024 target labels, 0 at 32 and 128 (Table 5, 23 August 2026).
  • Decision cost at a 0.5 threshold: 0.507 raw versus 0.728 with always-on Platt; coverage 0.74 versus 0.37 (paper, 23 August 2026).
  • GPT-4 judge versus human agreement over 80% (Zheng et al., 2023, older models).

Try it in 10 minutes

Take the 30 labelled cases from Monday’s try-it and ask your judge for a confidence on each. Split them by topic. Fit one temperature on all of them and one on each topic’s half (use fit_temperature above), then compare Brier on the held-out half. If the per-topic temperatures differ a lot, or one lands on the grid’s edge, your judge isn’t one instrument across your domains.

Check yourself

1 Why isn't a large accuracy gap between two domains a reliable reason to refuse to transfer a calibration map?
The leak-free correlation with harm was only r = 0.25 (0.09 on a second generator), and identical gaps produced opposite outcomes.
2 What beat a calibration map fitted on lots of source data?
Temperature scaling fitted on 16 labels from the target domain: Δ Brier −0.171 and 9% harm, against −0.052 and 34% for source-fitted Platt.
3 What else must you redo after recalibrating a judge that gates on confidence?
Re-pick the threshold and report coverage: always-on Platt moved expected decision cost from 0.507 to 0.728 and coverage from 0.74 to 0.37 at a fixed 0.5 threshold.

Evidence

SourceTypeDateWhat we checked
arXiv 2609.27954, abstract pageprimary23 August 2026Authors, date, licence (CC BY 4.0); no repository linked as of 3 November 2026
arXiv 2609.27954, HTML textprimary23 August 2026Setup, Tables 2, 4, 5 and 7, limitations, appendices H to K. Several figure assets are missing in the HTML
arXiv 2511.21140, abstract pageprimary31 May 2026 (v4)Read the abstract and venue; the full text was not read
MT-Bench paperprimary2023Cited for the 80% agreement figure, as quoted in Monday’s post
Temperature-fit snippetour run3 November 2026Ran it on synthetic data; it runs, and it hits the grid edge when the signal is weak

A calibrated judge is a small habit, not a project; if this week gets away from you, the 16 labels will still be there next week.

Researched and drafted with AI, checked against the linked sources. How we use AI →