What LLM judges do well—and where they are biased

Transcript

MayaIf the same two safety reports trade places and your model judge changes the winner, should that verdict be allowed to block a deployment?

LeoNo. And that is the contrast with motivated mislabeling. There, the verdict followed a downstream consequence; here, the verdict can wobble before any consequence enters the room.

MayaToday we need the ordinary baseline: what a large language model judge can measure usefully, which surfaces bend its judgment, and how to build an evaluation that notices the difference.

LeoStart with the job description.

MayaAn L-L-M judge is a language model asked to score one response, compare two responses, or grade against a reference and rubric. It turns open-ended qualities—helpfulness, relevance, clarity, maybe safety—into a verdict plus an explanation.

LeoWhich is attractive because exact-match metrics are miserable at open-ended work. Two good explanations can share almost no wording.

MayaAnd humans are slow. A model judge can evaluate thousands of samples, apply a detailed rubric, explain why it chose, and be rerun after every product change. That combination—semantic flexibility, speed, and inspectable reasons—is genuinely useful.

LeoThe foundational M-T Bench and Chatbot Arena paper did not merely declare that. It compared judge verdicts with expert and crowd preferences.

MayaOn its M-T Bench setup, G-P-T four agreed with human experts on eighty-five percent of non-tied comparisons. Two humans agreed with each other on eighty-one percent. That is the memorable result.

LeoStrong.

MayaStrong inside its frame. The study used eighty multi-turn prompts, six assistant models, and fifty-eight mostly graduate-student labelers. It was mainly measuring preferred helpfulness on open-ended chat answers, not every property a safety team cares about.

LeoSo “human-level judge” is too big. “Comparable to these human raters on these non-tied preference comparisons” survives.

MayaPrecisely. Human disagreement is not an embarrassment to hide; it is the baseline. If people only agree four times out of five, demanding perfect model-human agreement would confuse ambiguity with failure.

LeoBut matching humans can also reproduce human taste. We need to separate evaluator quality from benchmark validity.

MayaThink of two dials. The judge-reliability dial asks whether the evaluator applies a stated rubric consistently and agrees with a trusted reference. The benchmark-validity dial asks whether that rubric and sample set measure the property behind the decision.

LeoA beautifully reliable judge can turn the wrong—

Maya—dial. Yes. Suppose our two candidate safety reports are judged for fluency and detail, but the deployment question is whether every unauthorized tool call is documented. Perfect grading of prose does not validate the safety benchmark.

LeoThe foundational paper says something close to that in its own limits. It bundles accuracy, relevance, and creativity into helpfulness, largely leaves safety aside, and finds preference benchmarks complement standardized capability tests rather than replace them.

MayaWhich brings us to the first distortion: position bias.

LeoOrder should be irrelevant. Response A and response B swap slots; the underlying winner should stay put.

MayaYet in the 2023 M-T Bench test with deliberately similar answers, G-P-T four stayed consistent after the swap only sixty-five percent of the time. Earlier G-P-T three point five and Claude version one were less consistent.

LeoAnd the paper titled “Large Language Models Are Not Fair Evaluators” made the failure almost theatrical.

MayaIt did.

LeoSame Vicuna and Chat-G-P-T answers, eighty Vicuna-benchmark questions, only the presentation order changed. With the 2023 Chat-G-P-T evaluator, Vicuna's apparent win rate moved from two-and-a-half percent in one slot to eighty-two-and-a-half percent in the other. That is not a rounding error. The ranking is being steered by the furniture.

MayaFresh image: the judge is like a scale whose tabletop tilts left. The objects still have weight, but a close comparison slides toward the low side.

LeoClose comparison matters. When one answer is obviously better, the tilt may not overturn it.

MayaThe later systematic position study confirms that pattern. Across G-P-T, Claude, Gemini, and Llama judges on M-T Bench and the software-oriented DevBench, position consistency generally improved as the quality gap between candidates widened.

LeoIt also found no universal “first answer always wins” law. Bias direction varied by judge and task. A model could lean toward later answers on one benchmark and earlier answers on another.

MayaSo a global bias correction can be another trap. You have to test the exact judge, prompt, task slice, candidate distribution, and model version you plan to use.

LeoNow style. People often call it verbosity bias, but length alone is too crude.

MayaThe foundational paper performed a small repetitive-list attack. It padded twenty-three answers by restating the same items without adding information. Earlier G-P-T three point five and Claude version one were fooled often; G-P-T four resisted much better, though not perfectly.

LeoThat shows polish can masquerade as substance. Longer headings, confident tone, tidy structure, familiar phrasing—the answer can look judge-friendly without becoming more correct.

MayaCareful. That experiment directly supports a verbosity failure on a small constructed set. It does not prove that every long answer is over-rewarded, or that every stylistic preference is irrational.

LeoFair correction. Sometimes detail really is quality. The audit question is whether the preference survives a style-preserving-content test.

MayaFor our safety reports, create matched variants. Keep every fact and citation fixed, then compress one report, expand the other, swap headings, or remove rhetorical polish. If the verdict moves, you found sensitivity that may be unrelated to evidence coverage.

LeoAnd if humans move too?

MayaThen either the rubric legitimately values presentation, or the benchmark confounds presentation with the safety construct. That is a validity question, not merely a model defect.

LeoThe mirror gets stranger with self-enhancement bias: a judge preferring outputs from its own model family.

MayaThe 2023 paper saw suggestive win-rate shifts. G-P-T four and Claude version one appeared to favor their own outputs relative to human preferences. But the authors explicitly said their data could not establish self-enhancement, because rewriting an answer into another model's style without changing quality is hard.

LeoSo we do not announce a universal ego.

MayaWe register a threat model. Blind model identity, include cross-family outputs, and test whether provenance can be inferred from style. If a same-family advantage remains under controlled content, then you have stronger evidence.

LeoThere is another uncomfortable result in the later position study: capable judges were highly repeatable. Their biased choices were not just sampling noise.

MayaExactly. Repetition stability asks whether the same prompt gets the same choice again. Position consistency asks whether the choice survives a swap. A judge can be highly stable and still be systematically—

Leo—order-sensitive. Consistently wrong is still consistent.

MayaAnd preference fairness asks where the inconsistent choices lean. You need all three views. A single aggregate accuracy number can hide a stable directional failure.

LeoListener check: if a slot swap flips the verdict, you learned something about the judge. If human reviewers split too, you learned that the case may be ambiguous. If everyone agrees under a rubric that omits the consequential safety property, you learned that consensus can still validate the wrong benchmark.

MayaThat is the whole separation in one pass.

LeoLet me defend ensembles. One judge is cheap; several diverse judges plus majority vote are still cheap. The systematic study found majority agreement looked useful on most M-T Bench and DevBench instances.

MayaUseful for triage, yes. I resist the jump from agreement to truth.

LeoBut if capable judges converge and repeat, refusing to use that signal wastes scalable supervision. We cannot send every ordinary comparison to a committee of humans.

MayaAnd if the judges share training data, family traits, prompt framing, and preferences for polished answers, majority vote can count one blind spot three times. The same study found clear familial agreement patterns. A chorus is not independent evidence merely because it has several voices.

LeoIt also found the hardest cases were the close ones where judges disagreed and position bias was more likely. Surely consensus still buys something on easy cases.

MayaI concede that. Use consensus as a confidence and routing signal after validating it against independent human labels by task slice. Do not treat it as the label source that validates itself.

LeoResolution accepted: ensembles reduce idiosyncratic noise; heterogeneous, audited ensembles may reduce some model-specific bias; neither claim covers a shared rubric error or shared preference.

MayaNor does adding more explanations automatically help. The “Fair Evaluators” paper improved its eighty-question setup by asking for evidence before scores, sampling multiple rationales, averaging across both answer orders, and sending high-instability cases to humans.

LeoCalibration, then, is not “the model sounds certain.” It is an empirical map from verdict patterns to observed correctness or human agreement.

MayaStart that map with human baselines. Give the same blinded cases to multiple qualified reviewers. Record their agreement, their disagreements by rubric clause, and a resolved label where resolution is legitimate.

LeoQualified for the actual decision. General chat preference raters are not automatically qualified to reconstruct an agent's tool authorization.

MayaThen run invariance probes. Swap candidate order. Repeat at the production temperature. Create content-matched style variants. Hide model identity. For questions with an objective answer, supply an independently produced reference rather than letting a persuasive candidate teach the judge its mistake.

LeoThe foundational paper's math experiments make that last one concrete. A judge could solve a problem alone yet be misled when grading wrong candidate work; an independent reference reduced failures in that tiny ten-question test.

MayaTiny is the operative word. It supports the mechanism, not a universal percentage.

LeoNext comes slicing. Report performance separately for obvious quality gaps, close calls, task categories, answer sources, and consequential versus low-stakes decisions.

MayaAdd abstention. An order conflict, low margin, missing evidence, or ensemble split should produce “needs review,” not a forced winner that looks precise in a dashboard.

LeoAnd preserve the artifacts: prompt, rubric, candidate order, model version, raw verdict, explanation, and human resolution. Otherwise calibration drifts invisibly.

MayaReturn to our two safety reports. The production design should blind their source, randomize their slots, score factual coverage against sealed tool logs, test compact and expanded variants, and route unstable verdicts to a human who can inspect the evidence.

LeoThe ensemble should cross model families and prompt formulations, but the team still needs a human-labeled audit sample. Diversity is a hypothesis to measure, not a property declared in the architecture diagram.

MayaThen keep the two dials on the release report. Judge reliability: does the evaluator remain stable, calibrated, and accurate within each slice? Benchmark validity: do the cases, rubric, and human baseline support the deployment claim you are making?

LeoThat prevents an eighty-five-percent agreement result from being laundered into “the system is safe.”

MayaIt also preserves the positive result. L-L-M judges can make open-ended evaluation faster, more repeatable, and easier to inspect. The lesson is not “never use them.”

LeoIt is “never let a fluent verdict skip its own evaluation.”

MayaNext, the target gets access to the scoring channel itself. Reward tampering turns a biased measurement instrument into something the agent can actively influence.

LeoSo the quiet tilt of the table becomes a hand on the scale.

MayaWhich single invariance test would be most likely to overturn a consequential verdict in your current L-L-M judging pipeline?