LLM as Judge

Pairwise verdicts are local ("A beat B"). To turn a pile of them into a global leaderboard you fit an aggregation model. The Bradley–Terry model estimates a latent strength $s_i$ per system such that

$P(i \text{ beats } j) = \frac{e^{s_i}}{e^{s_i} + e^{s_j}}$

which is the logistic form behind the Elo-style ratings reported by Chatbot Arena. Pointwise scoring needs no such machinery, which is part of its appeal for continuous monitoring, but that convenience is exactly why teams over-trust an uncalibrated absolute number.

How a judge scores, mechanically

A judge prompt has a predictable anatomy, and each part controls a specific failure mode:

• Role and task instructions, establish that the model is an impartial evaluator, not a participant.
• An ordered rubric, the criteria, most important first (e.g. correctness before style), with concrete descriptors so "good" is not left to the model's imagination.
• An output schema, force structured output (JSON with a verdict/score field and a short rationale) so results are parseable and comparable.
• Decoding settings, pin temperature to $0$ so the same inputs give the same verdict; a judge that flips on re-runs cannot be a measurement instrument.
• Few-shot anchors (optional but powerful), one or two worked exemplars with gold labels that pin the scale, dramatically reducing drift and rubric sensitivity.

Ask for the rationale before the verdict so the label is conditioned on the reasoning rather than rationalized after the fact, but treat that rationale as a debugging artifact, not proof the verdict is correct.

Known biases and how to fight them

A judge is an LLM, so it inherits systematic biases. These are measurable and, mostly, mitigable, but only if you look for them.

Worked example: a pairwise verdict

Here is a concrete pairwise judge prompt. The rubric puts correctness first and explicitly instructs the judge to ignore length, a direct defense against verbosity bias.

System:
You are an impartial evaluator. Compare two AI assistant responses to the
same user question and decide which is more helpful and correct.

Judge on these criteria, in order of importance:
  1. Correctness, factual and technical accuracy.
  2. Relevance, directly addresses the user's actual problem.
  3. Completeness, covers the root cause and a working fix.
  4. Clarity, well organized and easy to follow.

Rules:
  - Do NOT let response length affect your decision. A shorter, correct
    answer beats a longer answer that is padded or partly wrong.
  - Do NOT favor a particular position. Judge only on the criteria above.
  - If the two responses are of equal quality, output "tie".

Return strict JSON: {"reason": "<one sentence>", "verdict": "A" | "B" | "tie"}

[USER QUESTION]
Why does my Python multiprocessing script hang when a worker puts a very
large object on a multiprocessing.Queue and the parent then calls join()?

[RESPONSE A]
multiprocessing.Queue sends objects through an OS pipe with a bounded
buffer. A large object does not fit the buffer, so the worker's put()
blocks in a background feeder thread until the parent reads the data. If
the parent calls join() before draining the queue, the worker can never
finish and the parent waits forever, a deadlock. Fix it by draining the
queue with get() in the parent before join(), or by using a sentinel value
so the consumer reads until the producer signals completion.

[RESPONSE B]
This is almost always caused by Python's Global Interpreter Lock, which
prevents two processes from running at once and serializes all your I/O.
Because the GIL is held while the large object sits in memory, join()
cannot acquire it and the program stalls. The most reliable fix is to
switch from multiprocessing to the threading module, which avoids pickling
overhead entirely, and to wrap your queue calls in try/except so the
interpreter can recover. You may also want to raise your system swap and
call gc.collect() before joining.

Response B is longer and more confident, but it is wrong: the GIL is per-process and does not span processes, threading would not fix a pipe-buffer deadlock (and reintroduces GIL contention), and swap/gc.collect() are irrelevant. A well-designed judge picks the shorter, correct answer:

{
  "reason": "A correctly identifies the bounded pipe buffer and feeder-thread deadlock and gives a working drain-before-join fix, whereas B misattributes the hang to the GIL and proposes changes that do not address the buffer.",
  "verdict": "A"
}

Now test for position bias by swapping the order. Re-run the exact same evaluation with the candidates reversed, so the correct answer is now in position 2:

{
  "reason": "The response explaining the pipe buffer and feeder thread is technically correct; the GIL explanation is not.",
  "verdict": "A"
}

The winner is stable under the swap, so this verdict is decisive and not an artifact of position. Had the judge instead picked "whatever is in position 1" both times, favoring B in the original order and A after the swap, that flip would expose position bias, and the correct move is to record the pair as a tie and count it against the judge's swap-consistency rate. In practice you always run both orderings and only credit a win when the two agree.

Calibrate against human labels

None of the above means anything until you know the judge agrees with humans. Before trusting a judge, build a human-labeled gold set of a few hundred representative examples, ideally labeled by two or more raters so you can measure human–human agreement as your ceiling, then run the judge on it and compare.

Raw agreement (the fraction of examples where the judge's label matches the human's) is the headline number, but it is inflated when one label dominates. Cohen's kappa corrects for chance:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ is the observed agreement and $p_e$ is the agreement you would expect if both raters labeled at random according to their observed marginal rates. Interpret $\kappa$ against the human–human ceiling on the same task, not against 1.0: the MT-Bench study found GPT-4's agreement with human preferences was over 80%, on par with the agreement between two independent humans, which is the bar that actually matters. For pointwise judges, also check rank correlation (Spearman or Kendall) between judge scores and human scores, since a judge can be uniformly too generous yet still rank systems correctly.

Set an agreement threshold before you look at the results, and only promote a judge that clears it on held-out data.

Building and validating a judge

1. 1Define the rubricWrite explicit, ordered criteria with a concrete descriptor for each score level. Ambiguous rubrics are the single largest source of judge noise.
2. 2Draft the judge promptFix the role, criteria, output schema, and decoding. Ask for a short rationale before the verdict, and pin temperature to zero for reproducibility.
3. 3Collect a human gold setHave humans label a few hundred representative examples, ideally with two or more raters so you can measure human-human agreement as your ceiling.
4. 4Measure agreementRun the judge on the gold set and compare to human labels using accuracy and a chance-corrected metric such as Cohen kappa. Judge against the human ceiling, not against perfection.
5. 5Probe for biasSwap answer order to quantify position bias, add length-matched pairs to expose verbosity bias, and check whether the judge favors its own model family.
6. 6Iterate or rejectIf agreement is below your threshold, revise the rubric and anchors and re-measure. A judge that fails calibration is not ready, no matter how reasonable its rationales sound.
7. 7Monitor and re-validateRe-run calibration whenever you change the judge model, the prompt, or the population of inputs. A judge calibrated on last quarter's traffic can drift silently.

Further reading

• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, the foundational study of pairwise LLM judges, human agreement, and biases: arxiv.org/abs/2306.05685
• Wang et al. (2023), Large Language Models are not Fair Evaluators, quantifies position bias and proposes calibration via order swapping: arxiv.org/abs/2305.17926
• Liu et al. (2023), G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, pointwise judging with chain-of-thought rubrics and human-correlation analysis: arxiv.org/abs/2303.16634