How to mitigate Verbosity Bias and Position Bias in LLM-as-a-Judge evaluations?

Short answer: Verbosity bias (favoring longer, verbose outputs) and position bias (favoring whichever candidate response is presented first in position A) are systematic LLM judge errors; they are mitigated by swapping candidate presentation order (position swapping), enforcing length-normalized scoring rubrics, and using multi-judge ensemble scoring.

Detail

Using LLMs to grade other LLMs introduces systematic cognitive biases:

Position Bias:  Judge prefers Candidate A simply because it appeared first in the prompt.
Verbosity Bias: Judge assigns 5/5 to a long rambling 500-word answer over a precise 20-word answer.

Mitigation Strategies

1. Position Swapping (Bidirectional Eval): Run evaluation twice, swapping candidates: Eval 1: (A, B) and Eval 2: (B, A). Accept win only if candidate wins in both positions.
2. Length-Controlled Rubrics: Instruct the judge explicitly: "Do not award higher scores for longer responses. Evaluate conciseness and precision."
3. Reference-Guided Rubrics: Provide explicit ground-truth reference answers to anchor judge scoring.

Example

Python position-swapped evaluator loop:

def evaluate_pairwise_unbiased(candidate_a: str, candidate_b: str, judge_fn) -> str:
    # Pass 1: A first, B second
    res1 = judge_fn(first=candidate_a, second=candidate_b) # Returns 'first' or 'second'
    # Pass 2: B first, A second
    res2 = judge_fn(first=candidate_b, second=candidate_a) # Returns 'first' or 'second'

    if res1 == "first" and res2 == "second":
        return "Model A Wins"
    elif res1 == "second" and res2 == "first":
        return "Model B Wins"
    else:
        return "Tie / Inconsistent Position Bias"

Interview tips

• Highlight Self-Preference Bias (LLM judges tend to award higher scores to completions generated by their own model family).
• Discuss using Few-Shot Judge Exemplars to calibrate rating scales.

Related Concepts

• [[How does Automatic Prompt Engineer (APE) optimize prompt selection using search?]] (#118): How does Automatic Prompt Engineer (APE) optimize prompt selection using search?
• [[What is human evaluation (RLHF human rating) and how do you design annotation rubrics?]] (#175): What is human evaluation (RLHF human rating) and how do you design annotation rubrics?
• [[How to implement Elo rating systems to benchmark model performance?]] (#179): How to implement Elo rating systems to benchmark model performance?

⬅ Back to Evaluation and Testing · All topics