How do you mitigate positional bias and self-preference in LLM-as-a-Judge?

Short answer: Positional bias and self-preference in LLM judges are mitigated by running pairwise evaluations in dual positions (swapping Model A and Model B), averaging bidirectional scores, setting low generation temperatures, using neutral reference rubrics, and ensembling distinct model families (e.g. GPT-4o + Claude 3.5 Sonnet) as judges.

Detail

LLM-as-a-Judge systems exhibit systematic biases that skew evaluation results:

Core Judge Biases

1. Position Bias: LLMs systematically favor whichever candidate response appears first (Candidate A) or second (Candidate B) in the evaluation prompt context, regardless of quality.
2. Self-Preference Bias: LLMs systematically rate candidate outputs generated by their own model family higher than outputs from competitor model families.
3. Verbosity Bias: LLMs systematically favor longer, wordy responses over concise, precise answers.

Mitigation Techniques

[Candidate Pair (A, B)] ──► Pass 1: Rate (A, B) ──┐
                                                 ├──► Average / Consolidate Score
[Candidate Pair (B, A)] ──► Pass 2: Rate (B, A) ──┘

• Bidirectional Swap Evaluation: Evaluate every candidate pair twice $(A, B)$ and $(B, A)$. Assign wins only if a candidate wins in both positions (or average position-adjusted win rates).
• Multi-Judge Ensembling: Combine scores from different model families (e.g., OpenAI + Anthropic + Meta) to cancel out vendor-specific self-preference bias.
• Strict Length Normalization: Include explicit prompt guidelines instructing the judge to penalize fluff and reward concise accuracy.

Example

Python implementation of swap-aware pairwise scoring:

def evaluate_pairwise_unbiased(llm_judge_fn, prompt: str, resp_a: str, resp_b: str) -> str:
    # Pass 1: Candidate A first
    winner_1 = llm_judge_fn(prompt, candidate_a=resp_a, candidate_b=resp_b) # returns 'A', 'B', or 'Tie'

    # Pass 2: Candidate B first (swapped)
    winner_2_raw = llm_judge_fn(prompt, candidate_a=resp_b, candidate_b=resp_a)
    winner_2 = 'A' if winner_2_raw == 'B' else ('B' if winner_2_raw == 'A' else 'Tie')

    if winner_1 == winner_2:
        return winner_1
    return "Tie (Positional Inconsistency Detected)"

Interview tips

• Highlight Cohen's Kappa ($\kappa$) or Spearman rank correlation to measure inter-annotator agreement between LLM judges and human annotators.
• Explain how calibration datasets (ground-truth human preferences) establish baseline judge reliability.

⬅ Back to Evaluation and Testing · All topics