Skip to content

Evaluation & Testing interview questions

LLM-as-a-Judge, Ragas metrics (Faithfulness, Relevance), benchmarks (MMLU, HumanEval), unit testing, and red teaming.

What to study

  • Evaluating RAG systems with Faithfulness, Context Recall, and Answer Relevance.
  • Reducing bias and variance in LLM-as-a-Judge setups.
  • Automated prompt regressions and CI/CD evaluation test suites.

Questions and answers

  1. How do Faithfulness, Context Recall, and Answer Relevance differ in Ragas?

    Intermediate · 3 min read

  2. What is LLM-as-a-Judge and how is it used for automated evaluations?

    Beginner · 2 min read

  3. What are standard LLM benchmarks like MMLU, GSM8K, and HumanEval?

    Beginner · 2 min read

  4. How do you build a CI/CD prompt regression testing suite?

    Intermediate · 2 min read

  5. How do you mitigate positional bias and self-preference in LLM-as-a-Judge?

    Advanced · 2 min read

  6. What is context precision vs context recall in RAG?

    Beginner · 2 min read

  7. What is faithfulness vs answer relevance in LLM evals?

    Beginner · 2 min read

  8. What is ground-truth reference data in evaluation harnesses?

    Beginner · 2 min read

  9. How do you measure inter-annotator agreement (Cohen's Kappa) with LLM judges?

    Intermediate · 2 min read

  10. How do you build an automated red teaming harness against jailbreaks?

    Advanced · 2 min read

  11. What is exact match (EM) vs F1 score in extraction and classification evals?

    Beginner · 2 min read

  12. What is BLEU and ROUGE scoring and why are they inadequate for modern LLM evaluation?

    Beginner · 2 min read

  13. What is assertion testing in LLM unit tests?

    Beginner · 2 min read

  14. What is synthetic evaluation dataset generation and when should you use it?

    Beginner · 2 min read

  15. What is human evaluation (RLHF human rating) and how do you design annotation rubrics?

    Beginner · 2 min read

  16. How to measure model hallucination rate using NLI (Natural Language Inference) entailment models?

    Intermediate · 2 min read

  17. How to mitigate Verbosity Bias and Position Bias in LLM-as-a-Judge evaluations?

    Intermediate · 2 min read

  18. How to build an automated RAG evaluation harness measuring Context Precision and Recall?

    Intermediate · 2 min read

  19. How to implement Elo rating systems to benchmark model performance?

    Advanced · 2 min read

  20. How to build automated adversarial red teaming engines to discover safety guardrail bypasses?

    Advanced · 2 min read

Study Evaluation & Testing with progress tracking