LLM Evaluation — Interview Grill

70+ active-recall questions. Pair with LLM_EVALUATION_DEEP_DIVE.md. Answer each in <60 seconds out loud. Mark any you can't answer cleanly and re-read the relevant section.

Section A — Why LLM eval is hard (Q1–8)

  1. Why is evaluating an LLM harder than evaluating a binary classifier?
  2. Give three reasons reference-based metrics like BLEU and ROUGE fail for instruction following.
  3. What is "Goodhart's law" and how does it apply to LLM benchmarks?
  4. Why does prompt sensitivity matter for benchmark reporting?
  5. What does it mean that "capability ≠ helpfulness"? Give an example.
  6. Why does an LLM-judge become useless when the testee approaches the judge in capability?
  7. Describe the offline / online gap and why benchmarks alone don't predict product success.
  8. Cost-and-latency-wise, what makes LLM eval different from traditional ML eval?

Section B — Taxonomy (Q9–14)

  1. Distinguish capability eval, product eval, and safety eval.
  2. Distinguish reference-based, reference-free, pairwise, and programmatic eval. Give one example of each.
  3. What's the difference between offline eval and shadow / canary deployment?
  4. What does "verifiable instruction following" mean? Why is IFEval valuable?
  5. When would you use a closed-form (multiple choice) eval vs an open-ended eval?
  6. What's the difference between token-level, output-level, conversation-level, and session-level evaluation?

Section C — Capability benchmarks (Q15–28)

  1. What does MMLU measure? Why is MMLU-Pro the modern replacement?
  2. What is GPQA-Diamond? What does it measure that MMLU-Pro doesn't?
  3. Why are GSM8K and HumanEval saturated? What replaced them?
  4. How is SWE-Bench-Verified different from SWE-Bench? Why does verification matter?
  5. Why is LiveCodeBench important relative to HumanEval?
  6. What does RULER measure? Why is it more informative than vanilla NIAH?
  7. What is "Lost in the Middle"? How would you test for it?
  8. Difference between MMMU and MM-Vet?
  9. What does GAIA measure? What's special about its construction?
  10. Why is TAU-bench an interesting agent eval?
  11. What is the difference between TruthfulQA and SimpleQA?
  12. What does XSTest measure? Why is over-refusal eval important?
  13. Roughly, what's a defensible capability eval suite for a frontier model in 2026?
  14. Why might you weight HumanEval+ over HumanEval?

Section D — Instruction following and chat quality (Q29–34)

  1. What's the difference between IFEval and MT-Bench?
  2. What does AlpacaEval 2 length-controlled correct for? Why is it necessary?
  3. Why does multi-turn evaluation reveal different weaknesses than single-turn?
  4. How do you test persona / system-prompt adherence?
  5. What's Arena-Hard-Auto and how does it relate to Chatbot Arena?
  6. Give three programmatic checks you would always include in a chat eval.

Section E — LLM-as-judge (Q35–46)

  1. What is LLM-as-judge? Why does it work at all?
  2. List five biases of LLM judges.
  3. How do you mitigate position bias in pairwise comparison?
  4. How do you mitigate length bias?
  5. How do you mitigate self-preference / family bias?
  6. Walk me through how you'd calibrate an LLM judge.
  7. What is a multi-judge ensemble and why use it?
  8. What is Prometheus / G-Eval / PandaLM and how do they differ from "ask GPT-4"?
  9. When does an LLM judge stop working?
  10. What's the typical structured output format for a pairwise judge?
  11. Why might you strip formatting (markdown, headers) before judging?
  12. Suppose your judge agreement with humans is κ=0.45 — what do you do?

Section F — Pairwise and ELO (Q47–53)

  1. Why is pairwise more reliable than absolute scoring for open-ended quality?
  2. Sketch the ELO update formula.
  3. How is ELO computed from pairwise comparisons in practice (Bradley-Terry)?
  4. What does Chatbot Arena measure? What are its limitations?
  5. Why does Arena-Hard-Auto correlate so well with Arena ELO at <1% the cost?
  6. To distinguish 50% from 55% pairwise win-rate at 95% confidence, roughly how many comparisons?
  7. Sketch a Bradley-Terry MLE in pseudo-code.

Section G — Open-ended generation eval (Q54–57)

  1. Why don't BLEU and ROUGE work for instruction following?
  2. When does BERTScore / COMET make sense?
  3. What rubric would you use for an LLM judge scoring open-ended responses?
  4. How do you measure diversity vs quality for creative tasks?

Section H — Factuality (Q58–66)

  1. Difference between TruthfulQA, SimpleQA, FactScore, LongFact?
  2. Walk through SAFE.
  3. What does RAGAS measure? List the four metrics.
  4. Distinguish citation existence from citation faithfulness.
  5. Why is calibration a factuality proxy?
  6. What is Expected Calibration Error?
  7. Why does RLHF often hurt calibration?
  8. What's FACTS Grounding?
  9. How would you eval the factuality of a long-form answer (no single ground truth)?

Section I — Contamination (Q67–73)

  1. What is benchmark contamination?
  2. List four ways contamination can happen.
  3. What is Min-K%-prob? How does it detect membership in training data?
  4. How do you build a contamination-resistant eval going forward?
  5. Why do "perturbation tests" detect memorization?
  6. What is a canary string and how is it used?
  7. What does it mean to "decontaminate" a benchmark?

Section J — Robustness and statistics (Q74–82)

  1. How do you measure prompt sensitivity?
  2. Why does few-shot ordering affect benchmark scores?
  3. What is BBQ? What does it measure?
  4. Approximately, the 95% CI half-width for accuracy on n=200, p=0.5?
  5. What's pass@k? When does it matter?
  6. Multiple-comparisons problem: if you eval on 20 benchmarks at α=0.05, how many false positives by chance?
  7. Why is reporting CIs alongside benchmark numbers important?
  8. If you sample 5 responses per prompt, what's the unit of analysis?
  9. Bootstrap CI vs Wilson interval — when would you use each?

Section K — Harnesses (Q83–86)

  1. What does lm-eval-harness do? Why is it the academic default?
  2. What's HELM and what makes it different from lm-eval-harness?
  3. What's Inspect (UK AISI) and when do you use it?
  4. Compare RAGAS, TruLens, DeepEval for RAG eval.

Section L — Online eval and A/B (Q87–95)

  1. What surrogate quality metrics would you log for a chat product?
  2. What does "regenerate rate" tell you?
  3. How do you sample production traffic for online eval?
  4. How do you size an A/B test for a chat product (binary success metric, p≈0.3, lift δ=2%)?
  5. What is CUPED? Why does it matter for LLM A/B tests?
  6. Why does latency matter as a guardrail in LLM A/B?
  7. Why is "selection bias from refusals" a concern?
  8. Walk through offline → shadow → canary → A/B for an LLM product.
  9. What's sequential testing (mSPRT)? When would you use it?

Section M — Product eval design (Q96–100)

  1. Walk me through designing the eval for a customer-support chatbot. Use the four-layer pattern.
  2. How do you build a 500-prompt golden set for a chatbot?
  3. How often do you refresh the golden set? Why?
  4. What does it mean to "calibrate the LLM judge to humans" for a product? Walk through.
  5. List five failure modes a good eval suite catches.

Quick fire (Q101–115)

  1. One line: what does IFEval measure?
  2. One line: what does RULER measure?
  3. One line: what does FactScore measure?
  4. One line: what is pass@k?
  5. One line: what is length-controlled win rate?
  6. One line: SimpleQA vs TruthfulQA.
  7. One line: SAFE.
  8. One line: Min-K%-prob.
  9. One line: ELO.
  10. One line: CUPED.
  11. One line: Lost in the Middle.
  12. One line: HELM.
  13. One line: Inspect framework.
  14. One line: Arena-Hard-Auto.
  15. One line: Bradley-Terry.

Self-grading

  • 90+ correct: ready for frontier-lab eval rounds.
  • 70–89: re-read §5 (judges), §11 (stats), §15 (case study).
  • 50–69: re-read full deep dive then redo.
  • <50: spend two days on the deep dive, then come back.

5-day drill plan

  • Day 1: §1–4 (why hard, taxonomy, knowledge benchmarks). Drill A, B, C.
  • Day 2: §5–7 (LLM judge, pairwise, open-ended). Drill E, F, G.
  • Day 3: §8–9 (factuality, contamination). Drill H, I.
  • Day 4: §11 + §13–14 (stats, online, A/B). Drill J, L.
  • Day 5: §15 case study + §16 senior signals + Quick fire. Whiteboard a product eval suite end-to-end.