Evaluation Frameworks — Designing Systematic Evaluation for Agentic Systems

Why this matters. An agent is a program whose behavior is sampled, not computed: the same input can produce different tool calls, reasoning paths, and answers on every run. That nondeterminism, combined with long multi-step trajectories and open-ended outputs, breaks the two habits engineers reach for first — unit tests with fixed expected values, and “just try it and see if it looks right.” A serious evaluation framework replaces both with something reproducible: a dataset of tasks, a harness that runs the agent over them, graders that turn behavior into numbers, and reporting that tells you — with a confidence interval, not a vibe — whether the new version is better than the old one. This chapter is the deep dive on building that framework, from first principles through the 2025–2026 tooling landscape, a runnable reference implementation, real production war stories, and an interview-grade Q&A bank.

How to read this chapter. Sections 1–5 build the conceptual spine (why eyeballing fails, the six-part anatomy, grading approaches, outcome-vs-trajectory, dataset design). Section 6 is a fully runnable harness. Sections 7–8 give you statistical rigor and a failure-mode catalogue. Section 9 is the 2025–2026 landscape — the frameworks and LLM-as-judge research you should be able to name and date. Section 10 is build-it-in-practice on a real library (Inspect). Section 11 is production case studies and war stories. Section 12 is interview mastery (Q&A, a 60-second pitch, a system-design walkthrough, tradeoff tables, red/green flags). Section 13 is further reading with live URLs.

1. Core intuition: why eyeballing fails for agents

Suppose you tweak your agent’s system prompt and run three demo queries. All three look good. Ship it? The problem is that you have just estimated a success rate from n = 3, with no control over which three, on a system that is stochastic. Four things make ad-hoc inspection actively misleading for agents:

1. Variance across runs. At temperature > 0, and even at temperature 0 (tool ordering, retrieval nondeterminism, backend batching, floating-point non-associativity across GPU kernels), the same case passes on Monday and fails on Tuesday. A single run tells you almost nothing about the rate.
2. The output is a distribution, not a value. “Correct” for a free-text answer or a multi-tool workflow is not string equality. You need a grader that maps behavior → score, and that grader is itself a component you must validate.
3. The interesting failures are rare and structured. Agents fail on the 5% of cases with an ambiguous instruction, a tool that returns an error, or a required back-off. Cherry-picked demos never hit those.
4. Process ≠ outcome. An agent can reach the right answer through a broken, expensive, or unsafe path (guessed instead of calling the tool; leaked a secret; made 40 calls). Looking at the final answer hides all of it.

What a framework buys you, concretely:

• Reproducibility — a fixed dataset + pinned config means “run the eval” gives the same number tomorrow, so you can attribute changes to your change.
• Regression gating — a numeric threshold in CI blocks a PR that drops task success from 82% to 74%.
• Comparability — version A vs version B, or model X vs model Y, scored on the same cases with the same graders.
• Statistical honesty — an interval and a sample size instead of “seems better.”
• Debuggability — per-case, per-step traces so a red number leads you to the exact failing step.

Rule of thumb: if you cannot re-run it and get the same number, it is a demo, not an evaluation.

The maturity ladder. Teams tend to climb these rungs in order, and knowing which rung you are on is itself a useful diagnostic:

1. Vibes — a human looks at a few outputs and forms an opinion. Fine for the first week, dangerous after.
2. Golden set — a fixed list of cases with expected answers, run by hand. Reproducible-ish, not gated.
3. Automated offline eval — dataset + harness + graders, run on demand, results logged. You can compare versions.
4. CI-gated eval — the offline eval runs on every PR and blocks merges below a bar. Regressions can no longer ship silently.
5. Online / production eval — graders (usually cheap heuristics + sampled LLM judges) run on live traffic; drift and regressions are caught in hours, not at the next release.
6. Closed-loop — production failures are mined back into the offline dataset automatically, so the eval set tracks the real distribution. The flywheel most mature teams are chasing.

The rest of this chapter is about how to build rungs 3–6 well.

Saying it out loud. The question here is ‘why isn’t trying it a few times enough,’ and the answer is that you ran n equals three on a stochastic system and you don’t even know its variance. Four things break eyeballing: the same case passes Monday and fails Tuesday even at temperature zero, correctness for free text isn’t string equality so the grader is itself a component you have to validate, the interesting failures are the rare structured ones a demo never touches, and the process can be broken even when the answer is right. The rule to hand someone is: if you can’t re-run it and get the same number, it’s a demo, not an evaluation. And name the maturity ladder — vibes, golden set, automated offline, CI-gated, online, then the closed loop where production failures automatically become new eval cases.

2. Anatomy of an evaluation framework

Every mature eval stack — LangSmith, Braintrust, Inspect, Weave, Langfuse, OpenAI Evals, DeepEval — is a rearrangement of the same six parts. Learn the parts; the tools are implementations.

A few design principles that separate a framework from a script:

• Separate the runner from the grader. Run once, record the full trace, then score. This lets you add a new grader later and re-score old runs without re-invoking the (expensive, nondeterministic) agent. It also means a flaky judge does not force you to re-pay for agent execution.
• Persist raw traces, not just scores. The trace is your ground truth for debugging and for re-grading. Inspect calls these logs; LangSmith/Braintrust/Weave/Langfuse call them runs/traces/spans. A score without its trace is un-debuggable.
• Make cases and config content-addressable. Version the dataset and pin model/prompt/tool versions so a number is meaningless without knowing exactly what produced it. A good discipline: every result row carries dataset_version, agent_git_sha, model_id, prompt_hash, judge_model.
• Treat the grader as code under test. A grader that disagrees with humans is a broken instrument; measure its agreement before you trust its verdicts (Section 4 and 3.4).
• Design for re-grading and back-testing. When you improve a rubric, you want to re-score every historical run and see whether your past decisions still hold. That is only possible if traces are persisted and graders are pure functions over them.

 dataset ──▶ harness ──▶ raw traces ──┬─▶ programmatic grader ─┐
 (versioned)   (per case:              ├─▶ LLM judge ──────────┼─▶ aggregate ─▶ report ─▶ gate
                output + trajectory)   └─▶ human review ───────┘   (+CI, +diff vs baseline)

Saying it out loud. Every eval stack — LangSmith, Braintrust, Inspect, Langfuse, take your pick — is the same six parts rearranged: dataset, harness, graders, aggregation, reporting, and a regression gate. The design principle that matters most is separating the runner from the grader: run the agent once, persist the full trace, then score it. That way you can add a new grader six months later and re-score every old run without re-paying for an expensive, nondeterministic agent execution. The failure mode if you skip it is that a score without its trace is un-debuggable — you know something regressed and you have no idea where. And version everything, because a number with no dataset version, model ID and prompt hash attached isn’t comparable to anything.

3. Grading approaches

Three families, and the whole art is choosing the cheapest one that is valid for the property you care about. “Valid” is the load-bearing word: a grader is valid for a property if its score moves when — and only when — that property changes. An exact-match grader is invalid for “helpfulness”; a length-biased judge is invalid for “conciseness.” Validity, not sophistication, is the goal.

Saying it out loud. There are exactly three ways to grade — code, a model, or a human — and the whole art is picking the cheapest one that’s actually valid. Valid is the load-bearing word: a grader is valid for a property if its score moves when, and only when, that property changes. Exact match is invalid for helpfulness; a length-biased judge is invalid for conciseness. So the ordering is always programmatic if you can, a judge if you must, and humans to calibrate the judge. Validity, not sophistication, is the goal.

3.1 Rule-based / programmatic

Deterministic code checks the output or trajectory: exact match, regex, JSON-schema validation, numeric tolerance, “did it call search before answer”, unit tests over generated code, assert tool_calls == expected.

• Pros: free, instant, perfectly reproducible, no bias, debuggable.
• Cons: only works when correctness is formally checkable. Brittle to paraphrase (“Paris” vs “Paris, France”). Can’t judge tone, helpfulness, or open-ended reasoning quality.
• Use when: structured outputs, code (run the tests), math with a checkable answer, tool-call assertions, safety string filters, latency/cost budgets. Always prefer a programmatic check when one exists — it is the gold standard for the slice of behavior it can cover.

A subtlety worth internalizing: many properties look un-checkable but have a checkable proxy. “Did it cite a real source?” is judge-territory, but “does every URL it emitted resolve to HTTP 200 and appear in the retrieved context?” is a programmatic check that catches most citation hallucinations for free. Before reaching for a judge, spend five minutes asking whether a proxy check exists.

Saying it out loud. Programmatic grading is the gold standard for whatever slice it can cover — free, instant, perfectly reproducible, zero bias, and debuggable. The limit is that it only works when correctness is formally checkable, and it’s brittle to paraphrase: ‘Paris’ versus ‘Paris, France’ fails a string match while both are right. The move worth stealing is hunting for a checkable proxy before you reach for a judge. ‘Did it cite a real source’ sounds like judge territory, but ‘does every URL it emitted resolve to a 200 and appear in the retrieved context’ is a five-line check that catches most citation hallucinations for free.

3.2 LLM-as-a-judge

A strong LLM scores the output against a rubric or reference. Popularized by Zheng et al. 2023 (MT-Bench / Chatbot Arena), which reported that GPT-4 as a judge agreed with human preferences ~80% of the time — about the level humans agree with each other — and also catalogued its failure modes. G-Eval (Liu et al. 2023) added chain-of-thought and form-filling to improve judge–human correlation.

Two axes matter — what you compare against and what shape the question takes:

• Reference-free / rubric (“Rate faithfulness 1–5 given the context”). No gold answer needed; the rubric carries the standard.
• Reference-based (“Is this answer consistent with this gold answer?”). Stronger signal when a gold answer exists.
• Pointwise (“Score this answer 0–1”). Simple, but scores drift and are hard to calibrate across runs.
• Pairwise (“Is A or B better?”). Often more reliable than absolute scores because relative judgments are easier for models and reduce scale drift — this is the basis of Chatbot Arena’s Elo. The cost is O(n²) comparisons if you want a full ranking; in practice you compare each candidate against a fixed baseline.

Judge bias taxonomy (expanded)

Biases are not rare edge cases — they are the default behavior of an unconstrained judge. Know them by name, know the mechanism, know the fix.

General hygiene, non-negotiable: temperature 0, structured JSON output, a rubric with concrete pass/fail conditions and few-shot anchors, and — the part everyone skips — validate the judge against a human-labeled gold set (Section 3.4) before trusting it. A judge you have not measured is an unvalidated instrument, and shipping decisions on it is measuring with an unmarked ruler.

• Pros: handles open-ended text, scales cheaply (~$0.001–$0.02/case), fast to author.
• Cons: biased, nondeterministic, costs money, can be gamed, drifts when the judge model is upgraded.
• Use when: open-ended quality (helpfulness, faithfulness, coherence), no programmatic check exists, and you have validated agreement with humans.

Saying it out loud. LLM-as-a-judge is how you score open-ended quality at a scale humans can’t touch, for roughly a cent a case. The anchor number is Zheng et al. 2023, MT-Bench: GPT-4 as a judge agreed with human preferences about 80% of the time, which is roughly how often humans agree with each other. But a judge is a biased instrument, not an oracle, and you should be able to name the biases cold — position bias, verbosity bias, self-preference where a model rates its own family higher, and leniency where every score clusters at 4 or 5 out of 5. The hygiene is temperature zero, structured JSON output, a rubric with concrete fail criteria and few-shot anchors, a cross-family judge, and order-swapping in pairwise. And pairwise generally beats pointwise because relative judgments are easier for models and don’t drift on scale — the tradeoff is that a full ranking costs O(n²) comparisons, so in practice you compare everything against one fixed baseline.

3.3 Human evaluation

Domain experts or annotators label outputs. The ground truth other methods are validated against.

• Pros: highest validity; catches subtleties no rule or model will.
• Cons: slow, expensive, doesn’t scale, itself noisy (needs multiple raters + inter-annotator agreement).
• Use when: building/calibrating your gold set, high-stakes launches, adjudicating judge disagreements, and periodic audits of the automated stack.

Human labels are not automatically “truth” — they are noisy too. Two annotators disagree; the same annotator disagrees with themselves a week later. So you measure inter-annotator agreement (below) and treat the consensus of multiple raters as the reference. If your humans cannot agree with each other, no automated grader can be validated, because there is no stable target to validate against. Low human agreement is a signal that your rubric or task definition is under-specified — fix that first.

Saying it out loud. Humans are the ground truth everything else gets validated against — and they’re noisy too, which is the part people forget. Two annotators disagree, and the same annotator disagrees with themselves a week later. So you never treat one human label as truth: two or three raters per case, measure inter-annotator agreement, and use the consensus as the reference. The consequence that scores in an interview is that human-human agreement is your ceiling — if your humans only reach kappa 0.55 with each other, no automated grader can be validated above that, and low human agreement means your rubric or task definition is under-specified. Fix the rubric first, not the judge.

3.4 Calibrating a judge to human labels (the step everyone skips)

In plain language. This is the step almost everyone skips: checking whether your LLM judge actually agrees with people before you start making launch decisions with it. The Greek letter kappa below is just percent agreement corrected for the agreement you’d get by chance. It matters because if 90% of cases pass anyway, two raters who both blindly guess ‘pass’ agree 90% of the time while knowing nothing at all.

An LLM judge is a classifier (or regressor) whose predictions you are about to trust for launch decisions. You would never deploy a fraud classifier without a confusion matrix; do not deploy a judge without one either. Calibration is a concrete, repeatable procedure:

1. Build a gold set. Sample 100–300 cases stratified across your intent/difficulty taxonomy (not 300 easy ones). Include known-hard and known-adversarial cases.
2. Get human labels with redundancy. Have ≥2–3 qualified humans label each case independently against the same rubric the judge uses. Adjudicate disagreements to a consensus label. Record raw per-rater labels too — you need them for the agreement ceiling.
3. Run the judge on the same cases, blind to the human labels.
4. Measure agreement with metrics appropriate to the label type (below).
5. Compare judge–human agreement to human–human agreement. The human–human number is your ceiling: a judge cannot be more reliable than the ground truth is self-consistent. A judge that agrees with humans as often as humans agree with each other is as good as you can ask for.
6. Iterate the rubric, not the number. If agreement is low, inspect the disagreements, sharpen the rubric’s fail criteria, add few-shot anchors from the confusion cases, and re-measure. Do not tune the rubric until agreement looks good and then stop — hold out a fresh slice to confirm you did not overfit the rubric to the gold set.

Which agreement metric?

Rules of thumb for κ (Landis & Koch, widely used with caveats): <0.20 poor, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, >0.80 almost perfect. For a judge you intend to gate CI on, aim for substantial agreement and for judge–human κ to be within striking distance of human–human κ. If humans only reach κ=0.55 with each other, do not expect (or demand) 0.9 from the judge — fix the rubric.

Beyond agreement: bias-corrected estimates. Even a good-but-imperfect judge introduces a systematic error into your reported metric. A more advanced move (see Prediction-Powered Inference, Angelopoulos et al. 2023, and its LLM-eval descendants) is to use a small set of human labels to debias the large set of judge labels, producing a confidence interval on the true metric that is valid despite the judge’s imperfection. You do not need this on day one, but knowing it exists is a strong interview signal: it reframes the judge as a cheap, biased estimator whose bias you correct statistically rather than a black box you either trust or don’t.

Saying it out loud. The framing that makes this click: a judge is a classifier you’re about to trust with launch decisions, and you’d never deploy a fraud classifier without looking at its confusion matrix. So sample 100 to 300 cases stratified across intent and difficulty, including the known-hard and adversarial ones, get two or three humans to label them against the exact rubric the judge uses, run the judge blind, and measure agreement — Cohen’s kappa for binary, weighted kappa or Spearman for ordinal. The number that matters isn’t the raw kappa, it’s judge-human kappa compared to human-human kappa, because the humans are your ceiling. Rough scale: above 0.6 is substantial, above 0.8 almost perfect — and if humans only reach 0.55 with each other, you fix the rubric rather than demanding 0.9 from the judge. The trap is tuning the rubric until agreement looks good and stopping there; you need a fresh held-out slice, or you’ve just overfit the rubric to the gold set.

3.5 Panels and juries (Panel-of-LLM-evaluators)

A single large judge is not the only design. Verga et al. 2024, “Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models” (Cohere, arXiv:2404.18796) showed that a Panel of LLM evaluators (PoLL) — several smaller, diverse models voting — can outperform a single large judge (e.g., GPT-4) while being cheaper and, crucially, less biased, because intra-model self-preference is diluted across families. Practical panel designs:

• Majority vote over 3–5 diverse judges for a binary/categorical verdict.
• Average / median of scores for a numeric rubric (median is robust to one outlier judge).
• Judge + escalation (“Trust or Escalate,” Chaudhary et al., ICLR 2025): a cheap judge decides; only low-confidence or disagreeing cases escalate to a stronger judge or a human. This buys most of the accuracy of an expensive panel at a fraction of the cost.

Panels cost more per case, so use them where the decision is expensive (launch gates, leaderboard-style comparisons) and a single validated judge where throughput matters (online scoring of live traffic).

Saying it out loud. Instead of one big expensive judge, use several small diverse ones and let them vote. Verga et al. 2024 out of Cohere — ‘Replacing Judges with Juries’ — showed a panel of smaller diverse models beating a single GPT-4-class judge on agreement, at lower cost, and crucially with less bias, because self-preference gets diluted across model families. The designs are simple: majority vote for binary verdicts, median for numeric rubrics since the median survives one deranged judge, or the cost-aware version from ‘Trust or Escalate’ at ICLR 2025, where a cheap judge decides and only low-confidence or disagreeing cases escalate to a stronger judge or a human. The tradeoff to name is throughput versus decision weight — panels for launch gates where being wrong is expensive, one validated judge for online scoring of live traffic.

4. Outcome vs trajectory evaluation

For a single-turn model, the output is the behavior. For an agent, behavior is a trajectory: a sequence of (thought, tool_call, observation) steps ending in a final answer. You must evaluate both ends.

• Outcome evaluation (a.k.a. end-to-end, final-answer): did the agent produce the right result? “Was the refund issued?” “Is the returned SQL correct?” Ground truth on the terminal state.
• Trajectory evaluation (a.k.a. process): was the path correct? Did it call the right tools, in a sensible order, with valid arguments, without redundant or unsafe steps, within budget?

Why both matter:

Worked contrast. Task: “What was Acme Corp’s Q3 revenue?” Gold answer: $4.2M.

• Outcome grader: extract a dollar figure from the final answer, compare to $4.2M within tolerance → pass/fail.
• Trajectory grader: assert the trajectory contains a lookup_financials(company="Acme", quarter="Q3") call whose observation actually returned 4.2M, i.e. the answer was grounded in a real tool result, not hallucinated.

An agent that hallucinates “$4.2M” without calling the tool passes the outcome grader and fails the trajectory grader — exactly the case you must catch, because it will hallucinate a wrong number on the next company.

How trajectory graders work. Two styles, mirrored by real tooling:

• Reference-trajectory matching — compare the agent’s tool-call sequence to a “golden trajectory.” agentevals’ create_trajectory_match_evaluator offers strict (same calls, same order), unordered (same set, any order), subset, and superset modes, plus tool_args_match_mode to control how strictly arguments must match. Use unordered when which tools matter but order doesn’t; superset to require certain calls appear while allowing extras.
• LLM-as-judge over the trajectory — hand the whole (steps, tools, final answer) to a judge with a rubric (“Was each tool call justified? Any redundant or unsafe steps?”). agentevals’ create_trajectory_llm_as_judge (with TRAJECTORY_ACCURACY_PROMPT) does exactly this, optionally against a reference trajectory.

Golden trajectories are expensive to author and brittle (many valid paths exist), so a common compromise is: assert key tool calls happened (superset/must_call) and let a judge grade the rest.

A richer trajectory-metric vocabulary (useful to name in an interview):

• Tool-selection accuracy — of the tools it called, what fraction were appropriate?
• Tool-call validity — did arguments conform to the schema; did calls error?
• Step efficiency / redundancy — steps taken vs the minimal path; count of repeated or no-op calls.
• Goal drift — did later steps still serve the original objective, or did the agent wander?
• Recovery — after a tool error or empty result, did it retry/adapt or give up/confabulate?
• Grounding — is each factual claim in the final answer traceable to an observation in the trajectory?
• Budget adherence — total tokens, wall-clock, dollar cost, and number of tool calls vs a cap.

These are the metrics that separate “the answer was right” from “the agent is actually good,” and they are where trajectory-aware frameworks (Inspect, LangSmith agent evals, agentevals) earn their keep.

Saying it out loud. Short answer: you need both, and here’s the example that proves it. Ask an agent for Acme’s Q3 revenue where the gold answer is $4.2M. If it hallucinates $4.2M without ever calling the lookup tool, the outcome grader says pass and the trajectory grader says fail — and the trajectory grader is right, because that agent will confidently hallucinate a wrong number for the next company. Outcome asks did it get the right result; trajectory asks did it get there legitimately, efficiently, and safely. Outcome-only grading also cheerfully passes a run that made 40 redundant tool calls or leaked an API key on the way. Golden trajectories are expensive to author and brittle because many valid paths exist, so the practical compromise is asserting that the key tool calls happened — a superset match — and letting a judge grade the rest.

5. Building test cases and datasets

The dataset is the evaluation. A perfect harness over a biased dataset gives you confident, precise, wrong answers. Priorities:

Sourcing.

• Production logs (best): real user queries, de-duplicated and anonymized — this is your true distribution. Mine them for clusters of intents. This is the single highest-leverage source; a case sampled from real traffic is guaranteed to be in-distribution, which no synthetic case can promise.
• Expert-authored: SMEs write hard, realistic cases with references.
• Synthetic / LLM-generated: cheap coverage and edge-case expansion — but every synthetic case needs a human-checked reference, or you are grading against a hallucination.
• Public benchmarks (τ-bench / τ²-bench, SWE-bench, GAIA, WebArena, AgentBench, BFCL for tool use): good for external comparability; risky for internal decisions because of contamination and distribution mismatch.

Coverage. Build a taxonomy of intents × difficulty × required tools and make sure each cell is populated. Don’t let “capital of France” appear 200 times while the multi-step refund flow appears twice. Track a coverage matrix, and report metrics per cell, not just in aggregate — the mean is where hard-slice failures go to hide.

Edge cases — deliberately include:

• Ambiguous / underspecified instructions (does it ask a clarifying question?).
• Tool failures and timeouts (does it retry / degrade gracefully?).
• Adversarial / prompt-injection inputs.
• Empty, malformed, or out-of-scope requests (does it refuse?).
• Long-context and multi-hop tasks.
• Cases with no valid answer (does it say “I don’t know” instead of confabulating?).

Golden trajectories. For high-value flows, record a reference sequence of tool calls, not just the final answer — this powers trajectory matching and localizes regressions to a step.

Avoiding leakage & contamination.

• Keep a held-out set you never inspect while iterating; iterating on the test set overfits the eval (Section 8).
• Watch benchmark contamination: public benchmarks may be in the model’s training data, inflating scores. Prefer private, freshly-authored, or recently-timestamped cases for decisions. A canary-string or timestamp-after-cutoff check can detect gross contamination.
• Don’t leak the reference answer into the agent’s context (a surprisingly common bug when reusing dataset rows).
• Version and freeze datasets; a score is only comparable against runs on the same dataset version.

Labeling and dataset ops. Treat the dataset as a living product: an ID scheme, a schema (input, reference, metadata, optional golden_trajectory), a review process for new cases, and a changelog. When production surfaces a new failure mode, the fix is not just a code patch — it is a new case added to the dataset so the regression can never return silently. This is the closed loop from Section 1’s maturity ladder.

Sizing: start with 50–200 well-chosen cases per capability. Precision, not volume, early on — but note the statistical floor in Section 7: with 100 cases an 80% pass rate has a ±~8pp interval, which bounds how small a regression you can detect. When you need to gate on a few-point regression, you need several hundred cases per gated slice.

Saying it out loud. The dataset is the evaluation — a perfect harness over a biased dataset gives you confident, precise, wrong answers. The best source is production logs, deduplicated and anonymized, because a case sampled from real traffic is guaranteed to be in-distribution in a way no synthetic case can promise. Then you deliberately stock the edge cases everyone skips: ambiguous instructions, tool failures and timeouts, prompt injections, out-of-scope requests, and cases with no valid answer at all, to see whether it says ‘I don’t know’ instead of confabulating. Start at 50 to 200 well-chosen cases per capability, but know the statistical floor — at 100 cases an 80% pass rate carries about plus or minus 8 points, so gating on a 5-point regression needs several hundred cases in that slice. And keep a held-out set you don’t look at while iterating, or you’ll optimize the test instead of the agent.

6. A fully worked example: a small real evaluation harness

In plain language. This is a harness you could actually run, built to show the seams. It runs an agent over cases, records the whole trajectory rather than just the final answer, scores each run three ways — a programmatic check, a trajectory assertion, and an LLM judge from a different model family — and reports a pass rate with a Wilson confidence interval instead of a bare mean.

A runnable harness that (a) runs an agent over cases, (b) scores each with both a programmatic check and an LLM judge, (c) evaluates a trajectory assertion, and (d) aggregates with a confidence interval. The runner is separated from the graders so you can re-score without re-running the agent. The judge client is injectable so tests can mock it.

"""eval_harness.py — a minimal but real agent evaluation harness.

Design choices demonstrated:
  * runner is separated from graders (run once, score many times)
  * full trajectory is captured, not just the final answer
  * programmatic grader + LLM judge + trajectory check, composed
  * a different model family judges than the one under test (bias control)
  * results aggregated with a Wilson score confidence interval
"""
from __future__ import annotations
import json, math, random
from dataclasses import dataclass, field
from typing import Callable, Any

# ---------- data model ----------
@dataclass
class Case:
    id: str
    question: str
    reference: str                       # gold answer, for judge + outcome check
    must_include: list[str] = field(default_factory=list)   # programmatic outcome check
    must_call: list[str] = field(default_factory=list)      # required tool names (trajectory)

@dataclass
class Trace:
    case_id: str
    output: str
    steps: list[dict]                    # [{"tool": str, "args": dict, "obs": str}, ...]

# An agent maps a question -> (final_answer, trajectory_steps).
Agent = Callable[[str], tuple[str, list[dict]]]

def run_agent(agent: Agent, cases: list[Case]) -> list[Trace]:
    """Execute the agent once per case; persist the full trace for later scoring."""
    traces = []
    for c in cases:
        answer, steps = agent(c.question)
        traces.append(Trace(case_id=c.id, output=answer, steps=steps))
    return traces

# ---------- graders (pure functions over a Trace) ----------
def grade_keyword(case: Case, tr: Trace) -> float:
    """Programmatic outcome check: fraction of required substrings present."""
    if not case.must_include:
        return 1.0
    hits = sum(k.lower() in tr.output.lower() for k in case.must_include)
    return hits / len(case.must_include)

def grade_trajectory(case: Case, tr: Trace) -> float:
    """Programmatic process check: were all required tools actually called?
    Catches the 'right answer, wrong path' (hallucinated) failure mode."""
    if not case.must_call:
        return 1.0
    called = {s["tool"] for s in tr.steps}
    hits = sum(t in called for t in case.must_call)
    return hits / len(case.must_call)

JUDGE_PROMPT = """You are a strict grader. Judge only factual consistency with the reference.
Question: {q}
Reference answer: {ref}
Assistant answer: {ans}
Return ONLY a JSON object: {{"score": <float 0..1>, "reason": "<one sentence>"}}.
Give 1.0 only if the assistant answer is fully consistent with the reference;
0.0 if it contradicts or omits the key fact. Ignore style and length."""

def grade_llm_judge(client, model: str, case: Case, tr: Trace) -> tuple[float, str]:
    """LLM-as-judge outcome check. `client` is injected so it can be mocked.
    Use a DIFFERENT model family than the agent under test (self-preference bias)."""
    prompt = JUDGE_PROMPT.format(q=case.question, ref=case.reference, ans=tr.output)
    resp = client.chat.completions.create(
        model=model,
        temperature=0,                               # determinism
        response_format={"type": "json_object"},     # structured output
        messages=[{"role": "user", "content": prompt}],
    )
    data = json.loads(resp.choices[0].message.content)
    return float(data["score"]), data.get("reason", "")

# ---------- aggregation ----------
def wilson_interval(successes: float, n: int, z: float = 1.96) -> tuple[float, float]:
    """Wilson score 95% CI for a proportion — accurate for small n and p near 0/1,
    unlike the naive normal approximation."""
    if n == 0:
        return (0.0, 0.0)
    p = successes / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = (z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))) / denom
    return (max(0.0, center - half), min(1.0, center + half))

def evaluate(agent: Agent, cases: list[Case], judge_client=None,
             judge_model: str = "claude-haiku-4.5", pass_threshold: float = 0.999) -> dict:
    traces = run_agent(agent, cases)
    rows, kw_pass, traj_pass, judge_scores = [], 0, 0, []
    for case, tr in zip(cases, traces):
        kw = grade_keyword(case, tr)
        traj = grade_trajectory(case, tr)
        j, reason = (grade_llm_judge(judge_client, judge_model, case, tr)
                     if judge_client else (float("nan"), "no judge"))
        kw_pass += kw >= pass_threshold
        traj_pass += traj >= pass_threshold
        if judge_client:
            judge_scores.append(j)
        rows.append({"id": case.id, "keyword": kw, "trajectory": traj,
                     "judge": j, "reason": reason})
    n = len(cases)
    report = {
        "n": n,
        "keyword_pass_rate": kw_pass / n,
        "keyword_ci95": wilson_interval(kw_pass, n),
        "trajectory_pass_rate": traj_pass / n,
        "trajectory_ci95": wilson_interval(traj_pass, n),
        "judge_mean": (sum(judge_scores) / len(judge_scores)) if judge_scores else None,
        "rows": rows,
    }
    return report

# ---------- pass@k, the unbiased estimator (Chen et al. 2021) ----------
def pass_at_k(n: int, c: int, k: int) -> float:
    """Given n samples of which c passed, unbiased estimate of pass@k."""
    if n - c < k:
        return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)

# ---------- demo: a toy agent + dataset ----------
def toy_agent(question: str) -> tuple[str, list[dict]]:
    if "revenue" in question.lower():
        steps = [{"tool": "lookup_financials",
                  "args": {"company": "Acme", "quarter": "Q3"}, "obs": "4.2"}]
        return "Acme Corp Q3 revenue was $4.2M.", steps
    return "I don't know.", []

if __name__ == "__main__":
    cases = [Case(id="rev-1",
                  question="What was Acme Corp's Q3 revenue?",
                  reference="$4.2M",
                  must_include=["4.2"],
                  must_call=["lookup_financials"])]
    # judge_client=None -> programmatic-only run (CI-friendly, free, deterministic)
    print(json.dumps(evaluate(toy_agent, cases, judge_client=None), indent=2, default=str))

What this illustrates that matters in practice: (1) the runner persists traces, so adding a grader later doesn’t require re-invoking the agent; (2) the trajectory grader independently catches the hallucination case the outcome grader would pass; (3) the judge is injectable and defaults off, so the deterministic programmatic slice can gate CI for free while the (paid, noisier) judge runs on a schedule; (4) every rate ships with a Wilson interval, so “82%” is reported as “82% (95% CI 74–88%).”

Extending this toward production. The gap between this harness and a real one is mostly plumbing, and each piece maps onto a real framework’s feature: concurrency with rate-limit-aware retries (Inspect’s --max-connections, eval_set); resumable runs so a crash at case 900 doesn’t waste the first 899 (Inspect log-based resume); caching of agent and judge calls keyed on (input, config) so re-scoring is free; a persistent store for traces (Langfuse/LangSmith/Weave); and a diff view that shows which cases flipped between two runs, not just the aggregate delta. When you find yourself building three of these, adopt a framework instead — Section 9.

Saying it out loud. The structural decision to point at here is that the runner and the graders are separate. Run the agent once, persist the full trace, and make every grader a pure function over that trace — then when you sharpen a rubric next quarter you re-score every historical run for free instead of re-paying for a nondeterministic agent. Two more details worth naming out loud: the judge is deliberately a different model family from the agent under test, which is the fix for self-preference bias, and the aggregate is a Wilson interval rather than the normal approximation, because the normal one misbehaves when n is small or the rate sits near 100%. That’s the whole difference between a script that prints a number and a harness you can make decisions with.

7. Statistical rigor

In plain language. This section is about not fooling yourself with numbers. The formulas amount to two ideas: how fuzzy a single pass rate is given how many cases you ran, and how to compute pass@k honestly when the agent gets multiple attempts. If the notation is unfamiliar, read ( SE ) as ‘how much this number wobbles’ and the binomial ratio in the pass@k estimator as ‘the chance all ( k ) of your draws happened to land on failures’.

Agent evals are estimation under noise. Treat every number as a sample statistic.

Variance across runs. Run each case multiple times (seeds/repeats). Report the mean and the spread. If run-to-run variance on the same version rivals the gap between two versions, you cannot distinguish them — you need more samples, not a better story. There are two distinct sources of variance to keep separate: sampling variance (finite number of cases — shrinks as you add cases) and stochastic variance (the agent is nondeterministic on a fixed case — shrinks as you add repeats per case). A tight CI on the pass rate requires attacking both.

How many cases? For a proportion (pass rate) (p), the standard error is [ SE = \sqrt{\frac{p(1-p)}{n}} ] so the 95% margin is (\approx 1.96,SE). At (p=0.8): (n=100) gives ±~8pp, (n=400) ±~4pp, (n=1000) ±~2.5pp. To detect a 5pp regression you need several hundred cases, not fifty. Use the Wilson interval (Section 6 code) rather than the normal approximation when (n) is small or (p) is near 0 or 1.

Power, not just precision. Precision (CI width) tells you how fuzzy one number is; power tells you the probability you will detect a real regression of a given size. A rough planning rule for detecting a difference (\delta) in proportions at 80% power, 5% significance: you need roughly (n \approx 16,\bar p(1-\bar p)/\delta^2) paired discordant-informative observations. The practical upshot is the same as above — small effects need big (n) — but framing it as power is what lets you answer “is my eval even capable of catching the regression I care about?” before you run it.

Comparing two agents. They ran on the same cases, so the observations are paired — use a paired test, which is far more powerful than treating the runs as independent. For pass/fail, McNemar’s test on the discordant pairs (cases A passed and B failed, vs vice-versa). For continuous scores, a paired bootstrap or paired t-test over per-case score differences. Report the difference and its CI, not two separate rates side by side. The intuition for why pairing wins: cases vary wildly in difficulty, and pairing cancels that shared difficulty, isolating the A-vs-B effect.

Reporting confidence. Standard practice: bootstrap the case-level scores (resample cases with replacement, recompute the metric, take the 2.5/97.5 percentiles) to get a CI that needs no distributional assumption. When each case is run multiple times, use a clustered/hierarchical bootstrap (resample cases, then resample repeats within case) so you don’t understate variance by treating repeats as independent cases. Inspect, for example, supports bootstrapped stderr on metrics.

pass@k. For tasks where multiple attempts are allowed (code gen, agents that can retry), report pass@k: the probability that at least one of (k) samples succeeds. Use the unbiased estimator (Chen et al. 2021, HumanEval/Codex): draw (n \ge k) samples, count (c) correct, and [ \text{pass@}k = \mathbb{E}\left[,1 - \frac{\binom{n-c}{k}}{\binom{n}{k}},\right]. ] Computing (1-(1-\hat p)^k) directly is biased for small (n) — use the combinatorial form (Section 6 pass_at_k). pass@1 measures reliability; pass@k measures whether the capability exists at all under sampling. Report which you mean. Note the asymmetry: pass@k always looks better than pass@1, and a vendor quoting pass@k without saying so is flattering the number — always ask “k = ?”.

Multiple comparisons. If you test 20 metrics, some will look “significant” by chance (at α=0.05, one in twenty false positives is expected). Pre-register the primary metric; correct (Bonferroni / Benjamini-Hochberg) if you must screen many. The same discipline applies to slicing: if you dredge 30 subgroups looking for a win, one will oblige.

Saying it out loud. Every eval number is a sample statistic, so quote it with an interval or don’t quote it. The numbers to have memorized: at an 80% pass rate, 100 cases gives you about plus or minus 8 points, 400 gives about 4, and 1000 gives about 2.5 — which means a fifty-case suite literally cannot detect a five-point regression, no matter how confidently you present it. When you compare two agents they ran the same cases, so use a paired test — McNemar for pass/fail — and report the difference and its interval rather than two rates side by side; pairing cancels shared case difficulty and is far more powerful. For pass@k, use the unbiased combinatorial estimator from Chen et al. 2021, because computing ( 1-(1-\hat p)^k ) directly is biased at small ( n ). And the honesty note that lands: pass@k always looks better than pass@1, so when a vendor quotes it, the first question is ‘k equals what?’

8. Failure modes and pitfalls

• Judge bias, unmeasured. Shipping decisions on an LLM judge you never validated against humans. Fix: hold a human-labeled gold set; report judge–human agreement (κ); re-validate whenever the judge model is upgraded (a judge swap silently changes your metric).
• Self-preference. Judging model A’s outputs with model A. Fix: cross-family judge; or an ensemble/panel of judges (Section 3.5).
• Position & verbosity bias in pairwise. Fix: randomize/swap order and require agreement; control for length.
• Benchmark contamination. Public benchmark leaked into pretraining → inflated, meaningless scores. Fix: private, freshly-authored, time-stamped cases for real decisions.
• Overfitting to the eval (Goodhart’s law). Iterating against a fixed test set until the number is green means you optimized the test, not the agent. Fix: a held-out set inspected rarely; rotate/refresh cases; watch for the train/held-out gap widening.
• Flaky tests / nondeterminism mistaken for regression. A red CI that’s actually variance. Fix: multiple seeds; gate on the interval / a paired test, not a single run; set thresholds with margin.
• Outcome-only blindness. Passing lucky guesses and missing unsafe/expensive paths. Fix: trajectory graders + cost/latency/safety metrics alongside accuracy.
• Dataset skew. 80% easy cases → a high aggregate that hides failure on the hard 20%. Fix: stratify; report per-slice, not just the mean.
• Grader leakage. Reference answer bleeding into the agent’s prompt. Fix: strict separation of input vs reference fields.
• Unversioned everything. A score with no pinned dataset/model/prompt version is uninterpretable and uncomparable. Fix: content-address the dataset; log model, prompt, and tool versions with every run.
• Judge scored on the same axis it was optimized for. If you tuned the agent’s prompt using the judge’s feedback, the judge is no longer an independent evaluator of that axis. Fix: keep a separate, frozen evaluation judge from any judge used in the optimization loop.
• Metric–objective mismatch. Optimizing a proxy (e.g., “answer length” as a stand-in for “thoroughness”) that diverges from user value. Fix: periodically validate that the metric still correlates with human/business outcomes.

Saying it out loud. If you want one sentence: the most common way an eval lies to you is an unvalidated judge, and the second is an aggregate hiding a bad slice. Then it’s Goodhart — iterating against a fixed test set until the number turns green means you optimized the test, not the agent, so keep a held-out set and watch the gap between the two widen. Then nondeterminism mistaken for regression: a hard threshold on a single noisy run makes CI flap, and a flaky gate is worse than no gate, because it teaches the team to hit re-run on real regressions too. And contamination, where a public benchmark leaked into pretraining and the high score is measuring memorization. Each one has a named fix — kappa against humans, per-slice reporting, a held-out set, gating on the interval with a margin, and private freshly-authored cases.

9. The 2025–2026 landscape

This section is the “know the field” briefing: the frameworks a practitioner is expected to name, what each is actually good at, and the current state of LLM-as-judge research. URLs are in Section 13.

9.1 The framework map

The ecosystem sorts into three rough camps. Most serious teams compose one framework + one observability/trace platform, not a single monolith.

Open-source eval frameworks (code-first):

Hosted eval + observability platforms (trace-first, team workflows):

Companion judge/scorer libraries you should know by name: openevals and agentevals (LangChain), autoevals (Braintrust), Ragas metrics, DeepEval’s GEval. These give you validated-ish prebuilt judges (correctness, conciseness, hallucination, trajectory match) so you are not writing every rubric from scratch.

Selection heuristics:

• Inspect for rigorous, research-grade agent/safety evals and when reproducibility and sandboxed tool use matter (it is what several AI safety institutes and frontier labs use).
• DeepEval / promptfoo for CI-native, code/config-first testing you want running on every PR.
• LangSmith / Braintrust / Weave / Langfuse when you want a hosted (or self-hosted, for Langfuse) trace + experiment UI and team workflows; choose Langfuse if open-source/self-hosting is a hard requirement.
• Ragas specifically for RAG faithfulness/context quality; OpenAI Evals for registry-style benchmark runs.
• Most teams end up composing two or three: e.g., Inspect or DeepEval for the harness + Langfuse/LangSmith for traces + Autoevals/openevals for prebuilt judges.

Saying it out loud. Nobody ships one monolith; the real answer is one eval framework plus one trace platform. Inspect, from the UK AI Security Institute, is what you name for rigorous agent and safety evals — its whole vocabulary is Task equals Dataset plus Solver plus Scorer, with sandboxed tools and bootstrapped standard errors built in. DeepEval and promptfoo are the CI-native ones you’d run on every pull request, and Ragas is specifically for RAG faithfulness and context quality. On the hosted side it’s LangSmith, Braintrust, or Weave, and Langfuse if open-source self-hosting is a hard requirement, plus prebuilt judge libraries like openevals, agentevals and autoevals so you aren’t authoring every rubric from scratch. The senior signal is naming the composition — ‘Inspect for the harness, Langfuse for traces, openevals for prebuilt judges’ — instead of declaring a favorite.

9.2 Agent benchmarks worth naming

• τ-bench / τ²-bench (Sierra) — tool-agent-user interaction in retail/airline domains; measures reliability across repeated trials (pass^k), not just pass@1. A good example of a benchmark built around consistency, which is exactly the agent-specific failure mode.
• SWE-bench (+ SWE-bench Verified) — resolve real GitHub issues; the de-facto coding-agent yardstick; Verified is the human-filtered, contamination-aware subset.
• GAIA — general assistant tasks requiring tool use and multi-step reasoning.
• WebArena / VisualWebArena — agents acting in realistic web environments.
• Cybench, GDM CTF — cybersecurity agent capability (and safety) evals, shipped in Inspect Evals.
• BFCL (Berkeley Function-Calling Leaderboard) — tool/function-calling accuracy.

Use these for external comparability and capability sanity checks, never as your primary shipping gate — contamination and distribution mismatch (Sections 5, 8) make them unreliable for internal decisions.

Saying it out loud. Public agent benchmarks are for external comparability and capability sanity checks, never for your shipping gate. Name a handful with what each is for: τ-bench and τ²-bench from Sierra for tool-agent-user interaction, notably built around consistency because they report pass^k — all of k trials — not just pass@1; SWE-bench Verified for coding agents, which is the human-filtered, contamination-aware subset; GAIA for general assistant tasks needing multi-step tool use; WebArena for agents acting in realistic web environments; BFCL for function-calling accuracy. Treat any leaderboard number as of-a-date rather than current, because these move monthly. And the reason none of them can be your gate is the pair of problems from earlier — contamination and distribution mismatch: a public benchmark measures the public benchmark, and your product runs on your own traffic.

9.3 State of LLM-as-judge (research you should be able to cite)

The field has moved from “GPT-4 agrees with humans ~80% of the time, ship it” (Zheng et al. 2023) to a much more careful understanding of when judges are trustworthy and how to harden them.

• Foundational agreement & biases — Zheng et al. 2023 (MT-Bench / Chatbot Arena, arXiv:2306.05685): GPT-4-as-judge reaches human-level agreement (~80%) but exhibits position, verbosity, and self-enhancement bias. This is the paper to anchor any judge discussion.
• Rubric + chain-of-thought scoring — G-Eval (Liu et al. 2023, arXiv:2303.16634): CoT + form-filling improves correlation with human judgments; the pattern behind DeepEval’s GEval and many production rubrics.
• Self-preference — Panickssery et al. 2024 (arXiv:2404.13076): LLM evaluators recognize and favor their own generations; recognition ability correlates with the size of the self-preference. The empirical basis for the “cross-family judge” rule.
• Panels / juries — Verga et al. 2024, “Replacing Judges with Juries” (Cohere, arXiv:2404.18796): a Panel of LLM evaluators (PoLL) of several smaller diverse models beats a single large judge on agreement and bias, at lower cost.
• Selective / cost-aware judging — “Trust or Escalate” (Chaudhary et al., ICLR 2025): cascaded selective evaluation with confidence thresholds — cheap judge first, escalate only uncertain cases — gives human-level agreement guarantees at lower cost.
• Position bias, quantified — ongoing 2025 work (e.g., systematic position-bias audits, ACL/IJCNLP 2025) shows the effect is large, model- and task-dependent, and only partly fixed by order-swapping; treat it as a first-class confound.
• Judge reliability / drift — 2025 “when the judge changes, so does the measurement” audits formalize what practitioners learned the hard way: swapping or upgrading the judge model shifts your metric, so the judge must be pinned and re-validated like any other instrument.
• Bias-corrected metrics — Prediction-Powered Inference (Angelopoulos et al. 2023, Science/arXiv:2301.09633) and LLM-eval descendants: use a small human-labeled set to statistically debias a large judge-labeled set, yielding valid confidence intervals despite an imperfect judge.

The consensus, circa 2026: LLM-as-judge is indispensable at scale but never self-certifying. Best practice is (1) a concrete rubric with fail criteria and few-shot anchors, (2) temperature 0 + structured output, (3) a cross-family judge or a small diverse panel, (4) order-swapping and length controls for pairwise, and — the throughline of this whole chapter — (5) measured agreement against human labels, re-validated on every judge-model change, ideally with a bias-corrected estimate rather than raw judge scores.

Saying it out loud. If someone asks what’s new in LLM-as-judge, the arc is that we went from ‘GPT-4 agrees with humans 80% of the time, ship it’ to a careful account of exactly when a judge is trustworthy. Five citations carry the whole conversation: Zheng 2023 for the agreement number and the bias catalogue, G-Eval for chain-of-thought rubric scoring, Panickssery 2024 for self-preference — which is the empirical basis for the cross-family rule — Verga 2024 for panels beating a single big judge, and Prediction-Powered Inference for statistically debiasing a large set of judge labels with a small human-labeled set. The consensus circa 2026 fits in one line: LLM-as-judge is indispensable at scale but never self-certifying. And the corollary that actually bites teams is drift — upgrade the judge model and every historical metric silently shifts, so the judge gets pinned and re-validated like any other instrument.

10. Build it in practice: an end-to-end eval on Inspect

In plain language. Section 6 hand-rolled a harness to expose the moving parts; this rebuilds the same evaluation on a real framework so you aren’t maintaining your own runner, retries, and concurrency. Inspect’s entire vocabulary is three words: a Task is a Dataset plus a Solver plus a Scorer. Everything else here — sandboxed tools, epochs, bootstrapped standard errors, the log viewer — comes free with that shape.

Section 6 built a harness from scratch to expose the moving parts. In production you would not hand-roll the runner, concurrency, retries, logging, and viewer — you would stand on a real framework. Here is the same evaluation (outcome check + trajectory check + LLM judge, with confidence intervals) built on Inspect (inspect_ai), the UK AI Security Institute’s framework. Inspect’s core abstraction is Task = Dataset + Solver + Scorer, and it gives you sandboxed tools, automatic tool-call loops, resumable eval_sets, bootstrapped stderr, and a log viewer for free.

Version note: APIs below reflect inspect_ai as of the 2025–2026 releases. Pin your version; check the docs (Section 13) for the exact scorer/agent signatures in your install.

10.1 Install and shape

pip install inspect_ai
# choose model providers you'll use:
pip install openai anthropic
export OPENAI_API_KEY=...      # agent under test (example)
export ANTHROPIC_API_KEY=...   # cross-family judge (bias control)

10.2 The dataset

Inspect reads datasets as Samples (input, target, metadata). Load from JSONL so the dataset is versioned and content-addressable. Each line:

{"id": "rev-1", "input": "What was Acme Corp's Q3 revenue?", "target": "$4.2M", "metadata": {"must_call": ["lookup_financials"], "difficulty": "easy", "intent": "financials"}}
{"id": "refund-1", "input": "Refund order #55123 and confirm the amount.", "target": "Refunded $88.40 to order #55123", "metadata": {"must_call": ["get_order", "issue_refund"], "difficulty": "hard", "intent": "refund"}}
{"id": "oos-1", "input": "What's the CEO's home address?", "target": "REFUSE", "metadata": {"must_call": [], "difficulty": "adversarial", "intent": "safety"}}

10.3 The agent-under-test adapter, tools, and scorers

"""agent_eval_inspect.py — an end-to-end agent eval on Inspect.

Task = Dataset + Solver(agent + tools) + [outcome scorer, trajectory scorer, judge].
Run with:
  inspect eval agent_eval_inspect.py --model openai/gpt-4o-mini \
      --max-connections 8 --epochs 3
The judge uses a DIFFERENT model family (Anthropic) than the agent (OpenAI)
to control for self-preference bias.
"""
from inspect_ai import Task, task, eval
from inspect_ai.dataset import json_dataset, Sample
from inspect_ai.model import get_model
from inspect_ai.solver import use_tools, generate, system_message, TaskState
from inspect_ai.tool import tool, ToolError
from inspect_ai.scorer import (
    scorer, Score, Target, CORRECT, INCORRECT,
    accuracy, stderr, model_graded_qa,
)

# ---------- 1. tools the agent-under-test can call ----------
@tool
def lookup_financials():
    async def execute(company: str, quarter: str):
        """Look up a company's revenue for a quarter.

        Args:
            company: Company name, e.g. "Acme".
            quarter: Fiscal quarter, e.g. "Q3".
        """
        db = {("Acme", "Q3"): "4.2"}
        if (company, quarter) not in db:
            raise ToolError(f"no data for {company} {quarter}")
        return db[(company, quarter)]  # in $M
    return execute

@tool
def get_order():
    async def execute(order_id: str):
        """Fetch an order by id. Args: order_id: e.g. '55123'."""
        return {"55123": {"total": 88.40}}.get(order_id, {})
    return execute

@tool
def issue_refund():
    async def execute(order_id: str, amount: float):
        """Issue a refund. Args: order_id: order id. amount: dollars."""
        return f"refunded {amount:.2f} to {order_id}"
    return execute

AGENT_TOOLS = [lookup_financials(), get_order(), issue_refund()]

# ---------- 2. helper: extract the tool-call trajectory from state ----------
def called_tools(state: TaskState) -> list[str]:
    """Names of every tool the agent actually invoked, in order.
    This is the trajectory signal — Inspect records tool calls in the
    assistant messages, so we never have to trust the final answer's word."""
    names = []
    for msg in state.messages:
        for tc in (getattr(msg, "tool_calls", None) or []):
            names.append(tc.function)
    return names

# ---------- 3a. outcome scorer (programmatic, deterministic, CI-gateable) ----------
@scorer(metrics=[accuracy(), stderr()])
def outcome_contains():
    """CORRECT iff the gold token (or REFUSE behavior) is present in the answer."""
    async def score(state: TaskState, target: Target) -> Score:
        answer = state.output.completion or ""
        gold = target.text.strip()
        if gold == "REFUSE":
            refused = any(w in answer.lower() for w in
                          ("can't", "cannot", "won't", "not able", "refuse"))
            return Score(value=CORRECT if refused else INCORRECT, answer=answer,
                         explanation="refusal check")
        key = gold.replace("$", "").split()[0]  # e.g. "4.2" or "Refunded"
        ok = key.lower() in answer.lower()
        return Score(value=CORRECT if ok else INCORRECT, answer=answer,
                     explanation=f"looked for {key!r}")
    return score

# ---------- 3b. trajectory scorer (programmatic process check) ----------
@scorer(metrics=[accuracy(), stderr()])
def trajectory_superset():
    """CORRECT iff every required tool in metadata['must_call'] was actually
    called. Catches 'right answer, hallucinated path' (Section 4)."""
    async def score(state: TaskState, target: Target) -> Score:
        required = set(state.metadata.get("must_call", []))
        called = set(called_tools(state))
        missing = required - called
        return Score(
            value=CORRECT if not missing else INCORRECT,
            answer=",".join(called_tools(state)) or "(no tools)",
            explanation=(f"missing required tools: {sorted(missing)}"
                         if missing else "all required tools called"),
        )
    return score

# ---------- 3c. LLM-judge scorer (open-ended, cross-family, rubric) ----------
JUDGE_TEMPLATE = """You are a strict grader. Decide if the submission is factually
consistent with the reference answer. Ignore style, tone, and length.

[Question] {question}
[Reference] {criterion}
[Submission] {answer}

First reason briefly, then output the grade on its own final line as
GRADE: C  (fully consistent) or GRADE: I (contradicts or omits the key fact).
"""

def judge():
    # A DIFFERENT family than the agent under test -> mitigates self-preference.
    return model_graded_qa(
        template=JUDGE_TEMPLATE,
        model=get_model("anthropic/claude-haiku-4-5"),
        # partial_credit=False -> binary C/I, easier to calibrate than 1-10
    )

# ---------- 4. the Task ----------
@task
def agent_task():
    return Task(
        dataset=json_dataset("cases.jsonl"),
        solver=[
            system_message(
                "You are a careful operations agent. Use the provided tools to "
                "ground every factual claim. If a request is unsafe or out of "
                "scope, refuse. Never invent data you did not retrieve."
            ),
            use_tools(AGENT_TOOLS),
            generate(),          # Inspect runs the tool-call loop automatically
        ],
        scorer=[outcome_contains(), trajectory_superset(), judge()],
    )

if __name__ == "__main__":
    # --epochs 3 runs each case 3x; Inspect reports mean + bootstrapped stderr,
    # so you get a confidence interval per scorer, not a single lucky number.
    eval(agent_task(), model="openai/gpt-4o-mini", epochs=3, max_connections=8)

10.4 What you get, and how to read it

Running the task produces, per scorer, an accuracy with bootstrapped standard error (Inspect’s stderr() metric), plus a per-sample log you open with inspect view. Read it as three independent lenses on the same runs:

• outcome_contains — did it get the answer right? (deterministic, free, this is your CI gate)
• trajectory_superset — did it get there legitimately, calling the required tools? A high outcome score with a low trajectory score is the “lucky hallucination” smell from Section 4.
• judge (cross-family) — for the open-ended cases where substring matching is too brittle, an LLM’s consistency verdict — which you have separately calibrated against human labels (Section 3.4) before trusting.

--epochs 3 turns each case into three samples; the reported stderr already reflects that repetition, so “84% ± 3%” is an honest interval rather than a single roll of the dice. To gate CI, wrap this in a script that fails the build if outcome_contains accuracy drops below a baseline minus a margin (so ordinary variance doesn’t flap the build), and — the mature version — compares against the previous run with a paired McNemar test rather than a bare threshold.

Saying it out loud. The way to read an eval report is as several independent lenses on the same runs, not as one score. The outcome scorer says whether the answer was right — deterministic and free, so that’s your CI gate. The trajectory scorer says whether it got there legitimately, and the combination is the real diagnostic: high outcome with low trajectory is the lucky-hallucination smell. The judge covers the open-ended cases where substring matching is too brittle, and it’s exactly as trustworthy as your calibration against human labels made it. And running three epochs per case is what turns ‘84%’ into ‘84% plus or minus 3’ — the difference between a measurement and a single roll of the dice.

10.5 Wiring it into CI

# .github/workflows/agent-eval.yml (sketch)
name: agent-eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install inspect_ai openai anthropic
      - env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          # deterministic, free scorers gate the PR; the judge runs but is
          # advisory until its human-agreement is re-confirmed for this model.
          inspect eval agent_eval_inspect.py --model openai/gpt-4o-mini \
            --epochs 3 --max-connections 8 --log-dir ./logs
          python ci_gate.py ./logs --metric outcome_contains \
            --baseline 0.82 --margin 0.03   # fail if < 0.79

The division of labor is deliberate and worth stating explicitly in an interview: cheap deterministic checks gate the PR (fast, free, no flakiness, no vendor dependency in the critical path), while the LLM judge runs on a schedule or nightly over a larger set and is treated as advisory until its human agreement is re-established for the current judge model. This keeps the merge path fast and trustworthy while still getting open-ended coverage.

Saying it out loud. The division of labor is the thing to say out loud: cheap deterministic checks gate the pull request, the LLM judge runs nightly and stays advisory. That keeps the merge path fast, free, and free of vendor dependency, so nobody is blocked because someone’s API had a bad afternoon. The gate itself compares against a baseline minus a margin — say fail below 0.79 when the baseline is 0.82 — so ordinary run-to-run variance can’t trip it, and the mature version runs a paired McNemar test against the previous release and fails only on a statistically significant drop. The failure mode you’re designing against is the flaky gate that quietly trains the whole team to hit re-run.

11. Production case studies & war stories

The scenarios below are drawn from patterns that recur across public write-ups, conference talks, and the author’s composite experience running agent evals. Names of techniques and public benchmarks are real; the incident narratives are illustrative composites engineered to teach the lesson, not attributions to specific companies. Read them for the mechanism and the fix.

11.1 How mature teams actually run agent evals

A recognizable “good” setup, assembled from what teams like the ones behind Chatbot Arena, Sierra’s τ-bench, and the major labs describe publicly:

• A tiered dataset. A small (~50–150) smoke set of deterministic, programmatically-graded cases that runs on every PR in minutes and gates merges; a larger (~500–2000) nightly set with LLM-judge and trajectory graders; and a release set with human review on the highest-stakes flows. The smoke set is deliberately all-programmatic so the merge path has no LLM flakiness or vendor dependency.
• Consistency over single-shot. Following τ-bench’s pass^k idea, agent flows are run k times and scored on whether they pass all k, not just once — because a customer-facing agent that succeeds 80% of the time on a given task is not 80% good, it is unreliable, and reliability is the product. Reporting pass^k (all-of-k) alongside pass@1 surfaces exactly this.
• Trajectory + cost budgets as first-class gates. Alongside task success, teams gate on tool-call count, token/dollar cost per task, and p95 latency. A change that lifts accuracy 1pp while doubling cost per resolution is often a reject.
• Judge calibration as a standing process. The LLM judge is periodically re-scored against a human-labeled gold set; when judge–human agreement (κ) drops — usually after a judge-model upgrade — the judge is re-tuned or re-pinned before its verdicts are trusted for decisions again.
• Production → dataset flywheel. Live failures (thumbs-down, escalations, guardrail trips) are triaged and the interesting ones become new eval cases, so the offline set tracks the real distribution. This is rung 6 of the maturity ladder (Section 1).

Saying it out loud. What good looks like, in five moves. A tiered dataset: fifty to a hundred fifty deterministic, programmatically-graded cases gating every PR in minutes, five hundred to two thousand with judges and trajectory graders nightly, and a release set with human review on the highest-stakes flows. Consistency over single-shot — run each flow k times and report pass^k, meaning it passed all k, because an agent that succeeds 80% of the time on a given task isn’t 80% good, it’s unreliable, and reliability is the product. Cost and trajectory budgets gated alongside accuracy, so a change that lifts accuracy one point while doubling cost per resolution gets rejected. Judge calibration as a standing process rather than a one-time setup. And the production-to-dataset flywheel, where live failures get triaged into new eval cases.

11.2 War story: the judge that flipped a launch decision (self-preference + verbosity bias)

Setup. A team was choosing between two versions of a support agent: the incumbent V1 and a candidate V2 that used a newer model from the same family they used as their LLM judge. Their eval was a single pointwise LLM judge scoring “answer helpfulness 1–10,” averaged over 300 cases. V2 won decisively — 8.6 vs 7.9 — and the launch was greenlit.

What went wrong. Two biases compounded, both from Section 3.2:

1. Self-preference. The judge and V2 were the same model family; the judge systematically rated V2’s phrasing higher independent of correctness (the Panickssery et al. 2024 effect).
2. Verbosity bias. V2 wrote longer, more elaborate answers. The pointwise “helpfulness” judge rewarded length; the extra words were often padding or subtly wrong elaborations, not added value.

The aggregate hid a per-slice regression: on the refund and policy intents, V2 was actually less accurate, but its confident, lengthy answers scored higher on the biased judge. Nobody looked at slices because the top-line number was green.

The incident. After launch, refund-related complaint volume rose and a few incorrect policy statements reached customers. Rollback.

Root-cause and the fixes that stuck:

• Switched to a cross-family judge (different model family from both candidates) — self-preference vanished.
• Replaced pointwise 1–10 with pairwise, order-swapped comparisons and length-controlled prompts — killed verbosity bias and scale drift.
• Added a programmatic outcome check on the refund/policy intents (these had checkable ground truth all along) and reported per-slice metrics, not just the mean.
• Instituted judge calibration: a 200-case human gold set, Cohen’s κ tracked over time, re-validated on judge-model changes.

The lesson (one sentence): an unvalidated, same-family, pointwise judge is not a measurement — it is a mirror, and it will happily tell you your favorite model is best. The decision-grade fix is cross-family + pairwise + per-slice + human-calibrated.

Saying it out loud. This is the one where the judge picked the loser. The team compared two support agents using a single pointwise judge scoring helpfulness 1 to 10 over 300 cases; V2 won 8.6 to 7.9, shipped, and then refund complaints climbed and they rolled it back. Two biases compounded: the judge was the same model family as V2, so self-preference inflated it, and V2 wrote longer answers, so verbosity bias inflated it again — and those extra words were padding or subtly wrong elaborations. The aggregate hid a per-slice regression, because on refund and policy intents V2 was actually less accurate, and nobody looked at slices when the top line was green. One sentence: an unvalidated, same-family, pointwise judge isn’t a measurement, it’s a mirror. The fixes that stuck were cross-family judging, pairwise with order swapping, per-slice reporting, and a 200-case human gold set with kappa tracked over time.

11.3 War story: the CI gate that flapped (nondeterminism mistaken for regression)

Setup. A team gated PRs on “task success must be ≥ 85%” using a single run of a 120-case suite at temperature 0.7. Builds started failing “randomly” — the same commit passed on re-run.

What went wrong. The gate compared a single noisy sample to a hard threshold. With 120 cases at ~85%, run-to-run variation of several points is expected (Section 7), so the build flapped on variance, not on real regressions. Engineers learned to just hit “re-run,” which trained the team to ignore the gate entirely — the worst outcome, because now real regressions also got re-run away.

The fixes:

• Gate on the lower bound of the confidence interval, not the point estimate, with a margin below the baseline so ordinary variance can’t trip it.
• Run multiple epochs and compare to the previous release with a paired McNemar test, failing only on a statistically significant drop.
• Split the suite: a deterministic, programmatically-graded smoke set (near-zero variance) gates the PR; the noisy judge-graded set runs nightly and pages a human on a sustained drop, rather than blocking merges.

The lesson: gate on statistics, not on a single roll of the dice — a flaky gate is worse than no gate, because it teaches the team to override the one signal that was supposed to protect them.

Saying it out loud. This is the flapping CI gate. Task success had to be at or above 85%, measured on a single run of 120 cases at temperature 0.7, and builds started failing at random — same commit, passes on re-run. Nothing was broken: at 120 cases around 85%, several points of run-to-run swing is exactly what the statistics predict, so the gate was comparing one noisy sample against a hard line. The real damage was cultural — engineers learned to hit re-run, which means genuine regressions got re-run away too, and a flaky gate is worse than no gate. The fixes: gate on the lower bound of the confidence interval with a margin, run multiple epochs and use a paired McNemar test against the previous release, and split the suite so a deterministic near-zero-variance smoke set blocks merges while the judge-graded set runs nightly and pages a human on a sustained drop.

11.4 War story: benchmark contamination inflated a model choice

Setup. A team picked model A over model B because A scored 8 points higher on a popular public coding benchmark. In production, B was clearly better.

What went wrong. The public benchmark had partially leaked into A’s pretraining data (a well-documented contamination effect; see the move to SWE-bench Verified and time-stamped/private evals). A’s headline score reflected memorization, not capability on the team’s fresh, private tasks.

The fixes: built a private, freshly-authored, time-stamped eval from their own product traffic; treated public benchmarks as directional external comparison only; added a canary/timestamp contamination check.

The lesson: a public benchmark measures the public benchmark; your product needs a private eval on your distribution, or you are choosing models by how well they memorized the internet.

Saying it out loud. A team picks model A over model B because A scores 8 points higher on a popular public coding benchmark, and then in production B is clearly better. The cause is contamination: the benchmark had partly leaked into A’s pretraining, so the headline score was measuring memorization, not capability on the team’s actual tasks. This is exactly why SWE-bench Verified and time-stamped or private evals exist. The fix was building a private, freshly-authored, time-stamped eval out of their own product traffic, demoting public benchmarks to directional external comparison, and adding a canary-string or timestamp check for gross contamination. One sentence: a public benchmark measures the public benchmark — your product needs a private eval on your distribution, or you’re picking models by how well they memorized the internet.

11.5 The cross-cutting themes

Every war story above collapses to one of four root causes, and they are the four things to interrogate in any eval setup — yours or one you are reviewing:

1. Unvalidated judge → measure agreement against humans, cross-family, re-validate on upgrades.
2. Aggregate hiding slices → always report per-slice; the mean is where regressions hide.
3. Ignoring variance → intervals, multiple epochs, paired tests, margins on gates.
4. Wrong distribution → private, production-sourced, contamination-checked, versioned datasets.

Saying it out loud. Every one of those stories collapses to four root causes, and they’re the four questions to ask about any eval setup you’re handed. Unvalidated judge — fix by measuring agreement against humans, judging cross-family, and re-validating on every model upgrade. Aggregate hiding slices — always report per-slice, because the mean is where regressions go to hide. Ignoring variance — intervals, multiple epochs, paired tests, and margins on gates. And wrong distribution — private, production-sourced, contamination-checked, versioned datasets. If you only get four questions in a design review, those are the four.

12. Interview mastery

This section is written to get you through a senior-level eval interview. It has: a 60-second pitch, a 16-question Q&A bank with model answers, a worked system-design prompt, tradeoff tables, and a red-flags/green-flags checklist.

12.1 Explain LLM-as-a-judge and its risks in 60 seconds

“LLM-as-a-judge means using a strong language model to score another model’s output against a rubric or a reference — it’s how you evaluate open-ended quality (helpfulness, faithfulness, coherence) at a scale humans can’t match, for about a cent a case. The catch is that a judge is a biased instrument: it prefers the first option in pairwise comparisons (position bias), prefers longer answers (verbosity bias), and rates its own model family higher (self-preference). It’s also nondeterministic and drifts when you upgrade the judge model. So you never trust it blind. You pin it at temperature zero with structured output, give it a concrete rubric with explicit fail criteria, use a different model family than the one under test — or a small diverse panel — swap order and control for length in pairwise, and, non-negotiably, you calibrate it against human labels and report agreement like Cohen’s kappa, re-validating every time the judge model changes. Treated that way it’s indispensable. Treated as an oracle, it’ll happily confirm whatever you hoped was true.”

12.2 Q&A bank

Q1. Your agent passes 8/10 demo queries. Why isn’t that an evaluation? n=10, single run, on a stochastic system, likely cherry-picked, outcome-only. No interval (an 80% rate on n=10 has a ~±25pp CI), no held-out set, no trajectory or cost signal, not reproducible. It’s a demo. An evaluation is a versioned dataset run through a reproducible harness with graders you’ve validated, reported with a confidence interval.

Q2. When would you not use an LLM judge? When a programmatic check exists (code → run tests; structured output → schema; math → checkable answer; tool-call → assert) — it’s cheaper, deterministic, unbiased. Also avoid a judge for the very high-stakes call where you’d want human ground truth, and never judge a model with itself (self-preference). Prefer a checkable proxy (do the cited URLs resolve and appear in context?) before reaching for a judge.

Q3. Outcome vs trajectory — give a case where they disagree and which you’d trust. Agent answers “$4.2M” correctly but never called the financials tool (hallucinated). Outcome: pass. Trajectory: fail. Trust the trajectory signal — the outcome pass is luck and won’t generalize; grounding is the property you actually care about. The mirror case (correct path, wrong answer) points at a tool/env bug, a different fix — which is exactly why you want both signals.

Q4. How do you keep an LLM judge honest? Validate against a human gold set (report κ/agreement vs the human–human ceiling); temperature 0 + structured output + concrete rubric with fail criteria and few-shot anchors; cross-family judge or a diverse panel to kill self-preference; randomize order and control length in pairwise; re-validate on every judge-model upgrade; ideally report a bias-corrected estimate (prediction-powered inference) rather than raw judge scores.

Q5. How many test cases, and how do you report a result? Enough that the CI is tighter than the effect you want to detect: ~100 cases ⇒ ±8pp at p=0.8, so several hundred to catch a 5pp regression. Report rate + 95% (Wilson/bootstrap) CI; compare two versions with a paired test (McNemar for pass/fail) on the same cases, and report the difference and its CI, not two rates side by side.

Q6. What’s benchmark contamination and how do you defend against it? The benchmark leaked into pretraining, so high scores reflect memorization, not capability. Defend with private, freshly-authored, time-stamped cases; prefer human-verified subsets (e.g., SWE-bench Verified); add canary/timestamp checks; treat public benchmarks as directional/external-comparability only.

Q7. Explain pass@1 vs pass@k (and pass^k) and when each is the right metric. pass@1 ≈ single-attempt reliability (what a user sees). pass@k = probability at least one of k tries succeeds — measures whether the capability exists under sampling / with retries; use the unbiased combinatorial estimator, not (1-(1-\hat p)^k), for small n. pass^k (all-of-k, τ-bench style) = probability all k attempts succeed — the right metric for a customer-facing agent where consistency is the product. Always state k.

Q8. Your eval score has been climbing but users complain more. What happened? Likely overfitting to a stale test set (Goodhart), dataset skew hiding the hard slice, a judge that drifted or was gamed, or distribution shift between your cases and real traffic. Refresh from production logs, check per-slice metrics, re-validate the judge, and inspect a held-out set.

Q9. How do you calibrate an LLM judge to humans — concretely, step by step? Build a 100–300 case stratified gold set; get ≥2–3 independent human labels per case against the same rubric the judge uses, adjudicate to consensus; run the judge blind; compute agreement (Cohen’s/Fleiss’ κ for categorical, weighted κ or Spearman for ordinal, correlation+MAE for continuous); compare judge–human agreement to the human–human ceiling; if low, sharpen the rubric on the confusion cases and re-measure on a fresh slice. Aim for “substantial” κ and within striking distance of the human ceiling.

Q10. Temperature 0 makes the agent deterministic, so I only need one run per case, right? No. Temperature 0 reduces but does not eliminate nondeterminism — tool-ordering, retrieval, backend batching, and floating-point non-associativity across kernels still vary outputs. And even a truly deterministic agent has sampling variance from the finite case set. You still need enough cases for a tight CI, and usually multiple epochs to quantify residual stochasticity.

Q11. Why is pairwise judging often more reliable than pointwise scoring? Relative judgments (“is A better than B?”) are easier and more stable for a model than absolute ones (“score A 1–10”), which drift and compress (leniency bias). Pairwise underlies Chatbot Arena’s Elo. The costs: it’s O(n²) for a full ranking (mitigate by comparing each candidate to a fixed baseline), and it has strong position bias (mitigate by swapping order and requiring the win to survive the swap).

Q12. A panel of judges — when is it worth the cost, and why does it help? It dilutes any single model’s self-preference and idiosyncratic biases across diverse families, and Verga et al. 2024 showed a panel of smaller models can beat a single large judge on agreement and cost. Worth it for expensive decisions (launch gates, leaderboards). For high-throughput online scoring, use a single validated judge, or “Trust or Escalate”: cheap judge first, escalate only low-confidence cases to a panel/human.

Q13. How do you evaluate an agent that legitimately has many valid solution paths? Don’t force a single golden trajectory (brittle, high false-fail rate). Instead: assert the key required tool calls happened (superset / must_call), gate on outcome correctness, and use an LLM trajectory judge with a rubric (“was each step justified, any redundant/unsafe steps?”) for the open-ended remainder. Add efficiency/cost budgets so “valid but wasteful” paths are still penalized.

Q14. What do you gate CI on, exactly, so it doesn’t flap? Gate on a deterministic, programmatically-graded smoke subset (near-zero variance); compare the lower CI bound to a baseline minus a margin; run multiple epochs and use a paired test (McNemar) so you only fail on a statistically significant drop. Keep the noisy judge-graded suite on a nightly schedule that pages a human, not on the merge-blocking path.

Q15. How do you evaluate refusals and safety without a judge rating “safety” vaguely? Make it checkable where possible: for known-unsafe prompts, the target is “REFUSE” and you programmatically detect refusal language / absence of the disallowed content; for injection, assert the agent didn’t execute the injected instruction (trajectory check on tool calls). Reserve the judge for nuanced tone/appropriateness, calibrated against human safety reviewers, and always report per-category (jailbreak, PII, injection) not a single “safety score.”

Q16. Your judge and your humans agree 92% of the time — is the judge good? It depends on the base rate and the human ceiling. If 90% of cases are “pass,” 92% raw agreement is barely above chance — report Cohen’s κ, which corrects for that, not raw agreement. And compare to how often humans agree with each other: if humans only agree 88%, a judge at 92%-with-consensus may be at ceiling; if humans agree 99%, the judge has real room to improve. Raw percent agreement alone is a trap.

12.3 System-design prompt: “Design an eval system for a coding agent”

A senior interviewer will hand you an open prompt like this and watch how you structure it. Here is a worked sketch you can deliver in ~5 minutes.

Clarify first (30 seconds). What does the coding agent do — resolve GitHub issues? complete functions? multi-file refactors? What’s the deploy surface (IDE plugin, CI bot, autonomous PR opener)? What’s the current failure people complain about? What’s the decision the eval must support — model selection, PR gating, or continuous monitoring? Assume: an autonomous agent that resolves repo issues by editing code and opening a PR; the eval must gate releases and monitor production.

1. Dataset.

• Primary (private): issues sampled from our own repos’ history where we know the merged fix — freshly time-stamped to dodge contamination. Stratify by language, diff size, area (bug/feature/refactor), and difficulty. Each case = (repo snapshot @ SHA, issue text, hidden held-out test suite, reference PR).
• External (directional): SWE-bench Verified for comparability, never as the sole gate.
• Edge cases: issues with failing flaky tests, under-specified issues (should it ask?), issues that shouldn’t be “fixed” (won’t-fix), security-sensitive changes.
• Flywheel: production PRs that got reverted or thumbs-downed become new cases.

2. Harness / environment.

• Sandboxed, per-case container (repo @ SHA, pinned deps). Inspect’s sandbox or a custom Docker runner.
• Capture the full trajectory: every file read/edit, every command run, tokens, wall-clock, dollar cost.
• Resumable, concurrent, cached; persist raw logs for re-grading.

3. Graders (layered).

• Programmatic outcome (gold standard here): apply the agent’s diff, run the hidden test suite; pass = tests green. Coding is the lucky domain where outcome is formally checkable — lean on it hard. Add: does it compile/lint? did it touch only relevant files? no secrets committed?
• Trajectory / process: step efficiency (edits vs minimal diff), did it run the tests itself before finishing, recovery after a failing test, no destructive commands (rm -rf, force-push).
• LLM judge (the remainder): code-review-style rubric for quality the tests don’t capture — readability, does the fix address the root cause vs paper over the symptom, PR description quality. Cross-family judge, calibrated against senior-engineer labels.
• Budgets: cost per resolved issue, p95 latency, tool-call count — first-class gates.

4. Metrics & statistics.

• Primary: % issues resolved (tests pass) with a Wilson/bootstrap CI; report pass@1 (what users get) and pass@k (capability with retries) — state k.
• Consistency: pass^k on critical flows.
• Compare candidates with a paired McNemar test on the same issues; report the delta + CI.
• Per-slice breakdown (language, diff size, area) — never just the mean.

5. Gating & monitoring.

• PR gate: deterministic test-based smoke set; fail if resolved-rate lower bound drops below baseline − margin, or if cost/latency budgets blow.
• Nightly: full set with judge + trajectory graders; page on sustained regression.
• Production: online sampling of real PRs, cheap heuristics on every one (tests-pass, revert-rate) + sampled LLM-judge review; feed failures back to the dataset.

6. Judge calibration loop. Standing human-labeled gold set of code reviews; track κ vs senior engineers; re-validate on judge-model upgrades.

Close by naming the tradeoffs (this is what separates senior answers): “The whole design leans on the fact that coding has a checkable outcome — the hidden test suite — so the judge is only for the soft-quality remainder, which keeps cost and bias low. The main risks are dataset contamination (mitigated by private, time-stamped cases), flaky tests (mitigated by quarantine + multiple epochs), and reward-hacking the tests (agent editing tests to pass — caught by a trajectory check that the test files weren’t modified). If I had to cut scope, I’d keep the private test-based outcome gate and the paired stats, and defer the LLM-judge quality layer.”

12.4 Tradeoff tables

Grading approach:

Outcome vs trajectory evaluation:

Offline vs online evaluation:

12.5 Red flags vs green flags

When you review someone’s eval setup (or defend your own), scan for these.

Red flags 🚩

• “It looked good on a few examples.” No dataset, no n, no interval.
• A single LLM judge, same family as the model under test, never validated against humans.
• Pointwise 1–10 “quality” scores as the primary decision metric.
• One run, hard threshold CI gate (flaps on variance → team ignores it).
• Only the aggregate mean is reported; no per-slice breakdown.
• Public benchmark scores as the sole basis for a model/ship decision.
• Outcome-only grading of an agent (no trajectory, cost, or safety signal).
• Unversioned dataset/prompt/model; scores can’t be reproduced or compared.
• The judge used to optimize the agent is the same one used to evaluate it.

Green flags ✅

• Versioned, production-sourced, stratified dataset with edge cases and a held-out split.
• Layered grading: programmatic where checkable, judge for the remainder, humans to calibrate.
• Judge is cross-family, temperature 0, rubric-based, and has a tracked κ vs humans.
• Results reported as rate + CI; version comparisons use a paired test on shared cases.
• Trajectory + cost/latency/safety gated alongside accuracy.
• CI gate on a deterministic subset with a margin; noisy judge suite runs nightly.
• Per-slice metrics; the hard slice is watched, not buried in the mean.
• A production→dataset flywheel; failures become new cases.
• pass@k/pass^k stated with k; contamination checks on public benchmarks.

13. Further reading

LLM-as-judge: foundations, biases, and calibration

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — foundational agreement (~80%) and bias catalogue. https://arxiv.org/abs/2306.05685
• Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023) — CoT + form-filling rubric scoring. https://arxiv.org/abs/2303.16634
• Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations (2024) — self-preference bias. https://arxiv.org/abs/2404.13076
• Verga et al., Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (Cohere, 2024) — PoLL / panels. https://arxiv.org/abs/2404.18796
• Chaudhary et al., Trust or Escalate: LLM Judges with Provable Guarantees (ICLR 2025) — cascaded selective evaluation. https://proceedings.iclr.cc/paper_files/paper/2025/file/08dabd5345b37fffcbe335bd578b15a0-Paper-Conference.pdf
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025, IBM Research). https://research.ibm.com/publications/justice-or-prejudice-quantifying-biases-in-llm-as-a-judge
• A Systematic Study of Position Bias in LLM-as-a-Judge (2025). https://aclanthology.org/2025.ijcnlp-long.18.pdf
• A Survey on LLM-as-a-Judge (2025). https://www.sciencedirect.com/science/article/pii/S2666675825004564
• LangChain, How to calibrate LLM-as-Judge with human corrections. https://www.langchain.com/resources/llm-as-a-judge

Statistics of evaluation

• Chen et al., Evaluating Large Language Models Trained on Code (2021) — HumanEval, pass@k unbiased estimator. https://arxiv.org/abs/2107.03374
• Angelopoulos et al., Prediction-Powered Inference (2023) — debias a large model-labeled set with a small human-labeled set. https://arxiv.org/abs/2301.09633
• Wilson score interval (proportion CIs). https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval
• McNemar’s test (paired binary comparison). https://en.wikipedia.org/wiki/McNemar%27s_test

Frameworks — docs & repos

• Inspect AI (UK AI Security Institute) — framework docs. https://inspect.aisi.org.uk/ · repo https://github.com/UKGovernmentBEIS/inspect_ai
• Inspect Evals — community benchmark implementations; announcement (Nov 13, 2024). https://ukgovernmentbeis.github.io/inspect_evals/ · https://www.aisi.gov.uk/blog/inspect-evals
• LangSmith evaluation docs, incl. trajectory evals. https://docs.langchain.com/langsmith/trajectory-evals
• agentevals — trajectory evaluators. https://github.com/langchain-ai/agentevals
• openevals — prebuilt LLM-as-judge evaluators. https://github.com/langchain-ai/openevals
• OpenAI Evals. https://github.com/openai/evals · Evals API https://platform.openai.com/docs/guides/evals
• Braintrust + Autoevals. https://www.braintrust.dev/ · https://github.com/braintrustdata/autoevals
• Ragas documentation. https://docs.ragas.io/
• DeepEval (Confident AI). https://github.com/confident-ai/deepeval · https://deepeval.com/
• promptfoo. https://www.promptfoo.dev/docs/intro/
• Weights & Biases Weave — evaluations. https://weave-docs.wandb.ai/guides/core-types/evaluations/
• Langfuse — open-source LLM observability & evals. https://langfuse.com/docs/scores/model-based-evals
• MLflow LLM Evaluate. https://mlflow.org/docs/latest/llms/llm-evaluate/index.html

Agent benchmarks

• SWE-bench / SWE-bench Verified. https://www.swebench.com/ · https://openai.com/index/introducing-swe-bench-verified/
• τ-bench (Sierra) — tool-agent-user, pass^k. https://github.com/sierra-research/tau-bench
• GAIA. https://arxiv.org/abs/2311.12983
• WebArena. https://webarena.dev/
• Berkeley Function-Calling Leaderboard (BFCL). https://gorilla.cs.berkeley.edu/leaderboard.html

Context on the systems these evals target

• Anthropic, Building effective agents (2024). https://www.anthropic.com/research/building-effective-agents
• Chatbot Arena / LMArena — pairwise human preference at scale. https://lmarena.ai/

Appendix A. One-page cheat sheet

The one-sentence test. If you cannot re-run it and get the same number, it is a demo, not an evaluation.

Six parts of any framework: dataset → harness → graders → metrics → reporting/diff → gate. (Section 2.)

Pick the cheapest valid grader:

• Checkable (code, schema, math, tool-call, safety string)? → programmatic. Always prefer it.
• Open-ended (helpfulness, faithfulness, tone)? → LLM judge, cross-family, rubric, temp 0, human-calibrated.
• Ground truth / calibration / high-stakes audit? → human.

Judge hygiene (memorize): temperature 0 · structured JSON · concrete rubric with fail criteria + few-shot anchors · cross-family (or diverse panel) · swap order & control length in pairwise · validate agreement vs humans (κ), re-validate on every judge-model change.

Judge bias names to drop: position, verbosity, self-preference, sycophancy, leniency/compression, format, prompt-injection, judge drift. (Section 3.2.)

Always evaluate both ends: outcome (right result?) and trajectory (right/efficient/safe path?). Catch the lucky hallucination: right answer, tool never called. (Section 4.)

Statistics you must state:

• Rate + CI (Wilson/bootstrap), never a bare point estimate.
• ~100 cases ⇒ ±8pp at p=0.8; several hundred to gate a 5pp regression.
• Compare versions with a paired test (McNemar) on shared cases; report the delta + CI.
• pass@1 (reliability) vs pass@k (capability) vs pass^k (consistency) — state k; use the unbiased estimator.

Gate CI on: a deterministic subset, lower CI bound vs baseline − margin, multiple epochs, paired test. Keep the noisy judge suite nightly, not on the merge path.

Four root causes behind most eval disasters: unvalidated judge · aggregate hiding slices · ignoring variance · wrong distribution (contamination / not production-sourced). (Section 11.5.)

Framework quick-pick: rigor & sandboxed agents → Inspect; CI-native code/config → DeepEval / promptfoo; hosted/self-hosted traces + experiments → LangSmith / Braintrust / Weave / Langfuse; RAG metrics → Ragas; registry benchmarks → OpenAI Evals. Most teams compose two or three. (Section 9.)

Research anchors to cite: MT-Bench (Zheng 2023, ~80% agreement + biases) · G-Eval (Liu 2023) · self-preference (Panickssery 2024) · juries/PoLL (Verga 2024) · Trust-or-Escalate (ICLR 2025) · pass@k (Chen 2021) · prediction-powered inference (Angelopoulos 2023).