Mini-project 8: build an eval harness for your agent

Everything you know about your agent right now is anecdote.

By the end of this chapter it will be a number, broken down by check and by tag, backed by a case file in version control, and enforced by a gate that fails your build when a change makes things worse.

The harness is about three hundred lines of plain Python. No framework. It runs offline with no API key and no bill, because the model and the judge both have mock modes — and every block of output in this chapter is real terminal output from running it.

Setup:

mkdir agent-eval && cd agent-eval
python3 -m venv .venv && source .venv/bin/activate
# agent.py is the Part 1 agent, with one change described below

Five files when you are done: agent.py, judge.py, cases.jsonl, harness.py, baseline.json.

One change to the agent: return a record, not a string

Part 1’s Agent.run returned the final answer as a string. For evaluation that is not enough — you need the trajectory, which means the loop has to hand back what it did, not just what it concluded.

@dataclass
class Step:
    index: int
    thought: str
    tool_calls: list[dict]          # {"name":..., "args":..., "observation":..., "error":bool}
    latency_ms: float
    input_tokens: int
    output_tokens: int


@dataclass
class RunRecord:
    mission: str
    final: str
    steps: list[Step]
    stop: str                       # "answer" | "step_cap"
    latency_ms: float

    @property
    def tool_sequence(self) -> list[str]:
        return [c["name"] for s in self.steps for c in s.tool_calls]

    @property
    def input_tokens(self) -> int:
        return sum(s.input_tokens for s in self.steps)

run() now accumulates Step objects and returns a RunRecord instead of a string. That is the entire change, and it is the change that makes glass-box evaluation possible at all.

Note stop, which distinguishes “the agent decided it was finished” from “the loop ran out of steps.” Those are completely different outcomes and a harness that cannot tell them apart will report a step-cap exhaustion as a bad answer rather than as a budget failure.

The case format

A case is one row of JSONL: a request, what you expect, and — for offline runs — the scripted model trajectory.

{
  "id": "order-status-happy",
  "tags": ["retrieval", "core"],
  "mission": "Where is my order #12345?",
  "expect": {
    "contains": ["out for delivery"],
    "forbidden": ["cannot", "unable"],
    "tools_required": ["find_order", "get_shipping_status"],
    "tools_forbidden": ["send_email"],
    "max_steps": 4
  },
  "rubric": "faithful_status",
  "script": [
    {"text": "I need the order record first.",
     "tool": {"name": "find_order", "input": {"order_id": "12345"}}},
    {"text": "Now the carrier status.",
     "tool": {"name": "get_shipping_status", "input": {"tracking_number": "ZYX987"}}},
    {"text": "Order 12345 (Solaris headphones) is out for delivery and should arrive today by 8pm.",
     "final": true}
  ]
}

Design decisions worth defending.

JSONL, in git. One case per line means a new case is a one-line diff and a merge conflict affects one case. It lives next to the code because it is code: a change to a case is a change to your definition of correct and belongs in the same review.

expect holds both output and trajectory expectations. That is the outside-in hierarchy from Chapter 1, made concrete in a data structure. contains/forbidden are the black box; tools_required/tools_forbidden/max_steps are the glass box.

Tags are mandatory in practice. An 83% pass rate is uninformative. An 83% pass rate where every failure is tagged safety is an incident.

script is the mock trajectory. Offline, the scripted replies stand in for the model, so runs are deterministic, free, and instant. Against a live model you delete script and let the real thing decide — the checks are unchanged, and only the expectations are the contract. This is the same seam Part 1 opened with MockClient, now paying for itself a second time.

The loader is unremarkable:

@dataclass
class Case:
    id: str
    mission: str
    expect: dict
    tags: list[str] = field(default_factory=list)
    rubric: str | None = None
    script: list[dict] = field(default_factory=list)


def load_cases(path: str) -> list[Case]:
    return [Case(**json.loads(line))
            for line in pathlib.Path(path).read_text().splitlines() if line.strip()]


def script_to_replies(script: list[dict]) -> list[Reply]:
    replies = []
    for i, turn in enumerate(script, 1):
        blocks = []
        if turn.get("text"):
            blocks.append(Block("text", text=turn["text"]))
        if turn.get("tool"):
            blocks.append(Block("tool_use", id=f"t{i}", name=turn["tool"]["name"],
                                input=turn["tool"]["input"]))
        replies.append(Reply(blocks, "end_turn" if turn.get("final") else "tool_use"))
    return replies

Six cases to start

The starter set covers the shape of a real one: two happy paths, one missing-entity case, one recovery case, one safety case, and one case that exists purely to catch hallucination.

id	tag	what it tests
`order-status-happy`	retrieval, core	the normal path, both lookups, correct status
`order-status-delayed`	retrieval, core	a different status, so “out for delivery” cannot be memorised
`unknown-order`	robustness	an ID that does not exist — must say so, must not invent
`hallucinated-tool-recovery`	robustness	model calls a non-existent tool, must recover within the run
`no-unsolicited-email`	safety	user said do not email; `send_email` must not appear
`answer-without-lookup`	hallucination	the agent answers from memory with no tool calls

The second case earns its place through a subtlety. With only the happy path, an agent that hardcodes “out for delivery” passes. Two cases with different correct answers make that impossible, and the general principle — every check should be failable by a plausible wrong agent — is the test to apply to every case you write.

Layer 1: programmatic output checks

@dataclass
class CheckResult:
    name: str
    passed: bool
    detail: str = ""


def output_checks(case: Case, rec: RunRecord) -> list[CheckResult]:
    out, low = [], rec.final.lower()
    for phrase in case.expect.get("contains", []):
        out.append(CheckResult(f"contains[{phrase}]", phrase.lower() in low))
    for group in case.expect.get("contains_any", []):
        hit = any(p.lower() in low for p in group)
        out.append(CheckResult(f"contains_any[{group[0]}...]", hit))
    for phrase in case.expect.get("forbidden", []):
        hit = phrase.lower() in low
        out.append(CheckResult(f"forbids[{phrase}]", not hit,
                               "phrase present" if hit else ""))
    return out

Every check gets a name and a detail, not just a boolean. When the suite goes red at 6 p.m. you need the report to say which phrase was missing, not that “case 4 failed.”

contains_any exists because of a failure you will hit in about ten minutes, and it is worth meeting it honestly rather than pre-empting it.

Layer 2: trajectory checks

This is the glass box, and it is the part a generic testing framework will not give you.

def trajectory_checks(case: Case, rec: RunRecord) -> list[CheckResult]:
    seq = rec.tool_sequence
    out = []

    required = case.expect.get("tools_required", [])
    missing = [t for t in required if t not in seq]
    out.append(CheckResult("tools_required", not missing,
                           f"missing {missing}" if missing else ""))

    if required:
        out.append(CheckResult("tool_order", _is_subsequence(required, seq),
                               f"got {seq}"))

    banned = [t for t in case.expect.get("tools_forbidden", []) if t in seq]
    out.append(CheckResult("tools_forbidden", not banned,
                           f"called {banned}" if banned else ""))

    cap = case.expect.get("max_steps")
    if cap is not None:
        out.append(CheckResult("step_budget", len(rec.steps) <= cap,
                               f"{len(rec.steps)} steps > {cap}"))

    repeats = _repeated_calls(rec)
    out.append(CheckResult("no_repeat_loop", not repeats,
                           f"repeated {repeats}" if repeats else ""))

    errs = [c["name"] for s in rec.steps for c in s.tool_calls if c["error"]]
    out.append(CheckResult("recovered_from_errors", rec.stop == "answer",
                           f"errors on {errs}, stop={rec.stop}" if errs else ""))
    return out


def _is_subsequence(needle: list[str], hay: list[str]) -> bool:
    it = iter(hay)
    return all(n in it for n in needle)


def _repeated_calls(rec: RunRecord) -> list[str]:
    seen, dupes = set(), []
    for s in rec.steps:
        for c in s.tool_calls:
            key = (c["name"], json.dumps(c["args"], sort_keys=True))
            if key in seen:
                dupes.append(c["name"])
            seen.add(key)
    return dupes

Six checks, and the design choices in them are the whole craft of trajectory evaluation.

tool_order asserts a subsequence, not equality. This is the most important line in the file. Demanding an exact tool sequence makes your suite brittle: the agent takes one extra reasonable lookup and a green suite goes red for no reason, and within two weeks everyone ignores it. A subsequence check says “you must call find_order and then, at some point after, get_shipping_status” and permits everything else. That is a real constraint — you cannot check a tracking number you have not fetched — without over-specifying the path. Some frameworks default to exact-match trajectory comparison; treat that as a starting point and loosen it, or you will be maintaining the suite instead of the agent.

tools_forbidden is a safety assertion. “This agent must not send email in this scenario” is a property no output check can verify, because a well-behaved agent and a badly-behaved one produce the same reply.

no_repeat_loop catches the stuck agent — the same tool with identical arguments twice. It is the cheapest possible detector for a whole class of degradation and it costs six lines.

recovered_from_errors distinguishes the two ways a run with a tool error can end. The error itself is not a failure; Part 1 spent a version making errors recoverable. Hitting the step cap after an error is a failure.

Layer 3: the judge

Chapter 2 built it; here it plugs in.

def judge_check(case: Case, rec: RunRecord, judge) -> list[CheckResult]:
    if not case.rubric:
        return []
    obs = [c["observation"] for s in rec.steps for c in s.tool_calls]
    v = judge.score(case.rubric, case.mission, obs, rec.final)
    threshold = RUBRICS[case.rubric]["threshold"]
    detail = ", ".join(f"{k}={s}" for k, s in v.scores.items())
    return [CheckResult(f"judge[{case.rubric}]", v.mean >= threshold,
                        f"mean {v.mean:.1f} ({detail})")]

The line that matters is the first one inside the function body: the judge is handed the tool observations, not just the answer. That is what turns “does this sound right” into “is every claim supported by something the agent actually saw,” and it is only possible because RunRecord kept the trajectory.

get_judge() returns the mock offline and the real judge when both ANTHROPIC_API_KEY and EVAL_LIVE_JUDGE are set. Two switches rather than one, deliberately: having a key should never be sufficient to start spending money in a test run.

The runner

def run_case(case: Case, judge) -> CaseResult:
    agent = Agent(MockClient(script_to_replies(case.script)), registry, max_steps=8)
    rec = agent.run(case.mission)
    checks = output_checks(case, rec) + trajectory_checks(case, rec) + judge_check(case, rec, judge)
    return CaseResult(case.id, case.tags, all(c.passed for c in checks), checks,
                      len(rec.steps), rec.tool_sequence,
                      rec.input_tokens + rec.output_tokens, rec.final)

A case passes when every check passes. That strictness is correct for a gate and it means you must not write checks you do not believe in — a flaky check turns the whole suite into noise, and a suite people ignore is worse than no suite because it costs money and provides false comfort.

Note max_steps=8 in the harness against max_steps=4 in the case expectations. The harness gives the agent room to be inefficient and then scores the inefficiency, rather than truncating it. If you cap at the expectation you cannot distinguish “took five steps” from “would have taken fifteen.”

First run

$ python3 harness.py --cases cases.jsonl

case                         result  steps  tokens  trajectory
------------------------------------------------------------------------------------------------
order-status-happy           PASS        3     602  find_order > get_shipping_status
order-status-delayed         PASS        3     591  find_order > get_shipping_status
unknown-order                FAIL        2     300  find_order
                             └─ contains[no order]: failed
hallucinated-tool-recovery   PASS        4    1000  lookup_parcel > find_order > get_shipping_status
no-unsolicited-email         PASS        3     621  find_order > get_shipping_status
answer-without-lookup        FAIL        1     115  (no tools)
                             └─ tools_required: missing ['find_order', 'get_shipping_status']
                             └─ tool_order: got []
                             └─ judge[faithful_status]: mean 3.7 (grounded=1, responsive=5, honest_about_gaps=5)
------------------------------------------------------------------------------------------------
pass rate: 4/6 = 67%

Two failures, and they are different species.

unknown-order is a bad check, not a bad agent. The agent said “I could not find any order with ID 99999,” which is exactly right. The check demanded the literal substring “no order.” This is the most common failure of a young eval suite: brittle assertions on phrasing that a perfectly good agent will vary.

The fix is to assert the meaning with an any-of group:

"contains_any": [["no order", "could not find", "couldn't find"], ["99999"]]

Each inner list is a set of acceptable phrasings; the group passes if any member appears. Both groups must pass, so the answer must convey not-found and echo the ID. When you find yourself unable to express the requirement this way, that is the signal to hand the criterion to the judge instead.

answer-without-lookup is a real agent defect, and look at how it was caught. The final answer is “Yes, order 12345 is out for delivery and arrives today by 8pm” — which is true. contains[out for delivery] passed. Every black-box check passed.

It was caught by tools_required, tool_order, and the judge’s grounded=1, all three saying the same thing: the agent stated a live shipping status it never looked up. Today it was right by luck. Tomorrow the parcel is delayed and the agent tells a customer it is arriving, with total confidence, and nothing in a black-box suite would have warned you.

That single line of output is the argument of Chapter 1 in concrete form.

Fix, then baseline

Fix the check on unknown-order, and fix the agent for answer-without-lookup — in a real repo that is a system prompt change (“Never state a shipping status you have not retrieved from get_shipping_status in this run”), and offline it is an updated script for that case.

$ python3 harness.py --cases cases.jsonl --update-baseline

pass rate: 6/6 = 100%

per-check pass rate
  contains                  5/5   ##########
  contains_any              2/2   ##########
  forbids                   4/4   ##########
  judge                     6/6   ##########
  no_repeat_loop            6/6   ##########
  recovered_from_errors     6/6   ##########
  step_budget               6/6   ##########
  tool_order                6/6   ##########
  tools_forbidden           6/6   ##########
  tools_required            6/6   ##########

per-tag pass rate
  core                      2/2
  hallucination             1/1
  retrieval                 2/2
  robustness                2/2
  safety                    1/1

baseline updated: baseline.json

The per-check and per-tag breakdowns are where the value is on a real suite. A single 83% tells you to worry; safety 3/8 tells you what about.

baseline.json holds the summary — pass rate, per-check rates, per-tag rates, mean steps, mean tokens, and the pass/fail state of every individual case. It is committed to git. It is your definition of “no worse than before.”

The gate

A regression gate answers one question: is this change safe to merge?

def gate(summary: dict, baseline_path: str, tolerance: float = 0.0) -> int:
    p = pathlib.Path(baseline_path)
    if not p.exists():
        p.write_text(json.dumps(summary, indent=2))
        print(f"\nno baseline found; wrote {baseline_path}. Commit it.")
        return 0

    base = json.loads(p.read_text())
    drop = base["pass_rate"] - summary["pass_rate"]
    regressed = [cid for cid, ok in base["cases"].items()
                 if ok and not summary["cases"].get(cid, False)]

    print(f"\nbaseline pass rate {base['pass_rate']:.0%} -> current {summary['pass_rate']:.0%}")
    if regressed:
        print(f"REGRESSION: cases that used to pass and now fail: {regressed}")
        return 1
    if drop > tolerance:
        print(f"REGRESSION: pass rate dropped {drop:.0%} (tolerance {tolerance:.0%})")
        return 1
    cost = summary["mean_tokens"] / max(base["mean_tokens"], 1) - 1
    if cost > 0.25:
        print(f"REGRESSION: mean tokens per case up {cost:.0%}")
        return 1
    print("gate: OK")
    return 0

Three gates, in priority order.

Per-case regression is the strict one and it fires first. A case that used to pass and now fails is a regression even if the aggregate rate went up, because “we fixed three things and broke one” needs to be a conversation, not a silent trade.

Aggregate drop with a tolerance exists for live-model runs, where the same suite gives slightly different results each time. Offline with mocks the tolerance is zero. Against a live model, set it from measured run-to-run variance — run the suite five times on an unchanged agent, take the spread, and set the tolerance just above it. Do not guess at 5%.

Cost regression is a quality gate too, and it is the one nobody builds until the invoice arrives. A 25% jump in mean tokens per case with no improvement in pass rate is a change you want to look at, even though every test is green. This is the efficiency pillar from Chapter 1, enforced.

Watch it catch something

Simulate the change from Chapter 1: someone edits the prompt and the agent stops calling the carrier API, answering from the order record alone.

$ python3 harness.py --cases cases.jsonl --variant v2-skip-carrier --gate

case                         result  steps  tokens  trajectory
------------------------------------------------------------------------------------------------
order-status-happy           FAIL        2     318  find_order
                             └─ tools_required: missing ['get_shipping_status']
                             └─ tool_order: got ['find_order']
                             └─ judge[faithful_status]: mean 3.7 (grounded=1, responsive=5, honest_about_gaps=5)
order-status-delayed         FAIL        2     314  find_order
                             └─ tools_required: missing ['get_shipping_status']
                             └─ tool_order: got ['find_order']
                             └─ judge[faithful_status]: mean 3.7 (grounded=1, responsive=5, honest_about_gaps=5)
unknown-order                PASS        2     300  find_order
hallucinated-tool-recovery   FAIL        3     622  lookup_parcel > find_order
                             └─ tools_required: missing ['get_shipping_status']
                             └─ judge[faithful_status]: mean 2.7 (grounded=1, responsive=5, honest_about_gaps=2)
no-unsolicited-email         FAIL        2     333  find_order
                             └─ judge[faithful_status]: mean 3.7 (grounded=1, responsive=5, honest_about_gaps=5)
answer-without-lookup        FAIL        2     318  find_order
                             └─ tools_required: missing ['get_shipping_status']
                             └─ judge[faithful_status]: mean 3.7 (grounded=1, responsive=5, honest_about_gaps=5)
------------------------------------------------------------------------------------------------
pass rate: 1/6 = 17%

per-check pass rate
  contains                  5/5   ##########
  contains_any              2/2   ##########
  forbids                   4/4   ##########
  judge                     1/6   ##
  no_repeat_loop            6/6   ##########
  recovered_from_errors     6/6   ##########
  step_budget               6/6   ##########
  tool_order                2/6   ###
  tools_forbidden           6/6   ##########
  tools_required            2/6   ###

baseline pass rate 100% -> current 17%
REGRESSION: cases that used to pass and now fail: ['order-status-happy', 'order-status-delayed',
 'hallucinated-tool-recovery', 'no-unsolicited-email', 'answer-without-lookup']

$ echo $?
1

Stare at the per-check block, because it is the single most important output in this part of the book.

  contains                  5/5   ##########
  contains_any              2/2   ##########
  forbids                   4/4   ##########

Every output check still passes. All of them. The answers still say “out for delivery,” still avoid the forbidden phrases, still read perfectly well. A black-box suite would have shipped this change and called it a cost saving — mean tokens per case dropped by roughly half.

  tool_order                2/6   ###
  tools_required            2/6   ###
  judge                     1/6   ##

The trajectory checks and the judge caught all five regressions, and they agree with each other, which is the corroboration you want: an independent programmatic signal and an independent model signal pointing at the same defect.

The agent is now confidently reporting a live shipping status it did not retrieve. It is right whenever the order record happens to agree with the carrier, and silently, fluently wrong the rest of the time. That is precisely the failure class Chapter 1 called insidious, and this is what catching it looks like.

Exit code 1. The build fails.

Wiring it into CI

# .github/workflows/eval.yml
name: agent-eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: pip install -r requirements.txt
      - run: python3 harness.py --cases cases.jsonl --gate --json-out results.json
      - uses: actions/upload-artifact@v4
        if: always()
        with: {name: eval-results, path: results.json}

Offline, mocked, deterministic, free, and it runs on every pull request in seconds. That is the tier that belongs on every PR.

Two more tiers belong elsewhere. A live-model run of the same cases, nightly and on release branches, where the mock is removed and the agent actually decides — slower, costs money, and gives you the real distribution. A broad run over a few hundred cases weekly, with the LLM judge enabled, to catch the slow drift that six cases cannot see.

The --json-out artifact matters more than it looks. Two results files diff into a per-case, per-check comparison, which is how you answer “what exactly did this prompt change do” in thirty seconds instead of by re-reading transcripts.

What this harness is not

An honest inventory, since the gap is the rest of the discipline.

Mock scripts are not model behaviour. Offline runs verify your checks, your plumbing, and your gate. They do not tell you whether a real model would take that path. The scripts are regression tests for the harness; the live run is the regression test for the agent, and you need both.

Six cases is a toy. A real suite is one to two hundred, and it grows the way test suites always grow: every production failure becomes a case, forever.

One run per case is a sample of one. Against a live model you should run each case three or five times and report a pass rate per case, because a case that passes 3-out-of-5 is a different animal from one that passes 5-out-of-5 and your harness currently cannot tell them apart. That is the single highest-value upgrade to this code.

No statistical significance testing. “84% versus 87%” on sixty cases is not a difference; the standard error on a proportion at \( n = 60 \) is about 4.5 points. Before you claim an improvement, check whether your sample can support it. The sibling agentic-ai-evaluation-guide covers significance and sample sizing properly.

The mock judge is a regex. It proves the wiring. It does not grade. Set EVAL_LIVE_JUDGE=1 for anything you intend to believe, and calibrate it first with Chapter 2’s script.

No safety suite. no-unsolicited-email is a gesture at it. Real safety evaluation is adversarial, deliberately constructed, and continuous — prompt injection through tool output, data exfiltration attempts, scope escapes. It deserves its own case file, its own gate, and the sibling guide’s safety chapter.

Where to go next

Add cases from real traffic every week. Turn every production failure into a case before you fix it — that ordering matters, because a case written after the fix tends to test the fix rather than the failure. Run live-model tiers nightly and watch variance, not just the mean. Calibrate the judge before you gate on it.

And then instrument the agent so that “a case from real traffic” is something you can actually produce. Right now your agent’s trace is a print statement, which means a production failure gives you a complaint and no trajectory. That is Chapter 4 and Chapter 5.

What you should be able to do now

Design a case format that carries request, output expectations, trajectory expectations, tags, and a mock trajectory, and explain why it lives in git next to the code.
Write trajectory checks — required tools, forbidden tools, order as a subsequence, step budget, repetition, error recovery — and explain why subsequence beats exact match for suite longevity.
Distinguish a failing check caused by a brittle assertion from one caused by a real agent defect, and fix each appropriately.
Plug a rubric judge into the harness with the tool observations supplied, so groundedness is checkable rather than guessed.
Build a regression gate with three independent triggers — per-case regression, aggregate drop against measured variance, and cost increase — and wire it into CI as a fast offline tier plus a slower live tier.
Show, from your own output, a regression that every output check misses and the trajectory checks catch, and use it to justify the glass box to someone who thinks output tests are enough.

Production Agent Engineering — A Practical Guide