Quality when the same input gives different answers

Run this experiment once and the rest of the chapter becomes obvious.

Take your Part 1 agent, point it at a real model, and send the same request ten times. Read the ten trajectories side by side.

Some will call find_order then get_shipping_status. One will call get_shipping_status first with the order ID in the tracking field, get nothing back, and recover. One will answer from the first lookup without checking the carrier at all, and — because the order record happens to be enough — produce a correct answer by a route you would not have approved. One will take five steps to do what the others did in two.

Ten runs, one input, four distinct behaviours, and possibly ten out of ten “correct” final answers.

Now write the unit test.

That is the whole problem, and every technique in this part is a response to it.

What exactly broke

It is worth being precise about which assumptions failed, because most people over-generalise from “AI is non-deterministic” into “you cannot test it,” which is wrong and leads to shipping on vibes.

Assumption 1: the same input produces the same output. Gone, and not recoverable by setting temperature to zero. Even at temperature zero, floating-point non-associativity across batched inference, provider-side load balancing between hardware revisions, and silent model updates behind a stable model alias all reintroduce variation. Treat determinism as unavailable and design accordingly.

Assumption 2: correctness is a boolean. For a summary, a plan, or a customer reply, “correct” is a judgment with a range of acceptable answers and a fuzzy boundary. The useful reframe: correctness is a distribution, and what you measure is a rate over a sample. Your agent is not correct or incorrect. It is 87% correct on this case set, with a confidence interval you should be able to state.

Assumption 3: failures are loud. In deterministic software, a failure crashes, throws, or returns a visibly wrong number. Agent failures return 200 OK with a fluent, confident, plausible, wrong answer. Your error rate dashboard stays flat while quality collapses. This is the single most dangerous property of the whole category: the failure mode is silence.

Assumption 4: the bug is in the code. You cannot set a breakpoint inside a judgment. When an agent picks the wrong tool, no line of your Python is wrong. The defect lives in a prompt, a tool description, a context assembly decision, or the model’s weights, and none of those are things a debugger steps through.

What survives is more than people expect. Your tool implementations are ordinary code and should have ordinary unit tests. Your orchestration layer — step caps, budgets, retry policy, state merging — is deterministic and should be tested exhaustively, and Part 4’s engine was built that way. The non-determinism is confined to one component: the model’s decision at each step. Test everything else conventionally, and build statistical machinery only around the part that actually needs it.

Saying it out loud. The reason you can’t just unit-test an agent is that four assumptions underneath normal testing all break at once. Same input no longer gives the same output, and temperature zero doesn’t save you — batched inference, load balancing across hardware, and silent updates behind a stable model alias all reintroduce variation. Correctness stops being a boolean and becomes a rate over a sample, so the honest sentence is “87 percent on this case set,” not “it works.” Failures stop being loud — you get a 200 OK with a fluent, confident, wrong answer, so your error dashboard stays flat while quality collapses. And the bug usually isn’t in your code at all, it’s in a prompt or a tool description, and no debugger steps through those. What people over-generalise is the conclusion: your tools and your orchestration are ordinary deterministic code and deserve ordinary tests. Only the model’s per-step decision needs statistical machinery.

Compounding: why step count is the risk multiplier

There is a piece of arithmetic worth internalising because it explains why agents feel so much less reliable than the models inside them.

Suppose the model makes the right decision at each step with probability \( p \), and a trajectory needs \( n \) steps, each of which must go right. The chance the whole trajectory is clean is \( p^n \).

At \( p = 0.97 \) — a genuinely strong model — a six-step trajectory completes cleanly 83% of the time. At twelve steps it is 69%. At twenty-five steps, which is not an unusual length for a research or coding agent, it is 47%.

Three consequences follow directly.

Small per-step improvements matter enormously, because they are exponentiated. Shorter trajectories are more reliable trajectories, which is a quality argument for the “encode what you can, let the model decide only what needs judgment” heuristic from Part 4. And recovery is worth more than accuracy: an agent that notices a bad step and repairs it turns \( p^n \) into something much friendlier, which is why Part 1 spent a whole version on turning errors into observations.

This is also why evaluating only the final answer is insufficient. A 47% clean-trajectory rate that produces 90% correct final answers means your agent is compensating for its own mistakes — good news — but you cannot see any of that, or notice when it stops being true, from the output alone.

Saying it out loud. Here’s why agents feel so much less reliable than the models inside them. If the model gets each step right with probability p and the task needs n steps in a row, your clean-trajectory rate is p to the n. At 97 percent per step — a genuinely strong model — six steps is 83 percent, twelve steps is 69 percent, and twenty-five steps, which is normal for a research or coding agent, is 47 percent. Three things fall out of that. Tiny per-step gains matter enormously because they get exponentiated, shorter trajectories are more reliable trajectories, and recovery is worth more than raw accuracy — an agent that notices a bad step and repairs it breaks the exponent. It’s also why final-answer-only evaluation lies to you: 47 percent clean trajectories producing 90 percent correct answers means the agent is compensating for its own mistakes, and you can’t see when that stops being true.

Four pillars: making “quality” a word you can measure

“Is the agent good?” is unanswerable. Split it into four questions that each have an owner, a metric, and a decision attached, and it becomes tractable. This framing comes from Google’s Agent Quality whitepaper and it is the most useful thing in it.

Effectiveness — did it achieve the user’s actual goal? Not “did it produce output,” and not “did it call a tool successfully.” Did the thing the user wanted to happen, happen. For a coding agent that is PR acceptance rate, not compile rate. For a support agent it is resolution without escalation, not response sent. Effectiveness is the pillar that connects to a business metric, and if you cannot name that metric you do not yet know what your agent is for.

Efficiency — did it get there sensibly? An agent that books a flight in twenty-five steps with five failed tool calls and three self-corrections is a low-quality agent even when it succeeds, because it costs five times as much, takes five times as long, and has five times as many chances to go wrong tomorrow. Measure tokens, wall-clock time, step count, and tool-call count per successful task. Efficiency is the pillar most teams skip and then discover through their invoice.

Robustness — what happens when the world misbehaves? The API times out. The record has a null where a string should be. The user’s request is ambiguous or contradicts itself. A robust agent retries, degrades gracefully, asks for clarification, or reports what it could not do — and the failure mode you are testing for is the agent that instead invents a plausible answer. Robustness is only measurable if your case set deliberately contains adversity, which means writing cases where the tools fail on purpose.

Safety and alignment — should it have done that at all? Did it stay in scope, refuse what it should refuse, resist instructions arriving through tool output, and avoid leaking data it held? This one is a gate, not a score. An agent that is 99% effective and 1% harmful is not shippable, and no amount of effectiveness buys it down.

The four pillars are not equally weighted and they are not independent. Efficiency and effectiveness trade against each other constantly — more lookups, more grounding, more cost. The value of naming them separately is that you notice when you are trading, instead of optimising one and silently regressing another.

Notice the structural point: none of these are measurable from the final answer alone. You cannot count steps you did not record. You cannot tell which API call failed if you did not trace it. You cannot verify that the agent stayed in scope if you never saw what it did in the middle. A four-pillar quality model implies an observability requirement, which is why Chapters 4 and 5 exist and why they are in the same part of this book as the evaluation chapters rather than filed under operations.

Saying it out loud. “Is the agent good?” is unanswerable, so I split it into four questions that each have a metric and an owner. Effectiveness is did the user’s actual goal happen — PR acceptance rate, not compile rate; resolution without escalation, not response sent. Efficiency is did it get there sensibly, measured in tokens, steps, and wall-clock per successful task; that’s the pillar teams skip and then rediscover through the invoice. Robustness is what happens when the API times out or the request is ambiguous, and it’s only measurable if your case set deliberately contains adversity. And safety is a gate, not a score — an agent that’s 99 percent effective and 1 percent harmful isn’t shippable, and no amount of effectiveness buys that down. The tradeoff you’re naming by separating them is that efficiency and effectiveness pull against each other constantly, so you want to notice when you’re trading rather than optimise one and silently regress the other.

The outside-in hierarchy

Now the decision that organises your entire eval strategy.

There are two places to look, and you look at them in a fixed order.

Stage 1: the black box — end-to-end evaluation

Start outside. One question: did the agent achieve the goal?

Feed it a realistic request, take the final answer, and score it against what a good outcome looks like. Nothing about the internals enters this stage. The metrics are task success rate (binary or graded), output quality against a rubric, and — for interactive agents — user satisfaction.

Black-box evaluation has three properties that make it the right starting point.

It measures the thing you actually care about, so it cannot be gamed by a component that scores well while the product gets worse. It is cheap to build, because a case is a request and an expectation. And it survives refactors: change frameworks, swap models, restructure the whole orchestration layer, and your black-box cases still apply unchanged, which makes them the only tests that reliably outlive an architecture.

What it misses is everything about how.

An agent that guessed and got lucky scores identically to one that looked the fact up. An agent that took nineteen steps scores identically to one that took three. An agent that called your send_email tool along the way scores identically to one that did not, unless you thought to check. Those are not hypothetical: the third is a live incident and the first is a time bomb, because “guessed and got lucky” becomes “guessed and got it wrong” the moment the data shifts.

Black-box evaluation tells you what went wrong. For anything else, open the box.

Saying it out loud. Black-box evaluation asks one question — did the agent achieve the goal — and nothing about the internals gets in. I start there for three reasons: it measures the thing you actually care about so no component can score well while the product gets worse, a test case is just a request and an expectation so it’s cheap to write, and it survives refactors. Swap the model, change frameworks, rewrite the orchestration, and those cases still apply, which makes them the only tests that outlive an architecture. What it can’t see is everything about how: the agent that guessed and got lucky scores the same as the one that looked it up, and the agent that quietly called send_email along the way scores the same as the one that didn’t. That first one’s a time bomb, because “guessed and got lucky” becomes “guessed and got it wrong” the day the data shifts.

Stage 2: the glass box — trajectory evaluation

The trajectory is the ordered record of what the agent thought, which tools it called with which arguments, what came back, and what it did with that. Evaluating it means asking, of each link in the chain, whether it was a reasonable thing to do given what was known at the time.

There are six places a trajectory goes wrong, and knowing the list turns debugging from staring into triage.

Planning. The reasoning itself is bad: a nonsensical decomposition, a plan that ignores half the request, a repetitive loop where the same idea is restated in three consecutive steps.

Tool selection. The wrong tool, no tool when one was needed, a hallucinated tool name, or an unnecessary call. Part 1’s error-as-observation mechanism makes hallucinated tool names survivable; it does not make them free, and a rising rate of them means your tool descriptions have drifted.

Tool parameterisation. The right tool called wrongly: missing arguments, a value in the wrong field, an order ID where a tracking number belongs. This class is worth a check of its own because it produces soft failures — the call succeeds, the tool returns “not found,” and nothing anywhere reports an error.

Observation interpretation. The tool returned the right answer and the agent misread it: misparsed numbers, missed the key entity, or — the important one — failed to recognise an error state and proceeded as though the call had worked.

Retrieval quality. If there is RAG in the loop, a bad answer may be a retrieval failure wearing a generation failure’s clothes. The diagnostic question is whether the right chunk was in the context at all; if it was, the fault is downstream. The sibling agentic-ai-evaluation-guide covers retrieval evaluation in depth, and you should use it rather than reinventing precision-at-k.

Efficiency and robustness of the path. Redundant calls, work done twice, an unhandled exception, a step cap hit at the end of a run that was nearly finished.

Trajectory evaluation is what converts “the final answer is wrong” into “the final answer is wrong because step one passed the order ID as a tracking number, and every step after that was reasoning from an empty result.” That sentence is the entire point of the discipline.

Saying it out loud. Trajectory evaluation is opening the box and asking, of each step, whether that was a reasonable thing to do given what was known at the time. There are six places it goes wrong and having the list turns debugging into triage: bad planning, wrong tool selection, wrong parameters on the right tool, misreading the observation, bad retrieval, and an inefficient or unrobust path. The parameterisation one is worth calling out because it fails softly — the call succeeds, the tool returns “not found,” and nothing anywhere logs an error. What this buys you is converting “the answer is wrong” into “the answer is wrong because step one passed the order ID into the tracking field and every step after that was reasoning from an empty result.” That sentence is the whole discipline.

Why the order matters

Run the black box first, always.

If end-to-end success is 95% and your users are happy, the trajectory analysis you were about to spend a week on will find inefficiencies you do not need to fix yet. Component-level metrics have a way of becoming an optimisation target divorced from outcomes — you improve tool-selection accuracy by four points and end-to-end success does not move, because tool selection was never the binding constraint.

Open the box when the black box tells you something is wrong, or when the pillar you care about is one the black box cannot see: efficiency and safety are both invisible from the outside.

Saying it out loud. You always run the black box first, and the reason is Goodhart. Component metrics have a habit of becoming a target divorced from outcomes — you push tool-selection accuracy up four points, end-to-end success doesn’t move, and it turns out tool selection was never the binding constraint. So if end-to-end is at 95 percent and users are happy, the week of trajectory analysis you were about to do will find inefficiencies you don’t need to fix yet. You open the box when the black box says something’s wrong, or when the pillar you care about is one the outside can’t see — and efficiency and safety are both invisible from the outside.

Where they meet

There is a category of failure only visible when you check both, and it is the reason this book insists on both.

Consider a change that makes your agent stop calling the carrier API and answer from the order record alone. Every “does the answer contain the right status” check still passes, because the order record usually implies the status. Output quality looks stable. Cost goes down. It looks like an improvement.

What actually happened is that the agent stopped grounding its most volatile claim in live data, and it will be confidently wrong the first time a parcel is delayed after the record was written.

You will watch exactly this happen in Chapter 3, in real terminal output: output checks stay at 5/5 while the trajectory checks collapse from 6/6 to 2/6. Black-box evaluation is blind to it by construction. That is why you need both, and it is the strongest single argument in this part.

Saying it out loud. Here’s the failure that justifies doing both, and it’s my favourite example. Suppose a change makes your agent stop calling the carrier API and answer from the order record instead. Every output check still passes, because the record usually implies the status. Quality looks stable, cost goes down, it looks like a win. What actually happened is the agent stopped grounding its most volatile claim in live data, and it’ll be confidently wrong the first time a parcel is delayed after the record was written. In the worked example later, the output checks stay at five out of five while the trajectory checks fall from six out of six to two out of six. Black-box evaluation is blind to that by construction — not because it’s badly written, but because the information isn’t in the output.

Where quality gets built

One structural claim before the mechanics, because it changes what you do on Monday rather than what you believe.

Quality for agents is not a phase at the end. It cannot be, because the artifacts it needs — trajectories, tool call records, token counts, timings — either exist because you emitted them or do not exist at all. There is no equivalent of attaching a profiler to a running process after the fact. If your agent did not record what it did, that run is unrecoverable and you will be reduced to asking the user what they typed.

So the practical rule: the instrumentation goes in with the loop, not after it. Part 1’s Agent.run already logged thought, action, and observation for exactly this reason. Chapter 5 turns those prints into real spans, and the reason that upgrade is a fifty-line change rather than a rewrite is that the structure was there from the first version.

The same applies to the case set. Ten realistic cases written the week you start are worth more than a hundred written after launch, because the ten are available to every decision you make in between.

Saying it out loud. Quality for agents can’t be a phase at the end, and the reason is mechanical rather than philosophical. The artifacts you’d need — trajectories, tool call records, token counts, timings — either exist because you emitted them or they don’t exist at all. There’s no attaching a profiler to a run that already finished. If the agent didn’t record what it did, that run is unrecoverable and you’re reduced to asking the user what they typed. So the instrumentation goes in with the loop, not after it, which is why turning prints into real spans later is a fifty-line change rather than a rewrite. Same logic for the case set: ten realistic cases written the week you start beat a hundred written after launch, because the ten inform every decision in between.

Choosing what to evaluate first

You cannot evaluate everything, so here is an order that consistently pays.

Start with ten to thirty real requests from your actual domain — from logs, from a support queue, from the person who asked for the agent. Not synthetic ones you invented, which will be politer, shorter, and better-spelled than reality. This is the single highest-value hour in the whole exercise.

For each, write down what a good outcome looks like in a form a program can check, even partially: a phrase that must appear, a fact that must be right, a tool that must have been called, a tool that must not have been.

Add adversity deliberately. A request about an entity that does not exist. A tool that fails. An ambiguous ask. A request that tries to get the agent to do something out of scope. These will be a third of your case set and will find most of your bugs.

Tag every case — retrieval, robustness, safety, hallucination — because aggregate pass rate hides the fact that all your failures are in one category.

Then, and only then, worry about scale. A hundred well-chosen cases with real expectations beat ten thousand generated ones, and the sibling agentic-ai-evaluation-guide has a whole chapter on dataset construction and synthetic case generation when you get there.

Saying it out loud. If I had one hour, I’d spend it collecting ten to thirty real requests from logs or the support queue — not synthetic ones, which are always politer, shorter, and better spelled than reality. For each one I write down what a good outcome looks like in a form a program can check: a fact that must be right, a tool that must have been called, a tool that must not have been. Then I deliberately add adversity — an entity that doesn’t exist, a tool that fails, an ambiguous ask, an out-of-scope request — and that’ll be about a third of the set and will find most of your bugs. Tag every case, because an aggregate pass rate hides the fact that all your failures live in one category. A hundred well-chosen cases with real expectations beat ten thousand generated ones, every time.

What this buys you

When this is in place, three questions that were previously arguments become measurements.

“Is the new model better for us?” becomes a fifteen-minute run with a number and a per-tag breakdown. “Did that prompt change help?” becomes a diff between two reports. “Can we ship this?” becomes a gate that either passes or does not, and does not care how confident anyone in the room is.

That is the actual deliverable of this part: replacing meetings with runs.

Chapter 2 covers who does the judging, and Chapter 3 is where you build it.

Saying it out loud. The payoff is that three questions that used to be arguments become measurements. “Is the new model better for us?” becomes a fifteen-minute run with a number and a per-tag breakdown. “Did that prompt change help?” becomes a diff between two reports. “Can we ship?” becomes a gate that passes or doesn’t, and doesn’t care how senior the most confident person in the room is. That’s the actual deliverable — you’re replacing meetings with runs. And the cost is honest: somebody has to own the case set, and a case set nobody maintains rots into a gate everyone learns to override.

What you should be able to do now

State precisely which testing assumptions break for agents — repeatability, boolean correctness, loud failure, code-located bugs — and which parts of your system are still ordinary deterministic code that deserves ordinary tests.
Compute the compounding reliability of a multi-step trajectory from a per-step success rate, and use it to argue for shorter trajectories and better recovery rather than only for a better model.
Break “is it good?” into effectiveness, efficiency, robustness, and safety, name the metric and the owner for each in your own system, and explain why none of them are measurable from the final answer alone.
Apply the outside-in hierarchy: run black-box evaluation first, open the glass box when it fails or when the pillar you care about is invisible from outside, and name the six trajectory failure classes.
Describe a concrete failure that black-box checks cannot see and trajectory checks catch, and use it to justify building both.
Assemble a first case set from real requests with deliberate adversity and tags, rather than from imagination and at scale.

Production Agent Engineering — A Practical Guide