Part 5 — Quality and Observability
At the end of Part 1 you ran your agent once, read the output, and it was right.
That feeling — “it worked when I tried it” — is the most expensive feeling in this field.
It is not evidence. You ran one input through a system whose defining property is that the same input does not produce the same output twice. You observed one sample from a distribution and concluded something about the distribution. If a colleague told you they had validated a payments service by calling it once, you would not accept it, and an agent is a payments service with a random number generator wired into the control flow.
Here is what “it worked when I tried it” cannot tell you.
It cannot tell you the success rate on the other ninety-nine requests your users will actually send. It cannot tell you whether the answer was right for the right reason — whether the agent looked the fact up or guessed it and got lucky. It cannot tell you what the change you are about to merge does to any of that. It cannot tell you what the run cost, how long it took at p99, or how close it came to the step cap. And when it fails at 3 a.m. for a customer you cannot reach, it gives you nothing to debug with.
Everything in Part 5 exists to replace that feeling with numbers you can defend and traces you can read.
What this part covers
Chapter 1 — Quality when the same input gives different answers. Why the testing instincts that work on deterministic software mislead you here, stated precisely rather than hand-wavily. The four pillars that make “quality” a measurable word: effectiveness, efficiency, robustness, safety. Then the decision that structures everything else — the outside-in hierarchy, black-box evaluation of the final answer versus glass-box evaluation of the trajectory, what each catches, what each misses, and why shipping without both leaves a specific class of failure invisible.
Chapter 2 — Who does the judging. Programmatic checks, LLM judges, agent judges, humans, and real users, with an honest account of what each costs and what each is bad at. How to actually build an LLM judge: rubric design, forced structure, calibration against human labels, and the biases — position, verbosity, self-preference — that make an uncalibrated judge worse than no judge. Runnable code, including a position-swap harness and a calibration script that computes agreement with human labels.
Chapter 3 — Mini-project 8: build an eval harness. The build. A case format, a runner, programmatic checks, trajectory checks, an LLM judge with a mock mode, a report, and a regression gate that exits non-zero when a change makes the agent worse. Every output in the chapter is real terminal output from code you can run offline with no API key.
Chapter 4 — Observability: seeing inside the agent’s mind. Monitoring versus observability, stated crisply enough to act on. The three pillars — logs, traces, metrics — specialised for agents: what to log per step and what must never touch your logs, what a good agent trace contains, and the difference between system metrics that page an SRE and quality metrics that page a product owner. Plus the current state of the OpenTelemetry GenAI semantic conventions, which is the standard your instrumentation should speak.
Chapter 5 — Mini-project 9: instrument your agent with tracing. The second build. Real OpenTelemetry spans around the loop, the model calls, and the tool calls, with token, cost, latency, and error attributes drawn from the GenAI conventions. A trace viewer in forty lines so you can see the tree in your terminal, an export path to a hosted backend, and a worked debugging walkthrough: a bad run, its trace, and the exact reasoning that takes you from “the answer was wrong” to “step one passed an order ID where a tracking number belonged.”
What you will have built
Two artifacts, both running against the agent you have been growing since Part 1.
An eval harness: a versioned case file, a runner that executes the agent over every case, three layers of checking (programmatic assertions, trajectory assertions, and a rubric judge), a scored report broken down by check and by tag, and a CI gate that fails a build on regression.
A traced agent: nested spans for the agent run, each model call, and each tool call, carrying token counts, cost, latency, and error status, exportable to any OpenTelemetry backend, with an in-terminal viewer for when you just want to look.
A note on scope
This book has a sibling repository, agentic-ai-evaluation-guide, which covers evaluation as a subject in its own right across twelve chapters — evaluation frameworks, metrics and benchmarks, tool-use and reasoning evaluation, safety evaluation, multi-agent evaluation, real-world testing, automated evaluation pipelines, dataset construction, tooling, and production monitoring.
Part 5 is deliberately not a second copy of that. Its job is the builder’s slice: exactly enough theory to make correct decisions about your own system, and then the code you wire into it this afternoon. Where you need depth — benchmark contamination, inter-annotator agreement methodology, red-teaming programmes, the full metric zoo — the sibling guide is named at the point where it matters and you should go read it there.
Where this sits
Parts 1 through 4 made your agent capable. It loops, it calls tools, it remembers, it orchestrates.
Part 5 is what makes it trustworthy, and the distinction is not decorative. Capability is what you demo. Trustworthiness is what lets you deploy on a Friday, hand the pager to someone else, and change the prompt next month without holding your breath.
Start with Chapter 1.