Observability: seeing inside the agent’s mind

Your agent gave a customer a wrong answer at 03:14 this morning.

You know because they complained. The dashboard is green: no exceptions, 200s across the board, p99 latency normal, error rate zero. Your logs contain the request ID, the final response, and nothing else.

What did the agent do? Which tool did it call first? What did that tool return? Was the wrong fact hallucinated, or did it come back wrong from a system you do not own?

You cannot answer any of it, and — this is the part that hurts — you cannot reproduce it either, because running the same request now will produce a different trajectory.

The run happened once and you did not record it. That failure is now permanently unavailable to you.

This chapter is about not being in that position.

Monitoring versus observability

The distinction is thrown around loosely; here is the version that changes what you build.

Monitoring answers questions you thought of in advance. You decided that p99 latency, error rate, and queue depth mattered, you built dashboards for them, and you set alerts. It tells you that something is wrong, fast, for the failure modes you anticipated.

Observability is the property that you can answer questions you did not think of in advance, from data the system already emitted. “Show me every run last Tuesday where the agent called get_shipping_status before find_order, and tell me how many of those hit the step cap” is a question nobody built a dashboard for. If your telemetry can answer it without a deploy, you are observable.

For deterministic software, monitoring gets you a long way, because the failure modes are enumerable: it crashed, it slowed down, it ran out of memory. For agents it does not, because the interesting failures produce no error at all. The agent that skipped a lookup, the agent that took nineteen steps to do a two-step job, the agent that misread a tool result — every one of those returns 200 OK in normal latency, and every one is invisible to monitoring by construction.

The practical consequence: you are not instrumenting for uptime, you are instrumenting for reconstruction. The bar is that any past run can be replayed on a screen, step by step, with everything the agent saw and everything it decided. Design to that bar and the dashboards fall out for free. Design to the dashboards and you will find yourself, at 03:14, with a green wall and a wrong answer.

Saying it out loud. Monitoring answers questions you thought of in advance — latency, error rate, queue depth — and it tells you fast that something you anticipated has broken. Observability is the property that you can answer a question you didn’t think of, from data the system already emitted, without shipping a deploy. For normal software monitoring gets you a long way because the failures are enumerable: it crashed, it slowed down, it ran out of memory. For agents it doesn’t, because the interesting failures return 200 OK at normal latency — the agent that skipped a lookup, took nineteen steps for a two-step job, or misread a tool result. So the framing I’d use is that you’re not instrumenting for uptime, you’re instrumenting for reconstruction: the bar is that any past run can be replayed on a screen step by step. Design to that bar and the dashboards fall out for free.

Pillar 1: logs — the agent’s diary

A log is a timestamped record of one discrete event. Logs tell you what happened at a point in time.

The upgrade from print() is not “use the logging module.” It is structured logging: every record is a JSON object with a stable set of fields, so it can be queried rather than grepped.

log.info("tool_call_completed", extra={
    "run_id": run_id,
    "trace_id": trace_id,
    "step": 2,
    "tool": "get_shipping_status",
    "args_hash": "sha256:4f2b...",
    "duration_ms": 412,
    "ok": True,
    "result_bytes": 143,
})

The difference is not aesthetic. "how often does get_shipping_status take over a second" is a query against that record and an impossible question against print(f"called {name}").

Saying it out loud. A log is a timestamped record of one discrete event, and the upgrade that matters isn’t switching from print to the logging module — it’s making every record a JSON object with stable fields, so you can query it instead of grepping it. The difference is concrete: “how often does get_shipping_status take over a second” is a one-line query against a structured record and an impossible question against a formatted string. Logs tell you what happened at a point in time, which is necessary but not sufficient, because the thing you usually need is how the points connect.

What to log at each step

Per agent run: a run ID, the trace ID, the user or tenant ID, the agent name and version, the model and its version, the prompt or config version, the entry point, and the outcome — answered, step-capped, budget-exhausted, errored.

Per step: the step index, the model’s stated reasoning, which tools it requested with which arguments, what each returned, the token counts, the latency, and any errors.

Per tool call: the tool name, argument shape, duration, success or failure, and the size of the result.

Two habits pay for themselves repeatedly.

Log the intent before the action, and the outcome after. Two records, not one. This is what distinguishes “the agent decided not to call the tool” from “the agent called the tool and the process died mid-call” — which are the same absence of a log line if you only log completions.

Log the version of everything that can change behaviour. Model, prompt, tool schema, retrieval index. Six weeks from now, “quality dropped on the 14th” is only actionable if you can join it to “the prompt version changed on the 14th.”

Saying it out loud. Per run you want a run ID, trace ID, tenant, agent version, model version, prompt version, and the outcome — answered, step-capped, budget-exhausted, or errored. Per step, the reasoning, the tool calls with arguments, what came back, tokens, latency, errors. Two habits pay for themselves over and over. Log the intent before the action and the outcome after, as two records, because otherwise “the agent decided not to call the tool” and “the agent called it and the process died mid-call” are the same missing log line. And log the version of everything that can change behaviour, because six weeks later “quality dropped on the 14th” is only actionable if you can join it to “the prompt version changed on the 14th.”

What must never go in

This is the part with legal consequences, so it is a checklist rather than advice.

Never log raw credentials, tokens, or keys — including ones that arrived inside a tool argument, which is the path people miss. Never log full payment details. Redact or tokenise personal data before it reaches storage, not in a cleanup job afterwards; once it is in your log store it is in your backups, your replicas, and your vendor’s system. Be deliberate about full prompt and response content: it is enormously useful for debugging and it is the highest-risk data you hold, because user messages contain whatever users typed.

The workable pattern is a redaction layer between your instrumentation and your exporter, applied by field name and by regex, with content capture off by default and switchable per environment:

REDACT_KEYS = {"to", "email", "body", "phone"}

def _safe_args(args: dict) -> str:
    return json.dumps({k: ("[redacted]" if k in REDACT_KEYS else v) for k, v in args.items()})

Also truncate. A tool that returns a 400KB document will otherwise put 400KB into every span, every log line, and every export, and you will discover this through your observability bill. Truncate at a few kilobytes, record the original length, and store the full artifact separately with a reference if you need it.

Saying it out loud. This is the bit with legal consequences, so it’s a checklist rather than a judgment call. No raw credentials or tokens, including ones that arrived inside a tool argument — that’s the path people miss. No full payment details. Redact personal data before it reaches storage, not in a cleanup job afterwards, because once it’s in the log store it’s in your backups, your replicas, and your vendor’s system. Full prompt and response capture is the highest-value debugging data and the highest-risk data you hold, so it goes behind a redaction layer and stays off by default. And truncate everything: one tool returning a 400KB document puts 400KB into every span and every export, and you’ll find out via your observability bill.

Sampling

Full-detail capture on every production request is usually too expensive. The standard policy, and it is a good one:

100% of runs that errored, hit the step cap, or received negative user feedback.
100% of runs on a canary or a new prompt version.
1–10% of ordinary successful runs, sampled randomly.
Metrics on everything, always, because they are cheap.

The asymmetry is the point: failures are rare and precious, successes are common and interchangeable.

Saying it out loud. You can’t afford full-detail capture on every production request, and you don’t need it. The policy is asymmetric on purpose: 100 percent of runs that errored, hit the step cap, or got negative feedback, 100 percent of canary and new-prompt-version runs, and somewhere between 1 and 10 percent of ordinary successes sampled at random. Metrics stay on everything because they’re cheap. The reasoning is that failures are rare and precious and successes are common and interchangeable — losing a random successful run costs you nothing, and losing the one run that went wrong costs you the whole investigation.

Pillar 2: traces — following the footsteps

If logs are diary entries, a trace is the narrative that connects them.

A trace is one end-to-end operation — one agent run — decomposed into nested spans, each a named unit of work with a start time, a duration, a status, and a bag of attributes. Spans nest, so the trace is a tree, and the tree is the shape of what your agent actually did.

Consider the failure from the top of the chapter. Isolated logs give you WARN: no shipment found and INFO: run ended at step cap, and neither explains anything. The trace gives you this:

invoke_agent
├── chat                    -> wants get_shipping_status(tracking_number="12345")
├── execute_tool            -> "No shipment found for 12345."
├── chat                    -> wants get_shipping_status(tracking_number="12345")
├── execute_tool            -> "No shipment found for 12345."
├── chat                    -> wants lookup_parcel(...)          [ERROR: no such tool]
└── chat                    -> step cap

The order ID went into the tracking field, find_order was never called, and everything after step one was reasoning from an empty result. Root cause in five seconds, from structure alone, without reading a single prompt.

That is why tracing is the pillar to build first if you can only build one. It is also why Part 1’s loop printed thought, action, and observation from version three — that print statement was a trace with the wrong exporter.

Saying it out loud. If logs are diary entries, a trace is the narrative connecting them — one run decomposed into nested spans, each with a start, a duration, a status, and attributes. And the reason it’s the pillar to build first is that structure alone often gives you root cause. Take the classic: isolated logs say “no shipment found” and “run ended at step cap,” which explains nothing, while the trace tree shows the order ID went into the tracking field, find_order was never called, and every step after the first was reasoning from an empty result. That’s five seconds of reading versus an afternoon of guessing, and you never had to open a prompt.

What a good agent trace contains

Span hierarchy that mirrors the agent’s structure. A root span for the run, a child per model call, a child per tool call, and — if you have sub-agents — a child span per delegated task, so the parallel fan-out from Part 4 shows up as parallel spans rather than a flat list.

Attributes that make spans queryable. Model name, token counts in and out, cost, latency, tool name, argument summary, result summary, finish reason, error type. The rule: if you might one day want to filter or group by it, it is an attribute.

Status. Every span is OK or ERROR, and an error span carries an error.type you can group by. Take care here — the interesting agent failures are soft: the tool ran fine and returned “not found.” Span status OK, content says no. If you only surface hard errors, the failure above is invisible again, so put the result summary on the span and make sure your viewer shows it.

Correlation IDs. A trace ID that appears in your logs, your user feedback events, and your eval records. This one field is what turns a thumbs-down into a reproducible case, and it is the highest-leverage line of code in your instrumentation.

Context propagation across process boundaries. If the agent calls a service that is also instrumented, the trace should continue into it. This is what OpenTelemetry’s context propagation does for you and why you should not invent your own trace format.

Saying it out loud. A good agent trace has a span hierarchy that mirrors what the agent actually did — root span for the run, a child per model call, a child per tool call, a child per delegated sub-agent so parallel work shows as parallel spans. Attributes on every span for anything you might want to filter or group by later: model, tokens, cost, latency, tool name, argument summary, result summary, finish reason. And a correlation ID that appears in your logs, your feedback events, and your eval records — that single field is what turns a thumbs-down into a reproducible case. The subtlety to name is soft failure: the tool ran fine and returned “not found,” so span status is OK and the content says no. If your viewer only surfaces hard errors, the most common agent failure is invisible all over again.

The GenAI semantic conventions

If you name your attributes yourself, your traces are legible only to your own tooling. There is a standard: the OpenTelemetry GenAI semantic conventions, which define attribute names for model calls, tool calls, and agent operations.

Current state, as of this writing in 2026, and it matters that you know it: the GenAI conventions now live in their own repository, open-telemetry/semantic-conventions-genai, and they are still marked Development, not Stable (https://github.com/open-telemetry/semantic-conventions-genai). Only shared core attributes like error.type and server.address are stable. Names have moved across releases — gen_ai.system became gen_ai.provider.name, and prompt_tokens/completion_tokens became input_tokens/output_tokens.

Use them anyway. A convention that moves is still enormously better than a private vocabulary, every serious backend ingests them, and the migrations are mechanical. Just pin your semantic-conventions package version, read the attribute names from the package constants rather than typing string literals, and expect to update.

The pieces you will use:

Attribute	Meaning
`gen_ai.operation.name`	`chat`, `execute_tool`, `invoke_agent`, `create_agent`, `embeddings`, `invoke_workflow`
`gen_ai.provider.name`	the provider, e.g. `anthropic`
`gen_ai.request.model` / `gen_ai.response.model`	requested and actual model
`gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`	token counts
`gen_ai.response.finish_reasons`	why generation stopped
`gen_ai.tool.name` / `gen_ai.tool.call.id`	which tool, which call
`gen_ai.tool.call.arguments` / `gen_ai.tool.call.result`	the arguments and result
`gen_ai.agent.name` / `gen_ai.conversation.id`	agent identity and session
`gen_ai.input.messages` / `gen_ai.output.messages`	full content capture, opt-in
`gen_ai.evaluation.name` / `gen_ai.evaluation.score.value`	evaluation results attached to a span

Span naming follows {operation} {model} for inference spans — chat claude-sonnet-4-5 — and the tool-execution operation is execute_tool.

Two notes with teeth.

Content capture — the full messages — is opt-in, controlled by environment variables, and off by default for exactly the privacy reasons above. Turn it on in development, think hard before turning it on in production, and if you do, put your redaction layer in front of it.

That last row is newer and underused: the conventions now include attributes for evaluation results, which means the judge scores from Chapter 2 can be attached to the very span they judged. Your quality metrics and your system metrics end up in one queryable store, and “show me the p99 latency of runs that scored below 4 on groundedness” becomes a single query. That is the join everyone wants and almost nobody builds.

Saying it out loud. If you name your telemetry attributes yourself, your traces are legible only to your own tooling, so use the OpenTelemetry GenAI semantic conventions. The honest caveat is that they’re still marked Development rather than Stable, they live in their own repo now, and names have moved between releases — gen_ai.system became gen_ai.provider.name, prompt_tokens became input_tokens. Use them anyway: a moving convention beats a private vocabulary, every serious backend ingests them, and the migrations are mechanical. Pin the package version and read names from constants rather than typing string literals. The row people miss is the evaluation attributes, which let you hang judge scores on the very span they judged — that’s what makes “show me p99 latency for runs that scored below 4 on groundedness” a single query instead of a data-engineering project.

Where the traces go

You need a backend, and the useful thing to know is that the export format is standardised, so the choice is reversible.

OpenTelemetry Collector plus any OTLP backend — Jaeger, Tempo, or a commercial APM. Maximum control, no LLM-specific UI.

LLM-native platforms, which give you a trajectory-shaped view, prompt playgrounds, dataset management, and judge integrations on top of traces. Langfuse accepts standard OTLP at /api/public/otel with Basic auth and maps the GenAI conventions onto its data model, which means any OTel SDK can feed it (https://langfuse.com/docs/opentelemetry/get-started). Arize Phoenix and LangSmith occupy similar ground. Opik — Apache-2.0, self-hostable with one script, with tracing, datasets, judge metrics, and online evaluation rules in one product — is the one this course uses, and Chapter 5 shows it (https://github.com/comet-ml/opik).

The decision procedure: instrument with the OpenTelemetry API and the GenAI conventions, then choose an exporter. If the platform disappoints, you change a URL, not your codebase. Do not accept an SDK that requires you to write your instrumentation in its private vocabulary.

Saying it out loud. The nice thing about the backend decision is that it’s reversible, because the export format is standardised. So instrument with the OpenTelemetry API and the GenAI conventions, then pick an exporter — a plain OTLP backend like Jaeger or Tempo if you want maximum control and no LLM-specific UI, or an LLM-native platform if you want a trajectory-shaped view, dataset management, and judge integrations on top. If the platform disappoints, you change a URL rather than your codebase. The rule I’d hold to is refusing any SDK that makes you write your instrumentation in its private vocabulary, because that turns a reversible decision into a rewrite.

Pillar 3: metrics — the health report

Metrics are aggregations over time. They are not a separate data source: they are what you get when you count and average the attributes already on your spans.

The split that matters for agents is between two families that page different people.

Saying it out loud. Metrics aren’t a separate data source — they’re what you get when you count and average attributes that are already on your spans, which is why instrumenting for reconstruction gives you the dashboards for free. The split that matters for agents is between system metrics and quality metrics, because they page different people on different timescales. System metrics answer “is it up and what does it cost,” quality metrics answer “is it still good,” and the second set will move for weeks without the first set twitching. If you only build one family, you’ll be the team with the green wall and the wrong answer at 03:14.

System metrics: vital signs

Computed directly from span attributes, cheap, real-time, and owned by whoever carries the pager.

Latency p50 and p99 of the root span. Report both; p50 is the experience you designed and p99 is the one people complain about.
Error rate — traces containing any ERROR span, split by error.type.
Tokens per run, input and output separately, because they price differently.
Cost per run and per successful run. The second number is the honest one: if a third of your runs fail and get retried, your cost per outcome is 50% higher than your cost per run.
Steps per run, with the distribution, not just the mean. A rising tail is the earliest signal of degradation you will get.
Step-cap exhaustion rate. Should be near zero. If it is climbing, something upstream broke.
Tool call frequency and failure rate per tool. Tells you which tools matter and which are flaky.
Repetition rate — runs containing the same tool called twice with identical arguments. A pure waste signal and trivially computable from spans.

Saying it out loud. System metrics come straight off span attributes, so they’re cheap and real-time, and they belong to whoever carries the pager. Latency at p50 and p99 — report both, because p50 is the experience you designed and p99 is the one people complain about. Error rate split by error type. Tokens in and out separately, because they price differently. Cost per successful run rather than cost per run, which is the honest number: if a third of runs fail and get retried, your cost per outcome is 50 percent higher than your cost per attempt. Steps per run as a distribution rather than a mean, because a rising tail is the earliest degradation signal you’ll get. And step-cap exhaustion rate, which should sit near zero — if it’s climbing, something upstream already broke.

Quality metrics: judging the decisions

Second-order, computed by running the judgment machinery from Chapters 2 and 3 over sampled production traces. Slower, costlier, and owned by whoever owns the product.

Task success rate, from a judge, from implicit user behaviour, or from a business event.
Groundedness / hallucination rate — claims unsupported by anything the agent retrieved.
Trajectory adherence — how often the agent followed the intended tool path, which is your production-side version of Chapter 3’s trajectory checks.
Helpfulness, from a judge or from user feedback.
Safety violation rate, which should be zero and alerts at any nonzero value.
Escalation and abandonment rate, both of which are effectiveness proxies you already have.

Two dashboards, not one, because the alerts are different in kind.

P99 latency > 3s for 5 minutes is an operational alert: something is broken now, wake someone. Groundedness score down 10% over 24 hours is a quality alert: the system is healthy and getting worse, and the response is an investigation, not a restart. Putting both on one dashboard guarantees one of the two audiences stops looking at it.

Saying it out loud. Quality metrics are second-order: you run the judging machinery over a sample of production traces, so they’re slower and costlier and they belong to whoever owns the product, not the pager. Task success, groundedness, trajectory adherence, helpfulness, safety violations, escalation and abandonment. The reason I insist on two dashboards rather than one is that the alerts are different in kind. “p99 over three seconds for five minutes” means something is broken right now, wake someone up. “Groundedness down 10 percent over 24 hours” means the system is perfectly healthy and quietly getting worse, and the response is an investigation, not a restart. Put both on one dashboard and one of the two audiences stops looking at it.

From telemetry to decisions

Data you do not act on is a storage bill. Four loops turn it into decisions, and each one is small.

The failure-to-case loop. A run errors, hits the cap, or gets a thumbs-down. Its trace goes into a review queue. A human tags it. The tagged case is added to the eval set from Chapter 3. Now that failure can never silently return. This is the single highest-value loop in this chapter and it is a queue and a foreign key.

The drift watch. Score a sample of production traces with the judge nightly. Plot it. Annotate the chart with prompt, model, and index version changes. Quality drift almost always coincides with a deploy, and the annotated chart is what makes that visible in seconds rather than in a week of bisecting.

The cost hunt. Group spans by tool and by step index and look at where tokens go. The answers are consistently unglamorous: a tool returning an enormous payload that gets carried in context for the rest of the run, a redundant lookup, a summarisation step that could go to a smaller model. Part 3’s context techniques are the fix; this is how you find where to apply them.

The tool-health loop. Failure rate and latency per tool, tracked over time. A tool whose failure rate is climbing degrades the agent long before it breaks it, because the model burns steps recovering. This is where the compounding arithmetic from Chapter 1 shows up in your invoice.

The connecting insight: evaluation and observability are the same system viewed from two ends. Observability captures trajectories; evaluation judges them. Traces feed eval cases; eval scores annotate traces. Build them apart and you will spend a quarter joining them; build them together and you have a flywheel where every production failure permanently improves the suite.

Chapter 5 is where you wire it.

Saying it out loud. Data you don’t act on is just a storage bill, so there are four small loops that turn it into decisions. The failure-to-case loop is the big one: a run errors or gets a thumbs-down, its trace goes to a review queue, a human tags it, and the tagged case joins the eval set so that failure can never silently come back — that’s a queue and a foreign key, not a project. Then the drift watch, where you judge a nightly sample and annotate the chart with prompt and model versions, because quality drift almost always coincides with a deploy. The cost hunt, grouping spans by tool to find the enormous payload being carried in context for the rest of the run. And tool health, because a tool whose failure rate is creeping up degrades the agent long before it breaks it — the model just burns steps recovering. The connecting idea is that evaluation and observability are one system seen from two ends: traces feed eval cases, eval scores annotate traces. Build them apart and you’ll spend a quarter joining them.

What you should be able to do now

State the difference between monitoring and observability in terms of anticipated versus unanticipated questions, and explain why agent failures are invisible to monitoring by construction.
Specify what to log per run, per step, and per tool call; apply a redaction and truncation layer; and set a sampling policy that keeps 100% of failures and a small fraction of successes.
Design a span hierarchy for an agent run — root, model calls, tool calls, sub-agents — and list the attributes that make those spans queryable, including soft-failure results that do not set an error status.
Use the OpenTelemetry GenAI semantic conventions with an accurate view of their stability, reading names from package constants rather than string literals, and keep content capture opt-in.
Separate system metrics from quality metrics, assign each to an owner and a dashboard, and write an alert for each kind that would actually fire on a real degradation.
Build the failure-to-case loop: trace ID on every feedback event, failures into a review queue, reviewed failures into the eval set from Chapter 3.

Production Agent Engineering — A Practical Guide