Operating an agent in production

The agent is live. The interesting part now is that nobody is watching it.

At 3 a.m. on a Sunday it is having several hundred conversations you will never read, choosing tools you did not anticipate, in orders you did not design, spending money nobody approved. That is not a bug — it is the property you shipped it for. It is also why operating an agent is a different job from operating a service.

A traditional service does what it was told. An agent does what it decided. Operating it means running a loop: observe what it is doing, act to keep it healthy and safe right now, and evolve it so today’s problem stops being a problem.

Observe and act are reflexes, measured in seconds and minutes. Evolve is strategy, measured in days. Teams that only do the first two run a permanent, slowly worsening incident.

Observe

Three pillars, and the middle one is the one people under-build.

Logs are the factual diary: every tool call with its arguments, every error, every decision point, every step boundary.

Traces are the narrative connecting them: one request ID threading the model calls, tool executions, and sub-agent handoffs into a causal path with durations attached. For an agent this is not a nice-to-have. There is no breakpoint you can set inside a model’s reasoning; the trace is your debugger, and you cannot retrofit it during an incident.

Metrics are the aggregate report card: rates, percentiles, distributions, cost.

Part 1 told you to emit a trace from day one. Here is what it needs to contain to be operationally useful, as opposed to merely present:

Field	Why you will want it at 3 a.m.
`run_id`, `session_id`, `user_id` (hashed)	Reconstruct one conversation, or all of one user’s
`agent_version`, `prompt_sha`, `model_id`	Which build did this — the manifest from Chapter 2
`flags` (resolved)	Reproduce the configuration that produced the failure
Per step: tool name, arguments, latency, outcome	The trajectory
Per step: input/output/cache tokens, cost	Attribute spend to a step, not just a run
Terminal reason	`answered`, `step_cap`, `budget`, `timeout`, `error`, `escalated`
Authorization decisions	Every allow, deny, and confirm from Chapter 4

Instrument with OpenTelemetry rather than a bespoke format. The GenAI semantic conventions give you agreed attribute names for model calls, token counts, and tool executions, which means your traces are readable by tools you have not chosen yet.

The metrics worth a dashboard, as opposed to the ones worth a query:

Task success rate, from a sampled judge or an explicit outcome signal. Steps per successful task — the thrash detector. Tool call success rate, per tool. Terminal reason distribution — a rising step_cap share means the agent is failing to finish, and it moves before user complaints do. Cost per successful task, which gets its own section below. p50 and p95 end-to-end latency, split by number of steps, because a p95 dominated by 12-step runs is a different problem from one dominated by a slow tool. Escalation and refusal rates. Authorization denials, per user and per tool — flat and boring normally, and spiky during an attack.

Two alerting rules that will save you real money.

Alert on rate of change, not just thresholds. “Cost per hour is 3x this time yesterday” catches a runaway at 3 a.m.; a static threshold set high enough not to be noisy catches it at 3 p.m. after it has run all night.

Alert on distribution shifts, not just averages. A tool that goes from 2% to 20% of all calls is a behaviour change, and it is what both a bad deploy and a successful exploit look like from the outside.

The sibling agentic-ai-evaluation-guide covers observability and evaluation instrumentation in depth. This chapter assumes the telemetry exists and is about what you do with it.

Saying it out loud. Three pillars — logs, traces, metrics — and traces are the one people under-build. There’s no breakpoint you can set inside a model’s reasoning, so the trace is your debugger, and you cannot retrofit it during an incident. What makes a trace operationally useful rather than merely present is a specific field list: the run and session IDs, the agent version and prompt hash and model ID, the resolved feature flags so you can reproduce the configuration, per-step tool and token and cost, the terminal reason, and every authorization decision. Two alerting rules save real money. Alert on rate of change, not just thresholds — “cost per hour is three times this time yesterday” catches a runaway at 3 a.m., while a static threshold set high enough not to be noisy catches it at 3 p.m. after it ran all night. And alert on distribution shifts, because a tool going from 2 percent to 20 percent of calls is what both a bad deploy and a successful exploit look like from outside.

Act: the levers

Observation without action is an expensive dashboard. Here are the levers, and the honest test of your operational maturity is how many of them you can pull without a deploy.

Traffic. Shift percentages between versions, or roll back. Chapter 3. Feature flags. Disable a tool, switch a prompt variant, change the confirmation threshold, tighten max_steps. Chapter 3. Rate limits and quotas. Per user, per tenant, per tool. Your first response to abuse and to a runaway loop. Model routing. Move a class of traffic to a cheaper or faster model. Budgets. Hard caps per run, per user per day, per tenant per month. Circuit breakers. Trip a failing dependency out of the tool belt so the agent degrades instead of hanging. Cache policy. Turn caching on or up when cost or latency spikes. Human queue depth. Lower the confirmation threshold so more actions route to people, when you have lost confidence and not yet lost the ability to serve.

That last one is underrated. “Route everything over $50 to a human” is a dial, and turning it down is a graceful way to keep operating during an investigation.

Saying it out loud. Observation without action is an expensive dashboard, and the honest test of operational maturity is how many levers you can pull without a deploy. Traffic shifting and rollback. Feature flags to kill a tool or swap a prompt variant. Rate limits and quotas per user, tenant, and tool. Model routing. Hard budgets. Circuit breakers on failing dependencies so the agent degrades instead of hanging. Cache policy. And the underrated one — human queue depth. “Route everything over 50 dollars to a human” is a dial, not a switch, and turning it down is a graceful way to keep operating while you investigate, instead of the binary choice between running blind and turning the product off.

Managing system health

Scale

The foundation is one architectural decision: the agent process holds no state.

Session, memory, and task state live outside the process — Redis, Postgres, a managed session store, whatever you already run. Then any instance can serve any request, autoscaling works, a deploy does not lose conversations, and a crash costs one request rather than one user’s afternoon. You built the store in Part 3; this is why.

Long-running work goes asynchronous. An agent task that takes four minutes should not hold an HTTP connection open. Accept the request, return a task ID, do the work on a queue, and let the client poll or receive a webhook. This also happens to be the shape A2A standardises, which is Chapter 6.

Concurrency limits belong per-instance and per-dependency. Your agent can hold a lot of in-flight model calls; the flaky vendor API behind one of its tools cannot, and an agent that retries enthusiastically is an excellent denial-of-service tool against your own suppliers.

Retries need two properties or they make things worse. Exponential backoff with jitter, so a shared outage does not produce a synchronised retry storm. And idempotency keys on every action tool, so a retry after an ambiguous timeout does not send the second email.

The container, autoscaling, and traffic mechanics under all of this are the serving guide’s territory. What is agent-specific is the statelessness requirement and the idempotency requirement, and both are design decisions you make long before you deploy.

Saying it out loud. The foundation is one architectural decision: the agent process holds no state. Session, memory, and task state live outside it, so any instance serves any request, autoscaling works, a deploy doesn’t lose conversations, and a crash costs one request rather than somebody’s afternoon. Long work goes asynchronous — a four-minute task shouldn’t hold an HTTP connection open, so you return a task ID and let the client poll or take a webhook. Concurrency limits go per-instance and per-dependency, because your agent can hold plenty of in-flight model calls but the flaky vendor API behind one tool cannot, and an agent that retries enthusiastically is an excellent denial-of-service tool against your own suppliers. And retries need backoff with jitter plus idempotency keys on every action tool, or a retry after an ambiguous timeout sends the second email.

Latency

Agent latency is dominated by step count, not by any single call. Six sequential model calls at 1.8 seconds each is eleven seconds, and no amount of infrastructure tuning fixes that.

The interventions that actually move it, in order of effect:

Reduce steps. Better tool descriptions, so it picks right the first time. Tools that return what is needed in one call instead of three. Pre-fetching the obvious context before the loop starts. Parallelise independent tool calls. If the model requests three lookups in one turn, execute them concurrently. This is a change in your orchestration layer, and it is often the single biggest win. Stream. Time-to-first-token is what users perceive. An agent that narrates “checking your order…” feels twice as fast as one that goes silent for eleven seconds. Route the cheap steps to a fast model. Classification, summarisation of a tool result, deciding relevance — a small model does these in a third the time. Cache the prompt prefix. System prompt plus tool schemas is a large, unchanging block re-sent on every step.

Saying it out loud. Agent latency is dominated by step count, not by any one call — six sequential model calls at 1.8 seconds each is eleven seconds, and no amount of infrastructure tuning fixes that. So the interventions that move it are about steps. Reduce them with better tool descriptions and tools that return what’s needed in one call instead of three. Parallelise independent tool calls when the model requests several in a turn, which is an orchestration change and often the single biggest win. Stream, because time-to-first-token is what users actually perceive and an agent that says “checking your order” feels twice as fast as one that goes silent for eleven seconds. Then route cheap steps to a fast model and cache the prompt prefix.

Cost

This is where agents surprise people, and the surprise has a specific cause.

An agent re-sends its accumulated context on every step. So a run of $ n $ steps does not cost $ n $ times one call — it costs roughly the sum of a growing context, which is quadratic in the number of steps. A trajectory that wanders to twelve steps instead of four costs far more than three times as much.

Which is why the metric to run your business on is not cost per call and not cost per run. It is cost per successful task:

\[ C_{\text{success}} = \frac{C_{\text{attempt}}}{s} \]

where $ C_{\text{attempt}} $ is the mean cost of an attempt and $ s $ is the task success rate.

Put numbers in it, because the implication is counterintuitive. At $0.031 per attempt and a 72% success rate, each successful task costs $0.043. Now suppose a bigger, more expensive model raises the attempt cost 40% to $0.043 but lifts success to 91%. Cost per success: $0.047. Barely worse, and if a failed task costs you a human support contact at several dollars, the expensive model is dramatically cheaper overall.

A cheaper model that fails more often is frequently more expensive. You cannot see that with a cost-per-call dashboard, and this is the number one reason cost optimisation programs make agents worse.

Now the levers, roughly in order of return on effort.

Prompt caching. Your system prompt and tool schemas are a large static prefix re-sent every step. Caching that prefix is close to free money: on Anthropic’s models a cache read costs 0.1x the base input rate against a 1.25x write for the five-minute TTL, so a prefix read even twice is already ahead. Order your context static-first so the cacheable prefix is as long as possible.

Model routing. Not every step needs your best model. Route by step type — planning and final synthesis to the strong model, classification and observation-summarising to a small one.

def choose_model(step: str, complexity: float, flags) -> str:
    if step in ("classify", "summarize_observation", "extract"):
        return flags.small_model            # ~10x cheaper, ~3x faster
    if step == "plan" and complexity > 0.7:
        return flags.strong_model
    return flags.default_model

Measure this with the eval suite before shipping it, because routing is exactly the kind of change that looks free on the cost dashboard and costs you four points of success rate.

Context discipline. Summarise large tool observations instead of carrying them raw. Externalise big artifacts and carry references. Trim aggressively, pin the mission. This is Part 3 applied to your bill, and on long trajectories it is worth more than model routing.

Step budgets. A hard cap is a cost control as much as a safety control. Track the distribution of steps per run; the tail is where your money goes.

Semantic caching of whole answers. For agents with repetitive traffic — internal helpdesks especially — caching by normalised question can eliminate a real fraction of calls. Be careful: cache keys must include the principal and any personalised context, or you have built a data leak with excellent latency.

Batching. Where latency does not matter — nightly evals, bulk classification, backfills — batch APIs are typically half price.

Budgets to enforce in code, not in a dashboard:

@dataclass
class Budget:
    max_usd_per_run: float = 0.50
    max_usd_per_user_day: float = 5.00
    max_usd_per_tenant_month: float = 2000.00

Per-run is your runaway-loop protection. Per-user-day is your abuse protection. Per-tenant-month is your “we cannot lose money on this customer” protection. All three should degrade gracefully — switch to a cheaper model, then refuse politely, then escalate to a human — rather than returning a 500.

Saying it out loud. The metric to run the business on is cost per successful task, not cost per call and not cost per run — attempt cost divided by success rate. Put numbers on it, because the implication is counterintuitive. At 3.1 cents per attempt and 72 percent success, each success costs 4.3 cents. Move to a model that’s 40 percent more expensive per attempt but lifts success to 91 percent, and cost per success is 4.7 cents — barely worse, and if a failed task means a human support contact costing several dollars, the expensive model is dramatically cheaper overall. So a cheaper model that fails more often is frequently more expensive, and you cannot see that on a cost-per-call dashboard. That’s the number one reason cost optimisation programs make agents worse. The lever order is prompt caching first, since a cached prefix reads at a tenth of the input rate against a 1.25x write and pays for itself on the second read; then model routing, measured on the eval suite because it’s exactly the change that looks free on the cost dashboard and quietly costs four points of success rate; then context discipline, step budgets, semantic caching, and batching.

Managing risk in production

Chapter 4 built the defenses and the playbook. Operationally, what you are doing day to day is watching for three signatures:

A shift in the tool-call distribution. The most reliable early indicator of both a bad deploy and an active exploit. A spike in authorization denials, especially concentrated on one principal. That is someone probing. Anomalous cost or step counts for a single user. Either abuse, or a loop, and both need the same first response.

Keep the read-only circuit breaker one flag flip away, and rehearse it. A containment mechanism nobody has used is a containment mechanism that does not work; run a game day where you disable write tools in production on purpose and confirm the agent degrades the way you think it does.

Saying it out loud. Day to day, risk management is watching for three signatures. A shift in the tool-call distribution, which is the most reliable early indicator of both a bad deploy and an active exploit. A spike in authorization denials concentrated on one principal, which is somebody probing. And anomalous cost or step counts for a single user, which is either abuse or a loop and needs the same first response either way. And keep the read-only circuit breaker one flag flip away and rehearse it — a containment mechanism nobody has ever used is a containment mechanism that doesn’t work, so run a game day where you disable write tools in production on purpose and confirm the agent degrades the way you think it does.

Evolve

Observe and act keep the system standing. Evolve is what makes it better, and it is the phase that separates a product from a maintained demo.

The question that starts it is not “what broke.” It is “how do we make this class of problem stop happening.”

Saying it out loud. Observe and act keep the system standing; evolve is what makes it better, and it’s what separates a product from a maintained demo. The question isn’t “what broke,” it’s “how do we make this class of problem stop happening.” And the punchline is about velocity: the classic example is a retail agent where 15 percent of users hit an error on one request type — logs surface it, the failure becomes a test case, an engineer adds a better tool, and the fix is live in 48 hours. Same insight in an organisation where deploying takes three weeks of manual validation, and you improve ten times slower. So the CI/CD pipeline isn’t a deployment convenience, it’s the engine of evolution, and its cycle time is the speed limit on how fast your agent gets better.

The workflow

Three steps, and the middle one is the one that compounds.

1. Analyse production data. Not by reading random traces — by clustering the failures.

The mechanical version: take every run whose terminal reason was not answered, plus a sample of answered runs the judge scored low, embed the initial request, cluster, and label the clusters.

You are looking for:

Requests that consistently fail — a capability gap.
Requests that succeed but take twice the normal steps — a tool design problem.
Tools with elevated error rates — an integration problem.
Repeated escalations on the same topic — a missing tool or a policy gap.
Requests the agent handles that it was never designed for — usually your best product signal.

A weekly hour on this is one of the highest-return hours available to an engineering team.

2. Turn failures into eval cases.

This is the compounding step.

A production failure that produces only a fix produces a fix. A production failure that produces a test case produces a fix and permanent protection against its return — and your eval set grows toward the real input distribution instead of staying frozen at whatever you imagined before launch.

Make it mechanical:

def failure_to_eval_case(run: dict, expected: str, reviewer: str) -> dict:
    """Promote a production failure into a golden-dataset case."""
    return {
        "case_id": f"prod-{run['run_id'][:8]}",
        "source": "production",
        "captured_at": run["started_at"],
        "input": redact(run["input"]),               # PII out before it hits the repo
        "context": {k: run["context"][k] for k in ("user_tier", "locale")},
        "expected": expected,                        # a human wrote this
        "expected_tools": run.get("should_have_called", []),
        "tags": ["regression", run["failure_cluster"]],
        "added_by": reviewer,
    }

Three details in there matter.

Redaction is not optional — production inputs contain personal data and your eval set lives in a git repository forever. The expectation is written by a human; if you let a model label its own failures you have built a machine that agrees with itself. And the tag records the cluster, so six months from now you can ask which failure families you have actually fixed.

The governance rule that makes this stick, from Chapter 1: no production failure closes without a case.

3. Refine and deploy.

Now the pipeline from Chapter 2 earns its keep. Commit the improvement — a prompt refinement, a new tool, a tightened policy, a better tool description — and it runs through the full gate, including the new case, and rolls out through the schedule in Chapter 3.

The velocity here is the whole point. The whitepaper’s example is a retail agent where 15% of users hit an error on a particular request type: the logs surface it, the failure becomes a test case, an engineer refines the prompt and adds a better tool, and the fix is live inside 48 hours.

Compare that to the same insight in an organisation where deploying takes three weeks of manual validation. Same insight, same engineer, and one of those organisations improves 10x faster.

The CI/CD pipeline is not a deployment convenience. It is the engine of evolution, and its cycle time is the speed limit on how fast your agent gets better.

Saying it out loud. The evolution loop is three steps and the middle one compounds. First, analyse production data by clustering failures rather than reading random traces — take every run that didn’t end in “answered” plus the low-scored ones, embed the request, cluster, label. You’re looking for requests that consistently fail, requests that succeed at twice the normal step count, tools with elevated error rates, repeated escalations on one topic, and requests the agent handles that it was never designed for, which is usually your best product signal. Second, turn failures into eval cases, which is the step that compounds: a failure that produces only a fix produces a fix, and a failure that produces a test case produces permanent protection plus an eval set that drifts toward the real input distribution. Redact before it lands in the repo, and have a human write the expectation — let a model label its own failures and you’ve built a machine that agrees with itself. Third, ship it through the gate.

What evolves

Not just prompts.

Tool descriptions, when the trace shows the wrong tool being chosen. Usually the cheapest fix available. Tool granularity, when three calls are consistently made together — merge them. New tools, when escalations cluster on something the agent simply cannot do. The context strategy, when quality degrades with conversation length. Model routing, when the traces show a cheap model handling a class of step perfectly well. Guardrails and policy, when denials show a legitimate pattern being blocked, or an illegitimate one getting through. The eval set itself, which is a living artifact, not a launch deliverable.

Saying it out loud. It isn’t just prompts, and that’s the point worth making. Tool descriptions evolve when the trace shows the wrong tool being chosen, and that’s usually the cheapest fix available. Tool granularity evolves when three calls are always made together — merge them. New tools appear when escalations cluster on something the agent simply can’t do. The context strategy evolves when quality degrades with conversation length. Model routing evolves when traces show a cheap model handling a class of step perfectly well. Guardrails evolve when denials show a legitimate pattern being blocked, or an illegitimate one getting through. And the eval set evolves, because it’s a living artifact rather than a launch deliverable.

Security evolves the same way

The loop is identical, and the whitepaper is right to call it out separately because teams treat security as a fixed checklist.

Observe: monitoring catches a novel injection that got past your filters. Act: contain it with the circuit breaker. Evolve: the attack becomes a permanent adversarial eval case, the guardrail is refined, the change goes through the pipeline and validates against the expanded suite.

The result is a posture that gets stronger with every attack rather than a checklist that ages.

Saying it out loud. Security runs the identical loop, and it’s worth saying separately because teams treat it as a fixed checklist. Observe: monitoring catches a novel injection that got past the filters. Act: contain it with the circuit breaker in seconds. Evolve: the attack becomes a permanent adversarial eval case, the guardrail is refined, and the change goes through the pipeline validating against the expanded suite. The result is a posture that gets stronger with every attack rather than a checklist that ages badly. And it’s the same argument as everywhere else in this part — the thing that makes it work is cycle time, not cleverness.

The whole lifecycle, in one paragraph

Worth being able to recite, because it is the argument of this entire part.

An engineer works in a fast local loop with a scripted model and no infrastructure. A change enters the pipeline, where cheap checks run first and an evaluation gate compares it against the production baseline on a versioned golden dataset. One artifact is built, validated in staging, approved by a human, and promoted unchanged. It reaches users through shadow mode, then a canary with quality gates, then a staged rollout with written kill criteria and a rollback that takes seconds. In production, observability captures every trajectory, operational levers keep cost and risk in bounds without a deploy, and every failure becomes a new eval case that feeds the next turn of the loop.

That cycle is AgentOps. The pieces are not individually clever. Having all of them, and a short cycle time around the loop, is the difference between an agent you demo and an agent your business runs on.

Saying it out loud. If I had to recite the whole thing: an engineer works in a fast local loop with a scripted model and no infrastructure. A change enters the pipeline, cheap checks run first, and an evaluation gate compares it against the production baseline on a versioned golden dataset. One artifact gets built, validated in staging, approved by a human, and promoted unchanged. It reaches users through shadow mode, then a canary with quality gates, then a staged rollout with written kill criteria and a rollback measured in seconds. In production, observability captures every trajectory, operational levers keep cost and risk in bounds without a deploy, and every failure becomes an eval case feeding the next turn. None of those pieces is individually clever. Having all of them, with a short cycle time around the loop, is the difference between an agent you demo and an agent your business runs on.

What you should be able to do now

Specify the trace fields an agent needs to be debuggable at 3 a.m., including version, resolved flags, per-step cost, terminal reason, and authorization decisions.
Choose alerting rules based on rate of change and distribution shift rather than static thresholds, and explain what each one catches that the other misses.
List the operational levers you can pull without a deploy, and audit your own system for which ones you are missing.
Compute cost per successful task, and use it to show that a cheaper model with a lower success rate can cost more overall.
Apply the cost levers in order — prompt caching, model routing, context discipline, step budgets, semantic caching, batching — and name the specific risk each one carries.
Enforce per-run, per-user-day, and per-tenant-month budgets in code with graceful degradation rather than errors.
Run the evolution workflow: cluster production failures, promote them into redacted eval cases with human-written expectations, and ship the fix through the gate.
Explain why the CI/CD pipeline’s cycle time is the speed limit on how fast your agent improves.

Production Agent Engineering — A Practical Guide