Shipping without breaking things

Here is the incident that teaches this chapter, and it has happened to a lot of teams.

You ship a new agent version. Latency is fine. Error rate is zero. CPU is flat, memory is flat, the dashboards are green, and nobody pages anybody.

Four days later support notices that refund requests have doubled. The new prompt made the agent slightly more agreeable, and it has been telling customers they qualify for refunds they do not qualify for. Every one of those requests returned HTTP 200 in 900 milliseconds.

Your infrastructure metrics cannot see this failure. That is the central fact about rolling out agents, and every strategy below exists to work around it.

A traditional deploy is safe when the service is up. An agent deploy is safe when the service is up and still good at its job, and “good at its job” is a measurement you have to build deliberately, because nothing in your existing stack produces it.

The mechanics of traffic splitting — load balancers, Kubernetes rollouts, service meshes, canary infrastructure — are covered properly in the sibling llm-serving-inference-guide. This chapter is about what is specific to agents: what to compare, what to watch, and when to stop.

Shadow mode: run it without letting it act

Before a single user sees the new version, you can run it against real traffic.

Shadow mode sends a copy of each production request to the candidate version, discards its response, and logs it. The user gets the old version’s answer. The candidate never touches a customer.

This is the highest-value and most underused technique available to you, because it gets you real input distribution — the actual weird things people type — with zero blast radius.

The agent-specific wrinkle is the one that trips everyone up: your agent has action tools. A shadow run that calls issue_refund has issued a refund, and “we discarded the response” is no comfort.

So shadowing an agent requires a mode where action tools are stubbed:

class ShadowToolRegistry:
    """Wraps the real registry. Reads pass through; writes are recorded, not executed."""

    def __init__(self, real, effects: dict[str, str]) -> None:
        self.real, self.effects = real, effects
        self.would_have: list[dict] = []

    def call(self, name: str, args: dict) -> str:
        if self.effects.get(name) == "write":
            self.would_have.append({"tool": name, "args": args})
            return f"OK (shadow: {name} recorded, not executed)"
        return self.real.call(name, args)

Note what the stub returns. It returns a plausible success, because if it returns an error the agent will react to the error and the shadow trajectory stops resembling what the real one would have been. You are simulating, and the simulation has to be convincing to the model.

The would_have list is the interesting output. “The candidate would have issued 34 refunds where production issued 19” is a finding you got for free, before anyone was affected.

What to compare between shadow and production, per request:

Final answer agreement — an LLM judge scoring whether the two responses say the same thing, with disagreements sampled for human review.
Trajectory divergence — did they call the same tools, in roughly the same order, the same number of times.
Action divergence — the would_have list against what production actually did. This is the one that catches the refund story.
Cost and latency — real numbers on real traffic, which your eval set only approximates.

Shadow mode’s limitation is worth stating plainly. It cannot tell you whether users like the new answers, because no user saw them. It tells you what changed, not whether the change is good. For that you need real traffic, which is the next section.

Saying it out loud. Shadow mode is the most underused technique available to you: mirror a copy of every production request to the new version, throw its answer away, log everything. The user gets the old version’s response, the candidate never touches a customer, and you get the real input distribution — the actual weird things people type — with zero blast radius. The agent-specific wrinkle is that your agent has action tools, so a shadow run that calls issue_refund has issued a refund, and “we discarded the response” is no comfort. So writes get stubbed, and the stub has to return a plausible success rather than an error, otherwise the model reacts to the error and the shadow trajectory stops resembling the real one. The output you want is the would-have-done log: “the candidate would have issued 34 refunds where production issued 19” is a finding you got for free. The limit is that nobody saw those answers, so shadow tells you what changed, never whether the change is good.

Canary with a quality gate

A canary sends a small percentage of real traffic to the new version and watches. Standard practice, and every deployment platform gives it to you.

The standard gate is where agents differ. Infrastructure canary analysis watches error rate, latency percentiles, and saturation. Keep all of that — and understand it will not fire for the failure mode you actually fear.

Add these, and treat them as first-class:

Tool call error rate. Not HTTP errors — tool-level failures, including the ones your registry catches and turns into observations the model reads. A version whose tool errors doubled is a version that is confused about arguments.

Steps per successful task. If the candidate takes 6.2 steps where the baseline took 4.1, it is thrashing. It may still produce right answers, at 50% more cost and latency, and this metric sees it before your bill does.

Tool selection distribution. The histogram of which tools get called. A shifted distribution is a behaviour change, whether or not you intended one. This is also your injection detector — a sudden spike in an unusual tool is what an exploit looks like from the outside.

Human-escalation rate. How often the agent hands off, refuses, or ends without resolving. Cheap to compute, and it moves early when quality drops.

Automated quality score on a sample. Take 2–5% of canary conversations, run an LLM judge over them against a rubric, and track the score. This is the only metric on the list that directly measures the thing you care about, and it costs pennies at that sample rate.

A business proxy, if you have one. Refund rate. Resolution rate. Repeat-contact rate within 24 hours. Conversion. Slower to move — often days — but this is what actually matters, and it is the metric the refund incident would have shown up in.

The whole point of this list: stack the fast, noisy proxies in front of the slow, true signal. Tool error rate moves in minutes. The judge score moves in hours. The business metric moves in days. Your rollout schedule should be paced by which signals have had time to speak.

Saying it out loud. A canary is easy — small percentage of real traffic to the new version. What’s different for agents is the gate. Keep error rate and latency, and understand they will not fire for the failure you actually fear. So you add tool call error rate, steps per successful task, the tool selection histogram, escalation rate, a judge score on two to five percent of conversations, and a business proxy like refund rate or repeat contacts. The organising idea is to stack the fast noisy proxies in front of the slow true signal: tool error rate moves in minutes, the judge score in hours, the business metric in days. And your rollout schedule should be paced by which signals have had time to speak — if you complete the rollout in six hours and your business proxy takes two days, you never measured it.

Comparing fairly

Two errors will corrupt your canary analysis if you let them.

Do not compare canary against yesterday. Compare canary against the baseline serving concurrently. Traffic on Tuesday morning is not traffic on Saturday night, and a 3-point difference between time periods tells you about the periods.

Do not compare unmatched populations. If your canary routes by hash of user ID, you are fine. If it routes by “whoever hits the new region,” you have confounded the version with the geography. Pin the assignment to something stable and orthogonal to the thing you are measuring.

Both of these are just A/B testing hygiene, and they are worth restating because agent teams tend to arrive from an ML background where the experiment design is someone else’s job.

Saying it out loud. Two mistakes will quietly corrupt your canary analysis. Don’t compare the canary against yesterday — compare it against the baseline serving concurrently, because traffic on Tuesday morning isn’t traffic on Saturday night and a three-point gap between time periods is telling you about the periods. And don’t compare unmatched populations: if you route by hash of user ID you’re fine, but if you route by “whoever hits the new region” you’ve confounded version with geography. It’s ordinary A/B hygiene, and it’s worth restating because agent teams often arrive from a background where experiment design was somebody else’s job.

Staged rollout: the schedule

A canary that stays at 5% forever is not a rollout. The schedule is the plan for expanding it, and it should be written down before you start, because the pressure to skip a stage arrives exactly when you are tired.

A shape that works for a customer-facing agent:

Stage	Traffic	Hold	What has to be true to advance
Shadow	0% (mirrored)	1–3 days	Action divergence explained, no unexplained trajectory changes
Internal	Employees only	1–2 days	No qualitative complaints, smoke tests green
Canary	1–5%	2–24 h	Tool error rate and steps-per-task flat, judge score within band
Expand	25%	24 h	Judge score flat, escalation rate flat, cost per task acceptable
Majority	50%	24–48 h	Business proxy has moved into view and has not degraded
Full	100%	—	Promote baseline, archive the old revision

Two rules about the schedule.

Hold long enough for the slow signal. If your business proxy takes 48 hours to move, a rollout that completes in six hours never measured it. That is not a rollout, it is a deploy with extra steps.

Never advance during a window when nobody is watching. The 100% step at 17:00 on a Friday is a genre of incident with its own folklore.

For a high-traffic consumer product these holds compress; for a low-traffic internal agent they stretch, because at 200 requests a day a 5% canary produces 10 samples and you cannot conclude anything from 10 samples. Which brings up the honest limitation: if your traffic is low, canary analysis is statistically hopeless, and shadow mode plus a strong eval suite is a better use of your time.

Saying it out loud. A canary parked at five percent forever isn’t a rollout — the schedule is the plan for expanding it, and you write it down before you start, because the pressure to skip a stage arrives exactly when you’re tired. Two rules. Hold long enough for the slowest signal you actually rely on, or you’ve done a deploy with extra steps. And never advance during a window when nobody’s watching; the hundred-percent step at 5pm on a Friday is a genre of incident with its own folklore. The honest limitation is volume: at 200 requests a day, a five percent canary gives you ten samples, and you can’t conclude anything from ten samples. If your traffic is low, canary analysis is statistically hopeless and shadow mode plus a strong eval suite is a much better use of your time.

Blue-green, and why agents complicate it

Blue-green runs two full environments and flips traffic between them. Instant cutover, instant rollback, no partial state.

It works for agents with one large caveat: sessions.

Your agent is stateful. A user is three turns into a conversation when you flip, and now their next turn is handled by a different version with a different prompt and possibly a different memory schema. At best the tone changes mid-conversation. At worst the new version cannot deserialize the state the old one wrote.

Three mitigations, in order of preference:

Version your session state and make new readers tolerant of old writes. This is just schema evolution, and it is the durable fix.

Pin a session to a version for its lifetime. Route on session ID, not request. Costs you a slower rollout, because long sessions keep the old version alive.

Drain. Stop assigning new sessions to blue, let existing ones finish, then flip. Fine when sessions are minutes; useless when they are days.

If your agent runs long-horizon tasks — hours or days, spanning deploys — this stops being a rollout question and becomes an architecture question. The sibling agentic-ai-evaluation-guide has a long-horizon-operations track that covers it properly.

Saying it out loud. Blue-green is two full environments with an instant flip, and it works for agents with one big caveat: sessions. Your agent is stateful, so a user three turns into a conversation gets their fourth turn handled by a different version with a different prompt — at best the tone changes mid-conversation, at worst the new version can’t deserialize the state the old one wrote. Three fixes in order of preference: version the session state and make new readers tolerant of old writes, which is just schema evolution and the durable answer; pin a session to a version for its lifetime, which costs you a slower rollout because long sessions keep the old version alive; or drain, which is fine when sessions are minutes and useless when they’re days. If your tasks run for hours or days across deploys, this stops being a rollout question and becomes an architecture question.

Feature flags: the finest-grained control you have

Traffic percentages are a blunt instrument. Flags let you ship code that is dark and turn on specific behaviour for specific cohorts.

For agents the useful granularity is not “new version on/off.” It is per-capability:

@dataclass(frozen=True)
class AgentFlags:
    prompt_variant: str = "v7"          # which system prompt
    enable_refund_tool: bool = False    # dark-launch a new action tool
    max_steps: int = 6                  # tune the budget without a deploy
    model: str = "claude-sonnet-4-5-20250929"
    require_confirm_over: float = 100.0 # human gate threshold
    judge_sample_rate: float = 0.02


def flags_for(user_id: str, service) -> AgentFlags:
    """Resolved once per request, logged with the trace, never cached across requests."""
    return service.evaluate(AgentFlags, subject=user_id)

Three things this buys you that a traffic split does not.

A new tool can be dark-launched. Register issue_refund, leave it off for everyone, enable it for your own account, then for support staff, then for 1% of customers. The riskiest part of an agent change is usually a new action tool, and this is the only mechanism that lets you roll that out independently of everything else.

Budgets become tunable without a deploy. When cost spikes at 2 a.m., dropping max_steps from 8 to 5 is a config change, not a release.

The circuit breaker exists before you need it. enable_refund_tool = false pushed globally disables one capability in seconds while leaving the rest of the agent working. Chapter 4 calls this the first move in the security playbook, and it only exists if you built the flag in advance.

Two disciplines keep flags from becoming their own outage. Log the resolved flag set with every trace — otherwise you cannot reproduce a failure, because you do not know what configuration produced it. And put an expiry on every flag; a flag that has been at 100% for four months is dead code with a runtime lookup attached.

Saying it out loud. Traffic percentages are blunt. Flags let you ship code dark and switch a specific capability on for a specific cohort, and for agents the useful granularity isn’t “new version on or off,” it’s per-capability. Three things that buys you. You can dark-launch a single action tool — register issue_refund, enable it for your own account, then support staff, then one percent of customers — and since a new action tool is usually the riskiest part of the change, that’s the only mechanism that rolls it out independently. Budgets become tunable without a deploy, so cost spiking at 2 a.m. means dropping max_steps from eight to five, not cutting a release. And the circuit breaker exists before you need it: flipping one capability off globally in seconds while the rest keeps working. Two disciplines keep flags from becoming their own outage — log the resolved flag set with every trace, or you can’t reproduce a failure because you don’t know what config produced it, and put an expiry on every flag, because one that’s been at a hundred percent for four months is dead code with a runtime lookup attached.

Rollback

The measure of a rollback is not whether it exists. It is how long it takes and whether you have done it.

Under sixty seconds, one command, no build step. If rollback requires re-running CI, it is not a rollback, it is a redeploy, and it will take twenty minutes you do not have.

On a platform with revision-based traffic control this is a single call:

# roll back to a known-good revision, immediately
gcloud run services update-traffic support-agent \
  --region europe-west1 --to-revisions support-agent-00042-abc=100

The equivalent exists everywhere: a previous Kubernetes ReplicaSet, a previous task definition, a previous Lambda alias. The point is that the old version is still there, warm, and one command away.

Three things that quietly break rollback for agents:

Prompts stored outside the artifact. You roll back the container and the prompt stays new, because it lives in a database someone edits through a UI. Now you are running an untested combination. Keep prompts in the image.

Migrated state. The new version wrote session or memory records in a schema the old version cannot read. Rolling back the code does not roll back the data. Make schema changes additive and deploy them a release ahead of the code that needs them — the standard expand/contract discipline, which applies unchanged here.

Flags that outlived the rollback. You roll back the code, and the flag enabling the new tool is still on. Include flag state in your rollback runbook.

Saying it out loud. The measure of a rollback isn’t whether it exists, it’s how long it takes and whether you’ve actually done it. The bar is under sixty seconds, one command, no build step — if it requires re-running CI it’s a redeploy, and it’ll take twenty minutes you don’t have. The old revision should still be there, warm, one traffic-shift call away. Three things quietly break rollback for agents. Prompts stored outside the artifact, so you roll back the container and the prompt stays new and now you’re running an untested combination. Migrated state, where the new version wrote records in a schema the old one can’t read — rolling back code doesn’t roll back data, so make schema changes additive and ship them a release ahead. And flags that outlived the rollback, still enabling the new tool after the code is gone.

Kill criteria

Write these before the rollout, not during it. The purpose is to make the decision to roll back mechanical, so that a tired engineer at 3 a.m. does not have to be brave.

A workable set:

Roll back immediately, no discussion:

Any confirmed safety incident — data exposed, an action taken that should not have been possible, a successful injection.
Error rate above 2x baseline sustained for five minutes.
p95 latency above 2x baseline sustained for ten minutes.
Cost per hour above 3x forecast.

Roll back after a look, within thirty minutes:

Judge quality score down more than 5 points against the concurrent baseline.
Tool error rate up more than 50%.
Human escalation rate up more than 30%.
Steps per successful task up more than 40%.

Halt the rollout, hold at current percentage, investigate:

Any of the above at half the threshold.
A single unexplained metric movement, even a favourable one. Unexplained is unexplained.

Two properties make these usable. Every threshold is measured against the concurrently serving baseline, not against last week. And every one is wired into an alert that names the rollback command in its body — the person who gets paged should not have to look it up.

Saying it out loud. You write kill criteria before the rollout, not during it, and the purpose is to make the decision mechanical so a tired engineer at 3 a.m. doesn’t have to be brave. Three tiers. Roll back immediately with no discussion on any confirmed safety incident, error rate over twice baseline for five minutes, p95 over twice baseline for ten, or cost over three times forecast. Roll back within thirty minutes after a look if the judge score drops more than five points, tool errors are up 50 percent, escalations up 30, or steps per task up 40. And halt in place to investigate on any unexplained metric movement — including a favourable one, because unexplained is unexplained. Two properties make them usable: every threshold is against the concurrently serving baseline rather than last week, and every alert body names the rollback command so nobody has to look it up.

What a rollout looks like end to end

Putting the pieces in order, for a change of any consequence:

The pipeline from Chapter 2 produces one artifact that has passed the eval gate.
Deploy it with no traffic, under a tag, and smoke-test it by name.
Mirror production traffic to it in shadow mode for a day or two, with action tools stubbed. Review the action divergence.
Route internal users to it. Read some transcripts yourself; this step is qualitative on purpose.
Move to 5% of real traffic. Watch tool error rate and steps per task for the first hour; the judge score after a few.
Expand on the written schedule, holding long enough at each step for the slowest signal you rely on.
At 100%, promote the eval results to be the new baseline, and keep the previous revision warm for a week.

Step 7 is the one people skip. If you do not promote the baseline, your next comparison is against a version that has not been live for a month, and your gate slowly stops meaning anything.

Saying it out loud. End to end it’s seven steps: one artifact that passed the eval gate, deployed with no traffic under a tag and smoke-tested by name; a day or two of shadow with writes stubbed, reviewing the action divergence; internal users, where you read some transcripts yourself because that step is qualitative on purpose; five percent of real traffic watching tool errors and steps per task in the first hour and the judge score after a few; then expansion on the written schedule. The step people skip is the last one — promoting the new eval results to be the baseline. Skip it and your next comparison is against a version that hasn’t been live in a month, and your gate slowly stops meaning anything.

What you should be able to do now

Explain why healthy infrastructure metrics are insufficient evidence that an agent deploy went well, with a concrete failure that produces only HTTP 200s.
Build a shadow-mode harness that stubs action tools with plausible successes, and use the resulting “would have done” log to compare candidate against production before any user is exposed.
Specify a canary gate that includes tool error rate, steps per successful task, tool selection distribution, escalation rate, a sampled judge score, and a business proxy — and order your rollout stages by how fast each signal responds.
Write a staged rollout schedule with hold times justified by the slowest signal, and say when your traffic volume is too low for canary analysis to mean anything.
Handle stateful sessions across a version flip using schema tolerance, session pinning, or draining, and pick the right one for your session lifetime.
Design per-capability feature flags that let you dark-launch a single action tool and act as a circuit breaker, and log the resolved flag set with every trace.
Get rollback under sixty seconds and one command, and name the three things — external prompts, migrated state, sticky flags — that silently break it.
Write kill criteria in advance, measured against the concurrently serving baseline, at three severity tiers.

Production Agent Engineering — A Practical Guide