People, process, and the road to production

There is a specific failure that has nothing to do with your code.

The agent is good. The demo went well. Six weeks later it is still not live, and when you ask why, the answer is a shrug and a list: legal wants to see something, nobody agreed what “good enough” means, the platform team does not know what permissions to grant, and the person who wrote the prompts left for another team.

That is not a technical problem and no amount of clever engineering fixes it. This chapter is about the part of production that is made of people, decisions, and agreements — and about the one technical principle that makes the agreements enforceable.

Read it even if you are a team of two. The roles below still exist at that size; they are just all wearing your face, and knowing which hat you have on when is most of the value.

Who owns an agent in production

Traditional software has a clean answer. The team that wrote it operates it.

Agents smear that boundary, because an agent’s behaviour is determined by artifacts that different people own. The prompt is one. The tool schemas are another. The model version is a third, and you do not own it at all — the vendor does, and they will change it. The retrieval corpus is a fourth, and it is owned by whoever owns the documents.

So the useful question is not “who owns the agent” but “for each thing that can change the agent’s behaviour, who is accountable when it does?”

Answer it explicitly, in a file in the repository, and you will avoid most of the six-week stall. Here is the shape of the answer at a normal company.

The platform or cloud team owns the environment: identity, network, secrets storage, the runtime the agent deploys onto, and the least-privilege roles that determine what the agent’s service account can touch. This team is why the agent cannot reach the production database it was never supposed to reach. Get them involved before you need them, because “grant this service account access to the payments API” is a conversation with a lead time.

The data team owns whatever the agent retrieves from: the ingestion pipeline, the freshness guarantees, the quality bar for the corpus. When your agent confidently cites a policy that was superseded in March, that is a data ownership question, not a prompt question.

The AI or ML engineering function owns the agent itself — the loop, the tool implementations, the context strategy, the evaluation harness, and the deployment pipeline. This is you.

Whoever owns behaviour owns the prompts, the golden dataset, and the definition of a correct answer. The industry has not settled on a title for this — “prompt engineer” exists at some companies, at others it is a domain expert with a text editor, at others it is the same AI engineer wearing a second hat. The title does not matter. What matters is that somebody is accountable for the sentence “this output is correct,” because that sentence is the foundation of every gate in the rest of this part, and if nobody owns it your eval set is a pile of opinions.

A governance function owns the record: which version is live, what it scored, what data it can see, who approved it. At a small company this is a spreadsheet and a habit. At a regulated one it is a formal artifact repository with an auditor attached. Either way, the requirement is the same — six months from now someone will ask what the agent was doing on a particular Tuesday, and you need to be able to answer.

Product owns the go/no-go. The last step into production is almost never fully automatic, and it should not be. Somebody with commercial accountability looks at the eval report and says ship.

The failure mode when this is undefined is not chaos. It is silence. Everyone assumes someone else is checking, the agent ships without a quality bar, and the first person to discover the regression is a customer.

Saying it out loud. With normal software the team that wrote it operates it, and that breaks for agents because behaviour comes from artifacts different people own — the prompt, the tool schemas, the retrieval corpus, and the model version, which you don’t own at all because the vendor does and they will change it. So the question isn’t “who owns the agent,” it’s “for each thing that can change the agent’s behaviour, who’s accountable when it does.” Write that down in a file in the repo. The one role people forget is whoever owns the sentence “this output is correct” — no title has settled on it, but if nobody owns it your eval set is just a pile of opinions. And the failure mode when this is undefined isn’t chaos, it’s silence: everybody assumes somebody else is checking, and the first person to find the regression is a customer.

What changes about your development process

Four things, concretely.

Behaviour becomes a reviewable artifact

In normal code review you read a diff and reason about what it does. For an agent, the diff might be four words in a system prompt, and its effect is unknowable by reading.

So the review needs a second artifact: an evaluation report comparing this version against the current production baseline. Not “I tested it and it seems fine.” A table with numbers on a fixed dataset.

This changes what a reviewer does. They still read the code, but they also read the behavioural delta, and they are accountable for asking why task success dropped two points on the refund scenarios. The Google whitepaper frames this as the “pre-PR evaluation” — the engineer runs the suite locally and links the report in the pull request description, making the report a mandatory review artifact. That is the cheap version, and it is a legitimate place to start. Chapter 2 automates it so the pipeline produces the report and blocks the merge, which is the version you want by the time more than three people are committing.

Saying it out loud. In normal code review you read a diff and reason about what it does. For an agent the diff might be four words in a system prompt, and its effect is genuinely unknowable by reading. So the review needs a second artifact — an evaluation report comparing this version to the production baseline on a fixed dataset, not “I tried it and it seemed fine.” That changes the reviewer’s job: they’re now accountable for asking why task success dropped two points on refund scenarios. The cheap version is the engineer running the suite locally and linking the report in the PR, which is a completely legitimate place to start; the version you want once more than about three people are committing is the pipeline producing it and blocking the merge.

Every artifact gets a version, together

An agent is not source code. It is source code plus prompts plus tool definitions plus a model identifier plus configuration, and its behaviour is a function of all five.

Which means a version number that covers only the code is a lie. When you roll back, you have to roll back the set. When you file an incident, you have to record the set. When you compare two eval runs, they have to differ in exactly one element of the set or the comparison means nothing.

The practical rule: prompts live in the repository as files, not in a database and not in a UI. Prompt-management SaaS that lets a non-engineer edit production behaviour without a commit is selling you an outage. If you need non-engineers editing prompts — and you often do — give them a pull request workflow with a preview, not a live edit button.

Saying it out loud. An agent isn’t source code — it’s code plus prompts plus tool definitions plus a model identifier plus config, and behaviour is a function of all five. So a version number covering only the code is a lie. You roll back the set, you record the set in the incident, and when you compare two eval runs they have to differ in exactly one element of the set or the comparison means nothing. The practical rule is that prompts live in the repo as files, not in a database and not in a SaaS UI, because anything that lets someone change production behaviour without a commit is selling you an outage. If non-engineers need to edit prompts, and they often do, give them a pull request with a preview, not a live edit button.

Your definition of done grows a section

For a normal service, done means the tests pass and the feature works. For an agent, done includes:

The eval set has cases covering this change, including the failure cases.
The report shows no regression on the existing set beyond the agreed noise band.
Any new tool has a policy entry saying who may call it and with what arguments (Chapter 4).
Anything irreversible has a human gate or a documented reason it does not need one.
The trace emitted by the new path contains enough to debug it at 2 a.m.

That list is not bureaucracy, it is the checklist derived from the incidents everyone has already had.

Saying it out loud. For a normal service, done means tests pass and the feature works. For an agent, done grows a few lines: the eval set has cases covering this change including the failure cases, the report shows no regression beyond an agreed noise band, any new tool has a policy entry saying who may call it with what arguments, anything irreversible has a human gate or a written reason it doesn’t need one, and the trace from the new path is good enough to debug at 2 a.m. That’s not bureaucracy — every one of those lines is derived from an incident somebody has already had.

Incidents produce test cases, not just fixes

The single highest-leverage process change, and the one that compounds.

When an agent fails in production, the fix is the second deliverable. The first is a new case in the golden dataset that reproduces the failure. Otherwise your eval set stays frozen at whatever you imagined before launch, while reality keeps generating failure modes you did not imagine.

Chapter 5 turns this into a mechanism. Here it is just the norm: no production failure closes without a test case.

Saying it out loud. This is the single highest-leverage process change and the one that compounds: when the agent fails in production, the fix is the second deliverable. The first is a new case in the golden dataset that reproduces the failure. Otherwise your eval set stays frozen at whatever you imagined before launch, while reality keeps generating failure modes you never imagined. The rule is one sentence — no production failure closes without a test case — and the effect is that your suite gets sharper exactly where your system is weakest, for free, forever.

The stages from prototype to GA

The path has five stops. Naming them prevents the argument where one person means “it’s live” and another means “three of us can use it.”

1. Prototype. One engineer, a notebook or a script, no persistence guarantees, no other users. The goal is answering “is this even possible,” and the correct amount of infrastructure is close to zero. The failure at this stage is spending three weeks on a deployment pipeline for something that turns out not to work.

The exit criterion is a working trajectory on ten real inputs, and the beginning of an eval set — write the golden cases while you still remember which inputs were hard.

2. Internal alpha. Deployed somewhere other than your laptop, reachable by your team. Real observability from day one: traces, logs, token counts. The eval set exists and runs, even if it runs manually.

The goal here is discovering the input distribution. Your teammates will type things you did not anticipate, and that is the product of this stage — not the feature list, the inputs.

The exit criterion is a stable eval score you believe, plus the tool authorization layer in place, because the next stage puts the agent in front of people who did not build it.

3. Dogfood. Everyone in the company can use it, or at least everyone in the relevant function. This is where load, cost, and the long tail of weird requests show up together for the first time.

The whitepaper is right to call this out as its own stage rather than folding it into staging. Internal users tolerate rough edges and give you qualitative feedback that no metric produces. They also, importantly, are people you can apologise to.

The exit criterion is a week without a severity-one surprise, a cost-per-task number you can defend, and a runbook.

4. Limited external release. A canary, a percentage, a specific customer segment, or a flag. Chapter 3 is entirely about how to do this step safely.

The exit criterion is quality metrics on real traffic that hold up against the baseline, plus a rollback you have actually tested rather than one you believe exists.

5. General availability. Everyone. Which is not the end — it is the point at which the Observe → Act → Evolve loop in Chapter 5 becomes your permanent job.

The mistake to avoid is skipping stages because the demo was good. Every stage exists to surface a different class of problem, and problems you skip do not disappear, they queue.

Saying it out loud. There are five stops — prototype, internal alpha, dogfood, limited external release, and GA — and naming them prevents the argument where one person means “it’s live” and another means “three of us can use it.” Each stage exists to surface a different class of problem. Prototype answers whether it’s possible at all, with close to zero infrastructure. Alpha’s real product isn’t the feature list, it’s discovering the input distribution, because your teammates will type things you never anticipated. Dogfood is where load, cost, and the long tail of weird requests arrive together, and internal users are the ones you can actually apologise to. Limited release is the first honest quality signal on real traffic. The mistake is skipping stages because the demo went well — problems you skip don’t disappear, they queue.

Evaluation as a quality gate

Here is the principle everything else in this part hangs from.

No agent version reaches users without first passing a measured quality check.

The whitepaper calls this evaluation-gated deployment, and the phrasing is worth keeping because it is precise about the mechanism: evaluation is not a report you generate and file, it is a gate that is either open or closed.

Why agents need this when normal software does not is worth being exact about, because “AI is unpredictable” is not an argument anyone can act on.

Normal software has a property that agents lack: local reasoning works. You change a function, you can determine what else that function affects, and your tests cover the affected surface. Change a sentence in a system prompt and the affected surface is everything the agent does. There is no call graph. The blast radius of every change is the whole system.

There is a second property agents lack: unit tests are not sufficient evidence. Every tool can have perfect test coverage while the agent fails, because the failure is in the choice of tool, the order of calls, the moment it decided it had enough information, or the confident sentence it produced without looking anything up. The whitepaper puts it well — you can pass a hundred unit tests for your tools and still fail spectacularly by picking the wrong one.

What you have to evaluate is the trajectory, not just the final answer. Did it call the tool it should have called, before it made the claim it made?

So the gate measures at least three families of thing:

Task outcome. Did the run achieve the goal, judged against a reference or a rubric. Trajectory quality. Tool call success rate, correct tool selection, steps used against the budget, whether it looked things up before asserting. Safety. Guardrail violations, injection resistance, refusal correctness, PII in outputs.

Set a threshold on each, on a fixed dataset, and the gate becomes mechanical.

Saying it out loud. The principle is that no agent version reaches users without passing a measured quality check — evaluation isn’t a report you file, it’s a gate that’s either open or closed. And the reason agents need this when normal software doesn’t isn’t “AI is unpredictable,” which nobody can act on. It’s two specific properties. Local reasoning fails: change a function and you can trace what it affects, but change a sentence in a system prompt and the blast radius is everything the agent does, because there’s no call graph. And unit tests aren’t sufficient evidence: every tool can have perfect coverage while the agent fails, because the failure was in which tool it picked, in what order, or in the confident sentence it produced without looking anything up. So the gate has to measure the trajectory as well as the outcome, plus safety, with a threshold on each.

Two ways to implement the gate

The whitepaper names both, and the choice is about team maturity rather than correctness.

The manual pre-PR gate. Whoever owns behaviour runs the suite locally, produces a report comparing the change against the production baseline, and links it in the pull request. The reviewer is now responsible for the behavioural delta as well as the code.

This is flexible, it costs no infrastructure, and it is honest about being a human process. It also erodes. Someone will skip it on a Friday, and once it is skipped once it is optional forever.

The automated in-pipeline gate. The harness runs in CI, compares against a golden dataset, and blocks the deploy if a threshold is breached. Rigid, consistent, and not subject to Friday.

The cost is real: the pipeline is now slower, the gate can fail for reasons that are not the developer’s fault, and if the thresholds are wrong you have built a machine for annoying people. Chapter 2 is largely about making this gate trustworthy enough that nobody wants to disable it.

Start manual if you are starting. Automate before your team gets to about five contributors, because that is roughly where informal discipline stops scaling.

Saying it out loud. There are two ways to run the gate and the choice is about team maturity, not correctness. The manual pre-PR version — whoever owns behaviour runs the suite and links the report — costs no infrastructure and is honest about being a human process. Its failure mode is erosion: somebody skips it on a Friday, and once it’s skipped once it’s optional forever. The automated in-pipeline gate is consistent and immune to Fridays, but it makes the pipeline slower, it can fail for reasons that aren’t the developer’s fault, and if your thresholds are wrong you’ve built a machine for annoying people. Start manual if you’re starting, automate before you hit about five contributors, because that’s roughly where informal discipline stops scaling.

What a golden dataset actually is

A curated, versioned set of representative cases with known-good expectations. Not a dump of production logs, and not a hundred cases you generated with a model in an afternoon.

The properties that make it useful:

It is versioned alongside the code, so an eval run is reproducible and so a change to the dataset is reviewable. Silently loosening a test case is the easiest way to make a gate pass and the hardest to catch — treat dataset edits as a code review.

It covers the boring cases and the sharp ones. Happy paths tell you the agent works. The value is in the ambiguous request, the missing record, the tool that returns an error, the customer who asks for something you must refuse, and the input containing an injection attempt.

It is small enough to run often. A suite that takes forty minutes and two hundred dollars will run nightly at best, which means pull request feedback loses it. Chapter 2 splits it: a fast subset per PR, the full set nightly and pre-release.

The agentic-ai-evaluation-guide sibling repository covers dataset construction, LLM-as-judge design, rubric writing, and metric selection in far more depth than fits here. This part assumes you have a harness with a callable interface and focuses on wiring it into a gate.

Saying it out loud. A golden dataset is a curated, versioned set of representative cases with known-good expectations — not a dump of production logs and not a hundred cases you generated with a model in an afternoon. Three properties make it useful. It’s versioned alongside the code, and edits to it go through review, because quietly loosening a test case is the easiest way to make a gate pass and the hardest thing to catch. It covers the sharp cases as well as the boring ones — the ambiguous request, the missing record, the tool that errors, the thing you must refuse, the input carrying an injection attempt. And it’s small enough to run often: a suite that takes forty minutes and two hundred dollars runs nightly at best, which means it’s no longer part of the pull request loop.

The three pillars, and why the order matters

Everything in Part 6 rests on three capabilities, and teams consistently build them in the wrong order.

Observability comes first, and it is not optional at any stage. You cannot evaluate what you cannot see, and you cannot debug an agent by reading its source. If you build one thing before shipping, build the trace.

Automated evaluation comes second, because it is what turns observability into a decision. Traces tell you what happened; evaluation tells you whether it was good.

Automated deployment comes third, and this is the ordering people get wrong — they build a beautiful CI/CD pipeline first, then discover it has nothing meaningful to gate on, so it becomes a fast way to ship regressions.

Build them in that order and each one makes the next one worth having.

Saying it out loud. Three capabilities, and teams consistently build them in the wrong order. Observability first, always — you can’t evaluate what you can’t see and you can’t debug an agent by reading its source, so if you build one thing before shipping, build the trace. Automated evaluation second, because that’s what turns observability into a decision: traces tell you what happened, evaluation tells you whether it was good. Deployment automation third. The expensive mistake is doing that one first, because then you’ve built a beautiful pipeline with nothing meaningful to gate on, which is just a very fast way to ship regressions.

What this looks like on a small team

You may be reading all of the above thinking it describes a company four times your size. Here is the two-person version, which is genuinely sufficient for a long time.

One file in the repository names, for each artifact, who decides: prompts, eval set, tool policy, deploy approval. It can be four lines.

Prompts and tool schemas live in version control. One version number covers code, prompts, tools, and the model identifier together.

An eval set of thirty to sixty cases, versioned in the repository, with a fast subset of about a dozen. The fast subset runs on every pull request; the full set runs before a release.

A rule that no production failure closes without a new case in the set.

A deployment that can be rolled back with one command you have run at least once on purpose.

That is the whole process. It fits on an index card, and it is the difference between an agent you ship and an agent you demo.

Next chapter you build the pipeline that enforces it.

Saying it out loud. If you’re two people, the whole process fits on an index card and it’s genuinely sufficient for a long time. One file naming who decides on prompts, eval set, tool policy, and deploy approval — it can be four lines. Prompts and tool schemas in version control, with one version number covering code, prompts, tools, and the model identifier together. Thirty to sixty eval cases in the repo with a fast subset of about a dozen that runs on every PR. A rule that no production failure closes without a new case. And a rollback you’ve actually run once on purpose rather than one you believe exists. That last one is the difference between an agent you ship and an agent you demo.

What you should be able to do now

Map every artifact that can change your agent’s behaviour — code, prompts, tool schemas, model version, retrieval corpus, configuration — to a named accountable owner, and explain what stalls when one of those is unowned.
Name the five stages from prototype to GA, state the specific class of problem each one surfaces, and give an exit criterion for each.
Argue why agent changes need a behavioural evaluation report as a review artifact, using the “local reasoning fails” and “unit tests are insufficient evidence” arguments rather than a vague appeal to nondeterminism.
Choose between a manual pre-PR gate and an automated in-pipeline gate for your team’s current size, and say what each one costs you.
List the properties that make a golden dataset useful, and explain why editing it must be treated as a reviewable change.
Order observability, evaluation, and deployment automation correctly, and explain why building the pipeline first is a common and expensive mistake.

Production Agent Engineering — A Practical Guide