Multi-agent systems: when one agent isn’t enough

The pitch is seductive and you have heard it a hundred times.

Instead of one overloaded agent trying to do everything, build a team. A researcher, a writer, a critic, a manager. Each one focused, each one simple, each one easy to test — a digital org chart, mirroring how humans divide work. The Google whitepaper puts it as a “team of specialists,” and it is not wrong to.

Here is what the pitch leaves out.

You have replaced function calls with network calls between components that communicate in natural language, that can each independently hallucinate, that have no shared memory unless you build it, that cannot see each other’s failures unless you propagate them, and whose combined behaviour you can no longer reason about by reading any single file. Every one of those is a distributed-systems problem. Distributed systems are hard when the components are deterministic. Yours are not.

This chapter is the honest version. When the split genuinely pays, when it is expensive theatre, what the topologies actually cost, what a handoff has to carry, and the failure modes that only exist once there is more than one agent.

The default answer is one agent

Start here, because it is the position you should have to be argued out of.

A single agent with a well-designed tool belt handles more than people expect. Twenty tools is fine if the descriptions are good — that is a Part 2 problem, not an architecture problem. Long tasks are fine if you manage context. Multiple domains are fine if the system prompt is organised.

The reflex to reach for multiple agents usually shows up for one of three reasons, and all three have cheaper fixes.

“The prompt is getting too long.” That is a context engineering problem. Splitting it across three agents does not delete the instructions; it distributes them and adds coordination on top.

“There are too many tools and it picks the wrong one.” That is a tool design problem. Two tools whose descriptions overlap will confuse a supervisor deciding which agent to route to just as reliably as they confuse a single agent choosing between them.

“It’s slow.” Sometimes real, and this is the one case where multi-agent is the direct answer — but only if the slow parts are genuinely independent. If step B needs step A’s output, running them “in parallel” as two agents buys you nothing but overhead.

A second agent should be the answer to a question you can state in one sentence, and the sentence should not be “it would be more elegant.”

Saying it out loud. My default is one agent, and I want to be argued out of it rather than into a team. The three reasons people reach for multiple agents — the prompt’s too long, it keeps picking the wrong tool, it’s too slow — are a context problem, a tool-design problem, and a parallelism problem, and only the third is actually answered by adding an agent. Splitting a long prompt across three agents doesn’t delete the instructions, it distributes them and adds coordination on top. And the moment you add a second agent you’ve turned function calls into network calls between components that talk in English and can each hallucinate independently — that’s a distributed system, and distributed systems are already hard when the parts are deterministic.

Three tests for when it pays

Here are the cases where the split earns its cost. If none applies, you are building a distributed monolith.

Test 1: genuinely parallel independent subtasks

The work decomposes into pieces that do not need each other’s results.

Research a company across ten sources. Review a pull request across security, performance, and style. Summarise forty documents.

The gain is latency, and it is real: \( n \) independent subtasks that each take \( t \) seconds finish in about \( t \) rather than \( nt \). The test is strict, though. If worker B needs worker A’s finding to know what to look for, you have a sequential pipeline wearing a parallel costume, and you will pay for the coordination without collecting the speedup.

Watch for the hidden dependency where two workers need the same fact. Both will go and get it. Now you have paid twice and, worse, they may get different answers and you have to reconcile them.

Saying it out loud. The first test is whether the work really breaks into pieces that don’t need each other’s results — research a company across ten sources, review a PR for security and performance and style. When that holds, the win is latency: n subtasks that each take t seconds finish in about t instead of n times t. But the test is strict. If worker B needs worker A’s finding to know what to look for, you’ve got a sequential pipeline in a parallel costume, and you’ll pay the coordination cost without collecting the speedup. Watch for the sneaky version too, where two workers need the same fact — now you’ve fetched it twice and they might come back with different answers you have to reconcile.

Test 2: genuinely distinct skill or tool sets

Not “different topics” — different capabilities.

A code-writing agent that needs a sandbox, a filesystem, and a test runner. A financial-analysis agent that needs a database, a spreadsheet engine, and a pricing API. These have different system prompts, different tool belts, different failure modes, different evaluation sets, and often different models — a cheap fast model for classification, an expensive one for synthesis.

Model-tier routing alone can justify the split. If 80% of your traffic is handled by a small model and 20% escalates, that is a real cost argument, not an aesthetic one.

The test that fails: “one agent for billing questions, one for shipping questions” where both have the same three tools and near-identical prompts. That is a switch statement you have built out of language models.

Saying it out loud. The second test is about different capabilities, not different topics. A coding agent that needs a sandbox and a test runner versus a financial agent that needs a database and a pricing API — those have different prompts, different tools, different failure modes, and often different models. Model-tier routing on its own can justify the split: if eighty percent of traffic clears on a small cheap model and twenty percent escalates, that’s a real cost argument. The version that fails the test is one agent for billing and one for shipping when both have the same three tools and nearly identical prompts — that’s a switch statement you’ve built out of language models.

Test 3: context isolation

This is the strongest argument and the least discussed.

A sub-agent that reads two hundred pages and returns one paragraph has kept 199 pages of noise out of the main agent’s context window. The main agent stays sharp because it never saw the raw material.

The mechanism is a hard boundary: the worker’s transcript does not become the supervisor’s transcript. Only its distilled output crosses. That distillation is the product, and it is why sub-agents are so effective for research and search — they are compression with reasoning attached.

Get this wrong — pass the worker’s full transcript up — and you have all the cost of multi-agent with none of the benefit, plus a supervisor whose context is now larger than a single agent’s would have been.

Saying it out loud. The strongest argument for a sub-agent is the one people talk about least: it keeps noise out of the main agent’s context. A worker reads two hundred pages and hands back one paragraph, so the supervisor stays sharp because it never saw the raw material. The mechanism is a hard boundary — the worker’s transcript does not become the supervisor’s transcript, only its distilled output crosses. That distillation is the actual product; sub-agents are compression with reasoning attached. Get it wrong and pass the full transcript up, and you’ve got all the cost of multi-agent with none of the benefit, plus a supervisor whose context is now bigger than a single agent’s would have been.

Four topologies

Each with what it is good for and what it will do to you.

Orchestrator-worker (supervisor)

One agent decomposes the task, dispatches sub-tasks to workers, collects results, and synthesizes. Workers do not talk to each other. The whitepaper calls this the Coordinator pattern; you will also see “supervisor” and “manager.”

This is the workhorse. It maps cleanly onto the fan-out and fan-in you built in the last chapter, control is centralised in one place you can read, and adding a worker does not change any other worker.

Its weakness is that the supervisor is the whole system. If it decomposes badly — sub-tasks that overlap, or that leave a gap, or that are underspecified — every worker does the wrong thing in parallel, efficiently. The supervisor also becomes a context bottleneck: it holds the plan, every returned result, and the synthesis.

Design rule: the supervisor’s decomposition prompt is the highest-leverage text in the system. Spend disproportionate effort on it, and put the sub-task list in a structured output schema so you can inspect and test it independently of everything downstream.

Saying it out loud. Orchestrator-worker is the workhorse: one agent decomposes the task, fans it out to workers who don’t talk to each other, then collects and synthesizes. It’s popular because control is centralised in one file you can read, and adding a worker doesn’t change any other worker. The weakness is that the supervisor is basically the whole system — if it decomposes badly, with overlapping sub-tasks or a gap, every worker does the wrong thing in parallel, very efficiently. So the decomposition prompt is the highest-leverage text you own, and I’d put the sub-task list in a structured output schema so I can test the plan on its own before anything downstream runs.

Hierarchical

Supervisors of supervisors. A top-level coordinator delegates to mid-level leads who delegate to workers. Google’s co-scientist system, described in the whitepaper, runs this shape with a supervisor allocating work and compute across a fleet of specialists over hours or days.

Use it when the task tree is genuinely deep and the mid-level nodes add real decomposition value. Be aware that error and cost compound multiplicatively with depth: at 90% reliability per level, three levels gives you \( 0.9^3 \approx 73% \). And debugging is now archaeology across three layers of natural-language handoffs.

Most teams that build three levels needed two.

Saying it out loud. Hierarchical is supervisors of supervisors, and you want it only when the task tree is genuinely deep enough that the middle layer adds real decomposition value. The reason to be suspicious is that reliability compounds multiplicatively with depth — at ninety percent per level, three levels is about seventy-three percent end to end. And debugging becomes archaeology across three layers of natural-language handoffs, where the interesting information only exists if you logged the packets deliberately. My rule of thumb is that most teams who built three levels needed two.

Peer handoff

No coordinator. Agents transfer control directly to each other — a triage agent hands to a billing agent, which hands to a refunds agent. Control moves; there is no return.

This is the OpenAI Agents SDK’s model, where handoffs are exposed to the model as tools, generated automatically with names like transfer_to_refunds_agent (https://openai.github.io/openai-agents-python/handoffs/). That framing is elegant: the model already knows how to call tools, so it already knows how to hand off.

from agents import Agent, handoff

refunds = Agent(name="Refunds", instructions="...")
triage = Agent(
    name="Triage",
    instructions="Route the customer to the right specialist.",
    handoffs=[refunds],
)

Handoff is the right shape for conversational routing, where one agent should own the user at any moment. It is the wrong shape for anything needing aggregation, because nobody is left holding the whole picture.

The failure mode is ping-pong: A hands to B, B decides it is A’s problem, hands back. Cap the number of handoffs per session, and log the handoff chain — a chain longer than three is a routing-design bug that will not fix itself.

Saying it out loud. Peer handoff has no coordinator — agents transfer control directly, triage hands to billing, billing hands to refunds, and there’s no return trip. The elegant part is that frameworks expose the handoff as a tool call, so the model already knows how to do it. It’s the right shape for conversational routing where exactly one agent should own the user at any moment, and the wrong shape for anything that needs aggregation, because nobody is left holding the whole picture. The failure mode to name is ping-pong: A hands to B, B decides it’s A’s problem and hands back. Cap handoffs per session and log the chain — anything longer than three is a routing-design bug that won’t fix itself.

Blackboard / shared state

Agents read and write a shared structured workspace instead of messaging each other. An agent picks up work when the state satisfies its precondition.

This is what your workflow engine already is: the state object is the blackboard, reducers are the write protocol, and conditional edges are the preconditions. It is the most debuggable topology, because at any moment there is one artifact that tells you everything the system knows.

The cost is that it only works when you can define the schema up front. Free-form collaboration does not fit in a dataclass. The mitigation for concurrent writes is exactly the reducer discipline from the last chapter — declare it or the engine raises.

There is a fifth shape, “group chat,” where several agents share a conversation and a manager picks who speaks next — AG2’s GroupChat with a GroupChatManager, or AutoGen’s SelectorGroupChat (https://docs.ag2.ai/latest/docs/user-guide/advanced-concepts/groupchat/groupchat/). It is excellent for prototyping and exploration. Be careful shipping it: token cost grows with the square of the conversation because every agent reads everything, and “who should speak next” is a decision with no ground truth to evaluate against.

Saying it out loud. In the blackboard shape, agents don’t message each other at all — they read and write one shared structured workspace, and an agent picks up work when the state meets its precondition. If you’ve built a graph engine you already have this: the state object is the blackboard, reducers are the write protocol, conditional edges are the preconditions. It’s the most debuggable topology because at any moment there’s one artifact that tells you everything the system knows. The cost is that you have to define the schema up front, and genuinely free-form collaboration doesn’t fit in a dataclass. The related shape to be careful with is group chat, where every agent reads everything — token cost grows with the square of the conversation, and “who speaks next” is a decision with no ground truth to evaluate against.

What a handoff has to carry

The most common multi-agent failure is not a bad topology. It is information lost at a boundary.

When agent A hands work to agent B, everything B needs must be in the handoff, because B cannot see A’s context. Teams get this wrong by assuming the sub-task description is enough. It is not, and here is the checklist.

The objective, stated as one testable sentence. Not “look into the billing situation.” “Establish what the customer was actually charged for order 12345 and whether a refund has been issued.” The difference is whether you can tell if the worker succeeded.

The resolved inputs. Actual values, not references. {"order_id": "12345"}, not “the order the customer mentioned.” The worker has never seen the customer’s message.

The constraints. Tool allowlist, step budget, cost budget, deadline, and anything it must not do. Budgets belong in the packet because they belong to the sub-task, not the agent — the same worker gets a bigger budget for a harder job.

The return contract. What shape the answer should take, and how long. “Three bullet findings, each citing the tool result it came from” produces something the supervisor can actually use. Absent this, workers write essays, and the supervisor’s context fills with prose.

Provenance. Who asked, which run, which attempt. You need this the first time a worker misbehaves and you have to reconstruct why it was called.

Write it as a typed object, not a formatted string:

@dataclass
class Handoff:
    sender: str
    recipient: str
    objective: str                 # one sentence, testable
    inputs: dict                   # facts, already resolved
    tool_allow: list[str]          # qualified prefixes: "orders.", "kb."
    max_steps: int = 4
    return_contract: str = "3 bullet findings, each with the tool result it came from"

The value of the dataclass is not tidiness. It is that the boundary is now inspectable, loggable, diffable, and testable without running a model. When a worker returns nonsense you can look at exactly what it was given and immediately tell whether the bug is in the decomposition or the worker.

The return path deserves the same treatment. A Report with a status, findings, a note, and steps used — not a free-text blob. A supervisor that has to parse prose to find out whether a worker succeeded will eventually get it wrong.

Saying it out loud. The most common multi-agent failure isn’t a bad topology, it’s information lost at a boundary. When A hands to B, B cannot see A’s context, so everything B needs has to be in the packet: a one-sentence testable objective, the actual resolved values rather than references like “the order the customer mentioned,” the constraints and budgets, the return contract, and provenance. Make it a typed object, not a formatted string — then the boundary is loggable, diffable, and testable without running a model. The payoff is diagnostic: when a worker returns nonsense, you look at exactly what it was given and instantly know whether the bug is in the decomposition or in the worker.

Failure modes you only get with multiple agents

Cascading errors

Worker A returns a plausible wrong fact. The supervisor has no way to check it — that is why it delegated — so it flows into the synthesis and into everything downstream. Confidence is preserved at every hop while accuracy is not.

Mitigate by having workers cite the tool result behind each finding, so the supervisor can spot an unsupported claim, and by adding a verification node for anything consequential. Structurally: never let a claim cross a boundary without its evidence.

Saying it out loud. Cascading errors are what happens when a worker returns a plausible wrong fact and the supervisor has no way to check it — which is the whole reason it delegated in the first place. So the error flows into the synthesis and everything downstream, and the nasty part is that confidence is preserved at every hop while accuracy isn’t. The structural fix is a rule: never let a claim cross a boundary without its evidence. Make workers cite the tool result behind each finding so an unsupported claim is visible, and add an explicit verification node for anything consequential.

Duplicated work

Two workers both need the customer record. Both fetch it. You pay twice, and if the data changed between the calls, they now disagree and the supervisor has to arbitrate between two of its own agents.

Mitigate by resolving shared facts before the fan-out and putting them in every packet’s inputs, and by caching reads at the tool layer keyed on arguments.

Saying it out loud. Duplicated work is two workers both needing the customer record and both going to fetch it. You pay twice, which is annoying, but the real problem is that if the data changed in between they now disagree, and the supervisor has to arbitrate between two of its own agents with no basis for choosing. The fix is to resolve shared facts before the fan-out and stamp them into every packet’s inputs, plus cache reads at the tool layer keyed on arguments. It’s the same consistency problem you’d have in any distributed system, just with a language model doing the arbitration.

Responsibility diffusion

Nobody owns the answer. The supervisor assumed workers verified their findings; workers assumed the supervisor would. This shows up as a system that is individually correct at every step and collectively wrong.

Mitigate by naming one node responsible for each quality property, in the graph. If accuracy matters, there is a verification node and its name is in the diagram.

Saying it out loud. Responsibility diffusion is when nobody owns the answer. The supervisor assumed the workers verified their findings, the workers assumed the supervisor would, and you end up with a system that’s individually correct at every step and collectively wrong. The fix isn’t a prompt reminding everyone to be careful, it’s naming one node responsible for each quality property in the graph itself. If accuracy matters, there is a verification node and its name is on the diagram — otherwise “who checks this?” has no answer you can point at.

Context loss at the boundary

The user said “and don’t email them, they’ve asked us to stop.” The supervisor decomposed into three sub-tasks and that constraint appeared in none of the packets. A worker with an email tool now has no idea.

Mitigate with a constraints field that is propagated to every packet by construction, and by keeping tool allowlists narrow so a worker physically cannot do the thing you forgot to forbid. Allowlists are the more reliable of the two: a prompt can be forgotten, and a missing tool cannot be called.

Saying it out loud. Context loss at the boundary is the one that actually hurts customers. The user says “and don’t email them, they asked us to stop,” the supervisor splits the work into three sub-tasks, and that constraint appears in none of the packets — so a worker with an email tool has no idea. Two mitigations, and one is stronger than the other. You propagate a constraints field into every packet by construction, and you keep tool allowlists narrow. The allowlist is the more reliable of the two, because a prompt can be forgotten and a missing tool cannot be called.

Coordination overhead exceeding the win

Five workers, each returning a paragraph, and the supervisor now reasons over five paragraphs plus its plan. Total tokens exceed what a single agent would have used. This is the theatre case, and the tell is that your token count went up and your quality did not.

Measure it. If you cannot show the multi-agent version beating the single-agent baseline on your eval set, you have added complexity for nothing. That baseline is not optional — build it first.

Saying it out loud. The last failure mode is the theatre case: five workers each return a paragraph, the supervisor now reasons over five paragraphs plus its plan, and total tokens exceed what a single agent would have spent. The tell is that your token count went up and your quality didn’t. This is more common than the marketing suggests — multi-agent setups frequently underperform a single agent at the same token budget, and most headline wins you read about were never token-matched in the first place. So build the single-agent baseline first and make the multi-agent version beat it on your eval set. If it can’t, you’ve added distributed-systems complexity for nothing.

The arithmetic

Be concrete about what this costs, because “it’s more expensive” is not actionable.

Take a single agent that does six sequential steps at roughly 3,000 tokens each, dominated by re-sending accumulated context.

Now the multi-agent version: a supervisor that plans (1 call), three workers doing two steps each (6 calls), and a synthesis (1 call). Eight model calls against six. Each worker’s context is smaller than the single agent’s would have been at the same point — that is the isolation benefit — but the supervisor pays for the plan and the three returned reports.

Two rules of thumb hold up in practice.

Token cost typically rises. Anthropic reported roughly 4x the tokens of a single agent for their multi-agent research system, and 15x a plain chat interaction (https://www.anthropic.com/engineering/multi-agent-research-system). Your ratio will differ, but plan for a multiple, not a discount. The justification has to be quality or latency, never cost.

Latency improves only for the parallel portion. Amdahl’s law, applied to agents: if a fraction \( p \) of the work is parallelisable across \( n \) workers, your speedup is bounded by \( 1 / ((1-p) + p/n) \). Planning and synthesis are the serial part and they are not small. With \( p = 0.6 \) and three workers, the ceiling is 1.67x — and that is before coordination overhead. Meanwhile p99 latency is now governed by your slowest worker, so one straggler erases the gain for the unluckiest requests.

And two costs that do not show on a dashboard.

Evaluation gets harder. You now need per-agent eval sets and end-to-end ones, plus a way to attribute an end-to-end failure to a specific agent. Budget for this before you build.

Debugging gets harder. A single trace becomes a tree of traces, and the interesting information — what was in each handoff packet — is only there if you logged it deliberately. Log every packet and every report. You will need them within a week.

Saying it out loud. Here’s the arithmetic I’d give. Token cost almost always goes up — Anthropic reported roughly four times a single agent’s tokens for their multi-agent research system, and about fifteen times a plain chat turn — so the justification has to be quality or latency, never cost. Latency only improves for the genuinely parallel portion, which is Amdahl’s law applied to agents: if sixty percent parallelises across three workers, your ceiling is about 1.67x before any coordination overhead, and p99 is now set by your slowest worker. And the honest framing is that at an equal token budget a well-built single agent often wins outright, because the published comparisons usually aren’t token-matched. The two costs that never show on a dashboard are that evaluation now needs per-agent suites plus end-to-end ones with failure attribution, and that a single trace has become a tree of traces.

Interoperability, briefly

Everything above assumes agents you own, in one process.

When the agents belong to different teams or different companies, you need a protocol. The Agent2Agent (A2A) protocol is the open standard for this: agents publish an Agent Card, a JSON document advertising capabilities, endpoint, and auth requirements, and interact through long-running asynchronous tasks with streaming updates rather than single request-response calls (https://a2a-protocol.org/latest/specification/).

The distinction from MCP is the one to hold onto, and the whitepaper states it plainly: agents are not tools. MCP gives an agent access to capabilities — transactional, request-response, you call it and it returns. A2A connects agents that are each doing their own reasoning, over interactions that can take minutes and produce intermediate updates.

If your “multi-agent system” is one process fanning out to three functions, you do not need A2A and adding it is pure cost. If it spans organisational boundaries with independent deploy cycles, you need something like it, and building your own is a bigger project than it looks.

Saying it out loud. MCP and A2A get confused constantly, and the clean line is that agents are not tools. MCP gives one agent access to capabilities — transactional, request-response, you call it and it returns. A2A connects agents that are each doing their own reasoning, over long-running asynchronous tasks with streaming updates, where an agent publishes an Agent Card advertising what it can do, its endpoint, and its auth. So the practical test is organisational: if your multi-agent system is one process fanning out to three functions, A2A is pure cost. If it spans teams or companies with independent deploy cycles, you need something like it, and rolling your own is a much bigger project than it looks.

Patterns, and where to find them

This chapter deliberately does not enumerate patterns. Reflection, planning, iterative refinement, ensembling, generator-critic, tool-use routing, and the rest are covered properly — twenty-one of them, with trade-offs and worked examples — in the companion repository, the agentic-ai-evaluation-guide design-patterns playbook. Read that for the catalogue.

What you need from this part is the machinery underneath. Every one of those patterns is nodes, edges, shared state, and a budget. Reflection is a cycle with a termination condition. Generator-critic is two nodes and a conditional edge. Ensembling is a fan-out with a reducer that votes. Once the engine is real, adopting a pattern is an afternoon.

Next chapter you build the orchestrator-worker system for real, on the engine from Chapter 2, with typed handoff packets, per-agent budgets and allowlists, tools drawn from three separate MCP servers, and a worker whose backend is down.

Saying it out loud. I’d resist reciting a pattern catalogue, because once you have the engine, the patterns are trivial to express. Reflection is a cycle with a termination condition. Generator-critic is two nodes and a conditional edge. Ensembling is a fan-out with a reducer that votes. That’s the real point: the machinery underneath is nodes, edges, shared state, and a budget, and adopting a named pattern on top of that is an afternoon’s work. Someone who has only memorised the pattern names can’t tell you what happens when the critic never converges; someone who’s built the engine reaches straight for the step cap.

What you should be able to do now

Apply the three tests — independent parallel subtasks, genuinely distinct skill and tool sets, context isolation — and defend a decision to use one agent instead of several.
Choose between orchestrator-worker, hierarchical, peer handoff, and blackboard topologies for a specific problem, and state the specific failure each one is prone to.
Specify a handoff packet completely: objective, resolved inputs, constraints and budgets, return contract, and provenance — and explain what breaks when each is missing.
Name the multi-agent-specific failure modes and give a structural mitigation for each, rather than a prompt-level one.
Estimate the token and latency impact of a proposed decomposition before building it, using the serial fraction and the reported token multiples, and insist on a single-agent baseline to beat.
Explain why MCP and A2A solve different problems, and when a system genuinely needs the latter.

Production Agent Engineering — A Practical Guide