Model, tools, orchestration: the anatomy underneath every agent

Every agent you will ever build or read is made of the same three parts.

A model that reasons. Tools that act. An orchestration layer that runs the loop, holds the state, and decides when to stop.

Frameworks differ in which of these they hide and how well. They do not differ in having them. Learn the anatomy and every framework becomes a skin over something you already understand.

This chapter goes through all three, then covers the two levers you actually pull to steer the thing: the instructions you write, and the context you assemble.

The model: the reasoning core

The model is what turns a goal and a pile of context into a decision about what to do next. That is its entire job in an agent. Not to know things — to decide things.

This reframing matters more than it sounds. People pick models by asking “which one is smartest,” which is roughly the wrong question. The right question is “which one makes good next-step decisions given messy context and a list of tools.”

Saying it out loud. Every agent is the same three parts: a model that reasons, tools that act, and an orchestration layer that runs the loop, holds the state, and decides when to stop. Frameworks differ in which of those they hide, not in having them. The reframe I’d push on the model piece is that its job in an agent isn’t to know things, it’s to decide things — what’s the next step given messy context and a list of tools. That’s why “which model is smartest” is close to the wrong question, and why a leaderboard rank doesn’t tell you much about how something behaves on turn nine of a trajectory.

What actually predicts agent performance

Benchmark leaderboards are a weak predictor of how a model behaves in a loop. The properties that matter are narrower and less glamorous:

Tool-calling reliability. Does it emit well-formed calls with the right argument names, every time? A model that gets this right 97% of the time sounds excellent and is miserable in practice, because a six-step trajectory has an 83% chance of completing cleanly and you will spend your life debugging the other 17%.

Instruction adherence under pressure. Your system prompt says “always look up the order before stating its status.” Does it still obey that on turn nine, with eight thousand tokens of accumulated observations in front of it? Many models obey instructions beautifully at turn one and drift by turn six.

Knowing when to stop. Underconfident models call the same tool repeatedly, seeking a certainty that is not coming. Overconfident models answer after one lookup when they needed three. Both are expensive; the second is dangerous.

Recovery from bad observations. You hand it ERROR: no tool named 'lookup_parcel'. Does it read that, pick a real tool, and proceed? Or does it call the imaginary tool again? You will test this directly in the next chapter.

The way to evaluate these is not to read a leaderboard. It is to take twenty real requests from your own domain, run them through two or three candidate models, and read the traces. If your agent writes code, test it against your codebase. If it processes claims, test it on your document formats. Generic benchmarks measure generic ability; you are shipping something specific.

Saying it out loud. What predicts agent performance is narrower and less glamorous than benchmark scores: tool-calling reliability, instruction adherence deep into a trajectory, knowing when to stop, and recovering from a bad observation. The number I’d lead with is that a model which emits well-formed tool calls 97% of the time sounds excellent but gives a six-step trajectory only about an 83% chance of completing cleanly — and you’ll spend your life on that other 17%. Lots of models follow the system prompt beautifully at turn one and drift by turn six. So the way to evaluate isn’t reading a leaderboard, it’s taking twenty real requests from your own domain, running them through two or three candidates, and reading the traces.

Model choice as an architectural decision

You do not have to pick one. A mature agent architecture routes.

Use a strong, expensive model for the hard parts — the initial plan, the ambiguous judgment call, the final synthesis. Route the high-volume, low-difficulty work — classifying intent, summarizing an observation, deciding whether a document is relevant — to something small and fast. In the Claude family that is roughly Opus or Sonnet for the former and Haiku for the latter; every provider has an equivalent tier structure.

The economics are stark. If nine out of ten calls in your trajectory are “summarize this tool output,” routing those to a model that costs a tenth as much and responds in a third the time changes the entire viability of the product.

There is a corresponding decision about multimodal input. If your agent needs to handle images or audio, you can either use a natively multimodal model — one call, the model sees the image directly — or convert to text first with a dedicated vision or speech API and reason over the text. The native path is simpler and usually better. The conversion path lets you pick a best-of-breed component for each modality and swap them independently, at the cost of an extra hop and a lossy transformation. Start native; split it out only when you can name the specific quality problem that forces the split.

Saying it out loud. You don’t have to pick one model — a mature architecture routes. Expensive model for the hard parts, the initial plan, the ambiguous judgment, the final synthesis; small fast model for the high-volume grunt work like classifying intent or summarizing a tool output. The economics are stark: if nine of ten calls in a trajectory are “summarize this,” moving those to something that costs a tenth as much and answers in a third the time can be the difference between a viable product and one you can’t afford to run. Same logic on multimodal — start with a natively multimodal model and only split out a dedicated vision or speech step when you can name the specific quality problem forcing it, because that path adds a hop and a lossy conversion.

Plan for replacement

The model you choose today will be obsolete within a year. That is not pessimism, it is the observed rate.

The engineering implication is concrete: do not let model-specific behavior leak into your agent’s structure. Put the model behind a thin interface — one function that takes system instructions, messages, and tool specs, and returns a decision. You will build exactly that interface in the next chapter, and it is the reason the code there can run against a mock or a live API by swapping one object.

Then build an evaluation set, so that “should we upgrade” is a fifteen-minute question with a numerical answer rather than a quarter-long project.

Saying it out loud. Whatever model you pick today is obsolete within a year — that’s the observed rate, not pessimism. So the engineering rule is: don’t let model-specific behavior leak into the structure of your agent. Put it behind a thin interface — one function that takes instructions, messages, and tool specs and returns a decision — so swapping providers is swapping one object, and you can run the same loop against a mock in tests. Then build an eval set, so “should we upgrade” is a fifteen-minute question with a number attached instead of a quarter-long project.

Tools: the hands

Tools connect the model’s reasoning to reality. They are how it learns things that are not in its weights and how it causes things to happen.

A tool has three parts, and getting all three right is most of the work:

A contract — a name, a description, and a schema for its inputs, written for a model to read.
An implementation — the actual function, which runs on your infrastructure under your control.
An observation — whatever comes back, rendered as text the model can reason over.

The contract is the part engineers under-invest in. Your tool description is not documentation, it is prompt. It is loaded into the model’s context on every single call, and it is the only thing telling the model when this tool is the right choice.

Compare:

# Bad: says what it is, not when to use it
"description": "Gets order data."

# Good: says when, and what comes back
"description": (
    "Look up an order by its ID. Use this before making any statement about "
    "an order's contents or status. Returns a JSON record including the "
    "customer name, items, and the carrier tracking number. Returns a "
    "not-found message if the ID doesn't exist."
)

The second version prevents a whole class of failure — the model answering from memory instead of looking up — without a word of code. When your agent misbehaves, fixing the tool description is frequently the cheapest available repair.

Saying it out loud. A tool has three parts: a contract the model reads, an implementation that runs on your infrastructure, and an observation that comes back as text it can reason over. The part engineers under-invest in is the contract, because a tool description isn’t documentation — it’s prompt. It’s loaded into context on every call and it’s the only thing telling the model when this tool is the right choice. “Gets order data” tells it what the tool is; “look up an order by ID, use this before making any statement about an order’s status, returns a JSON record with the tracking number” tells it when. That second version kills a whole class of failure — answering from memory instead of looking it up — without a line of code, which is why editing the description is often the cheapest repair available.

The distinction that matters: retrieval vs action

Split your tools into two categories and treat them completely differently.

Retrieval tools read the world. Search, database queries, RAG over your document store, natural-language-to-SQL, fetching a record. They ground the agent in fact — the single most effective countermeasure to hallucination is making the agent look things up before it speaks.

Retrieval tools are safe to call speculatively. If the model calls one unnecessarily you have wasted a few hundred milliseconds and some tokens. Nobody gets paged. This means you can be generous with them: give the agent good search, let it retry, let it explore.

Action tools change the world. Send the email, issue the refund, update the CRM record, delete the file, execute the code, place the order.

These are categorically different, and the difference is that they are not idempotent and frequently not reversible. A retrieval tool called twice returns the same answer twice. An action tool called twice sends two emails. Agents call tools twice more often than you would like — a confused model that does not recognize an observation as a success will happily retry.

The practical rules I would enforce on any production system:

Every action tool is marked as such in the registry, and the orchestration layer knows which is which.
Every action tool validates its own inputs against policy in code, before doing anything. Not in the prompt. The model’s judgment is an input to the decision, never the decision. A prompt saying “never refund more than $500” is a suggestion; an if amount > 500: raise is a rule.
High-consequence action tools pause for a human. This is itself implementable as a tool — ask_for_confirmation(summary) — which suspends the agent, surfaces the proposal to a person through whatever channel you have, and resumes on approval. That is the human-in-the-loop pattern, and it is what makes an agent shippable in a regulated context.
Action tools are idempotent where you can manage it. Accept a client-supplied idempotency key so a duplicate call is a no-op rather than a duplicate side effect.

The reason this distinction is architectural rather than a style preference: it is what lets you give an agent broad autonomy safely. Wide latitude to read, narrow and gated latitude to write. That asymmetry is the single most useful safety property you can build in, and it costs almost nothing.

Saying it out loud. Split your tools into two buckets and treat them completely differently. Retrieval tools read the world, and they’re safe to call speculatively — a wasted call costs you a few hundred milliseconds and some tokens, nobody gets paged — so be generous, let the agent search and retry, because making it look things up is the single best countermeasure to hallucination. Action tools change the world, and the difference that matters is that they’re not idempotent and often not reversible: a retrieval tool called twice returns the same answer, an action tool called twice sends two emails. And agents do retry, because a confused model that doesn’t recognize a success will happily call again. So action tools validate against policy in code, take an idempotency key, and pause for a human when the stakes are high. Wide latitude to read, narrow gated latitude to write — that asymmetry is the cheapest safety property you can build in.

Function calling: how the wiring works

The mechanism connecting the model to your Python function is function calling, and it is simpler than it looks.

You send the model a list of tool specifications alongside your messages. Each spec is a name, a description, and a JSON Schema for the parameters. The model, instead of returning text, can return a structured request: call find_order with {"order_id": "12345"}.

Here is the critical part that trips people up: the model does not execute anything. It emits a request. Your code decides whether to honor it, runs the function, and sends the result back as another message. Every single tool execution passes through code you wrote, which is exactly where your guardrails belong.

Two standards are worth knowing by name. OpenAPI specifications are the long-standing way to describe an HTTP API in a machine-readable contract, and they translate almost directly into tool schemas — if your internal services have OpenAPI specs, you have most of a toolset already. MCP, the Model Context Protocol, is the newer open standard for tool discovery and connection; it lets an agent connect to a tool server and find out what is available at runtime rather than having everything hardcoded (https://modelcontextprotocol.io/).

Some models also ship native tools — search, code execution — where the provider runs the tool inside the inference call and you never see the round trip. Convenient, and worth using, but note the trade-off: you cannot inspect or gate what you cannot see.

Saying it out loud. Function calling is simpler than it sounds. You send the model your messages plus a list of tool specs — name, description, JSON Schema for the parameters — and instead of returning text it can return a structured request like call find_order with this order ID. The part that trips people up is that the model never executes anything. It emits a request; your code decides whether to honor it, runs the function, and feeds the result back as another message. Every tool execution passes through code you wrote, which is exactly where your guardrails belong. Worth knowing two names too: OpenAPI specs translate almost directly into tool schemas, and MCP is the open standard for discovering tools at runtime. The tradeoff with provider-native tools, where the search or code execution happens inside the inference call, is that you can’t inspect or gate what you can’t see.

The orchestration layer: the loop, the state, the limits

If the model reasons and the tools act, the orchestration layer is the code that makes them a system. It is the part you write, and it is where your engineering judgment lives.

It has three responsibilities.

Running the loop. Assemble a context, call the model, inspect what came back. If it is a tool request, execute it and append the observation. If it is a final answer, return it. Repeat. That is the ReAct pattern — reason and act, interleaved — and you will write it in the next chapter.

Holding the state. The message list is the agent’s working memory. Everything the agent knows about the current task lives there: the mission, every thought, every tool call, every observation. Managing that list well is the craft. Let it grow unbounded and you pay linearly-growing cost per step and eventually blow the context window. Trim it carelessly and you drop the fact the agent needed.

Enforcing the limits. This is the part beginners omit and then learn about painfully.

Step cap. A hard maximum on loop iterations. Non-negotiable. Without it, a confused agent runs until something else breaks.
Token or cost budget. Track spend per run and abort past a threshold. Trajectory cost grows superlinearly because every step resends the accumulated history.
Wall-clock timeout. Some tools hang. The loop must not.
Repetition detection. If the agent has called the same tool with the same arguments three times, it is stuck. Notice, and intervene.

Notice that all four of these are deterministic, hard-coded controls sitting outside the model’s reasoning. That is deliberate. Anything the model can talk itself out of is not a limit.

A production orchestration layer also emits a trace — a structured record of every step, every prompt, every tool call, every observation, with timing and token counts. You cannot set a breakpoint inside a model’s reasoning. The trace is your debugger, and building it in from day one is much easier than retrofitting it later.

Saying it out loud. The orchestration layer is the code you actually write, and it does three things. It runs the loop — assemble context, call the model, execute a tool if one was requested, append the observation, repeat. It holds the state, which is really the message list, and managing that list well is the craft: let it grow unbounded and cost per step grows with it until you blow the window; trim it carelessly and you drop the fact the agent needed. And it enforces the limits — step cap, cost budget, wall-clock timeout, and repetition detection when the same tool gets called with the same arguments three times. All four are deterministic checks sitting outside the model’s reasoning, and that’s deliberate: anything the model can talk itself out of isn’t a limit. It also emits the trace, which is your only debugger, because you can’t set a breakpoint inside a model’s reasoning.

Design choice 1: instruct with persona and domain knowledge

The system prompt is your highest-leverage lever, and it is not a command. It is the agent’s constitution — the standing rules that apply to every decision it makes.

A weak system prompt says what the agent is. A strong one says how it behaves in the situations you know are coming.

Things that belong in it:

Identity and scope. Who this agent is and what it does not do. “You are a support agent for Solaris Audio. You handle order status, returns, and product questions. For billing disputes, hand off to a human.”

Tool usage policy. Not just which tools exist — the schemas cover that — but when. “Always look up an order before making any statement about it. Never state a delivery date you have not retrieved.”

Output contract. Format, length, tone. If a downstream system parses the output, specify the schema exactly.

Hard constraints, phrased as refusals. “Never quote a price you have not retrieved from the pricing tool. If a customer asks for a discount, say you’ll escalate rather than offering one.”

Two or three worked examples. This is the highest-value content per token in the entire prompt. A short example of a tricky case — a customer asking about an order that does not exist, an ambiguous request — teaches behavior that paragraphs of instruction do not.

Persona is not decoration. “You are a meticulous claims processor who never approves anything without documentation” measurably changes tool-calling behavior, because it gives the model a coherent stance to reason from rather than a list of rules to satisfy independently.

The one thing to remember: the system prompt is not a security boundary. Anything you write there can be argued with, and content that arrives through a tool observation can do the arguing. Rules you actually need enforced go in code.

Saying it out loud. The system prompt is your highest-leverage lever, and it’s a constitution rather than a command — standing rules that apply to every decision. A weak one says what the agent is; a strong one says how it behaves in the situations you already know are coming. So: identity and scope, when to use which tool, the output contract, hard constraints phrased as refusals, and two or three worked examples of tricky cases, which are the highest-value tokens in the whole prompt. Persona isn’t decoration either — “a meticulous claims processor who never approves anything without documentation” measurably changes tool-calling behavior, because it gives the model a stance to reason from instead of a checklist. The thing I’d end on is the failure mode: the system prompt is not a security boundary. Anything you write there can be argued with, and text arriving through a tool observation is what does the arguing.

Design choice 2: augment with context

The agent’s “memory” is not a database. It is whatever your orchestration layer put into the context window on this particular call.

Splitting it into two kinds clarifies almost everything.

Short-term memory is the running scratchpad for the current task: the mission, the accumulated thought/action/observation triples, the conversation so far. It lives in your message list and dies when the task ends. It is what the model needs to decide the next step.

The engineering problem is size. By step ten of a research trajectory you may be carrying tens of thousands of tokens of raw tool output, most of it irrelevant to the current decision, all of it being paid for on every subsequent call. The three techniques that work:

Summarize as you go. After a tool returns a large observation, run a cheap model call that distills it, and carry the distillation forward. This is Level 2’s “context engineering” made concrete.
Window with pinning. Keep the most recent N exchanges verbatim, plus a permanently pinned summary of everything older, plus the mission itself which never gets trimmed.
Externalize. Write the big artifact to a file or a store and carry a reference. The agent can re-read it with a tool if it needs to. Do not carry a 40KB document in context on the chance it becomes relevant.

Long-term memory persists across sessions: the user’s preferences, what happened last time, an outcome from three weeks ago.

Architecturally, long-term memory is almost always just another retrieval tool — a vector store or a search index the agent can query, plus a pre-fetch step that pulls the obviously-relevant facts into context before reasoning starts. That is the whole trick. Once you see memory as a tool, you already know how to build it.

The two access patterns are worth naming separately. Pre-fetch puts likely-relevant memory into context automatically at the start — the user’s name, their tier, their open tickets. Active recall gives the agent a search_memory tool it can call when it realizes it needs something. Production systems use both, because pre-fetch handles the predictable and active recall handles everything else.

Saying it out loud. An agent’s memory isn’t a database — it’s whatever your orchestration layer decided to put in the context window on this call. Short-term is the scratchpad for the current task and it dies when the task ends; the engineering problem there is size, because by step ten you’re carrying tens of thousands of tokens of raw tool output and paying for all of it on every subsequent call. Three things fix that: summarize each big observation with a cheap model call and carry the distillation, window the recent turns while pinning a summary and the original mission, and externalize big artifacts to a store so you carry a reference instead of 40KB of document. Long-term memory is almost always just another retrieval tool, and once you see it that way you already know how to build it. Use both access patterns — pre-fetch for the predictable stuff like the user’s tier, active recall for everything else.

Putting it together

Here is the whole architecture in one paragraph, which is worth being able to recite.

The orchestration layer assembles a context — system instructions, mission, memory, history, tool specs — and hands it to the model. The model returns either a final answer or a request to call a tool. If it is a tool request, the orchestration layer validates it, executes it under whatever guardrails apply, formats the result as an observation, appends it to the state, and loops. Every iteration checks the step cap, the budget, and the clock. Every iteration emits a trace record. The loop exits on a final answer, an exhausted limit, or a human intervention.

That is an agent. Everything else in this book is a refinement of that paragraph.

In the next chapter you write it.

Saying it out loud. If I had to give the whole architecture in one breath: the orchestration layer assembles a context — system instructions, the mission, memory, history, tool specs — and hands it to the model. The model returns either a final answer or a request to call a tool. If it’s a tool request, your code validates it, executes it under whatever guardrails apply, formats the result as an observation, appends it, and loops. Every iteration checks the step cap, the budget, and the clock, and every iteration emits a trace record. The loop exits on a final answer, an exhausted limit, or a human stepping in. That’s an agent — everything else is a refinement of that paragraph.

What you should be able to do now

Decompose any agent — yours or someone else’s — into model, tools, and orchestration, and say which component a given bug belongs to.
Choose and justify a model for an agent based on tool-calling reliability, instruction adherence, and stopping behavior rather than benchmark scores, and design a routing strategy that sends cheap work to a cheap model.
Classify every tool in a system as retrieval or action, and apply the right safety posture to each — generous latitude for reads, code-enforced validation and human gates for writes.
Write a tool description that steers model behavior, and explain why the description is prompt rather than documentation.
List the four hard limits every orchestration loop needs (step cap, cost budget, timeout, repetition detection) and explain why each must live outside the model’s reasoning.
Design a context strategy: what gets pre-fetched, what gets summarized as the trajectory grows, what gets externalized, and what is available through active recall.

Production Agent Engineering — A Practical Guide