Context engineering: the discipline of what goes in the window

Prompt engineering was a real skill and it is not the skill you need anymore.

Prompt engineering is about crafting one excellent block of text — usually the system instructions — and iterating on its wording until the model behaves. It is mostly static. You write it once, you tune it, you ship it, and it is the same on every request.

That works beautifully for a single-turn assistant and falls apart the moment you build an agent. An agent’s most important input is not the prompt you wrote in November. It is the twenty-three thousand tokens of accumulated history, tool output, retrieved documents, and remembered user facts that your orchestration layer assembled about four milliseconds ago, dynamically, for this request only.

Context engineering is the practice of assembling that payload.

It is a different discipline with different failure modes. Prompt engineering fails by being vague. Context engineering fails by including the wrong things, in the wrong order, at the wrong length, and it fails silently — the model produces something plausible, you have no red squiggly line, and you find out three weeks later that the agent has been confidently quoting a stale memory.

There is a cooking analogy that the Google whitepaper uses and I have never found a better one. Prompt engineering is writing the recipe. Context engineering is the mise en place — having every ingredient prepped, measured, and within reach before the pan gets hot. A great recipe with the wrong ingredients on the counter still produces a bad dinner.

Saying it out loud. Prompt engineering is writing one excellent block of static text and tuning the wording until the model behaves. Context engineering is assembling the whole payload, dynamically, for this one request — and in an agent that payload is the thing that matters, because the most important input isn’t the system prompt you wrote in November, it’s the twenty-odd thousand tokens of history, tool output, retrieved documents, and remembered user facts your orchestration layer built four milliseconds ago. The failure modes are different too: prompt engineering fails by being vague, and you can see that. Context engineering fails by including the wrong things at the wrong length, and it fails silently — no red squiggle, just an agent confidently quoting a stale memory, which you find out about three weeks later.

The budget framing

Here is the mental model I want you to carry through the rest of this part.

Every token in the context window is a token you paid for, waited for, and spent attention on. They all compete.

Not “compete” in a hand-wavy sense. Compete in three concrete, measurable ways.

Money. Providers bill per input token, and an agent resends its whole accumulated context on every step of its loop. If a trajectory takes eight tool calls, the observation from step one is billed eight times. This is why a tool that returns 30 KB of JSON is not a 30 KB problem — it is a 30 KB × remaining-turns problem, which is a point worth carrying over from Part 2.

Time. Time-to-first-token grows with input length. Users notice.

Attention. This is the one people underweight. A model’s ability to locate and use a specific instruction degrades as the surrounding text grows. The whitepaper calls this context rot: the phenomenon where a model’s attention to critical information diminishes as context expands. It is not a hard cliff, it is a gradient, and it starts well below the advertised window size.

The practical consequence: a bigger context window is a budget increase, not a solution. Treat context the way you would treat a latency budget or a memory budget in an embedded system. Know roughly what each component costs. Know which components you can cut. Know what breaks when you cut them.

A concrete exercise, and I mean actually do this on your own agent: instrument your orchestration layer to log the token count of each context component on every model call.

def context_budget(payload: dict) -> dict:
    """Rough per-component token accounting. Log this on every model call."""
    def approx_tokens(x) -> int:
        # Good enough for budgeting: ~4 chars per token for English text.
        # Swap in a real tokenizer (tiktoken, the provider's count-tokens
        # endpoint) when you need accuracy rather than proportions.
        return len(json.dumps(x)) // 4

    return {
        "system_instructions": approx_tokens(payload["system"]),
        "tool_schemas":        approx_tokens(payload["tools"]),
        "memories":            approx_tokens(payload.get("memories", [])),
        "retrieved_docs":      approx_tokens(payload.get("docs", [])),
        "history":             approx_tokens(payload["messages"][:-1]),
        "user_message":        approx_tokens(payload["messages"][-1]),
    }

The first time you run this on a real agent, the number that shocks you will be history or tool_schemas, and it will not be the one you expected. I have seen production agents spending 40% of every request on tool schemas for tools the agent never calls.

Saying it out loud. The framing I carry is that every token in the window is one you paid for, waited for, and spent attention on — and they all compete. Money, because providers bill per input token and the agent resends its whole context every step, so an observation from step one gets billed eight times over an eight-call trajectory. Time, because time-to-first-token grows with input length. And attention, which is the one people underweight: a model’s ability to find and use a specific instruction degrades as the surrounding text grows. That’s context rot, and it’s a gradient that starts well below the advertised window size, not a cliff at the limit. So a bigger window is a budget increase, not a solution. If you instrument per-component token counts, the number that shocks you usually isn’t the one you expected — I’ve seen production agents spending 40% of every request on schemas for tools they never call.

What actually occupies the window

The whitepaper offers a taxonomy that is worth internalizing, because it groups context by function rather than by where it came from. Three groups.

Context that guides reasoning

This tells the model how to think and what it is allowed to do.

System instructions — persona, capabilities, constraints, output contract. Written by you, static or semi-static.
Tool definitions — the name, description, and JSON Schema for every tool you expose. Written by you, but their size is often a surprise.
Few-shot examples — demonstrations that teach behavior through in-context learning rather than instruction.

The interesting move here, and one that most people never make: few-shot examples do not have to be static. Hardcoding three examples means every request pays for all three, and at most one of them is relevant. Selecting two examples from a library of forty, based on similarity to the current request, costs the same number of tokens and works considerably better.

Evidential and factual data

This is what the model reasons over — the evidence.

Long-term memory — persisted knowledge about this user, gathered across sessions. Chapters 4 and 5.
External knowledge — documents and records retrieved from a knowledge base, typically via RAG.
Tool outputs — whatever your tools returned.
Sub-agent outputs — conclusions handed back by specialized agents you delegated to.
Artifacts — non-textual data: files, images, audio associated with the user or session.

Immediate conversational information

This grounds the model in the task at hand.

Conversation history — the turn-by-turn record.
State / scratchpad — structured working data for this conversation. The shopping cart. The draft. The list of files already reviewed.
The user’s prompt — the immediate query.

Saying it out loud. It helps to group what’s in the window by function rather than by where it came from. There’s context that guides reasoning — system instructions, tool definitions, few-shot examples. There’s evidential data the model reasons over — long-term memory, retrieved documents, tool outputs, sub-agent conclusions. And there’s the immediate conversation — history, scratchpad state, the user’s actual question. The move most people never make is on the few-shot examples: they don’t have to be static. Hardcoding three means every request pays for all three and at most one is relevant, whereas selecting two from a library of forty by similarity to the current request costs the same tokens and works considerably better.

Which of these you actually control

Sort the list by how much leverage you have, because that is what determines where to spend engineering effort.

Total control, low effort: system instructions, tool definitions, few-shot examples. You write these. If they are bloated it is because you have not looked at them recently. Auditing your tool schemas and deleting the four tools nobody calls is the cheapest context win available and takes an afternoon.

Total control, real effort: memories, retrieved documents, state. You decide what to fetch, how many, how to rank them, and how to format them. This is where most of the actual work in this part lives.

Partial control: tool outputs. The tool decides what it returns, but you wrote the tool. Truncation, summarization, and returning a handle instead of a payload are all your decisions. This is the single largest source of accidental context bloat in real systems.

Least control: the user’s message and the conversation history. You cannot stop a user from pasting a 12,000-token log file. You can decide what happens to it afterwards — which is the subject of Chapter 3.

Saying it out loud. Sort those components by how much leverage you actually have, because that’s where the engineering effort should go. System instructions, tool definitions, and examples are total control at low effort — if they’re bloated it’s because nobody has looked recently, and deleting the four tools nobody calls is the cheapest context win available and takes an afternoon. Memories, retrieved documents, and state are total control at real effort, and that’s where most of the work lives. Tool outputs are partial control, but remember you wrote the tool, so truncation and returning a handle are your decisions — and this is the single biggest source of accidental bloat in real systems. The only thing you genuinely don’t control is the user pasting a 12,000-token log file, and even then you control what happens to it next.

The loop that never changes

Every turn of every agent runs the same four-phase cycle. Once you see it, you will see it in every framework.

1. Fetch context. Retrieve what might be relevant: memories for this user, RAG documents matching the query, recent session events. Dynamic retrieval uses the user’s message and metadata to decide what to pull.

2. Prepare context. Assemble the final payload. This step is blocking and on the hot path — the model call cannot start until the payload is ready. Every millisecond you spend here is a millisecond of user-visible latency, which is why aggressive retrieval strategies with reranking are so often the wrong choice for interactive agents.

3. Invoke model and tools. The ReAct loop from Part 1. Model output and tool results append to the context as you go.

4. Upload context. Persist what the turn produced: append events to the session, push the transcript to the memory manager. This step should be non-blocking — the user already has their answer, and there is no reason to make them wait while an LLM extracts memories in the background.

Two of those four phases are where all the interesting engineering is. Phase 1 decides what is available. Phase 2 decides what makes the cut.

Saying it out loud. Every turn of every agent runs the same four phases: fetch context, prepare context, invoke the model and tools, then upload what the turn produced. The two that matter are the first two — fetch decides what’s available, prepare decides what makes the cut. The performance detail worth naming is that prepare is blocking and on the hot path: the model call can’t start until the payload is assembled, so every millisecond of clever retrieval and reranking is user-visible latency. That’s exactly why aggressive multi-stage retrieval is so often the wrong choice for an interactive agent. And the last phase, persisting events and extracting memories, should be non-blocking — the user already has their answer, so don’t make them wait while an LLM writes memories in the background.

Structured outputs are a context tool

Here is a technique that people file under “output formatting” and should file under “context engineering,” because its main value is on the input side of the next call.

If your model returns free-form prose, you have to either keep that prose verbatim in the history or parse it with something fragile. If your model returns a validated object, you can store six fields instead of six hundred tokens, and you can reconstruct exactly what you need on the next turn.

Structured output is the mechanism. You give the provider a JSON Schema — in Python, almost always generated from a Pydantic model — and the provider constrains generation so the response conforms to it. Not “asks nicely.” Constrains.

All three major providers support this today, with slightly different call shapes.

from pydantic import BaseModel, Field
from typing import Literal

class ExtractedFact(BaseModel):
    """One durable fact worth remembering about the user."""
    fact: str = Field(description="Stated in third person, e.g. 'The user prefers window seats.'")
    category: Literal["preference", "identity", "goal", "constraint"]
    confidence: float = Field(ge=0.0, le=1.0)

class Extraction(BaseModel):
    facts: list[ExtractedFact]

OpenAI, using the Responses API — the schema goes in text_format:

from openai import OpenAI
client = OpenAI()

response = client.responses.parse(
    model="gpt-5.6",
    input=[
        {"role": "system", "content": "Extract durable facts about the user."},
        {"role": "user", "content": transcript},
    ],
    text_format=Extraction,
)
extraction = response.output_parsed          # a validated Extraction instance

On the older Chat Completions API the same thing is client.chat.completions.parse(..., response_format=Extraction), and the result is on completion.choices[0].message.parsed. Both are current; the Responses API is the one OpenAI is building on.

Gemini, where the schema goes in the generation config and the parsed object comes back on .parsed:

from google import genai
client = genai.Client()

response = client.models.generate_content(
    model="gemini-3-flash",
    contents=transcript,
    config={
        "response_mime_type": "application/json",
        "response_schema": Extraction,
    },
)
extraction = response.parsed

Claude, where you get the same effect through a tool definition — you declare a tool whose input schema is your Pydantic model, and force the model to call it:

import anthropic
client = anthropic.Anthropic()

msg = client.messages.create(
    model="claude-opus-5",
    max_tokens=2048,
    tools=[{
        "name": "record_extraction",
        "description": "Record the durable facts extracted from the conversation.",
        "input_schema": Extraction.model_json_schema(),
    }],
    tool_choice={"type": "tool", "name": "record_extraction"},
    messages=[{"role": "user", "content": transcript}],
)
extraction = Extraction.model_validate(msg.content[0].input)

Three things to notice, because they generalize.

The description on each field is prompt. Same lesson as tool descriptions in Part 2. Field(description="Stated in third person...") measurably changes what comes back. An undocumented field is a field the model will fill in a format you did not want.

Literal and numeric bounds are free accuracy. Enumerating the allowed categories eliminates an entire class of “the model invented a new category” bug, at zero token cost relative to describing them in prose.

Validation is not optional even with constrained decoding. The provider guarantees the output is syntactically valid against the schema. It guarantees nothing about whether confidence: 0.99 is honest. Structural validity is a floor, not a ceiling — run your semantic checks anyway.

You will use this exact pattern in Chapter 6 to turn a conversation into memory records.

Saying it out loud. People file structured output under formatting; it belongs under context engineering, because its real value is on the input side of the next call. If the model returns prose, you either keep the prose verbatim in history or parse it with something fragile. If it returns a validated object, you store six fields instead of six hundred tokens. The mechanism is that you hand the provider a JSON Schema — usually generated from a Pydantic model — and it constrains generation, it doesn’t just ask nicely. Three things generalize: the description on each field is prompt, so an undocumented field gets filled in a format you didn’t want; enum and numeric bounds are free accuracy, killing the invented-category bug at zero token cost; and validation is still not optional, because the provider guarantees the output is syntactically valid against the schema and guarantees nothing about whether a confidence of 0.99 is honest.

The three levers

Every context engineering technique in the rest of this part is one of three things. When you are stuck, walk the list.

Selection — send less by choosing better

Do not include everything you have. Include what is relevant to this decision.

Retrieve the top 5 memories, not all 200. Include the two few-shot examples that resemble this request. Register only the tools this agent role can actually use, rather than the union of every tool in your system. Drop tool results from ten steps ago that have already been acted on.

Selection is almost always the first lever to reach for, because it is lossless with respect to what you keep — you are not degrading anything, you are just not sending the irrelevant parts. Its cost is the risk of selecting wrongly, and the latency of whatever ranking you do.

Saying it out loud. Selection is sending less by choosing better — top five memories instead of all two hundred, the two examples that resemble this request, only the tools this agent role can actually use, dropping tool results from ten steps ago that have already been acted on. It’s the first lever I reach for because it’s lossless with respect to what you keep: you’re not degrading anything, you’re just not sending the irrelevant parts. What it costs you is the risk of selecting wrongly — a dropped memory that mattered — plus the latency of whatever ranking you do to make the choice.

Compression — send less by shrinking

Represent the same information in fewer tokens.

Summarize the first forty turns into a paragraph. Replace a raw tool result with the three fields the model actually needs. Store a structured profile instead of a transcript. Extract memories rather than carrying the conversation.

Compression is lossy by definition, and the entire craft is controlling what you lose. The version of this that works in production is not “summarize the history” — it is “summarize the history while guaranteeing that these seven fields survive verbatim.” You will build exactly that in Chapter 3.

Saying it out loud. Compression is sending less by shrinking — summarize the first forty turns into a paragraph, replace a raw tool result with the three fields that matter, store a structured profile instead of a transcript. The thing to be honest about is that compression is lossy by definition, and the entire craft is controlling what you lose. The version that works in production isn’t “summarize the history,” it’s “summarize the history while guaranteeing these seven fields survive verbatim” — the order ID, the amount, the deadline, whatever would be catastrophic to paraphrase. Unconstrained summarization is where agents quietly lose the one number the whole task depended on.

Isolation — send less by splitting the work

Give a different context window to a different piece of work.

Delegate the document-analysis subtask to a sub-agent that gets its own clean window, and let it return a two-paragraph conclusion rather than dumping forty pages into the parent’s history. Write a large artifact to a file and pass a path. Run the extraction step in a separate LLM call with its own focused prompt instead of asking one call to do the task and remember the facts.

Isolation is the most powerful lever and the most expensive in complexity. It buys you clean windows and parallelism at the cost of coordination — the sub-agent does not know what the parent knows, and deciding what to pass across the boundary is its own design problem.

Saying it out loud. Isolation is sending less by splitting the work — give a different piece of work its own clean context window. Hand the document analysis to a sub-agent that returns two paragraphs instead of dumping forty pages into the parent’s history; write the big artifact to a file and pass a path; run extraction as a separate focused call rather than asking one call to do the task and remember the facts. It’s the most powerful lever and the most expensive in complexity. The tradeoff is coordination: the sub-agent doesn’t know what the parent knows, and deciding what crosses that boundary is its own design problem — which is exactly where multi-agent systems most often fail.

Three levers. Selection, compression, isolation. Sessions (Chapter 2) and compaction (Chapter 3) are mostly selection and compression. Memory (Chapters 4–6) is compression plus selection, applied across time rather than within a conversation.

The habit

The habit I want you to leave this chapter with is small and it will change how you debug.

When your agent misbehaves, do not start by editing the prompt. Start by dumping the exact context that was sent on the call that went wrong. All of it. Every message, every schema, every injected memory, with token counts.

You will find, far more often than you expect, that the answer is sitting right there. The instruction you thought was in the system prompt was overwritten by a memory. The tool result the model needed was truncated at 500 characters. The fact it “hallucinated” was in the window, from a memory extracted six weeks ago that is no longer true.

The model is not mysterious. It read what you sent it. Your job is to know what you sent.

Saying it out loud. The habit I’d want someone to take away is about debugging. When your agent misbehaves, don’t start by editing the prompt — start by dumping the exact context that was sent on the call that went wrong. All of it, every message, every schema, every injected memory, with token counts. Far more often than you’d expect the answer is just sitting there: the instruction you thought was in the system prompt got overridden by a memory, the tool result got truncated at 500 characters, or the fact it “hallucinated” was genuinely in the window, from a memory extracted six weeks ago that isn’t true anymore. The model isn’t mysterious. It read what you sent it. Your job is to know what you sent.

What you should be able to do now

Explain why a larger context window does not solve the context problem, naming the three costs — money, latency, and context rot — and roughly when each starts to bite.
Break any agent’s context payload into the three functional groups (reasoning guidance, evidential data, immediate conversation) and identify which components you control and how tightly.
Instrument an agent to log per-component token counts on every model call, and read the result to find the largest avoidable consumer.
Define a Pydantic schema and get validated structured output from OpenAI, Gemini, or Claude, and explain why field descriptions and Literal types are doing prompt-engineering work.
Classify any proposed context optimization as selection, compression, or isolation, and state what it costs you.

Production Agent Engineering — A Practical Guide