Part 3 — Context Engineering
A language model has no memory.
That sentence sounds like a limitation you already knew about, and it is, but almost nobody takes it literally enough. The model does not remember the last thing it said to you. It does not remember your name, the tool it called four steps ago, or the fact that you told it twice already to stop suggesting the same thing. Every single call to a model is the first call it has ever seen.
Everything that looks like memory — the agent recalling your preference, picking up a task where it left off, knowing that step three failed — is an illusion produced by your code. Your code assembled a payload, put the relevant history and facts inside it, and shipped it off. The model read the payload, reasoned over exactly what was there, and emitted a response. Then it forgot everything again.
Context engineering is the discipline of deciding what goes in that payload.
It is the work that stands between a demo that impresses people for ten turns and a system that is still coherent on turn two hundred, or on Tuesday next week when the same user comes back.
Context is the scarcest resource you have
Every agent has a budget, and the budget is the context window.
It is finite, it is shared, and everything competes for it. Your system instructions compete with the tool schemas. The tool schemas compete with the twelve thousand tokens of JSON that the search tool returned on step four. That JSON competes with the user’s actual question. And the user’s actual question competes with the six retrieved memories you helpfully pre-fetched, three of which are stale.
The naive response is to buy a bigger window. Models now ship with context windows measured in hundreds of thousands of tokens, and it is tempting to conclude that the problem is solved. It is not, for three reasons that you will feel in production.
Cost scales with what you send. Providers bill per input token, and an agent re-sends its accumulated history on every step of the loop. A ten-step trajectory carrying fifty thousand tokens of history is not fifty thousand tokens; it is closer to half a million.
Latency scales with what you send. Time-to-first-token rises with input length. A user waiting four seconds for the agent to start typing does not care that you had budget headroom.
Quality degrades before the window fills. This is the one that surprises people. Models get measurably worse at finding the relevant fact as the surrounding text grows — an effect the Google whitepaper this part draws on calls context rot. A model that reliably follows an instruction at eight thousand tokens will start dropping it at eighty thousand, long before it hits any hard limit. More context is not more capability. Past a point it is less.
So the goal is not to fill the window. The goal is stated more precisely: give the model no more and no less than what it needs to make the next decision correctly.
The two systems this part is about
Almost all of the machinery of context engineering resolves into two components, and keeping them mentally separate will save you a lot of confusion.
A session is the container for one conversation. It holds the chronological record of what happened — user messages, model replies, tool calls, tool results — plus a small structured scratchpad of working state. It is scoped to one continuous interaction, it is on the hot path of every turn, and it dies or expires when the conversation ends. Think of it as the desk you are working at right now: covered in everything you need for this task, and messy in a way that is fine because it is temporary.
Memory is what survives. It is not the raw transcript. It is extracted information — facts, preferences, summaries, procedures — distilled from conversations and persisted so that the next conversation, next week, can start from somewhere other than zero. Think of it as the filing cabinet: you do not shove the whole desk into it, you go through the desk, throw away the drafts, and file the two documents that mattered.
Conflating these two is the single most common architectural mistake in this area. Sessions are verbatim, short-lived, and framework-specific. Memory is processed, long-lived, and deliberately framework-agnostic. They need different storage, different lifecycles, different privacy controls, and different code.
What this part covers
Chapter 1 — Context engineering: the discipline of what goes in the window. The budget framing, taken seriously. What actually occupies your context window, component by component, and which components you control. Structured outputs as a context tool, with the current APIs from three providers. Then the three levers you have — selection, compression, isolation — which every technique in the rest of this part is an instance of.
Chapter 2 — Sessions and state. What belongs in a session and what does not. Session state versus conversation history versus memory, made crisp enough that you can classify any piece of data in about five seconds. How the major frameworks disagree about all of this, and why that disagreement becomes a real problem the moment you have two agents from two frameworks that need to collaborate. Then the production checklist: persistence, ordering, concurrency, expiry, size limits, isolation, and PII. You build a session store with pluggable backends.
Chapter 3 — Managing long conversations. What specifically degrades as history grows, and the honest cost of every fix. Truncation, sliding windows, summarization and compaction, selective retrieval of past turns, and prompt caching — with the actual current pricing shape from the providers, because caching changes the arithmetic enough to change your architecture. When to compact, and when compaction is the wrong answer and you should end the session with a handoff instead. You build a compaction strategy that preserves a schema of critical fields, so that the lossy step stops losing the things you cannot afford to lose.
Chapter 4 — Memory systems: what the agent remembers between conversations. The conceptual chapter. Semantic, episodic, and procedural memory. Structured versus unstructured content. The three organization patterns — collections, structured profiles, rolling summaries — and when each one is right. Vector stores versus knowledge graphs. Explicit versus implicit creation, internal versus external management. Memory scope, which is the setting most likely to cause a data leak if you get it wrong. And a precise statement of how memory differs from RAG and from session state.
Chapter 5 — Memory generation, provenance, and retrieval. The mechanics. Extraction — turning noisy conversation into candidate memories — and consolidation, which is the hard part: merging duplicates, resolving contradictions, letting things decay. Provenance and lineage, which is where memory systems become trustworthy or do not. When to trigger generation, and why it belongs in the background. Retrieval scoring and timing. And the placement question at inference: memories in the system instructions behave differently from memories in the conversation history, and the difference is not subtle.
Chapter 6 — Mini-project 5: build a memory system. You build the whole thing. Extraction from a conversation, embedding-backed storage, consolidation that detects and resolves a genuine contradiction, scoped retrieval, provenance on every record, and injection into the next turn’s context. It runs offline with a deterministic mock embedder, so no API key is required. Then the same system rewritten against mem0 and ChromaDB, so you can go either way.
What you will have built by the end
A session store you can put behind any agent, with a swappable backend and the production concerns handled.
A compaction strategy that shrinks a conversation without dropping the six fields your business logic depends on.
A memory system with extraction, consolidation, provenance, and scoped retrieval — small enough to read in one sitting, and structurally identical to what the commercial memory managers are doing.
And the habit that matters more than any of them: when your agent does something stupid, your first question stops being “why did the model do that” and becomes “what exactly was in the window when it decided.”
That question has an answer. You can print it.
Start with Chapter 1.