Memory systems: what the agent remembers between conversations

Sessions handle the now. Memory handles everything else.

A memory is a snapshot of meaningful information extracted from a conversation or another data source, condensed into a form that is useful later, and persisted across sessions.

Read that definition again and notice what it does not say. It does not say “the conversation history.” A memory is not the transcript. It is what you concluded from the transcript.

This is the distinction the whole chapter rests on, and some frameworks blur it by calling the session “short-term memory.” For our purposes: a session is raw dialogue; a memory is extracted information. They have different lifetimes, different storage, different formats, and different failure modes, and the systems you build for them share almost no code.

The relationship between them is symbiotic and worth stating explicitly. Sessions are the primary source of memories — you mine conversations to produce them. And memories are one of the main strategies for keeping sessions small — a session-scoped memory can replace two hundred turns of transcript. Each one feeds the other.

Saying it out loud. Sessions handle the now; memory handles everything else. A memory is a snapshot of meaningful information pulled out of a conversation, condensed into something useful later, and persisted across sessions — and the important thing is what that doesn’t say. A memory is not the transcript. It’s what you concluded from the transcript. Some frameworks blur this by calling the session “short-term memory,” but they have different lifetimes, different storage, different formats, and different failure modes, and the systems you build for them share almost no code. The relationship is symbiotic: sessions are where memories come from, and memories are one of the main ways you keep sessions small, because one session-scoped memory can stand in for two hundred turns of transcript.

What memory buys you

Four capabilities, and it is worth being clear about which one you are actually building for, because they pull the design in different directions.

Personalization. The obvious one. Remembering that a user prefers window seats, supports a particular team, always wants their code in TypeScript. This is what makes an agent feel like it knows you rather than meeting you fresh every morning.

Context window management. A memory scoped to one session — “the user is booking New York to Paris, Nov 7–14, direct flights only, middle seat” — replaces a very long transcript with three lines. This is compaction by another name, which is why Chapter 3 and this chapter keep touching.

Data mining and insight. Aggregate memories across many users, privacy-preservingly, and you learn things about your product. Forty users this week generated a memory about the return policy on one specific item. That is a signal, and you would never have found it in the raw logs.

Self-improvement. The agent records which strategies and tool sequences led to good outcomes, and builds a playbook. This is procedural memory, covered at the end of this chapter, and it is the least developed area commercially and the most interesting.

Saying it out loud. Memory buys you four things, and it’s worth knowing which one you’re actually building for because they pull the design in different directions. Personalization is the obvious one — the agent knows you rather than meeting you fresh every morning. Context window management is the underrated one, because a memory scoped to one session replaces a very long transcript with three lines, which is just compaction wearing a different hat. Data mining is the one nobody plans for: aggregate memories across users and you learn that forty people this week hit the same return-policy confusion, which you’d never have found in raw logs. And self-improvement, where the agent builds a playbook of what worked, is the least commercially developed and the most interesting.

Memory vs RAG vs session state

Three systems that all “retrieve things and put them in the context window,” and people conflate them constantly. Here is the crisp version.

RAG makes the agent an expert on facts. Memory makes it an expert on the user. Session state tracks where we are right now.

The differences are architectural, not cosmetic:

	RAG	Memory	Session state
Purpose	Inject external factual knowledge	Personalize and persist across sessions	Track the current task
Source	A static, pre-indexed corpus — docs, wikis, PDFs	The dialogue between user and agent	The current conversation
Isolation	Usually shared and global, read-only	Almost always scoped per user	Scoped to one session
Certainty	Authoritative	Inferred, so inherently uncertain	Known
Write pattern	Batch, offline, administrative	Event-driven — per turn, per session, or agent-triggered	Every turn
Read pattern	As-a-tool, when the query needs facts	As-a-tool, or statically at turn start	Always
Format	Natural-language chunks	Natural-language snippets or structured profiles	Structured dict
Preparation	Chunking and indexing	Extraction and consolidation	None

The row that matters most is preparation.

RAG’s data preparation is chunking and indexing — mechanical, deterministic, and it does not care what the chunks say. Memory’s data preparation is extraction and consolidation — an LLM decides what is meaningful, then another LLM decides how it fits with what you already believe.

That is why a memory manager is not a vector database. It uses one, the way a web application uses Postgres. Its actual value is the active curation: deciding what to remember, noticing that a new fact contradicts an old one, and doing something sensible about it. If your “memory system” is a vector store you write every user message into, you have built a lossy search index over your logs, not a memory.

The analogy from the whitepaper is worth keeping. RAG is the research librarian in a large public library: expert on the world’s facts, knows nothing about you. Memory is the personal assistant with a private notebook: knows nothing about the world, expert on you. A serious agent needs both, and they are different hires.

Saying it out loud. The crisp version is: RAG makes the agent an expert on facts, memory makes it an expert on the user, and session state tracks where we are right now. The differences are architectural, not cosmetic — RAG is a shared, read-only, pre-indexed corpus that’s authoritative; memory is per-user, inferred, and therefore inherently uncertain. The row that matters most is data preparation. RAG’s prep is chunking and indexing: mechanical, deterministic, doesn’t care what the chunks say. Memory’s prep is extraction and consolidation, where one model decides what’s meaningful and another decides how it fits with what you already believe. That’s why a memory manager is not a vector database — it uses one the way a web app uses Postgres. If your memory system is a vector store you dump every user message into, you’ve built a lossy search index over your logs.

The anatomy of a memory

A single memory has two parts.

Content — the substance extracted from the source. It is deliberately framework-agnostic: simple structures any agent can ingest. It comes in two flavors:

Structured: a dict or JSON object with a schema you defined. {"seat_preference": "window"}. Precise, queryable, easy to validate, hard to extend to nuance.
Unstructured: a natural language sentence capturing the essence of something. "The user prefers a window seat." Flexible, expressive, harder to query exactly, and the thing you actually inject into a prompt.

Most systems use both — structured for the stable profile attributes, unstructured for everything else.

Metadata — context about the memory. A unique ID. The owner. Labels describing the content or its source. Timestamps. Confidence. Provenance.

Metadata is where beginners under-invest and it is what separates a memory store you can operate from one you cannot. Without an owner you cannot enforce isolation. Without a timestamp you cannot decay. Without provenance you cannot resolve a contradiction, and you cannot honor a deletion request.

And one universal rule, which is easy to state and easy to violate: memories are descriptive, not predictive. Record "The user asked about vegan options twice". Do not record "The user is vegan". The first is a fact. The second is an inference presented as a fact, and when it is wrong it is wrong confidently and permanently.

Saying it out loud. A memory has two parts: content and metadata. Content is either structured — a dict with a schema you defined, precise and queryable but hard to extend to nuance — or unstructured, a natural-language sentence that’s flexible and is the thing you actually inject into a prompt. Most systems use both. Metadata is where beginners under-invest, and it’s what separates a store you can operate from one you can’t: without an owner you can’t enforce isolation, without a timestamp you can’t decay anything, and without provenance you can’t resolve a contradiction or honor a deletion request. The one rule I’d hammer is that memories are descriptive, not predictive. Record that the user asked about vegan options twice; do not record that the user is vegan. The first is a fact, the second is an inference presented as a fact, and when it’s wrong it’s wrong confidently and permanently.

Types of information: declarative and procedural

From cognitive science, and it is a genuinely useful split.

Declarative memory is “knowing what.” Facts, figures, events — anything the agent can explicitly state. If the memory answers a what question, it is declarative.

It subdivides:

Semantic memory — general knowledge and stable facts. “The user’s company operates in the EU.” “Project Halyard is the mobile redesign.”
Episodic memory — specific events and their circumstances. “On March 3rd, the user reported that the export failed with a timeout, and we resolved it by increasing the batch limit.”

The distinction matters for retrieval. Semantic memories are broadly relevant and cheap to keep resident — they belong in a profile you always load. Episodic memories are relevant occasionally and expensive to keep resident — they belong in a searchable collection you query when something looks related.

Procedural memory is “knowing how.” Skills and workflows. The right sequence of tool calls to book a trip. The approach that works for debugging this class of failure. If the memory answers a how question, it is procedural.

Almost every commercial memory platform is built for declarative memory. Procedural memory is a different problem, and the end of this chapter says why.

Saying it out loud. Borrowing from cognitive science, declarative memory is knowing what and procedural memory is knowing how. Declarative splits again into semantic — stable general facts like which market the user’s company operates in — and episodic, specific events with their circumstances, like the export failing with a timeout on March 3rd and how it got fixed. That split isn’t academic, it drives retrieval: semantic memories are broadly relevant and cheap to keep resident, so they belong in a profile you always load, while episodic memories are only occasionally relevant and expensive to keep resident, so they belong in a searchable collection you query when something looks related. Procedural memory is the sequence of moves that works, and it’s worth knowing that essentially every commercial memory platform is built for declarative memory only.

Organization patterns

You have memories. How do you arrange them? Three patterns, and the choice determines how retrieval works.

Collections

A pool of self-contained natural-language memories for one user. Each one is a distinct observation, event, or summary. Several may relate to the same topic.

mem_01  "The user prefers window seats on flights over 3 hours."
mem_02  "The user travels to Berlin roughly monthly for work."
mem_03  "The user was frustrated by the March 3rd export timeout."
mem_04  "The user's team uses TypeScript, not JavaScript."

Retrieval is a search problem — semantic similarity over a large, loosely structured pool. Good for: episodic memory, open-ended domains, anything where you cannot enumerate the fields in advance. Bad at: guaranteeing that a specific fact is present. Search might not surface it.

Structured user profile

A set of core facts, like a contact card that keeps getting updated.

{
  "seat_preference": "window",
  "home_airport": "CDG",
  "dietary": ["no shellfish"],
  "preferred_language": "en",
  "tier": "gold"
}

Retrieval is a lookup, not a search. Fetch the profile, or one attribute of it. Fast and deterministic. Good for: stable, enumerable attributes. Preferences. Account details. Anything you want guaranteed present in every context. Bad at: anything not in the schema. Every new kind of fact is a migration.

Rolling summary

One single evolving document that summarizes the entire relationship. Instead of creating new memories, the manager continuously rewrites this one.

The user is a senior engineer at a Berlin fintech, working on a payments
migration. Prefers direct technical answers without preamble. Has raised
three support issues, all related to webhook delivery. Currently blocked on
a rate-limit question from March 3rd.

Retrieval is trivial — there is one document. Good for: session compaction, keeping a compact always-present picture, avoiding retrieval latency entirely. Bad at: precision and scale. Every update is a rewrite, so it is expensive to maintain and gets progressively lossier — the same recursive-summarization decay from Chapter 3.

Which to use

In practice: a structured profile for the things you must always have, plus a collection for everything else. The profile guarantees presence for the ten attributes you enumerated. The collection catches the long tail you could not enumerate. Rolling summaries are best treated as a session-compaction technique rather than a primary long-term store.

Saying it out loud. Three ways to arrange memories, and the choice decides how retrieval works. A collection is a pool of self-contained natural-language memories, so retrieval is a search problem — great for episodic stuff and open-ended domains you can’t enumerate, bad at guaranteeing a specific fact is present, because search might just not surface it. A structured profile is a contact card that keeps getting updated, so retrieval is a lookup rather than a search — fast, deterministic, guaranteed present, but every new kind of fact is a schema migration. A rolling summary is one evolving document, trivially retrieved and always present, but every update is a rewrite so it’s expensive and gets progressively lossier, which is the same recursive-summarization decay from the compaction chapter. In practice I’d run a structured profile for the ten things I must always have plus a collection for the long tail, and treat rolling summaries as a session-compaction technique rather than a long-term store.

Storage architectures

Two, plus the hybrid.

Vector databases. Memories become embedding vectors; retrieval finds the nearest neighbours to a query embedding. This is the common case, and it is right for unstructured natural-language memories where meaning matters more than exact wording. “What does this user like to eat” will find “The user avoids shellfish” without the word “shellfish” appearing in the query.

Its weakness is relational reasoning. Vector search finds things similar to your query. It cannot follow a chain — “who does this person report to, and what is that person’s team working on” is two hops, and similarity search does not do hops.

Knowledge graphs. Memories as entities (nodes) and relationships (edges), often as knowledge triples: (user, works_at, Acme), (Acme, headquartered_in, Berlin). Retrieval traverses. Multi-hop questions become graph queries, which is exactly what vector search cannot do.

Its weakness is that extraction is much harder — you have to identify entities, resolve them to existing nodes, and type the relationships — and fuzzy conceptual queries do not map well onto traversal.

Hybrid. Enrich the graph’s entities with embeddings, so you can do both: semantic search to find the entry point, then traversal to explore from there. More capable, more infrastructure. Zep and mem0’s graph mode are both in this territory.

Start with a vector store. Move to a graph when you can name a specific question your agent needs to answer that requires more than one hop. “It would be more sophisticated” is not that question.

Saying it out loud. Two storage architectures plus a hybrid. A vector database turns memories into embeddings and retrieves nearest neighbours, which is right for unstructured natural-language memories where meaning matters more than wording — asking what the user likes to eat will surface “avoids shellfish” without the word shellfish appearing anywhere in the query. Its weakness is relational reasoning: similarity search finds things like your query, it can’t follow a chain, so “who does this person report to and what is that person’s team working on” is two hops and vector search doesn’t do hops. A knowledge graph stores entities and relationships and retrieval traverses, which handles exactly that, at the cost of much harder extraction — you have to resolve entities to existing nodes and type the relationships — and it’s bad at fuzzy conceptual queries. So start with a vector store, and move to a graph when you can name the specific multi-hop question your agent has to answer. “It would be more sophisticated” is not that question.

Creation mechanisms

Two orthogonal axes, and they get confused with each other.

Explicit vs implicit — how the information was elicited.

Explicit: the user directly instructs the agent to remember. “Remember that my anniversary is October 26th.” High trust. Unambiguous intent. Rare.

Implicit: the agent infers something from the conversation without being told to. “My anniversary is next week, can you help me find a gift?” → the agent extracts an approximate date. Lower trust. Much more common. Where most of the value is, and most of the errors.

Internal vs external — where the extraction logic lives.

Internal: memory management built into the agent framework. Convenient to start with, usually thin on features. Can still use external storage — the point is that the generation logic is yours.

External: a dedicated memory service (Vertex AI Agent Engine Memory Bank, mem0, Zep). Your agent makes API calls to store, retrieve, and consolidate. You get semantic search, entity extraction, and automatic consolidation without building them.

The default advice: use an external service unless you have an unusual requirement, because consolidation is genuinely hard and you will underestimate it. Build it yourself once, though — Chapter 6 — because you need to know what the service is doing before you can debug it.

Saying it out loud. There are two independent axes here. Explicit versus implicit is about how the information was elicited: explicit means the user said remember this, which is high trust and rare; implicit means the agent inferred it from conversation, which is where most of the value lives and also most of the errors. Internal versus external is about where the extraction logic lives — built into your framework, or a dedicated memory service. My default is to use an external service, because consolidation is genuinely hard and you’ll underestimate it, but build it yourself once so you know what the service is doing when you have to debug it. And I’d be honest about the evidence here rather than repeating vendor claims: independent audits of the standard long-conversation memory benchmark found substantial errors in its answer key and an LLM judge that accepts a majority of deliberately wrong answers, and the leading memory-layer paper’s own table shows plain full-context beating it on accuracy. So the real case for a memory layer is latency and cost — you’re not resending a hundred thousand tokens every turn — not that it makes the agent more correct.

Memory scope

This is the setting most likely to cause a security incident, so read this section twice.

Scope answers: who or what does this memory describe, and therefore who is allowed to see it?

User-level scope. Tied to a user ID, persists across all their sessions. "The user prefers the middle seat." This is the default and the most common. It is what makes an agent feel continuous.

Session-level scope. Insights extracted from one specific session, isolated to that session. "The user is shopping for New York to Paris tickets between Nov 7 and Nov 14, prefers direct flights and the middle seat." This is compaction: the processed insight replaces the verbose transcript. Distinct from the raw session log — it holds conclusions, not dialogue.

Application-level scope (global). Accessible to every user of the application. "The codename Halyard refers to the mobile redesign project." Used for shared context, system-wide announcements, and baseline common knowledge. Procedural memories often live here, because a workflow that works for one user usually works for all of them.

Now the warning.

Application-scoped memories are a data-exfiltration channel.

If a memory generated from user A’s conversation gets stored at application scope, user B can retrieve it. That is not a hypothetical; it is the natural consequence of a scope bug in an extraction pipeline, and the extraction pipeline is an LLM, which means the bug can be induced by a user who wants it to happen.

The controls, all of which you need:

Application-scoped writes require explicit, deliberate authorization. Never a default, never inferable by the extraction LLM.
Anything written at application scope is aggressively anonymized and stripped of anything user-specific first.
Every memory record carries its scope in metadata, and the retrieval layer filters by scope and owner in code, not in a prompt.

The general principle from Chapter 1 of Part 1 applies unchanged: a rule the model can talk itself out of is not a rule. Scope filtering is a WHERE clause, not an instruction.

Saying it out loud. Scope answers who a memory describes and therefore who is allowed to see it, and it’s the setting most likely to cause a security incident. User-level is the default and what makes an agent feel continuous. Session-level is really compaction — the processed insight replacing a verbose transcript. Application-level is shared across every user, which is useful for things like what an internal project codename means. And that last one is a data-exfiltration channel: if a memory generated from user A’s conversation lands at application scope, user B can retrieve it. That’s not hypothetical, it’s the natural result of a scope bug in an extraction pipeline — and the extraction pipeline is an LLM, so the bug can be deliberately induced by a user who wants it to happen. The controls are that application-scoped writes need explicit authorization and are never inferable by the extraction model, anything written there is anonymized first, and the retrieval layer filters by scope and owner in code. Scope filtering is a WHERE clause, not an instruction — a rule the model can talk itself out of isn’t a rule.

Multimodal memory

The key distinction here is between the data a memory is derived from and the data it is stored as.

Memory from a multimodal source is the common case. The agent processes an image, a voice memo, a video — and produces a textual memory. It does not keep the audio file. It transcribes, interprets, and stores: "The user expressed frustration about a shipping delay."

Memory with multimodal content is the harder case. The memory itself contains the media. The user uploads an image and says “remember this design for our logo,” and the memory record holds the image.

Almost all production memory managers do the first and not the second, and the reason is boring and correct: generating, indexing, and retrieving binary content requires specialized models and infrastructure, whereas converting everything to text gives you one searchable format and one embedding space.

The practical pattern is a hybrid. Store the textual insight as the memory content — that is what gets embedded, retrieved, and injected — and keep a URI reference to the original artifact in metadata. The text is searchable; the artifact is retrievable when something actually needs the pixels. This is the same “return a handle, not the payload” principle from Part 2, applied to storage.

Saying it out loud. The distinction that matters is between what a memory is derived from and what it’s stored as. Memory from a multimodal source is the common case: the agent processes an image or a voice memo and produces a textual memory — it doesn’t keep the audio, it transcribes and interprets. Memory with multimodal content, where the record itself holds the image, is the harder case, and almost no production system does it. The reason is boring and correct: generating, indexing, and retrieving binary content needs specialized models and infrastructure, whereas converting everything to text gives you one searchable format and one embedding space. So the practical pattern is a hybrid — store the textual insight as the content that gets embedded and injected, and keep a URI to the original artifact in metadata. It’s the same return-a-handle-not-the-payload principle from tool design, applied to storage.

Procedural memories

Everything above is about declarative memory — the “what.” Procedural memory is the “how,” and it is a genuinely different problem.

The reason: storing the “how” is not an information retrieval problem, it is a reasoning augmentation problem.

A declarative memory is a fact you inject so the model knows something. A procedural memory is a plan you inject so the model does something in a particular way. The whole lifecycle differs:

Extraction must distill a reusable strategy from a successful run, not a fact from a conversation. “When the export times out, increase the batch limit before investigating the network” is a procedure. Getting an LLM to produce that from a trace requires a very different prompt than “extract user preferences.”

Consolidation curates workflows rather than merging facts. Integrating a newly successful method with the existing playbook. Patching a step that turns out to be wrong. Pruning a procedure that stopped working when an API changed. This is closer to code review than to deduplication.

Retrieval fetches a plan relevant to the task at hand, not data relevant to a question. The schema is usually different — a procedural memory has trigger conditions, steps, and preconditions, where a declarative memory has a sentence.

It is natural to compare this to fine-tuning, and the comparison is illuminating. Fine-tuning is slow, offline, and changes model weights. Procedural memory is fast, online, and changes the prompt — the agent adapts by having the right playbook injected, via in-context learning, with no training run. You can ship a procedural memory fix in the time it takes to write one.

The honest state of the field: commercial memory platforms are built for declarative memory and do not really handle this. If you want procedural memory today, you are building it, and you should treat the playbook store as a separate system with its own schema rather than trying to force it into a memory manager designed for user facts.

Saying it out loud. Procedural memory is knowing how, and it’s a genuinely different problem, because storing a how isn’t an information retrieval problem — it’s a reasoning augmentation problem. A declarative memory is a fact you inject so the model knows something; a procedural memory is a plan you inject so the model does something a particular way. Every stage differs: extraction has to distill a reusable strategy from a successful run rather than a fact from a chat, consolidation is closer to code review than to deduplication because you’re patching steps and pruning procedures that broke when an API changed, and retrieval fetches a plan matching the task rather than data matching a question. The useful comparison is fine-tuning: that’s slow, offline, and changes weights, whereas procedural memory is fast, online, and changes the prompt — you can ship a fix in the time it takes to write one. The honest state of the field is that commercial platforms don’t really handle this, so if you want it today you’re building it, and you should keep the playbook store as its own system rather than forcing it into a memory manager designed for user facts.

What you should be able to do now

State the difference between memory, RAG, and session state in one sentence each, and explain why the difference in data preparation is the one that actually matters.
Explain why a memory manager is not a vector database, and identify a “memory system” that is really just a search index over logs.
Classify a memory as semantic, episodic, or procedural, and say how that classification changes where you store it and when you retrieve it.
Choose among collections, structured profiles, and rolling summaries for a given use case, and defend the choice on retrieval characteristics.
Choose between a vector store and a knowledge graph by naming a specific multi-hop question that forces the graph.
Set the correct scope for a memory, and describe the exfiltration risk of application-level scope along with the controls that mitigate it.
Explain the difference between memory from a multimodal source and memory with multimodal content, and implement the text-plus-URI hybrid.

Production Agent Engineering — A Practical Guide