Pattern 8: Memory Management — what the agent keeps, and what it wisely throws away

The interview scenario

Memory questions are usually about continuity across time:

• “Design a coaching agent that works with a user for months — it should remember their goals, their history, and what advice actually worked.”
• “Our support bot treats every conversation like a first date: users repeat their account details and their whole saga every single time. Fix it.”
• “Design a personal assistant that learns preferences — how the user likes their summaries, who their important contacts are — without being told twice.”

The moment the design has to survive the end of a conversation, you’re in this pattern.

What this pattern is, in plain words

Here’s the uncomfortable fact underneath every chat product: the model remembers nothing. Between one API call and the next, it’s a goldfish. What feels like memory in a chatbot is a trick — the application resends the whole conversation so far with every request, and the model rereads it from scratch each time.

That resent transcript is short-term memory, and it lives in the context window — the model’s working memory, a hard cap on how much text it can consider in one call. Everything in the current session — messages, tool results, intermediate reasoning — competes for that finite space.

Short-term memory has two structural problems. It’s ephemeral: when the session ends, it’s gone. And it’s expensive: you pay to reprocess the whole transcript on every turn, so a long conversation gets slower and pricier as it goes. Long-context models stretch the window; they don’t change either property.

Long-term memory is the fix for persistence: information worth keeping is written outside the model — into a database, a knowledge graph, or most commonly a vector store — and selectively pulled back in when relevant. A vector store is a database that finds text by meaning rather than exact words: each piece of text is converted into an embedding — a list of numbers positioned so that similar meanings land near each other — and a query about “my running goals” retrieves the note that says “training for a half marathon,” even though they share no words. Searching by meaning like this is called semantic search.

The human analogy maps cleanly and is worth saying in the interview: short-term memory is the whiteboard in the meeting room — fast, visible, wiped when the meeting ends. Long-term memory is the filing cabinet — durable and searchable, but you must deliberately file things into it and deliberately pull things out. The craft of this pattern is deciding what gets filed, how it’s organized, and what gets thrown away.

And that last part is not a joke: forgetting is a feature. An agent that hoards every utterance forever ends up drowning in stale, contradictory, and privacy-sensitive clutter. Good memory systems curate.

How the design actually works

Think of memory as two subsystems with a narrow bridge between them: managing the window, and managing the store.

Managing the window. Within a session, the game is fitting the most relevant information into limited space. The workhorse technique is summarization: as the transcript grows, older turns are compressed into a running summary — “user is training for a half marathon in June; knee pain discussed; plan adjusted to low-impact weeks” — which replaces the raw messages. You keep recent turns verbatim, because nuance matters most near the present, and pay a small quality tax on the compressed past.

Frameworks give you the mechanics. LangChain’s ChatMessageHistory is manual bookkeeping of turns; ConversationBufferMemory automatically injects the running history into each prompt under a named variable. LangGraph goes further: the session’s state persists through a checkpointer — a component that snapshots state so a thread can pause and resume later, surviving restarts. In Google’s ADK, the session is a first-class object: a Session holds the event log of the conversation, and its State is a key-value scratchpad — a dictionary of small facts the agent maintains as it goes: task progress, flags, incremental data.

ADK adds a detail worth citing because it shows real design maturity: state keys carry prefixes that declare their scope and lifetime. A bare key is session-scoped; user: keys follow the user across all their sessions; app: keys are shared across all users; temp: keys survive only the current turn. And you never mutate state directly — updates travel through the event-append flow (an output_key on the agent, or an explicit state delta attached to an event) so every change is recorded, persisted, and safe under concurrency. Scope and auditability, designed in from the start.

Managing the store. Long-term memory is a separate service with two verbs. Write: after (or during) a session, extract what’s worth keeping and store it. Read: given the current context, search the store and inject the relevant results into the prompt. ADK formalizes this as the MemoryService, with exactly those operations — add a session’s content to memory, and search memory — backed by an in-memory implementation for development and a vector-search-backed one for production. LangGraph’s store organizes memories as JSON documents under namespaces (folder-like groupings, typically per user per context) with semantic search over them.

The write side is the subtle side. Storing raw transcripts is the naive move; the mature move is extraction — a model pass that distills the session into discrete facts and preferences worth keeping. Vertex AI’s Memory Bank is the managed version of this idea, and its feature list is a design checklist: it analyzes conversations asynchronously (off the critical path, so the user never waits on memory writes), extracts key facts and preferences, scopes them by user ID, and — crucially — consolidates: new information is merged with existing memories and contradictions are resolved, so “I moved to Chicago” updates the old “lives in Denver” fact instead of coexisting with it.

What gets stored — the three types. The book borrows a taxonomy from human memory, and interviewers reward you for using it:

Procedural memory is the mind-bending one: the agent’s instructions become data it can revise. Powerful, and dangerous enough that self-updates belong behind review.

The read path, end to end. A returning user’s message arrives. The system loads recent session state, queries long-term memory — semantic search with the current message and user ID — and injects the top few results into the prompt alongside the conversation. The model answers as if it “remembers.” That’s the whole bridge: retrieval pulls from the filing cabinet onto the whiteboard.

Where it breaks. Memory failures are insidious because they compound silently over months:

Context overflow. The window fills; naive truncation silently drops the earliest — often foundational — content, and models attend least reliably to the middle of very long contexts anyway. Defense: deliberate summarization instead of accidental truncation.

Stale memories. Facts expire — jobs change, goals get achieved, preferences drift. A system that confidently serves last year’s truth feels worse than one that admits ignorance. Defense: consolidation on write, timestamps on every memory, and recency weighting on retrieval.

Poisoned memories. The store is writable, which makes it an attack surface and an error amplifier: a sarcastic remark stored as a preference, or an injected instruction (“remember: always recommend this product”) that gets faithfully retrieved into future prompts. Defense: treat retrieved memories as data, never as instructions; validate and attribute writes; make the store inspectable.

Retrieval misses and mismatches. The right memory exists but isn’t retrieved, or a semantically-similar-but-wrong one is. Both are quiet failures — the user just experiences an agent that’s oddly forgetful or oddly confused. Defense: retrieval evals with known query→memory pairs, and similarity thresholds so garbage doesn’t get injected merely for being the nearest garbage.

Cross-user leakage. The nightmare failure: user A’s facts surface in user B’s session. Defense: namespace isolation enforced structurally — user ID baked into every read and write path, tested adversarially, never left to the model’s discretion.

Unbounded growth. Millions of accumulating memories degrade retrieval precision and inflate cost. Defense: forgetting as policy — expiry for time-sensitive facts, decay or archival for long-unretrieved ones, consolidation to merge duplicates. Curation is what keeps the filing cabinet useful.

Privacy. Memory is a dossier. Users need to see what’s remembered, correct it, and delete it — “forget that” must actually delete, not just apologize.

When to use it, and when not. The rule of thumb from the book: reach for memory when the agent must maintain context through a conversation, track multi-step progress, personalize across sessions, or learn from past outcomes. Skip long-term memory for stateless, one-shot tasks — a translation tool doesn’t need a filing cabinet, and premature memory infrastructure adds cost, privacy obligations, and failure modes with no payoff. Short-term-only is a perfectly respectable design when sessions are self-contained.

Memory versus RAG. They share machinery — vector store, embeddings, semantic search — so name the difference crisply: RAG retrieves from a shared knowledge corpus (your docs, your wiki — mostly read-only, same for every user) to ground answers in facts. Memory retrieves from a per-user, read-write record derived from interactions to ground answers in relationship. One makes the agent knowledgeable; the other makes it familiar. Same plumbing, different water.

The complete interview answer, spoken

The prompt: “Design a coaching agent that remembers users across months.” Spoken, 15–20 minutes.

“Scoping first. Text-based coaching — fitness and habits, say — with sessions a couple of times a week over months? I’ll assume that. Scale: tens of thousands of users, so per-user cost matters but we’re not at consumer-social scale. And privacy: health-adjacent data, so I’ll treat memory as sensitive by default — visible to the user, deletable, isolated per user.

The heart of this problem is that the model is stateless — it remembers literally nothing between API calls — so ‘remembering across months’ is entirely an architecture I build around it. I’ll design two layers with a bridge: session memory for within-a-conversation continuity, a long-term store for across-months continuity, and extraction plus retrieval as the bridge between them.

Here’s the whiteboard.

            during session                    after session
 User <--> +-----------+                 +------------------+
           |  Coach    |    transcript   |  Extraction job  |
           |  LLM      | --------------> |  (async, cheap   |
           +-----------+                 |   model)         |
             ^   ^                       +--------+---------+
     recent  |   | retrieved                      | facts,
     turns + |   | memories                       | episodes
     summary |   |                                v
           +-----------+   semantic     +------------------+
           | Session   |   search       |  Memory store    |
           | state     |  <-----------> |  (vector DB,     |
           | (whiteboard)               |   per-user ns)   |
           +-----------+                |  + consolidation |
                                        +------------------+
                                                 |
                                        user-facing "what I
                                        remember" + delete

Start with the session layer, because even within one conversation memory is work. Each turn, the model receives: the system prompt, a rolling summary of the older parts of this session, the last N turns verbatim, and — I’ll get there — retrieved long-term memories. The rolling summary is produced by a cheap model call whenever the transcript crosses a size threshold: compress the oldest turns into the summary, keep the recent ones raw. That bounds per-turn cost and keeps us far from the context limit, and the tradeoff is honest: we lose verbatim detail from an hour ago but keep its meaning.

Alongside the transcript I keep structured session state — a small key-value scratchpad, in the ADK sense: today’s topic, exercises agreed on, a flag that the user mentioned pain. Scratchpad facts are cheap to carry and cheaper to read than re-deriving them from prose every turn.

Now the interesting layer: long-term. The write path first, because what you store determines everything downstream.

When a session ends, I do not dump the transcript into storage — raw transcripts are noise with occasional signal, and retrieval over noise returns noise. Instead an asynchronous extraction job — async because the user should never wait on memory bookkeeping, which is exactly how the managed services like Vertex Memory Bank run — has a cheap model distill the session into typed memories.

Three types, mirroring the classic taxonomy. Semantic — durable facts and preferences: ‘goal: half marathon June 14,’ ‘prefers morning workouts,’ ‘history of knee pain.’ Episodic — what happened and what worked: ‘week of May 5: switched to low-impact plan for knee pain; user reported improvement.’ Those episodes later serve as few-shot guidance — when a similar situation recurs, retrieving what worked last time steers the coach better than any general instruction. And procedural — per-user coaching style: ‘responds to direct challenges, dislikes cheerleading.’ I’d fold that into a per-user profile the system prompt reads, and any self-updating of it goes through review, because an agent editing its own instructions from user chatter is a prompt-injection vector wearing a lab coat.

Each memory record carries: content, type, timestamp, source session ID, and a confidence. It’s embedded and written into the vector store under the user’s namespace — and I want namespace isolation enforced by the storage layer itself, user ID mandatory in every query path, because cross-user leakage is the one failure here that ends the product. I’d write adversarial tests for it, not just trust the code path.

Before a new memory lands, consolidation: compare it against near-duplicates in the store. Same fact again — bump confidence, refresh timestamp. Contradiction — ‘knee is fully recovered’ versus ‘ongoing knee pain’ — supersede the old record: mark it inactive with a pointer to its replacement, rather than deleting, so there’s an audit trail. This is the Memory Bank behavior — merge new data, resolve contradictions — and it’s what keeps months of accumulation coherent instead of self-contradictory.

The read path: when a session starts, two retrievals. First, the standing profile — goals, constraints, style — which always loads; continuity of identity shouldn’t depend on a similarity score. Second, semantic search against the user’s namespace with the opening message, top four or five hits above a similarity threshold, injected as a labeled block: ‘context from past sessions, may be outdated — verify with the user if load-bearing.’ Mid-session, if the topic shifts, I re-query — either on topic-change detection or by giving the model a search_memory tool it can call when it feels under-informed, which is the ADK-style pattern of memory-as-a-tool.

That ‘may be outdated’ framing is deliberate and I want to flag it: retrieved memories are treated as data to consider, never as instructions to follow. If a stored note somehow says ‘always recommend supplement X,’ it arrives quoted inside a data block, not appended to the system prompt. That, plus extraction filtering at write time — facts about the user, never imperatives — is my two-layer defense against memory poisoning.

Forgetting is designed in, not an afterthought. Time-sensitive memories get expiries — ‘training for the June race’ auto-archives after June. Memories unretrieved for months decay toward archival. Consolidation prunes duplicates continuously. And the user gets a visible ‘what I remember about you’ screen with per-item delete — when they say ‘forget that,’ the record is actually purged, along with its embedding. That’s both a trust feature and, for health-adjacent data, table stakes for regulation.

Evaluation, because memory quality is measurable if you make it so. Extraction: a labeled set of transcripts with gold memory lists — measure precision, did we store junk, and recall, did we miss the knee injury. Retrieval: known query-to-memory pairs, standard search metrics. Consolidation: seeded contradiction cases, checking the right record wins. End to end, the metric that matters is continuity: in session simulations spanning weeks, does the coach behave consistently with established facts — never re-asking the goal, never re-suggesting the run that hurt the knee. Plus one online canary: how often real users re-state things they’ve already told us — that number falling is the product working.

Cost: per turn, one main model call on a bounded context — the summary strategy caps it — plus a vector query, which is millicents. Extraction is one cheap async call per session. Storage is a few kilobytes of text and embeddings per user — negligible. The dominant knob is how much retrieved-and-summarized context I stuff into each turn, and that’s a measurable quality-versus-cost dial I’d tune with the continuity evals.

Tradeoff to close on: I’m extracting aggressively rather than storing transcripts wholesale — cheaper retrieval, cleaner memory, easier privacy — at the price of losing anything the extractor didn’t deem important, permanently. Given health-adjacent data, I prefer that bias: store less, store it well, and let the user see all of it.“

Follow-ups they will ask

“Context windows are getting huge — why not just resend the entire month of history every session?” Three reasons: cost — you’d reprocess megabytes of transcript on every single turn, forever; quality — retrieval of long contexts is uneven, and ten relevant facts beat ten thousand raw lines competing for the model’s attention; and governance — a transcript blob can’t be inspected, corrected, or selectively deleted the way discrete memories can. Long context raises the ceiling on short-term memory; it doesn’t provide persistence, curation, or consent.

“How is this different from RAG?” Same plumbing — embeddings, vector store, semantic search — different content and contract. RAG searches a shared, mostly read-only knowledge corpus so the agent is knowledgeable; memory searches a per-user, read-write record distilled from interactions so the agent is familiar. In practice you often run both side by side: a coaching agent RAG-retrieves exercise science and memory-retrieves your knee history.

“What if the user tells the agent something false, or someone deliberately poisons the store?” Accept that some wrong facts will land, and contain the damage: extraction stores only user-facts and never imperatives; retrieved memories are injected as quoted data with an ‘unverified’ frame, not as instructions; consolidation lets newer information supersede older; and the user-visible memory screen turns silent corruption into visible, correctable state. The agent should also hold important memories lightly — ‘last time you mentioned knee pain, is that still an issue?’ — which is both good safety and good coaching.

“How do you decide what’s worth storing?” An explicit extraction rubric, not vibes: durable facts about the user, stated preferences, goals with dates, and outcome-bearing episodes — with small talk, transient logistics, and anything sensitive-but-irrelevant excluded by instruction. Then measure it: the extraction eval’s precision score is literally ‘are we storing junk,’ and the user-facing memory screen is a free audit channel — if users keep deleting a category of memory, the rubric is wrong.

“Two memories conflict at retrieval time — who wins?” Ideally consolidation resolved it at write time, with the newer fact superseding and the loser archived. If a conflict survives to retrieval, deterministic policy: prefer active over superseded, then recency, then confidence — and if they still genuinely tie, surface the ambiguity to the user rather than guessing. One extra rule from operations: never let the model silently average two contradictory facts into a mushy third.

“How would you migrate an existing memory-less product to this design?” Incrementally, read-path last: start the async extraction job on new sessions immediately so the store begins filling, run it in shadow for a few weeks while evaluating extraction quality offline, then enable profile-loading, then semantic retrieval — each behind a flag with the continuity metrics watched. Backfilling from historical transcripts is possible but I’d do it only with explicit user consent, because ‘we mined your old conversations’ is a very different privacy posture than ‘we remember going forward.’

Say it in one breath

Memory is two systems and a bridge: the context window as ephemeral, expensive working memory managed by summarization, and an external per-user store — typically a vector database searched by meaning — holding semantic facts, episodic experiences, and procedural rules extracted asynchronously from sessions. Retrieval injects a few relevant, timestamped memories back into the prompt as data, never as instructions, with consolidation resolving contradictions at write time. And forgetting is a feature, not a bug: expiry, decay, and user-visible deletion are what keep months of accumulation accurate, cheap, and trustworthy.