Memory generation, provenance, and retrieval

Chapter 4 was about what a memory is. This one is about the machinery: how memories get made, how you know whether to trust them, how you find the right ones, and where you put them once you have.

The useful frame is that memory generation is an LLM-driven ETL pipeline. Extract meaningful content from noisy source data, transform it by reconciling it with what you already know, load it into durable storage. The novelty is that both the “what is meaningful” decision and the “how does this fit” decision are made by a model rather than by rules you wrote.

That is precisely what separates a memory manager from a database. With a database, you write the INSERT and the UPDATE and you decide when each applies. With a memory manager, an LLM looks at new information and existing information and decides whether this is a create, an update, a merge, or a delete.

Four stages.

Ingestion — raw source data arrives, usually a conversation transcript.
Extraction and filtering — an LLM pulls out content matching a definition of “meaningful.” If nothing matches, nothing is created. This is the crucial part: it does not extract everything.
Consolidation — the new insights are reconciled against existing memories. Merge, update, delete, or create.
Storage — persist to a vector store or graph.

Stages 2 and 3 are where all the difficulty is, so they get a section each.

Saying it out loud. The frame I’d use is that memory generation is an LLM-driven ETL pipeline: extract meaningful content out of noisy source data, transform it by reconciling it with what you already believe, load it into durable storage. The novelty is that both the what-is-meaningful decision and the how-does-this-fit decision are made by a model instead of by rules you wrote — and that’s exactly what separates a memory manager from a database. With a database you write the INSERT and the UPDATE and you decide when each applies; with a memory manager an LLM looks at the new and the existing information and decides whether this is a create, an update, a merge, or a delete. Four stages — ingestion, extraction, consolidation, storage — and essentially all the difficulty is in the middle two.

Extraction

Extraction answers one question: what in this conversation is meaningful enough to become a memory?

It is not summarization. Summarization compresses everything proportionally. Extraction is targeted filtering — it separates the signal (facts, preferences, goals, commitments) from the noise (greetings, acknowledgments, “let me check that for you”), and it throws away most of the input.

The word doing the work is meaningful, and it has no universal definition. What a customer support agent needs to remember — order numbers, reported defects, promises made — has almost nothing in common with what a wellness coach needs to remember — long-term goals, emotional states, what the user tried last month. Defining “meaningful” for your domain is the single highest-leverage decision in the whole memory system. Get it wrong and you have either a store full of noise or a store missing the thing that mattered.

Three mechanisms for telling the extraction model what you want.

Saying it out loud. Extraction answers one question: what in this conversation is worth remembering? It’s not summarization — summarization compresses everything proportionally, whereas extraction is targeted filtering that separates facts, preferences, goals, and commitments from greetings and “let me check that for you,” and throws away most of the input. The word doing all the work is “meaningful,” and it has no universal definition: what a support agent must remember — order numbers, defects, promises made — has almost nothing in common with what a wellness coach must remember. Defining meaningful for your domain is the single highest-leverage decision in the whole memory system, because getting it wrong gives you either a store full of noise or a store missing the one thing that mattered.

Schema and template-based extraction

Give the LLM a JSON schema and use structured output to constrain generation. This is the Pydantic pattern from Chapter 1, applied here:

from pydantic import BaseModel, Field
from typing import Literal

class Memory(BaseModel):
    fact: str = Field(description=(
        "One durable fact, stated in the third person, self-contained enough "
        "to be understood without the conversation. E.g. 'The user prefers "
        "window seats on flights longer than three hours.'"
    ))
    topic: Literal["preference", "identity", "goal", "constraint", "issue"]
    confidence: float = Field(ge=0.0, le=1.0, description=(
        "1.0 if the user stated this directly; lower if inferred."
    ))

class Extraction(BaseModel):
    memories: list[Memory] = Field(description=(
        "Empty list if the conversation contains nothing durable. Do not "
        "invent memories to avoid returning an empty list."
    ))

Two things in there are doing real work and are easy to leave out.

“Self-contained enough to be understood without the conversation” prevents the single most common extraction failure: memories like "The user said yes to the second option." Second option in what? That memory is useless in three weeks and actively misleading when retrieved alongside an unrelated conversation.

“Do not invent memories to avoid returning an empty list” prevents the second most common failure. Models are strongly biased toward producing output. Without explicit permission to return nothing, you will get memories extracted from conversations that contained no information whatsoever, and your store will fill with "The user greeted the agent."

Saying it out loud. The most controllable way to tell an extractor what you want is a schema plus structured output, so generation is constrained rather than merely encouraged. Two lines in that schema do enormous work and are easy to leave out. First, requiring each fact to be self-contained enough to understand without the conversation — otherwise you get memories like “the user said yes to the second option,” which is useless in three weeks and actively misleading when it surfaces next to an unrelated conversation. Second, explicitly permitting an empty list and saying not to invent memories to avoid returning one. Models are strongly biased toward producing output, so without that permission your store fills up with “the user greeted the agent.”

Natural language topic definitions

Rather than a rigid schema, describe the topic in prose and let the model interpret it. This is what managed services expose as configuration:

memory_topics = [
    {"managed_memory_topic": {"managed_topic_enum": "USER_PERSONAL_INFO"}},
    {"custom_memory_topic": {
        "label": "business_feedback",
        "description": (
            "Specific user feedback about their experience at the coffee shop: "
            "opinions on drinks, food, pastries, ambiance, staff friendliness, "
            "service speed, cleanliness, and suggestions for improvement."
        ),
    }},
]

More flexible than a schema, less deterministic. Good for topics where the interesting variation is in the content rather than the structure.

Few-shot examples

Show the model input conversations paired with the ideal memories extracted from them.

This is the most effective mechanism for nuanced or unusual topics — the ones where you can recognize a good extraction but cannot describe the rule. Two or three well-chosen examples routinely outperform a paragraph of instruction, at a fraction of the tokens.

example = {
    "conversation": [
        {"role": "model", "text": "Welcome back to The Daily Grind! How was your visit?"},
        {"role": "user",  "text": "The drip coffee was lukewarm today, which was a "
                                   "bummer. And the music was way too loud."},
    ],
    "expected_memories": [
        {"fact": "The user reported that the drip coffee was lukewarm."},
        {"fact": "The user felt the music in the shop was too loud."},
    ],
}

Look closely at that expected output, because it is teaching two things at once. Two separate memories rather than one combined one — atomicity, which makes retrieval and later contradiction-handling far easier. And “reported that” / “felt that” rather than “the coffee was lukewarm” — the memory records that the user said something, not that it is objectively true. That framing is not pedantry. It is the difference between a memory that stays correct forever and one that becomes false the moment the shop fixes the coffee machine.

Saying it out loud. For nuanced topics — the ones where you can recognize a good extraction but can’t state the rule — few-shot examples beat instructions, and two or three well-chosen pairs routinely outperform a paragraph at a fraction of the tokens. The thing to notice is that a good example teaches two things at once. It shows two separate memories rather than one combined one, which is atomicity, and that makes retrieval and later contradiction-handling far easier. And it phrases things as “the user reported that the coffee was lukewarm” rather than “the coffee was lukewarm” — recording that the user said something, not that it’s objectively true. That’s not pedantry; it’s the difference between a memory that stays correct forever and one that becomes false the moment the shop fixes the machine.

A practical efficiency trick

Running extraction on the full verbose transcript every turn is wasteful. The pattern most managers use instead: maintain a rolling summary of the conversation and feed that, plus the most recent turns, into the extraction prompt.

The summary carries enough context for the model to interpret the new turns correctly. The recent turns carry the new information. You get the same extraction quality without reprocessing forty turns of dialogue every five minutes.

Consolidation

Extraction is the easy half.

Consolidation is where a pile of extracted facts becomes a coherent picture of a person, and it is the stage that most homegrown memory systems skip and then regret.

Without it, the store degrades in four specific ways.

Duplication. “I need a flight to NYC” in January and “I’m planning a trip to New York” in March produce two memories that mean the same thing. Now retrieval returns both, wasting context, and any counting or reasoning over memories is wrong.

Contradiction. “I’m vegetarian” in January. “I’ve started eating fish again” in June. Both memories sit in the store. Retrieval surfaces both. The model gets to pick, and it picks unpredictably.

Failure to evolve. “The user is interested in marketing” is true but crude. Three conversations later, the right memory is “The user is leading a marketing project focused on Q4 customer acquisition.” A store without consolidation keeps the crude one alongside the good one forever.

Relevance decay. A memory about a meeting two years ago is not as useful as one from last week, and eventually it is noise. An agent that never forgets accumulates an ever-growing store where the signal-to-noise ratio only goes down.

Saying it out loud. Extraction is the easy half. Consolidation is where a pile of extracted facts becomes a coherent picture of a person, and it’s the stage homegrown systems skip and then regret. Without it the store degrades in four specific ways. Duplication: “I need a flight to NYC” in January and “planning a trip to New York” in March become two memories meaning the same thing, so retrieval wastes context and any counting over memories is wrong. Contradiction: “I’m vegetarian” in January and “I’ve started eating fish again” in June both sit there, and the model picks between them unpredictably. Failure to evolve: the crude early memory lives on next to the good refined one forever. And relevance decay, where an agent that never forgets accumulates a store whose signal-to-noise ratio only goes down.

The algorithm

Consolidation is a retrieve-then-decide loop.

Step 1: find the candidates. For each newly extracted memory, search the existing store for similar memories. Those are the ones that might need to change. This is the step people get wrong by making it too narrow — a high similarity threshold means contradictions never get detected, because “I’m vegetarian” and “I eat fish now” are not especially similar as strings.

Step 2: ask an LLM what to do. Present the new information and the candidate existing memories together, and ask for operations:

CREATE — the insight is novel and unrelated to anything existing.
UPDATE — an existing memory should be modified with new or corrected information.
DELETE / INVALIDATE — new information makes an existing memory incorrect or irrelevant.

Step 3: apply as a transaction. Translate the decisions into database operations, atomically. Partial application is how you end up with both the old and new version of a contradicting fact.

Here is the shape of it:

class Op(BaseModel):
    action: Literal["CREATE", "UPDATE", "DELETE", "NOOP"]
    target_id: str | None = Field(description="Existing memory ID for UPDATE/DELETE.")
    content: str | None = Field(description="New content for CREATE/UPDATE.")
    reason: str = Field(description="Why. Logged for debugging, not shown to users.")

And the prompt that produces them, whose rules are the whole policy:

You maintain a user's memory store. Below are existing memories and a newly
extracted candidate. Decide what operations keep the store coherent.

- If the candidate says the same thing as an existing memory, NOOP.
- If the candidate refines or extends an existing memory, UPDATE it.
- If the candidate contradicts an existing memory, UPDATE the existing memory
  to the newer state. Prefer newer information over older.
- If the candidate is unrelated to everything shown, CREATE.
- Never CREATE something that duplicates an existing memory.

That reason field is not decoration. When your memory store contains something wrong six months from now, the reason string is how you find out which consolidation decision produced it. It costs a few tokens and it is the difference between debugging and guessing.

Saying it out loud. Consolidation is a retrieve-then-decide loop. For each newly extracted memory, search the existing store for anything similar — those are the ones that might need to change. Then hand the new information and the candidates to a model and ask for operations: create, update, or delete. Then apply them as a transaction, because partial application is exactly how you end up holding both the old and the new version of a contradicting fact. The step people get wrong is the first one, by setting the similarity threshold too high — “I’m vegetarian” and “I eat fish now” aren’t especially similar as strings, so a narrow search means contradictions are never even detected. And log a reason on every operation: when your store contains something wrong six months later, that string is the difference between debugging and guessing.

Forgetting is a feature

Two mechanisms, and you want both.

Instruct the LLM to defer to newer information during consolidation. That handles the contradictions you detect.

Set a TTL for automatic deletion. That handles the ones you do not — the memories that are not contradicted by anything, just quietly irrelevant.

Beyond that, proactive pruning triggered by:

Time-based decay — importance falls with age. A memory about last week beats one from two years ago, other things equal.
Low confidence — a memory created from a weak inference and never corroborated is a good candidate for removal.
Irrelevance — as the picture of the user gets richer, older trivial memories stop earning their storage.

A memory system that only ever adds is not a memory system. It is a log with a search box.

Saying it out loud. Forgetting is a feature, and you want two mechanisms. Instruct the model to prefer newer information during consolidation, which handles the contradictions you actually detect. And set a TTL for automatic deletion, which handles the ones you don’t — memories nothing contradicts, just quietly irrelevant. On top of that, prune proactively on time-based decay, on low confidence where a weak inference was never corroborated, and on irrelevance as the picture of the user gets richer. The line I’d end on is that a memory system that only ever adds isn’t a memory system, it’s a log with a search box.

Provenance and lineage

The machine learning axiom is “garbage in, garbage out.” With LLMs it is worse: garbage in, confident garbage out.

For an agent to reason well over its memories — and for the consolidation step above to make good decisions — it needs to know how much to trust each one. Trust comes from provenance: a record of where a memory came from and what has happened to it since.

This gets complicated because consolidation destroys the simple one-to-one mapping. One memory can blend information from several sources. One source can produce several memories. It is a many-to-many graph, and if you do not track it deliberately, you cannot reconstruct it later.

Saying it out loud. The old machine learning line is garbage in, garbage out; with LLMs it’s worse, it’s garbage in, confident garbage out. So for an agent to reason well over its memories — and for consolidation to make good calls — it needs to know how much to trust each one, and trust comes from provenance: where a memory came from and what’s happened to it since. The reason this gets hard is that consolidation destroys the simple one-to-one mapping. One memory can blend several sources and one source can produce several memories, so it’s a many-to-many graph, and if you don’t track it deliberately at write time you can’t reconstruct it later at all.

Source type determines base trust

Three categories, in descending order of trustworthiness:

Bootstrapped data. Pre-loaded from internal systems — a CRM, a user profile, an account record. High trust: it came from a system of record, not from a model’s interpretation of a sentence. Its main use is solving the cold-start problem — giving a brand new user a personalized experience before they have said anything.

User input. Either explicit (a form, a settings page, a direct “remember this”) which is high trust, or implicit (extracted from conversation) which is meaningfully lower. Implicit extraction is where most memories come from and where most errors come from.

Tool output. Data returned by an external tool call. Generating memories from tool output is generally a bad idea. The whitepaper is blunt about this and it is right: those memories are brittle and go stale fast. A stock price, an inventory count, an order status — these are facts about a system that changes, and freezing them into long-term memory means confidently telling a user something that stopped being true on Tuesday. Cache them short-term; do not remember them.

Saying it out loud. Three source types, in descending order of trust. Bootstrapped data from a system of record — a CRM, an account record — is the most trustworthy, and its real value is solving cold start, giving a brand new user a personalized experience before they’ve said anything. User input splits: explicit, where they filled in a form or said remember this, is high trust; implicit, extracted from conversation, is meaningfully lower, and that’s where most memories and most errors come from. Tool output is the one to be blunt about: generating long-term memories from tool results is generally a bad idea. A stock price, an inventory count, an order status — those are facts about a system that changes, so freezing them into memory means confidently telling a user something that stopped being true on Tuesday. Cache them short-term; don’t remember them.

Lineage during memory management

Provenance solves two operational problems that are otherwise unsolvable.

Conflict resolution. When sources disagree, you need a policy, and the policy needs source metadata to work with:

Trust hierarchy — the CRM record beats an inference from conversation.
Recency — newer beats older, all else equal.
Corroboration — three independent sources agreeing beats one source asserting.

Most systems use a blend. The important thing is that it is an explicit policy, applied in code at consolidation time, rather than an emergent property of whatever the LLM felt like doing.

Deleting derived data. A user revokes access to a data source — disconnects their calendar, deletes their CRM record, exercises a right to erasure. Now what happens to the memories derived from it?

Deleting every memory that source ever touched is over-aggressive: a memory that blended four sources loses everything because one of them was withdrawn. The correct approach is to regenerate the affected memories from the remaining valid sources. It is computationally expensive and it is the right answer, and it is only possible if you recorded which sources contributed to which memories.

This is the concrete reason provenance is not optional in any system with real users. It is a compliance requirement wearing an architecture costume.

Saying it out loud. Provenance solves two problems that are otherwise unsolvable. First, conflict resolution: when sources disagree you need an explicit policy with source metadata to work from — a trust hierarchy where the CRM record beats an inference from chat, recency as a tiebreak, and corroboration where three independent sources beat one assertion. The important part is that it’s a policy applied in code at consolidation time, not an emergent property of whatever the model felt like doing. Second, deleting derived data: a user disconnects their calendar or exercises a right to erasure, and now what happens to memories derived from it? Deleting every memory that source ever touched is over-aggressive, since a memory blending four sources loses everything because one was withdrawn. The correct move is regenerating the affected memories from the remaining valid sources — expensive, and only possible if you recorded which sources contributed to which memory. Provenance is a compliance requirement wearing an architecture costume.

Confidence evolves

Confidence should not be a number you set at creation and never touch.

It increases through corroboration — the same fact arriving from a second trusted source. It decreases with age, as memories go stale. It drops when contradictory information appears. Below a floor, the memory gets archived or deleted.

Lineage during inference

Here is the part people skip: this all matters at inference time too, not just during curation.

When you inject memories into a prompt, inject them with their confidence and, where relevant, their age and source. Not for the user — these are internal — but so the model can weigh them.

<MEMORIES>
- [confidence: high, source: account record] The user's plan is Enterprise.
- [confidence: high, stated 2 days ago] The user is migrating to the new API.
- [confidence: low, inferred 4 months ago] The user may prefer email over chat.
</MEMORIES>

A model given that block behaves noticeably better than one given three bare sentences. It will lean on the first two and treat the third as a weak prior rather than a fact — which is exactly right, and is behavior you got for the price of three annotations.

Saying it out loud. The part people skip is that provenance matters at inference time too, not just during curation. When you inject memories into the prompt, inject them with confidence and, where it matters, age and source — not for the user, these are internal, but so the model can weigh them. Give it a block where one line says high confidence from the account record, another says stated two days ago, and a third says low confidence, inferred four months ago, and it behaves noticeably better than with three bare sentences: it leans on the first two and treats the third as a weak prior rather than a fact. That’s exactly the behavior you want, and you bought it for the price of three annotations.

Triggering generation

The memory manager automates extraction and consolidation once you invoke it. Deciding when to invoke it is your job, and it is a real tradeoff: freshness against cost and latency.

Session completion. Generate once, at the end. Cheapest. Lowest fidelity — the model summarizes a large block at once and detail gets lost. And you have no memories mid-session, which is bad for long sessions.

Turn cadence. Every N turns. The pragmatic default. Good enough for most systems.

Real-time. After every turn. Highest fidelity, highest cost, and it needs careful handling to avoid latency.

Explicit command. The user says “remember this.” Always support this regardless of what else you do. It is high-trust, unambiguous, and users expect it to work.

One trap worth naming: do not reprocess the same events repeatedly. If you run generation every five turns and each run ingests the whole conversation, you are paying to re-extract the same first ten turns over and over. Track a watermark of what has been ingested and only send what is new (plus enough overlap for context).

Saying it out loud. Deciding when to run generation is your call, and it’s a straight freshness-versus-cost tradeoff. At session completion is cheapest but lowest fidelity, because the model summarizes one big block and detail gets lost, and you have no memories at all mid-session, which hurts on long ones. Every N turns is the pragmatic default and good enough for most systems. After every turn is highest fidelity and highest cost and needs care to avoid latency. And always support an explicit “remember this,” whatever else you do, because it’s high trust, unambiguous, and users expect it to work. The trap worth naming is reprocessing: if you run generation every five turns and each run ingests the whole conversation, you’re paying to re-extract the same first ten turns over and over. Track a watermark and send only what’s new, plus a little overlap for context.

Memory-as-a-tool

The more sophisticated pattern: let the agent decide.

Expose memory generation as a tool. The agent, mid-conversation, notices something worth persisting and calls it.

def remember(fact: str, tool_context) -> dict:
    """Persist a durable fact about the user for future conversations.

    Call this when the user reveals a stable preference, a constraint, a
    long-term goal, or a correction to something previously established.
    Do NOT call this for transient details (today's weather, the current
    page they are on) or for anything already in your context.

    Args:
        fact: One self-contained sentence in the third person, e.g.
            "The user prefers window seats on flights over three hours."
    """
    memory_client.generate(
        direct_memories_source={"direct_memories": [{"fact": fact}]},
        scope={"user_id": tool_context.user_id, "app_name": tool_context.app_name},
        config={"wait_for_completion": False},   # background
    )
    return {"status": "ok"}

Note where the responsibility moved. With a managed pipeline, the memory manager decides what is meaningful. With this pattern, the agent decides — which means you decide, through the tool description. That description is doing the same job the topic definitions were doing earlier, which is the Part 2 lesson arriving again: the tool description is a prompt.

There is a middle option too, which is often the best of both. The agent extracts the fact (it has the full conversational context and knows what matters) and hands it to the memory manager for consolidation only — so you get agent-quality extraction plus managed merge-and-deduplicate. That is what direct_memories_source above is doing.

Saying it out loud. The more sophisticated pattern is to let the agent decide — expose memory writing as a tool it can call mid-conversation when it notices something worth keeping. Notice where the responsibility moved: with a managed pipeline the memory manager decides what’s meaningful, and with this pattern the agent decides, which really means you decide, through the tool description. That description is doing exactly the job the topic definitions were doing — the tool description is a prompt, again. And there’s a middle option that’s often best of both: the agent extracts the fact, because it has the full conversational context and knows what matters, and hands it to the memory service for consolidation only. Agent-quality extraction, managed merge-and-deduplicate.

Background vs blocking

Memory generation must be asynchronous. This is not a preference.

Generation means LLM calls plus database writes. Blocking a user’s response on that is unacceptable — you are adding seconds of latency to deliver zero value to the current turn, because the memory being written cannot possibly help the answer already being produced.

The architecture:

The agent responds to the user. Done. User is happy.
The agent makes a non-blocking call to the memory service, pushing raw source data.
The memory service acknowledges immediately, queues the work, and does the expensive extraction and consolidation on its own time.
Memories are persisted.
A later turn retrieves them.

The consequence worth internalizing: this makes the memory pipeline failure-isolated. If the memory service is down, slow, or throwing errors, the agent still answers. You lose some memories, which you can backfill. You do not lose the product.

The corollary is a real behavior to plan for: memories written at the end of turn 5 may not be retrievable at the start of turn 6. Eventual consistency is the price. Handle it by keeping the current session’s information in the session (where it is immediately available) and letting memory serve the next conversation.

Saying it out loud. Memory generation has to be asynchronous, and that’s not a preference. Generation means model calls plus database writes, and blocking the user’s answer on that adds seconds of latency to deliver exactly zero value to the current turn — the memory being written cannot possibly help the response already being produced. So: answer the user, then fire a non-blocking call to the memory service, which acknowledges immediately and does the expensive work on its own time. The consequence worth internalizing is that this makes the memory pipeline failure-isolated — if the memory service is down or slow, the agent still answers, and you lose some memories you can backfill rather than losing the product. The corollary you have to plan for is that memories written at the end of turn 5 may not be retrievable at the start of turn 6. Eventual consistency is the price, and you handle it by keeping the current session’s information in the session and letting memory serve the next conversation.

Retrieval

Generation puts things in. Retrieval is what makes them useful, and it has its own failure mode: retrieving the wrong memories is worse than retrieving none. An irrelevant memory in the context does not sit there harmlessly; it pulls the model toward a topic that was not being discussed.

How you retrieve depends on how you organized (Chapter 4). For a structured profile it is a lookup — fetch the profile, done. For a collection it is a search problem, and that is where the engineering is.

Saying it out loud. Generation puts things in; retrieval is what makes them useful, and it has its own failure mode — retrieving the wrong memories is worse than retrieving none at all. An irrelevant memory doesn’t sit there harmlessly, it actively pulls the model toward a topic nobody was discussing. How you retrieve follows from how you organized: a structured profile is a lookup, you just fetch it, and a collection is a search problem, which is where all the engineering lives.

Score on three dimensions, not one

The common mistake is ranking by vector similarity alone.

Relevance — semantic similarity to the current conversation. Necessary, not sufficient.
Recency — how recently the memory was created or last corroborated.
Importance — how significant this memory is in general, typically assigned at generation time rather than computed at retrieval.

Similarity alone will happily surface a memory that is conceptually adjacent but eight months old and trivial, over one that is slightly less similar and central to who this user is.

A blended score:

def score(memory, query_embedding, now):
    relevance = cosine(memory.embedding, query_embedding)      # 0..1
    age_days  = (now - memory.created_at) / 86400
    recency   = 0.5 ** (age_days / memory.half_life_days)      # exponential decay
    return 0.6 * relevance + 0.25 * recency + 0.15 * memory.importance

Those weights are a starting point, not a recommendation — tune them against your own evaluation set. The structural point is that there are three terms.

Note half_life_days living on the memory rather than being a global constant. “The user’s name is Ada” should decay very slowly. “The user is currently debugging a webhook issue” should decay in days. A single global decay rate gets both of those wrong.

Saying it out loud. The common mistake is ranking by vector similarity alone. You want three terms: relevance, which is semantic similarity to the current conversation and is necessary but not sufficient; recency, how recently the memory was created or corroborated; and importance, usually assigned at generation time rather than computed at retrieval. Similarity alone will happily surface something conceptually adjacent but eight months old and trivial over something slightly less similar but central to who this person is. The detail I’d point at is putting the decay half-life on the individual memory rather than using a global constant — “the user’s name is Ada” should decay very slowly and “the user is currently debugging a webhook issue” should decay in days, and any single global rate gets both of those wrong.

Query rewriting. Use an LLM to turn an ambiguous user message into a better search query, or expand it into several queries covering different facets. Improves results meaningfully. Costs an LLM call before retrieval, on the hot path.

Reranking. Retrieve a broad candidate set (say top 50) by similarity, then have an LLM re-order the shortlist. More accurate. Also an extra call.

Fine-tuned retrievers. Train a retriever on your domain. Best quality if you have labeled data. Significant cost and ongoing maintenance.

All three add latency to the hot path, which makes them a poor fit for interactive agents. Where the memories are stable, cache the retrieval results — the expensive computation happens once and subsequent identical queries skip it entirely.

But the honest advice is the whitepaper’s: the best retrieval improvement is better generation. A store full of atomic, well-scoped, deduplicated memories retrieves well with plain similarity search. A store full of duplicates and vague sentences will not be rescued by a reranker. If retrieval quality is your problem, look upstream first.

Saying it out loud. There are fancier options — rewriting the query with a model call, retrieving a broad candidate set and reranking the shortlist, or training a domain-specific retriever — and they all improve results and they all add latency to the hot path, which makes them a poor fit for interactive agents. If your memories are stable, cache the retrieval results so the expensive part happens once. But the honest advice is that the best retrieval improvement is better generation. A store of atomic, well-scoped, deduplicated memories retrieves well with plain similarity search, and a store full of duplicates and vague sentences will not be rescued by a reranker. If retrieval quality is your problem, look upstream first.

Timing: proactive or reactive

Proactive (static) retrieval loads memories automatically at the start of every turn. Context is always available; no extra model call. The cost is latency on every turn including the many that need no memory at all. Mitigate by caching — memories are static within a turn, so this caches well.

def retrieve_memories_callback(callback_context, llm_request):
    response = client.agent_engines.memories.retrieve(
        name=AGENT_ENGINE,
        scope={"user_id": callback_context.user_id, "app_name": APP_NAME},
    )
    memories = [f"* {m.memory.fact}" for m in response]
    if not memories:
        return
    llm_request.config.system_instruction += (
        "\n\nHere is information you know about the user:\n" + "\n".join(memories)
    )

agent = LlmAgent(..., before_model_callback=retrieve_memories_callback)

Reactive retrieval (memory-as-a-tool) gives the agent a search tool and lets it decide. More efficient in aggregate — you only pay when memory is actually needed. Costs an extra round trip when it is used.

Its real weakness is subtle: the agent does not know what it does not know. It cannot decide to look something up if it has no idea anything is stored. The mitigation is to say so in the tool description:

def search_memory(query: str, tool_context) -> list[str]:
    """Search what you know about this user from previous conversations.

    The following kinds of information may be available:
    * Stated preferences (dietary, seating, communication style, language)
    * Past issues they reported and how those were resolved
    * Ongoing projects and goals they have mentioned
    * Account details they have confirmed

    Use this when the user refers to something from a previous conversation,
    or when a personalized answer would clearly be better than a generic one.
    """
    return tool_context.search_memory(query).memories

Enumerating the categories converts “the agent guesses whether to search” into “the agent checks whether its need matches a listed category,” which is a much easier decision.

In practice: use both. Proactively load the stable profile — it is small, always relevant, and caches. Give the agent a tool for the long tail of episodic memories.

Saying it out loud. Two timings. Proactive retrieval loads memories automatically at the start of every turn, so context is always there and there’s no extra model call — the cost is latency on every turn including the many that need no memory, which you mitigate with caching since memories are static within a turn. Reactive retrieval hands the agent a search tool and lets it decide, which is more efficient in aggregate because you only pay when memory is needed, but costs a round trip when used. Its real weakness is subtle: the agent doesn’t know what it doesn’t know, so it can’t decide to look something up if it has no idea anything is stored. The mitigation is to enumerate the categories right in the tool description — stated preferences, past issues and how they were resolved, ongoing projects — which converts “guess whether to search” into “check whether my need matches a listed category.” In practice use both: proactively load the small stable profile, and give the agent a tool for the long tail.

Inference: where you put the memories changes the behavior

You have the memories. Now, where in the payload?

This is not a formatting question. Placement changes how the model treats them.

In the system instructions

Append retrieved memories to the system prompt, behind a preamble.

from jinja2 import Template

template = Template("""{{ system_instructions }}

<MEMORIES>
Here is information you know about this user:
{% for m in memories %}* {{ m.fact }}
{% endfor %}</MEMORIES>
""")
prompt = template.render(system_instructions=BASE, memories=retrieved)

Advantages. High authority — system instructions carry weight, and the model treats these as foundational rather than as something that came up. Clean separation — the dialogue stays a dialogue. Ideal for stable global information: the user profile, their tier, their language.

Costs, all of them real.

Over-influence. The model may try to relate everything back to memories that are sitting in its core instructions. A user asks a generic question about pricing and gets an answer awkwardly threaded through their stated interest in hiking. This is a genuine failure mode and it looks unhinged to users.

Framework support. You need the ability to construct the system prompt dynamically before each call. Not every framework makes this easy.

Incompatible with memory-as-a-tool. The system prompt is finalized before the model runs, which is before it could possibly decide to call a retrieval tool. You cannot put tool-retrieved memories in the system prompt of the same call.

Poor multimodal handling. Most APIs accept only text in the system instruction. An image memory has nowhere to go.

Cache implications. From Chapter 3: rewriting the system prompt on every turn invalidates your prompt cache. If memories change per turn, put them after your cache breakpoint, or accept the cost.

Saying it out loud. Putting memories in the system instructions gives them high authority — the model treats them as foundational rather than as something that came up — and keeps the dialogue clean, which makes it ideal for stable global things like the profile, the tier, the language. The costs are all real though. Over-influence is the big one: the model starts relating everything back to memories sitting in its core instructions, so a generic pricing question comes back awkwardly threaded through the user’s stated interest in hiking, and that looks unhinged. It’s also fundamentally incompatible with memory-as-a-tool, because the system prompt is finalized before the model runs, which is before it could possibly decide to call a retrieval tool. And rewriting the system prompt every turn invalidates your prompt cache, so per-turn memories belong after your cache breakpoint or you accept the bill.

In the conversation history

Inject memories as messages — either before the whole history, or immediately before the latest user message.

Advantages. Compatible with memory-as-a-tool: tool results land in the conversation naturally, which makes this the only option for reactively retrieved memories. Handles multimodal content, since message content blocks accept images. Lower authority, which is sometimes exactly what you want — a transient episodic memory should be weaker than a standing instruction.

Costs.

Dialogue injection. The headline risk: the model may treat an injected memory as something that was actually said in this conversation. It then says “as you mentioned earlier” about something the user said six weeks ago in a different session, which is unsettling.

Noise. Retrieved memories that turn out to be irrelevant sit in the dialogue confusing the model, and you pay for them every turn thereafter.

Perspective. If you inject under the user role, memories must be written in first person or the transcript reads bizarrely. {"role": "user", "content": "The user prefers window seats"} is a user talking about themselves in the third person, which no human does, and models notice.

Saying it out loud. Putting memories in the conversation history is the only option for reactively retrieved ones, since tool results land there naturally, and it handles multimodal content because message blocks accept images. It also carries lower authority, which is sometimes exactly right — a transient episodic memory should be weaker than a standing instruction. The headline risk is dialogue injection: the model treats an injected memory as something actually said in this conversation, so it says “as you mentioned earlier” about something the user said six weeks ago in a different session, which is unsettling for the user. There’s also a perspective trap — if you inject under the user role, the memory has to be in first person, because a user message saying “the user prefers window seats” is a person talking about themselves in the third person, which no human does and models notice.

The hybrid, which is what you should build

System instructions for stable, global memories. The profile. Preferences. Anything that should always be present and always carry weight.

Conversation history or tool results for transient, episodic memories. The specific past incident that happens to be relevant right now.

That split maps cleanly onto the semantic/episodic distinction from Chapter 4, and onto the caching layout from Chapter 3: stable memories sit above your cache breakpoint and get cached; episodic ones sit below and change freely.

Three architectural decisions lining up in the same direction is usually a sign you have found the right seam.

Saying it out loud. What you should actually build is the hybrid: system instructions for stable global memories — the profile, the preferences, anything that should always be present and always carry weight — and the conversation history or tool results for transient episodic ones, the specific past incident that happens to matter right now. What I like about that split is that it lines up with three separate things at once. It matches the semantic-versus-episodic distinction, it matches the retrieval split between proactive and reactive, and it matches the caching layout, since stable memories sit above your cache breakpoint and episodic ones sit below and change freely. Three architectural decisions pointing the same direction is usually a sign you’ve found the right seam.

What you should be able to do now

Write an extraction schema that produces atomic, self-contained memories and explicitly permits an empty result.
Explain why consolidation is the hard stage, and implement a retrieve-then-decide loop producing CREATE / UPDATE / DELETE operations applied transactionally.
Design a forgetting policy combining LLM-driven contradiction resolution, TTL, and confidence-based pruning.
Record provenance on every memory, and use it both to resolve conflicts during consolidation and to weight reliability at inference time.
Explain why deleting a revoked data source means regenerating derived memories rather than deleting everything it touched.
Choose a generation trigger and justify it on the cost/fidelity tradeoff, and explain why generation must be non-blocking and what eventual consistency means for turn N+1.
Implement blended retrieval scoring over relevance, recency, and importance, with per-memory decay rates.
Decide between system-instruction and conversation-history placement for a given class of memory, naming the specific behavioral difference — over-influence versus dialogue injection.

Production Agent Engineering — A Practical Guide