Managing long conversations

A session that starts as an immutable log of everything works beautifully for about twenty turns.

Then it stops working, and it stops working in four separate ways at once.

What actually degrades

The hard limit. Every model has a maximum context size. Exceed it and the API call fails outright. This is the least interesting failure because it is loud, deterministic, and easy to detect. It is also the last one you hit.

Cost. You pay per input token, on every call, and an agent resends its accumulated history each time. A conversation carrying 60,000 tokens of history costs 60,000 input tokens for the next turn, and the turn after, and the turn after that. The cost of a conversation is not linear in its length; it is roughly quadratic.

Latency. More input means more time before the first output token appears. Users experience this as the agent getting sluggish as the conversation goes on, which is a bad thing for it to do, because that is exactly when they are most invested.

Quality. This is the one that actually decides your architecture. As tokens accumulate, two things happen. Noise increases — there is more irrelevant material for the model’s attention to be diluted across. And autoregressive errors compound — a wrong statement on turn nine sits in the context on turn ten, where it looks exactly as authoritative as everything else.

Note the order in which these bite. Quality degrades first, then latency and cost become uncomfortable, and only much later does the hard limit fire. If your policy is “compact when the API errors,” you have been shipping a degraded agent for a long time before you noticed.

The packing analogy from the whitepaper is a good one: the context window is a suitcase. Overpack and it is heavy and you cannot find anything. Underpack and you left your passport at home. Success is not about how much you can carry.

Saying it out loud. Four things degrade as history grows, and they bite in an order people get backwards. Quality goes first, because noise dilutes attention and a wrong statement on turn nine sits in the context on turn ten looking exactly as authoritative as everything true. Then latency and cost get uncomfortable — you resend the whole history every turn, so cost over a conversation is roughly quadratic in its length, not linear. The hard context limit fires last, and it’s the least interesting failure because it’s loud and deterministic. There’s strong recent evidence for the quality part: when a task is split across multiple turns instead of stated all at once, model performance drops around 39%, and that decomposes into roughly a 16% fall in aptitude and a 112% rise in unreliability — same model, wildly less predictable. The mechanism is premature commitment: it locks onto an early wrong interpretation and never revisits it. Turning temperature down to zero doesn’t fix it. So if your policy is “compact when the API errors,” you’ve been shipping a degraded agent for a long time before you noticed.

The techniques, with their real costs

There are four families, and they trade off cleanly against each other.

Truncation and sliding windows

The simplest thing that could possibly work: keep the last N turns, drop the rest.

The token-based variant is slightly better — walk backwards from the most recent message, accumulating until you hit a token budget, then cut. This adapts to message size instead of assuming all turns are equal.

Cost: it is unconditionally lossy and it does not know what it is dropping. The user’s shipping address, given on turn 3, is gone on turn 30. The agent will either invent one or ask again, and both are bad.

When it is right: short-horizon conversations, or agents where old turns genuinely do not matter — a translation bot, a code-completion assistant, anything where the relevant window is inherently narrow.

Most frameworks give you this for free. In ADK it is a plugin that filters the context without touching stored events:

from google.adk.apps import App
from google.adk.plugins.context_filter_plugin import ContextFilterPlugin

app = App(
    name="support_app",
    root_agent=agent,
    plugins=[ContextFilterPlugin(num_invocations_to_keep=10)],
)

Note the important property: this changes what is sent, not what is stored. Your session history stays complete. That is the history-versus-context distinction from Chapter 2, and it is what makes truncation recoverable rather than destructive.

Saying it out loud. Truncation is the simplest thing that could work — keep the last N turns, drop the rest, or better, walk backwards accumulating until you hit a token budget so it adapts to message size. The cost is that it’s unconditionally lossy and it has no idea what it’s dropping: the shipping address the user gave on turn 3 is gone on turn 30, and the agent will either invent one or ask again. It’s the right answer for short-horizon work where old turns genuinely don’t matter — translation, code completion. The property that makes it survivable is that it changes what you send, not what you store: the session history stays complete, so truncation is recoverable rather than destructive.

Summarization and compaction

Replace older messages with a model-generated summary. The summary sits at the front of the context, the recent messages stay verbatim behind it. As the conversation grows, you summarize again — often folding the previous summary into the new one, which is why this is called recursive summarization.

This is strictly better than truncation on information density and strictly worse on everything else.

Cost 1: it is lossy in an unpredictable way. Truncation loses old things, which is at least a rule you can reason about. Summarization loses whatever the summarizing model decided was unimportant, which you cannot predict and will not notice until the agent gets something wrong.

Cost 2: it costs a model call. Which is money and latency, which is why it must happen in the background.

Cost 3: it destroys your prompt cache. Rewriting the front of the context invalidates every cached prefix. More on this below, because it changes the arithmetic more than people expect.

Two engineering requirements make compaction survivable in production.

Do it asynchronously and persist the result. Summarization is expensive. Do not make the user wait for it, and do not recompute it on every turn. Generate it in the background, write it to the session, reuse it.

Track exactly which events the summary covers. Store the index or event ID range. Without that bookkeeping you will send both the summary and the original messages it summarized, which is worse than doing nothing.

ADK exposes this as configuration:

from google.adk.apps import App
from google.adk.apps.app import EventsCompactionConfig

app = App(
    name="support_app",
    root_agent=agent,
    events_compaction_config=EventsCompactionConfig(
        compaction_interval=5,   # summarize every 5 invocations
        overlap_size=1,          # re-read one prior turn for continuity
    ),
)

That overlap_size is a small detail worth understanding. Compacting strictly disjoint blocks tends to produce summaries that lose the thread across boundaries. Overlapping by a turn gives the summarizer enough context to connect the new block to the old one.

Saying it out loud. Compaction replaces old messages with a model-generated summary, with recent turns kept verbatim behind it, and folding the previous summary into the new one is why it’s called recursive. It beats truncation on information density and loses on everything else. It’s lossy in an unpredictable way — truncation drops old things, which is at least a rule you can reason about, whereas summarization drops whatever the summarizer decided didn’t matter, and you find out when the agent gets something wrong. It costs a model call, so it has to happen in the background. And it destroys your prompt cache, because you rewrote the front of the context. Two requirements make it survivable: do it asynchronously and persist the result rather than recomputing, and track exactly which events the summary covers — because without that bookkeeping you’ll send the summary and the original messages it summarized, which is worse than doing nothing.

When to trigger

Three trigger families, and you will probably want two of them.

Count-based — compact when tokens or turns exceed a threshold. Simple, predictable, and honestly good enough for most systems.

Time-based — compact after a period of inactivity. This is the clever one, because it is free: the user is not waiting, so the latency cost is zero. If someone stops typing for fifteen minutes, that is an excellent moment to do expensive work.

Event-based — compact when a task, sub-goal, or topic concludes. This produces the highest-quality summaries because the boundary is semantically meaningful rather than arbitrary. It requires the agent to detect the boundary, which is its own problem.

Combine count-based as a safety net with time-based as the primary path, and you get good summaries most of the time and never blow the window.

Saying it out loud. There are three ways to decide when to compact and you want two of them. Count-based fires on a token or turn threshold — simple, predictable, honestly good enough for most systems. Time-based fires after a period of inactivity, and it’s the clever one because it’s effectively free: nobody is waiting, so the latency cost is zero, and someone going quiet for fifteen minutes is a great moment to do expensive work. Event-based fires when a task or topic concludes, which gives the best summaries because the boundary is semantically meaningful, but it needs the agent to detect the boundary, which is its own problem. My default is time-based as the primary path with count-based as the safety net — good summaries most of the time, and you never blow the window.

Selective retrieval of past turns

Instead of compressing history, index it and retrieve from it.

Embed each turn or each block of turns, store them, and when a new user message arrives, retrieve the handful of past turns most relevant to it. The context then contains: system instructions, the last few turns verbatim, and three older turns pulled in because they matter right now.

Cost: retrieval latency on the hot path, and a real risk of incoherence. Non-contiguous conversation fragments read strangely to a model. Turn 4 followed by turn 47 with nothing in between can produce confident nonsense about what was agreed.

When it is right: very long-running relationships where the conversation covers many distinct topics and you can afford the retrieval step. Note that at this point you have essentially reinvented memory, which is the honest conclusion — see Chapters 4 through 6.

Saying it out loud. Instead of compressing the history you can index it and retrieve from it — embed each turn or block, and when a new message arrives pull the handful of past turns that actually relate. So the context is instructions, the last few turns verbatim, and three older turns that matter right now. The costs are retrieval latency sitting on the hot path and a real risk of incoherence: turn 4 followed by turn 47 with nothing in between reads strangely to a model and can produce confident nonsense about what was agreed. It’s right for long-running relationships spanning many topics — and the honest observation is that at this point you’ve reinvented memory, which is a fine place to end up as long as you know that’s what you did.

Prompt caching, and why it changes the math

This is the technique that most changes how you should think about the others, and it is frequently misunderstood.

Providers will cache a prefix of your prompt server-side. On a subsequent request whose prefix matches exactly, they skip re-processing it and bill you far less.

The Anthropic API shape, which is representative:

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    system=[
        {"type": "text", "text": STABLE_SYSTEM_INSTRUCTIONS},
        {"type": "text", "text": LARGE_POLICY_DOCUMENT,
         "cache_control": {"type": "ephemeral"}},
    ],
    messages=conversation,
)

The cache_control marker says “cache everything up to and including this block.” The pricing shape is the part to internalize, expressed as multipliers on the base input price:

Operation	Multiplier on base input price
Cache write, 5-minute TTL (default)	1.25×
Cache write, 1-hour TTL (`"ttl": "1h"`)	2×
Cache read	0.1×

So a cache hit costs a tenth of a normal input token, and a cache write costs a quarter more than one. Break-even on the 5-minute TTL is roughly the second read. There is a minimum cacheable prefix — currently 512 tokens on Claude Opus 5, 1,024 on Sonnet 5 and Opus 4.8, higher on some others — and prompts below it silently go uncached rather than erroring.

Now the part that matters architecturally.

Caching only works on an exact, unchanged prefix. The moment you rewrite the front of your context, every cached token behind that point is invalidated.

Which means: compaction and caching are in direct tension. Compacting the head of your context saves you tokens and destroys your cache in the same operation. On a conversation where the cache would have been hitting, aggressive compaction can make things more expensive, not less.

The resolution is layout. Order your context from most stable to least stable:

System instructions (never change)
Tool schemas (change on deploy)
Long-lived retrieved documents and stable user profile memories
← cache breakpoint here
Conversation history
Current user message

Everything above the breakpoint is cached and cheap. Compaction only rewrites things below it. This is why “put the memories in the system prompt” is not only a behavioral choice — it is a caching choice, and we will return to it in Chapter 5.

One more current option, on the Anthropic API: server-side context editing, which clears old tool results for you while preserving cache-friendliness better than a wholesale rewrite would.

response = client.beta.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    messages=messages,
    tools=tools,
    betas=["context-management-2025-06-27"],
    context_management={"edits": [{
        "type": "clear_tool_uses_20250919",
        "trigger": {"type": "input_tokens", "value": 30000},
        "keep": {"type": "tool_uses", "value": 3},
        "exclude_tools": ["web_search"],
    }]},
)
print(response.context_management)   # tells you what it actually cleared

That is worth knowing about because old tool results are usually the largest and most disposable thing in an agent’s context, and clearing them is a much better first move than summarizing the dialogue.

Saying it out loud. Prompt caching is the thing that changes how you should think about all the rest. Providers cache a prefix of your prompt server-side, and if the next request’s prefix matches exactly they skip reprocessing and bill you far less — on the Anthropic shape a cache read is about a tenth of a normal input token and a write costs about a quarter more, so you break even around the second read. The architectural consequence is that caching only works on an exact unchanged prefix, which puts compaction and caching in direct tension: compacting the head of your context saves tokens and destroys your cache in the same move, and on a conversation that was hitting cache, aggressive compaction can make things more expensive. The resolution is layout — order from most stable to least: system instructions, tool schemas, stable profile memories, then your cache breakpoint, then history and the current message. Everything above the breakpoint stays cheap and compaction only touches what’s below it. And if you just want a quick win, clearing old tool results is usually a better first move than summarizing dialogue, because they’re the biggest and most disposable thing in the window.

Compact, or start fresh with a handoff?

There is a decision point people miss entirely, and it is a real one.

Compaction assumes the conversation should continue. Sometimes it should not.

Compact when the conversation is one continuous task, the recent turns depend on the earlier ones, and the user experience is a single unbroken thread.

Start fresh with a handoff when a task has completed and a new one is beginning, when the topic has changed substantially, when the trajectory has gone wrong and the history is full of failed attempts, or when a specialized sub-agent should take over.

A handoff is a clean break: you end the session, write a small structured record of everything the next session needs, and start a new session seeded with that record and nothing else.

The difference from compaction is a difference in what you keep. Compaction keeps a lossy version of everything. A handoff keeps a complete version of a defined, small set of things.

@dataclass
class Handoff:
    goal: str                       # what the next session is trying to achieve
    facts: dict[str, Any]           # everything established, structured
    decisions: list[str]            # what was agreed, and by whom
    open_questions: list[str]       # what is still unresolved
    do_not_repeat: list[str]        # approaches already tried and failed

That do_not_repeat field is the one that earns its place. The single biggest advantage of a fresh session is escaping a context full of failed attempts — but only if you carry forward the lesson rather than the transcript.

The rule I would apply: if you cannot say in one sentence why the old history is needed, hand off instead of compacting. A clean 800-token context beats a compacted 12,000-token one on quality, latency, and cost simultaneously. That combination is rare enough that you should take it when it is offered.

Saying it out loud. There’s a decision people miss entirely: compaction assumes the conversation should continue, and sometimes it shouldn’t. Compact when it’s one continuous task and recent turns depend on earlier ones. Start fresh with a handoff when a task finished and a new one is starting, when the topic changed, or when the trajectory went wrong and the history is full of failed attempts. A handoff is a clean break — you end the session, write a small structured record, and seed a new session with that and nothing else. The difference is what you keep: compaction keeps a lossy version of everything, a handoff keeps a complete version of a small defined set of things — the goal, the established facts, the decisions, the open questions, and crucially a do-not-repeat list, because escaping a context full of failed attempts only helps if you carry the lesson rather than the transcript. My rule is that if you can’t say in one sentence why the old history is needed, hand off — a clean 800-token context beats a compacted 12,000-token one on quality, latency, and cost at the same time, and that combination is rare enough to take when it’s offered.

Build it: schema-preserving compaction

Naive compaction — “summarize the old messages” — loses the specific facts your business logic depends on, and it loses them silently. The fix is to stop asking one operation to do two jobs.

Summarize the prose. Extract the fields. Carry the fields verbatim.

Start by writing down what must survive. This dataclass is a contract, and it is the thing you tune when compaction turns out to have lost something.

from dataclasses import dataclass, field, asdict
from typing import Optional

@dataclass
class CriticalFacts:
    """Fields the agent cannot function without. Never summarized away.

    Add a field here the day you find the agent forgetting something. This
    dataclass is the thing you tune when compaction loses information.
    """
    user_id: Optional[str] = None
    open_order_id: Optional[str] = None
    shipping_address: Optional[str] = None
    refund_authorized_eur: Optional[float] = None
    confirmed_by_user: list[str] = field(default_factory=list)
    unresolved_questions: list[str] = field(default_factory=list)

    def merge(self, other) -> "CriticalFacts":
        """Later values win for scalars; lists accumulate without duplicates."""
        ...

    def render(self) -> str:
        """One '- key: value' line per set field, under an authoritative header."""
        ...

merge encodes the update policy: for a scalar, newer wins; for a list, accumulate. That is a decision, not a default. If the user changes their shipping address on turn 40, you want the new one. If they confirm two separate things, you want both.

Now the compactor. It takes two callables — a summarizer and an extractor — which in production are two model calls and in the demo below are offline stand-ins.

@dataclass
class Compaction:
    """The durable record of one compaction. Persist this, not just the text."""
    summary: str
    facts: CriticalFacts
    covers_upto: int          # exclusive index into the original message list
    replaced_tokens: int
    summary_tokens: int


class Compactor:
    def __init__(self, summarize, extract, *, trigger_tokens=3000,
                 keep_recent=6, overlap=1):
        self.summarize = summarize
        self.extract = extract
        self.trigger_tokens = trigger_tokens
        self.keep_recent = keep_recent
        self.overlap = overlap

    def should_compact(self, messages: list[dict]) -> bool:
        return count_tokens(messages) > self.trigger_tokens

    def compact(self, messages: list[dict], prior=None) -> Compaction:
        """Fold everything except the last `keep_recent` messages into a summary."""
        cut = max(0, len(messages) - self.keep_recent)
        start = max(0, prior.covers_upto - self.overlap) if prior else 0
        window = messages[start:cut]
        if not window:
            return prior or Compaction("", CriticalFacts(), 0, 0, 0)

        facts = self.extract(window)
        if prior:
            facts = prior.facts.merge(facts)

        summary = self.summarize(
            ([{"role": "system", "content": "Previously: " + prior.summary}] if prior else [])
            + window
        )
        return Compaction(
            summary=summary,
            facts=facts,
            covers_upto=cut,
            replaced_tokens=count_tokens(messages[:cut]),
            summary_tokens=count_tokens([{"content": summary}]),
        )

    def build_context(self, system: str, messages: list[dict], c) -> list[dict]:
        """Assemble what actually goes to the model."""
        if c is None:
            return [{"role": "system", "content": system}] + messages
        head = (f"{system}\n\n{c.facts.render()}\n\n"
                f"SUMMARY OF EARLIER CONVERSATION:\n{c.summary}")
        return [{"role": "system", "content": head}] + messages[c.covers_upto:]

Three design points worth stating explicitly.

covers_upto is the bookkeeping the whitepaper insists on. build_context slices messages[c.covers_upto:], so the messages folded into the summary are never sent twice.

prior makes it recursive. The second compaction reads the first summary as input and merges the first fact set forward, so nothing has to survive more than one summarization hop to reach turn 200.

build_context puts the facts above the summary and labels them authoritative. Ordering is not decorative — the facts are precise and the summary is lossy, and when they disagree you want the model deferring to the precise one.

For the extractor, a real system uses a structured-output call with the Pydantic pattern from Chapter 1 — one model call whose schema is CriticalFacts itself. For the demo below, a handful of regexes stand in (\b(ORD-\d+)\b for the order ID, ship(?:ping)? (?:it )?to ([^.]+) for the address, and so on) so that every line runs with no API key.

Running it on a seventeen-message support conversation that includes three bulky tool results:

before compaction: 464 tokens, 17 messages
should_compact: True

=== CONTEXT SENT TO MODEL ===
[system]
You are a support agent for Solaris Audio.

CARRIED-FORWARD FACTS (authoritative, do not contradict):
- open_order_id: ORD-991
- shipping_address: 44 Rue Lafayette, Paris 75009
- refund_authorized_eur: 24.0
- confirmed_by_user: ['Yes, that works for me.']
- unresolved_questions: ['Actually how long will delivery take?']

SUMMARY OF EARLIER CONVERSATION:
The customer contacted support and the conversation covered: order,
damaged, warranty, address, delivery, refund. (13 messages folded.)

[user]      Confirmed, thank you.
[assistant] The replacement is booked and the refund is queued.
[user]      One more thing - does the warranty restart on the replacement?
[assistant] Let me check the warranty policy for replacement units.

after compaction: 170 tokens, 5 messages
replaced 416 tokens with 33 of summary

464 tokens down to 170, a 63% reduction, on a conversation that is deliberately short — the ratio gets much better as conversations grow, because the recent window stays fixed while the compacted portion grows.

Now the comparison that justifies all of this. Here is what a summary-only compaction would have preserved:

=== NAIVE SUMMARY-ONLY COMPACTION (what you lose) ===
The customer contacted support and the conversation covered: order,
damaged, warranty, address, delivery, refund. (17 messages folded.)
  open_order_id          in naive summary? False   schema-preserved: ORD-991
  shipping_address       in naive summary? False   schema-preserved: 44 Rue Lafayette, Paris 75009
  refund_authorized_eur  in naive summary? False   schema-preserved: 24.0

The summary knows the conversation was about an address. It does not contain the address. An agent working from that summary alone will ask the customer for their address again, on turn eighteen, having already confirmed it on turn five. Everyone has experienced this from the customer side and it is infuriating.

And the recursion, after the conversation continues and compacts a second time:

=== AFTER A SECOND COMPACTION ===
CARRIED-FORWARD FACTS (authoritative, do not contradict):
- open_order_id: ORD-991
- shipping_address: 44 Rue Lafayette, Paris 75009
- refund_authorized_eur: 24.0
- confirmed_by_user: ['Yes, that works for me.', 'Confirmed, thank you.']
- unresolved_questions: ['Actually how long will delivery take?', 'One more thing - does the warranty restart on the replacemen']

Facts accumulated rather than degrading. That is the property you want, and it is the property naive summarization does not have — each summarization hop is another chance to drop something, and after four hops the details are gone.

One honest caveat, visible in that output: unresolved_questions is growing and nothing ever removes an entry. A real implementation needs the extractor to also mark questions as answered, or you have built a list that only grows. That is the same problem as memory consolidation, arriving early — which is a good segue, because consolidation is exactly what Chapter 5 is about.

Saying it out loud. Naive compaction loses the exact facts your business logic depends on, and it loses them silently — so stop asking one operation to do two jobs. Summarize the prose, extract the fields, and carry the fields verbatim. In practice that means writing down a small schema of things that must survive — the open order ID, the shipping address, the authorized refund amount, what the user confirmed — and treating that schema as a contract you tune the day you catch the agent forgetting something. Three details make it work: track which messages the summary covers so you never send them twice, feed the previous summary and fact set into the next compaction so it’s recursive and nothing has to survive more than one hop, and put the facts above the summary labeled authoritative, because the facts are precise and the summary is lossy and you want the model deferring to the precise one when they disagree. On a short demo conversation that’s a 63% token reduction, and the ratio only improves as conversations grow. The failure it prevents is concrete: a summary knows the conversation was about an address, but it doesn’t contain the address, so the agent asks the customer for it again on turn eighteen having confirmed it on turn five.

What you should be able to do now

Name the four things that degrade as conversation history grows, and put them in the order you will actually encounter them.
Choose between truncation, summarization, selective retrieval, and caching for a given agent, and state honestly what each one costs.
Explain why compaction must be asynchronous and why you must record which events a summary covers.
Lay out a context so that a prompt cache actually hits, and explain why compaction and caching pull against each other.
Decide between compacting a session and ending it with a structured handoff, and write the handoff record.
Implement compaction that preserves a declared schema of critical fields verbatim, and demonstrate what a summary-only approach would have lost.

Production Agent Engineering — A Practical Guide