Context Compaction
AI Engineer path · lesson 84 of 124When a conversation context window fills up, compaction replaces verbose history with a concise rolling summary. Learn how compaction works, what survives vs. what gets dropped, and how to pair it with prompt caching for maximum efficiency.
Add agent turns until the window fills. When usage passes the 75 % threshold, compaction kicks in and older turns are replaced by a summary.
Key insight
Compaction fires at 75 % full rather than 100 % — leaving headroom for the model to think and for tool outputs to land. Waiting for a full window means the next turn overflows before compaction completes.
Compact a long conversation
Order the compaction pass.
Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.
- 1Drop a step here
- 2Drop a step here
- 3Drop a step here
- 4Drop a step here
- 5Drop a step here
- 6Drop a step here
Test your knowledge
At what fill percentage do most systems trigger context compaction?
The short version
Context compaction is like a smart meeting note-taker: instead of transcribing every word, they capture decisions, action items, and open questions. When the notebook fills up, the transcript gets replaced by a crisp summary — you lose the verbatim banter but keep everything that matters for the next step.
In more depth
Why Context Compaction Exists
Every LLM has a finite context window. As a conversation grows, costs rise linearly and latency climbs. Past a threshold (typically 75–80 % full), continuing becomes expensive and risky due to the "lost-in-the-middle" attention penalty. Compaction solves this by replacing older turns with a compact rolling summary.
What Compaction Preserves
A good compaction summary retains:
It drops:
A well-executed compaction can achieve a 90–97 % token reduction while preserving all decision-relevant information.
Compaction Strategies
Rolling summary — the default and most common. When the window exceeds a threshold, the oldest N turns are distilled into a ~500-token summary block. Simple and effective for most use cases.
Sliding window — keeps the last k turns verbatim and discards everything older. Fast and predictable but loses important early context if the conversation spans many topics.
Hierarchical / sub-agents — each sub-agent works in its own fresh context and returns a compact result to the orchestrator. Naturally bounded; each node never sees the full history.
External memory + RAG — important facts are written to a vector store and retrieved on demand. Theoretically unbounded but adds retrieval latency and infrastructure complexity.
Compaction + Prompt Caching
The most cost-efficient architecture combines both:
This yields cheap stable context and a bounded dynamic window simultaneously.
Interview Questions
References & Further Reading
Love this visualization?
Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.
100% free • No credit card required