agent-loop
Context compaction: choosing what to drop when the window fills
Learn how LLM context compaction selectively removes or summarizes information when token limits are reached, keeping AI agents effective during long tasks.
What happens when your LLM agent hits the context limit mid-task
You've handed your AI agent a 200-page specification document, a dozen chat messages, three code files, and a list of action steps. It's two-thirds through the task when the context window maxes out. What does it forget—and who decides?
LLM context compaction is the process of selectively removing or summarizing information in an AI agent's working memory when the token limit approaches capacity. Unlike simple truncation (which chops off the oldest messages) or naive summarization (which blurs all the detail), compaction uses heuristics or learned policies to retain what matters most for the task at hand. The goal: keep the agent coherent, effective and on-mission even when the conversation or data set grows beyond the model's native window.
Why context limits matter for real work
Modern large language models ship with context windows ranging from 8,000 to over 200,000 tokens. That sounds generous—until you factor in a typical enterprise workflow:
- A single engineering issue with full thread history, linked pull requests, code diffs and build logs can easily consume 50,000 tokens.
- A sales agent reviewing deal notes, email threads, CRM records and product documentation routinely needs 100,000+ tokens per conversation.
- Customer support threads that pull in previous tickets, knowledge base articles and screen recordings grow linearly with each interaction.
When the window fills, the model must either refuse new input, discard old context, or compress what it holds. Refusal breaks the task. Naive discard loses critical state. Compaction is the compromise that keeps the loop running.
Core strategies for deciding what to drop
Recency vs. relevance
The simplest heuristic: keep recent messages, drop the oldest. This works when the conversation is linear and the most recent exchanges contain the task definition. It fails when earlier context—such as the original user requirement or a key decision—is essential for downstream steps.
A relevance-based approach scores each message or document chunk against the current task goal, then evicts the lowest-scoring items. Scoring might use embedding similarity to the active query, keyword matches to task parameters, or learned importance weights from a small ranker model.
Role-aware retention
In a structured agent loop—where messages carry roles like system, user, assistant and tool—you can pin certain roles as high priority. System prompts and the most recent user instruction are often non-negotiable. Tool call results from the last few steps matter more than intermediate reasoning. Older assistant reasoning can be summarized or dropped entirely.
Semantic summarization
Instead of deleting chunks outright, replace clusters of related messages with a terse summary. For instance, a ten-message debugging exchange might compress to "User reported 404 error; agent traced route config; deployed fix; confirmed resolution." The original detail is lost, but the causal chain survives.
Hierarchical buckets
Divide context into tiers: a pinned tier (system prompt, task definition), a working tier (recent tool outputs, active sub-tasks), and a background tier (older conversation, reference documents). Evict from background first, compress working memory second, touch pinned content last.
Practical checklist for implementing compaction
1. Measure token usage per turn. Instrument your agent loop to log the exact token count before each model call. Set a threshold—typically 85–90 % of the window—to trigger compaction.
2. Define task-critical content. Tag or classify which messages must survive: system instructions, the user's original request, open action items, unresolved errors.
3. Choose a compaction method. Start with recency-based eviction plus a pinned tier for critical messages. Add relevance scoring if tasks frequently reference earlier context.
4. Test with long-running traces. Simulate a 50-turn conversation or a multi-step workflow that involves reading large documents. Verify the agent still completes the task correctly after compaction.
5. Log what was dropped. Store discarded messages in a separate audit stream so you can diagnose failures caused by missing context.
6. Monitor task success rates. Track whether compaction degrades outcomes—longer resolution times, repeated questions, incorrect outputs—and adjust your retention heuristics accordingly.
Example: compacting a support ticket thread
Imagine a support agent handling a customer issue over twenty messages. By turn fifteen, the context holds:
- System prompt and agent playbook (2,000 tokens, pinned)
- Original issue description (500 tokens, pinned)
- Twelve exchanges of troubleshooting steps (18,000 tokens)
- Three knowledge base articles (9,000 tokens)
- Recent tool outputs from a log query (6,000 tokens)
When the 35,000-token threshold is hit, a compaction pass:
1. Summarizes the twelve troubleshooting exchanges into a single 800-token paragraph: "User confirmed browser version; agent checked plugin conflicts; escalated to backend team; received fix ETA."
2. Keeps the two most relevant KB articles (5,000 tokens) based on embedding similarity to the active sub-task; drops the third.
3. Retains all recent tool outputs and the pinned content.
New total: 14,300 tokens. The agent can continue for another dozen turns before the next compaction cycle.
Trade-offs and failure modes
Loss of nuance: Summarization risks stripping out the exact error message, edge case or user phrasing that matters for resolution. Pinning the original issue and most recent user messages helps, but intermediate detail can still vanish.
Recency bias: Keeping only recent context can erase root-cause information from early in the conversation, forcing the agent to ask questions the user already answered.
Computational overhead: Scoring every message for relevance, generating summaries, or running a ranker model adds latency. For latency-sensitive applications, simple recency-based eviction may be the only practical option.
Cycle thrashing: If compaction is too aggressive, the agent may lose context needed for the next turn, triggering an immediate second compaction. This can degrade into a loop of forgetting and re-asking.
How Goatfied agents handle long-running tasks
GOATs—the AI agents you hire for specific roles in Goatfied—operate within structured playbooks and escalation rules that naturally limit unbounded context growth. Each GOAT has a defined mission, decision guardrails and a monthly budget, so tasks are scoped rather than open-ended. When a GOAT encounters a multi-step workflow in Operations or a complex incident in Observability, the platform breaks the work into discrete approval steps. This structure reduces the need for aggressive compaction, because each step operates with a fresh, task-specific context window rather than dragging along the entire conversation history.
For workflows that do require long-term memory—such as an IT agent tracking an issue across days or a sales agent nurturing a deal over weeks—Goatfied stores structured state in the relevant MANAGE suite (IT issues, sales deals) and retrieves only the necessary records on demand. This approach keeps the active context lean while preserving full audit trails and historical data outside the model's working memory.
When to compact vs. when to architect around the limit
If your agent regularly hits the context ceiling, ask whether the task design is the real problem. A monolithic "do everything" agent will always struggle with long traces. Alternatives:
- Chain of specialists: Route the task through a sequence of narrow agents, each with a focused context.
- State extraction: After each major step, write key decisions and outputs to a database or structured log, then start the next step with a clean window and a pointer to the stored state.
- Human-in-the-loop breakpoints: Insert approval or review steps that let you reset context while keeping the workflow auditable.
Compaction is a runtime tactic, not a strategy. It buys breathing room, but if you're compacting every five turns, the architecture likely needs a redesign.
Frequently asked questions
What's the difference between compaction and truncation?
Truncation simply drops the oldest messages when the window fills, with no regard for importance. Compaction uses heuristics—recency, relevance, role, semantic similarity—to selectively evict or summarize content, aiming to preserve what matters most for task completion.
Can I use a longer context window instead of compacting?
Longer windows delay the problem but don't eliminate it. Tasks that involve reading dozens of documents, processing logs or iterating over many turns will eventually hit any fixed limit. Compaction becomes necessary when the alternative is terminating the agent mid-task.
How do I know if my compaction strategy is working?
Measure task success rate, number of turns to completion, and whether the agent asks redundant questions (a sign it forgot earlier context). Log which messages were evicted and spot-check whether critical information was lost. A/B test different retention policies on the same task set.
Does compaction introduce hallucinations?
Summarization can introduce errors if the summary misrepresents the original content. To mitigate this, keep summaries terse and factual, prefer bullet points over prose, and retain verbatim copies of critical data (error codes, timestamps, user requests) in the pinned tier rather than summarizing them.
Should I compact proactively or wait until the limit is hit?
Proactive compaction—triggered at 85–90 % capacity—gives you headroom for the next model response and tool outputs. Waiting until you're at 100 % risks an error mid-turn or forces an emergency eviction that may drop something important.