A support agent has been working through a customer case for two hours. It has read the contract, queried three systems and drafted a reply. Then it quotes a refund amount from the wrong invoice, and nobody can say why, because every fact it needed was somewhere in the session.
Teams usually respond by buying a larger context window or by trimming the history harder. Both can help, and both can cause the exact failure they were meant to prevent. The discipline that sits between them is context engineering: deciding, step by step, what the model sees.
For teams that run models on their own hardware, our local inference work is where this becomes a hardware question, because every token kept in context occupies GPU memory for as long as the session lives.
Curated context and careful compaction can raise accuracy and cut cost, but compaction that is too aggressive silently removes the facts an agent needs, so the work is to decide what must survive and then test that it does. This article sets out the evidence, the trade-offs and a checklist a technical lead can apply this week.
Scope note: the six parts of an agent harness, including a short context section, are covered in agent harness architecture. Here we go deeper on context alone.
Why a Larger Window Is Not Better Recall
Context rot is the informal name for output quality falling as the input grows. The most careful public test is Chroma’s Context Rot report of July 2025. It evaluated 18 models and states that performance consistently degrades with increasing input length, even on simple retrieval tasks. Lower-similarity targets degraded faster, and a single distractor hurt accuracy while four hurt more. Chroma builds a vector database, so read the report as vendor research, though its methods are laid out in detail.
A second line of evidence comes from the NoLiMa benchmark, published at ICML 2025. It tested 13 models that claim support for at least 128K tokens, using questions that cannot be answered by matching keywords. According to the authors, GPT-4o fell from 99.3% at short lengths to 69.7% at 32K tokens, and 11 of the 13 models dropped below half of their short-context baseline at 32K.
Position inside the window is more disputed. Liu et al. found that models do best when relevant information sits at the start or end of the input. In their 20-document test, GPT-3.5-Turbo fell from 75.8% to 53.8% when the answer moved to the middle. That study covers 2023 models. Chroma, testing newer ones, reported no notable variation by needle position. Treat “lost in the middle” as something to check on your own model rather than a rule.
A caution on dates. Everything above was measured on 2025 or older models. As of November 2026, Anthropic lists a 1M-token window for its current models, and the recall tests above predate that generation. Capacity is a specification. Recall is something you have to measure.
What Context Engineering Covers
Anthropic’s guidance on effective context engineering frames context as a finite resource with diminishing marginal returns. Its documentation on compaction says the active context is kept small because response quality degrades as a conversation grows. These are statements of vendor position, but they match the independent measurements above.
The techniques fall into a small set, each with its own failure mode.
| Technique | What it does | Where it goes wrong |
|---|---|---|
| Just-in-time retrieval | Loads documents or records only when a step needs them | Slower than pre-computed retrieval, and the agent can search poorly |
| Tool-result clearing | Removes old tool outputs once they have been used | Cleared content is gone from the model’s view |
| Compaction | Summarises a long history and continues from the summary | Details, identifiers and earlier instructions can disappear |
| Structured notes | Writes state to a file outside the window | Notes go stale, or the agent forgets to read them |
| Subagents | Hands a subtask to a fresh context and takes back a short result | Extra cost, and the summary returned can omit what matters |
Anthropic’s post makes the same point about compaction in its own words: overly aggressive compaction can lose subtle but critical context. The rest of this article is largely about where that line sits.
What Curation Saves in Money
Cost is the better-measured side of the argument. Every token in context is billed on input, and in self-hosted setups it also occupies memory.
Hosted models. Anthropic’s documentation (as of November 2026) lists cache reads at 0.1 times the base input price, or 0.05 times on Opus 5.5 and Sonnet 5.5, with a default cache lifetime of five minutes. A stable prefix of instructions is therefore cheap to resend. Anthropic’s context editing documentation warns that clearing old tool results invalidates cached prompt prefixes, so the cleared amount has to be large enough to justify rewriting the cache. Prices and beta features change, so check them before you commit to a design.
Self-hosted models. The memory held for past tokens, the KV cache, grows linearly with context. Sebastian Raschka’s worked example for Qwen3-8B (36 layers, 8 KV heads, head dimension 128, bf16) comes to 144 KiB per token. At 32,768 tokens that is about 4.5 GiB for a single sequence. This is the logical size of the cache and excludes allocator overhead, fragmentation, batching and quantisation, so real deployments differ. Even so, it shows why a session that keeps 30,000 tokens of stale tool output costs GPU memory that another session could have used. Our article on self-hosted AI cost control shows where these costs sit in a full budget.
Measured savings. The JetBrains Complexity Trap study compared strategies for a software-engineering agent on SWE-bench Verified across five model configurations. Observation masking, which hides old tool outputs but keeps the agent’s own reasoning and actions, cut instance cost by 50 to 57% in most configurations. LLM summarisation cut cost by 41 to 55% in most configurations. The cost result is solid. The accuracy result is not, as the next section shows.
Anthropic reported in its context management post of September 2025 that memory plus context editing improved performance by 39% in its internal agentic-search evaluation, that context editing alone gave 29%, and that token use fell by 84% in a 100-turn web-search test. These are Anthropic’s own evaluations. The post names no public benchmark and does not state the baseline, so read them as a direction rather than a forecast for your workload.
Where Compaction Loses Facts
Compaction trades detail for space, and the detail it drops is rarely the detail you would have picked.
Factory’s evaluation of compression, published in December 2025, scored three summarisation methods across sessions totalling 36,611 messages on a 0 to 5 scale with an LLM judge. Overall scores were 3.70 for Factory, 3.44 for Anthropic and 3.35 for OpenAI. The artifact dimension, which covers which files and objects the agent touched, was the lowest for all of them: 2.45, 2.33 and 2.19. The article itself says artifact tracking remains an unsolved problem. Two caveats apply. Factory evaluated its own product against two competitors, and the judge was a language model.
JetBrains found the same fragility from another direction. LLM summarisation lowered the solve rate for Gemini 2.5 Flash with thinking from 40.4% to 31.4%, and for Qwen3-32B from 17.0% to 16.0%, a drop the paper does not treat as significant. The authors also report that summarisation consistently leads to longer trajectories, which the authors attribute to summaries masking failure signals and encouraging the agent to keep going. Masking was not safe either: it lowered the solve rate for Qwen3-32B and for Gemini 2.5 Flash with thinking, and raised it for other configurations.
Anthropic’s post on long-running agents reaches a practical conclusion. It credits compaction as a context management capability and still states that compaction is not sufficient for long-running work. Its remedy is ordinary files: an initializer step, a progress file and git history, so that a new session can resume from written state. A map file such as AGENTS.md, telling each session where things live, is Orange ITS’s own addition and not part of Anthropic’s recipe.
What does this look like in a business process? The facts that an agent must never lose tend to be small and exact: a customer number, an approved discount, a rule such as “never contact this client by phone”. A summary is a poor place to keep them.
A Context Budget Checklist for a Technical Lead
Think of the window as a budget with four accounts. Decide what goes into each account before the agent runs, then verify afterwards.
Keep in context at all times
- The task definition and the rules the agent must follow, kept short.
- Tool descriptions for the tools in actual use. Remove the rest.
- The current step and the immediate previous result.
Retrieve on demand
- Policy documents, product data, contract text and customer history. Fetch the relevant section when a step needs it and drop it afterwards.
- Reference tables. Query the record instead of pasting the table.
- Anything over a few thousand tokens that only one step uses.
Clear once used
- Large tool outputs such as full search results, file listings and API responses. Anthropic’s context editing (a beta feature as of November 2026) clears old tool results at a configurable trigger. Its documented defaults are a 100,000 input-token trigger and the three most recent tool uses kept. The client keeps the full history, but the model no longer sees what was cleared.
- Dead ends and abandoned attempts, which are the kind of distraction Chroma found harmful.
Write to notes outside the window
- Decisions, open questions and the status of each subtask.
- Identifiers the agent will need again, kept in a structured file.
- A progress log so the next session, or the next model, can resume without the history.
Never leave to a summary
- Instructions and prohibitions given by a person during the session.
- Identifiers such as customer numbers, order IDs, invoice numbers and file paths.
- Amounts, dates and thresholds that drive a decision.
- Approvals, with who gave them and what they covered.
Write these to a note or a database field the harness controls, and re-inject them after any compaction. The model should read them from there rather than recall them.
Test compaction like any other change
Build a small set of cases where a fact introduced early must be used late, then run it before and after you change compaction settings, model or instructions. Draw cases from real failures, as described in our guides on testing and evals for AI agents and on building evals from real tickets. Score the outcome, such as the correct refund amount, and ignore how fluent the summary looks.
Two measurements are worth tracking from the start. One is the share of must-survive facts still correct after compaction. The other is cost per completed task, since a cheaper run that needs more steps may not be cheaper overall.
What Is Still Contested
Two questions have no settled answer, and a sensible plan accounts for both.
Summary versus masking. Factory favours structured summaries, and its own evaluation supports that. JetBrains found that masking matched or beat LLM summarisation at lower cost in its tests, but its solve-rate results changed sign across configurations, so neither approach wins outright. Factory’s evaluation uses its own product and scoring. A fair reading is that masking is the cheaper default for tool-heavy agents and summaries need the testing described above. Your workload may differ from SWE-bench, so measure on it.
Context files. An ETH Zurich study (Gloaguen et al., arXiv 2602.11988) tested context files for coding agents. It found that they raised inference cost by over 20% on average and did not generally improve task success. LLM-generated files cost 20% and 23% more on two benchmarks, and developer-written files cost at most 19% more. Developer-written files gained 2.4% on average, a difference the authors report as not significant. They add that such files are useful for specifying non-standard practices. The evidence covers Python repositories and coding agents. The study does not cover other agent workloads, so there is no basis to extend the result to a support or finance agent, or to dismiss it.
Instruction files are widely recommended in practitioner writing, which sits in tension with the study. Both can be true: a short file that states what the agent cannot infer helps, and a long overview that repeats the codebase adds cost. The practical rule is to make each line earn its place, and the way to know is to run your eval set with and without it.
Measure Before You Trim
Context engineering is easy to treat as a tuning exercise, and the evidence says it is closer to a trade with a measurable price on each side. Fewer tokens lower cost and, in some setups, reduce the distraction that degrades answers. Fewer tokens also mean fewer places where a customer number or an approval can live.
That leaves a clear order of work. Decide which facts must survive, store them outside the window, apply the cheapest trimming that your tests allow, and rerun the same test set whenever the model, the window or the compaction setting changes. A team that can show that its refund amounts and approvals survive a compaction has an agent it can trust with longer sessions, and a team that cannot show it has a cost saving it has not yet verified.
Frequently asked questions
What is context engineering?
Context engineering is the practice of deciding what information an AI agent sees at each step and what it does not. Anthropic describes context as a finite resource with diminishing marginal returns, so the goal is the smallest set of information that still gets the job done. It covers instructions, retrieved documents, tool results, notes and the history that is carried forward.
What is context rot?
Context rot is the decline in answer quality as the input grows. Chroma tested 18 models and reported that performance consistently degrades with increasing input length, and that distractors make it worse. The results come from 2025 models on controlled tasks, so measure the effect on your own agent before you design around it.
Does a bigger context window solve the problem?
No. As of November 2026 Anthropic lists a 1M-token window for its current models, but window size measures capacity and says nothing about recall. Chroma and the NoLiMa authors both found accuracy falling as input grows on 2025 models. A larger window also means more tokens to pay for, so curation still reduces cost.
Is context compaction safe for production agents?
Only with testing. Compaction reduces cost, but summaries drop details. Factory scored artifact tracking lowest for every method it evaluated, and JetBrains saw LLM summarisation lower the solve rate in some configurations. Protect instructions, identifiers, amounts and approvals by writing them to external notes, and test that these facts survive compaction.
Should we write a long instruction file for our agent?
Keep it short and test it. An ETH Zurich study of coding agents found that context files raised inference cost by over 20% on average and did not generally improve task success. Developer-written files fared better than generated ones and helped with non-standard practices. The study covers coding only, so run your own comparison.