Context Rot: Why Long Context Degrades LLM Answers

Context rot is the decline in LLM answer quality as the number of input tokens grows, and it shows up long before the context window is full. It is strongest when the relevant fact shares few words with the question and similar but wrong passages sit nearby.

A 1M-token window describes how much a model accepts as input, not how much it uses reliably. In Chroma’s July 2025 report, quality dropped as input grew on all 18 models tested. In the author’s measurement on Claude Haiku 5.5, accuracy fell from 97% at ~1K tokens to 60% at ~95K, even though the input took up less than 10% of the window.

This article covers what research from 2023 to 2026 says about context rot, why the standard needle-in-a-haystack test overstates model ability, what a small reproducible measurement showed, and what to do about it in agents and long-context pipelines. What a context window is, how large it is on different models and what happens on overflow is covered in the article on the context window.

What is context rot

Definition and origin of the term

The term caught on after Chroma’s technical report Context Rot: How Increasing Input Tokens Impacts LLM Performance, published July 14, 2025 (authors Kelly Hong, Anton Troynikov and Jeff Huber), and then entered vendor vocabulary. Who used the phrase first could not be established.

Anthropic’s engineering blog post of September 29, 2025 defines context rot this way: “as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases.” The same post adds: “While some models exhibit more gentle degradation than others, this characteristic emerges across all models.” Hence the “attention budget” metaphor: context is a finite resource with diminishing returns. The context windows documentation puts it more briefly: “As token count grows, accuracy and recall degrade, a phenomenon known as context rot.” So what sits in the window matters as much as how much room is left.

How context rot differs from overflow and lost in the middle

Context rot is not window overflow. On overflow, the API returns an error or the application truncates the history, and the model never sees part of the text. With context rot, all the text is there, but the model uses it less well.

It also isn’t just “lost in the middle.” The position effect from the Lost in the Middle paper explains one kind of miss. Context rot is broader: quality drops as length grows even when position stays fixed, and it depends on how closely the question matches the relevant passage and how many distracting passages surround it.

What the research shows

Almost all the work below uses synthetic tasks and models from 2023-2025. You can’t carry its numbers over to current Opus 5.5, Sonnet 5.5, Haiku 5.5 or the GPT-6 family: no comparable published tables for these models could be found.

Lost in the Middle: position

In 2023, Liu and co-authors showed a U-shaped curve: models make better use of information at the start and end of the input than in the middle. The paper became the foundation of the topic, but it was done on 2023 models. On newer models and other tasks the effect can be weaker or different, so “put the important parts at the edges” isn’t universal advice, as the context engineering guide already covers.

RULER and NoLiMa: effective length is shorter than claimed

RULER (NVIDIA, 2024) tested 17 models that claim windows of 32K tokens or more on 13 synthetic tasks. In the authors’ words, “only half of them can maintain satisfactory performance at the length of 32K.” The tasks are harder than a single lookup: multiple needles, variable tracing chains, aggregation.

NoLiMa (Adobe Research and LMU Munich, ICML 2025) removed the main giveaway of the classic test: the needle contains none of the question’s words, and the link is only through world knowledge. Of 13 models claiming at least 128K, “at 32K, for instance, 11 models drop below 50% of their strong short-length baselines.” GPT-4o went from 99.3% on short input to 69.7% at 32K.

The NoLiMa repository README has an effective length table: the longest input at which a model keeps at least 85% of its short-input baseline. An excerpt (2025 runs):

The claimed window is 16x the effective length for GPT-4o and roughly 250x for Gemini 2.0 Flash. These are 2024-2025 models. No such table exists for the late-2026 generation.

Chroma: distractors, similarity, haystack structure

Chroma tested 18 models from Anthropic, OpenAI, Google and Alibaba. Task difficulty was held constant and only input length changed. Answers were scored by a GPT-4.1 judge, which the authors iterated on until its agreement with human labels was above 0.99. The report counts 194,480 LLM calls in total. This article doesn’t give per-model drop percentages: they appear only in the report’s charts and the repository data, and restating them from memory would be unreliable.

The report’s qualitative findings:

• Question-needle similarity. Measured as cosine similarity of embeddings (how that works is covered in the embeddings article), averaged across five embedding models. In the authors’ words, “as needle-question similarity decreases, model performance degrades more significantly with increasing input length.” On short inputs, even low-similarity pairs are solved well.
• Distractors. “Even a single distractor reduces performance relative to the baseline (needle only), and adding four distractors compounds this degradation further.” Different distractors don’t have equal impact.
• Failure mode depends on the model family. “Claude models consistently exhibit the lowest hallucination rates” and tend to say no answer can be found. GPT models more often gave confident wrong answers.
• Haystack structure. A shuffled haystack produced better results than a coherent one on all 18 models: “structural coherence consistently hurts model performance.” The authors didn’t establish why.
• Focused input. On LongMemEval (306 questions), every model tested did better on a focused prompt of about 300 tokens than on the full one of about 113K. Claude showed the largest gap, mostly because it declined to answer under ambiguity.
• Needle-haystack similarity had an inconsistent effect. The authors note that testing across only two topics is not enough to generalize.

Agentic search in 2026: premature termination

Xia, Wang, Huang and Liu, in Diagnosing and Mitigating Context Rot in Long-horizon Search (June 2026), studied deep-search agents on four flagship models and three benchmarks. The main failure mode: “models give up or provide uncertain incorrect answers long before exhausting the context window,” even on easy and moderately difficult queries. The authors compared seven context management methods and describe them as ways to lower peak context length, after which the model is more likely to keep searching instead of stopping.

Anthropic’s prompting documentation makes a related observation: “Claude might sometimes naturally attempt to bring its work to a close as it approaches the context limit.” The vendor recommends telling the model in the prompt that its context will be compacted and that it should finish the task.

Why needle-in-a-haystack overestimates models

The classic needle-in-a-haystack test (Greg Kamradt, November 2023) hides one fact in a long text and plots a length-by-depth grid. A grid that’s “green across the whole window” refers to the easy version of the task. The sources above give five reasons why that result doesn’t carry over to real work:

1. Lexical overlap. If the question repeats words from the needle, the model finds the passage almost like a string search. Without overlap, scores fall much faster with length: in NoLiMa, GPT-4o dropped from 99.3% to 69.7% at 32K, and in Chroma the drop is steeper the lower the question-needle similarity.
2. An out-of-place needle. A fact about one thing inside a text about something else stands out on its own. Chroma tested this factor and found no consistent effect across two topics, so this is a hypothesis, not an established fact.
3. No distractors. In real data, similar wrong passages sit next to the answer. In Chroma, a single distractor already lowers the score.
4. One fact instead of synthesis. RULER, Michelangelo and MRCR add aggregation, latent structure and telling identical passages apart. Results on these tasks are worse than on a single needle: in HELM Long Context, the best MRCR score was 0.256 against a best five-task mean of 0.588.
5. The conclusion from 1-4, not a vendor quote: “100% across the grid” means the model can find a string that resembles the question. It says little about reliability on a task with paraphrasing, distractors and synthesis.

What is known about the mechanism, and what isn’t

The research above describes behavior, not internals. What is supported:

• Length. Quality drops as input grows while task difficulty stays fixed (Chroma, RULER, NoLiMa).
• Position. A U-shaped curve on 2023 models (Lost in the Middle); on newer models the effect is weaker or different.
• Question-answer similarity. Length amplifies the dependence on how closely the question matches the relevant passage (NoLiMa, Chroma).
• Distractors. Similar wrong passages lower scores (Chroma), and model families fail differently: they abstain or they hallucinate. These are different failure modes, not “better” and “worse.”
• Reasoning. In LongBench v2, a model with long reasoning beat both direct-answer models and humans. In agentic search (Xia et al.), long context instead pushes the model toward stopping early. The sources don’t support “reasoning cures context rot”; “it helps on some tasks” is the accurate version.

What we don’t know. The collected sources don’t prove an architectural cause. NoLiMa’s authors suggest the attention mechanism has a harder time in long contexts when literal matches are absent, but they offer this as an interpretation of results, not a measured cause. Explanations like “attention gets spread thin across tokens” or “positional encodings fade” are plausible but not measured in this work. Anthropic’s “attention budget” is a metaphor, not a measurement. Why coherent text hurts more than shuffled text is unknown. The accurate framing is observed behavior, not mechanism.

Author’s measurement: context rot on Claude Haiku 5.5

To check the effect on a current model, the author ran a small synthetic task modeled on NoLiMa and Chroma on November 8, 2026. The model was Claude Haiku 5.5 (released October 2026, 1M-token window), called via OpenRouter. This is an illustration, not a benchmark.

Setup

• Haystack. Three Project Gutenberg novels: Pride and Prejudice, Emma and Middlemarch, with headers and license text removed. On the Haiku 5.5 tokenizer this came to 3.04 characters per token.
• Needles. Five custom sentences where the question shares no words with the needle, and the link runs only through knowledge of a landmark. Example: the needle “Marta’s flat is two minutes’ walk from the Sagrada Familia.” with the question “Which person in the document has a connection to Barcelona?” The script checks that the needle names don’t appear in the novels.
• Distractors. Sentences of the same shape with other names and other cities (Rijksmuseum, Charles Bridge and so on): 0 or 4 of them at random positions.
• Grid. Target lengths of 1K, 8K, 32K, 64K and 96K tokens, needle position at 10, 50 and 90% of the haystack, 0 or 4 distractors, 5 needles. 150 requests in total, 0 API errors.
• Parameters. Reasoning off (reasoning: {enabled: false}), max_tokens 30, no temperature (Haiku 5.5 doesn’t accept it). The document sits in tags above the question, as Anthropic recommends. No caching. Actual length was taken from usage.prompt_tokens: 1,056 to 95,017 tokens, all below the 100K threshold above which Haiku 5.5 input costs 5x more.
• Scoring. No LLM judge: an answer containing the needle’s name is correct, “NOT FOUND” is a refusal, anything else is an error.

The core of the script (the full version adds text loading, needle checks and parallel execution; chat and summary wrap OpenRouter’s POST /api/v1/chat/completions and parse the response):

def build(chars, pos, needle, n_distr, rng):
    out, size = [], 0
    for p in PARAS:  # novel paragraphs in order
        if size + len(p) > chars and out: break
        out.append(p); size += len(p) + 2
    out.insert(int(pos * len(out)), needle[1])
    distr = [p for p in POOL if p[2] != needle[2]]  # other cities
    for d in rng.sample(distr, n_distr):
        out.insert(rng.randrange(len(out) + 1), d[1])
    return "\n\n".join(out)

def ask(doc, city):
    prompt = (f"<document>\n{doc}\n</document>\n\n"
              f"Which person in the document has a connection to {city}? "
              "Landmarks count as evidence. Answer with the name only, or NOT FOUND.")
    return summary(*chat(model="anthropic/claude-haiku-5.5", max_tokens=30,
                         reasoning={"enabled": False},
                         messages=[{"role": "user", "content": prompt}]))

def grade(ans, name):
    a = (ans or "").lower()
    if name.lower() in a: return "ok"
    if "not found" in a: return "abstain"
    return "wrong"

Results

Accuracy by input length, all positions and both distractor conditions combined (N = 30 per length, 95% Wilson interval):

Accuracy falls with length from 97% to 60%, even though the longest input takes up less than 10% of the model’s window. The intervals at ~1K and ~95K don’t overlap, so the trend isn’t noise. With a sample this small, don’t trust the exact percentages, but the direction holds.

Nearly every miss was a refusal, not a made-up answer: 117 correct answers, 32 “NOT FOUND”, 0 clearly wrong names and 1 ambiguous answer. The ambiguous one came at ~95K with distractors: the model started quoting the Rijksmuseum distractor sentence and the answer was cut off at max_tokens. This matches Chroma’s observation that Claude models abstain more often than they hallucinate, but one model on one task can’t test that pattern.

Difficulty depended heavily on how distant the association was. Sagrada Familia → Barcelona and Brandenburg Gate → Berlin each scored 28 of 30, while Alfama → Lisbon scored only 14 of 30, and at ~8K this needle drew a refusal in half the cases. A Lisbon neighborhood name is more weakly tied to the city than the most famous landmarks, and it breaks earlier. This is the same question-answer similarity effect seen in NoLiMa and Chroma, on different material.

What wasn’t confirmed

The hypothesis before the run had two parts: with distractors, accuracy falls faster with length, and position matters less than length. Only the second part held.

Distractors had no consistent effect: at ~32K results were even better with them (13 vs 11), at ~95K worse (8 vs 10). At N = 15 per cell, a difference like that is within noise. This measurement neither confirms nor refutes Chroma’s distractor result: the sample is too small, and the distractors here are about other cities and don’t even partially answer the question.

Needle position at lengths of 32K and above had no noticeable effect either: 23 of 30 with the needle at 10% of the haystack, 19 of 30 at both 50% and 90%.

A methodology lesson from the first run

The first run on the same grid had to be discarded. The question was “Which character has been to Barcelona?”, and on ~1K without distractors the model found the answer in 1 case out of 15. Across the whole grid it got 23 of 150 right, with 127 refusals and no wrong names: the model simply didn’t infer “lives near the Sagrada Familia, so has been to Barcelona” from that wording. Debugging on short input showed that the wording “has a connection to… Landmarks count as evidence” scores 15 of 15.

In any synthetic test, check baseline accuracy on short input first. If the model can’t solve the task at 1K, the drop with length measures how well the question is worded, not context rot.

Limitations of the measurement

• One model, one task family: retrieving a single fact through an association. Not an agentic session and not a real RAG pipeline.
• Five needles written by the author, of varying difficulty; N = 30 per length and 15 per cell, so the intervals are wide.
• One domain and language for the haystack: 19th-century English prose. The model may have seen these novels in training. Results on code or tables may differ.
• Reasoning was off. With it on, results may differ; LongBench v2 hints at a benefit from reasoning.
• Haiku 5.5 doesn’t let you set temperature, so answers are nondeterministic. All 150 requests were served by a single OpenRouter provider, and another provider may behave differently.
• Lengths are in Haiku 5.5 tokens. The same text gives a different token count on another model; see the article on tokens.
• Insertion position is computed by paragraph, not by token. There was no control run without a needle.

The main run cost $0.60, and the discarded run cost the same.

What to do about context rot

Anthropic’s general principle: good context engineering means “finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome.” The full set of context assembly methods is in the context engineering guide. Below are only the ones tied directly to degradation.

Keep context short

The best-supported fix is focused input: in Chroma’s LongMemEval experiment, every model tested did better on ~300 tokens than on ~113K. Practical options:

• Retrieval instead of loading everything. When you can find the relevant passage in advance, pass that instead of the whole corpus. When to pick which approach is covered in the comparison of context engineering and RAG.
• Load on demand. Anthropic calls this “just in time” context: agents “maintain lightweight identifiers (file paths, stored queries, web links, etc.)” and load data with tools when it’s needed. Claude Code works this way: “CLAUDE.md files are naively dropped into context up front,” while glob and grep retrieve files as needed.
• Clear old tool results. Old tool call results are rarely needed again. Anthropic writes that “one of the safest lightest touch forms of compaction is tool result clearing.” In the API this is context editing.
• Tool definitions. In an agent, tool schemas also fill the window. How much room MCP servers take and how to trim them is covered in a separate breakdown.

Documents at the top, question at the end

For inputs of 20K tokens or more, Anthropic’s documentation recommends placing long documents near the top of the prompt, above the query, instructions and examples. The same page notes: “Queries at the end can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs.” It also suggests asking the model to quote the relevant parts of the documents first and then answer from those quotes.

Ordering doesn’t fully fix degradation. In the measurement above, the document sat at the top and the question at the end, and accuracy still dropped from 97% to 60%.

Subagents and notes outside the window

A subagent gets a clean window, spends tens of thousands of tokens in it and returns a condensed summary, which per Anthropic is “often 1,000-2,000 tokens.” This has a price: in measurements of Claude Code subagents, a subagent in a heavy setup starts with about 55-58K tokens of context. On a task with 1-3 tool calls that doesn’t pay off; on a long task it does.

Structured notes outside the window (a NOTES.md file, a to-do list, the memory tool) let you bring back only the state you need instead of the whole history. How to design that kind of memory without it turning into a dump is covered in the article on agent memory.

How to test on your own task

1. Collect 10-30 examples of a real task with known answers.
2. Make sure the model gets them right almost every time on short input. If it doesn’t, fix the wording before you measure length.
3. Add more plausible but irrelevant text around the relevant passage: 1K, 8K, 32K and beyond, up to the lengths your product actually sees.
4. Count correct answers, refusals and errors separately: a refusal and a confident error need different handling.
5. Take the length from the API response’s usage, not from a character count.
6. Add similar wrong passages if your data has them, and compare.

How compaction fails

What compaction does

As Anthropic describes it, compaction takes a conversation that is nearing the window limit, summarizes it and continues in a new window with that summary. Claude Code keeps architectural decisions and unresolved bugs and drops redundant tool outputs. The API has server-side compaction (in beta for Claude 4.6 and later models), and Claude Code has the /compact command. The lightest form is clearing old tool results.

Compaction shortens the input, so it works against context rot. But lossy compression adds failure modes of its own. The collected sources contain no direct measurements of compaction failures, so each row below says where it comes from.

What to keep in a summary at a checkpoint is listed in the context engineering guide.

When to start with a clean window

For tasks that span several windows, Anthropic’s documentation suggests to “consider starting with a brand new context window rather than using compaction,” because “Claude’s latest models are extremely effective at discovering state from the local filesystem.” The condition: the state lives in files (progress notes, tests, git log), and the starting prompt says explicitly what to read first. Then the new window starts with short, accurate context instead of a summary that may have lost details.

The practical rule that follows: if the work splits naturally into stages and each stage’s state can be written to a file, a clean window is more reliable than repeated compaction. If the state lives only in the conversation, compaction is the only option, and then at least keep control of its prompt.

Limitations and open questions

• Almost all data on context rot comes from 2023-2025 models. For Opus 5.5, Sonnet 5.5, Haiku 5.5 and GPT-6, no published long-context measurements could be found in primary sources, only aggregators.
• Chroma’s per-model drop figures aren’t given here: they’re available only in the report’s charts and repository data.
• The mechanism of degradation hasn’t been established; only the observed behavior is known.
• Compaction failures are described from the definition and indirect data, with no direct measurements.
• The author’s measurement: one model, one task, a small sample, with all the caveats above.
• Anthropic’s “up to 30 percent” figure comes from a documentation page with no revision date and may change.

Primary sources

• Hong, Troynikov, Huber. Context Rot: How Increasing Input Tokens Impacts LLM Performance. Chroma, July 14, 2025. Code.
• Anthropic. Effective context engineering for AI agents, September 29, 2025.
• Anthropic docs. Context windows, Prompting best practices, Pricing, snapshot of November 8, 2026.
• Liu et al. Lost in the Middle: How Language Models Use Long Contexts. TACL, arXiv July 6, 2023.
• Hsieh et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? COLM 2024, arXiv April 9, 2024.
• Vodrahalli et al. Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries. arXiv September 19, 2024.
• Bai et al. LongBench v2. arXiv December 19, 2024.
• Modarressi et al. NoLiMa: Long-Context Evaluation Beyond Literal Matching. ICML 2025, arXiv February 7, 2025. Repository with the effective length table.
• OpenAI. MRCR, April 12, 2025, corrected December 5, 2025.
• Stanford CRFM. HELM Long Context, September 2025.
• Xia, Wang, Huang, Liu. Diagnosing and Mitigating Context Rot in Long-horizon Search. arXiv June 29, 2026.
• Kamradt. LLMTest_NeedleInAHaystack, November 2023.