[AI Context #2] Why Long Context Ruins Answers

In Part 1, we saw why the context window is finite. But looking at today’s model specs raises a question: models with 1 million tokens already exist, so why not just feed in the entire codebase or document? Is there any point in being careful?

This is the misconception we’ll challenge in this part. Fitting inside the context window and using the content well are completely different problems. Multiple studies have confirmed that model performance declines as context grows, and anyone who has run agents for a long time has felt it firsthand: responses become scattered toward the end of a session, and the agent tries to fix bugs it already fixed.

lost in the middle: the middle goes unread

The 2023 paper “Lost in the Middle” by Stanford researchers is one of the most frequently cited results in this area. When the researchers embedded answer documents at various positions in a long context and measured how well models found them, accuracy followed a U-shaped curve. Models found information at the beginning and end well, but retrieval dropped sharply for information in the middle. In severe cases, performance with the answer placed in the middle was lower than when the document was omitted entirely.

Models do “read” all 20 documents. They just don’t allocate attention evenly. The pattern is similar to a person focusing on the introduction and conclusion of a thick report while merely skimming the main body; it also appears in attention weights.

The “needle in a haystack” scores often shown in model announcements can hide this problem. The test embeds one sentence in a long text and asks the model to retrieve it verbatim; most current models score close to perfectly. Real-world work is not simple retrieval. When the task requires connecting and reasoning across information from multiple places, the same model’s score slips as context grows.

context rot: length itself degrades performance

In 2025, vector DB company Chroma published a technical report naming this phenomenon “context rot.” They tested 18 leading models on the same task while increasing only the input length, and performance consistently declined simply because the input got longer, even when task difficulty stayed the same. This happened within the context-window specification.

Several causes overlap. First is the structural limitation of attention. Attention weights are like a fixed budget, so as the number of tokens grows, each token receives a thinner share. The more irrelevant content is mixed in, the more the important signal gets buried. Next is the training-distribution problem. Most text the model saw during training consists of short documents, so inputs hundreds of thousands of tokens long are unfamiliar situations. The model may support that length according to its specification, but that does not guarantee reasoning quality at that length.

In practice, you need an intuition for the “effective context.” Even with a 200,000-token specification, it is safer to assume that the range where quality holds on complex reasoning tasks is much shorter. The spec number means “you can put this much in without an error,” not “you can put this much in and the model will remain smart.”

An agent’s context isn’t merely long; it’s messy

Agents make the problem one step worse. As we saw in Part 1, an agent’s context accumulates tool-call results, and this accumulation is not merely long—it also contains contradictions.

Consider a typical scenario. An agent reads a file to fix a bug, edits it, and reads it again. The context now contains the same file both before and after the edit. It also contains failed and successful test logs. There is no guarantee which one the model will reference for its next decision. This is how it sees the old version, says “There’s still a bug,” and starts a fix that is already complete.

These failure patterns have names. context poisoning is when incorrect information, such as a hallucinated summary, enters the context and contaminates later decisions in a chain; context distraction is when accumulated history becomes so long that the model is drawn into repeating past patterns instead of following new instructions; context confusion is when similar but different information causes confusion; and context clash is when contradictory information conflicts. Even if you don’t know the names, the symptoms are familiar. An agent that repeats the same mistake unusually often late in a session or does what it was told not to do is usually exhibiting one of these four failures.

Signals of quality degradation

If you notice the following signals in a long session, suspect a context problem.

• It reopens problems that were already solved or repeatedly reads the same file
• It starts violating rules set early in the session, such as coding conventions or files not to touch
• Responses become verbose and drift toward a topic from much earlier rather than the immediately preceding question
• It ignores information just provided and answers using outdated information somewhere in the context

The important point is that the model is not “getting tired.” It is a stateless machine that rereads everything from the beginning on every turn. The model has not changed; the state of the context it must read has. Output quality drops because the input is long, messy, and contradictory.

Summary

• Being in the context does not mean information will be used well. Both lost in the middle, where retrieval of middle information declines, and context rot, where length itself degrades performance, have been confirmed by research.
• The specification number is an upper bound, not a quality guarantee. The more complex the reasoning required, the shorter the effective context should be considered compared with the specification.
• An agent’s context accumulates tool-call debris and can contain contradictions. The four failure patterns—poisoning, distraction, confusion, and clash—are the main culprits behind late-session quality degradation.

Now that diagnosis is complete, the next part begins the treatment. We start with the basic tools, /clear and /compact. Both “clear the context,” but their operating principles are completely different, and using them incorrectly can erase the work context instead. We’ll establish when to clear and when to compact.

Continue reading

• The complete guide to /clear vs /compact: when to clear and when to compact an AI agent’s context
• [Agent Design #2] Context Engineering: How to Use the Context Window Like a Budget