![Cover image for [AI Context #1] Why Do Agents Forget Instructions?](/assets/images/posts/04ff8585-2d8d-44f7-87e9-2a975fd9176a/1.jpg)
When using a coding agent like Claude Code, you may see a warning such as “Context left until auto-compact: 8%” at the bottom of the screen. Chat-based AI does not usually show this. Yet surprisingly few people can explain why the number decreases or what happens when it reaches zero. If you explicitly said “Don’t touch the tests” at the start of a session and, an hour later, the agent is casually editing test files, this number is almost certainly involved.
This is the first article in a series covering AI-agent context, from fundamentals to practice. Practical techniques such as when to use /compact and /clear, and why to split work among subagents, all rest on the constraint that “the context window is finite.” So this first installment starts at the foundation. We unpack what a context window really is and why it cannot grow indefinitely using three keys: tokens, attention, and the KV cache.
Tokens: The Units AI Uses to Count Text
The unit of a context window is neither a character nor a word, but a token. An LLM cannot read text as a whole; its tokenizer receives a sequence of small pieces. The splitting rule comes from BPE (Byte Pair Encoding) algorithms, whose principle is simple: character combinations that frequently occur together in the training data are grouped into one chunk. Common combinations such as “the” or “ing” become a single token, while rare words are split into multiple pieces.
This reveals an important fact for Korean users. Because tokenizer training data is overwhelmingly English-heavy, Korean generally uses 1.5–2 times more tokens than English for the same content. A specification of “200,000 context-window tokens” amounts to two full-length novels for English documents, but the usable capacity feels far smaller with Korean documents.
A context window is the maximum amount the model can read at once, measured in tokens. The system prompt, conversation history, attached documents, and the answer the model is currently generating must all fit within this limit.
Why Does an Agent Fill Its Context So Quickly?
There is another fundamental fact to understand about LLMs: the model is a stateless machine. Once it handles a request, it stores its contents nowhere. From the model’s perspective, yesterday’s conversation—and even the previous turn—never existed.
So how does a conversation continue? The answer is almost comically simple. On every turn, the app sends the entire history again, from the system prompt through the present. A model writing its tenth answer has not “remembered” the previous nine conversations; it has just read them again from the beginning.
For chat-based AI, this history consists mainly of sentences typed by a person and the model’s replies, so it grows slowly. Agents are different. Each agent turn includes tool calls. Reading a file adds the entire file contents to the context; running tests adds hundreds of lines of failure logs; searching adds all search results. The user may have entered only “Fix the bug,” but if the agent reads ten files and runs the build three times, that single turn consumes tens of thousands of tokens. That is why coding-agent interfaces show the remaining context. What takes days to exhaust in chat can happen in an hour or two with an agent.
When the limit is reached, the app has to discard something. It may trim the oldest turns or compress them into a summary. If an instruction such as “Don’t touch the tests,” established early in the session, gets pushed out during this process, the model starts the next turn as if it never made that promise. Its memory has not deteriorated; there was never memory to begin with, and the instruction was simply pushed outside the readable range.
Finite Reason #1: Attention’s Quadratic Cost
Why not simply increase the window to around 100 million tokens? The transformer’s core operation, attention, gets in the way.
When processing a new token, attention calculates how relevant it is to every preceding token. If a variable named user appears in code, this is what enables the model to connect it with a declaration 3,000 lines earlier. It is the source of an LLM’s apparent ability to understand context—and also the source of its cost.
Because every token looks at every other token, computation grows with the square of the sequence length. If the context becomes 10 times longer, attention computation jumps 100 times. Optimizations such as sparse attention, sliding windows, and FlashAttention have enabled models with 1 million tokens, but the fundamental fact remains: the ability to reference the full context carries a sharply increasing cost as length grows.
Finite Reason #2: The Memory Bill Called the KV Cache
Computation is not the only problem. Memory comes with a separate bill called the KV cache (Key-Value cache).
Recomputing the entire context from scratch every time the model generates one token would be wasteful. Instead, the intermediate attention results calculated for each token—the Key and Value vectors—are stored in GPU memory and reused. That is the KV cache. The problem is that this cache is stored separately for every layer and every attention head, so it grows in direct proportion to context length.
For a rough sense of scale, an open-source model with around 70 billion parameters uses about 300 KB of KV cache per token even with the memory-saving technique GQA. Filling the context with 128,000 tokens makes the cache alone roughly 40 GB, consuming the memory of an entire high-end GPU independently of the model weights. This is why expanding the context window is a hardware-and-money problem, not merely a software setting.
This cost is reflected directly in user fees. LLM APIs charge in proportion to input-token counts, and when combined with resending the entire history every turn, this explains why long sessions become more expensive and slower over time.
Summary
- A context window is the maximum number of tokens a model can read at once. Korean uses 1.5–2 times more tokens than English, so you should not take the specification at face value.
- LLMs are stateless. Conversational memory is a reread of the entire history on every turn, and because agents also accumulate tool-call results, their context fills much faster than chat context.
- The window is finite because attention computation grows quadratically with length, while the KV cache occupies GPU memory in direct proportion to length. Context is both a capability and a cost.
The next installment challenges a common assumption: that bigger context is always better. In reality, multiple studies have confirmed that model performance declines as context grows. We cover lost in the middle, context rot, and why the specification number differs from effective capacity.