![Cover image for [AI Context #6] Agent Memory Beyond Context (Plans, Memory, RAG)](/assets/images/posts/9c901582-b6d3-47a4-a582-ff7c6393053d/1.jpg)
This installment is about that destination. Where and how should knowledge that must survive the end of a session be stored? The context window is a workbench, while the file system is a warehouse. The workbench is small, expensive, and cleared when the session ends. The warehouse is large, cheap, and persistent. Good agent operation is ultimately logistics design between the workbench and the warehouse.
This installment’s topic is that destination. Where and how to store knowledge that must survive after a session ends. The context window is a workbench; the file system is a warehouse. The workbench is small, expensive, and cleared when a session ends. The warehouse is large, cheap, and persistent. Good agent operation is ultimately logistics design between the workbench and the warehouse.
Plan Files: The Lifeline of Long Tasks
The most basic form of externalization is a plan file. At the start of a task, have the agent create a file such as PLAN.md containing the goals, confirmed decisions, and remaining steps, then update it as work progresses.
This single file solves several problems at once. First, it survives session resets. Whether you run /clear because the context is full or close your laptop and start a new session the next day, reading the plan file lets the work continue. Unlike the uncertain summary produced by auto-compact in Part 3, it is an accurate record whose contents you control directly.
There is also a side effect. If the agent updates and rereads the plan file at every step, the overall goal repeatedly appears near the end of the context. As discussed in Part 2, the model’s attention is strongly drawn to the context’s tail. By exploiting this property, you keep “what was I doing?” in view. This is the remedy for agents that lose the original goal and go off on tangents during tasks spanning dozens of steps. In practice, agent products handling long tasks use this pattern: they keep a to-do list in a file and revise it continuously.
Memory: Knowledge That Accumulates Across Sessions
If a plan file extends the life of one task, memory accumulates knowledge across the entire project. Agents including Claude Code support memory-directory features, and the principle is simple: write each discovered fact to its own file, keep only a summary index loaded continuously, and read the full text when a related task needs it. Part 4’s pointer principle applies here too.
The key issue here is what qualifies for storage. If you write down everything, memory quickly becomes a dump. The bar is “information that cannot be derived elsewhere.” Code structure fails because it can be learned by reading the code; past commits fail because Git remembers them. By contrast, operational rules such as “this project’s dev server must always run in the main worktree,” preferences such as “the user wants formal language,” and environment-specific traps discovered through painful trial and error are worth storing because they are recorded nowhere else.
Stale memories also need to be part of the plan. A fact written six months ago is not guaranteed to remain true. The read/write rules for memory should include “if a referenced memory conflicts with reality, update or delete it.” This prevents incorrect memories from returning as the context poisoning described in Part 2.
RAG: When the Warehouse Is Library-Sized
Plan files and memory cover a few dozen files. But what if the warehouse is the size of a library? Thousands of internal wiki pages, tens of thousands of papers, or hundreds of thousands of customer inquiries. The information you need is certainly somewhere inside, but you cannot load it all into the context.
The technique used here is RAG (Retrieval-Augmented Generation). When a question arrives, the system first searches the warehouse for relevant fragments, then places only a few of them in the context to generate an answer. Search commonly uses embeddings. By converting text into coordinates in semantic space, a question about “refund rules” and a document titled “payment cancellation policy” can land near each other despite different wording, allowing retrieval even when their keywords do not overlap.
For a while, RAG was treated as the only answer for long documents, but coding agents produced an interesting reversal. Agents that can use tools were found to locate needed code quite effectively by repeatedly using grep and exploring files, even without embedding search. Practical thinking today is therefore hybrid. Direct agent exploration is strong for code with precise identifiers, while embedding search is strong for piles of natural-language documents with varied wording. Either way, the principle is the same: do not load everything; load only what is relevant at query time.
Summary
- The context is a workbench, and files are a warehouse. The basic practice is to send knowledge that must remain after a session ends out of the context and into files.
- Plan files rescue long tasks from session resets and repeatedly expose the goal near the end of the context, preventing drift.
- Store only information that cannot be derived elsewhere in memory, keep only the index loaded continuously, and manage stale entries with update and deletion rules.
- When the warehouse is very large, use RAG to load only relevant fragments at query time. Direct exploration is strong for code; embedding search is strong for natural-language document collections.
Only the final installment of the series remains. So far, we have managed context for quality; the last installment is about money. It covers how prompt-cache behavior can make API costs differ by multiples depending on how the same context is built, and why the beginning of the context must not be altered carelessly.