[AI Context #7] Cut API Costs with Prompt Caching

The final part of the series is about money. If you have run an agent through an API, you may have noticed something odd on the bill. Input tokens cost vastly more than output tokens. Given the architecture from Part 1, that is only natural. Every turn resends everything from the system prompt through the full conversation history, so in a 50-turn agent session, the same system prompt is effectively billed 50 times.

Prompt caching is the mechanism that cuts down this repeated cost. Used well, it can reduce input costs to one-tenth and speed up responses, but there is one condition: you must build the context in a cache-friendly way. This part examines how it works and the easy mistakes that unknowingly wipe out the cache.

Do not recompute the same prefix

We already saw half of the principle in Part 1. When a model processes input, it produces intermediate attention results for each token, which accumulate in GPU memory as the KV cache. Prompt caching keeps this KV cache after a request ends instead of discarding it, then reuses the entire computation for a region when the next request begins with the same content.

See how well this fits agent conversations. The input on turn 10 is “system prompt + turns 1–9 + new message.” The input on turn 11 is “system prompt + turns 1–10 + new message.” The beginning is completely identical. With a cache, the server retrieves the results already computed through turn 10 and only computes the newly appended tail.

Billing follows the same structure. With the Anthropic API, cached input tokens cost only 10% of the list price. Newly written cache tokens incur a 25% surcharge, but that is quickly offset in agent patterns that write once and read many times. In a session lasting hundreds of turns, caching can mean a several-fold difference in cost. Processing time also skips the cached region, noticeably reducing time to first token.

One condition: the prefix must not change by even one character

There is no free lunch. The condition for cache reuse is prefix matching. Caching applies only to the identical contiguous region from the very start of the input; from the first point where the content differs onward, everything is recomputed.

Part 1 also explains why. In attention, each token’s KV values depend on every preceding token. If even one earlier token changes, the computed results for every token after it change as well, so the cache cannot be used. That is why modifying the beginning of the context is so expensive.

The principle for cache-friendly context can be summarized in one sentence: put stable content first, variable content later, and never modify history—only append to it (append-only).

Careless mistakes that wipe out the cache

The principle is simple, but there are many subtle ways to violate it.

The first is a variable value inside the system prompt. What happens if you insert the current time down to the second into the system prompt? Since the very beginning of the context changes with every request, the cache hit rate becomes 0%. If you must include it, use a coarser granularity such as the date, or place it toward the end of the context. Session IDs and random values are the same kind of landmine.

The second is changing tool definitions. Tool schemas usually appear at the very top of the context, so adding or removing a tool midway through a session invalidates the cache for everything after it. In Part 4, I recommended cleaning up MCP servers; from a caching perspective, this adds one more line: clean them up before the session starts, and leave them alone afterward.

The third is editing history in the middle. Some people try to save context by modifying a previous turn or deleting old turns, but that breaks the prefix and destroys the cache from that point onward. Trying to save a few thousand tokens can easily become a lopsided trade that incurs recomputation for the entire session. If you need to remove something from history, it is usually better to close out the current thread and clear it wholesale.

Finally, Part 3’s /compact looks different from a caching perspective as well. Compression replaces the entire history with a new summary, completely resetting the cache. The later turns do benefit in the long run because they build a new cache on top of shorter history, but remember that “compact is not free housekeeping; it is an event with a cache-rebuilding cost.” That is why /clear at each completed thread is often cheaper than frequent compacting.

Series summary: seven ways to think about context

Let’s fold the principles running through all seven parts into one line each.

1. The context window is finite, and its limits come from the physics of attention computation and the KV cache.
2. Just because content fits does not mean it will be used well. Length itself degrades quality.
3. Make /clear the default at every task boundary, and use /compact with an instruction only when continuity is needed.
4. The system prompt, CLAUDE.md, and tool definitions are fixed costs. Keep only what is always true, and keep it short.
5. Isolate work that is heavy in process but light in conclusion in a subagent’s context.
6. Externalize knowledge that must survive across sessions into files: plan files, memory, and RAG when it grows large.
7. Context must be built append-only for caching to pay off. Put stable content first and variable content later.

Reduced to one sentence: context is not the model’s capability but a budget designed by the user. This principle will remain valid even in the era of million-token contexts. As the window grows, the judgment to choose what to place inside it will determine the performance gap.