Summary: Although token cache reads enjoy a 90% discount, deliberately evicting stale tool outputs at each turn boundary, even though it disrupts the LLM provider's prefix cache, can lower total inference expenses by nearly 50% while safeguarding against hard context-window failures. This post explores how accepting the upfront cost of breaking the cache eliminates the accumulation of "dead" tokens, offering a surprisingly cost-effective strategy for agentic workflows.
The Context Accumulation Tax
Autoregressive LLMs are stateless. To maintain task continuity, an autonomous coding agent must append every new action to the conversation history and resend the entire payload on every model turn (this was in fact the genius of ChatGPT when it was released almost four years ago). Without intervention, token consumption scales at O(N²) relative to the number of turns.
In realistic coding tasks, this context payload is not dominated by user prompts or high-level reasoning. It is dominated by ephemeral tool telemetry. In our benchmarking exercise, raw file reads and bash logs alone accounted for over 80% of the accumulated tool context.
The utility of this tool telemetry data decays rapidly:
- When an agent runs a test suite, it needs the verbatim bash output to write the immediate patch.
- Once the patch is written and the user's prompt is resolved, the agent almost never needs to read that exact 1,000-line test log again.
Yet, by default, standard agent architectures retain these dead tokens indefinitely.
The False Security of Prompt Caching
Modern LLM providers—including Anthropic and OpenAI's latest frontier models—use prefix-matching KV caches to discount repetitive context. Across these frontier tiers, a standard model prices a cache read at roughly 0.1× the base input cost, but penalizes a new cache write (prefix invalidation) at 1.25×.
Because cache reads are so cheap, conventional wisdom dictates that you should never mutate the context prefix. Agents retain all historical tool outputs to avoid triggering the 1.25× cache-write penalty.
However, there’s a tipping point.
Let T be the token volume of an obsolete tool output. Let K be the number of subsequent model calls in the remainder of the session. Let S be the size of an eviction stub (e.g., ~3 tokens).
If we retain the dead output to preserve the cache, the cumulative read tax is:
If we evict the output, we pay a one-time penalty to rewrite the downstream prefix, but subsequent calls only pay for the tiny stub S:
The Insight: Replaying hundreds of thousands of dead tokens across dozens of subsequent calls (K > 50) incurs costs that far exceed the one-time penalty of rewriting the cache prefix.
Architecture: Turn-Boundary Eviction
To capitalize on this, we implemented turn-boundary eviction. This operates differently from traditional monolithic context compaction, which waits for the context to hit a dangerous ceiling (e.g., 900k tokens) before using an expensive LLM call to summarize the history.
Instead, turn-boundary eviction operates deterministically and silently at the exact boundary where an agent finishes a user prompt:
- Intra-Prompt Protection: While the agent is looping to solve the current prompt (reading files, running bash commands), all tool outputs remain in full. This ensures zero loss of operational fidelity and maintains 100% cache hits during the active task.
- Inter-Prompt Eviction: The moment the agent hands control back to the user, our harness clears every tool item that hasn't been cleared.
There is no size limit and no grace period. Identifiers and tool names are preserved so the schema remains valid, but the payloads are gutted:
| Item | Before | After |
|---|---|---|
| Tool result | Full output | [evicted] (~3 tokens) |
| Edit/Write call | Path, oldText, newText | { path } |
| Bash call | Full command | First 60 characters |
| Reasoning / User text | Full text | Never touched |
Cache Dynamics in Practice
Because eviction happens once per prompt completion, it forces exactly one cache rewrite per prompt.
Furthermore, because the resulting stubs are tiny, the actual rewrite footprint is small. In our session, across 111 clearing passes, the extra cache rewrite volume was only 140k tokens total. This means the agent paid roughly 1 token in rewrite penalties for every 6 tokens of dead context it permanently eliminated. After that single rewrite, the new, much smaller prefix stabilizes for the subsequent prompts’ inner loops.
Empirical Results
We measured this against an extensive, multi-hour developer session consisting of hundreds of complex agent interactions and over a thousand tool executions.
Without eviction, long debugging runs act like a snowball. Less than halfway through the session, the unpruned context would have breached standard provider context limits, causing hard runtime failures. With turn-boundary eviction enabled, peak context footprint was reduced by over 40%, keeping the session safely within operational boundaries.
| Metric | Actual (With Eviction) | Counterfactual (Without Eviction)* |
|---|---|---|
| Gross Context Tokens Sent | Baseline (1×) | 2.2×–3.2× higher |
| Cache Reads | Baseline (1×) | 2.3×–3.3× higher |
| Cache Writes | Baseline (1×) | 1.5×–1.9× higher |
| Total Session Cost | Baseline (1×) | +75% to +137% more expensive |
*Counterfactual is presented as a range between theoretical chars/4 estimation and calibrated logged growth.
Trade-offs and Failure Modes
The turn-boundary eviction approach is highly effective, but it is not flawless. Our data revealed three specific trade-offs:
- The Re-Read Tax: Eviction is lossy. In this session, the agent made numerous read calls to files it had already read in an earlier prompt. Because the file was evicted to an
[evicted]stub, the agent had to spend time and tokens pulling the data from disk again. - Intra-Prompt Vulnerability: Because eviction only happens at the boundary of a user prompt, a model that gets stuck in a massive debugging loop within a single prompt will still bloat the context window until the turn resolves.
- Contextual Blindspots: An all-or-nothing 3-token stub removes valuable metadata. If an agent needs to know if a historical test passed or failed, or what line numbers a file contained, the
[evicted]string provides zero clues.
Future Work: Semantic Stubs
The natural evolution of this architecture is to replace the strict [evicted] string with semantic stubs. By keeping a one-line summary—such as an exit code, a line-count range, or a truncated signature—we can eliminate the blind spots and reduce the re-read tax, while still capturing 99% of the mathematical cost savings of eviction.