Bury Once, Nibble Cheap: Prompt Caching in an Agent Harness埋一次，吃一冬：agent harness 里的 prompt cache

Some background. JollySammy recently looked up a word and has been insufferable ever since: the computer-science term cache turns out to be his family trade. When a squirrel buries food for winter, biologists literally call it caching. Cones go in bottom to top, new ones only on top, and as long as the buried layers stay untouched he can trot past the pile each morning, count just the cones added since yesterday, and be done before breakfast. The winter his cousin helpfully “reorganized” the bottom layer, he spent three days re-checking every cone above it.

The running example is, as usual in this series, the harness JollySammy keeps in his burrow, and the discussion mostly runs under Anthropic’s caching rules.

1. An agent’s bill is nothing like a chat bill

Every turn, we resend the whole history to the model: the system prompt, every tool definition, every message and tool result so far. The longer the session, the more each turn resends, so over a 50-turn coding session the total input tokens grow roughly with the square of the turn count (turn N carries everything the previous N−1 turns accumulated). The prompt cache changes this arithmetic. A prefix the server has already cached is re-read at 0.1× the base input price on Anthropic’s explicit cache, or 0.5× on OpenAI’s implicit one, and time-to-first-token drops as well. That makes the hit rate the single biggest cost knob on the harness, and it reaches back into every context-engineering decision: what may enter the system prompt, where volatile facts go, when history may be rewritten.

Two turn-20 bills show the stakes. A cache-healthy session:

turn 20 usage (healthy session):
{
  "input_tokens": 412,                  // only the new turn, full price
  "cache_read_input_tokens": 45210,     // all history re-read at 0.1x
  "cache_creation_input_tokens": 380    // small incremental write
}

The same session after a per-turn timestamp sneaks into the system prompt:

turn 20 usage (cache destroyed every turn):
{
  "input_tokens": 0,
  "cache_read_input_tokens": 0,
  "cache_creation_input_tokens": 45990  // rewriting the whole cache at 1.25x
}

The first bill charges forty-five thousand tokens at a tenth of the price. The second charges them at full price plus a 25% cache-write premium, every single turn. The conversations are identical. The difference is one misplaced timestamp, and that gap is this whole post.

2. The server only recognizes a byte-identical prefix

An Anthropic request participates in the cache in a fixed order: tools, then system, then messages. We place a cache_control breakpoint somewhere in the request, and the server hashes everything from the request’s first byte up to that marker and stores it. On the next request it compares from the longest breakpoint down: if a prefix hash matches, that span bills at 0.1×; if it doesn’t, the span counts as brand new, bills at full price, and gets re-cached at a 1.25× write price.

The from-byte-zero definition makes invalidation chain. One upstream byte changes, and every breakpoint after it dies, because each later hash contains the earlier bytes. This is exactly JollySammy’s pile: tools at the bottom, system on top of it, messages above that. Pull a cone from the bottom layer and everything above collapses.

A concrete failure looks like this:

--- WRONG: volatile line inside the stable prefix ---
You are a coding agent.
Current time: 2026-08-18 14:32:08     <-- changes every turn
Tool usage rules: ...
Security policy: ...

The timestamp changes each turn, so everything from that line onward fails the hash each turn, and the entire system prompt plus the whole message history gets rewritten every time. The fix moves volatile content behind the stable content:

--- RIGHT: volatile content pushed behind the stable prefix ---
You are a coding agent.
Tool usage rules: ...
Security policy: ...
__SYSTEM_PROMPT_DYNAMIC_BOUNDARY__    <-- seam marker
gitStatus: branch=main, 2 files modified

The first half never changes and the breakpoint sits at its end. The second half may change however it likes; it lives after the breakpoint, so the changes can’t reach the cached zone.

Section remark: split the request into a stable zone and a volatile zone, and keep the stable zone byte-stable.

3. Stability as structure, not discipline

Reminding everyone in code review to keep timestamps out of the system prompt does not hold. CC-Py classifies content at the assembly layer, so volatile content structurally cannot enter the prefix. System-prompt modules fall into the three groups STATIC, DYNAMIC, and APPENDED, with a boundary sentinel (that fixed __SYSTEM_PROMPT_DYNAMIC_BOUNDARY__ line) inserted at the seam between static and volatile content; we walked the full assembly pipeline in the system-prompt layers post. At packing time, extract_cacheable_prefix finds the sentinel in the serialized string and cuts out everything before it as the static prefix. No sentinel means no cacheable content, full stop. And the most volatile facts, git status above all, never enter the prefix at all: append_system_context appends them to the tail of the system prompt each turn. The source comment says it plainly, git context is “intentionally NOT injected” into the prefix.

4. Two breakpoints, and why only two

The assembly layer guarantees a stable prefix; right before the request goes out, _apply_cache_control translates that structure into markers Anthropic understands. It does two things. First, it splits the system message from one plain string into two text blocks: the static prefix carrying a {"type": "ephemeral"} breakpoint, and the volatile tail carrying nothing, left outside the cached zone. Second, it puts a breakpoint on the last tool definition. Tools sit at the very front of the request, the prefix of the prefix, and since a breakpoint means “cache everything up to here”, one marker on the tail element covers the entire tool block:

# jollysammy/api/llm_client.py (abridged)
if full.startswith(prefix):
    blocks.append({"type": "text", "text": prefix,
                   "cache_control": breakpoint})   # static prefix
    remainder = full[len(prefix):].lstrip("\n")
    if remainder:
        blocks.append({"type": "text", "text": remainder})  # volatile tail
...
if api_tools:
    new_tools = [dict(t) for t in api_tools]
    new_tools[-1]["cache_control"] = dict(breakpoint)  # cache whole tool block

A breakpoint on every tool would buy nothing: it burns the four-per-request quota, and the server computes one extra hash per marker. Pure overhead, zero benefit.

5. Three turns on the meter

Turn 1 goes out: tools (8k tokens) plus static prefix (3k) plus volatile tail (0.2k) plus one user message. The server has never seen this prefix, so it hashes and writes at both breakpoints:

turn 1 usage:
{
  "input_tokens": 300,                  // volatile tail + user message
  "cache_creation_input_tokens": 11000, // tools + static prefix, written
  "cache_read_input_tokens": 0
}

Turn 2 goes out with tools and static prefix unchanged to the byte. The tail’s git status changed, and messages grew by three. The server compares breakpoint by breakpoint: tools hit, static prefix hit:

turn 2 usage:
{
  "input_tokens": 2100,                 // tail + all messages, full price
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 11000      // the whole stable zone at 0.1x
}

Before turn 3, someone adds a new tool. The bytes of tools changed, the bottom cone moved, both breakpoints die:

turn 3 usage (a tool definition changed):
{
  "input_tokens": 2400,
  "cache_creation_input_tokens": 11300, // everything re-written at 1.25x
  "cache_read_input_tokens": 0
}

The rule reads straight off these three turns. While the stable zone holds, we pay full price only for the increment. One changed byte in the stable zone, and the whole cached zone gets rewritten, with the 25% write premium on top.

6. The cache killers we’ve actually met

Volatile facts written into the prefix. Timestamps, build numbers, git state; the sentinel plus tail-append handles this class.

An unstable tool list. Order drift when definitions get rebuilt, MCP servers coming and going and changing the tool set: both alter the prefix of the prefix and zero out the entire request’s cache. The tool registration order has to be deterministic and the tool set stable within a session.

Rewriting the middle of history. Compaction and truncating old tool results both do it. The cheaper alternative avoids the rewrite from the start: large tool outputs go into files on disk, and history keeps only the path plus a short summary, so the bytes already sent never change. The order in which parallel results land in history has to stay stable too, the parallel tool calls post touched this at its end.

Switching models mid-session. The cache is per-model. Per-turn model reselection and the 529-overload fallback both cold-start everything that follows.

Idling past the TTL. Anthropic’s default is 5 minutes, and every hit refreshes the clock, so an active session’s cache effectively never expires. A human-approval pause longer than 5 minutes comes back to a full rewrite.

Two hard boundaries round out the list. A prefix below the minimum cacheable length (1024 tokens for most models, 2048 for the Haiku class) doesn’t cache at all, breakpoint or not. And explicit caching allows at most 4 breakpoints per request. CC-Py uses two, tools and system; production harnesses typically spend the other two on the message side, one on the last message and one on the second-to-last user message, so the history itself hits incrementally turn over turn. That’s a real gap in this port: its messages bill at full price every turn.

7. Reading the bill without fooling ourselves

Sooner or later someone asks what the cache actually saved. The two vendors report the same concept in entirely different shapes:

--- Anthropic response ---
"usage": {
  "input_tokens": 412,            // does NOT include cache fields
  "cache_creation_input_tokens": 380,
  "cache_read_input_tokens": 45210
}

--- OpenAI response ---
"usage": {
  "prompt_tokens": 46002,         // DOES include cached tokens
  "prompt_tokens_details": { "cached_tokens": 42000 }
}

The semantic difference is one sentence, and getting it wrong wrecks everything downstream. Anthropic’s input_tokens excludes the cache reads and writes, so the true input is the sum of all three fields. OpenAI’s prompt_tokens includes cached_tokens, which is a subset of it. Put Anthropic’s input_tokens on a dashboard as “context length” and we read 412 where the context is actually 412 + 380 + 45210 = 46002. Add OpenAI’s two fields together and we double-count. The hit-rate formulas differ accordingly:

anthropic: hit = read / (input + write + read)          -> 45210/46002 ~ 98%
openai:    hit = cached_tokens / prompt_tokens          -> 42000/46002 ~ 91%

CC-Py’s _extract_usage adapts to both shapes: it reads the Anthropic keys first and falls back to prompt_tokens_details.cached_tokens when they’re absent.

Before doing any accounting, the meter itself needs checking. This port has one precise blind spot: the /v1/responses adapter reads only the Anthropic-shaped keys, while that API actually reports hits under input_tokens_details.cached_tokens. On that path the cache metric shows a permanent zero. What looks like “no hits” is “field never read”, and without checking the observation chain first we’d spend an afternoon optimizing a problem that doesn’t exist.

Streaming adds one more trap we’ve already dissected in the streaming reconstruction post: streamed usage events carry running totals, so within one message each field takes the latest non-zero value, and only across messages do we add.

8. The break-even condition is favorable

A write costs 25% extra; a hit saves 90%. With base input price p, the net saving is

net = 0.9 * read_tokens * p  -  0.25 * write_tokens * p

so a prefix pays for its own write the first time it gets reused, and every reuse after that is profit. A multi-turn agent resends this turn’s prefix next turn with near certainty, which is why caching-on is essentially always right for agents. (The 1-hour-TTL breakpoint writes at 2×; it only pays off for workloads whose request gaps predictably exceed 5 minutes, human-approval flows for instance.)

The pricing table has a deliberate quirk:

# jollysammy/core/query_engine.py (abridged)
_MODEL_PRICING = {
  "claude-sonnet-4-5": {"input": 3.0, "output": 15.0,
                        "cache_read": 0.30, "cache_write": 3.75},
  # gpt-4o rows carry 0.0 -- OpenAI already includes cached tokens
  # inside prompt_tokens; adding them again would double-count.
  "gpt-4o": {"input": 2.50, "output": 10.0,
             "cache_read": 0.0, "cache_write": 0.0},
}

The Sonnet row encodes exactly the official multipliers, 0.30/3.0 = 0.1× and 3.75/3.0 = 1.25×. The zeros in the gpt-4o row are intentional: the cached tokens are already inside prompt_tokens, and charging them again via cached_tokens would be double billing. The cost is that the table doesn’t model OpenAI’s implicit 50% discount, so estimates on that path run high, a directional error we can live with.

In production we log the (input, write, read) triple per request, tagged by session, agent type, and model, and watch two curves. In-session hit rate should climb past 90% within a few turns and stay there. And the cache-write share should stay small; a persistently high write share means the prefix is being rewritten, and every dollar of write is a hit that should have happened and didn’t. The hit rate never reaches 100%, since each turn’s fresh user message and the previous assistant output are always new tokens at full price.

9. Subagents ride the same meter

When we dispatch a subagent from a dedicated agent definition, that agent’s prompt becomes the static base of the subagent’s own system prompt and then runs through exactly the same sentinel-and-breakpoint pipeline as the main loop. So every agent definition is its own cache entry, with its own stable prefix, independent of the main session’s.

One consequence follows immediately: repeated dispatches of the same agent definition share one cache entry. A review pipeline that opens 10 parallel verifiers of the same type pays the cache write on the first request, and the other nine read at 0.1×, the way a print shop charges a plate-making fee for the first copy and paper money for the rest. The precondition is identical tool subsets. Different tool lists mean different tools blocks, and since tools are the very front of the prefix, the cache keys diverge from the first byte; ten verifiers become ten cache entries and nobody rides anybody’s cache.

Fork-style subagents flip the intuition here. Inheriting the full parent context sounds expensive, tens of thousands of tokens on the very first request, but those tokens are precisely the prefix the parent already wrote into the cache, so a fork’s first request hits almost entirely:

--- fork subagent (inherits full parent history) ---
first request usage:
{ "input_tokens": 900, "cache_read_input_tokens": 48000,
  "cache_creation_input_tokens": 400 }     // nearly free start

--- fresh dedicated subagent (own prompt, first of its kind) ---
first request usage:
{ "input_tokens": 200, "cache_read_input_tokens": 0,
  "cache_creation_input_tokens": 12500 }   // cold start, pays the write

A fork’s marginal input cost is mostly the 0.1× re-read. What’s expensive is the context it grows on its own afterwards: new tokens, full price, and thrown away when its session ends.

The cache also sharpens an architecture argument, agent definitions should be few and stable. Merging 50 micro-specialized agents into 5 general ones with parameterized instructions, differences in the user prompt, common ground in the system prompt, immediately improves the hit structure. A system-side byte may be re-read at a tenth of the price hundreds of times; a user-side byte is consumed by this one session and never again.

10. Racing cache writes in a concurrent fan-out, and the fixes

The main session dispatches 8 same-type subagents, and they fire their first requests in the same millisecond. From the server’s view, 8 requests arrive carrying the same prefix, but the cache doesn’t hold it yet, the first write hasn’t landed, so all 8 are judged misses. All 8 bill at full price, each pays its own 1.25× write premium, and the cache ends up holding exactly one copy. Money spent 8 times, goods delivered once. This is the racing cache write:

naive fan-out (8 agents at once):
  8 x (12000 tokens full price + 12000 x 1.25 write premium)
staggered fan-out:
  1 x (12000 full + write premium)     <- the warm-up
  7 x (12000 x 0.1 cache read)         <- the rest ride the cache

Three fixes, all cheap. Warm up: send one agent first (or a scout request with a tiny max_tokens) to write the prefix into the cache, then release the remaining seven. Micro-stagger: a few hundred milliseconds of spacing inside the batch, enough for the first response’s cache write to land. Keep definitions stable: put agent system prompts and tool subsets under version control, so a deploy doesn’t silently change the cache key. One changed key, and every warm-up was for nothing.