Deep dive · for builders
How cached context, tools, documents, and thinking modes shape AI application design
If you call the API directly, write agents, or ship anything that assembles prompts for a model, prompt caching is not a billing footnote. It is one of the places where model mechanics quietly turn into architecture.
Assumes two facts from Where Have All the Tokens Gone? — that a request carries the whole conversation on every turn, and that reuse of the unchanged opening is what makes it affordable. If you already know both, you are in the right place.
What you should be able to do by the end. Explain what a cache hit actually skips, spot the quiet changes that destroy reuse, design request layouts that cache well, and decide when caching is worth caring about at all.
Vendor note. The mechanics are general; the names, limits, headers, and controls in the examples are Claude/Anthropic-flavored. Treat the API details as examples to verify against your provider, not universal rules.
Self-hosted note. This guide assumes a managed model API where the provider — e.g., Anthropic, OpenAI, or Google — owns the inference engine, cache policy, billing meters, and the final prompt format the model receives. If you run your own model and serving stack, the same prefill and KV-cache mechanics still matter, but the engineering problem changes: you own memory pressure, batching, eviction, cache lifetime, routing, and observability. That deserves its own guide.
Five stages of inference, the bill they produce, and the caching mechanism behind the trick. Everything later depends on this picture, so we start by opening the box.
Where the cache comes from
When you send a message to an AI model, it does not read the way a person does and then compose a complete answer. Your request moves through a small pipeline. The text is broken into fragments called tokens. Those tokens are converted into embeddings — long lists of numbers the model can process. The model then reads the entire prompt in a stage called prefill. As it reads, it creates a working memory called the KV cache. Finally, it generates the response one token at a time, consulting — and adding to — that memory as it decides what comes next.
In simplified form:
The KV cache is not a stop along the way — it is the bridge. Prefill fills it once; generation keeps reading from it, and adding to it, one token at a time. That is why the two bars overlap under the KV cache instead of meeting at a clean edge: the teal bar's claim on it ends where the orange bar's claim begins.
Hold onto that split: first the model reads, then the model writes. That distinction explains why long prompts can delay the first response, why output streams gradually, why cached input can cost less, and why input and output tokens are priced differently.
Before going further, here are the same five words again, slower and with less hand-waving.
The model has no notion of letters, or even of words. It works from a fixed vocabulary of text fragments — common words are usually one fragment, rarer or longer ones break into several. The first step is just chopping your text into that vocabulary. Those fragments are the unit everything downstream is counted and billed in, and producing them costs essentially nothing.
Each fragment is then swapped for a long list of numbers. That is the whole step: turning a piece of text into something arithmetic can be done on. Fragments that tend to appear in similar company end up with similar numbers, which is where the model’s sense of what relates to what begins. Also close to free.
Now the model itself runs over the input you already sent. Every fragment is weighed against every fragment before it, repeatedly, building up an account of what the text says and which parts bear on which. This is the model reading the request, not writing the answer yet. Almost all the input-side cost is spent here.
That reading leaves notes behind: for every fragment, a record of what the model worked out about it. The notes exist so that writing the answer does not mean re-reading the prompt from the top for every new word. During generation, new answer fragments add notes to this live working memory too. Prompt caching is the separate provider feature that can keep the input-side notes for reuse on a later request.
Now the model writes, one fragment at a time. Each one is chosen by consulting the notes, and each adds a note of its own before the next begins. This step is usually called decode or generation, not prefill. Reading happened once, for the whole prompt; writing happens step by step, once per fragment. That asymmetry is why the two halves of your bill look so different — which is the next module.
Let's walk through a couple of turns and the distinction gets easier.
A turn is one request in and one response out.
Your app sends the instructions, tools, conversation so far, and the user's latest message. The model reads that input during prefill and builds the KV cache, the notes it will use while writing the answer. Billing: normal input cost. If the beginning is likely to repeat — for example, the same tools, system instructions, and earlier history — the provider can save those notes for later.
The model writes the answer. Billing: normal output cost, because this answer is fresh work for this request.
Your app sends the conversation again, now including the previous answer as history. That answer is no longer “output” in the billing sense; it is text inside the new input. You already paid the output rate when the model generated it on turn 1. Now, because your app sent it back, it is billed again as input. If the beginning is unchanged, the provider can load the saved KV cache instead of running prefill over it again. Billing: cheaper cache-read rate for the reused beginning, which can include the previous answer; normal input rate for the latest user message at the end.
The model writes the next answer. Billing: normal output cost again, because this new answer is generated fresh too.
One subtlety is worth naming: while the model is writing an answer, it also keeps a live KV cache for the tokens it has just generated. That helps this request keep writing coherently. It is not the same thing as a prompt-cache hit on the next request.
For prompt caching, earlier model text only becomes reusable after your app sends it back as input. That is the rule: prompt caching saves rereading; it does not save the model from generating the next answer.
From here on, I will use prefix for that reusable beginning of the request: the part that starts at the very first token and stays unchanged long enough for the provider to recognize it again.
Most trouble with caching starts as a mental model carried in from somewhere else. These are the three it gets confused with, and clearing them changes how the rest of this guide reads.
Request anatomy
Now that you know how a request is processed, look at what is actually inside one. In a normal ChatGPT or Claude conversation, the message you type is only one piece of what gets sent to the model.
Before each response, the application assembles a complete request. I will call that application layer the harness: the code that gathers system instructions, tool definitions, skill or agent instructions, safety rules, your preferences or profile, relevant conversation history, retrieved documents, tool results, and finally your latest message.
system instructionstool definitionsskill / agent instructionssafety rulesyour preferences & profileconversation historyretrieved documentstool resultsyour latest message
The model has to process this entire assembled request before it can generate the first output token — none of it is optional reading. Prompt caching does not make the request disappear, and it does not make new material free. It changes the rate for the pieces the provider can recognize and reuse.
Stable pieces like system instructions, tool schemas, and your profile may qualify for a cheaper cache-read rate on later requests. New material — your latest message, recent tool results, any instruction that just changed — is still processed at the normal input rate.
Once you look at the request as pieces, the input bill splits into three tiers: ordinary input, cache write, and cache read.
| Tier | What happened to this piece | Roughly |
|---|---|---|
| Input miss | No cache involved at all — an ordinary prefill | $5 / M |
| Cache write | First time this reusable beginning is seen; its KV tensors get stored for reuse | ~$6.25 / M |
| Cache read | A previous write is reused instead of recomputed | ~$0.50 / M |
The rate moves in only one direction from a plain miss: a write costs more than doing nothing special, not less, because the provider is not just running prefill — it is also storing the resulting tensors so a later request can skip that work. A hit is where the saving actually shows up, at roughly a tenth of the base rate. That write premium matters: a piece that never repeats pays the write price and never reaches the hit tier that earns it back. Caching is a bet that a prefix will be read again soon enough to justify storing it.
Other providers expose the same idea differently. OpenAI may show uncached input and cached input rather than this exact write/read table. Gemini separates cached token use from storage duration. The pricing shape is provider-specific; the mental model is the same: reused input can be cheaper, while new input and output are still billed normally.
When should you care? Care when you have a large stable prefix, repeated calls, the same model, and enough reuse within the cache lifetime. Care less when prompts are short, requests are one-off, the prefix changes every time, or most of the spend is long generated answers. In those cases, caching may be technically available and still not be the lever that matters.
The other half of the bill
Input, at least, has tiers you can move between. Output has exactly one price, and every major vendor sets it at a multiple of input — five to eight times, depending on the model. They did not agree on that number between them, which is the tell: it is not a pricing convention. It falls out of the two stages in the pipeline above, and it decides what caching can and cannot save you.
On Claude Opus 5, input runs $5 per million tokens at its base rate and output runs $25 — a 5× multiple that OpenAI and Google land within a couple of points of, on their own models, independently. Absolute rates move constantly; the ratio is the durable part, and unlike input’s three tiers, there is no cheaper way to buy it.
Three words carry the rest of this part. Two of them name the stages; the third is the unit those stages are measured in.
One trip through the model, start to finish. Everything below is counted in these.
Also called prompt processing. The model reads your entire input in a single pass and builds the KV cache from it.
Also called generation. The model produces output tokens one at a time, each in its own pass, reusing that KV cache instead of reading the prompt again. You will also see inference used loosely for this stage — strictly it means the whole request, prefill included.
Reading a prompt takes a single pass, however long the prompt is. Writing an answer takes one pass per token — and no amount of hardware changes that.
Every token is already in front of the model — you supplied them, so there is nothing to wait for.
So they all go through together, in one trip. 100,000 tokens still cost a single pass.
None of it exists yet, so every pass has to wait for the one before it to produce the token it needs.
Nothing here can be parallelized away. 1,000 tokens cost 1,000 passes.
That one-pass versus many-passes split is the useful intuition, but it is not the whole story. Reading a prompt lets the system spread a lot of work across many input tokens at once. Writing an answer is different: each new token depends on the tokens that came before it, so the model has to keep stepping forward one token at a time.
Providers make this more efficient behind the scenes by serving many requests together, but they cannot turn generation into the same kind of bulk read that prefill is. That is why output can stay several times more expensive per token even after all the engineering tricks are applied.
This creates an odd-looking bill. Output tokens cost more one by one, but a real request often contains far more input than output. A request with 100,000 input tokens and 1,000 output tokens can still spend between twelve and twenty times more on input overall. So both statements are true: output is expensive per token, and input can dominate the total bill.
| Scenario | Input | Output | What dominates cost |
|---|---|---|---|
| First time you send a large context | 100k fresh | 1k answer | Input reading |
| Another question with the same context | 100k read | 1k answer | New input and output |
| Deep reasoning on a shorter context | 20k cached | 10k reasoning + 1k answer | Output generation |
Knowing when to stop
This is the one short detour into output mechanics, because caching only helps input. To know whether caching matters for your workload, you need to know what can still run away on the output side. There is no completion check anywhere in the loop.
Every decode step produces a probability distribution over the whole vocabulary. One entry in that vocabulary is a special end-of-sequence token — not a signal the model raises, just an ordinary token it can predict like any other. Generation continues until that token is the one selected. Nothing else stops it.
The capital of France is ▮
The capital of France is Paris. ▮
Illustrative distributions, not measured. The shape is the point: mid-sentence, ending is almost impossible; after a complete answer, it dominates. Bars below a few per cent are drawn at a floor width so they stay visible — read the numbers, not the lengths.
The token gets there through training. Documents in the corpus end, and the boundary is marked; the model learns where that marker tends to fall. Instruction tuning and reinforcement learning then push on exactly that — a large part of what post-training does is move end-of-sequence probability to where a person would have stopped.
“It does not know it is done” is true, but not mysterious. There is no completion check — correct. But the probability of ending is conditioned on everything written so far, and post-training shapes it deliberately. The model is not verifying that it answered you; it is reproducing where finished text stops. Those come apart exactly when you would expect: unusual formats, tasks with no natural ending, instructions it has not internalised.
There is also an outside limit. Every model has a maximum number of tokens it can
generate in one response, and your request may set a lower ceiling with a parameter
like max_tokens or max_output_tokens. So generation can
stop for two very different reasons: the model predicted an ending, or the output
budget cut it off.
Only the first is the model ending naturally.
The others are outside controls or handovers. Each response carries a
stop_reason naming which one fired.
stop_reason before touching the content array.
The operational rule. A truncated answer and a finished answer can look
identical if you only read the text. Any pipeline that parses, stores, or acts on
model output should check stop_reason before it trusts the content.
Prefix matching
What gets stored is not text — it is the KV cache: the key and value tensors from the transformer's prefill pass. Everything else follows from that one fact.
A breakpoint is the cache marker your application harness places in the
request. It tells the provider, “cache everything from the beginning of the
request through here.” In the example below, the breakpoint sits after
msg 2, so msg 3 is outside the cached part.
msg 3
cache read
the stored KV cache is loaded; msg 3 is read normally
A cache hit has to start at the beginning because each later token depends on the tokens before it. Change something early, and the notes after that point no longer match. Append something at the end, and the earlier notes can still be reused. Prefix matching is not a design preference; it falls out of how attention works.
Vendor note. Breakpoints are Claude/Anthropic-style API controls, not a universal model feature. The harness chooses where to place them, and Anthropic currently allows up to four per request. OpenAI-style prompt caching is more automatic: you usually do not place breakpoints yourself; the provider reports how many input tokens were served from cache. The general rule is shared, though: reusable cache spans start at the beginning of the request.
Cache lifetime is provider-specific. Claude/Anthropic exposes 5-minute and 1-hour TTL options on cache controls. OpenAI-style prompt caching is usually automatic and has its own retention behavior. The general lesson is simpler: cached input is temporary, and a long enough gap turns the next request back into a cache miss.
The silent floor
Each provider sets a minimum cacheable prefix size for each supported model. A prefix below that minimum produces no cache entry, no warning, and no error. The request simply costs full price forever, which makes this one of the hardest caching bugs to notice.
The number itself is provider- and model-specific. It can run from the low hundreds of tokens to the low thousands depending which model you call, and it may change across a model family's version history. Look it up for the specific model in your request rather than assuming last quarter's number still applies.
How to detect it. Check the provider's cache-creation metric on the first
request. On Claude/Anthropic-style APIs, that is
usage.cache_creation_input_tokens. If it comes back
0 when you expected a write, your prefix may be under the
floor — not misconfigured. Either make the reusable prefix longer so the
span clears the minimum, or accept that this prompt is too small to be worth
caching.
Caching is fragile in specific, knowable ways. These are the changes that cost you everything, and the ones that cost you nothing.
What breaks it
A change wipes its own tier and everything below it. Which means some edits are cheap, while changes near the front of the request are expensive.
| Change | Tools | System | Messages |
|---|---|---|---|
| Add or remove a tool · switch model | LOST | LOST | LOST |
| Edit the system prompt | KEPT | LOST | LOST |
| Change tool_choice · toggle thinking | KEPT | KEPT | LOST |
| Append a message | KEPT | KEPT | LOST |
Read the table from left to right. A cache entry only survives while everything before it stays the same. If you change something near the front of the request, every cached span that depends on it becomes unusable. If you change something near the end, earlier spans can still survive. That is why invalidation is tiered instead of all-or-nothing.
Tools render at position 0 — so adding a single tool destroys the entire cache. Model switches are equally total, since caches are model-scoped.
datetime.now() in the system promptjson.dumps() without sorted keysThe invalidation people forget
When a long conversation gets summarized to make room, the message array is no longer just growing at the end. Older turns are replaced by a summary, so the next request is no longer byte-for-byte the same history. Any cached span that included the old history can no longer match.
You detect the threshold, make a model call to summarize the older turns, and rebuild the array yourself. Full control over what is kept and when it fires — and full responsibility for the cache damage, since you decide where the rewrite starts.
Some providers expose compaction as an API feature. Anthropic, for example, offers server-side compaction as a beta: opt in, and the API summarizes earlier context for you automatically as a conversation approaches a trigger threshold.
Claude/Anthropic-specific note. With Anthropic's server-side compaction, you must append the whole response back into your messages each turn, not just the text you extracted from it. The response carries the blocks the API uses to replace the compacted history on the next request. Pull out the text and append only that, and those blocks are gone: no error, no warning, and the conversation quietly loses the summarized state.
Design around it, not against it. Compaction is unavoidable on long-horizon work — turn counts are unbounded and windows are not. What you can control is where the rewrite starts. Keep your frozen prefix (tools, system, standing instructions) entirely outside the compacted span and it survives untouched, so you lose the conversation cache but keep the expensive one. A compaction implementation that rewrites from position 0 throws away everything, every time.
You usually cannot control the provider's final rendering of a request. What you control is what goes into each field, which pieces stay stable, and where volatile material appears.
Prompt assembly
Written for readers building on the API. It assumes Part 1 — prefix matching, provider-specific cache controls, and the size floor.
Different providers expose different controls. Claude-style APIs give you explicit cache breakpoints. OpenAI-style prompt caching is more automatic on many models. But the shared rule is the same: cache matching happens against the beginning of the final request, so the harness still matters. Put stable material early, keep noisy material late, and use cache markers where your provider gives them to you.
Stable content must physically precede volatile content, because a change invalidates everything after it. Sort your prompt by how often each piece changes, then place cache markers at the seams when your provider exposes them.
Cache markers are an implementation detail. Some providers let you place explicit markers; others cache the prefix automatically and only report how much was reused. Do not let the API shape distract from the strategy: stable beginnings are easier to reuse, volatile endings are cheaper to change, and giant prefixes are still giant even when they cache well.
The model knows the general idea of APIs, CRMs, SQL databases, and calendars from training. It does not know your live tool surface unless the harness describes it. Native tools, REST or OpenAPI definitions, MCP-provided toolsets, and skills all have to be introduced somehow — and whatever gets introduced becomes part of the request.
A connected tool is not automatically context. Your app may have an MCP server loaded in memory, or a REST client ready to call, while the prompt contains only a short name and description. That is good. The expensive move is advertising the entire capability surface up front: every endpoint, every parameter, every object definition, every custom field.
The CRM case is the warning sign. Thousands of custom fields and
object definitions can overwhelm the useful context before the user has asked
anything CRM-related. Better designs expose discovery tools:
search_crm_fields, describe_object,
list_relevant_fields, fetch_schema_slice. Let the
model ask for the slice it needs instead of making it reread the whole CRM
universe every turn.
Skills follow the same shape. A skill is a good home for long procedures, field guides, and vendor playbooks because a short description can be visible first and the full body can load only when needed. Keep skill descriptions precise and non-overlapping; otherwise the menu of skills becomes its own kind of noisy prefix.
Side note: deep research is still fan-out. Many AI products now have a “deep research” or “researcher” mode: Claude Research, ChatGPT Deep Research, Gemini Deep Research, Microsoft Researcher, Perplexity Deep Research. The names differ, but the shape is similar: the harness plans, searches, reads many sources, and may run multiple workers behind one visible request. That can be useful, but it is not magic. Each worker still has a context window, model calls, tool calls, and output. I learned this the expensive way: one broad request spawned 101 subagents in parallel and turned into a $150 lesson. Put a budget around fan-out before you let the harness do it for you.
A document is not outside the prompt just because it came from a file picker, drive connector, or knowledge base. If the model is reading it in this turn, some representation of that document is in context. That includes useful text, but it can also include layout noise, repeated headers, embedded metadata, OCR mistakes, table artifacts, and page furniture.
| Need | Avoid | Prefer |
|---|---|---|
| One answer from a long PDF | Attach the whole PDF and hope the model finds the paragraph. | Extract the relevant pages or section, then pass a clean text or Markdown version. |
| A company knowledge corpus | Try to fit the corpus into the context window. | Use search or retrieval first, then place only the relevant passages in the request. |
| A reusable policy or runbook | Paste a fresh copy into every request. | Keep a stable, cleaned version that can sit early in the prefix or live behind retrieval. |
| Precise facts or citations | Rely on a summary that hides where details came from. | Retrieve small excerpts with source names, dates, page numbers, or section headings. |
Markdown often beats raw document shape. Converting a PDF, slide deck, or exported report to clean Markdown can remove visual clutter while keeping headings, lists, tables, and source references. It is not magic compression; it is context hygiene. The goal is to give the model the words and structure it needs, not every artifact of how the document happened to be printed.
Retrieval is not a workaround for thinking. It is a way to decide what deserves to enter the request. A RAG system, search index, or document lookup tool can hold far more than the context window, but the model still reasons over the pieces you actually put in front of it. Retrieval changes the selection problem; it does not make context infinite.
Measure the result. Log your provider's cached-input metric and watch how it changes when you add tools, schemas, documents, or retrieval results. A falling hit rate usually means the beginning of the request is mutating, or that too much volatile material has crept into the prefix.
Fast, thinking, extended, adaptive, effort — many vendors ship some version of this control. It matters here because thinking is output, and output is the side prompt caching cannot save. It can also change the rendered request enough to disturb cache hits.
Thinking modes
The thinking toggle sounds like it should make the model take several rounds, check itself, and stop when satisfied. The actual behavior is simpler: one request still produces one response. The model just writes more hidden output before it starts writing the answer you see.
Think of it as hidden generation. Thinking is produced token by token, using the same output machinery as the visible answer. More effort means more generated tokens inside the same turn, not another conversation round.
Effort is a request setting. The same model can run with one effort level on this request and another on the next. The model constrains which modes are available; the request picks one of them.
The shared ceiling
You already met this trap in Part 1, as a one-line warning under
max_tokens: thinking and the visible answer share one ceiling, and
thinking runs first. Here is what that looks like happening. Three effort levels,
one ceiling — raising effort moves the boundary inside the turn, it does not
add turns.
The trap, in full. max_tokens bounds thinking and visible text
together, and thinking runs first. Size the ceiling around the answer you expect
and a high effort level can consume it before the answer starts. The fix is not a
smaller effort level by reflex — it is a ceiling sized for both halves,
which on demanding agentic work means 64k and up, not the 4k that felt
generous when you were only counting prose.
The bill
Thinking tokens and answer tokens are the same line item at the same price. There is no separate, cheaper tier for reasoning. The model working out its approach and the model telling you the answer are, to the invoice, both output.
Hiding the reasoning does not change that. Products may show a summary, a collapsed trace, or nothing at all, but billing follows the full hidden generation, not the small amount of reasoning text you were allowed to see.
Log the invisible half. Put thinking_tokens next to
cache_read_input_tokens in your metrics. A workload where thinking is
most of the output is a workload where caching has less left to save, because the
expensive side of the bill is being generated fresh every time.
Where to put it
Reasoning can live inside the model, or it can live in your harness. People often call the second pattern multi-turn reasoning: the app breaks the work into several model calls, with tool results and intermediate conclusions passed forward as message history. That choice is often made for product feel, but it also changes the bill.
| Question | Inside the model | Multi-turn reasoning in the harness |
|---|---|---|
| What happens? | One request, one prefill, then hidden reasoning and visible answer are generated as output. | The app plans, calls the model, reads tools, appends what happened, and calls the model again. |
| Who configures it? | The model must support a thinking or effort mode; the request chooses the level. | Your harness owns the loop: when to call, what to save, when to stop, and what to pass forward. |
| What gets billed? | The deliberation is output. It is fresh work inside that response. | Each call has input and output. Prior steps become input on later calls, often cacheable if the prefix stays stable. |
| Latency shape | Usually one round trip, but the visible answer may start later because hidden reasoning runs first. | Several round trips, plus tool latency, but the user or UI can sometimes see progress between steps. |
| Cache behavior | Prompt caching does not save the current hidden reasoning; it is generated now. | Intermediate conclusions and tool results can become cacheable later when sent back as message history. |
| Best fit | Deep thinking inside one answer, with minimal orchestration. | Workflows that need tools, checkpoints, recovery, auditability, or visible step-by-step progress. |
Which is cheaper is arithmetic, not doctrine. Thinking pays once at the output rate. An orchestrator pays across several calls, but repeated history may move to the cache-read rate. The answer depends on turn count, token volume, tool latency, and how much of the intermediate work is worth preserving.
The cache warning. Pick an effort level at the start of a cached session and hold it. Changing thinking settings midstream can change how the request is rendered, which commonly costs you message-cache reuse. Thinking blocks you pass back on later turns also become input, so they follow the same cache rules as the rest of the message history.
Context is the substrate the rest of the ecosystem is built on. Here is what stops being mysterious once you can see it.
The payoff
This is why the foundation is worth the read. Each of these is a question people argue about, and each becomes tractable once you can reason about what is in the window and what it costs.
“Thinking mode or an orchestrator loop?” is unanswerable in the abstract and trivial once framed as paid once at the output rate versus paid repeatedly at the cache-read rate, at a given turn count. The previous section is that framing applied.
An agent is a loop that accumulates context. Subagents exist because context is finite and isolatable. Compaction and memory exist because turn counts are unbounded. Every agent framework is a different answer to the same constraint.
Every connected server is tokens in the prefix on every turn, forever. Deferred loading exists precisely because of that. “Should I connect this server?” turns out to be a context question with an arithmetic answer.
Per-request cost is dominated by the prefix, not the completion. Once you know that, cache hit rate becomes a first-class metric to instrument and alert on — and capacity planning becomes arithmetic instead of guesswork.
Should a fact live in the system prompt, in a retrieved document, in a tool result, in a memory file, or in the weights? That is one question — where does this belong, given what it costs to carry — and it now has a decision procedure rather than a vibe.
The discipline has a name. Deciding what occupies the window, in what order, at what refresh rate, for what cost, is increasingly called context engineering. It is turning into the load-bearing skill for building on models — less about clever wording than about managing a scarce, expensive, re-transmitted resource. You now have the mental model it rests on.