← Andy's AI Field Notes

Field guide

Where have all the tokens gone?

How the context window really works — what fills it, what it costs, and how to stop burning through it

Author: Andy Harjanto, Summer 2026 · 20 minutes to read

One afternoon I hit my usage limit mid-session and had no idea why. This is what I found underneath — and the handful of habits that came out of it.

No prior knowledge assumed. The examples use Claude Code — that is where I hit the wall — but the broad pattern is the same in Codex CLI, Gemini CLI, Cursor, and ordinary chat assistants. Each provider has its own cache policy, commands, and price list; the mental model is what carries across.

Part 1

Why I went looking

This started as an interruption I didn't understand. If you have hit the same wall — or just want to know what you are paying for — the rest of the guide is the answer I put together.

Where this started

The afternoon it stopped

It was late afternoon. I was deep in a Claude Code session — reading source, pulling logs, running a few custom tools of my own, chasing a problem across traces — when it stopped and told me I was over my limit.

!
You have reached your usage limit. Ask your administrator to increase your AI budget.

It didn't read like an error. It felt like the power going out. Nothing to debug, nothing to retry, no account of what I had done to earn it. Just: done for now.

That bothered me more than the interruption did. I had no mental model of what I had been spending. Tokens went somewhere, cost accumulated, and the whole mechanism stayed invisible right up until the moment it wasn't. I couldn't have told you whether I had been careless or simply unlucky.

So I went looking. What follows is what I found — not a survey of the literature, but the working model I wish I'd had that afternoon, and the handful of habits that came out of it.

Three things I wanted to know

Everything after this answers one of them. If you only care about one, go straight there.

01
Where do the tokens actually go?
Parts 2–3
02
Why does it add up so fast?
Part 4
03
What could I have done differently?
Part 5

The short version

Three facts, and most of it follows

If you read nothing else, read this. Everything later in the guide is these three sentences with the mechanism filled in.

Storage
LocalTranscripts, CLAUDE.md, and memory files live in ~/.claude/
Processing
RemoteThe model itself holds no state between calls
Transmission
Every turnThe entire window is serialized and re-sent each time

The consequence that surprises people. Because processing is remote and the model holds no state, the entire conversation has to reach the model on every turn. When you call an API you re-send it yourself; a polished product may move the bytes around more cleverly behind the scenes, but the model still reads all of it, every time, and you are billed accordingly. That single fact drives the cost curve, the latency curve, and the existence of caching.

Part 2

Where you meet it

The same pattern appears across the tools you use. What changes is how much you control and how much you can see.

Canonical scenarios

Four surfaces, one mechanism

Whichever of these you use, the request that reaches the model has the same shape. The difference is who assembles it — and therefore who can make it cheaper.

Surface Who assembles the request What you can observe What you control
Claude Desktop / claude.ai The app, entirely Nothing exposed What you type, and when to start a new chat
Claude Code (CLI or desktop) The harness — tools, files, skills, memory /context shows the live breakdown Which files and tools enter; when to clear or delegate
Direct API calls You — you build the array Usage counters on every response Everything, including cache breakpoints
An agent harness you build You, across many turns and subagents Whatever you instrument Everything — including the mistakes

Read the table as a gradient of responsibility. At the top, the product makes every decision for you and does it well. At the bottom, you own the prompt assembly, the cache strategy, and the failure modes. The basic sequence in Part 3 appears in all four rows — which is why understanding them helps whether you are a user or a builder.

Part 3

Under the hood

One question, followed from the moment you press enter to the moment the answer appears. This is the core of the guide.

The sequence

What actually happens between keystroke and answer

Six steps, start to finish. One happens once per session; the other five repeat on every single turn. That split is where most of the surprise lives — and it is the only genuine sequence in this guide, so it is the only thing numbered.

  1. Your configuration is read from disk Once per session

    In Claude Code today: CLAUDE.md, the memory index, your settings, every connected server, and the list of available skills are read off the filesystem and assembled into one block of standing instructions. Other tools load their own equivalents the same way.

    Why it mattersThis happens exactly once. Edit any of those files while a session is running and the session will not notice — it is working from the copy it took at startup.

  2. Your message joins the conversation Every turn

    The harness holds the whole conversation in memory. Your new message, plus any results from tools it ran on your behalf, get appended to the end of it.

    Why it mattersNothing is removed as you go — the conversation only grows, until it gets full enough that the system starts summarizing the older parts away. Tool results (file contents, command output, fetched pages) are usually far larger than anything you typed.

  3. The whole thing is packed up Every turn

    All of it — the standing instructions, every tool definition, every previous message, every file that was read along the way — is serialized into a single payload.

    Why it matters“All of it” is literal. Today's APIs have no way to send only what changed since last time — not a limit of the technology, just how the interfaces work.

  4. The payload is sent Every turn

    It crosses the network to the API and the connection stays open for the reply.

    Why it mattersThis is the step people assume gets optimized away. It does not. The full payload goes over the wire on every single turn, forever.

  5. The server checks whether it has seen this opening before Every turn

    It compares the beginning of your payload against what it processed recently. Seen it before, unchanged? Reuse the earlier work. Never seen it, or something changed? Do the work now, and save it for next time.

    Why it mattersThis single step is the entire difference between a cheap turn and an expensive one. Part 4 explains how it works.

  6. The answer is generated and streamed back Every turn

    Only the new part of the prompt needs working through; the answer itself is always written fresh. The response streams to your screen while it is still being written, and the harness appends the finished turn to a log file on your disk.

    Why it mattersThe live conversation is the copy held in memory — the log is written but never read during the session. It earns its keep afterwards: that file is what lets you resume this conversation tomorrow, search it months from now, or recover it if the process crashes. Think of it as a flight recorder, not a working file.

The distinction that trips people up

Read once. Send always. Process once.

Steps 1, 4 and 5 above hide a distinction worth pulling out on its own — collapsing these three into one idea causes more confusion than anything else in this guide. Here they are traced across a twenty-turn session, for a single file: CLAUDE.md.

1×

Read from disk

Opened at session start. Never re-opened — which is why editing it mid-session changes nothing.

20×

Sent over the network

Its text rides in every single request payload, along with everything else in the window.

1×

Processed by the model

Computed on the first turn. The remaining nineteen are served from cache.

Staleness follows directly. Because files are read once and held in memory, the model reasons about the snapshot it captured. Anything that changes on disk mid-session must be explicitly re-read.

Example

Walkthrough: two turns, six lanes, one cache

Now watch those six steps actually run. This is one real exchange — a question, an answer, a follow-up — with every participant given its own lane. Time runs left to right. The dashed band is the wire: everything above it is your machine, everything below it is Anthropic's. You do not need to memorize every lane; just watch where the full payload travels, and where caching changes the work.

cache miss — full price cache hit — roughly a tenth ↓ away from you  ·  ↑ back toward you blank cell = that actor is idle

Look at column 05. “Fix it and run the suite.” is six words that mean nothing on their own — no file, no test name, no diagnosis. It only works because all of turn 1 travels down the wire again alongside it. That is what the payload is buying: the model is stateless, so context is not remembered, it is re-supplied.

Read the wire lane next. It fires in column 02 and again in column 05, carrying the same earlier bytes again. That is the fact people find most surprising: caching does nothing for the network. You send the whole window on every single turn, forever.

Columns 03 and 06 mention a cache hit and a miss — Part 4 explains those properly. For now: the first time, everything is computed; the second time, most of it is reused.

The divergence is entirely below the wire, in columns 03 and 06. Identical input arrives, and is either prefilled from scratch or loaded from tensors computed twenty seconds earlier. That is why every optimization in this guide is about keeping the prefix byte-identical — you are not trying to send less, you are trying to send the same thing so the expensive half can be skipped.

My context window

Before and after: 3% full, then 98%

Here is the part I could not see on the afternoon it stopped. The window does not fill up because of what I typed. It fills up because of what the assistant read on my behalf.

At session startbefore I typed anything
33.4k3% full
Four hours inafter real work
978k98% full
  • Standing overhead33.4k
  • The conversation~45k
  • Tool results~900k

The overhead never moved. It is the same 33.4k in both bars. What I typed all afternoon came to about 45k. Everything else — more than nine-tenths of the window — was the source, the logs, the traces, and the output of every tool I ran, pulled in on my behalf one call at a time.

The 3% bar is measured. The 98% bar is illustrative — a realistic afternoon rather than a captured trace — but the proportions are the point, and they are not exaggerated.

What the 3% is made of

  • System tools14.3k
  • MCP tools8.5k
  • Skills5.7k
  • System prompt4.9k
  • Memory files90

Loaded before you type a word, and roughly constant for the whole session. Worth knowing, but it is not what runs you out of room.

The number that misleads. Your session may also list ~328k tokens of deferred tool schemas. That is not charged — only tool names are held, and the full schema loads on demand. It is why a session advertising 360k of tooling reports 33.4k of actual use.

What a nearly-full window feels like

Nothing announces this. There is no warning at 50%, no yellow light at 80%. The symptoms arrive gradually and are easy to blame on something else.

Every turn costs more than the last

You are now paying for ~978k input tokens on every exchange, including the ones where you typed four words. Cost per turn is set by how full the window is, not by how much you asked for.

Replies take longer to start

There is more to send and more to read before the first word comes back. Even when the reuse machinery in Part 4 is working perfectly, a near-full window is slower off the mark than an empty one.

Older material starts being summarized away

To make room, the system condenses earlier parts of the conversation. This keeps you working, but you do not choose what gets compressed — and details you cared about can quietly lose their precision.

Answers can get vaguer

The five thousand tokens that actually matter are now competing with nine hundred thousand that mostly do not. More context is not automatically better context.

And then it stops

Which is where this guide started. Not a crash, not a bug — a budget consumed mostly by material I never chose to load and never needed to keep.

The landmine. Most repositories hold at least one file that would eat the window in a single read. A 1.6 MB bundled doc is roughly 400,000 tokens — 40% of the window, gone in one tool call, permanently. Lockfiles, bundles, coverage output, vendored docs. Part 5 is largely about not doing this.

Part 4

Why caching exists, and how it works

Start with the bill, because that is the problem caching was built to solve. Then the idea itself, in plain terms — the machinery is in an appendix if you want it.

The economics

What the next twenty turns cost at 98% full

Picking up exactly where the gauge above left off. The window is already nearly full; now you keep working. Every turn from here re-sends about 978k tokens — whether you typed four words or four hundred — so the cost per turn is set by how full the window is, not by what you asked for. Input cost at $5 per million tokens as of 25 July 2026, comparing a cached session against an uncached one. Same conversation, same result.

Without caching $100.00
With caching $19.50
Multipliers against the base input price. “5-minute” and “1-hour” are how long a cache entry survives between turns — not how long your session lasts. The figures above assume the 1-hour setting.
OperationMultiplierPer 1M tokens
Uncached input1×$5.00
Cache write · 5-min TTL1.25×$6.25
Cache write · 1-hour TTL2×$10.00
Cache read~0.1×$0.50

Where $19.50 comes from: one cache write at $10.00, then nineteen reads at $0.50 each. Only the stable prefix is discounted — anything appended after the last breakpoint is billed at full rate, every turn.

The idea

How the saving actually happens

You do not need the machinery to make good decisions here. The idea fits in two rules, and those two rules explain most of the advice in the rest of this guide.

Before it can answer, the model has to read your entire request. That reading is the expensive part — and by the second turn, most of what it is reading is word-for-word identical to the first.

So the service keeps its work. When a request arrives that begins with exactly the same text as one it handled recently, it reuses what it already worked out and only reads through the genuinely new part at the end. That reuse is what costs a tenth instead of full price.

Two rules follow, and they are the whole story

It only ever works from the beginning

The reuse covers a run of text starting at the very top of your request. It cannot pick up a passage from the middle. Change something near the start and everything after it has to be redone.

“Exactly the same” means exactly

One different character is enough — a timestamp, a reordered list, one tool added or removed. As far as the reuse is concerned that is a different request, and the work starts over.

And what is actually being saved

Not the text. What gets kept is the model's processed form of it — the intermediate result of having read those particular words in that particular order. Both rules fall out of that one fact. Each word's processed form depends on every word before it, so the reuse has to begin at the beginning; and a single character changed early makes every processed form after it different, so the match has to be exact rather than approximate. It is not a design choice someone made. It is the only scheme that could work.

What it means in practice. A long, stable conversation is cheap to continue and expensive to disturb. Keeping the early part of a session untouched is worth more than keeping it short — which is exactly backwards from most people's instinct, and is why Part 5 is about what counts as “disturbing” it.

Five things people mix up

Caching is one of several mechanisms that get loosely called “memory,” and they get conflated constantly. They solve different problems and live in different places.

Mechanism What it is Lives Solves
Conversation history The turns so far, re-sent every time In the request Continuity within one session
Prompt caching Reuse of the work already done on an unchanged opening Briefly on the provider's servers Cost and speed — nothing else
Memory files Facts you deliberately wrote down On your disk, loaded at startup Continuity across sessions
Retrieval (RAG) Fetching relevant material and putting it in the request A search index or database Data far larger than the window
Embeddings A way of finding things by meaning rather than keyword A vector index One way of doing the fetching above

The one that matters most here. Prompt caching never changes what the model can see or remember — it only changes what the provider has to recompute. If caching worked perfectly and you understood nothing else in this guide, the model would behave identically; you would just pay less.

What it actually knows comes from the other four: the conversation you have had so far, the notes you deliberately saved, and whatever was fetched and placed in front of it. Caching changes the bill. The others change the answer.

Coming next

That is as far as a user needs to go. If you build on this — writing an agent, or calling the API directly — there is a companion guide picking up exactly here: how the saved work is stored and looked up, how several save-points nest inside one another, the minimum size below which nothing is saved at all, precisely which changes throw it away and which are free, how to lay out a prompt so it caches well, and when retrieval beats a bigger window.

Prompt caching, from the inside — 30 minutes, written for builders.

Part 5

What to actually do differently

Everything above was explanation. This is the part you can use tomorrow — thirteen habits that follow directly from how the window works, in rough order of how much they save you.

Day to day

Thirteen habits, and why each one works

None of these require you to think about tokens. They are just what falls out of the fact that every turn re-sends everything, and that reuse only survives if the opening stays untouched. Two of them name Claude Code commands (/clear, /context) — other tools have their own equivalents, and the habit is the same either way.

Start and end sessions on purpose

The single biggest lever, and the one most people never touch.

New topic? Start a new session.
Continuing an old one drags the whole previous conversation into every turn of the new one — you pay for it repeatedly, and it is not helping.
Finished a coding thread? /clear it.
Once a bug is fixed and verified, everything you read to get there is dead weight. Clearing resets to a clean window. But clearing is destructive — if you might come back, open a separate session instead and switch between them.
One session per project.
Context from one repository almost never helps in another, and it rides along on every request regardless.

Be picky about what goes in

This is where the window actually goes. Nine-tenths of it, in the example earlier.

Don't paste a whole document when you need one section.
A 100-page PDF dropped in to answer one question sits in the window for the rest of the session and is re-sent on every subsequent turn. Paste the section, or say which part matters.
Never let it read generated files.
Lockfiles, build output, coverage reports, minified bundles, vendored documentation. One of these can consume 40% of the window in a single read, permanently, and none of it is information you wanted.
Ask it to search first, read second.
“Find where the timeout is configured” costs a fraction of “read these twelve files.” Let it locate the thing, then read only what it found.
For big files, ask for the part you need.
A specific function, a range of lines, the section under a particular heading. Whole-file reads are the default and are usually wrong.

Push the heavy reading elsewhere

The trick most people do not know exists.

For broad exploration, ask it to use a subagent.
A subagent reads in its own separate window and reports back a summary. It can work through thirty files and your session grows by a paragraph. Phrasing as simple as “search the codebase and summarize what you find” usually triggers it.

Notice the warning signs

Nothing alerts you. These are the signals.

Replies getting slower is the tell.
It is rarely the network. A filling window is slower to send and slower to read before the first word comes back.
Run /context when you wonder.
It shows exactly how full you are and what is taking the room. Cheap to check, and the only way to see the thing that is otherwise invisible.
Vaguer answers can mean a crowded window.
If responses start hedging or losing detail on a long session, the material that matters is competing with a great deal that does not. Starting fresh often beats pushing harder.

Protect what you would lose

The failure that costs the most is not cost.

Write conclusions down before you clear, walk away, or let it compact.
Files survive; the reasoning does not. Everything worked out in a session lives nowhere else unless you save it to a file, a doc, or a commit message — and compaction can blur the details while the session is still running.
Say when a file has changed underneath it.
It is holding the copy it read earlier, not the file on disk. If you edited something since, say so and ask it to re-read — otherwise it is confidently reasoning about the old version.

If you only take one thing. A long, stable session is cheap to continue and expensive to disturb, and what fills it is almost never what you typed — it is what got read on your behalf. Be deliberate about the second, and start fresh more often than feels necessary.

Where have all the tokens gone?
Long time prefilling.
Where have all the KVs gone?
Long time caching.
The answer, my friend,
is hiding in attention.
The answer is hiding in attention.