Saving your tokens

1. The noise, and the thing under it

Every launch is sold as a revolution. The fundamentals haven't moved.

We're in the loud part of the cycle. Kimi K3, GPT-5.6 and Fable 5 landed inside a few weeks of each other, every benchmark framed as a step change and every comment thread insisting last month's model is a relic. A lot of that volume is timing. With IPOs circling, there's real money in making every release sound like it rewrote the field, so the sensational read gets amplified and the boring read gets buried.

The boring read is the true one. Nothing about doing this well has changed since the days of plain data work. Good structure up front still produces better results at the end. Define the question, shape the inputs, decide the method before you run anything — and your output is cleaner and your bill is smaller. Skip that and hand the thinking to the model, and it will happily think, at length, on your account. I know because I did exactly that, and it cost me five million tokens.

2. Where the money actually goes

You are billed for the whole conversation, on every single call.

Here's the part people miss. A model has no memory between calls. To keep a conversation going, the entire history — system prompt, every earlier message, every tool result — is sent again with each new step. You pay input tokens on all of it, every time. So the cost of a call isn't the size of your new message. It's the size of everything said so far.

That means a long context doesn't cost a bit more, it compounds. Ten turns of real work can bill like fifty, because turn ten re-sends turns one through nine underneath it. Let an agent loop run with dozens of research bots dumping their raw findings back into the thread, and the context balloons, and every subsequent step drags that whole weight along. The diagram below is the shape of it: the new message is a sliver, the re-sent history is the bulk.

3. My five-million-token World Cup run

Above-average result, well below-average discipline.

I ran a World Cup prediction on Fable 5 and let it think for itself. No fixed method, no cap on the loop — just a goal and dozens of back-end research bots feeding findings in. It worked, in the sense that I finished 10th out of 300, comfortably above average.¹ It also burned through five million tokens, because I'd handed the model the one job I should have kept: deciding how to do the work.

Every bot returned its research as raw prose, and all of it piled into the running context. The model re-read the whole growing heap on each step to decide the next one. That's the compounding curve above, running unsupervised for hours. The result was genuinely good; the cost was indefensible for what it was.

What would have kept it well under a million

None of this needed a cheaper model or a cleverer prompt. It needed structure decided before the first call:

• Fix the method first. Write the ranking approach down — the factors, the weighting, the steps — so the model executes a plan instead of inventing one live. Most of the five million went on the model deliberating about method.
• Summarise before it enters the context. Each bot should return three lines, not three pages. Roll raw research into a short structured note — a stat line per team — and feed the model that, not the transcripts.
• Cap the loop. A hard step limit and a clear stop condition. Open-ended agents wander, and every wandering step re-bills the whole context.
• Carry a window, not the whole history. Keep a running summary plus the last few messages, and let older raw turns drop off rather than paying for them forever.
• Split the work by model. Cheap, small model for the bulk research and cleanup; the expensive frontier model only for the final judgement call. More on that split below.
• Cache the stable prefix. The system prompt and fixed instructions don't change between calls; prompt caching bills the repeated prefix at a fraction of the rate.

Same 10th place, a fraction of the tokens. The structure was the saving, not the hardware.

4. The curated context layer

Don't feed the model the history. Feed it a briefing.

Most of the fixes above are one idea wearing different hats, and it deserves its own name: a curated context layer. A deliberate stage that sits between the raw material — transcripts, tool output, research dumps — and the model's context window, so only a distilled version of the truth ever gets in. Anthropic's engineering team calls the wider discipline context engineering, and their framing is the right one: context is a finite budget you're curating on every single call, not a bin that fills up on its own.²

In practice the layer has two mechanics. Compaction: when a conversation gets long, summarise it and restart the window from the summary, carrying the decisions forward and dropping the noise. And distillation at the boundary: let sub-agents burn tens of thousands of tokens exploring, but only ever let a short structured note back through — Anthropic's multi-agent systems hand back one to two thousand tokens from searches that cost far more. Their published evaluations put a number on the whole approach: on a hundred-turn task, context management cut token consumption by roughly 84%, while letting runs finish that would otherwise have died of context exhaustion.² My World Cup run was the control group. Nobody curated anything, and the bill shows it.

The part that surprised me reading further: the curated layer isn't just cheaper, it's smarter. Chroma's "context rot" research tested eighteen frontier models and found accuracy falls as input grows — even on trivially simple retrieval, and well before the advertised window limit.³ A model reading a two-page briefing genuinely outperforms the same model reading the two hundred pages the briefing came from. A long context window is a capacity, not an instruction to fill it.

If you've built data platforms, this should feel familiar, because it's the oldest pattern we have: raw data lands, a staging layer cleans it, a curated layer serves it, and nobody points the CEO's dashboard at the landing tables. The context window is the dashboard. Stop pointing it at the raw feed.

5. Cheaper models, and running your own

Not every step needs a frontier model. Many need no API at all.

Once the structure is sound, the next lever is not paying premium rates for cheap work. Most steps in an agent — reformat this, extract those fields, summarise that page — are easily handled by a small model. Reserve the flagship for the hard reasoning at the end. And for a lot of that routine work you can drop the API bill to zero by running a model on your own machine.

This is where people tangle two different things, so it's worth being precise: the engine is the software that runs a model; the model is the downloaded intelligence it runs. They're chosen separately, and the same engine will run many different models.

Ollama and LM Studio are the two easy ways in. Ollama is a lean command-line tool with an API that mimics the big providers, so pointing existing code at it is a one-line change. LM Studio is a desktop app with a proper interface for browsing, downloading and chatting to models. Both are conveniences over the same engine: llama.cpp, the C++ project that actually runs the model on your CPU and GPU and defined the GGUF file format these tools download. You rarely touch it directly, but it's what's doing the work.

vLLM is the different animal. It's built for throughput — one GPU server answering many people's requests at once — so it's what you reach for when a whole team needs a shared local model, not a laptop setup. Different tool, different job.

The models themselves are the interchangeable part. Gemma (Google), Llama (Meta), Qwen (Alibaba) and Mistral (Mistral AI) are open-weight families you download and point any of these engines at. Trying a different one is swapping a file, not changing your setup.

The one hardware limit that actually bites

Everything hinges on VRAM.

A model has to fit in your graphics card's memory to run fast. That single number sets what's realistic. On a common 8GB consumer GPU, the honest ceiling is a 7B-to-8B model at 4-bit quantisation — "quantisation" meaning the weights are compressed to about half a byte each, which shrinks a 7B model to roughly 4GB and leaves room for your context. It's a genuine quality drop from a frontier model, but for routine steps in a pipeline it's often more than enough, and it runs for free on hardware you already own.

6. Kimi K3, and paying once instead of per token

Downloadable weights turn a metered bill into a fixed cost.

This is what makes Kimi K3 interesting beyond the benchmark noise. Alongside the hosted API, you can download the model and run it yourself, which removes the per-token charge entirely.⁴ For a workload heavy enough that the API bill hurts, that changes the maths: instead of paying for every token forever, you pay once for the hardware to serve it.

The honest caveat is that a frontier-scale model is nothing like a 7B on a laptop — it wants serious GPU capacity, which is why vLLM on a real server exists. So "eliminate API cost" is true, but you're trading a metered bill for an infrastructure one. Whether that's cheaper depends entirely on your volume. It's the same call as owning versus renting, and the answer is a spreadsheet, not a headline.

7. The part that never changed

Strip away the launch noise and the advice is dull, which is how you know it's right. Decide the method before you run the model. Keep the context lean and structured. Send small models the small jobs, and save the expensive one for the decision that needs it. Run locally when the work is routine and the volume is high. Every one of those is a front-loaded structure choice, the same discipline that separated a good data project from an expensive one long before any of this. New models, same habit: sort the structure out first, and the bill takes care of itself.

Notes & sources

1. The World Cup prediction run and its 10th-of-300 result are described in my post on catching AI getting things wrong. LinkedIn, on the Fable 5 run. ↩
2. Context engineering, compaction and sub-agent distillation — including agents returning 1,000–2,000-token summaries from far larger searches, and context management cutting token consumption by roughly 84% on a hundred-turn agent task. Anthropic, "Effective context engineering for AI agents". ↩
3. "Context rot": across 18 frontier models, output accuracy degrades as input length grows, even on simple retrieval tasks, well before the advertised context limit. Chroma, "Context Rot: How Increasing Input Tokens Impacts LLM Performance". ↩
4. Kimi K3 is offered both as a hosted API and as downloadable open weights you can self-host, which removes the per-token API charge in favour of running the model on your own hardware. Framing per the recent Kimi K3 release; check the model card for the current licence and weight availability. ↩

← All projects