The Token Economy

Mehran Mozaffari·Updated 1 September 2026

9 resources0 related posts

Tokens Became an Engineering Constraint When Caching Split the Price of Reading

The token economy is the discipline that treats every LLM call as a priced resource with structure: not "how much does GPT cost" but "what does this category of token cost under this condition," and what my architecture can do to move spend from expensive conditions to cheap ones. A few years ago an API price sheet had two numbers per model — input, output — and cost management meant picking a smaller model. Today the two-number price sheet is dead. The leading providers now bill the same input token at as many as five different rates depending on whether it hits a cache, when it arrives, whether it was written to a cache in the first place, which service tier served it, and whether it was submitted through a batch endpoint. DeepSeek's current sheet is the extreme case: the same million input tokens cost $0.007 on a cache hit off-peak and $0.44 on a cache miss at peak — a 63× spread inside one model's price table, before you even consider a different vendor.

That spread is what makes this a field rather than a shopping decision. It created a new engineering role — call it cost-aware inference architecture — with its own tooling layer (gateways and observability platforms), its own optimization moves (prefix caching, batch routing, tier shaping, prompt compression, quantized self-hosting), and its own failure modes. What gets miscategorized into this field matters. Subscription plans (per-seat coding assistants, monthly chat tiers) are a pricing shape adjacent to this economy, but they hide tokens behind a flat fee and give you no lever to pull, so they are relevant here only as the thing teams graduate from. GPU-hour purchasing for self-hosting is the opposite edge: it converts per-token cost into capital cost and utilization math, which this dossier touches (through vLLM) but does not center. The center is the metered API, because that is where most production spend lives and where the levers are sharpest.

The Price Sheet Is Now a Rate Structure, Not a Number

Read three current price pages and the pattern is unmistakable. Every major provider has converged on a matrix of modifiers stacked onto a base rate, and the modifiers are individually larger than most base-rate differences people agonize over.

Anthropic's prompt-caching page documents the structure explicitly: cache writes cost 1.25× base input for the 5-minute TTL and 2× for the 1-hour TTL; cache reads cost 0.1× base input; and these multipliers stack with the Batch API discount and data-residency pricing. On Claude Sonnet 5 at $2 per million input tokens, a fully-cached read costs $0.20, and a 1-hour cache write costs $4. Claude Opus 5 runs $5 in / $25 out; Claude Fable 5 sits at $10 / $50; Haiku 4.5 at $1 / $5. The opus-to-haiku spread is 5× on input and output; the cache spread on a single model is 10×. Choosing the right model is worth 5×; caching the right prefix is worth 10×, and they compose.

Gemini's pricing page carries the same structure with two additions worth stealing. First, service tiers with names: Standard, Batch (50% off), Flex (also half price, for latency-tolerant work), and Priority (1.8×, for guaranteed capacity). Second, scheduled pricing: Gemini 3.7 Flash is $0.75 in / $3.75 out per million "through December 31, 2026" and doubles on January 1, 2027 — intro pricing with a published expiry, which turns a cost model into a dated contract clause. Gemini also meters context caching differently from everyone else: a discounted read price ($0.075/MTok) plus a storage price ($0.50 per million tokens per hour through 2026), which makes cache retention an explicitly billed resource rather than a free side effect.

DeepSeek contributes the time dimension: peak hours (01:00–04:00 and 06:00–10:00 UTC, Monday through Friday) bill at 2× the off-peak rates across the board, and its cache-hit discount — roughly 31× at the V4 Flash tier — is the largest published. The model context (1M tokens, 384K max output) and concurrency limits (2,500 for Flash, 500 for Pro) complete the picture: the price sheet now encodes cache behavior, time-of-day, service tier, batchability, and capacity. The comparison table below puts the current shapes side by side.

	Anthropic (Sonnet 5)	Google (Gemini 3.7 Flash)	DeepSeek (V4 Flash)	OpenRouter (aggregator)	Groq (neocloud)
Base input, $/MTok	$2.00	$0.75 (intro, doubles 2027)	$0.22 miss / $0.007 hit (off-peak)	Pass-through, ~500+ models	Model-specific, premium for speed
Base output, $/MTok	$10.00	$3.75 (intro)	$0.66 off-peak / $1.32 peak	Pass-through	Model-specific
Cache discount	Reads at 0.1× base	~0.1× read + $0.50/MTok/hr storage	~31× off-peak (hit vs miss)	Provider-dependent	Provider-dependent
Cache write premium	1.25× (5m TTL), 2× (1h TTL)	None stated (storage billed)	None stated (implicit caching)	—	—
Batch discount	Yes (stacks with caching)	50%	Not published	Provider-dependent	—
Time-of-day pricing	No	No	2× peak vs off-peak	—	—
Service tiers	—	Standard / Batch / Flex / Priority	Peak / off-peak	Varies by provider	Speed-tiered
Terms quirk	TTL clock starts at request start	Free-tier data trains models; paid does not	Granted balance spent first	One API, many vendors	LPU speed positioning

My position after reading all of these: most teams audit base rates when the actual cost lever is the modifier matrix, because a 10–30× caching spread dwarfs the 2–5× model-choice spread that gets all the attention.

Prompt Caching Is the Biggest Discount Most Teams Never Claim

The mechanics are now standardized enough to describe once. A request's prompt — tools, then system, then messages, in that order — forms a prefix. If a recently served request shares that prefix, the provider reuses the computed state and bills the shared tokens at the cache-read rate instead of the base rate. Anthropic's implementation is the most fully specified: caching is explicit or automatic (a single top-level cache_control moves the breakpoint forward as the conversation grows), the default TTL is 5 minutes refreshed free on every hit, a 1-hour TTL costs double on writes, and — the detail that bites people — the TTL clock starts when the request starts, not when the response ends, so a 4-minute streaming response leaves roughly 1 minute of cache life for the follow-up. DeepSeek's implementation is implicit and more aggressive: every request's prefix is cached server-side, hits are detected automatically, and the discount lands without any client code — but you do not control what gets cached. Gemini's version adds the storage meter, making long-lived caches an explicit rental.

The engineering consequence is a reordering of prompt architecture. Anything stable — system instructions, tool schemas, retrieved documents, the growing conversation history — belongs early in the prompt so it falls inside the cached prefix; anything per-request belongs at the end. Teams that interleave dated content into their system prompt (a "today is..." line, a rotating knowledge snapshot) break their own cache on every request and pay full price while believing they have caching enabled. My position: prefix-ordering plus explicit cache breakpoints is the highest-leverage, lowest-risk optimization in the entire field — a one-day change that routinely halves multi-turn application spend — and the TTL-clock subtlety is the reason naive implementations quietly miss most of the discount.

sequenceDiagram
    participant A as Application
    participant P as Provider
    A->>P: request 1 (tools, system, 20 documents)
    P->>P: full prefill at base input price
    P->>P: write prefix to cache (billed 1.25x on Anthropic)
    P-->>A: response
    A->>P: request 2 within 5 minutes (same prefix, new question)
    P->>P: prefix hit, billed at 0.1x base
    P-->>A: response (cheaper and faster)
    Note over A,P: TTL is measured from request start<br/>a 4 minute stream leaves about 1 minute of cache life
    A->>P: request 3 after TTL expiry
    P->>P: full prefill again at base price

The Aggregator Layer Turned Prices Into a Commodity and Routing Into a Discipline

Between applications and providers sits a routing layer that did not exist in the two-number era. OpenRouter exposes more than 500 models through one API, which changes the cost conversation from "which vendor" to "which model at which price at this moment," including open-weight models served by competing inference providers at different rates. LiteLLM (57.7k stars, MIT) is the self-hosted version of the same idea: a gateway with a Rust core and Python SDK that speaks 100+ provider APIs in OpenAI or native format and layers on exactly the functions this economy needs — cost tracking per key and per team, budgets and spend ceilings, fallbacks across providers, and load balancing across deployments. Its rise tells you something about where the field's center of gravity is: when token prices became a rate structure with a dozen knobs, teams needed a place to enforce policy (this key may only call cheap models; this route falls back from premium to standard at midnight; this team's monthly budget hard-stops at $X), and the gateway is where policy lives.

My position: the gateway layer is the one piece of token-economy infrastructure I would install before optimizing anything else, because it converts invisible spend into queryable data and makes every downstream optimization enforceable rather than aspirational. The counter-position worth stating: a gateway is also a new failure point in your latency path, and LiteLLM's Rust-core rewrite is a direct response to teams discovering that a Python gateway in front of a fast model adds real milliseconds at scale. Route through one, but measure the tax.

Serving Efficiency Is Where Your Own Costs Are Made

The self-hosted side of the economy runs on vLLM (90.7k stars), the high-throughput, memory-efficient inference and serving engine that much of the open-model provider ecosystem builds on. vLLM is where per-token cost is manufactured: its memory management (paged KV cache), continuous batching, and quantization support determine how many tokens a given GPU-hour yields, which is the number that converts a $2.50/H100-hour rental into a cents-per-million-tokens figure. The open-weight model releases (DeepSeek's among them, MIT-licensed weights) plus vLLM means the theoretical floor on token prices is now public knowledge — which is precisely why the API providers' cache discounts keep deepening: the floor is visible.

On the demand side, LLMLingua (Microsoft, 6.6k stars, MIT) attacks the problem from the other end: prompt compression that claims up to 20× reduction with minimal performance loss, operating on both prompts and KV cache. The cost logic is clean — a 5× smaller prompt is 5× cheaper at uncached input rates and proportionally faster — but the engineering caution is equally real: compression is lossy in ways that correlate with the exact content (rare entities, long-range references) that production tasks depend on, so it needs task-level evaluation, not benchmark-level trust. My position: compression is a rounding-error optimizer compared to caching and routing for most workloads, and I would deploy it only for genuinely bulky, repetitive retrieval payloads where its hit rate is measured per pipeline, not per paper.

Quantization belongs in the same bucket: serving quantized open weights (or using providers' quantized tiers) trades a few points of quality for a large discount at self-host scale. It is a lever with a measurable quality cost, which makes it an evaluation problem first and a finance problem second — the opposite of caching, which is free money.

Observability: You Cannot Optimize What You Do Not Meter

Langfuse (34k stars, MIT core with an ee/ enterprise directory) is the open-source standard for the metering half: traces with token counts and costs per request, per feature, per user; evaluations; prompt management; OpenTelemetry-compatible ingestion that also plays with LiteLLM and the OpenAI SDK. The reason this category exists at all is that provider consoles answer "what did I spend" but not "what did this feature spend" or "which user cost us $400 this week" — and per-feature cost attribution is the prerequisite for every optimization above. My position: wire the cost ledger into tracing before you touch a single prompt; every cache-rate saving is unverifiable without it, and in my experience the ledger immediately surfaces the one rogue loop or unbounded context window that accounts for a third of the bill.

There is also a hidden cost class that observability makes visible: the terms-of-service shape. Gemini's free tier explicitly uses your content to improve Google's products, while the paid tier does not — which means the true price of "free" is your data, and any workload touching sensitive content must be on the paid tier regardless of volume. Anthropic's 1-hour cache premium and data-residency multipliers are the same phenomenon from the other side: compliance and retention are priced, and the invoice for them shows up in the token rate.

Choosing by Constraint: The Decision Tree That Actually Matters

Constraint one: is the workload multi-turn or prefix-heavy? If yes, provider choice is dominated by cache economics — Anthropic's explicit TTL control, DeepSeek's 31× implicit hits, Gemini's storage-priced long-lived caches — and prompt architecture (stable content early, breakpoints set) matters more than the model choice. If the workload is single-shot and stateless, caching is irrelevant and the levers collapse to base rate, batch routing, and model choice.

Constraint two: is the workload latency-tolerant? Everything deferrable belongs on batch: Gemini's batch tier is literally half price, Anthropic's batch discount stacks with caching, and off-peak scheduling on DeepSeek halves rates again inside specific UTC windows. A nightly summarization job that runs at Standard rates is leaving 50–75% on the table, and the fix is a cron expression.

Constraint three: do you need to enforce policy across many consumers? Then the self-hosted gateway (LiteLLM) or the aggregator (OpenRouter) is the architecture, chosen by whether you need budgets and guardrails inside your perimeter (gateway) or maximum model breadth with minimum vendor lock (aggregator). Small teams on a single provider with two engineers can skip this layer; the moment there are three services and two teams calling models, they cannot.

Constraint four: how sensitive is the data, and how far can the open-weight path go? Data-sensitive workloads push toward paid tiers with no-training terms, residency options, or self-hosting on vLLM — where licence terms on the weights (MIT for DeepSeek's, permissive across the major open releases) make self-hosting legally frictionless even when operationally heavy. The self-host decision is utilization math: predictable high volume favors it, spiky low volume never does.

The choice compresses into one decision flow, evaluated in order — each gate can end the analysis, and the order matters because caching changes the denominator that every later decision divides:

flowchart TD
    START[One request to price] --> MULTI{Multi-turn or<br/>prefix-heavy?}
    MULTI -->|yes| CACHE[Reorder prompt for prefix caching<br/>set cache breakpoints explicitly]
    CACHE --> BASE[Effective input rate is now the cache-read rate]
    MULTI -->|no| BASE2[Effective input rate is the base rate]
    BASE --> DEFER{Latency tolerant?}
    BASE2 --> DEFER
    DEFER -->|yes| BATCH[Route to batch or off-peak windows<br/>typically 50 percent off]
    DEFER -->|no| TIER[Standard or priority tier<br/>priority costs about 1.8x]
    BATCH --> POLICY{Many teams or services?}
    TIER --> POLICY
    POLICY -->|yes| GW[Gateway with budgets, guardrails, fallbacks]
    POLICY -->|no| DIRECT[Direct provider API is fine]
    GW --> SENS{Data sensitive?}
    DIRECT --> SENS
    SENS -->|yes| PAID[Paid tier with no-training terms,<br/>residency options, or self-host on vLLM]
    SENS -->|no| DONE[Ship, then watch the cache-hit-rate metric]
    PAID --> DONE

Where It Breaks

Cache misses you pay for without noticing. The prefix cache is exact: one changed byte in the system prompt, a timestamp, a rotated API key echoed into instructions, or reordered few-shot examples invalidates the prefix and re-bills the full input at base rate. Trigger: any dynamic content injected before the stable content. The failure is invisible in aggregate spend for weeks because each miss is only 2–10× a hit, multiplied by millions of requests. The observability layer's cache-hit-rate metric is the detector; without it, this is the dominant silent burn.

TTL expiry masquerading as flaky costs. Anthropic's 5-minute TTL measured from request start means a slow streaming response can consume its own cache window; a follow-up arriving 90 seconds after a 4-minute stream pays full price again. Trigger: long generations followed by quick user replies. The symptom looks like random cost spikes; the cause is arithmetic.

Thinking tokens billed as output. Reasoning models bill their hidden deliberation as output tokens, so a model switch that looks like a price cut on the input column can triple the output column. Trigger: migrating a workload to a thinking-mode default (DeepSeek's V4 defaults thinking on; Gemini bills thinking tokens inside output price). The fix is measuring blended cost per task, not per request, which only the tracing layer can see.

Batch-routing correctness bugs. Half-price batch tiers require jobs that tolerate hours of delay and idempotent re-submission; the classic failure is routing latency-sensitive traffic into a batch queue during an incident, or double-billing by re-submitting jobs that were actually processing. Trigger: incident-driven config changes made without a rollback plan.

Budget ceilings that break production. LiteLLM-style hard spend limits do exactly what they say: when a team hits the ceiling, API calls start failing, and the failure surfaces as an outage in a feature nobody remembered was LLM-backed. Trigger: a retry loop in one service consuming the shared budget. Ceilings need alerting tiers before the hard stop, and per-service budgets, not one global one.

Aggregator drift. Routing through 500+ models means a provider can deprecate a model, change a rate, or shift quality under a version name you pinned loosely. Trigger: unpinned model aliases in production configs. The price of breadth is that your cost model and your quality baseline both need re-validation whenever the underlying route changes.

Open Questions

The first genuine unknown is where cache pricing converges. Three incompatible models exist today — Anthropic's write-premium/read-discount with explicit TTLs, DeepSeek's free implicit caching, Gemini's storage-metered rental — and they imply different application architectures (explicit prefix management vs none vs cache-rental optimization). Providers are watching each other; whichever model wins will quietly become a standard that shapes how everyone writes prompts.

The second is whether scheduled pricing (Gemini's doubling on January 1, 2027; DeepSeek's peak windows) becomes normal, turning token procurement into something resembling commodity hedging — with forward commitments, reserved-throughput contracts (Gemini Enterprise's provisioned throughput), and off-peak scheduling as standard financial practice. The tooling for this barely exists; the pricing pages already assume it.

The third is the floor question: as open-weight models plus vLLM make marginal token cost public and small, what sustains closed-model premiums? The visible answer so far is quality-at-the-frontier, speed (Groq's LPU positioning), and compliance — but caching discounts keep compressing the effective price of the closed models too, and the endgame shape of this economy — whether tokens become a near-free commodity with premiums only at the frontier — is the single biggest unresolved variable in every team's long-range cost plan.