Living dossier

LLMs & Reasoning

Back to topics•Blog
Mehran Mozaffari·
11 resources0 related posts

Reasoning Became a Billing Line, Not a Capability Checkbox

The defining change in this field is that "reasoning" stopped being a model category and became a metered dimension of every model. The mechanism is test-time compute: before producing the visible answer, the model generates internal chain-of-thought tokens — planning, trying approaches, recovering from ambiguity — and those tokens are real compute with real prices. OpenAI's reasoning documentation states the billing consequence without euphemism: reasoning tokens "are not visible via the API, [but] they still occupy space in the model's context window and are billed as output tokens." Gemini's pricing table bills "output price (including thinking tokens)" in the same line. DeepSeek's V4 models ship with thinking enabled by default at the high effort level, which means the default invocation of a cheap model quietly produces paid deliberation. What used to be a product decision ("should I use the reasoning model?") is now a per-request economic decision ("how much thinking does this call deserve?"), and none of the major APIs answer it for you.

This reframing matters because the measurement literature says the trade is real but bounded. METR's time-horizon research — the most cited measurement program in the field — tracks the length of human-time tasks that models complete at 50% and 80% reliability, and its data shows models succeed at nearly 100% of tasks under four minutes while failing more than 90% of tasks over four hours. Reasoning effort widens the distribution; it does not repeal it. ARC Prize's leaderboard draws the same conclusion from a different direction: its "reasoning systems" trend lines — the same model evaluated at successive reasoning levels — display "asymptotic behavior as thinking time increases," which is the polite way of saying that past a point, more thinking buys almost nothing. The engineering discipline this dossier covers is spending deliberation where its curve is still rising and refusing it where the curve has flattened, because the flattening is billed at output-token rates.

What gets miscategorized into this field: chain-of-thought prompting of non-reasoning models (a useful technique, but a different mechanism from trained reasoning modes), and benchmark reasoning scores quoted without their cost axis — a model that scores 2 points higher at 8× the token spend is usually a worse product, and ARC's cost-per-task axis exists precisely to expose that.

Every Provider Invented Its Own Knob, and the Knobs Do Not Agree

The control surfaces diverge so much that porting a reasoning configuration between providers is a design task, not a find-and-replace. OpenAI offers the widest ladder — none, minimal, low, medium, high, xhigh, max — with a documented use-case table (voice and classification at none; tool use and planning at low; agentic coding at medium; long-horizon research at high; deep research at xhigh and max), a separate reasoning.mode switch between standard and pro execution on GPT-5.6 models, and the explicit warning that defaults are model-dependent (gpt-5.5 defaults to medium). Anthropic moved the field in the opposite direction: manual budget_tokens control — minimum 1,024 tokens, must be less than max_tokens — is now deprecated on Claude 4.6 and rejected outright on 4.7 and later, replaced by adaptive thinking where the model decides; effort parameters shape the response instead. Gemini exposes thinking levels with cryptographically signed thought blocks. DeepSeek exposes a toggle plus low/high/max — and then documents, in a table, that requesting medium or high or xhigh all map to the same actual effort of high. Qwen3, open-weight under Apache-2.0, made thinking a property of the chat template (enable_thinking=True) with runtime soft switches (/think, /no_think in the user message) and a smooth thinking-budget curve.

OpenAI (GPT-5.5/5.6) Anthropic (Claude 4.6+) Google (Gemini 3.x) DeepSeek (V4) Qwen3 (open weights)
Control surface reasoning.effort: none→max; reasoning.mode: standard/pro Adaptive thinking (model-decided); legacy budget_tokens deprecated/rejected Thinking levels + thinking_summaries config Toggle + reasoning_effort low/high/max enable_thinking template flag; /think /no_think per turn
Default state medium (model-dependent) Adaptive Model-dependent Thinking ON, effort high Thinking ON
Chain-of-thought visibility Hidden; reasoning_tokens in usage only Summarized thinking blocks; internal state encrypted signature always present; summary optional, may be empty Full CoT in reasoning_content Raw CoT between <think> tags
Multi-turn CoT handling Pre-GPT-5.6: dropped; GPT-5.6: rendered by default, reasoning.context to choose Encrypted blocks passed back verbatim Signed blocks round-trip Dropped without tools; required pass-back with tools (400 otherwise) Model follows most recent /think instruction; CoT in context
Effort honesty Full ladder, per-model defaults documented Deprecated manual budgets in favor of adaptivity Levels per model medium→high mapping published Budget scaling curve published
Weights Closed Closed Closed Open weights, MIT Apache-2.0

My positions, model by model. OpenAI's ladder is the most operationally mature — the effort table is a routing policy in documentation form — but the GPT-5.6 pro mode's billing ("aggregates the model work... and bills those tokens at the selected model's standard token rates") makes it a per-task premium that needs its own eval gate. Anthropic's adaptive pivot is the most intellectually honest: after two generations of exposing budget_tokens, they concluded users were bad at picking budgets and made the model pick — correct for most adopters, a loss for the latency-sensitive minority who genuinely needed the fixed ceiling (manual mode "remains useful when your workload requires predictable latency"). DeepSeek's published effort-mapping table — where medium, high, and xhigh all silently mean high — is either transparency or an admission that the knob has fewer stops than the API suggests; either way it is unique in the field and should be read before writing effort-routing logic against their API. Qwen3 is the field's reference for open reasoning: the four-stage post-training pipeline (long-CoT cold start, reasoning RL, thinking-mode fusion, general RL) is published, the hybrid modes are switchable mid-conversation, and the 30B-A3B MoE runs on a single high-end GPU. Gemini's signed thought blocks — an encrypted signature on every thought step, present "even when the model performs minimal reasoning" — are the strongest statement of any provider that raw chain-of-thought is now a controlled artifact, not an output.

Thinking Tokens Are Output Tokens: The Cost Model of Deliberation

The billing stack-up deserves its own arithmetic, because reasoning inverts the usual input-heavy cost profile of LLM applications. Three consequences follow from "thinking is billed as output." First, output rates are 4–5× input rates across the major providers (Claude Sonnet 5: $2 in / $10 out; Gemini 3.7 Flash: $0.75 in / $3.75 out at intro pricing), so a response that spends 8,000 reasoning tokens to answer from a 2,000-token prompt costs multiples of its naive estimate. Second, invisible spend is real spend: OpenAI recommends reserving at least 25,000 tokens for reasoning and outputs when starting out, and documents the failure mode where generation hits max_output_tokens before any visible output exists — "you could incur costs for input and reasoning tokens without receiving a visible response." Third, the modifiers stack: DeepSeek's thinking-default-on interacts with its peak/off-peak pricing (2× during peak UTC windows) and its ~31× cache-hit discount, which means the same request can vary in cost by an order of magnitude depending on hour and cache state, before effort is even considered.

The current OpenAI sheet makes the arithmetic concrete, and its structure is worth reading closely because it prices reasoning's failure modes directly. On gpt-5.5, input is $5.00 per million tokens (short context) against $30.00 output — a 6× output multiple that is exactly the multiplier your reasoning tokens pay — doubling to $10/$45 past the 272K context threshold. The GPT-5.6 family prices the effort decision as a product line instead: sol at $4/$20, terra at $2/$12, and luna at $0.20/$1.20, a 20× output spread within one model generation — which reframes "which effort level" as often the wrong question next to "which model size," since luna at max effort can undercut sol at low on cost by an order of magnitude. The pro-mode premium has a published shape too: gpt-5.5-pro bills $30/$180 against gpt-5.5's $5/$30 — the aggregated-model-work billing of pro mode materialized as exactly 6× on both columns — and the historical arc runs from o1-pro's $150/$600 through o3-pro's $20/$80 to sol's $20/$20-spread, a frontier-reasoning price deflation of roughly 8× in two years. The modifiers from the general token economy all appear here with reasoning attached: batch and flex tiers halve everything (sol drops to $2/$10), the new Fast mode doubles it ($8/$40 on sol), data-residency endpoints add a 10% uplift on post-March-2026 models, and — new on the 5.6 line — cache writes are separately priced ($5.00/MTok on sol against $0.40 cached reads, the same 12.5× write premium Anthropic pioneered), which means reasoning-heavy multi-turn workloads now pay for cache population explicitly rather than as an implicit 1.25× surcharge. Long-context pricing also doubles cache economics ($0.80 cached reads past 272K), so a reasoning agent with a growing context pays premium rates on precisely the tokens its thinking is most likely to need.

The engineering discipline this dossier covers is spending deliberation where its curve is still rising and refusing it where the curve has flattened, because the flattening is billed at output-token rates — and because the bill arrives whether or not the extra thought changed the answer, the only sustainable cost policy is measured per workload rather than assumed per model.

Chain of Thought Is Ephemeral by Design, and That Shapes Your Architecture

The providers disagree about whether you get to see the reasoning, but they agree — absolutely, and with implementation teeth — about whether you get to keep it. DeepSeek is the most explicit: in multi-turn conversation without tools, reasoning content from previous turns "will be ignored and will not be concatenated into the context." With tools, the rule hardens into an error: reasoning_content must be passed back fully on every subsequent request, "even for turns where the model did not perform a tool call," or the API returns a 400. Gemini's thought steps carry a mandatory encrypted signature whose stated purpose is representing "the model's internal reasoning state" — the summary is optional, the signature always round-trips. Anthropic's adaptive-thinking blocks pass back as opaque encrypted content. The design intent across all four is identical: the model may use its prior deliberation, but the deliberation is not yours to read, edit, or persist as plain text.

Three architectural consequences. Multi-turn chat with reasoning models is cheaper and shorter than the visible transcript suggests, because dropped CoT does not accumulate — but tool-using agents have the opposite obligation, and a middleware that strips unknown fields from assistant messages will break DeepSeek's thinking-plus-tools flow with a 400 that takes a debugging session to trace to its cause. Caching interacts strangely: Anthropic documents that extended thinking cannot combine with max_tokens: 0 cache pre-warming, and DeepSeek's ephemeral CoT means the cacheable prefix ends where the turn's reasoning began. And observability changes meaning: the reasoning you can log (Qwen3's raw <think> blocks, DeepSeek's reasoning_content) is the model's own artifact with no fidelity guarantee from the closed providers, whose summaries are explicitly permitted to be empty "depending on... whether the model performed enough reasoning."

flowchart TD
    REQ[Request arrives] --> CLASS{Task class}
    CLASS -->|voice, classification| NONE[Effort none or low<br/>latency critical]
    CLASS -->|tool use, planning| LOW[Effort low<br/>modest deliberation]
    CLASS -->|agentic coding, research| MED[Effort medium or high]
    CLASS -->|deep research, security review| HIGH[Effort xhigh or max<br/>or pro mode]
    NONE --> THINK[Model generates reasoning tokens<br/>billed as output]
    LOW --> THINK
    MED --> THINK
    HIGH --> THINK
    THINK --> TOOLS{Tool calls needed?}
    TOOLS -->|yes| PASSBACK[Pass reasoning_content back verbatim<br/>or the API returns 400]
    PASSBACK --> THINK
    TOOLS -->|no| ANSWER[Final answer]
    ANSWER --> LEDGER[Usage ledger<br/>reasoning_tokens vs output_tokens]
    LEDGER --> EVAL{Eval shows win<br/>at this effort?}
    EVAL -->|no| DOWN[Step effort down one level]
    DOWN --> CLASS

The tool-calling contract — where chain-of-thought stops being ephemeral and becomes a load-bearing protocol field — is the clearest way to see the design. DeepSeek's flow, where a missed pass-back is a hard 400, makes the hidden state machine explicit:

sequenceDiagram
    participant Dev as Developer code
    participant DS as DeepSeek V4 thinking mode
    participant T as Tool
    Dev->>DS: user question with tools attached
    DS-->>Dev: reasoning_content (thinking) plus tool_call, content empty
    Dev->>T: execute the requested tool
    T-->>Dev: tool result
    Dev->>DS: messages plus prior reasoning_content passed back verbatim
    DS-->>Dev: new reasoning_content plus second tool_call
    Dev->>T: execute again
    T-->>Dev: tool result
    Dev->>DS: full history with every reasoning_content intact
    DS-->>Dev: reasoning_content plus final content, no tool calls
    Note over Dev,DS: drop any reasoning_content from any turn<br/>and the API rejects the whole request with a 400

The Benchmarks Moved From Answers to Adaptation

Evaluation is where reasoning claims go to be priced, and the two measurement programs worth watching have both moved past static accuracy. ARC Prize evolved ARC-AGI from passive fluid-intelligence puzzles (ARC-AGI-1, ARC-AGI-2) to ARC-AGI-3, which "challenges AI agents to adapt on the fly to novel interactive environments" — the benchmark is now an agent harness, not a question set. Its leaderboard's central chart is cost-per-task against score, with three visible populations: reasoning systems drawn as connected trend lines across effort levels, base LLMs as single-shot points, and Kaggle competition systems that solved tasks under a $50 compute budget. The chart's own interpretation guide states the thesis this dossier keeps returning to: "true intelligence isn't just about solving problems, but solving them efficiently." METR's time-horizon series measures the complementary axis — task length at fixed reliability — with the 7-month doubling trend and its Time Horizon 1.1 refresh, and its SWE-bench-Verified replication doubling in under 3 months.

The practical use of both: treat reasoning-effort selection as an empirical curve per workload, not a preference. ARC's trend lines give the shape (asymptotic), METR gives the reliability framing (50% success is not production), and the only defensible routing policy is measured — run your task distribution at two adjacent effort levels, compute cost-per-completed-task, and let the ledger pick. My position on benchmarks generally: ARC-AGI-2-class scores quoted without cost-per-task, and reasoning-model gains quoted without latency, are marketing; the leaderboard's own "$10,000 to run" disclosure line is the level of accounting rigor applications should imitate.

Where It Breaks

Overthinking at the margins. Reasoning modes burn hundreds to thousands of tokens on trivial inputs — a classification request that needs one label can spend 2,000 reasoning tokens deciding how to format it. Trigger: effort left at provider defaults (DeepSeek's default is thinking-on at high) across high-volume, low-complexity traffic. The symptom is a bill that doubles after a model upgrade; the detector is the reasoning-token field in usage, charted per task class.

Incomplete responses that still bill. When reasoning exhausts max_output_tokens before the answer begins, OpenAI returns status incomplete with the input and reasoning already charged. Trigger: tight max_output_tokens caps carried over from non-reasoning models, or reasoning-hungry prompts at high effort. The 25,000-token reservation guideline exists because this failure is common enough to document.

The 400-Error CoT contract. DeepSeek's tool-calling mode requires every prior reasoning_content to be passed back verbatim, in order, forever. Trigger: any serialization layer that drops unknown message fields — most OpenAI-SDK wrappers written before reasoning modes. The failure is a 400 with a message that names the field but not the fix, and it surfaces weeks after deployment when a new turn shape first occurs.

Budget illusions on manual modes. Anthropic's legacy budget_tokens is "a target rather than a strict cap" — actual usage varies, Claude may stop well before the budget, and max_tokens remains the real ceiling. Teams that treated the budget as a cost control found latency and spend varying anyway. Trigger: legacy manual mode on models that still accept it; the durable fix is the migration to adaptive thinking and cost control at the ledger, not the prompt.

Effort knobs that do not stop where they say. DeepSeek maps medium and high requests to the same high internal effort. Trigger: routing logic written against the documented ladder that assumes distinct stops. Any effort-based pricing or latency SLO built on that API's mid-ladder is built on a fiction the vendor documents but does not fix.

Fluent reasoning, wrong answer. Chain-of-thought is generated text, not a proof; models produce confident, well-structured deliberation that arrives at errors, and hidden-CoT providers summarize it through their own filter. Trigger: high-stakes extraction from reasoning traces (compliance, audit) that assumes the thinking is faithful. The research field has no settled faithfulness guarantee, and neither does any vendor's documentation — treat reasoning traces as model-internal artifacts that happen to be visible, not as explanations you can rely on.

Benchmark transfer failure. Effort settings tuned to leaderboard tasks do not transfer to yours: ARC's trend lines are asymptotic on ARC's tasks, and your task's curve may flatten earlier or later. Trigger: adopting an effort level because it won a benchmark. The cure is the same ledger-driven per-workload curve, measured on your traffic.

Open Questions

The first is standardization of the effort surface. Five vendors, five incompatible ladders, one published effort-mapping quirk, and one provider retreating from manual control entirely — routing logic that spans providers (which every gateway and aggregator now does) is translating between semantics that no standards body has reconciled. Whether a common effort vocabulary stabilizes (the OpenAI-style ladder is closest) or whether adaptive thinking makes the whole question moot is unresolved, and the answer changes gateway design.

The second is the faithfulness question moving from research to regulation. If reasoning traces are increasingly encrypted (Gemini's signatures), dropped (DeepSeek multi-turn), or summarized at the vendor's discretion (Anthropic), then the auditability of AI decisions is degrading at exactly the moment deployments in regulated domains are scaling. Whether chain-of-thought becomes a required disclosure, a licensed artifact, or quietly disappears behind entirely opaque interfaces is a policy variable every long-horizon architecture should hedge.

The third is whether open-weight reasoning closes the gap for good. Qwen3 published its hybrid-mode training recipe (four stages, Apache-2.0 weights) and DeepSeek publishes MIT-licensed weights with thinking modes at commodity prices; the closed labs' remaining advantage is frontier reliability at high effort and the interactive benchmarks (ARC-AGI-3) that require harnesses rather than checkpoints. If the open recipe reproduces at the next capability tier, reasoning becomes a commodity feature with a token price — and the durable differentiators shift to context handling, tool reliability, and the cost curve, which is where the rest of this dossier's tooling already lives.

Resources

Provider reasoning documentation

Pricing and cost structures

Measurement