Bhaskar
← All writing
first principlesAug 19, 2026· 32 min read

When Should an LLM Think Longer?

A first-principles guide to reasoning systems, verification, and adaptive test-time compute

Updated August 2026

An LLM receives a problem, generates one answer, and stops. That is the cheapest possible inference policy. It is also the correct default.

The burden of proof belongs to every extra token, candidate, critique, search branch, verifier call, and model escalation that we add. More inference is useful only when it changes the probability of success enough to justify its latency and cost. A longer response is not evidence of better reasoning. It may be a better search, a redundant monologue, or a confident walk toward the wrong answer.

This article derives an adaptive reasoning gateway from that constraint. The gateway does not ask every model to “think harder.” It diagnoses what could go wrong, spends the smallest useful amount of compute, obtains independent evidence where possible, and stops as soon as the evidence is strong enough.

The governing objective is:

[ \text{choose strategy } s^*(x) = \arg\max_s \frac{\mathbb{E}[\text{task utility}\mid x,s]

  • \lambda_r,\mathbb{E}[\text{risk}\mid x,s]} {\mathbb{E}[\text{cost}\mid x,s]} ]

subject to hard limits on latency, money, tool permissions, and safety.

In plain language: for this particular request, choose the inference strategy that buys the most trustworthy success per dollar without violating the service contract.

1. Begin with one answer in one pass

Suppose a user asks:

A service handles 240 requests per second. Traffic grows by 25%, and each request causes two database writes. How many writes per second must the database sustain?

A direct model call maps input (x) to one output (y):

[ y \sim p_\theta(y\mid x) ]

With deterministic decoding, or a low-temperature approximation to it, the system returns one highly probable sequence. It pays for one prompt, one generation, and no verification.

This baseline has important advantages:

  • minimum latency and cost;
  • simple traces and failure analysis;
  • no selector or verifier that can introduce a second error;
  • no disagreement-resolution policy;
  • no illusion that repeated model opinions are independent evidence.

For an easy, familiar, low-stakes request, this may already be optimal. Before adding computation, measure it.

What exactly failed?

“The model got it wrong” is too coarse. Separate three failure classes:

  1. Generation failure: the model had enough information and a viable solution but expressed the answer incorrectly. Examples include malformed JSON, omitted fields, an invalid citation format, or a correct plan translated into syntactically broken code.
  2. Reasoning failure: the model selected a bad decomposition, made an invalid inference, performed arithmetic incorrectly, or committed early to a path that could not reach the answer.
  3. Verification failure: the system produced a candidate that could have been rejected or corrected, but it had no reliable check—or its checker accepted the wrong answer.

The distinction determines the remedy. Constrained decoding may fix generation. Additional candidates or search may fix reasoning. Tests, retrieval, or a verifier may fix selection. Asking for a longer explanation is not a universal treatment.

For the example above, the model must compute (240\times1.25\times2=600). If it outputs "writes_per_second": without a value, that is generation. If it computes 25% as 25 requests, that is reasoning. If two candidates say 530 and 600 but the system chooses 530, that is verification.

2. Task difficulty is a property of a model-task pair

A problem is not intrinsically “easy” or “hard.” It is easy or hard for a particular model, prompt, context, toolset, and output contract.

A five-line Python function may be easy for a coding model but hard for a small general model. A current tax-rate question may require no deep deduction yet remain impossible without fresh sources. A 100-page contract may contain the answer explicitly but impose difficult retrieval and evidence-alignment work.

Useful difficulty signals include:

  • number and dependency depth of required steps;
  • novelty relative to previously evaluated examples;
  • input length and distractor density;
  • ambiguity or missing information;
  • need to combine several documents or tools;
  • existence of strict constraints;
  • first-pass verifier failures;
  • disagreement among independently sampled answers;
  • historical error rate for the same task cluster.

Difficulty should therefore be estimated twice:

  • before generation, from request features and historical performance;
  • after generation, from candidate agreement and verification evidence.

The second estimate is often more informative. A task that looked hard may yield a candidate that compiles, passes every test, and satisfies a schema. A task that looked easy may produce three incompatible answers.

Uncertainty is not the same as difficulty

Difficulty concerns expected work. Uncertainty concerns how little confidence we should place in a particular prediction. They interact but are not identical.

  • A hard theorem may have a mechanically checked proof, so the final answer can have low uncertainty.
  • An easy-sounding question about an undocumented production incident may remain highly uncertain.
  • A model may be confidently wrong on an out-of-distribution request.

Raw verbal confidence—“I am 95% certain”—is not a probability until calibrated. Research has shown that models can sometimes predict whether their answers are correct when elicited in suitable formats, but calibration can deteriorate on new task distributions. The practical conclusion is not “trust model confidence”; it is “measure whether each confidence signal predicts correctness on your workload” (Kadavath et al., 2022).

3. Parallel compute: generate more than one candidate

The smallest increase beyond one-shot inference is to sample again.

For (N) candidates:

[ y_1,\dots,y_N \sim p_\theta(y\mid x; T,\text{seed}) ]

Sampling helps only if two conditions hold:

  1. the model sometimes produces a correct candidate; and
  2. the system can recognize or aggregate correct candidates better than chance.

Without the first, sampling creates many wrong answers. Without the second, it creates an expensive selection problem.

Best-of-N

Best-of-N generates (N) complete candidates, scores each with a reward or verifier (V(x,y_i)), and returns:

[ y^*=\arg\max_i V(x,y_i) ]

This is useful when errors vary across samples and candidate quality can be ranked. Code generation is the cleanest example: generate several implementations, run tests, discard failures, and rank survivors by additional criteria such as security checks or complexity.

The probability that at least one candidate is correct appears to improve quickly. If every sample had independent correctness probability (p), then:

[ P(\text{at least one correct})=1-(1-p)^N ]

At (p=0.4), five independent samples would give a 92.2% chance of containing a correct answer. But LLM samples are not independent.

Candidate diversity

Changing a random seed does not guarantee meaningful diversity. Samples can share the same misconception because they come from the same weights, prompt, context, training distribution, and retrieved evidence.

Useful diversity can come from:

  • moderate sampling temperature;
  • different decompositions or representations;
  • distinct retrieved evidence;
  • different tools;
  • different model families or sizes;
  • explicit roles with genuinely different information—not decorative personas;
  • separate attempts that do not see earlier candidates.

There is a trade-off. Too little diversity produces duplicates. Too much produces low-probability nonsense. Measure semantic or structural diversity among valid candidates, not merely lexical difference.

Self-consistency and majority voting

Self-consistency samples multiple reasoning paths, extracts their final answers, and chooses the answer with the greatest total support. The original work reported large benchmark gains, including +17.9 percentage points on GSM8K in its experimental setting (Wang et al., 2022).

For answers (a), simple voting computes:

[ \hat a=\arg\max_a\sum_{i=1}^{N}\mathbf 1[\text{answer}(y_i)=a] ]

This works best when:

  • the task has one canonical answer;
  • different valid reasoning routes converge on it;
  • incorrect routes disperse across several wrong answers;
  • answer normalization is reliable.

Majority voting is not equivalent to Best-of-N. Voting uses agreement as the selection signal. Best-of-N uses an explicit scorer. Voting is awkward for essays, plans, and code solutions that are semantically equivalent but textually different. It is natural for arithmetic, classification, and normalized short answers.

Correlated errors: the central limitation

If all candidates misread “25% growth” the same way, a 10–0 vote only measures consensus among correlated samples. It does not establish truth.

A useful approximation from correlated statistical samples is the effective sample size:

[ N_{\text{eff}}\approx \frac{N}{1+(N-1)\rho} ]

where (\rho) represents average error correlation. With (N=10) and (\rho=0.5), the effective evidence is only about 1.82 independent samples. This formula is an intuition, not a literal estimator for every LLM workload, but it explains why ten near-duplicates do not buy ten times the confidence.

Common causes of correlation are:

  • the same missing fact;
  • the same misleading prompt premise;
  • prompt injection in shared retrieved content;
  • the same flawed heuristic learned during training;
  • the same candidate being shown to all critics;
  • a judge model from the same family sharing the generator’s blind spots.

When correlation is high, spend compute on a different information channel—a calculator, tests, search, a database query, or a differently trained verifier—not on more paraphrases.

4. Sequential compute: critique, revise, and reflect

Parallel strategies explore width. Sequential strategies spend compute on depth.

Sequential refinement

The minimal loop is:

  1. generate a draft;
  2. identify a concrete defect;
  3. revise only in response to that defect;
  4. stop when a check passes or the budget ends.

The key is step 2. “Make this better” has no stopping target. “The JSON violates this schema at items[2].price” supplies actionable evidence.

Critique and revise

Self-Refine demonstrated a feedback-and-revision loop without updating model weights, reporting average absolute improvements of roughly 20% across the paper’s evaluated tasks (Madaan et al., 2023). The mechanism is plausible because evaluation and generation are different conditional tasks: a model that misses an error while composing may notice it when attention is focused on a rubric.

But a critique is still a model output. It can:

  • invent a defect in a correct answer;
  • praise a wrong answer;
  • focus on style instead of correctness;
  • anchor on the original approach;
  • cause a revision to replace a correct result with an incorrect one.

Therefore preserve the original candidate, score both versions independently, and require the revision to cite which failed constraint it repairs. Never overwrite the only good candidate merely because a critic sounded persuasive.

Reflection

Reflection is broader than critique. It summarizes why an attempt failed and proposes a changed strategy for a subsequent attempt. In agent systems, the reflection can be stored and used across trials. Reflexion, for example, used linguistic feedback in episodic memory rather than changing model weights (Shinn et al., 2023).

Use the terms precisely:

  • critique: what is wrong with this candidate?
  • revision: produce a corrected candidate.
  • reflection: what failure pattern and strategy change should influence the next attempt?

Reflection helps when the system receives informative feedback from an environment, such as a compiler error or failed browser action. Reflection without new evidence can become a longer restatement of the same misconception.

5. Verification is what turns more generation into a system

Generating candidates increases recall: the correct answer may appear somewhere. Verification must convert that recall into reliable precision.

Outcome verification

An outcome verifier checks the final result without endorsing the path.

Examples:

  • does the numeric answer equal 600?
  • does the function pass hidden tests?
  • is the returned JSON valid against the schema?
  • does the database query return the expected rows?
  • does the plan satisfy every stated constraint?

Outcome verification is cheap and powerful when the endpoint is mechanically checkable. It cannot distinguish a lucky answer from sound reasoning, and a weak test suite can accept a broken implementation.

Process verification

A process verifier scores intermediate steps: whether each transformation is valid, whether evidence supports the next claim, or whether a tool action follows from the current state.

Process verification is useful when:

  • a wrong intermediate step can cause expensive downstream actions;
  • the final answer is hard to evaluate directly;
  • partial credit helps search prune bad paths;
  • auditability of the procedure matters.

OpenAI’s process-supervision work reported better mathematical reasoning than outcome-only supervision in its setting by rewarding correct intermediate steps (OpenAI, 2023). That does not imply that a visible reasoning narrative is a faithful window into a model’s internal computation. A chain may omit causes, rationalize an answer after the fact, or contain errors even when the final answer is correct. Evaluate traces as observable artifacts, not as ground truth about the model’s mind.

The verification ladder

Prefer the strongest, most independent, and cheapest verifier available.

1. Deterministic verifiers

These implement explicit rules:

  • JSON Schema or Pydantic validation;
  • type checking;
  • regex and format checks;
  • arithmetic recomputation;
  • database constraints;
  • state-machine invariants;
  • policy rules encoded in code;
  • cryptographic or checksum comparisons.

They are reproducible and inexpensive. Their limitation is coverage: they can only verify the properties you encoded.

2. Code execution as verification

Generated code can be compiled and run in a sandbox against unit, property, integration, mutation, and security tests. HumanEval helped establish functional correctness via execution-based tests and the pass@k family of metrics (Chen et al., 2021).

Execution proves only behavior exercised by the test oracle. A malicious program can pass weak tests. A correct algorithm can still violate performance, privacy, or operational constraints. Sandboxing, resource limits, network restrictions, and disposable environments are mandatory because candidate code is untrusted.

3. Tool-assisted verification

Use authoritative systems for the facts they own:

  • calculator for arithmetic;
  • database for account state;
  • compiler for syntax and types;
  • calendar for availability;
  • package registry for versions;
  • official API for current prices or status.

The model should formulate the query and interpret the result; it should not impersonate the tool by guessing its output. ReAct showed how interleaving reasoning with external actions could reduce hallucination and error propagation on fact-oriented tasks in its experiments (Yao et al., 2022).

4. Retrieval as verification

Retrieval supplies external evidence and provenance. The system should decompose a candidate into checkable claims, retrieve authoritative sources for each claim, and test entailment, freshness, and source quality.

Retrieval is not automatically verification. A search result may be irrelevant, stale, adversarial, or merely repeat the same false claim. The original RAG formulation combined parametric generation with non-parametric memory to improve knowledge-intensive generation (Lewis et al., 2020); a production verifier additionally needs source-policy enforcement and claim-level evidence alignment.

5. Model-based verifiers

For qualities such as completeness, nuance, policy interpretation, or explanatory clarity, a model judge may be necessary. Give it:

  • the original task;
  • an explicit rubric with observable criteria;
  • candidate answers in randomized order;
  • authoritative evidence when factuality matters;
  • an abstain or “insufficient evidence” option;
  • structured scores and defect labels.

Evaluate the verifier itself against human or deterministic ground truth. Measure false acceptance and false rejection separately. A 90% accurate verifier can still be dangerous if nearly all of its errors accept harmful outputs.

Training a verifier to rank sampled math solutions improved over generation alone in early work (Cobbe et al., 2021). In application engineering, however, a judge must be treated as another fallible model—not an oracle.

Independence matters more than ceremony

“The model checked its own answer” is weak evidence when the checker sees the same context and shares the same blind spot. Verification improves when it introduces one or more of:

  • a deterministic oracle;
  • new authoritative data;
  • execution feedback;
  • hidden tests;
  • a model trained for verification;
  • a genuinely different model family;
  • human review for cases outside machine competence.

6. Search over solutions

Best-of-N samples complete answers. Search allocates compute to partial solutions.

Represent reasoning as states (z), candidate actions (a), a transition that produces the next state, and a value estimate (V(z)). Search repeatedly expands promising states and rejects bad ones before paying to complete them.

Tree-search intuition

A tree begins at the problem. Each node is a partial solution, and each edge is a proposed next step. The system can:

  1. generate several next steps;
  2. score the resulting partial states;
  3. expand the most promising states;
  4. backtrack from dead ends;
  5. stop on a verified solution or exhausted budget.

Tree of Thoughts demonstrated this pattern on tasks requiring exploration and lookahead; on the paper’s Game of 24 setup, it reported 74% success compared with 4% for a chain-of-thought baseline using GPT-4 (Yao et al., 2023). The result illustrates task dependence, not a universal 70-point gain.

Tree search helps when early decisions have large downstream consequences, partial states can be evaluated, and backtracking is valuable. It is wasteful when the answer is a short lookup or when partial-state scoring is no better than guessing.

Beam-search intuition

Beam search keeps only the best (B) partial paths at each depth. With beam width 1, it resembles greedy sequential reasoning. A wider beam preserves alternatives while bounding memory and compute.

The danger is verifier myopia. If the value function incorrectly penalizes a temporarily awkward but ultimately correct path, the beam prunes it permanently. Increasing (B) reduces premature pruning but increases cost. A diverse beam—limiting near-duplicate branches—can be more useful than simply widening it.

Width versus depth

Spend compute on width when independent attempts often vary and an outcome verifier is strong. Spend it on depth when the task benefits from iterative correction, feedback, or long dependencies. Spend it on search when partial paths can be scored and early choices must be reconsidered.

There is no universally best test-time scaling method. Inference-scaling studies find that the best allocation changes with problem difficulty and the quality of the verifier. A compute-optimal policy in one study was more than four times as efficient as a uniform Best-of-N baseline under its evaluation setup (Snell et al., 2024).

7. Allocate compute adaptively

A fixed policy—always generate eight samples, always use high reasoning effort—spends too much on easy tasks and too little on hard ones.

Difficulty estimation

Build a lightweight difficulty model from features available before generation:

request features
  task family
  input length
  constraint count
  estimated reasoning depth
  ambiguity markers
  required tools
  nearest evaluated examples and their failure rates
  business risk
        ↓
predicted P(direct answer succeeds)

Train it on out-of-fold predictions from your actual gateway. Do not label difficulty using token length alone. Long document extraction can be easy with good retrieval; a two-sentence logic trap can be hard.

Confidence estimation

Combine signals rather than trusting one number:

  • calibrated task-family success probability;
  • verifier score;
  • test coverage and outcomes;
  • normalized vote margin;
  • semantic agreement among candidates;
  • retrieval support and source freshness;
  • distance from the router’s training distribution;
  • whether the answer changed after critique;
  • presence of unresolved constraints.

A simple learned meta-model can estimate:

[ \hat p=P(\text{candidate is correct}\mid \text{signals}) ]

Calibration

If outputs assigned confidence 0.8 are correct only 0.6 of the time, the system is overconfident. Reliability diagrams plot predicted confidence against empirical accuracy. Useful metrics include Brier score, log loss, expected calibration error, and—most importantly for gateways—risk-coverage curves.

Coverage is the fraction of requests answered without escalation or abstention. Selective risk is the error rate among those answered. A good confidence mechanism allows the system to cover more requests at the same risk.

Calibrate separately by task family, model version, language, and strategy. Recalibrate after model or prompt changes. Monitor out-of-distribution traffic because in-distribution calibration does not guarantee transfer.

Early stopping

Stop when additional expected value is lower than marginal cost:

[ \mathbb{E}[\Delta U\mid \text{current state}] \leq \lambda_c\Delta C+\lambda_l\Delta L ]

Operational stopping rules can be concrete:

  • a deterministic verifier passes with sufficient coverage;
  • a candidate has independent authoritative support;
  • the calibrated correctness probability exceeds the threshold for this risk tier;
  • the same normalized answer wins by a prevalidated vote margin;
  • two successive refinements yield no verifier improvement;
  • remaining money, token, or latency budget is insufficient for the next action;
  • no allowed strategy has positive expected value;
  • the system must abstain or escalate.

Do not stop merely because the model says “final answer.” Do not continue merely because budget remains.

Budget-aware reasoning

A reasoning budget is multidimensional:

maximum wall-clock latency
maximum input/output/reasoning tokens
maximum model spend
maximum candidates
maximum refinement rounds
maximum tool calls
maximum search nodes
deadline for first token and final result

Recent systems expose direct reasoning-effort controls. OpenAI’s current reasoning API documentation, for example, exposes reasoning effort and notes that max_output_tokens bounds generated tokens including reasoning tokens (OpenAI reasoning guide). Budget forcing research has also shown that forcing a reasoning model to continue can improve some difficult math results; s1 reported an AIME24 increase from 50% to 57% in its setup (Muennighoff et al., 2025). Neither result licenses forcing every answer to be longer.

8. Model routing, cascades, and fallback policies

Test-time optimization includes choosing which model receives each unit of compute.

Small-model versus large-model routing

A small model may handle classification, extraction, formatting, and familiar code patterns cheaply. A larger reasoning model may be better for ambiguous architecture decisions, difficult debugging, or multi-document synthesis.

Route based on predicted utility, not prestige:

[ m^*=\arg\max_m \left( \widehat{P}(\text{success}\mid x,m)\cdot V_x-C_m-\lambda L_m \right) ]

where (V_x) is the value of success for request (x).

The router needs held-out evaluation data. A heuristic such as “long prompt means large model” will miss short but hard tasks and overpay for long but mechanical ones.

Cascaded inference

A cascade starts cheaply and escalates conditionally:

  1. small model, direct answer;
  2. deterministic or lightweight verification;
  3. revision or several samples if the defect seems recoverable;
  4. stronger model if uncertainty remains;
  5. tool or human escalation for high-risk unresolved cases.

FrugalGPT demonstrated that learned cascades could sharply reduce benchmark cost while retaining or improving accuracy in its studied configurations (Chen et al., 2023). Production gains depend on traffic mix, price changes, router accuracy, and whether the early model’s failure can be detected.

Fallback policies

A fallback must respond to a named failure, not generic discomfort.

Observed failure Useful next action
Invalid structure Repair or constrained regeneration
Arithmetic mismatch Calculator or executable expression
Tests fail locally Critique using test output, then revise
Candidates disagree Independent verifier or new evidence
Missing current fact Retrieve authoritative source
Small model uncertain on a hard cluster Fresh call to stronger model
Verifier lacks evidence Abstain or request missing information
High-risk action Human approval, regardless of confidence

Escalating a flawed answer by handing its full narrative to a stronger model can anchor the stronger model. When possible, provide the original problem plus objective failure evidence, and ask for a fresh solution before showing the earlier candidate.

9. Evaluate reasoning without confusing traces for truth

The gateway has two observable products: a final result and a trajectory.

Final-answer evaluation

Choose the strongest task-specific evaluator:

  • exact match after normalization;
  • deterministic field assertions;
  • unit or integration tests;
  • database-state assertions;
  • claim-level factual support;
  • human rubric;
  • blinded pairwise evaluation;
  • downstream business outcome.

Measure both pass@1 and strategy-level success. Best-of-N can have high pass@N—a correct answer exists somewhere—while selection accuracy remains poor. That is a verifier problem, not a generator victory.

Reasoning-trace evaluation

Do not grade a trace primarily by length or eloquence. Inspect:

  • validity of observable intermediate claims;
  • use of supplied evidence;
  • contradiction rate;
  • unnecessary steps and loops;
  • recovery from explicit feedback;
  • tool choice and argument correctness;
  • whether the trace supports the final answer;
  • whether hidden prompt perturbations change the answer without being acknowledged.

A trace can be useful for debugging without being a faithful causal explanation. Current research continues to find unfaithful reasoning traces, so operational controls must rely on verifiable state and outcomes, not presumed access to internal cognition (Arcuschin et al., 2025).

Latency-quality trade-off

Report a curve, not one accuracy number. At each strategy and budget, measure:

  • p50, p95, and p99 end-to-end latency;
  • time to first useful result;
  • generation and tool time separately;
  • success rate and severity-weighted failure rate;
  • timeout and cancellation rate.

Parallel candidates reduce wall-clock time relative to sequential sampling but increase instantaneous resource demand, rate-limit pressure, and spend. Sequential refinement has lower concurrency but accumulates latency.

Cost-quality trade-off

For each request (i), record actual input, cached, output, and reasoning tokens; tool fees; model calls; and compute time. Then calculate:

[ \text{success per dollar} =\frac{\sum_i \mathbf 1[\text{success}_i]}{\sum_i \text{cost}_i} ]

Also report cost per successful task:

[ \frac{\sum_i \text{cost}_i}{\sum_i \mathbf 1[\text{success}_i]} ]

The second metric is often easier to interpret. Segment both by task difficulty and risk. An average can hide a gateway that saves money on trivial requests but catastrophically under-computes hard ones.

10. A controlled experiment for the gateway

Do not compare strategies on different prompts, different model versions, or different judging rules. Build a stratified, frozen evaluation set with categories such as:

  • deterministic arithmetic;
  • code generation with hidden tests;
  • current factual questions with source requirements;
  • multi-step policy interpretation;
  • open-ended architecture recommendations.

For every example, define success before running the experiment. Freeze prompts, model snapshots where possible, sampling settings, tool versions, timeouts, and pricing. Randomize candidate order for model judges. Run enough repeated samples to estimate variance. Bootstrap confidence intervals at the request level, not the candidate level.

Compare these arms:

  1. one direct answer;
  2. one answer plus sequential critique/revision;
  3. Best-of-N with a fixed verifier;
  4. self-consistency with normalized voting;
  5. one answer plus relevant tool verification;
  6. N candidates selected by a learned verifier;
  7. direct small model with stronger-model fallback;
  8. the adaptive policy.

Track:

  • task success;
  • verifier accuracy;
  • false acceptance and false rejection;
  • false confidence: high-confidence failures;
  • p50/p95/p99 latency;
  • tokens and monetary cost;
  • success per dollar;
  • tool-use error;
  • escalation and abstention rates.

An illustrative result table

The numbers below are hypothetical, included to show how to reason about results; they are not research findings or claims about a particular model.

Strategy Success False-confidence rate p95 latency Mean cost/request Successful tasks per $1
Direct 72% 11% 1.8 s $0.010 72.0
Critique + revise 76% 9% 4.0 s $0.021 36.2
Best-of-5 83% 7% 4.8 s $0.052 16.0
Self-consistency-5 80% 8% 4.5 s $0.048 16.7
Tool verified 89% 3% 3.1 s $0.018 49.4
Strong model direct 87% 5% 5.4 s $0.070 12.4
Adaptive gateway 88% 3% 4.2 s $0.024 36.7

The correct conclusion is not “adaptive wins everything.” In this illustration:

  • direct generation is cheapest per success but too risky if the false-confidence requirement is below 5%;
  • tool verification dominates additional sampling where a tool oracle exists;
  • the strong model has good quality but poor economics;
  • the gateway nearly matches tool-verified quality across a mixed workload while covering tasks without tool oracles;
  • Best-of-5 is retained only for task segments where its incremental success justifies the cost.

Now inspect results by difficulty. If direct success is 96% on easy tasks and 38% on hard tasks, spending five samples everywhere is obviously wasteful. The adaptive policy should send easy tasks directly and reserve search, verification, or escalation for cases whose expected marginal value is positive.

11. When additional reasoning hurts

More inference can reduce quality for several reasons:

  1. Overthinking: the model abandons an initially correct answer after unnecessary reconsideration.
  2. Error amplification: a false early assumption becomes the premise for every later step.
  3. Correlated sampling: several outputs repeat one misconception and create false consensus.
  4. Verifier bias: a fluent wrong answer receives a higher score than a terse correct one.
  5. Context pollution: candidates, critiques, and retrieved documents crowd out the original constraints.
  6. Anchoring: later attempts imitate the first rather than independently solve the task.
  7. Search error: a poor value model prunes the correct branch.
  8. Tool misuse: extra calls introduce stale, irrelevant, or adversarial data.
  9. Latency failure: a theoretically better answer arrives after the user or upstream system times out.
  10. Budget displacement: money spent on easy cases is unavailable for genuinely difficult ones.
  11. Non-verifiable refinement: prose becomes smoother while factual correctness stays flat.
  12. Distribution shift: forced long reasoning pushes the model away from behavior on which it was trained and evaluated.

Recent preprint evidence reports diminishing returns and cases where extended reasoning changes correct answers to incorrect ones, reinforcing the need for task-dependent stopping rather than uniform long budgets (Zhou et al., 2026). Treat this as an active research area and validate the effect on your own models.

The practical rule is simple:

Extra compute without new candidates, new evidence, stronger verification, or purposeful search is usually just extra surface area for error.

12. Build the adaptive reasoning gateway

The gateway is a policy and evidence system around model calls. It should not expose hidden reasoning or depend on storing private internal chains. Record concise candidate artifacts, actions, scores, evidence, and decisions sufficient for evaluation and audit.

Core components

flowchart TD
    A["Request + constraints"] --> B["Difficulty and risk estimator"]
    B --> C["Budgeted strategy policy"]
    C --> D["Candidate generator"]
    D --> E["Verifier ladder"]
    E --> F{"Confidence sufficient?"}
    F -- Yes --> G["Return answer + evidence"]
    F -- No, budget remains --> C
    F -- No, budget exhausted --> H["Escalate or abstain"]
  1. Request normalizer: identifies task family, constraints, freshness needs, risk tier, and allowed tools.
  2. Difficulty estimator: predicts direct-answer success and out-of-distribution risk.
  3. Budget manager: tracks wall time, tokens, money, calls, and concurrency.
  4. Strategy policy: chooses direct, revise, sample, search, tool-check, or stronger-model fallback.
  5. Candidate store: keeps every candidate immutable with model and prompt metadata.
  6. Verifier registry: maps task properties to deterministic, execution, retrieval, tool, and model verifiers.
  7. Confidence calibrator: turns observed signals into empirical correctness probabilities.
  8. Stopping controller: accepts, continues, escalates, or abstains.
  9. Experiment logger: records enough information to compare policies reproducibly.

Minimal data model

from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Literal

class Decision(str, Enum):
    ACCEPT = "accept"
    CONTINUE = "continue"
    ESCALATE = "escalate"
    ABSTAIN = "abstain"

@dataclass(frozen=True)
class Budget:
    max_cost_usd: float
    max_latency_ms: int
    max_total_tokens: int
    max_candidates: int
    max_tool_calls: int

@dataclass(frozen=True)
class Candidate:
    id: str
    model: str
    strategy: str
    answer: Any
    usage: dict[str, int]
    latency_ms: int
    parent_id: str | None = None

@dataclass(frozen=True)
class Verification:
    candidate_id: str
    verifier: str
    verifier_type: Literal["deterministic", "execution", "tool", "retrieval", "model"]
    passed: bool | None
    score: float | None
    defects: tuple[str, ...]
    evidence_refs: tuple[str, ...]

@dataclass
class RunState:
    request_id: str
    task_family: str
    risk_tier: str
    budget: Budget
    candidates: list[Candidate] = field(default_factory=list)
    verifications: list[Verification] = field(default_factory=list)
    cost_usd: float = 0.0
    total_tokens: int = 0
    tool_calls: int = 0
    decision: Decision = Decision.CONTINUE

Store explicit summaries or structured intermediate states needed by your workflow. Avoid designing observability around access to a provider’s private reasoning tokens. Current APIs may meter internal reasoning while returning only answers or summaries; the application contract should depend on outcomes, evidence, and action traces.

Strategy policy

Start with rules before training a router:

def choose_strategy(profile, state):
    if profile.has_deterministic_oracle:
        return "direct_then_deterministic_verify"

    if profile.requires_fresh_facts:
        return "retrieve_then_generate_then_claim_verify"

    if profile.is_code and profile.tests_available:
        return "best_of_n_execute_select"

    if profile.predicted_direct_success >= 0.95 and profile.risk_tier == "low":
        return "direct"

    if profile.canonical_answer and profile.expected_error_diversity >= 0.5:
        return "self_consistency"

    if profile.partial_states_are_scorable and profile.requires_backtracking:
        return "bounded_tree_search"

    if profile.failure_is_likely_revisable:
        return "critique_then_revise"

    return "strong_model_fallback"

Then replace thresholds with a policy learned from logged, out-of-fold results. Never train on labels produced by the same unvalidated judge that the policy will optimize; the router will learn the judge’s biases.

Candidate generation

Each strategy must define how it creates genuine alternatives:

  • direct: one low-variance candidate;
  • revision: immutable original plus a defect-targeted revision;
  • Best-of-N: independent complete candidates, hidden from one another;
  • self-consistency: diverse solution paths plus canonical answer extraction;
  • search: bounded partial-state expansion with diversity constraints;
  • fallback: fresh stronger-model attempt, optionally followed by comparison.

Verification order

Run cheap, decisive checks first:

schema and safety gates
        ↓
deterministic invariants
        ↓
execution or authoritative tools
        ↓
retrieval and evidence alignment
        ↓
model rubric judge
        ↓
human review or abstention

A failed hard constraint should immediately reject a candidate. A soft judge score should not override a failed unit test or an authoritative database result.

Early-stopping controller

def decide(state, calibrated_p_correct, hard_checks_passed, next_action_value):
    threshold = 0.995 if state.risk_tier == "high" else 0.90

    if hard_checks_passed and calibrated_p_correct >= threshold:
        return Decision.ACCEPT

    if budget_exhausted(state):
        return Decision.ESCALATE if state.risk_tier == "high" else Decision.ABSTAIN

    if next_action_value <= 0:
        return Decision.ESCALATE if state.risk_tier == "high" else Decision.ABSTAIN

    return Decision.CONTINUE

The numerical thresholds must come from risk requirements and validation data. They are not model-prompt constants.

Logging candidates and decisions

For every run, record:

  • request and task-family IDs, with sensitive content redacted or access-controlled;
  • policy and experiment-arm version;
  • model snapshot, parameters, and prompt/template version;
  • candidate parentage and normalized answer;
  • verifier version, result, defects, evidence references, and latency;
  • token and cost ledger;
  • stop reason;
  • final answer, escalation, or abstention;
  • delayed ground truth and user correction when available.

This makes it possible to answer: Did the generator fail to include a correct candidate? Did the verifier mis-rank it? Did the router under-allocate compute? Did early stopping accept false confidence?

13. Production monitoring

Offline benchmark gains can disappear when traffic, model versions, sources, or prices change. Monitor the gateway as a policy under changing conditions.

Quality and calibration

  • success and severity-weighted failure by task family;
  • verifier false-accept and false-reject rates;
  • high-confidence error rate;
  • calibration error and risk-coverage curves;
  • candidate disagreement and semantic diversity;
  • answer-change rate after critique—and whether changes help;
  • escalation, abstention, and human-overturn rates.

Efficiency

  • cost and tokens per request and per success;
  • p50/p95/p99 latency by strategy;
  • candidates and refinement rounds per request;
  • early-stop savings versus the maximum policy;
  • tool-call count, failure, retry, and cache-hit rates;
  • queueing and provider rate-limit pressure.

Drift and incidents

  • changes after model, prompt, verifier, retriever, or price updates;
  • task-distribution and language drift;
  • source freshness and retrieval-quality drift;
  • systematic disagreement between models and tools;
  • loops, repeated critiques, and token-budget exhaustion;
  • one task cluster consuming disproportionate compute;
  • safety-gate or permission violations.

Deploy policy changes behind experiment flags. Shadow new routers before allowing them to control spend. Keep a direct or previous-policy rollback. Re-run the frozen evaluation set whenever any generator, judge, prompt, tool, or threshold changes.

14. The mental model to retain

Test-time optimization is not “make the model talk longer.” It is controlled allocation of inference resources.

You now have seven distinct levers:

  1. One-shot generation for easy, low-risk tasks.
  2. Parallel sampling when independent attempts can expose a correct candidate.
  3. Self-consistency when answers are canonical and wrong paths disperse.
  4. Sequential refinement when specific, detectable defects can be repaired.
  5. Verification when rules, execution, tools, sources, judges, or humans can add evidence.
  6. Search when partial solutions are scorable and backtracking matters.
  7. Routing and cascades when task difficulty justifies different models or budgets.

The gateway should ask four questions after every increment of compute:

  1. What failure are we trying to correct?
  2. What genuinely new information or search coverage will the next action add?
  3. How will we know whether it helped?
  4. Is its expected improvement worth its marginal latency, cost, and risk?

If the system cannot answer those questions, it should not spend the compute.

The strongest reasoning system is therefore not the one that always thinks longest. It is the one that knows when a direct answer is sufficient, when disagreement demands evidence, when a verifier is stronger than another sample, when a small model should yield to a larger one, and when no amount of additional inference can replace missing information or human judgment.

Research and technical sources