When2Think AI Adaptive Reasoning Efficiency
The Moment a System Is Right but Gives the Wrong Answer
A large reasoning model faces a simple arithmetic problem: 2 + 2. It generates a chain of thought spanning hundreds of tokens — restating the problem, considering alternative interpretations, verifying its work, second-guessing the verification. The answer, when it finally arrives, is correct. But the computation spent to reach it dwarfs the computation the problem required by orders of magnitude. The system was right. It was also wasteful in a way that compounds across millions of queries.
Now reverse the scenario. A genuinely difficult problem arrives — one requiring multi-step inference, careful decomposition, sustained attention. The same model, trained to be efficient, answers directly. No chain of thought. No intermediate steps. The answer is wrong. The system was efficient. It was also incorrect in a way that erodes trust.
This is the efficiency tax, and it is the problem that a new framework called When2Think addresses (arXiv:2609.19671v2). The paper, submitted to arXiv in September 2025 by Jaejun Shim and three co-authors, proposes a method for instance-adaptive computation allocation in large reasoning models. [2] The core insight is deceptively simple: not every problem deserves the same amount of thinking. The implementation, however, requires navigating a fundamental tension in how these systems are trained.
The Structural Asymmetry Between Thinking Too Much and Thinking Too Little
Large Reasoning Models (LRMs) have demonstrated remarkable capabilities on complex tasks by generating explicit chains of thought — intermediate reasoning steps that allow the model to decompose problems, explore alternatives, and verify conclusions. The approach has proven effective. It has also proven expensive.
The cost is not merely financial, though token generation carries real computational overhead. The deeper cost is computation allocation. A model that spends equal effort on trivial and difficult problems is misallocating its most valuable resource: computation. The When2Think paper identifies this as a core inefficiency in current LRM architectures. Models “often overthink easy problems and underthink hard ones,” leading to inefficient computation allocation. [2] Existing approaches to this problem fall into two categories. Some methods regulate the amount of generated computation — capping token counts, pruning reasoning chains, or applying early stopping criteria. Others select between direct answering and explicit reasoning — a binary choice between thinking and not thinking. Neither approach jointly controls whether to reason and how much computation to allocate within reasoning. The result is a difficulty-dependent loss in accuracy when computation is reduced. This is the efficiency tax.
The asymmetry is structural. Overthinking wastes resources but rarely produces wrong answers. Underthinking saves resources but frequently produces wrong answers. A system optimized purely for efficiency will underthink. A system optimized purely for accuracy will overthink. The challenge is finding the allocation that minimizes the efficiency tax across a distribution of problems with varying difficulty.
What When2Think Actually Does
The framework proposed in the paper is called When2Think, described as an RLVR-based post-training framework for instance-adaptive computation allocation. RLVR stands for Reinforcement Learning with Verifiable Rewards — a training paradigm where the model receives feedback based on whether its output can be verified as correct or incorrect, rather than based on a learned reward model’s judgment.
The core mechanism is Instance-level Difficulty-Aware Control (IDAC). It uses cached reference statistics of success and token cost to modulate a correctness-gated efficiency bonus based on generated token count. In simpler terms: the system maintains a cache of how often similar problems were solved correctly and how many tokens were typically required. When a new problem arrives, this cached information informs a bonus structure that rewards correct answers while penalizing unnecessary token generation.
The correctness gate is crucial. The efficiency bonus only applies when the answer is correct. A model cannot game the system by generating fewer tokens and producing wrong answers. The reward structure creates a gradient toward the most efficient path to correctness, not merely the shortest path.
Two additional mechanisms support the framework. Importance sampling supports exploration of Think and NoThink modes — allowing the model to try both reasoning and direct answering during training, with the sampling adjusted to focus on informative cases. Batch-Wise Standardization constructs standardized advantages for critic-free optimization. This means the framework does not require a learned critic — a separate model that estimates the value of different actions — which simplifies training and reduces computational overhead.
The framework also requires neither a learned reward model nor a learned critic. Offline reference caching avoids online reference-model queries during policy updates. This architectural choice matters for practical deployment: it means the system can be trained without the infrastructure that reward-model-based approaches typically demand.
The Numbers from AIME24
The paper reports results on AIME24, a benchmark consisting of problems from the American Invitational Mathematics Examination (AIME). [1] On this benchmark, When2Think improves Pass@3 — a metric measuring whether at least one of three generated answers is correct (Pass@k) — by 10.0 percentage points, while reducing token usage by 27.9%. [2] Pass@3 improved by 10.0 percentage points, and token usage was reduced by 27.9%. The key finding is that instance-adaptive computation allocation indicates better accuracy than fixed allocation strategies at equivalent or lower token budgets. The efficiency tax is reduced, not merely shifted.
This matters because the standard trade-off in reasoning model deployment has been between accuracy and cost. When2Think demonstrates that the trade-off is not fixed. By allocating computation based on instance difficulty, the framework achieves better outcomes on both dimensions simultaneously for the distribution of problems it was trained on.
The AIME24 results are particularly relevant because the benchmark contains problems of varying difficulty. Some require multi-step reasoning; others are more straightforward. A fixed allocation strategy will inevitably overthink the easy problems and underthink the hard ones. An adaptive strategy can, in principle, match allocation to difficulty.
Why This Is Harder Than It Sounds
The challenge in building such a system is not the concept — it is the implementation. Difficulty is not directly observable. The model must infer it from the problem statement and from cached statistics about similar problems. The inference is imperfect, and errors compound.
Consider the feedback loop. If the model incorrectly classifies a hard problem as easy, it will underthink and likely produce a wrong answer. The correctness gate prevents this from being rewarded, but the training signal is weaker for such cases because the model never explored the reasoning path that would have led to the correct answer. The importance sampling mechanism addresses this by ensuring exploration of both Think and NoThink modes, but the fundamental challenge remains: the system must learn to recognize difficulty without having experienced the full range of difficulty during training.
The cached reference statistics help. By maintaining a record of success rates and token costs for similar problems, the system can make informed estimates about new problems. But the cache is only as good as the similarity metric used to retrieve from it. Problems that are superficially similar but structurally different will receive inappropriate allocation.
The Batch-Wise Standardization mechanism provides a form of normalization that helps stabilize training across batches with varying difficulty distributions. This is a technical detail, but it matters for reproducibility: without it, the training signal would be noisy and the learned policy would be unstable.
The Broader Context of Efficiency in AI Reasoning
The When2Think paper arrives at a moment when the computational cost of AI reasoning has become a central concern. The largest reasoning models generate thousands of tokens for problems that once required only a few (arXiv:2609.19671v2). The capability gains are real, but so are the costs.
The efficiency tax described in the paper is not merely an engineering problem. It reflects a deeper issue in how these systems are trained. Standard training objectives reward correctness without penalizing unnecessary computation. The result is a model that learns to think more than necessary because thinking more is never penalized. When2Think’s correctness-gated efficiency bonus changes this incentive structure.
The approach is related to but distinct from other efficiency-focused methods. Speculative decoding, for example, uses a smaller model to draft outputs that a larger model verifies (Speculative Decoding) — an architecture-level optimization. Quantization reduces the precision of weights and activations (Quantization) — a numerical optimization. When2Think operates at the level of reasoning allocation — a cognitive optimization. The three approaches are complementary rather than competing.
The framework also connects to a broader question in AI research: how should systems allocate limited computational resources across tasks of varying difficulty? This is not merely a technical question. It is a question about the structure of intelligence itself. Biological brains allocate attention selectively, focusing resources on what matters and ignoring what does not (Attention). When2Think is an attempt to build this selectivity into artificial reasoning systems.
What the Framework Does Not Claim
The paper is careful in its claims. It does not assert that When2Think solves the problem of computation allocation in general. It does not claim that the efficiency gains observed on AIME24 will transfer to all domains. It does not argue that the framework eliminates the need for human oversight or that it produces interpretable reasoning traces.
What it does claim is more modest and more defensible: that instance-adaptive computation allocation, implemented through the IDAC mechanism, reduces the efficiency tax on verifiable-reward tasks. The AIME24 results support this claim. The architectural choices — no learned reward model, no learned critic, offline reference caching — make the framework practical to implement.
The limitations are implicit in the design. The framework requires verifiable rewards, which means it applies to domains where correctness can be automatically checked. Mathematics is one such domain. Many real-world tasks are not. The cached reference statistics require a distribution of similar problems to be useful. A model facing entirely novel problems has no cache to draw from.
The correctness gate, while preventing gaming, also creates a potential blind spot. If the model consistently fails on a particular class of problems, it will never receive the efficiency bonus for those problems because it never produces correct answers. The importance sampling mechanism mitigates this by ensuring exploration, but the fundamental challenge of learning from failure remains.
The Question of Responsibility When a System Gets It Wrong
The When2Think framework raises a question that extends beyond its technical details. If a system is trained to allocate computation adaptively — to think more about hard problems and less about easy ones — what happens when it misjudges difficulty? Who is responsible for the wrong answer that results?
The efficiency tax is not merely a cost. It is a measure of the system’s failure to match its computational effort to the problem’s demands. When the system underthinks a hard problem, the failure is one of allocation. The model had the capability to solve the problem but did not deploy it. The error is not in the weights but in the policy.
This is a new kind of failure mode. Traditional AI errors arise from insufficient capability — the model cannot solve the problem. Adaptive computation errors arise from misallocation — the model could solve the problem but did not try hard enough. The distinction matters for how we think about accountability. A system that fails because it cannot do better is different from a system that fails because it chose not to.
The correctness gate in When2Think’s training objective ensures that the model is never rewarded for wrong answers, regardless of how few tokens they required. But the gate does not eliminate the possibility of wrong answers from underthinking. It merely ensures that such answers are not reinforced. At deployment time, the system may still misjudge difficulty and produce errors that a fixed-allocation system would have avoided.
The paper does not address this question directly. Its focus is on the training framework and the empirical results. But the framework’s design choices — the correctness gate, the importance sampling, the cached reference statistics — are all responses to the challenge of learning adaptive allocation without sacrificing accuracy. The efficiency tax is the measure of how well they succeed.
What the Results Suggest About the Future of Reasoning Systems
The AIME24 results, while specific to one benchmark, point toward a broader possibility. If reasoning systems can learn to allocate computation adaptively, the trade-off between accuracy and efficiency becomes less rigid. The efficiency tax can be reduced, not merely accepted.
This has implications for deployment. A system that thinks more about hard problems and less about easy ones can serve a wider range of queries within a fixed computational budget. It can respond quickly to straightforward requests and take more time for complex ones. The user experience improves not because the system is faster in general but because it is faster where speed matters and more thorough where thoroughness matters.
The framework’s avoidance of learned reward models and critics is also significant. These components add complexity and computational overhead to training. By using cached reference statistics and batch-wise standardization instead, When2Think reduces the infrastructure required to implement adaptive computation allocation. This makes the approach more accessible to researchers and practitioners without large-scale training resources.
The paper’s contribution is not a breakthrough in reasoning capability. It is a refinement in reasoning efficiency. The distinction matters. The field has made remarkable progress in what reasoning models can do. When2Think addresses how they do it — how much computation they allocate, when they reason explicitly, and when they answer directly. These are questions of policy rather than capability, and they are increasingly important as reasoning models move from research demonstrations to production systems.
The Efficiency Tax as a Design Constraint
The concept of the efficiency tax provides a useful frame for thinking about reasoning system design. Any system that reduces computation will pay some accuracy cost. The question is how large that cost is and whether it can be minimized.
When2Think’s approach is to make the tax difficulty-dependent. Easy problems should incur little or no tax because they require little computation. Hard problems should incur a tax only when the reduced computation is insufficient for correctness. The IDAC mechanism implements this logic through cached statistics and correctness-gated bonuses.
The framework does not eliminate the efficiency tax. It reduces it by making allocation more precise. The remaining tax reflects the irreducible uncertainty in difficulty estimation. Some problems will always be misclassified. Some hard problems will look easy. Some easy problems will look hard. The system’s performance depends on how well it navigates this uncertainty.
The AIME24 results suggest that the navigation is effective for the distribution of problems in that benchmark. Whether it generalizes to other domains — code generation, scientific reasoning, legal analysis — is an open question. The framework’s requirements — verifiable rewards, a distribution of similar problems, cached reference statistics — define the conditions under which it can be applied.
The Architecture of Selective Attention
When2Think is, at its core, an architecture for selective attention. It decides which problems deserve sustained reasoning and which can be answered directly. This selectivity is what distinguishes it from fixed-allocation approaches.
The human analogy is imperfect but instructive. When a person encounters a problem, they do not apply the same cognitive effort regardless of difficulty. They assess the problem, draw on past experience with similar problems, and allocate attention accordingly. Easy problems receive minimal attention. Hard problems receive sustained focus. The allocation is adaptive and largely automatic.
When2Think attempts to build this adaptivity into artificial reasoning systems. The cached reference statistics serve as a form of experience. The importance sampling ensures exploration of both reasoning and direct answering. The correctness gate ensures that efficiency does not come at the cost of accuracy. The batch-wise standardization stabilizes the learning process.
The framework is not a model of human cognition. It is an engineering solution to a specific problem in AI reasoning. But the problem it solves — how to allocate limited computation across tasks of varying difficulty — is one that any intelligent system must confront. The solution When2Think proposes is one approach among many. Its contribution is to demonstrate that the approach works on a benchmark where correctness is verifiable and difficulty varies.
What Comes Next
The paper does not speculate about future work, but the framework’s design suggests several directions. The cached reference statistics could be extended to incorporate more nuanced difficulty signals — not just success rates and token costs but also problem features that predict difficulty. The importance sampling could be refined to focus exploration on cases where the difficulty estimate is most uncertain. The correctness gate could be supplemented with partial credit for reasoning traces that approach correct answers without reaching them.
These are engineering refinements rather than conceptual breakthroughs. The core idea — instance-adaptive computation allocation with correctness-gated efficiency bonuses — is established by the paper’s results. The question is how far it can be pushed.
The broader context is a field that has spent years pushing the boundaries of what reasoning models can do. The next phase may be about pushing the boundaries of how efficiently they do it. When2Think is a contribution to that phase. Its results on AIME24 are modest but meaningful. They suggest that the efficiency tax is not fixed — that it can be reduced through better allocation. For a field where computational costs are increasingly central, that is a finding worth noting.
The system that knows when not to think is not a system that thinks less. It is a system that thinks more precisely — allocating its computational resources where they are needed and withholding them where they are not. The AIME24 results suggest it succeeds, at least in part. The rest is a matter of refinement, extension, and the slow work of turning a promising framework into a reliable tool.