The Claim That Started It
Microsoft’s Phi-4-reasoning arrives with a proposition that sounds almost moral: intelligence need not be bought with size. The 14-billion-parameter model, described in a technical report posted in April 2025, is presented as evidence that a small model can reason through mathematics, science, and code if it is taught the right way. [1] The report’s own framing is narrower than the marketing: it claims strong performance on complex reasoning tasks, not understanding. That is the claim on the surface. Underneath it sits a more interesting question, one the report itself keeps circling without ever quite naming: what does it mean for a model to look like it is reasoning, and how would we know the difference between the appearance and the thing?
The gap is not a scandal. It is a structural feature of how these systems are built, trained, and measured. And Phi-4-reasoning, precisely because it is small, well-documented, and honest about its own limits, is an unusually clear window into it.
What the Model Actually Does
Phi-4-reasoning is a fine-tuned version of the existing Phi-4 model. The training recipe is described in the technical report as supervised fine-tuning on a curated set of “teachable” prompts — problems selected to sit at the edge of what the base model can already handle, neither too easy nor too hard. For each selected prompt, reasoning demonstrations were generated using OpenAI’s o3-mini. The model learned to produce long chains of thought by imitating those demonstrations.
A second variant, Phi-4-reasoning-plus, adds a short phase of outcome-based reinforcement learning on roughly 6,000 math-focused problems with verifiable solutions. The plus version generates longer reasoning traces and scores higher on math benchmarks, at the cost of using about 1.5 times more tokens on average than the base reasoning model.
The reported gains over Phi-4 are large. Accuracy on math benchmarks such as AIME 2025 and OmniMath improves by more than 50 percentage points over the base Phi-4 model. [1] Coding performance on LiveCodeBench rises by more than 25 points over the base Phi-4 model. [1]. Algorithmic and planning tasks — the traveling salesman problem, 3SAT, calendar planning — improve by 30 to 60 points over the base Phi-4 model, even though the training did not deliberately target those skills. [1]. The models also outperform DeepSeek-R1-Distill-Llama-70B, a model five times their size, on most benchmarks, and approach the full 671-billion-parameter DeepSeek-R1 on AIME 2025. [1].
Those are the numbers. They are real, and they are impressive. The question is what they mean.
The Difference Between Format and Skill
One of the report’s most revealing observations concerns the relationship between format and skill. The model learns the superficial structure of reasoning — the tags that mark where thinking begins and ends — from demonstrations generated by another model. But the quality of the reasoning inside those tags is what the training is meant to improve. The report’s authors read the benchmark gains as evidence that the model is not merely copying format but acquiring reasoning as a learned skill — an interpretation, not a proof.
That reading is plausible. It is also, by the authors’ own framing, an interpretation rather than a proof. The distinction matters because it is exactly the kind of gap this article is about. A system that produces the visible markers of deliberation can be mistaken for a system that deliberates, and the training data — long chains of thought generated by another model — provides the template for the markers before it provides anything like the substance.
The report does not claim the model understands. It claims the model’s chain-of-thought block becomes more effective over training. Those are different statements, and the space between them is where overconfidence in AI usually begins.
The Teacher in the Room
There is a detail in the methodology that deserves more attention than it typically gets. The reasoning traces the model learned from were not written by humans. They were generated by o3-mini. The report notes that o3-mini in high-effort mode was a stronger teacher than in medium mode, and that a better teacher combined with a longer 32k-token context produced better results. [1].
This is distillation: a smaller model learning to imitate the outputs of a larger one. It works. The evidence is in the benchmark scores. But it also means the model’s reasoning style is inherited, not discovered. When Phi-4-reasoning produces a step-by-step solution, it is reproducing a pattern of problem-solving that another system generated. The pattern can be genuinely useful. It can also carry the teacher’s mistakes, and the student has no independent way to check them.
The report is candid about a related risk. It notes that the model tends to regurgitate variations of its safety guidelines inside the thinking block, and that the effect of this on safety is an active topic of research. A model that repeats its instructions to itself while appearing to reason is a small, concrete example of the larger problem: the visible process and the actual process are not the same thing.
The Measurement Problem the Authors Admit
Perhaps the most honest section of the report concerns evaluation itself. The authors point out that benchmark scores on small datasets vary substantially across repeated runs. Two runs of an average-of-five evaluation on AIME 2025 can differ by up to 5 to 10 percentage points. [1] That is enough to make side-by-side comparisons unreliable — not just for Phi-4-reasoning, but for every model in the comparison, including those from OpenAI and DeepSeek.
This is a remarkable admission to find in a paper that also reports competitive benchmark results. It means the headline numbers, taken at face value, overstate their own precision. The report responds by reporting standard deviations, running 50 to 64 generations on AIME 2025, and adding larger benchmarks. [1]. That is good practice. It is also an acknowledgment that the scoreboard, which is how most people encounter these models, is a blurrier picture than it looks.
The authors go further. They compare different evaluation strategies — average, majority, best-of-N, worst-of-N — and find significant gaps between typical and optimal generations. Those gaps suggest room to improve performance through better decoding. They also suggest something less comfortable: the same model can look excellent or mediocre depending on how you choose to sample it. The number you report is a choice.
Where the Reasoning Thins Out
The report’s benchmark analysis contains a finding that undercuts the smooth story of general reasoning ability. Improvements are comparatively smaller in biology and chemistry than in mathematics and physics. [1]. Even within mathematics, discrete mathematics shows relatively modest gains. These patterns hold across the models evaluated.
In other words, the reasoning skill that transferred so impressively to traveling salesman problems and calendar planning did not transfer evenly. It appears to have transferred where the structure of the problem resembled the training data, and less where it did not. The model’s apparent generality has a shape, and the shape follows the data.
This is not a failure. It is a boundary. But a boundary is exactly what a reader needs to know when deciding how much to trust a system that presents its answers with the same confident step-by-step format regardless of whether it is on familiar ground.
The Plus
Version and the Cost of Thinking Longer Phi-4-reasoning-plus illustrates the trade-off in its purest form. Reinforcement learning on verifiable math problems pushes accuracy higher, and it does so by generating longer reasoning traces: the plus version is 15 percent more accurate than Phi-4-reasoning across 50 runs on AIME 2025, but its traces are about 1.5 times longer on average. [1].
The report reads this as a sign that reinforcement learning exploration and verification could be improved for broader task sets. A less generous reading is that the model learned to think longer where it was rewarded for thinking longer, and the extra length did not generalize as cleanly as the extra accuracy did. The plus version is better at math because it was trained to be better at math. The mechanism is reinforcement, and reinforcement shapes behavior toward the reward.
What “Small” Buys and What It Does Not
The case for a 14-billion-parameter reasoning model is practical. Smaller models are cheaper to run, faster to respond, and easier to deploy in constrained environments. They expand the options for where inference can happen. None of that is in dispute.
What the report does not claim — and what readers should not infer — is that small models are inherently more trustworthy or more transparent. A small model can produce a confident, well-formatted, step-by-step answer that is wrong at the first step and consistent all the way to the end. The report itself warns that a model may carefully make mistakes based on incorrect premises, and that verification for each use case is necessary. [1].
That caution is the heart of the matter. The model’s output format invites trust. The model’s actual reliability depends on the problem, the domain, and the premises. The format does not signal which is which.
The Workflow That Keeps
Humans in the Loop The report’s practical recommendation is modest and worth stating plainly: entrust AI with preparatory work — organizing documents, generating candidate calculations, drafting code, building checklists — and keep humans responsible for judging whether the premises hold, whether to adopt the results, and who bears the responsibility.
This is not a rejection of the technology. It is a description of where the technology actually sits. The model is useful precisely because it can do the preparatory work at scale. It is risky precisely because its output looks finished. The workflow that works is the one that treats the output as a draft, not a verdict.
The emphasis on clearly defining the scope of what is entrusted is the operational version of the gap this article has been tracing. The model does not signal where its competence ends. The people using it have to.
The Clarity That Comes From the Report Itself
The most valuable thing about the Phi-4-reasoning technical report is not its benchmark table. It is the honesty of its caveats. The authors state plainly that benchmark results do not guarantee performance in real applications, that evaluation methodology needs to be more rigorous, that the model can make careful mistakes, and that safety behavior around chain-of-thought is an open research question. [1].
Those statements do not diminish the achievement. They locate it. A model that can reason through AIME problems at 14 billion parameters is a genuine milestone. A model that does so by imitating a larger teacher, evaluated on benchmarks whose variance the authors themselves flag, is a milestone with a shape and a boundary.
The gap between appearance and substance is not a flaw unique to this model. It is the condition of all language models that produce reasoning-shaped text. What makes Phi-4-reasoning worth studying is that its creators documented the gap instead of papering over it. The step-by-step answer is a performance — useful, often accurate, and not the same as understanding.