A correct answer is not a correct thought
A vision-language model presented with a matrix of shapes can select the right missing tile while its written reasoning describes a rule that does not exist. [1] That is the central discomfort in a 2026 study from Yildiz Technical University, which set out not to rank models but to catch them in the act of pretending. The authors built a 5,240-sample evaluation framework spanning BİLSEM, MM-IQ, Visual Puzzles and PuzzleVQA, and the finding that organizes everything else is deceptively simple: accuracy and reasoning come apart. A model can look competent on the answer sheet and be incoherent in the margin (arXiv).
This is the shape deception takes in machine learning — not a lie told, but a gap between what the output claims and what the process actually did. The study’s authors observed that a correct answer does not always mean the reasoning chain is valid, and they treated that observation as a measurement problem rather than an anecdote [1]. [1] The result is a paper about the distance between performance and understanding, and about how training can widen it.
What the benchmark actually asks
The four datasets were chosen for their spread of difficulty, and the human accuracy figures attached to them tell you why. PuzzleVQA sits at an average human success rate of 91 percent; Visual Puzzles at 75 percent; MM-IQ at 51 percent, which makes it the discriminating set. The BİLSEM material — 806 original Turkish questions drawn from preparation exams for gifted students — was added for linguistic and logical variety. Each source was split 80-20 for training and testing, and a unified 400-sample subset was drawn from the combined test pool for the benchmark runs (arXiv).
The datasets differ in another way that matters for the deception angle. PuzzleVQA ships with three-stage solution paths — a caption describing the visual elements, an explanation naming the underlying rule, a deduction applying it — and BİLSEM solutions were written by hand. MM-IQ and Visual Puzzles contain only questions and options, no reasoning at all. So the study had access to ground-truth rationales for some tasks and none for others, which is precisely the condition under which a model’s fluent-sounding explanation becomes hardest to audit.
The judge that reads the reasoning, not the answer
Because a binary correct/incorrect score hides everything the study wanted to see, the team adopted an LLM-as-a-Judge protocol, scoring generated rationales on three axes: logical consistency between steps and the visual input, conciseness and clarity, and alignment with the reference solution (arXiv). The choice of evaluator was itself iterative — initial trials with Cosmos Gemma 9B gave inconsistent results, and the team moved to the Gemini 2.5 Flash API for stability. Solution pairs were presented in A-B format without labels, so the judge would not know which model produced which, and a calibration pass on PuzzleVQA compared near-reference, incorrect, and identical solutions to check the evaluator’s behavior.
What the judge exposed is the paper’s most quotable structural finding: some models construct a fundamentally flawed reasoning chain while still arriving at the correct answer, while others establish the correct logic and then fail at the final selection. Accuracy, in other words, is a coin that lands heads for two entirely different reasons — and only one of them is skill. The study’s framing of this is blunt: evaluating only the final answer is insufficient for determining the validity of the model’s internal reasoning [1]
Training that makes the deception worse
The fine-tuning experiments began with a sensible baseline. Qwen3VL-8B and Gemma3-12B were trained on synthetic solutions generated by Gemini 3 Pro, using LoRA to keep costs down and reduce catastrophic forgetting. Qwen3VL-8B outperformed Gemma3-12B across every dataset mixture, and the best configuration — PuzzleVQA plus MM-IQ, the “3pro” mix — lifted general success from a 24.7 percent baseline to 48.8 percent, close to a doubling (arXiv). On the headline metric, supervised fine-tuning looks like a triumph.
Then the team added synthetic answers from Qwen3VL-Flash and the picture inverted. Standard SFT on the combined 3pro-plus-QwenFlash data dropped general success to 32.25 percent; QwenFlash alone fell to 30.50 percent. Two data-quality interventions followed — strict regex filtering to strip samples with wrong final answers, and a MultiSFT strategy that regenerates extra solution traces per problem before filtering. MultiSFT recovered the combined setting to 40.50 percent and the QwenFlash-only setting to 32.00 percent, beating regex-only preprocessing in both cases. Neither augmentation surpassed the original Gemini-only baseline.
The collapse, in numbers
The output-level diagnostics make the failure concrete. The SFT (3pro) model produced empty outputs on just 1.00 percent of the 400-sample test set, with an average output length of 215.3 words. Adding QwenFlash traces pushed empty outputs to 42.25 percent and the repetition rate to 52.00 percent. The QwenFlash-only model showed the same pathology: 42.25 percent empty outputs, 51.25 percent repetition. MultiSFT (3pro plus QwenFlash) clawed the empty-output rate back to 6.25 percent and repetition to 13.25 percent (arXiv).
A representative failure begins with a reasonable visual observation and then repeats the same partial rule indefinitely, never converging. Read against the deception angle, this is the inverse of the confident wrong answer: a model that has learned the surface form of reasoning — the cadence, the step markers, the hedging — and lost the ability to stop. Weaker fine-tuned models, the authors note, produced long but low-information explanations, cycling through weak hypotheses and sometimes never emitting a parseable final answer. Fluency, here, is not a signal of thought. It is a residue of it.
Why the baseline beats the student
The study’s most counterintuitive result is that baseline models often exhibited more stable reasoning than their fine-tuned counterparts. This lines up with the “SFT memorizes, RL generalizes” thesis the authors cite from prior comparative work by Chu et al. (2025), and it cuts against the instinct that more training on more examples must produce a better reasoner [1]. When the supervised traces are themselves unstable, the student inherits the instability along with the format. The paper’s own summary is that SFT is a double-edged sword: without high-quality, diverse reasoning traces, it can trigger collapse (arXiv).
What the gap costs
The study’s own remedy proposals are modest and worth stating plainly. More diverse synthetic supervision, sampling distinct reasoning paths from a broader array of models rather than one generator. A shift from SFT toward reinforcement learning with Group Relative Policy Optimization, rewarding both final-answer correctness and the structural integrity of the rationale. Domain-specific pretraining for Turkish-language tasks like BİLSEM. Scaling the benchmark to much larger models to test whether abstract reasoning is an emergent property of scale or requires architectural change in how vision and logic are joined (arXiv).
The closing line of the paper is the one that should travel: while multimodal models can “see,” true “thinking” requires moving beyond the imitation of patterns toward a verifiable, self-correcting logic [1]. That is a statement about a gap, and the gap is the story. A system that produces the right answer and the wrong reason is not half-right. It is a system whose apparent competence and actual process have come apart — and the training methods meant to close that distance can, under the wrong data, tear it wider.