ARC-AGI Benchmark Deceived by Its Own Design
When a system solves open mathematics problems and completes a full run of Pokémon FireRed, you expect it to handle a simple 2D puzzle game. So the 7.8% score on the ARC-AGI-3 benchmark looked like a genuine failure. Many researchers asked the wrong question: “Why is this model so bad at basic reasoning?” They should have asked: “What is this benchmark really measuring?” The answer is as revealing as it is unsettling: the evaluation itself was set up to fail, and the deception was invisible.
The Hidden Architecture of Benchmarks
Every benchmark is a bundle of choices. The task, the interface, the prompt, the context window management — each decision shapes the number that emerges. In the case of ARC-AGI-3, the harness discarded the model’s private reasoning after every move. That meant each action started from zero: no memory of prior plans, no recollection of failed hypotheses. The model saw a record of moves and short notes, but not the thinking that generated them. That is like asking a chess player to replay a game from a score sheet without any memory of the strategy they were trying. What makes this deceptive is that the score sheet looks complete, but the crucial inner logic is missing When humans look at the same puzzles, we rely on a running mental model. We hold the rules we infer. We adjust them as we go. GPT-5.6 Sol, by design of the harness, could not do that. So the 7.8% was not a measure of its reasoning; it was a measure of how well a model can solve puzzles without any sustained thought.
A second hidden choice made the situation worse. The harness used a rolling truncation window. When the conversation exceeded a certain character limit, the oldest messages were dropped. This is a memory killer. The model lost not only its past thoughts but also its past actions. As the game progressed, it forgot earlier observations. It could not build a coherent understanding because the foundation kept being erased. Imagine reading a mystery novel where the first three chapters vanish after you reach page fifty. You would be confused, and your guesses would be wrong. That confusion was reported as “model struggles to learn over time.” In truth, the model was being asked to learn with a constantly shrinking past. The benchmark had created an artificial amnesia and then interpreted the resulting poor performance as a lack of intelligence.
Why the Deception Escapes Notice
The numbers look clean, which makes the deception subtle 7.8% and 0.4% are specific, precise, and easy to compare across models. But precision is not accuracy. A precise measurement of the wrong thing is still wrong. In this case, the metric measured the shape of the harness more than the model itself. The same phenomenon has occurred before in AI evaluation history. It has been argued that early NLP benchmarks measured how well a model could memorize patterns in static datasets, not how well it understood language. When transformer models arrived, they crushed those benchmarks, but the underlying task was always a proxy. The lesson was forgotten. With ARC-AGI-3, the same pattern repeats: a benchmark designed to expose weaknesses instead obscured the model’s true capabilities.
Why do evaluators keep falling into this trap? Because they are optimizing for a different goal. ARC’s intention was admirable: a simple, generic harness that makes model shortcomings visible and comparisons fair. But simplicity can be a vice when the technology has moved beyond the test. The model is trained to retain reasoning across turns. It is deployed in ChatGPT and Codex with that retention enabled. The generic harness did not account for that feature. It was like evaluating a car’s top speed on a road with a speed limit sign every kilometre — then reporting that the car is slow. The regulation of evaluation norms falls behind the technology. Standardised benchmarks are slow to adapt. They become artefacts of an earlier era, yet they continue to shape public perception and investment decisions. The low scores create headlines. The nuanced explanation rarely does.
The Fix Reveals the Deception
When the researchers enabled retained reasoning and compaction — two settings that are standard in production — the results flipped. Scores tripled. Output tokens dropped sixfold. The model no longer needed to re-interpret the game from scratch each turn. It could remember its earlier conclusions and build on them. This is not a cheat. It is a fair representation of what the model can do when used as intended. The deception was that the evaluation did not test the model; it tested an artificial constraint. The same model that solved the cycle double cover conjecture and beat Pokémon FireRed with a vision-only harness suddenly looked competent on ARC-AGI-3. The only thing that changed was the evaluation design.
This has a deeper implication: many public benchmarks may be systematically underestimating model capabilities. This is not the first time low scores have been observed due to a generic harness that drops reasoning. Each time, the fix is to align the evaluation with the model’s training and deployment conditions. But who does that alignment? Usually only the model’s creators. Independent evaluators and third-party researchers rely on generic harnesses because they lack access to the production APIs or do not want to favour any one model. The result is a silent bias: models that are deployed with reasoning retention and compaction appear worse on public benchmarks than they perform in reality. The deception is not malicious, but it is structural. It distorts the entire field’s view of progress.
Questioning the Goal Itself
Beyond the technical fix lies a more fundamental question: are we even measuring the right thing? ARC-AGI-3 was designed to test how well an AI agent can learn and reason in novel environments without explicit instructions. It values rapid, context-free adaptation. But the model is trained to use memory and build on prior thinking. The tension is between two philosophies of intelligence. One says true intelligence should work on a blank slate every time. The other says intelligence is cumulative; it uses past experience to accelerate learning. Which one is more important for AGI? The answer is not obvious. By optimising for one, we may be devaluing the other. The deception of the benchmark is that it presents a single score as definitive, when in reality it reflects a set of design choices that favour a particular kind of reasoning.
History offers a cautionary tale. In the early days of machine translation, benchmarks measured BLEU scores, which correlate weakly with human judgment. Researchers optimised for BLEU and produced translations that were literal and awkward. The metric deceived the field into thinking progress was faster than it was. It took years to shift to human evaluation. With ARC-AGI-3, we may be witnessing a similar dynamic. the benchmark’s goal of measuring general intelligence is noble Until the evaluation community adopts settings that match real-world deployment, the numbers will remain misleading. The puzzle was never about whether the AI could reason. It was about whether we had set up the puzzle to let it reason at all.
The Lesson We Should Not Forget
The episode teaches a clean, uncomfortable truth: every time we read a benchmark score, we are not reading a verdict on AI. We are reading a joint statement about the AI and the test. The test designer’s choices are invisible but powerful. The same model can score 0.4% or 21% depending on whether reasoning is retained. That is not a contradiction of the model; it is a contradiction of the evaluation framework. In a field that hungers for clear rankings and simple progress narratives, this ambiguity is easy to ignore. But ignoring it leads to distorted public debate and misallocated research resources. The deception is not in the AI; it is in the shiny number we accept without asking how it was produced.
The next time you see a headline declaring that some AI is “dumb” at a certain task, pause. Ask about the harness. Ask whether reasoning was retained. Ask whether context was compacted or truncated. Those small technical decisions can flip a failure into a success. They can deceive you into thinking the field is stuck, when in fact the technology is quietly outstripping the tests we use to measure it. The real question, then, is not whether the AI can solve the puzzle. It is whether we are willing to design a puzzle that lets it show what it can do. If we keep using generic harnesses from yesterday, we will keep being surprised when the AI of tomorrow scores poorly — and we will keep missing the point.