Most discussions about machine intelligence begin with the model. How many parameters, how much training data, how well it reasons, how fast it runs. That framing has become so habitual that it now functions as a kind of default assumption — that the model is the mind and everything else is plumbing. A controlled study on the ARC-AGI-3 benchmark asks a different question. [1]
The Scaffold Is the Skill
The paper, by Jens Lehmann and Sahar Vahdati, opens with a definition that sounds mundane until you consider its implications (Lehmann & Vahdati, arXiv:2610.30499). [1] A harness, they write, is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model’s behalf. Language models, the authors note, rarely act alone. The behavior of any agentic system therefore depends on both its harness and its underlying model.
That sentence carries more weight than it appears to. If behavior depends on two things, then improving the system can happen through two routes: better weights or better scaffolding. The second route has traditionally belonged to people. Someone inspects failures, redesigns representations, adds tools, tunes control flow. The paper states the consequence bluntly: if each novel task requires a human engineer to do this work, then what improves is not the system’s own capacity to adapt but the engineer’s understanding, transferred into the system by hand. The agent’s learning remains confined to a supplied mechanism whose improvement depends on the engineer.
This is the human role the study sets out to test. Not the researcher who trains the model, not the annotator who labels data, but the engineer who builds the wrapper — the person whose craft is judgment about what the model needs in order to act well. The experiment asks whether a frontier model can take on part of that engineering work itself, using its own execution experience to improve the software through which it acts.
A Benchmark Built to Resist Shortcuts
ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. The paper describes the setup in concrete terms: at each turn, the agent observes one or more 64-by-64 color grids, together with the current game state, progress across levels, and available actions. Depending on the game, the interface offers directional inputs, a game-specific input, coordinate clicks, or undo. The agent is not told what the visible objects represent, how these inputs affect them, or what completes a level.
The example the authors use is a game called FT09. The agent must infer that small patterned tiles specify which neighboring tiles must match or differ from a reference color. Clicking cycles tile colors, which need to be matched to target patterns. Later levels introduce controls that change several tiles together. None of this is explained. It must be discovered through interaction.
The benchmark’s main metric, Relative Human Action Efficiency, combines level completion with action efficiency relative to human baselines. Credit goes only to completed levels, and solutions that use more environment actions than hidden human baselines are discounted. Later levels receive greater weight. The measure is designed so that flailing around until something works earns little.
What makes the benchmark a revealing test of self-improvement is that effective designs are known to exist. Harnesses developed under human steering already reach 100 RHAE on the public games with GPT-5.6 Sol and Opus 5, and the paper cites further human-steered systems that report the same ceiling. By contrast, Sol scores 38.3 on the public set in a simple benchmark harness with retained reasoning and compaction enabled, and an off-the-shelf Claude Code baseline scores 42.83 with Opus 4.8/Fable 5. The ceiling is not hypothetical. Someone has already built the scaffolding that gets there. The question is whether a model can find it alone.
How the Experiment Strips Away the Human
The protocol is deliberately austere. Each experimental condition pairs a model with a reasoning-effort setting and uses that profile for both harness development and gameplay. During construction and revision, the model operates through OpenCode, a general-purpose coding harness that provides tools to inspect and edit files and run commands. That coding environment remains fixed throughout. During gameplay, the model is called through the generated task harness — the artifact being autonomously constructed and improved.
Each run receives a sandboxed workspace with Python, preinstalled scientific libraries, and shell tools. The repository includes documentation for interacting with the game, a client for calling the assigned model, and model and pricing specifications. The model must create a launch script that starts its solver on a specified game without human input. The solver architecture is otherwise unspecified. The harness controls how much work is performed by deterministic code and when the model is consulted.
There is a qualification gate, but it is not about quality. A candidate must run and interact with the game within fixed action and level limits, regardless of its score. If it fails, the same model receives its failure report and execution logs and may repair it for at most ten cycles. Task performance is measured later, in fixed-snapshot evaluations.
Then comes the part that removes the engineer entirely. The qualified initial system runs on five development games. Revision 1 receives a fixed bundle containing its own source, traces, scores, actions, and inference costs. It may change any generated file or preserve the parent. The same procedure produces revision 2. No human edits or natural-language guidance are introduced between stages.
Every frozen version is then evaluated on the five development games and twenty held-out public games. The held-out games provide no feedback for construction, revision, or selection. The study covers nine model-effort conditions across seven model families, with three independent constructions per condition — 27 lineages, 81 frozen snapshots, and 2,025 game evaluations. All 27 runs completed construction and both revisions.
What the Models Actually Built
The generated systems span markedly different interaction strategies. The paper lists them: compact direct model controllers coexist with heuristic-first systems, strategist-executor hybrids, persistent manuals, explicit world models, replay systems, deterministic planners, and inference-budget managers. Architectures also vary across replicates of the same condition.
This diversity matters for the question of whether a human role has been displaced. The harness engineer’s value has always rested partly on taste — knowing which representation to try, which tool to add, when to let code run and when to ask the model. The models produced all of these design families on their own, from a minimal repository and a written interface contract.
The revisions show the same pattern. Across all 27 lineages, the authors audited all 54 parent-to-descendant transitions and labeled each as Preserve, Extend, Reorient, or Replace. Strategy change was common. Twenty-two of 27 first revisions reoriented the harness, four extended it, and one primarily preserved or restored the incumbent. Second revisions included 12 reorientations, 12 extensions, and three preserves. No lineage replaced its harness family; revisions repeatedly reorganized memory, perception, batching, replay, and resource allocation.
In other words, the models did not merely tweak parameters. They restructured the software that determines their own capabilities, and they did so in ways that resemble what a competent engineer would attempt.
The Gap Between Doing and Improving
Here the study turns, and the turn is the point. Across the 27 lineages, harness performance ranged from zero to near-ceiling scores. Construction, in other words, is not the bottleneck. But two revisions raised mean held-out RHAE by only 2.80 points, with a 95 percent bootstrap interval of minus 1.44 to 7.35 — compatible with no average benefit or a useful positive gain.
Development gains overstate the transfer. Revision 2 raised mean development RHAE by 8.42 points but held-out RHAE by only 2.80. The mean adaptation gap — the difference between improvement on the five feedback-generating games and improvement on the twenty held-out games — is 5.62 points, with a 95 percent interval of 1.78 to 9.77. Observed strict-sign changes comprised 15 gains, 12 losses, and no ties.
The aggregate hides opposing effects at the level of individual games. Averaging across runs, held-out performance improved on 13 games and declined on seven. The largest mean gain was 20.44 RHAE points; the largest decline was 15.03. A model that improved on the games it could see sometimes got worse on the games it could not.
Cost tells a similar story. Across all 27 lineages with complete selected-game cost meters, nine revision-2 descendants improved both held-out RHAE and cost per game. [1] Six gained RHAE at higher cost, nine became cheaper while losing RHAE, and three worsened on both dimensions. Mean cost fell by $2.72 per game, but the median change was only minus $0.54. The lower average expenditure combines revisions that achieve more at lower cost and revisions that spend less while also solving less effectively.
A post hoc rule selecting the snapshot with the highest development RHAE yields a mean held-out gain of 4.53 points over the initial harness, with a 95 percent interval of 0.81 to 8.68 — compared with 2.80 for always deploying the final revision. Mean deployment cost under that rule is $9.15 rather than $7.57 per game. The rule remains to be evaluated prospectively.
What the Revisions Reveal About Judgment
The authors went further than scores. They performed an outcome-blind, reliability-audited analysis of observable capabilities in 54 revision episodes, connecting development feedback to code changes and relating those changes to performance on games withheld from revision. The codebook reads like a description of engineering judgment itself: does the updater inspect item-specific support before editing? Does it connect cited evidence to a mechanism and prediction? Does it use targeted inspection, stated alternatives, refutable expectations, and result-governed narrowing? Does it enact plans with eligible-feedback coverage and final-edit alignment? Does it establish executable scope, fallbacks, baseline checks, comparison, narrowing, rollback, and no-change decisions?
The findings suggest that these behaviors appear in the revisions but do not reliably produce improvement. Construction strength does not ensure consistent improvement. In 36 episodes, the updater links the final result to its change and hands off. Yet the link between evidence and outcome remains loose.
The paper’s own framing is careful here. It separates four capabilities: constructing an executable harness, repairing interface and lifecycle failures, mapping execution evidence to a concrete update, and selecting updates that improve feedback-held-out tasks. The first three do not imply the fourth. A capable updater must also preserve working behavior and abstain when evidence is weak.
That last clause describes a disposition, not a technique. Knowing when not to change something — when the evidence is too thin, when a working behavior should be protected — is exactly the judgment that experienced engineers exercise and that automated systems notoriously lack. The study finds that models can generate the change but not reliably judge which change to keep.
The Human Ceiling That Remains
The paper is explicit that human-steered systems demonstrate further headroom, although their development and selection procedures differ. This is not a small caveat. It means the benchmark’s ceiling has been reached by people building harnesses, and the autonomous route has not closed the distance.
The comparison is not entirely like-for-like, and the authors say so. Human-steered harnesses may use different development procedures, different selection criteria, more time, more compute. But the direction of the gap is clear: the engineering work that produced 100 RHAE involved human decisions about representation, memory, and control flow that the models did not reproduce from their own execution evidence.
There is a broader context here that the study does not need to invoke but that sharpens its significance. ARC-AGI-3 has become a proving ground where the harness, not the model, is increasingly understood as the decisive variable. Human-steered harnesses reach the ceiling; off-the-shelf baselines sit far below it. The difference between 38.3 and 100 is not, in my reading, a difference in intelligence. It is a difference in scaffolding — and scaffolding has, until now, been a human craft.
Why the Distinction Matters Beyond the Benchmark
The study’s contribution is not that models cannot write harnesses. They clearly can, and the diversity of what they build is striking. The contribution is that the capacity to build an agent and the capacity to improve it must be evaluated separately. A system that can construct a working harness from a minimal repository has demonstrated something real. A system that can then revise that harness in ways that generalize to unseen games has demonstrated something else — something the study finds only weakly present under its two-revision protocol.
This distinction has practical consequences for how the displacement of engineering judgment will actually unfold. If construction and improvement were the same skill, the harness engineer’s role would already be obsolete. The models build. But the role also involves knowing which build to trust, which revision to deploy, when to stop. The study’s post hoc selection rule — picking the snapshot with the highest development score — improved held-out performance over always deploying the final revision, but it remains to be evaluated prospectively. Even the selection step is not settled.
The paper’s conclusion states the gap directly: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities. [1]. That sentence describes a division of labor that is technically solved but socially unresolved — an interpretation, not a claim the paper itself makes. The software can be written by the model. The judgment about whether the software is better remains, for now, a human question — and the study shows that the models, left to their own execution evidence, do not reliably answer it.
The Engineer’s Residual
What remains for the human in this picture is not the writing of code. It is the setting of the criteria by which code is judged, the decision about which games count as evidence, the choice of when a revision has earned deployment. The study’s protocol fixes all of these in advance — five development games, twenty held-out games, two revisions, a fixed coding bootstrap — precisely because leaving them to the model would confound the measurement. The human judgment is not eliminated from the experiment; it is moved upstream, into the design of the test.
That relocation is the real finding. The harness engineer’s craft has not vanished. It has been compressed into the protocol: the choice of benchmark, the split between development and held-out games, the metric that weights completion against efficiency, the decision to measure cost alongside performance. These are judgments about what matters, and they determine what the models can learn. The models can now execute the engineering work. They cannot yet decide what the work is for.
The study’s 27 lineages, 81 snapshots, and 2,025 evaluations amount to a careful demonstration that the boundary has moved but not disappeared. [1]. The model writes the harness. The human decides what counts as a harness worth keeping. Until the second capacity is automated — and the evidence here suggests it is not — the engineer’s role is not superfluous. It is upstream, smaller, and harder to see — arguably the most precarious position of all.