Machines Learning Depth Erases Human Proof of Spatial Understanding
The moment a model stops being a tool and starts being a colleague is rarely announced. It happens quietly, in the weights of a neural network, in the way a system trained to answer questions gets retrained to take actions. A vision-language model that once described what it saw now reaches for what it can do. And somewhere in that transition, something disappears — not the model’s ability, but our ability to tell what it still knows.
The new study from Alexander Hackett and colleagues, published in Science, takes a weight-matched pair of models and asks a disarmingly simple question: after a vision-language model is post-trained to become a robot-controlling vision-language-action model, does it still understand depth? [1] The researchers’ findings, detailed in their paper, reveal a stark divergence between task performance and internal spatial representation The answer is not just “worse.” The answer is a story about how capability and understanding can diverge, and about how the very act of making a model useful for a task can strip away the evidence that it ever grasped the world at all.
The Floor and the Cliff
Consider what happens when you probe a neural network layer by layer. In the base vision-language model, depth decodability — the ability to extract spatial distance information from the model’s internal representations — improves steadily as you move toward the final layers. The model is building a coherent picture of three-dimensional space, layer by layer, like a draftsman adding perspective lines to a sketch. The information is there, increasingly accessible, increasingly refined.
The vision-language-action model tells a different story. From the very first layer, it decodes depth worse than its base counterpart. The researchers call this persistent gap “the floor” — a baseline deficit that never closes, no matter how deep you go. [1] This deficit is present from the outset, suggesting the post-training process immediately rewires the model’s representational priorities But the more striking finding is what happens at the end. Where the base model’s depth decodability peaks in its terminal layers, the action-trained model’s collapses. They call this “the cliff,” and it is not a gentle slope. It is a drop-off, a sudden loss of spatial information precisely where the model should be most prepared to act on it.
This is not a story about a model failing at a task. It is a story about a model succeeding at a different task, and in doing so, discarding something its predecessor carried. The robot that this model controls can still navigate, still grasp, still manipulate objects. But the internal representation that once encoded “how far away is that cup” has been overwritten, repurposed, or simply abandoned.
Where the Judgment Goes
The deeper question is what this means for the humans who once performed this kind of spatial reasoning themselves. For most of human history, judging distance was a survival skill — reading the depth of a ravine, the reach of a branch, the gap between two moving objects. We outsourced that judgment to machines gradually: first to cameras and rangefinders, then to software that could estimate depth from images, and now to models that fold depth perception into a larger action policy without ever making it explicit.
The cliff suggests something uncomfortable. When a model is trained to act, it does not need to know depth in the way a human knows depth. It needs to know enough to reach, to avoid, to place. The model is not explaining the world to anyone; it is just moving through it. And if no one is asking it to explain, the explanation quietly disappears.
This is the pattern that should worry us, not because robots will take our jobs, but because they will take our questions. A human worker who estimates depth on a loading dock can be asked: how did you know that gap was too wide? The answer involves experience, attention, maybe a lifetime of near-misses. A vision-language-action model that performs the same task cannot answer that question, not because it is secretive, but because the information is simply no longer there in a form anyone can extract.
The Mechanism of Disappearance
The researchers do not stop at describing the cliff; they locate its cause. Through careful ablation studies, detailed in their paper, they show that the late-layer collapse comes from interference in the MLP writes — the parts of the network that transform and integrate information. [1] This mechanistic finding is crucial, as it moves the discussion from observation to causation When they ablate these late-layer MLP writes in the action-trained model, most of the terminal depth decodability returns. The same intervention in the base model produces no comparable recovery. The cliff is not a general degradation; it is a specific overwriting.
This matters because it tells us that the loss is not inevitable. It is a design choice, encoded in the architecture and the training procedure. The action post-training process prioritizes motor commands and task completion over spatial representation, and the model obliges by reallocating its computational resources. Depth becomes implicit, distributed, unrecoverable — not because the model cannot represent it, but because the training objective no longer rewards representing it in a decodable way.
The institutional dimension here is hard to ignore. This is not a failure mode that a single engineer would notice, because it does not show up in task performance. The robot still works. The model still acts. The cliff is only visible to someone probing the internals with a specific question — someone like Hackett, who had the curiosity and the resources to ask whether the model still knew something it was no longer being asked to demonstrate. How many other capabilities are silently dropped in the same way, across the thousands of models being post-trained for deployment right now? The study underscores a growing need for transparency in model development, where internal state audits become as routine as performance benchmarks?
The Open Question
The study leaves us with a question that reassesses everything before it. If we cannot tell, from the outside, whether a model has lost a capability that it never needed to display, then how do we know what any deployed system still understands? The floor and the cliff are not just properties of this one model pair; they are symptoms of a broader epistemic problem. We are building systems that act on the world, and we are losing the ability to verify what they know about it.
For the human whose judgment was once the bottleneck — the quality inspector, the crane operator, the surgical assistant — this is not a story about being replaced by a better machine. It is a story about being replaced by a machine that does not need to justify itself. The human could be asked, could explain, could be held accountable. The model cannot, not because it is unwilling, but because the knowledge that would allow it to explain has been optimized away.
The cliff is not a bug. It is the shape of a trade we are making without quite agreeing to its terms: in exchange for models that can act, we are accepting models that cannot account for what they know. And the people who once provided that account — whose judgment was the interface between perception and action — are not being replaced by better judgment. They are being replaced by a system that has made judgment itself unnecessary, and in doing so, has made it invisible.
Sources
1. Science