Somewhere in a server farm, a software system is rehearsing your kitchen. It has learned the height of your counter from a photograph, the way light falls through a window at four in the afternoon, and the awkward geometry of a drawer that never quite closes. None of this was planned as a gift to robotics. It arrived as a byproduct of a game you played a decade ago, and it is now becoming one of the most practical tools in artificial intelligence.
The Accidental Archive
In 2016, millions of people walked their neighborhoods at dusk, phones raised, hunting creatures that only existed on screens. Pokémon Go turned ordinary streets into a shared augmented playground. What players did not calculate was that every scan of a park bench or a storefront fed a growing visual record of the physical world. That record now has a second life.
Niantic, the creator of the game, sold it last year and pivoted hard into one of Silicon Valley’s buzziest AI topics right now: world models. [1] Brian McClendon, chief technology officer of Niantic Spatial, frames the mission plainly: the work is about reconstructing the real world. [1] The trove of environmental data gathered during years of play is being repurposed to build digital replicas. Unlike language models, which traffic in text, these systems ingest visual input and produce environments that simulate how objects move and collide.
The gain here is not abstract. A robot that has never entered a building can rehearse inside a faithful copy of it first. McClendon describes a workflow in which someone photographs the space where a robot is meant to operate, generates a three-dimensional model from those images, and then runs millions of simulated passes through it. [1] The machine learns the room’s dimensions, its obstacles, its quirks — all before its wheels or legs touch the actual floor.
Practice Without Consequence
The value of that rehearsal is easiest to see where mistakes are expensive. Oliver Cameron, the CEO of Odyssey, an AI lab focused on world models, draws the distinction between the two families of model with a simple comparison. A language model receives a sentence and predicts the next word. A world model receives an image of the world and predicts the next visual state of it. The output is a simulation, but one that looks and feels real enough to train against.
For driverless cars, the stakes sharpen further. Jamie Shotton, chief scientist at Wayve, describes the requirement in physical terms: a world model has to react to the actions taken in the world, which means it must grasp physics, cause and effect, and how other objects and people alter their behavior in response. Wayve’s generative model draws partly on dashcam footage, and the company uses it to rebuild hazardous roads and unsafe conditions that would be impractical to stage in real life.
Here the improvement is concrete rather than promotional. A vehicle’s software can be tested against a reconstruction of a flooded underpass or a chaotic intersection without putting anyone on the road. The simulation becomes a rehearsal space for situations too dangerous to schedule and too rare to encounter often enough in ordinary driving.
The Human Cost of a Synthetic World
There is a less comfortable side to this progress, and it sits in the labor market rather than the laboratory. To simulate the physical world, the companies building these systems need recordings of it. Earlier this year, a Wired reporter strapped an iPhone to his forehead and recorded hours of video in his apartment, filming his hands folding laundry, washing dishes, and pouring a Diet Coke. He sold the footage through an app for a modest sum. As more firms chase better world models, more of this piecework is likely to appear.
The arrangement is worth naming precisely. The same technology that promises to spare robots from costly real-world errors is generating a new category of gig work, in which ordinary people record ordinary domestic tasks so that machines can learn them. As more companies attempt to build better world models, even more gig work may demand this kind of data collection.
Why Simulation Beats Instruction
The mechanism behind all of this explains why the approach works where earlier methods struggled. Teaching a machine through explicit rules is brittle; the world contains too many exceptions. Teaching it through demonstration is slow and expensive. Simulation offers a third path: generate a plausible environment, let the system act inside it, and let it fail repeatedly at no physical cost.
That is why the field has attracted serious money and serious people. Yann LeCun, Meta’s former chief AI scientist, recently raised over a billion dollars for a startup aimed at building AI that understands the physical world. Fei-Fei Li, the Stanford computer scientist, launched World Labs in 2024, and AMD acquired the company this year for over eight billion dollars. Tencent has invested in its own world models, and Google continues to publicize improvements to its versions.
The commercial logic is straightforward once the mechanism is clear. Whoever builds the most faithful simulator holds the training ground for the next generation of machines, whether those machines drive, walk, or render entire game worlds. Physics and three-dimensional rendering are already core to video games, which makes the overlap between entertainment and robotics less strange than it first appears.
What the Machines Still Miss
The enthusiasm has limits that the technology’s own architecture imposes. A world model is only as good as the world it has seen. Reconstructions built from dashcam footage reflect the roads those cameras traveled. Simulations assembled from phone videos of apartments reflect the apartments of the people who recorded them. The system’s understanding of cause and effect is learned from observation, not derived from first principles, which means it can be confidently wrong in situations that fall outside its training.
Shotton’s formulation captures the difficulty without overselling the solution. Reacting to actions, understanding physics, tracking how people respond to a vehicle’s movement — these are demanding requirements, and meeting them in simulation is not the same as meeting them on a wet street at night. The gap between a convincing replica and a reliable one is where the real engineering lives.
The Comparison That Matters
Set the two trajectories side by side and the difference becomes visible. In one, the visual data people generate is harvested through games and apps and converted into training grounds that benefit the companies assembling them. In the other, the same data could be gathered with clearer consent and fairer payment for the people whose homes and habits made the simulation possible.
The technology itself does not decide which path is taken. What it does offer is a genuine advance: machines that can learn the shape of a room, the physics of a collision, the sequence of a dangerous maneuver, all without breaking anything real. That is a meaningful gain, and it deserves to be described accurately rather than inflated. The question that remains is how much of our physical life these systems will ingest, and who decides the terms.
