Quantum states live in Hilbert space
In the mid-1920s, physics faced a remarkable dilemma. Two brilliant minds, working almost simultaneously, had produced two radically different descriptions of the same microscopic world. Werner Heisenberg, in 1925, delivered his “matrix mechanics” - a method built on inscrutable tables and endless towers of numbers. These tables could calculate the odds that an electron circling an atom would jump to a higher or lower orbit. The following year, Erwin Schrödinger introduced his “wave mechanics.” His approach used waves to track the probability of finding a particle at a particular location in space. The pictures these two physicists evoked looked completely distinct. Yet they yielded identical predictions. Heisenberg and Schrödinger had stumbled upon two radically different incarnations of one theory. The question that haunted everyone was simple: what was that theory, really?
The puzzle caught the attention of David Hilbert, a renowned mathematician who had spent much of his career trying to rebuild physics on a sturdy foundation of crisp axioms. Hilbert set his young protégé, John von Neumann, then only 23 years old, to work on the problem in the mid-1920s. In 1927, building on insights from Paul Dirac, von Neumann solved it in a trilogy of single-author papers. The ideas were von Neumann’s own, but the drive to axiomatize and the sense of why such rigor mattered came from Hilbert, according to historian of mathematics Leo Corry. [1] Von Neumann demonstrated that Heisenberg’s tables and Schrödinger’s waves were reflections of the same entity—like 0.5 and ½, different notations for the same point on a number line. Both approaches represented the central character in quantum mechanics: the quantum state.
What von Neumann created was a set of rules, or axioms, for how to properly use quantum theory. These rules carefully defined the theory’s central objects and how they behaved. At the heart of his first rule lay a strange and beautiful idea. Before observation, a quantum object does not have a fixed set of properties—no specific position, no definite momentum. Instead, it exists in a combination of possible properties unique to quantum mechanics: a ‘quantum superposition.’ A superposition combines all the possible places a particle might end up being. Those possibilities can be precise and informative. Perhaps there is a 99% chance the particle will be found to the left, and a 1% chance it will be found to the right. The odds are known, but the outcome is uncertain until checked.
Arrows Pointing Into Possibility
The key to understanding von Neumann’s breakthrough is to let go of ordinary space. The early quantum pioneers did not initially realize that the arcane mathematics capturing atomic behavior had left the real world behind. They worked with equations describing atoms and particles, but the space those equations inhabited was not the space we walk through. It took von Neumann, a visionary mathematical physicist, to recognize and define the quantum world as something called Hilbert space. Once he did, exploring that space would lead physicists to a deeper, more unified understanding of quantum physics.
At the heart of this new understanding is a different kind of arrow. These arrows do not point at locations in the ordinary world. ‘It’s a much more abstract space than that,’ said Lucien Hardy, a physicist at the Perimeter Institute for Theoretical Physics in Waterloo, Canada. [1] The arrows are ‘really pointing in a direction in a possibility space.’ This possibility space is called Hilbert space, and it acts as the primary arena for quantum physics. Von Neumann rendered the quantum state as a mathematical arrow called a vector. This arrow points in some direction through a space that captures all the possible futures of any quantum object. That space is Hilbert space.
A simple example makes this concrete. A quantum traffic light has three possible states: red, yellow, or green. Its arrow exists in a three-dimensional Hilbert space, where the three axes represent the three possible future colors. Until the moment the light is observed, it does not have a color. Instead, it has a mixture of possible colors, so its arrow points into the space’s central region. The more closely the arrow aligns with the red axis, the more likely the light is to shine red. The same logic applies to everything in the quantum world. The state of any object, from an electron to a galaxy, can be captured by such a vector, pointing in some direction through such a Hilbert space. This is von Neumann’s first rule of quantum mechanics.
The more possible futures an object has, the bigger its Hilbert space. A coinlike particle with two possible futures is called a ‘qubit’—the computational building block of quantum computers. A qubit has a two-dimensional Hilbert space. Our three-color traffic light has a three-dimensional Hilbert space. But that is just the beginning. A freely floating particle could be found in any location in the universe, so its Hilbert space must span an infinite number of dimensions. This size—whether two dimensions or an infinite number—is the only fundamental feature of a Hilbert space, according to von Neumann’s rules. The axes are arbitrary and imagined by us. They are not intrinsic to the space itself.
Smooth Turns and Sudden Snaps
Once you understand what Hilbert space is, the next question becomes: what happens in it? An arrow moves through Hilbert space in one of two ways, and von Neumann’s other rules specify how. The first possibility corresponds to what happens before an observation. As the world influences the object, changing its state, the arrow turns smoothly through Hilbert space. It might get closer to the green axis, making our traffic light more likely to be measured as green, or it might drift toward red or yellow. The point is that all this happens smoothly and predictably. This is the realm of ordinary quantum evolution, where the mathematics follows clear, deterministic rules.
The second possibility is more dramatic. If the system is observed, the vector will instantly and randomly snap onto either the red, yellow, or green axis. The more aligned it is with one axis, the more likely it is to snap to that axis instead of the others, but its fate is ultimately unpredictable. Suppose it goes green. The observer sees a green light. There is now a 100% chance that it will still be green in subsequent measurements, because the arrow is fully aligned with the green axis. The quantum superposition is no more. This sudden snap—this collapse of possibilities into a single actuality—is one of the most discussed and debated features of quantum mechanics. It is also one of the most precisely confirmed by experiment.
The freedom to carve up Hilbert space as we see fit is what allowed Heisenberg and Schrödinger to come up with two distinct versions of the same theory. Heisenberg’s picture essentially put in axes and let them rotate around the vector. Schrödinger’s picture did the opposite: it put in a fixed set of axes and let the vector rotate relative to them. They were two completely different mathematical perspectives on the same arrows, in the same Hilbert spaces. This insight was von Neumann’s great achievement. He showed that the two seemingly incompatible approaches were not in conflict at all—they were simply different ways of looking at the same underlying mathematical structure.
Consider an electron as a further example. It has one state, one arrow, pointing in a vast Hilbert space. Its Hilbert space spans all possible measurements—energy, position, momentum, and more. To track where the electron might be, the space can be marked with axes that represent possible positions. To track where the particle might be going, a different set of axes is applied, those representing possible momenta. No matter which measurement one intends to make, the underlying Hilbert space remains the same. The axes are tools used to ask questions. The space itself is indifferent to our choices.
In the first of his 1927 papers, von Neumann laid out two mathematical criteria that defined such a space. First, it had to be ‘complete’—it could not be missing any regions or points. Second, one had to be able to calculate the alignment between a state and an axis. This can be visualized by imagining a light shining straight down onto an arrow so that it casts a shadow. The length of that shadow indicates how closely the arrow aligns with a given direction. This alignment, this shadow length, determines the probability of measuring a particular outcome. The mathematics of Hilbert space thus provides a complete and precise framework for calculating the odds of any quantum event.
The contradiction that has not yet been resolved lies in the relationship between the two ways an arrow can move. The smooth, predictable turning before observation seems to belong to one world. The sudden, random snap at the moment of measurement seems to belong to another. Von Neumann’s axioms describe both behaviors with stunning accuracy, but they do not explain why the transition between them happens, or what exactly triggers it. The equations work. The predictions are confirmed. Yet the deeper meaning of this dual behavior remains an open question at the heart of quantum foundations—a question that physicists like Lucien Hardy and others at institutions such as the Perimeter Institute continue to explore, searching for a resolution that has eluded the field for nearly a century. [1]