Two Lenses on the Same Low Score
Two research groups at Yale University approach the same puzzle from opposite ends. John Sous asks whether the exams are broken. Logan Wright asks whether the laboratory can be rebuilt so a machine can walk through it. Same institution, same concern about artificial intelligence in physics, two entirely different repair manuals.
The starting point is a number that looks damning. Cutting-edge AI models scored as low as 30 per cent on six different tests designed specifically to assess their ability on tough physics problems. [1] A 30 per cent score reads like failure. It reads like a machine that can chat about physics but cannot do physics. That reading, according to Sous, is wrong.
He and his colleagues went through the benchmark problems one by one. They found a litany of errors and ambiguous definitions. Once corrected, scores soared to as high as 94 per cent. [1] A bad exam tells you about the exam. Sous puts the conclusion bluntly: anything that is well defined, even if it is very hard, will likely be handled very well by AI — arguably better than the best experts. [1] He says his data backs this up. Well defined is the load-bearing phrase, and everything that follows hangs from it.
Where Physics Refuses to Be Pinned Down
Well defined is precisely what much of physics is not. The physicist Vladimir Arnold once said that mathematics is the part of physics where experiments are cheap. It marks a border. On one side sit problems you can check without touching nature. On the other side sit problems where nature has the final vote, and nature does not accept formal proofs.
Sous draws the line himself. Interesting physics, he says, comes when we find something confusing — some phenomenon we observe in nature, a material or the cosmos that we do not understand. The physicist’s job is then to build a model, an approximation or a simulation that captures and predicts that phenomenon. He calls this a very artistic, a very creative process. The open question he poses is whether AI can take the next step and do the full job of a theoretical physicist: define, create, explore, test. His own answer is that the picture is not clear yet.
The reason mathematics sits at the tip of the spear is a technique called formalisation. It turns a mathematical idea into computer code that can be methodically checked. The process is so thorough that if a theorem comes through intact, it is considered correct beyond all reasonable doubt. It lets AI process a problem and know when it has arrived at a correct solution. Physics has seen some advances in using formalisation to test its claims, but it is early days in comparison. Physics problems are often messier and harder to pin down.
Rebuilding the Laboratory Around Code
Wright’s answer is not to fix the exam but to rebuild the room. His principle is simple: create experiments where the inputs and outputs can be computer controlled. Then hand that control to an AI that can pull the appropriate levers, explore the problem space and arrive at meaningful results.
His description of the machine is deliberately unglamorous. It has no hands and no eyes. It interacts with the world through language and code. So if an experiment can be set up where all observations reduce to code and all actions reduce to code, the AI becomes genuinely useful.
In Wright’s laboratory, the AI model Claude controls laser pulses aimed at a rubidium atom, proposes what output a given pulse will produce, and then tests that proposal.
Wright is explicit about the limits. Not all of the universe can be explored this way.
Mathematics already has its flood of machine-made proofs and its argument about what the flood is worth. Physics has a corrected benchmark showing the machines were better than the tests suggested, a formalisation toolkit that works only where problems can be pinned down, and one laboratory where a language model spends the night turning a rubidium atom into data. Wright expects the next frontier to be the constrained domains where code can capture every observation and every action.