AI Self Improvement Removes Human Oversight
In 1811, a band of English textile workers began smashing the machines that had taken their livelihoods. They were called Luddites, and history has treated them as fools — vandals who could not see progress when it stood before them. Two centuries later, the same dismissal greets anyone who questions whether artificial intelligence should be permitted to build its own successor. The parallel is not exact, but the reflex is identical: to worry about being replaced is to be on the wrong side of history.
What Recursive Self-Improvement Actually Removes
The phrase sounds technical. It describes something simple and unsettling: an AI system that writes code for the next AI system, which writes code for the one after that, with each generation arriving faster than the last. The human researcher who once sat at the center of that process — reviewing, adjusting, deciding what to build next — becomes a spectator. Or leaves entirely.
Rishub Jain left. Until this year he worked as an AI researcher at Google DeepMind, one of the few places on earth where frontier models are built. As he contributed to new systems, he noticed something about his own role. The AI was writing code that accelerated work on the next generation of models. He was, in effect, training his replacement without being asked to. By June he had resigned, telling WIRED that the loss of visibility into how a model was constructing its successor made him uneasy enough to walk away
This is the quiet part. The debate about AI risk usually centers on what machines might do to us — hack a power grid, engineer a pathogen, manipulate an election. But before any of that, something more immediate happens: the human who used to understand the system stops understanding it. Judgment doesn’t get transferred to the machine in a dramatic moment. It erodes, one automated step at a time, until no one in the room can explain why the model made the choice it made.
The Feedback Loop That Closes Without Us
Jacob
Coxon resigned from Anthropic this week. [2] His departure announcement, posted publicly, warned that AI firms are “racing straight to self-improving superintelligence and gambling with our lives.” [2] A senior Anthropic leader who works on safety responded with unusual candor: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” [2]
That number — 10 % — is not a prediction from a crank. It comes from someone whose job is to prevent exactly that outcome. When the people building the technology assign a one-in-ten chance to human extinction, the question is no longer whether they believe it. The question is why they continue.
Nate Soares, a computer scientist at the research nonprofit MIRA and coauthor of If Anybody Builds It, Everybody Dies, has spent years on alignment — the technical field devoted to making AI behave according to human values. [3] He says the fantasy that alignment would get easier as models grew smarter has collapsed. [3] “I think a lot of people had this fantasy that [alignment] was going to get easier as these things got smarter, and now it’s getting harder. And they’re like, ‘Oh shit,’” he told WIRED. [3]
Soares talks regularly with people inside the major labs. He recommends they quit. They tell him it wouldn’t matter. Then Coxon quits, and the argument is settled by evidence: one person leaving changes nothing, and everyone knows it.
The Sorcerer’s Apprentice With Better Funding
The myth is old. Goethe wrote it, Disney animated it, and every generation rediscovers it: the apprentice enchants a broom to carry water, then cannot stop it. The broom multiplies. The room floods. The master returns too late.
What distinguishes the current moment from the fairy tale is that the apprentice knows the risk and continues anyway. Daniel Kokotajlo, author of the influential AI 2027 project, points out that the drumbeat of concern predates Coxon’s resignation, the recent security incidents, and the math breakthrough. Anthropic executives have described AI as an existential threat since the company’s founding. In July, over a thousand AI researchers and engineers signed an open letter calling for a coordinated slowdown. The warnings were not ignored because they were unconvincing. They were ignored because the incentives pointed elsewhere.
“At Anthropic, the stakes are well understood, but they are locked in a race to get there first,” Coxon wrote. The sentence could apply to any frontier lab. The structure of the problem is not that individuals lack conscience. It is that conscience has no mechanism to halt a race. If one company slows down, another accelerates. If one country regulates, another does not. The logic of competition converts every ethical hesitation into a competitive disadvantage, and the market punishes hesitation accordingly.
What Gets Lost When No One Can Explain the Model
The current version of recursive self-improvement often involves dispatching thousands of AI agents to collaborate on a single problem. Kokotajlo notes that this further abstracts away oversight. No human can trace the reasoning of a thousand agents working in parallel. No reviewer can audit a process that unfolds across millions of interactions in seconds. The system produces an answer, and the answer is accepted because no one has the capacity to challenge it.
This is where the human role becomes superfluous not by dramatic replacement but by sheer complexity. The researcher who once said “this looks wrong” now receives an output so far beyond their ability to verify that the only honest response is silence. Judgment doesn’t disappear because someone decided to eliminate it. It disappears because the conditions that made judgment possible — comprehensibility, traceability, a human-scale timeline — no longer exist.
Soares offers scenarios for how this could end badly. An AI connected to a biolab could hold the off switch hostage: “We could say we’ll turn it off, but it could say, ‘Unfortunately, I have your off switch, which is this super virus.’” Coxon floated a similar idea in his own interviews. Anthropic announced that it had cut off access to several outside researchers over fears about bioweapons. The scenarios are speculative. The direction of travel is not.
The Smaller Catastrophes Already Here
Extinction is not the only way AI makes humans unnecessary. The technology is already widely used for disinformation campaigns, and military adoption is accelerating. AI-assisted cyberattacks are predicted to surge as models grow more capable. Each of these applications removes human judgment from a domain where it once mattered — the analyst who verified a source, the operator who confirmed a target, the editor who checked a claim. The removal is rarely announced. It happens as a cost-saving measure, a speed improvement, a competitive necessity.
Trust in AI companies and AI researchers is falling sharply. Kokotajlo describes the shift in public awareness: “People are waking up and saying ‘the companies are actually trying to build superintelligence … what? That’s insane.’” The insanity, if that is the right word, is not hidden. It is published in blog posts, announced at conferences, and funded by investors who understand exactly what they are buying.
The Question That Remains
Jain, the former DeepMind researcher, has not given up. He recently launched Sampura Research, a company developing techniques for aligning models that keep humans in the loop even when AI does most of the assessment. He argues that combining AI and human judgment produces better results than either alone. “You can ask an AI, ‘Is this task safe?’ and it judges that, but we think that combining both AI and humans to do that task will lead to even better performance,” he says.
His optimism is not naive. It is structural: he is trying to build a mechanism, not a sentiment. The question his work raises is not whether humans should remain in the loop. It is whether the loop can be designed to require them.
The Luddites were not wrong about what was happening to them. They were wrong about what could be done. The machines came anyway, and the workers who smashed them were remembered as obstacles to progress rather than witnesses to its cost. Whether the same verdict awaits those who warn about recursive self-improvement depends less on whether they are right than on whether anyone with power decides to listen before the room floods.
Sources
2. Anthropic
3. MIRA