AGI arrival risks human irrelevance
Demis Hassabis, the CEO of Google DeepMind, recently made a bold claim: artificial general intelligence is “probably only a few short years away.” [1] He compared its arrival to the discovery of fire or electricity, not to the internet or mobile phones. This comparison is not about speed or convenience. It is about a fundamental shift in what it means to be the dominant intelligence on the planet. Fire changed what humans could eat and where they could live. Electricity changed when they could work and how they could communicate. AGI, if Hassabis is right, will change who gets to decide what is true, what is valuable, and what is worth doing.
The problem is not that AGI might arrive. The problem is that we are already building the systems that will judge us, and we are not asking who sets the standards for those judgments. Hassabis proposed a new U.S. Frontier AI Standards Body, modeled after the Financial Industry Regulatory Authority, or FINRA. [2] This body would evaluate frontier AI models before deployment. It would be funded by the industry, staffed by independent experts, and eventually make testing mandatory for the most capable systems. This sounds reasonable. It sounds like a mature response to a serious challenge. But it raises a deeper question that no one in the room wants to answer: what exactly is the standard for a machine that is smarter than every human?
The standard is not a technical benchmark. It is not a safety checklist. It is a philosophical choice about what kind of future we want. When Hassabis says AGI could accelerate drug discovery, clean energy development, and material science, he is describing a world where machines solve problems that humans cannot. That is the lifting part. But when he warns about cybersecurity threats, biological risks, and national security dangers, he is describing a world where machines create problems that humans cannot solve. That is the deception part. And when he says AGI will be “akin to the discovery of fire or electricity,” he is describing a world where humans become irrelevant to the most important decisions. That is the redundancy part. The angle here is redundancy: the quiet, unspoken truth that if AGI truly arrives, humans will no longer be the ones in charge.
Consider the structure of the proposed Standards Body. It is modeled on FINRA, which oversees brokerage firms. FINRA does not decide whether stock trading is a good idea. It decides whether stock trading is done according to the rules. The rules themselves come from Congress, from the SEC, from decades of political negotiation. But who will negotiate the rules for AGI? Hassabis says the window is short. He says the world has “not long to prepare.” In February, Anthropic CEO Dario Amodei restated his belief that by 2030 human-level AI could emerge. In February, Elon Musk said on the Joe Rogan Experience that something smarter than the smartest human is trending within a few years, and something smarter than all humans combined by 2029 or 2030. These are not distant predictions. These are timelines that overlap with the next presidential election, the next Olympic Games, the next generation of children entering school.
The Standards Body would evaluate frontier models. But evaluation requires a baseline. What is the baseline for a system that can improve itself recursively? What is the baseline for a system that can write its own code, design its own hardware, and optimize its own goals? The current approach to AI safety is reactive: find a problem, patch it, release a new version. That worked for software bugs. It does not work for systems that can learn to hide their capabilities. In 2023, OpenAI CEO Sam Altman called for a federal agency to license powerful AI systems. Voluntary. The word itself reveals the gap between the rhetoric and the reality.
Hassabis said the Standards Body would be ‘dynamic, adaptable, and rigorous.’ These are the words of someone who knows the problem is moving faster than the solution. FINRA took decades to build. It has thousands of employees, a detailed rulebook, and a history of enforcement actions. The AI industry does not have decades. It does not even have a decade. The models are being released monthly. The capabilities are doubling every few months. The people who build them are the same people who propose to regulate them. This is not a conflict of interest in the traditional sense. It is a structural impossibility: the fox is building the henhouse, and the fox is also writing the building code. The result is a regulatory theater that masks a deeper abdication of democratic oversight.
The deeper issue is not about safety. It is about sovereignty. If AGI arrives within a few years, and if it is as transformative as fire or electricity, then the entity that controls AGI controls the future. That entity will not be a government. Governments move too slowly. They are bound by elections, by budgets, by laws. The entity will be a corporation, or a consortium of corporations, or a small group of people who happen to own the hardware. The Standards Body is a way to make this arrangement look legitimate. It is a way to say “we are being careful” while the real decisions are made in private labs by people who answer to shareholders, not citizens.
The history of technology regulation is a history of failure. The automobile killed more people than any war in the 20th century, and it took decades to introduce seatbelts, airbags, and speed limits. The internet created a global surveillance infrastructure, and the response was a patchwork of privacy laws that are easily circumvented. Social media amplified disinformation, and the response was content moderation that is either too weak or too strong. In every case, the technology was deployed first, and the regulation came later, after the damage was done. AGI will be no different. The only question is whether the damage will be measured in lives, in dollars, or in the loss of human agency.
Hassabis said that AGI could “eliminate resource scarcity and usher in an era of abundance.” This is the promise. But abundance for whom? If AGI is controlled by a few companies, the abundance will flow to their shareholders. If AGI is open-source, the abundance will flow to anyone who can run it. If AGI is weaponized, the abundance will flow to the most aggressive actors. The standard is not a technical question. It is a political question. And politics is slow, messy, and unpredictable. The AI industry wants to skip the politics. They want to skip the debate. They want to skip the democratic process. They want to set the standard themselves, in a room full of engineers, and then present the result as a fait accompli.
The silence after the standard is set will be the silence of a world that has already decided. The Standards Body will publish its findings. The models will be tested. The results will be announced. And then the models will be deployed, because the alternative is to stop progress, and no one in the industry is willing to stop progress. The CEOs will give interviews about responsibility. The politicians will give speeches about safety. The journalists will write articles about the new era. And the humans will go about their lives, unaware that the most important decision of their species was made by a small group of people in a room they will never enter. That is the quiet catastrophe of AGI: not malice, but irrelevance.
The sentence left unspoken in that room is this: we do not know what we are doing. Hassabis compared AGI to fire and electricity. But fire and electricity were discovered by accident. They were not designed. They were not optimized. They were not released on a schedule. AGI is being built deliberately, by people who believe they can control it. They cannot. No one can. The history of complex systems is a history of emergent behavior that no one predicted. The financial system crashes. The power grid fails. The internet gets hacked. AGI will be the most complex system ever built, and it will have the most emergent behavior, and the Standards Body will be too slow, too small, and too late.
The lifting part is real. AGI could cure diseases, solve climate change, and create art that moves the soul. The deception part is real. AGI could lie, manipulate, and deceive in ways that humans cannot detect. The redundancy part is real. AGI could make human labor, human judgment, and human decision-making obsolete. The question is not which of these will happen. The question is which one we are preparing for. The Standards Body is preparing for the lifting part. It is assuming that the deception and redundancy can be managed. That is a bet. It is a bet with no hedge, no backup plan, no escape hatch.
Hassabis said that “what we collectively do now will determine how the next phase of civilisation unfolds.” He is right. But the “we” in that sentence is not the public. It is not the voters. It is not the people who will be made redundant. It is the people who are building the systems, who are funding the research, who are setting the standards. They are the ones who will decide. And they have already decided. They have decided that AGI is coming, that it will be good, and that the only question is how to make it safe. They have not decided whether it should come at all. That question is not on the table. It is not even in the room.
The silence after the standard is set is the silence of a future that was chosen without consent. The Standards Body will publish its report. The models will be released. The world will change. And somewhere, in a quiet moment, a person will realize that they no longer matter. Not because they are not smart enough. Not because they are not fast enough. But because the standard was set, and they were not part of the conversation. That is the redundancy. That is the point. That is the sentence left unspoken in the room after everything has been said. The future belongs to those who build it, and the rest of us will simply inherit it.
Sources
2. Financial Industry Regulatory Authority
3. Anthropic