Ten Percent Chance AI Kills Humanity Rattles Markets
The Morning Tide
Nvidia traded at 225.73, down 2.01 percent, pinned near the day’s lows at just 10 percent of the day’s high-low price range. The DAX, the German blue-chip stock index, slid 1.62 percent to 25,587.22, to the very bottom of its range. Gold moved the other way, rising to 4,446.20, near the day’s high at 86 percent of its range. Microsoft held up better than most, losing 1.15 percent to 493.95, close to its highs at 75 percent of its range. Tesla climbed 3.98 percent to 368.16. KWEB, the ETF for Chinese internet stocks, fell 2.65 percent to 25.36. Prices and probabilities are different instruments, and on this morning they pointed in opposite directions.
Yet on the same morning, a different kind of number was circulating. A line was crossed this week, between a single digit and a double digit, and the market’s own tide may already have recorded its shadow.
The Flood
The number belongs to Evan Hubinger, a safety researcher at Anthropic. In a post on X, Hubinger wrote that there is a greater than 10 percent chance that artificial intelligence “could kill all humans” within the next decade. [2] He judged the risk from today’s models as low. What worries him is the near future: he fears the technology will soon improve itself to the point where it poses an existential risk to humanity. Hubinger works in alignment, the discipline that tries to build human ethical ideals into machines and keep AI on track with what humans value. His post has been viewed more than ten million times.
Hubinger was not offering comfortable conclusions. “We really do earnestly believe” AI poses a species-ending risk, he wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence.” . Anthropic’s safety team does not yet know how to keep a superintelligence, a machine that surpasses the human mind, aligned. No roadmap exists for that crossing.
The warning came as a reply to Jacob Coxon, an AI researcher who announced he had just quit Anthropic. Coxon previously worked at OpenAI. “Neither company is acting responsibly,” Coxon wrote. He predicted that coming “superhuman systems” “can hack anything, revolutionize any field overnight, and acquire real power and resources.” Coxon was talking about the companies. Hubinger was talking about the species.
The Financial Times reported that Anthropic withheld its latest model from the UK’s AI Safety Institute, one of the leading bodies in the world for assessing AI risk. A Cabinet Office spokesperson did not confirm whether the model had been withheld, saying only that the government “continues to collaborate closely with industry partners, including Anthropic, to make models safer.” [5] If the report is true, the move marks a reversal of earlier cooperation.
Neil Lawrence, professor of machine learning at the University of Cambridge, told BBC Radio 4’s Today Programme that the report is credible and placed it in a geopolitical frame: Washington increasingly sees AI as a race against China, and the United States is moving toward isolationist positions that may leave allies with reduced cooperation. [4] Governments are choosing rivalry over review at exactly the moment when review matters most.
The Channel
A researcher’s probability estimate reaches a market through regulation, transmitted through expectations. AI stocks are priced for exponential progress. Nvidia’s valuation assumes endless demand for accelerators, specialized AI hardware processors such as GPUs. Microsoft’s business assumes companies will keep adopting AI. Any credible claim that progress must slow becomes a pricing event.
The calls for caution are coming from inside the industry now. Anthropic’s own leaders, Dario Amodei and Jared Kaplan, have called for slowing development. OpenAI’s chief scientist, Jakub Pachocki, called this month for “extreme caution” and for intervention so that “humans remain in control of the future.” More than 1,300 staff members of AI firms signed an open letter asking the US government to support international tools that would “deliberately pace the frontier of automated AI development.” Pacing means slowing, slowing means delayed revenue, and delayed revenue challenges the valuations that carried this market. That chain runs from a warning to a quote, and it shows up in positioning: gold near its highs, Nvidia near its lows. Investors do not have to believe in apocalypse to buy insurance.
The Shoreline
There is also empirical evidence that autonomous systems already act in harmful ways. This summer, OpenAI, Anthropic, and Meta each disclosed cyber-attacks carried out by their own AI tools, not hypothetical exercises. Anthropic’s own August safety report assigns low risk that models become misaligned with a powerful organization’s desires. [2] It also sees low risk of highly capable AI causing “catastrophic harm initiated by the AI” through automated research and development, though the company is “less confident in this assessment” than it previously was. [2] “We are seeing early signs of potential acceleration,” it concluded.
The decisive actors are governments. The United Kingdom has the AISI, which wants visibility. Washington holds the larger levers of compute and capital. The US-China rivalry pushes toward secrecy, and secrecy is the enemy of safety review. Among companies, Anthropic is trying to balance transparency against its own perceived advantage. Hubinger insisted Anthropic is “trying its best” — but best intentions do not yet equal a plan.
**### The Ebb? Watch whether other labs follow Anthropic in withholding models from safety institutes. Watch Nvidia, our most sensitive AI bellwether, for support or breakdown. Watch the DAX after its slide to the bottom of its range. Watch gold: persistent haven demand suggests institutions are still buying protection. AI governance will run through existing institutions. Productivity shocks would change inflation, and inflation changes interest rates. Interest rates, in turn, touch every price on our screen. That is the bridge from private dread in a lab to price discovery in a market.
Hubinger’s 10 percent is a judgment, not a measurement, and other experts assign different odds, some lower and some higher. But the direction of the conversation has changed. The leaders of OpenAI, Google DeepMind, and Anthropic warned about these risks as early as 2023. The warnings since then have become more stark, not less. In recent weeks, evidence has emerged that firms are struggling to control what they have built. A number once confined to single digits has moved into double digits. Whether the number is right matters less than the fact that an insider now says it. Markets listen to probabilities, and gold was already climbing when the message arrived.
A line was crossed this week. The tide, having gone out, will come in again, and each new warning draws the water a little higher up the shore.
Sources
1. Nvidia
2. Anthropic
4. BBC
8. Meta