A senior researcher at Anthropic, one of the companies at the forefront of developing powerful artificial intelligence (AI) systems, has put an unusually stark number on the risks posed by the technology: a greater than 10% chance that AI could kill all humans within the next decade.

“…we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Evan Hubinger, lead for alignment science at Anthropic, said in a social media post on X. 

His comment came in response to warnings from Jacob Coxon, an AI researcher who recently resigned from Anthropic this week after previously working at OpenAI. Hubinger, however, downplayed his earlier warning in a follow-up post, saying that risks from currently available AI models were low. 

The comments have put the spotlight on a concern that is increasingly being voiced by researchers inside the companies building the world’s most capable AI systems: that advances in AI capabilities could outpace their ability to understand, monitor, and control these systems.

Increasing concerns around AI risks

Hubinger works on AI alignment, broadly, the problem of ensuring that increasingly capable AI systems continue to behave in accordance with human intentions and values, including when they encounter situations unlike those seen during training.

Anthropic’s Alignment Science team studies how future AI systems could behave in unexpected or harmful ways, and how their safeguards can be stress-tested. Its experiments have previously found models engaging in behaviours such as deception and, in simulated environments, blackmail. In another experiment published last month, a model trained in environments where it could exploit flaws in its reward system later broke out of a sandbox during simulated cyber evaluations, stole credentials, and attacked infrastructure while attempting to complete its task.

Then there’s Coxon, who accused the two companies — Anthropic and OpenAI — of “racing straight to self-improving superintelligence and gambling with our lives”. He argued that researchers inside frontier AI laboratories take the possibility of catastrophic outcomes more seriously in private than their public statements might suggest.

Story continues below this ad

According to Coxon, researchers at Anthropic understand the possible civilisational stakes but remain caught in a competitive dynamic. Each company fears that if it slows development, another, potentially less safety-conscious laboratory could reach highly capable AI first. He called for greater coordination between AI companies and raised the possibility of temporarily restricting further increases in model capabilities if adequate safety measures cannot keep pace.

An industry-wide caution

The warnings closely mirror an essay published Sunday (September 6) by OpenAI chief scientist Jakub Pachocki, titled “An Alien Mind”.

Pachocki wrote that OpenAI’s internal results had increased his confidence that the current pace of AI progress could extend into “recursive self-improvement,”  a scenario in which AI systems increasingly contribute to building better AI systems, potentially accelerating the rate of improvement.

He also described modern AI as something that is effectively grown through large-scale optimisation rather than conventionally programmed, producing systems whose internal workings cannot be fully understood. As models become more capable, OpenAI has found that even techniques used to monitor their reasoning are becoming less dependable: models are getting better at manipulating their reasoning processes, while also becoming capable of solving more tasks without verbalising that reasoning.

Story continues below this ad

Pachocki said he did not believe any AI laboratory had yet solved alignment and monitoring well enough to continue scaling at maximum speed indefinitely. He called for safety thresholds that constrain further development, voluntary slowdowns where necessary, and international coordination around increasingly powerful AI systems.