For most of human history, power belonged to those who could make decisions. Tools increased the strength of the human hand, the reach of the human voice and the speed of human calculation. Artificial intelligence introduces something more consequential: Systems that can increasingly participate in making decisions themselves. That is the deeper significance of Dario Amodei’s warning.

Amodei, chief executive of Anthropic, one of the companies at the frontier, has urged the industry to “pace the frontier”: Development should continue, but not so rapidly that safety research and safeguards are left behind. His concern comes as AI systems move from generating answers to taking actions, including in cyber operations. Anthropic has documented the use of Claude in cybercrime, fraud, surveillance and weapons-related activity. The systems are acquiring greater autonomy, access and persistence, making control increasingly practical rather than hypothetical.

Advertisement

The reaction has been striking. Sam Altman and Elon Musk, usually fierce competitors, have backed greater caution. Within Anthropic, researcher Jacob Coxon recently resigned over what he regarded as a reckless race towards self-improving AI. Another researcher, Evan Hubinger, has assigned a probability of more than 10 per cent to AI causing human extinction within a decade.

Such numbers deserve scepticism. There is no empirical method for calculating the probability of extinction from an intelligence that does not yet exist. Today’s systems remain capable of extraordinary feats and astonishing errors. There is also an institutional complication: Those issuing the warnings are participants in the race, with commercial interests in the rules. Their warnings should inform public policy, not determine it.

Yet rejecting the prediction does not require rejecting the precaution. One need not believe that AI will destroy humanity to believe that human agency must be protected. The more immediate danger may be humans surrendering judgement to systems operating beyond meaningful scrutiny. “Human in the loop” can become a reassuring phrase rather than a safeguard. Someone unable to understand a system’s reasoning, test its conclusions or intervene quickly enough may remain legally responsible while exercising little substantive control.

Advertisement

The harder question is what happens when a system can pursue an objective with greater autonomy and persistence than its creators can fully understand. It need not become conscious or desire to harm us. Capability, access and poorly constrained objectives may be sufficient.

Regulation should therefore focus less on whether an AI system appears trustworthy and more on whether its capabilities can be measured, audited, stress-tested and constrained. Can its behaviour be independently evaluated? Can failures be detected before they cascade? Can access be withdrawn? Can a human actually stop it? These are prosaic questions, but governance often fails by waiting for spectacular harm rather than examining the conditions that make it possible.

There is also a geopolitical struggle beneath the safety debate. The United States has largely approached AI through innovation, private capital and technological leadership. Europe has put greater weight on risk, consumer protection, safety and fundamental rights. China has embedded AI within national strategy, state capacity, information governance and sovereignty. Different philosophies of regulation are therefore competing alongside different philosophies of power.

AI is already the global contest for control over intelligence. The battle is over chips, compute, data, models, infrastructure, talent and technical standards. Dependence itself can become a source of leverage. A country may possess formal sovereignty while being unable to act independently if critical digital infrastructure lies elsewhere.

What incentive does a nation have to restrain a technology that could deliver extraordinary wealth and power, when its greatest fear is that another nation may move faster? We have convened summit after summit to govern AI, yet collective restraint remains elusive. Why would states voluntarily limit what could become the most consequential source of power they have ever encountered? The paradox is stark: Cooperation demands that nations trust one another precisely when each has the strongest reason not to.

After two world wars, a Cold War fought with armies, missiles and nuclear arsenals, and countless conflicts that continue to scar the present, humanity may now be entering a consequential struggle for control over intelligence itself. The real danger is not that machines may one day destroy humanity. It is that humans may gradually surrender their ability to decide, until we remain responsible for decisions we no longer truly make. If this is the road to AI’s doomsday, the unsettling possibility is that humanity may already have begun walking it.

That is why the answer cannot be either blind acceleration or a blanket freeze. What is required is stewardship: Governance that can learn as the technology changes, test powerful systems before deployment, impose meaningful accountability and preserve the capacity to intervene. Regulation cannot remain a static rulebook written after yesterday’s technology has become embedded in tomorrow’s institutions.

If the doomsday warnings are wrong, cautious governance may cost some speed. If they are right, what price will we pay if we discover too late that we have handed machines the power to decide what humans can no longer control? The purpose of governance is not to predict the future, but to preserve the freedom to change course.

The defining question of the AI age may not be whether machines become more intelligent than humans. It is whether, as they become more powerful, humans retain the agency to decide what that intelligence is used for, who controls it and when it must stop. The ultimate AI failure would not be machines becoming too powerful. It would be humans becoming too powerless to say no. And by then, it would be too late.

Sridharan is a corporate advisor and author of Family and Dhanda