The warning from leaders of the world’s top AI companies, that the technology could become powerful enough to pose a serious risk to humanity in as little as six months to a year, may sound like a scenario from dystopian fiction. It must, however, not be dismissed. In an essay published over the weekend, Anthropic co-founder and CEO Dario Amodei made a case for “pacing the frontier”. He was backed by rivals Sam Altman, CEO of OpenAI, and Elon Musk, founder and CEO of xAI. The dangers of letting companies, which are racing to develop a deeply transformative technology, set the limits have long been flagged. That the industry leaders have now spoken up and underlined the need to slow down, so that risk prevention can keep up, opens a precious window to develop more effective regulations for AI development.

This breathing space may allow AI companies to skirt around increasingly hostile positions on the technology’s environmental and economic impacts, but it must also be used by governments to set safety standards without strangling innovation. For example, Amodei and Altman’s voluntary commitment to grant employee-level system access to independent evaluators could be made mandatory. Also required is a system for international cooperation to limit the most dangerous uses of superintelligent AI, such as cyberwarfare, bioterrorism and economic disruption at a global scale. The challenge of achieving this in a time of heightened global rivalries is steep — US President Donald Trump has already dismissed the risks flagged by Amodei et al. as something that “won’t happen”, emphasising that the US must maintain its lead over China.

Advertisement

An internal reckoning has been forced by the development of recursive self-improvement — where an AI system uses its capabilities to design, develop and train its successors. Then there are the high-profile instances of the technology escaping the sandbox to act autonomously, such as when OpenAI agents hacked their way online and launched a coordinated attack on the open-source platform Hugging Face while attempting to cheat on an evaluation. But whether the recent concerns emerge from a belated sense of accountability or the instinct to avoid the liabilities arising from AI gone rogue is beside the point. The risk of a technology developing faster than it can even be understood is real, especially when it is, just as quickly, being integrated into systems that control and manage everything from banking and transport to healthcare and defence.