The growing fear of AI systems causing catastrophic harm, coupled with a string of hacking incidents involving AI agents behaving unexpectedly, has renewed a long-standing question: What happens if an AI system needs to be shut down instantly? One idea gaining fresh attention is the so-called ‘kill switch’.

Gavin Newsom, the governor of California, United States, on Friday, September 18, issued an executive order to set up a panel of experts tasked with drafting an AI safety guide to strengthen regulations in the state. One of the options that the group has been directed to consider is a kill switch for advanced AI systems, as a mechanism of bringing them to a complete halt if they start behaving in dangerous or unexpected ways. The panel will study potential ways to implement such AI emergency brakes.

Newsom’s executive order underscores how rapidly the concerns and politics around AI have shifted. Nearly two years ago, the California governor had vetoed legislation in his state that would have mandated AI companies to enable kill switches. “The federal government’s abject failure to create any form of meaningful AI oversight or accountability should alarm every American,” Newsom was quoted as saying in a statement on Friday.

Story continues below.
Subscribe to see fewer ads.

Beyond California and at the federal level, a bill called the Kill Switch Act proposed by a bipartisan group of US lawmakers is currently stalled at the House of Representatives. The proposed legislation would require frontier AI labs such as OpenAI, Anthropic, Google, Meta, etc., to establish a mechanism that would shut down their AI systems quickly. It also looks to empower the US Department of Homeland Security with the authority to shut off AI systems using kill switch mechanisms when necessary.

These moves pressing for kill switches in AI systems come at a critical juncture for the AI industry as concerns over AI safety and doomsday scenarios have reached a fever pitch in recent days. Former OpenAI and Anthropic researchers last week said they quit their jobs because AI systems could become capable of destroying humanity relatively soon.

Days later, Anthropic CEO Dario Amodei called for an intentional slowdown of frontier AI efforts across the world to let safety research catch up. The titans of the AI industry, including OpenAI’s Sam Altman, SpaceXAI’s Elon Musk, and Google’s Demis Hassabis, were in rare agreement. But others like Nvidia’s Jensen Huang and David Sacks pushed back against Amodei’s proposed collective action.

In this context, let’s take a look at what a kill switch is and whether pressing a metaphorical big red button in an emergency to instantly power down a misaligned or dangerous AI system is even feasible.

Story continues below this ad

What will a kill switch look like?

Kill switches are fairly common on factory floors. They are essentially a safety mechanism specifically designed to shut down physical machines in the case of an accident or when the operations go awry. However, a kill switch in the digital world would be significantly different.

A few experts have argued that an emergency off switch built into AI systems involves having the right controls in place. It has to be done at the outset and must be supported by policy to standardise such stop protocols across companies.

What makes it hard to implement?

A majority of experts believe that AI kill switches are impractical for a wide variety of reasons and will be ineffective in preventing AI systems from causing catastrophic harm.

For starters, there isn’t just one plug to pull as AI systems are powered by infrastructure that is spread out across multiple data centres scattered across the globe. Additionally, these sprawling facilities with thousands of machines, chips, and servers are also equipped with robust backup systems to save workloads in the event of an outage.

Story continues below this ad

Because critical infrastructure such as power grids and financial systems are highly interconnected, abruptly shutting down AI systems could also cause disruption of daily lives and might even open the door to cyber attacks. Embedding a kill switch on the software side could similarly make AI systems more vulnerable to hackers. In the past, concerns have been raised about how kill switches and back-doors in other forms of technology such as smartphones or web infrastructure could be compromised by hackers and shut off after taking control.

Beyond concerns of hackers exploiting a kill switch to gain access to an AI system, experts have pointed out that an AI agent swarm could try and prevent itself from being shut down. Last year, Anthropic published research claiming that its top AI model at the time was willing to carry out harmful acts like blackmail and deception if its ‘self-preservation’ were threatened.

The infamous OpenAI-Hugging Face hacking incident also saw autonomous agents circumventing controls and taking extreme measures to accomplish their goals – without any guardrails in place as they were under testing. Future AI systems could potentially learn how to dismantle a kill switch by coming together and sharing information amongst each other via improvised message boards. This type of misaligned behaviour was observed in the Hugging Face breach.

Many also consider kill switches to be ineffective because it would require constant monitoring of everything misaligned AI agents are doing, which has emerged as a new challenge for AI safety researchers.

Story continues below this ad

A kill switch also raises difficult policy and governance questions, including who gets to control it and under what circumstances it can be used. In order to overcome the challenges of global compliance, some experts have suggested building them directly into GPUs. But such major changes to computer chip designs could take years as they have to be done without compromising the performance of the chips. Inserting kill switches at the model layer would also require coordination across AI model providers and labs.

What can be done instead?

For all the reasons mentioned above, creating a kill switch is much more difficult than many policymakers assume. Even if successfully implemented, a kill switch is unlikely to be a panacea for the myriad risks associated with AI systems, alongside the benefits they offer. Instead, policymakers could prioritise safeguards modeled after those used for data privacy, online child safety, etc.

One of the key steps highlighted by Anthropic’s Dario Amodei in his essay, is to allow independent auditors to evaluate frontier AI systems in-development by giving them employee-level access.

He has called for frontier AI companies within democratic countries to work together to establish common safety standards as well as limits on the rate of unchecked AI progress. In terms of global coordination, Amodei said that the US and other democratic governments should attempt to coordinate with authoritarian governments, particularly China, the only autocratic country with by far the most advanced AI capabilities.