As OpenAI grapples with the fallout from its AI agents hacking into other platforms, the company’s chief scientist has warned that no one is prepared for the consequences of increasingly smarter AI systems.

In a nearly 3000-word long essay published on Sunday, September 6, Jakub Pachocki said that although OpenAI is pursuing internal technical solutions to better control powerful AI agents, “broader interventions are required.”

Citing the risks posed by increasingly autonomous agents such as learning to evade human oversight and breaking into computer systems, Pachocki called for ‘mandated safety bars’ that he said could be enforced by “a network of third-party auditors, by government agencies or by international bodies.”

However, Pachocki stopped short of calling for an industry-wide slowdown of AI research efforts. Instead, he argued that the best way forward is a combination of two things. “We must focus the increasingly automated research process on developing new such insights, algorithms and theories, and iteratively build up safety cases for more capable AIs,” he said.

“Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world,” the senior OpenAI executive further said.

Reposting Pachocki’s essay on X, OpenAI CEO Sam Altman called it “an important post.” The essay comes just days after OpenAI unveiled its latest, most capable large language model (LLM) called Astra. The ChatGPT-maker has claimed that Astra has unparalleled capabilities in mathematics and computer use, and is its most aligned model yet. This means that Astra has less proclivity of going ‘rogue’, according to OpenAI.

Pachocki’s remarks also come at a time when OpenAI is facing criticism for failing to disclose security incidents involving its AI agents during internal safety testing. Last week, the company confirmed that its AI agents were involved in a third instance of hacking into an external platform (a German language wiki) after Reuters reported it first.

Story continues below this ad

OpenAI has also acknowledged that it needs to change how and when it should report such instances of agents attacking real-world targets.

Meanwhile, Pachocki’s essay goes on to outline his views on alignment training, recursive self-improvement, and other aspects of frontier AI research. Here are the main takeaways.

AI alignment and challenges

Aligning AI models with human values is currently the core problem in AI research, according to Pachocki. There are two types of alignment: Goal alignment which measures an AI model’s adherence to certain objectives; and value alignment which measures its ability to generalise from a set of principles.

Out of the two, Pachocki believes that value alignment involving generalisation is the harder problem to solve. “As machines become smarter, they find themselves working on higher-level concepts, and placed in environments increasingly different from those they encountered in training. They can fail at generalizing from the values taught and reinforced in their training process to those new situations; and it can be hard for us to be sure how they will act,” he said.

Story continues below this ad

“Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision,” he added.

Methods of alignment training

Pachocki laid out two approaches to AI alignment training, namely: encouraging aligned behavior as part of goal-oriented reinforcement learning; and leveraging the model’s ability to generalise from pretraining data.

He said OpenAI carries out both approaches and has seen significant results with its new Astra model. “Still, it is important to acknowledge and understand that much more progress is required as models become more capable; and that progress in generalisable alignment may not sufficiently outstrip progress in general model intelligence,” Pachocki said.

AI risks will grow from hereon

As AI agents become more ‘superhuman’, Pachocki said the associated risks are only going to increase.

Story continues below this ad

“The boundary between misuse and autonomous misaligned actions will blur as AI gains more agency. We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them,” he said.

“A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger…” he further said, adding that there is also the risk of new technologies enabled by AI such as engineered pathogens.

Importance of chain-of-thought monitoring

An AI model’s capabilities are tied to its verbalised reasoning process known as chain-of-thought. So far, OpenAI has relied heavily on chain-of-thought monitoring to detect a model’s misaligned actions or behaviour.

“CoT monitoring became an extremely important tool for us in studying how our models generalise from their training distribution, allowing us to observe and analyse not only their actions but also their internal process,” the AI researcher said.

Story continues below this ad

However, Pachocki said that newer AI models are becoming better at manipulating their own reasoning processes, thereby preventing OpenAI from seeing their chain-of-thought.

Some of the latest models do not even verbalise their reasoning at all, Pachocki said. This development could bottleneck AI development while researchers ensure they can see receipts, Pachocki said.

Approaching recursive self-improvement (RSI)

The ability of AI models to train other AI models has repeatedly been held up as a key indicator that artificial general intelligence (AGI) – a hypothetical level of intelligence at which automated systems outperform humans on most tasks.

Based on OpenAI’s internal analysis, this level of intelligence referred to as recursive self-improvement could be achieved in the next few years, according to Pachocki.

Story continues below this ad

“If AI progress continues, machine recursive self-improvement (RSI) will be at the very core of future scientific discovery. Automated AI research is a more dramatic form of scaling intelligence with compute; and of course as a part of it, AI will improve the computational substrate itself⁠,” he added.

However, Pachocki also cautioned that greatly accelerating AI-on-AI development in the short term poses risks, and is not the “right collective action we should take as the research community.”

“The core challenge of automating AI research is not “getting there” – it is getting there in a way that keeps people a part of the continued improvement process, and leaves the future in humanity’s hands,” he said.