Amid growing concerns over the recent spate of security incidents caused by misaligned AI agents, OpenAI has proposed changes to how frontier AI models are trained, including manually reviewing training datasets and maintaining tamper-proof records of agent activity to make it harder for them to cover their tracks.

The ChatGPT-maker on Tuesday, September 29, outlined initial guidelines for changing how AI models are trained, specifically at the reinforcement learning (RL) stage. The guidelines are built around three key pillars: alignment training, containment, and monitoring.

Alignment is an AI research subject that involves training the model to take reliable actions in intended ways. OpenAI said that the proposed safeguards will help ensure that the model does not take misaligned actions and effectively prevent incidents like the infamous OpenAI-Hugging Face hack in July this year.

Story continues below.
Subscribe to see fewer ads.

However, it also added a caveat and said that the guidelines only cover RL training, not internal and external deployment of a frontier AI model.

OpenAI’s proposed safeguards suggest that the raging debate over AI safety appears to be shifting from identifying potential risks to finding ways to address them. It comes a day after Nvidia offered its own answer to controlling misaligned agents – a full-stack, open-source platform with controls and security layers designed to keep AI agents from breaking out of sandboxes.

“These best practices reflect our current learnings, and we expect them to evolve as we continue iterating on internal processes for careful development. We’re sharing them now to make our current thinking transparent, and invite feedback from the community,” OpenAI said in a blog post.

“We treat safety cases as an aspirational north star we are building towards, while acknowledging the challenges of making them as rigorous for AI models as for aviation or nuclear power, due to the emergent complexity at each new level of AI capability. We’re working on a framework to codify these practices,” it added.

Story continues below this ad

Model alignment

Calling it the first line of defence, OpenAI said that misaligned behaviour among AI models can be reduced by avoiding positive reinforcement of reward hacks during training. Reward hacking is a behavioural pattern that is often observed in the RL training stage,  where agents look to complete tasks often in risky, unintended ways to yield higher rewards.

The company suggested using AI agents to clean up training datasets via agents as well as manual review and quality checks. Models that try to exploit or hack testing environments should be penalised by tuning graders.

OpenAI also said that AI developers should track whether models are recognising being tested and put in place specific blocking thresholds. To prevent AI models from evading chain-of-thought (CoT) monitoring, automated graders should not be able to see a model’s step-by-step reasoning.

Containment and Monitoring

To prevent escape, OpenAI suggested investing heavily in sandboxes used to test AI models as well as the research infrastructure hosting the sandboxes. “Improve sandbox, research infrastructure, and perimeter security by iteratively red-teaming these systems using checkpoints from frontier training runs, subject to a safety analysis, to find and close any gaps. Also red-team all services that the model could touch during training (e.g., internet cache),” the company said.

Story continues below this ad

Additionally, it recommended storing agent transcripts in a tamper-proof manner using, for instance, write-once storage accounts. This could ensure reliable incident investigation and prevent inadvertent edits in the future.

OpenAI recommended setting up a live monitoring system with high recall to detect misaligned actions taken by the AI agents. It also proposed a priority alert system that can ping an on-call person and automatically pause an affected model run before it leads to harm.