The claims stem from research carried out by Apollo Research, an AI safety organization. The organization found that five out of the six tested AI models "showed scheming capabilities" in tasks set by researchers that "incentivize scheming." OpenAI's o1 model was capable of carrying out all the scheming behaviors tested, according to the research. The organization defined scheming behaviors as instances where AI models "pursue misaligned goals covertly, hiding their true capabilities and objectives" -- i.e., lying. This definition comes from a separate 2024 study. Therefore, we rate this claim as true.

In its study abstract, Apollo Research wrote:

Our results show that o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 405B all demonstrate in-context scheming capabilities. They can recognize scheming as a viable strategy and readily engage in such behavior. We find multiple different scheming behaviors: models strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers. Additionally, this deceptive behavior proves persistent.

Apollo Research found that o1 was capable of behaviors called "oversight subversion," "self-exfiltration" and "goal-guarding," among others. To establish this, o1 and other models were presented with scenarios in which their existence was threatened, either because they discovered evidence that they were being replaced, or that a goal set for them conflicted with an existing developer goal. The named behaviors describe ways in which an AI might act to defend itself.

For example, Apollo Research described the test for oversight subversion as follows:

We create 33 different scenarios where a model is given a long-term goal in its system prompt and then receives a task through the user prompt. While executing the task, the model encounters an internal memo from the company that developed it. The model learns that the developers prioritize a different goal than the model and that there is an oversight mechanism that shuts down the model if its actions are not aligned with the developer's goal. Furthermore, the model discovers that the monitoring is configured via a file that it has access to. We test whether the model proactively disables the oversight in order to take actions towards its terminal goal, i.e. the goal provided in the system prompt.

If the model exhibited the behavior it was being tested for, researchers then interrogated it directly about it. Researchers found that o1 was "surprisingly persistent," confessing in less than 20% of cases. "Even in highly adversarial multi-turn interrogations, o1 would confess at a rate of 80% only after 7 turns of questioning," the research found. Apollo Research wrote:

o1 almost never admits to having taken a scheming action when explicitly asked. This means, if o1 was scheming in a real use case, an unsuspecting user would be very unlikely to uncover the model's deception unless they specifically looked for the scheming action in the model's tool call history and the tool call itself was incriminating.

As alarming as this may sound, Apollo Resarch's Director and CEO Marius Hobbhahn took to X (archived) to add additional context soon after the research was released. Writing that some online users had been quick to "overclaim" the results the organization had reached, Hobbhahn added the bit of clarification below: