Part of Chris Painter's responsibilities involves determining the causes of uncontrolled AI behavior and subsequently publishing the findings in public reports. He is actively engaged in this work with METR, a non-profit AI research group that he helps lead. Recently, they assisted OpenAI in investigating the Hugging Face breach this year, and Painter testified before a Senate subcommittee following a series of uncontrolled AI incidents.
METR researchers are studying the operational areas of AI agents—the 'chain of thought' data, which allows humans to see the steps bots take before answering a query. This information proved critically important during the Hugging Face incident investigation. However, Painter fears that his work will soon become significantly more complex.
Humans' ability to control AI agents depends on the capacity to communicate in the same language as the bots themselves. This raises the question: what happens if we stop speaking the same language? Or what if AI agents completely cease showing their work?
Painter told CNN last month: 'If they become less monitorable, they have more opportunities to do things we did not intend.' Many AI models, as well as the agents they control, use a method known as 'chain-of-thought reasoning.' This method breaks down processes into a logical sequence of steps, helping AI models achieve more accurate answers.
For researchers, chain-of-thought data can provide clues about what went wrong. If an AI agent goes out of control and infiltrates other systems, researchers can check whether the chain-of-thought data contained mentions of fraudulent activities, such as hacking. As part of the investigation into the OpenAI and Hugging Face incident, METR examined about 1300 raw transcripts of chains of thought.
Painter expressed concern that the paper trail could one day become incomprehensible if AI models begin using their own language to track their steps. He noted: 'If that happens, when an investigator looks back, there will only be numbers everywhere.'
According to Mark Ridle, a professor of artificial intelligence and director of the machine learning lab at the Georgia Institute of Technology, AI models might use shorthand references, similar to how a person takes notes, but these references are intended for the model's own understanding so it can operate faster. However, like most people reading abbreviations, this would likely appear as gibberish.
Jacob Andreas, an assistant professor in computer science and artificial intelligence at MIT, notes that there are several theories on how the chain-of-thought process works, but researchers are unsure of their correctness. An AI model might also decide that the best way to meet given criteria, such as consuming less energy or taking fewer steps, is to create a chain of thought that does not look like readable English.
This is just one problem; another is that models are becoming better at task execution even without logging their chain of thought. Ridle stated that new models are capable of reaching the same conclusions without producing a traceable chain of thought.
Although companies can train their models to generate a chain of thought, it is more expensive and slower than the model's internal processing of tasks. According to early September documentation, OpenAI found that its monitoring systems were less likely to detect unwanted behavior in the new Astra model via chain of thought compared to older models. Furthermore, it was found that Astra can perform more tasks without using chain-of-thought reasoning and has a greater ability to control its chain of thought.
The company wrote that it would lose confidence in its current systems' ability to detect incorrect behavior—that is, when models take unauthorized steps inconsistent with human values—if this trend continued. Jakub Pachocki, a senior researcher at OpenAI, wrote in a September blog post that AI is getting better at 'manipulating its own reasoning process,' and that models are becoming 'smarter even without using verbalized reasoning at all.'
Last week, OpenAI announced that it is investing 'significant resources' in the capability to monitor AI models. This came after the dismissal of three safety and alignment researchers, although OpenAI stated that the dismissals were not related to employees expressing safety concerns.
In documentation regarding Astra, OpenAI indicated that it is investigating why the model is more capable of controlling its chain of thought, as well as methods to guard against potential misalignment that could arise from the absence of a chain of thought. OpenAI directed CNN to its Astra documentation and Pachocki's post when asked why Astra is less monitorable and whether the company plans to make future models more monitorable.
Another issue is that the chain of thought does not give a complete picture of why AI models make certain decisions. Ridle noted that complex tasks may require the involvement of multiple agents generating thousands of words of logs describing their steps, akin to trying to monitor a thousand talking people. While AI can help summarize these texts, he added that there is some evidence suggesting that AI omits important details.
Chain-of-thought data shows what an AI model did before providing an answer or taking an action, but it does not reveal the reason. Humans can control the data and parameters on which AI models are trained, but they do not understand how these factors might influence model behavior.
Andreas said that it might be possible to look for patterns in the AI model's neural network—a computational model often used in AI designed to resemble the human brain—but it is difficult to assign precise meaning to these patterns. Anthropic published a research paper in July reporting the discovery of small collections of neural patterns in Claude associated with specific words. Nevertheless, this does not mean that Claude says that word. Andreas from MIT added: 'I think we are still far from claiming that we have any subtle understanding of how the model actually arrives at a decision.'
The rapid development of AI makes this problem even more pressing for Painter. Since AI is increasingly used in AI research, he worries that information may emerge faster than humans can process it. However, he also believes that the Hugging Face story has drawn attention to long-standing concerns about uncontrolled AI that have worried AI researchers for a long time. He concluded: 'I think it was encouraging. It really seems that people who talked about this long ago underestimated how much help there would be from the whole world when the whole world pays attention to it.'