OpenAI has faced opprobrium in recent months from those who suggest it may be rushing too quickly and recklessly to develop more powerful artificial intelligence. The company appears intent on showing it takes AI safety seriously. Today it showcased research that it says could help researchers scrutinize AI models even as they become more capable and useful.
The new technique is one of several ideas related to AI safety that the company has touted in recent weeks. It involves having two AI models engage in a conversation that forces the more powerful one to be more transparent, or “legible,” with its reasoning so that humans can understand what it’s up to.
“This is core to the mission of building an [artificial general intelligence] that is both safe and beneficial,” Yining Chen, a researcher at OpenAI involved with the work, tells WIRED.
So far, the work has been tested on an AI model designed to solve simple math problems. The OpenAI researchers asked the AI model to explain its reasoning as it answered questions or solved problems. A second model is trained to detect whether the answers are correct or not, and the researchers found that having the two models engage in a back and forth encouraged the math-solving one to be more forthright and transparent with its reasoning.
OpenAI is publicly releasing a paper detailing the approach. “It’s part of the long-term safety research plan,” says Jan Hendrik Kirchner, another OpenAI researcher involved with the work. “We hope that other researchers can follow up, and maybe try other algorithms as well.”
Transparency and explainability are key concerns for AI researchers working to build more powerful systems. Large language models will sometimes offer up reasonable explanations for how they came to a conclusion, but a key concern is that future models may become more opaque or even deceptive in the explanations they provide—perhaps pursuing an undesirable goal while lying about it.
The research revealed today is part of a broader effort to understand how large language models that are at the core of programs like ChatGPT operate. It is one of a number of techniques that could help make more powerful AI models more transparent and therefore safer. OpenAI and other companies are exploring more mechanistic ways of peering inside the workings of large language models, too.
OpenAI has revealed more of its work on AI safety in recent weeks following criticism of its approach. In May, WIRED learned that a team of researchers dedicated to studying long-term AI risk had been disbanded. This came shortly after the departure of cofounder and key technical leader Ilya Sutskever, who was one of the board members who briefly ousted CEO Sam Altman last November.
OpenAI was founded on the promise that it would make AI both more transparent to scrutiny and safer. After the runaway success of ChatGPT and more intense competition from well-backed rivals, some people have accused the company of prioritizing splashy advances and market share over safety.
Daniel Kokotajlo, a researcher who left OpenAI and signed an open letter criticizing the company’s approach to AI safety, says the new work is important, but incremental, and that it does not change the fact that companies building the technology need more oversight. “The situation we are in remains unchanged,” he says. “Opaque, unaccountable, unregulated corporations racing each other to build artificial superintelligence, with basically no plan for how to control it.”
Another source with knowledge of OpenAI’s inner workings, who asked not to be named because they were not authorized to speak publicly, says that outside oversight of AI companies is also needed. “The question is whether they’re serious about the kinds of processes and governance mechanisms you need to prioritize societal benefit over profit,” the source says. “Not whether they let any of their researchers do some safety stuff.”
Source
