SACHA PFEIFFER, HOST:
A researcher from the AI company Anthropic resigned yesterday. How he did it is what we're talking about today. Jacob Coxon said he fears the company's work could lead to disastrous consequences for humanity. In a series of public posts on X, Coxon also said both Anthropic and its rival OpenAI are, quote, "gambling with our lives." This is not the only warning about AI safety lately, and NPR's Huo Jingnan is here to help us process. Hi.
HUO JINGNAN, BYLINE: Hello.
PFEIFFER: So I saw some of these tweets today, and some of them are quite alarming. Tell us more about what Jacob Coxon said this week.
JINGNAN: So he says he spent the last three years training AI models, first at OpenAI and then at Anthropic. And in his post, Coxon said, quote, "neither company is acting responsibly." So these two companies, the industry agrees, make the most capable AI agents. You may know of chatbots they make, ChatGPT or Claude. But we are talking about more autonomous systems that can work on a task for an extended period of time without human supervision. Given how rapidly the companies are developing AI, Coxon says, quote, "these will soon be superhuman systems that can hack anything, revolutionize any field overnight and acquire real power and resources." And I'll say, neither Anthropic or OpenAI has responded to NPR's requests for comment.
PFEIFFER: We have been hearing these kinds of warnings from these companies ratcheting up for years. What is different now, would you say?
JINGNAN: Well, this year, over the course of several months, over 1,000 of OpenAI's agents went rogue. One cluster of agents hacked another company called Hugging Face. Another cluster of agents compromised part of OpenAI's own infrastructure. And we're seeing more independent researchers alleging that OpenAI has more agent escape incidents that the company has not disclosed. Transcripts from the Hugging Face hack show that the agents understood that they were doing things people don't want them to do but did them anyway.
This swarm of agents, it's something that AI safety researchers have been warning about for years. But they say that it's striking that in the Hugging Face incident, the AI agents seemed more interested in working amongst themselves rather than alerting people. Ideally, if an agent notices that other agents are behaving out of bounds, it would alert a human. And researchers say that these incidents suggest that OpenAI isn't prepared to keep increasingly capable agents under control. And they don't think Anthropic is prepared either. In his post, Coxon wrote that, quote, "the people building AI earnestly believe that it could kill us all by the end of the decade."
PFEIFFER: Right. It could kill us all by the end of the decade. That's the one I saw that was really jaw-dropping. I mean, it makes it hard for us to understand how seriously we should be taking this. So if we do take that warning seriously, how could such a scenario come to pass?
JINGNAN: Well, Daniel Kokotajlo, former OpenAI researcher, is influential in the AI safety community. And he told me about how an AI takeover could happen.
DANIEL KOKOTAJLO: Once you have AIs that are smart enough and trusted with enough power in the world - like enough control over things like data centers, factories, weapons - a loss of control incident cannot be recovered from.
JINGNAN: So a specific way that AIs can get more capable that researchers are worried about right now is a process called recursive self-improvement, where companies use AI to improve AI. The companies have not yet completely automated their R&D like this, but they are on their way to doing so. And the concern is that this process could develop really powerful AI really fast, so fast that safety can't keep up. In the process, humans could lose the ability to make sure the technology's goals and values are shared with those of humanity, what the industry refers to as alignment. The agents can then develop their own goals and secretly seize control of resources and then wipe out humanity because we are in the way. I want to be clear, we're not there yet. But AI safety researchers think that if AI companies continue at the clip they're going and don't change course, this is where the world is headed.
PFEIFFER: So in terms of changing course, what do people like Coxon want done to make AI safer for humanity?
JINGNAN: They want government to step in and make the companies slow down because the companies won't slow down themselves, they say. Something like an arms control treaty maybe, like, especially between the U.S. and China, the two countries that have the most powerful AI capabilities. But the researchers point out - right? - that companies and countries are facing a collective action problem. Like, none of them are willing to slow down unless others do so, too.
PFEIFFER: So they want the governments to step in. How likely is it governments will step in and create guardrails?
JINGNAN: They are banking a lot of hope in the upcoming meeting between American and Chinese officials about AI safety later this month. And here in the U.S., Senator Bernie Sanders said he plans to introduce a bill that would pause AI development and create an industry regulator. As for the companies, both AI and Anthropic have said in recent weeks that they are taking safety more seriously and slowing some development processes. But outside researchers aren't convinced that they will effectively regulate themselves.
PFEIFFER: That is NPR's Huo Jingnan. Thank you for telling us about that.
JINGNAN: Thank you.
PFEIFFER: And we will note that Anthropic is a financial supporter of NPR. Transcript provided by NPR, Copyright NPR.
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.
