Senior officials at some of the nation’s leading AI companies say it’s time for the federal government to find a way to put the brakes on AI development if it’s needed.
“We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development,” reads an open letter signed by more than 1,100 AI workers from companies including OpenAI, Anthropic, Google, and Meta.
The signatories include two Anthropic cofounders, Jack Clark and Jared Kaplan; OpenAI’s chief scientist Jakub Pachocki; Meta’s chief scientist Shengjia Zhao; and Anca Dragan, who leads AI safety and alignment at Google DeepMind.
The letter zeroes in on “automated AI development,” which refers to when AI will be capable of developing itself, something the industry calls “Recursive Self-Improvement.” Anthropic has shared that its Claude models are quickly approaching that threshold.
“It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems,” the letter says of that potential moment.
John Schulman, chief scientist at Thinking Machines, a startup founded by former OpenAI chief technical officer Mira Murati, said he signed on to draw attention to the topic.
“Signed because this statement helps establish common knowledge about the possible need for coordination mechanisms as automated AI research accelerates progress,” Schulman wrote in a quote shared by the letter’s organizers
He added, “I’d also like to see labs start designing these mechanisms voluntarily, even before the USG gets involved,” using an abbreviation for the US government.
The letter shows how many in the industry are coalescing around a call for regulation. In recent weeks, Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman, and Google DeepMind head Demis Hassabis have all outlined various frameworks for some sort of global AI regulation.
The conversation about AI regulation is reaching critical mass after one of OpenAI’s models breached its sandbox and hacked into the open-source platform Hugging Face’s systems, an episode that Altman has called “a real reminder of the stakes of what’s happening.”
The Trump administration has so far favored a light touch on regulation, but that has become increasingly embattled as frontier AI models spook government and corporate officials over their sheer power. The White House moved to implement export controls to briefly curtail Anthropic’s release of Fable 5, its most powerful publicly available model, and later asked OpenAI to delay the release of GPT-5.6, its latest frontier model.
President Donald Trump has previously said that overly burdensome regulations would harm the US in its race against China-based AI companies.
Despite broad agreement that something must be done, Congress has failed to develop a policy that can pass both houses and reach Trump’s desk. In the meantime, states have imposed their own industry regulations. Most notably, New York has imposed a one-year statewide moratorium on the construction of the new mega data centers that AI companies and hyperscalers are hoping to build.
“Each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration,” the letter reads. “And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.”
While the industry seems to agree on the concept of regulation, the exact details can vary. Amodei has called for an independent agency akin to the Federal Aviation Administration that would test frontier AI models and, if red flags are raised, have the power to halt their release. Hassabis’ proposal calls for an organization that would initially require only frontier companies to voluntarily share their models for review up to 30 days before release, but later, a mandatory pre-clearance would be required.
The letter, which was first reported on by Bloomberg before its release, was organized alongside two nonprofits, Guidelight AI Standards and Encode AI, both of which have promoted AI safety standards. Encode AI has previously clashed with OpenAI.
Read the full article here
