OpenAI has set the goal of making an AI tool that could research its own improvements. Now, the company is preparing for the accompanying dangers.
The potential for AI systems to achieve so-called “recursive self-improvement” has come to the forefront for AI leaders, after coding tools from OpenAI and Anthropic leaped forward over the last six months. Google DeepMind CEO Demis Hassabis also said this week that humanity now stands at the “foothills of the singularity,” the moment when AI begins to improve itself and outpaces human intelligence.
OpenAI, which is aiming to go public this year, recently posted a job listing that seeks a safety researcher to grapple with what happens when an AI can train better versions of itself.
The posting for OpenAI’s Preparedness safety team went up this month, job aggregator sites said. It offers a whopping pay package of $295,000 to $445,000 and seeks “strong technical executors to support preparations for recursive self-improvement.”
“This work relies on reasoning about problems that might exist in the future, but might not exist now,” the listing says. “So it’s especially important that people in this role are tasteful and strategic.”
OpenAI and Anthropic did not respond to requests for comment from Business Insider.
Top AI labs race to build self-training models
Models from labs like OpenAI and Anthropic have improved at dizzying rates, as measured by the complexity of the problems they can handle. Researchers at METR, a lab that studies model capabilities, wrote in March that the length of a task that frontier AI models can complete doubles about every seven months — meaning these models can increasingly perform work that takes a long time for humans.
The implication, METR wrote, is that AI agents will be able to handle a “large fraction” of the software work that takes human coders days or weeks to complete.
OpenAI is actively chasing this vision — selling its Codex coding tool to companies is a huge revenue driver. It’s also hoping to automate its own research work. CEO Sam Altman said in October that the company had set goals of running an “automated AI research intern” on hundreds of thousands of chips by this coming September and a “true automated AI researcher by March of 2028.”
“We may totally fail at this goal,” Altman wrote on X, “but given the extraordinary potential impacts we think it is in the public interest to be transparent about this.”
In April, Anthropic published research on using AI models to oversee stronger AI models, with promising but limited results. In May, the company’s cofounder and policy head, Jack Clark, wrote that he thinks there’s a roughly 60% chance of seeing AI research and development without human involvement by the end of 2028.
OpenAI is preparing for self-improving AI
If AI models can train themselves, it’s possible to imagine a sci-fi dystopia where they skyrocket in capability, escape containment, and wreak widespread havoc — a fear of the AI safety movement. METR’s CEO, Elizabeth Barnes, wrote on Friday that in her opinion, “any ‘reasonable’ civilization would clearly be taking things much more slowly and carefully with AI.”
OpenAI’s job posting hints at how the company is preparing for a world where AI models can rapidly improve themselves.
It says the researcher could focus on defending OpenAI’s models from data poisoning — attempts to corrupt an AI model through the dataset it’s trained on. The employee might also make tools to interpret models’ reasoning or experiment with models to understand their safety and dangers.
The posting also says the researcher could “track progress toward automation of technical staff,” including measuring the usage of AI coding tools.
OpenAI’s Preparedness team is tasked with preventing severe harms from AI. The company’s jobs page includes other roles on that team, including for automated red-teaming to test OpenAI’s cybersecurity, biological and chemical risks, and agentic AI threats.
“This is urgent, fast-paced work that has far-reaching implications for the company and for society,” the Preparedness postings say.
Have a tip? Contact this reporter via email at scouncil@businessinsider.com, or over text, Signal, Telegram, or WhatsApp at 415-757-8198. Use a personal email address, a nonwork WiFi network, and a nonwork device; here’s our guide to sharing information securely.
Read the full article here
