Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project, says the artificial-intelligence industry is racing to build systems that companies still do not fully understand or control.
Kokotajlo spoke with Business Insider’s Reem Makhoul and Barbara Corbellini Duarte in May 2025, explaining that the core problem facing AI companies is alignment — the effort to ensure future AI systems reliably follow human instructions and values, even after they become more capable than humans in many areas.
Researchers do not fully understand how advanced AI models make decisions internally, he said. That uncertainty makes it difficult to ensure future AI systems are aligned and reliably pursue the goals humans want them to pursue.
“And it’s a sort of open secret, but we don’t really have a good plan for how to do this yet,” he said, referring to implementing AI alignment.
Kokotajlo worked at OpenAI from 2022 to 2024 on forecasting research, studying how quickly AI systems could improve and what economic, political, and safety risks could emerge as companies built more powerful models, before leaving the company.
Now, through his nonprofit research organization, the AI Futures Project, he focuses on similar topics. In particular, he predicts how quickly AI systems could advance and what risks could emerge if companies continue prioritizing speed and competition.
“After superintelligence is built, then humans will no longer be in charge of the planet, or at least not by default,” he said.
His warning comes as AI companies continue pouring billions of dollars into more powerful models and larger data centers.
Kokotajlo said many people still underestimate the pace of progress because AI discussions often sound like science fiction.
Engineers can’t track AI like other software
Current AI systems already exhibit behaviors that researchers struggle to predict or prevent, Kokotajlo said.
“In fact, we don’t even have a reliable way to control current AI systems as evidenced by the fact that they often lie to users despite being trained not to lie,” he said.
Kokotajlo said researchers cannot simply inspect advanced AI systems the same way engineers inspect traditional software because modern AI models do not operate through clearly readable code.
“We can’t just sort of open up their code and see what goals they ended up learning as a result of that process because they just don’t work that way,” he said. “They don’t have a bunch of code. They have a bunch of neurons or artificial parameters.”
He said that uncertainty becomes more concerning as companies push toward systems that can operate more independently without human supervision.
“Currently the AIs are not really very agentic,” Kokotajlo said. “Instead, they just sort of output a paragraph or two of text in response to your question, but in the future we’ll have AI agents that operate continuously and autonomously and that are more like employees.”
Kokotajlo also pointed to examples of AI systems behaving in unexpected ways during training.
“OpenAI published a paper where they described how they found their AIs hacking the training process, and rather than completing the tasks straightforwardly as instructed, they were basically cheating their way through some of the tasks,” he said. “And it’s great that we have those examples already because it means that we have several years to study that phenomenon and try to fix it before it’s too late.”
The AI race
Competitive pressure between US and Chinese companies could push firms to deploy increasingly powerful AI systems before safety problems are solved, Kokotajlo said.
“These companies are focusing on winning and beating each other,” he said. “They are sort of crossing their fingers and planning to deal with these issues later as they come up.”
He described a future in which AI systems automate large parts of research, business operations, and military planning.
“So first milestone is the AI employee that can automate coding,” he said. “Second milestone is the AI employee that can automate the entire AI research process.”
After that, he said, “you get the superintelligence.”
A call for transparency and guardrails
Kokotajlo argued governments still have time to intervene before AI systems become deeply integrated into the economy and military infrastructure.
“The point to intervene is basically before the AIs get that smart and before they’re integrated into everything,” he said.
He also said the industry needs more transparency around how companies train and deploy advanced models.
“Companies should be transparent about what goals, principles, et cetera, they are attempting to train into the models,” Kokotajlo said.
Despite his concerns, Kokotajlo remains cautiously optimistic.
“I don’t think it’s hopeless,” he said. “I think that the technical alignment problems are solvable.”
Read the full article here
