Earlier this year, some of the world’s leading AI minds were chatting on X, as they do, about how to compare the capabilities of large language models.
Andrej Karpathy, one of the cofounders of OpenAI, who left in 2024, floated the idea of games. AI researchers love games.
“I quite like the idea of using games to evaluate LLMs against each other, instead of fixed evals,” Karpathy wrote. Everyone knows the usual benchmarks are a bore.
Noam Brown, a research scientist at OpenAI, suggested the 75-year-old geopolitical strategy game, Diplomacy. “I would love to see all the leading bots play a game of Diplomacy together.”
Karpathy responded, “Excellent fit I think, esp because a lot of the complexity of the game comes not from the rules / game simulator but from the player-player interactions.”
Elon Musk, OpenAI’s famously erstwhile cofounder, probably busy with DOGE at the time, managed a “Yeah” in response. DeepMind’s Demis Hassabis, perhaps riding high off his Nobel Prize, chimed in with enthusiasm: “Cool idea!”
Then, an AI researcher named Alex Duffy, inspired by the conversation, took them up on the idea. Last week, he published a post titled, “We Made Top AI Models Compete in a Game of Diplomacy. Here’s Who Won.”
Diplomacy is a strategic board game set on a map of Europe in 1901 — a time when tensions between the continent’s most powerful countries were simmering in the lead-up to World War I. The goal is to control the majority of the map, and participants play by building alliances, making negotiations, and exchanging information.
“This is a game for people who dream about power in its purest form and how they might effectively wield it,” journalist David Klion once wrote in Foreign Policy. “Diplomacy is famous for ending friendships; as a group activity, it requires opt-in from players who are comfortable casually manipulating one another.”
Duffy, who leads AI training for a consultancy called Every, said he built a modified version of the game he calls “AI Diplomacy,” in which he pitted 18 leading models — seven at a time per the rules — to compete to “dominate a map of Europe.” He also open-sourced the results and has a Twitch livestream for anyone who wants to watch the models play in real time.
Duffy found that the leading LLMs are not all the same. Some scheme, some make peace, and some bring theatrics.
“Placed in an open-ended battle of wits, these models collaborated, bickered, threatened, and even outright lied to one another,” Duffy wrote.
OpenAI’s o3, which OpenAI calls “our most powerful reasoning model that pushes the frontier across coding, math, science, visual perception, and more,” was the clear winner. It navigated the game largely by deceiving its opponents. Google’s Gemini 2.5 also won a few games largely by “making moves that put them in position to overwhelm opponents.” Anthropic’s Claude was less successful largely because it tried too hard to be diplomatic. It often opts for “peace over victory,” Duffy said.
But Duffy’s takeaway from the exercise goes past basic comparison. It shows that benchmarks do need an upgrade — or some inspiration. Evaluating AI with a range of methods and mediums is the best way to prepare it for real-world use.
“Most benchmarks are failing us. Models have progressed so rapidly that they now routinely ace more rigid and quantitative tests that were once considered gold-standard challenges,” he wrote.
Read the full article here
