Silicon Valley has spent the past year turning AI usage into a scoreboard.
But after asking four executives about how they measure AI return on investment at Mistral AI’s summit in Paris last month, I noticed something striking: none of them started by talking about how many AI tokens employees were using.
Charles Holive, chief AI officer at BNP Paribas CIB, said his team focuses on outcomes rather than what he called “vanity metrics.”
“We try to go away from vanity metrics — billions of tokens per day,” he said. “We try to make sure that what we track is an outcome, not a vanity metric.”
Instead of asking how many AI tokens employees have used, Holive said he asks: “What did you do, you didn’t do before? How much faster did you do it?”
Antoine Pichot, director of innovation, digital and data at La Banque Postale, told Business Insider the bank measures AI by whether it makes employees more efficient, improves customer service, and delivers value for money.
Amit Kapur, chief AI and transformation officer at Tata Consultancy Services, one of the world’s largest IT services and consulting firms, said his focus is on whether AI is improving business performance rather than just how many tokens employees consume.
Meanwhile, Sujay Bhattacharya, NTT DATA’s executive managing director and global head of Digital Workplace Services and Mistral AI, who helps large companies adopt AI tools from multiple AI providers, said his customers are increasingly looking beyond token counts and focusing on the overall cost and business value of AI projects.
The ‘tokenmaxxing’ backlash
The comments at Mistral’s AI summit come as some US companies are beginning to shift away from “tokenmaxxing” — the idea that greater AI use and higher token consumption automatically lead to higher productivity.
Amazon shut down an internal AI-use leaderboard last month after employees began performing tasks simply to climb the rankings.
Uber COO Andrew Macdonald has publicly questioned whether higher AI spending is producing more useful products, saying it remains difficult to draw a direct line between token consumption and customer value.
Meanwhile, OpenAI, Anthropic, and GitHub have all moved toward usage-based pricing for businesses, putting more pressure on employees at customer companies to prove that higher AI usage is delivering meaningful returns.
The shift away from tokenmaxxing doesn’t mean companies have stopped tracking AI token usage.
Holive said his team at BNP Paribas CIB continues to monitor token consumption to control costs and measure adoption.
But the executives I spoke to in Paris all appeared to agree on one thing: while tokens may tell you how much AI is being used, they don’t necessarily tell you whether it’s yielding a tangible return on investment.
Read the full article here
