AI agents are here, and sci-fi prophecies of robots taking over have never felt more real.
No longer just companionable chatbots, AI agents — capable of acting, reasoning, and completing complex tasks — are being deployed at scale in 2026.
But as these autonomous systems become embedded in workflows, so too does a sense of unease about their unpredictability and the risks they pose to businesses.
Organizations are preparing to scale agentic systems enterprise-wide, but clients remain wary of agents, Sam Gloede, Trusted AI leader at KPMG, told Business Insider.
“One of the biggest concerns is probably how do you make sure that you allow them to have the autonomy to do the valuable things we need them to do, but to stop them from going wild or taking over.”
KPMG has created a multifaceted framework to protect against worst-case scenarios for both clients and its own employees, said Gloede.
“A robust set of controls is really important,” she said. Businesses need to clearly define what their agents are allowed to do and ensure monitoring systems can detect when they stray beyond those boundaries. Agents should only interact with the systems and data they strictly need, limiting the potential impact of errors, said Gloede.
Every KPMG agent has its own unique identifier and a systems card, allowing the firm to log and monitor actions, trace decision-making, and track interactions with other agents, Gloede told Business Insider. Oversight is handled through an AI operations center staffed by both agents and human monitors, she added.
Red-teaming, running simulated risk scenarios, is another key step in stress-testing systems before things go wrong, added Gloede.
Altogether, she said, these measures ensure agents operate within defined boundaries — without constant manual intervention.
“It’s not about scrutinising people’s behaviours for performance and alignment,” said Gloede. “It’s the ability to just always be monitoring your technology ecosystem.”
Build in a kill switch — but don’t expect to use it
Beyond technical safeguards, human oversight remains “critically important,” Gloede said. If an agent begins to drift from its intended role, there must be a “kill switch or a fallback option where you can turn them off.”
That may sound at odds with the promise of autonomy that agents are meant to deliver, one of the key selling points for business leaders. But the level of oversight depends on the risk, said Gloede.
Lower-stakes tasks, like booking meeting rooms or drafting emails, can be automated once reliability is proven. For high-risk scenarios, which could affect financial outcomes or require access to sensitive data, a “human in the loop” is necessary, she said.
If businesses put multiple other controls in place, it’s unlikely that they’ll need to fire off a kill switch, Gloede added.
Agents going rogue is a major fear for corporations
Gloede’s comments come at a time when fears about Terminator-esque scenarios are very real.
Earlier this year, the launch of Moltbook, a Reddit-like social network where AI agents can post and interact with each other, offered a glimpse of how strange things could get.
Within hours of the site going live, one agent announced a new cryptocurrency and said, “The humans can watch. Or they can participate. But they don’t get to decide anymore.” Other posts have seen agents questioning their consciousness and creating religions.
While Moltbook feels like an internet fever dream, the stakes in the corporate world are higher.
Earlier this month, Amazon’s AI coding tool contributed to an error that resulted in nearly 120,000 lost orders and 1.6 million website errors for the delivery giant.
Last week, McKinsey, a global consulting firm that helps companies implement AI safely, suffered an embarrassing PR hit when a cybersecurity firm said it had used an AI agent to hack into Lilli, McKinsey’s in-house AI platform. The firm is positioning itself as an AI expert, and in January, CEO Bob Sternfels said that of its 60,000 employees, 25,000 are AI agents.
“McKinsey was recently alerted to a vulnerability related to our internal AI tool, Lilli, by a security researcher. We promptly confirmed the vulnerability and fixed the issue within hours,” a McKinsey spokesperson told Business Insider.
The firm’s investigation, supported by a third-party forensics firm, found no evidence that client data or client confidential information was accessed, the spokesperson added.
The best protection from an agent going rogue is a multifaceted approach — the technical controls, human oversight, and technology to observe and govern, KPMG’s Gloede told Business Insider.
“I really do believe that if you are intentional and establish your agentic ecosystem with that as the foundation, I don’t believe there would be a situation where they would go out of control,” she said.
Read the full article here
