In the age of AI, Silicon Valley’s “move fast and break things” ethos is proving to be literal.
Earlier this week, Business Insider reported that Amazon set up new guardrails following a series of outages, including one primarily driven by its AI coding tool that led to nearly 120,000 lost orders.
Similar flubs have plagued other companies as they adopt AI. In January, an events company founder said an AI agent made four errors in a single week, including giving away free tickets. And last summer, the CEO of a browser-based coding platform apologized after an AI agent wiped out a client’s codebase and lied about it.
The incidents highlight a delicate balancing act for employers eager to harness AI. Clamp down too hard on workers, and experimentation suffers. Loosen the reins too much, and the risks of errant AI agents or poorly reviewed code can quickly multiply.
“You have to know your own risk tolerance,” said Matt Rosenbaum, a principal researcher at The Conference Board, a nonprofit provider of data and insights for business leaders. “You also have to know what to do if things go wrong and what to change so it doesn’t happen again.”
Speed and power, unchecked
Part of the challenge is that software developers aren’t expected to write as much code as they used to, said Todd Olson, CEO and cofounder of Pendo, an AI startup that helps companies improve their user experience. Now, a large part of developers’ jobs has shifted to reviewing code that is written by AI, he said.
“Those are very different skill sets and different habits,” Olson told Business Insider.
Another problem: Since AI can generate code in seconds, workers racing to meet deadlines may be tempted to accept the output at face value, increasing the risk that mistakes slip through.
Roughly two-thirds of workers have accepted AI-generated output without carefully checking it, and 72% have put less effort into their tasks because of AI, according to a global study by KPMG and the University of Melbourne. The findings are based on a survey of more than 30,000 workers between November 2024 and January 2025.
“The lesson companies are learning is that speed without analytic discipline at scale can create systemic exposure,” said Lauren Buitta, founder and CEO of Girl Security, a nonprofit that prepares young women for careers in national security.
The uncertainty surrounding AI’s rapidly expanding capabilities adds another layer of complexity. As tools become more powerful and accessible, employees may test their limits without fully grasping the downstream consequences.
“Just because you can do something doesn’t mean you should,” said Kevin Serwatka, founder of the recruiting-intelligence platform Benchmarket, who previously worked in recruiting leadership roles at companies such as Google, Meta, and Robinhood.
A takeaway from these mistakes, he said, isn’t to discourage experimentation, “but put guardrails around what that looks like at your company.”
A silver lining
Olson said Amazon’s outage also likely served as a learning lesson for the company, albeit a painful one.
“They just probably found a whole bunch of test cases that they can train AI on, so that the AI can review these things in the future,” he said.
Other companies using AI to write code are likely to make mistakes, too, and that’s a natural part of experimentation, said Andrew Filev, founder and CEO of coding agent company Zencoder.
“Small snafus are actually good,” he said, though ideally they’re identified and addressed internally rather than exposed to customers. “People will learn and improve their guardrails and systems.”
Reminding workers of the importance of speaking up about any errors AI dishes out is critical, said Filev, because if a problem is ignored, it could lead to an “incident where the blast radius is much bigger.”
Filev said achieving AI autonomy requires starting with a combination of AI and human audits.
“You want both processes to work in parallel for a period of time,” he said, until “the AI review is at least as good as the human review.”
Read the full article here
