Amazon is beefing up internal guardrails after recent outages hit the company’s e-commerce operation, including one disruption tied to its AI coding assistant Q.
Dave Treadwell, Amazon’s SVP of e-commerce services, told staff on Tuesday that a “trend of incidents” emerged since the third quarter of 2025, including “several major” incidents in the last few weeks, according to an internal document obtained by Business Insider. At least one of those disruptions were tied to Amazon’s AI coding assistant Q, while others exposed deeper issues, another internal document explained.
Problems included what he described as “high blast radius changes,” where software updates propagated broadly because control planes lacked suitable safeguards. (A control plane guides how data flows across a computer network).
In other cases, data corruption took hours to unwind. Some failures were traced back to basic mechanisms, such as a requirement to have two people authorize code changes, that were either lacking or bypassed.
In response, Amazon is introducing tighter controls that will require engineers to document code changes more thoroughly and secure additional approvals. At the same time, the company is developing other safeguards designed to introduce what executives described as “controlled friction” into the code-change review process.
“We are implementing temporary safety practices which will introduce controlled friction to changes in the most important parts of the Retail experience,” Treadwell wrote in the document on Tuesday. “In parallel, we will invest in more durable solutions including both deterministic and agentic safeguards.”
The Amazon snafus are an example of how generative AI is upending how software is written, checked, and shipped. AI coding services, such as Claude Code and Amazon’s Q and Kiro offerings, help engineers produce way more code than in the past. However, this code still needs to be checked for bugs and other potential issues before putting it out into the world. When this avalanche of new code hits traditional software-review processes, problems can emerge.
“Agentic” versus “deterministic”
Amazon’s Treadwell wrote that the company’s new code guardrails will combine AI-driven, “agentic” tools with more predictable, rules-based “deterministic” systems.
This tackles one of the core issues with AI models. These powerful new services are not deterministic. That means you can ask the same question twice and an AI model may spit out slightly different answers. That sometimes makes this technology inappropriate for corporate workflows that must be 100% accurate every time. That includes core Enterprise Resource Planning software systems and likely applies to crucial things such as product, price, ordering, and transaction data on a giant e-commerce marketplace, such as the one operated by Amazon.
Earlier on Tuesday, Treadwell held a meeting with some Amazon employees to discuss how to address the recent problems. An Amazon spokesperson told Business Insider the meeting was part of a regular weekly review, and that the Amazon Web Services cloud business was not involved in any of these incidents.
“As part of normal business, the meeting will include a review of the availability of our website and app as we focus on continual improvement,” the spokesperson said.
GenAI will “accelerate exposure”
Some of the most severe disruptions occurred last week, according to the internal documents.
On March 2, customers across Amazon marketplaces saw incorrect delivery times when adding items to their carts. The incident led to nearly 120,000 lost orders and roughly 1.6 million website errors. Amazon’s AI tool Q was one of the primary contributors that triggered the event, according to an internal review.
“GenAI’s usage in control plane operations will accelerate exposure of sharp edges and places where guardrails do not exist,” one internal document said of the March 2 incident. “We need investment in control plane safety.”
On March 5, another outage caused a 99% drop in orders across Amazon’s North American marketplaces, resulting in 6.3 million lost orders, one of the internal documents stated. One key factor was a production change that was deployed without using a formal documentation and approval process called Modeled Change Management.
“No automated pre-deployment validation,” the document said. “Single authorized operator could execute a high-blast-radius config change with no guardrails.”
An Amazon spokesperson told Business Insider that only 1 incident reviewed on Tuesday was AI-related, and none of them involved AI-written code.
A 90-day safety reset
Now, Amazon is rolling out a 90-day, temporary safety guideline that will serve as an addendum to the existing policies, according to one of the internal documents.
The new policy targets approximately 335 “Tier-1 systems,” or services that can directly impact consumers, that have experienced multiple order-impacting incidents since last year and are owned by VP-level organizations.
Under the new policy, Amazon engineers must get two people to review their work before making any coding changes. They also have to use an internal documenting and approval tool and an automated coding system that strictly adheres to Amazon’s central reliability engineering rules.
Amazon is also notifying all Tier-1 system owners, as well as Director- and VP-level leaders, instructing them to audit all production code change activities within their organizations.
The Amazon spokesperson told Business Insider that it is not accurate to say junior and mid-level engineers are required to get sign off from senior engineers for any AI-assisted changes.
The Financial Times previously reported that Amazon was holding a “deep dive” meeting about the outages on Tuesday and that the company’s Kiro AI coding tool partly caused a 13-hour AWS service outage in December.
Have a tip? Contact this reporter via email at ekim@businessinsider.com or Signal, Telegram, or WhatsApp at 650-942-3061. Use a personal email address, a nonwork WiFi network, and a nonwork device; here’s our guide to sharing information securely.
Read the full article here
