- Published on
Amazon AGI director says AI agent reliability, not capability, is blocking enterprise deployment at VB Transform 2026
- Authors
- Name
- DeepHorus
Amazon’s director of AGI autonomy, Bryan Silverthorn, revealed at VB Transform 2026 that the primary bottleneck for enterprise AI agent deployment is reliability, not capability. Citing Cisco data, he noted that 85% of enterprises are piloting AI agents, but only 5% have shipped them to production. This stark gap highlights a critical challenge: agents that excel in internal evaluations often fail in real-world customer environments.
Silverthorn, who joined Amazon through its acquisition of Adept AI and now leads multimodal agent training, argued that reliability must be broken into four distinct dimensions: consistency, robustness, predictability, and safety. He credited this framework to research from Princeton, explaining that it helps unpack factors frequently tangled in evaluations. Without addressing these dimensions, enterprises risk deploying agents that perform well in controlled tests but collapse unpredictably.
A key example involved a customer deploying an agent for software QA that extracted serial numbers from screens. The agent worked flawlessly for two months, then intermittently read wrong numbers due to a vision encoder’s sensitivity to screen position changes. Silverthorn emphasized that the root cause was a measurement failure, not a model flaw. VentureBeat’s proprietary research reinforced this: half of surveyed companies shipped agents that passed internal evals but failed real customers, while most enterprises track uptime over accuracy.
The implications are profound, as enterprises default to model makers’ evaluations, leaving testing strategies as a gamble. Silverthorn stressed that teams must identify their dimensions of variability and match measurement rigor to application stakes. Without rigorous guardrails, agents can cause significant operational risks, from incorrect data extraction to costly errors in automated workflows.
Silverthorn’s most memorable prescription was cultural: inside Amazon’s AGI lab, agents are literally called “interns.” The philosophy treats agents as powerful but occasionally clueless, requiring management skills like asking what could go wrong, adding backups, and consciously accepting certain risks. “You can ask the intern, ‘Hey, what might you do wrong here? How might you mitigate your negative outcomes?’” he said. Amazon’s lab embraces trade-offs, occasionally tolerating agents running wrong experiments for research velocity.
Looking ahead, Silverthorn was candid about current limits. Self-improving AI remains a “loaded term,” with fully autonomous improvement distant. Computer use is a core focus, as seen with a trucking customer using browser automation for warranty claims, but future agents will integrate multiple tools like MCP and APIs. LLM-as-judge techniques are promising but only part of a broader alignment strategy.
For enterprises stuck in pilot purgatory, the path forward is clear: prioritize reliability measurement over benchmarking. By adopting Silverthorn’s four-dimension framework and treating agents as interns requiring oversight, companies can bridge the gap between pilot and production. As Silverthorn concluded, the answer is not better models but better management of variability and risk.
