- Published on
Enterprises that got burned by a bad eval are the most likely to remove humans from the loop, not the least
- Authors
- Name
- DeepHorus
Enterprises are rapidly losing their skepticism of automated AI evaluation toolsâeven as the systems they are meant to safeguard continue to fail at the same rate. According to new data from the VentureBeat Pulse Research agentic reliability tracker, trust in automated evaluation nearly tripled in July, rising from 5% to 13% of organizations. Yet the percentage of enterprises that shipped an agent that passed internal tests and then caused a customer-facing failure held steady at just under half, statistically unchanged from June. The findings point to a widening gap between confidence and evidence, driven largely by companies that have not yet experienced a high-profile failure.
The July survey marks the second wave of the tracker and the first with an identical instrument to the prior month, allowing reliable month-over-month comparisons. Among 108 enterprises with 100 or more employees, the share of organizations fully trusting automated evaluation jumped from 5% to 13%. Meanwhile, the most-cited limitation in Juneâthat evaluations fail to align with real-world outcomesâfell from 29% to 19%, no longer the leading objection. But the actual failure rate barely moved: 49% of firms reported a false-confidence failure in the past year, down only slightly from Juneâs 50%, with 24% experiencing it more than once.
Cross-tabulated data reveal that the new trust comes almost entirely from inexperienced organizations. Among companies that have not been burned by a bad evaluation, 24% fully trust automated evaluation; among those that have been burned, only 4% do. Strikingly, being burned does not slow the march toward autonomyâit accelerates it. Some 85% of enterprises that experienced a failure already allow zero-human deployment or are engineering toward it, compared with 61% of those that have not. Overall, the autonomy trajectory remained flat at 67%, but the population driving it has shifted toward organizations with the most direct evidence that evaluations miss real-world conditions.
The vendor landscape is showing signs of consolidation and maturity. The share of enterprises running no dedicated evaluation tooling fell from 17% to 12%, while specialist platforms gained ground: Braintrust nearly doubled to 15% and DeepEval reached 17%. Switching intent cooled, with 44% planning no change, up from 36%. Ease of integration overtook cost as the top selection factor, surging from 27% to 39%. This suggests enterprises are shifting from price-driven experimentation to fit-driven adoption, a hallmark of a maturing market.
âWhat we are seeing is a classic overconfidence cycle,â said an analyst familiar with the data, who asked to remain anonymous. âThe tools are improving, but the underlying evaluation science hasnât caught up with the speed of deployment. The newest adopters are the most optimistic, and the veterans know why.â The coming months will test whether the rising trust is warranted or a repeating pattern of early enthusiasm followed by corrective failures.
As agentic AI becomes mainstream, the findings underscore a broader tension: the push for autonomous systems is colliding with the need for reliable evaluation. The data suggests that reliability failures do not deter adoptionâthey normalize it. With more enterprises moving toward zero-human operation, the pressure on evaluation tooling to keep pace will only intensify. The next waves of the tracker will reveal whether confidence and reality eventually converge, or whether the gap becomes a permanent feature of the industry.
