Enterprises are increasingly letting AI agents act on their own while the tests meant to keep those agents in check are losing credibility, a mismatch that researchers call the evaluation gap. A recent survey of 157 midsize and larger firms shows that half of them have shipped an agent that passed internal checks but later failed a customer, and only five percent claim to fully trust automated evaluations today.
Agents pass tests but still break in production
In the past twelve months, 50 % of respondents said an AI feature cleared their pre‑deployment evaluations and then caused a customer‑facing error. Another 24 % reported the same problem more than once, indicating a recurring disconnect between test outcomes and real‑world performance.
Related: Zero trust architecture must now move at agent speed
These incidents are costly because they happen after the agent has already been released, forcing companies to scramble for fixes while customers experience broken workflows or incorrect answers. The data suggest that passing an evaluation no longer guarantees a working agent.
Trust in automated evaluation is limited
When asked which flaw most undermines confidence in automated tests, 29 % of firms pointed to poor alignment with real‑world outcomes—the very issue that explains the failures above. Overall, just five percent say they fully trust the automated evaluation process.
Zero‑human deployment is on the rise
Despite the trust gap, 66 % of organizations already allow—or are building toward—fully automated deployments that require no human review. Thirty‑four percent have approved this approach for low‑risk agents, and another 33 % are engineering pipelines to enable it within the next year. Only 22 % say they will not permit such deployments in the foreseeable future.
Related: Intuit to showcase AI infrastructure rebuild at VB Transform 2026
Large enterprises are not lagging behind smaller firms; 70 % of the biggest respondents are moving toward zero‑human deployment, compared with 64 % of mid‑size companies. This trend shows that the push for autonomy is not limited to startups or niche players.
Provider‑native solutions lead the pack, with 17 % of firms using model providers’ native evals and another 17 % reporting no dedicated evaluation tooling at all.
Related: AI coding tools spawn new software supply risk
Production monitoring also falls short of checking output quality. About a quarter of the surveyed companies run real‑time quality checks on live production traffic.
Future shifts in evaluation strategy
Nearly two‑thirds of firms already permit fully automated, zero‑human‑in‑the‑loop deployment for low‑risk agents, and many are actively engineering toward broader automated deployment within the next twelve months. The evaluation stack remains fragmented and immature, highlighting the need for more robust tools that align test results with real‑world outcomes.
Leave a Reply