An Agent Can Pass Every Test and Still Fail Its First Real Customer
Nubank screened agent configurations across 16,000+ simulated conversations before going live — a data point that captures how far production reliability has diverged from benchmark performance.
The gap between agents that pass evals and agents that survive production is becoming the defining engineering problem of the year. "An AI agent can pass every test and still fail with its first real customer," wrote @KostyaAI, pointing to Nubank's practice of screening agent configurations across more than 16,000 simulated conversations before a single live test. That number is the story: it takes five figures of synthetic dialogue to develop enough confidence to expose an agent to one real person.
Unlock the full briefing
Get every story in today's briefing, the full archive, and the daily AI intelligence brief.
All stories today
Full archive
Daily brief
Cancel anytime. Payments powered by Stripe.