Alibaba's SWE-CI Benchmark Exposes a Critical Gap: AI Agents Break Working Code 75% of the Time During Maintenance
A new benchmark testing AI agents on real codebase evolution — not just greenfield tasks — finds that three-quarters of agent interventions introduce regressions. Only two models cross the 50% zero-regression threshold.
The AI coding agent hype has a maintenance problem. Alibaba's SWE-CI benchmark, highlighted by @northStar0x7, tests something most benchmarks ignore: whether agents can modify a codebase over time without breaking things that already work. The results are sobering. Across the benchmark, 75% of agent-generated changes broke previously functioning code. Only two models — both variants of Claude Opus — achieved a zero-regression rate above 50%.
Unlock the full briefing
Get every story in today's briefing, the full archive, and the daily AI intelligence brief.
All stories today
Full archive
Daily brief
Cancel anytime. Payments powered by Stripe.