Every Frontier LLM Scores 0% on ProgramBench, a New Test of Whole-Program Synthesis
The team behind SWE-Bench dropped a new benchmark that asks models to recreate real software like SQLite and FFmpeg from scratch — and no model can do it at all.
The creators of SWE-Bench have released ProgramBench, a new evaluation that asks LLMs to generate entire programs — real-world codebases like SQLite, FFmpeg, and other production software — from scratch, without internet access or starter code. As @jyangballin detailed, the benchmark consists of 200 rigorous, whole-repo generation tasks. The results are striking: every frontier model scores 0%.
Unlock the full briefing
Get every story in today's briefing, the full archive, and the daily AI intelligence brief.
All stories today
Full archive
Daily brief
Cancel anytime. Payments powered by Stripe.