New Long-Horizon Benchmark Shows Frontier Agents Solve Barely Half of Sustained Terminal Tasks

Long-Horizon Terminal-Bench reports a best mean reward of 0.505, with 29 of 46 tasks never solved by any agent tested — a sober counterweight to the self-improving-agent hype.

Amid a wave of demos claiming agents that rewrite their own code overnight, a new benchmark offers a needed reality check. @Yucheng__Shi introduced Long-Horizon Terminal-Bench (LHTB), a test suite designed to measure how agents perform on sustained, multi-step terminal work rather than the short, tidy tasks that populate marketing reels. The headline number is unflattering: the best mean reward was only 0.505, and 29 of 46 tasks were never solved by any agent evaluated.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae.

Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Aenean nec lorem. In porttitor. Donec laoreet nonummy augue. Suspendisse dui purus, scelerisque at, vulputate vitae, pretium mattis, nunc. Mauris eget neque at sem venenatis eleifend. Ut nonummy. Fusce aliquet pede non pede. Suspendisse dapibus lorem pellentesque magna integer malesuada.

Curabitur sed tortor. Integer aliquam adipiscing lacus. Ut nec urna et arcu imperdiet ullamcorper. Duis at lacus. Quisque purus sapien, gravida non, sollicitudin a, malesuada id, erat. Etiam vestibulum massa rutrum magna. Cras convallis convallis dolor. Quisque tincidunt pede ac urna nunc lectus fermentum diam, at placerat a purus rutrum et vulputate risus nisl sit amet nunc.

Maecenas lorem. Pellentesque pretium lectus id turpis. Etiam sapien elit, consequat eget, tristique non, venenatis quis, ante. Fusce wisi. Phasellus faucibus molestie nisl. Fusce eget urna. Curabitur vitae diam non enim vestibulum interdum nulla id dolor aliquet vulputate. Donec vitae sapien ut libero venenatis faucibus nullam quis ante etiam sit amet orci eget eros faucibus tincidunt.

Unlock the full briefing

Get every story in today's briefing, the full archive, and the daily AI intelligence brief.

All stories today

Full archive

Daily brief