Nouveau Recherche PDF, HTML, DOCX et bien d'autres formats.

Prépublication · 2026

Higher-Scoring Coding Models Did Not Get More Work Done

Yining Hua — Monde

Source : arXiv

Résumé

New coding models advertise themselves on score improvements of coding benchmarks. Yet no study has tested if a model that scores higher on these benchmarks gets more real work done. On June 12, 2026, a U.S. export control directive banned access to Claude Fable 5 for 19 days, which forced users to switch to other models. During this time, most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5). We treat this suspension as a natural experiment, where the people and projects stayed the same and only the model changed, so we could isolate the effect…

Actions

Citation