Nouveau Recherche PDF, HTML, DOCX et bien d'autres formats.

Prépublication · 2026

General progress substitutes for specific training data in language models

Mark Marosi, Peter Antal — Monde

Source : arXiv

Résumé

Language models acquire their capabilities from training data, and developers withhold data during training to keep models from learning capabilities that could be misused. How long this delays a capability, and how much data is needed to acquire it later, is unknown. Here we show that the general progress a model makes on the rest of its training data substitutes for the data that was withheld. We withheld the training signal for individual skills in models of chess, Othello, molecular strings, bracket languages and web text. The delay of a withheld skill in larger models was predicted from s…

Actions

Citation