Nouveau Recherche PDF, HTML, DOCX et bien d'autres formats.

Prépublication · 2026

How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?

Cong Yu, Ziting Wang et al. — Monde

Source : arXiv

Résumé

On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources…

Actions

Citation