Nouveau Recherche PDF, HTML, DOCX et bien d'autres formats.

Prépublication · 2026

Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO

Zeyu Leo Liu, Yang Li et al. — Monde

Source : arXiv

Résumé

On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the t…

Actions

Citation