arXiv cs.CLPaper
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
The mechanism matters more than the headline. OPD doesn't expand student capabilities; it reshapes the reward landscape, which can backfire into reward hacking when the teacher rewards behaviors it rarely exhibits. If you're using OPD in production post-training, the real lever is not the distillation itself but cleaning your implicit reward signal. Masking bad outputs during training or using SFT initialization are concrete fixes.