在策略蒸馏(OPD) 为语言模型训练后, 提供密集的, 每个token监督，但其有效性受到教师质量的瓶颈: 外部教师遭受分布不匹配,，而具有特权条件的自蒸馏受到上下文学习能力的限制。我们通过\textbf{S}elf-\textbf{E}xtrapolating Policy Distillation),提出\textbf{RISE} (\textbf{R}ecursive \textbf{I}mprovement，它直接从模型'自己的RLVR训练轨迹构造一个合成教师。通过推断当前检查点和尾随锚点之间的位移（在参数空间或输出 logit 空间中），RISE 将稀疏结果引发的参数更新转换为密集的token级目标,，无需任何外部模型或特权条件。 RISE 将 RLVR 和 OPD 结合在一个互补的循环中: 结果奖励将外推推向正确的推理,，而外推的教师则完善token级别的决策。此外，, 由于随着学生的改进，教师每次迭代都会更新, 蒸馏成为一种递归改进机制，而不是一次性压缩步骤。涵盖数学推理, 多域 STEM, 代码生成, 和多轮代理任务的实验表明，RISE 在所有设置上都优于仅 RLVR 训练和策略自蒸馏。
On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose \textbf{RISE} (\textbf{R}ecursive \textbf{I}mprovement via \textbf{S}elf-\textbf{E}xtrapolating Policy Distillation), which constructs a synthetic teacher directly from the model的 own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor---in parameter space or output logit space---RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)