具有可验证奖励的强化学习(RLVR) 使用自动可验证的结果信号, 提高了大型语言模型的推理能力，但这些信号通常是稀疏的并且处于序列级别。按策略自蒸馏 (OPSD) 通过在学生访问的前缀上查询特权教师并提供密集的令牌级分布监督来缓解这种稀疏性。尽管这种密集的监督减轻了信号稀疏性,，但我们发现标准 OPSD 仍然没有充分利用推出的时间结构。它为每个局部散度分配相同的系数,，无论其位置或发生的散度顺序如何。在同政策自回归一代,中，相同的发散幅度可以遵循不同的差异历史,，反映教师和学生之间不匹配的不同演变。由于单独的局部标量无法区分这些时间上下文, 标准 OPSD 无法使其令牌级别权重适应已实现的差异序列。为了解决这个限制,，我们提出了分歧自适应监管视野(DASH)。 DASH 将每个局部蒸馏信号与序列级均值之间的差距映射到自适应传播门，然后使用这些门来控制向后多步聚合。通过这样做，, DASH 根据生成过程中局部分歧的演变来调整代币级别的监督权重。对三个模型规模的三个数学推理基准的实验表明，DASH 比我们在所有三个规模的每个基准上重新运行的匹配的普通 OPSD 有所改进。 DASH 重复使用 OPSD 已计算, 的教师和学生分布，因此增益不需要额外的教师或学生前向传递。此 https URL 代码:
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level. On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision. Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout. It assigns every local divergence the same coefficient, regardless of its position or the divergence sequence in which it occurs. In on-policy autoregressive generation, the same divergence magnitude can follow different discrepancy histories, reflecting different evolutions of the mismatch between the teacher and student. Since the local scalar alone cannot distinguish these temporal contexts, standard OPSD cannot adapt its token-level weights to the realized discrepancy sequence. To address this limitation, we propose Divergence-Adaptive Supervision Horizons (DASH). DASH maps the gap between each local distillation signal and the sequence-level mean to an adaptive propagation gate and then uses these gates to control backward multi-step aggregation. By doing so, DASH adjusts token-level supervision weights according to how local divergences evolve during generation. Experiments on three mathematical reasoning benchmarks across three model scales show that DASH improves over our matched vanilla OPSD reruns on every benchmark at all three scales. DASH reuses the teacher and student distributions that OPSD already computes, so the gains require no additional teacher or student forward pass. Code: this https URL
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)