推理语言模型的后训练通常由具有可验证奖励的监督蒸馏和强化学习驱动。蒸馏通常依赖于思想链注释，这些注释的获取成本很高，而且本身可能很嘈杂,不完整,或部分不正确;即使最终解决方案是正确的,不完美的基本原理也会干扰学习。另一方面，经过验证的奖励, 的强化学习, 通常将评估反馈压缩为标量信号,，从而模糊了响应的哪些方面应该改进。我们提出 \textbf{Rubric-Conditioned Self-Distillation}, 一个框架，该框架将 rubrics 合并为结构化, 细粒度反馈，用于策略自蒸馏。我们的方法以标准级别的标准为条件调整教师模型，并使用它为 27% 学生自己的采样轨迹提供令牌级别的指导。这种设计避免了将单一参考原理作为唯一的监督目标。相反，, 规则指定了强烈的响应应该满足,，从而在推理过程中比标量奖励优化实现更细粒度的信用分配。我们使用两阶段管道实例化该框架，该管道首先学习生成特定于任务的评分标准，然后训练评分标准引导的推理器。我们对一系列不同的科学推理基准进行了评估，结果表明，以标题为条件的自蒸馏有效地将标题级标准转化为推理过程中的代币级指导, 平均超过 GRPO 1.0 分，超过 OPSD 0.9 分。

Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved. We propose \textbf{Rubric-Conditioned Self-Distillation}, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student的 own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization. We instantiate this framework with a two-stage pipeline that first learns to generate task-specific rubrics and then trains a rubric-guided reasoner. We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.

科目: 人工智能 (cs.AI); 计算和语言 (cs.CL)

Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)