将大型语言模型 (LLMs) 与人类价值观保持一致通常会降低其一般能力,，称为对齐税。现有方法通过平衡双重目标, 来缓解这一问题，该目标严重依赖于大量通用数据或辅助奖励模型。在本文,中，我们认为,是因为安全特征在输出分布中本质上是稀疏的,对齐需要局部修改而不是全局权衡。为此,，我们提出 SafeSteer,，它执行仅限于安全令牌的策略蒸馏。首先,我们通过激活转向构建一个安全教师。基于这位老师,我们开发了安全令牌选择算法。因此，, SafeSteer 在训练期间限制对这些令牌的反向 KL 惩罚，以保留一般功能。不同模型的实验结果表明，与现有方法相比，我们的 SafeSteer 在安全性和通用能力之间实现了卓越的权衡,，在七个安全基准上实现了强大的安全性能，而在五个通用能力基准上仅出现了最小的退化。值得注意的是, SafeSteer 仅需要 100 个有害样本，无需使用任何通用数据, 少于之前基线使用数据的 1%, 大大降低了对齐成本。更多详细信息请参见我们的项目页面 https URL。
Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax. Existing methods mitigate this by balancing dual objectives, which heavily rely on massive general-purpose data or auxiliary reward models. In this paper, we argue that, because safety features are inherently sparse within the output distribution, alignment requires localized modifications rather than global trade-offs. To this end, we propose SafeSteer, which performs on-policy distillation confined to safety tokens. First, we construct a safety teacher via activation steering. Based on this teacher, we develop a safety token selection algorithm. Consequently, SafeSteer restricts the reverse KL penalty to these tokens during training to preserve general capabilities. Experimental results across diverse models show that our SafeSteer achieves a superior trade-off between safety and general capability compared with existing methods, attaining strong safety performance on seven safety benchmarks with only minimal degradation on five general capability benchmarks. Notably, SafeSteer requires only 100 harmful samples without using any general-purpose data, less than 1% of what previous baselines used, considerably reducing alignment cost. More details are on our project page at this https URL.
科目: 人工智能 (cs.AI); 计算和语言 (cs.CL)
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)