我们提出了一种确定性伴随匹配框架，该框架将基于流的生成模型的人类偏好对齐制定为速度场的最优控制问题。在当前策略,下，我们可以直接将控制回归到值梯度引起的目标，从而得到一个简单而稳定的训练目标。基于这个观点,，我们引入了一种截断伴随方案，该方案将计算集中在轨迹,的末端部分，其中与奖励相关的信号集中,，这在保持对齐质量的同时节省了大量的计算量。我们进一步推广了标准的基于 KL 的正则化, 之外的框架，允许在对齐强度和分布保留之间进行更灵活的权衡。 SiT-XL/2 和 FLUX.2-Klein-4B 上的实验表明，多个比对指标, 具有一致的增益，并且多样性和模式保留得到显着改善。
We propose a deterministic adjoint matching framework that formulates human preference alignment for flow-based generative models as an optimal control problem over velocity fields. One can directly regress the control toward a value-gradient-induced target under the current policy, leading to a simple and stable training objective. Building on this perspective, we introduce a truncated adjoint scheme that focuses computation on the terminal portion of the trajectory, where reward-relevant signals concentrate, which yields substantial computational savings while preserving alignment quality. We further generalize the framework beyond standard KL-based regularization, allowing more flexible trade-offs between alignment strength and distributional preservation. Experiments on SiT-XL/2 and FLUX.2-Klein-4B demonstrate consistent gains across multiple alignment metrics, along with substantially improved diversity and mode preservation.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)