线性注意力用固定大小的循环状态,取代了softmax注意力的无界缓存，将序列混合减少到线性时间，并将解码减少到恒定内存。困难的部分不仅仅是忘记, 的内容，而是如何在不扰乱现有关联的情况下编辑此压缩内存。 Delta 规则模型在写入新值, 之前减去当前读取，Kimi Delta Attention (KDA) 通过通道衰减来锐化遗忘。但主动编辑仍然使用单个标量门来控制两个不同的事情: 在键侧删除多少旧内容以及在值侧提交多少新内容。我们引入了门控 DeltaNet-2,，它通过继承自适应遗忘和通道式衰减来概括门控 DeltaNet 和 KDA，同时解决它们的共同限制,（擦除和写入之间的标量关系）。门控 Delta 规则 2 通过通道擦除门 b_t 和通道写门 w_t, 将这些角色分开，当两个门都崩溃到相同标量时，减少到 KDA；当衰减也崩溃时，减少到门控 DeltaNet。我们推导出一个快速权重更新视图,，一种分块 WY 算法，其通道方式衰减被吸收到不对称擦除因子, 中，以及一个保持高效并行训练的门感知反向传递。在 100B FineWeb-Edu 令牌, 门控 DeltaNet-2 上训练的 1.3B 参数下，在语言建模, 常识推理, 和检索方面，Mamba-2, 门控 DeltaNet, KDA, 和 Mamba-3 变体中取得了最强的总体结果。它的优势在长上下文 RULER 大海捞针基准, 上最为明显，它改进了评估的多键检索设置，并在循环和混合设置中保持强大。代码可从此 https URL 获取。
Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at this https URL.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)