线性注意力和状态空间语言模型将前缀压缩为固定大小的循环状态,，产生 O(1) 内存，但代价是有损精确内存:，当许多键值关联竞争, 早期事实被覆盖并且针召回率降低时。受补充学习系统, 的启发，我们为线性注意力提供海马补充。 HOLA (海马线性注意力) 将通常的 delta 规则状态保留为压缩存储器，并添加有界精确 KV 缓存, 形成半参数测试时间存储器:，状态模型线性可压缩结构,，而缓存存储不应强制通过该状态的关联。缓存在没有学习驱逐模块的情况下写入,，保留具有大 beta 的令牌 * ||e||, 实际提交给状态的预测残差; 解耦的 RMSNorm-gamma 缓存读取，然后将这些精确的 KV 对转换为锐检索而不是软平均。在 15B SlimPajama 令牌上训练的 340M 参数, HOLA 将维基文本困惑度从 27.32 降低到 22.92 (-16.1%), 低于全注意力 Transformer++ (26.88), 并将 LAMBADA 困惑度从 30.95 提高到30.26。它还实现了最佳的线性上下文检索，并且比 GDN 或 RULER 大海捞针上匹配的 HOLA+recency 缓存更加稳健，召回高达 32k 令牌(16x 其训练长度)。
Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement. HOLA (Hippocampal Linear Attention) keeps the usual delta-rule state as a compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory: the state models linearly compressible structure, while the cache stores associations that should not be forced through that state. The cache writes without a learned eviction module, keeping tokens with large beta * ||e||, the prediction residual actually committed to the state; a decoupled RMSNorm-gamma cache read then turns these exact KV pairs into sharp retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92 (-16.1%), below a full-attention Transformer++ (26.88), and improves LAMBADA perplexity from 30.95 to 30.26. It also achieves the best linear in-context retrieval and remains much more robust than GDN or a matched HOLA+recency cache on RULER needle-in-a-haystack recall out to 32k tokens (16x its training length).
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)