每个 Transformer 架构都致力于学习语义嵌入空间中的丰富表示，但旋转位置嵌入(RoPE) 所作用的旋转流形已被视为固定, 手工制作的结构,，仅由离散序数索引填充。我们认为，这种旋转空间是注意力机制中被很大程度上忽视的第二个表达维度,，其系统探索可能为基于注意力的架构打开一扇新的大门。与复数的类比很有启发性:，就像引入虚轴（与实线正交且独立）一样，解锁了曾经被认为不可能的新代数结构, 将旋转流形视为可学习的, 信号条件空间，打开了注意力的正交自由度。在此框架,中，令牌嵌入对表示的语义(real)组件进行编码——令牌意味着什么——而旋转对其动态(imaginary)组件进行编码——它如何在时间,位置,和上下文中与每个其他令牌相关。我们引入 SIREN-RoPE,，这是这个想法, 的具体实例，它通过双分支正弦表示网络 (SIREN) 用异构信号（连续时间戳, 循环时间模式, 和分类元数据）填充旋转维度。作为概念证明,，我们使用生成推荐器作为排名模型, 对来自主要社交网络的生产规模新闻提要数据集进行评估，证明激活此隐藏维度可以在校准和排名目标上产生一致的改进，而计算开销可以忽略不计。我们邀请社区不要将旋转空间视为已解决的位置编码细节,，而是将其视为未开发的轴，其丰富的结构可能与代数中证明的虚数单位一样对注意力产生重要影响。

Every Transformer architecture dedicates enormous capacity to learning rich representations in semantic embedding space -- yet the rotation manifold acted upon by Rotary Positional Embeddings (RoPE) has been treated as a fixed, hand-crafted structure, populated only by discrete ordinal indices. We argue that this rotation space is a largely overlooked second dimension of expressivity in the attention mechanism, one whose systematic exploration may open a new door for attention-based architectures. The analogy to complex numbers is instructive: just as introducing the imaginary axis -- orthogonal to and independent of the real line -- unlocked new algebraic structure once believed impossible, treating the rotation manifold as a learnable, signal-conditioned space opens an orthogonal degree of freedom in attention. In this framing, the token embedding encodes the semantic (real) component of a representation -- what a token means -- while the rotation encodes its dynamic (imaginary) component -- how it relates to every other token across time, position, and context. We introduce SIREN-RoPE, a concrete instantiation of this idea, which populates the rotation dimension with heterogeneous signals -- continuous timestamps, cyclical temporal patterns, and categorical metadata -- via a dual-branch Sinusoidal Representation Network (SIREN). As a proof of concept, we evaluate on a production-scale news feed dataset from a major social network using a generative recommender as the ranking model, demonstrating that activating this hidden dimension yields consistent improvements across calibration and ranking objectives with negligible computational overhead. We invite the community to view the rotation space not as a solved positional-encoding detail, but as an untapped axis whose rich structure may prove as consequential for attention as the imaginary unit proved for algebra.

科目:人工智能(cs.AI)

Subjects: Artificial Intelligence (cs.AI)