循环架构为学习需要组合推理的任务的逐步过程提供了归纳偏差。循环达到的有效层数决定了这些模型找到的解决方案的质量。与深度架构一样，2C 循环架构很容易出现由深度引起的信号传播问题，因为停止决策被推迟。在本文,中，我们使用预范数层和残差缩放来解决这个信号传播问题。基于这些架构修改,，我们提出 FPRM, 一种基于 Transformer 的定点推理模型，该模型使用定点收敛作为循环架构中的端到端停止机制。我们表明，定点停止允许 FPRM 调整其计算以适应任务难度。 FPRM 对常见推理基准,（即数独, 迷宫, 状态跟踪, 和ARC-AGI）有效。
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address this signal propagation issue using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM, a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to task difficulty. FPRM is effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking, and ARC-AGI.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)