强化学习 (RL) 已应用于改进大型语言模型 (LLM) 推理,，但由于缺乏受控, 可扩展环境，对训练如何随任务难度进行扩展的系统研究受到阻碍。观察到的 LLM 在长视野推理中的缺点提出了这样的前景：它们是自回归变压器架构的基础。为了解决这个,，我们引入了 ScaleLogic, 一个综合逻辑推理框架，它提供对难度:所需证明规划的深度(的两个轴的独立控制，即,范围)和底层逻辑的表达能力。我们提出的框架支持广泛的逻辑:，从简单的仅蕴涵逻辑("if-then")到更具表现力的一阶推理，结合连词("and"),或析取("or"),否定("not"),和通用量化("对于所有")。使用此框架,，我们表明 RL 训练计算 $T$ 遵循关于推理深度 $D$ ($T \propto D^{\gamma}$, $R^{2} > 0.99$), 的幂律，并且缩放指数$\gamma$ 随着逻辑表达, 从$1.04$ 单调增加到$2.60$。在下游数学和一般推理基准, 上，更具表现力的训练设置会产生更大的性能增益 (，高达 $+10.66$ 点)，并且与表现力较低的设置相比，计算效率更高的传输, 表明模型的训练内容, 而不仅仅是训练的数量, 决定了下游传输。我们进一步表明，幂律关系在多种 RL 方法中成立,，并且基于课程的培训大大提高了扩展效率。更广泛地说,，我们的结果表明，LLM 在长期推理方面的缺点并不是底层架构, 的根本，可以通过改进培训方法和数据来解决。
Reinforcement learning (RL) has been applied to improve large language model (LLM) reasoning, yet the systematic study of how training scales with task difficulty has been hampered by the lack of controlled, scalable environments. Observed LLM shortcomings in long-horizon reasoning have raised the prospect that they are fundamental to the autoregressive transformer architecture. To address this, we introduce ScaleLogic, a synthetic logical reasoning framework that offers independent control over two axes of difficulty: the depth of the required proof planning (i.e., the horizon) and the expressiveness of the underlying logic. Our proposed framework supports a wide range of logics: from simple implication-only logic ("if-then") towards more expressive first-order reasoning with conjunction ("and"), disjunction ("or"), negation ("not"), and universal quantification ("for all"). Using this framework, we show that the RL training compute $T$ follows a power law with respect to reasoning depth $D$ ($T \propto D^{\gamma}$, $R^{2} > 0.99$), and that the scaling exponent $\gamma$ increases monotonically with logical expressiveness, from $1.04$ to $2.60$. On downstream mathematics and general reasoning benchmarks, more expressive training settings yield both larger performance gains (up to $+10.66$ points) and more compute-efficient transfer compared to less expressive settings, demonstrating that what a model is trained on, not just how much it is trained, shapes downstream transfer. We further show that the power-law relationship holds across multiple RL methods, and curriculum-based training substantially improves scaling efficiency. More broadly, our results demonstrate that LLM shortcomings in long-horizon reasoning are not fundamental to the underlying architecture, and can be addressed by improved training methodology and data.
科目: 人工智能 (cs.AI); 计算和语言 (cs.CL)
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)