Layer dropout (a.k.a.随机深度) 已被证明可以在语言和视觉转换器中实现更快的训练,、更高的准确性, 以及对零样本层修剪的鲁棒性。然而，随着模型和数据集的扩展，, 的 dropout（尤其是层 dropout）已从大型语言模型 (LLMs) 预训练配方中基本消失。虽然之前的一些工作报告称，dropout 会降低准确性,，但没有全面的研究量化,，更不用说减轻, 这种影响了。在这项研究,中，我们表明层辍学应该用于最先进的LLM培训,，为培训和培训后的好处建立最佳实践和规模分析。具体来说，, 具有最佳层分布, 时间安排, 和优化器超参数,，我们观察到在相同的训练 FLOPs 下，层 dropout 会导致较低的损失。对于给定数量的训练步骤, LLM 可以实现较低或相似的验证损失，同时节省高达 25% 的训练 FLOP。此外，, 层丢失可实现显着的训练后优化,，例如提前退出, 中间层跳过, 和自推测解码,，从而产生高达 1.5 倍的推理加速，而准确性损失可以忽略不计。通过超过 2400 个训练实验,，模型涵盖从 271M 到 8.2B 参数的模型以及高达 160B 个标记, 的数据集，我们证明这些发现可以可靠地扩展到大规模训练体系。所有预训练实验均在 Cerebras CS-3 系统上运行。

Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout - particularly layer dropout - has largely disappeared from large language models (LLMs) pre-training recipes. While some prior work has reported that dropout can degrade accuracy, no comprehensive study has quantified, let alone mitigated, this effect. In this study, we show that layer dropout should be used in state-of-the-art LLM training, establishing best practices and scaling analysis for both training and post-training benefits. Concretely, with optimal layer distribution, time schedule, and optimizer hyperparameters, we observe that at the same training FLOPs layer dropout leads to lower loss. For a given number of training steps, LLMs can achieve lower or similar validation loss while saving upto 25% of training FLOPs. Moreover, layer dropout enables significant post-training optimizations, such as early exit, intermediate-layer skipping, and self-speculative decoding, yielding up to 1.5x inference speedup with negligible accuracy loss. Across more than 2400 training experiments, spanning models from 271M to 8.2B parameters and datasets up to 160B tokens, we demonstrate that these findings extend reliably to large-scale training regimes. All pre-training experiments were run on Cerebras CS-3 systems.

科目:人工智能(cs.AI)

Subjects: Artificial Intelligence (cs.AI)