揭秘数据组织以加强法学硕士培训

大型语言模型 (LLMs) 已经彻底改变了各个领域,，但它们的训练效率在很大程度上依赖于有效的数据管理。虽然数据选择已被广泛研究,，但用于增强培训的战略数据组织仍然是一个未充分探索的领域,，特别是因为当前的法学硕士通常只接受一个或几个时期的培训。本文通过重用最初为数据效率, 生成的预先计算的样本级分数，系统地探讨了数据组织对 LLM 训练的影响，从而将额外的计算开销降至最低。我们确定并正式确定了优化数据组织:边界锐化,循环调度,课程连续性,和本地多样性的四个关键准则。在他们,的指导下，我们引入了两种新颖的数据排序方法，称为 STR 和 SAW。涵盖预训练和 SFT 阶段, 的不同模型规模和数据大小, 的广泛实验验证了我们总结的指南的有效性。它们还证明了我们提出的数据排序方法在增强 LLM 培训的稳定性和性能方面的稳健性。 Github 链接: 此 https URL

Large Language Models (LLMs) have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation. While data selection has been widely studied, the strategic data organization for enhanced training remains an underexplored area, particularly since current LLMs are often trained for only one or a few epochs. This paper systematically explores the influence of data organization on LLM training by reusing pre-computed sample-level scores originally generated for data efficiency, thereby incurring minimal additional computational overhead. We identify and formalize four key guidelines for optimizing data organization: Boundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity. Guided by them, we introduce two novel data ordering methods termed STR and SAW. Extensive experiments across different model scales and data sizes, encompassing both pre-training and SFT stages, validate the effectiveness of our summarized guidelines. They also demonstrate the robustness of our proposed data ordering methods in enhancing the stability and performance of LLM training. Github Link: this https URL

科目: 人工智能 (cs.AI); 计算和语言 (cs.CL)

Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)