中期培训已成为现代 LLM 发展, 的一个重要阶段，在最终培训后使用大规模策划的混合物来增强能力。其数据选择问题是明显的:，数据在接近预训练规模的预训练式目标下进行了优化,，但针对下游功能进行了策划，并从具有不同格式和训练角色的异构源中提取。因此，, 有效选择需要可扩展性和源自适应语义标准。现有的基于模型的方法可以很好地扩展,，但仅提供隐式质量信号。语义选择方法提供更强的判断,，但通常假设固定的标题或标准化的数据格式。为了解决这种不匹配,，我们提出 MIRA, 是一个基于自锚式标题发现的源感知过滤框架。关键思想是使标题构建成为数据选择的一部分: MIRA 首先发现每个源组应该评估什么,，然后将这些判断提炼到可扩展的学生评分器中以进行全语料库过滤。在具有 21 个源和 5 个源组的面向代码的中期训练中，, MIRA 在 9 个代码基准测试中优于选择基线，并且在仅使用一半标记的情况下与完整语料库的运行相匹配。
Mid-training has become an important stage in modern LLM development, using large-scale curated mixtures to strengthen capabilities before final post-training. Its data selection problem is distinct: the data are optimized under a pretraining-style objective at near-pretraining scale, but are curated toward downstream capabilities and drawn from heterogeneous sources with different formats and training roles. As a result, effective selection requires both scalability and source-adaptive semantic criteria. Existing model-based methods scale well, but provide only implicit quality signals. Semantic selection methods offer stronger judgments, but usually assume fixed rubrics or standardized data formats. To address this mismatch, we propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery. The key idea is to make rubric construction part of data selection: MIRA first discovers what should be evaluated for each source group, then distills those judgments into scalable student scorers for full-corpus filtering. On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks and matches the full-corpus run while using only half the tokens.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)