由于上下文学习, 自我改进, 和长期代理工作流程的需求，语言模型 (LMs) 的上下文长度急剧增加了,。然而，现有的长上下文语料库, 以书籍, 学术文章, 和代码存储库, 为主，这些资源是有限的，并且在远程依赖关系中通常是稀缺的。在这项工作,中，我们引入了OctoLong,一个上下文工程管道，它配备了AST解析器,、语言服务器后端,和包管理器，以促进代码引用的递归检索,，从而能够管理长度为数百万个令牌的依赖丰富的代码上下文。然后，我们通过对~50B令牌混合物进行上下文扩展中期训练，训练OctoLong-Instruct,一套功能强大的长上下文开放LM,，这些长上下文开放LM,源自大小从600M到14B参数,不等的基本模型，其中~50B令牌混合物包含OctoLong代码上下文~6.2B令牌，然后是指令调整的~10B令牌。我们针对 18 个最先进的开放权重长上下文 LM 进行的训练消融和实验评估表明，用 OctoLong 数据取代 12% 的传统上下文扩展语料库，可以在远程检索, 长期状态跟踪, 存储库级代码理解, 和下游代理任务, 方面产生巨大收益，同时还增强了短上下文编码场景中的 API 使用。
Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.
科目: 人工智能 (cs.AI); 机器学习 (cs.LG); 软件工程 (cs.SE)
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)