LiveCodeBench (LCB) 最近已成为广泛采用的基准，用于在代码生成任务上评估大型语言模型 (LLMs)。通过策划竞争性编程问题, 不断向集合, 添加新问题并按发布日期过滤它们, LCB 提供污染感知评估并提供编码能力的整体视图。然而, LCB 仍然仅限于Python,，这留下了LLM 是否可以泛化现实软件工程所需的各种编程语言的问题。我们引入了 Multi-LCB, 基准，用于评估包括 Python 在内的 12 种编程语言, 的法学硕士。 Multi-LCB 将 LCB 数据集中的 Python 任务转换为其他语言的等效任务，同时保留 LCB的 污染控制和评估协议。因为它与原始 LCB 格式完全兼容, 多 LCB 将自动跟踪未来的 LCB 更新, 能够系统地评估跨语言代码生成能力，并要求模型维持远远超出 Python 的性能。我们评估了 24 个法学硕士在 Multi-LCB, 上的指令和推理，发现了 Python 过度拟合, 语言特定污染, 的证据以及多语言性能的巨大差异。我们的结果将 Multi-LCB 确立为多编程语言代码评估, 的严格新基准，直接解决 LCB的主要限制并暴露当前 LLM 能力的关键差距。

LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB的 contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB的 primary limitation and exposing critical gaps in current LLM capabilities.

科目: 人工智能 (cs.AI); 编程语言 (cs.PL)

Subjects: Artificial Intelligence (cs.AI); Programming Languages (cs.PL)