发现因果规律并将其应用于构建功能系统（发现到应用循环）是通用智能, 的标志，但评估这种能力却因科学发现与现实世界工程之间巨大的复杂性差距而受到阻碍。我们引入了 SciCrafter, 一个基于 Minecraft 的基准测试，它通过参数化的红石电路任务来操作这个循环。代理必须以指定的模式 (e.g., 同时或按定时顺序点燃灯); 缩放目标参数大大增加了构建复杂性和所需知识, 迫使真正的发现而不是依赖于记忆的解决方案。在通用代码代理支架, 下评估包括 GPT-5.2, Gemini-3-Pro, 和 Claude-Opus-4.5 在内的前沿模型，我们发现所有模型的成功率均稳定在大约 26% 左右。为了诊断这些失败,，我们将循环分解为四种能力——知识差距识别,实验发现,知识巩固,和知识应用——并设计有针对性的干预措施，其边际贡献作为相应差距的代理。我们的分析表明，尽管通用知识应用能力仍然是所有模型,中前沿模型的最大差距，但知识差距识别开始成为主要障碍——表明瓶颈正在从正确解决问题转向为当前人工智能提出正确问题。我们发布 SciCrafter 作为未来人工智能系统研究的诊断探针，引导整个发现到应用循环。

Discovering causal regularities and applying them to build functional systems--the discovery-to-application loop--is a hallmark of general intelligence, yet evaluating this capacity has been hindered by the vast complexity gap between scientific discovery and real-world engineering. We introduce SciCrafter, a Minecraft-based benchmark that operationalizes this loop through parameterized redstone circuit tasks. Agents must ignite lamps in specified patterns (e.g., simultaneously or in timed sequences); scaling target parameters substantially increases construction complexity and required knowledge, forcing genuine discovery rather than reliance on memorized solutions. Evaluating frontier models including GPT-5.2, Gemini-3-Pro, and Claude-Opus-4.5 under a general-purpose code agent scaffold, we find that all plateau at approximately 26% success rate. To diagnose these failures, we decompose the loop into four capacities--knowledge gap identification, experimental discovery, knowledge consolidation, and knowledge application--and design targeted interventions whose marginal contributions serve as proxies for corresponding gaps. Our analysis reveals that although the general knowledge application capability still remains as the biggest gap across all models, for frontier models the knowledge gap identification starts to become a major hurdle--indicating the bottleneck is shifting from solving problems right to raising the right problems for current AI. We release SciCrafter as a diagnostic probe for future research on AI systems that navigate the full discovery-to-application loop.

科目:人工智能(cs.AI)

Subjects: Artificial Intelligence (cs.AI)