最近的工作使用 LLM 代理来重现实证社会科学结果，并可访问数据和代码。我们通过询问:他们能否在仅给出论文的方法描述和原始数据的情况下重现结果?我们开发了一个代理复制系统，从论文中提取结构化方法描述,在严格的信息隔离下运行重新实现 - 代理永远不会看到原始代码,结果,或论文 - 并实现复制输出与原始结果的确定性,单元级比较。错误归因步骤通过系统链追踪差异，以确定根本原因。通过对 48 篇论文上的四个代理支架和四个法学硕士进行了人工验证的可重复性, 评估，我们发现代理可以在很大程度上恢复已发表的结果,，但模型, 支架, 和论文之间的性能差异很大。根本原因分析表明，失败既源于代理人的错误，也源于论文本身的不规范。
Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper的 methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation -- agents never see the original code, results, or paper -- and enables deterministic, cell-level comparison of reproduced outputs to the original results. An error attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis reveals that failures stem both from agent errors and from underspecification in the papers themselves.
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)