部署在企业环境中的代理必须跨结构化 API 和文档集合, 进行推理，但现有基准测试会单独评估这些功能。我们引入 VAKRA (e\textbf{V}aluating \textbf{A}PI 和 \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), 超过 $8{,}000$ 的基准跨 $62$ 个域的可执行 API，任务跨越三种难度不断增加的设置: 多样化的 API 交互方式, 基于结构化 API 的多跳推理, 以及具有自然语言工具使用策略约束的多源推理。通过针对容纳多个有效路径的实时 API, 重新执行预测的工具调用来验证正确性。使用固定的 ReAct 工具将模型功能与代理架构隔离,，我们评估前沿模型和开放权重模型，发现即使是最好的模型，在单跳端点式任务上也只能达到 70.4\%，而在组合 API 上则下降到 50--51\%; 性能随着推理深度的增加而下降超过 50\%,，并且策略约束的问题会暴露严重的失败(as无法回答的查询) 低至 2.4\%。跟踪分析显示失败集中在语言介导的推理 - 实体消歧, 跨源基础, 而不是工具调用机制。代码可通过 https URL 获取。数据集可通过 https URL 获取
Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4\% on single-hop endpoint-style tasks and drops to 50--51\% on compositional APIs; performance degrades by over 50\% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4\% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available this https URL. Dataset is available this https URL
科目:人工智能(cs.AI)
Subjects: Artificial Intelligence (cs.AI)