计算机使用代理 (CUAs) 可以通过原子 GUI 操作,（例如单击和键入,）和高级工具调用,（例如基于 API 的文件操作,）进行操作，但这种混合操作空间通常使他们不确定何时继续 GUI 操作或切换到工具,，从而导致执行路径不理想。这一困难源于高质量交错 GUI 工具轨迹, 的稀缺、收集真实工具轨迹, 的成本和脆弱性以及缺乏对 GUI 工具路径选择的轨迹级监督。在本文,中，我们提出了 ToolCUA, 一种端到端代理，旨在通过分阶段训练范例来学习最佳的 GUI-工具路径选择。我们首先引入交错式 GUI 工具轨迹缩放管道，该管道重新利用丰富的静态 GUI 轨迹并合成一个基础工具库,，无需手动工程或实际工具轨迹收集即可实现多种 GUI 工具轨迹。然后，我们执行工具引导 GUI RFT,，将预热 SFT 与单轮 RL 相结合，以改进关键 GUI-工具切换点的决策。最后,我们在高保真GUI工具环境中通过在线代理强化学习优化ToolCUA,以工具高效路径奖励为指导，鼓励适当的工具使用和更短的执行路径。 OSWorld-MCP 上的实验表明，ToolCUA 的准确率达到了 46.85%,，比基线, 相对提高了约 66%，在可比规模的模型中建立了新的最先进水平。与仅 GUI 设置相比，它还提高了 3.9%,，展示了有效的 GUI 工具编排。结果进一步表明，混合行动空间中的训练对于现实世界的数字代理来说是一个有前途的范例。在这里开源:这个https URL

Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with GUI actions or switch to tools, leading to suboptimal execution paths. This difficulty stems from the scarcity of high-quality interleaved GUI-Tool trajectories, the cost and brittleness of collecting real tool trajectories, and the lack of trajectory-level supervision for GUI-Tool path selection. In this paper, we propose ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm. We first introduce an Interleaved GUI-Tool Trajectory Scaling Pipeline that repurposes abundant static GUI trajectories and synthesizes a grounded tool library, enabling diverse GUI-Tool trajectories without manual engineering or real tool-trajectory collection. We then perform Tool-Bootstrapped GUI RFT, combining warmup SFT with single-turn RL to improve decisions at critical GUI-Tool switching points. Finally, we optimize ToolCUA with Online Agentic RL in a high-fidelity GUI-Tool environment, guided by a Tool-Efficient Path Reward that encourages appropriate tool use and shorter execution paths. Experiments on OSWorld-MCP show that ToolCUA achieves 46.85% accuracy, a relative improvement of approximately 66% over the baseline, establishing a new state of the art among models of comparable scale. It also improves by 3.9% over GUI-only settings, demonstrating effective GUI-Tool orchestration. The results further suggest that training in a hybrid action space is a promising paradigm for real-world digital agents. Open-sourced here: this https URL

科目:人工智能(cs.AI)

Subjects: Artificial Intelligence (cs.AI)