基于 LLM 的代理越来越多地被部署来处理流任务,，但它们通常仍然是一次性的问题解决者，无法从过去的交互中学习。从经验中提炼出来的可重用技能为自我进化,提供了天然的基础，其中高质量的技能管理是关键瓶颈。现有方法要么依赖于手动技能管理, 规定启发式技能操作, 要么训练短期技能操作。然而, 他们仍然很难从间接和延迟的反馈中学习复杂的长期管理政策。为了应对这一挑战,，我们提出了 SkillOS, 一种经验驱动的 RL 训练方案，用于学习自我进化代理的技能管理。 SkillOS 将一个用于检索和应用技能的冻结代理执行器与一个可训练的技能管理者配对，后者根据积累的经验更新外部 SkillRepo。为了提供管理, 的学习信号，我们设计了复合奖励，并根据与技能相关的任务依赖性, 对分组任务流进行训练，其中早期轨迹更新 SkillRepo,，稍后相关任务评估这些更新。在多轮代理任务和单轮推理任务, 中，SkillOS 在有效性和效率方面始终优于无内存和基于强内存的基线,，学习的技能管理者可以在不同的执行器主干和任务域中进行泛化。进一步的分析表明，学习的管理者会产生更有针对性的技能使用,，而 SkillRepo 中的技能会随着时间的推移演变成结构更丰富的 Markdown 文件，这些文件会编码更高级别的元技能。
LLM-based agents are increasingly deployed to handle streaming tasks, yet they often remain one-off problem solvers that fail to learn from past interactions. Reusable skills distilled from experience provide a natural substrate for self-evolution, where high-quality skill curation serves as the key bottleneck. Existing approaches either rely on manual skill curation, prescribe heuristic skill operations, or train for short-horizon skill operations. However, they still struggle to learn complex long-term curation policies from indirect and delayed feedback. To tackle this challenge, we propose SkillOS, an experience-driven RL training recipe for learning skill curation in self-evolving agents. SkillOS pairs a frozen agent executor that retrieves and applies skills with a trainable skill curator that updates an external SkillRepo from accumulated experience. To provide learning signals for curation, we design composite rewards and train on grouped task streams based on skill-relevant task dependencies, where earlier trajectories update the SkillRepo, and later related tasks evaluate these updates. Across multi-turn agentic tasks and single-turn reasoning tasks, SkillOS consistently outperforms memory-free and strong memory-based baselines in both effectiveness and efficiency, with the learned skill curator generalizing across different executor backbones and task domains. Further analyses show that the learned curator produces more targeted skill use, while the skills in SkillRepo evolve into more richly structured Markdown files that encode higher-level meta-skills over time.
科目: 人工智能 (cs.AI); 计算和语言 (cs.CL)
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)