Skip to content
Back to search

OpenSkill

Open-World Self-Evolution for LLM Agents — agents that build both their skills and their own verification signals from scratch, with no target-task supervision. (Code coming soon.)

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/OpenLAIR/OpenSkill with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/OpenLAIR/OpenSkill'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/openlair--openskill/badge.svg)](/listings/openlair--openskill)

HTML

<a href="/listings/openlair--openskill"><img src="/api/listings/openlair--openskill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requirements

- Python **3.12+** and [`uv`](https://docs.astral.sh/uv/) - **Docker** (each task runs in its own container) - API keys: **Anthropic** (skill creation & evaluation), **Gemini** (deep research & search grounding), optionally **OpenAI** (Codex evaluation)

Installation

git clone https://github.com/OpenLAIR/OpenSkill.git
cd OpenSkill
uv sync --locked          # pinned dependency tree, including the Harbor sandbox framework
cp .env.example .env      # then fill in your API keys

All dependency versions (including the Harbor evaluation framework and the `google-genai` SDK) are pinned in `uv.lock` to the exact versions the release was validated with.

Repository layout

OpenSkill/
├── libs/
│   ├── openskill/               # the OpenSkill pipeline
│   │   ├── pipeline.py          #   Stage 1–3 orchestrator 
│   │   ├── run_pipeline.py      #   CLI: evolve skills from scratch
│   │   ├── run_eval.py          #   CLI: zero-shot evaluation of existing skills
│   │   ├── openskill_evolution.py #  in-container skill-creator agent 
│   │   ├── agent_planner.py     #   research-query synthesis        
│   │   ├── skill_planner.py     #   multi-skill plan + references   
│   │   ├── deep_research_client.py # deep-research retrieval        
│   │   └── targeted_dr.py       #   gap-vs-bug diagnosis & retrieval 
│   └── terminus_agent/          # agent runtime: skill discovery, virtual verifier,
│                                #   pytest parsing, evaluation-agent wrappers
├── .claude/skills/skill-creator/ # meta-skill injected into the creator agent
├── tasks/                       # 84 SkillsBench tasks — no skills
├── tasks-evolved/               # the same tasks + skills evolved by OpenSkill + retrieved docs
└── pyproject.toml / uv.lock / LICENSE / .env.example

Two copies of the benchmark ship with the repo:

- **`tasks/`** — clean task definitions (instruction, environment, hidden tests). The input for evolving skills from scratch. - **`tasks-evolved/`** — identical tasks with the **released evolved skills** (`environment/skills/evo-*`) and the retrieved knowledge documents (`environment/doc/`) used in the paper. The input for reproducing the main results.

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 274 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. # 🧭 OpenSkill
  3. ### Open-World Self-Evolution for LLM Agents
  4. *An agent that builds both its skills and its own verification signals from scratch — using only a task prompt and open-world resources, with **no target-task supervision**.*
  5. [![Website](https://img.shields.io/badge/🌐_Website-openlair.github.io/openskill-2f64b3)](https://openlair.github.io/openskill/)
  6. [![arXiv](https://img.shields.io/badge/arXiv-2606.06741-b31b1b)](https://arxiv.org/abs/2606.06741)
  7. [![Hugging Face Papers](https://img.shields.io/badge/🤗_HF_Papers-2606.06741-ffcc4d)](https://huggingface.co/papers/2606.06741)
  8. [![Code](https://img.shields.io/badge/💻_Code-repository-2f8c5b)](https://github.com/OpenLAIR/OpenSkill)
  9. [![License](https://img.shields.io/badge/License-Apache_2.0-blue)](LICENSE)
  10. </div>
  11. ---
  12. ## 📰 News
  13. - **2026/08** 🎉 OpenSkill is accepted by **EMNLP 2026**!
  14. - **2026/05** 🚀 Initial release of OpenSkill.
  15. ---
  16. ## Overview
  17. Self-evolving agents need to adapt *after* deployment — but existing methods assume a usable learning loop is already there: curated skills, successful trajectories, or verifier signals. Real open-world deployments may offer **none of these**, only a task prompt.
  18. **OpenSkill** studies *open-world self-evolution*: an agent must build **both its skills and its own verification signals from scratch**, drawing on open-world resources but **no target-task supervision**. Target-task supervision is reserved strictly for final evaluation.
  19. <div align="center">
  20. <table>
  21. <tr>
  22. <td align="center" width="33%"><b>📈 Scalable</b><br/><sub>Skills are sourced from the open world, not bounded by a human's or model's prior knowledge.</sub></td>
  23. <td align="center" width="33%"><b>🌐 Grounded</b><br/><sub>Knowledge and verification anchors come from real docs, repositories, and the web.</sub></td>
  24. <td align="center" width="33%"><b>🔒 Supervision-free</b><br/><sub>No gold answers, rewards, or verifier outputs during learning — a leakage barrier keeps them out.</sub></td>
  25. </tr>
  26. </table>
  27. </div>
  28. ---
  29. ## The Idea — a new paradigm for self-evolving skills
  30. Unlike human-curated, LLM-generated, or supervised self-evolution, OpenSkill acquires skills from the open world and verifies them with self-built virtual tasks — making it **simultaneously scalable, grounded, and supervision-free**. Prior paradigms each miss at least one of these properties.
  31. <div align="center">
  32. <img src="assets/paradigms.png" alt="Four paradigms for self-evolving agent skills: Human-Curated, LLM-Generated, Supervised Self-Evolution, and Ours: Open-World." width="92%"/>
  33. </div>
  34. ---
  35. ## How OpenSkill works
  36. Given only a **task prompt**, a **base model**, **tool access**, and **open-world resources**, OpenSkill bootstraps a learning loop from scratch in three stages.
  37. | Stage | Name | What happens |
  38. |:-----:|:-----|:-------------|
  39. | **01** | **Open-world knowledge acquisition** | Retrieves task-relevant knowledge and independent verification anchors from docs, repos, papers, and the web — then drafts a structured skill plan. |
  40. | **02** | **Leakage-free skill evolution** | Drafts skills and refines them in a sandbox against self-built virtual tests grounded in the anchors, fixing bugs and knowledge gaps over up to three rounds. |
  41. | **03** | **Zero-shot target evaluation** | Deploys the *frozen* skill to the target agent. Ground-truth tests are unlocked only here, at final evaluation — never during construction. |
  42. <div align="center">
  43. <img src="assets/framework.png" alt="OpenSkill framework overview: open-world knowledge acquisition, leakage-free evolution loop with a virtual-task verifier and diagnostic retriever, and final evaluation." width="95%"/>
  44. <br/><sub>A leakage barrier keeps target supervision out of skill construction, unlocking it only for final evaluation.</sub>
  45. </div>
  46. ---
  47. ## 🚀 Getting Started
  48. ### Requirements
  49. - Python **3.12+** and [`uv`](https://docs.astral.sh/uv/)
  50. - **Docker** (each task runs in its own container)
  51. - API keys: **Anthropic** (skill creation & evaluation), **Gemini** (deep research & search grounding), optionally **OpenAI** (Codex evaluation)
  52. ### Installation
  53. ```bash
  54. git clone https://github.com/OpenLAIR/OpenSkill.git
  55. cd OpenSkill
  56. uv sync --locked # pinned dependency tree, including the Harbor sandbox framework
  57. cp .env.example .env # then fill in your API keys
  58. ```
  59. All dependency versions (including the Harbor evaluation framework and the `google-genai` SDK) are pinned in `uv.lock` to the exact versions the release was validated with.
  60. ### Repository layout
  61. ```
  62. OpenSkill/
  63. ├── libs/
  64. │ ├── openskill/ # the OpenSkill pipeline
  65. │ │ ├── pipeline.py # Stage 1–3 orchestrator
  66. │ │ ├── run_pipeline.py # CLI: evolve skills from scratch
  67. │ │ ├── run_eval.py # CLI: zero-shot evaluation of existing skills
  68. │ │ ├── openskill_evolution.py # in-container skill-creator agent
  69. │ │ ├── agent_planner.py # research-query synthesis
  70. │ │ ├── skill_planner.py # multi-skill plan + references
  71. │ │ ├── deep_research_client.py # deep-research retrieval
  72. │ │ └── targeted_dr.py # gap-vs-bug diagnosis & retrieval
  73. │ └── terminus_agent/ # agent runtime: skill discovery, virtual verifier,
  74. │ # pytest parsing, evaluation-agent wrappers
  75. ├── .claude/skills/skill-creator/ # meta-skill injected into the creator agent
  76. ├── tasks/ # 84 SkillsBench tasks — no skills
  77. ├── tasks-evolved/ # the same tasks + skills evolved by OpenSkill + retrieved docs
  78. └── pyproject.toml / uv.lock / LICENSE / .env.example
  79. ```
  80. Two copies of the benchmark ship with the repo:
  81. - **`tasks/`** — clean task definitions (instruction, environment, hidden tests). The input for evolving skills from scratch.
  82. - **`tasks-evolved/`** — identical tasks with the **released evolved skills** (`environment/skills/evo-*`) and the retrieved knowledge documents (`environment/doc/`) used in the paper. The input for reproducing the main results.
  83. ---
  84. ## 📊 Reproducing the main results
  85. Evaluate the released skills with a fresh target agent (Claude Code + Opus 4.6, the paper's main setting). Ground-truth tests run only in this stage:
  86. ```bash
  87. uv run python -m libs.openskill.run_eval \
  88. --tasks 3d-scan-calc \
  89. --tasks-dir tasks-evolved \
  90. --eval-agent claude-code \
  91. --model anthropic/claude-opus-4-6 \
  92. --eval-runs 5
  93. ```
  94. - `--eval-runs 5` follows the paper protocol (n_eval = 5 independent zero-shot runs per task).
  95. - `--eval-agent` selects the target harness: `claude-code` or `codex`.
  96. - To sweep the whole benchmark, pass `--tasks $(ls tasks-evolved)`.
  97. - On machines with slow disks, add `--timeout-multiplier 4` to scale the per-task Docker build and agent timeouts.
  98. Results are appended to `results/eval_results.csv` (per-task reward and test-level accuracy).
  99. ---
  100. ## 🧬 Evolving skills from scratch
  101. Run the full three-stage pipeline on a clean task — deep research, skill planning, verification-anchor retrieval, sandboxed evolution against the virtual verifier, then a 5-run evaluation of the newly evolved skills:
  102. ```bash
  103. uv run python -m libs.openskill.run_pipeline \
  104. --tasks 3d-scan-calc \
  105. --eval-runs 5
  106. ```
  107. The pipeline copies the task from `tasks/` into an isolated `workspace/`, strips any pre-existing skills or documents, and never exposes the hidden test suite to the creator or the verifier. Evolved skills land in `workspace/<task>/environment/skills/evo-*`; the full evolution trace (interventions, virtual-test results, verifier sessions) is stored under `jobs/<job-name>/`.
  108. > [!NOTE]
  109. > Skill evolution is stochastic. On tasks whose instructions under-specify conventions
  110. > (units, interface variants, event definitions), a single evolution run can converge to a
  111. > convention that differs from the hidden tests, so per-task rewards vary across runs.
  112. > The skills in `tasks-evolved/` are the validated versions used for the paper's numbers;
  113. > use them for exact reproduction, and expect run-to-run variance when re-evolving from scratch.
  114. ---
  115. ## Results — best automated pass rate on every setting
  116. On **SkillsBench** (11 domains) OpenSkill beats the strongest closed-world baseline by **+8.9 / +8.8** points and lands within **1–3 points of the human upper bound** — while honoring the no-supervision constraint.
  117. <div align="center">
  118. | Metric | Value |
  119. |:------|:------|
  120. | Overall pass rate on **Opus 4.6** | **43.6%** &nbsp;(+8.9 over best baseline) |
  121. | Overall pass rate on **GPT 5.2** | **42.1%** &nbsp;(+8.8 over best baseline) |
  122. | GT test intents covered by self-built verifier | **88.9%** |
  123. | Domains best / tied-best on Opus 4.6 | **8 / 11** |
  124. </div>
  125. **SkillsBench — overall average pass rate (%)** &nbsp;*(Human = reference upper bound, excluded from ranking)*
  126. | Target agent | No Skill | Self-Gen | CoT | Skill-Creator | AutoSkill | Memento | **OpenSkill** | Human |
  127. |:-------------|:--------:|:--------:|:---:|:-------------:|:---------:|:-------:|:-------------:|:-----:|
  128. | **Opus 4.6** (Claude Code) | 25.5 | 23.9 | 23.9 | 34.7 | 24.7 | 30.1 | **43.6** | *44.5* |
  129. | **GPT 5.2** (Codex) | 25.0 | 32.2 | 33.3 | 29.2 | 11.2 | 15.6 | **42.1** | *44.8* |
  130. > Beyond SkillsBench, OpenSkill is also the best automated method on **SocialMaze** (82.7% / 70.7%) and **ScienceWorld** (90.0% / 85.3%) across both target agents.
  131. ---
  132. <details>
  133. <summary><h2>Analysis — skills transfer, the verifier aligns, every component matters &nbsp;<sub>(click to expand)</sub></h2></summary>
  134. <br/>
  135. <table>
  136. <tr>
  137. <td width="50%" valign="top">
  138. **RQ1 — Transferability**
  139. Skills generated by Opus 4.6 transfer **as-is** to four weaker models, improving by **+5.5 to +14.8** points over no-skill with no model-specific adaptation.
  140. </td>
  141. <td width="50%" valign="top">
  142. **RQ2 — Virtual verifier quality**
  143. Without ever seeing ground-truth tests, the verifier reaches **80.5%** recall against GT-positive outcomes, **60.7%** overall agreement, and covers **88.9%** of GT test intents.
  144. </td>
  145. </tr>
  146. </table>
  147. **RQ3 — Component contribution.** On SocialMaze, reward peaks at three refinement rounds; open-world query and the virtual verifier each improve over a parametric-only baseline and are largely complementary.
  148. <div align="center">
  149. <img src="assets/transfer.png" alt="Transferability of Opus 4.6-generated skills to four weaker models." width="48%"/>
  150. &nbsp;
  151. <img src="assets/ablations.png" alt="Ablations on SocialMaze: reward vs refinement iterations, and component contributions." width="48%"/>
  152. </div>
  153. </details>
  154. ---
  155. ## 🗺️ Roadmap
  156. ### 🟢 Released
  157. - [x] Project page & overview — [openlair.github.io/openskill](https://openlair.github.io/openskill/)
  158. - [x] Paper preprint (arXiv) — [arXiv:2606.06741](https://arxiv.org/abs/2606.06741)
  159. - [x] Core **OpenSkill** framework code (knowledge acquisition → skill evolution → evaluation)
  160. - [x] Reproduction scripts for the SkillsBench main results
  161. - [x] Evolved skills and retrieved knowledge documents for all 84 SkillsBench tasks (`tasks-evolved/`)
  162. ---
  163. ## Citation
  164. ```bibtex
  165. @misc{yan2026openskillopenworldselfevolutionllm,
  166. title = {OpenSkill: Open-World Self-Evolution for LLM Agents},
  167. author = {Zhiling Yan and Dingjie Song and Hanrong Zhang and Wei Liang and Yuxuan Zhang and Yutong Dai and Lifang He and Philip S. Yu and Ran Xu and Xiang Li and Lichao Sun},
  168. year = {2026},
  169. eprint = {2606.06741},
  170. archivePrefix = {arXiv},
  171. primaryClass = {cs.AI},
  172. url = {https://arxiv.org/abs/2606.06741}
  173. }
  174. ```
  175. <details>
  176. <summary><h2>Authors &amp; Affiliations &nbsp;<sub>(click to expand)</sub></h2></summary>
  177. <br/>
  178. Zhiling Yan<sup>1,\*</sup>, Dingjie Song<sup>1,\*</sup>, Hanrong Zhang<sup>2</sup>, Wei Liang<sup>1</sup>, Yuxuan Zhang<sup>3,4</sup>, Yutong Dai<sup>5</sup>, Lifang He<sup>1</sup>, Philip S. Yu<sup>2</sup>, Ran Xu<sup>5</sup>, Xiang Li<sup>6</sup>, Lichao Sun<sup>1,†</sup>
  179. <sup>1</sup> Lehigh University &nbsp;·&nbsp; <sup>2</sup> University of Illinois Chicago &nbsp;·&nbsp; <sup>3</sup> University of British Columbia &nbsp;·&nbsp; <sup>4</sup> Vector Institute &nbsp;·&nbsp; <sup>5</sup> Salesforce AI Research &nbsp;·&nbsp; <sup>6</sup> Massachusetts General Hospital & Harvard Medical School
  180. <sub>\* Equal contribution &nbsp;&nbsp; † Corresponding author</sub>
  181. <div align="center">
  182. <br/>
  183. <img src="assets/logos/lu.png" height="34"/> &nbsp;&nbsp;
  184. <img src="assets/logos/uic.png" height="34"/> &nbsp;&nbsp;
  185. <img src="assets/logos/ubc.jpg" height="40"/> &nbsp;&nbsp;
  186. <img src="assets/logos/sf-logo.png" height="34"/> &nbsp;&nbsp;
  187. <img src="assets/logos/mgh.png" height="34"/> &nbsp;&nbsp;
  188. <img src="assets/logos/harvard.png" height="34"/>
  189. </div>
  190. </details>
  191. ---
  192. <div align="center">
  193. <sub><b>OpenSkill</b> · Open-World Self-Evolution for LLM Agents · 2026 · <a href="https://github.com/OpenLAIR">OpenLAIR</a></sub>
  194. </div>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.