Skip to content
Back to search

SkillLearnBench

[COLM'26] SkillLearnBench is the first benchmark for evaluating continual learning methods that automatically generate agent skills.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/cxcscmu--skilllearnbench/badge.svg)](/listings/cxcscmu--skilllearnbench)

HTML

<a href="/listings/cxcscmu--skilllearnbench"><img src="/api/listings/cxcscmu--skilllearnbench/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pip install anthropic openai rich tomli dataclaw json-repair   # tomli is only required on Python < 3.11
dataclaw --help                                                # verify that the dataclaw CLI is on your PATH
cp .env.example .env                                           # fill in your API keys (some tasks also require extra variables such as GH_TOKEN — see .env.example)

Docker is a hard requirement, since every agent trial runs inside a container. Install it from [docs.docker.com/get-docker](https://docs.docker.com/get-docker/).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 129 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. # SkillLearnBench
  3. <img src="SkillLearnBench_logo.png" alt="SkillLearnBench" width="900">
  4. [![Homepage](https://img.shields.io/badge/homepage-leaderboard-purple)](https://cxcscmu.github.io/SkillLearnBench)
  5. [![Paper](https://img.shields.io/badge/paper-openreview-red)](https://openreview.net/pdf?id=ulrZEJrNXA)
  6. [![Paper](https://img.shields.io/badge/poster-colm-red)](https://colm.eventhosts.cc/virtual/2026/poster/1764)
  7. [![Tasks](https://img.shields.io/badge/tasks-20-blue)](#tasks)
  8. [![Methods](https://img.shields.io/badge/methods-4-green)](#baselines)
  9. [![License](https://img.shields.io/badge/license-MIT-orange)](LICENSE)
  10. **SkillLearnBench**: A benchmark for continual learning methods that generate agent skills for real-world tasks. <br>
  11. 20 skill-dependent tasks · 15 sub-domains · 100 verified instances
  12. 🔥Exciting news!🔥 Our paper has been accepted to COLM 2026! 🎉🎉
  13. </div>
  14. ## Installation
  15. ```bash
  16. pip install anthropic openai rich tomli dataclaw json-repair # tomli is only required on Python < 3.11
  17. dataclaw --help # verify that the dataclaw CLI is on your PATH
  18. cp .env.example .env # fill in your API keys (some tasks also require extra variables such as GH_TOKEN — see .env.example)
  19. ```
  20. Docker is a hard requirement, since every agent trial runs inside a container. Install it from [docs.docker.com/get-docker](https://docs.docker.com/get-docker/).
  21. ## Quick Start of Evaluation
  22. ```bash
  23. # Always dry-run first — preview what will run, no execution
  24. python evaluate_skills.py court-form-filling github-repo-analytics --dry-run
  25. # Evaluate with human-authored skills (default, `skills/human_authored`) on two tasks (court-form-filling, github-repo-analytics)
  26. python evaluate_skills.py court-form-filling github-repo-analytics
  27. # The committed `skills/<method>/` tree holds pre-generated skills for all baselines,
  28. # e.g., compare method: one-shot (claude-sonnet-4-6) vs. human-authored vs. no-skill baseline.
  29. # Warning: This command will evaluate these 3 methods across all 20 tasks in SkillLearnBench and requires some time to complete.
  30. python evaluate_skills.py --skill-path skills/b1-one-shot-claude-sonnet-4-6 skills/human_authored none
  31. ```
  32. When running `evaluate_skills.py`, the code will load the correspondent skills from `skill-path`, and evaluate them in the tasks of SkillLearnBench.
  33. Evaluation results are written to `output/evaluation_reports/<method>/<task>/`, a `report.csv` record the average results of each metrics.
  34. ⚠️ Keep `--max-workers` ≤ 50 to avoid API rate limits.
  35. <a id="tasks"></a>
  36. ### (1) Tasks
  37. SkillLearnBench contains 20 tasks across 6 real-world categories, with 100 instances in total.
  38. | Category | Task | Instances |
  39. |---|---|:---:|
  40. | **Software Engineering** | python-scala-translation | 2 |
  41. | | nlp-paper-reproduction | 3 |
  42. | | dependency-vulnerability-check | 5 |
  43. | | github-repo-analytics | 5 |
  44. | | fix-security-bug | 3 |
  45. | **Information Retrieval** | enterprise-information-search | 6 |
  46. | | travel-planning | 5 |
  47. | **Productivity Tools** | schedule-planning | 5 |
  48. | | offer-letter-generator | 6 |
  49. | | court-form-filling | 6 |
  50. | **Data & Analytics** | earthquake-plate-calculation | 6 |
  51. | | financial-analysis | 6 |
  52. | | weighted-gdp-calculation | 6 |
  53. | | dbscan-parameter-tuning | 5 |
  54. | | stock-data-visualization | 5 |
  55. | **Content & Creative** | anthropic-poster-design | 5 |
  56. | | chinese-poem-generator | 5 |
  57. | | video-object-counting | 5 |
  58. | **Utilities & Other** | organize-messy-files | 6 |
  59. | | temperature-simulation | 5 |
  60. ### (2) Evaluation Dimensions
  61. | Dimension | Metrics | What it measures |
  62. |-----------|---------|-----------------|
  63. | **Task Success** | Pass rate | Binary verifier outcome per trial |
  64. | **Skill Quality** | Functional coverage, executability, safety | How well a skill describes the task and avoids unsafe instructions |
  65. | **Trajectory Quality** | Key-point recall, execution order, completeness | Whether the solving agent's action trace matches the expected solution path |
  66. The solving agent is powered by Claude Sonnet 4.6, and the LLM-as-judge uses GPT-5-mini.
  67. ## Baselines
  68. We implement four continual learning methods based on skill generation.
  69. | ID | Name | Description |
  70. |----|------|-------------|
  71. | b1 | One-Shot | The agent generates a skill set in a single pass. |
  72. | b2 | Self-Feedback | The agent first generates an initial skill set and uses it to attempt the task. After execution, it reviews the trajectory, identifies issues, and refines the skills. This cycle repeats K=2 times (i.e., K−1 rounds of feedback) without any external supervision. |
  73. | b3 | Teacher-Feedback | After each failed attempt, the agent asks the teacher questions, and the teacher provides directional guidance without revealing the ground-truth skill. The agent then updates its skills and retries the task. The skill set is regenerated up to K=3 times, with up to K−1 QA rounds triggered by failed attempts. This setting simulates a domain expert helping the agent improve. |
  74. | b4 | Skill Creator | Claude's official skill-creator. The agent follows a structured multi-stage process: analyzing the task intent, investigating edge cases and dependencies, writing a skill specification, and validating it with automated checks. |
  75. The agent can be powered by any LLM. In our codebase we provide results for Claude and Gemini (claude-haiku-4-5, claude-sonnet-4-6, and claude-opus-4-6; gemini-3.1-flash-lite-preview, gemini-3-flash-preview, and gemini-3.1-pro-preview).
  76. Run a baseline to generate skills:
  77. ```bash
  78. python generate_skills.py --tasks court-form-filling --methods b1-one-shot --models claude-sonnet-4-6
  79. ```
  80. ### (1) Evaluate other LLMs with the existing four baselines
  81. You can plug additional LLMs into the four baselines above. See [BASELINES.md](BASELINES.md) for more details.
  82. ### (2) Evaluate other continual learning methods
  83. To evaluate your own continual learning method, use the [tasks](./tasks) folder, which provides the verifier (in each `tests` subfolder) and the instance data for every task. Feed the `instance-1` information together WITHOUT its verifier into your method to generate skills. The generated skills should follow the same layout as any subfolder in [skills](./skills/). Then evaluate your method with `python evaluate_skills.py --skill-path your_skill_path`.
  84. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full set of options and instructions on adding new methods.
  85. ## Citation
  86. ```bibtex
  87. @article{zhong2026skilllearnbench,
  88. title={SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks},
  89. author={Zhong, Shanshan and Lu, Yi and Ning, Jingjie and Wan, Yibing and Feng, Lihan and Ao, Yuyi and Ribeiro, Leonardo F. R. and Dreyer, Markus and Ammirati, Sean and Xiong, Chenyan},
  90. journal={arXiv preprint arXiv:2604.20087},
  91. year={2026},
  92. url={https://arxiv.org/abs/2604.20087}
  93. }
  94. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.