Skip to content
Back to search

upskill

Generate and evaluate agent skills for code agents like Claude Code, Open Code, OpenAI Codex

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/huggingface--upskill/badge.svg)](/listings/huggingface--upskill)

HTML

<a href="/listings/huggingface--upskill"><img src="/api/listings/huggingface--upskill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install upskill:

uv pip install upskill
# or just use uv
uvx upskill

Create a new skill

upskill generate "write good git commit messages"
# or based on previous agent traces
upskill generate "document the pattern" --from ./trace.md
# Skills are saved to ./skills/{skill-name}/ by default

Generate a skill with a teaching model and evaluate it on a student model.

upskill generate "write good git commit messages" --model sonnet --eval-model haiku

Benchmark a set of models against a skill.

upskill eval ./skills/git-commit-messages/ -m haiku -m sonnet
# logs pretty printed to the terminal

View the results later.

upskill runs --skill git-commit-messages

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 619 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <img width="1920" height="1080" alt="upskill_banner" src="https://github.com/user-attachments/assets/b71fd417-7d23-4f5d-aa89-06ea6b284d1b" />
  2. # UPskill
  3. Generate and evaluate agent skills based on traces with agents. Create skills with teacher models (expensive/slow) that student models (cheap/fast) can use to perform harder tasks reliably.
  4. > [!TIP]
  5. >
  6. > UPskill v2 - recommended default config file now runs evaluations on Hugging Face Jobs. Make sure
  7. > to set your `HF_TOKEN` and use `--artifact-repo <dataset-name>` for job creation and result capture
  8. ## Quick Start
  9. Install upskill:
  10. ```bash
  11. uv pip install upskill
  12. # or just use uv
  13. uvx upskill
  14. ```
  15. Create a new skill
  16. ```bash
  17. upskill generate "write good git commit messages"
  18. # or based on previous agent traces
  19. upskill generate "document the pattern" --from ./trace.md
  20. # Skills are saved to ./skills/{skill-name}/ by default
  21. ```
  22. Generate a skill with a teaching model and evaluate it on a student model.
  23. ```bash
  24. upskill generate "write good git commit messages" --model sonnet --eval-model haiku
  25. ```
  26. Benchmark a set of models against a skill.
  27. ```bash
  28. upskill eval ./skills/git-commit-messages/ -m haiku -m sonnet
  29. # logs pretty printed to the terminal
  30. ```
  31. View the results later.
  32. ```bash
  33. upskill runs --skill git-commit-messages
  34. ```
  35. ## Development checks
  36. This repo uses a CI flow inspired by `fast-agent` with separate format, lint, typecheck, and test
  37. stages.
  38. Install dev dependencies:
  39. ```bash
  40. uv sync --extra dev
  41. ```
  42. Run the quality gates locally:
  43. ```bash
  44. uv run scripts/format.py
  45. uv run scripts/lint.py
  46. uv run scripts/typecheck.py
  47. uv run scripts/cpd.py --check
  48. uv run --extra dev pytest -v
  49. ```
  50. Or use the helper script to run the whole sequence:
  51. ```bash
  52. uv run scripts/check.py
  53. ```
  54. Add `--sync` to include `uv sync --extra dev`, or `--skip-tests` for a faster static-only pass.
  55. To auto-format before re-running checks:
  56. ```bash
  57. uv run --extra dev scripts/format.py --write
  58. ```
  59. Current enforced standards:
  60. - `ruff format --check` for formatting
  61. - `ruff check` for style, imports, modernization, bugbear, simplify, and import-hygiene rules
  62. - cyclomatic complexity via Ruff `C90` with `max-complexity = 15`
  63. - `ty check` across `src`, `tests`, and `scripts`
  64. - `pmd cpd` via `scripts/cpd.py --check` to flag duplicated code in `src/`
  65. - `pytest` for the test suite
  66. CI enforcement lives in `.github/workflows/ci.yml` and runs on pushes and pull requests targeting
  67. `main`.
  68. ## Model Handling Overview
  69. upskill uses distinct phases with explicit model roles:
  70. - **Skill generation**: create/refine `SKILL.md`
  71. - **Test generation**: create synthetic evaluation cases
  72. - **Evaluation**: run tests against evaluator model(s)
  73. - **Benchmark**: repeated evaluation across multiple runs/models
  74. Model flags by command:
  75. | Command | Flag | Meaning |
  76. |---|---|---|
  77. | `generate` | `--model` | Skill generation/refinement model |
  78. | `generate` | `--test-gen-model` | Test generation model override |
  79. | `generate` | `--eval-model` | Optional extra cross-model eval pass |
  80. | `eval` | `-m/--model` | Evaluation model(s) (repeatable) |
  81. | `eval` | `--test-gen-model` | Test generation model override (when tests are generated) |
  82. | `benchmark` | `-m/--model` | Evaluation model(s) to benchmark |
  83. | `benchmark` | `--test-gen-model` | Test generation model override (when tests are generated) |
  84. | `runs` / `plot` | `-m/--model` | Historical results filter only |
  85. `upskill eval` enters **benchmark mode** whenever you pass multiple `-m` values or `--runs > 1`.
  86. In benchmark mode, baseline comparison is always off; `--no-baseline` is redundant.
  87. ## Commands
  88. ### `upskill generate`
  89. Generate a skill from a task description with automatic evaluation and refinement.
  90. ```bash
  91. upskill generate TASK [OPTIONS]
  92. ```
  93. **Arguments:**
  94. - `TASK` - Description of what the skill should teach
  95. **Options:**
  96. - `-e, --example` - Input -> output example (can be repeated)
  97. - `-f, --from PATH` - Improve from existing skill dir or agent trace file (auto-detected)
  98. - `-m, --model MODEL` - Skill generation model (e.g., 'sonnet', 'haiku', 'anthropic.claude-sonnet-4-20250514')
  99. - `--test-gen-model MODEL` - Override test generation model for this run
  100. - `-o, --output PATH` - Output directory for skill
  101. - `--no-eval` - Skip evaluation and refinement
  102. - `--eval-model MODEL` - Different model to evaluate skill on
  103. - `--executor [local|jobs]` - Execution backend for evaluation/refinement; overrides config
  104. - `--artifact-repo TEXT` - Dataset repo for remote fast-agent job artifacts (required with `--executor jobs`)
  105. - `--max-parallel N` - Max concurrent evaluation executions; overrides config
  106. - `--runs-dir PATH` - Directory for run logs (default: ./runs)
  107. - `--log-runs / --no-log-runs` - Log run data (default: enabled)
  108. **Examples:**
  109. ```bash
  110. # Basic usage
  111. upskill generate "parse JSON Schema files"
  112. # Make and evaluate skills for less powerful models
  113. upskill generate "write git commits" --model sonnet --eval-model haiku
  114. # Remote execution on Hugging Face Jobs
  115. upskill generate "parse invoices" --executor jobs --artifact-repo <user>/upskill-tests
  116. # Improve an existing skill (auto-detected as directory)
  117. upskill generate "add more error handling examples" --from ./skills/api-errors/
  118. # Generate from an agent trace file (auto-detected as file)
  119. upskill generate "document the pattern" --from ./trace.json
  120. # Skip evaluation during generation (evaluate separately with upskill eval)
  121. upskill generate "parse YAML" --no-eval
  122. ```
  123. **Output:**
  124. ```
  125. Generating skill with sonnet...
  126. Generating test cases...
  127. Evaluating on sonnet... (attempt 1)
  128. 60% -> 100% (+40%) OK
  129. git-commit-messages
  130. Write clear, conventional commit messages that follow best practices.
  131. SKILL.md ~450 tokens
  132. baseline ████████████░░░░░░░░ 60%
  133. with skill ████████████████████ 100% (+40%)
  134. tokens: 1200 → 800 (-33%)
  135. Saved to ./skills/git-commit-messages
  136. ```
  137. ### `upskill eval`
  138. Evaluate an existing skill against test cases. Supports single-model evaluation with baseline comparison, or multi-model benchmarking.
  139. ```bash
  140. upskill eval SKILL_PATH [OPTIONS]
  141. ```
  142. **Arguments:**
  143. - `SKILL_PATH` - Path to skill directory containing SKILL.md
  144. **Options:**
  145. - `-t, --tests PATH` - Test cases JSON file
  146. - `-m, --model MODEL` - Model(s) to evaluate against (repeatable for multi-model benchmarking)
  147. - `--test-gen-model MODEL` - Override test generation model when tests must be generated
  148. - `--runs N` - Number of runs per model; overrides config
  149. - `--no-baseline` - Skip baseline comparison (simple eval mode only; ignored in benchmark mode)
  150. - `-v, --verbose` - Show per-test results
  151. - `--executor [local|jobs]` - Execution backend for evaluation; overrides config
  152. - `--max-parallel N` - Max concurrent evaluation executions; overrides config
  153. - `--log-runs / --no-log-runs` - Log run data (default: enabled)
  154. - `--runs-dir PATH` - Directory for run logs
  155. **Examples:**
  156. ```bash
  157. # Basic evaluation with baseline comparison
  158. upskill eval ./skills/my-skill/
  159. # With verbose output
  160. upskill eval ./skills/my-skill/ -v
  161. # Custom test cases
  162. upskill eval ./skills/my-skill/ --tests ./tests.json
  163. # Evaluate on specific model
  164. upskill eval ./skills/my-skill/ -m haiku
  165. # Multi-model benchmarking (compare models)
  166. upskill eval ./skills/my-skill/ -m haiku -m sonnet
  167. # Multiple runs per model for statistical significance
  168. upskill eval ./skills/my-skill/ -m haiku -m sonnet --runs 5
  169. # Evaluate a local model configured in fast-agent
  170. upskill eval ./skills/my-skill/ -m generic.my-model
  171. # Skip baseline (just test with skill)
  172. upskill eval ./skills/my-skill/ --no-baseline
  173. # Benchmark mode is triggered by multiple models OR --runs > 1
  174. upskill eval ./skills/my-skill/ -m haiku --runs 5
  175. # Disable run logging
  176. upskill eval ./skills/my-skill/ --no-log-runs
  177. ```
  178. **Benchmark output:**
  179. ```
  180. Evaluating my-skill across 2 model(s)
  181. 3 test case(s), 5 run(s) per model
  182. haiku
  183. Pass rate: 4/5 (80%) Avg assertions: 2.8/3
  184. sonnet
  185. Pass rate: 5/5 (100%) Avg assertions: 3.0/3
  186. ┏━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
  187. ┃ Model ┃ Pass Rate ┃ Avg Assertions ┃ Avg Tokens ┃
  188. ┡━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
  189. │ haiku │ 4/5 │ 2.8/3 │ 1250 │
  190. │ sonnet │ 5/5 │ 3.0/3 │ 1890 │
  191. └────────┴───────────┴────────────────┴────────────┘
  192. ```
  193. **Test cases JSON format:**
  194. ```json
  195. [
  196. {"input": "Write a commit for adding login", "expected": {"contains": ["feat", "login"]}},
  197. {"input": "Fix the null pointer bug", "expected": {"contains": ["fix", "bug"]}}
  198. ]
  199. ```
  200. ### `upskill list`
  201. List all generated skills in a tree view.
  202. ```bash
  203. upskill list [OPTIONS]
  204. ```
  205. **Options:**
  206. - `-d, --dir PATH` - Skills directory to list
  207. - `-v, --verbose` - Show skill contents preview
  208. **Examples:**
  209. ```bash
  210. # List skills in default directory
  211. upskill list
  212. # List from custom directory
  213. upskill list -d ./my-skills/
  214. # Show preview of skill contents
  215. upskill list -v
  216. ```
  217. **Output:**
  218. ```
  219. ./skills
  220. ├── git-commit-messages
  221. │ ├── Write clear, conventional commit messages...
  222. │ └── files
  223. │ └── SKILL.md
  224. ├── api-error-handling
  225. │ ├── Handle API errors gracefully with proper logging...
  226. │ └── files
  227. │ ├── SKILL.md
  228. │ └── references/error-codes.md
  229. └── yaml-parsing
  230. ├── Parse YAML files safely with schema validation...
  231. └── files
  232. ├── SKILL.md
  233. └── scripts/validate.py
  234. ```
  235. ### `upskill runs`
  236. View run results as a plot, or export to CSV. By default, shows a visual comparison of baseline vs with-skill performance.
  237. ```bash
  238. upskill runs [OPTIONS]
  239. ```
  240. **Options:**
  241. - `-d, --dir PATH` - Runs directory
  242. - `-s, --skill TEXT` - Filter by skill name(s) (repeatable)
  243. - `-m, --model TEXT` - Filter historical run data by model(s) (repeatable)
  244. - `--metric [success|tokens]` - Metric to display (default: success)
  245. - `--csv PATH` - Export to CSV instead of plot
  246. **Examples:**
  247. ```bash
  248. # View results plot (default)
  249. upskill runs
  250. # Filter by skill and models
  251. upskill runs -s my-skill -m haiku -m sonnet
  252. # Show token usage instead of success rate
  253. upskill runs --metric tokens
  254. # Export to CSV
  255. upskill runs --csv ./results.csv
  256. # Custom runs directory
  257. upskill runs -d ./my-runs/
  258. ```
  259. **Plot output:**
  260. ```
  261. skill: git-commit-messages
  262. haiku
  263. baseline ████████████░░░░░░░░ 60%
  264. with skill ████████████████░░░░ 80% (+20%)
  265. sonnet
  266. baseline ████████████░░░░░░░░ 60%
  267. with skill ████████████████████ 100% (+40%)
  268. ```
  269. **Matrix view (multiple skills and models):**
  270. ```
  271. ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
  272. ┃ skill ┃ haiku ┃ sonnet ┃
  273. ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
  274. │ git-commit-messages │ 60%→80% │ 60%→100% │
  275. │ api-error-handling │ 40%→70% │ 50%→90% │
  276. │ yaml-parsing │ 70%→90% │ 80%→100% │
  277. └─────────────────────┴──────────────┴──────────────┘
  278. ```
  279. ## Skill Output Format
  280. Skills are saved in a standard directory format:
  281. ```
  282. ./skills/{skill-name}/
  283. ├── SKILL.md # Main skill instructions
  284. ├── references/ # Supporting documents (optional)
  285. └── scripts/ # Executable scripts (optional)
  286. ```
  287. **Example SKILL.md:**
  288. ```markdown
  289. # git-commit-messages
  290. Write clear, conventional commit messages that follow best practices.
  291. ## Instructions
  292. This skill teaches how to write effective git commit messages
  293. following the Conventional Commits specification.
  294. ## Format
  295. Commit messages should follow this structure:
  296. <type>(<scope>): <subject>
  297. <body>
  298. <footer>
  299. ## Types
  300. - `feat`: New feature
  301. - `fix`: Bug fix
  302. - `docs`: Documentation changes
  303. ...
  304. ## Examples
  305. ### Simple feature commit
  306. feat(auth): add password reset functionality
  307. ### Bug fix with explanation
  308. fix(api): handle null response from user service
  309. The user service can return null when not found.
  310. Added proper null checking to prevent crashes.
  311. Closes #123
  312. ```
  313. ## Run Logging
  314. By default, upskill logs all runs to `./runs/`. Each run creates:
  315. ```
  316. ./runs/
  317. ├── 2025_01_21_15_30/ # Batch folder (timestamp)
  318. │ ├── run_1/
  319. │ │ ├── run_metadata.json # Model, task, timing
  320. │ │ └── run_result.json # Pass/fail, assertions, tokens
  321. │ ├── run_2/
  322. │ │ └── ...
  323. │ └── batch_summary.json # Aggregate results
  324. └── results.csv # Summary CSV (after `upskill runs`)
  325. ```
  326. Disable with `--no-log-runs`.
  327. ## Configuration
  328. ### upskill config (`./upskill.config.yaml`)
  329. ```yaml
  330. skill_generation_model: sonnet # Default skill generation model
  331. eval_model: haiku # Default evaluation model (optional)
  332. test_gen_model: null # Optional test generation model
  333. skills_dir: ./skills # Where to save skills
  334. runs_dir: ./runs # Where to save run logs
  335. max_refine_attempts: 2 # Refinement iterations
  336. executor: local # Default execution backend
  337. num_runs: 1 # Default eval/benchmark runs when --runs is omitted
  338. max_parallel: 5 # Default concurrent evaluation executions
  339. jobs_secrets: HF_TOKEN # Comma-separated HF Jobs env var names to forward
  340. jobs_image: ghcr.io/astral-sh/uv:python3.13-bookworm # HF Jobs container image
  341. # fastagent_config: ./fastagent.config.yaml # Optional FastAgent config override
  342. ```
  343. `test_gen_model` fallback behavior:
  344. - CLI `--test-gen-model` overrides config for a single run.
  345. - If set, test generation uses `test_gen_model`.
  346. - If unset, test generation falls back to `skill_generation_model`.
  347. - For `eval`/`benchmark`, this intentionally uses `skill_generation_model` (not `eval_model`) so generated tests stay
  348. stable when sweeping multiple evaluation models.
  349. Backward compatibility: `model` is still accepted in config files as a legacy alias for
  350. `skill_generation_model`.
  351. CLI flags override config values for execution settings:
  352. - `--executor` overrides `executor`
  353. - `--runs` overrides `num_runs`
  354. - `--max-parallel` overrides `max_parallel`
  355. - `--jobs-secrets` overrides `jobs_secrets`
  356. If you set `executor: jobs`, you still need the required jobs-specific CLI inputs such as
  357. `--artifact-repo`.
  358. `jobs_secrets` is a comma-separated list of environment variable names to forward into
  359. remote HF Jobs runs. It should contain secret names such as `HF_TOKEN` or
  360. `ANTHROPIC_API_KEY`, not literal secret values.
  361. `jobs_image` controls which container image HF Jobs uses for remote execution.
  362. Config lookup order:
  363. 1. `UPSKILL_CONFIG` environment variable (path)
  364. 2. `./upskill.config.yaml` (project local)
  365. 3. `~/.config/upskill/config.yaml` (legacy fallback)
  366. ### FastAgent config (`fastagent.config.yaml`)
  367. Place in your project directory to customize FastAgent settings:
  368. ```yaml
  369. default_model: sonnet
  370. logger:
  371. progress_display: true
  372. show_chat: false
  373. streaming: markdown
  374. # MCP servers (optional)
  375. mcp:
  376. servers:
  377. fetch:
  378. command: "uvx"
  379. args: ["mcp-server-fetch"]
  380. ```
  381. ## Environment Variables
  382. ```bash
  383. # Required for Anthropic models
  384. ANTHROPIC_API_KEY=sk-ant-...
  385. # Required for OpenAI models
  386. OPENAI_API_KEY=sk-...
  387. # Optional: custom endpoints
  388. ANTHROPIC_BASE_URL=http://localhost:8080
  389. OPENAI_API_BASE=http://localhost:11434/v1
  390. # For local models (generic provider)
  391. GENERIC_BASE_URL=http://localhost:8080/v1
  392. GENERIC_API_KEY=local # Optional, defaults to "local"
  393. ```
  394. ## Python API
  395. ```python
  396. from upskill import (
  397. generate_skill,
  398. generate_tests,
  399. evaluate_skill,
  400. refine_skill,
  401. Config,
  402. )
  403. # Load configuration
  404. config = Config.load()
  405. # Generate a skill
  406. skill = await generate_skill(
  407. "parse JSON Schema files",
  408. model="sonnet",
  409. config=config,
  410. )
  411. # Generate test cases
  412. tests = await generate_tests("parse JSON Schema files")
  413. # Evaluate the skill
  414. results = await evaluate_skill(
  415. skill,
  416. tests,
  417. model="haiku",
  418. config=config,
  419. )
  420. print(f"Skill lift: {results.skill_lift:.0%}")
  421. print(f"Token savings: {results.token_savings:.0%}")
  422. print(f"Is beneficial: {results.is_beneficial}")
  423. # Refine based on failures
  424. if not results.is_beneficial:
  425. from upskill.evaluate import get_failure_descriptions
  426. failures = get_failure_descriptions(results)
  427. improved_skill = await refine_skill(skill, failures)
  428. ```
  429. ## Model Format
  430. upskill uses FastAgent model format:
  431. ```
  432. <provider>.<model>.<reasoning_effort?>
  433. ```
  434. **Examples:**
  435. - `sonnet` - Anthropic Claude Sonnet (alias)
  436. - `haiku` - Anthropic Claude Haiku (alias)
  437. - `opus` - Anthropic Claude Opus (alias)
  438. - `anthropic.claude-sonnet-4-20250514` - Full model name
  439. - `openai.gpt-4.1` - OpenAI GPT-4.1
  440. - `openai.o3-mini.low` - OpenAI o3-mini with low reasoning effort
  441. - `generic.llama3.2:latest` - Local model via Ollama
  442. - `generic.my-model` - Local model via llama.cpp or other OpenAI-compatible server
  443. ## Local Models
  444. upskill supports local models through any OpenAI-compatible endpoint (llama.cpp, vLLM, etc.).
  445. **With llama.cpp server:**
  446. ```bash
  447. # Start llama.cpp server
  448. ./llama-server -m model.gguf --port 8080
  449. # Configure endpoint via fast-agent config/env, then evaluate
  450. upskill eval ./skills/my-skill/ --model generic.my-model
  451. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.