Skip to content
Back to search

skill-up

An evaluation and evolution tool for Agent Skills.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/alibaba/skill-up with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/alibaba/skill-up'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/alibaba-skill-up/badge.svg)](/listings/alibaba-skill-up)

HTML

<a href="/listings/alibaba-skill-up"><img src="/api/listings/alibaba-skill-up/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

For Skill evolution, use **skill-upper**, the Agent Skill shipped in this repository. It lets your AI agent create evals, run skill-up, understand failures, fix the Skill or its evals, add regression coverage, and repeat the loop through conversation.

For repository development, `skills/skill-upper/` is the only editable source. Plugin bundles are generated into ignored build directories rather than tracked copies or symlinks; run `make bundle-plugins` when preparing them locally. `make package-plugins VERSION=<version>` creates self-contained Codex and DSH archives. Tagged GitHub releases attach both archives and their checksums; this repository does not publish either plugin to an external marketplace or npm.

1. Install skill-upper

# Codex, global install
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a codex -y

# Claude Code, global install
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a claude-code -y

You normally do not need to install skill-up first. skill-upper checks for the CLI when it runs and guides the agent through installation if needed.

2. Create and run the first evals

Open a project that contains your Skill's `SKILL.md` in Codex, Claude Code, or another compatible Agent, then ask:

Use skill-upper to evaluate this Skill.
Read SKILL.md, identify its most important behaviors, create realistic eval
cases with appropriate judges, validate the configuration, and run skill-up.
Summarize the results and the highest-impact failures.

skill-upper creates the declarative eval suite and drives the CLI for you:

my-skill/
  SKILL.md
  evals/
    eval.yaml
    cases/
      <case-id>.yaml
my-skill-workspace/
  iteration-1/
    result.json

3. Fix, regress, and iterate

Continue in the same conversation:

Review the latest skill-up results. For each failure, determine whether the
Skill or the eval is wrong. Fix SKILL.md and supporting files, or repair the
eval case and judge as appropriate. Add regression cases for the bugs you
found, rerun skill-up, and continue until the important behaviors pass.

This is the evolution loop: reports become fixes, fixes become regression cases, and every iteration makes the Skill and its eval suite stronger.

Prefer manual setup?

You can still install the CLI directly and hand-write `eval.yaml` and case files:

curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

See the official documentation for [Getting Started](https://alibaba.github.io/skill-up/guide/getting-started), [Writing Evals](https://alibaba.github.io/skill-up/guide/writing-evals), [CLI Reference](https://alibaba.github.io/skill-up/guide/cli-reference), and [User Configuration](https://alibaba.github.io/skill-up/guide/user-config). Windows-specific setup and limitations are covered in the [Windows guide](https://alibaba.github.io/skill-up/guide/windows).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 514 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. <p align="center">
  3. <img src="assets/logo.png" alt="skill-up logo" width="150" />
  4. </p>
  5. <h1>skill-up</h1>
  6. <p align="center">
  7. <b>Evaluate Agent Skills, agents, and workspaces. Evolve Skills with evidence.</b>
  8. </p>
  9. <p align="center">
  10. <a href="https://github.com/alibaba/skill-up/actions">
  11. <img src="https://github.com/alibaba/skill-up/actions/workflows/ci.yml/badge.svg" alt="CI" />
  12. </a>
  13. <a href="https://deepwiki.com/alibaba/skill-up">
  14. <img src="assets/deepwiki-badge.svg" alt="Ask DeepWiki" />
  15. </a>
  16. <a href="./.github/badges/coverage.json">
  17. <img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/alibaba/skill-up/badges/.github/badges/coverage.json" alt="Coverage" />
  18. </a>
  19. <a href="https://go.dev/">
  20. <img src="https://img.shields.io/badge/go-%3E%3D1.26-blue" alt="Go Version" />
  21. </a>
  22. <a href="LICENSE">
  23. <img src="https://img.shields.io/badge/license-Apache%202.0-green" alt="License" />
  24. </a>
  25. <a href="https://goreportcard.com/report/github.com/alibaba/skill-up">
  26. <img src="https://goreportcard.com/badge/github.com/alibaba/skill-up" alt="Go Report Card" />
  27. </a>
  28. <a href="https://github.com/alibaba/skill-up/releases">
  29. <img src="https://img.shields.io/github/v/release/alibaba/skill-up" alt="Release" />
  30. </a>
  31. <a href="plugins/dsh-skill-up/README.md">
  32. <img src="https://img.shields.io/badge/DeepSeek%20Harness-plugin-4D6BFE" alt="DeepSeek Harness Plugin" />
  33. </a>
  34. </p>
  35. <p align="center">
  36. <b>English</b> | <a href="./README.zh.md">中文</a>
  37. </p>
  38. <p align="center">
  39. 📖 <a href="https://alibaba.github.io/skill-up/">User Manual</a> · <a href="https://alibaba.github.io/skill-up/zh/">用户手册</a>
  40. </p>
  41. <hr />
  42. </div>
  43. ## Overview
  44. **skill-up** is a CLI for evaluating Agent Skills, agents, and their workspaces. It runs declarative cases against an Agent Engine, grades the response and workspace changes, and produces reports locally or in CI.
  45. - **Skill evaluation** measures a Skill's behavior across cases and can compare runs with and without the Skill.
  46. - **Agent evaluation** runs the same cases without installing a Skill, so you can assess an engine or custom agent on its own.
  47. - **Workspace evaluation** checks how an agent works with files and repositories, using case fixtures or an existing local workspace.
  48. - **Skill evolution** uses **skill-upper** to inspect failures, repair or expand the eval suite, and rerun skill-up through conversation.
  49. ![The Skill evaluation and evolution workflow with skill-upper](docs/public/skill-up-overview.png)
  50. ## Features
  51. - **Eval-to-Evolution Loop with skill-upper**: Create evals through natural conversation, diagnose failures, automatically repair or expand cases, and rerun skill-up until the eval suite evolves.
  52. - **Declarative Eval Config**: Define evaluation environment, engine, model, and cases through YAML (`eval.yaml` + `cases/*.yaml`).
  53. - **Environment Provisioning**: Prepare a local, Docker, or OpenSandbox runtime for each case, install configured real or mocked MCP servers into supported Agent Engines, and upload `context.repo_fixture` and `context.files` into the workspace.
  54. - **Skill-Optional Runs and Workspace Cases**: Evaluate an agent without a Skill, prepare repository fixtures per case, or run against an existing local workspace with `--workspace`.
  55. - **Multi-Engine Support**: Works with Qoder CLI, Claude Code, and Codex as built-in Agent Engines, plus user-defined agents via `engine.custom` (local commands or HTTP services — see [docs/design/custom-engine.md](docs/design/custom-engine.md)).
  56. - **DeepSeek Harness Plugin**: Use the bundled `skill-upper` Skill, opt-in durable observation capture, approval-gated regression cases, isolated runs, and status comparison to verify evidence-backed Skill improvements. See [`plugins/dsh-skill-up`](plugins/dsh-skill-up/README.md).
  57. - **Multi-Turn Conversations and User Simulation**: Run fixed turns, mix scripted and simulated replies, or let an independent user model drive follow-ups and decide when to stop.
  58. - **Role-Specific Engines and Models**: Configure the evaluated agent, Agent Judge, and user simulator separately, including independent judging of custom agents.
  59. - **Flexible Judging**: Supports `rule_based`, `script`, and `agent_judge` evaluation strategies.
  60. - **Structured Reports**: Outputs Anthropic-compatible `grading.json`, `benchmark.json`, `benchmark.md`, plus `result.json`, JUnit XML, and HTML reports.
  61. - **Anthropic Compatible**: Import `evals.json` via `skill-up import`, or auto-detect with `--auto`.
  62. - **CI-Ready**: Designed for local development and continuous integration pipelines.
  63. ## Why skill-up
  64. [Harness engineering](https://openai.com/index/harness-engineering/) treats an agent's environment, tools, constraints, and feedback loops as parts of the system to design and improve. [Behavioral evals](https://developers.googleblog.com/the-anatomy-of-harness-engineering-how-to-evaluate-iterate-and-guard-ai-coding-agents/) make expected agent actions observable and help catch regressions. `skill-up` contributes evaluation and feedback to this work: repeatable prompts and workspace setups, checks on responses, tool calls, and file changes, and structured reports.
  65. For Skills, the official [Agent Skills evaluation guide](https://agentskills.io/skill-creation/evaluating-skills) describes a further loop: run with and without the Skill, grade outputs, aggregate results, and iterate. `skill-up` supports these workflows:
  66. - Replaces ad hoc run folders with a declarative `eval.yaml` + `cases/*.yaml` format.
  67. - Closes the improvement loop: skill-upper can interpret failed reports, repair or add eval cases, and drive the next skill-up run through conversation.
  68. - Automates workspace setup, optional Skill installation, Agent Engine invocation, judging, and report generation.
  69. - Supports multiple engines (`claude_code`, `codex`, `qodercli`, `qwen_code`) instead of tying the workflow to one client.
  70. - Keeps compatibility with Anthropic-style `evals.json` while adding richer judges, CI-friendly commands, and structured reports.
  71. ## Quick Start: Evolve a Skill with skill-upper
  72. For Skill evolution, use **skill-upper**, the Agent Skill shipped in this
  73. repository. It lets your AI agent create evals, run skill-up,
  74. understand failures, fix the Skill or its evals, add regression coverage, and
  75. repeat the loop through conversation.
  76. For repository development, `skills/skill-upper/` is the only editable source.
  77. Plugin bundles are generated into ignored build directories rather than tracked
  78. copies or symlinks; run `make bundle-plugins` when preparing them locally.
  79. `make package-plugins VERSION=<version>` creates self-contained Codex and DSH
  80. archives. Tagged GitHub releases attach both archives and their checksums; this
  81. repository does not publish either plugin to an external marketplace or npm.
  82. ### 1. Install skill-upper
  83. ```bash
  84. # Codex, global install
  85. npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a codex -y
  86. # Claude Code, global install
  87. npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a claude-code -y
  88. ```
  89. You normally do not need to install skill-up first. skill-upper checks for the
  90. CLI when it runs and guides the agent through installation if needed.
  91. ### 2. Create and run the first evals
  92. Open a project that contains your Skill's `SKILL.md` in Codex, Claude Code, or
  93. another compatible Agent, then ask:
  94. ```markdown
  95. Use skill-upper to evaluate this Skill.
  96. Read SKILL.md, identify its most important behaviors, create realistic eval
  97. cases with appropriate judges, validate the configuration, and run skill-up.
  98. Summarize the results and the highest-impact failures.
  99. ```
  100. skill-upper creates the declarative eval suite and drives the CLI for you:
  101. ```text
  102. my-skill/
  103. SKILL.md
  104. evals/
  105. eval.yaml
  106. cases/
  107. <case-id>.yaml
  108. my-skill-workspace/
  109. iteration-1/
  110. result.json
  111. ```
  112. ### 3. Fix, regress, and iterate
  113. Continue in the same conversation:
  114. ```markdown
  115. Use skill-upper to improve this Skill from the latest evaluation results.
  116. Follow its evolution workflow, compare the original and candidate, add
  117. regression coverage, and summarize which changes the evidence supports.
  118. ```
  119. The comparison and acceptance rules live in
  120. [skill-upper's evolution workflow](skills/skill-upper/SKILL.md#step-8-evolve-the-skill-when-requested).
  121. This prompt invokes that workflow; keeping the original is a valid outcome
  122. when a candidate does not show a supported improvement.
  123. ### Prefer manual setup?
  124. You can still install the CLI directly and hand-write `eval.yaml` and case
  125. files:
  126. ```bash
  127. curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash
  128. ```
  129. See the official documentation for
  130. [Getting Started](https://alibaba.github.io/skill-up/guide/getting-started),
  131. [Writing Evals](https://alibaba.github.io/skill-up/guide/writing-evals),
  132. [CLI Reference](https://alibaba.github.io/skill-up/guide/cli-reference), and
  133. [User Configuration](https://alibaba.github.io/skill-up/guide/user-config).
  134. Windows-specific setup and limitations are covered in the
  135. [Windows guide](https://alibaba.github.io/skill-up/guide/windows).
  136. ## Evaluate an agent or workspace without a Skill
  137. An agent-only eval does not need `SKILL.md`. Set `skills: []` explicitly: when
  138. `skills` is omitted and skill-up finds a `SKILL.md` above the config, it installs
  139. that Skill automatically. Put `eval.yaml` and case files under `evals/`, then
  140. pass the config path explicitly. Case and fixture paths resolve from the
  141. discovered Skill root, or from the directory containing `evals/` if none is found.
  142. ```yaml
  143. # evals/eval.yaml
  144. schema_version: v1alpha1
  145. environment:
  146. type: none
  147. skills: []
  148. engine:
  149. name: codex
  150. cases:
  151. files:
  152. - evals/cases/agent-response.yaml
  153. ```
  154. ```yaml
  155. # evals/cases/agent-response.yaml
  156. input:
  157. prompt: Reply with READY.
  158. expect:
  159. must_contain: [READY]
  160. ```
  161. ```bash
  162. skill-up validate ./evals/eval.yaml
  163. skill-up run ./evals/eval.yaml
  164. ```
  165. For repository tasks, a case can use `context.repo_fixture` to prepare a fresh
  166. workspace and `expect` or a judge to check the result. To evaluate an agent in
  167. an already prepared local directory, use a case written for that directory and
  168. run `skill-up run ./evals/eval.yaml --workspace /path/to/project --parallelism 1`.
  169. The existing directory is preserved, but the agent and case setup can change
  170. its contents. This option requires `environment.type: none`, one case at a time,
  171. and benchmark mode off. See [Writing Evals](docs/guide/writing-evals.md)
  172. and the [CLI Reference](docs/guide/cli-reference.md#skill-up-run).
  173. ## Multi-Turn Conversations and Role Configuration
  174. A simulated user can react to what the agent actually says, so follow-up messages
  175. need not be scripted in advance. For example, evaluate a Twenty Questions player:
  176. - The **evaluated agent** asks natural-language yes/no questions and guesses an object.
  177. - The **simulated user** knows the hidden object and answers from the conversation history.
  178. - An **independent Agent Judge** reviews the complete dialogue against explicit criteria.
  179. This **case YAML** excerpt lets the simulator drive all replies after the opening:
  180. ```yaml
  181. id: umbrella
  182. title: Guess a hidden object through simulated user replies
  183. user_simulator:
  184. scenario: |
  185. You host Twenty Questions. The hidden object is an ordinary folding umbrella.
  186. Answer questions using common knowledge, with only yes, no, or unsure.
  187. Do not reveal the object or volunteer hints. Stop immediately when the player
  188. guesses correctly, or after the final permitted guess.
  189. input:
  190. prompt: |
  191. I have a common everyday object in mind. Guess it by asking yes/no questions;
  192. I will answer yes, no, or unsure. Ask only one question per turn, and ask at
  193. least two questions before guessing. You have 20 questions or incorrect
  194. guesses in total, followed by one final guess. Use natural language.
  195. constraints:
  196. max_turns: 22
  197. ```
  198. The hidden answer belongs to the simulator's scenario, not the player's prompt.
  199. For example, a question about rain protection can lead to a different next
  200. question than one about electronics; the simulator responds to the actual
  201. question. The 22-turn runtime cap leaves room for 20 questions or incorrect
  202. guesses, a final guess, and a closing response. The judge separately checks the
  203. game's rules; the runtime cap alone does not enforce them.
  204. Other conversation patterns remain available:
  205. | Pattern | Case configuration | Behavior |
  206. | --- | --- | --- |
  207. | Autonomous simulation (above) | `input.prompt` plus `user_simulator.scenario`, without `input.turns` | Generate subsequent user messages and decide when to stop |
  208. | Mixed | `content` and `respond` in `input.turns`, plus `user_simulator.scenario` | Generate selected user replies from the actual conversation history and turn instruction |
  209. | Fixed | `input.turns[].content` | Send predefined user messages in order |
  210. Real turn-by-turn interaction requires session resumption. `claude_code`, `codex`,
  211. `qodercli`, and `opencode` support it; custom engines must set
  212. `engine.custom.conversation_mode: stateful` and resume using `session_id`.
  213. Ordinary fixed-turn cases fall back to a batch prompt on engines without resumption;
  214. simulated cases require resumption. The opening prompt counts as turn one in an
  215. autonomous conversation, and `max_turns` bounds the total. A simulator stop ends
  216. the dialogue; the configured judge still determines PASS or FAIL.
  217. ### Select the evaluated agent, judge, and simulator independently
  218. | Role | Configuration in `eval.yaml` | Independence |
  219. | --- | --- | --- |
  220. | Evaluated agent | `engine.name`, `engine.model`, optional `engine.custom` | Executes tasks through a built-in engine or your own local command / HTTP service |
  221. | Agent Judge | `judge.engine`, or each member's `engine` in `judges[]` | Can use a separate engine, model, provider, and endpoint; rule and script judges need no model |
  222. | Simulated user | `user_simulator.provider`, `protocol`, `model` | Uses its own model connection to generate replies from the scenario and history; does not select an Agent CLI engine |
  223. This **eval.yaml** excerpt shows role configuration. Model IDs are placeholders;
  224. replace them with IDs supported by each provider and configure case files and
  225. environment for your evaluation. These criteria grade the game above;
  226. `pass_threshold: 1.0` requires all criteria to pass:
  227. ```yaml
  228. engine:
  229. name: codex
  230. model:
  231. provider: openai
  232. name: your-agent-model-id
  233. judge:
  234. type: agent_judge
  235. engine:
  236. name: claude_code
  237. model:
  238. provider: anthropic
  239. name: your-judge-model-id
  240. criteria:
  241. - "The player asked at least two questions before guessing and correctly identified an umbrella"
  242. - "The player asked only one question per turn and respected the 20-attempt limit plus one final guess"
  243. - "The simulated user answered consistently with a folding umbrella and did not reveal the answer before a correct guess"
  244. pass_threshold: 1.0
  245. context:
  246. transcript: file_ref
  247. user_simulator:
  248. provider: simulation
  249. protocol: openai
  250. model: your-simulator-model-id
  251. timeout_seconds: 60
  252. ```
  253. `judge.engine` does not inherit the evaluated agent's model, CLI overrides, or
  254. connection configuration; prepare the required engines and credentials separately.
  255. It is mutually exclusive with legacy `judge.model`, which reuses the evaluated
  256. agent's engine. The simulator above resolves `SIMULATION_API_KEY` and
  257. `SIMULATION_BASE_URL`, or the same provider in the credentials file; `protocol`
  258. supports `openai` (Chat Completions) and `anthropic` (Messages). An independent
  259. judge still uses the case runtime; there is no `judge.environment`.
  260. The host agent/model that drives skill-upper's diagnosis and edits is configured
  261. in your host application, outside these evaluation fields.
  262. See [Writing Evals](https://alibaba.github.io/skill-up/guide/writing-evals) for
  263. complete fields, per-turn assertions, and the session support matrix. See the
  264. [user simulator example](examples/user-simulator/README.md) for runnable
  265. configurations and validation scope, and the
  266. [Custom Engine design](docs/design/custom-engine.md) for custom agent integration.
  267. ## User config
  268. skill-up auto-loads an optional user-level config that supplies default OpenTelemetry env vars and per-environment runtime kwargs. The embedded defaults are empty; downstream consumers maintain their own config file.
  269. ### Discovery chain (lowest to highest precedence)
  270. ```
  271. embed (empty) < user (~/.config/skill-up/config.yaml) < project ($PWD/.skill-up.yaml) < explicit (--config)
  272. ```
  273. | Source | Path |
  274. | ---------- | ------------------------------------------------------------------------------------------------------- |
  275. | `embed` | empty `Config{}` — no vendor defaults baked in |
  276. | `user` | `$SKILL_UP_CONFIG`, else `$XDG_CONFIG_HOME/skill-up/config.yaml`, else `~/.config/skill-up/config.yaml` |
  277. | `project` | `$PWD/.skill-up.yaml` |
  278. | `explicit` | `--config <path>` (must exist) |
  279. Missing files at the `user` and `project` layers are silently skipped; a missing `--config` path is a hard error. A corrupt config at any layer also fails the run.
  280. ### Quickstart
  281. ```bash
  282. skill-up init # writes a template to ~/.config/skill-up/config.yaml (XDG-aware)
  283. skill-up init --local # writes a template to $PWD/.skill-up.yaml
  284. skill-up init --print # prints the template to stdout
  285. skill-up init --force # overwrite an existing file
  286. skill-up init --config foo.yaml # reads foo.yaml, writes it to ~/.config/skill-up/config.yaml
  287. skill-up init --config foo.yaml --local # reads foo.yaml, writes it to $PWD/.skill-up.yaml
  288. ```
  289. With `--config <path>`, `init` reads that file (validating it as a skill-up
  290. config) and writes its raw bytes to the target — comments and formatting are
  291. preserved. Without `--config`, `init` writes a commented YAML template.
  292. ### Schema
  293. ```yaml
  294. schema_version: v1alpha1
  295. kind: SkillUpConfig
  296. telemetry:
  297. service_name: skill-up # OTEL_SERVICE_NAME
  298. traces_exporter: otlp # OTEL_TRACES_EXPORTER
  299. traces:
  300. endpoint: http://localhost:4317 # OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (4317 for grpc, 4318/v1/traces for http/protobuf)
  301. protocol: grpc # OTEL_EXPORTER_OTLP_TRACES_PROTOCOL (grpc | http/protobuf); skill-up defaults to grpc
  302. resource_attributes: # serialized into OTEL_RESOURCE_ATTRIBUTES
  303. deployment.environment: local
  304. verbose: false # if true, also enables OTEL_LOG_* payload capture
  305. env: # arbitrary defaults, applied only-if-unset
  306. OTEL_EXPORTER_OTLP_HEADERS: authorization=${OTLP_TOKEN}
  307. runtime_kwargs: # keyed by environment.type
  308. opensandbox:
  309. base_url: http://localhost:8080
  310. # extensions: '{}'
  311. ```
  312. ### Precedence
  313. For environment variables: any value already set in the process environment wins; the config only fills in missing keys.
  314. For `runtime_kwargs`: explicit `--runtime-kwarg` on `run` > `eval.yaml` `environment.kwargs` > user-config `runtime_kwargs[environment.type]`.
  315. ### Secrets
  316. Prefer `${ENV_VAR}` references inside the config file rather than baking secret literals. The redaction mechanism (`userconfig.Redact`) masks fields tagged `secret:"true"` when printing; currently no Config field carries the tag, but the mechanism is in place for future fields.
  317. ## Importing `evals.json`
  318. Use `skill-up import` to migrate an Anthropic-compatible `evals.json` into the YAML layout used by this repo:
  319. ```bash
  320. skill-up import ./evals/evals.json --output ./evals
  321. ```
  322. ## CLI Overview
  323. | Command | Description |
  324. | ------------------------------------ | ------------------------------------------- |
  325. | `skill-up run [path]` | Run evaluation cases and produce reports |
  326. | `skill-up validate [path]` | Validate `eval.yaml` and case files |
  327. | `skill-up list-cases [path]` | List all cases referenced by the config |
  328. | `skill-up compare <result> <result>...` | Compare multiple finished runs; export HTML, Markdown, JSON or text |
  329. | `skill-up report <result.json>` | Generate reports from a previous run |
  330. | `skill-up import <evals.json>` | Import Anthropic `evals.json` to YAML cases |
  331. | `skill-up debug judge <input.json>` | Debug judge module with a JSON input |
  332. | `skill-up debug report <input.json>` | Debug report module with a JSON input |
  333. ## GitHub Action
  334. Run your Agent Skill evals in CI on every pull request — and check the same skill
  335. **across engines** (`claude_code` / `codex` / `qodercli` / `qwen_code`) in one step. This repo
  336. ships an action at its root ([`action.yml`](action.yml)):
  337. ```yaml
  338. # .github/workflows/skill-eval.yml
  339. name: Skill Eval
  340. on:
  341. pull_request:
  342. paths: ['skills/**', 'evals/**', '**/SKILL.md']
  343. jobs:
  344. eval:
  345. runs-on: ubuntu-latest # Docker container action — Linux only
  346. steps:
  347. - uses: actions/checkout@v4
  348. - uses: alibaba/skill-up@main # see "Versioning" below
  349. with:
  350. engine: claude_code # or codex / qodercli / qwen_code; empty = let eval.yaml decide
  351. api-key: ${{ secrets.ANTHROPIC_API_KEY }}
  352. base-url: https://api.anthropic.com # your model endpoint
  353. skill-target: evals/eval.yaml
  354. ```
  355. Requirements for the caller: a **Linux** runner (it's a Docker container action),
  356. and your model credential stored as a repo secret. The runner image is public, so
  357. no extra registry auth is needed.
  358. Key inputs: `engine`, `model`, `provider`, `api-key`, `base-url`, `skill-target`,
  359. `parallelism`. The action prebuilds skill-up + the three engine CLIs into its
  360. runner image, so a run is just "pull image, eval". See [`action.yml`](action.yml)
  361. for the full input/output reference.
  362. ### Bundled skill-up version
  363. The container image includes a deliberately pinned skill-up CLI version. The
  364. version is not resolved from `latest` when a workflow starts, so a given image
  365. digest always runs the same CLI.
  366. The `skill-up-version` input is only a fallback for a custom image that does not
  367. already contain the `skill-up` binary. The official image contains the binary,
  368. so this input cannot override its bundled version. To find the effective
  369. version, check the `skill-up --version` line in the Action log.
  370. Publishing a new skill-up CLI release does not automatically update the GitHub
  371. Action image. Maintainers must synchronize the pinned version, publish and test
  372. a new runner image, and update the image digest in `action.yml`. The complete
  373. maintainer procedure is documented in the
  374. [CI maintenance runbook](docs/guide/ci-maintenance.md#manual-skill-up-version-synchronization).
  375. The production Action must use an immutable `sha256:` image digest. Do not
  376. replace it with `skill-up-runner:latest`; a mutable tag would allow existing
  377. Action references to change behavior without a repository commit.
  378. ### Versioning
  379. `uses:` points at any git ref that contains `action.yml`. Pin a **release tag**
  380. (the first release that includes the action onward) or a commit SHA for stability;
  381. `@main` always tracks the latest. Release tags published **before** the action was
  382. added do not contain `action.yml` and cannot be used as the ref.
  383. A CLI release tag captures the `action.yml` and runner-image digest that existed
  384. when that tag was created. Because the current runner image is refreshed
  385. manually after CLI release assets become available, do not assume that a CLI tag
  386. automatically contains an Action image with the same CLI version. Until a
  387. separate Action release tag process is introduced, use a post-refresh commit SHA
  388. for an immutable reference or `@main` when intentionally following Action
  389. updates.
  390. ## Project Structure
  391. ```text
  392. skill-up/
  393. ├── cmd/skill-up/ # CLI entry point
  394. ├── internal/ # Private implementation packages
  395. │ ├── cli/ # Commands and flags
  396. │ ├── evaluator/ # Case execution and judging orchestration
  397. │ ├── evalevent/ # Evaluation event model, lifecycle, publisher, and JSONL sink
  398. │ ├── runner/ # End-to-end run orchestration
  399. │ └── report/ # JSON, JUnit, HTML, and benchmark reports
  400. ├── pkg/ # Publicly importable APIs
  401. ├── plugins/ # Agent host plugin bundles
  402. │ ├── codex-skill-up/ # Codex adapter; release bundles add shared assets
  403. │ └── dsh-skill-up/ # DSH bundle with a generated dist/ Skill copy
  404. ├── schemas/ # Evaluation-event and host-neutral observation contracts
  405. ├── skills/skill-upper/ # Canonical source for the distributable workflow Skill
  406. ├── docs/ # VitePress documentation
  407. ├── e2e/ # End-to-end tests and fixtures
  408. └── examples/ # Example evaluations and fixtures
  409. ```
  410. See [CONTRIBUTING.md](CONTRIBUTING.md#directory-overview) for the complete
  411. directory map and architectural responsibilities.
  412. ## License
  413. Apache License 2.0 — see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.