Skip to content
Back to search

skillgrade

"Unit tests" for your agent skills

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mgechev--skillgrade/badge.svg)](/listings/mgechev--skillgrade)

HTML

<a href="/listings/mgechev--skillgrade"><img src="/api/listings/mgechev--skillgrade/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**Prerequisites**: Node.js 20+, Docker

npm i -g skillgrade

**1. Initialize** — go to your skill directory (must have `SKILL.md`) and scaffold:

cd my-skill/
GEMINI_API_KEY=your-key skillgrade init    # or ANTHROPIC_API_KEY / OPENAI_API_KEY
# Use --force to overwrite an existing eval.yaml

Generates `eval.yaml` with AI-powered tasks and graders. Without an API key, creates a well-commented template.

**2. Edit** — customize `eval.yaml` for your skill (see [eval.yaml Reference](#evalyaml-reference)).

**3. Run**:

GEMINI_API_KEY=your-key skillgrade --smoke

The agent is auto-detected from your API key: `GEMINI_API_KEY` → Gemini, `ANTHROPIC_API_KEY` → Claude, `OPENAI_API_KEY` → Codex. Override with `--agent=claude`.

**4. Review**:

skillgrade preview          # CLI report
skillgrade preview browser  # web UI → http://localhost:3847

Reports are saved to `$TMPDIR/skillgrade/<skill-name>/results/`. Override with `--output=DIR`.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 684 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Skillgrade
  2. An eval harness for AI coding agents. Define tasks, run any agent, score with deterministic scripts, LLM judges, or human reviewers.
  3. Works with Gemini CLI, Claude Code, Codex CLI, OpenCode, any [ACP](https://agentclientprotocol.com/)-compatible agent, or any command-line tool — with or without [Agent Skills](https://agentskills.io/home).
  4. See [examples/](examples/) — [superlint](examples/superlint/) (simple) and [angular-modern](examples/angular-modern/) (TypeScript grader).
  5. ![Browser Preview](https://raw.githubusercontent.com/mgechev/skillgrade/main/assets/browser-preview.png)
  6. ## Quick Start
  7. **Prerequisites**: Node.js 20+, Docker
  8. ```bash
  9. npm i -g skillgrade
  10. ```
  11. **1. Initialize** — scaffold an `eval.yaml` in any directory:
  12. ```bash
  13. cd my-evals/
  14. skillgrade init
  15. ```
  16. If a `SKILL.md` is present and an API key is set, `init` generates AI-powered tasks tailored to the skill. Otherwise, it creates a ready-to-edit template.
  17. **2. Edit** — define your tasks in `eval.yaml` (see [eval.yaml Reference](#evalyaml-reference)).
  18. **3. Run**:
  19. ```bash
  20. GEMINI_API_KEY=your-key skillgrade --smoke
  21. ```
  22. The agent is auto-detected from your API key: `GEMINI_API_KEY` → Gemini, `ANTHROPIC_API_KEY` → Claude, `OPENAI_API_KEY` → Codex. Override with `--agent=claude`.
  23. **4. Review**:
  24. ```bash
  25. skillgrade preview # CLI report
  26. skillgrade preview browser # web UI → http://localhost:3847
  27. ```
  28. Reports are saved to `$TMPDIR/skillgrade/<eval-name>/results/`. Override with `--output=DIR`.
  29. ## Presets
  30. | Flag | Trials | Use Case |
  31. |------|--------|----------|
  32. | `--smoke` | 5 | Quick capability check |
  33. | `--reliable` | 15 | Reliable pass rate estimate |
  34. | `--regression` | 30 | High-confidence regression detection |
  35. ## Options
  36. | Flag | Description |
  37. |------|-------------|
  38. | `--eval=NAME[,NAME]` | Run specific evals by name (comma-separated) |
  39. | `--grader=TYPE` | Run only graders of a type (`deterministic`, `llm_rubric`, or `human`) |
  40. | `--trials=N` | Override trial count |
  41. | `--parallel=N` | Run trials concurrently |
  42. | `--agent=gemini\|claude\|codex\|acp\|opencode\|command` | Override agent (default: auto-detect from API key) |
  43. | `--model=NAME` | Model the agent answers with (`gemini`, `claude`, `codex`, `opencode`). Default: whatever the agent CLI is configured to use |
  44. | `--provider=docker\|local` | Override provider |
  45. | `--acp-command=CMD` | ACP agent command (e.g., `gemini --acp`) |
  46. | `--command=CMD` | Command to run for the `command` agent (e.g., `node mycli.js`) |
  47. | `--opencode-agent=NAME` | OpenCode agent (build\|plan\|explore) |
  48. | `--opencode-model=MODEL` | OpenCode model (provider/model format) |
  49. | `--output=DIR` | Output directory (default: `$TMPDIR/skillgrade`) |
  50. | `--format=FORMAT` | Export results (`json`, `junit`, `csv`, `markdown` — comma-separate for multiple) |
  51. | `--validate` | Verify graders using reference solutions |
  52. | `--ci` | CI mode: exit non-zero if below threshold |
  53. | `--threshold=0.8` | Pass rate threshold for CI mode |
  54. | `--preview` | Show CLI results after running |
  55. ## eval.yaml Reference
  56. ```yaml
  57. version: "1"
  58. # Optional: path to a skill directory. Omit for non-skill evals.
  59. # skill: path/to/my-skill
  60. defaults:
  61. agent: gemini # gemini | claude | codex | acp | opencode | command
  62. model: claude-opus-5 # model the agent answers with (gemini, claude, codex, opencode)
  63. provider: docker # docker | local
  64. trials: 5
  65. timeout: 300 # seconds
  66. threshold: 0.8 # for --ci mode
  67. grader_model: gemini-3-flash-preview # default LLM grader model
  68. grader_provider: gemini # default LLM grader provider: gemini | anthropic | openai
  69. command: node mycli.js # command to run when agent is 'command' (see Custom Command Agent)
  70. acp: # ACP agent configuration (optional)
  71. command: gemini --acp # command to start ACP-compatible agent
  72. env: # optional environment variables
  73. DEBUG: "1"
  74. docker:
  75. base: node:20-slim
  76. setup: | # extra commands run during image build
  77. apt-get update && apt-get install -y jq
  78. environment: # container resource limits
  79. cpus: 2
  80. memory_mb: 2048
  81. tasks:
  82. - name: fix-linting-errors
  83. instruction: |
  84. Use the superlint tool to fix coding standard violations in app.js.
  85. workspace: # files copied into the container
  86. - src: fixtures/broken-app.js
  87. dest: app.js
  88. - src: bin/superlint
  89. dest: /usr/local/bin/superlint
  90. chmod: "+x"
  91. context: # reference material for the agent
  92. - src: specs/api-schema.yaml # (not scored — just gives the agent context)
  93. dest: api-schema.yaml
  94. graders:
  95. - type: deterministic
  96. setup: npm install typescript # grader-specific deps (optional)
  97. run: npx ts-node graders/check.ts
  98. weight: 0.7
  99. - type: llm_rubric
  100. rubric: |
  101. Did the agent follow the check → fix → verify workflow?
  102. provider: gemini # optional: gemini (default) | anthropic | openai
  103. model: gemini-3.5-flash # optional model override
  104. weight: 0.3
  105. # Per-task overrides (optional)
  106. agent: claude
  107. model: claude-sonnet-5 # override the model for this task only
  108. grader_provider: anthropic # override default LLM grader provider
  109. trials: 10
  110. timeout: 600
  111. ```
  112. String values (`instruction`, `rubric`, `run`) support **file references** — if the value is a valid file path, its contents are read automatically:
  113. ```yaml
  114. instruction: instructions/fix-linting.md
  115. rubric: rubrics/workflow-quality.md
  116. ```
  117. ### Workspace vs Context
  118. Both `workspace` and `context` copy files into the agent's working directory, but they serve different roles:
  119. - **`workspace`** — evaluation fixtures. These are the files the agent works on and graders check against (source code, configs, test inputs).
  120. - **`context`** — reference material. Docs, specs, or schemas that shape how the agent approaches the task but aren't directly graded.
  121. ```yaml
  122. tasks:
  123. - name: implement-api
  124. workspace:
  125. - src: fixtures/app.js # the file the agent edits
  126. dest: app.js
  127. context:
  128. - src: specs/openapi.yaml # reference the agent can read
  129. dest: openapi.yaml
  130. ```
  131. ### Multi-turn evals
  132. A task can define `turns` instead of a single `instruction`. Each turn has its own instruction and optional graders. The agent's workspace persists across turns — turn 2 sees everything turn 1 produced.
  133. ```yaml
  134. tasks:
  135. - name: plan-then-implement
  136. turns:
  137. - instruction: |
  138. Read the codebase and propose a refactoring plan. Write it to plan.md.
  139. graders:
  140. - type: llm_rubric
  141. rubric: Is the plan well-structured and comprehensive?
  142. weight: 0.3
  143. - instruction: |
  144. Implement the refactoring plan you wrote in plan.md.
  145. graders:
  146. - type: deterministic
  147. run: bash graders/check-refactor.sh
  148. weight: 0.7
  149. workspace:
  150. - src: fixtures/legacy-app/
  151. dest: .
  152. ```
  153. A task must have either `instruction` or `turns` — never both. Grader results from all turns are aggregated into the trial's final reward.
  154. Agents that implement the optional `runTurn()` method receive the turn index and all previous outputs for context. Agents without it fall back to the standard `run()` on each turn.
  155. ### Dataset / batch support
  156. Expand a single task template into many tasks by loading rows from a JSONL or CSV file:
  157. ```yaml
  158. tasks:
  159. - name: code-review
  160. dataset: datasets/review-cases.jsonl
  161. instruction: |
  162. Review this code and identify bugs:
  163. ```
  164. {{code}}
  165. ```
  166. graders:
  167. - type: deterministic
  168. run: node graders/check-review.mjs
  169. weight: 1.0
  170. ```
  171. `datasets/review-cases.jsonl`:
  172. ```json
  173. {"code": "if (x = 5) { ... }", "expected_issues": "assignment in condition"}
  174. {"code": "arr[arr.length]", "expected_issues": "off-by-one"}
  175. ```
  176. This expands to `code-review/0`, `code-review/1`, etc. Template variables use `{{key}}` syntax — each key in the row is interpolated into the instruction. Each row's full data becomes the task's `expected` field, so graders can access it via `SKILLGRADE_INPUT`.
  177. CSV files are also supported — the first row is treated as headers.
  178. ### Reference output and metadata
  179. A task can carry the row's answer key and a set of labels:
  180. ```yaml
  181. - name: easy--tooltip-token
  182. instruction: |
  183. Report the background colour token of the Tooltip's default variant.
  184. Write {"token": "..."} to answer.json.
  185. expected: # the answer key — graders only, never the agent
  186. token: Brand/100
  187. variants: [default, hover]
  188. metadata: # labels for --filter, recorded with the results
  189. tier: easy
  190. form: open
  191. tags: [smoke]
  192. graders:
  193. - type: deterministic
  194. run: node graders/check-token.mjs
  195. ```
  196. Both are optional: a task without `expected` is scored purely on what its
  197. graders measure, so golden-truth and metric-style tasks live in the same suite.
  198. skillgrade **delivers `expected`, it never interprets it** — comparison is the
  199. grader's job. A deterministic grader receives the task context as one JSON
  200. document in `SKILLGRADE_INPUT`, so `expected` keeps its structure:
  201. ```js
  202. // graders/check-token.mjs — one script for every token task
  203. const { task, trial, expected, metadata } = JSON.parse(process.env.SKILLGRADE_INPUT);
  204. const answer = JSON.parse(fs.readFileSync('answer.json', 'utf8')); // cwd is the workspace
  205. console.log(JSON.stringify({
  206. score: answer.token === expected.token ? 1 : 0,
  207. details: `${task}: want ${expected.token}, got ${answer.token ?? '(none)'}`,
  208. }));
  209. ```
  210. ```bash
  211. # or from a shell grader
  212. want=$(jq -r .expected.token <<< "$SKILLGRADE_INPUT")
  213. ```
  214. An `llm_rubric` grader gets `expected` as an `## Expected Output` section in its
  215. prompt. Both channels are built after the agent process has exited, and neither
  216. writes to the workspace — unlike `run:` and `rubric:`, which are staged into the
  217. agent's working directory before it starts, so keep answer keys out of those.
  218. ### Selecting which tasks to run
  219. `metadata` is what `--filter` selects on. Values are OR within a key and AND
  220. across keys; `--filter` and `--not-filter` are repeatable:
  221. ```bash
  222. skillgrade --filter=tier=easy,medium # easy OR medium
  223. skillgrade --filter=tier=hard --filter=form=refuse # hard AND refuse
  224. skillgrade --filter=tags=smoke --not-filter=tags=flaky
  225. skillgrade --filter-pattern='^easy--' # regex over task names
  226. skillgrade --filter=tier=easy --list # print the selection, run nothing
  227. ```
  228. A filter on a key that no task declares is an error rather than a silent
  229. match-everything — `--filter=teir=easy` should not quietly run the whole suite.
  230. Filters work the same whether the tasks are inline or imported.
  231. ### Splitting eval.yaml across files
  232. Any section can live in another file. `$import` takes a file, a directory (every
  233. `.yaml`/`.yml` inside it, sorted), a glob, or a list of those:
  234. ```yaml
  235. version: "1"
  236. defaults:
  237. $import: shared/defaults.yaml # merged in place — keys below win
  238. trials: 3
  239. tasks:
  240. - $import: evals/easy/*.yaml # one task per file, or a file holding a list
  241. - $import: evals/hard # a whole directory
  242. trials: 10 # applied to every task it imports
  243. - name: still-inline # inline tasks keep working
  244. instruction: ...
  245. graders: [...]
  246. ```
  247. Each imported file is a normal YAML document — a single task, a list of tasks, or
  248. an object for a section like `defaults`. Imported files may import further files;
  249. paths are relative to the file that contains the `$import`, and cycles are an error.
  250. A task keeps working when you move it into its own file: **its relative paths
  251. resolve against its own directory first, then the eval root.** So
  252. `evals/easy/one.yaml` can say `instruction: instruction.md` for the file next to it
  253. while still pointing `run: node graders/check.mjs` at the shared graders directory
  254. at the root.
  255. ## Graders
  256. Skillgrade ships with three grader types. Mix and weight them to score any dimension — correctness, code quality, workflow compliance, human judgment.
  257. ### Deterministic
  258. Runs a command and parses JSON from stdout:
  259. ```yaml
  260. - type: deterministic
  261. run: bash graders/check.sh
  262. weight: 0.7
  263. ```
  264. Output format:
  265. ```json
  266. {
  267. "score": 0.67,
  268. "details": "2/3 checks passed",
  269. "checks": [
  270. {"name": "file-created", "passed": true, "message": "Output file exists"},
  271. {"name": "content-correct", "passed": false, "message": "Missing expected output"}
  272. ]
  273. }
  274. ```
  275. `score` (0.0–1.0) and `details` are required. `checks` is optional.
  276. **Bash example:**
  277. ```bash
  278. #!/bin/bash
  279. passed=0; total=2
  280. c1_pass=false c1_msg="File missing"
  281. c2_pass=false c2_msg="Content wrong"
  282. if test -f output.txt; then
  283. passed=$((passed + 1)); c1_pass=true; c1_msg="File exists"
  284. fi
  285. if grep -q "expected" output.txt 2>/dev/null; then
  286. passed=$((passed + 1)); c2_pass=true; c2_msg="Content correct"
  287. fi
  288. score=$(awk "BEGIN {printf \"%.2f\", $passed/$total}")
  289. echo "{\"score\":$score,\"details\":\"$passed/$total passed\",\"checks\":[{\"name\":\"file\",\"passed\":$c1_pass,\"message\":\"$c1_msg\"},{\"name\":\"content\",\"passed\":$c2_pass,\"message\":\"$c2_msg\"}]}"
  290. ```
  291. > Use `awk` for arithmetic — `bc` is not available in `node:20-slim`.
  292. ### LLM Rubric
  293. Evaluates the agent's session transcript against qualitative criteria:
  294. ```yaml
  295. - type: llm_rubric
  296. rubric: |
  297. Workflow Compliance (0-0.5):
  298. - Did the agent follow the mandatory 3-step workflow?
  299. Efficiency (0-0.5):
  300. - Completed in ≤5 commands?
  301. weight: 0.3
  302. provider: gemini # gemini (default) | anthropic | openai
  303. model: gemini-2.0-flash # optional, auto-detected from API key
  304. ```
  305. The `provider` field selects which LLM API to call:
  306. | Provider | API Key Env Var | Base URL Env Var (optional) | Default Model |
  307. |------------|---------------------|-----------------------------|----------------------------|
  308. | `gemini` | `GEMINI_API_KEY` | - | Dynamically resolved latest Flash model (via API) |
  309. | `anthropic`| `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | Dynamically resolved latest Haiku model (via API) |
  310. | `openai` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | Dynamically resolved latest Mini/Flash model (via API) |
  311. `ANTHROPIC_BASE_URL` and `OPENAI_BASE_URL` enable custom/self-hosted endpoints (Ollama, vLLM, etc.). They apply to both LLM grading and `skillgrade init`.
  312. ### Human
  313. Pauses the eval for manual review. The grader writes a context file with the agent's output and waits for a reviewer to score it:
  314. ```yaml
  315. - type: human
  316. prompt: |
  317. Does the generated UI match the design spec?
  318. Check layout, colors, and responsive behavior.
  319. weight: 0.5
  320. ```
  321. When a human grader runs, it:
  322. 1. Writes the instruction, agent output, and commands to `<workspace>/.review/<task>-<trial>.json`
  323. 2. Prints a banner with the file path and scoring instructions
  324. 3. Polls for `<workspace>/.review/<task>-<trial>.result.json`
  325. The reviewer writes a standard grader result:
  326. ```json
  327. {"score": 0.8, "details": "Layout matches, colors are off on mobile"}
  328. ```
  329. Useful for evaluating subjective quality — design fidelity, UX, documentation clarity — that automated checks can't reliably measure.
  330. ### Combining Graders
  331. ```yaml
  332. graders:
  333. - type: deterministic
  334. run: bash graders/check.sh
  335. weight: 0.7 # 70% — did it work?
  336. - type: llm_rubric
  337. rubric: rubrics/quality.md
  338. weight: 0.2 # 20% — was the approach good?
  339. - type: human
  340. prompt: Is the output production-ready?
  341. weight: 0.1 # 10% — human sign-off
  342. ```
  343. Final reward = `Σ (grader_score × weight) / Σ weight`
  344. ## Extending Skillgrade
  345. All four extension points use the same registry pattern — register a factory function, use it by name:
  346. ```typescript
  347. import {
  348. registerGrader, registerAgent, registerProvider, registerReporter,
  349. Grader, Reporter
  350. } from 'skillgrade';
  351. // Custom grader
  352. class MyGrader implements Grader {
  353. async grade(workspace, provider, config, taskPath, sessionLog) {
  354. return { grader_type: 'my_grader', score: 0.9, weight: config.weight, details: 'Passed' };
  355. }
  356. }
  357. registerGrader('my_grader', () => new MyGrader());
  358. // Custom agent
  359. registerAgent('my-agent', (config) => new MyAgent(config));
  360. // Custom provider (e.g., SSH, E2B, cloud VMs)
  361. registerProvider('ssh', (config) => new SSHProvider(config));
  362. // Custom reporter
  363. class DashboardReporter implements Reporter {
  364. async report(results, outputDir) {
  365. // push to your dashboard
  366. return path.join(outputDir, 'dashboard.json');
  367. }
  368. }
  369. registerReporter('dashboard', () => new DashboardReporter());
  370. ```
  371. ## Report Formats
  372. Export results in multiple formats — use one or combine them:
  373. ```bash
  374. skillgrade --smoke --format=json # JSON summary
  375. skillgrade --smoke --format=junit # JUnit XML for CI dashboards
  376. skillgrade --smoke --format=csv # CSV for spreadsheets
  377. skillgrade --smoke --format=markdown # Markdown report
  378. skillgrade --smoke --format=json,junit,markdown # all three at once
  379. ```
  380. | Format | Output File | Use Case |
  381. |--------|-------------|----------|
  382. | `json` | `summary.json` | Dashboards, trend tracking, programmatic access |
  383. | `junit` | `junit.xml` | CI systems (GitHub Actions, Jenkins, CircleCI) |
  384. | `csv` | `results.csv` | Spreadsheet analysis, data pipelines |
  385. | `markdown` | `report.md` | Documentation, PR comments, Slack posts |
  386. ## CI Integration
  387. Use `--provider=local` in CI — the runner is already an ephemeral sandbox, so Docker adds overhead without benefit.
  388. ```yaml
  389. # .github/workflows/skillgrade.yml
  390. - run: |
  391. npm i -g skillgrade
  392. cd evals/
  393. GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }} skillgrade --regression --ci --provider=local
  394. ```
  395. Exits with code 1 if pass rate falls below `--threshold` (default: 0.8).
  396. > **Tip**: Use `docker` (the default) for local development to protect your machine. In CI, `local` is faster and simpler.
  397. ## Environment Variables
  398. | Variable | Used by |
  399. |----------|---------|
  400. | `GEMINI_API_KEY` | Agent execution, LLM grading (`provider: gemini`), `skillgrade init` |
  401. | `ANTHROPIC_API_KEY` | Agent execution, LLM grading (`provider: anthropic`), `skillgrade init` |
  402. | `OPENAI_API_KEY` | Agent execution (Codex), LLM grading (`provider: openai`), `skillgrade init` |
  403. | `ANTHROPIC_BASE_URL` | LLM grading (`provider: anthropic`), `skillgrade init` — custom Anthropic-compatible endpoint |
  404. | `OPENAI_BASE_URL` | LLM grading (`provider: openai`), `skillgrade init` — custom OpenAI-compatible endpoint (Ollama, vLLM, etc.) |
  405. | `GEMINI_MODEL` | Override the default model used for Gemini LLM grading (defaults to dynamic API lookup; throws if resolution fails) |
  406. | `INIT_GEMINI_MODEL` | Override the model used for Gemini in `skillgrade init` (defaults to `GEMINI_MODEL` or dynamic API lookup; throws if resolution fails) |
  407. | `ANTHROPIC_MODEL` | Override the default model used for Anthropic LLM grading (defaults to dynamic API lookup; throws if resolution fails) |
  408. | `INIT_ANTHROPIC_MODEL` | Override the model used for Anthropic in `skillgrade init` (defaults to `ANTHROPIC_MODEL` or dynamic API lookup; throws if resolution fails) |
  409. | `OPENAI_MODEL` | Override the default model used for OpenAI LLM grading (defaults to dynamic API lookup; throws if resolution fails) |
  410. | `INIT_OPENAI_MODEL` | Override the model used for OpenAI in `skillgrade init` (defaults to `OPENAI_MODEL` or dynamic API lookup; throws if resolution fails) |
  411. Variables are also loaded from `.env` in the working directory. Shell values override `.env`. All values are **redacted** from persisted session logs.
  412. ## Custom Command Agent
  413. Point skillgrade at **any command** — a custom script, an orchestrator over the Claude/OpenAI SDKs, or any CLI tool — without forking the package or implementing an ACP server.
  414. ### Quick Start
  415. ```bash
  416. skillgrade --agent=command --command="node mycli.js"
  417. ```
  418. Or in `eval.yaml`:
  419. ```yaml
  420. defaults:
  421. agent: command
  422. command: "node mycli.js"
  423. provider: local # run on the host; or use docker + docker.setup to install your CLI
  424. ```
  425. `command` can also be set per task to override the default.
  426. ### How the instruction reaches your command
  427. The task instruction is **piped to your command's stdin** (skillgrade writes it to `/tmp/.prompt.md`, then runs `cat /tmp/.prompt.md | <command>` inside the workspace directory). If your CLI takes the prompt as an argument instead, wrap it in a one-line script that reads stdin.
  428. Your command runs in the workspace and is free to read/edit files there — graders score the resulting workspace state (and any live checks), not your command's stdout, so any agent slots in cleanly.
  429. ### Docker vs local
  430. - **`provider: local`** is the simplest fit for a custom agent: your command runs on the host with your tools already installed.
  431. - **`provider: docker`** still works — skillgrade does **not** auto-install anything for the `command` agent, so install your CLI and dependencies via `docker.setup`:
  432. ```yaml
  433. defaults:
  434. agent: command
  435. command: "mycli run"
  436. docker:
  437. base: node:20-slim
  438. setup: "npm install -g my-cli-package"
  439. ```
  440. ## OpenCode Agent
  441. [OpenCode](https://opencode.ai/) is an AI coding agent that supports multiple AI models and specialized subagents.
  442. ### Quick Start
  443. ```bash
  444. # Use OpenCode with default agent and model
  445. skillgrade --agent=opencode
  446. # Specify OpenCode agent (build|plan|explore)
  447. skillgrade --agent=opencode --opencode-agent=build
  448. # Specify both agent and model (provider/model format)
  449. skillgrade --agent=opencode --opencode-agent=build --opencode-model=anthropic/claude-sonnet-4-20250514
  450. ```
  451. ### OpenCode Agents
  452. | Agent | Description |
  453. |-------|-------------|
  454. | `build` | Default primary agent with full tool access |
  455. | `plan` | Read-only planning/analysis agent |
  456. | `explore` | Fast codebase exploration agent |
  457. ### OpenCode Models
  458. Models are specified in `provider/model` format:
  459. | Model | Format |
  460. |-------|--------|
  461. | Claude Sonnet 4 | `anthropic/claude-sonnet-4-20250514` |
  462. | GPT 5.1 Codex | `opencode/gpt-5.1-codex` |
  463. ### CLI Options
  464. | Flag | Description |
  465. |------|-------------|
  466. | `--agent=opencode` | Use OpenCode agent |
  467. | `--opencode-agent=NAME` | OpenCode agent (build\|plan\|explore) |
  468. | `--opencode-model=MODEL` | OpenCode model (provider/model format) |
  469. ### How It Works
  470. 1. skillgrade invokes OpenCode CLI with `opencode run`
  471. 2. Passes instruction via temp file to avoid shell escaping issues
  472. 3. Supports both agent and model specification
  473. 4. Works with `--provider=docker` or `--provider=local`
  474. ## ACP Agent
  475. [Agent Client Protocol (ACP)](https://agentclientprotocol.com/) is an open protocol that standardizes communication between AI coding agents and clients. Using an ACP-compatible agent allows you to evaluate agents without managing API keys directly.
  476. ### Quick Start
  477. ```bash
  478. # Use Gemini CLI in ACP mode (requires gemini CLI installed)
  479. skillgrade --agent=acp --acp-command="gemini --acp"
  480. # Or configure in eval.yaml
  481. ```
  482. ```yaml
  483. defaults:
  484. agent: acp
  485. acp:
  486. command: gemini --acp
  487. ```
  488. ### ACP-Compatible Agents
  489. Any agent that supports the ACP protocol can be used:
  490. | Agent | Command |
  491. |-------|---------|
  492. | Gemini CLI | `gemini --acp` |
  493. | Other ACP agents | Check agent documentation |
  494. ### How It Works
  495. 1. skillgrade starts the ACP agent as a subprocess
  496. 2. Communication happens via JSON-RPC 2.0 over stdio
  497. 3. No API key required — authentication is handled by the ACP agent
  498. 4. Works best with `--provider=local` since the ACP agent needs to be available in your environment
  499. ### CLI Options
  500. | Flag | Description |
  501. |------|-------------|
  502. | `--agent=acp` | Use ACP-compatible agent |
  503. | `--acp-command=CMD` | Command to start the ACP agent |
  504. The `--acp-command` can also be set in `eval.yaml` under `defaults.acp.command`.
  505. ## Best Practices
  506. - **Grade outcomes, not steps.** Check that the file was fixed, not that the agent ran a specific command.
  507. - **Instructions must name output files.** If the grader checks for `output.html`, the instruction must tell the agent to save as `output.html`.
  508. - **Validate graders first.** Use `--validate` with a reference solution before running real evals.
  509. - **Start small.** 3–5 well-designed tasks beat 50 noisy ones.
  510. - **Combine grader types.** Deterministic for correctness, LLM for quality, human for subjective judgment — weight each to match what matters.
  511. For a comprehensive guide on writing high-quality skills, check out [skills-best-practices](https://github.com/mgechev/skills-best-practices/). You can also install the skill creator skill to help author skills:
  512. ```bash
  513. npx skills add mgechev/skills-best-practices
  514. ```
  515. ## License
  516. MIT
  517. ---
  518. *Inspired by [SkillsBench](https://arxiv.org/html/2602.12670v1) and [Demystifying Evals for AI Agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents).*

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.