Skip to content
Back to search

agent-skills-eval

A test runner for agentskills.io-style AI agent skills

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/darkrishabh/agent-skills-eval with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/darkrishabh/agent-skills-eval'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/darkrishabh--agent-skills-eval/badge.svg)](/listings/darkrishabh--agent-skills-eval)

HTML

<a href="/listings/darkrishabh--agent-skills-eval"><img src="/api/listings/darkrishabh--agent-skills-eval/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

npx agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge gpt-4o-mini \
  --baseline \
  --strict

That's it. Point it at a folder of skills, give it a target model and a judge model, and it produces a workspace with full artifacts and a static HTML report.

agent-skills-workspace/
└── iteration-1/
    ├── meta.json            # run metadata
    ├── benchmark.json       # rolled-up pass/fail per skill
    ├── eval-basic/
    │   ├── with_skill/      # output, timing, judge grading
    │   └── without_skill/   # ↑ same, with the skill stripped
    └── report/
        └── index.html       # the visual report

Open `iteration-1/report/index.html` and you have a real, evidence-backed answer to "is my skill working?"

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 452 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. <img src="https://github.com/user-attachments/assets/094b8e11-e19e-4c96-ae82-ba701cfcf7e3" alt="agent-skills-eval — a test runner for Agent Skills" width="100%" />
  3. <br />
  4. # agent-skills-eval
  5. **A test runner for [Agent Skills](https://agentskills.io).**
  6. Write a `SKILL.md`, drop in some evals, and find out — empirically — whether your skill actually makes the model better at the task.
  7. [![npm version](https://img.shields.io/npm/v/agent-skills-eval.svg?style=flat-square&logo=npm&label=npm)](https://www.npmjs.com/package/agent-skills-eval)
  8. [![CI](https://img.shields.io/github/actions/workflow/status/darkrishabh/agent-skills-eval/ci.yml?style=flat-square&logo=github&label=ci)](https://github.com/darkrishabh/agent-skills-eval/actions/workflows/ci.yml)
  9. [![license: MIT](https://img.shields.io/badge/license-MIT-green?style=flat-square)](LICENSE)
  10. [![node](https://img.shields.io/node/v/agent-skills-eval.svg?style=flat-square&logo=nodedotjs&logoColor=white)](package.json)
  11. [![docs](https://img.shields.io/badge/docs-GitHub%20Pages-0f766e?style=flat-square)](https://darkrishabh.github.io/agent-skills-eval/)
  12. [![TypeScript](https://img.shields.io/badge/TypeScript-3178C6?style=flat-square&logo=typescript&logoColor=white)](https://www.typescriptlang.org/)
  13. [Documentation](https://darkrishabh.github.io/agent-skills-eval/) · [Quickstart](#quickstart) · [SDK](#sdk) · [agentskills.io](https://agentskills.io)
  14. </div>
  15. ---
  16. ## Why this exists
  17. [Agent Skills](https://agentskills.io) — the open standard from Anthropic for giving agents domain knowledge — make it easy to ship a `SKILL.md` and assume your agent is now better at the task. The hard part is *proving* it.
  18. `agent-skills-eval` is the missing piece. It runs your skill against the same prompts twice — once `with_skill` loaded into context, once `without_skill` (baseline) — has a judge model grade both outputs, and gives you a side-by-side report. If the skill doesn't make a measurable difference, you'll see it. If it does, you have receipts.
  19. It's the test framework for the Agent Skills ecosystem, separated from any specific agent runtime so it works wherever your skills do.
  20. ## Quickstart
  21. ```bash
  22. npx agent-skills-eval ./skills \
  23. --target gpt-4o-mini \
  24. --judge gpt-4o-mini \
  25. --baseline \
  26. --strict
  27. ```
  28. That's it. Point it at a folder of skills, give it a target model and a judge model, and it produces a workspace with full artifacts and a static HTML report.
  29. ```text
  30. agent-skills-workspace/
  31. └── iteration-1/
  32. ├── meta.json # run metadata
  33. ├── benchmark.json # rolled-up pass/fail per skill
  34. ├── eval-basic/
  35. │ ├── with_skill/ # output, timing, judge grading
  36. │ └── without_skill/ # ↑ same, with the skill stripped
  37. └── report/
  38. └── index.html # the visual report
  39. ```
  40. Open `iteration-1/report/index.html` and you have a real, evidence-backed answer to "is my skill working?"
  41. ## What you get
  42. | | |
  43. |---|---|
  44. | **`with_skill` vs `without_skill`** | Every eval runs both ways so you can see the actual lift from the skill — or its absence. |
  45. | **Judge-graded outputs** | Use any chat model as a judge. Pass/fail with cited assertions, not vibes. |
  46. | **TypeScript SDK + CLI** | One-liner CLI for CI, full SDK for custom pipelines, custom providers, and dashboards. |
  47. | **OpenAI-compatible by default** | Works out of the box with OpenAI, Together, Groq, Anthropic via OpenAI-compat layers, local Llama servers — anything that speaks the OpenAI chat API. |
  48. | **Tool-call assertions** | Deterministic checks for agents that call tools, not just generate text. |
  49. | **Portable artifacts** | JSON + JSONL all the way down. Run today, diff tomorrow. Plug into your own dashboard. |
  50. | **Static HTML reports** | A drop-in report site you can publish anywhere — no infrastructure. |
  51. | **Fully spec-compliant** | Implements the full [agentskills.io specification](https://agentskills.io/specification): `SKILL.md` validation, `evals/evals.json`, official `iteration-N` artifact layout, frontmatter rules. |
  52. ## Install
  53. ```bash
  54. npm install agent-skills-eval
  55. ```
  56. Or run directly without installing:
  57. ```bash
  58. npx agent-skills-eval --help
  59. ```
  60. ## How it works
  61. The mental model is straightforward. For every eval defined in your skill:
  62. ```
  63. ┌─────────────────────────────┐
  64. │ same prompt │
  65. └───────────────┬─────────────┘
  66. │
  67. ┌───────────────┴─────────────┐
  68. ▼ ▼
  69. ┌──────────────┐ ┌──────────────┐
  70. │ with_skill │ │without_skill │
  71. │ SKILL.md in │ │ baseline, │
  72. │ context │ │ no skill │
  73. └──────┬───────┘ └──────┬───────┘
  74. │ │
  75. ▼ ▼
  76. target model target model
  77. │ │
  78. ▼ ▼
  79. output output
  80. │ │
  81. └──────────┬──────────────────┘
  82. ▼
  83. ┌─────────────┐
  84. │ judge │ scores both against
  85. │ model │ the same assertions
  86. └──────┬──────┘
  87. ▼
  88. pass / fail per side
  89. ```
  90. The judge sees the eval's `expected_output` and `assertions` and grades each side independently. The `--baseline` flag is what enables the comparison; without it you only get the `with_skill` run.
  91. ## YAML config
  92. For anything beyond a quick command, drop a config file at the root of your project:
  93. ```yaml
  94. # agent-skills-eval.yaml
  95. root: ./skills
  96. workspace: ./agent-skills-workspace
  97. baseline: true
  98. target: gpt-4o-mini
  99. judge: gpt-4o-mini
  100. baseUrl: https://api.openai.com/v1
  101. apiKeyEnv: OPENAI_API_KEY
  102. include:
  103. - "skills/**"
  104. exclude:
  105. - "**/draft-*"
  106. evalIds:
  107. - "basic"
  108. concurrency: 4
  109. layout: iteration
  110. strict: true
  111. report:
  112. enabled: true
  113. title: Agent Skills Report
  114. logging:
  115. format: pretty # pretty | jsonl | silent
  116. verbose: false
  117. color: auto
  118. targetParams:
  119. temperature: 0
  120. judgeParams:
  121. temperature: 0
  122. ```
  123. ```bash
  124. OPENAI_API_KEY=... npx agent-skills-eval --config agent-skills-eval.yaml
  125. ```
  126. CLI flags always override config values.
  127. ### Options reference
  128. These are configuration-file options and CLI defaults. See `--help` for the
  129. available CLI flags; `targetParams`, `judgeParams`, and `logging.snippetLength`
  130. are config-only. Logging flags use `--log-format`, `--log-file`, `--verbose`,
  131. and `--no-color`; report flags use `--report`, `--no-report`, `--report-title`,
  132. and `--report-output`. Supplied CLI flags override matching config values.
  133. | Option | Type | Default | What it does |
  134. |---|---|---|---|
  135. | `root` | string | `.` | Directory scanned recursively for `SKILL.md` files. Positional CLI arg. |
  136. | `workspace` | string | `./agent-skills-workspace` | Output directory for run artifacts (per-eval outputs, `grading.json`, `benchmark.json`, `meta.json`). |
  137. | `baseline` | boolean | `false` | When `true`, runs each eval both `with_skill` and `without_skill` so the report shows the lift the skill provides. When `false`, only `with_skill` runs. |
  138. | `target` | string | `gpt-4o-mini` | Model under evaluation — the one the skill is meant to help. |
  139. | `judge` | string | value of `target` | Model that grades the rubric assertions. Set it to a stronger model than `target` for more reliable grading. |
  140. | `baseUrl` | string | `OPENAI_BASE_URL` env, else **required** | Base URL of the OpenAI-compatible API for both target and judge. |
  141. | `apiKeyEnv` | string | `OPENAI_API_KEY` | Name of the environment variable holding the API key (the key itself is never written to config). |
  142. | `include` | string[] | all discovered skills | Glob(s) matched against each skill's path; only matching skills run. Repeatable `--include` on the CLI. |
  143. | `exclude` | string[] | none | Glob(s) matched against each skill's path; matching skills are skipped. Applied after `include`. Repeatable `--exclude`. |
  144. | `evalIds` | string[] | all cases | Select IDs within every selected skill. Repeat `--eval-id` on the CLI; SDK also accepts numeric IDs. Missing IDs fail the run. |
  145. | `concurrency` | number | `4` | Number of eval cases run in parallel. Must be a positive integer. |
  146. | `strict` | boolean | `false` | Validate each `SKILL.md` against the agentskills.io schema before running; fail on violations. |
  147. | `layout` | `flat` \| `iteration` | `iteration` | `iteration` allocates `iteration-N/` locally; with `CI=true`, it resets `iteration-1/`; `flat` writes directly into the workspace, overwriting the previous run. |
  148. | `report` | boolean \| object | `true` | `true`/`false` toggles HTML report generation. As an object, takes `enabled`, `title`, and `output`. |
  149. | `report.enabled` | boolean | `true` | Whether to generate the static HTML report. |
  150. | `report.title` | string | none | Title shown at the top of the HTML report. |
  151. | `report.output` | string | `<run workspace>/report` | Directory the HTML report is written to. |
  152. | `logging.format` | `pretty` \| `jsonl` \| `silent` | `pretty` | `pretty` prints the rich console reporter; `jsonl` emits machine-readable events; `silent` suppresses progress output (the final JSON result still goes to stdout). |
  153. | `logging.verbose` | boolean | `false` | Print full prompts, outputs, and judge prompts instead of snippets (`pretty` format only). |
  154. | `logging.color` | boolean \| `auto` | `auto` | ANSI color: `auto` enables it on a TTY, `true`/`false` force it. |
  155. | `logging.snippetLength` | number | `200` | Max length of inline prompt/output snippets in non-verbose `pretty` output. |
  156. | `logging.file` | string | none | Path to write JSONL event logs to (used with `--log-file`). |
  157. | `targetParams` | object | none | Inference params (e.g. `temperature`) passed through to the target model. |
  158. | `judgeParams` | object | none | Inference params passed through to the judge model. |
  159. To iterate on one case, pass `--eval-id` one or more times:
  160. ```bash
  161. OPENAI_API_KEY=... npx agent-skills-eval --config agent-skills-eval.yaml --eval-id basic
  162. ```
  163. ## SDK
  164. For programmatic use — CI pipelines, custom dashboards, multi-skill rollups — drive the evaluator from TypeScript:
  165. ```ts
  166. import {
  167. OpenAICompatibleProvider,
  168. consoleReporter,
  169. evaluateSkills,
  170. } from "agent-skills-eval";
  171. const provider = new OpenAICompatibleProvider({
  172. baseUrl: "https://api.openai.com/v1",
  173. apiKey: process.env.OPENAI_API_KEY!,
  174. model: "gpt-4o-mini",
  175. providerName: "openai",
  176. });
  177. const result = await evaluateSkills({
  178. root: "./skills",
  179. workspace: "./agent-skills-workspace",
  180. baseline: true,
  181. concurrency: 4,
  182. workspaceLayout: "iteration",
  183. strict: true,
  184. target: { model: provider.model, provider },
  185. judge: { model: provider.model, provider },
  186. onEvent: consoleReporter(),
  187. });
  188. console.log(result);
  189. ```
  190. Stream events to a file as JSONL for downstream analysis:
  191. ```ts
  192. import { jsonlReporter } from "agent-skills-eval";
  193. const reporter = jsonlReporter({ file: "./events.jsonl" });
  194. await evaluateSkills({ /* ... */ onEvent: reporter.onEvent });
  195. await reporter.close();
  196. ```
  197. Load YAML config programmatically:
  198. ```ts
  199. import { loadConfigFile } from "agent-skills-eval";
  200. const config = loadConfigFile("./agent-skills-eval.yaml");
  201. ```
  202. ## Custom providers
  203. Bring any backend by implementing the `Provider` interface — five fields, one method:
  204. ```ts
  205. import type { Provider, ProviderResult } from "agent-skills-eval";
  206. export const provider: Provider = {
  207. name: "my-provider",
  208. model: "my-model",
  209. async complete(prompt: string): Promise<ProviderResult> {
  210. return {
  211. provider: "my-provider",
  212. model: "my-model",
  213. output: "model output",
  214. latencyMs: 0,
  215. inputTokens: 0,
  216. outputTokens: 0,
  217. costUsd: 0,
  218. };
  219. },
  220. };
  221. ```
  222. Useful for: local model servers (Ollama, vLLM, llama.cpp), proprietary internal APIs, mock providers in unit tests, or routing layers in front of multiple providers.
  223. ## Skill layout
  224. A skill is a folder. The minimum is a `SKILL.md`. Add `evals/evals.json` and you can evaluate it.
  225. ```text
  226. my-skill/
  227. ├── SKILL.md
  228. ├── references/
  229. │ └── notes.md
  230. ├── scripts/
  231. │ └── helper.sh
  232. └── evals/
  233. ├── evals.json
  234. └── files/
  235. └── input.csv
  236. ```
  237. `SKILL.md`:
  238. ```markdown
  239. ---
  240. name: my-skill
  241. description: Analyze small CSV files.
  242. license: MIT
  243. compatibility: Works with text-capable chat models.
  244. ---
  245. When given a CSV file, identify the most important trend and cite the
  246. relevant rows.
  247. ```
  248. `evals/evals.json`:
  249. ```json
  250. {
  251. "skill_name": "my-skill",
  252. "evals": [
  253. {
  254. "id": "basic",
  255. "name": "basic behavior",
  256. "prompt": "Use the attached data to summarize revenue.",
  257. "files": ["evals/files/input.csv"],
  258. "expected_output": "The response identifies the highest revenue month.",
  259. "assertions": [
  260. "The output identifies the highest revenue month."
  261. ]
  262. }
  263. ]
  264. }
  265. ```
  266. If you skip `assertions` but provide `expected_output`, the SDK promotes the expected output into a judge assertion automatically — so a minimal agentskills.io eval file produces meaningful pass/fail grading without extra work.
  267. ## CLI options
  268. ```bash
  269. npx agent-skills-eval [root] \
  270. --config agent-skills-eval.yaml \
  271. --workspace ./agent-skills-workspace \
  272. --baseline \
  273. --target gpt-4o-mini \
  274. --judge gpt-4o-mini \
  275. --base-url https://api.openai.com/v1 \
  276. --api-key-env OPENAI_API_KEY \
  277. --include "skills/**" \
  278. --exclude "**/draft-*" \
  279. --eval-id basic \
  280. --concurrency 4 \
  281. --layout iteration \
  282. --strict \
  283. --log-format pretty \
  284. --report
  285. ```
  286. **Logging modes**: `pretty` for humans, `jsonl` for machines, `silent` for quiet CI.
  287. ## Reports
  288. The static HTML report is built from disk artifacts and shows everything you'd want for skill iteration:
  289. - Pass rate by skill and by eval
  290. - Assertion-by-assertion grading evidence with judge reasoning
  291. - Full target output, side by side for `with_skill` and `without_skill`
  292. - Prompt and judge prompt details
  293. - Timing and token usage
  294. - Tool calls when present
  295. Use `--report-output` (or `report.output` in YAML) to choose where the report lands.
  296. ## agentskills.io compatibility
  297. Implements the [agentskills.io](https://agentskills.io) specification end to end:
  298. - `SKILL.md` YAML frontmatter — required `name` and `description`, optional `license`, `compatibility`, `metadata`, `allowed-tools`
  299. - Strict validation: name length, lowercase-hyphenated format, parent-directory match, description length, compatibility length
  300. - Optional `scripts/`, `references/`, and `assets/` directories — markdown references included in skill context, scripts exposed by manifest
  301. - `evals/evals.json` schema: `skill_name`, `evals[].id`, `prompt`, `expected_output`, `files`, `assertions`
  302. - Official artifact layout: `iteration-N/<eval>/<mode>/outputs`, `timing.json`, `grading.json`, `benchmark.json`
  303. - Baseline comparison via `with_skill` and `without_skill`
  304. Beyond the spec, this SDK adds: per-eval `defaults`, model `params`, tool definitions, deterministic `tool_assertions`, and a flat `workspaceLayout: "flat"` for multi-skill dashboards.
  305. ## Examples
  306. See [`examples/basic-skill`](examples/basic-skill) for a complete skill folder, and [`examples/agent-skills-eval.yaml`](examples/agent-skills-eval.yaml) for a reference config.
  307. [`examples/agent-qa-evidence`](examples/agent-qa-evidence) reviews run evidence from [agent-qa](https://github.com/vostride/agent-qa), a natural-language web/mobile testing tool with advanced memidex memory and self-improvement. Its six cases distinguish selection-only output, incomplete records, healed runs, setup failures, and completed passes and failures. Select it with `--include agent-qa-evidence` when the root is `./examples`, using the usual target/judge and provider configuration.
  308. The fixtures are synthetic normalized evidence, not live application results or exact CLI response schemas. This example evaluates evidence interpretation without installing or running agent-qa. Running the evaluator still calls the configured target and judge models; no model scores or comparative improvement are bundled.
  309. ## Development
  310. ```bash
  311. npm ci
  312. npm test
  313. npm pack --dry-run
  314. ```
  315. ## Documentation
  316. Full docs live at **[darkrishabh.github.io/agent-skills-eval](https://darkrishabh.github.io/agent-skills-eval/)** (sources in [`docs/`](docs)). Local preview:
  317. ```bash
  318. python3 -m http.server 8080 --directory docs
  319. ```
  320. ## Contributing
  321. Issues, PRs, and skill examples are all welcome. See [CONTRIBUTING.md](CONTRIBUTING.md), [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md), and [SECURITY.md](SECURITY.md).
  322. ## License
  323. MIT. See [LICENSE](LICENSE).
  324. ---
  325. <div align="center">
  326. Built for the [Agent Skills](https://agentskills.io) ecosystem.
  327. </div>
  328. ## Integration boundaries
  329. See [the artifact contract](docs/artifact-contract.md) for output layouts, field
  330. semantics, adapter guidance, and provenance limits. External result-format
  331. adapters can consume these files without adding dependencies to this runner.
  332. The built-in OpenAI-compatible provider makes a single chat-completion request
  333. per target run (plus retries). Function tool definitions let you grade the
  334. returned tool-call requests; the provider does not connect to MCP servers,
  335. execute tools, or run a multi-turn agent loop. To evaluate a skill that needs
  336. real MCP execution today, implement a custom `Provider` that owns the tool loop
  337. and returns actual output and captured `toolCalls` for grading. Declaring a
  338. function tool is not evidence that its operation executed successfully.
  339. This runner evaluates behavior supplied by a provider; it does not authenticate
  340. agent identity, detect clones, or attest to an untampered runtime. Integrations
  341. that need identity verification must perform it before starting the evaluation
  342. and retain the evidence separately. Passing scores do not establish identity
  343. or regulatory compliance.
  344. Outgoing requests from `OpenAICompatibleProvider` carry
  345. `User-Agent: agent-skills-eval/<package-version> (+https://github.com/darkrishabh/agent-skills-eval; node/<node-version>)`.
  346. This applies to target and judge calls and retries. SDK callers can override it
  347. through `extraHeaders` using any capitalization of `User-Agent`.
  348. The proposed native-agent lifecycle is available for contributor review through
  349. [the experimental runtime foundation](docs/runtime-foundation.md). It is a
  350. separate SDK entry point; the native adapters and CLI integration are still under
  351. development with the authors of PRs #25 and #35.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.