Skip to content
Back to search

SkillEvaluator

Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/NVIDIA/SkillEvaluator with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/NVIDIA/SkillEvaluator'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nvidia--skillevaluator/badge.svg)](/listings/nvidia--skillevaluator)

HTML

<a href="/listings/nvidia--skillevaluator"><img src="/api/listings/nvidia--skillevaluator/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install with [uv](https://docs.astral.sh/uv/):

uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

**NVIDIA Build: set just two variables.** [API key](https://build.nvidia.com):

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'

This key covers chat, Tier 2 embeddings, and supported Tier 3 agents in Docker or local mode.

For gateways, set `SKILL_EVAL_LLM_PROVIDER=openai-compatible`, `SKILL_EVAL_LLM_BASE_URL`, and `SKILL_EVAL_LLM_API_KEY`. [Model defaults are overridable](docs/configuration.mdx#choose-an-llm-provider); the gateway must support the selected models and APIs.

Run all tiers on `./my-skill`, a directory containing `SKILL.md`:

skillevaluator validate ./my-skill

Run individual tiers:

skillevaluator tier1 ./my-skill
skillevaluator tier2 ./my-skill
skillevaluator tier3 ./my-skill

- **Tier 1** runs static checks, including dependency checks, plus rubric, LLM security analysis, and finding verification when a provider is configured. Without a provider, LLM stages are reported as skipped. Use `--no-llm` to opt out of LLM checks. - **Tier 2** checks for repeated content inside the skill. Add `--catalog ./skill-catalog.json` to compare against other skills too; without a catalog, the output explicitly says inter-skill comparison did not run. - **Tier 3** creates one starter evaluation case if no dataset or task source exists, then runs the provider-native agent in Docker with and without the skill. Valid existing sources are reused unchanged; invalid ones produce an error. Review starter cases; they provide limited coverage.

A tier without a path shows help. Expert commands such as `tier1 security-scan`, `tier2 similarity-check`, and `tier3 create-eval-dataset` remain available.

Runtime requirements

`[all]` installs Python extras; full Tier 1 needs Semgrep, SkillSpector, and Gitleaks, and Tier 3 needs an agent runtime and sandbox. Follow the [installation guide](https://docs.nvidia.com/skills/skillevaluator/installation), then check readiness:

skillevaluator doctor --env-mode docker

For a keyless check without external scanners, run `skillevaluator quality-check ./my-skill`. If the command is unavailable, run `uv tool update-shell` and reopen your terminal.

Other providers and optional model overrides

OpenAI uses two variables and defaults to Codex for Tier 3:

export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY='sk-...'

NVIDIA Build defaults to `nvidia/nemotron-3-super-120b-a12b` for evaluator chat, Tier 3 agent execution, and judging. Its embedding default is `nvidia/nemotron-3-embed-1b`; OpenAI defaults to `gpt-5.6-sol` and `text-embedding-3-small`. These are configured defaults, not automatically updated selections of the latest models. To select a different evaluator model:

export SKILL_EVAL_LLM_MODEL='your-provider-model-id'

The standard judge inherits that model unless overridden. Tier 3 defaults to OpenCode for NVIDIA Build, Codex for OpenAI, and Claude Code for Anthropic. Use `--agents` and `--agent-model` to override the agent and its model separately.

Anthropic and Bedrock need a separate embedding provider for Tier 2. Custom OpenAI-compatible endpoints also require explicit endpoint and model settings. See [Providers & Credentials](https://docs.nvidia.com/skills/skillevaluator/configuration) for these setups and advanced overrides.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 333 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. # ![SkillEvaluator](https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/ad2815fe78c045db91d230681d32863698da08cf/docs/assets/skillevaluator-wordmark.svg)
  3. <h4>Evaluate and improve AI agent skills and plugins: quality, security, overlap, compatibility, and real impact across agents</h4>
  4. [![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](LICENSE)
  5. [![Python](https://img.shields.io/badge/Python-3.12%20%7C%203.13%20%7C%203.14-blue.svg)](https://www.python.org/)
  6. [![PyPI](https://img.shields.io/pypi/v/skillevaluator.svg?label=PyPI)](https://pypi.org/project/skillevaluator/)
  7. [![Reusable CI templates](https://img.shields.io/badge/Reusable%20CI-templates-blue.svg)](https://docs.nvidia.com/skills/skillevaluator/repository-integration#copy-a-template)
  8. [![Documentation](https://img.shields.io/badge/Documentation-docs.nvidia.com-blue.svg)](https://docs.nvidia.com/skills/skillevaluator/)
  9. [![Paper](https://img.shields.io/badge/arXiv-2608.20614-b31b1b.svg)](https://arxiv.org/abs/2608.20614)
  10. [![Blog](https://img.shields.io/badge/Blog-NVIDIA%20Developer-76B900.svg)](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/)
  11. [![Stars](https://img.shields.io/github/stars/NVIDIA/SkillEvaluator)](https://github.com/NVIDIA/SkillEvaluator/stargazers)
  12. [![Forks](https://img.shields.io/github/forks/NVIDIA/SkillEvaluator)](https://github.com/NVIDIA/SkillEvaluator/forks)
  13. </div>
  14. > 📺 **Livestream:** [Ask the Experts: Evaluating Agent Skills](https://www.youtube.com/watch?v=4rI0zATFxYI)
  15. SkillEvaluator checks AI agent skills and plugins, and measures a skill's effect
  16. on task results: start with quality checks, then compare runs with and without a skill.
  17. Catch skill issues before merge. Connect your repository to SkillEvaluator with a
  18. [CI template](https://docs.nvidia.com/skills/skillevaluator/repository-integration#copy-a-template)
  19. for automated quality and security gates, without building or maintaining
  20. evaluation code.
  21. Agent skills are folders of instructions and supporting files defined by the
  22. [Agent Skills specification](https://agentskills.io/). SkillEvaluator is part of
  23. the [NVIDIA Verified Skills pipeline](https://github.com/NVIDIA/skills).
  24. ## Get started
  25. Install the CLI in a Python 3.12–3.14 environment:
  26. ```bash
  27. pip install skillevaluator
  28. ```
  29. <details><summary>Or use uv</summary>
  30. ```bash
  31. uv tool install --python 3.13 skillevaluator
  32. ```
  33. </details>
  34. For uv setup or direct GitHub source installation, see
  35. [Installation](https://docs.nvidia.com/skills/skillevaluator/installation).
  36. Then check skills from your terminal, CI, or agent:
  37. <table><tbody><tr valign="top">
  38. <td width="33%">
  39. <p><img src="https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/928cc9d483fe9fef2d5011c3ca8f132cda46730a/docs/assets/get-started-cli.svg" width="250" alt="Run the CLI directly"></p>
  40. Run checks backed by **500+ static rules and detection patterns**, without an
  41. LLM key. Complete [scanner setup](https://docs.nvidia.com/skills/skillevaluator/installation-options#system-tools)
  42. for full coverage:
  43. ```bash
  44. skillevaluator validate \
  45. ./my-skill --tiers 1
  46. ```
  47. <details><summary>Run all three tiers</summary>
  48. After [all-tier setup with NVIDIA Build](#run-all-three-tiers-with-a-free-nvidia-build-key),
  49. remove `--tiers 1` to add deduplication and live agent evaluation to measure
  50. skill lift:
  51. ```bash
  52. skillevaluator validate \
  53. ./my-skill
  54. ```
  55. </details>
  56. </td>
  57. <td width="33%">
  58. <p><img src="https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/928cc9d483fe9fef2d5011c3ca8f132cda46730a/docs/assets/get-started-ci.svg" width="250" alt="Add it to your repo via CI"></p>
  59. Check every skill change. Run in your GitHub repo to open a setup PR:
  60. ```bash
  61. skillevaluator ci init
  62. ```
  63. [Guided setup and CI templates](#bring-skill-evaluation-to-your-repository) for GitHub and GitLab.
  64. </td>
  65. <td width="33%">
  66. <p><img src="https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/928cc9d483fe9fef2d5011c3ca8f132cda46730a/docs/assets/get-started-agent.svg" width="250" alt="Wire it to your agent"></p>
  67. Add the skill to Claude Code, Codex or another agent:
  68. ```bash
  69. npx skills add \
  70. NVIDIA/SkillEvaluator
  71. ```
  72. Then ask: *"Is my skill any good?"*
  73. See the [agent guide](https://docs.nvidia.com/skills/skillevaluator/agent-skill).
  74. </td>
  75. </tr></tbody></table>
  76. ### ⚡ [Harness check](https://docs.nvidia.com/skills/skillevaluator/harness-check): will your agent load your skill?
  77. 🔑 **No API key** · 🤖 **No agent run**
  78. ```bash
  79. skillevaluator harness-check ./my-skill
  80. ```
  81. Illustrative summary of versioned profiles. Full output can include notes and schema diagnostics.
  82. ```text
  83. ╭─ Harness compatibility · my-skill ──────────────────────────────────────────────────────╮
  84. │ ⚠️ codex degraded Description cut to 1,021 chars + '...' in the skill list │
  85. │ ❌ claude-code not loaded .agents/skills is not scanned │
  86. │ ✅ opencode loads │
  87. │ ✅ hermes loads │
  88. │ ✅ openclaw loads │
  89. │ ❌ copilot-cli not loaded description is 1,028 UTF-16 units, 4 over the 1,024 limit │
  90. ╰────────────────────────────────────────────────────────────────── ❌ 2 of 6 not loaded ─╯
  91. ```
  92. `skillevaluator validate` includes Harness Check in Tier 1 alongside **quality,
  93. schema, version, PII, license, Unicode and script linting**. Security and code
  94. integrity checks use **SkillSpector, Gitleaks, Bandit and optional Semgrep**.
  95. [See all Tier 1 checks](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#the-checks);
  96. use `--tiers 1` to run only this tier.
  97. 🧪 **Ready to go deeper?** Try a
  98. [live agent evaluation](https://docs.nvidia.com/skills/skillevaluator/tier3-live-evaluation) (Tier 3).
  99. ## Quickstart
  100. ### Check your skill without a key
  101. ```bash
  102. skillevaluator quality-check ./my-skill
  103. ```
  104. No API key, scanner, GPU or Docker is needed. It prints a score out of 100 with a
  105. grade and the findings behind it; the passing threshold defaults to 70. This is
  106. static quality only; publication evidence stays `INCOMPLETE`. Security and agent impact need additional checks. See the
  107. [keyless walkthrough](docs/quickstart.mdx#run-without-an-llm).
  108. No skill yet? Clone the repository and use the bundled
  109. `src/skillevaluator/tier3/reference_skills/text-analyzer`. Scores can differ by release.
  110. ### Run all three tiers with a free NVIDIA Build key
  111. Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for
  112. **free prototyping**, subject to [access terms and rate limits](https://docs.api.nvidia.com/nim/docs/product).
  113. 1. Sign in at [build.nvidia.com](https://build.nvidia.com/) and complete enrollment.
  114. 2. Open a model page, select **Get API Key**, and generate your key.
  115. 3. Complete [scanner setup](docs/installation-options.mdx#system-tools), start Docker, and [prepare a reviewed dataset](docs/prepare-dataset.mdx). Install every tier, set the provider and key, then validate:
  116. ```bash
  117. pip install "skillevaluator[all]"
  118. export SKILL_EVAL_LLM_PROVIDER=nv_build
  119. export NVIDIA_API_KEY='nvapi-...'
  120. skillevaluator validate ./my-skill
  121. ```
  122. Without `--tiers`, `validate` runs all three. The key covers embeddings and chat
  123. in Tier 2 and supported agents in Tier 3; add `--llm` for model checks in Tier 1.
  124. For uv or GitHub installations, use the matching [all-tier install](https://docs.nvidia.com/skills/skillevaluator/installation#all-tiers) instead of the pip command above. Tier 3 also
  125. needs Docker running; add `--tiers 1,2` to skip it. More in
  126. [Run the combined pipeline](#run-the-combined-pipeline), the
  127. [Build walkthrough](docs/quickstart.mdx#use-nvidia-build) and [provider configuration](docs/configuration.mdx).
  128. ## Bring skill evaluation to your repository
  129. **Minimal setup. No evaluation code to write or maintain.** Choose your skill
  130. folders and thresholds to get automated validation, security scanning and
  131. downloadable reports for changed skills in GitHub or GitLab.
  132. **[11 CI checks](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#the-checks) backed by 500+ static rules and detection patterns:**
  133. quality, schema, version, PII, license, Unicode, script linting,
  134. [Harness Check](https://docs.nvidia.com/skills/skillevaluator/harness-check),
  135. security, code integrity and dependency auditing.<br>
  136. **Scanners:** [SkillSpector](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#security--vulnerability-scanning),
  137. [Gitleaks, Bandit and Semgrep](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#code-integrity--code-risk-secrets-and-hygiene),
  138. and [pip-audit](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#dependency--cve-audit-opt-in).
  139. **GitHub guided setup:** after [installing the CLI](#get-started), run inside
  140. your checked-out repository:
  141. ```bash
  142. skillevaluator ci init
  143. ```
  144. Choose checks and optional hosted feedback, then review the setup PR.
  145. [CLI setup guide](https://docs.nvidia.com/skills/skillevaluator/cli-onboarding).
  146. **Manual setup:** copy a filled-in template from [CI templates](ci/bundles/README.md).
  147. No local CLI needed.
  148. | Your repository | Ready-to-use template | Copy destination |
  149. | --- | --- | --- |
  150. | GitHub | [Skill checks](ci/bundles/1.0.1/skill-checks.yml) | `.github/workflows/skill-checks.yml` |
  151. | GitLab | [Skill checks](ci/bundles/1.0.1/gitlab-skill-checks.yml) | `.gitlab-ci.yml`, or merge into your existing pipeline |
  152. | GitHub PR comments without hosting | [Optional comments](ci/bundles/1.0.1/comment-skill-checks.yml) | Second workflow; complete [comment setup](ci/bundles/1.0.1/README.md#pr-comment-setup) |
  153. | GitHub hosted reports and PR comments | [Optional publisher](ci/bundles/1.0.1/publish-skill-reports.yml) | Second workflow; complete [publisher setup](ci/bundles/1.0.1/README.md#one-time-publisher-setup) |
  154. Choose one optional feedback workflow. These templates run Tier 1 quality and
  155. security checks without a model key; see [all-tier CI setup](https://docs.nvidia.com/skills/skillevaluator/advanced-ci)
  156. to add deduplication and live agent evaluation.
  157. See [GitHub](https://docs.nvidia.com/skills/skillevaluator/github-actions) and
  158. [GitLab](https://docs.nvidia.com/skills/skillevaluator/gitlab-ci) setup,
  159. [template reference](ci/templates/README.md) and [report examples](ci/examples/README.md).
  160. ## Three-tier overview
  161. ![Three-tier pipeline: skill, validation, deduplication, live evaluation, reports](https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/ad2815fe78c045db91d230681d32863698da08cf/docs/assets/three-tier-overview.svg)
  162. Each tier is independent. Use the [all-tier installation](https://docs.nvidia.com/skills/skillevaluator/installation#all-tiers) for all three:
  163. | Tier | Question | Example after setup | Additional setup |
  164. | --- | --- | --- | --- |
  165. | Tier 1: Validation | Is the skill safe, well-formed and [harness-compatible](docs/tier1-reference.mdx#the-checks)? | `skillevaluator tier1 ./my-skill --no-llm` | External scanners for full static coverage; a provider and model clients enable LLM checks |
  166. | Tier 1: Harness Check | Will the agent load the skill? | `skillevaluator harness-check ./my-skill` | No API key or agent run |
  167. | Tier 2: Deduplication | Does guidance repeat or overlap? | `skillevaluator tier2 ./my-skill` | Embeddings and chat; add `--catalog ./skill-catalog.json` for inter-skill comparison |
  168. | Tier 3: Live Agent Evaluation | Does the skill help the agent? | `skillevaluator tier3 ./my-skill` | Provider access, a supported agent, and a running sandbox (Docker by default) |
  169. Prepare [scanners](docs/installation-options.mdx#system-tools) for your installed release; Semgrep is optional.
  170. `--no-llm` does not remove scanner requirements; a scanner that cannot run is
  171. `INCOMPLETE` and exits nonzero.
  172. [SkillSpector](https://github.com/NVIDIA/SkillSpector) provides specialized skill
  173. security scanning; [Harbor](https://github.com/harbor-framework/harbor), the
  174. open-source agent evaluation framework, powers sandboxed agent runs.
  175. Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses
  176. sources or creates one skill-grounded case with limited coverage. See
  177. [tier guides](https://docs.nvidia.com/skills/skillevaluator/).
  178. ## Run the combined pipeline
  179. After setting up the provider, scanners, and agent sandbox, run all three tiers:
  180. ```bash
  181. skillevaluator doctor --env-mode docker
  182. skillevaluator validate ./my-skill
  183. ```
  184. `doctor` checks readiness, not inference success. `validate` runs all tiers with
  185. autopilot for skills. Use `--tiers 1,2` to skip live evaluation or `--tiers 1` for
  186. static checks. `--no-autopilot` requires an existing source. Other artifact types
  187. retain their applicable stages.
  188. Tier 1 gates `validate`. Tier 2 gates unless `--no-block-on-dedup` is set;
  189. Tier 3 gates only with `--block-on-agent-eval`. Model and managed-sandbox usage
  190. can incur charges; see [runtime and cost](docs/tier3-run-reference.mdx#plan-for-cost).
  191. Standalone generation needs substantive `evals/EVAL.md` guidance, a `--prompt`
  192. file with task requests and outcomes, or matching retained traces. `--force`
  193. below replaces the autopilot starter; `--refine` refines it. Explicit
  194. `--grounding skill` permits skill-only starters. Review cases; templates omit
  195. unauthored negatives.
  196. ```bash
  197. skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force
  198. ```
  199. ## Run from a source checkout
  200. For an editable environment, follow the [development setup](docs/developer-guide.mdx#set-up-a-development-environment).
  201. ## Research and citation
  202. Resources: [paper](https://arxiv.org/abs/2608.20614),
  203. [PDF](https://arxiv.org/pdf/2608.20614),
  204. [DOI](https://doi.org/10.48550/arXiv.2608.20614),
  205. [BibTeX](https://arxiv.org/bibtex/2608.20614), and
  206. [NVIDIA Developer blog](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/).
  207. Cite the paper and record the SkillEvaluator release or commit; see [CITATION.cff](CITATION.cff).
  208. ## Documentation
  209. See [the documentation](https://docs.nvidia.com/skills/skillevaluator/) for
  210. setup, tier guides, reports, CI integration, CLI reference, and
  211. [performance dashboard](docs/reports.mdx#performance-dashboard).
  212. ## Installation and third-party software
  213. This project will download and install additional third-party open source
  214. software projects. Review the license terms of these open source projects before
  215. use.
  216. ## Contributing
  217. Read [CONTRIBUTING.md](CONTRIBUTING.md) and run:
  218. ```bash
  219. make lint && make test && make build
  220. ```
  221. See [project governance](GOVERNANCE.md) and [Code of Conduct](CODE_OF_CONDUCT.md).
  222. ## Support
  223. Support level: **Experimental**. Community-supported, best-effort, with no SLA
  224. or NVIDIA enterprise support entitlement. Report bugs and feature requests through
  225. [GitHub Issues](https://github.com/NVIDIA/SkillEvaluator/issues); see
  226. [SUPPORT.md](SUPPORT.md).
  227. ## Security
  228. Report suspected vulnerabilities using the private process in
  229. [SECURITY.md](SECURITY.md). Do not disclose security issues in a public GitHub
  230. issue.
  231. ## Releases
  232. See [CHANGELOG.md](CHANGELOG.md) and
  233. [GitHub Releases](https://github.com/NVIDIA/SkillEvaluator/releases).
  234. ## License
  235. Apache License 2.0: see [LICENSE](LICENSE), [NOTICE](NOTICE), and
  236. [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
  237. Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.