SkillEvaluator
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/NVIDIA/SkillEvaluator with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/NVIDIA/SkillEvaluator'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/nvidia--skillevaluator)HTML
<a href="/listings/nvidia--skillevaluator"><img src="/api/listings/nvidia--skillevaluator/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Install with [uv](https://docs.astral.sh/uv/):
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
**NVIDIA Build: set just two variables.** [API key](https://build.nvidia.com):
export SKILL_EVAL_LLM_PROVIDER=nv_build export NVIDIA_API_KEY='nvapi-...'
This key covers chat, Tier 2 embeddings, and supported Tier 3 agents in Docker or local mode.
For gateways, set `SKILL_EVAL_LLM_PROVIDER=openai-compatible`, `SKILL_EVAL_LLM_BASE_URL`, and `SKILL_EVAL_LLM_API_KEY`. [Model defaults are overridable](docs/configuration.mdx#choose-an-llm-provider); the gateway must support the selected models and APIs.
Run all tiers on `./my-skill`, a directory containing `SKILL.md`:
skillevaluator validate ./my-skill
Run individual tiers:
skillevaluator tier1 ./my-skill skillevaluator tier2 ./my-skill skillevaluator tier3 ./my-skill
- **Tier 1** runs static checks, including dependency checks, plus rubric, LLM security analysis, and finding verification when a provider is configured. Without a provider, LLM stages are reported as skipped. Use `--no-llm` to opt out of LLM checks. - **Tier 2** checks for repeated content inside the skill. Add `--catalog ./skill-catalog.json` to compare against other skills too; without a catalog, the output explicitly says inter-skill comparison did not run. - **Tier 3** creates one starter evaluation case if no dataset or task source exists, then runs the provider-native agent in Docker with and without the skill. Valid existing sources are reused unchanged; invalid ones produce an error. Review starter cases; they provide limited coverage.
A tier without a path shows help. Expert commands such as `tier1 security-scan`, `tier2 similarity-check`, and `tier3 create-eval-dataset` remain available.
Runtime requirements
`[all]` installs Python extras; full Tier 1 needs Semgrep, SkillSpector, and Gitleaks, and Tier 3 needs an agent runtime and sandbox. Follow the [installation guide](https://docs.nvidia.com/skills/skillevaluator/installation), then check readiness:
skillevaluator doctor --env-mode docker
For a keyless check without external scanners, run `skillevaluator quality-check ./my-skill`. If the command is unavailable, run `uv tool update-shell` and reopen your terminal.
Other providers and optional model overrides
OpenAI uses two variables and defaults to Codex for Tier 3:
export SKILL_EVAL_LLM_PROVIDER=openai export OPENAI_API_KEY='sk-...'
NVIDIA Build defaults to `nvidia/nemotron-3-super-120b-a12b` for evaluator chat, Tier 3 agent execution, and judging. Its embedding default is `nvidia/nemotron-3-embed-1b`; OpenAI defaults to `gpt-5.6-sol` and `text-embedding-3-small`. These are configured defaults, not automatically updated selections of the latest models. To select a different evaluator model:
export SKILL_EVAL_LLM_MODEL='your-provider-model-id'
The standard judge inherits that model unless overridden. Tier 3 defaults to OpenCode for NVIDIA Build, Codex for OpenAI, and Claude Code for Anthropic. Use `--agents` and `--agent-model` to override the agent and its model separately.
Anthropic and Bedrock need a separate embedding provider for Tier 2. Custom OpenAI-compatible endpoints also require explicit endpoint and model settings. See [Providers & Credentials](https://docs.nvidia.com/skills/skillevaluator/configuration) for these setups and advanced overrides.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 333 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <div align="center">
- # 
- <h4>Evaluate and improve AI agent skills and plugins: quality, security, overlap, compatibility, and real impact across agents</h4>
- [](LICENSE)
- [](https://www.python.org/)
- [](https://pypi.org/project/skillevaluator/)
- [](https://docs.nvidia.com/skills/skillevaluator/repository-integration#copy-a-template)
- [](https://docs.nvidia.com/skills/skillevaluator/)
- [](https://arxiv.org/abs/2608.20614)
- [](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/)
- [](https://github.com/NVIDIA/SkillEvaluator/stargazers)
- [](https://github.com/NVIDIA/SkillEvaluator/forks)
- </div>
- > 📺 **Livestream:** [Ask the Experts: Evaluating Agent Skills](https://www.youtube.com/watch?v=4rI0zATFxYI)
- SkillEvaluator checks AI agent skills and plugins, and measures a skill's effect
- on task results: start with quality checks, then compare runs with and without a skill.
- Catch skill issues before merge. Connect your repository to SkillEvaluator with a
- [CI template](https://docs.nvidia.com/skills/skillevaluator/repository-integration#copy-a-template)
- for automated quality and security gates, without building or maintaining
- evaluation code.
- Agent skills are folders of instructions and supporting files defined by the
- [Agent Skills specification](https://agentskills.io/). SkillEvaluator is part of
- the [NVIDIA Verified Skills pipeline](https://github.com/NVIDIA/skills).
- ## Get started
- Install the CLI in a Python 3.12–3.14 environment:
- ```bash
- pip install skillevaluator
- ```
- <details><summary>Or use uv</summary>
- ```bash
- uv tool install --python 3.13 skillevaluator
- ```
- </details>
- For uv setup or direct GitHub source installation, see
- [Installation](https://docs.nvidia.com/skills/skillevaluator/installation).
- Then check skills from your terminal, CI, or agent:
- <table><tbody><tr valign="top">
- <td width="33%">
- <p><img src="https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/928cc9d483fe9fef2d5011c3ca8f132cda46730a/docs/assets/get-started-cli.svg" width="250" alt="Run the CLI directly"></p>
- Run checks backed by **500+ static rules and detection patterns**, without an
- LLM key. Complete [scanner setup](https://docs.nvidia.com/skills/skillevaluator/installation-options#system-tools)
- for full coverage:
- ```bash
- skillevaluator validate \
- ./my-skill --tiers 1
- ```
- <details><summary>Run all three tiers</summary>
- After [all-tier setup with NVIDIA Build](#run-all-three-tiers-with-a-free-nvidia-build-key),
- remove `--tiers 1` to add deduplication and live agent evaluation to measure
- skill lift:
- ```bash
- skillevaluator validate \
- ./my-skill
- ```
- </details>
- </td>
- <td width="33%">
- <p><img src="https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/928cc9d483fe9fef2d5011c3ca8f132cda46730a/docs/assets/get-started-ci.svg" width="250" alt="Add it to your repo via CI"></p>
- Check every skill change. Run in your GitHub repo to open a setup PR:
- ```bash
- skillevaluator ci init
- ```
- [Guided setup and CI templates](#bring-skill-evaluation-to-your-repository) for GitHub and GitLab.
- </td>
- <td width="33%">
- <p><img src="https://raw.githubusercontent.com/NVIDIA/SkillEvaluator/928cc9d483fe9fef2d5011c3ca8f132cda46730a/docs/assets/get-started-agent.svg" width="250" alt="Wire it to your agent"></p>
- Add the skill to Claude Code, Codex or another agent:
- ```bash
- npx skills add \
- NVIDIA/SkillEvaluator
- ```
- Then ask: *"Is my skill any good?"*
- See the [agent guide](https://docs.nvidia.com/skills/skillevaluator/agent-skill).
- </td>
- </tr></tbody></table>
- ### ⚡ [Harness check](https://docs.nvidia.com/skills/skillevaluator/harness-check): will your agent load your skill?
- 🔑 **No API key** · 🤖 **No agent run**
- ```bash
- skillevaluator harness-check ./my-skill
- ```
- Illustrative summary of versioned profiles. Full output can include notes and schema diagnostics.
- ```text
- ╭─ Harness compatibility · my-skill ──────────────────────────────────────────────────────╮
- │ ⚠️ codex degraded Description cut to 1,021 chars + '...' in the skill list │
- │ ❌ claude-code not loaded .agents/skills is not scanned │
- │ ✅ opencode loads │
- │ ✅ hermes loads │
- │ ✅ openclaw loads │
- │ ❌ copilot-cli not loaded description is 1,028 UTF-16 units, 4 over the 1,024 limit │
- ╰────────────────────────────────────────────────────────────────── ❌ 2 of 6 not loaded ─╯
- ```
- `skillevaluator validate` includes Harness Check in Tier 1 alongside **quality,
- schema, version, PII, license, Unicode and script linting**. Security and code
- integrity checks use **SkillSpector, Gitleaks, Bandit and optional Semgrep**.
- [See all Tier 1 checks](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#the-checks);
- use `--tiers 1` to run only this tier.
- 🧪 **Ready to go deeper?** Try a
- [live agent evaluation](https://docs.nvidia.com/skills/skillevaluator/tier3-live-evaluation) (Tier 3).
- ## Quickstart
- ### Check your skill without a key
- ```bash
- skillevaluator quality-check ./my-skill
- ```
- No API key, scanner, GPU or Docker is needed. It prints a score out of 100 with a
- grade and the findings behind it; the passing threshold defaults to 70. This is
- static quality only; publication evidence stays `INCOMPLETE`. Security and agent impact need additional checks. See the
- [keyless walkthrough](docs/quickstart.mdx#run-without-an-llm).
- No skill yet? Clone the repository and use the bundled
- `src/skillevaluator/tier3/reference_skills/text-analyzer`. Scores can differ by release.
- ### Run all three tiers with a free NVIDIA Build key
- Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for
- **free prototyping**, subject to [access terms and rate limits](https://docs.api.nvidia.com/nim/docs/product).
- 1. Sign in at [build.nvidia.com](https://build.nvidia.com/) and complete enrollment.
- 2. Open a model page, select **Get API Key**, and generate your key.
- 3. Complete [scanner setup](docs/installation-options.mdx#system-tools), start Docker, and [prepare a reviewed dataset](docs/prepare-dataset.mdx). Install every tier, set the provider and key, then validate:
- ```bash
- pip install "skillevaluator[all]"
- export SKILL_EVAL_LLM_PROVIDER=nv_build
- export NVIDIA_API_KEY='nvapi-...'
- skillevaluator validate ./my-skill
- ```
- Without `--tiers`, `validate` runs all three. The key covers embeddings and chat
- in Tier 2 and supported agents in Tier 3; add `--llm` for model checks in Tier 1.
- For uv or GitHub installations, use the matching [all-tier install](https://docs.nvidia.com/skills/skillevaluator/installation#all-tiers) instead of the pip command above. Tier 3 also
- needs Docker running; add `--tiers 1,2` to skip it. More in
- [Run the combined pipeline](#run-the-combined-pipeline), the
- [Build walkthrough](docs/quickstart.mdx#use-nvidia-build) and [provider configuration](docs/configuration.mdx).
- ## Bring skill evaluation to your repository
- **Minimal setup. No evaluation code to write or maintain.** Choose your skill
- folders and thresholds to get automated validation, security scanning and
- downloadable reports for changed skills in GitHub or GitLab.
- **[11 CI checks](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#the-checks) backed by 500+ static rules and detection patterns:**
- quality, schema, version, PII, license, Unicode, script linting,
- [Harness Check](https://docs.nvidia.com/skills/skillevaluator/harness-check),
- security, code integrity and dependency auditing.<br>
- **Scanners:** [SkillSpector](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#security--vulnerability-scanning),
- [Gitleaks, Bandit and Semgrep](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#code-integrity--code-risk-secrets-and-hygiene),
- and [pip-audit](https://docs.nvidia.com/skills/skillevaluator/tier1-reference#dependency--cve-audit-opt-in).
- **GitHub guided setup:** after [installing the CLI](#get-started), run inside
- your checked-out repository:
- ```bash
- skillevaluator ci init
- ```
- Choose checks and optional hosted feedback, then review the setup PR.
- [CLI setup guide](https://docs.nvidia.com/skills/skillevaluator/cli-onboarding).
- **Manual setup:** copy a filled-in template from [CI templates](ci/bundles/README.md).
- No local CLI needed.
- | Your repository | Ready-to-use template | Copy destination |
- | --- | --- | --- |
- | GitHub | [Skill checks](ci/bundles/1.0.1/skill-checks.yml) | `.github/workflows/skill-checks.yml` |
- | GitLab | [Skill checks](ci/bundles/1.0.1/gitlab-skill-checks.yml) | `.gitlab-ci.yml`, or merge into your existing pipeline |
- | GitHub PR comments without hosting | [Optional comments](ci/bundles/1.0.1/comment-skill-checks.yml) | Second workflow; complete [comment setup](ci/bundles/1.0.1/README.md#pr-comment-setup) |
- | GitHub hosted reports and PR comments | [Optional publisher](ci/bundles/1.0.1/publish-skill-reports.yml) | Second workflow; complete [publisher setup](ci/bundles/1.0.1/README.md#one-time-publisher-setup) |
- Choose one optional feedback workflow. These templates run Tier 1 quality and
- security checks without a model key; see [all-tier CI setup](https://docs.nvidia.com/skills/skillevaluator/advanced-ci)
- to add deduplication and live agent evaluation.
- See [GitHub](https://docs.nvidia.com/skills/skillevaluator/github-actions) and
- [GitLab](https://docs.nvidia.com/skills/skillevaluator/gitlab-ci) setup,
- [template reference](ci/templates/README.md) and [report examples](ci/examples/README.md).
- ## Three-tier overview
- 
- Each tier is independent. Use the [all-tier installation](https://docs.nvidia.com/skills/skillevaluator/installation#all-tiers) for all three:
- | Tier | Question | Example after setup | Additional setup |
- | --- | --- | --- | --- |
- | Tier 1: Validation | Is the skill safe, well-formed and [harness-compatible](docs/tier1-reference.mdx#the-checks)? | `skillevaluator tier1 ./my-skill --no-llm` | External scanners for full static coverage; a provider and model clients enable LLM checks |
- | Tier 1: Harness Check | Will the agent load the skill? | `skillevaluator harness-check ./my-skill` | No API key or agent run |
- | Tier 2: Deduplication | Does guidance repeat or overlap? | `skillevaluator tier2 ./my-skill` | Embeddings and chat; add `--catalog ./skill-catalog.json` for inter-skill comparison |
- | Tier 3: Live Agent Evaluation | Does the skill help the agent? | `skillevaluator tier3 ./my-skill` | Provider access, a supported agent, and a running sandbox (Docker by default) |
- Prepare [scanners](docs/installation-options.mdx#system-tools) for your installed release; Semgrep is optional.
- `--no-llm` does not remove scanner requirements; a scanner that cannot run is
- `INCOMPLETE` and exits nonzero.
- [SkillSpector](https://github.com/NVIDIA/SkillSpector) provides specialized skill
- security scanning; [Harbor](https://github.com/harbor-framework/harbor), the
- open-source agent evaluation framework, powers sandboxed agent runs.
- Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses
- sources or creates one skill-grounded case with limited coverage. See
- [tier guides](https://docs.nvidia.com/skills/skillevaluator/).
- ## Run the combined pipeline
- After setting up the provider, scanners, and agent sandbox, run all three tiers:
- ```bash
- skillevaluator doctor --env-mode docker
- skillevaluator validate ./my-skill
- ```
- `doctor` checks readiness, not inference success. `validate` runs all tiers with
- autopilot for skills. Use `--tiers 1,2` to skip live evaluation or `--tiers 1` for
- static checks. `--no-autopilot` requires an existing source. Other artifact types
- retain their applicable stages.
- Tier 1 gates `validate`. Tier 2 gates unless `--no-block-on-dedup` is set;
- Tier 3 gates only with `--block-on-agent-eval`. Model and managed-sandbox usage
- can incur charges; see [runtime and cost](docs/tier3-run-reference.mdx#plan-for-cost).
- Standalone generation needs substantive `evals/EVAL.md` guidance, a `--prompt`
- file with task requests and outcomes, or matching retained traces. `--force`
- below replaces the autopilot starter; `--refine` refines it. Explicit
- `--grounding skill` permits skill-only starters. Review cases; templates omit
- unauthored negatives.
- ```bash
- skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force
- ```
- ## Run from a source checkout
- For an editable environment, follow the [development setup](docs/developer-guide.mdx#set-up-a-development-environment).
- ## Research and citation
- Resources: [paper](https://arxiv.org/abs/2608.20614),
- [PDF](https://arxiv.org/pdf/2608.20614),
- [DOI](https://doi.org/10.48550/arXiv.2608.20614),
- [BibTeX](https://arxiv.org/bibtex/2608.20614), and
- [NVIDIA Developer blog](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/).
- Cite the paper and record the SkillEvaluator release or commit; see [CITATION.cff](CITATION.cff).
- ## Documentation
- See [the documentation](https://docs.nvidia.com/skills/skillevaluator/) for
- setup, tier guides, reports, CI integration, CLI reference, and
- [performance dashboard](docs/reports.mdx#performance-dashboard).
- ## Installation and third-party software
- This project will download and install additional third-party open source
- software projects. Review the license terms of these open source projects before
- use.
- ## Contributing
- Read [CONTRIBUTING.md](CONTRIBUTING.md) and run:
- ```bash
- make lint && make test && make build
- ```
- See [project governance](GOVERNANCE.md) and [Code of Conduct](CODE_OF_CONDUCT.md).
- ## Support
- Support level: **Experimental**. Community-supported, best-effort, with no SLA
- or NVIDIA enterprise support entitlement. Report bugs and feature requests through
- [GitHub Issues](https://github.com/NVIDIA/SkillEvaluator/issues); see
- [SUPPORT.md](SUPPORT.md).
- ## Security
- Report suspected vulnerabilities using the private process in
- [SECURITY.md](SECURITY.md). Do not disclose security issues in a public GitHub
- issue.
- ## Releases
- See [CHANGELOG.md](CHANGELOG.md) and
- [GitHub Releases](https://github.com/NVIDIA/SkillEvaluator/releases).
- ## License
- Apache License 2.0: see [LICENSE](LICENSE), [NOTICE](NOTICE), and
- [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
- Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.