Test a skill
Last reviewed October 24, 2026
A skill can fail in two ways, and they need different tests. It can sit unused when it should run, or run when it should not. And when it does run, it can do no better than the agent would have done without it. It is easy to test only that it runs when asked, and stop there.
Does it trigger when it should?
Write about twenty prompts: half that should use the skill and half that should not. The useful ones in the second half are near misses, prompts that share words with the skill but want something else ("what does git describe do?" for a release-notes skill). Run each prompt three times, because the choice is not deterministic, and count a prompt as passing when the result matches more often than not (optimizing descriptions).
In Claude Code you can check one prompt from the command line. This prints the skill's name if the agent chose it:
claude -p "Draft release notes for this repository." \
--output-format stream-json --verbose --tools Skill \
| jq -r 'select(.type == "assistant") | .message.content[]
| select(.type == "tool_use" and .name == "Skill") | .input.skill'
writing-release-notes--tools Skill lets the agent do nothing but choose, so the check is quick and cannot change your files. The near miss above printed nothing, which is the result you want.
When a prompt fails, rewrite the description around the intent it missed. Do not paste the failing prompt's words into it; that makes the test pass and the skill worse, the classic way to overfit. Keep a few prompts aside that you never tune against, and judge the description by those.
Does it help when it runs?
Run the same tasks with the skill and without the skill, and compare. Start with two or three realistic cases, and grade each against assertions you wrote down beforehand, with evidence for every pass. The standard keeps these in evals/evals.json (evaluating skills):
{
"skill_name": "writing-release-notes",
"evals": [
{
"id": 1,
"prompt": "Write release notes for everything since v1.4.0.",
"expected_output": "Added, Changed and Fixed sections; no refactors or test-only commits",
"assertions": ["groups changes under Added, Changed and Fixed", "asks before writing CHANGELOG.md"]
}
]
}Run the baseline in a fresh session. The conversation in which you wrote the skill is full of context that hides what the skill leaves out (Claude Code skills).
Let skill-creator run the tests
Anthropic's skill-creator, installed on the write step, automates both halves. It keeps the cases in evals/evals.json, runs each in a clean subagent, grades the assertions, and benchmarks pass rate, time and tokens with the skill against without it. It can also compare two versions of a skill blind, and generate should-trigger and should-not-trigger prompts to tune the description.
Test on every model you expect
Instructions that one model follows well can be too terse for a smaller one, or padding for a larger one. Anthropic's advice is to test with every model the skill will run on, Haiku, Sonnet and Opus included (best practices), and the same goes for every agent you list in your README.
When it never triggers
- Ask the agent what skills it has. If yours is missing, the folder is in the wrong place or the frontmatter did not parse.
- Run the validator from the write step, or
claude plugin validate .claude/skillsin Claude Code, which reports frontmatter it cannot read. - Check that the description uses the words people say, not the words in your code.
- Check the folder name matches
nameexactly. A mismatch fails validation.
Building something else? See the guides to a plugin or marketplace and an MCP server.