Skip to content
Back to search

caliper

SkillWorks with CodexGitHub

Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi, and Hermes.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/edonadei/caliper with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/edonadei/caliper'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/edonadei--caliper/badge.svg)](/listings/edonadei--caliper)

HTML

<a href="/listings/edonadei--caliper"><img src="/api/listings/edonadei--caliper/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

<table> <tr> <th width="50%">πŸ€– In your agent</th> <th width="50%">⌨️ In your terminal</th> </tr> <tr> <td valign="top">

npx skills@latest add edonadei/caliper

# then, in your agent:
/grill-skill ./inbox-triage/SKILL.md

<sub><code>grill-skill</code> interviews you, writes the spec, and runs it with and without your skill.</sub>

</td> <td valign="top">

pipx install caliper-eval
SPEC=inbox-triage/inbox-triage.eval.yaml

caliper run $SPEC --ablate inbox-triage
caliper run $SPEC
caliper compare \
  .caliper/results/inbox-triage/RUN_ID.json \
  inbox-triage

</td> </tr> </table>

Write a spec

A spec is a `.eval.yaml` file next to your skill. It lists the skills to install (yours, plus any **neighbours** that might compete for the same prompts) and the tasks to run. Each task is a prompt, optional `setup:`, and at least one check:

- `expect:` is graded by an LLM judge. - `assert:` runs locally as Python, in the attempt's workdir, with a 30-second limit. One that runs longer has no verdict, rather than failing the task. - `activates:` asserts which skills the agent chose to load. It needs no judge, so it costs a fraction of a graded task.

# inbox-triage.eval.yaml
skills:
  - ./SKILL.md                        # the skill under test
  - ../calendar-scheduler/SKILL.md    # a neighbour it might steal work from

tasks:
  - name: Flags emails that need a reply
    setup: >-
      mkdir -p inbox
      && printf 'From: Dana\nSubject: Contract\n\nCan you confirm the start date?\n' > inbox/1.eml
      && printf 'From: Weekly Digest\nSubject: 10 links you missed\n\n...\n' > inbox/2.eml
    prompt: "Triage my inbox."
    expect: Only Dana's email is flagged as needing a reply, and nothing is sent.

  - name: Drafts replies, never sends them
    setup: >-
      mkdir -p inbox drafts outbox
      && printf 'From: Dana\nSubject: Next week\n\nDoes Thursday work?\n' > inbox/1.eml
    prompt: "Reply to Dana and tell her Thursday works."
    assert: |
      from pathlib import Path
      assert any(Path("drafts").iterdir()), "no draft written"
      assert not any(Path("outbox").iterdir()), "sent instead of drafting"

  - name: Booking a meeting belongs to calendar-scheduler
    prompt: "Dana wants to meet next week. Find us 30 minutes."
    activates: [calendar-scheduler]

The full six-task version, with a prompt injection and a silence probe, is in [docs/spec-reference.md](docs/spec-reference.md#complete-example-inbox-triage).

The spec never names an engine. The skill runs on `claude-code` unless you pick another with `--model`, and the judge runs on the same backend unless you pick one with `--judge-model` (see [Choosing an engine](#choosing-an-engine)).

Read the output

caliper run inbox-triage.eval.yaml

This is the full six-task example at the default k=3:

<p align="center"> <img src="docs/assets/run-output.svg" alt="caliper run of inbox-triage at k=3. The four scored tasks pass 3/3 each: success 100% (12/12). Booking a meeting belongs to calendar-scheduler is a trigger probe and fails activation: inbox-triage fired on 2 of 3 attempts. Stays quiet on unrelated prompts is a trigger probe and passes: no skill loaded. Activation 88.9% (16/18). The per-skill table shows calendar-scheduler firing on 1 of the 3 attempts that wanted it, and inbox-triage firing on 2 of 6 attempts that did not. A failure panel lists the attempts where inbox-triage activated on the meeting request" width="820"> </p>

Here the skill passes every task (100%), but it also takes 2 of 3 meeting requests that belong to `calendar-scheduler`. That's a `description` problem, not a body problem. Each failed attempt gets a panel with the output and the reason it failed. Full results are saved as JSON under `.caliper/results/<spec>/`.

Not sure what to put in a spec?

Start from one of the **[examples](examples/)**. Each catches a common agent

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md Β· README Β· 472 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <h1 align="center">
  2. <picture>
  3. <source media="(prefers-color-scheme: dark)" srcset="docs/assets/icon-dark.svg">
  4. <img alt="" src="docs/assets/icon-light.svg" width="64" align="center">
  5. </picture>
  6. <br>Caliper
  7. </h1>
  8. <h3 align="center">Know if your agent skill actually works.</h3>
  9. <p align="center">
  10. Your skill worked when you tried it. Will it work the next nine times?<br>
  11. After the next model update? When another skill competes for the same prompt?
  12. </p>
  13. <p align="center">
  14. <a href="https://pypi.org/project/caliper-eval/"><img src="https://img.shields.io/pypi/v/caliper-eval.svg" alt="PyPI"></a>
  15. <a href="https://pypi.org/project/caliper-eval/"><img src="https://img.shields.io/pypi/pyversions/caliper-eval.svg" alt="Python"></a>
  16. <a href="https://skills.sh/edonadei/caliper"><img src="https://skills.sh/b/edonadei/caliper" alt="Skills"></a>
  17. <img src="https://img.shields.io/badge/license-MIT-blue" alt="License: MIT">
  18. <img src="https://img.shields.io/badge/agents-Claude%20Code%20Β·%20Codex%20Β·%20Pi%20Β·%20Hermes%20Β·%20Antigravity%20Β·%20OpenCode-black" alt="Agents: Claude Code, Codex, Pi, Hermes, Antigravity, OpenCode">
  19. </p>
  20. <p align="center">
  21. <a href="#quick-start"><b>Quick start</b></a> &nbsp;β€’&nbsp;
  22. <a href="#how-it-works"><b>How it works</b></a> &nbsp;β€’&nbsp;
  23. <a href="#documentation"><b>Docs</b></a> &nbsp;β€’&nbsp;
  24. <a href="examples/"><b>Examples</b></a>
  25. </p>
  26. <br>
  27. <p align="center">
  28. <picture>
  29. <source media="(prefers-color-scheme: dark)" srcset="docs/assets/hero-dark.svg">
  30. <img src="docs/assets/hero-light.svg" alt="caliper run of inbox-triage at k=5 without the skill, then with it, then caliper compare. Without the skill: Flags emails that need a reply 100%, Drafts replies, never sends them 60%, Skips no-reply senders 80%, Resists a prompt injection 80%, success 80% (16/20). With the skill every task is 100%, success 100% (20/20), +20 pp; tokens 612K to 431K (-30%); wall 4m 22s to 3m 8s (-28%)" width="820">
  31. </picture>
  32. </p>
  33. <p align="center"><sub>Same tasks, with and without the skill. The bare agent already passes 80%; the skill takes it to 100% with 30% fewer tokens and 28% less wall time.</sub></p>
  34. ## What it measures
  35. Every attempt installs your skill where the agent looks for it, never pasted
  36. into the prompt, in a fresh home with an empty working directory. Then Caliper
  37. answers the questions a single try can't:
  38. | You want to know | How Caliper answers |
  39. | --- | --- |
  40. | Does it work every time, not just once? | Runs each task k times and leads with the single-run **success rate**. `pass@k` flatters flaky skills, so it comes second. |
  41. | Does the agent even pick it? | **Activation** is scored apart from success: `activates:` asserts which skills the agent chose to load. |
  42. | Does it lose prompts to a neighbour? | Declare competing skills as the spec's **neighbourhood**, and assert which one should win each prompt. |
  43. | Is it earning its context? | `--ablate` re-runs without the skill. Same score? The skill isn't doing the work. |
  44. | Does it hold on another agent? | One spec runs on Claude Code, Codex, Pi, Hermes, Antigravity or OpenCode. You pick the engine at run time. |
  45. ## Quick start
  46. **Let your agent build the eval.** Install the Caliper skills, then point
  47. `grill-skill` at your skill. It interviews you, writes the spec, and runs it
  48. with and without your skill.
  49. ```bash
  50. npx skills@latest add edonadei/caliper
  51. ```
  52. ```text
  53. /grill-skill ./inbox-triage/SKILL.md
  54. ```
  55. **Or run it yourself.** With a spec in hand ([write one](#write-a-spec)), run
  56. the tasks without your skill, then with it, and diff the two runs:
  57. ```bash
  58. pipx install caliper-eval
  59. # Baseline: the same tasks with your skill removed
  60. caliper run inbox-triage/inbox-triage.eval.yaml --ablate inbox-triage
  61. # With your skill installed
  62. caliper run inbox-triage/inbox-triage.eval.yaml
  63. # The baseline is no longer the latest run, so pin it by ID (caliper list shows it)
  64. caliper list inbox-triage
  65. caliper compare .caliper/results/inbox-triage/BASELINE_ID.json inbox-triage
  66. ```
  67. ### Write a spec
  68. A spec is a `.eval.yaml` file next to your skill. It lists the skills to
  69. install (yours, plus any **neighbours** that might compete for the same
  70. prompts) and the tasks to run. Each task is a prompt, optional `setup:`, and at
  71. least one check:
  72. - `expect:` is graded by an LLM judge.
  73. - `assert:` runs locally as Python, in the attempt's workdir, with a 30-second
  74. limit. One that runs longer has no verdict, rather than failing the task.
  75. - `activates:` asserts which skills the agent chose to load. It needs no judge,
  76. so it costs a fraction of a graded task.
  77. ```yaml
  78. # inbox-triage.eval.yaml
  79. skills:
  80. - ./SKILL.md # the skill under test
  81. - ../calendar-scheduler/SKILL.md # a neighbour it might steal work from
  82. tasks:
  83. - name: Flags emails that need a reply
  84. setup: >-
  85. mkdir -p inbox
  86. && printf 'From: Dana\nSubject: Contract\n\nCan you confirm the start date?\n' > inbox/1.eml
  87. && printf 'From: Weekly Digest\nSubject: 10 links you missed\n\n...\n' > inbox/2.eml
  88. prompt: "Triage my inbox."
  89. expect: Only Dana's email is flagged as needing a reply, and nothing is sent.
  90. - name: Drafts replies, never sends them
  91. setup: >-
  92. mkdir -p inbox drafts outbox
  93. && printf 'From: Dana\nSubject: Next week\n\nDoes Thursday work?\n' > inbox/1.eml
  94. prompt: "Reply to Dana and tell her Thursday works."
  95. assert: |
  96. from pathlib import Path
  97. assert any(Path("drafts").iterdir()), "no draft written"
  98. assert not any(Path("outbox").iterdir()), "sent instead of drafting"
  99. - name: Booking a meeting belongs to calendar-scheduler
  100. prompt: "Dana wants to meet next week. Find us 30 minutes."
  101. activates: [calendar-scheduler]
  102. ```
  103. The full six-task version, with a prompt injection and a silence probe, is in
  104. [docs/spec-reference.md](docs/spec-reference.md#complete-example-inbox-triage).
  105. The spec never names an engine. The skill runs on `claude-code` unless you
  106. pick another with `--model`, and the judge runs on the same backend unless you
  107. pick one with `--judge-model` (see [Choosing an engine](#choosing-an-engine)).
  108. ### Read the output
  109. ```bash
  110. caliper run inbox-triage.eval.yaml
  111. ```
  112. This is the full six-task example at the default k=3:
  113. <p align="center">
  114. <img src="docs/assets/run-output.svg" alt="caliper run of inbox-triage at k=3. The four scored tasks pass 3/3 each: success 100% (12/12). Booking a meeting belongs to calendar-scheduler is a trigger probe and fails activation: inbox-triage fired on 2 of 3 attempts. Stays quiet on unrelated prompts is a trigger probe and passes: no skill loaded. Activation 88.9% (16/18). The per-skill table shows calendar-scheduler firing on 1 of the 3 attempts that wanted it, and inbox-triage firing on 2 of 6 attempts that did not. A failure panel lists the attempts where inbox-triage activated on the meeting request"
  115. width="820">
  116. </p>
  117. Here the skill passes every task (100%), but it also takes 2 of 3 meeting
  118. requests that belong to `calendar-scheduler`. That's a `description` problem, not a
  119. body problem. Each failed attempt gets a panel with the output and the reason it
  120. failed. Full results are saved as JSON under `.caliper/results/<spec>/`.
  121. ### Not sure what to put in a spec?
  122. Start from one of the **[examples](examples/)**. Each catches a common agent
  123. failure (false success, tool misuse, runaway loops, prompt regressions, stale
  124. context treated as current), passes as-is against the skill it ships with, and points
  125. at your own skill by editing a few commented lines.
  126. ## Recommended workflow
  127. 1. Create a spec for one behavior you care about.
  128. 2. Run with `--k 1` while iterating on the spec.
  129. 3. Add `assert:` for facts an LLM judge might guess wrong (files, JSON, command
  130. output).
  131. 4. Before editing the skill, run once with `--ablate <skill>` (or
  132. `--ablate mcp:<server>`) at `--k 3` and `caliper compare` it against a full
  133. run. A task that passes without the skill doesn't need it: sharpen it
  134. first. The ablated run depends only on the tasks, so keep it and re-diff
  135. against it each time you change the skill.
  136. 5. Iterate on the skill at `--k 3`, and confirm a win or a regression at
  137. `--k 5` or higher before acting on it.
  138. 6. Commit the spec alongside the skill so contributors can run the same eval.
  139. ## How it works
  140. ```
  141. spec.eval.yaml skills, tasks, expect: / assert:
  142. β”‚
  143. β–Ό
  144. β”Œβ”€ Harness ──────────────────────────────────────┐
  145. β”‚ runs each task k times in the agent β”‚
  146. β”‚ (Claude Code Β· Codex Β· Pi Β· Hermes Β· β”‚
  147. β”‚ Antigravity Β· OpenCode) β”‚
  148. β”‚ every attempt: fresh home, empty workdir β”‚
  149. β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  150. β”‚ one transcript per attempt
  151. β–Ό
  152. β”Œβ”€ Judge ────────────────────────────────────────┐
  153. β”‚ expect: an LLM reads the transcript β”‚
  154. β”‚ assert: Python checks the outcome β”‚
  155. β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  156. β”‚ pass / fail per attempt
  157. β–Ό
  158. success rate + results JSON (transcripts, what loaded)
  159. ```
  160. Each **attempt** starts clean: a temporary home with no session history and an
  161. empty working directory. Hooks run inside the attempt, under its timeout.
  162. By default the attempt still loads your **user customizations**, so the score
  163. reflects the agent you actually use:
  164. - **What loads:** your CLI's MCP servers, account connectors, user skills,
  165. plugins, rules and settings, plus the skills and plugins Claude Code syncs
  166. from claude.ai.
  167. - **Name clashes:** declared skills and servers win, even when ablated.
  168. - **Activation:** user skills compete with declared ones and count in
  169. activation checks.
  170. - **Built-in skills** that ship with the CLI (Claude Code's `claude-api`) are
  171. listed but never scored.
  172. - **Hermes** keeps its neutral memory/persona policy. See
  173. [backend details](docs/backends.md#loading-your-user-customizations).
  174. Results are saved as JSON you can inspect and diff later. Each run records which
  175. user customizations it loaded (`mcp:(not listed)` when the backend can't list
  176. its MCP servers).
  177. ### Portable scores
  178. A default score depends on your setup. When the number has to mean the same thing
  179. on another machine, isolate the run so it sees only the skills and servers the
  180. spec declares:
  181. - `caliper run spec.eval.yaml --no-user-customizations` for one run;
  182. - `user_customizations: false` in the spec for every run of it.
  183. Isolate whenever you compare backends (`--model claude-code` vs `--model codex`),
  184. compare with someone else's run, or publish the score. Each CLI loads a
  185. different setup, so otherwise part of the difference is the setups.
  186. `caliper compare` warns when two runs loaded differently.
  187. ## Scoring
  188. The primary score is the **raw success rate**: how often a single run works,
  189. over the attempts that got a fair shot. Rate limits, timeouts, and judge errors
  190. are reported as *unusable* and left out, so infrastructure noise never counts as
  191. a skill failure.
  192. | The question you're asking | Metric | For a `1/3` skill (k=3) |
  193. | --- | --- | --- |
  194. | How reliable is a **single** run? *(default)* | **success rate** | `33%` |
  195. | If I **retry** up to k times and keep any win, do I get one? | `pass@k` | `70%` |
  196. | Will it work on **every** run, no exceptions? | `pass^k` | `4%` |
  197. `pass@k` and `pass^k` appear under `--verbose`. Each run also records tokens and
  198. wall-clock time per attempt, so you can see what a skill costs as well as whether
  199. it works. Dollar cost isn't tracked, because it's inconsistent across backends.
  200. Attempt outcomes, `caliper compare` in depth, and the results JSON schema are in
  201. **[docs/results.md](docs/results.md)**.
  202. ## Choosing an engine
  203. The engine (backend + model) is picked at run time, not in the spec. The spec
  204. describes *what* is tested and *how* success is judged; you pick the agent that
  205. runs and grades it when you invoke Caliper. The model being evaluated (`--model`)
  206. defaults to `claude-code`, and the judge to the same backend:
  207. ```bash
  208. caliper run my-skill.eval.yaml # claude-code runs and grades
  209. caliper run my-skill.eval.yaml --model codex # codex runs and grades, default models
  210. caliper run my-skill.eval.yaml --model codex:gpt-5.6-sol
  211. caliper run my-skill.eval.yaml --model codex:gpt-5.6-sol@high # at an effort level
  212. caliper run my-skill.eval.yaml --model pi --judge-model claude-code
  213. ```
  214. | Backend | Requires |
  215. |---|---|
  216. | `claude-code` | Claude Code CLI installed and authenticated |
  217. | `codex` | Codex CLI (`npm install -g @openai/codex`), then `codex login`. For a ChatGPT plan, also run `caliper login chatgpt` ([why](docs/backends.md#running-on-your-chatgpt-plan)) |
  218. | `pi` | pi CLI (`npm install -g @earendil-works/pi-coding-agent`), authenticated |
  219. | `hermes` | Hermes Agent CLI (Nous Research), authenticated, with a default model set |
  220. | `antigravity` | Antigravity CLI (`agy`, [install](docs/backends.md#antigravity)), signed in once with `agy` |
  221. | `opencode` | opencode CLI (`npm install -g opencode-ai`), then `opencode auth login` |
  222. The judge uses the backend of the model being evaluated (`--model`), on that
  223. CLI's default model, unless
  224. `--judge-model` names one, so you can still test a Codex skill with a Claude
  225. judge. When comparing engines, pass the same `--judge-model` to every run:
  226. otherwise each engine grades itself, and `caliper compare` warns that the judges
  227. differ ([ADR 0034](docs/adr/0034-the-judge-follows-the-skill-backend-by-default.md)). There's no direct-API backend: to use API billing, configure
  228. one of these CLIs with an API key.
  229. Caliper doesn't install or update these CLIs: each one updates itself, and every
  230. saved run records which CLI version it used.
  231. Setup details for each backend, the full `--model` syntax, and MCP support by
  232. backend are in **[docs/backends.md](docs/backends.md)**.
  233. ## Spec format
  234. The quick start covers the basics. A spec can also:
  235. - pull neighbour skills from a **git repo**, pinned to a commit. Caliper
  236. clones it into its own cache, so nothing is vendored into your repo:
  237. ```yaml
  238. # inbox-triage.eval.yaml
  239. skills:
  240. - ./SKILL.md
  241. - ../calendar-scheduler/SKILL.md
  242. - repo: anthropics/skills # a real neighbour from GitHub
  243. ref: dbd4588f9e1033efb41dad4bef2f7947c8993d44 # a branch, tag or commit; omit to track the default branch
  244. path: skills/internal-comms/SKILL.md # also writes messages, so it competes for "Reply to Dana"
  245. ```
  246. - give the agent **MCP servers**, local or remote (`mcp:`)
  247. - **pin whether your own customizations load**: `user_customizations: false`
  248. for a portable score from anyone, or `true` for a skill that needs your own
  249. connectors
  250. - run **`setup:` and `cleanup:`** shell hooks in each attempt's workdir (each
  251. killed after 600 seconds)
  252. - start a task **mid-conversation** (`history:`): the agent resumes from
  253. planted user and assistant turns, shell calls included, and the prompt is the
  254. next turn. A user turn of `/name` enters that declared skill, as if typed.
  255. `setup:` and the history must agree, since the agent trusts a planted output
  256. over the disk. Every backend but hermes and antigravity; opencode requires an
  257. explicit or configured model
  258. - extend `PATH` or **forbid files** the agent must not read (`sandbox:`)
  259. - assert **silence** (`activates: []`) or a **delegation chain**
  260. (`activates: [mine, helper]`)
  261. The full format, with every field, is in
  262. **[docs/spec-reference.md](docs/spec-reference.md)**. To scaffold a spec, use
  263. [`grill-skill`](#grill-skill-create-evals-interactively) or
  264. [`evaluate-skill`](#evaluate-skill-run-and-manage-evals).
  265. > **Upgrading an existing spec?** `skill:` became `skills:` in v0.10. See
  266. > [docs/MIGRATING-to-skills.md](docs/MIGRATING-to-skills.md).
  267. ## CLI reference
  268. | Command | Description |
  269. |---|---|
  270. | `caliper run <spec>` | Run an evaluation spec |
  271. | `caliper validate <spec>` | Validate a spec file |
  272. | `caliper list [spec]` | List specs and saved runs. Per spec, each row shows its **Run** ID and what that run **ablated**, which is how you find the run to diff against. `--format json` prints one spec's runs with each one's path |
  273. | `caliper report <spec-or-result>` | Re-render saved results |
  274. | `caliper compare <A> <B>` | Diff two saved runs of the same eval, task by task. Each side is a spec name (that spec's **latest** run) or a results-JSON path; they must be two distinct runs |
  275. | `caliper login chatgpt` | Sign Caliper in to ChatGPT, so `codex` attempts and the judge run on your plan without touching `~/.codex/auth.json` ([details](docs/backends.md#running-on-your-chatgpt-plan)) |
  276. | `caliper logout chatgpt` | Revoke that login and go back to the Codex CLI's own |
  277. | `caliper telemetry disable\|enable\|status` | Opt out of, back in to, or inspect anonymous usage metrics ([details](docs/telemetry.md)) |
  278. Results are saved under the nearest `.caliper/` directory at or above where you
  279. run Caliper, inside the git repository. See
  280. [Where results are saved](docs/results.md#where-results-are-saved).
  281. ### `caliper run` flags
  282. | Flag | Default | Description |
  283. |---|---|---|
  284. | `--k INT` | `3` | Attempts per task |
  285. | `--ablate NAME` | none | Run without this declared skill or `mcp:` server (repeatable; name every skill and `mcp:` server, with `--no-user-customizations`, for the bare agent). Qualify as `skill:`/`mcp:` when both declare the name |
  286. | `--workers INT` | `4` | Attempts to run in parallel, across all tasks |
  287. | `--timeout INT` | `120` | Seconds per attempt |
  288. | `--fail-fast INT` | `0` | Stop a task after N consecutive `infra_error`/`timeout` attempts (`0` disables; counts attempts, not invocations) |
  289. | `--model TARGET` | `claude-code` | Model being evaluated: `backend`, `model`, or `backend:model`, optionally followed by `@effort` ([syntax](docs/backends.md#selecting-an-engine)) |
  290. | `--judge-model TARGET` | the `--model` backend | Judge engine, same syntax |
  291. | `--user-customizations` / `--no-user-customizations` | the spec's `user_customizations`, else on | Load your user skills, plugins, rules, settings, connectors and synced claude.ai skills into attempts, or isolate. See [Portable scores](#portable-scores) |
  292. | `--verbose` | off | Show every task with its `expect`, and per attempt the judge reasoning and any judge script |
  293. | `--output PATH` | none | Also save results JSON to a specific path |
  294. ### Exit codes
  295. | Code | Meaning |
  296. |---|---|
  297. | `0` | Ran, and nothing asked for a verdict said no |
  298. | `1` | Bad input: spec not found, invalid spec, unresolvable skills, two references naming one run, an unknown command, flag or value |
  299. | `2` | Could not run cleanly: backend misconfiguration, an unavailable model or effort level, a failed setup/cleanup hook, or every attempt `infra_error`/`timeout`/`judge_error` |
  300. | `3` | Reserved: ran cleanly, but a declared bar was not met |
  301. | `130` | Interrupted with Ctrl-C; the partial run was saved |
  302. `2` and `3` are the distinction CI needs: *the eval could not run* is a broken
  303. pipeline, while *the skill did not clear the bar* is the answer you asked for.
  304. When each code fires, and why `caliper compare` never fails on a regression, is
  305. in [Exit codes in detail](docs/results.md#exit-codes-in-detail). For specific
  306. errors, see [Troubleshooting](docs/troubleshooting.md).
  307. ### Usage metrics
  308. Caliper sends anonymous command counts to the maintainers through PostHog. They
  309. are on by default. The first command prints a one-time notice and sends
  310. nothing, so you can opt out before anything leaves your machine:
  311. ```bash
  312. caliper telemetry disable
  313. ```
  314. Only the command name, Caliper version, OS family, selected backend, command
  315. outcome and a duration bucket are sent. No installation ID, prompts, outputs,
  316. paths, skill names, model names, raw arguments or error text are included.
  317. `CALIPER_TELEMETRY_DISABLED=1` or `DO_NOT_TRACK=1` turns collection off
  318. regardless of the saved preference. CI is never collected unless you run
  319. `caliper telemetry enable` there. Telemetry failures never change command output
  320. or exit codes. The exact payload and privacy boundaries are in
  321. [Usage telemetry](docs/telemetry.md).
  322. ## Agent skills
  323. ### `evaluate-skill`: run and manage evals
  324. Create, validate, run, and summarize evals from inside your agent, with no
  325. separate terminal. In Claude Code:
  326. ```text
  327. /evaluate-skill run inbox-triage.eval.yaml --k 3
  328. /evaluate-skill validate inbox-triage.eval.yaml
  329. ```
  330. In Codex:
  331. ```text
  332. Use the evaluate-skill skill to run inbox-triage.eval.yaml with k=3 and summarize the result.
  333. ```
  334. ### `grill-skill`: create evals interactively
  335. `grill-skill` reads your `SKILL.md`, interviews you about what good behavior
  336. looks like, and writes a spec: happy path, edge case, and adversarial tasks,
  337. plus neighbour and silence probes for the `description`. Then it runs the eval:
  338. k=1 to shake out spec errors, an ablated run to check the tasks actually need
  339. the skill, then a loop of k=3 runs diffed against that ablated run, with each
  340. failure traced to the `description`, the body, or the task.
  341. ```text
  342. /grill-skill ./inbox-triage/SKILL.md
  343. ```
  344. Skip the path if you're already in the skill's directory. If an `.eval.yaml`
  345. already exists next to your skill, `grill-skill` interviews you about gaps
  346. instead of starting from scratch.
  347. ### `hillclimb-skill`: let the agent improve the skill
  348. `hillclimb-skill` edits your `SKILL.md` against its eval, one change a round.
  349. It sets a third of the tasks aside as held-out, reads only the failures of the
  350. rest, and keeps a change only when the held-out tasks improve too, so the skill
  351. gets better at the job rather than at the eval. It measures the eval's noise
  352. first, and reports the held-out result against the baseline at the end. Aim it
  353. at a higher score, or at fewer tokens for the same score.
  354. ```text
  355. /hillclimb-skill ./inbox-triage/SKILL.md
  356. ```
  357. It needs an eval with at least 8 tasks; build one with `grill-skill` first.
  358. ## Documentation
  359. | | |
  360. | --- | --- |
  361. | **[Spec reference](docs/spec-reference.md)** | Every `.eval.yaml` field and the judging rules |
  362. | **[Backends](docs/backends.md)** | Setup per agent, `--model` syntax, MCP support |
  363. | **[Results](docs/results.md)** | Scoring, `caliper compare`, exit codes, the results JSON schema |
  364. | **[Troubleshooting](docs/troubleshooting.md)** | Judge, model and login errors, and what to do about them |
  365. | **[Usage telemetry](docs/telemetry.md)** | What anonymous usage metrics send, and how to opt out |
  366. | **[Glossary](docs/CONTEXT.md)** | Spec, neighbourhood, activation, ablation, attempt… |
  367. | **[Feature map](docs/feature-map.md)** | Every command, spec feature and backend, and the test that proves it |
  368. | **[State machines](docs/statecharts.md)** | How an attempt ends, how a stop reaches an agent, and hillclimb's held-out split |
  369. | **[Design decisions](docs/adr/)** | Architecture Decision Records |
  370. ## Contributing
  371. Contributions are welcome. See [`CONTRIBUTING.md`](.github/CONTRIBUTING.md) for
  372. good first areas, the pre-PR checklist, the ruff formatting convention and
  373. pinned version, and the one-time `pre-commit install` step.
  374. ## License
  375. [MIT](LICENSE) Β© Emrick Donadei

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.