Skip to content
Back to search

calibre-labs

MarketplaceWorks with Claude CodeGitHub

Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Calibre-Labs/reforge-ai-evals with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Calibre-Labs/reforge-ai-evals'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/calibre-labs--reforge-ai-evals/badge.svg)](/listings/calibre-labs--reforge-ai-evals)

HTML

<a href="/listings/calibre-labs--reforge-ai-evals"><img src="/api/listings/calibre-labs--reforge-ai-evals/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**Any coding agent**, with [npx skills](https://github.com/vercel-labs/skills):

npx skills add https://github.com/Calibre-Labs/reforge-ai-evals

Add `--skill trace-codes` to install one skill, and run `npx skills update` to get new versions.

**Claude Code plugin:**

claude plugin marketplace add Calibre-Labs/reforge-ai-evals
claude plugin install ai-evals@calibre-labs

**Manual:** copy any folder in [`skills/`](skills/) into your agent's skills directory. The skills read and write plain files (markdown, CSV, JSON, HTML), so you don't need an eval platform account.

Then ask your agent something like:

Write a v1 eval rubric from my PRD.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 16 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "calibre-labs",
  3. "metadata": {
  4. "description": "Skills from Calibre Labs for evaluating AI products"
  5. },
  6. "owner": {
  7. "name": "Calibre Labs"
  8. },
  9. "plugins": [
  10. {
  11. "name": "ai-evals",
  12. "source": "./",
  13. "description": "Skills for evaluating an AI product, from the eval rubric in the PRD to judges checked against your labels."
  14. }
  15. ]
  16. }
README.md · README · 133 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # AI Evals Toolkit
  2. Skills that guide AI coding agents (Claude Code, Codex, Cursor and others) to build evals for your AI product: eval rubrics, eval datasets, trace analysis, code-based evaluators, LLM-as-a-judge prompts and judge alignment.
  3. From the [Reforge AI Evals course](https://www.reforge.com/courses/ai-evals) by Sandhya Hegde and Justin Bauer of [Calibre](https://calibrelabs.ai).
  4. ## How it works
  5. Your coding agent can do most of the eval work if it has the right instructions. Two steps stay with you: **reading traces** and **labeling outputs** to calibrate judges. The skills cover the rest.
  6. Start with **[eval-guide](skills/eval-guide/SKILL.md)**. It looks at where you are and loads the right skill.
  7. New to evals, or want to learn the method before using it on your own product? Ask **[eval-coach](skills/eval-coach/SKILL.md)** a question ("what's a trace code?", "when do I need an LLM judge instead of a code check?") or say "teach me evals". It answers from the course's FAQs, course reader and live sessions, draws the diagrams, and walks you through every skill on Mapper, the course agent.
  8. ![The AI evals loop: your coding agent runs six steps with skills, and two steps, reading traces and labeling a sample, stay with you](docs/workflow.png)
  9. | Step | Skill | What it does |
  10. |------|-------|--------------|
  11. | 1. Define good | [eval-rubric](skills/eval-rubric/SKILL.md) | Writes the eval rubric into your PRD: v1 before traces, v2 after |
  12. | 2. Build test inputs | [uig](skills/uig/SKILL.md) | Designs a diverse eval dataset with a User Input Grid and finds coverage gaps |
  13. | 3. Read traces | **You** | Annotate what went wrong, in your own words |
  14. | Optional, for step 3 | [trace-review-app](skills/trace-review-app/SKILL.md) | Designs and builds a one-file review app for your product's traces, with notes, labels, trace codes and exports (draft) |
  15. | 4. Find failure modes | [trace-codes](skills/trace-codes/SKILL.md) | Clusters your notes into named yes/no failure codes |
  16. | 5. Write evaluators | [eval-code](skills/eval-code/SKILL.md), [eval-llm-judge](skills/eval-llm-judge/SKILL.md) | Code checks for objective failures, LLM judges for subjective ones |
  17. | 6. Label a sample | **You** | Mark outputs pass or fail |
  18. | 7. Calibrate judges | [llm-align](skills/llm-align/SKILL.md) | Measures each judge against your labels (TPR/TNR) and fixes its prompt |
  19. | 8. Grow the dataset | [ticket-to-eval](skills/ticket-to-eval/SKILL.md) | Turns support tickets and bad traces into regression eval rows |
  20. | Any time | [eval-coach](skills/eval-coach/SKILL.md) | Answers questions from the course material and teaches each skill in short lessons on Mapper |
  21. Coming next: `eval-run` (run the agent and scorers over a dataset), `eval-trace` (tool calls and multi-step traces) and `eval-monitor` (score production and report drift).
  22. ## Install
  23. **Any coding agent**, with [npx skills](https://github.com/vercel-labs/skills):
  24. ```bash
  25. npx skills add https://github.com/Calibre-Labs/reforge-ai-evals
  26. ```
  27. Add `--skill trace-codes` to install one skill, and run `npx skills update` to get new versions.
  28. **Claude Code plugin:**
  29. ```bash
  30. claude plugin marketplace add Calibre-Labs/reforge-ai-evals
  31. claude plugin install ai-evals@calibre-labs
  32. ```
  33. **Manual:** copy any folder in [`skills/`](skills/) into your agent's skills directory. The skills read and write plain files (markdown, CSV, JSON, HTML), so you don't need an eval platform account.
  34. Then ask your agent something like:
  35. ```
  36. Write a v1 eval rubric from my PRD.
  37. ```
  38. or, to learn first:
  39. ```
  40. Teach me evals.
  41. ```
  42. ## Worked examples
  43. - **[Templates](templates/):** an AI PRD template, plus PRDs, v1 and v2 eval rubrics, simulated sessions and annotated traces for three agents: Corner (consumer shopping), Ledger (enterprise finance) and Mapper (market research, with 10 traces to annotate).
  44. - **[Eval pages](templates/eval-pages/):** templates for a results page and a scorer page (code check or LLM judge), with Mapper's Week 3 scorers as the worked example.
  45. - **[Market Map User Input Grid](docs/uig-market-map.md):** a complete worked UIG for the single-prompt market research agent from the spring 2026 cohort.
  46. ## FAQ
  47. <details>
  48. <summary><b>What is an AI eval?</b></summary>
  49. An AI eval is a repeatable test that scores an AI feature's outputs against defined criteria, so you can tell whether a prompt, model or agent change made the product better or worse. A useful setup has three parts: a dataset of realistic inputs, evaluators that score each output, and a loop that reruns them on every change.
  50. </details>
  51. <details>
  52. <summary><b>What is LLM-as-a-judge?</b></summary>
  53. Using a language model to grade another model's output against a written criterion. It's for semantic qualities code can't check, such as whether an answer is correct or an ambiguous request was handled well. A good judge tests one criterion, has binary PASS/FAIL definitions and few-shot examples (including a borderline case), and writes its critique before its score.
  54. </details>
  55. <details>
  56. <summary><b>When should I use a code-based evaluator instead of an LLM judge?</b></summary>
  57. Use code for anything structural and deterministic: counts, required sections, citations, formats. It's fast, free and never drifts. Use an LLM judge for meaning: correctness, relevance, tone, edge-case handling. Most products need both.
  58. </details>
  59. <details>
  60. <summary><b>How do I know whether my LLM judge is trustworthy?</b></summary>
  61. Label 20–40 outputs by hand (roughly half PASS, half FAIL), run the judge on the same outputs, and compute TPR (how often it agrees when a human said PASS) and TNR (how often it agrees when a human said FAIL). Aim for both above 90%, with 80% as the minimum. Track them separately: a judge with high overall accuracy can still be too lenient.
  62. </details>
  63. <details>
  64. <summary><b>What is a User Input Grid?</b></summary>
  65. A method for designing eval datasets that cover the real diversity of inputs an agent will face. List the dimensions along which user inputs vary (such as query type, domain, phrasing style and time frame) and the values each can take, then check that your dataset hits every value several times. It replaces datasets built from engineer intuition, which overfit to what engineers know, or from raw logs, which overfit to easy, common cases. See the [Market Map example](docs/uig-market-map.md).
  66. </details>
  67. <details>
  68. <summary><b>How big should an eval dataset be?</b></summary>
  69. Coverage matters more than size. 30–80 inputs that hit every value of every dimension at least 3 times usually beat 500 random ones. Tag every row with its dimension values so you can slice results and see where the agent fails.
  70. </details>
  71. <details>
  72. <summary><b>What goes in an AI PRD and an eval rubric?</b></summary>
  73. An AI PRD specifies the agent's behavior, not screens, including how much autonomy it has and the release thresholds that gate a launch. The eval rubric lists the criteria the agent is judged on, with the definition of success and known risks next to the criteria they justify. Write v1 from the PRD before reading any traces, then sharpen it into v2 after trace analysis.
  74. </details>
  75. <details>
  76. <summary><b>What are the most common eval mistakes?</b></summary>
  77. - Writing evaluators before reading traces, so they test what you imagined instead of how the agent actually fails.
  78. - Compound judges that check several things at once, which are uninterpretable when they fail.
  79. - Trusting a judge without checking it against human labels.
  80. - Datasets built from intuition or raw logs instead of deliberate coverage.
  81. - Running evals ad hoc instead of on every prompt or model change.
  82. </details>
  83. <details>
  84. <summary><b>Do I need a particular eval platform or model?</b></summary>
  85. No. The skills work in any coding agent that supports skills, and they read and write plain files. `llm-align` and `ticket-to-eval` can also read from platforms such as Braintrust or LangSmith if your data lives there.
  86. </details>
  87. ## About
  88. Built by Sandhya Hegde and Justin Bauer, co-founders of [Calibre](https://calibrelabs.ai), an applied AI research and consulting firm. Calibre writes about AI product quality at [blog.calibrelabs.ai](https://blog.calibrelabs.ai). The [Reforge AI Evals course](https://www.reforge.com/courses/ai-evals) also covers production monitoring, agent success metrics and the eval-driven roadmap.
  89. Code is [MIT](LICENSE). Skills, templates, datasets and docs are [CC BY 4.0](LICENSE-CONTENT). To cite this repo, use GitHub's "Cite this repository" button.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.