Skip to content
Back to search

ai-evals-course

MarketplaceWorks with Claude CodeGitHub

Skills that guide AI coding agents to help you build product-specific AI evals.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ai-evals-course--evals-skills/badge.svg)](/listings/ai-evals-course--evals-skills)

HTML

<a href="/listings/ai-evals-course--evals-skills"><img src="/api/listings/ai-evals-course--evals-skills/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install with [npx skills](https://github.com/vercel-labs/skills):

npx skills add https://github.com/ai-evals-course/evals-skills

Install one skill only:

npx skills add https://github.com/ai-evals-course/evals-skills --skill error-discovery

Check for updates:

npx skills check
npx skills update

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 16 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "ai-evals-course",
  3. "metadata": {
  4. "description": "Skills for building LLM evaluations"
  5. },
  6. "owner": {
  7. "name": "Shreya Shankar and Hamel Husain"
  8. },
  9. "plugins": [
  10. {
  11. "name": "evals",
  12. "source": "./",
  13. "description": "Skills for building LLM evals. Starts with interactive error discovery: build a review app, sample diverse traces, and organize human annotations into failure modes."
  14. }
  15. ]
  16. }
README.md · README · 87 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Eval Skills
  2. Skills that guide AI coding agents to help you build [product-specific](https://hamel.dev/notes/llm/evals/) AI evals (not foundation model benchmarks).
  3. These skills guard against common mistakes we've seen helping 50+ companies and teaching thousands of students in our [AI Evals course](https://maven.com/parlance-labs/evals?promoCode=evals-info-url).
  4. ## Why skills for evals
  5. There are many [easily avoidable footguns](https://hamel.dev/blog/posts/revenge/) in evals. These skills help you avoid them.
  6. **evals-start** is the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two:
  7. - **[eval-audit](skills/eval-audit/SKILL.md)**, if you already have an eval pipeline. It inspects your setup and recommends next steps.
  8. - **[error-discovery](skills/error-discovery/SKILL.md)**, if you have traces but haven't analyzed them yet. It builds an annotation interface and helps you sample traces. Watch [Shreya's walkthrough](https://youtu.be/tqUDjc1HzO4).
  9. ## Installation
  10. Install with [npx skills](https://github.com/vercel-labs/skills):
  11. ```bash
  12. npx skills add https://github.com/ai-evals-course/evals-skills
  13. ```
  14. Install one skill only:
  15. ```bash
  16. npx skills add https://github.com/ai-evals-course/evals-skills --skill error-discovery
  17. ```
  18. Check for updates:
  19. ```bash
  20. npx skills check
  21. npx skills update
  22. ```
  23. ## Available skills
  24. | Skill | What it does |
  25. |-------|-------------|
  26. | [evals-start](skills/evals-start/SKILL.md) | Entry point. Routes to the skill that matches your situation |
  27. | [eval-audit](skills/eval-audit/SKILL.md) | Audit an eval pipeline and surface problems with prioritized severity |
  28. | [error-discovery](skills/error-discovery/SKILL.md) | Build a review app, select diverse samples, and organize your notes into failure modes |
  29. | [generate-synthetic-data](skills/generate-synthetic-data/SKILL.md) | Create diverse synthetic test inputs using dimension-based tuple generation |
  30. | [write-code-eval](skills/write-code-eval/SKILL.md) | Write code checks for objective failure modes |
  31. | [write-judge-prompt](skills/write-judge-prompt/SKILL.md) | Design LLM-as-Judge evaluators for subjective quality criteria |
  32. | [validate-evaluator](skills/validate-evaluator/SKILL.md) | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |
  33. | [evaluate-rag](skills/evaluate-rag/SKILL.md) | Evaluate retrieval and generation quality in RAG pipelines |
  34. | [build-review-interface](skills/build-review-interface/SKILL.md) | Build custom annotation interfaces for human trace review |
  35. ## The error-discovery skill
  36. The `error-discovery` skill in the plugin is the most important. Error discovery involves qualitative and quantitative analysis of your traces to find failure modes. You should only write evals after doing this step.
  37. This skill makes an AI agent run error analysis on a dataset. Point it at a
  38. JSONL/CSV/JSON file of LLM outputs or traces, and it:
  39. 1. Reads the dataset and figures out the content type (articles, agent traces,
  40. code, structured output, etc.).
  41. 2. Designs visual encoding based on what varies in the data. Uses Gestalt
  42. principles (color for categories, spacing for hierarchy, opacity for
  43. importance).
  44. 3. Builds a single-file HTML review app served by a Python stdlib server. No
  45. dependencies.
  46. 4. Clusters the data and picks a diverse initial sample (cluster reps + random
  47. picks).
  48. 5. Runs an interactive loop: monitors annotations, categorizes failure modes,
  49. proposes new samples to increase coverage.
  50. ![Error Discovery Skill Workflow](workflow.png)
  51. You read and leave free-text notes. The agent sorts them into failure modes,
  52. tracks coverage, and picks new samples to fill gaps.
  53. Once installed, point the agent at a dataset:
  54. ```
  55. Can you help me do error analysis on traces.jsonl?
  56. ```
  57. ## Write your own skills
  58. These skills address common eval mistakes. Start here, then write skills for your own data and domain. Matt Pocock's [writing-for-agents skill](https://github.com/mattpocock/skills/tree/main/skills/productivity/writing-for-agents) explains how to write instructions for agents.
  59. ## Beyond these skills
  60. These skills cover eval work that applies across projects. The [AI Evals course](https://maven.com/parlance-labs/evals?promoCode=evals-info-url) also covers production monitoring, regression suites, and cost.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.