Skip to content
Back to search

hamelsmu-evals-skills

MarketplaceWorks with Claude CodeGitHub

Skills for AI Evals to compliment the course: AI Evals For Engineers & PMs

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/hamelsmu-evals-skills/badge.svg)](/listings/hamelsmu-evals-skills)

HTML

<a href="/listings/hamelsmu-evals-skills"><img src="/api/listings/hamelsmu-evals-skills/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

In Claude Code, run these two commands:

# Step 1: Register the plugin repository
/plugin marketplace add hamelsmu/evals-skills

# Step 2: Install the plugin
/plugin install evals-skills@hamelsmu-evals-skills

To upgrade:

/plugin update evals-skills@hamelsmu-evals-skills

After installation, restart Claude Code. The skills will appear as `/evals-skills:<skill-name>`.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 16 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "hamelsmu-evals-skills",
  3. "metadata": {
  4. "description": "Skills for building LLM evaluations"
  5. },
  6. "owner": {
  7. "name": "Hamel Husain"
  8. },
  9. "plugins": [
  10. {
  11. "name": "evals-skills",
  12. "source": "./",
  13. "description": "Skills for building LLM evaluations: pipeline audit, error analysis, synthetic data generation, LLM-as-Judge design, evaluator validation, RAG evaluation, and annotation interfaces."
  14. }
  15. ]
  16. }
README.md · README · 85 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Eval Skills for AI Coding Agents
  2. > [!IMPORTANT]
  3. > This repository is deprecated. The skills moved to [ai-evals-course/evals-skills](https://github.com/ai-evals-course/evals-skills), a plugin called `evals` that Shreya Shankar and Hamel Husain maintain together. Install it with:
  4. >
  5. > ```bash
  6. > npx skills add https://github.com/ai-evals-course/evals-skills
  7. > ```
  8. Skills that guide AI coding agents to help you build LLM evaluations.
  9. These skills guard against common mistakes I've seen helping 50+ companies and teaching students in our [AI Evals course](https://maven.com/parlance-labs/evals?promoCode=evals-info-url). If you're new to evals, see [questions.md](questions.md) for free resources on the fundamentals.
  10. ## New to Evals? Start Here
  11. If you are new to evals, start with the `eval-audit` skill. Give your coding agent these instructions:
  12. > Install the eval skills plugin from https://github.com/hamelsmu/evals-skills, then run /evals-skills:eval-audit on my eval pipeline. Investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. Use other skills in the plugin as recommended by the audit.
  13. The audit isn't a complete solution, but it will catch common problems we've seen in evals. It will also recommend other skills to use to fix the problems.
  14. ## Installation
  15. In Claude Code, run these two commands:
  16. ```bash
  17. # Step 1: Register the plugin repository
  18. /plugin marketplace add hamelsmu/evals-skills
  19. # Step 2: Install the plugin
  20. /plugin install evals-skills@hamelsmu-evals-skills
  21. ```
  22. To upgrade:
  23. ```bash
  24. /plugin update evals-skills@hamelsmu-evals-skills
  25. ```
  26. After installation, restart Claude Code. The skills will appear as `/evals-skills:<skill-name>`.
  27. ## Installation (npx skills)
  28. If you use the open Skills CLI, install from this repo with:
  29. ```bash
  30. npx skills add https://github.com/hamelsmu/evals-skills
  31. ```
  32. Install one skill only:
  33. ```bash
  34. npx skills add https://github.com/hamelsmu/evals-skills --skill eval-audit
  35. ```
  36. Check for updates:
  37. ```bash
  38. npx skills check
  39. npx skills update
  40. ```
  41. ## Available Skills
  42. | Skill | What it does |
  43. |-------|-------------|
  44. | eval-audit | Audit an eval pipeline and surface problems with prioritized severity |
  45. | error-analysis | Guide the user through reading traces and categorizing failures |
  46. | generate-synthetic-data | Create diverse synthetic test inputs using dimension-based tuple generation |
  47. | write-judge-prompt | Design LLM-as-Judge evaluators for subjective quality criteria |
  48. | validate-evaluator | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |
  49. | evaluate-rag | Evaluate retrieval and generation quality in RAG pipelines |
  50. | build-review-interface | Build custom annotation interfaces for human trace review |
  51. Invoke a skill with `/evals-skills:skill-name`, e.g., `/evals-skills:error-analysis`.
  52. ## Write Your Own Skills
  53. These skills are a starting point and only encode common mistakes that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them. Start here, then write your own.
  54. The [meta-skill](meta-skill.md) can help you ground custom skills.
  55. ## Beyond These Skills
  56. These skills handle the parts of eval work that generalize across projects. Much of the process doesn't: production monitoring, CI/CD integration, data analysis, and much more. The [course](https://maven.com/parlance-labs/evals?promoCode=evals-info-url) covers all of it.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.