hamelsmu-evals-skills
Skills for AI Evals to compliment the course: AI Evals For Engineers & PMs
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/hamelsmu-evals-skills)HTML
<a href="/listings/hamelsmu-evals-skills"><img src="/api/listings/hamelsmu-evals-skills/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
In Claude Code, run these two commands:
# Step 1: Register the plugin repository /plugin marketplace add hamelsmu/evals-skills # Step 2: Install the plugin /plugin install evals-skills@hamelsmu-evals-skills
To upgrade:
/plugin update evals-skills@hamelsmu-evals-skills
After installation, restart Claude Code. The skills will appear as `/evals-skills:<skill-name>`.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 16 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- {
- "name": "hamelsmu-evals-skills",
- "metadata": {
- "description": "Skills for building LLM evaluations"
- },
- "owner": {
- "name": "Hamel Husain"
- },
- "plugins": [
- {
- "name": "evals-skills",
- "source": "./",
- "description": "Skills for building LLM evaluations: pipeline audit, error analysis, synthetic data generation, LLM-as-Judge design, evaluator validation, RAG evaluation, and annotation interfaces."
- }
- ]
- }
README.md · README · 85 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # Eval Skills for AI Coding Agents
- > [!IMPORTANT]
- > This repository is deprecated. The skills moved to [ai-evals-course/evals-skills](https://github.com/ai-evals-course/evals-skills), a plugin called `evals` that Shreya Shankar and Hamel Husain maintain together. Install it with:
- >
- > ```bash
- > npx skills add https://github.com/ai-evals-course/evals-skills
- > ```
- Skills that guide AI coding agents to help you build LLM evaluations.
- These skills guard against common mistakes I've seen helping 50+ companies and teaching students in our [AI Evals course](https://maven.com/parlance-labs/evals?promoCode=evals-info-url). If you're new to evals, see [questions.md](questions.md) for free resources on the fundamentals.
- ## New to Evals? Start Here
- If you are new to evals, start with the `eval-audit` skill. Give your coding agent these instructions:
- > Install the eval skills plugin from https://github.com/hamelsmu/evals-skills, then run /evals-skills:eval-audit on my eval pipeline. Investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. Use other skills in the plugin as recommended by the audit.
- The audit isn't a complete solution, but it will catch common problems we've seen in evals. It will also recommend other skills to use to fix the problems.
- ## Installation
- In Claude Code, run these two commands:
- ```bash
- # Step 1: Register the plugin repository
- /plugin marketplace add hamelsmu/evals-skills
- # Step 2: Install the plugin
- /plugin install evals-skills@hamelsmu-evals-skills
- ```
- To upgrade:
- ```bash
- /plugin update evals-skills@hamelsmu-evals-skills
- ```
- After installation, restart Claude Code. The skills will appear as `/evals-skills:<skill-name>`.
- ## Installation (npx skills)
- If you use the open Skills CLI, install from this repo with:
- ```bash
- npx skills add https://github.com/hamelsmu/evals-skills
- ```
- Install one skill only:
- ```bash
- npx skills add https://github.com/hamelsmu/evals-skills --skill eval-audit
- ```
- Check for updates:
- ```bash
- npx skills check
- npx skills update
- ```
- ## Available Skills
- | Skill | What it does |
- |-------|-------------|
- | eval-audit | Audit an eval pipeline and surface problems with prioritized severity |
- | error-analysis | Guide the user through reading traces and categorizing failures |
- | generate-synthetic-data | Create diverse synthetic test inputs using dimension-based tuple generation |
- | write-judge-prompt | Design LLM-as-Judge evaluators for subjective quality criteria |
- | validate-evaluator | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |
- | evaluate-rag | Evaluate retrieval and generation quality in RAG pipelines |
- | build-review-interface | Build custom annotation interfaces for human trace review |
- Invoke a skill with `/evals-skills:skill-name`, e.g., `/evals-skills:error-analysis`.
- ## Write Your Own Skills
- These skills are a starting point and only encode common mistakes that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them. Start here, then write your own.
- The [meta-skill](meta-skill.md) can help you ground custom skills.
- ## Beyond These Skills
- These skills handle the parts of eval work that generalize across projects. Much of the process doesn't: production monitoring, CI/CD integration, data analysis, and much more. The [course](https://maven.com/parlance-labs/evals?promoCode=evals-info-url) covers all of it.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.