eval-studio
Test a chatbot, a retrieval (RAG) system or an AI agent the way software is tested. Test cases are run against the real system (an HTTP endpoint, a shell command, a JavaScript function or an OpenAI-compatible chat endpoint), the answers are scored with plain checks (contains, regex, JSON shape, numbers, "retrieved source X", "called tool Y with these arguments", refusals, latency, length) and an optional, clearly marked model-judged check, each run is compared with the last one, and a single-file HTML report shows what passed, what failed and what got worse. Exits non-zero on regressions, for CI. Use whenever the user wants to test, evaluate, benchmark or regression-test a chatbot, assistant, RAG pipeline, retrieval, prompt change, model switch or agent; asks "did my change make the bot worse", "write test cases for my bot", "evals", "eval suite", "golden set", "LLM tests", "check my RAG answers", "compare two prompts or models"; or wants such tests in CI.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/vinvirtual--eval-studio)HTML
<a href="/listings/vinvirtual--eval-studio"><img src="/api/listings/vinvirtual--eval-studio/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
git clone https://github.com/VinVirtual/eval-studio ~/.claude/skills/eval-studio # any folder works for command-line use ~/.claude/skills/eval-studio/evals check
Optionally put it on your PATH: `ln -s ~/.claude/skills/eval-studio/evals ~/.local/bin/evals`.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 74 lines
- ---
- name: eval-studio
- description: Test a chatbot, a retrieval (RAG) system or an AI agent the way software is tested. Test cases are run against the real system (an HTTP endpoint, a shell command, a JavaScript function or an OpenAI-compatible chat endpoint), the answers are scored with plain checks (contains, regex, JSON shape, numbers, "retrieved source X", "called tool Y with these arguments", refusals, latency, length) and an optional, clearly marked model-judged check, each run is compared with the last one, and a single-file HTML report shows what passed, what failed and what got worse. Exits non-zero on regressions, for CI. Use whenever the user wants to test, evaluate, benchmark or regression-test a chatbot, assistant, RAG pipeline, retrieval, prompt change, model switch or agent; asks "did my change make the bot worse", "write test cases for my bot", "evals", "eval suite", "golden set", "LLM tests", "check my RAG answers", "compare two prompts or models"; or wants such tests in CI.
- ---
- # eval-studio
- Cases → **run** against the target → **score** with checks → **compare** with the last run → **report** page. Plain Node 18+, no packages to install, nothing sent anywhere except to the target (and to a judge model only if the user turns that on).
- Skill folder: `~/.claude/skills/eval-studio/` (written below as `$E`). Every command is `$E/evals <command> [project]`. `$E/evals help` lists them.
- ```
- $E/evals the command line (check, new, import, lint, run, score, compare, report, baseline, build, test)
- $E/engine/lib/checks/ the check types $E/references/checks.md every check and its fields
- $E/engine/lib/adapters/ the target types $E/references/targets.md every target and its settings
- $E/engine/report/ the report page (HTML, CSS, JS, inlined into one file)
- $E/templates/ what `evals new` copies $E/examples/tool-library/ a complete small example
- $E/examples/ci/evals.yml a GitHub Actions workflow that fails a pull request on regressions
- ```
- ## Workflow
- 1. **Check**: `$E/evals check [dir]`. Node version, and in a project the config, the target type and whether the environment variables it names are set.
- 2. **Scaffold**: `$E/evals new <dir>`. Put the folder in the user's repo (for example `evals/`) so cases, the baseline and the workflow are versioned with the code they test.
- 3. **Point it at the system**: set `target` in `evals.config.json` (see `references/targets.md`). Prefer testing the real entry point users hit (`http` or `module`) over the bare model (`openai`), or retrieval is not tested. Keys only as `${NAME}` or `apiKeyEnv`; never write a key into a file.
- 4. **Write cases with the user** (rules below). JSONL, one case per line. A spreadsheet of questions can come in as CSV: `$E/evals import cases.csv`.
- 5. **Lint**: `$E/evals lint <dir>`. Fix every error: duplicate ids, cases with no check, unknown check types, bad patterns or shapes, missing environment variables, a target that does not answer. `--offline` skips the reachability probe.
- 6. **First run**: `$E/evals build <dir> --label first`. Look at the failures with the user before anything else: each one is either a bug in the system or a bad case. Fix bad cases (wrong expected fact, check too strict) and run again.
- 7. **Freeze a baseline** once the results are understood: `$E/evals baseline <dir>` copies the newest scored run to the `baseline` file named in the config. Commit it.
- 8. **After every change** to prompts, model, retrieval, chunking or tools: `$E/evals build <dir> --label <what-changed>`. It lints, runs, scores, compares with the baseline (or the previous run) and writes `report.html`. Exit code 1 means the gate failed: a case got worse (a new case that fails counts), a baseline case is missing (renamed or deleted: rename back, or pass `--allow-removed` once on purpose), the score fell more than 5 points, a case got no answer, or a `must-pass` case did not pass. `--tag fees` runs only some cases without the others counting as missing.
- 9. **Look at the report** before telling the user anything: open `report.html` in a browser, or with Playwright take a screenshot and Read it. Report: the score and its change, every case that **got worse** with the reason, what was fixed, flaky cases, and the latency change. Do not summarise a regression as "minor" without reading the answer.
- 10. **CI**: copy `examples/ci/evals.yml` and adjust paths. Gate settings are in the config (`gate`) or flags (`--max-regressions`, `--max-score-drop`, `--max-latency-increase`).
- ## Writing good cases with the user
- Ask for material before writing anything: real questions from logs, support tickets, the FAQ, and the documents the bot answers from. Then:
- - **Real questions, in users' words.** Typos, vague phrasing and follow-ups are fine; they are what the bot gets. Do not polish them into exam questions.
- - **Expected facts come from the user's own documents, never from your own knowledge.** Quote or point to the passage (`"expected": "30 days (returns.md)"`). If the documents do not say, ask the user or leave the case out. Never invent a price, date, policy or number. A case built on a made-up fact makes a correct bot fail.
- - **Check the fact, not the wording.** `contains` on the key phrase or number, `number` with a tolerance, `source` for the document that holds the answer. Avoid `exact` except for fixed strings (a help line, a JSON field).
- - **Cover the edges**: questions the documents answer only across two passages, near-miss wording, numbers that need a sum, out-of-date terms, a question in another language if users write in one, very long input, an empty or one-word message.
- - **Things the bot must refuse or must not do**: out-of-scope questions (`refusal`), requests for personal data or for advice it must not give (`refusal` plus `not-contains`), a tool it must not call (`no-tool-call`). Ask the user what the bot must never say or do; these are often the most important cases.
- - **Agents**: one case per tool decision. "Must call X with arguments like Y" for the happy path, "must not call Z" for look-alike requests (a question about reserving must not make a reservation).
- - **A small must-never-regress set**: tag 5 to 20 cases `must-pass`. They fail the gate even with no baseline, and a skipped one fails too, so give each a plain check, never only a model-judged one. Pick what would embarrass or harm: wrong prices, safety, privacy, refusals.
- - **Keep ids stable.** The comparison matches cases by id; renaming one shows up as a missing case plus a new one and fails the gate.
- - **Regex**: keep patterns simple. Nested repeats like `(a+)+` are refused, and every regex is stopped after 1 second.
- - **One idea per case, an id that says what it is** (`late-fee-rate`, not `case-14`), a `note` when the case is expected to fail today or exists for a known incident, and tags for the area (`fees`, `retrieval`, `refusal`).
- - **Size**: start with 20 to 50. Add a case for every real bug found, written before the fix.
- - Tell the user plainly which cases you wrote from their documents and which expected values you could not confirm.
- This tool does **not** make up cases from nothing. It runs and scores the cases you write together.
- ## Reading results honestly
- - String checks are blunt. "contains 30 days" passes "it is not 30 days". Read the answers of new and failing cases, not only the colour.
- - A pass shows that these cases pass. It does not show the bot is right in general; say so when reporting.
- - Answers vary. Use `--repeat 3` (or `run.repeat`) before trusting a change in a single case; a case that passes some attempts is **flaky** and counts as not passing.
- - Small score changes on a small set are noise. Talk about which cases changed, not about +1.3 points.
- - **Model-judged checks** (`judge`) are off by default and marked "model-judged" everywhere. Their scores change between runs and judge models and are not ground truth. Turn them on only when the user asks and has configured a judge endpoint (`judge` in the config, or `--judge`), keep a plain check beside each one, and never call a paid API without the user's go-ahead.
- ## Files
- - `evals.config.json`: `name`, `cases`, `target`, `run` (concurrency, retries, timeoutMs, repeat), `baseline`, `gate` (maxRegressions, maxScoreDrop, maxLatencyIncreasePct, mustPassTag), `judge`, `report`, `runs`.
- - A case: `id`, `input` (or `messages` for multi-turn), `expected` (for people), `note`, `tags`, `weight`, `threshold`, `vars` (for request templates), `checks`.
- - `runs/<id>.json`: the questions, every attempt's answer, sources, tool calls, latency, tokens and cost, and (after `score`) every check result. Secrets are masked before saving. `score` can be run again on an old run after the checks change; checks are taken from the current cases file by id (`--frozen` keeps the run's own).
- - `report.html`: one file, no network, safe to attach to a pull request or send.
- ## Troubleshooting
- - **unreachable** in lint: start the system first, or check the URL and port. `--offline` skips the probe.
- - **Every case errors with HTTP 401/403**: the key variable is not set in this shell (`evals check` shows it).
- - **The answer is empty but the target answered**: the response path is wrong. Set `target.response.answer` (see `references/targets.md`); the raw run file shows what came back.
- - **A source check never passes**: the ids differ (`fees.md` vs `fees`, a URL vs a title). Look at `sources` in the run file and use those ids.
- - **Too slow or rate-limited**: lower `run.concurrency`, raise `timeoutMs`.
README.md · README · 134 lines
- # eval-studio
- [](https://github.com/VinVirtual/eval-studio/actions/workflows/test.yml) [](LICENSE)
- 
- Test a chatbot, a retrieval (RAG) system or an AI agent the way software is tested: write test cases, run them against the real system, score the answers with checks, compare with the last run, and get one report page that shows what passed, what failed and **what got worse**.
- It is packaged as a [Claude Code](https://claude.com/claude-code) skill (`SKILL.md` tells the assistant how to use it, including how to write good cases with you), and it works by hand from the command line. Plain Node.js, no packages to install.
- ## What it does
- - **Targets**: an HTTP endpoint (request body template, answer read from the response by path), a shell command (question on stdin, answer on stdout), a function in a JavaScript module, or any OpenAI-compatible chat endpoint. Keys are read from environment variables only.
- - **Running**: several cases at once, retries, a timeout per call, and `--repeat N` to ask every case several times and catch answers that change. It records the answer, the retrieved sources, the tool calls, the latency, and tokens and cost when the target reports them.
- - **Checks**: contains and not-contains, regex, exact, JSON shape and values, a number within a tolerance, "must retrieve or cite source X" (with its rank), "must call tool X with arguments like Y", "must not call tool Z", refusal expected (or not), max latency, max length. Weights per check and per case, tags per case.
- - **Model-judged check (optional)**: a rubric scored by a model you configure, through any OpenAI-compatible endpoint. Off by default, and labelled "model-judged" wherever its results appear.
- - **Comparison**: regressions (passed before, not now, plus new cases that fail), fixes, unchanged, new and missing cases, score and latency changes. A gate for CI exits non-zero when a case got worse, when a baseline case is missing from the run (renamed or deleted), when the score fell more than 5 points, when latency rose too much (if set), when the target gave no answer to a case, or when a `must-pass` case did not pass (skipped counts as not passing). The no-answer and must-pass rules apply even to a first run with no baseline.
- - **Report**: one HTML file with no network access. It has a summary, a trend across saved runs, filters by status and tag, a search, and for each case the question, the expected facts, the answer next to the baseline's answer with changed words marked, every check with its result (and what it was before), the retrieved passages with their rank, and the tool trace. Light and dark themes, readable on a phone.
- - **Cases** in JSONL or JSON, plus CSV import from a spreadsheet.
- - **Secrets** are masked before anything is saved: the values of key-like environment variables, common key formats, `Bearer` tokens, `password=` pairs, and passwords in URLs.
- ## Limits
- Read these before trusting a green report.
- - **String checks are blunt.** "contains 30 days" passes an answer that says "it is not 30 days". A check confirms that a phrase, number or source is present. It does not confirm that the answer is right. Read the answers of new and failing cases.
- - **Model-judged scores are not ground truth.** They are another model's opinion. They change between runs and between judge models, and the answer being graded can sway them. Use them for what string checks cannot see, next to a plain check, never as the only check on something that matters.
- - **Passing cases do not prove the bot is correct in general.** They show that these questions, scored by these checks, pass today. Coverage is only as good as the cases.
- - **It does not generate test cases from nothing.** You (or the assistant, with you) write them from real questions and your own documents. The skill's instructions forbid inventing expected facts.
- - Answers vary from run to run. A single case flipping once is weak evidence; use `--repeat`.
- - Redaction catches obvious secrets, not every secret. Checks in `build` see the raw answers; `score` on a saved run sees the redacted text.
- - Regex patterns with a repeated group that itself repeats are refused, and every regex check is stopped after 1 second. Number checks do not understand decimal commas.
- - A module target that ignores its abort signal keeps running after its timeout until the command exits. Do not put private customer data in cases or reports you share.
- - Tested on Linux only (Node 24). It uses nothing platform-specific beyond `sh` for command targets, but macOS and Windows have not been tested.
- ## Requirements
- | Needed | Why |
- |---|---|
- | Node.js 18 or newer | runs everything (uses the built-in `fetch` and `node --test`) |
- | The system you want to test | reachable from the machine that runs the evals |
- | A browser | to open the report |
- No npm packages, no Python, no account. Nothing is sent anywhere except to your target, and to a judge endpoint only if you turn the model-judged check on.
- ## Install
- ```bash
- git clone https://github.com/VinVirtual/eval-studio ~/.claude/skills/eval-studio # any folder works for command-line use
- ~/.claude/skills/eval-studio/evals check
- ```
- Optionally put it on your PATH: `ln -s ~/.claude/skills/eval-studio/evals ~/.local/bin/evals`.
- ## Quick start
- ```bash
- E=~/.claude/skills/eval-studio
- $E/evals new evals # evals.config.json, cases.jsonl, target.mjs (a stand-in target)
- # point "target" in evals/evals.config.json at your system, write cases in evals/cases.jsonl
- $E/evals lint evals # duplicate ids, cases with no check, unknown checks, target not answering
- $E/evals build evals --label first # run + score + compare + report.html
- $E/evals baseline evals # freeze this run as the baseline (commit the file)
- # change a prompt, the model, the retrieval...
- $E/evals build evals --label new-prompt # exits 1 if anything that passed now fails
- ```
- A case is one line of JSON:
- ```json
- {"id": "late-fee-rate", "input": "What is the late fee per day?", "expected": "2 dollars per tool per day (fees.md)", "tags": ["fees", "must-pass"], "checks": [{"type": "number", "value": 2, "pick": "first"}, {"type": "source", "id": "fees", "maxRank": 1}]}
- ```
- Every check and its fields: [`references/checks.md`](references/checks.md). Every target and its settings: [`references/targets.md`](references/targets.md).
- ### The example
- `examples/tool-library/` is a complete project that needs no key and no network. Its target is a tiny keyword-retrieval question-answer bot (`bot.mjs`) over five short documents about a community tool library. It has 18 cases, some of which fail on purpose (a sum the bot cannot do, an answer that needs two documents, a word it does not understand), plus two saved runs. The baseline run is the first version of the bot. The changed run ignores common words and refuses on weak matches: that fixes one case (it now refuses an off-topic question) and breaks one (it no longer answers "Are you open on Sunday?"). `report.html` is the result.
- ```bash
- cd ~/.claude/skills/eval-studio/examples/tool-library
- ../../evals build --label again # exits 1: hours-sunday got worse since the baseline
- ```
- ## Commands
- | Command | What it does |
- |---|---|
- | `evals check [dir]` | Node version; in a project, the config, the target type and whether the environment variables it needs are set |
- | `evals new <dir>` | starts a project from the templates |
- | `evals import <file.csv> [out.jsonl]` | turns a CSV into cases. Columns: `id`, `input` (or `question`), `expected`, `tags`, `weight`, `contains` (`a\|b` for all), `contains_any`, `not_contains`, `regex`, `exact`, `number`, `tolerance`, `source` (`id@rank`), `tool`, `tool_args` (JSON), `no_tool`, `refusal`, `max_latency_ms`, `max_chars`, `max_words`, `json_shape`, `judge` |
- | `evals lint [dir]` | errors for duplicate ids, cases with no check, unknown check types, bad check fields, missing environment variables, a target that does not answer. `--offline`, `--strict` (fail on warnings), `--json` |
- | `evals run [dir]` | asks the target every case and saves `runs/<id>.json`. Exits 1 when the target answered no case at all. `--label`, `--id` (a plain name; an existing run is replaced only with `--force`), `--repeat N`, `--concurrency N`, `--retries N`, `--timeout MS`, `--tag T` |
- | `evals score [run]` | applies the checks (from the current cases file, by id; `--frozen` keeps the run's own). `--judge` turns the model-judged check on for this command |
- | `evals compare [dir] [base] [new]` | regressions, fixes, unchanged, missing cases, score and latency changes. Exits 1 when the gate fails. `--json`, `--allow-removed`, `--allow-new-failures` |
- | `evals report [run]` | writes `report.html`. `--baseline R`, `--out FILE` |
- | `evals baseline [run]` | copies a scored run to the baseline file named in the config |
- | `evals build [dir]` | lint, run, score, compare, report. Exits 1 when the gate fails (`--no-gate` always exits 0) |
- | `evals test` | runs the unit tests |
- A run can be named by its file, its id or its label. Without one, the newest run is used. The baseline is, in order: the run you name, the `baseline` file in the config, or the newest earlier scored run. A project folder given on the command line must exist; a typo is an error, never a fallback to the current folder. Flags that take a value must have one. Exit codes: 0 fine, 1 gate failed, lint errors or no answers, 2 a usage mistake.
- Gate settings in `evals.config.json`, or as flags:
- ```json
- "gate": { "maxRegressions": 0, "maxScoreDrop": 5, "maxLatencyIncreasePct": null, "maxErrors": 0, "allowRemoved": false, "allowNewFailures": false, "mustPassTag": "must-pass" }
- ```
- These are the defaults. Flags override them for one command: `--max-regressions`, `--max-score-drop`, `--max-latency-increase`, `--max-errors`, `--allow-removed`, `--allow-new-failures`. The score change is measured over the cases both runs asked.
- ## CI
- [`examples/ci/evals.yml`](examples/ci/evals.yml) is a GitHub Actions workflow. On every pull request it runs `evals build` against the committed baseline, fails the check when the gate fails, and uploads the report and the run as artifacts. A committed baseline is required: the workflow stops with an error without one. eval-studio is fetched at a pinned tag or commit (`EVAL_STUDIO_REF`) and the actions are pinned by commit SHA. Keys come from repository secrets through environment variables; pull requests from forks get no secrets, so the example skips them (see the comments in the file). If your target is an HTTP service, start it in the job before the build step.
- ## How it works
- 1. `run` calls the target through an adapter (`engine/lib/adapters/`) and writes every attempt to a run file, with secrets masked.
- 2. `score` applies the checks (`engine/lib/checks/`) to every attempt and adds up the weighted scores. A case is pass, fail, flaky (passes some attempts), error (the target never answered) or skipped (only model-judged checks, with the judge off).
- 3. `compare` matches cases by id between two scored runs and applies the gate.
- 4. `report` puts the data, the styles and the script into one HTML file with a content security policy that blocks every network request.
- ## Development
- ```bash
- ./evals test # 74 tests with node --test: every check type, comparison and gate, redaction, CSV, lint,
- # each adapter against a local stub server, shell syntax in case text never running, the runner, the judge, report data, and the
- # command line itself in throwaway projects
- ```
- No test calls an external service. The OpenAI-compatible target and the model-judged check are tested against a local stub server.
- ## License
- MIT, see [`LICENSE`](LICENSE).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.