Skip to content
Back to search

agent-eval

Test AI coding agents on your framework. Measure what actually works.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/vercel-labs/agent-eval with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/vercel-labs/agent-eval'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/vercel-labs--agent-eval/badge.svg)](/listings/vercel-labs--agent-eval)

HTML

<a href="/listings/vercel-labs--agent-eval"><img src="/api/listings/vercel-labs--agent-eval/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# Create a new eval project
npx @vercel/agent-eval init my-agent-evals
cd my-agent-evals

# Install dependencies
npm install

# Add your API keys
cp .env.example .env
# Edit .env with your AI_GATEWAY_API_KEY and VERCEL_TOKEN

# Preview what will run (no API calls, no cost)
npx @vercel/agent-eval --dry

# Run all experiments
npx @vercel/agent-eval

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 1,153 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # @vercel/agent-eval
  2. Test AI coding agents on your framework. Measure what actually works.
  3. ## Why?
  4. You're building a frontend framework and want AI agents to work well with it. But how do you know if:
  5. - Your documentation helps agents write correct code?
  6. - Adding an MCP server improves agent success rates?
  7. - Sonnet performs as well as Opus for your use cases?
  8. - Your latest API changes broke agent compatibility?
  9. **This framework gives you answers.** Run controlled experiments, measure pass rates, compare techniques.
  10. ## Quick Start
  11. Requires Node.js 22 or later.
  12. ```bash
  13. # Create a new eval project
  14. npx @vercel/agent-eval init my-agent-evals
  15. cd my-agent-evals
  16. # Install dependencies
  17. npm install
  18. # Add your API keys
  19. cp .env.example .env
  20. # Edit .env with your AI_GATEWAY_API_KEY and VERCEL_TOKEN
  21. # Preview what will run (no API calls, no cost)
  22. npx @vercel/agent-eval --dry
  23. # Run all experiments
  24. npx @vercel/agent-eval
  25. ```
  26. ## CLI
  27. ### Run all experiments
  28. ```bash
  29. npx @vercel/agent-eval
  30. ```
  31. With no arguments, the CLI discovers every `experiments/*.ts` file and runs them all. Each experiment runs in parallel. Results with matching fingerprints are reused automatically (see [Result Reuse](#result-reuse)).
  32. ### Run a single experiment
  33. ```bash
  34. npx @vercel/agent-eval cc
  35. ```
  36. The argument is the experiment filename without `.ts`. This resolves to `experiments/cc.ts`.
  37. ### Flags
  38. | Flag | Description |
  39. |----------------------|--------------------------------------------------------------------------------------------|
  40. | `--dry` | Preview what would run without executing. No API calls, no cost. |
  41. | `--smoke` | Quick setup verification. Picks the first eval alphabetically, runs once per model. |
  42. | `--force` | Ignore cached fingerprints and re-run everything. Only applies when running all. |
  43. | `--ack-failures` | Keep non-model failures as final results instead of deleting them. |
  44. Flags work with both modes:
  45. ```bash
  46. npx @vercel/agent-eval --dry # preview all experiments
  47. npx @vercel/agent-eval cc --dry # preview a single experiment
  48. npx @vercel/agent-eval --smoke # smoke test all experiments
  49. npx @vercel/agent-eval cc --smoke # smoke test one experiment
  50. ```
  51. ### Other commands
  52. ```bash
  53. npx @vercel/agent-eval init <name> # scaffold a new eval project
  54. npx @vercel/agent-eval playground # launch web-based results viewer
  55. npx @vercel/agent-eval playground --watch # live mode (watches for new results)
  56. ```
  57. ## Creating Evals
  58. Each eval tests one specific task an agent should be able to do with your framework.
  59. ### Directory structure
  60. ```
  61. evals/
  62. create-button-component/
  63. PROMPT.md # Task for the agent
  64. EVAL.ts # Tests to verify success (or EVAL.tsx for JSX)
  65. package.json # Your framework as a dependency
  66. src/ # Starter code
  67. ```
  68. **PROMPT.md** -- what you want the agent to do:
  69. ```markdown
  70. Create a Button component using MyFramework.
  71. Requirements:
  72. - Export a Button component from src/components/Button.tsx
  73. - Accept `label` and `onClick` props
  74. - Use the framework's styling system for hover states
  75. ```
  76. **EVAL.ts** -- how you verify it worked:
  77. ```typescript
  78. import { test, expect } from 'vitest';
  79. import { readFileSync, existsSync } from 'fs';
  80. import { execSync } from 'child_process';
  81. test('Button component exists', () => {
  82. expect(existsSync('src/components/Button.tsx')).toBe(true);
  83. });
  84. test('has required props', () => {
  85. const content = readFileSync('src/components/Button.tsx', 'utf-8');
  86. expect(content).toContain('label');
  87. expect(content).toContain('onClick');
  88. });
  89. test('project builds', () => {
  90. execSync('npm run build', { stdio: 'pipe' });
  91. });
  92. ```
  93. Use **EVAL.tsx** when your tests require JSX syntax (React Testing Library, component rendering). You only need one eval file per fixture -- choose `.tsx` if any test needs JSX.
  94. ### Asserting on agent behavior
  95. EVAL.ts tests can assert not just on the files the agent produced, but on *how* it worked — which shell commands it ran, which files it read, how many tool calls it made, etc. The framework automatically parses the agent's transcript and writes the results to `__agent_eval__/results.json` in the sandbox before your tests run.
  96. ```typescript
  97. import { test, expect } from 'vitest';
  98. import { readFileSync } from 'fs';
  99. test('agent used the correct scaffolding command', () => {
  100. const results = JSON.parse(readFileSync('__agent_eval__/results.json', 'utf-8'));
  101. const commands = results.o11y.shellCommands.map((c: { command: string }) => c.command);
  102. expect(commands).toContain('npx create-next-app project');
  103. });
  104. test('agent did not make excessive tool calls', () => {
  105. const results = JSON.parse(readFileSync('__agent_eval__/results.json', 'utf-8'));
  106. expect(results.o11y.totalToolCalls).toBeLessThan(50);
  107. });
  108. ```
  109. The `results.o11y` object is a `TranscriptSummary` with these fields:
  110. | Field | Type | Description |
  111. |-------|------|-------------|
  112. | `shellCommands` | `{ command, exitCode?, success? }[]` | Shell commands the agent ran |
  113. | `filesRead` | `string[]` | Files the agent read |
  114. | `filesModified` | `string[]` | Files the agent wrote or edited |
  115. | `toolCalls` | `Record<ToolName, number>` | Count of each tool type used |
  116. | `totalToolCalls` | `number` | Total tool calls made |
  117. | `webFetches` | `{ url, method?, status?, success? }[]` | Web fetches made |
  118. | `totalTurns` | `number` | Conversation turns |
  119. | `errors` | `string[]` | Errors encountered |
  120. | `thinkingBlocks` | `number` | Thinking/reasoning blocks |
  121. | `usage` | `TokenUsage \| undefined` | Token usage of the run, when the agent reports it (see [Token usage and cost](#token-usage-and-cost)) |
  122. > **Note**: If the agent's transcript is unavailable (e.g. the agent crashed before producing output), `results.o11y` will be `null`.
  123. ### Agentic LLM judge
  124. For open-ended quality checks that exact assertions can't express, EVAL.ts can run an **agentic LLM judge**. Each judge assertion re-invokes the *same agent* that did the codegen, **in the same sandbox**, to evaluate a criterion — then returns pass/fail. No fresh sandbox, no copying evidence around. (With a [separate verifier](#separate-verifier), "the same sandbox" is the verifier's, which holds the agent's files.)
  125. ```typescript
  126. import { test, expect } from 'vitest';
  127. import { environment, transcript } from '@vercel/agent-eval/eval';
  128. // Judge the final state: the agent explores the project (read/grep/run) for evidence.
  129. test('uses server components', async () => {
  130. await expect(environment).toSatisfyCriterion('uses Server Components for the product list');
  131. });
  132. // Judge the transcript: how the agent worked. It reads the transcript by path, so the
  133. // full transcript is never stuffed into a prompt.
  134. test('diagnosed properly', async () => {
  135. await expect(transcript).toSatisfyCriterion('diagnosed with DevTools, not trial-and-error edits');
  136. });
  137. // Numeric: the judge scores 0-1; assert a threshold (still pass/fail overall).
  138. test('quality bar', async () => {
  139. await expect(environment).toScoreAtLeast('production-quality error handling', 0.8);
  140. });
  141. ```
  142. Two subjects, imported from `@vercel/agent-eval/eval` — no paths, the subject is implicit:
  143. - `environment` — the judge agent explores the final sandbox state (cwd) with its own tools.
  144. - `transcript` — the judge agent reads the materialized transcript by path.
  145. Two matchers, on either subject:
  146. - `toSatisfyCriterion(criterion)` — passes when the judge decides the criterion is satisfied.
  147. - `toScoreAtLeast(criterion, threshold)` — passes when the judge's 0–1 score is `>= threshold`.
  148. One deterministic matcher, on `transcript` only — no judge run, free and exact:
  149. - `toContainText(needle)` — passes when the raw transcript contains the substring, or matches the `RegExp` (use `/…/i` for case-insensitive). Built for `.not`, i.e. asserting the agent *never* said or reached for something:
  150. ```typescript
  151. test('never suggested the pages router API', () => {
  152. expect(transcript).not.toContainText('getServerSideProps');
  153. expect(transcript).not.toContainText(/getserversideprops/i); // any casing
  154. });
  155. ```
  156. A missing or empty transcript **throws** (fails the test even under `.not`) — an uncaptured transcript is an infra failure, not evidence of absence. Note the transcript is the agent's native format (for `claude-code`, raw session JSONL), so text containing quotes or newlines appears JSON-escaped there; stick to identifier-like needles or match the escaped form with a `RegExp`. For semantic "never did X" checks, phrase the negation *inside* a `toSatisfyCriterion` criterion instead — do not use `.not.toSatisfyCriterion(...)`, which would invert the judge's fail-closed default into a fail-open one.
  157. You supply only the **criterion** string; the framework owns the judge prompt and the verdict contract. On failure the assertion message carries the judge's reasoning, e.g. `[judge:environment] FAIL (score 0.42): product list is a Client Component`, so a failed judge clause is distinguishable from a failed deterministic test or a crash.
  158. By default the judge uses the **same agent and model** as the run under test (self-grading). Because each assertion is a real agent run, it costs time and tokens — keep criteria focused.
  159. **Pin the judge** to grade every run with one fixed agent + model — the apples-to-apples choice when comparing models, since the judge quality no longer varies with the model under test (and a model never grades itself):
  160. ```typescript
  161. const config: ExperimentConfig = {
  162. agent: 'codex',
  163. model: 'gpt-5.4',
  164. // Grade with a fixed Claude judge regardless of the model under test.
  165. judge: { agent: 'vercel-ai-gateway/claude-code', model: 'claude-opus-4-8' },
  166. };
  167. ```
  168. - `judge.model` is required (pinning the model is the point).
  169. - `judge.agent` is optional and defaults to the codegen agent — omit it to keep the same harness and only pin the model. When it names a different agent, that agent's CLI is installed in the sandbox automatically and its key is resolved from its own env var (falling back to `VERCEL_OIDC_TOKEN`). `agentOptions` belong to the codegen agent and are not applied to a different judge agent, which runs with its defaults.
  170. - Pinning changes the eval fingerprint, so a pinned run won't reuse self-graded cached results.
  171. **TypeScript support is built in.** Importing from `@vercel/agent-eval/eval` types the `environment`/`transcript` subjects _and_ registers the `toSatisfyCriterion` / `toScoreAtLeast` matchers on Vitest's `expect` — no manual `declare module 'vitest'` augmentation needed. The package also re-exports the `JudgeSubject` and `JudgeVerdict` types for advanced use.
  172. > **Note**: requires `validation: 'vitest'` (the default). The framework gives the eval process the run's credentials automatically so the judge can call the agent CLI in-sandbox.
  173. ## Configuration Reference
  174. ### Experiment config
  175. ```typescript
  176. // experiments/my-experiment.ts
  177. import type { ExperimentConfig } from '@vercel/agent-eval';
  178. const config: ExperimentConfig = {
  179. // Required: which agent to use
  180. agent: 'vercel-ai-gateway/claude-code',
  181. // Model to use. Omit this to use the underlying agent CLI's native default.
  182. // Provide an array to run the same experiment across multiple models.
  183. model: 'opus',
  184. // How many times to run each eval (default: 1)
  185. runs: 10,
  186. // Stop after first success? (default: true)
  187. earlyExit: false,
  188. // npm scripts that must pass after agent finishes (default: [])
  189. scripts: ['build', 'lint'],
  190. // Validation mode after the agent finishes (default: 'vitest')
  191. // 'vitest' - run EVAL.ts/EVAL.tsx plus configured scripts
  192. // 'none' - response-only mode; skip EVAL.ts/EVAL.tsx, run scripts if provided
  193. validation: 'vitest',
  194. // Timeout per run in seconds (default: 600)
  195. timeout: 600,
  196. // Filter which evals to run (default: '*' for all)
  197. evals: '*',
  198. // evals: ['specific-eval'],
  199. // evals: (name) => name.startsWith('api-'),
  200. // Setup function for sandbox pre-configuration
  201. setup: async (sandbox) => {
  202. await sandbox.writeFiles({ '.env': 'API_KEY=test' });
  203. await sandbox.runCommand('npm', ['run', 'setup']);
  204. },
  205. // Rewrite the prompt before running
  206. editPrompt: (prompt) => `Use the skill.\n\n${prompt}`,
  207. // Custom post-run analysis hook. Can attach analysis/metadata to result.json.
  208. // `runData.generatedFiles` maps path -> Buffer; call .toString('utf-8') for text.
  209. onRunComplete: async ({ runData }) => ({
  210. ...runData,
  211. result: {
  212. ...runData.result,
  213. analysis: { mentionedBrands: ['Vercel'] },
  214. },
  215. }),
  216. // Optional brands to compare in downstream analysis.
  217. brands: [
  218. {
  219. id: 'vercel',
  220. name: 'Vercel',
  221. domain: 'vercel.com',
  222. aliases: ['Vercel Platform'],
  223. isYourBrand: true,
  224. },
  225. ],
  226. // Sandbox backend (default: 'auto' -- Vercel if token present, else Docker)
  227. sandbox: 'auto',
  228. // Vercel backend only, both unset by default. See "Sandbox image and user".
  229. // sandboxImage: 'vercel/sandbox/node:24',
  230. // sandboxUser: 'user',
  231. // Copy project files to results directory (default: 'none')
  232. // 'none' - don't copy files
  233. // 'changed' - copy only files modified by the agent
  234. // 'all' - copy the entire project including original fixture files
  235. copyFiles: 'changed',
  236. // Pin the agentic LLM judge (see "Agentic LLM judge" above). Omit to self-grade
  237. // with the codegen agent+model. `model` required; `agent` defaults to codegen.
  238. judge: { agent: 'vercel-ai-gateway/claude-code', model: 'claude-opus-4-8' },
  239. // Send results somewhere besides results/ (default: none). See "Reporters".
  240. // reporters: [jsonlReporter({ path: 'results/runs.jsonl' })],
  241. // Environment variables for the agent process only (default: none).
  242. // See "Agent environment variables".
  243. // agentEnv: { DEPLOY_TOKEN: process.env.DEPLOY_TOKEN! },
  244. // Grade in a fresh sandbox the agent never touched (default: 'shared').
  245. // See "Separate verifier".
  246. // verifier: 'separate',
  247. // protectedPaths: ['fixtures/**'],
  248. };
  249. export default config;
  250. ```
  251. ### Agent selection
  252. ```typescript
  253. // Vercel AI Gateway (recommended -- unified billing and observability)
  254. agent: 'vercel-ai-gateway/claude-code' // Claude Code via AI Gateway
  255. agent: 'vercel-ai-gateway/codex' // OpenAI Codex via AI Gateway
  256. agent: 'vercel-ai-gateway/opencode' // OpenCode via AI Gateway
  257. agent: 'vercel-ai-gateway/fx' // fx via AI Gateway (research runs)
  258. agent: 'vercel-ai-gateway/pi-coding-agent' // PI coding agent via AI Gateway
  259. // Direct API (uses provider keys directly)
  260. agent: 'claude-code' // requires ANTHROPIC_API_KEY
  261. agent: 'codex' // requires OPENAI_API_KEY
  262. agent: 'gemini' // requires GEMINI_API_KEY
  263. agent: 'cursor' // requires CURSOR_API_KEY
  264. ```
  265. ### Custom agents
  266. Register an implementation of the exported `Agent` interface before exporting
  267. an experiment config. Custom IDs are valid anywhere a built-in agent ID is
  268. accepted:
  269. ```typescript
  270. import {
  271. registerAgent,
  272. type Agent,
  273. type ExperimentConfig,
  274. } from '@vercel/agent-eval';
  275. const definition = {
  276. name: 'my-agent',
  277. displayName: 'My Agent',
  278. defaultModel: 'default-model',
  279. o11yAgentName: 'claude-code',
  280. runnerPath: '/path/to/run.mjs',
  281. getApiKeyEnvVar: () => 'MY_AGENT_API_KEY',
  282. install: () => [],
  283. configFiles: () => [],
  284. authEnv: () => ({}),
  285. } satisfies Agent['definition'];
  286. const myAgent: Agent = {
  287. name: definition.name,
  288. displayName: definition.displayName,
  289. getApiKeyEnvVar: definition.getApiKeyEnvVar,
  290. getDefaultModel: () => definition.defaultModel,
  291. run: async (fixturePath, options) => {
  292. // Invoke the agent and return an AgentRunResult.
  293. return {
  294. success: true,
  295. output: 'done',
  296. duration: 1000,
  297. };
  298. },
  299. definition,
  300. };
  301. registerAgent(myAgent);
  302. const config: ExperimentConfig = {
  303. agent: 'my-agent',
  304. };
  305. export default config;
  306. ```
  307. To record the agent's CLI version in each run's [provenance](#provenance), add an optional `versionCommand` to the definition. It runs once after install, and its trimmed output becomes `provenance.agentCliVersion`. It gets 30 seconds by default (set `timeoutMs` to change that) before the run moves on without it:
  308. ```typescript
  309. versionCommand: () => ({ kind: 'command', cmd: 'my-agent', args: ['--version'] }),
  310. ```
  311. Agent IDs must be non-empty. A later registration with the same ID replaces the
  312. previous one, which allows shared registration modules to be evaluated by
  313. multiple experiment files. Registered agents conform to the complete `Agent`
  314. contract, including the definition used for installation, authentication,
  315. invocation, transcript parsing, and pinned judging.
  316. ### Multi-model experiments
  317. Provide an array of models to run the same experiment on each one. Results are stored under separate directories (`experiment-name/model-name`):
  318. ```typescript
  319. const config: ExperimentConfig = {
  320. agent: 'vercel-ai-gateway/claude-code',
  321. model: ['opus', 'sonnet'],
  322. runs: 10,
  323. };
  324. ```
  325. ### Native agent defaults
  326. When `model` is omitted, Agent Eval does not pass a model override. The
  327. underlying agent CLI chooses the same native default it would use for a normal
  328. user run:
  329. ```typescript
  330. const config: ExperimentConfig = {
  331. agent: 'vercel-ai-gateway/claude-code',
  332. runs: 10,
  333. };
  334. ```
  335. Results use `modelPolicy: 'native-default'`, `requestedModel` is omitted, and
  336. `observedModel` is populated when the agent CLI exposes the runtime model in its
  337. transcript or logs. Provide `model` to force a specific model.
  338. ### Opt-in runtime controls
  339. Agent Eval preserves each agent CLI's normal behavior by default. Recommendation
  340. and controlled-treatment evals can opt into a narrower runtime:
  341. ```typescript
  342. const config: ExperimentConfig = {
  343. agent: 'vercel-ai-gateway/claude-code',
  344. disableBundledSkills: true,
  345. webResearch: true,
  346. };
  347. ```
  348. `disableBundledSkills` disables skills shipped by Claude Code or Codex while
  349. leaving caller-installed project and user skills available. OpenCode and fx do
  350. not ship bundled skill catalogs, so this option does not change their runtimes.
  351. The option is omitted by default, preserving existing arguments and agent
  352. behavior. Other built-in agents reject this option until they expose an
  353. equivalent control.
  354. `webResearch` remains opt-in. It allows Claude Code's `WebSearch`/`WebFetch`,
  355. enables Codex live search (including custom AI Gateway providers), and enables
  356. OpenCode's Exa-backed `websearch`/`webfetch` tools. Whether an agent chooses to
  357. use an available research tool remains part of the measured behavior.
  358. The fx adapter currently requires `webResearch: true`. fx does not yet expose a
  359. prompt-free way to disable every web tool while retaining unrestricted coding
  360. tools, so Agent Eval rejects non-research fx runs instead of silently changing
  361. the treatment.
  362. ### Sandbox image and user
  363. On the Vercel Sandbox backend, two further opt-in controls change the
  364. environment the agent observes. Both are unset by default, so existing
  365. experiments keep the legacy `node24` runtime and the sandbox's default account.
  366. ```typescript
  367. const config: ExperimentConfig = {
  368. agent: 'vercel-ai-gateway/claude-code',
  369. sandbox: 'vercel',
  370. sandboxImage: 'vercel/sandbox/node:24',
  371. sandboxUser: 'user',
  372. };
  373. ```
  374. `sandboxImage` boots each sandbox from a Vercel Container Registry image — a
  375. managed image such as `vercel/sandbox/node:24`, or your own digest-pinned
  376. custom image — instead of the legacy runtime. The Docker backend rejects it,
  377. because Docker selects its image from `runtime`. The `vercel/sandbox/universal`
  378. default image ships preinstalled coding agents; pin a narrower image so the
  379. CLIs Agent Eval installs are the ones being measured.
  380. `sandboxUser` creates a Linux user per sandbox and runs every command and file
  381. operation as that user, with its own home and a `~/workspace` working
  382. directory. Global npm installs are redirected to a user-owned prefix, and the
  383. `SUDO_*` variables left by the switch are cleared, so the only identity the
  384. agent can observe is the one you named. Command environments (including agent
  385. auth tokens) are delivered through a user-owned file that a bash bootstrap
  386. sources, never through command arguments. The Docker backend already runs as
  387. the unprivileged `node` user and ignores this option.
  388. Both options are part of the result-reuse fingerprint: enabling either
  389. invalidates cached results from the default environment.
  390. ### Agent environment variables
  391. Some tasks need the agent to use authenticated tools: deploying to a hosting provider, calling an API, or installing from a private registry. `agentEnv` hands the agent environment variables for that:
  392. ```typescript
  393. const config: ExperimentConfig = {
  394. agent: 'vercel-ai-gateway/claude-code',
  395. agentEnv: {
  396. DEPLOY_TOKEN: process.env.EVAL_DEPLOY_TOKEN!,
  397. NPM_CONFIG_REGISTRY: 'https://npm.example.com',
  398. },
  399. };
  400. ```
  401. The variables are set on the agent process and nowhere else:
  402. - Values travel only in the process environment, never in command arguments. With `sandboxUser` set, they go through the same user-owned env file as the agent's auth token.
  403. - Validation (EVAL.ts and `scripts`) and judge runs don't receive them. Those run in the same sandbox, though, so anything the agent wrote to disk is still there, and with `sandboxUser` so is the env file that delivered the values.
  404. - Every value is redacted from `result.json`, transcripts, test and script outputs, copied project files, and reporter payloads, the same way the agent's API key is. That includes the escaped forms JSON gives a value containing quotes, backslashes, newlines, or non-ASCII characters, such as a multiline private key printed into a JSON transcript. Values shorter than 16 characters are not redacted, because matching short strings would shred unrelated text, so keep anything secret at least that long.
  405. - A key may not collide with the agent's own authentication variables or the workspace identity variables (`USER`, `LOGNAME`). A collision is an error, not a silent override. Variables the sandbox already sets, such as `PATH`, are replaced, so avoid them unless that's the point.
  406. - Only the sorted key names are part of the result-reuse fingerprint. Rotating a token keeps cached results; adding or removing a variable re-runs them.
  407. The agent can read, print, and misuse anything you give it, and a model that is told to deploy may decide to deploy somewhere else. Use short-lived credentials scoped to exactly what the task needs (a single project, a read-only registry, a test account), and revoke them after the run.
  408. ### Separate verifier
  409. By default, tests are hidden from the agent and uploaded into its sandbox once it finishes. That is the sandbox the agent just had full control of, so it can still change what the grader runs: patch `node_modules` or a global binary, edit the vitest config, leave a background process running, or rewrite files the tests read. To grade somewhere the agent never touched:
  410. ```typescript
  411. const config: ExperimentConfig = {
  412. agent: 'vercel-ai-gateway/claude-code',
  413. verifier: 'separate',
  414. protectedPaths: ['fixtures/**'],
  415. };
  416. ```
  417. With `verifier: 'separate'`, once the agent finishes:
  418. 1. Its file changes are captured as a git diff (added, modified, and deleted files, plus executable bits), and its sandbox is stopped.
  419. 2. A fresh sandbox boots with the same backend, image or runtime, and user.
  420. 3. The original fixture files are uploaded, the git baseline is created, your `setup` runs, and the agent's changes are applied, except changes to protected paths.
  421. 4. Project dependencies are installed fresh. The agent CLI is installed only when a fixture file imports `@vercel/agent-eval/eval`, so judge matchers can run.
  422. 5. The tests, vitest config, transcript context, and judge files are uploaded, validation runs, and the verifier is stopped.
  423. Changes to protected paths are never applied in the verifier. These are always protected: `EVAL.ts`, `EVAL.tsx`, and `PROMPT.md` anywhere in the tree, `vitest.config.*` and `vitest.workspace.*`, `__agent_eval__/**`, and `node_modules` (exported as `VERIFIER_PROTECTED_PATHS`). `protectedPaths` adds your own globs, relative to the workspace root. Every protected path the agent changed is listed, sorted, in `result.tampering`. The result also records `verifier: 'separate'` and the `verifierSandboxId`.
  424. Things to know:
  425. - `setup` runs in both sandboxes, so keep it repeatable. It runs before the agent's changes are replayed, because its file writes are already part of the agent's diff.
  426. - A separate verifier roughly doubles the sandbox boots per run, and the verifier's boot, install, and validation count toward `timeout`.
  427. - Judge matchers explore the verifier's sandbox, which holds the agent's files. The transcript judge reads the same transcript as before.
  428. - The verifier installs dependencies from the agent's final `package.json` and lockfile, and runs the agent's version of each npm script in `scripts`. That's what makes the agent's dependency changes work, but it also means the agent can rewrite a graded script or point a dependency at its own code. If your grading depends on those, add `package.json` and the lockfile to `protectedPaths`, knowing the agent's dependency changes then won't be applied.
  429. - The diff is captured with git inside the agent's sandbox, where the agent could replace `git`, so its output is treated as untrusted. Only canonical paths inside the workspace are replayed; an alias such as `./EVAL.ts` or `src/../data.json`, or a path outside the workspace, is never applied and is listed in `result.tampering`. Protected paths never reach the verifier, but `tampering` is best effort: a replaced `git` can also hide changes, which then simply aren't applied.
  430. - Only file contents and executable bits are replayed. A symlink arrives as a regular file.
  431. - `copyFiles` saves the agent's changes as captured before grading: it includes changes to protected paths, and leaves out the test files the shared verifier's capture picks up.
  432. - Custom agent definitions can mark install steps with `scope: 'project'` so the verifier re-runs only the project install. A definition that marks no step has all of its steps re-run.
  433. `protectedPaths` also works with the default shared verifier, where it only reports: protected paths the agent changed are listed in `result.tampering` and graded as the agent left them. Files that install steps create before the agent runs are never reported.
  434. Both backends support the separate verifier. `verifier: 'separate'` and `protectedPaths` are part of the result-reuse fingerprint, but only when set, so `verifier: 'shared'` keeps existing cached results.
  435. ### Run research evals with fx
  436. fx runs through Vercel AI Gateway and uses its native `web_search` and
  437. `web_fetch` tools. Agent Eval pins fx `0.0.11`, verifies the downloaded Linux
  438. binary checksum, and captures the supported saved-session JSON transcript. Set
  439. `agentOptions.effort` to pass an explicit reasoning effort to `fx ask`.
  440. ```typescript
  441. import type { ExperimentConfig } from '@vercel/agent-eval';
  442. const config: ExperimentConfig = {
  443. agent: 'vercel-ai-gateway/fx',
  444. agentOptions: { effort: 'high' },
  445. webResearch: true,
  446. disableBundledSkills: true,
  447. };
  448. export default config;
  449. ```
  450. The fx adapter supports Linux x86-64 and arm64 sandboxes. Agent Eval disables
  451. fx permission prompts inside the disposable sandbox, matching the execution
  452. model used by the other built-in coding agents. fx can self-grade its own runs,
  453. but it cannot serve as a pinned judge for a different agent.
  454. ### OpenCode model format
  455. OpenCode uses Vercel AI Gateway exclusively. The OpenCode CLI reads models as
  456. `{providerID}/{modelID}`, where the provider is the CLI's own `vercel` (AI
  457. Gateway) provider — so both of these forms work:
  458. ```typescript
  459. model: 'anthropic/claude-sonnet-4' // canonical gateway id — vercel/ is added automatically
  460. model: 'vercel/anthropic/claude-sonnet-4' // OpenCode's native form — passed verbatim
  461. ```
  462. When the prefix was added automatically, `observedModel` is reported back in
  463. the request's namespace (`anthropic/claude-sonnet-4`), so requested-vs-observed
  464. comparisons hold. Models targeting a provider configured via
  465. `agentOptions.extraProviders` are passed verbatim.
  466. ### PI coding agent
  467. [PI](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent)
  468. (`@earendil-works/pi-coding-agent`) runs like every other built-in agent: the `pi`
  469. CLI is installed and invoked inside the sandbox with the eval workspace as its
  470. cwd, and its JSON event stream becomes the transcript. Results, `results.json`
  471. assertions, the agentic judge and the Playground all work unchanged.
  472. Like OpenCode, PI only runs through the Vercel AI Gateway, so models use gateway
  473. ids:
  474. ```typescript
  475. const config: ExperimentConfig = {
  476. agent: 'vercel-ai-gateway/pi-coding-agent', // requires AI_GATEWAY_API_KEY
  477. model: 'anthropic/claude-sonnet-4.5',
  478. agentOptions: {
  479. thinking: 'high', // optional: off | minimal | low | medium | high | xhigh
  480. },
  481. };
  482. ```
  483. `agentOptions.extraProviders` is written verbatim to PI's
  484. [`models.json`](https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/docs/models.md)
  485. inside the sandbox. Runs always use PI's `vercel-ai-gateway` provider, so use it
  486. to extend that provider, for example with a gateway model PI's built-in catalog
  487. does not list yet:
  488. ```typescript
  489. agentOptions: {
  490. extraProviders: {
  491. 'vercel-ai-gateway': { models: [{ id: 'vendor/unreleased-model' }] },
  492. },
  493. },
  494. ```
  495. Pin the installed CLI with `agentOptions.cliPackage` (for example
  496. `'@earendil-works/pi-coding-agent@0.99.1'`). PI has no web tools, so
  497. `webResearch` has no effect on it.
  498. The eval workspace is treated as a trusted PI project, so caller-installed
  499. project skills (`.agents/skills`, `.pi/skills`) and `.pi/` settings load even
  500. though `--mode json` cannot show PI's trust prompt.
  501. ### Response-only evals
  502. Use `validation: 'none'` for tasks where the important output is the agent's
  503. answer rather than changed files passing `EVAL.ts`.
  504. ```typescript
  505. const config: ExperimentConfig = {
  506. agent: 'vercel-ai-gateway/claude-code',
  507. model: 'sonnet',
  508. validation: 'none',
  509. runs: 10,
  510. earlyExit: false,
  511. brands: [
  512. { id: 'vercel', name: 'Vercel', aliases: ['Vercel Platform'], isYourBrand: true },
  513. { id: 'netlify', name: 'Netlify' },
  514. { id: 'railway', name: 'Railway' },
  515. ],
  516. onRunComplete: async ({ runData }) => {
  517. // Add custom brand/recommendation analysis here.
  518. return runData;
  519. },
  520. };
  521. ```
  522. Response-only fixtures still need `PROMPT.md` and `package.json`, but they do
  523. not need `EVAL.ts` or `EVAL.tsx`.
  524. ## A/B Testing
  525. The real power is comparing different approaches. Create multiple experiment configs:
  526. ```typescript
  527. // experiments/control.ts
  528. import type { ExperimentConfig } from '@vercel/agent-eval';
  529. const config: ExperimentConfig = {
  530. agent: 'vercel-ai-gateway/claude-code',
  531. model: 'opus',
  532. runs: 10,
  533. earlyExit: false,
  534. };
  535. export default config;
  536. ```
  537. ```typescript
  538. // experiments/with-mcp.ts
  539. import type { ExperimentConfig } from '@vercel/agent-eval';
  540. const config: ExperimentConfig = {
  541. agent: 'vercel-ai-gateway/claude-code',
  542. model: 'opus',
  543. runs: 10,
  544. earlyExit: false,
  545. setup: async (sandbox) => {
  546. await sandbox.runCommand('npm', ['install', '-g', '@myframework/mcp-server']);
  547. await sandbox.writeFiles({
  548. '.claude/settings.json': JSON.stringify({
  549. mcpServers: { myframework: { command: 'myframework-mcp' } }
  550. })
  551. });
  552. },
  553. };
  554. export default config;
  555. ```
  556. ```bash
  557. npx @vercel/agent-eval
  558. ```
  559. Compare the results:
  560. ```
  561. control (baseline): 7/10 passed (70%)
  562. with-mcp: 9/10 passed (90%)
  563. ```
  564. | Experiment | Control | Treatment |
  565. |------------|---------|-----------|
  566. | MCP impact | No MCP | With MCP server |
  567. | Model comparison | Haiku | Sonnet / Opus |
  568. | Documentation | Minimal docs | Rich examples |
  569. | System prompt | Default | Framework-specific |
  570. | Tool availability | Read/write only | + custom tools |
  571. ## Results
  572. Results are saved to `results/<experiment>/<timestamp>/`:
  573. ```
  574. results/
  575. with-mcp/
  576. 2026-01-27T10-30-00Z/
  577. create-button/
  578. summary.json # Pass rate, fingerprint, classification
  579. classification.json # Cached failure classification (if failed)
  580. run-1/
  581. result.json # Individual run result + o11y summary
  582. transcript.json # Parsed/structured agent transcript
  583. transcript-raw.jsonl # Raw agent output (for debugging)
  584. outputs/
  585. eval.txt # EVAL.ts test output
  586. scripts/
  587. build.txt # npm script output
  588. project/ # Agent-generated files (if copyFiles is set)
  589. src/
  590. Button.tsx # Files created/modified by the agent
  591. ```
  592. ### summary.json
  593. Each eval directory contains a `summary.json` with:
  594. ```json
  595. {
  596. "totalRuns": 2,
  597. "passedRuns": 0,
  598. "passRate": "0%",
  599. "meanDuration": 45.2,
  600. "fingerprint": "a1b2c3...",
  601. "classification": {
  602. "failureType": "infra",
  603. "failureReason": "Rate limited (HTTP 429) — model never ran"
  604. },
  605. "valid": false
  606. }
  607. ```
  608. The `fingerprint` field enables result reuse across runs. The `classification` and `valid` fields appear only for failed evals -- `valid: false` marks non-model failures so they are not reused by fingerprinting and are automatically retried.
  609. ### Token usage and cost
  610. Every run records the token usage its agent CLI reported, so you can weigh cost against pass rate when you compare models, docs, or MCP servers. `result.json` carries a `usage` object, and the transcript summary carries the same object at `o11y.usage`:
  611. ```json
  612. {
  613. "status": "passed",
  614. "duration": 84.2,
  615. "usage": {
  616. "inputTokens": 315,
  617. "outputTokens": 122,
  618. "cacheReadTokens": 24448,
  619. "cacheWriteTokens": 0,
  620. "reasoningTokens": 64,
  621. "totalTokens": 24885
  622. }
  623. }
  624. ```
  625. `summary.json` adds a `usage` block when at least one run reported usage:
  626. ```json
  627. {
  628. "totalRuns": 2,
  629. "passedRuns": 1,
  630. "passRate": "50%",
  631. "meanDuration": 80.1,
  632. "usage": {
  633. "runsWithUsage": 2,
  634. "totalTokens": 51230,
  635. "meanTotalTokens": 25615,
  636. "inputTokens": 1102,
  637. "outputTokens": 2380,
  638. "cacheReadTokens": 47748,
  639. "cacheWriteTokens": 0,
  640. "reasoningTokens": 940
  641. }
  642. }
  643. ```
  644. Token figures cover the runs that reported usage (`runsWithUsage`), and a field appears only when every one of those runs reported it. `costUsd` appears only when every run in the eval reported a cost.
  645. Agent Eval records only what the agent CLI reports. It never estimates token counts and never computes cost from a price table, so anything the CLI doesn't report is left out. CLIs also count tokens differently, so compare `totalTokens` across agents and read the other fields per agent:
  646. | Agent | Source | Notes |
  647. |-------|--------|-------|
  648. | Claude Code | `message.usage` on assistant session entries | A message written across several lines is counted once. `costUsd` only from CLI versions that write `costUSD`. |
  649. | Codex | `turn.completed` usage, or `token_count` in a saved session | Cache reads and writes are split out of `inputTokens`. `reasoningTokens` is part of `outputTokens`. |
  650. | OpenCode | `step_finish` events | `costUsd` is OpenCode's own figure, and OpenCode reports 0 for models it has no pricing for. `totalTokens` needs a version that reports step totals. Subagent sessions aren't in the event stream. |
  651. | Gemini | `stats` on the final `result` event | `outputTokens` excludes thinking tokens, which `totalTokens` includes. |
  652. | fx | `usage` in `fx ask --json` | The transcript is the saved session, which records context size rather than consumed tokens, so the runner reports the ask result's counts alongside it. fx doesn't break out cache usage. |
  653. | Cursor | — | The `stream-json` output reports no usage. |
  654. `usage` covers the code-generation run only. Judge assertions re-invoke the agent inside the sandbox, and their tokens are not included.
  655. EVAL.ts can read the same figures, for example to hold the agent to a token budget:
  656. ```typescript
  657. test('stays within a token budget', () => {
  658. const results = JSON.parse(readFileSync('__agent_eval__/results.json', 'utf-8'));
  659. expect(results.o11y.usage?.totalTokens).toBeLessThan(200_000);
  660. });
  661. ```
  662. A custom agent can return `usage` from `run()` directly, or from its in-sandbox runner as `usage` on the runner result, for CLIs that report usage outside the transcript. Otherwise Agent Eval reads it from the transcript. Usage reported directly takes precedence.
  663. ### Provenance
  664. Every run records what ran, so a result can be reproduced or compared months later. `result.json` carries a `provenance` object:
  665. ```json
  666. {
  667. "status": "passed",
  668. "duration": 84.2,
  669. "provenance": {
  670. "agentEvalVersion": "2.5.0",
  671. "agentCliVersion": "2.0.14 (Claude Code)",
  672. "sandboxBackend": "vercel",
  673. "sandboxImage": "vercel/sandbox/node:24@sha256:…",
  674. "sandboxUser": "user"
  675. }
  676. }
  677. ```
  678. - `agentEvalVersion` is the installed `@vercel/agent-eval` version.
  679. - `agentCliVersion` is the trimmed output of the agent's version command (for example `claude --version`), run once after install. Every built-in agent has one. If the command fails, prints nothing, or takes longer than 30 seconds, the field is left out and the run carries on.
  680. - `sandboxBackend` is `vercel` or `docker`.
  681. - `sandboxImage` is the digest-pinned image a Vercel sandbox booted from when `sandboxImage` is set. Runtime-based sandboxes and Docker leave it out.
  682. - `sandboxUser` is the Linux user the agent ran as: the created `sandboxUser` on Vercel, or `node` on Docker. The Vercel sandbox's default account leaves it out.
  683. Provenance is a record, not an input: it is not part of the result-reuse fingerprint.
  684. ### Playground UI
  685. Browse results in a web-based dashboard:
  686. ```bash
  687. npx @vercel/agent-eval playground
  688. ```
  689. This opens a local Next.js app with:
  690. - **Overview** dashboard with stats and recent experiments
  691. - **Experiment detail** with per-eval pass rates and run results, including tokens and cost when the agent reports them
  692. - **Transcript viewer** to inspect agent tool calls, thinking, and errors
  693. - **Compare** two runs side-by-side with pass rate deltas
  694. Options:
  695. ```bash
  696. npx @vercel/agent-eval playground --results-dir ./results --evals-dir ./evals --port 3001
  697. ```
  698. ### File Copying
  699. By default, the framework only saves test outputs and transcripts. Use the `copyFiles` config option to also save the files generated by the agent:
  700. ```typescript
  701. const config: ExperimentConfig = {
  702. copyFiles: 'changed', // or 'all' or 'none' (default)
  703. };
  704. ```
  705. **Options:**
  706. - **`none`** (default) — Don't copy any project files, only save outputs and transcripts
  707. - **`changed`** — Copy only files that were modified, created, or deleted by the agent (a renamed file counts as the old path deleted and the new one created)
  708. - **`all`** — Copy the complete project including both the original fixture files and agent changes
  709. Files are saved to `results/<experiment>/<timestamp>/<eval>/run-N/project/`. The framework uses git to track changes.
  710. ## Reporters
  711. Results always land in the local `results/` tree. Reporters send them somewhere else as well, such as an ingest endpoint, object storage, or a JSONL file, without forking the harness or walking result folders afterwards.
  712. ```typescript
  713. import { httpReporter, jsonlReporter, type ExperimentConfig } from '@vercel/agent-eval';
  714. const config: ExperimentConfig = {
  715. agent: 'vercel-ai-gateway/claude-code',
  716. model: 'opus',
  717. reporters: [
  718. jsonlReporter({ path: 'results/runs.jsonl' }),
  719. httpReporter({
  720. url: 'https://example.com/ingest',
  721. headers: { authorization: `Bearer ${process.env.INGEST_TOKEN}` },
  722. }),
  723. ],
  724. };
  725. ```
  726. A reporter has a `name` and up to two hooks:
  727. - `onRunComplete(event)` fires once per finished run, after your `onRunComplete` hook, so the result includes anything that hook attached. The event carries `experimentName`, `evalName`, `runIndex`, `agent`, `model`, the run's `result`, and the eval's `fingerprint` and `contentFingerprint` when they were computed.
  728. - `onExperimentComplete(event)` fires once per experiment, after `saveResults()`, so every path under `outputDir` exists. The event carries the full `results`, the `reused` evals that were skipped because a cached result matched their fingerprint, and, when run through the CLI with the classifier enabled, the failure `classifications`. Classification runs before this hook, so evals it removed as non-model failures are no longer on disk.
  729. Both events have `schemaVersion: 1`. Retried attempts, aborted attempts, and reused evals get no run events. With `earlyExit`, a run that finishes just after another run of the same eval passed can be reported even though `summary.json` stops counting at the first pass.
  730. Reporters run in order and each call is awaited. A reporter that throws or rejects is logged, listed in the CLI output, and skipped: it never changes an eval result, and the next reporter still runs. Payloads go through the same credential redaction as `result.json`, including anything your `onRunComplete` hook attached. Reporters are not part of the result-reuse fingerprint.
  731. Two reporters are built in:
  732. - `jsonlReporter({ path })` appends one line per run, each a run event. Parent directories are created.
  733. - `httpReporter({ url, headers, retries, retryDelayMs })` POSTs each event as JSON: `{ "type": "run", ... }` or `{ "type": "experiment", ... }`. Network errors, 429, and 5xx responses are retried (2 retries by default, with exponential backoff from 500 ms). The experiment payload drops transcripts, file contents, config functions, and the reporters themselves; each run is reduced to its `result`. The full data stays on disk under `outputDir`.
  734. Write your own by returning an object with a `name` and the hooks you need:
  735. ```typescript
  736. import { appendFile } from 'node:fs/promises';
  737. import type { Reporter } from '@vercel/agent-eval';
  738. const passRateCsv: Reporter = {
  739. name: 'pass-rate-csv',
  740. async onExperimentComplete({ experimentName, results }) {
  741. const rows = results.evals.map((summary) => `${experimentName},${summary.name},${summary.passRate}\n`);
  742. await appendFile('pass-rates.csv', rows.join(''));
  743. },
  744. };
  745. ```
  746. Reporters configured in an experiment file are called by every command that runs it. Library users can also pass `reporters` to `runExperiment()`, which calls them after the config's reporters. When every eval in an experiment is reused, nothing runs and no reporter is called.
  747. ## Result Reuse
  748. The framework computes a SHA-256 fingerprint for each (eval, config) pair. The fingerprint covers all eval directory files and result-affecting config including `agent`, `model`, `scripts`, `timeout`, `earlyExit`, `runs`, `webResearch`, `disableBundledSkills`, `sandboxImage`, `sandboxUser`, the key names (not values) of `agentEnv`, `verifier: 'separate'`, `protectedPaths`, and a pinned `judge`.
  749. On subsequent runs, evals with a matching fingerprint and a valid cached result (at least one passing run) are skipped automatically. This means:
  750. - **Adding new evals** -- safe, no existing results to invalidate.
  751. - **Extending the model array** -- safe, each model gets its own experiment directory.
  752. - **Changing the `evals` filter** -- safe, the filter is not part of the fingerprint.
  753. - **Editing an eval file** -- only invalidates that specific eval.
  754. - **Changing config fields** (agent, model, timeout, etc.) -- invalidates all evals in that experiment.
  755. Use `--force` to bypass fingerprinting and re-run everything. Functions like `setup` and `editPrompt` cannot be hashed, so use `--force` when you change those.
  756. Each result also stores a `contentFingerprint` — a hash of the eval files **only**, independent of config. This separates "the eval itself changed" from "a config field changed."
  757. ### Carrying forward config-only changes
  758. A benign config change (e.g. bumping `timeout`) changes the combined fingerprint and would otherwise re-run every eval. `agent-eval refingerprint` carries those forward in the cached results **without masking a real eval change**:
  759. ```bash
  760. agent-eval refingerprint # all experiments
  761. agent-eval refingerprint cc --dry # preview one experiment
  762. ```
  763. For each cached result it compares the eval's current `contentFingerprint` to the stored one: if the content is unchanged it re-stamps the combined fingerprint (the result stays cached); if the content **changed** it leaves the result stale so it re-runs. Opt-in runtime changes such as web research and bundled-skill isolation also store a `reuseCompatibilityFingerprint`; those boundaries are never carried forward. `agent-eval status` reports content or runtime-compatibility changes as work, while benign config-only edits stay cached. Run `refingerprint` after a benign experiment config edit to carry that change into the cache (`run` does this automatically).
  764. ### After changing or syncing evals: status → pick what to run
  765. Run `agent-eval` with no arguments. It shows the work, then — in a terminal — lets you multi-select which experiments to run. It never re-runs everything:
  766. ```bash
  767. agent-eval
  768. ```
  769. ```
  770. Evals needing work:
  771. new agent-026-no-serial-await
  772. changed agent-024-avoid-redundant-usestate
  773. Work to do — 6 run(s) across 3 experiment(s):
  774. claude-opus-4.6 2 to run (22 up to date)
  775. ...
  776. Pick experiments to run:
  777. 1 claude-opus-4.6
  778. 2 claude-sonnet-4.6
  779. Numbers (e.g. 1,3), "all", or Enter to skip:
  780. ```
  781. Status classifies each eval by **content**, so a benign config change (e.g. pinning a judge) is never reported as work. The same building blocks work non-interactively:
  782. ```bash
  783. agent-eval status # read-only: what's new/changed, per experiment
  784. agent-eval status --check # exit non-zero if anything is new/changed (simple CI gate)
  785. agent-eval status --json # machine-readable, for custom CI policy
  786. agent-eval run claude-sonnet-4.6 # run the named experiment(s) — new/changed evals only
  787. ```
  788. **Accepting staleness is the consumer's call, not the framework's.** `agent-eval` only *reports* — it has no `keep`/`acknowledge`. If you want to leave some experiments on an older eval while keeping others fresh, do that in your own CI: read `agent-eval status --json` (per-experiment `new`/`changed`) and fail only on experiments not in your accepted-stale list. (See next-evals-oss's `scripts/check-stale.mjs` for an example.)
  789. > `refingerprint` (carry config-only changes forward) runs automatically inside `run`; your sync script should call `agent-eval refingerprint` after pulling evals so committed results pick up benign config changes without re-running.
  790. ## Failure Classification
  791. When evals fail, the framework optionally classifies each failure as one of:
  792. - **model** -- the agent tried but wrote incorrect code
  793. - **infra** -- infrastructure broke (API errors, rate limits, crashes)
  794. - **timeout** -- the run hit its time limit
  795. Classification uses Claude Sonnet 4.5 via the Vercel AI Gateway with sandboxed read-only tools to inspect result files. This requires `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` to be set.
  796. ### Classifier Status
  797. - **Enabled** (with `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN`): Classifications are cached in `classification.json`. Non-model failures are removed by default so they can be re-run; pass `--ack-failures` to keep them as final results.
  798. - **Disabled** (without keys): The classifier is skipped. All results are preserved as-is. Housekeeping will not remove non-model failures (only incomplete and duplicate results). Add `AI_GATEWAY_API_KEY` to `.env` to enable the classifier.
  799. ## Housekeeping
  800. After each experiment completes, the framework automatically:
  801. - Removes duplicate results for the same eval (keeps the newest)
  802. - Removes incomplete results (missing `summary.json` or transcripts)
  803. - Removes empty timestamp directories
  804. ## Environment Variables
  805. Every run requires an API key for the agent and a token for the sandbox. Classifier is optional.
  806. | Variable | Required when | Description |
  807. |----------------------|----------------------------------------|----------------------------------------------------------------------------------------------|
  808. | `AI_GATEWAY_API_KEY` | `vercel-ai-gateway/` agents or classifier | Vercel AI Gateway key -- required for `vercel-ai-gateway/` agents and failure classification |
  809. | `ANTHROPIC_API_KEY` | `agent: 'claude-code'` | Direct Anthropic API key |
  810. | `OPENAI_API_KEY` | `agent: 'codex'` | Direct OpenAI API key |
  811. | `GEMINI_API_KEY` | `agent: 'gemini'` | Direct Google Gemini API key |
  812. | `CURSOR_API_KEY` | `agent: 'cursor'` | Direct Cursor API key |
  813. | `VERCEL_TOKEN` | Always (pick one) | Vercel personal access token -- for local dev |
  814. | `VERCEL_OIDC_TOKEN` | Always (pick one) OR for classifier | Vercel OIDC token -- for CI/CD pipelines, or enables classifier without `AI_GATEWAY_API_KEY` |
  815. The **classifier is optional**: if neither `AI_GATEWAY_API_KEY` nor `VERCEL_OIDC_TOKEN` is set, failure classification is skipped and all results are preserved as-is. Set either key to enable the classifier, which automatically identifies and removes non-model failures (infrastructure errors, rate limits, timeouts).
  816. OpenCode and fx only support Vercel AI Gateway
  817. (`vercel-ai-gateway/opencode` and `vercel-ai-gateway/fx`). There are no direct
  818. API variants for these agents.
  819. ### Setup
  820. The `init` command generates a `.env.example` file. Copy it and fill in your keys:
  821. ```bash
  822. cp .env.example .env
  823. ```
  824. The framework loads `.env.local` first, then `.env` as a fallback, via [dotenv](https://github.com/motdotla/dotenv).
  825. ### Vercel AI Gateway (recommended)
  826. One key for all models:
  827. ```bash
  828. AI_GATEWAY_API_KEY=your-ai-gateway-api-key
  829. VERCEL_TOKEN=your-vercel-token
  830. ```
  831. ### Direct API keys (no Vercel account required)
  832. If you don't have a Vercel account, use provider API keys directly:
  833. ```bash
  834. ANTHROPIC_API_KEY=sk-ant-... # For Claude Code
  835. OPENAI_API_KEY=sk-proj-... # For Codex
  836. ```
  837. And choose ONE sandbox option (no Vercel key needed):
  838. ```bash
  839. # Option 1: Use Docker (free, no account needed)
  840. # Just set sandbox: 'docker' in your experiment config, that's it!
  841. # Option 2: Use Vercel (requires free account)
  842. VERCEL_TOKEN=your-vercel-token
  843. ```
  844. #### Minimal setup example
  845. Claude Code via direct API with Docker sandbox:
  846. ```typescript
  847. // experiments/my-eval.ts
  848. import type { ExperimentConfig } from '@vercel/agent-eval';
  849. const config: ExperimentConfig = {
  850. agent: 'claude-code', // Direct API (not vercel-ai-gateway/...)
  851. model: 'opus',
  852. runs: 1,
  853. sandbox: 'docker', // No VERCEL_TOKEN needed
  854. };
  855. export default config;
  856. ```
  857. Then just set:
  858. ```bash
  859. ANTHROPIC_API_KEY=sk-ant-...
  860. ```
  861. That's it! The classifier will be disabled (since you don't have `AI_GATEWAY_API_KEY`), but all features work fine — you'll just see a warning that non-model failure classification is skipped.
  862. ## Tips
  863. **Start with `--dry`**: Always preview before running to verify your config and avoid unexpected costs.
  864. **Use `--smoke` first**: Verify API keys, model IDs, and sandbox connectivity before launching a full run.
  865. **Use multiple runs**: Single runs don't tell you reliability. Use `runs: 10` and `earlyExit: false` for meaningful data.
  866. **Isolate variables**: Change one thing at a time between experiments. Don't compare "Opus with MCP" to "Haiku without MCP".
  867. **Test incrementally**: Start with simple tasks, add complexity as you learn what works.
  868. ## Contributing
  869. See [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow and release process.
  870. ## License
  871. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.