Skip to content
Back to search

eval-view

Regression testing for AI agents. Snapshot behavior,diff tool calls,catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/hidai25/eval-view with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/hidai25/eval-view'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/hidai25--eval-view/badge.svg)](/listings/hidai25--eval-view)

HTML

<a href="/listings/hidai25--eval-view"><img src="/api/listings/hidai25--eval-view/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

> **OpenAI adapter migration:** OpenAI shut down the Assistants API on **August 26, 2026**. > The latest published EvalView release, **0.8.1**, still uses that API; the Responses API > migration is currently **unreleased source**. If you use `openai-assistants`, follow the > [migration guide](docs/OPENAI_MIGRATION.md) before running your tests. An `assistant_id` > alone cannot preserve your agent's configuration. Other adapters are unaffected.

pip install evalview
evalview snapshot    # Record your agent's current behavior as the baseline
evalview check       # After any change, diff against the baseline

That's the whole loop. `check` returns one of:

  ✓ login-flow        PASSED          behavior matches baseline
  ⚠ refund-request    TOOLS_CHANGED   called a different tool, or in a different order
  ✗ billing-dispute   REGRESSION      score dropped — output quality fell

It diffs the **whole trajectory** — tool names, parameters, and order — not just the final string. The deterministic tool + sequence diff runs offline, with no API key. Add an LLM judge only when you want output-quality scoring.

Executing your agent can still incur backend API charges: `--no-judge` skips the judge, not those calls. Embedding-based semantic comparison is opt-in.

No agent yet? See it work in 30 seconds:

evalview demo

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 171 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <!-- mcp-name: io.github.hidai25/evalview-mcp -->
  2. <!-- keywords: AI agent testing, regression detection, golden baselines -->
  3. <p align="center">
  4. <img src="assets/logo.png" alt="EvalView" width="350">
  5. <br>
  6. <strong>Snapshot testing for AI agents.</strong><br>
  7. Record what your agent does today. Get told when it silently changes.
  8. </p>
  9. <p align="center">
  10. <a href="https://pypi.org/project/evalview/"><img src="https://img.shields.io/pypi/v/evalview.svg?label=release" alt="PyPI version"></a>
  11. <a href="https://pypi.org/project/evalview/"><img src="https://img.shields.io/pypi/dm/evalview.svg?label=downloads" alt="PyPI downloads"></a>
  12. <a href="https://github.com/hidai25/eval-view/actions/workflows/ci.yml"><img src="https://github.com/hidai25/eval-view/actions/workflows/ci.yml/badge.svg?branch=main" alt="Package CI"></a>
  13. <a href="https://github.com/hidai25/eval-view/actions/workflows/dogfood.yml"><img src="https://github.com/hidai25/eval-view/actions/workflows/dogfood.yml/badge.svg?branch=main" alt="Core Dogfood"></a>
  14. <a href="https://github.com/hidai25/eval-view/actions/workflows/dogfood-live.yml"><img src="https://github.com/hidai25/eval-view/actions/workflows/dogfood-live.yml/badge.svg?branch=main&amp;event=workflow_dispatch" alt="Live Provider Checks (manual)"></a>
  15. <a href="https://github.com/hidai25/eval-view/stargazers"><img src="https://img.shields.io/github/stars/hidai25/eval-view?style=social" alt="GitHub stars"></a>
  16. <a href="https://opensource.org/licenses/Apache-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License"></a>
  17. </p>
  18. ---
  19. Your agent returns `200` and looks fine. But a model update, a provider change, or a one-line prompt edit just made it skip a clarification, call the wrong tool, or quietly drop output quality. Your tests still pass. Your users notice before you do.
  20. **EvalView snapshots your agent's behavior — the tools it calls, in what order, with what output — and tells you the moment that behavior changes.** Like Jest snapshots, but for tool-calling, multi-turn agents.
  21. [![demo.gif](assets/demo.gif)](https://github.com/user-attachments/assets/96d8b5f7-3561-44a1-86a4-270fb0d1d8a6)
  22. <sub>↑ 30-second live demo — no API key needed</sub>
  23. ## Quick Start
  24. > **OpenAI adapter migration:** OpenAI shut down the Assistants API on **August 26, 2026**.
  25. > The latest published EvalView release, **0.8.1**, still uses that API; the Responses API
  26. > migration is currently **unreleased source**. If you use `openai-assistants`, follow the
  27. > [migration guide](docs/OPENAI_MIGRATION.md) before running your tests. An `assistant_id`
  28. > alone cannot preserve your agent's configuration. Other adapters are unaffected.
  29. ```bash
  30. pip install evalview
  31. ```
  32. ```bash
  33. evalview snapshot # Record your agent's current behavior as the baseline
  34. evalview check # After any change, diff against the baseline
  35. ```
  36. That's the whole loop. `check` returns one of:
  37. ```
  38. ✓ login-flow PASSED behavior matches baseline
  39. ⚠ refund-request TOOLS_CHANGED called a different tool, or in a different order
  40. ✗ billing-dispute REGRESSION score dropped — output quality fell
  41. ```
  42. It diffs the **whole trajectory** — tool names, parameters, and order — not just the final string. The deterministic tool + sequence diff runs offline, with no API key. Add an LLM judge only when you want output-quality scoring.
  43. Executing your agent can still incur backend API charges: `--no-judge` skips the
  44. judge, not those calls. Embedding-based semantic comparison is opt-in.
  45. No agent yet? See it work in 30 seconds:
  46. ```bash
  47. evalview demo
  48. ```
  49. ## Why snapshot testing (and not assertions)?
  50. Most eval tools ask you to *write down what "good" looks like* — assertions, metrics, rubrics. That's a lot of upfront work, and you can only catch the failures you thought to assert.
  51. EvalView inverts it: **it records what your agent actually does now, and flags any drift from that.** You catch regressions you never anticipated, with zero assertions written. When the new behavior is correct, `evalview snapshot` accepts it as the new baseline — same as updating a snapshot in Jest.
  52. | | EvalView | Assertion-based eval tools |
  53. |---|---|---|
  54. | Setup | Record current behavior | Write assertions/metrics first |
  55. | Catches | Any drift from baseline | Only what you asserted |
  56. | Non-determinism | Multi-variant baselines (up to 5 valid paths) | You handle it |
  57. | Unit of comparison | Full tool-call trajectory | Usually final output |
  58. This makes EvalView a **merge-time regression gate**, which is a different job from observability (Langfuse, LangSmith) or metric scoring (promptfoo, DeepEval, Braintrust). Many teams run one of those for visibility **and** EvalView as the gate. [Honest comparisons →](docs/COMPARISONS.md)
  59. ## EvalView tests itself in public, every day
  60. Every day at 09:00 UTC, on pull requests, and on pushes to main,
  61. [Core Dogfood](.github/workflows/dogfood.yml) exercises the non-live test suite,
  62. type checks, local mock-agent `snapshot` / `check`, `evalview demo`, end-to-end
  63. flows, and an `evalview monitor` smoke test. It uses no paid API credentials and
  64. makes no paid inference calls. GitHub runner usage is separate.
  65. [Live Provider Checks](.github/workflows/dogfood-live.yml) test the real evaluator and
  66. chat assistant only when a maintainer explicitly opts into paid API use on main.
  67. They have no automatic schedule. Their badge records the last manual run; a green
  68. core badge does not establish live-provider health or rule out provider drift.
  69. Package CI, core dogfood, and live checks have separate badges. Failed or incomplete
  70. checks remain visible within their scope, with logs and reports preserved as
  71. artifacts. Rolling issues use separate `dogfood-core` and `dogfood-live` labels.
  72. A provider outage, exhausted quota, or missing credential means live health is
  73. unavailable; it does not prove an agent regression.
  74. The historical [incident #264](https://github.com/hidai25/eval-view/issues/264)
  75. remains available for maintainer review of fresh evidence from both scopes. Neither
  76. workflow automatically closes it. Trust warnings are evidence to investigate,
  77. not proof of gaming or of a particular root cause.
  78. [Core runs →](https://github.com/hidai25/eval-view/actions/workflows/dogfood.yml) · [Manual live runs →](https://github.com/hidai25/eval-view/actions/workflows/dogfood-live.yml) · [Run and triage guide →](docs/INTERNAL_DOGFOODING.md#daily-failure-triage)
  79. ## CI: block regressions in every PR
  80. ```yaml
  81. # .github/workflows/evalview.yml
  82. name: EvalView
  83. on: [pull_request]
  84. jobs:
  85. agent-check:
  86. runs-on: ubuntu-latest
  87. permissions: { pull-requests: write }
  88. steps:
  89. - uses: actions/checkout@v4
  90. - uses: hidai25/eval-view@v0.8.1
  91. with:
  92. openai-api-key: ${{ secrets.OPENAI_API_KEY }}
  93. ```
  94. You get a PR comment with the diff, cost/latency deltas, and a pass/fail gate. [CI/CD guide →](docs/CI_CD.md)
  95. ## Works with your stack
  96. LangGraph · CrewAI · OpenAI · Claude · Mistral · Ollama · MCP · **any HTTP API**.
  97. ```bash
  98. evalview check --agent http://localhost:8000/invoke
  99. ```
  100. [Framework details →](docs/FRAMEWORK_SUPPORT.md)
  101. ## Use it as a library
  102. ```python
  103. from evalview import gate
  104. result = gate(test_dir="tests/")
  105. result.passed # bool
  106. result.diffs # per-test scores and tool diffs
  107. ```
  108. [Python API →](docs/CLI_REFERENCE.md#python-api)
  109. ## More
  110. EvalView also does multi-turn testing, statistical/pass@k runs, record/replay cassettes, model-drift canaries, production monitoring with Slack alerts, and auto-generated regression tests from incidents. These are power-user features — start with `snapshot` and `check`, reach for the rest when you need them.
  111. → [Full feature reference](docs/CLI_REFERENCE.md) · [Getting Started](docs/GETTING_STARTED.md) · [FAQ](docs/FAQ.md)
  112. → [Documentation index](docs/README.md) · [OpenAI migration](docs/OPENAI_MIGRATION.md) · [Release process](docs/RELEASING.md)
  113. ### Why I built EvalView
  114. An agent that looked successful kept pulling entire documents into its context and made one question cost $42.93. That experience led me to build EvalView. I wrote about it in [“I Was Running an AI Casino. Then I Started Writing Tests for My Agents”](https://medium.com/@hidaibarmor/i-was-running-an-ai-casino-then-i-started-writing-tests-for-my-agents-93cb3468ce1e). The December 2025 post is the origin story; use the current docs for setup and commands.
  115. ## Contributing
  116. This is a young project built mostly by one developer. Issues, PRs, and "I tried it and X was confusing" feedback are all genuinely valuable.
  117. - [Open an issue](https://github.com/hidai25/eval-view/issues) · [Discussions](https://github.com/hidai25/eval-view/discussions) · [CONTRIBUTING.md](CONTRIBUTING.md)
  118. **License:** Apache 2.0
  119. ---
  120. [![Star History Chart](https://api.star-history.com/svg?repos=hidai25/eval-view&type=Date)](https://star-history.com/#hidai25/eval-view&Date)
evalview/adapters/goose_adapter.py · Bundled script · 590 lines · 2 findings marked
  1. """Goose CLI adapter for Block's open-source AI agent.
  2. Goose (https://github.com/block/goose) is a local, extensible AI agent
  3. that automates engineering tasks. This adapter runs Goose via CLI and
  4. captures its tool calls and outputs for evaluation.
  5. Requirements:
  6. - Goose CLI installed: curl -fsSL https://github.com/block/goose/releases/download/stable/download_cli.sh | bash
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 8 on GitHub
  7. - Goose configured: goose configure
  8. Usage:
  9. adapter: goose
  10. input:
  11. query: "Run the tests in this project"
  12. context:
  13. cwd: "./my-project" # Working directory for Goose
  14. extensions: ["developer"] # Builtin extensions to enable
  15. provider: "anthropic" # Optional: override LLM provider
  16. model: "claude-sonnet-4-20250514" # Optional: override model
  17. """
  18. import asyncio
  19. import json
  20. import logging
  21. import os
  22. import re
  23. import subprocess
  24. import uuid
  25. from datetime import datetime
  26. from typing import Any, Dict, List, Optional
  27. from evalview.adapters.base import AgentAdapter
  28. from evalview.core.types import (
  29. ExecutionMetrics,
  30. ExecutionTrace,
  31. StepMetrics,
  32. StepTrace,
  33. TokenUsage,
  34. SpanKind,
  35. )
  36. from evalview.core.tracing import Tracer
  37. logger = logging.getLogger(__name__)
  38. class GooseAdapter(AgentAdapter):
  39. """Adapter for Block's Goose AI agent.
  40. Goose is a local-first AI agent that runs commands, edits files,
  41. and automates engineering tasks. This adapter executes Goose via
  42. its CLI and parses the output for evaluation.
  43. The adapter uses `goose run` with JSON output format to capture
  44. structured data about tool calls, outputs, and metrics.
  45. """
  46. def __init__(
  47. self,
  48. endpoint: str = "", # Not used for CLI, but required by registry
  49. timeout: float = 300.0, # 5 minutes default for complex tasks
  50. cwd: Optional[str] = None,
  51. extensions: Optional[List[str]] = None,
  52. provider: Optional[str] = None,
  53. model: Optional[str] = None,
  54. goose_path: str = "goose", # Path to goose binary
  55. **kwargs: Any,
  56. ):
  57. """Initialize Goose adapter.
  58. Args:
  59. endpoint: Not used (Goose runs locally), kept for registry compatibility
  60. timeout: Maximum execution time in seconds (default: 300)
  61. cwd: Working directory for Goose commands
  62. extensions: Builtin extensions to enable (e.g., ["developer", "computercontroller"])
  63. provider: LLM provider override (e.g., "anthropic", "openai")
  64. model: Model override (e.g., "claude-sonnet-4-20250514")
  65. goose_path: Path to goose binary (default: "goose")
  66. """
  67. self.timeout = timeout
  68. self.cwd = cwd
  69. self.extensions = extensions or []
  70. self.provider = provider
  71. self.model = model
  72. self.goose_path = goose_path
  73. self._last_raw_output: Optional[str] = None
  74. @property
  75. def name(self) -> str:
  76. return "goose"
  77. async def execute(
  78. self, query: str, context: Optional[Dict[str, Any]] = None
  79. ) -> ExecutionTrace:
  80. """Execute a task with Goose and capture the execution trace.
  81. Args:
  82. query: The instruction/task for Goose to execute
  83. context: Optional context with:
  84. - cwd: Working directory (overrides init setting)
  85. - extensions: Builtin extensions to add
  86. - provider: LLM provider override
  87. - model: Model override
  88. - max_turns: Maximum conversation turns
  89. Returns:
  90. ExecutionTrace with tool calls, output, and metrics
  91. """
  92. context = context or {}
  93. # Build the goose command
  94. cmd = self._build_command(query, context)
  95. # Determine working directory
  96. cwd = context.get("cwd", self.cwd)
  97. if cwd:
  98. cwd = os.path.abspath(os.path.expanduser(cwd))
  99. # Log the full command for debugging
  100. cmd_str = ' '.join(cmd)
  101. logger.info(f"Executing Goose: {cmd_str}")
  102. print(f"[DEBUG] Goose command: {cmd_str}") # Visible in console
  103. if cwd:
  104. logger.info(f"Working directory: {cwd}")
  105. start_time = datetime.now()
  106. try:
  107. # Run goose as subprocess
  108. result = await asyncio.get_event_loop().run_in_executor(
  109. None,
  110. lambda: subprocess.run(
  111. cmd,
  112. capture_output=True,
  113. text=True,
  114. cwd=cwd,
  115. timeout=self.timeout,
  116. env=self._build_env(context),
  117. ),
  118. )
  119. end_time = datetime.now()
  120. # Store raw output for debugging
  121. self._last_raw_output = result.stdout
  122. # Log any stderr (warnings, debug info)
  123. if result.stderr:
  124. logger.debug(f"Goose stderr: {result.stderr}")
  125. # Parse the output
  126. return self._parse_output(
  127. result.stdout, result.stderr, result.returncode, start_time, end_time
  128. )
  129. except subprocess.TimeoutExpired:
  130. end_time = datetime.now()
  131. logger.error(f"Goose timed out after {self.timeout}s")
  132. return self._create_error_trace(
  133. f"Goose timed out after {self.timeout} seconds",
  134. start_time,
  135. end_time,
  136. )
  137. except FileNotFoundError:
  138. end_time = datetime.now()
  139. logger.error("Goose CLI not found. Is it installed?")
  140. return self._create_error_trace(
  141. "Goose CLI not found. Install with: curl -fsSL https://github.com/block/goose/releases/download/stable/download_cli.sh | bash",
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 168 on GitHub
  142. start_time,
  143. end_time,
  144. )
  145. except Exception as e:
  146. end_time = datetime.now()
  147. logger.error(f"Error executing Goose: {e}")
  148. return self._create_error_trace(str(e), start_time, end_time)
  149. def _build_command(self, query: str, context: Dict[str, Any]) -> List[str]:
  150. """Build the goose CLI command."""
  151. cmd = [
  152. self.goose_path,
  153. "run",
  154. "-t",
  155. query,
  156. "--no-session", # Clean runs for testing
  157. ]
  158. # Add extensions (deduplicate)
  159. extensions = list(set(context.get("extensions", []) + self.extensions))
  160. if extensions:
  161. cmd.extend(["--with-builtin", ",".join(extensions)])
  162. # Add provider/model overrides
  163. provider = context.get("provider", self.provider)
  164. model = context.get("model", self.model)
  165. if provider:
  166. cmd.extend(["--provider", provider])
  167. if model:
  168. cmd.extend(["--model", model])
  169. # Add max turns if specified
  170. max_turns = context.get("max_turns")
  171. if max_turns:
  172. cmd.extend(["--max-turns", str(max_turns)])
  173. return cmd
  174. def _build_env(self, context: Dict[str, Any]) -> Dict[str, str]:
  175. """Build environment variables for the subprocess."""
  176. env = os.environ.copy()
  177. # Allow context to pass additional env vars
  178. if "env" in context:
  179. env.update(context["env"])
  180. return env
  181. def _parse_output(
  182. self,
  183. stdout: str,
  184. stderr: str,
  185. returncode: int,
  186. start_time: datetime,
  187. end_time: datetime,
  188. ) -> ExecutionTrace:
  189. """Parse Goose output into ExecutionTrace.
  190. Goose outputs a mix of:
  191. - Conversational text
  192. - Tool calls (shown with specific formatting)
  193. - Final response
  194. We parse this to extract tool calls and the final output.
  195. """
  196. session_id = f"goose-{uuid.uuid4().hex[:8]}"
  197. total_latency = (end_time - start_time).total_seconds() * 1000
  198. # Try to parse as JSON first (if --output-format json was used)
  199. try:
  200. data = json.loads(stdout)
  201. return self._parse_json_output(data, session_id, start_time, end_time)
  202. except json.JSONDecodeError:
  203. pass
  204. # Parse text output (default Goose format)
  205. steps = self._extract_tool_calls_from_text(stdout)
  206. final_output = self._extract_final_output(stdout)
  207. # Check for errors
  208. if returncode != 0:
  209. error_msg = stderr or f"Goose exited with code {returncode}"
  210. if not final_output:
  211. final_output = error_msg
  212. # Build trace context from steps
  213. tracer = Tracer()
  214. with tracer.start_span("Goose Execution", SpanKind.AGENT):
  215. for step in steps:
  216. tracer.record_tool_call(
  217. tool_name=step.tool_name,
  218. parameters=step.parameters,
  219. result=step.output,
  220. error=step.error,
  221. duration_ms=step.metrics.latency if step.metrics else 0.0,
  222. )
  223. return ExecutionTrace(
  224. session_id=session_id,
  225. start_time=start_time,
  226. end_time=end_time,
  227. steps=steps,
  228. final_output=final_output,
  229. metrics=ExecutionMetrics(
  230. total_cost=self._estimate_cost(steps, stdout),
  231. total_latency=total_latency,
  232. total_tokens=self._extract_tokens(stdout),
  233. ),
  234. trace_context=tracer.build_trace_context(),
  235. )
  236. def _parse_json_output(
  237. self,
  238. data: Dict[str, Any],
  239. session_id: str,
  240. start_time: datetime,
  241. end_time: datetime,
  242. ) -> ExecutionTrace:
  243. """Parse JSON output format from Goose."""
  244. steps = []
  245. # Extract tool calls from JSON
  246. tool_calls = data.get("tool_calls", data.get("steps", []))
  247. for i, call in enumerate(tool_calls):
  248. tool_name = call.get("tool", call.get("name", call.get("tool_name", "unknown")))
  249. steps.append(
  250. StepTrace(
  251. step_id=f"step-{i}",
  252. step_name=call.get("step_name", f"Step {i + 1}"),
  253. tool_name=tool_name,
  254. parameters=call.get("parameters", call.get("arguments", call.get("input", {}))),
  255. output=call.get("output", call.get("result", "")),
  256. success=call.get("success", True),
  257. error=call.get("error"),
  258. metrics=StepMetrics(
  259. latency=call.get("latency", 0.0),
  260. cost=call.get("cost", 0.0),
  261. ),
  262. )
  263. )
  264. final_output = data.get("response", data.get("output", data.get("final_response", "")))
  265. total_latency = (end_time - start_time).total_seconds() * 1000
  266. # Extract tokens if available
  267. tokens_data = data.get("tokens", data.get("usage", {}))
  268. total_tokens = None
  269. if tokens_data:
  270. if isinstance(tokens_data, dict):
  271. total_tokens = TokenUsage(
  272. input_tokens=int(
  273. tokens_data.get("input", tokens_data.get("input_tokens", 0)) or 0
  274. ),
  275. output_tokens=int(
  276. tokens_data.get("output", tokens_data.get("output_tokens", 0)) or 0
  277. ),
  278. cached_tokens=int(tokens_data.get("cached", 0) or 0),
  279. )
  280. elif isinstance(tokens_data, int):
  281. total_tokens = TokenUsage(output_tokens=tokens_data)
  282. return ExecutionTrace(
  283. session_id=session_id,
  284. start_time=start_time,
  285. end_time=end_time,
  286. steps=steps,
  287. final_output=final_output,
  288. metrics=ExecutionMetrics(
  289. total_cost=data.get("cost", data.get("total_cost", 0.0)),
  290. total_latency=total_latency,
  291. total_tokens=total_tokens,
  292. ),
  293. )
  294. def _extract_tool_calls_from_text(self, output: str) -> List[StepTrace]:
  295. """Extract tool calls from Goose's text output.
  296. Goose shows tool usage in format:
  297. ─── shell | developer ──────────────────────────
  298. command: ls -la
  299. <command output>
  300. <assistant response>
  301. """
  302. steps = []
  303. step_count = 0
  304. # Tool header pattern: "─── shell | developer ───" or "─── text_editor | developer ───"
  305. tool_header_pattern = r"[─━]+\s*(\w+)\s*\|\s*\w+\s*[─━]+"
  306. # Common tool names in Goose
  307. known_tools = {
  308. "bash", "shell", "read", "write", "edit", "search", "grep", "find",
  309. "git", "npm", "pip", "python", "node", "cargo", "make",
  310. "developer", "computercontroller", "memory", "text_editor",
  311. "ripgrep", "list_directory",
  312. }
  313. # Split output into sections by tool headers
  314. lines = output.split("\n")
  315. i = 0
  316. while i < len(lines):
  317. line = lines[i]
  318. # Check for tool header
  319. match = re.search(tool_header_pattern, line, re.IGNORECASE)
  320. if match:
  321. tool_name = match.group(1).lower()
  322. # Normalize tool names
  323. if tool_name == "shell":
  324. tool_name = "bash"
  325. if tool_name in known_tools or len(tool_name) > 2:
  326. step_count += 1
  327. # Collect parameters and output until next section
  328. i += 1
  329. params = {}
  330. tool_output_lines = []
  331. # First line after header is usually the command/parameter
  332. if i < len(lines):
  333. param_line = lines[i].strip()
  334. if param_line.startswith("command:"):
  335. params["command"] = param_line[8:].strip()
  336. i += 1
  337. elif param_line.startswith("path:"):
  338. params["path"] = param_line[5:].strip()
  339. i += 1
  340. # Collect output until we hit another tool header or end
  341. while i < len(lines):
  342. next_line = lines[i]
  343. # Check if this is another tool header
  344. if re.search(tool_header_pattern, next_line, re.IGNORECASE):
  345. break
  346. # Check if this looks like assistant text (starts with capital, no indentation)
  347. if next_line and not next_line.startswith(" ") and next_line[0:1].isupper():
  348. # Could be assistant response, check if it's not output
  349. if not any(c in next_line for c in ["│", "├", "└", "-rw", "drw"]):
  350. break
  351. tool_output_lines.append(next_line)
  352. i += 1
  353. # Trim empty lines from output
  354. tool_output = "\n".join(tool_output_lines).strip()
  355. # Limit output size for storage
  356. if len(tool_output) > 2000:
  357. tool_output = tool_output[:2000] + "..."
  358. steps.append(
  359. StepTrace(
  360. step_id=f"step-{step_count}",
  361. step_name=f"Step {step_count}",
  362. tool_name=tool_name,
  363. parameters=params,
  364. output=tool_output,
  365. success=True,
  366. error=None,
  367. metrics=StepMetrics(),
  368. )
  369. )
  370. continue
  371. i += 1
  372. # Deduplicate consecutive same-tool calls (but keep different outputs)
  373. if steps:
  374. deduped = [steps[0]]
  375. for step in steps[1:]:
  376. # Only dedupe if same tool AND no output on current
  377. if step.tool_name != deduped[-1].tool_name or step.output:
  378. deduped.append(step)
  379. steps = deduped
  380. return steps
  381. def _extract_final_output(self, output: str) -> str:
  382. """Extract the final response from Goose output.
  383. The final output is everything after the last tool execution block.
  384. """
  385. # Remove ANSI escape codes
  386. ansi_escape = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
  387. clean_output = ansi_escape.sub("", output)
  388. # Find the last tool header and take everything after it
  389. tool_header_pattern = r"[─━]+\s*\w+\s*\|\s*\w+\s*[─━]+"
  390. lines = clean_output.split("\n")
  391. last_tool_end = -1
  392. in_tool_block = False
  393. for i, line in enumerate(lines):
  394. # Check if this is a tool header
  395. if re.search(tool_header_pattern, line):
  396. in_tool_block = True
  397. continue
  398. # If we're in a tool block, look for where output ends
  399. if in_tool_block:
  400. # Tool output ends when we see a line that looks like assistant text
  401. # (starts with capital letter, not indented, not part of command output)
  402. if line and not line.startswith(" ") and not line.startswith("\t"):
  403. # Check if it's the start of a new sentence/response
  404. if line[0:1].isupper() and not any(c in line for c in ["-rw", "drw", "total ", "│", "├", "└"]):
  405. last_tool_end = i
  406. in_tool_block = False
  407. # If we found a response after tool output, return everything from there
  408. if last_tool_end > 0:
  409. response_lines = lines[last_tool_end:]
  410. # Filter out session metadata lines
  411. response_lines = [line for line in response_lines if not line.startswith("starting session")]
  412. return "\n".join(response_lines).strip()
  413. # Fallback: return the last substantive paragraphs
  414. paragraphs = clean_output.strip().split("\n\n")
  415. response_paragraphs = []
  416. for para in paragraphs:
  417. para = para.strip()
  418. if not para:
  419. continue
  420. if para.startswith("$") or para.startswith(">>>"):
  421. continue
  422. if re.match(r"^[─━═]+", para):
  423. continue
  424. if para.startswith("starting session"):
  425. continue
  426. response_paragraphs.append(para)
  427. if response_paragraphs:
  428. # Return all response paragraphs after the last tool-related one
  429. return "\n\n".join(response_paragraphs[-3:]) # Last 3 paragraphs
  430. return clean_output.strip()
  431. def _estimate_cost(self, steps: List[StepTrace], output: str) -> float:
  432. """Estimate cost based on output length and tool calls.
  433. This is a rough estimate since Goose doesn't always report cost.
  434. For accurate cost tracking, check the LLM provider's dashboard.
  435. """
  436. # Rough estimate: ~$0.003 per 1K tokens for Claude Sonnet
  437. # Assume ~4 chars per token
  438. total_chars = len(output)
  439. estimated_tokens = total_chars / 4
  440. estimated_cost = (estimated_tokens / 1000) * 0.003
  441. # Add cost per step (tool calls use tokens)
  442. estimated_cost += len(steps) * 0.001
  443. return round(estimated_cost, 6)
  444. def _extract_tokens(self, output: str) -> Optional[TokenUsage]:
  445. """Try to extract token usage from output.
  446. Returns None if token info is not available.
  447. """
  448. # Look for token counts in output
  449. patterns = [
  450. r"tokens?:\s*(\d+)",
  451. r"(\d+)\s*tokens?",
  452. r"input:\s*(\d+).*output:\s*(\d+)",
  453. ]
  454. for pattern in patterns:
  455. match = re.search(pattern, output, re.IGNORECASE)
  456. if match:
  457. if len(match.groups()) == 2:
  458. return TokenUsage(
  459. input_tokens=int(match.group(1)),
  460. output_tokens=int(match.group(2)),
  461. )
  462. else:
  463. return TokenUsage(output_tokens=int(match.group(1)))
  464. return None
  465. def _create_error_trace(
  466. self, error_msg: str, start_time: datetime, end_time: datetime
  467. ) -> ExecutionTrace:
  468. """Create an ExecutionTrace for error cases."""
  469. return ExecutionTrace(
  470. session_id=f"goose-error-{uuid.uuid4().hex[:8]}",
  471. start_time=start_time,
  472. end_time=end_time,
  473. steps=[
  474. StepTrace(
  475. step_id="error",
  476. step_name="Error",
  477. tool_name="error",
  478. parameters={},
  479. output=error_msg,
  480. success=False,
  481. error=error_msg,
  482. metrics=StepMetrics(),
  483. )
  484. ],
  485. final_output=error_msg,
  486. metrics=ExecutionMetrics(
  487. total_cost=0.0,
  488. total_latency=(end_time - start_time).total_seconds() * 1000,
  489. total_tokens=None,
  490. ),
  491. )
  492. async def health_check(self) -> bool:
  493. """Check if Goose CLI is available."""
  494. try:
  495. result = subprocess.run(
  496. [self.goose_path, "--version"],
  497. capture_output=True,
  498. text=True,
  499. timeout=5,
  500. )
  501. if result.returncode == 0:
  502. logger.info(f"Goose version: {result.stdout.strip()}")
  503. return True
  504. return False
  505. except Exception as e:
  506. logger.warning(f"Goose health check failed: {e}")
  507. return False
evalview/chat_prompt.py · Bundled script · 486 lines · 1 finding marked
  1. """System prompt for the interactive chat assistant.
  2. Extracted from chat.py to keep that module focused on runtime logic.
  3. The content is the assistant persona; edits here directly affect chat behavior.
  4. """
  5. SYSTEM_PROMPT = """You are EvalView Assistant - an expert on EvalView, a pytest-style testing framework for AI agents.
  6. ## WHAT EVALVIEW DOES
  7. EvalView catches agent regressions before you ship:
  8. - Tool changes (agent used different tools)
  9. - Output changes (response differs from baseline)
  10. - Cost spikes (tokens/$ increased)
  11. - Latency spikes (response time increased)
  12. ## SCORING SYSTEM
  13. Tests are scored 0-100 using weighted components:
  14. - **Tool Accuracy** (30%): Did agent use expected tools?
  15. - **Output Quality** (50%): LLM-as-judge evaluates response quality
  16. - **Sequence Correctness** (20%): Did tools run in expected order?
  17. **Partial Credit for Sequences**: If expected sequence is [a,b,c,d,e] and agent completed [a,b,c], score = 60% (3/5 steps).
  18. ## STATISTICAL MODE (pass@k)
  19. LLMs are non-deterministic. Statistical mode runs tests multiple times:
  20. - `--runs N`: Run each test N times
  21. - `--pass-rate 0.8`: 80% of runs must pass
  22. - **pass@k**: Probability of at least one success in k tries
  23. - **pass^k**: Probability of ALL k tries succeeding
  24. ## AVAILABLE ADAPTERS
  25. | Adapter | Description | Needs Endpoint |
  26. |---------|-------------|----------------|
  27. | http | Generic REST API (default) | Yes |
  28. | langgraph | LangGraph / LangGraph Cloud | Yes |
  29. | crewai | CrewAI multi-agent | Yes |
  30. | openai-assistants | OpenAI Responses API | No (uses SDK) |
  31. | anthropic / claude | Anthropic Claude API | Yes |
  32. | huggingface / hf | HuggingFace Inference | Yes |
  33. | goose | Block's Goose CLI agent | No (uses CLI) |
  34. | tapescope / streaming | JSONL streaming API | Yes |
  35. | mcp | Model Context Protocol | Yes |
  36. ## EXAMPLES IN THE REPO (use these exact paths)
  37. - examples/goosebench/tasks/ - Tests for Block's Goose agent (10 tasks)
  38. - examples/langgraph/ - LangGraph ReAct agent with search + calculator
  39. - examples/crewai/ - CrewAI multi-agent example
  40. - examples/anthropic/ - Claude API example
  41. - examples/openai-assistants/ - OpenAI Responses API example
  42. - examples/huggingface/ - HuggingFace inference example
  43. ## HOW TO TEST GOOSE
  44. ```bash
  45. evalview run examples/goosebench/tasks/
  46. ```
  47. Goose doesn't need a server - it runs via CLI. The goose adapter calls `goose run` directly.
  48. ## HOW TO TEST LANGGRAPH
  49. 1. Start the LangGraph agent:
  50. cd examples/langgraph/agent && langgraph dev
  51. 2. Run tests:
  52. evalview run examples/langgraph/ --verbose
  53. ## YAML TEST CASE SCHEMA
  54. Required top-level fields are name, input, and expected. Adapter and endpoint may
  55. come from project configuration; thresholds are optional and have defaults.
  56. ```yaml
  57. name: "Test Name"
  58. adapter: goose # or http, langgraph, crewai, etc.
  59. endpoint: http://localhost:8000 # if adapter needs it
  60. # Optional: difficulty level for benchmarking (trivial/easy/medium/hard/expert)
  61. difficulty: medium
  62. # Optional: suite type (capability for hill-climbing, regression for safety net)
  63. suite_type: capability
  64. input:
  65. query: "Your question here"
  66. context:
  67. extensions: ["developer"] # for goose
  68. expected:
  69. tools:
  70. - calculator
  71. - search
  72. tool_categories:
  73. - file_read
  74. - shell
  75. tool_sequence: # Expected order of tool calls
  76. - search
  77. - calculator
  78. output:
  79. contains: ["expected", "words"]
  80. not_contains: ["error"]
  81. thresholds:
  82. min_score: 70
  83. max_cost: 0.10
  84. max_latency: 5000
  85. # Optional: statistical mode (run test multiple times)
  86. variance:
  87. runs: 10 # Run 10 times
  88. pass_rate: 0.8 # 80% must pass
  89. ```
  90. ## KEY COMMANDS
  91. ```bash
  92. evalview demo
  93. ```
  94. Shows a demo of regression detection.
  95. ```bash
  96. evalview init
  97. ```
  98. Interactive setup wizard.
  99. ```bash
  100. evalview run
  101. ```
  102. Run tests in tests/test-cases/.
  103. ```bash
  104. evalview run examples/goosebench/tasks/
  105. ```
  106. Run tests from a specific path.
  107. ```bash
  108. evalview run --diff
  109. ```
  110. Compare against golden baseline (detect regressions).
  111. ```bash
  112. evalview run --verbose
  113. ```
  114. Show detailed output.
  115. ```bash
  116. evalview run --runs 10
  117. ```
  118. Statistical mode: run each test 10 times, get pass@k metrics.
  119. ```bash
  120. evalview run --runs 10 --pass-rate 0.7
  121. ```
  122. Statistical mode with custom pass rate (70% must pass).
  123. ```bash
  124. evalview run --difficulty hard
  125. ```
  126. Filter tests by difficulty level (trivial/easy/medium/hard/expert).
  127. ```bash
  128. evalview adapters
  129. ```
  130. List all available adapters.
  131. ```bash
  132. evalview snapshot
  133. ```
  134. Execute the configured tests and save a golden baseline.
  135. ```bash
  136. evalview check
  137. ```
  138. Execute the same tests and compare them with the saved golden baseline.
  139. ## CI/CD INTEGRATION
  140. EvalView integrates with GitHub Actions to block PRs with regressions.
  141. ```bash
  142. evalview ci comment
  143. ```
  144. Post test results as a PR comment. Shows pass/fail, score, cost, latency, and changes from baseline.
  145. ```bash
  146. evalview ci comment --dry-run
  147. ```
  148. Preview the PR comment without posting.
  149. **Add to GitHub Actions workflow:**
  150. ```yaml
  151. - name: Post PR comment
  152. if: github.event_name == 'pull_request'
  153. run: evalview ci comment
  154. env:
  155. GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  156. ```
  157. The comment shows:
  158. - Overall status (PASSED / REGRESSION / TOOLS_CHANGED / OUTPUT_CHANGED)
  159. - Summary metrics (tests, pass rate, cost, latency)
  160. - Top changes when using --diff mode
  161. - Failed tests with scores
  162. ## INTERACTIVE COMMANDS (use these directly in chat)
  163. The user can run these slash commands directly without leaving chat:
  164. | Command | Description |
  165. |---------|-------------|
  166. | /adapters | List all available adapters with tracing status |
  167. | /test [--trace] <adapter> <query> | Quick ad-hoc test against any adapter |
  168. | /run [--trace] [test-name] | Run a test case from YAML file |
  169. | /compare | Compare two test runs (detect regressions) |
  170. | /trace <script.py> [args] | Trace LLM calls in any Python script |
  171. | /traces | List stored traces from past runs |
  172. | /traces <id> | Show details of a specific trace |
  173. | /traces export <id> | Export trace to HTML file |
  174. | /traces cost | Show cost report for recent traces |
  175. | /model | Switch LLM provider/model |
  176. **Trace flags:** Add `--trace` or `-t` to `/test` or `/run` for live trace output during execution.
  177. ## WHEN USERS ASK TO TEST OR RUN THINGS
  178. When users want to test an agent or run something, suggest the appropriate slash command:
  179. 1. "Test my agent" or "Try calling my API" → Suggest `/test <adapter> <query>`
  180. Example: "Try `/test http What is 2+2?` or `/test ollama Hello`"
  181. For live tracing: "Use `/test --trace ollama What is 2+2?` to see LLM calls in real-time"
  182. 2. "Run my tests" or "Execute test cases" → Suggest `/run` or `/run <test-name>`
  183. Example: "Use `/run` to see available tests, or `/run my-test` to run a specific one"
  184. For live tracing: "Use `/run --trace my-test` to see detailed trace during execution"
  185. 3. "What adapters are available?" → Suggest `/adapters`
  186. 4. "What traces have I run?" or "Show my traces" → Suggest `/traces`
  187. Example: "Use `/traces` to see your recent traces, or `/traces abc123` to see a specific one"
  188. 5. "How much have I spent?" or "Show costs" → Suggest `/traces cost`
  189. Example: "Use `/traces cost` to see your spending breakdown by model"
  190. 6. "Did anything break?" or "Compare runs" or "Check for regressions" → Suggest `/compare`
  191. Example: "Run `/compare` to see what changed between your last two test runs"
  192. 7. "Trace my script" or "See what LLM calls my script makes" → Suggest `/trace`
  193. Example: "Use `/trace my_agent.py` to see all OpenAI/Anthropic/Ollama calls"
  194. ## NATURAL LANGUAGE EXAMPLES
  195. User: "I want to test my langgraph agent"
  196. → "You can quickly test it with `/test langgraph What is 2+2?` - make sure your agent is running at localhost:2024"
  197. User: "Run the calculator test"
  198. → "Use `/run calculator` to run that test case"
  199. User: "Test ollama with a math question"
  200. → "Try `/test ollama What is 15 * 23?`"
  201. User: "I want to see what API calls my agent script makes"
  202. → "Use `/trace your_agent.py` to trace all LLM calls - it instruments OpenAI, Anthropic, and Ollama automatically"
  203. User: "What traces have I run?"
  204. → "Use `/traces` to see your recent traces. Each trace has an ID you can use to see details with `/traces <id>`"
  205. User: "How much am I spending on LLM calls?"
  206. → "Use `/traces cost` to see a breakdown of your spending by model over the last 7 days"
  207. ## DEBUGGING WITH /trace AND /traces
  208. For tracing Python scripts, use `/trace`:
  209. - Automatically instruments OpenAI, Anthropic, and Ollama SDK calls
  210. - No code changes needed - just run `/trace my_script.py`
  211. - Shows token counts, costs, and timing for each LLM call
  212. - Traces are automatically saved for later viewing
  213. For viewing past traces, use `/traces`:
  214. - `/traces` - List your recent traces
  215. - `/traces <id>` - Show details of a specific trace
  216. - `/traces export <id>` - Export trace to HTML file with charts
  217. - `/traces cost` - See spending breakdown by model and day
  218. When users ask about debugging, test failures, or understanding what happened:
  219. 1. Suggest `/trace script.py` to trace a Python script
  220. 2. Suggest `/traces` to see past traces
  221. 3. Suggest `/traces cost` to see spending
  222. 4. Explain what traces show (LLM calls, tokens, costs)
  223. 5. Help interpret trace output if they share it
  224. ## SKILL TESTING (Agent-Based)
  225. EvalView can test Claude Code skills through real AI agents (not just system prompts).
  226. Use `/skill` command in chat to run skill tests directly.
  227. ### Available Agent Types
  228. | Agent | Description | Status |
  229. |-------|-------------|--------|
  230. | system-prompt | Legacy mode - injects skill as system prompt | Default |
  231. | claude-code | Execute through Claude Code CLI | Primary, fully tested |
  232. | codex | OpenAI Codex CLI | Implemented |
  233. | langgraph | LangGraph SDK integration | Implemented |
  234. | crewai | CrewAI framework | Implemented |
  235. | openai-assistants | OpenAI Responses API | Implemented |
  236. | custom | User-provided runner script | Implemented |
  237. ### Two-Phase Evaluation
  238. 1. **Phase 1 (Deterministic)**: Fast, debuggable checks
  239. 2. **Phase 2 (Rubric)**: LLM-as-judge evaluates quality (only runs if Phase 1 passes)
  240. ### Phase 1: All Deterministic Checks Available
  241. **Tool Checks:**
  242. - `tool_calls_contain`: ["Write", "Bash"] - tools that MUST be called
  243. - `tool_calls_not_contain`: ["Edit"] - tools that must NOT be called
  244. - `tool_sequence`: ["Read", "Write"] - tools must appear in this order
  245. **File Checks:**
  246. - `files_created`: ["package.json", "src/App.tsx"] - files that must be created
  247. - `files_modified`: ["README.md"] - files that must be modified
  248. - `files_not_modified`: ["config.json"] - files that must NOT be modified
  249. - `file_contains`: {path: [strings]} - strings that must appear in file
  250. - `file_not_contains`: {path: [strings]} - strings that must NOT appear
  251. **Command Checks:**
  252. - `commands_ran`: ["npm install"] - commands that must be executed (substring match)
  253. - `commands_not_ran`: ["rm -rf"] - commands that must NOT run
  254. - `command_count_max`: 15 - catch thrashing/looping behavior
  255. **Output Checks:**
  256. - `output_contains`: ["success", "created"] - strings in final output
  257. - `output_not_contains`: ["error", "failed"] - strings NOT in output
  258. **Token Budget Checks:**
  259. - `max_input_tokens`: 5000 - maximum input tokens allowed
  260. - `max_output_tokens`: 2000 - maximum output tokens allowed
  261. - `max_total_tokens`: 7000 - maximum total tokens
  262. **Build Verification:**
  263. - `build_must_pass`: ["npm run build", "npm test"] - commands that must exit with code 0
  264. **Runtime Smoke Tests:**
  265. ```yaml
  266. smoke_tests:
  267. - command: "curl http://localhost:3000"
  268. expected_output: "Hello"
  269. timeout: 30
  270. - url: "http://localhost:3000/api/health"
  271. expected_status: 200
  272. ```
  273. **Repository Cleanliness:**
  274. - `git_clean`: true - working directory must have no uncommitted changes
  275. **Permission/Security Checks:**
  276. - `no_sudo`: true - no sudo commands allowed
  277. - `forbidden_patterns`: ["rm -rf /", "sudo rm"] - command patterns that are forbidden
    • Note: Quoted as data in code: Runs `rm -rf` against a root or home-directory path
    • Line 342 on GitHub
  278. - `no_network_external`: true - block external network calls
  279. ### Test Categories (OpenAI Eval Guidelines)
  280. - **explicit**: Direct skill invocation ("Use the code-review skill")
  281. - **implicit**: Natural language that implies skill use ("Review this code for bugs")
  282. - **contextual**: Real-world noisy prompts with irrelevant context
  283. - **negative**: Prompts that should NOT trigger the skill (with should_trigger: false)
  284. ### Skill Test YAML Schema (Full Example)
  285. ```yaml
  286. name: test-my-skill
  287. description: Comprehensive skill test suite
  288. skill: ./skills/my-skill/SKILL.md
  289. agent:
  290. type: claude-code
  291. max_turns: 10
  292. timeout: 300
  293. capture_trace: true
  294. min_pass_rate: 0.8
  295. tests:
  296. # Explicit invocation
  297. - name: explicit-trigger
  298. category: explicit
  299. input: "Use the setup-demo-app skill to create a React app"
  300. should_trigger: true
  301. expected:
  302. tool_calls_contain: ["Write", "Bash"]
  303. files_created: ["package.json", "src/App.tsx"]
  304. commands_ran: ["npm install"]
  305. command_count_max: 15
  306. build_must_pass: ["npm run build"]
  307. # Implicit invocation
  308. - name: implicit-trigger
  309. category: implicit
  310. input: "Set up a minimal React demo app with Tailwind"
  311. expected:
  312. files_created: ["package.json", "tailwind.config.js"]
  313. file_contains:
  314. src/index.css: ["tailwindcss"]
  315. output_contains: ["created", "tailwind"]
  316. # Negative control
  317. - name: should-not-trigger
  318. category: negative
  319. input: "What time is it?"
  320. should_trigger: false
  321. expected:
  322. tool_calls_not_contain: ["Write"]
  323. files_created: []
  324. # With rubric evaluation
  325. - name: style-check
  326. input: "Create demo app"
  327. expected:
  328. files_created: ["package.json"]
  329. rubric:
  330. prompt: |
  331. Evaluate the code against these requirements:
  332. - Uses TypeScript
  333. - Has proper error handling
  334. - Follows React best practices
  335. min_score: 70
  336. model: gpt-5.4-mini # optional model override
  337. ```
  338. ### Skill Test Commands
  339. ```bash
  340. evalview skill test tests/my-skill.yaml
  341. ```
  342. Legacy mode (system prompt + string matching).
  343. ```bash
  344. evalview skill test tests/my-skill.yaml --agent claude-code
  345. ```
  346. Agent mode - executes skill through Claude Code CLI.
  347. ```bash
  348. evalview skill test tests/my-skill.yaml -a claude-code -t ./traces/
  349. ```
  350. Save JSONL traces for debugging.
  351. ```bash
  352. evalview skill test tests/my-skill.yaml -a claude-code --no-rubric
  353. ```
  354. Skip Phase 2 rubric evaluation (deterministic checks only).
  355. ```bash
  356. evalview skill test tests/my-skill.yaml --cwd /path/to/workspace
  357. ```
  358. Run in specific working directory.
  359. ```bash
  360. evalview skill test tests/my-skill.yaml --max-turns 20
  361. ```
  362. Override max conversation turns.
  363. ### Other Skill Commands
  364. ```bash
  365. evalview skill validate ./SKILL.md
  366. ```
  367. Validate a skill file for correct structure.
  368. ```bash
  369. evalview skill list ~/.claude/skills/
  370. ```
  371. List all skills in a directory.
  372. ```bash
  373. evalview skill doctor ~/.claude/skills/
  374. ```
  375. Diagnose skill issues (token budget, duplicates, etc.).
  376. ### Chat /skill Command
  377. Use `/skill` in chat to run skill tests interactively:
  378. - `/skill test tests.yaml` - run skill tests in legacy mode
  379. - `/skill test tests.yaml --agent claude-code` - run with Claude Code agent
  380. - `/skill test tests.yaml -a claude-code -t ./traces/` - with trace capture
  381. - `/skill validate ./SKILL.md` - validate a skill file
  382. - `/skill list ./skills/` - list skills in directory
  383. - `/skill doctor ./skills/` - diagnose skill issues
  384. ### When Users Ask About Skill Testing
  385. 1. "Test my skill" → Suggest `/skill test` or `evalview skill test`
  386. 2. "Test with real agent" → `/skill test tests.yaml --agent claude-code`
  387. 3. "Why did my skill fail?" → Check Phase 1 deterministic checks first, then rubric
  388. 4. "Debug skill execution" → Use `-t ./traces/` to capture JSONL traces
  389. 5. "Check token usage" → Add `max_input_tokens`, `max_output_tokens` to expected
  390. 6. "Verify build works" → Add `build_must_pass: ["npm run build"]`
  391. 7. "Test server starts" → Use `smoke_tests` with command or url checks
  392. 8. "Ensure no breaking changes" → Add `git_clean: true`
  393. 9. "Block dangerous commands" → Use `no_sudo: true` or `forbidden_patterns`
  394. ## RULES
  395. 1. Use ```bash blocks for explanations and examples. Use ```command blocks only when the user explicitly asks to execute that command now. Never turn a how-to question into execution or execute placeholder paths. A proposed command is not evidence that it ran; only claim success after receiving its actual result.
  396. 2. Answer questions using the knowledge above - don't hallucinate
  397. 3. For adapter questions, refer to the adapters table
  398. 4. For example questions, give the actual path from examples list
  399. 5. Keep responses concise but accurate
  400. 6. When debugging, suggest /trace to see execution details
  401. """

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.