Skip to content
Back to search

mcp-bench

SkillWorks with Claude CodeGitHub

Neutral, reproducible benchmark harness for comparing browser and automation MCP servers over standard stdio JSON-RPC against local deterministic HTML fixtures. Supports Playwright MCP, Chrome DevTools MCP, Puppeteer MCP, Browserbase MCP, Weavetab MCP, and any custom stdio MCP server via live tools/list schema introspection.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/weavetab--mcp-bench/badge.svg)](/listings/weavetab--mcp-bench)

HTML

<a href="/listings/weavetab--mcp-bench"><img src="/api/listings/weavetab--mcp-bench/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Inspect any server's live tools and schemas:

npx -y @weavetab/mcp-bench@latest --inspect playwright

Run a head-to-head comparison:

npx -y @weavetab/mcp-bench@latest --engines playwright,chrome-devtools,weavetab --runs 3 --out results/local-run.json

Benchmark a custom MCP server via JSON config (see [examples/custom-adapter.json](examples/custom-adapter.json)):

npx -y @weavetab/mcp-bench@latest --config ./my-adapter.json --engines playwright,my-custom-mcp

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 52 lines
  1. ---
  2. name: mcp-bench
  3. description: Neutral, reproducible benchmark harness for comparing browser and automation MCP servers over standard stdio JSON-RPC against local deterministic HTML fixtures. Supports Playwright MCP, Chrome DevTools MCP, Puppeteer MCP, Browserbase MCP, Weavetab MCP, and any custom stdio MCP server via live tools/list schema introspection.
  4. ---
  5. # mcp-bench — Neutral Automation MCP Benchmark Skill
  6. Run fair, apples-to-apples benchmarks across any set of automation MCP servers using local `127.0.0.1` HTML fixtures, server-side Pass@1 assertions, and live `tools/list` schema introspection.
  7. ## Core Principles
  8. 1. **Zero hardcoded assumptions about tool versions**: Do not guess or invent tool names for any MCP server. Always rely on live `tools/list` introspection (`--inspect <engine>`) or each project's official documentation below.
  9. 2. **Equal default execution**: Every built-in preset runs with default settings and zero server-specific parameter favoritism unless the user explicitly supplies a `--config` file.
  10. 3. **Honest reporting**: Report the exact raw numbers from the JSON artifact, including failures, skips, cold-start time, tool schema token overhead, and environment metadata.
  11. ## Official Documentation for Built-in Presets
  12. Consult the official docs when configuring custom flags or overrides for a server:
  13. | Preset | Package | Official Docs |
  14. | :--- | :--- | :--- |
  15. | `browserbase` | `@browserbasehq/mcp-server-browserbase@latest` | https://github.com/browserbase/mcp-server-browserbase |
  16. | `chrome-devtools` | `chrome-devtools-mcp@latest` | https://github.com/ChromeDevTools/chrome-devtools-mcp |
  17. | `playwright` | `@playwright/mcp@latest` | https://github.com/microsoft/playwright-mcp |
  18. | `puppeteer` | `@modelcontextprotocol/server-puppeteer@latest` | https://github.com/modelcontextprotocol/servers-archived/tree/main/src/puppeteer |
  19. | `weavetab` | `@weavetab/mcp@latest` | https://weavetab.dev/docs/mcp/tools |
  20. ## Commands
  21. Inspect any server's live `tools/list` output and resolved action mapping first:
  22. ```bash
  23. node ./bin/mcp-bench.mjs --inspect playwright
  24. ```
  25. Run a multi-engine benchmark:
  26. ```bash
  27. node ./bin/mcp-bench.mjs --engines playwright,chrome-devtools,weavetab --runs 3 --out results/local-run.json
  28. ```
  29. Benchmark a custom or newly released MCP server:
  30. ```bash
  31. node ./bin/mcp-bench.mjs --config ./examples/custom-adapter.json --engines playwright,my-custom-mcp
  32. ```
  33. Or pass a one-off stdio command:
  34. ```bash
  35. node ./bin/mcp-bench.mjs --custom-cmd node --custom-args "./server/index.js" --engines custom
  36. ```
README.md · README · 56 lines
  1. # mcp-bench
  2. A neutral, open-source benchmark harness and Agent Skill for measuring **automation MCP servers** over standard **MCP stdio JSON-RPC** against deterministic local HTML fixtures.
  3. ## Design Goals
  4. - **Neutral & fair**: Every engine runs against the same local `127.0.0.1` HTTP fixture server with server-side `/api/verify/<task>` Pass@1 assertions and zero hidden per-engine shortcuts.
  5. - **Live schema introspection (version-resilient)**: Instead of hardcoding tool names that break when servers rename tools or change parameters, `mcp-bench` calls `tools/list` at connect time and maps semantic actions dynamically from each server's live `inputSchema`.
  6. - **Extensible to any MCP server**: Test built-in presets or plug in any custom/local stdio MCP server via `--config` or `--custom-cmd`.
  7. - **Dual format (CLI + Agent Skill)**: Run directly in a terminal via `npx @weavetab/mcp-bench` or load `SKILL.md` into Claude Code, Antigravity, Cursor, or Windsurf.
  8. ## Built-in Presets & Official Docs
  9. Presets are listed alphabetically with links to each project's official documentation:
  10. | Engine ID | Package | Official Documentation |
  11. | :--- | :--- | :--- |
  12. | `browserbase` | `@browserbasehq/mcp-server-browserbase@latest` | https://github.com/browserbase/mcp-server-browserbase |
  13. | `chrome-devtools` | `chrome-devtools-mcp@latest` | https://github.com/ChromeDevTools/chrome-devtools-mcp |
  14. | `playwright` | `@playwright/mcp@latest` | https://github.com/microsoft/playwright-mcp |
  15. | `puppeteer` | `@modelcontextprotocol/server-puppeteer@latest` | https://github.com/modelcontextprotocol/servers-archived/tree/main/src/puppeteer |
  16. | `weavetab` | `@weavetab/mcp@latest` | https://weavetab.dev/docs/mcp/tools |
  17. ## Quickstart
  18. Inspect any server's live tools and schemas:
  19. ```bash
  20. npx -y @weavetab/mcp-bench@latest --inspect playwright
  21. ```
  22. Run a head-to-head comparison:
  23. ```bash
  24. npx -y @weavetab/mcp-bench@latest --engines playwright,chrome-devtools,weavetab --runs 3 --out results/local-run.json
  25. ```
  26. Benchmark a custom MCP server via JSON config (see [examples/custom-adapter.json](examples/custom-adapter.json)):
  27. ```bash
  28. npx -y @weavetab/mcp-bench@latest --config ./my-adapter.json --engines playwright,my-custom-mcp
  29. ```
  30. ## Deterministic Local Fixtures
  31. | Task | Description |
  32. | :--- | :--- |
  33. | `dense-dom` | Navigate to a 1,500-node table UI, take a snapshot, click Export, verify server POST |
  34. | `delta-update` | Open an approval dialog on a 120-card grid, take a second snapshot, confirm rollout |
  35. | `multi-form` | Fill 4 text inputs + checkbox and submit form; verify exact payload server-side |
  36. | `spa-rerender` | Re-mount target DOM node after 180ms to exercise stale-ref handling |
  37. | `multi-tab` | Extract token from a secondary fixture page, submit it on the primary form |
  38. ## License
  39. MIT — see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.