mcp-bench
Neutral, reproducible benchmark harness for comparing browser and automation MCP servers over standard stdio JSON-RPC against local deterministic HTML fixtures. Supports Playwright MCP, Chrome DevTools MCP, Puppeteer MCP, Browserbase MCP, Weavetab MCP, and any custom stdio MCP server via live tools/list schema introspection.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/weavetab--mcp-bench)HTML
<a href="/listings/weavetab--mcp-bench"><img src="/api/listings/weavetab--mcp-bench/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Inspect any server's live tools and schemas:
npx -y @weavetab/mcp-bench@latest --inspect playwright
Run a head-to-head comparison:
npx -y @weavetab/mcp-bench@latest --engines playwright,chrome-devtools,weavetab --runs 3 --out results/local-run.json
Benchmark a custom MCP server via JSON config (see [examples/custom-adapter.json](examples/custom-adapter.json)):
npx -y @weavetab/mcp-bench@latest --config ./my-adapter.json --engines playwright,my-custom-mcp
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 52 lines
- ---
- name: mcp-bench
- description: Neutral, reproducible benchmark harness for comparing browser and automation MCP servers over standard stdio JSON-RPC against local deterministic HTML fixtures. Supports Playwright MCP, Chrome DevTools MCP, Puppeteer MCP, Browserbase MCP, Weavetab MCP, and any custom stdio MCP server via live tools/list schema introspection.
- ---
- # mcp-bench — Neutral Automation MCP Benchmark Skill
- Run fair, apples-to-apples benchmarks across any set of automation MCP servers using local `127.0.0.1` HTML fixtures, server-side Pass@1 assertions, and live `tools/list` schema introspection.
- ## Core Principles
- 1. **Zero hardcoded assumptions about tool versions**: Do not guess or invent tool names for any MCP server. Always rely on live `tools/list` introspection (`--inspect <engine>`) or each project's official documentation below.
- 2. **Equal default execution**: Every built-in preset runs with default settings and zero server-specific parameter favoritism unless the user explicitly supplies a `--config` file.
- 3. **Honest reporting**: Report the exact raw numbers from the JSON artifact, including failures, skips, cold-start time, tool schema token overhead, and environment metadata.
- ## Official Documentation for Built-in Presets
- Consult the official docs when configuring custom flags or overrides for a server:
- | Preset | Package | Official Docs |
- | :--- | :--- | :--- |
- | `browserbase` | `@browserbasehq/mcp-server-browserbase@latest` | https://github.com/browserbase/mcp-server-browserbase |
- | `chrome-devtools` | `chrome-devtools-mcp@latest` | https://github.com/ChromeDevTools/chrome-devtools-mcp |
- | `playwright` | `@playwright/mcp@latest` | https://github.com/microsoft/playwright-mcp |
- | `puppeteer` | `@modelcontextprotocol/server-puppeteer@latest` | https://github.com/modelcontextprotocol/servers-archived/tree/main/src/puppeteer |
- | `weavetab` | `@weavetab/mcp@latest` | https://weavetab.dev/docs/mcp/tools |
- ## Commands
- Inspect any server's live `tools/list` output and resolved action mapping first:
- ```bash
- node ./bin/mcp-bench.mjs --inspect playwright
- ```
- Run a multi-engine benchmark:
- ```bash
- node ./bin/mcp-bench.mjs --engines playwright,chrome-devtools,weavetab --runs 3 --out results/local-run.json
- ```
- Benchmark a custom or newly released MCP server:
- ```bash
- node ./bin/mcp-bench.mjs --config ./examples/custom-adapter.json --engines playwright,my-custom-mcp
- ```
- Or pass a one-off stdio command:
- ```bash
- node ./bin/mcp-bench.mjs --custom-cmd node --custom-args "./server/index.js" --engines custom
- ```
README.md · README · 56 lines
- # mcp-bench
- A neutral, open-source benchmark harness and Agent Skill for measuring **automation MCP servers** over standard **MCP stdio JSON-RPC** against deterministic local HTML fixtures.
- ## Design Goals
- - **Neutral & fair**: Every engine runs against the same local `127.0.0.1` HTTP fixture server with server-side `/api/verify/<task>` Pass@1 assertions and zero hidden per-engine shortcuts.
- - **Live schema introspection (version-resilient)**: Instead of hardcoding tool names that break when servers rename tools or change parameters, `mcp-bench` calls `tools/list` at connect time and maps semantic actions dynamically from each server's live `inputSchema`.
- - **Extensible to any MCP server**: Test built-in presets or plug in any custom/local stdio MCP server via `--config` or `--custom-cmd`.
- - **Dual format (CLI + Agent Skill)**: Run directly in a terminal via `npx @weavetab/mcp-bench` or load `SKILL.md` into Claude Code, Antigravity, Cursor, or Windsurf.
- ## Built-in Presets & Official Docs
- Presets are listed alphabetically with links to each project's official documentation:
- | Engine ID | Package | Official Documentation |
- | :--- | :--- | :--- |
- | `browserbase` | `@browserbasehq/mcp-server-browserbase@latest` | https://github.com/browserbase/mcp-server-browserbase |
- | `chrome-devtools` | `chrome-devtools-mcp@latest` | https://github.com/ChromeDevTools/chrome-devtools-mcp |
- | `playwright` | `@playwright/mcp@latest` | https://github.com/microsoft/playwright-mcp |
- | `puppeteer` | `@modelcontextprotocol/server-puppeteer@latest` | https://github.com/modelcontextprotocol/servers-archived/tree/main/src/puppeteer |
- | `weavetab` | `@weavetab/mcp@latest` | https://weavetab.dev/docs/mcp/tools |
- ## Quickstart
- Inspect any server's live tools and schemas:
- ```bash
- npx -y @weavetab/mcp-bench@latest --inspect playwright
- ```
- Run a head-to-head comparison:
- ```bash
- npx -y @weavetab/mcp-bench@latest --engines playwright,chrome-devtools,weavetab --runs 3 --out results/local-run.json
- ```
- Benchmark a custom MCP server via JSON config (see [examples/custom-adapter.json](examples/custom-adapter.json)):
- ```bash
- npx -y @weavetab/mcp-bench@latest --config ./my-adapter.json --engines playwright,my-custom-mcp
- ```
- ## Deterministic Local Fixtures
- | Task | Description |
- | :--- | :--- |
- | `dense-dom` | Navigate to a 1,500-node table UI, take a snapshot, click Export, verify server POST |
- | `delta-update` | Open an approval dialog on a 120-card grid, take a second snapshot, confirm rollout |
- | `multi-form` | Fill 4 text inputs + checkbox and submit form; verify exact payload server-side |
- | `spa-rerender` | Re-mount target DOM node after 180ms to exercise stale-ref handling |
- | `multi-tab` | Extract token from a secondary fixture page, submit it on the primary form |
- ## License
- MIT — see [LICENSE](LICENSE).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.