Skip to content
Back to search

mcp-code-mode-bench

Measure MCP Code Mode against one-by-one tool calls and tool search: tokens, requests, time and cost, on the Claude API and Claude Code

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/yeyo11--mcp-code-mode-bench/badge.svg)](/listings/yeyo11--mcp-code-mode-bench)

HTML

<a href="/listings/yeyo11--mcp-code-mode-bench"><img src="/api/listings/yeyo11--mcp-code-mode-bench/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 171 lines
  1. # mcp-code-mode-bench
  2. How many tokens, model round trips and seconds does an agent spend on the same task, against the
  3. same MCP server, when it calls the tools one by one, and when it writes code that calls them? And
  4. when does Code Mode **not** pay off?
  5. This repository measures it, with a mock MCP server whose answers are known in advance, so every run
  6. is graded by code. Write-up: [With Code Mode, my agent makes twice the model requests and spends 44% less than without it](https://josegarrido.dev/en/code-mode-twice-the-model-requests-44-percent-cheaper/?utm_source=github&utm_medium=referral&utm_campaign=code-mode-doble-de-peticiones-44-por-ciento-menos-coste).
  7. ## The three "Code Modes"
  8. 1. **Cloudflare Code Mode** (Sept 2025) named the idea: turn MCP tools into a TypeScript API, let the model write code against it, run it in a V8 isolate ([blog](https://blog.cloudflare.com/code-mode/), [the API in ~1,000 tokens](https://blog.cloudflare.com/code-mode-mcp/)). Not benchmarked here: another stack, and deploying needs Workers Paid.
  9. 2. **Claude API programmatic tool calling** (stable, no beta header): Claude writes Python in the code execution container and calls your tools from it, so intermediate results never enter its context ([docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling); the pattern in [Anthropic's engineering post](https://www.anthropic.com/engineering/code-execution-with-mcp)). Mode C.
  10. 3. **Claude Code Code Mode** (experimental, v2.1.323+): Claude writes JavaScript that calls Bash, Read, Edit and MCP tools through a `REPL` tool, on with `CLAUDE_CODE_EXPERIMENTAL_CODE_MODE=1` ([docs](https://code.claude.com/docs/en/code-mode)). Mode E.
  11. The idea itself is older: [CodeAct](https://arxiv.org/abs/2402.01030) (2024) used executable code as the agent's action space.
  12. ## What is in the repository
  13. ```
  14. server/ mock google-docs-mcp: same 6 tools, schemas and output text, over seeded fixtures; --padding N
  15. tasks/ T1-T3 prompts, expected answers computed from the fixtures, and the checker
  16. bench/ API harness (modes A-C), Claude Code runner (modes D-E), report
  17. results/ raw/*.jsonl, one line per run, and results.md (generated)
  18. test/ npm test: fixtures, checker, server, harness (recorded responses, no API key)
  19. ```
  20. **The server** mirrors [`@yeyo11/google-docs-mcp`](https://github.com/yeyo11/google-docs-mcp)
  21. (`list_documents`, `list_comments`, `get_comments_summary`, `get_comment`, `get_document`,
  22. `get_auth_status`), answering from 20 invented documents of an invented team's "Q4 planning" folder:
  23. 300 comments with authors, dates, open/resolved status, reply threads, reopened threads, deleted
  24. comments and quoted text. "Today" is fixed at 2026-11-20, so answers never drift. `--padding N`
  25. adds N plausible filler tools (calendar, sheets, gmail, CRM, tickets...) to grow the catalogue,
  26. because with 6 tools the definitions cost little.
  27. **Tasks** (expected answers in [`tasks/expected.json`](tasks/expected.json)):
  28. | Task | What | Shape |
  29. |---|---|---|
  30. | T1 | In every document, which open comments have had no reply for more than 7 days? Group by document, count per author. | 1 + 20 calls, large results, small answer: Code Mode's case |
  31. | T2 | How many open comments does the Budget 2027 document have? | One or two calls: the control, where Code Mode should not pay off |
  32. | T3 | T2 with `--padding 200` (206 tools) | What tool definitions alone cost |
  33. Every prompt asks the model to end with a JSON block; the checker grades it by exact set and count
  34. match.
  35. **Modes**:
  36. | Mode | Where | What it is |
  37. |---|---|---|
  38. | A. Classic | Claude API, own loop | Every tool result goes through the model |
  39. | B. Classic + tool search | Claude API | `tool_search_tool_bm25_20251119`; every tool `defer_loading: true` except `list_documents` |
  40. | C. Programmatic tool calling | Claude API | `code_execution_20260120` + `allowed_callers` on every tool |
  41. | D. Claude Code, classic | `claude -p`, or an interactive session measured from its transcript | What most people use daily |
  42. | E. Claude Code, Code Mode | Interactive session with `CLAUDE_CODE_EXPERIMENTAL_CODE_MODE=1`, measured from its transcript | Experimental; does not run under `-p` |
  43. Same model, same effort (`medium`) and same prompt in every mode. Prompt caching (automatic, 5-minute
  44. TTL) is on in modes A-C, as any real agent would use it.
  45. **Per run** the harness records input tokens summed over **every** request (uncached + cache writes
  46. + cache reads, each also on its own), output tokens, number of model requests, tool calls (direct vs
  47. programmatic), wall time, correct yes/no, and cost in USD from the
  48. [pricing page](https://platform.claude.com/docs/en/about-claude/pricing) (`bench/src/prices.ts`).
  49. ## Run it
  50. Node 20+. Modes A-C need a Claude API key in `ANTHROPIC_API_KEY`; modes D-E need Claude Code
  51. v2.1.323 or later, signed in.
  52. ```sh
  53. npm install
  54. npm test # builds, then checks fixtures, checker, server and harness; free
  55. # Look at the exact request a mode would send, without sending it (no key needed)
  56. npm run bench -- api --dry-run --mode C --task T2
  57. # Debug the harness cheaply on Haiku 5.5, one repetition
  58. npm run bench -- api --model claude-haiku-5-5 --mode A,B,C --task T2 --reps 1 --smoke
  59. # One command per mode (5 repetitions of T1-T3; Sonnet 5.5 by default)
  60. npm run bench -- api --mode A
  61. npm run bench -- api --mode B
  62. npm run bench -- api --mode C
  63. npm run bench -- claude-code --mode D
  64. npm run bench -- session-command --mode E --task T1 # mode E is interactive: see docs/claude-code-code-mode.md
  65. npm run bench -- transcript --latest --mode E --task T1 --rep 1
  66. # Or all API modes interleaved in one run, which is fairer (drift over time hits every mode alike)
  67. npm run bench -- api --mode A,B,C --task T1,T2,T3 --reps 5 --cap 9
  68. npm run report # writes results/results.md
  69. ```
  70. The API harness keeps a running cost and **stops the whole run** once it reaches `--cap` (9 USD by
  71. default). `--spent` adds what earlier invocations already spent. Other flags: `--model`, `--effort`,
  72. `--no-cache`, `--max-requests`, `--batch`. `npm run bench -- tasks` prints the prompts and answers;
  73. `npm run bench -- tools --padding 200` lists the catalogue.
  74. ## Results
  75. Run on 2026-11-21 with Sonnet 5.5 at `medium` effort. Every run below answered correctly. Median
  76. (min–max) per cell; the full generated table, with cold and warm cache in separate rows, is in
  77. [`results/results.md`](results/results.md), and every run is a line in [`results/raw/`](results/raw/).
  78. **Claude API, 5 repetitions per cell, warm cache:**
  79. | Task | A. One by one | B. Tool search | C. Programmatic tool calling |
  80. |---|---|---|---|
  81. | T1: 20 documents, 21 calls | $0.281 · 77k in · 79 s | $0.283 · 80k in · 75 s | **$0.060** · 44k in · 57 s |
  82. | T2: one document | $0.018 · 7 s | **$0.006** · 7 s | $0.012 · 14 s |
  83. | T3: one document, 206 tools | $0.016 · 118k in | **$0.005** · 11k in | $0.029 · 161k in |
  84. **Claude Code 2.1.330, interactive sessions, T1, 3 repetitions** (cost is Claude Code's estimate at API
  85. prices; on a subscription it is quota, not money):
  86. | Mode | Cost | Model requests | Wall time |
  87. |---|---|---|---|
  88. | D. Classic | $0.41 | 4 | 85 s |
  89. | E. Code Mode | **$0.09** | 31 | **38 s** |
  90. What the numbers say:
  91. - **Chained calls whose results only need filtering are Code Mode's case.** T1 is about 4.7× cheaper
  92. through the API and about 4.5× in Claude Code. Not because there are fewer requests (there are many
  93. more) but because the 20 results stay in a variable of the script instead of the context, and the
  94. model writes far fewer output tokens.
  95. - **One call doesn't pay off.** Programmatic tool calling takes twice as long, and tool search is
  96. cheaper.
  97. - **A large catalogue is not Code Mode's problem to solve.** With 206 tools, every definition is still
  98. loaded in mode C; tool search (mode B) is about 3× cheaper. Cloudflare's "search, then execute"
  99. works because it combines both ideas.
  100. - **Tokens are not money.** With a warm cache, the 118k tokens of T3's definitions cost under 2 cents.
  101. ## Caveats
  102. - **Tokens are not money.** Most of what Code Mode saves in a long loop are cache reads, which cost
  103. 5 % of the input price on Sonnet 5.5. A big drop in tokens can be a small drop in dollars; the
  104. table shows both, and cold and warm cache separately.
  105. - **T2 is there for Code Mode not to pay off.** One or two small calls do not repay the code execution
  106. container and the extra turn to write the script. Anthropic's docs say so ("weak fit"); publishing it
  107. is part of the point.
  108. - **Programmatic tool results are not billed** as input tokens on the Claude API ("Only the final code
  109. execution result and Claude's response count"). That is the real effect, not an accounting trick,
  110. but it is why mode C's input can look tiny.
  111. - **Code execution container time is not in the cost.** Each organization gets 1,550 free
  112. container-hours a month; these runs stay inside it.
  113. - **Tool search is a fair rival, not a strawman.** Mode B solves the definition cost of a large
  114. catalogue without any code; T3 is where it competes with C.
  115. - **Claude Code's numbers are its own.** `total_cost_usd` is Claude Code's client-side estimate; on a
  116. subscription nothing is billed per run. Its system prompt is not the API harness's, so compare D
  117. with E, not D with A.
  118. - **Claude Code Code Mode does not run under `-p`.** The docs require an interactive terminal and a
  119. one-time opt-in, and under `claude -p` (v2.1.330) the `REPL` tool never appears. The headless runner
  120. therefore refuses mode E (it stops at Claude Code's init event and exits with code 4), and mode E is
  121. measured from interactive transcripts with the manual steps in
  122. [`docs/claude-code-code-mode.md`](docs/claude-code-code-mode.md). A mode E record without a `REPL`
  123. call is reported as invalid, never as Code Mode.
  124. - **Claude Code isolation.** Runs use `--strict-mcp-config`, `--setting-sources ""` (no user hooks,
  125. plugins or settings), an empty working folder, and only the mock server's tools, plus `ToolSearch` in
  126. D (Claude Code defers MCP tools behind it) and `REPL` in E.
  127. Claude Code may also trim or spill very large MCP results; that is recorded, not hidden.
  128. - **Mock latency is zero.** The real server waits on Google's APIs; every extra round trip would cost
  129. more wall time there than here, in every mode.
  130. - **One model, one server, three tasks.** Sonnet 5.5 at `medium` effort. Mode C writes Python; mode E
  131. writes JavaScript. Five repetitions per cell is enough to see a direction, not to settle a decimal.
  132. - **No refusal fallbacks.** A fallback would switch models mid-benchmark, so a refusal ends the run as
  133. an error.
  134. ## License
  135. [MIT](LICENSE) © José Antonio Garrido

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.