Skip to content
Back to search

MCPBench

The evaluation benchmark on MCP servers

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/modelscope--mcpbench/badge.svg)](/listings/modelscope--mcpbench)

HTML

<a href="/listings/modelscope--mcpbench"><img src="/api/listings/modelscope--mcpbench/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

The framework requires Python version >= 3.11, nodejs and jq.

conda create -n mcpbench python=3.11 -y
conda activate mcpbench
pip install -r requirements.txt

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 179 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <h1 align="center">
  2. 🦊 MCPBench: A Benchmark for Evaluating MCP Servers
  3. </h1>
  4. <div align="center">
  5. [![Documentation][docs-image]][docs-url]
  6. [![Package License][package-license-image]][package-license-url]
  7. </div>
  8. <div align="center">
  9. <h4 align="center">
  10. [中文](https://github.com/modelscope/MCPBench/blob/main/README_zh.md) |
  11. [English](https://github.com/modelscope/MCPBench/blob/main/README.md)
  12. </h4>
  13. </div>
  14. MCPBench is an evaluation framework for MCP Servers. It supports the evaluation of three types of servers: Web Search, Database Query and GAIA, and is compatible with both local and remote MCP Servers. The framework primarily evaluates different MCP Servers (such as Brave Search, DuckDuckGo, etc.) in terms of task completion accuracy, latency, and token consumption under the same LLM and Agent configurations. Here is the [evaluation report](https://arxiv.org/abs/2504.11094).
  15. <img src="assets/figure1.png" alt="MCPBench Overview" width="600"/>
  16. > The implementation refers to [LangProBe: a Language Programs Benchmark](https://arxiv.org/abs/2502.20315).\
  17. > Big thanks to Qingxu Fu for the initial implementation!
  18. <hr>
  19. # 📋 Table of Contents
  20. - [🔥 News](#news)
  21. - [🛠️ Installation](#installation)
  22. - [🚀 Quick Start](#quick-start)
  23. - [Launch MCP Server](#launch-mcp-server)
  24. - [Launch Evaluation](#launch-evaluation)
  25. - [🧂 Datasets and Experiments](#datasets-and-experiments)
  26. - [🚰 Cite](#cite)
  27. # 🔥 News
  28. + `Sep. 1, 2025` 🌟 Modelscope AI hackathon will be hold on Sep. 23rd, ref: https://modelscope.cn/active/aihackathon-mcp-agent
  29. + `Apr. 29, 2025` 🌟 Update the code for evaluating the MCP Server Package within GAIA.
  30. + `Apr. 14, 2025` 🌟 We are proud to announce that MCPBench is now open-sourced.
  31. # 🛠️ Installation
  32. The framework requires Python version >= 3.11, nodejs and jq.
  33. ```bash
  34. conda create -n mcpbench python=3.11 -y
  35. conda activate mcpbench
  36. pip install -r requirements.txt
  37. ```
  38. # 🚀 Quick Start
  39. Please first determine the type of MCP server you want to use:
  40. - If it is a remote host (accessed via **SSE**, such as [ModelScope](https://modelscope.cn/mcp), [Smithery](https://smithery.ai), or localhost), you can directly conduct the [evaluation](#launch-evaluation).
  41. - If it is started locally (accessed via npx using **STDIO**), you need to launch it.
  42. ## Launch MCP Server (optional for stdio)
  43. First, you need to write the following configuration:
  44. ```json
  45. {
  46. "mcp_pool": [
  47. {
  48. "name": "firecrawl",
  49. "run_config": [
  50. {
  51. "command": "npx -y firecrawl-mcp",
  52. "args": "FIRECRAWL_API_KEY=xxx",
  53. "port": 8005
  54. }
  55. ]
  56. }
  57. ]
  58. }
  59. ```
  60. Save this config file in the `configs` folder and launch it using:
  61. ```bash
  62. sh launch_mcps_as_sse.sh YOUR_CONFIG_FILE
  63. ```
  64. For example, save the above configuration in the `configs/firecrawl.json` file and launch it using:
  65. ```bash
  66. sh launch_mcps_as_sse.sh firecrawl.json
  67. ```
  68. ## Launch Evaluation
  69. To evaluate the MCP Server's performance, you need to set up the necessary MCP Server information. the code will automatically detect the tools and parameters in the Server, so you don't need to configure them manually, like:
  70. ```json
  71. {
  72. "mcp_pool": [
  73. {
  74. "name": "Remote MCP example",
  75. "url": "url from https://modelscope.cn/mcp or https://smithery.ai"
  76. },
  77. {
  78. "name": "firecrawl (Local run example)",
  79. "run_config": [
  80. {
  81. "command": "npx -y firecrawl-mcp",
  82. "args": "FIRECRAWL_API_KEY=xxx",
  83. "port": 8005
  84. }
  85. ]
  86. }
  87. ]
  88. }
  89. ```
  90. To evaluate the MCP Server's performance on WebSearch tasks:
  91. ```bash
  92. sh evaluation_websearch.sh YOUR_CONFIG_FILE
  93. ```
  94. To evaluate the MCP Server's performance on Database Query tasks:
  95. ```bash
  96. sh evaluation_db.sh YOUR_CONFIG_FILE
  97. ```
  98. To evaluate the MCP Server's performance on GAIA tasks:
  99. ```bash
  100. sh evaluation_gaia.sh YOUR_CONFIG_FILE
  101. ```
  102. For example, save the above configuration in the `configs/firecrawl.json` file and launch it using:
  103. ```bash
  104. sh evaluation_websearch.sh firecrawl.json
  105. ```
  106. # Datasets and Experimental Results
  107. Our framework provides two datasets for evaluation. For the WebSearch task, the dataset is located at `MCPBench/langProBe/WebSearch/data/websearch_600.jsonl`, containing 200 QA pairs each from [Frames](https://arxiv.org/abs/2409.12941), news, and technology domains. Our framework for automatically constructing evaluation datasets will be open-sourced later.
  108. For the Database Query task, the dataset is located at `MCPBench/langProBe/DB/data/car_bi.jsonl`. You can add your own dataset in the following format:
  109. ```json
  110. {
  111. "unique_id": "",
  112. "Prompt": "",
  113. "Answer": ""
  114. }
  115. ```
  116. We have evaluated mainstream MCP Servers on both tasks. For detailed experimental results, please refer to [Documentation](https://arxiv.org/abs/2504.11094)
  117. # 🚰 Cite
  118. If you find this work useful, please consider citing our project or giving us a 🌟:
  119. ```bibtex
  120. @misc{mcpbench,
  121. title={MCPBench: A Benchmark for Evaluating MCP Servers},
  122. author={Zhiling Luo, Xiaorong Shi, Xuanrui Lin, Jinyang Gao},
  123. howpublished = {\url{https://github.com/modelscope/MCPBench}},
  124. year={2025}
  125. }
  126. ```
  127. Alternatively, you may reference our report.
  128. ```bibtex
  129. @article{mcpbench_report,
  130. title={Evaluation Report on MCP Servers},
  131. author={Zhiling Luo, Xiaorong Shi, Xuanrui Lin, Jinyang Gao},
  132. year={2025},
  133. journal={arXiv preprint arXiv:2504.11094},
  134. url={https://arxiv.org/abs/2504.11094},
  135. primaryClass={cs.AI}
  136. }
  137. ```
  138. [docs-image]: https://img.shields.io/badge/Documentation-EB3ECC
  139. [docs-url]: https://arxiv.org/abs/2504.11094
  140. [package-license-image]: https://img.shields.io/badge/License-Apache_2.0-blue.svg
  141. [package-license-url]: https://github.com/modelscope/MCPBench/blob/main/LICENSE

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.