Skip to content
Back to search

mcp-bench

MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/accenture--mcp-bench/badge.svg)](/listings/accenture--mcp-bench)

HTML

<a href="/listings/accenture--mcp-bench"><img src="/api/listings/accenture--mcp-bench/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Installation

1. **Clone the repository**

git clone https://github.com/accenture/mcp-bench.git
cd mcp-bench

2. **Install dependencies**

conda create -n mcpbench python=3.10
conda activate mcpbench
cd mcp_servers
# Install MCP server dependencies
bash ./install.sh
cd ..

3. **Set up environment variables**

# Create .env file with API keys
# Default setup uses both OpenRouter and Azure OpenAI
# For Azure OpenAI, you also need to set your API version in file benchmark_config.yaml (line205)
# For OpenRouter-only setup, see "Optional: Using only OpenRouter API" section below
cat > .env << EOF
export OPENROUTER_API_KEY="your_openrouterkey_here"
export AZURE_OPENAI_API_KEY="your_azureopenai_apikey_here"
export AZURE_OPENAI_ENDPOINT="your_azureopenai_endpoint_here"
EOF

4. **Configure MCP Server API Keys**

Some MCP servers require external API keys to function properly. These keys are automatically loaded from `./mcp_servers/api_key`. You should set these keys by yourself in file `./mcp_servers/api_key`:

# View configured API keys
cat ./mcp_servers/api_key

Required API keys include (These API keys are free and easy to get. You can get all of them within 10 mins): - `NPS_API_KEY`: National Park Service API key (for nationalparks server) - [Get API key](https://www.nps.gov/subjects/developer/get-started.htm) - `NASA_API_KEY`: NASA Open Data API key (for nasa-mcp server) - [Get API key](https://api.nasa.gov/) - `HF_TOKEN`: Hugging Face token (for huggingface-mcp-server) - [Get token](https://huggingface.co/docs/hub/security-tokens) - `GOOGLE_MAPS_API_KEY`: Google Maps API key (for mcp-google-map server) - [Get API key](https://developers.google.com/maps) - `NCI_API_KEY`: National Cancer Institute API key (for biomcp server) - [Get API key](https://clinicaltrialsapi.cancer.gov/signin) This api key registration website might require US IP to open, see Issue #10 if you have difficulies for getting this api key.

Basic Usage

# 1. Verify all MCP servers can be connected
##You should see "28/28 servers connected" 
##and "All successfully connected servers returned tools!" after running this
python ./utils/collect_mcp_info.py

# 2. List available models
source .env
python run_benchmark.py --list-models 

# 3. Run benchmark (gpt-oss-20b as an example)
##Must use o4-mini as judge model (hard-coded in line 429-436 in ./benchmark/runner.py) to reproduce the results.
## run all tasks
source .env
python run_benchmark.py --models gpt-oss-20b

## single server tasks
source .env
python run_benchmark.py --models gpt-oss-20b \
--tasks-file tasks/mcpbench_tasks_single_runner_format.json

## two server tasks
source .env
python run_benchmark.py --models gpt-oss-20b \
--tasks-file tasks/mcpbench_tasks_multi_2server_runner_format.json

## three server tasks
source .env
python run_benchmark.py --models gpt-oss-20b \
--tasks-file tasks/mcpbench_tasks_multi_3server_runner_format.json

Optional: Add other model providers

To add new models from OpenRouter:

1. **Find your model on OpenRouter** - Visit [OpenRouter Models](https://openrouter.ai/models) to browse available models - Copy the model ID (e.g., `anthropic/claude-sonnet-4` or `meta-llama/llama-3.3-70b-instruct`)

2. **Add the model configuration** - Edit `llm/factory.py` and add your model in the OpenRouter section (around line 152) - Follow this pattern:

   configs["your-model-name"] = ModelConfig(
       name="your-model-name",
       provider_type="openrouter",
       api_key=os.getenv("OPENROUTER_API_KEY"),
       base_url="https://openrouter.ai/api/v1",
       model_name="provider/model-id"  # The exact model ID from OpenRouter
   )

3. **Verify the model is available**

   source .env
   python run_benchmark.py --list-models
   # Your new model should appear in the list

4. **Run benchmark with your model**

   source .env

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 349 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
  2. [![arXiv](https://img.shields.io/badge/arXiv-2508.20453-b31b1b.svg)](https://arxiv.org/abs/2508.20453)
  3. [![Leaderboard](https://img.shields.io/badge/🤗%20Hugging%20Face-Leaderboard-FFD21E)](https://huggingface.co/spaces/mcpbench/mcp-bench)
  4. [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
  5. [![Python Version](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
  6. [![MCP Protocol](https://img.shields.io/badge/MCP-Protocol-green)](https://github.com/anthropics/mcp)
  7. ![MCP-Bench](./images/mcpbench_intro.png)
  8. ## Overview
  9. MCP-Bench is a comprehensive evaluation framework designed to assess Large Language Models' (LLMs) capabilities in tool-use scenarios through the Model Context Protocol (MCP). This benchmark provides an end-to-end pipeline for evaluating how effectively different LLMs can discover, select, and utilize tools to solve real-world tasks.
  10. ## News
  11. * [2025-09] MCP-Bench is accepted to NeurIPS 2025 Workshop on Scaling Environments for Agents.
  12. ## Leaderboard
  13. | Rank | Model | Overall Score |
  14. |------|-------|---------------|
  15. | 1 | gpt-5 | 0.749 |
  16. | 2 | o3 | 0.715 |
  17. | 3 | gpt-oss-120b | 0.692 |
  18. | 4 | gemini-2.5-pro | 0.690 |
  19. | 5 | claude-sonnet-4 | 0.681 |
  20. | 6 | qwen3-235b-a22b-2507 | 0.678 |
  21. | 7 | glm-4.5 | 0.668 |
  22. | 8 | gpt-oss-20b | 0.654 |
  23. | 9 | kimi-k2 | 0.629 |
  24. | 10 | qwen3-30b-a3b-instruct-2507 | 0.627 |
  25. | 11 | gemini-2.5-flash-lite | 0.598 |
  26. | 12 | gpt-4o | 0.595 |
  27. | 13 | gemma-3-27b-it | 0.582 |
  28. | 14 | llama-3-3-70b-instruct | 0.558 |
  29. | 15 | gpt-4o-mini | 0.557 |
  30. | 16 | mistral-small-2503 | 0.530 |
  31. | 17 | llama-3-1-70b-instruct | 0.510 |
  32. | 18 | nova-micro-v1 | 0.508 |
  33. | 19 | llama-3-2-90b-vision-instruct | 0.495 |
  34. | 20 | llama-3-1-8b-instruct | 0.428 |
  35. *Overall Score represents the average performance across all evaluation dimensions including rule-based schema understanding, LLM-judged (o4-mini as judge model) task completion, tool usage, and planning effectiveness. Scores are averaged across single-server and multi-server settings.*
  36. ## Quick Start
  37. ### Installation
  38. 1. **Clone the repository**
  39. ```bash
  40. git clone https://github.com/accenture/mcp-bench.git
  41. cd mcp-bench
  42. ```
  43. 2. **Install dependencies**
  44. ```bash
  45. conda create -n mcpbench python=3.10
  46. conda activate mcpbench
  47. cd mcp_servers
  48. # Install MCP server dependencies
  49. bash ./install.sh
  50. cd ..
  51. ```
  52. 3. **Set up environment variables**
  53. ```bash
  54. # Create .env file with API keys
  55. # Default setup uses both OpenRouter and Azure OpenAI
  56. # For Azure OpenAI, you also need to set your API version in file benchmark_config.yaml (line205)
  57. # For OpenRouter-only setup, see "Optional: Using only OpenRouter API" section below
  58. cat > .env << EOF
  59. export OPENROUTER_API_KEY="your_openrouterkey_here"
  60. export AZURE_OPENAI_API_KEY="your_azureopenai_apikey_here"
  61. export AZURE_OPENAI_ENDPOINT="your_azureopenai_endpoint_here"
  62. EOF
  63. ```
  64. 4. **Configure MCP Server API Keys**
  65. Some MCP servers require external API keys to function properly. These keys are automatically loaded from `./mcp_servers/api_key`. You should set these keys by yourself in file `./mcp_servers/api_key`:
  66. ```bash
  67. # View configured API keys
  68. cat ./mcp_servers/api_key
  69. ```
  70. Required API keys include (These API keys are free and easy to get. You can get all of them within 10 mins):
  71. - `NPS_API_KEY`: National Park Service API key (for nationalparks server) - [Get API key](https://www.nps.gov/subjects/developer/get-started.htm)
  72. - `NASA_API_KEY`: NASA Open Data API key (for nasa-mcp server) - [Get API key](https://api.nasa.gov/)
  73. - `HF_TOKEN`: Hugging Face token (for huggingface-mcp-server) - [Get token](https://huggingface.co/docs/hub/security-tokens)
  74. - `GOOGLE_MAPS_API_KEY`: Google Maps API key (for mcp-google-map server) - [Get API key](https://developers.google.com/maps)
  75. - `NCI_API_KEY`: National Cancer Institute API key (for biomcp server) - [Get API key](https://clinicaltrialsapi.cancer.gov/signin) This api key registration website might require US IP to open, see Issue #10 if you have difficulies for getting this api key.
  76. ### Basic Usage
  77. ```bash
  78. # 1. Verify all MCP servers can be connected
  79. ##You should see "28/28 servers connected"
  80. ##and "All successfully connected servers returned tools!" after running this
  81. python ./utils/collect_mcp_info.py
  82. # 2. List available models
  83. source .env
  84. python run_benchmark.py --list-models
  85. # 3. Run benchmark (gpt-oss-20b as an example)
  86. ##Must use o4-mini as judge model (hard-coded in line 429-436 in ./benchmark/runner.py) to reproduce the results.
  87. ## run all tasks
  88. source .env
  89. python run_benchmark.py --models gpt-oss-20b
  90. ## single server tasks
  91. source .env
  92. python run_benchmark.py --models gpt-oss-20b \
  93. --tasks-file tasks/mcpbench_tasks_single_runner_format.json
  94. ## two server tasks
  95. source .env
  96. python run_benchmark.py --models gpt-oss-20b \
  97. --tasks-file tasks/mcpbench_tasks_multi_2server_runner_format.json
  98. ## three server tasks
  99. source .env
  100. python run_benchmark.py --models gpt-oss-20b \
  101. --tasks-file tasks/mcpbench_tasks_multi_3server_runner_format.json
  102. ```
  103. ### Optional: Add other model providers
  104. To add new models from OpenRouter:
  105. 1. **Find your model on OpenRouter**
  106. - Visit [OpenRouter Models](https://openrouter.ai/models) to browse available models
  107. - Copy the model ID (e.g., `anthropic/claude-sonnet-4` or `meta-llama/llama-3.3-70b-instruct`)
  108. 2. **Add the model configuration**
  109. - Edit `llm/factory.py` and add your model in the OpenRouter section (around line 152)
  110. - Follow this pattern:
  111. ```python
  112. configs["your-model-name"] = ModelConfig(
  113. name="your-model-name",
  114. provider_type="openrouter",
  115. api_key=os.getenv("OPENROUTER_API_KEY"),
  116. base_url="https://openrouter.ai/api/v1",
  117. model_name="provider/model-id" # The exact model ID from OpenRouter
  118. )
  119. ```
  120. 3. **Verify the model is available**
  121. ```bash
  122. source .env
  123. python run_benchmark.py --list-models
  124. # Your new model should appear in the list
  125. ```
  126. 4. **Run benchmark with your model**
  127. ```bash
  128. source .env
  129. python run_benchmark.py --models your-model-name
  130. ```
  131. ### Optional: Using only OpenRouter API
  132. If you only want to use OpenRouter without Azure:
  133. 1. **Set up .env file with only OpenRouter:**
  134. ```bash
  135. cat > .env << EOF
  136. OPENROUTER_API_KEY=your_openrouterkey_here
  137. EOF
  138. ```
  139. 2. **Modify the code to access Azure models through OpenRouter:**
  140. Edit `llm/factory.py` and comment out the Azure section (lines 69-101), then add Azure models through OpenRouter instead:
  141. ```python
  142. # Comment out or remove the Azure section (lines 69-109)
  143. # if os.getenv("AZURE_OPENAI_API_KEY") and os.getenv("AZURE_OPENAI_ENDPOINT"):
  144. # configs["o4-mini"] = ModelConfig(...)
  145. # ...
  146. # Add Azure models through OpenRouter (in the OpenRouter section around line 106)
  147. if os.getenv("OPENROUTER_API_KEY"):
  148. # Add OpenAI models via OpenRouter
  149. configs["gpt-4o"] = ModelConfig(
  150. name="gpt-4o",
  151. provider_type="openrouter",
  152. api_key=os.getenv("OPENROUTER_API_KEY"),
  153. base_url="https://openrouter.ai/api/v1",
  154. model_name="openai/gpt-4o"
  155. )
  156. configs["gpt-4o-mini"] = ModelConfig(
  157. name="gpt-4o-mini",
  158. provider_type="openrouter",
  159. api_key=os.getenv("OPENROUTER_API_KEY"),
  160. base_url="https://openrouter.ai/api/v1",
  161. model_name="openai/gpt-4o-mini"
  162. )
  163. configs["o3"] = ModelConfig(
  164. name="o3",
  165. provider_type="openrouter",
  166. api_key=os.getenv("OPENROUTER_API_KEY"),
  167. base_url="https://openrouter.ai/api/v1",
  168. model_name="openai/o3"
  169. )
  170. configs["o4-mini"] = ModelConfig(
  171. name="o4-mini",
  172. provider_type="openrouter",
  173. api_key=os.getenv("OPENROUTER_API_KEY"),
  174. base_url="https://openrouter.ai/api/v1",
  175. model_name="openai/o4-mini"
  176. )
  177. configs["gpt-5"] = ModelConfig(
  178. name="gpt-5",
  179. provider_type="openrouter",
  180. api_key=os.getenv("OPENROUTER_API_KEY"),
  181. base_url="https://openrouter.ai/api/v1",
  182. model_name="openai/gpt-5"
  183. )
  184. # Keep existing OpenRouter models...
  185. ```
  186. This way all models will be accessed through OpenRouter's unified API.
  187. ## MCP Servers
  188. MCP-Bench includes 28 diverse MCP servers:
  189. - [BioMCP](https://github.com/genomoncology/biomcp) - Biomedical research data, clinical trials, and health information
  190. - [Bibliomantic](https://github.com/d4nshields/bibliomantic-mcp-server) - I Ching divination, hexagrams, and mystical guidance
  191. - [Call for Papers](https://github.com/iremert/call-for-papers-mcp) - Academic conference submissions and call announcements
  192. - [Car Price Evaluator](https://github.com/yusaaztrk/car-price-mcp-main) - Vehicle valuation and automotive market analysis
  193. - [Context7](https://github.com/upstash/context7) - Project context management and documentation services
  194. - [DEX Paprika](https://github.com/coinpaprika/dexpaprika-mcp) - Cryptocurrency DeFi analytics and decentralized exchange data
  195. - [FruityVice](https://github.com/CelalKhalilov/fruityvice-mcp) - Comprehensive fruit nutrition information and dietary data
  196. - [Game Trends](https://github.com/halismertkir/game-trends-mcp) - Gaming industry statistics and trend analysis
  197. - [Google Maps](https://github.com/cablate/mcp-google-map) - Location services, geocoding, and mapping functionality
  198. - [Huge Icons](https://github.com/hugeicons/mcp-server) - Icon search, management, and design resources
  199. - [Hugging Face](https://github.com/shreyaskarnik/huggingface-mcp-server) - Machine learning models, datasets, and AI capabilities
  200. - [Math MCP](https://github.com/EthanHenrickson/math-mcp) - Mathematical calculations and computational operations
  201. - [Medical Calculator](https://github.com/vitaldb/medcalc) - Clinical calculation tools and medical formulas
  202. - [Metropolitan Museum](https://github.com/mikechao/metmuseum-mcp) - Art collection database and museum information
  203. - [Movie Recommender](https://github.com/iremert/movie-recommender-mcp) - Film recommendations and movie metadata
  204. - [NASA Data](https://github.com/AnCode666/nasa-mcp) - Space mission data and astronomical information
  205. - [National Parks](https://github.com/KyrieTangSheng/mcp-server-nationalparks) - US National Parks information and visitor services
  206. - [NixOS](https://github.com/utensils/mcp-nixos) - Package management and system configuration tools
  207. - [OKX Exchange](https://github.com/esshka/okx-mcp) - Cryptocurrency trading data and market information
  208. - [OpenAPI Explorer](https://github.com/janwilmake/openapi-mcp-server) - API specification exploration and testing tools
  209. - [OSINT Intelligence](https://github.com/himanshusanecha/mcp-osint-server) - Open source intelligence gathering and analysis
  210. - [Paper Search](https://github.com/openags/paper-search-mcp) - Academic paper search across multiple research databases
  211. - [Reddit](https://github.com/dumyCq/mcp-reddit) - Social media content and community discussions
  212. - [Scientific Computing](https://github.com/Aman-Amith-Shastry/scientific_computation_mcp) - Advanced mathematical computations and data analysis
  213. - [Time MCP](https://github.com/dumyCq/time-mcp) - Date, time utilities, and timezone conversions
  214. - [Unit Converter](https://github.com/zazencodes/unit-converter-mcp) - Measurement conversions across different unit systems
  215. - [Weather Data](https://github.com/HarunGuclu/weather_mcp) - Weather forecasts and meteorological information
  216. - [Wikipedia](https://github.com/Rudra-ravi/wikipedia-mcp) - Encyclopedia content search and retrieval
  217. ## Project Structure
  218. ```
  219. mcp-bench/
  220. ├── agent/ # Task execution agents
  221. │ ├── __init__.py
  222. │ ├── executor.py # Multi-round task executor with retry logic
  223. │ └── execution_context.py # Execution context management
  224. ├── benchmark/ # Evaluation framework
  225. │ ├── __init__.py
  226. │ ├── evaluator.py # LLM-as-judge evaluation metrics
  227. │ ├── runner.py # Benchmark orchestrator
  228. │ ├── results_aggregator.py # Results aggregation and statistics
  229. │ └── results_formatter.py # Results formatting and display
  230. ├── config/ # Configuration management
  231. │ ├── __init__.py
  232. │ ├── benchmark_config.yaml # Benchmark configuration
  233. │ └── config_loader.py # Configuration loader
  234. ├── llm/ # LLM provider abstractions
  235. │ ├── __init__.py
  236. │ ├── factory.py # Model factory for multiple providers
  237. │ └── provider.py # Unified provider interface
  238. ├── mcp_modules/ # MCP server management
  239. │ ├── __init__.py
  240. │ ├── connector.py # Server connection handling
  241. │ ├── server_manager.py # Multi-server orchestration
  242. │ ├── server_manager_persistent.py # Persistent connection manager
  243. │ └── tool_cache.py # Tool call caching mechanism
  244. ├── synthesis/ # Task generation
  245. │ ├── __init__.py
  246. │ ├── task_synthesis.py # Task generation with fuzzy conversion
  247. │ ├── generate_benchmark_tasks.py # Batch task generation script
  248. │ ├── benchmark_generator.py # Unified benchmark task generator
  249. │ ├── README.md # Task synthesis documentation
  250. │ └── split_combinations/ # Server combination splits
  251. │ ├── mcp_2server_combinations.json
  252. │ └── mcp_3server_combinations.json
  253. ├── utils/ # Utilities
  254. │ ├── __init__.py
  255. │ ├── collect_mcp_info.py # Server discovery and tool collection
  256. │ ├── local_server_config.py # Local server configuration
  257. │ └── error_handler.py # Error handling utilities
  258. ├── tasks/ # Benchmark task files
  259. │ ├── mcpbench_tasks_single_runner_format.json
  260. │ ├── mcpbench_tasks_multi_2server_runner_format.json
  261. │ └── mcpbench_tasks_multi_3server_runner_format.json
  262. ├── mcp_servers/ # MCP server implementations (28 servers)
  263. │ ├── api_key # API keys configuration file
  264. │ ├── commands.json # Server command configurations
  265. │ ├── install.sh # Installation script for all servers
  266. │ ├── requirements.txt # Python dependencies
  267. │ └── [28 server directories]
  268. ├── cache/ # Tool call cache directory (auto-created)
  269. ├── run_benchmark.py # Main benchmark runner script
  270. ├── README.md # Project documentation
  271. ├── .gitignore # Git ignore configuration
  272. └── .gitmodules # Git submodules configuration
  273. ```
  274. ## Citation
  275. If you use MCP-Bench in your research, please cite:
  276. ```bibtex
  277. @article{wang2025mcpbench,
  278. title={MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers},
  279. author={Wang, Zhenting and Chang, Qi and Patel, Hemani and Biju, Shashank and Wu, Cheng-En and Liu, Quan and Ding, Aolin and Rezazadeh, Alireza and Shah, Ankit and Bao, Yujia and Siow, Eugene},
  280. journal={arXiv preprint arXiv:2508.20453},
  281. year={2025}
  282. }
  283. ```
  284. ## Star History
  285. [![Star History Chart](https://api.star-history.com/svg?repos=accenture/mcp-bench&type=Date)](https://star-history.com/#accenture/mcp-bench&Date)
  286. ## Acknowledgments
  287. - Built on the [Model Context Protocol](https://github.com/anthropics/mcp) by Anthropic
  288. - Thanks to all open-sourced MCP servers implemetation used

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.