Skip to content
Back to search

mcp-atlas

MCP Atlas

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/scaleapi/mcp-atlas with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/scaleapi/mcp-atlas'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/scaleapi--mcp-atlas/badge.svg)](/listings/scaleapi--mcp-atlas)

HTML

<a href="/listings/scaleapi--mcp-atlas"><img src="/api/listings/scaleapi--mcp-atlas/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires [docker](https://www.docker.com/products/docker-desktop/), [jq](https://jqlang.org/download/), and Python 3.10+.

git clone git@github.com:scaleapi/mcp-atlas.git && cd mcp-atlas

1. Configure

cp env.template .env

Set in `.env`: - `LLM_API_KEY` — key for the model under evaluation (comma-separated keys are rotated per request). - `LLM_BASE_URL` — any OpenAI Chat-Completions-compatible endpoint (a [LiteLLM](https://docs.litellm.ai/) proxy, OpenAI, Anthropic-via-LiteLLM, Azure, or a self-hosted vLLM/TGI server). - `EVAL_LLM_API_KEY` / `EVAL_LLM_BASE_URL` / `EVAL_LLM_MODEL` — *optional* judge settings for scoring and diagnostics; fall back to `LLM_*`, with the judge defaulting to `gemini/gemini-3.1-pro-preview`. - `MCP_SANDBOX_URL` — *optional*, defaults to `http://localhost:1984`.

> The agent harness was rewritten from Python to TypeScript in v2.0.0 — see [`CHANGELOG.md`](CHANGELOG.md).

2. Start the MCP servers

Allocate at least 8 GB (10 GB+ recommended) to Docker.

**Option A — prebuilt image (recommended):**

docker pull ghcr.io/scaleapi/mcp-atlas:1.2.7
docker tag ghcr.io/scaleapi/mcp-atlas:1.2.7 agent-environment:latest
make run-docker

**Option B — build from source** (only if you're modifying the server set, pinned versions, or baked-in data):

make build && make run-docker

Neither bakes in API keys — both inject them at runtime from `.env`. Startup takes 1+ minute; wait for `Uvicorn running on http://0.0.0.0:1984`. The 20 no-key servers are enabled by default; key-gated servers turn on when their keys are present. Verify:

curl -s http://localhost:1984/enabled-servers | jq -c

3. Start the agent harness (new terminal)

make install-harness
make run-harness

Starts the TypeScript harness on port 3001, exposing `/v2/mcp_eval/run_agent` — the multi-turn agent loop that runs the model against the sandbox until it finishes or hits a limit.

4. Smoke-test one task (new terminal)

Expected answer: "Customer".

curl -X POST http://localhost:3001/v2/mcp_eval/run_agent \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "What is the first word of the file at /data/Barber Shop.csv?"}],
    "enabledTools": ["filesystem_read_text_file"],
    "image": "ghcr.io/scaleapi/mcp-atlas:1.2.7"
  }' | jq

5. Run the full eval

make install-python   # one-time: deps for run_eval, scoring, diagnostics
python run_eval.py --model "openai/gpt-4o" --output outputs.csv

Pulls the 500-task dataset from HuggingFace by default; pass `--input tasks.csv` for a local CSV (`TASK`, `PROMPT`, `ENABLED_TOOLS` columns). Reruns skip already-completed `task_id`s, so an interrupted run resumes by rerunning the same command. Output columns: `task_id`, `raw_conversation_history`, `response`.

Keep each run's artifacts together by writing `--output` into a per-run directory and pointing the scoring/diagnosis steps at the same directory.

Configuration

Override any default per run:

| Flag | Default | What it does | |------|---------|--------------| | `--max-turns N` | `256` | Max agent-loop iterations per task. | | `--max-tool-calls N` | `100` | Max total tool calls per task. | | `--tool-output-cap N` | uncapped | Truncate each tool result to N characters before it's fed back to the model. | | `--context-window-management compact` | off | Summarize older turns once the conversation grows large. | | `--extra-llm-params '<json>'` | none | Forward a JSON object verbatim into the completion request (e.g. reasoning level). | | `--system-prompt "..."` | none | Prepend a system message to every task. | | `--concurrency N` | `5` | Tasks run in parallel. | | `--timeout S` | `1800` | Per-task timeout, in seconds. | | `--num-tasks N` | all | Run only the first N tasks. | | `--input PATH` | HuggingFace | Use a local CSV instead of `ScaleAI/MCP-Atlas`. |

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 192 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
  2. MCP-Atlas evaluates how well AI agents use tools to complete real-world tasks, across 36 Model Context Protocol (MCP) servers in a reproducible Docker sandbox, scored with an LLM-as-judge.
  3. - Paper: [arxiv.org/abs/2602.00933](https://arxiv.org/abs/2602.00933) ([local copy](assets/MCP_Atlas.pdf))
  4. - Leaderboard: [scale.com/leaderboard/mcp_atlas](https://scale.com/leaderboard/mcp_atlas)
  5. - Dataset: [huggingface.co/datasets/ScaleAI/MCP-Atlas](https://huggingface.co/datasets/ScaleAI/MCP-Atlas)
  6. ## Overview
  7. - **36 real MCP servers** spanning search, code execution, databases, APIs, and productivity tools — 20 need no setup, 11 require API keys, and 5 require API keys plus data setup (see [`data_exports/README.md`](data_exports/README.md)). All are open-source and version-pinned for reproducibility.
  8. - **500 tasks** with ground-truth expected tool calls and answers.
  9. - **LLM-as-judge scoring** reporting pass rate and coverage, plus per-task failure-mode diagnostics.
  10. Server definitions are in [`mcp_server_template.json`](services/agent-environment/src/agent_environment/mcp_server_template.json); a full list of the 36 servers and 307 tools is [here](https://gist.github.com/geobio/d0272d41ea395376233f1617a3988860).
  11. ## Quick Start
  12. Requires [docker](https://www.docker.com/products/docker-desktop/), [jq](https://jqlang.org/download/), and Python 3.10+.
  13. ```bash
  14. git clone git@github.com:scaleapi/mcp-atlas.git && cd mcp-atlas
  15. ```
  16. ### 1. Configure
  17. ```bash
  18. cp env.template .env
  19. ```
  20. Set in `.env`:
  21. - `LLM_API_KEY` — key for the model under evaluation (comma-separated keys are rotated per request).
  22. - `LLM_BASE_URL` — any OpenAI Chat-Completions-compatible endpoint (a [LiteLLM](https://docs.litellm.ai/) proxy, OpenAI, Anthropic-via-LiteLLM, Azure, or a self-hosted vLLM/TGI server).
  23. - `EVAL_LLM_API_KEY` / `EVAL_LLM_BASE_URL` / `EVAL_LLM_MODEL` — *optional* judge settings for scoring and diagnostics; fall back to `LLM_*`, with the judge defaulting to `gemini/gemini-3.1-pro-preview`.
  24. - `MCP_SANDBOX_URL` — *optional*, defaults to `http://localhost:1984`.
  25. - `SANDBOX_AUTH_TOKEN` — shared secret the sandbox API requires (it runs tools with your API keys). `make run-docker` and `run_all.sh` generate one in `.env` if it's empty; the harness, `run_eval.py` and `test_servers.py` read it from `.env` and send it as a bearer token.
  26. > The agent harness was rewritten from Python to TypeScript in v2.0.0 — see [`CHANGELOG.md`](CHANGELOG.md).
  27. ### 2. Start the MCP servers
  28. Allocate at least 8 GB (10 GB+ recommended) to Docker.
  29. **Option A — prebuilt image (recommended):**
  30. ```bash
  31. docker pull ghcr.io/scaleapi/mcp-atlas:1.2.8
  32. docker tag ghcr.io/scaleapi/mcp-atlas:1.2.8 agent-environment:latest
  33. make run-docker
  34. ```
  35. > `1.2.8` is the first prebuilt image that requires `SANDBOX_AUTH_TOKEN` on every API call. Earlier tags serve the tool API with no authentication, so never publish their port beyond loopback.
  36. **Option B — build from source** (only if you're modifying the server set, pinned versions, or baked-in data):
  37. ```bash
  38. make build && make run-docker
  39. ```
  40. Neither bakes in API keys — both inject them at runtime from `.env`. Startup takes 1+ minute; wait for `Uvicorn running on http://0.0.0.0:1984`. The 20 no-key servers are enabled by default; key-gated servers turn on when their keys are present. Verify:
  41. ```bash
  42. export $(grep -E '^SANDBOX_AUTH_TOKEN=' .env)
  43. curl -s -H "Authorization: Bearer $SANDBOX_AUTH_TOKEN" http://localhost:1984/enabled-servers | jq -c
  44. ```
  45. ### 3. Start the agent harness (new terminal)
  46. ```bash
  47. make install-harness
  48. make run-harness
  49. ```
  50. Starts the TypeScript harness on port 3001, exposing `/v2/mcp_eval/run_agent` — the multi-turn agent loop that runs the model against the sandbox until it finishes or hits a limit.
  51. ### 4. Smoke-test one task (new terminal)
  52. Expected answer: "Customer".
  53. ```bash
  54. curl -X POST http://localhost:3001/v2/mcp_eval/run_agent \
  55. -H "Content-Type: application/json" \
  56. -d '{
  57. "model": "openai/gpt-4o",
  58. "messages": [{"role": "user", "content": "What is the first word of the file at /data/Barber Shop.csv?"}],
  59. "enabledTools": ["filesystem_read_text_file"],
  60. "image": "ghcr.io/scaleapi/mcp-atlas:1.2.8"
  61. }' | jq
  62. ```
  63. ### 5. Run the full eval
  64. ```bash
  65. make install-python # one-time: deps for run_eval, scoring, diagnostics
  66. python run_eval.py --model "openai/gpt-4o" --output outputs.csv
  67. ```
  68. Pulls the 500-task dataset from HuggingFace by default; pass `--input tasks.csv` for a local CSV (`TASK`, `PROMPT`, `ENABLED_TOOLS` columns). Reruns skip already-completed `task_id`s, so an interrupted run resumes by rerunning the same command. Output columns: `task_id`, `raw_conversation_history`, `response`.
  69. Keep each run's artifacts together by writing `--output` into a per-run directory and pointing the scoring/diagnosis steps at the same directory.
  70. #### Configuration
  71. Override any default per run:
  72. | Flag | Default | What it does |
  73. |------|---------|--------------|
  74. | `--max-turns N` | `256` | Max agent-loop iterations per task. |
  75. | `--max-tool-calls N` | `100` | Max total tool calls per task. |
  76. | `--tool-output-cap N` | uncapped | Truncate each tool result to N characters before it's fed back to the model. |
  77. | `--context-window-management compact` | off | Summarize older turns once the conversation grows large. |
  78. | `--extra-llm-params '<json>'` | none | Forward a JSON object verbatim into the completion request (e.g. reasoning level). |
  79. | `--system-prompt "..."` | none | Prepend a system message to every task. |
  80. | `--concurrency N` | `5` | Tasks run in parallel. |
  81. | `--timeout S` | `1800` | Per-task timeout, in seconds. |
  82. | `--num-tasks N` | all | Run only the first N tasks. |
  83. | `--input PATH` | HuggingFace | Use a local CSV instead of `ScaleAI/MCP-Atlas`. |
  84. | `--image NAME` | `ghcr.io/scaleapi/mcp-atlas:1.2.8` | Sandbox image. |
  85. | `--skip-health-check` | off | Skip the pre-flight health check (one real call per server; the run aborts if any server fails). |
  86. - `--extra-llm-params` sets reasoning/provider-specific options, e.g. `--extra-llm-params '{"reasoning_effort": "high"}'` (use whatever key your provider expects; default is the provider's own).
  87. - Harness request timeouts are env-configurable for slow models: `TOOL_CALL_TIMEOUT_MS` (60000), `LIST_TOOLS_TIMEOUT_MS` (180000), `LLM_TIMEOUT_MS` (600000).
  88. - Each run writes a `run_config.json` beside the output CSV; the scorer embeds it into `coverage_stats_*.json` so every result is traceable to its configuration.
  89. ### 6. Score
  90. ```bash
  91. python services/scoring/score_claims.py \
  92. --groundtruth-file path/to/groundtruth.csv \
  93. --model-file outputs.csv \
  94. --model-name your-model \
  95. --output-dir results/your-model
  96. ```
  97. LLM-as-judge claim-coverage scoring (default judge `gemini/gemini-3.1-pro-preview`). The ground-truth file is the HuggingFace dataset exported to CSV (columns `TASK`, `PROMPT`, `GTFA_CLAIMS`), or the same `--input` CSV if you ran locally. Outputs `scored_<model>.csv`, `coverage_stats_<model>_*.json` (pass rates at 0.50 and 0.75 coverage thresholds), and a coverage histogram. `--concurrency` auto-tunes per judge model.
  98. ### 6b. Diagnose failures (optional)
  99. ```bash
  100. python services/diagnostics/single_model_diagnostic.py --scored-file scored_<model>.csv --verbose
  101. ```
  102. Classifies each failing task into one of 11 failure modes (4 tool-call + 7 cognitive) over an enriched trajectory, and writes a `diagnosis_*.csv` plus a model-level narrative.
  103. ### 7. Evaluate another model
  104. Change `LLM_API_KEY` / `LLM_BASE_URL` in `.env`, restart the harness, and rerun with a different `--model`. See [LiteLLM providers](https://docs.litellm.ai/docs/providers) for model names.
  105. ## Scaling throughput
  106. A single sandbox handles concurrent tasks comfortably, and **you can run several evals in parallel against it.** The agent loop is I/O-bound — most of each task's time is spent waiting on the model, not calling tools — so one sandbox stays well under capacity at typical concurrency. Raise `--concurrency` or launch multiple runs as needed; reach for the scale-out options below only when the sandbox itself becomes the bottleneck (very high concurrency, or tool-heavy workloads where some MCP servers degrade under load):
  107. **Shard across independent stacks (simplest).** Run several sandbox + harness pairs on different ports, point `run_eval.py` at a slice of the tasks for each, then concatenate the output CSVs. Each task runs end-to-end on one stack, so within-task state (filesystem, memory, git) stays consistent. The harness's `.env` does not override variables already set in the environment, so per-stack `PORT` / `MCP_SANDBOX_URL` overrides just work:
  108. ```bash
  109. # Stack A — sandbox on 1984, harness on 3001
  110. docker run -d -p 127.0.0.1:1984:1984 --env-file .env ghcr.io/scaleapi/mcp-atlas:1.2.8
  111. PORT=3001 MCP_SANDBOX_URL=http://localhost:1984 make run-harness
  112. # Stack B — sandbox on 1985, harness on 3002
  113. docker run -d -p 127.0.0.1:1985:1984 --env-file .env ghcr.io/scaleapi/mcp-atlas:1.2.8
  114. PORT=3002 MCP_SANDBOX_URL=http://localhost:1985 make run-harness
  115. # Run each half of the dataset against its own harness, then concatenate
  116. HARNESS_URL=http://localhost:3001 python run_eval.py --input tasks_part_a.csv --output out_a.csv --model "<model>"
  117. HARNESS_URL=http://localhost:3002 python run_eval.py --input tasks_part_b.csv --output out_b.csv --model "<model>"
  118. ```
  119. **Point at an orchestrator (scales furthest).** Because the harness reaches the sandbox solely through `MCP_SANDBOX_URL`, you can point it at a service that provisions an ephemeral sandbox per task — no harness changes; any HTTP endpoint implementing the agent-environment API works. The harness sends `Authorization: Bearer $SANDBOX_AUTH_TOKEN` on every call, so the orchestrator should check it (and pass it to the sandboxes it starts).
  120. > **One rule when adding sandboxes:** keep all of a task's tool calls on the same sandbox. Per-call load-balancing across replicas breaks stateful tools (filesystem, memory, git, MongoDB), which assume a consistent view within a task.
  121. ## What's Included
  122. - **Agent harness** (`services/agent-harness/`, TypeScript) — multi-turn agent loop, talks to the sandbox via `MCP_SANDBOX_URL`.
  123. - **Agent environment** (`services/agent-environment/`, Python) — Dockerized sandbox serving the 36 MCP servers over HTTP.
  124. - **Scoring** (`services/scoring/`, Python) — LLM-as-judge claim-coverage scoring.
  125. - **Diagnostics** (`services/diagnostics/`, Python) — failure-mode classification across an 11-mode taxonomy.
  126. ## Citation
  127. If you use MCP-Atlas in your research, please cite:
  128. ```bibtex
  129. @misc{bandi2026mcpatlas,
  130. title = {MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers},
  131. author = {Bandi, Chaithanya and Dumitru, Razvan-Gabriel and Hertzberg, Ben and Agarwal, Divyansh and Boo, Geobio and Polakam, Tejas and Hassaan, Sami and Da, Jeff and Kim, HiJae and Gupta, Vipul and Sharma, Manasi and Park, Andrew and Dimakis, Martin and Hernandez Montoya, Ernesto Gabriel and Rambado, Dan and Salazar, Ivan and Cruz, Rafael and Rezaei, MohammadHossein and Rane, Chetan and Levin, Ben and Zhang, Daniel Yue and Kenstler, Brad and Liu, Bing},
  132. year = {2026},
  133. eprint = {2602.00933},
  134. archivePrefix = {arXiv},
  135. primaryClass = {cs.SE},
  136. url = {https://arxiv.org/abs/2602.00933}
  137. }
  138. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.