Skip to content
Back to search

OSWorld-MCP

OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/x-plug--osworld-mcp/badge.svg)](/listings/x-plug--osworld-mcp)

HTML

<a href="/listings/x-plug--osworld-mcp"><img src="/api/listings/x-plug--osworld-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1️⃣ Preparation: Code Setup

# Clone OSWorld base repo
git clone https://github.com/xlang-ai/OSWorld.git

# Clone OSWorld-MCP
git clone https://github.com/X-PLUG/OSWorld-MCP.git

Integrate **OSWorld-MCP** files into OSWorld to enable MCP support.

---

2️⃣ Preparation: Docker Environment

1. Copy MCP files into `/home` inside Docker:

/home/
└── mcp_server/
└── osworld_mcp_client.py

2. Install dependencies:

pip install -r requirements.txt

3. Install [Node.js](https://nodejs.org/en/download/) 4. Launch MCP server:

cd mcp_server
bash debug_server.sh

A successful launch opens the local MCP debug UI in your browser.

---

3️⃣ Running Evaluation

Example: Evaluate **Claude 4 Sonnet** (15 steps):

python run_multienv_e2e.py \
    --api_url <your_api_url> \
    --api_key <your_api_key> \
    --model 'claude-sonnet-4-20250514-thinking' \
    --test_all_meta_path 'evaluation_examples/test_all.json' \
    --num_envs 1 \
    --action_space mcp \
    --max_steps 15 \
    --max_trajectory_length 15

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 140 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents
  2. ## 🔔 Updates
  3. **2026-05-13:** You Might also be interested: **ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents**. ![Website](https://x-plug.github.io/ToolCUA/)
  4. **2026-05-09:** Fixed bugs in LibreOffice MCP tools. New model results added: Kimi-K2.5, Gemini-3.1-Pro, Claude-4.5-Sonnet, and Qwen3.5-Plus.
  5. **2026-01-26:** OSWorld-MCP is accepted to **ICLR 2026**! 🎉
  6. **2025-10-28:** We released our paper and project page! 🎉
  7. 📄 [Read the Paper](https://arxiv.org/abs/2510.24563) &nbsp;|&nbsp; 🌐 [Visit the Project Page](https://osworld-mcp.github.io)
  8. ---
  9. ## 📑 Overview & Key Highlights
  10. OSWorld-MCP is a comprehensive and fair benchmark for evaluating computer-use agents in real-world scenarios.
  11. It jointly measures **Model Context Protocol (MCP)** tool invocation capabilities, **graphical user interface (GUI)** operation skills, and **decision-making** performance.
  12. Designed as an extension of **OSWorld**, it significantly improves realism, balance, and comparability in evaluation.
  13. **Key Features & Findings**
  14. - **158 validated MCP tools**, spanning **7 common applications** (LibreOffice Writer, Calc, Impress, VS Code, Google Chrome, VLC, OS utilities). Among them, **25 distractor tools** for robustness testing
  15. - **250 tool-beneficial tasks** → 69% of benchmark tasks benefit from MCP tools
  16. - Multi-round tool invocation possible, posing real decision-making challenges
  17. - **MCP tools boost model accuracy & efficiency** — e.g., OpenAI o3: 8.3% → 17.6% (15 steps)
  18. - Highest observed Tool Invocation Rate (**TIR**) = 33.3% (Claude-4-Sonnet, 50 steps) → indicating ample room for improvement
  19. - MCP tools improve agent metrics
  20. - Higher tool invocation correlates with higher accuracy
  21. - Combining tools introduces significant challenges
  22. **Architecture Overview**
  23. ![OSWorld-MCP Architecture](images/architecture.png)
  24. *Figure: OSWorld-MCP evaluation framework integrating GUI actions and MCP tool invocations.*
  25. ---
  26. ## ⚙️ Installation & Usage
  27. ### 1️⃣ Preparation: Code Setup
  28. ```bash
  29. # Clone OSWorld base repo
  30. git clone https://github.com/xlang-ai/OSWorld.git
  31. # Clone OSWorld-MCP
  32. git clone https://github.com/X-PLUG/OSWorld-MCP.git
  33. ```
  34. Integrate **OSWorld-MCP** files into OSWorld to enable MCP support.
  35. ---
  36. ### 2️⃣ Preparation: Docker Environment
  37. 1. Copy MCP files into `/home` inside Docker:
  38. ```
  39. /home/
  40. └── mcp_server/
  41. └── osworld_mcp_client.py
  42. ```
  43. 2. Install dependencies:
  44. ```bash
  45. pip install -r requirements.txt
  46. ```
  47. 3. Install [Node.js](https://nodejs.org/en/download/)
  48. 4. Launch MCP server:
  49. ```bash
  50. cd mcp_server
  51. bash debug_server.sh
  52. ```
  53. A successful launch opens the local MCP debug UI in your browser.
  54. ---
  55. ### 3️⃣ Running Evaluation
  56. Example: Evaluate **Claude 4 Sonnet** (15 steps):
  57. ```bash
  58. python run_multienv_e2e.py \
  59. --api_url <your_api_url> \
  60. --api_key <your_api_key> \
  61. --model 'claude-sonnet-4-20250514-thinking' \
  62. --test_all_meta_path 'evaluation_examples/test_all.json' \
  63. --num_envs 1 \
  64. --action_space mcp \
  65. --max_steps 15 \
  66. --max_trajectory_length 15
  67. ```
  68. ---
  69. ## 📐 Key Metrics
  70. 1. **Task Accuracy (Acc)** — % of tasks successfully completed.
  71. 2. **Tool Invocation Rate (TIR)** — correct decisions to use a tool or not.
  72. 3. **Average Completion Steps (ACS)** — average number of actions per completed task.
  73. ---
  74. ## 📊 Leaderboard (Sorted by Accuracy)
  75. 🔗 **Live Leaderboard:** [osworld-mcp.github.io](https://osworld-mcp.github.io)
  76. **Max Steps: 15**
  77. | Model / Agent | Acc | TIR | ACS |
  78. |----------------------|----------|-----------------------------|-----------------------------|
  79. | Agent-S2.5 | 42.1 | 30.0 | 10.0 |
  80. | Claude-4-Sonnet | 36.1 | 27.4 | 10.5 |
  81. | Qwen3-VL | 32.8 | 21.5 | 10.0 |
  82. | Seed1.5-VL | 30.7 | 21.0 | 10.1 |
  83. | OpenAI o3 | 17.6 | 11.6 | 11.9 |
  84. | Gemini-2.5-Pro | 17.4 | 12.2 | 11.6 |
  85. | Qwen2.5-VL | 14.5 | 10.1 | 14.0 |
  86. **Max Steps: 50**
  87. | Model / Agent | Acc | TIR | ACS |
  88. |----------------------|----------|-------|-------|
  89. | Agent-S2.5 | 49.5 | 35.3 | 17.0 |
  90. | Claude-4-Sonnet | 45.0 | 33.3 | 20.0 |
  91. | Qwen3-VL | 39.5 | 26.1 | 18.6 |
  92. | Seed1.5-VL | 38.2 | 25.1 | 22.3 |
  93. | Gemini-2.5-Pro | 25.7 | 16.8 | 31.0 |
  94. | OpenAI o3 | 24.1 | 16.0 | 33.0 |
  95. | Qwen2.5-VL | 15.6 | 9.3 | 39.0 |
  96. ---
  97. ## 📚 Citation
  98. ```bibtex
  99. @article{jia2025osworldmcp,
  100. title={OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents},
  101. author={Jia, Hongrui and Liao, Jitong and Zhang, Xi and Xu, Haiyang and Xie, Tianbao and Jiang, Chaoya and Yan, Ming and Liu, Si and Ye, Wei and Huang, Fei},
  102. year={2025},
  103. journal={arXiv preprint arXiv:2510.24563}
  104. }
  105. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.