Skip to content
Back to search

computer-use-mcp

High Performance MCP server + client for Windows and macOS computer control built in Rust. Screenshot, mouse, keyboard, clipboard, and app management — all in-process via Rust NAPI.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/zavora-ai--computer-use-mcp/badge.svg)](/listings/zavora-ai--computer-use-mcp)

HTML

<a href="/listings/zavora-ai--computer-use-mcp"><img src="/api/listings/zavora-ai--computer-use-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires **Node.js 20+** and an interactive desktop. The published package bundles native modules for macOS, Windows and Linux, on x64 and arm64.

Add this to your agent's MCP server configuration:

{
  "mcpServers": {
    "computer-use": {
      "command": "npx",
      "args": ["-y", "@zavora-ai/computer-use-mcp"]
    }
  }
}

For a client with a different configuration format, use `npx` as the command and `-y @zavora-ai/computer-use-mcp` as its arguments. See [agent configuration](AGENTS.md) for client-specific examples. No model API key is needed by the MCP server itself.

Before using desktop tools:

| Platform | Setup | |---|---| | macOS | Grant the host Accessibility and Screen Recording access. App scripting may also request Automation access. | | Windows | Run in a signed-in desktop session. Protected or elevated windows may need matching privileges. | | Linux | Use a graphical session with the required X11/Wayland utilities. Accessibility support needs AT-SPI. See [platform details](docs/ARCHITECTURE.md). |

Ask your agent to run `doctor` to check capabilities. Office must be installed and activated separately; its folder-access prompts are normal setup requirements. Downloads is not guaranteed to bypass those prompts.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 254 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Computer Use MCP
  2. Give an AI agent the ability to discover apps, understand their controls, and work
  3. on a desktop. Computer Use MCP connects MCP-compatible agents to native tools on
  4. **macOS, Windows and Linux**.
  5. Use app scripting and accessibility controls when available, and screenshots,
  6. mouse and keyboard input when the interface requires them.
  7. ## See it paint
  8. ![An original night landscape painted by Astra through Computer Use MCP](docs/assets/astra-painting.png)
  9. **Astra painted this through real mouse strokes.** The agent used the OpenAI
  10. Responses API to operate the bundled paint studio through this MCP: 105 strokes,
  11. seven colors and 15 model calls, followed by a verified PNG export. The result is
  12. an original Van Gogh-inspired landscape made through native pointer events.
  13. ```sh
  14. # From a clone of this repository, after the setup below:
  15. npm install --prefix agents/openai-agent
  16. npx playwright install chromium
  17. export OPENAI_API_KEY=your-key
  18. node agents/openai-agent/showcase.mjs paint --studio --turns 24 --tokens 200000
  19. ```
  20. This makes paid API calls. The recorded run used 191,734 input and 3,274 output
  21. tokens. [Run the demos and read their limits](agents/openai-agent/README.md).
  22. ## See it read
  23. ![DeepSeek Flash reading a scanned invoice and typing it into a form](docs/assets/deepseek-ledger-run.gif)
  24. **Thirty-eight seconds, ten fields, no answer key.** The invoice on the left is
  25. drawn on a canvas, so its text never reaches the accessibility tree — the only way
  26. to read it is to look. DeepSeek Flash reads it from screenshots, types each value
  27. through the accessibility API, and presses Verify; the local host scores the
  28. entries and tells it only which ones are wrong. This run scored 10 of 10 on the
  29. first Verify, over 12 model calls and 23 tool calls, at a 94% prompt-cache hit
  30. rate. Every entry is recorded with its `isTrusted` flag, so a submission that was
  31. not typed through the UI is refused.
  32. ```sh
  33. npm install --prefix agents/deepseek-agent
  34. npx playwright install chromium
  35. export DEEPSEEK_API_KEY=your-key
  36. node agents/deepseek-agent/showcase.mjs --seed 7 # the run above
  37. node agents/deepseek-agent/showcase.mjs --receipt random # a real receipt from the web
  38. ```
  39. [Full-resolution capture](docs/assets/deepseek-ledger-run.mp4) ·
  40. [how it works, and what it cannot fake](agents/deepseek-agent/README.md)
  41. ## See it plan
  42. ![The run console: a request with an attached reference on the left, the agent's activity feed below it, and its plan beside a live frame from Blender](docs/assets/blender-console-live.png)
  43. **Watch the work instead of reading tool calls.** The run console is an MCP App the
  44. agent reports into. It declares a plan, marks one task active at a time, and attaches
  45. real screen captures as it goes — the frame above is a live capture of Blender's own
  46. window.
  47. Underneath the conversation is everything it is doing: each thought, each tool call
  48. and each result, with timings. That feed is reported by the host driving the run
  49. rather than by the model, so it is complete — Blender's own calls appear in it too.
  50. You can type while it works. The message lands in the transcript that every run tool
  51. returns, so the agent reads it on its next call and can change course mid-task. Attach
  52. an image and it gets both the picture and the path the file was saved at, which is how
  53. Blender can open your reference as a texture rather than the agent working from its own
  54. description of it. `web_search` and `scrape` let it check a fact it does not know —
  55. both run locally, so they work with any model.
  56. Look at it right now, with no API key:
  57. ```sh
  58. npx -y -p @zavora-ai/computer-use-mcp computer-use-mcp-console
  59. ```
  60. That opens the console and drives a **scripted** walkthrough — the words are written
  61. into the host, but everything it looks at is real, so the frame on the right is a
  62. capture of your actual screen. For a live agent instead, start the host with
  63. `--no-demo` and point one at `http://127.0.0.1:4517/mcp`:
  64. ```sh
  65. computer-use-mcp-console --no-demo
  66. ```
  67. That endpoint serves the desktop tools and the run tools from the same server, so
  68. whatever the agent plans and captures appears in the browser. The screenshot above
  69. is [an ADK-Rust agent on DeepSeek Flash](https://github.com/zavora-ai/adk-rust/tree/main/examples/blender_console)
  70. refining a watch it modelled earlier in the same conversation.
  71. The console is loopback-only and unauthenticated: `/mcp` grants desktop control to
  72. anything that can reach the port. Fine on your own machine, never on a shared host.
  73. ## Set up your agent
  74. Requires **Node.js 20+** and an interactive desktop. npm installs the native module
  75. for your platform (macOS, Windows or Linux, on x64 or arm64) from its own package,
  76. `@zavora-ai/computer-use-mcp-<platform>-<arch>`, chosen by OS and CPU, so an install
  77. downloads one binary. Installing with optional dependencies turned off
  78. (`--omit=optional`) leaves it out, and the server says how to fix that.
  79. Add this to your agent's MCP server configuration:
  80. ```json
  81. {
  82. "mcpServers": {
  83. "computer-use": {
  84. "command": "npx",
  85. "args": ["-y", "@zavora-ai/computer-use-mcp"]
  86. }
  87. }
  88. }
  89. ```
  90. For a client with a different configuration format, use `npx` as the command and
  91. `-y @zavora-ai/computer-use-mcp` as its arguments. See [agent configuration](AGENTS.md)
  92. for Cursor, Claude Desktop, Codex and programmatic examples. No model API key is
  93. needed by the MCP server itself.
  94. Before using desktop tools:
  95. | Platform | Setup |
  96. |---|---|
  97. | macOS 14+ | Grant the host app Accessibility (called Device Control and Data Access from macOS 27) and Screen Recording access. The host is the app that launches the server (your terminal, Cursor or Claude Desktop), not `node`. Input Monitoring lets the user-active guard see when you are typing or moving the mouse. App scripting may also request Automation access. Until Accessibility is granted, input tools refuse with `accessibility_permission_denied` rather than silently doing nothing. The capture helper (fast captures and OCR) and the status card are compiled on first use, so install the Xcode Command Line Tools (`xcode-select --install`). |
  98. | Windows | Run in a signed-in desktop session. Protected or elevated windows may need matching privileges. Coordinates are physical pixels at any display scale. |
  99. | Linux | Use an X11 or Wayland session. On GNOME 46+ and KDE Plasma 6.1+, Wayland input goes through the desktop's RemoteDesktop portal: you approve once and the choice is remembered. Elsewhere on Wayland install `ydotool`. Accessibility needs AT-SPI (`python3-gi`, `gir1.2-atspi-2.0`). See [testing on Linux](docs/TESTING-LINUX.md) and [platform details](docs/ARCHITECTURE.md). |
  100. Ask your agent to run `doctor` to check capabilities. The `computer-use-mcp` command checks
  101. its own install first (dependencies, build, native module) and names the fix on stderr if
  102. something is missing. Office must be installed and activated separately; its folder-access
  103. prompts are normal setup requirements. Downloads is not guaranteed to bypass those prompts.
  104. ## What it can do
  105. | Capability | Examples |
  106. |---|---|
  107. | Discover applications | Find installed and running apps, identify their targets, and choose an automation approach. |
  108. | Read the interface | Capture a window, zoom into a region, inspect accessibility controls and find a button or field; read a window's text by on-device OCR when an app draws its own UI (macOS); wait for a window, for text, or for the screen to settle instead of polling screenshots. |
  109. | Operate applications | Click controls (`click` with button and count), fill forms, select menus, type text, drag paths and use keyboard shortcuts; set a session target once instead of naming the window on every call; run a known sequence in one call (`run_steps`). Every action says whether it was verified, and every refusal names a safe next call. |
  110. | Use app scripting | AppleScript/JXA on macOS, PowerShell on Windows, and bash or PowerShell on Linux. |
  111. | Manage desktop work | Target individual windows, switch focus, read/write the clipboard and inspect displays. |
  112. | Build agent hosts | Add persistent desktop sessions, verified workflows, an MCP App console, isolated browser contexts and supervised runtimes. |
  113. | Stay in control | A desktop status card (on by default on macOS and Windows; `COMPUTER_USE_STATUS_CARD=true` on Linux) shows what the agent is doing with Pause and Stop, and `ctrl+alt+escape` stops it. Moving the mouse or typing pauses the agent until you stop. A locked screen is refused by name, and on macOS and Windows the display stays awake while the agent works. Neither the card nor the agent's pointer appears in its screenshots (on Linux the card is painted out of them, and it reserves a strip of the desktop instead of floating on top). |
  114. The default profile, **`desktop`**, exposes 37 tools on macOS: observation,
  115. coordinates, OCR for apps that draw their own UI, waits and the session target.
  116. Set `COMPUTER_USE_PROFILE=full` for every tool (78 on macOS), or `core`, `ax`,
  117. `scripting` and `windows-admin` for other bounds. Tools that don't exist on your
  118. platform are not listed. Optional host services add their own tools. The server's
  119. instructions to the agent are generated for your platform and profile, so they only
  120. name tools that are there.
  121. Try asking your agent:
  122. - “Find my installed Office apps and explain how you can automate them.”
  123. - “Open a new text document, write a short meeting agenda and save it.”
  124. - “Inspect this application's controls and fill out the form I describe.”
  125. - “Create a new painting with a swirling night sky and export it.”
  126. Always specify the intended app or window. Prefer connectors, APIs or filesystem
  127. operations for work that does not need a desktop. Platform and application
  128. support varies; an available tool is not a promise that every app exposes usable
  129. controls.
  130. ## Use it from code
  131. ```js
  132. import { createComputerUseServer } from '@zavora-ai/computer-use-mcp'
  133. import { connectInProcess } from '@zavora-ai/computer-use-mcp/client'
  134. // The default profile is `desktop`; pass { profile: 'full' } for every tool.
  135. const client = await connectInProcess(createComputerUseServer())
  136. try {
  137. const apps = await client.discoverApplications({
  138. query: 'office',
  139. include_capabilities: true,
  140. })
  141. console.log(apps.content)
  142. } finally {
  143. await client.close()
  144. }
  145. ```
  146. [Application examples](examples/README.md) · [AI agent examples](agents/README.md)
  147. · [Responses showcases](agents/openai-agent/README.md)
  148. ## Architecture
  149. ```mermaid
  150. flowchart TD
  151. Agent[AI agent or MCP client] --> Server[MCP server]
  152. Server --> Policy[Tool permissions and target checks]
  153. Policy --> Session[Desktop session]
  154. Session --> Native[Native mouse, keyboard and screen tools]
  155. Session --> AX[Accessibility controls]
  156. Session --> Scripts[App scripting]
  157. Native --> Apps[Desktop applications]
  158. AX --> Apps
  159. Scripts --> Apps
  160. Server --> Hosts[Optional host services]
  161. Hosts --> Browser[Isolated browser]
  162. Hosts --> Tasks[Persistent tasks and session console]
  163. ```
  164. The TypeScript server handles tool requests, permissions and cancellation. A Rust
  165. native module connects to the operating system. Optional host services provide
  166. longer-lived sessions and isolated execution. Results return to the agent so it
  167. can observe, act and verify.
  168. For protocol details, deployment controls and the full component diagram, see
  169. [Architecture](docs/ARCHITECTURE.md) and the [host integration guide](docs/STRATEGY.md).
  170. Legacy MCP clients remain supported; newer protocol features are opt-in.
  171. ## Develop locally
  172. ```sh
  173. git clone https://github.com/zavora-ai/computer-use-mcp.git
  174. cd computer-use-mcp
  175. npm ci
  176. npm run build:ts
  177. npm test
  178. ```
  179. For live desktop use from a clone, install Rust and build your platform's native
  180. module: `npm run build:native` on macOS, `npm run build:native:win` on Windows x64,
  181. or `npm run build:native:linux` on Linux. Windows arm64 has `build:native:win:arm64`.
  182. The packaged npm install does not require a local Rust build.
  183. `npm run smoke` checks the local desktop; `npm run test:browser` tests a disposable
  184. browser fixture. These live checks need their documented platform prerequisites.
  185. [Development scripts](scripts/README.md) lists the maintained commands.
  186. ## Documentation and support
  187. - [v7.7.0 release notes](docs/releases/v7.7.0.md): honest results, `run_steps`, a person in control by default, faster captures, Linux on Wayland, a native package per platform
  188. - [v7.6.1 release notes](docs/releases/v7.6.1.md): the review fixes and the desktop status card
  189. - [v7.6.0 release notes](docs/releases/v7.6.0.md) (includes 7.5.0: [its notes](docs/releases/v7.5.0.md))
  190. - [v7.4.0 release notes and rollout checklist](docs/releases/v7.4.0.md)
  191. - [Reviews](docs/reviews/): the 2026-10-09 review and design pass behind 7.6.0, the 2026-10-11 three-pass review behind 7.6.1, and the 2026-10-23 competitor study behind 7.7.0
  192. - [v8 roadmap](docs/specs/v8/roadmap.md): what comes next
  193. - [Tool usage and agent setup](AGENTS.md)
  194. - [Testing on Linux](docs/TESTING-LINUX.md) and [in UTM virtual machines](docs/TESTING-IN-UTM.md)
  195. - [Architecture and platform support](docs/ARCHITECTURE.md)
  196. - [Sessions, Tasks, browser isolation and supervision](docs/STRATEGY.md)
  197. - [Token and interaction efficiency](docs/EFFICIENCY.md)
  198. - [Security and reporting vulnerabilities](SECURITY.md)
  199. - [Report a bug or request a feature](https://github.com/zavora-ai/computer-use-mcp/issues)
  200. MIT licensed. Desktop access can change real applications and files; configure
  201. permissions and target applications deliberately. The bundled HTTP server is
  202. loopback-only. Exposing a remote endpoint requires host-owned authentication.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.