Skip to content
Back to search

agent-desktop

Agent Desktop gives any agent reliable computer use on the desktop. Built with Rust, it sees any app's real UI structure through OS accessibility trees and operates it — refs stay stable and actions stay safe to retry, instead of guessing from pixels.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/vercel-labs/agent-desktop with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/vercel-labs/agent-desktop'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/lahfir--agent-desktop/badge.svg)](/listings/lahfir--agent-desktop)

HTML

<a href="/listings/lahfir--agent-desktop"><img src="/api/listings/lahfir--agent-desktop/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

npm (recommended)

npm install -g agent-desktop        # downloads prebuilt binary automatically

Or without installing:

npx agent-desktop snapshot --app Finder -i

From source

git clone https://github.com/lahfir/agent-desktop
cd agent-desktop
cargo build --release
cp target/release/agent-desktop /usr/local/bin/

Requires Rust 1.89+ and macOS 13.0+.

On macOS, application names use AppKit's localized display name. If it is unavailable, the inventory uses the executable name, then the bundle identifier, while retaining the application's process identity.

Permissions

macOS requires Accessibility permission. Screenshots also require Screen Recording permission, and the Notification Center opener requires Automation permission for System Events. Plain permission checks never prompt. Request missing permissions in a bounded isolated helper with:

agent-desktop permissions --request   # request missing permissions in an isolated helper

Permission fields are explicit objects, for example:

{
  "accessibility": { "state": "granted" },
  "screen_recording": { "state": "denied", "suggestion": "Grant Screen Recording permission" },
  "automation": { "state": "unknown" }
}

Automation reports `granted`, `denied`, or `unknown`; `unknown` means macOS would need to prompt or System Events could not be probed without prompting.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 190 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # agent-desktop
  2. Computer use for AI agents: see and operate desktop apps on macOS, Windows, and Linux through the OS accessibility tree.
  3. <p>
  4. <a href="https://vercel.com/labs#labs-products"><img alt="Vercel Labs Product" src="https://img.shields.io/badge/LABS-PRODUCT-0a0a0a.svg?style=for-the-badge&amp;logo=Vercel&amp;labelColor=000000" height="28"></a>
  5. <a href="https://www.npmjs.com/package/agent-desktop"><img alt="npm version: agent-desktop" src="https://img.shields.io/npm/v/agent-desktop.svg?style=for-the-badge&amp;labelColor=000000" height="28"></a>
  6. <a href="https://github.com/vercel-labs/agent-desktop/blob/main/LICENSE"><img alt="License: Apache-2.0" src="https://img.shields.io/github/license/vercel-labs/agent-desktop.svg?style=for-the-badge&amp;labelColor=000000" height="28"></a>
  7. <a href="https://www.npmjs.com/package/agent-desktop"><img alt="npm downloads per month: agent-desktop" src="https://img.shields.io/npm/dm/agent-desktop.svg?style=for-the-badge&amp;labelColor=000000&amp;label=npm%20downloads" height="28"></a>
  8. </p>
  9. agent-desktop gives an AI agent computer use without screenshots or pixel matching. It reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element the agent can act on. The agent clicks or types on a ref, reads the UI again to check the result, and repeats. The loop lives in your agent; agent-desktop has no model inside. It ships as a CLI and a C library.
  10. Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. Keep them out of public logs and uploads.
  11. ## Install
  12. ```bash
  13. npm install -g agent-desktop
  14. agent-desktop status # platform, permissions, supported surfaces
  15. agent-desktop permissions --guide # macOS: walks you through the Accessibility grant
  16. ```
  17. Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64.
  18. For direct execution without a JavaScript launcher, download the matching native archive from
  19. [GitHub Releases](https://github.com/vercel-labs/agent-desktop/releases) and keep its helper beside the executable.
  20. On macOS and Linux, Homebrew also works: `brew install agent-desktop`. There is no official Scoop package yet.
  21. The repository no longer maintains a custom Homebrew tap or Scoop bucket.
  22. Release archives and building from source are covered in [Installation](https://agent-browser.dev/desktop/installation).
  23. ## Quick start
  24. ```bash
  25. agent-desktop snapshot --app Finder -i # interactive elements, each with a ref
  26. agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot
  27. agent-desktop snapshot --app Finder -i # observe again to verify
  28. ```
  29. Each command prints one JSON envelope. A trimmed snapshot:
  30. ```json
  31. {
  32. "version": "2.6",
  33. "ok": true,
  34. "command": "snapshot",
  35. "data": {
  36. "app": "Finder",
  37. "snapshot_id": "s8f3k2p9",
  38. "ref_count": 59,
  39. "complete": true,
  40. "tree": {
  41. "role": "window",
  42. "name": "Downloads",
  43. "children": [
  44. { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
  45. ]
  46. }
  47. }
  48. }
  49. ```
  50. - **Refs belong to their snapshot.** A bare `@e8` is rejected with `INVALID_ARGS`. When the UI changes, an action
  51. returns `STALE_REF` or `AMBIGUOUS_TARGET`: take a new snapshot and use its refs.
  52. - **Several windows** give `AMBIGUOUS_TARGET`; `snapshot` lists the candidate window ids in `error.details`. Pass one
  53. with `--window-id`.
  54. - **Dense apps:** map first with `snapshot --skeleton --app Slack -i --compact`, then drill into one region with
  55. `snapshot --root @s8f3k2p9:e3 -i --compact`.
  56. - **Headless by default.** Ref actions use the accessibility API without moving the pointer or taking focus. The global
  57. `--headed` flag allows physical input.
  58. - **Post-action waits** observe the ref's original process. Window-owned refs stay pinned to their original window,
  59. including refs captured from a sheet, alert, popover, focused view, or Linux menu. macOS application menus keep
  60. their application-owned surface and need no document window. The wait reads the whole pinned window, so overlay
  61. selectors share its scope. [Wait scope and results](https://agent-browser.dev/desktop/configuration#post-action-wait-flags)
  62. describe the selector limits.
  63. - `type`, `set-value`, and `clipboard-set` accept `--stdin` for piped text, keeping it out of process arguments. Run secret steps as separate commands; batch JSON text appears in process arguments.
  64. - **PowerShell.** Quote every ref (`'@s8f3k2p9:e8'`), because a bare `@token` is the splatting operator.
  65. The full walkthrough is in the [quick start](https://agent-browser.dev/desktop/quick-start).
  66. ## How it works
  67. <picture>
  68. <source media="(prefers-color-scheme: dark)" srcset="assets/architecture-dark.png">
  69. <img alt="The calling agent runs the agent-desktop CLI or the C library. Both go through agent-desktop-core and the PlatformAdapter trait to one platform crate, chosen at compile time, which reads the target app through the AX API on macOS, UI Automation on Windows, or AT-SPI on Linux." src="assets/architecture-light.png" width="720">
  70. </picture>
  71. Your agent calls the CLI, or the C library in process. Both run the same core, which reaches the OS only through `PlatformAdapter`. Each binary is built for one OS and talks to that OS's accessibility API.
  72. ## For AI agents
  73. The binary carries guides that match its version:
  74. ```bash
  75. agent-desktop skills get desktop # the loop, refs, verification and retry rules
  76. agent-desktop skills get platform # what differs on the OS you run on
  77. agent-desktop <command> --help # flags for one command
  78. ```
  79. Install the skill into an agent harness from [skills.sh](https://skills.sh/vercel-labs/agent-desktop/agent-desktop):
  80. ```bash
  81. npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop
  82. ```
  83. Or add this to your `AGENTS.md`:
  84. ```markdown
  85. Use `agent-desktop` to see and operate desktop apps. Before the first use, run
  86. `agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.
  87. ```
  88. Retry a failed command only when `error.disposition.retry` is `safe`. See [JSON output](https://agent-browser.dev/desktop/json-output) for the envelope, error codes, and exit codes.
  89. ### MCP
  90. `agent-desktop mcp` serves every command as an MCP tool over stdio. A tool is named after its command (`desktop_snapshot`, `desktop_click`, `desktop_session_start`), takes that command's arguments, and returns its JSON envelope:
  91. ```json
  92. { "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } }
  93. ```
  94. Global flags given to `mcp`, such as `--session`, `--headed` or `--trace`, apply to every call except `sandbox exec`, whose guest command takes its own flags.
  95. ## Commands
  96. agent-desktop has 61 command names, 57 operational commands. The other four (`key-down`, `key-up`, `mouse-down`, `mouse-up`) refuse to run until a daemon can release a held key or button. [Commands](https://agent-browser.dev/desktop/commands) documents each one.
  97. | Group | Commands |
  98. |---|---|
  99. | Observe | `snapshot`, `find`, `get`, `is`, `screenshot`, `list-surfaces` |
  100. | Act on a ref | `click`, `double-click`, `triple-click`, `right-click`, `type`, `set-value`, `clear`, `focus`, `select`, `toggle`, `check`, `uncheck`, `expand`, `collapse`, `scroll`, `scroll-to` |
  101. | Keyboard and mouse | `press`, `hover`, `drag`, `mouse-move`, `mouse-click`, `mouse-wheel`, `key-down`, `key-up`, `mouse-down`, `mouse-up` |
  102. | Apps and windows | `launch`, `close-app`, `list-apps`, `list-windows`, `list-displays`, `focus-window`, `resize-window`, `move-window`, `minimize`, `maximize`, `restore`, `open-system-surface` |
  103. | Notifications | `list-notifications`, `dismiss-notification`, `dismiss-all-notifications`, `notification-action` |
  104. | Clipboard | `clipboard-get`, `clipboard-set`, `clipboard-clear` |
  105. | Wait | `wait` |
  106. | Sessions | `session`, `trace`, `cursor-overlay` |
  107. | Sandboxes | `sandbox` |
  108. | System | `status`, `permissions`, `version`, `batch`, `skills` |
  109. ## Platform support
  110. | Platform | Requirement | Accessibility API |
  111. |---|---|---|
  112. | [macOS](https://agent-browser.dev/desktop/platforms/macos) | macOS 13 or later, Intel or Apple Silicon | AX API |
  113. | [Windows](https://agent-browser.dev/desktop/platforms/windows) | Windows 10 1809 or later, x64 or ARM64 | UI Automation |
  114. | [Linux](https://agent-browser.dev/desktop/platforms/linux) | Ubuntu 24.04 or later, x86_64 or aarch64, X11 or Wayland | AT-SPI |
  115. Each platform page lists what works headless, what needs `--headed`, and what refuses.
  116. ## C library
  117. Each release ships `libagent_desktop_ffi`, so a program can call agent-desktop in process. See the [C library docs](https://agent-browser.dev/desktop/ffi).
  118. ```python
  119. import ctypes
  120. lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
  121. lib.ad_adapter_create.restype = ctypes.c_void_p
  122. lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
  123. assert lib.ad_init(7) == 0 # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library
  124. adapter = lib.ad_adapter_create()
  125. lib.ad_adapter_destroy(adapter)
  126. ```
  127. ## Development
  128. Each platform adapter builds only on its own OS, so scope cargo to the host's packages. On macOS:
  129. ```bash
  130. HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi)
  131. cargo test "${HOST_PKGS[@]}"
  132. cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings
  133. ```
  134. On Windows, use `agent-desktop-windows` instead of `agent-desktop-macos`; on Linux, `agent-desktop-linux`. [CONTRIBUTING.md](CONTRIBUTING.md) lists every gate.
  135. ## Documentation
  136. - [Introduction](https://agent-browser.dev/desktop): what it is and how it works
  137. - [Snapshots and refs](https://agent-browser.dev/desktop/snapshots-and-refs)
  138. - [Errors and recovery](https://agent-browser.dev/desktop/errors-and-recovery)
  139. - [Skills](https://agent-browser.dev/desktop/skills)
  140. - [CONCEPTS.md](CONCEPTS.md): vocabulary
  141. - [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md), [SUPPORT.md](SUPPORT.md)
  142. ## License
  143. Apache-2.0

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.