Skip to content
Back to search

handy

MarketplaceWorks with Claude CodeGitHub

See and drive the real desktop: native mouse, keyboard, window, clipboard and coordinate-mapped screen control for LLMs and agents over MCP — Set-of-Mark targets, chained actions, permission toggle and a physical (0,0) failsafe.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/miisodev--handy/badge.svg)](/listings/miisodev--handy)

HTML

<a href="/listings/miisodev--handy"><img src="/api/listings/miisodev--handy/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires **Node.js 20+**.

1. One-Command Auto-Setup (Claude Desktop, Cursor, Windsurf, Claude Code)

npx -y @miisodev/handy install

Preview changes first with `--dry-run`, target a single host (`handy install cursor`), or remove cleanly (`handy uninstall`). Backups (`*.handy.bak`) are created automatically before modifying existing config files.

2. Prepackaged Plugin (Claude Code / Agent Marketplaces)

Installs the stdio MCP server, the `handy` operational skill (`skills/handy/SKILL.md`), and the `/handy:on`, `/handy:off`, `/handy:toggle`, `/handy:status`, `/handy:doctor`, and `/handy:windows` commands together:

claude plugin marketplace add miisodev/handy
claude plugin install handy

3. Global CLI + Live Control Panel

npm install -g @miisodev/handy
handy doctor                # verify input & capture readiness for your platform
handy ui                    # live Control Panel at http://127.0.0.1:7788

4. Manual Stdio MCP Configuration

{
  "mcpServers": {
    "handy": {
      "command": "npx",
      "args": ["-y", "@miisodev/handy", "mcp"]
    }
  }
}

Linux (Wayland) setup

Capture works out of the box on GNOME and any compositor whose portal serves `Screenshot`. Input needs a running `ydotoold` (uinput), which most distros ship:

sudo apt install ydotool            # or your distro's package
sudo systemctl enable --now ydotoold

If your distro only ships a *user-level* unit (some recent Ubuntu releases do), it cannot open `/dev/uinput`; run the daemon as a root system service with a user-owned socket instead:

sudo systemctl edit --force --full ydotoold   # then set:
# [Service]
# ExecStart=/usr/bin/ydotoold -p /run/user/1000/.ydotool_socket -P 0660 -o 1000:1000
# RuntimeDirectory=ydotoold

`handy doctor` tells you exactly which routes and tools it found — session type, capture routes, input tool.

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 23 lines
  1. {
  2. "name": "handy",
  3. "owner": {
  4. "name": "miisodev"
  5. },
  6. "metadata": {
  7. "description": "Real-time native mouse, keyboard, window, clipboard, and coordinate-mapped screen control MCP plugin for LLMs.",
  8. "version": "1.4.0"
  9. },
  10. "plugins": [
  11. {
  12. "name": "handy",
  13. "description": "Control mouse, keyboard, clipboard, and windows natively in real-time with chained actions, view the live screen or single windows with mapped (x, y) coordinates and frame diffing, and toggle permissions on/off.",
  14. "source": "./",
  15. "strict": false,
  16. "version": "1.4.0",
  17. "license": "MIT",
  18. "skills": [
  19. "./skills/handy"
  20. ]
  21. }
  22. ]
  23. }
README.md · README · 307 lines
  1. <div align="center">
  2. <img src="./public/handy.png" alt="Handy logo" width="128" />
  3. # handy
  4. **Give your agent eyes and hands — see and drive the real desktop.**
  5. Real-time native mouse, keyboard, window, clipboard and coordinate-mapped screen control for LLMs and agents, served over the [Model Context Protocol](https://modelcontextprotocol.io) as a stdio server and CLI — with numbered Set-of-Mark targets, chained multi-step actions, an instant on/off permission toggle and a physical `(0, 0)` corner-slam failsafe.
  6. [**Site**](https://miiso.dev/handy) · [Quick start](#quick-start) · [The tool surface](#the-tool-surface) · [How it works](#how-it-works) · [Privacy](PRIVACY.md)
  7. [![Version](https://img.shields.io/badge/version-1.4.0-0ea5e9?style=flat-square)](https://github.com/miisodev/handy/releases)
  8. [![CI](https://img.shields.io/github/actions/workflow/status/miisodev/handy/ci.yml?branch=main&style=flat-square&label=CI&color=0ea5e9)](https://github.com/miisodev/handy/actions/workflows/ci.yml)
  9. [![License: MIT](https://img.shields.io/badge/license-MIT-a1a1aa?style=flat-square)](./LICENSE)
  10. [![Platforms](https://img.shields.io/badge/platforms-Windows%20%7C%20macOS%20%7C%20Linux-334155?style=flat-square)](#deployment)
  11. [![MCP](https://img.shields.io/badge/protocol-MCP-0a0a0f?style=flat-square)](https://modelcontextprotocol.io)
  12. [![Sponsor](https://img.shields.io/badge/❤-Sponsor-ea4aaa?style=flat-square&logo=githubsponsors&logoColor=white)](https://github.com/sponsors/miisodev)
  13. </div>
  14. ---
  15. ## What is handy?
  16. Agents get screenshots, but few get honest control of the machine they are looking at. Coordinates are guessed from pixels, one-shot clicks land where nothing happened, and the human watching has no switch to stop it all.
  17. **Handy is an MCP server (with a full CLI alongside) that gives a model real-time native control of a desktop** — mouse, keyboard, clipboard and windows — paired with a live coordinate-mapped view of the screen: dual-contrast `(x, y)` rulers, active-window telemetry, pixel hex under the cursor, sub-2ms frame-to-frame diffs, and **Set-of-Mark** detection that numbers every interactive element on screen so the agent clicks `mark 13` instead of guessing `(631, 447)`. Up to 100 steps — focus a window, click, paste, wait for the UI to settle — run in **one atomic chained call**, with an after-shot that shows numbered trail markers where every action landed.
  18. It installs cleanly on any PC with zero C++ addon compilation (`node-gyp` free), speaks plain stdio to any MCP client, and treats permission as a product surface: a `~/.handy/permissions.json` toggle checked before every capture and every chain step, a `--view-only` mode, `--ttl` timers, a live loopback Control Panel, and a physical failsafe — slam your mouse into the top-left `(0, 0)` corner and any in-flight chain aborts instantly.
  19. ### Who it's for
  20. Developers and agent builders who want their assistant to actually operate the desktop — fill forms, drive native apps, verify what happened — rather than describe it. Any MCP-capable client works (Claude Desktop, Claude Code, Cursor, Windsurf, and everything else speaking MCP); the CLI makes the same surface usable from shell scripts and CI.
  21. ### From experiment to open source
  22. Handy began as the input layer for the author's own agents, then grew through daily real-machine use — real installs, real Wayland compositors, real failsafes — into a tool that behaves the same on Windows, macOS and Linux. What you're reading is the same code that runs the author's desktop every day, not a demo.
  23. ### The core principle
  24. > **An agent that can see and click must be interruptible.**
  25. Handy checks its permission store synchronously before every capture and before every individual step of a chain, and the failsafe reads the *real* cursor position each step, so a human can always stop the machine — by toggle, by corner-slam, or by timer. When the toggle is OFF, no MCP client can turn it back on by itself; `handy_permission({ "action": "off" })` remains a one-way stop switch.
  26. ### Design highlights
  27. - **Set-of-Mark vision (`marks: true`)** — `handy_see` detects, classifies (`control` / `text` / `icon` / `region`), numbers and outlines the interactive elements on screen in reading order; every pointer step then accepts `{ "mark": N }` and the chain DSL understands `click mark 5` / `click #5`. Marks resolve to the element's own centre at execution time, so a chain survives layout shift and HiDPI scaling. Detection is DPI-independent (~18ms at 1280x800, ~22ms at 2560x1440, ~49ms at 4K) because the work buffer is area-capped rather than dimension-capped.
  28. - **One atomic call for a whole interaction (`handy_act`)** — up to 100 sequential steps (`focus_window`, `move`, `click`, `drag`, `scroll`, `paste`, `type`, `key`, `mouse_down`/`up`, `key_down`/`up`, `wait`, `wait_for_change`, `wait_for_settled`, `wait_for_color`, `screenshot`) in a single call, with `screen`/`normalized` (0..1000)/`window` coordinate spaces and numbered amber trail markers on `screenshotAfter`.
  29. - **Human-like pointer motion (`motion: "human"`)** — Bézier arcs with Fitts's-law timing instead of teleporting straight lines, with the duration derived automatically from travel distance and target width when omitted.
  30. - **Post-action verification (`verifyChange: true`)** — a baseline capture before the chain and a frame diff after, so the receipt says whether the click actually did anything.
  31. - **Captures that survive real screens** — dual-contrast rulers legible on `#000` and `#fff` apps alike, area-average box-filter downscaling that keeps 1px text alive, and per-row PNG filter selection for ~26% smaller encodes.
  32. - **Fail-soft enhancements** — mark detection degrades to a plain screenshot (`marksError`) instead of failing `handy_see`, and stale marks fail loudly with the available indices instead of clicking somewhere arbitrary.
  33. - **Strict chain-DSL validation** — malformed pointer arguments throw with the offending segment quoted; out-of-range marks list what exists. Nothing degrades to "act wherever the cursor happens to be".
  34. ---
  35. ## How it works
  36. Handy is one TypeScript process with zero runtime dependencies beyond the MCP SDK and Zod. Each platform speaks its own native bridge — no wrappers, no VMs:
  37. | Platform | Backend | How |
  38. | --- | --- | --- |
  39. | **Windows 10/11** | `win32` | Persistent Win32 `SendInput` (`KEYEVENTF_UNICODE`, `MOUSEEVENTF_VIRTUALDESK`), `EnumWindows`/`SetForegroundWindow`, MTA-safe `OpenClipboard`, GDI `CopyFromScreen`, Per-Monitor V2 DPI awareness |
  40. | **macOS** | `darwin` | Persistent CoreGraphics `CGEvent` + `NSWorkspace` + `NSPasteboard`, in-memory `CGWindowListCreateImage` PNG capture (with `screencapture` fallback), Retina 2x logical-point mapping |
  41. | **Linux — X11** | `linux-x11` | Pure native X11 wire-protocol client (`XTEST`, `GetImage`, `BIG-REQUESTS`, `RANDR`, EWMH `_NET_CLIENT_LIST`/`_NET_ACTIVE_WINDOW`) — zero external packages |
  42. | **Linux — Wayland** | `linux-cli` | Auto-selected on Wayland sessions: screen capture through the freedesktop **screenshot portal** (works out of the box on GNOME, whose Shell screenshot API rejects third-party callers) or `grim` on wlroots; input via `ydotool` (uinput daemon) or `wtype`/`xdotool`; clipboard via `wl-clipboard`/`xclip`/`xsel` |
  43. | **Headless / CI** | `virtual` | Built-in 1280x800 multi-window interactive desktop workbench for safe agent sandboxing — no display required |
  44. Everything above the bridge is shared: `controller.ts` enforces permissions and the failsafe between steps, `marks.ts` detects and numbers UI elements, `overlay.ts` renders rulers and reticles, `png.ts` handles decode/encode, and the MCP layer (`mcp.ts`) wraps it all as six tools.
  45. A capture-and-act round trip looks like: the agent calls `handy_see { marks: true }` → the backend grabs a frame (portal on GNOME Wayland, native protocol on X11/Windows/macOS) → detection numbers the elements → the agent calls `handy_act` with a chain referencing `mark` indices → the controller re-checks permissions and the failsafe before each step → a receipt (and optionally a diff and after-shot) comes back.
  46. ---
  47. ## Quick start
  48. Requires **Node.js 20+**.
  49. ### 1. One-Command Auto-Setup (Claude Desktop, Cursor, Windsurf, Claude Code)
  50. ```bash
  51. npx -y @miisodev/handy install
  52. ```
  53. Preview changes first with `--dry-run`, target a single host (`handy install cursor`), or remove cleanly (`handy uninstall`). Backups (`*.handy.bak`) are created automatically before modifying existing config files.
  54. ### 2. Prepackaged Plugin (Claude Code / Agent Marketplaces)
  55. Installs the stdio MCP server, the `handy` operational skill (`skills/handy/SKILL.md`), and the `/handy:on`, `/handy:off`, `/handy:toggle`, `/handy:status`, `/handy:doctor`, and `/handy:windows` commands together:
  56. ```bash
  57. claude plugin marketplace add miisodev/handy
  58. claude plugin install handy
  59. ```
  60. ### 3. Global CLI + Live Control Panel
  61. ```bash
  62. npm install -g @miisodev/handy
  63. handy doctor # verify input & capture readiness for your platform
  64. handy ui # live Control Panel at http://127.0.0.1:7788
  65. ```
  66. ### 4. Manual Stdio MCP Configuration
  67. ```json
  68. {
  69. "mcpServers": {
  70. "handy": {
  71. "command": "npx",
  72. "args": ["-y", "@miisodev/handy", "mcp"]
  73. }
  74. }
  75. }
  76. ```
  77. ### Linux (Wayland) setup
  78. Capture works out of the box on GNOME and any compositor whose portal serves `Screenshot`. Input needs a running `ydotoold` (uinput), which most distros ship:
  79. ```bash
  80. sudo apt install ydotool # or your distro's package
  81. sudo systemctl enable --now ydotoold
  82. ```
  83. If your distro only ships a *user-level* unit (some recent Ubuntu releases do), it cannot open `/dev/uinput`; run the daemon as a root system service with a user-owned socket instead:
  84. ```bash
  85. sudo systemctl edit --force --full ydotoold # then set:
  86. # [Service]
  87. # ExecStart=/usr/bin/ydotoold -p /run/user/1000/.ydotool_socket -P 0660 -o 1000:1000
  88. # RuntimeDirectory=ydotoold
  89. ```
  90. `handy doctor` tells you exactly which routes and tools it found — session type, capture routes, input tool.
  91. ---
  92. ## The tool surface
  93. All returns are structured JSON. Input mistakes come back as informational errors the model can self-correct from; every receipt names what was executed, where, and how long it took.
  94. | Tool | Description |
  95. | --- | --- |
  96. | `handy_see` | Capture the live screen, a single `window`, or a zoomed `region` with dual-contrast `(x, y)` rulers, cursor reticle, `pixelUnderMouse`, `activeWindow`, `screenDiff`, and `highlightDiff`. With `marks: true` it also detects, numbers, and returns every interactive UI element (`elements`). |
  97. | `handy_act` | Execute single or chained native mouse, keyboard, paste, window-focus, and smart visual-wait (`wait_for_change`, `wait_for_settled`, `wait_for_color`) controls with `screen`/`normalized`/`window` coordinates, `mark` element references, `motion` profiles, numbered trail markers, and optional `verifyChange` post-conditions. |
  98. | `handy_windows` | List visible desktop windows (`title`, `app`, `bounds`, `focused`) or focus a window by `title`, `app`, or `id`. |
  99. | `handy_clipboard` | Read or write plain-text system clipboard directly. |
  100. | `handy_status` | Inspect live permission toggle state (`ON`/`OFF`), mode (`full`/`view_only`), OS backend, displays, active window, and cursor coordinates. |
  101. | `handy_permission` | Query permission status or immediately turn Handy `off`. |
  102. ### Example: See numbered elements, then act on them
  103. ```json
  104. // handy_see
  105. { "marks": true, "maxMarks": 30 }
  106. ```
  107. ```json
  108. // handy_act — marks 5 and 13 come from the capture above
  109. {
  110. "actions": [
  111. { "type": "click", "mark": 5 },
  112. { "type": "paste", "text": "Sam — ready for review 🚀" },
  113. { "type": "click", "mark": 13 },
  114. { "type": "wait_for_settled", "quietMs": 200 }
  115. ],
  116. "verifyChange": true,
  117. "screenshotAfter": true
  118. }
  119. ```
  120. `handy_see` returns each element's `index`, `kind`, bounds, `centerX`/`centerY`, and a `score`; marks stay valid in the server process until the next marked capture. The CLI re-detects on demand per invocation (detection is deterministic for an unchanged screen), and one chain can do both atomically:
  121. ```bash
  122. handy see --marks --out marked.png
  123. handy act 'click mark 5 -> paste "Sam" -> click mark 13 -> wait_for_settled' --screenshot-after after.png
  124. handy act '[{"type":"screenshot","marks":true},{"type":"click","mark":5}]'
  125. ```
  126. ### Standalone CLI
  127. ```bash
  128. handy see --out screen.png # coordinate-mapped capture
  129. handy see --window "Scratchpad" --highlight-diff --out shot.png
  130. handy see --marks --max-marks 20 --json # numbered UI elements
  131. handy windows && handy focus "Visual Studio Code"
  132. handy clipboard get | handy clipboard set "Copied from terminal"
  133. handy act 'focus "Scratchpad" -> click 800 420 -> paste "Hello 🌍" -> wait_for_settled'
  134. handy key ctrl+s # chords work on every platform bridge
  135. ```
  136. ---
  137. ## Deployment
  138. ### Permission controls & human failsafe
  139. Handy stores its permission toggle at `~/.handy/permissions.json` (`0600`) and checks it synchronously before every `handy_see` capture, `handy_windows`/`handy_clipboard` call, and before **every individual step** of a `handy_act` chain:
  140. ```bash
  141. handy off # immediately block all LLM screen viewing & input control
  142. handy on # full screen vision & mouse/keyboard control
  143. handy toggle # flip between ON and OFF
  144. handy on --view-only # allow screen viewing, block mouse & keyboard
  145. handy on --ttl 30m # auto-disable after 30 minutes
  146. handy status # toggle state, backend, windows & displays
  147. ```
  148. - **Physical failsafe**: slamming your mouse into the top-left `(0, 0)` corner aborts any in-flight chain and releases held keys/buttons immediately — the controller reads the *real* cursor position each step, so it works even when the agent commands a different pointer path.
  149. - **Prompt-injection guard**: when turned OFF by the user, an MCP client **cannot** re-enable Handy on its own; `handy_permission({ "action": "off" })` is always permitted as a one-way stop switch.
  150. ### Backend selection & environment
  151. The backend is picked automatically per platform; `HANDY_BACKEND` (`auto`, `virtual`, `linux-x11`, `linux-cli`/`wayland`, `darwin`, `win32`) overrides it. On a Wayland session the CLI backend is chosen even when `DISPLAY` is set (XWayland), because that is the only backend that can see the real desktop; on a pure X11 session the native X11 client takes over with full window listing and focus support. `HANDY_ENABLED=false` in the host environment is a second, independent kill switch.
  152. Known platform limits, stated plainly: `handy_windows`/`focus_window` enumerate windows through the X11/Wayland-of-record APIs (EWMH, `wmctrl`) and see XWayland windows only on GNOME Wayland — native windows there are not publicly enumerable by any client-side API. Capture and input are unaffected.
  153. ---
  154. ## Adapting to your environment
  155. A handful of values reflect the author's machine rather than yours. None are secrets:
  156. | What | Where | Make it yours |
  157. | --- | --- | --- |
  158. | **Linux input daemon** | `ydotool` + `ydotoold` (see [Linux (Wayland) setup](#linux-wayland-setup)) | Distros differ in how the daemon ships; `handy doctor` reports what it found |
  159. | **Capture tools (non-GNOME Wayland)** | `grim` (wlroots) / `maim` / `scrot` | Optional — the portal route needs nothing installed |
  160. | **Display override** | `HANDY_BACKEND` in the MCP host env | Only when auto-detection needs help |
  161. | **Permission store** | `~/.handy/permissions.json` | Created on first run; `0600` |
  162. | **Author · repo · funding** | `package.json`, the badges above | Your own details if you fork; the funding links support the original author |
  163. ---
  164. ## Architecture
  165. ```
  166. src/
  167. ├── index.ts public API barrel
  168. ├── mcp.ts six MCP tools over stdio + resources + prompts
  169. ├── controller.ts permission + failsafe enforcement between steps, chains, receipts
  170. ├── cli.ts the `handy` command surface
  171. ├── installer.ts host auto-setup (`handy install`, backups, uninstall)
  172. ├── marks.ts Set-of-Mark detection, DPI-independent, fail-soft
  173. ├── overlay.ts dual-contrast rulers, reticles, trail markers
  174. ├── png.ts decode/encode, box-filter resize, per-row filters
  175. ├── diff.ts frame diffing with early exit
  176. ├── permissions.ts ~/.handy/permissions.json store, TTLs, view-only
  177. ├── keys.ts per-platform key maps (X11 keysyms, Darwin, Win32 VK, Linux EV_KEY)
  178. ├── ui.ts loopback Control Panel (SSE)
  179. └── backends/
  180. ├── win32.ts SendInput, EnumWindows, GDI capture
  181. ├── darwin.ts CGEvent, NSWorkspace, CGWindowList
  182. ├── linux-x11.ts native X11 wire protocol client (XTEST, GetImage, RANDR, EWMH)
  183. ├── linux-cli.ts Wayland: portal capture + ydotool/wtype/xdotool/wl-clipboard
  184. └── virtual.ts headless 1280x800 workbench for CI
  185. ```
  186. Key properties: no native build steps (pure wire-protocol clients), permission checks synchronously before every effect, the failsafe reads the real cursor each step, mark references fail loudly when stale, and every route that can fail falls through with its error named.
  187. ## Development
  188. ```bash
  189. npm run typecheck # src + tests, strict
  190. npm run build # lib/, bin/handy.js, public/handy.png
  191. npm test # vitest across every backend bridge and the stdio MCP surface
  192. npm run validate # version-site agreement, MCPB manifest, plugin commands, skill references
  193. ```
  194. Real input is exercised against real desktop sessions — Windows SendInput, macOS CGEvent, Linux XTEST/uinput — with the `virtual` backend covering headless CI; a mock X11 server tests the binary protocol byte-for-byte.
  195. ## Contributing
  196. Handy is maintained by its author with contributions welcome: open an issue first for anything non-trivial, keep changes focused, match the existing code style, and update the docs a change touches — a tool change without its README/skill update is half a change. `npm run typecheck`, `npm run build` and `npm test` must pass clean; CI runs the same checks on every push and PR, on all three operating systems.
  197. ## Security
  198. Please do not report suspected vulnerabilities in a public issue — Handy is, by design, a tool that can click and type on someone's computer. Follow the private reporting process in [`SECURITY.md`](./SECURITY.md); never include screenshots or captured screen content in a report.
  199. ## Privacy and support
  200. Handy adds no analytics, no telemetry and no hosted account. Screen captures go only to the MCP client that asked for them; permission state lives in `~/.handy/`; the Control Panel binds to loopback only. The full [privacy policy](PRIVACY.md) says what is stored where. For help, open a [GitHub issue](https://github.com/miisodev/handy/issues) or email miisodev@gmail.com.
  201. ---
  202. ## Support the project
  203. Handy is built and maintained by one person. If it's useful to you, sponsorship directly funds its continued development:
  204. <div align="center">
  205. [![Sponsor on GitHub](https://img.shields.io/badge/Sponsor%20on%20GitHub-%E2%9D%A4-ea4aaa?style=for-the-badge&logo=githubsponsors)](https://github.com/sponsors/miisodev)
  206. [![Donate via PayPal](https://img.shields.io/badge/Donate-PayPal-009cde?style=for-the-badge&logo=paypal&logoColor=white)](https://paypal.me/miisodev?locale.x=en_US&country.x=ZA)
  207. </div>
  208. Starring the repo, reporting bugs, and spreading the word help too.
  209. ## License
  210. [MIT](./LICENSE) © [Kevin Miiso Novo](https://github.com/miisodev)
  211. ## Credits
  212. - [Model Context Protocol](https://modelcontextprotocol.io) — the open standard that lets one server drive every MCP-capable client.
  213. - The GNOME, wlroots and freedesktop portal teams — Wayland's capture and input story works because of them.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.