Skip to content
Back to search

mouse-driver-mcp

Human-shaped X11 pointer control for agents: armed sessions, panic/pause chords, a browser-anchored HUD and a local telemetry API. MCP server.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/crystalrockmarket--mouse-driver-mcp/badge.svg)](/listings/crystalrockmarket--mouse-driver-mcp)

HTML

<a href="/listings/crystalrockmarket--mouse-driver-mcp"><img src="/api/listings/crystalrockmarket--mouse-driver-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

cd ~/projects/mouse-driver-kituwa
python3 -m venv .venv
.venv/bin/pip install 'mcp==2.3.0' pytest
.venv/bin/python -m pytest -q

The HUD runs on the system Python because it needs GTK3 (`python3-gi`); the MCP server and CLI run in the venv.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 232 lines
  1. # Mouse Driver
  2. Human-shaped control of the real X11 pointer on **the desktop host** (the desktop box),
  3. with a small always-on-top companion HUD and a local telemetry API. Built for
  4. agents that need to operate the actual browser, not a headless replica.
  5. - Two 2560×1440 monitors, virtual screen 5120×1440, `DISPLAY=:0`, i3.
  6. - XTEST via `libX11`/`libXtst` (ctypes) — no `xdotool` in the motion path.
  7. - Opaque GTK3 HUD, no compositor needed.
  8. ## Safety first — the chords
  9. The driver only acts while **armed**, and three global chords let a human take
  10. control back instantly, from any application, without clicking the panel:
  11. | Chord | What it does | When to use it |
  12. | --- | --- | --- |
  13. | `Ctrl Alt Shift Esc` | **Panic.** Stops motion in one frame, releases any held button, disarms and **locks** arming. | Something looks wrong. Nothing can re-arm until you unlock. |
  14. | `Ctrl Alt Shift Enter` | **Unlock.** Clears the lock and leaves the driver disarmed. | Only after you have looked. |
  15. | `Ctrl Alt Shift H` | **Show / hide the HUD.** Works while the panel is hidden. |
  16. | `Ctrl Alt Shift R` | **Re-anchor the HUD.** Returns a dragged panel to auto mode inside the target window. |
  17. | `Ctrl Alt Shift K` | **Overlay keyboard.** Expands or collapses the on-screen keyboard inside the panel. |
  18. | `Ctrl Alt Shift P` | **Pause / resume.** Holds the pointer where it is (releasing any held button) and keeps the session armed with its TTL ticking. A second press resumes the interrupted move from the paused position. | Step in and look without ending the session. |
  19. | `Esc` `Esc` | **Disarm.** No lock, no sentinel. | A small mistake: wrong window, wrong target. |
  20. | `Ctrl Alt Shift P` (while paused, TTL expired) | Resume is refused (`arm expired`); the session stays disarmed. | — |
  21. **The lock is deliberately not an MCP tool.** `mouse_arm` refuses with
  22. `operator lock: press Ctrl+Alt+Shift+Enter to allow arming`, and no tool in the
  23. server can clear it. Physically holding `Ctrl Alt Shift Enter` at the keyboard
  24. is the only way back. The same applies to the pause: agents cannot resume
  25. themselves; the human chord or `POST /api/resume` (loopback, operator use) does.
  26. The panel shows the same four shortcuts as keycaps at all times and highlights
  27. the ones that matter for the current state (`ARMED`, `PAUSED`, `LOCKED`, `IDLE`).
  28. ## Install
  29. ```bash
  30. cd ~/projects/mouse-driver-kituwa
  31. python3 -m venv .venv
  32. .venv/bin/pip install 'mcp==2.3.0' pytest
  33. .venv/bin/python -m pytest -q
  34. ```
  35. The HUD runs on the system Python because it needs GTK3 (`python3-gi`); the MCP
  36. server and CLI run in the venv.
  37. ## Start
  38. ```bash
  39. # MCP server (stdio) — also starts the telemetry API on 127.0.0.1:8340
  40. bin/mouse-driver serve
  41. # HUD (needs the desktop session)
  42. bin/mouse-hud # click-through companion
  43. bin/mouse-hud --interactive # normal clicking, Disarm/STOP buttons visible
  44. # CLI
  45. bin/mouse-driver arm 300
  46. bin/mouse-driver move 1700 900
  47. bin/mouse-driver pause | resume | toggle
  48. bin/mouse-driver stop | panic | lock | unlock
  49. bin/mouse-driver status
  50. ```
  51. `deploy/mouse-driver-hud.service` is a **user** systemd unit draft (not
  52. installed): `Environment=DISPLAY=:0`, `XAUTHORITY=$HOME/.Xauthority`,
  53. `WantedBy=graphical-session.target`, `Restart=always`.
  54. ## MCP configuration (opencode)
  55. ```jsonc
  56. {
  57. "mcp": {
  58. "mouse-driver": {
  59. "type": "local",
  60. "command": [
  61. "$HOME/mouse-driver/.venv/bin/python",
  62. "-m",
  63. "mouse_driver.mcp_server"
  64. ],
  65. "environment": {
  66. "DISPLAY": ":0",
  67. "XAUTHORITY": "$HOME/.Xauthority"
  68. },
  69. "enabled": true
  70. }
  71. }
  72. }
  73. ```
  74. `DISPLAY` defaults to `:0` inside the process when unset (and `XAUTHORITY` to
  75. `~/.Xauthority` when that file exists), so the handshake succeeds even from a
  76. process that did not inherit the desktop environment. If X is genuinely
  77. unreachable the server still completes `initialize` and `tools/list`;
  78. `mouse_status` then reports `driver: "unavailable"` and every acting tool
  79. returns a JSON refusal.
  80. ## HUD
  81. The panel is a companion to the **target window**, not a screen-corner widget:
  82. - Anchored inside the top-right of the target's client area, 12 px in,
  83. clamped so it never leaves the window.
  84. - Width `min(430, max(320, 0.42 × window width))`, height ~200 px.
  85. - Re-reads the target geometry at each 8 Hz poll, so it follows moves, resizes,
  86. maximise/unmaximise and workspace changes.
  87. - Click-through: an empty X input shape, so it never blocks the page.
  88. - Override-redirect and keep-above: visible over a fullscreen target, never
  89. focused, and re-raised every 1.5 s only while the target is the active window.
  90. - Visible whenever `hud.visible` is true and the target has a mapped rectangle
  91. that intersects any monitor. Desktop index is ignored: with i3's per-output
  92. desktops two desktops can be physically visible at once.
  93. - Hidden when the target closes, minimises or its rectangle leaves every
  94. monitor; with no target at all it falls back to the bottom-right of the
  95. primary monitor and says “no target”.
  96. - A plain click on the header never moves the panel: the drag only starts after
  97. 3 px of pointer travel.
  98. - The **X** at the top-right hides the panel without quitting the process
  99. (`hud.visible=false`); `Ctrl Alt Shift H` brings it back. The X is the only
  100. clickable area besides the drag strip: the rest stays click-through.
  101. - The **⏸/▶ pause button** sits left of the X: 22×22, amber `▶` while paused,
  102. white `⏸` while running, grey and inert when disarmed or locked. It toggles
  103. `POST /api/pause` / `/api/resume` and matches the PAUSED pill within one poll.
  104. - The **⤡ resize grip** at the bottom-right scales the whole panel (CSS base
  105. table × scale, keyboard keycaps and canvas included): `s = width/430`, clamped
  106. 0.85–1.8 (366–774 px). Double-clicking it returns to 1.0. Scale persists in
  107. `data/hud.json` and via `POST /api/hud/scale` / `mouse-driver hud scale 1.3`.
  108. - The **⠿ grip** and `MOUSE DRIVER` line drag the panel, but it never floats
  109. free: the drag stores an offset from the target's top-right corner
  110. (`offset_x`, `offset_y`; default 12,12). The panel then follows the target
  111. through moves, resizes, maximise and workspace changes at that same offset.
  112. The footer reads `follow · auto` or `follow · custom`; `Ctrl Alt Shift R` or
  113. `mouse-driver hud reset` restores the 12,12 offset. Offsets are clamped so the
  114. panel stays inside the target (a smaller target aligns top-left and only
  115. overhangs as needed). Visibility, mode, offset, scale and last realised
  116. position persist in `data/hud.json` with the driver as the single writer.
  117. ![HUD over the browser](docs/images/hud-target.png)
  118. States, in precedence order: `LOCKED` (red) → `PAUSED` (amber) → `ARMED`
  119. (lime, with countdown) → `IDLE` (grey).
  120. ## Overlay keyboard
  121. The panel carries a display-only keyboard that lights up the keys the driver
  122. presses. It never accepts input and never blocks a click.
  123. - **Collapsed (default):** one row under the legend reads
  124. `RECENT KEYS Ctrl+Alt+Shift+T ×4 · Enter · Esc · A` — newest first, last six
  125. entries, consecutive repeats collapsed to `×N`. Panel ~430×170 (content-sized).
  126. - **Expanded** (`Ctrl Alt Shift K`): the same recent-keys row plus a 60 % layout
  127. (`Esc 1…0 - = Backspace`, `Tab Q…P [ ] \`, `Caps A…; ' Enter`,
  128. `Shift Z…/ Shift`, `Ctrl Alt Space Alt Fn` + arrows), 24 px keycaps. Panel
  129. 520×360. Keys light lime for ~140 ms; a combo lights every modifier it used.
  130. - The toggle lives in `data/keyboard.json` and `POST /api/keyboard/toggle`, so
  131. the CLI (`mouse-driver keyboard`) and the panel agree.
  132. - Physical key observation would need XRecord, which is not wired here; the
  133. strip is therefore labelled `RECENT KEYS` and carries driver-synthesised
  134. events only (`mouse_type`, `mouse_key`). A passive grab on typing keys was
  135. rejected on purpose.
  136. ## Repeat task
  137. The panel's `TASK` row and the driver share one model in `data/task.json`
  138. (1–20 steps of `key`/`type`/`wait`, repeat `infinite`/`count`/`duration`,
  139. `interval_s`). `mouse_task_set/start/stop/status`, `mouse-driver task ...` and
  140. the editor's Start/Stop all use it. A task never arms itself: start requires an
  141. armed session, and it stops on pause, lock, panic, STOP, disarm or an expired
  142. TTL, with hard caps of 10 000 cycles or 8 hours. The HUD editor opens from `▸`
  143. below `RECENT KEYS`; while open the panel takes input (for the entries) instead
  144. of being click-through, and `close` restores pass-through. Chords are unaffected
  145. because only modifier combos are grabbed.
  146. ## Tools
  147. | Tool | Purpose |
  148. | --- | --- |
  149. | `mouse_status` | Arming, lock, pause, TTL, cursor, screen, session totals, API URL. Read-only. |
  150. | `mouse_arm(ttl_s)` | Time-boxed arming, 1–900 s. Refused while locked. |
  151. | `mouse_disarm` | Disarm, aborting motion. Does not lock. |
  152. | `mouse_stop` | Sentinel + disarm (no lock). |
  153. | `mouse_where` | Cursor, screen size and monitor count. Read-only. |
  154. | `mouse_move(x,y,duration_s,curve)` | Fitts-duration curved minimum-jerk path with tremor and occasional overshoot. |
  155. | `mouse_click(button,count,x,y)` | Slowed 24 px approach, 35–90 ms hold. |
  156. | `mouse_drag(x1,y1,x2,y2,button)` | Press, human-shaped drag, release even on abort. |
  157. | `mouse_scroll(dy,dx)` | 4–9 decaying wheel events with 15–40 ms gaps. |
  158. | `mouse_type(text)` | XTEST keys at 55–140 ms per character; text never in argv. |
  159. | `mouse_key(combo)` | One xdotool-syntax combination. |
  160. | `mouse_list_windows` | Managed windows with geometry and title. Read-only. |
  161. | `mouse_focus_window(title)` | Focus by title substring; sets the HUD target. |
  162. | `mouse_telemetry(limit)` | `/api/state` payload plus recent sessions. Read-only. |
  163. | `mouse_replay_session(id,speed)` | Visual replay of a recorded path, no clicks. |
  164. | `mouse_task_set(name,steps,repeat,interval_s,enabled)` | Validate and save the repeat task. |
  165. | `mouse_task_start` / `mouse_task_stop` | Run or stop it (start requires arm). |
  166. | `mouse_task_status` | Task model plus runner status. Read-only. |
  167. There is intentionally no `mouse_unlock` and no `mouse_resume`.
  168. ## Telemetry API
  169. Loopback only (`127.0.0.1:8340`), threaded stdlib server, no CORS, no auth.
  170. Exact shapes are in [docs/DESIGN.md](docs/DESIGN.md#telemetry-json).
  171. ```
  172. GET /healthz
  173. GET /api/state
  174. GET /api/stream (SSE, same payload every 250 ms)
  175. GET /api/sessions
  176. GET /api/sessions/<id>/path
  177. POST /api/disarm /api/stop /api/pause /api/resume
  178. POST /api/panic /api/lock /api/unlock /api/keyboard/toggle
  179. ```
  180. ## Motion model
  181. - Duration: `0.12 + 0.22·log2(d/20 + 1)` s, clamped 0.18–1.60 s.
  182. - Cubic Bezier with perpendicular control points (curvature grows with distance).
  183. - Minimum-jerk profile `s = 10t³ − 15t⁴ + 6t⁵`.
  184. - 1.5–3 px tremor, fading to zero at both ends.
  185. - ~12 % of moves overshoot 2–4 % of the distance, pause 40–90 ms, then correct.
  186. - Clicks slow the final 24 px and settle 55–120 ms before button down.
  187. - Points emitted at 120 Hz; every loop re-reads the arming/pause/lock state and
  188. the `data/STOP` sentinel, so a human chord stops motion within ~8 ms.
  189. ## Limits
  190. - Single desktop, single X session; not multi-head aware beyond the virtual screen.
  191. - No clickjacking guard: while armed an agent can press anything the user could.
  192. - The panic chord cannot beat a frozen X server; the sentinel file remains the
  193. cross-process backstop.
  194. - Typing covers the active keymap plus an xdotool fallback for other scripts.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.