Skip to content
Back to search

wayland_app_control

MCP server for LLM interaction with GUI applications on KDE + Wayland

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/trevorcash--wayland-app-control/badge.svg)](/listings/trevorcash--wayland-app-control)

HTML

<a href="/listings/trevorcash--wayland-app-control"><img src="/api/listings/trevorcash--wayland-app-control/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 416 lines
  1. # wayland_app_control
  2. MCP server (built on [FastMCP](https://github.com/PrefectHQ/fastmcp)) that drives
  3. GUI applications inside a private, headless, nested Wayland session.
  4. The session is a `sway` compositor started with `WLR_BACKENDS=headless`. It has
  5. no parent seat, so host keyboard/mouse input can never reach the app. All input
  6. is injected through the compositor's own seat:
  7. | Channel | Mechanism |
  8. | --- | --- |
  9. | pointer | `swaymsg seat <seat> cursor set\|press\|release` + a long-lived `zwlr_virtual_pointer_v1` holder |
  10. | keyboard | `wtype` (virtual keyboard) |
  11. | screen | `grim` |
  12. The virtual-pointer helper is compiled on first use from C sources embedded in
  13. the script, so the only Python dependency is `fastmcp`.
  14. ## Requirements
  15. System tools (checked by `wyldappctrl_doctor`):
  16. ```bash
  17. # Debian / Ubuntu
  18. sudo apt install sway grim wtype libwayland-bin libwayland-dev build-essential
  19. # Fedora
  20. sudo dnf install sway grim wtype wayland-devel gcc
  21. # Arch
  22. sudo pacman -S sway grim wtype wayland gcc
  23. ```
  24. `sway` provides `swaymsg`; `libwayland-bin` provides `wayland-scanner`;
  25. `libwayland-dev` provides the `wayland-client` headers used to build the pointer
  26. holder.
  27. Python 3.11+ and FastMCP. The script carries PEP 723 inline metadata, so `uv`
  28. installs FastMCP automatically:
  29. ```bash
  30. uv run wayland_app_control.py
  31. ```
  32. Or install it yourself and run with any interpreter:
  33. ```bash
  34. pip install 'fastmcp>=2.0'
  35. python3 wayland_app_control.py
  36. ```
  37. The server speaks MCP over **stdio** and prints nothing to stdout except
  38. protocol traffic.
  39. ## Verify before wiring a harness
  40. ```bash
  41. # list tools (imports the file with the current interpreter)
  42. fastmcp list wayland_app_control.py
  43. # dependency check
  44. fastmcp call wayland_app_control.py wyldappctrl_doctor
  45. # one-call smoke test: start -> type -> Return -> screenshot -> stop
  46. fastmcp call wayland_app_control.py wyldappctrl_quick --input-json '{
  47. "command": "foot",
  48. "actions": [
  49. {"type": "type", "text": "echo hi"},
  50. {"type": "key", "key": "Return"},
  51. {"type": "wait", "ms": 400}
  52. ],
  53. "screenshot": true
  54. }'
  55. ```
  56. `fastmcp list` / `fastmcp call` import the script in-process, so `fastmcp` must
  57. be importable by the interpreter you invoke them with. Harnesses spawn the
  58. script as a subprocess, so they only need `uv` (or a Python with FastMCP) on
  59. `PATH`.
  60. `fastmcp call` spawns the server with a sanitized environment (no
  61. `XDG_RUNTIME_DIR`); the server falls back to `/run/user/<uid>` when that
  62. variable is missing. Harnesses inherit the full environment.
  63. ## Register with a harness
  64. Every harness below launches the server as a stdio subprocess. Use an absolute
  65. path to the script unless the harness starts children in the project directory
  66. (omp does; the others do not).
  67. ### omp
  68. This repo ships `.omp/mcp.json`, so omp discovers the server when started here:
  69. ```json
  70. {
  71. "$schema": "https://raw.githubusercontent.com/can1357/oh-my-pi/main/packages/coding-agent/src/config/mcp-schema.json",
  72. "mcpServers": {
  73. "wayland": {
  74. "type": "stdio",
  75. "command": "uv",
  76. "args": ["run", "./wayland_app_control.py"]
  77. }
  78. }
  79. }
  80. ```
  81. omp starts stdio children in the project directory, so the relative path
  82. resolves. For a global entry, put the same block in `~/.omp/mcp.json` with an
  83. absolute path.
  84. Verify inside omp:
  85. ```text
  86. /mcp reload # re-read .omp/mcp.json
  87. /mcp list # confirm the server and its source file
  88. /mcp test wayland # initialize + tools/list; reports identity and tool names
  89. ```
  90. `/mcp test` does not call a tool. Exercise a handler with a normal request
  91. ("Use the wayland server to run wyldappctrl_doctor"), then `/mcp reconnect
  92. wayland` after editing server code.
  93. omp-specific behavior this server already satisfies:
  94. - omp offers MCP revision `2025-11-25` and sends numeric JSON-RPC IDs; FastMCP
  95. negotiates the revision and accepts both ID forms.
  96. - omp namespaces tools by server and normalizes names to `[a-z_]`;
  97. `wyldappctrl_*` already conforms, so mounted names are stable.
  98. - Expected, user-correctable failures (for example `no session; call
  99. wyldappctrl_start first`) are raised as tool errors (`isError: true`), which
  100. omp surfaces as actionable tool errors. Protocol faults surface as MCP errors.
  101. - stdout carries protocol frames only: FastMCP logs to stderr and the banner is
  102. disabled.
  103. ### Claude Code
  104. ```bash
  105. claude mcp add --scope user wayland -- uv run /abs/path/wayland_app_control.py
  106. ```
  107. Project-scoped `.mcp.json`:
  108. ```json
  109. {
  110. "mcpServers": {
  111. "wayland": {
  112. "type": "stdio",
  113. "command": "uv",
  114. "args": ["run", "/abs/path/wayland_app_control.py"]
  115. }
  116. }
  117. }
  118. ```
  119. FastMCP helper (writes the same config, pins dependencies):
  120. ```bash
  121. fastmcp install claude-code wayland_app_control.py --name wayland
  122. ```
  123. ### Codex CLI
  124. ```bash
  125. codex mcp add wayland -- uv run /abs/path/wayland_app_control.py
  126. ```
  127. `~/.codex/config.toml` (or `.codex/config.toml` in a trusted project):
  128. ```toml
  129. [mcp_servers.wayland]
  130. command = "uv"
  131. args = ["run", "/abs/path/wayland_app_control.py"]
  132. ```
  133. Codex reads the server's `instructions` field at initialize; this server sends
  134. the start → launch → screenshot → input → log → stop workflow there.
  135. ### Cursor
  136. `~/.cursor/mcp.json` (or `.cursor/mcp.json` in a project):
  137. ```json
  138. {
  139. "mcpServers": {
  140. "wayland": {
  141. "type": "stdio",
  142. "command": "uv",
  143. "args": ["run", "/abs/path/wayland_app_control.py"]
  144. }
  145. }
  146. }
  147. ```
  148. ```bash
  149. fastmcp install cursor wayland_app_control.py --name wayland
  150. ```
  151. ### VS Code (Copilot)
  152. Workspace-root `.mcp.json` (portable `mcpServers` format) or
  153. `~/.copilot/mcp-config.json`:
  154. ```json
  155. {
  156. "mcpServers": {
  157. "wayland": {
  158. "type": "stdio",
  159. "command": "uv",
  160. "args": ["run", "/abs/path/wayland_app_control.py"],
  161. "tools": ["*"]
  162. }
  163. }
  164. }
  165. ```
  166. Do not use `.vscode/mcp.json` for new setups: it uses the deprecated
  167. editor-managed `servers` schema, not `mcpServers`.
  168. ### Gemini CLI
  169. ```bash
  170. gemini mcp add --scope user wayland uv run /abs/path/wayland_app_control.py
  171. ```
  172. `~/.gemini/settings.json`:
  173. ```json
  174. {
  175. "mcpServers": {
  176. "wayland": {
  177. "command": "uv",
  178. "args": ["run", "/abs/path/wayland_app_control.py"],
  179. "timeout": 600000
  180. }
  181. }
  182. }
  183. ```
  184. ```bash
  185. fastmcp install gemini-cli wayland_app_control.py --name wayland
  186. ```
  187. ### opencode
  188. ```bash
  189. opencode mcp add wayland -- uv run /abs/path/wayland_app_control.py
  190. ```
  191. `opencode.jsonc`:
  192. ```jsonc
  193. {
  194. "$schema": "https://opencode.ai/config.json",
  195. "mcp": {
  196. "servers": {
  197. "wayland": {
  198. "type": "local",
  199. "command": ["uv", "run", "/abs/path/wayland_app_control.py"]
  200. }
  201. }
  202. }
  203. }
  204. ```
  205. ### Zed
  206. `settings.json`:
  207. ```json
  208. {
  209. "context_servers": {
  210. "wayland": {
  211. "command": "uv",
  212. "args": ["run", "/abs/path/wayland_app_control.py"],
  213. "env": {}
  214. }
  215. }
  216. }
  217. ```
  218. ### Claude Desktop
  219. `claude_desktop_config.json`:
  220. ```json
  221. {
  222. "mcpServers": {
  223. "wayland": {
  224. "command": "uv",
  225. "args": ["run", "/abs/path/wayland_app_control.py"]
  226. }
  227. }
  228. }
  229. ```
  230. ```bash
  231. fastmcp install claude-desktop wayland_app_control.py --name wayland
  232. ```
  233. ### Goose
  234. ```bash
  235. fastmcp install goose wayland_app_control.py --name wayland
  236. ```
  237. ### Any other client
  238. Print a ready-to-paste `mcpServers` block:
  239. ```bash
  240. fastmcp install mcp-json wayland_app_control.py --name wayland
  241. ```
  242. Print the raw stdio command:
  243. ```bash
  244. fastmcp install stdio wayland_app_control.py
  245. ```
  246. ## Tools
  247. | Tool | Arguments | Returns |
  248. | --- | --- | --- |
  249. | `wyldappctrl_quick` | `command`, `args?`, `actions?`, `screenshot=true`, `width=1280`, `height=720`, `settle_ms=500`, `env?`, `cwd?` | `ok`, `pid`, `screenshot?`, `log_path`, `elapsed_ms`; on failure `ok=false`, `error`, `log` |
  250. | `wyldappctrl_doctor` | – | resolved paths for `sway`, `swaymsg`, `grim`, `wtype`, `wayland-scanner`, `cc`; `ok` / `missing` |
  251. | `wyldappctrl_start` | `width=1280`, `height=720`, `extra_config=""` | `display`, `output`, `seat`, `size`, `sway_pid`, `app_pid`, `app_log` |
  252. | `wyldappctrl_launch` | `command`, `args?`, `env?`, `cwd?` | `pid`, `log` |
  253. | `wyldappctrl_log` | `lines=60` | tail of the app's stdout/stderr |
  254. | `wyldappctrl_screenshot` | `path?` | PNG `path`, `bytes` |
  255. | `wyldappctrl_windows` | – | `views[]` with `title`, `app_id`, `focused`, `rect` |
  256. | `wyldappctrl_mouse` | `action` (`move`/`click`/`press`/`release`), `x?`, `y?`, `button="left"` | `moved` or `action` |
  257. | `wyldappctrl_type` | `text`, `delay_ms=80`, `settle_ms=250` | `typed` |
  258. | `wyldappctrl_key` | `key` (xkb name, e.g. `Return`, `Escape`, `Tab`, `F5`) | `key` |
  259. | `wyldappctrl_stop` | – | `stopped` |
  260. ### One-call quick run
  261. `wyldappctrl_quick` is the fast path: it stops any existing session, starts a
  262. fresh one, launches the app, runs the actions in order, optionally captures a
  263. screenshot, tears everything down, and returns `ok`.
  264. | Action `type` | Fields |
  265. | --- | --- |
  266. | `click` | `x`, `y`, `button` (`left` / `middle` / `right`) |
  267. | `move` | `x`, `y` |
  268. | `type` | `text` |
  269. | `key` | `key` (xkb name, e.g. `Return`) |
  270. | `wait` | `ms` |
  271. ```json
  272. {
  273. "command": "foot",
  274. "actions": [
  275. {"type": "wait", "ms": 500},
  276. {"type": "type", "text": "echo hi"},
  277. {"type": "key", "key": "Return"},
  278. {"type": "wait", "ms": 300}
  279. ],
  280. "screenshot": true
  281. }
  282. ```
  283. Returns `{"ok": true, "pid": ..., "screenshot": "/tmp/wayland-mcp/shot-001.png",
  284. "log_path": ..., "elapsed_ms": ...}`. A launch or action failure returns
  285. `ok: false` with `error` and the log tail; invalid action arguments raise a
  286. tool error. The session is always torn down.
  287. For step-by-step control:
  288. ```text
  289. wyldappctrl_start
  290. -> wyldappctrl_launch
  291. -> wyldappctrl_screenshot (confirm the window is up)
  292. -> wyldappctrl_mouse / wyldappctrl_type / wyldappctrl_key
  293. -> wyldappctrl_screenshot (confirm the interaction landed)
  294. -> wyldappctrl_log (on failure)
  295. -> wyldappctrl_stop
  296. ```
  297. Coordinates are output pixels with the origin at the top-left of the app window
  298. (no title bar). `click` sends press and release in separate frames so
  299. ImGui-style UIs register it. `type` and `key` sleep 100 ms after registering the
  300. virtual keyboard so the first event is not dropped.
  301. ## Session model
  302. - One server process owns exactly one session (module-level state). Each
  303. harness spawns its own process, so concurrent agents get independent sway
  304. instances; never point two clients at one process.
  305. - `wyldappctrl_start` is idempotent while sway is alive.
  306. - `wyldappctrl_launch` runs one app at a time; call `wyldappctrl_stop` first to
  307. replace it.
  308. - `wyldappctrl_quick` stops any existing session first, holds the session lock
  309. for the whole run, and always tears down.
  310. - `wyldappctrl_stop` terminates the app, the pointer holder and sway, and is
  311. safe to call repeatedly.
  312. - `XDG_RUNTIME_DIR` is used for sway's IPC socket
  313. (`$XDG_RUNTIME_DIR/sway-ipc.<uid>.<pid>.sock`); if unset, the server falls
  314. back to `/run/user/<uid>`.
  315. - Screenshots and logs are files on the same host:
  316. `$TMPDIR/wayland-mcp/{sway.conf,sway.log,app.log,shot-NNN.png}`.
  317. The compiled pointer holder is cached in
  318. `$XDG_CACHE_HOME/wayland-mcp/pointer-holder-<hash>`.
  319. ## Troubleshooting
  320. | Symptom | Cause / fix |
  321. | --- | --- |
  322. | `missing tools: ...` | Install the system packages listed above. |
  323. | `sway exited:` + log | Read `$TMPDIR/wayland-mcp/sway.log`. Headless software rendering is already forced via `WLR_RENDERER_ALLOW_SOFTWARE=1`. |
  324. | `timed out waiting for virtual pointer` | The compositor lacks `zwlr_virtual_pointer_manager_v1`; use a current sway. |
  325. | `timed out waiting for sway IPC socket` | `XDG_RUNTIME_DIR` is unset or not writable. |
  326. | `XDG_RUNTIME_DIR is not set` | The server falls back to `/run/user/<uid>`; if that directory is missing, set `XDG_RUNTIME_DIR` in the harness environment. |
  327. | Clicks/keys have no effect | Take a screenshot after each action; check `wyldappctrl_windows` for focus. |
  328. | Harness shows the server but no tools | The harness's `uv`/`python3` cannot import `fastmcp`; run `fastmcp list wayland_app_control.py` with the same interpreter. |

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.