Back to search
wayland_app_control
MCP serverGitHub
MCP server for LLM interaction with GUI applications on KDE + Wayland
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Checked today
Badge for your README
Markdown
[](/listings/trevorcash--wayland-app-control)HTML
<a href="/listings/trevorcash--wayland-app-control"><img src="/api/listings/trevorcash--wayland-app-control/badge.svg" alt="findsafeskills safety rating" /></a>Install
i
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 416 lines
- # wayland_app_control
- MCP server (built on [FastMCP](https://github.com/PrefectHQ/fastmcp)) that drives
- GUI applications inside a private, headless, nested Wayland session.
- The session is a `sway` compositor started with `WLR_BACKENDS=headless`. It has
- no parent seat, so host keyboard/mouse input can never reach the app. All input
- is injected through the compositor's own seat:
- | Channel | Mechanism |
- | --- | --- |
- | pointer | `swaymsg seat <seat> cursor set\|press\|release` + a long-lived `zwlr_virtual_pointer_v1` holder |
- | keyboard | `wtype` (virtual keyboard) |
- | screen | `grim` |
- The virtual-pointer helper is compiled on first use from C sources embedded in
- the script, so the only Python dependency is `fastmcp`.
- ## Requirements
- System tools (checked by `wyldappctrl_doctor`):
- ```bash
- # Debian / Ubuntu
- sudo apt install sway grim wtype libwayland-bin libwayland-dev build-essential
- # Fedora
- sudo dnf install sway grim wtype wayland-devel gcc
- # Arch
- sudo pacman -S sway grim wtype wayland gcc
- ```
- `sway` provides `swaymsg`; `libwayland-bin` provides `wayland-scanner`;
- `libwayland-dev` provides the `wayland-client` headers used to build the pointer
- holder.
- Python 3.11+ and FastMCP. The script carries PEP 723 inline metadata, so `uv`
- installs FastMCP automatically:
- ```bash
- uv run wayland_app_control.py
- ```
- Or install it yourself and run with any interpreter:
- ```bash
- pip install 'fastmcp>=2.0'
- python3 wayland_app_control.py
- ```
- The server speaks MCP over **stdio** and prints nothing to stdout except
- protocol traffic.
- ## Verify before wiring a harness
- ```bash
- # list tools (imports the file with the current interpreter)
- fastmcp list wayland_app_control.py
- # dependency check
- fastmcp call wayland_app_control.py wyldappctrl_doctor
- # one-call smoke test: start -> type -> Return -> screenshot -> stop
- fastmcp call wayland_app_control.py wyldappctrl_quick --input-json '{
- "command": "foot",
- "actions": [
- {"type": "type", "text": "echo hi"},
- {"type": "key", "key": "Return"},
- {"type": "wait", "ms": 400}
- ],
- "screenshot": true
- }'
- ```
- `fastmcp list` / `fastmcp call` import the script in-process, so `fastmcp` must
- be importable by the interpreter you invoke them with. Harnesses spawn the
- script as a subprocess, so they only need `uv` (or a Python with FastMCP) on
- `PATH`.
- `fastmcp call` spawns the server with a sanitized environment (no
- `XDG_RUNTIME_DIR`); the server falls back to `/run/user/<uid>` when that
- variable is missing. Harnesses inherit the full environment.
- ## Register with a harness
- Every harness below launches the server as a stdio subprocess. Use an absolute
- path to the script unless the harness starts children in the project directory
- (omp does; the others do not).
- ### omp
- This repo ships `.omp/mcp.json`, so omp discovers the server when started here:
- ```json
- {
- "$schema": "https://raw.githubusercontent.com/can1357/oh-my-pi/main/packages/coding-agent/src/config/mcp-schema.json",
- "mcpServers": {
- "wayland": {
- "type": "stdio",
- "command": "uv",
- "args": ["run", "./wayland_app_control.py"]
- }
- }
- }
- ```
- omp starts stdio children in the project directory, so the relative path
- resolves. For a global entry, put the same block in `~/.omp/mcp.json` with an
- absolute path.
- Verify inside omp:
- ```text
- /mcp reload # re-read .omp/mcp.json
- /mcp list # confirm the server and its source file
- /mcp test wayland # initialize + tools/list; reports identity and tool names
- ```
- `/mcp test` does not call a tool. Exercise a handler with a normal request
- ("Use the wayland server to run wyldappctrl_doctor"), then `/mcp reconnect
- wayland` after editing server code.
- omp-specific behavior this server already satisfies:
- - omp offers MCP revision `2025-11-25` and sends numeric JSON-RPC IDs; FastMCP
- negotiates the revision and accepts both ID forms.
- - omp namespaces tools by server and normalizes names to `[a-z_]`;
- `wyldappctrl_*` already conforms, so mounted names are stable.
- - Expected, user-correctable failures (for example `no session; call
- wyldappctrl_start first`) are raised as tool errors (`isError: true`), which
- omp surfaces as actionable tool errors. Protocol faults surface as MCP errors.
- - stdout carries protocol frames only: FastMCP logs to stderr and the banner is
- disabled.
- ### Claude Code
- ```bash
- claude mcp add --scope user wayland -- uv run /abs/path/wayland_app_control.py
- ```
- Project-scoped `.mcp.json`:
- ```json
- {
- "mcpServers": {
- "wayland": {
- "type": "stdio",
- "command": "uv",
- "args": ["run", "/abs/path/wayland_app_control.py"]
- }
- }
- }
- ```
- FastMCP helper (writes the same config, pins dependencies):
- ```bash
- fastmcp install claude-code wayland_app_control.py --name wayland
- ```
- ### Codex CLI
- ```bash
- codex mcp add wayland -- uv run /abs/path/wayland_app_control.py
- ```
- `~/.codex/config.toml` (or `.codex/config.toml` in a trusted project):
- ```toml
- [mcp_servers.wayland]
- command = "uv"
- args = ["run", "/abs/path/wayland_app_control.py"]
- ```
- Codex reads the server's `instructions` field at initialize; this server sends
- the start → launch → screenshot → input → log → stop workflow there.
- ### Cursor
- `~/.cursor/mcp.json` (or `.cursor/mcp.json` in a project):
- ```json
- {
- "mcpServers": {
- "wayland": {
- "type": "stdio",
- "command": "uv",
- "args": ["run", "/abs/path/wayland_app_control.py"]
- }
- }
- }
- ```
- ```bash
- fastmcp install cursor wayland_app_control.py --name wayland
- ```
- ### VS Code (Copilot)
- Workspace-root `.mcp.json` (portable `mcpServers` format) or
- `~/.copilot/mcp-config.json`:
- ```json
- {
- "mcpServers": {
- "wayland": {
- "type": "stdio",
- "command": "uv",
- "args": ["run", "/abs/path/wayland_app_control.py"],
- "tools": ["*"]
- }
- }
- }
- ```
- Do not use `.vscode/mcp.json` for new setups: it uses the deprecated
- editor-managed `servers` schema, not `mcpServers`.
- ### Gemini CLI
- ```bash
- gemini mcp add --scope user wayland uv run /abs/path/wayland_app_control.py
- ```
- `~/.gemini/settings.json`:
- ```json
- {
- "mcpServers": {
- "wayland": {
- "command": "uv",
- "args": ["run", "/abs/path/wayland_app_control.py"],
- "timeout": 600000
- }
- }
- }
- ```
- ```bash
- fastmcp install gemini-cli wayland_app_control.py --name wayland
- ```
- ### opencode
- ```bash
- opencode mcp add wayland -- uv run /abs/path/wayland_app_control.py
- ```
- `opencode.jsonc`:
- ```jsonc
- {
- "$schema": "https://opencode.ai/config.json",
- "mcp": {
- "servers": {
- "wayland": {
- "type": "local",
- "command": ["uv", "run", "/abs/path/wayland_app_control.py"]
- }
- }
- }
- }
- ```
- ### Zed
- `settings.json`:
- ```json
- {
- "context_servers": {
- "wayland": {
- "command": "uv",
- "args": ["run", "/abs/path/wayland_app_control.py"],
- "env": {}
- }
- }
- }
- ```
- ### Claude Desktop
- `claude_desktop_config.json`:
- ```json
- {
- "mcpServers": {
- "wayland": {
- "command": "uv",
- "args": ["run", "/abs/path/wayland_app_control.py"]
- }
- }
- }
- ```
- ```bash
- fastmcp install claude-desktop wayland_app_control.py --name wayland
- ```
- ### Goose
- ```bash
- fastmcp install goose wayland_app_control.py --name wayland
- ```
- ### Any other client
- Print a ready-to-paste `mcpServers` block:
- ```bash
- fastmcp install mcp-json wayland_app_control.py --name wayland
- ```
- Print the raw stdio command:
- ```bash
- fastmcp install stdio wayland_app_control.py
- ```
- ## Tools
- | Tool | Arguments | Returns |
- | --- | --- | --- |
- | `wyldappctrl_quick` | `command`, `args?`, `actions?`, `screenshot=true`, `width=1280`, `height=720`, `settle_ms=500`, `env?`, `cwd?` | `ok`, `pid`, `screenshot?`, `log_path`, `elapsed_ms`; on failure `ok=false`, `error`, `log` |
- | `wyldappctrl_doctor` | – | resolved paths for `sway`, `swaymsg`, `grim`, `wtype`, `wayland-scanner`, `cc`; `ok` / `missing` |
- | `wyldappctrl_start` | `width=1280`, `height=720`, `extra_config=""` | `display`, `output`, `seat`, `size`, `sway_pid`, `app_pid`, `app_log` |
- | `wyldappctrl_launch` | `command`, `args?`, `env?`, `cwd?` | `pid`, `log` |
- | `wyldappctrl_log` | `lines=60` | tail of the app's stdout/stderr |
- | `wyldappctrl_screenshot` | `path?` | PNG `path`, `bytes` |
- | `wyldappctrl_windows` | – | `views[]` with `title`, `app_id`, `focused`, `rect` |
- | `wyldappctrl_mouse` | `action` (`move`/`click`/`press`/`release`), `x?`, `y?`, `button="left"` | `moved` or `action` |
- | `wyldappctrl_type` | `text`, `delay_ms=80`, `settle_ms=250` | `typed` |
- | `wyldappctrl_key` | `key` (xkb name, e.g. `Return`, `Escape`, `Tab`, `F5`) | `key` |
- | `wyldappctrl_stop` | – | `stopped` |
- ### One-call quick run
- `wyldappctrl_quick` is the fast path: it stops any existing session, starts a
- fresh one, launches the app, runs the actions in order, optionally captures a
- screenshot, tears everything down, and returns `ok`.
- | Action `type` | Fields |
- | --- | --- |
- | `click` | `x`, `y`, `button` (`left` / `middle` / `right`) |
- | `move` | `x`, `y` |
- | `type` | `text` |
- | `key` | `key` (xkb name, e.g. `Return`) |
- | `wait` | `ms` |
- ```json
- {
- "command": "foot",
- "actions": [
- {"type": "wait", "ms": 500},
- {"type": "type", "text": "echo hi"},
- {"type": "key", "key": "Return"},
- {"type": "wait", "ms": 300}
- ],
- "screenshot": true
- }
- ```
- Returns `{"ok": true, "pid": ..., "screenshot": "/tmp/wayland-mcp/shot-001.png",
- "log_path": ..., "elapsed_ms": ...}`. A launch or action failure returns
- `ok: false` with `error` and the log tail; invalid action arguments raise a
- tool error. The session is always torn down.
- For step-by-step control:
- ```text
- wyldappctrl_start
- -> wyldappctrl_launch
- -> wyldappctrl_screenshot (confirm the window is up)
- -> wyldappctrl_mouse / wyldappctrl_type / wyldappctrl_key
- -> wyldappctrl_screenshot (confirm the interaction landed)
- -> wyldappctrl_log (on failure)
- -> wyldappctrl_stop
- ```
- Coordinates are output pixels with the origin at the top-left of the app window
- (no title bar). `click` sends press and release in separate frames so
- ImGui-style UIs register it. `type` and `key` sleep 100 ms after registering the
- virtual keyboard so the first event is not dropped.
- ## Session model
- - One server process owns exactly one session (module-level state). Each
- harness spawns its own process, so concurrent agents get independent sway
- instances; never point two clients at one process.
- - `wyldappctrl_start` is idempotent while sway is alive.
- - `wyldappctrl_launch` runs one app at a time; call `wyldappctrl_stop` first to
- replace it.
- - `wyldappctrl_quick` stops any existing session first, holds the session lock
- for the whole run, and always tears down.
- - `wyldappctrl_stop` terminates the app, the pointer holder and sway, and is
- safe to call repeatedly.
- - `XDG_RUNTIME_DIR` is used for sway's IPC socket
- (`$XDG_RUNTIME_DIR/sway-ipc.<uid>.<pid>.sock`); if unset, the server falls
- back to `/run/user/<uid>`.
- - Screenshots and logs are files on the same host:
- `$TMPDIR/wayland-mcp/{sway.conf,sway.log,app.log,shot-NNN.png}`.
- The compiled pointer holder is cached in
- `$XDG_CACHE_HOME/wayland-mcp/pointer-holder-<hash>`.
- ## Troubleshooting
- | Symptom | Cause / fix |
- | --- | --- |
- | `missing tools: ...` | Install the system packages listed above. |
- | `sway exited:` + log | Read `$TMPDIR/wayland-mcp/sway.log`. Headless software rendering is already forced via `WLR_RENDERER_ALLOW_SOFTWARE=1`. |
- | `timed out waiting for virtual pointer` | The compositor lacks `zwlr_virtual_pointer_manager_v1`; use a current sway. |
- | `timed out waiting for sway IPC socket` | `XDG_RUNTIME_DIR` is unset or not writable. |
- | `XDG_RUNTIME_DIR is not set` | The server falls back to `/run/user/<uid>`; if that directory is missing, set `XDG_RUNTIME_DIR` in the harness environment. |
- | Clicks/keys have no effect | Take a screenshot after each action; check `wyldappctrl_windows` for focus. |
- | Harness shows the server but no tools | The harness's `uv`/`python3` cannot import `fastmcp`; run `fastmcp list wayland_app_control.py` with the same interpreter. |
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.