Skip to content
Back to search

GhostDesk

Give any AI agent a full desktop — it sees the screen, clicks, types, and runs apps like a human. Automate anything with a UI: browsers, legacy software, internal tools. No API needed. One Docker command.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/yv17labs--ghostdesk/badge.svg)](/listings/yv17labs--ghostdesk)

HTML

<a href="/listings/yv17labs--ghostdesk"><img src="/api/listings/yv17labs--ghostdesk/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Run the container

One command, plain HTTP, no password. Fine for kicking the tires on a laptop you trust — **not fit for anything beyond that**. Ready to harden it? Jump to [Secure local run](#secure-local-run-tls--auth).

docker run -d --name ghostdesk-demo \
  --shm-size 2g \
  -p 3000:3000 \
  -p 6080:6080 \
  ghcr.io/yv17labs/ghostdesk:latest

The `latest` image ships with **Firefox**, the **foot** terminal, **mousepad** (text editor), **galculator**, and passwordless `sudo` for the `agent` user — enough to demo a browsing + note-taking workflow out of the box. Need a different app set? Build your own on top of `base` — see [Custom image](#custom-image).

The container boots in the dev posture: plain HTTP on both ports, every auth gate disarmed on purpose. You'll see warnings in the logs reminding you of that — they go away once you follow the secured path below.

2. Connect your agent

GhostDesk speaks [MCP](https://modelcontextprotocol.io/) over the Streamable HTTP transport — any MCP-compatible client can drive it. Point the one you already work from at `http://localhost:3000/mcp`: no headers, no auth, that is the whole demo posture.

**[Claude Code](https://claude.com/product/claude-code) / Claude Desktop**

{
  "mcpServers": {
    "ghostdesk": {
      "type": "http",
      "url": "http://localhost:3000/mcp"
    }
  }
}

**[Hermes Agent](https://hermes-agent.nousresearch.com), [OpenCode](https://opencode.ai/)**, or whatever you already drive your agents with — same URL, and it works very well.

**[SpecterChat](https://github.com/YV17labs/SpecterChat)** — the chat client we build for this, open source. Most chat UIs drop the image an MCP tool returns — they render it or they forward it to the model, rarely both — and a `screen_shot()` the model never sees is the whole product missing. SpecterChat displays it inline *and* sends it back as base64. It talks to any OpenAI-compatible endpoint (llama.cpp, vLLM, Ollama, LM Studio), so it takes either side of the next step; macOS, Linux and Windows builds are on its releases page.

3. Choose where the model runs — cloud or local

The client is settled; the model behind it is a choice, and neither answer is mandatory.

**In the cloud** — Claude, GPT, Gemini and the other frontier models drive GhostDesk out of the box: no header, nothing to install, the key your client already holds. If that is you, the setup is done — go to [step 4](#4-watch-your-agent-work).

**On your machine** — [Ollama](https://ollama.com/) is the recommended path: one installer, one model to pull, nothing to build and nothing to tune. Windows, macOS or Linux. No API bill, and no screenshot of your desktop leaving your network.

1. **Install Ollama** from [ollama.com/download](https://ollama.com/download) — the official app, downloaded like any other. Once installed it runs in the background, so there is no server to start. 2. **Pull the model.** In a terminal:

   ollama pull qwen3.8:27b

On Apple Silicon, pull [`qwen3.8:27b-mlx`](https://ollama.com/library/qwen3.8:27b-mlx) instead. 3. **Point your client at it.** Endpoint `http://localhost:11434/v1`, the tag you pulled as the model name, and the `GhostDesk-Model-Space: 1000` header on every MCP request — the Qwen family needs it, and the JSON for it is under [Coordinate space](#coordinate-space--ghostdesk-model-space-header).

That is the whole setup. **[`qwen3.8:27b`](https://ollama.com/library/qwen3.8:27b) is the default because it is the strongest** — best reasoning, best capability, and on desktop control a step the agent gets wrong costs more than the tokens it saves. When speed is what you need, [`qwen3.6:35b`](https://ollama.com/library/qwen3.6:35b) (on Apple Silicon, [`qwen3.6:35b-mlx`](https://ollama.com/library/qwen3.6:35b-mlx)) is the fast one: 35B parameters with only 3B active per token ([Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)), so tokens

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 1,163 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src="assets/banner.svg" alt="GhostDesk — Virtual Desktop Control" width="100%">
  3. </p>
  4. <p align="center">
  5. <img src="https://img.shields.io/badge/MCP-compatible-blueviolet?style=for-the-badge" alt="MCP Compatible">
  6. <img src="https://img.shields.io/badge/rust-1.97+-orange?style=for-the-badge&logo=rust&logoColor=white" alt="Rust 1.97+">
  7. <img src="https://img.shields.io/badge/built%20with-NestRS-7B6FDE?style=for-the-badge" alt="Built with NestRS">
  8. <img src="https://img.shields.io/badge/license-FSL--1.1--ALv2-blue?style=for-the-badge" alt="FSL-1.1-ALv2 License">
  9. <img src="https://img.shields.io/badge/platform-Docker%20%7C%20Linux%20%7C%20macOS%20%7C%20Windows-orange?style=for-the-badge" alt="Platform">
  10. </p>
  11. <p align="center">
  12. <strong>Give your AI agent eyes, hands, and a full desktop.</strong><br>
  13. An MCP server that lets LLM agents see the screen, move the mouse, type on the keyboard, launch apps, and run shell commands — in a sandboxed virtual desktop, or on the one in front of you.
  14. </p>
  15. <p align="center">
  16. <em>If a human can do it on a desktop, your agent can too.</em>
  17. </p>
  18. <p align="center">
  19. <video src="https://github.com/user-attachments/assets/304ba0b6-4ba8-49df-9825-a3bb78c2727e" controls muted playsinline width="960">
  20. Your browser does not support the video tag.
  21. </video>
  22. </p>
  23. <p align="center">
  24. <em>GhostDesk demo — from a single prompt (<strong>"open the browser, go to Google News, and tell me the latest headlines in the Technology section"</strong>), the agent launches Firefox, navigates to Google News, switches to the Technology section, and reports the latest stories back.</em>
  25. </p>
  26. ---
  27. **One binary, one MCP endpoint, three desktops.** The tool surface is the same
  28. everywhere — an agent calls `mouse_click` and `app_launch` without knowing
  29. which desktop it is on.
  30. | | **Linux** | **macOS** | **Windows** |
  31. |---|---|---|---|
  32. | **How it runs** | the Docker container, in one command | a native binary | a native binary |
  33. | **The desktop it drives** | a virtual one, shipped inside the image | the Mac in front of you | the PC in front of you |
  34. | **Sandboxed** | yes — disposable, one per agent | no | no |
  35. | **Driven through** | `zwlr_virtual_pointer_v1`, Sway IPC, `grim` | Quartz Event Services, Accessibility, `screencapture` | `SendInput`, GDI, `EnumWindows` |
  36. **Linux is the server.** The container is the deployment this README shows
  37. unless it says otherwise, and the only one of the three with a sandbox around
  38. it. macOS and Windows run the very same server against a real desk — no
  39. container, and therefore no isolation. Jump to
  40. [macOS](#run-on-macos-without-the-container) or
  41. [Windows](#run-on-windows-without-the-container), or read [what
  42. differs](#what-differs-from-the-container) between the three.
  43. <details>
  44. <summary><strong>Table of contents</strong></summary>
  45. - [Quick start](#quick-start)
  46. - [Tools](#tools)
  47. - [Model requirements](#model-requirements)
  48. - [Why GhostDesk?](#why-ghostdesk)
  49. - [How it works](#how-it-works)
  50. - [Secure local run (TLS + auth)](#secure-local-run-tls--auth)
  51. - [Running many agents](#running-many-agents)
  52. - [Custom image](#custom-image)
  53. - [Run on macOS, without the container](#run-on-macos-without-the-container)
  54. - [Run on Windows, without the
  55. container](#run-on-windows-without-the-container)
  56. - [What differs from the container](#what-differs-from-the-container)
  57. - [Configuration](#configuration)
  58. - [Security](#security)
  59. - [Troubleshooting](#troubleshooting)
  60. - [Build from source](#build-from-source)
  61. - [License](#license)
  62. </details>
  63. ---
  64. ## Quick start
  65. ### 1. Run the container
  66. One command, plain HTTP, no password. Fine for kicking the tires on a laptop
  67. you trust — **not fit for anything beyond that**. Ready to harden it? Jump to
  68. [Secure local run](#secure-local-run-tls--auth).
  69. ```bash
  70. docker run -d --name ghostdesk-demo \
  71. --shm-size 2g \
  72. -p 3000:3000 \
  73. -p 6080:6080 \
  74. ghcr.io/yv17labs/ghostdesk:latest
  75. ```
  76. The `latest` image ships with **Firefox**, the **foot** terminal, **mousepad**
  77. (text editor), **galculator**, and passwordless `sudo` for the `agent` user —
  78. enough to demo a browsing + note-taking workflow out of the box. Need a
  79. different app set? Build your own on top of `base` — see [Custom
  80. image](#custom-image).
  81. The container boots in the dev posture: plain HTTP on both ports, every auth
  82. gate disarmed on purpose. You'll see warnings in the logs reminding you of that
  83. — they go away once you follow the secured path below.
  84. ### 2. Connect your agent
  85. GhostDesk speaks [MCP](https://modelcontextprotocol.io/) over the Streamable
  86. HTTP transport — any MCP-compatible client can drive it. Point the one you
  87. already work from at `http://localhost:3000/mcp`: no headers, no auth, that is
  88. the whole demo posture.
  89. **[Claude Code](https://claude.com/product/claude-code) / Claude Desktop**
  90. ```json
  91. {
  92. "mcpServers": {
  93. "ghostdesk": {
  94. "type": "http",
  95. "url": "http://localhost:3000/mcp"
  96. }
  97. }
  98. }
  99. ```
  100. **[Hermes Agent](https://hermes-agent.nousresearch.com),
  101. [OpenCode](https://opencode.ai/)**, or whatever you already drive your agents
  102. with — same URL, and it works very well.
  103. **[SpecterChat](https://github.com/YV17labs/SpecterChat)** — the chat client
  104. we build for this, open source. Most chat UIs drop the image an MCP tool
  105. returns — they render it or they forward it to the model, rarely both — and a
  106. `screen_shot()` the model never sees is the whole product missing. SpecterChat
  107. displays it inline *and* sends it back as base64. It talks to any
  108. OpenAI-compatible endpoint (llama.cpp, vLLM, Ollama, LM Studio), so it takes
  109. either side of the next step; macOS, Linux and Windows builds are on its
  110. releases page.
  111. ### 3. Choose where the model runs — cloud or local
  112. The client is settled; the model behind it is a choice, and neither answer is
  113. mandatory.
  114. **In the cloud** — Claude, GPT, Gemini and the other frontier models drive
  115. GhostDesk out of the box: no header, nothing to install, the key your client
  116. already holds. If that is you, the setup is done — go to [step
  117. 4](#4-watch-your-agent-work).
  118. **On your machine** — [Ollama](https://ollama.com/) is the recommended path:
  119. one installer, one model to pull, nothing to build and nothing to tune.
  120. Windows, macOS or Linux. No API bill, and no screenshot of your desktop
  121. leaving your network.
  122. 1. **Install Ollama** from [ollama.com/download](https://ollama.com/download)
  123. — the official app, downloaded like any other. Once installed it runs in
  124. the background, so there is no server to start.
  125. 2. **Pull the model.** In a terminal:
  126. ```bash
  127. ollama pull qwen3.8:27b
  128. ```
  129. On Apple Silicon, pull
  130. [`qwen3.8:27b-mlx`](https://ollama.com/library/qwen3.8:27b-mlx) instead.
  131. 3. **Point your client at it.** Endpoint `http://localhost:11434/v1`, the tag
  132. you pulled as the model name, and the `GhostDesk-Model-Space: 1000` header
  133. on every MCP request — the Qwen family needs it, and the JSON for it is
  134. under [Coordinate space](#coordinate-space--ghostdesk-model-space-header).
  135. That is the whole setup. **[`qwen3.8:27b`](https://ollama.com/library/qwen3.8:27b)
  136. is the default because it is the strongest** — best reasoning, best capability,
  137. and on desktop control a step the agent gets wrong costs more than the tokens
  138. it saves. When speed is what you need,
  139. [`qwen3.6:35b`](https://ollama.com/library/qwen3.6:35b) (on Apple Silicon,
  140. [`qwen3.6:35b-mlx`](https://ollama.com/library/qwen3.6:35b-mlx)) is the fast
  141. one: 35B parameters with only 3B active per token
  142. ([Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)), so tokens
  143. per second is what you feel. All four tags are tested with GhostDesk, and the
  144. verdict is the same on every backend on this page. Want the flags in your own
  145. hands? That is [Your own inference
  146. server](#your-own-inference-server--the-tuning-path).
  147. ### 4. Watch your agent work
  148. Open `http://localhost:6080/` in your browser to see the virtual desktop in
  149. real time. No password prompt — the dev posture skips it.
  150. | Service | URL |
  151. |---------|-----|
  152. | MCP server | `http://localhost:3000/mcp` |
  153. | noVNC (browser) | `http://localhost:6080/` |
  154. | Health probes | `http://localhost:3000/health/{live,ready,startup}` |
  155. The probes are what the container's `HEALTHCHECK` reads, and they answer about
  156. the desktop rather than about the processes: `ready` goes down when the
  157. compositor stops answering the window seam, `live` when the connection behind
  158. the virtual pointer and keyboard is gone — a state in which every tool still
  159. replies and none of them does anything.
  160. Give your agent a first prompt to confirm the wiring is right:
  161. > *"Take a screenshot of the desktop, list the installed applications, then
  162. > open Firefox and go to wikipedia.org."*
  163. You should see Firefox launch in the noVNC tab, the URL bar fill in, and the
  164. page load — all under your agent's control.
  165. ### 5. When you're done
  166. ```bash
  167. docker stop ghostdesk-demo && docker rm ghostdesk-demo
  168. ```
  169. The demo run creates no named volume, so this leaves nothing behind.
  170. ---
  171. ## Tools
  172. Fourteen tools, named `verb_noun`, and this is the whole surface — no hidden
  173. endpoint, no second protocol. Defaults are in parentheses, `?` marks an
  174. optional parameter, and every coordinate is a **pixel offset in the last
  175. `screen_shot()`** — measured from that capture's own corner, so a capture taken
  176. with a `region` is answered in coordinates that only mean anything when the
  177. same `region` travels to the act using them. The moment the screen changes,
  178. coordinates computed from the previous capture are stale.
  179. ### Screen
  180. | Tool | Parameters | Returns |
  181. |---|---|---|
  182. | `screen_shot` | `region?` — `{x, y, width, height}`, cropped at native resolution · `format` `"webp"` \| `"png"` (`webp`) · `stabilize` bool (`true`) — wait up to 5 s for the screen to settle · `quality` 1–100 (`50`, WebP only; raise it for fine fonts or design surfaces) | one image block — `{"type": "image", "data": "<base64>", "mimeType": "image/webp"}`. A `region` capture is preceded by a text block naming that rectangle, since its coordinates start at the rectangle's corner |
  183. ### Mouse and keyboard
  184. The seven input tools answer with the same verdict, and `screen_changed` is the
  185. field worth branching on: `false` means the act landed on nothing. It is a
  186. signal, not an error — the answer is a fresh capture, never a retry at the same
  187. coordinates.
  188. ```json
  189. {"action": "Clicked left at (612, 335)", "screen_changed": true, "reaction_time_ms": 180}
  190. ```
  191. | Tool | Parameters |
  192. |---|---|
  193. | `mouse_move` | `x` int · `y` int · `region?` — the `screen_shot` region these were read off |
  194. | `mouse_click` | `x` · `y` · `button` `"left"` \| `"middle"` \| `"right"` (`left`) · `region?` |
  195. | `mouse_double_click` | `x` · `y` · `button` (`left`) · `region?` |
  196. | `mouse_drag` | `from_x` · `from_y` · `to_x` · `to_y` · `button` (`left`) · `region?` |
  197. | `mouse_scroll` | `x` · `y` · `direction` `"up"` \| `"down"` \| `"left"` \| `"right"` (`down`) · `amount` 1–5 wheel notches (`3`) · `region?` |
  198. | `key_type` | `text` string — Unicode, newlines and tabs, layout-independent |
  199. | `key_press` | `keys` string — one chord, `+` between tokens. Modifiers: `ctrl`/`control`, `alt`/`option`, `shift`, `super`/`meta`/`win`/`cmd`/`command`. Named keys: `return`/`enter`, `escape`/`esc`, `backspace`, `delete`, `tab`, `space`, `home`/`end`, `pageup`/`pagedown`, `left`/`right`/`up`/`down`, `f1`–`f12` |
  200. Past a sentence or two, `clipboard_set(text)` plus the paste chord beats
  201. `key_type`: it is instant, and immune to autocomplete and to the app's own key
  202. handlers.
  203. ### Clipboard
  204. | Tool | Parameters | Returns |
  205. |---|---|---|
  206. | `clipboard_get` | — | the clipboard as text — an empty string when it is empty or holds something that is not text |
  207. | `clipboard_set` | `text` string | `Clipboard set (N characters)` |
  208. ### Apps
  209. | Tool | Parameters | Returns |
  210. |---|---|---|
  211. | `app_list` | — | `result[]`, one entry per installed app: `name`, `exec`. This is the launch whitelist, and `exec` is the exact string `app_launch` takes |
  212. | `app_running` | — | `result[]`, one entry per real client window: `app`, `title`, `pid`, `focused` |
  213. | `app_launch` | `command` string — an `exec` from `app_list`, arguments not accepted · `wait_for_window` bool (`true`) | `pid`, `log_file`, `action`, plus `window` and `window_wait_ms` once a window appeared. No image: a window that exists is not a window that has drawn itself, so the frame is `screen_shot`'s to take when the agent is ready to look |
  214. | `app_status` | `pid` int — one returned by `app_launch` · `lines` int (`50`) | `pid`, `running`, `log_file`, `tail` — the tail of the captured stdout/stderr |
  215. `app_list` is a whitelist rather than a hint: `app_launch` refuses anything
  216. outside it, arguments included, whatever name the model sends.
  217. ### On the wire
  218. A call is ordinary MCP over Streamable HTTP — `POST /mcp`, whose `Accept`
  219. header has to name **both** `application/json` and `text/event-stream` or the
  220. endpoint answers `406`.
  221. ```jsonc
  222. // the params of a tools/call request
  223. {"name": "mouse_click", "arguments": {"x": 612, "y": 335}}
  224. ```
  225. The result carries its payload twice — once in `structuredContent`, once as a
  226. text block holding the same JSON, which is what a client older than structured
  227. output reads. `screen_shot` is the exception and answers with an image block.
  228. Two headers are GhostDesk's own: `Authorization: Bearer …`, required once a
  229. cert is mounted ([Secure local run](#secure-local-run-tls--auth)), and
  230. `GhostDesk-Model-Space`, for models that emit normalised coordinates ([Model
  231. requirements](#model-requirements)).
  232. ---
  233. ## Model requirements
  234. Your inference stack must cover four capabilities — all four are mandatory:
  235. 1. **Text + vision** — the agent perceives the desktop through screenshots and
  236. needs a model that can interpret them.
  237. 2. **Tool use** — GhostDesk exposes its tools as function calls; the model must
  238. be able to invoke them.
  239. 3. **MCP client** — the host needs to speak Streamable HTTP MCP to reach the
  240. GhostDesk server.
  241. 4. **WebP image support** — GhostDesk returns screenshots as WebP by default to
  242. keep payloads small and inference fast. A stack that can only decode PNG or
  243. JPEG will not work out of the box.
  244. Points 3 and 4 are where most stacks fall short, and the [Quick
  245. start](#quick-start) answers both: [SpecterChat](https://github.com/YV17labs/SpecterChat)
  246. on the client side, Ollama on the inference side. Below is the path for when
  247. you want the flags in your own hands.
  248. ### Coordinate space — `GhostDesk-Model-Space` header
  249. By default no header is needed: Claude and the other major frontier LLMs work
  250. out of the box. **Qwen3.x** need the client to send `GhostDesk-Model-Space:
  251. 1000` on every MCP request.
  252. Example MCP client config:
  253. ```json
  254. {
  255. "mcpServers": {
  256. "ghostdesk": {
  257. "url": "http://localhost:3000/mcp",
  258. "headers": {
  259. "GhostDesk-Model-Space": "1000"
  260. }
  261. }
  262. }
  263. }
  264. ```
  265. ### Your own inference server — the tuning path
  266. Everything below is optional: the same models as the [Quick
  267. start](#3-choose-where-the-model-runs--cloud-or-local), the same header, and you take on the
  268. flags. Three backends are tested here. Two are llama.cpp forks we
  269. maintain, both kept current with upstream, both adding the WebP decoding
  270. upstream still lacks — the day it lands there, they are archived and this
  271. points at upstream directly. The third is upstream mlx-vlm, on Apple Silicon.
  272. - **[YV17labs/llama-cpp-webp](https://github.com/YV17labs/llama-cpp-webp)** —
  273. branch `feature/webp`. WebP decoding and nothing else on top of upstream, so
  274. it stays close to master and inherits its backend work. It is the faster of
  275. the two forks on Metal and on CUDA — on an Apple Silicon Mac or an NVIDIA
  276. card, this is the one to run.
  277. - **[YV17labs/llama-cpp-turboquant-webp](https://github.com/YV17labs/llama-cpp-turboquant-webp)**
  278. — branch `feature/turboquant-webp`. The same WebP support plus the
  279. turbo-quant KV cache (`--cache-type-v turbo3`). Still maintained and still
  280. tracking upstream, but turbo quant is no longer where the interest is, and
  281. this is no longer the first recommendation.
  282. - **[Blaizzy/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)** — upstream as it
  283. ships, since it decodes WebP already and needs nothing from us. Apple
  284. Silicon only, and the MLX path rather than a llama.cpp one.
  285. Run whatever local model you like — nothing in GhostDesk is pinned to one.
  286. #### The commands
  287. One tested invocation per backend. They do not take the same flags, so each
  288. gets its own rather than one command with a switch — and the model in them is
  289. an example, not a requirement: swap in whatever you run.
  290. **`llama-cpp-webp`** — `--image-min-tokens 1024` is the one that matters for
  291. desktop control: it floors how much of the token budget a screenshot gets, and
  292. a screenshot the model reads at too coarse a scale is where off-target clicks
  293. come from.
  294. ```bash
  295. build/bin/llama-server \
  296. --model ~/Models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  297. --mmproj ~/Models/Qwen3.6-35B-A3B-mmproj-F16.gguf \
  298. --alias 'Qwen3.6-35B-A3B-Q4_K_M' \
  299. --host 127.0.0.1 --port 8080 \
  300. --ctx-size 131072 \
  301. --cache-type-k q8_0 --cache-type-v q8_0 \
  302. --flash-attn on \
  303. --image-min-tokens 1024 \
  304. --reasoning on --reasoning-format deepseek --reasoning-preserve \
  305. --jinja
  306. ```
  307. **`llama-cpp-turboquant-webp`** — the KV cache goes to `--cache-type-v turbo3`,
  308. and `--cache-reuse 256` keeps the prefix across turns, which a desktop session
  309. hits constantly: the conversation grows by one screenshot and one tool result
  310. at a time. `--spec-type draft-mtp` turns on the model's own
  311. multi-token-prediction draft head, so speculative decoding needs no second
  312. model loaded beside it.
  313. ```bash
  314. build/bin/llama-server \
  315. --model ~/Models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  316. --mmproj ~/Models/Qwen3.6-35B-A3B-mmproj-F16.gguf \
  317. --alias 'Qwen3.6-35B-A3B-Q4_K_M' \
  318. --host 127.0.0.1 --port 8080 \
  319. --ctx-size 131072 \
  320. --cache-type-k q8_0 --cache-type-v turbo3 \
  321. --flash-attn on \
  322. --spec-type draft-mtp --spec-draft-n-max 3 \
  323. --reasoning on --reasoning-format deepseek \
  324. --jinja --cache-reuse 256
  325. ```
  326. **`mlx-vlm`** — the flags are the same three ideas under other names:
  327. `--kv-bits 8 --kv-quant-scheme uniform` quantises the KV cache,
  328. `--enable-thinking` turns reasoning on, and `--draft-kind mtp` is the
  329. multi-token-prediction draft head. That last one is the difference that costs
  330. something: here the head is a second set of weights on disk (`--draft-model`),
  331. not a switch on the model already loaded.
  332. ```bash
  333. .venv/bin/mlx_vlm.server \
  334. --model ~/Models/Qwen3.6-35B-A3B-MLX-4bit \
  335. --host 127.0.0.1 --port 8080 \
  336. --kv-bits 8 --kv-quant-scheme uniform \
  337. --enable-thinking \
  338. --draft-model ~/Models/Qwen3.6-35B-A3B-MLX-mtp --draft-kind mtp
  339. ```
  340. Each of the three exposes an OpenAI-compatible endpoint on
  341. `http://127.0.0.1:8080`; point your MCP host's inference backend at it —
  342. SpecterChat's endpoint field takes that URL as is — and remember the
  343. `GhostDesk-Model-Space: 1000` header for the Qwen family.
  344. ---
  345. ## Why GhostDesk?
  346. Browser automation tools (Playwright, Puppeteer, Selenium…) were built for
  347. human test engineers driving a browser with selectors. They do one thing, and
  348. they do it well — inside the browser.
  349. GhostDesk is built from the other end: for **AI agents**, driving **everything
  350. a desktop runs**. Browsers, native apps, IDEs, terminals, office suites, legacy
  351. software, internal tools. If it renders pixels on screen, your agent can see it
  352. and use it — in one conversation, across many applications, without a line of
  353. glue code.
  354. You don't write selectors. You write a prompt:
  355. > *"Open the CRM, export last month's leads as CSV, open LibreOffice Calc,
  356. > build a pivot table, screenshot the chart, and email it to the team."*
  357. The agent opens the browser, logs in, downloads the file, switches to
  358. LibreOffice, processes the data, captures the result, composes the email, sends
  359. it. One prompt, multiple apps, fully autonomous — no glue code, no per-site
  360. scraper, no brittle selector chain.
  361. That is what *agents using a desktop* looks like.
  362. ### Runs on models you can actually host
  363. Desktop control needs to be **fast** — an agent that takes twelve seconds to
  364. decide where to click is unusable. The local path is first-class here, and it
  365. is three concrete things rather than a promise: screenshots ship as WebP so a
  366. capture costs a small payload, the coordinate space a model emits is one header
  367. away, and the inference side is Ollama out of the box — or, for those who
  368. want the flags, the llama.cpp forks that carry the WebP decoding upstream still
  369. lacks. No API bill, and no screenshot of your desktop leaving your network.
  370. Frontier models (Claude, GPT-4o, Gemini) work too and remain the smoothest path
  371. — but they are not the bar. See [Model requirements](#model-requirements) for
  372. the supported stacks and the one coordinate-space setting that matters.
  373. ---
  374. ## How it works
  375. GhostDesk drives a desktop and exposes it as an MCP server. The three ways to
  376. run it are the three columns at the top of this page — the container, a macOS
  377. binary, a Windows binary — and the tool surface is identical in all of them.
  378. The container is what the rest of this README shows unless it says otherwise.
  379. The agent perceives the screen by calling `screen_shot()`, which captures the
  380. full desktop at native resolution and returns it as WebP (or PNG). An optional
  381. `region=` argument can crop to a sub-rectangle when the agent explicitly wants
  382. to narrow its focus.
  383. This works with **any application** — web apps, native apps, legacy software,
  384. Canvas, WebGL.
  385. ### Built in Rust
  386. GhostDesk is a single compiled binary. It links `libc` and nothing else — no
  387. interpreter, no virtual environment, no package tree to harden at build time.
  388. The tool host mounts itself on the HTTP transport, each domain is a service
  389. resolved by type, and the whole dependency graph is verified at boot — a
  390. missing binding is a startup error, never a runtime surprise. The endpoint is
  391. **closed by default**: a guard has to bind before `/mcp` answers anything at
  392. all.
  393. The operating system sits behind five traits — input, screen, windows,
  394. clipboard, application catalogue — and each OS is one directory implementing
  395. those five, under [crates/platform/src](crates/platform/src). Nothing above
  396. that boundary names a desktop, which is what made the second one possible at
  397. all, and the third one routine.
  398. On Linux the compositor is driven from pure Rust: GhostDesk speaks
  399. `zwlr_virtual_pointer_v1` and `zwp_virtual_keyboard_v1` directly over the
  400. Wayland socket, with an XKB keymap it generates on the fly — which is why text
  401. entry produces identical output on a French AZERTY host and a US QWERTY one. On
  402. macOS the same five contracts are answered by Quartz Event Services, the
  403. Accessibility API, `screencapture` and the pasteboard. On Windows they are
  404. answered by `SendInput`, `EnumWindows`, GDI, the Win32 clipboard and the Start
  405. Menu — all in process, because Windows is the one of the three that ships no
  406. capture or clipboard tool to shell out to.
  407. ---
  408. ## Secure local run (TLS + auth)
  409. The Quick start above drops every gate so you can kick the tires in thirty
  410. seconds. The moment you want to expose this to anything beyond your own laptop
  411. — another machine on your LAN, a devcontainer port-forward on an untrusted
  412. network, a teammate's browser — flip to the secured posture: real TLS +
  413. bearer-token auth on MCP + password prompt on noVNC.
  414. GhostDesk couples **TLS and auth**: mount a cert and you get `wss://` +
  415. bearer-token on MCP + a single-password prompt on noVNC (see
  416. [Security](#security) → *Auth ≡ TLS*).
  417. [`mkcert`](https://github.com/FiloSottile/mkcert) issues a browser-trusted cert
  418. for `localhost` in two commands:
  419. ```bash
  420. # Issue a locally-trusted cert (first time only — installs a local CA in your trust store)
  421. mkcert -install
  422. mkdir -p tls
  423. mkcert -cert-file tls/server.crt -key-file tls/server.key localhost 127.0.0.1 ::1
  424. # Generate the MCP and VNC secrets
  425. export GHOSTDESK_AUTH__TOKEN=$(openssl rand -hex 32)
  426. export GHOSTDESK_VNC_PASSWORD=$(openssl rand -hex 16)
  427. ```
  428. Pick a container name that matches the agent's role — `sales-agent`,
  429. `research-agent`, `accounting-agent`… Below we use `my-agent` as a placeholder;
  430. replace it everywhere in the command.
  431. ```bash
  432. # Run the container — cert mounted, TLS + auth enabled everywhere
  433. docker run -d --name ghostdesk-my-agent \
  434. --restart unless-stopped \
  435. --cap-add SYS_ADMIN \
  436. --shm-size 2g \
  437. -p 3000:3000 \
  438. -p 6080:6080 \
  439. -v ghostdesk-my-agent-home:/home/agent \
  440. -v "$PWD/tls/server.crt:/etc/ghostdesk/tls/server.crt:ro" \
  441. -v "$PWD/tls/server.key:/etc/ghostdesk/tls/server.key:ro" \
  442. -e GHOSTDESK_AUTH__TOKEN \
  443. -e GHOSTDESK_VNC_PASSWORD \
  444. -e TZ=America/New_York \
  445. -e LANG=en_US.UTF-8 \
  446. ghcr.io/yv17labs/ghostdesk:latest
  447. echo "MCP token: $GHOSTDESK_AUTH__TOKEN"
  448. echo "VNC password: $GHOSTDESK_VNC_PASSWORD"
  449. ```
  450. Once the container is up, update your MCP client config — same shape as the
  451. demo, now over `https://` with a bearer token:
  452. **Claude Desktop / Claude Code**
  453. ```json
  454. {
  455. "mcpServers": {
  456. "ghostdesk": {
  457. "type": "http",
  458. "url": "https://localhost:3000/mcp",
  459. "headers": {
  460. "Authorization": "Bearer <paste $GHOSTDESK_AUTH__TOKEN here>"
  461. }
  462. }
  463. }
  464. }
  465. ```
  466. **Any other MCP-compatible client** — same URL, plus an `Authorization: Bearer
  467. <token>` header in whatever form your client accepts.
  468. Then open `https://localhost:6080/` in your browser — the `mkcert` CA installed
  469. by `mkcert -install` is already in your trust store, so the browser accepts the
  470. cert with no warning. noVNC will prompt for `$GHOSTDESK_VNC_PASSWORD`.
  471. > **Going to production?** Swap the `mkcert` leaf for a real cert, source both
  472. > secrets from your secret manager, and front port 6080 with an identity-aware
  473. > proxy — [SECURITY.md](SECURITY.md) has the full contract.
  474. > **`--cap-add SYS_ADMIN`** — Required by Electron apps (VS Code, Slack, etc.)
  475. > and other applications that need Linux user namespaces to run their sandbox.
  476. > Safe to remove if you don't need them.
  477. The named volume persists the agent's home directory across restarts — browser
  478. passwords, bookmarks, cookies, downloads, and desktop preferences are all
  479. preserved. On the first run, Docker automatically seeds the volume with the
  480. default configuration from the image.
  481. ---
  482. ## Running many agents
  483. One agent is one container. Two of them share nothing — not the filesystem, not
  484. the desktop, not the clipboard — so a second agent is a second port pair, a
  485. second volume and a second name. What differs between two of them is the system
  486. prompt you give the model, the applications in the image ([Custom
  487. image](#custom-image)), and the networks you attach the container to.
  488. ### Three agents, one compose file
  489. ```yaml
  490. # docker-compose.yml — 3 specialized agents, one command
  491. #
  492. # Prerequisites: the TLS cert + key at ./tls and the two secrets
  493. # (GHOSTDESK_AUTH__TOKEN, GHOSTDESK_VNC_PASSWORD) in your environment or a
  494. # .env file. Generate both exactly as shown in the Secure local run
  495. # section above. See SECURITY.md for the production secret-handling
  496. # contract.
  497. x-ghostdesk-defaults: &ghostdesk-defaults
  498. image: ghcr.io/yv17labs/ghostdesk:latest
  499. restart: unless-stopped
  500. cap_add: [SYS_ADMIN]
  501. shm_size: 2g
  502. environment:
  503. - GHOSTDESK_AUTH__TOKEN
  504. - GHOSTDESK_VNC_PASSWORD
  505. - TZ=America/New_York
  506. - LANG=en_US.UTF-8
  507. services:
  508. sales-agent:
  509. <<: *ghostdesk-defaults
  510. container_name: ghostdesk-sales-agent
  511. ports: ["3001:3000", "6081:6080"]
  512. volumes:
  513. - ghostdesk-sales-agent-home:/home/agent
  514. - ./tls/server.crt:/etc/ghostdesk/tls/server.crt:ro
  515. - ./tls/server.key:/etc/ghostdesk/tls/server.key:ro
  516. research-agent:
  517. <<: *ghostdesk-defaults
  518. container_name: ghostdesk-research-agent
  519. ports: ["3002:3000", "6082:6080"]
  520. volumes:
  521. - ghostdesk-research-agent-home:/home/agent
  522. - ./tls/server.crt:/etc/ghostdesk/tls/server.crt:ro
  523. - ./tls/server.key:/etc/ghostdesk/tls/server.key:ro
  524. accounting-agent:
  525. <<: *ghostdesk-defaults
  526. container_name: ghostdesk-accounting-agent
  527. ports: ["3003:3000", "6083:6080"]
  528. volumes:
  529. - ghostdesk-accounting-agent-home:/home/agent
  530. - ./tls/server.crt:/etc/ghostdesk/tls/server.crt:ro
  531. - ./tls/server.key:/etc/ghostdesk/tls/server.key:ro
  532. volumes:
  533. ghostdesk-sales-agent-home:
  534. ghostdesk-research-agent-home:
  535. ghostdesk-accounting-agent-home:
  536. ```
  537. ```bash
  538. docker compose up -d
  539. ```
  540. Each service is the same image with its name, its ports and its volume changed.
  541. Per agent, that costs:
  542. | Per agent | What it takes |
  543. |---|---|
  544. | **Two published ports** | `3000` for MCP, `6080` for noVNC — one pair per container, mapped to whatever the host has free |
  545. | **One named volume** | the agent's `/home/agent`: browser profile, cookies, downloads, desktop settings, kept across restarts |
  546. | **One `shm_size: 2g`** | shared memory for the browser and the other GPU-accelerated apps. It is a cap rather than a reservation — pages are allocated as they are touched — but every container may claim up to that much of the host's RAM |
  547. | **One desktop** | Sway, mako, wayvnc, websockify and the MCP server, under one supervisord |
  548. Nothing coordinates the instances: no scheduler, no shared state, no leader.
  549. Ten agents are ten `docker run`s, and stopping one is `docker rm`.
  550. ### Container isolation
  551. The container boundary is the only isolation GhostDesk has, and it is Docker's
  552. rather than the server's: separate filesystem, process and network namespaces,
  553. one volume per agent, and a `docker rm` that takes the desktop and everything
  554. the agent did to it. The MCP port and the noVNC port are the two doors through
  555. that boundary, which is why [Secure local run](#secure-local-run-tls--auth)
  556. puts TLS and a credential on both.
  557. Two things it does not give you, and both belong to the deployment.
  558. Segmentation *between* agents is the first: a container reaches whatever the
  559. networks you attached it to reach, so an agent that must not see the internet
  560. is one you attach only to Docker networks with no route off the host. Per-user
  561. identity on either door is the second — the token and the VNC password are one
  562. credential each, shared by every caller.
  563. [SECURITY.md](SECURITY.md#threat-model) draws the whole line, in scope against
  564. out of scope.
  565. ### Watching one work
  566. Every instance serves its own noVNC, so supervision is one browser tab per
  567. agent — `https://localhost:6081/` for the sales agent above, `6082` for
  568. research, `6083` for accounting (the compose file mounts a cert, so those are
  569. the secured posture's URLs). The tab is not read-only: take the mouse and
  570. keyboard whenever you want, and the agent's next `screen_shot()` sees whatever
  571. you left on screen.
  572. ---
  573. ## Custom image
  574. The `base` tag provides GhostDesk without any pre-installed GUI application —
  575. just the virtual desktop, VNC, and the MCP server. Use it to build your own
  576. image with only the tools you need:
  577. ```dockerfile
  578. FROM ghcr.io/yv17labs/ghostdesk:base
  579. RUN apt-get update \
  580. && apt-get install -y --no-install-recommends \
  581. chromium-browser \
  582. libreoffice-calc \
  583. && rm -rf /var/lib/apt/lists/*
  584. ```
  585. ```bash
  586. docker build -t my-agent .
  587. ```
  588. See the project's [Dockerfile](Dockerfile) for a complete example.
  589. | Tag | Description |
  590. |-----|-------------|
  591. | `latest`, `X.Y.Z`, `X.Y` | Full image — Firefox, foot terminal, mousepad, galculator, passwordless sudo |
  592. | `base`, `base-X.Y.Z`, `base-X.Y` | Minimal image — no GUI app, meant to be extended |
  593. ---
  594. ## Run on macOS, without the container
  595. The container ships a Linux desktop. GhostDesk is also just a binary, and on
  596. macOS that binary drives **the Mac in front of you** — the same tools, the same
  597. MCP endpoint, nothing in between. The five OS seams are answered by Quartz
  598. Event Services for the pointer and keyboard, the Accessibility API for windows,
  599. `screencapture` for frames, the pasteboard for the clipboard, and `.app`
  600. bundles for the catalogue.
  601. > **There is no sandbox on this path.** Every isolation guarantee in [Container
  602. > isolation](#container-isolation) belongs to the container. A native run hands
  603. > the agent your real mouse, your real keyboard, your real screen and your real
  604. > applications, with your own permissions. Run it on a machine you are willing
  605. > to hand over, and watch it.
  606. ### 1. Build and install
  607. From a clone of the repository — [Build from source](#build-from-source) has
  608. the toolchain it needs:
  609. ```bash
  610. cargo install --path apps/ghostdesk --locked
  611. ```
  612. ### 2. Grant the two permissions
  613. macOS gates input and capture behind privacy settings that **cannot be
  614. requested from code**, so you grant them by hand, once, in **System Settings ▸
  615. Privacy & Security**:
  616. | Setting | What it buys |
  617. |---------|--------------|
  618. | **Accessibility** | `mouse_*`, `key_*`, and every window operation |
  619. | **Screen Recording** | `screen_shot`, and window titles |
  620. Miss one and the affected tools fail naming the setting to open, rather than
  621. returning a black image or silently doing nothing.
  622. > **The grant is bound to the binary's path *and* its signature**, which is
  623. > macOS's rule and not ours: rebuild `ghostdesk` and the grant is revoked, so
  624. > both permissions have to be re-granted after every `cargo install`. Nothing
  625. > can script this away — TCC exists precisely so that no process can grant
  626. > itself the thing.
  627. ### 3. Run it
  628. ```bash
  629. NESTRS_ENV_PREFIX=GHOSTDESK GHOSTDESK_IDLE__TIMEOUT_SECS=0 ghostdesk
  630. ```
  631. Both variables are load-bearing:
  632. - `NESTRS_ENV_PREFIX=GHOSTDESK` has to be on the process — see
  633. [Configuration](#configuration). Without it every setting is read under its
  634. stock `NESTRS_*` name instead, and none of the `GHOSTDESK_*` names below
  635. reach the server.
  636. - `GHOSTDESK_IDLE__TIMEOUT_SECS=0` disarms the idle sweep. Armed, thirty
  637. minutes of MCP silence closes every open window — the right behaviour for a
  638. disposable container desktop, and the wrong one for your laptop.
  639. The server binds `127.0.0.1:3000` and serves with no token (posture
  640. `loopback_open`); point your MCP client at `http://localhost:3000/mcp` exactly
  641. as in [Connect your agent](#2-connect-your-agent). There is no noVNC endpoint — the
  642. desktop is the one you are looking at.
  643. ---
  644. ## Run on Windows, without the container
  645. The same binary once more, driving **the PC in front of you** — the same tools
  646. again, the same MCP endpoint. The five OS seams are answered by `SendInput` for
  647. the pointer and keyboard, `EnumWindows` and `WM_CLOSE` for windows, GDI for
  648. frames, the Win32 clipboard, and Start Menu shortcuts for the catalogue.
  649. Nothing here shells out to a helper process. Linux has `grim` and
  650. `wl-clipboard`, macOS has `screencapture` and `pbcopy`; Windows ships neither,
  651. so this is the one host that captures and copies in process — which also means
  652. a scaled capture is scaled by the *blit*, not by decoding and resizing a
  653. full-resolution frame afterwards.
  654. > **There is no sandbox on this path.** Every isolation guarantee in [Container
  655. > isolation](#container-isolation) belongs to the container. A native run hands
  656. > the agent your real mouse, your real keyboard, your real screen and your real
  657. > applications, with your own permissions. Run it on a machine you are willing
  658. > to hand over, and watch it.
  659. ### 1. Build and install
  660. GhostDesk links a WebP encoder written in C, so the build needs a C toolchain.
  661. Install the **Visual Studio Build Tools** with the *Desktop development with
  662. C++* workload (Visual Studio itself works too), then, from a clone of the
  663. repository:
  664. ```powershell
  665. cargo install --path apps/ghostdesk --locked
  666. ```
  667. ### 2. Grant nothing — but run it as yourself
  668. Windows puts none of this behind a privacy setting, so unlike macOS there is
  669. nothing to click. What it gates instead is **integrity level**, and two rules
  670. follow from that:
  671. - **Run GhostDesk as the signed-in user, in an interactive session.** A Windows
  672. service lives in session 0, which has no desktop at all. GhostDesk refuses to
  673. boot there, naming the reason, rather than accepting clicks that go nowhere.
  674. - **An unelevated GhostDesk cannot reach an elevated window** — Task Manager,
  675. an installer, anything started with *Run as administrator*. Windows discards
  676. that input in silence, which is exactly why the boot check exists. Starting
  677. GhostDesk elevated lifts the restriction and hands the agent an
  678. administrator's desktop; do that deliberately or not at all.
  679. While a UAC prompt or the lock screen is in front, the session belongs to
  680. Winlogon and **no** application can drive it — GhostDesk included. It reports
  681. that instead of reporting a healthy desk, so a supervisor sees a server that
  682. cannot work rather than one that appears to.
  683. ### 3. Run it
  684. ```powershell
  685. $env:NESTRS_ENV_PREFIX = "GHOSTDESK"
  686. $env:GHOSTDESK_IDLE__TIMEOUT_SECS = "0"
  687. ghostdesk
  688. ```
  689. Both variables are load-bearing, for the same two reasons they are on macOS:
  690. `NESTRS_ENV_PREFIX=GHOSTDESK` is what makes every `GHOSTDESK_*` name below
  691. reach the server, and `GHOSTDESK_IDLE__TIMEOUT_SECS=0` disarms the idle sweep,
  692. which would otherwise close your own windows after thirty minutes of MCP
  693. silence.
  694. The server binds `127.0.0.1:3000` and serves with no token (posture
  695. `loopback_open`); point your MCP client at `http://localhost:3000/mcp` exactly
  696. as in [Connect your agent](#2-connect-your-agent). There is no noVNC endpoint — the
  697. desktop is the one you are looking at.
  698. ---
  699. ## What differs from the container
  700. | | Container (Linux) | Native (macOS) | Native (Windows) |
  701. |---|---|---|---|
  702. | Desktop | virtual, disposable, sandboxed | yours | yours |
  703. | Primary modifier | `ctrl` | `cmd` | `ctrl` |
  704. | App catalogue | `.desktop` entries | `.app` bundles in `/Applications`, `/System/Applications`, their `Utilities`, and `~/Applications` | Start Menu shortcuts, machine-wide and per-user |
  705. | Supervision | noVNC on `:6080` | your own screen | your own screen |
  706. | Screen geometry | `GHOSTDESK_SCREEN__WIDTH` / `_HEIGHT` | the main display, at native pixel size | the primary display, at native pixel size |
  707. | Permissions | none | Accessibility + Screen Recording, granted by hand | none to grant — integrity level decides what it can reach |
  708. | Windows are closed by | Sway IPC | the Accessibility close button | `WM_CLOSE` |
  709. The modifier is not something you configure, and it is not cosmetic. The server
  710. publishes the desktop and its primary modifier in the tool descriptions, built
  711. from the same constant the key table presses, so the model is told `cmd+c` on
  712. macOS and `ctrl+c` on Linux and Windows. Every modifier *name* resolves on all
  713. three desktops — `ctrl`, `alt`, `option`, `super`, `meta`, `win`, `cmd`,
  714. `command` — but on macOS `ctrl+c` presses Control and puts a control character
  715. in the field, which is why the instruction is published rather than assumed.
  716. The app catalogue is a whitelist on all three, and it is what an agent may
  717. launch: nothing outside it can be started, whatever name the model sends. On
  718. Windows that means the Start Menu, and a Store application whose shortcut
  719. points at a package rather than at an executable stays out of it — a catalogue
  720. that listed what it cannot start would be a whitelist that lies.
  721. > **Running the binary on a Linux host instead of the container** works the
  722. > same way, with the Wayland stack's own expectations: a Sway session for the
  723. > window seam, `grim` for capture, `wl-clipboard` for the clipboard — see
  724. > [Build from source](#build-from-source) for the command and the caveat.
  725. ---
  726. ## Configuration
  727. Every variable GhostDesk reads is namespaced under `GHOSTDESK_*`. Standard
  728. POSIX variables (`TZ`, `LANG`) are kept as-is so the existing Unix ecosystem
  729. keeps working.
  730. The image sets `NESTRS_ENV_PREFIX=GHOSTDESK`, and that single variable is what
  731. makes every setting below read `GHOSTDESK_HTTP__PORT` rather than
  732. `NESTRS_HTTP__PORT`. There is no second spelling and no translation layer — one
  733. name, one place to look it up.
  734. Read them as `GHOSTDESK_<NAMESPACE>__<KEY>`: the double underscore separates
  735. the namespace from the setting. `http` and `mcp` are the server's transport
  736. namespaces; `screen`, `idle` and `auth` are GhostDesk's, one per feature module
  737. that owns settings. A single underscore (`GHOSTDESK_VNC_PASSWORD`) marks a
  738. container-level knob the entrypoint consumes itself, never reaching the server.
  739. `NESTRS_ENV_PREFIX` is the one name no prefix can rename, and it has to be on
  740. the process before the server starts — a `.env` file is read after it has
  741. already chosen which cascade to read. Both images bake it and the `Justfile`
  742. exports it, so a container run and a `nestrs run dev` both carry it; a binary
  743. you start any other way needs `NESTRS_ENV_PREFIX=GHOSTDESK` in its environment,
  744. or every variable below is read under its stock `NESTRS_*` name instead.
  745. ### Secrets (required under TLS — the prod container refuses to boot without them)
  746. | Variable | Description |
  747. |----------|-------------|
  748. | `GHOSTDESK_AUTH__TOKEN` | Bearer token required on every MCP request. Generate with `openssl rand -hex 32`. |
  749. | `GHOSTDESK_VNC_PASSWORD` | Password for wayvnc. RFB security type 2 carries a password and no username, so the noVNC overlay prompts for this one value. Generate with `openssl rand -hex 16`. |
  750. Both are plain environment variables. Wire them from your secret store
  751. (`secretKeyRef` on Kubernetes, Docker secrets / Vault / AWS SM on compose) —
  752. see [SECURITY.md](SECURITY.md#secrets-handling--rotation) for the full
  753. contract.
  754. ### Runtime knobs
  755. | Variable | Default | Description |
  756. |----------|---------|-------------|
  757. | `GHOSTDESK_HTTP__PORT` | `3000` | MCP server listening port |
  758. | `GHOSTDESK_HTTP__HOST` | `127.0.0.1` (standalone) / `0.0.0.0` (container) | Bind address for the MCP endpoint. Defaults to loopback per [MCP transports spec](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http); the container's entrypoint exports `0.0.0.0` so Docker's port-publishing layer can reach it. |
  759. | `GHOSTDESK_HTTP__CORS_ORIGINS` | *(empty)* | Comma-separated list of `Origin` headers accepted from browser clients (e.g. `https://app.example.com,https://localhost:8080`). Non-browser clients (Claude Desktop, SDKs, `curl`) send no `Origin` and are always allowed. Required for any browser-based MCP UI: without it no CORS layer is mounted at all, so the browser — not the server — refuses the response. The anti-DNS-rebinding control is the next row, and it is on by default. |
  760. | `GHOSTDESK_MCP__ALLOWED_HOSTS` | `localhost,127.0.0.1,::1` | Comma-separated `Host` header allow-list for the MCP endpoint. A request whose `Host` is not listed gets HTTP 403 — this is what stops a page on an attacker's origin from pointing its own hostname at a locally-running GhostDesk and calling your tools. **A deployment reached under a real hostname must name itself here.** Do not empty the list. |
  761. | `GHOSTDESK_TLS_CERT` | `/etc/ghostdesk/tls/server.crt` | Path to the TLS certificate. When the file exists, `websockify` and the MCP server auto-switch to `wss://` / `https://`. See [Security](#security). |
  762. | `GHOSTDESK_TLS_KEY` | `/etc/ghostdesk/tls/server.key` | Path to the TLS private key (matching `GHOSTDESK_TLS_CERT`). |
  763. | `GHOSTDESK_SCREEN__WIDTH` | `1280` | Virtual screen width in pixels. The fallback for a virtual screen with no display to ask — ignored on macOS and Windows, where the display reports its own pixel size. |
  764. | `GHOSTDESK_SCREEN__HEIGHT` | `1024` | Virtual screen height in pixels. Same fallback rule as the row above. |
  765. | `GHOSTDESK_IDLE__TIMEOUT_SECS` | `1800` | Seconds of MCP silence before all open client windows (Firefox, foot, mousepad…) are closed to free memory. Sway, mako, wayvnc and the MCP server itself are spared. Set to `0` to disable — **which you want on any native run**, macOS or Windows, where the windows it would close are your own. |
  766. | `TZ` | `America/New_York` | IANA timezone (POSIX standard, e.g. `Europe/Paris`) |
  767. | `LANG` | `en_US.UTF-8` | POSIX locale (e.g. `fr_FR.UTF-8`) |
  768. `GHOSTDESK_TLS_CERT` / `_KEY` are the exception that proves the rule: the
  769. entrypoint *probes* those paths to decide the posture, then hands the one it
  770. found to the server as `GHOSTDESK_HTTP__TLS_CERT_FILE`. Everything else you set
  771. reaches the server verbatim. A malformed value fails the boot naming the
  772. variable rather than silently falling back to a default.
  773. ### Pinned values (not configurable)
  774. | Variable | Value | Rationale |
  775. |----------|-------|-----------|
  776. | `GHOSTDESK_VNC_ADDRESS` | `127.0.0.1` | wayvnc is locked to loopback inside the container's netns; the VNC port is only reachable via the noVNC bridge on 6080. Override attempts are logged and ignored — see [SECURITY.md](SECURITY.md#transport-security). |
  777. ---
  778. ## Security
  779. GhostDesk owns two things: **transport encryption** and **authentication**.
  780. Everything else (rate limiting, SSO, WAF, session recording, brute-force
  781. protection, per-user identity on noVNC) is a reverse-proxy concern — the
  782. container is designed to run behind one, not directly on the internet.
  783. That posture, and the threat model behind it, assume the container. A binary
  784. run directly on [macOS](#run-on-macos-without-the-container) or
  785. [Windows](#run-on-windows-without-the-container) has no container boundary to
  786. lean on — see either section for what that costs you.
  787. The full threat model, the *Auth ≡ TLS* posture switch, the wayvnc
  788. RFB-type-2-inside-`wss://` rationale, the secrets handling contract, and the
  789. exhaustive in-scope / out-of-scope table all live in
  790. **[SECURITY.md](SECURITY.md)** — single source of truth. Start there before
  791. deploying to anything you don't fully trust.
  792. Reporting a vulnerability? Use GitHub's [private security
  793. advisory](../../security/advisories) — see [SECURITY.md §
  794. Reporting](SECURITY.md#reporting-security-vulnerabilities).
  795. ---
  796. ## Troubleshooting
  797. ### My agent's clicks land off-target by a huge margin
  798. Almost always a coordinate-space mismatch. Frontier models (Claude, GPT-4o,
  799. Gemini) need no header (default pass-through); the Qwen vision family needs the
  800. client to send `GhostDesk-Model-Space: 1000` on every MCP request. Full
  801. rationale in [Model requirements](#model-requirements) → *Coordinate space*.
  802. ### The container refuses to start with a secrets error
  803. The prod posture (cert mounted) **requires** both `GHOSTDESK_AUTH__TOKEN` and
  804. `GHOSTDESK_VNC_PASSWORD` to be set — GhostDesk refuses to boot without them on
  805. purpose, to prevent an unauthenticated prod container. Generate them as shown
  806. in [Secure local run](#secure-local-run-tls--auth) and pass them with `-e`. The
  807. demo posture (no cert) has no such requirement.
  808. ### noVNC shows a black screen or the desktop renders with graphical glitches
  809. You're probably short on shared memory. Browsers and other GPU-accelerated apps
  810. inside the container need a reasonable `/dev/shm` — `--shm-size 2g` is the
  811. baseline in every example and should not be trimmed. If you already have
  812. `--shm-size 2g`, check the container logs for wayvnc or compositor errors.
  813. ### On macOS, clicks do nothing or screenshots fail
  814. The two privacy permissions are missing. Grant **Accessibility** (input and
  815. windows) and **Screen Recording** (capture) in System Settings ▸ Privacy &
  816. Security, then restart the server — macOS applies the grant at process start.
  817. The failing tool names the setting it needs in its error, so read that rather
  818. than guessing which of the two it is. If both were working until you rebuilt:
  819. the grant is bound to the binary's signature, and a rebuild revokes it. Full
  820. walkthrough: [Run on macOS](#run-on-macos-without-the-container).
  821. ### On Windows, clicks and keystrokes go nowhere
  822. Three causes, and the server's own error names which one. **A UAC prompt or the
  823. lock screen is in front** — the session belongs to Winlogon and no application
  824. can drive it, so wait for it to be dismissed. **GhostDesk was started as a
  825. Windows service** — session 0 has no desktop; start it as the signed-in user
  826. instead. **The target window is elevated** — Task Manager, an installer,
  827. anything started with *Run as administrator* — and an unelevated process cannot
  828. reach it; everything else on the desktop still works. Full walkthrough: [Run on
  829. Windows](#run-on-windows-without-the-container).
  830. ### On Windows, an application is missing from `app_list`
  831. The catalogue is the Start Menu, and an entry only counts when its shortcut
  832. resolves to an executable that exists. Microsoft Store applications point at a
  833. package identity instead, so they are not listed and cannot be launched —
  834. install the desktop build of the application if the agent needs to drive it.
  835. ### Firefox / Electron apps fail to launch or crash immediately
  836. Electron-based apps (VS Code, Slack, Discord…) need Linux user namespaces for
  837. their sandbox. Add `--cap-add SYS_ADMIN` to your `docker run` (already present
  838. in the Secure local run example). Firefox itself works without it.
  839. ---
  840. ## Build from source
  841. The workspace builds with a plain `cargo build`.
  842. [`rust-toolchain.toml`](rust-toolchain.toml) pins the channel, so `rustup`
  843. resolves the same compiler everyone else has and there is no version to pick.
  844. The one native dependency is the WebP encoder, which is C — already buildable
  845. on a Linux dev box and with Xcode's command-line tools; on Windows it is the
  846. Visual Studio Build Tools with the *Desktop development with C++* workload.
  847. ```bash
  848. git clone https://github.com/YV17labs/GhostDesk.git
  849. cd GhostDesk
  850. cargo build --release # -> target/release/ghostdesk
  851. cargo test --workspace
  852. cargo clippy --workspace --all-targets -- -D warnings
  853. cargo fmt --all --check
  854. ```
  855. The [Justfile](Justfile) wraps those and a few more — `just --list` prints the
  856. set: `just check`, `just lint`, `just test unit`, and `just stack`, which
  857. brings the desktop up (Sway, mako, wayvnc, websockify, the MCP server) under
  858. one supervisord. `cargo install --path apps/ghostdesk --locked` is the same
  859. build installed into `~/.cargo/bin`, and it is the command the
  860. [macOS](#run-on-macos-without-the-container) and
  861. [Windows](#run-on-windows-without-the-container) sections call.
  862. ### Running what you built, on a Linux host
  863. The container ships a desktop; a Linux binary expects to find one. Give it a
  864. Sway session on the Wayland socket for the window seam, `grim` for capture and
  865. `wl-clipboard` for the clipboard, then:
  866. ```bash
  867. NESTRS_ENV_PREFIX=GHOSTDESK GHOSTDESK_IDLE__TIMEOUT_SECS=0 target/release/ghostdesk
  868. ```
  869. Both variables are load-bearing for the same two reasons they are on
  870. [macOS](#run-on-macos-without-the-container): the prefix is what makes every
  871. `GHOSTDESK_*` name reach the server, and the idle sweep would otherwise close
  872. your own windows after thirty minutes of MCP silence. The container exists so
  873. that you do not have to assemble that stack — reach for it unless you are
  874. working on the Linux backend itself.
  875. ### The repository
  876. ```
  877. apps/ghostdesk/ the binary: the composition root, and the endpoint's app-local half
  878. crates/features/ one folder per domain — auth, clipboard, host, idle, input,
  879. programs, screen — each with its own mcp/ adapter, the only
  880. place that knows about the wire
  881. crates/platform/ the OS substrate, and no framework types: input.rs, screen.rs,
  882. window.rs, clipboard.rs and desktop.rs are the five contracts,
  883. host.rs picks the backend for the compile target, and linux/,
  884. macos/ and windows/ are the three that answer them
  885. docker/ base image, services, entrypoint
  886. ```
  887. A fourth desktop is a fourth directory under `crates/platform/src/` and no
  888. change above it — that is the boundary [Built in Rust](#built-in-rust)
  889. describes, read from the filesystem.
  890. [CONTRIBUTING.md](CONTRIBUTING.md) carries the devcontainer setup, the test
  891. layout and the PR process; [AGENTS.md](AGENTS.md) carries the naming rules the
  892. workspace is checked against; [CHANGELOG.md](CHANGELOG.md) records what changed
  893. per release.
  894. ---
  895. ## License
  896. **[Functional Source License, Version 1.1, ALv2 Future
  897. License](https://fsl.software/) (FSL-1.1-ALv2)** — see [LICENSE](LICENSE) for
  898. the authoritative terms.
  899. **What this means in practice** *(informal summary — the LICENSE file governs;
  900. this is not legal advice)*:
  901. - **Permitted purposes** cover the use cases that matter for the vast majority
  902. of users: internal use and access inside your company, non-commercial
  903. education and research, and professional services you provide to a licensee
  904. who is using GhostDesk in accordance with the license. Self-hosting GhostDesk
  905. to run your own agents — even commercial, revenue-generating workflows that
  906. power *your* product — is a permitted internal use.
  907. - **Competing Use is prohibited.** You may not make GhostDesk available to
  908. others in a commercial product or service that substitutes for GhostDesk,
  909. substitutes for any product or service the project offers using GhostDesk, or
  910. provides the same or substantially similar functionality. In short: you
  911. cannot take GhostDesk and rebrand it, host it as a paid service, or build a
  912. competing desktop-automation-for-agents product from it.
  913. - **Apache 2.0 in two years.** Each released version of GhostDesk becomes
  914. available under the Apache License 2.0 on the second anniversary of its
  915. release, automatically and irrevocably. The Competing Use restriction only
  916. applies for those first two years.
  917. **Commercial licensing.** If your intended use falls under Competing Use — you
  918. want to resell GhostDesk, offer it as a managed service, or build a competing
  919. product — contact the maintainers to discuss a commercial license before
  920. deploying. Open a GitHub issue or reach out directly; we are happy to talk.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.