Skip to content
Back to search

som-window-mcp

Read-only MCP server that lets AI agents see Windows desktop UI: window list, UI Automation tree, and occlusion-proof screenshots.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nikita-pugach--som-window-mcp/badge.svg)](/listings/nikita-pugach--som-window-mcp)

HTML

<a href="/listings/nikita-pugach--som-window-mcp"><img src="/api/listings/nikita-pugach--som-window-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

uvx som-window-mcp

Or install the binary once:

uv tool install som-window-mcp

`uv tool install` places `som-window-mcp` in `~/.local/bin`; run `uv tool update-shell` once if the command is not found afterwards.

With pip:

pip install som-window-mcp

From a clone, for development:

git clone https://github.com/Nikita-Pugach/som-window-mcp
cd som-window-mcp
uv tool install .

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 229 lines
  1. # som-window-mcp
  2. **Read-only eyes for Windows desktop UI.** Let an AI agent see a window —
  3. including one covered by other windows — without guessing from full-screen
  4. pixels, and without giving it a mouse.
  5. [![PyPI](https://img.shields.io/pypi/v/som-window-mcp)](https://pypi.org/project/som-window-mcp/)
  6. [![Python versions](https://img.shields.io/pypi/pyversions/som-window-mcp)](https://pypi.org/project/som-window-mcp/)
  7. [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
  8. [![tests](https://github.com/Nikita-Pugach/som-window-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/Nikita-Pugach/som-window-mcp/actions/workflows/ci.yml)
  9. > Status: **v0.3** — five read-only tools: window list, accessibility tree,
  10. > single-element inspection, plain capture, and the numbered Set-of-Mark overlay.
  11. ![Explorer with every control numbered, captured while covered by other windows](docs/demo.gif)
  12. *One window, captured while it was covered by another. The marks come from the
  13. accessibility tree, not from OCR, so every number maps to an exact screen rect.*
  14. ## Why
  15. When you ask an agent to help with a desktop app, it has two bad options today:
  16. 1. **Screenshot the whole screen.** Expensive in tokens, and it captures your
  17. private windows along with the target one. If the app is behind your
  18. terminal, the agent cannot see it at all.
  19. 2. **Install a full desktop clicker.** That hands the model a mouse and a
  20. keyboard. Any text on screen can then talk the model into pressing `Win+R`
  21. and running something. It also tends to hang on blocked Windows calls.
  22. This server takes a third path: **look, never touch.** It reads the real Windows
  23. accessibility tree for structure, and `PrintWindow` for pixels, so a covered
  24. window still captures cleanly and only that window is in the image.
  25. ## Tools
  26. | Tool | What it returns |
  27. |---|---|
  28. | `list_windows` | Visible windows: `hwnd`, process, physical-pixel rect, whether the window is covered (`occluded`), whether it needs an elevated server (`elevated`) |
  29. | `get_ui_tree` | Compact text list of interactive elements — path, type, name, automation id, screen rect — with zoomable `max_depth` / `max_nodes` |
  30. | `inspect_element` | Full content of one element by path: value, bounded text, and a summary of its immediate children |
  31. | `capture_window` | One window's pixels, even while covered, plus the screen position of image pixel `(0,0)` |
  32. | `capture_som` | The same capture with a **numbered box on every interactive element**, plus a legend that maps each number to a real screen rect |
  33. There is no click, no typing, no mouse movement, and no code execution anywhere
  34. in this package.
  35. ## Set-of-Mark
  36. `capture_som` draws a numbered box on each interactive control. The boxes come
  37. from the accessibility tree, not from OCR or a detector, so they are pixel-exact
  38. and the legend carries real coordinates:
  39. ```text
  40. 7 0.4.1 Button "Открыть SDS" @450,320 90x28
  41. ```
  42. A vision model answers "7" instead of estimating a pixel; a clicker uses the
  43. rect and never looks at the image at all. Numbering follows reading order
  44. (top-to-bottom, left-to-right), and the tree is walked breadth-first so a fixed
  45. node budget reaches toolbars and lists before it disappears into one menu.
  46. Three filters keep the overlay readable:
  47. - **duplicates** — Windows Terminal exposes its caption buttons twice; overlapping
  48. controls of similar size are collapsed to the named one
  49. - **cells inside rows** — Explorer reports four column values per list row; the
  50. row is marked, its cells are not
  51. - **budget** — `max_marks` caps the count, and badge placement avoids collisions
  52. (verified: zero overlapping badges on a 60-mark Explorer window)
  53. ## Requirements
  54. - Windows 10/11 (x64)
  55. - Python 3.10+
  56. ## Install
  57. ```sh
  58. uvx som-window-mcp
  59. ```
  60. Or install the binary once:
  61. ```sh
  62. uv tool install som-window-mcp
  63. ```
  64. `uv tool install` places `som-window-mcp` in `~/.local/bin`; run
  65. `uv tool update-shell` once if the command is not found afterwards.
  66. With pip:
  67. ```sh
  68. pip install som-window-mcp
  69. ```
  70. From a clone, for development:
  71. ```sh
  72. git clone https://github.com/Nikita-Pugach/som-window-mcp
  73. cd som-window-mcp
  74. uv tool install .
  75. ```
  76. ## Configure
  77. ### OpenCode
  78. ```sh
  79. opencode mcp add som-window --global -- som-window-mcp
  80. ```
  81. Or by hand in `opencode.jsonc`:
  82. ```jsonc
  83. {
  84. "$schema": "https://opencode.ai/config.json",
  85. "mcp": {
  86. "servers": {
  87. "som-window": {
  88. "type": "local",
  89. "command": ["som-window-mcp"],
  90. },
  91. },
  92. },
  93. }
  94. ```
  95. ### Claude Code
  96. ```sh
  97. claude mcp add som-window -- som-window-mcp
  98. ```
  99. ### Any other MCP client
  100. ```json
  101. {
  102. "mcpServers": {
  103. "som-window": {
  104. "command": "som-window-mcp"
  105. }
  106. }
  107. }
  108. ```
  109. Running from a checkout instead of an installed package:
  110. ```json
  111. {
  112. "mcpServers": {
  113. "som-window": {
  114. "command": "python",
  115. "args": ["-m", "som_window_mcp.server"],
  116. "env": { "PYTHONPATH": "C:\\dev\\som-window-mcp\\src" }
  117. }
  118. }
  119. }
  120. ```
  121. ## Coordinate contract
  122. Everything is in **physical screen pixels**, and the server sets
  123. `PER_MONITOR_AWARE_V2` at startup, so coordinates match the screen on 100%,
  124. 125%, 150% and mixed-DPI multi-monitor setups.
  125. `capture_window` reports `origin`: the screen coordinate of image pixel `(0,0)`.
  126. ```text
  127. screen_x = origin_x + pixel_x
  128. screen_y = origin_y + pixel_y
  129. ```
  130. That is what a clicker needs, and it is the reason the capture is cropped by
  131. `DWMWA_EXTENDED_FRAME_BOUNDS`: without the crop every image carries a 9 px black
  132. frame from the invisible DWM resize border and every coordinate is off.
  133. ## Known limitations
  134. Measured, not guessed — see [`docs/uia-matrix.md`](docs/uia-matrix.md) for the
  135. raw numbers.
  136. | Case | Behaviour |
  137. |---|---|
  138. | Elevated window, non-elevated server | Windows blocks UIA across integrity levels. The tool says so explicitly instead of returning an empty tree. Run the server elevated to read it. |
  139. | Chrome, VS Code, Slack, Discord (Chromium/Electron) | No UI tree at all unless the app was started with `--force-renderer-accessibility` (8 nodes vs 173 measured). Screenshots work either way. |
  140. | Qt / Delphi apps | Partial or absent UIA tree. Screenshots still work. |
  141. | Minimised windows, some GPU surfaces | `PrintWindow` returns a blank frame; this is detected and reported instead of returning a useless image. |
  142. | Dense windows | About 2.5 ms per UIA node. Use `max_nodes`; a 4000-node walk takes ~10 s. |
  143. Read-only still means "sees your screen": captures are sent to your model
  144. provider. Use the window list to target a specific window, and keep password
  145. managers and banking apps out of the loop.
  146. ## How it works
  147. - `win32.py` — DPI awareness, window enumeration (DWM cloaked filter),
  148. `PrintWindow(PW_RENDERFULLCONTENT)`, `BGRX` decode to dodge the zero-alpha bug.
  149. - `uia_worker.py` — UI Automation runs in a **separate process**; a blocked COM
  150. call cannot be cancelled by a thread timeout, so the parent kills the process.
  151. - `worker.py` — persistent worker with hard timeout and transparent respawn
  152. (cold-importing `uiautomation` costs 1–2 s per call, so it stays alive).
  153. - `inspector.py` — window resolution and terse formatting. Ambiguous matches are
  154. an error, never a silent first hit.
  155. - `overlay.py` — Set-of-Mark: de-duplication, containment suppression, reading
  156. order, collision-free badge placement.
  157. - `capture.py` — blank-frame detection, downscale, smaller of PNG/JPEG.
  158. ## Roadmap
  159. - `0.4` — optional actions (`--enable-actions`) in a **separate** package, so
  160. this core stays provably read-only
  161. - `0.5` — `wait_for_element`, for UIs that settle slowly after a click
  162. ## Development
  163. ```sh
  164. pip install -e ".[dev]"
  165. pytest
  166. ```
  167. The unit tests cover window resolution, formatting, overlay selection and
  168. encoding. They do not touch Win32, so they run in a quarter of a second.
  169. ## License
  170. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.