Skip to content
Back to search

https://github.com/ctoldigital/axuse

SkillWorks with Claude CodeGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ctoldigital--axuse/badge.svg)](/listings/ctoldigital--axuse)

HTML

<a href="/listings/ctoldigital--axuse"><img src="/api/listings/ctoldigital--axuse/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

You need **Node.js 22+** and a Chromium-family browser (Chrome, Edge, Chromium, or Chrome for Testing; found automatically, or set `AXUSE_BROWSER=/path/to/chrome`). On a bare server with no browser and no sudo: `node src/cli.js install-browser` provisions a pinned Chrome for Testing under `~/.cache/axuse` — nothing else: axuse has no npm dependencies.

**1. Get it and check the environment**

git clone https://github.com/ctoldigital/axuse.git
cd axuse
node src/cli.js doctor        # finds a browser, launches it, takes a snapshot

For a daily Chrome/Edge session, use the guided bridge setup. It registers the native host, opens the extensions page, prints the exact folder to load, and can print your harness configuration in the same command:

node src/cli.js setup --browser chrome --harness codex

After clicking **Load unpacked** once, verify the connection with:

node src/cli.js bridge verify

**2. Add it to your harness** (use your absolute path to `src/cli.js`)

# Codex
codex mcp add axuse -- node /abs/path/axuse/src/cli.js mcp

# Claude Code
claude mcp add --scope user axuse -- node /abs/path/axuse/src/cli.js mcp

# OpenCode, Antigravity, any other MCP host: print the config block to paste
node src/cli.js print-config --harness opencode   # or agy | generic

**3. Ask for something in the browser**

> Open https://example.com, find the "More information" link and tell me > where it goes.

The agent calls `ax_open`, reads the accessibility snapshot, and acts on element refs like `[ref=e7]`. Each action returns only what changed.

**Optional settings** (environment variables for the MCP server)

| variable | effect | |---|---| | `AXUSE_SERVER_MODE` | `auto` (default): on a display-less Linux server, boot a private Xvfb and keep Chrome fully **headed** on it (anti-bot-safe; see [Servers, no desktop](docs/server-mode.md)) | | `AXUSE_DISPLAY_SIZE` | virtual display geometry for server mode (default `1920x1080x24`; `random` = a stable geometry per machine, so a fleet is not all 1920x1080) | | `AXUSE_TIMEZONE` | IANA `TZ` for the Chrome process — `Date`/`Intl` in pages and workers match the egress persona (validated by `doctor`) | | `AXUSE_LANG` | `--accept-lang` for Chrome — `navigator.language(s)` and the `Accept-Language` header match the egress persona | | `AXUSE_PROXY` | route every launched browser through a proxy (`http://`·`socks5://`, credentials supported — answered over CDP); `ax_open {proxy}` sets it per launch, and `ax_open {timezone, lang}` completes the per-instance persona | | `AXUSE_AMBIENT_MOTION` | `0` disables idle pointer park-fuzz (default: automatic in server mode — the agent's "thinking" pauses still show a live hand) | | `AXUSE_MAX_BROWSERS` / `AXUSE_MAX_TABS` | governance caps for massive operations (defaults 8 / 48) — typed failures instead of OOM spirals | | `AXUSE_NO_SANDBOX=1` | force `--no-sandbox` (auto for root/container server runs) | | `AXUSE_NEUTRALIZE_WEBDRIVER=0` | disable the `navigator.webdriver` posture-fix fallback (the `AutomationControlled` launch flag already fixes it at the source) | | `AXUSE_HEADLESS=1` | launch the browser without a window — **trusted internal pages only**: challenge/anti-bot pages detect headless first; servers get server mode instead | | `AXUSE_BROWSER` | path to the browser binary | | `AXUSE_PROFILE_DIR` | where temporary browser profiles go (default: OS temp; they are deleted on exit) | | `AXUSE_BRIDGE_DIR` | override the local bridge state directory (default: `~/Library/Application Support/axuse` on macOS) | | `AXUSE_ENABLE_MACOS_CUA=1` | enable macOS desktop tools (Accessibility + Screen Recording grants are still required) | | `AXUSE_BRIDGE_DEBUG=1` | trace the extension bridge |

To drive your own logged-in Chrome instead of a fresh browser, see [Driving your daily browser](#driving-your-daily-browser-the-codex-pattern).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 179 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # axuse — browser & computer use, done right
  2. Accessibility-first browser/computer use over MCP. Works in any MCP-capable
  3. harness (Codex, Claude Code, OpenCode, Antigravity, …). Zero dependencies.
  4. ## The contract (read once, then act)
  5. **Instrument, don't photograph.** The browser is controlled through CDP with
  6. structured accessibility snapshots. Screenshots are rationed. Coordinates are
  7. the last resort.
  8. 1. **Open → act → read the diff → act.** Every action returns the diff of
  9. what changed — you do not need to re-observe after acting. Elements carry
  10. their state: `checked`/`unchecked`, `expanded=…`, `selected`, `disabled`,
  11. `value="…"`, `haspopup`. Tables print one `row "a | b | c"` line per row.
  12. 2. **Batch, don't drip.** Forms: ONE `ax_fill` with every field (text,
  13. `<select>`, checkboxes/radios/switches as true/false, ARIA/native sliders as numbers (trusted Arrow-key convergence, no synthetic writes), custom dropdowns and
  14. autocomplete comboboxes by option text) plus `submit` — not one call per
  15. field.
  16. 3. **Search, don't scroll-and-read.** `ax_observe {find:"text"}` returns only
  17. the matching lines (+context, refs) from the whole page.
  18. `ax_scroll {until:"text"}` scrolls an infinite feed / lazy list in ONE call
  19. until the text appears and returns just that neighbourhood.
  20. 4. **Refs, not pixels.** Act on `{ref: "e7"}` from the latest observation.
  21. Refs are bound to the element and resolved live (after the element's box is
  22. stable), so clicks land even if the layout moved.
  23. 5. **Frames and tabs are handled for you.** Cross-origin iframes appear as
  24. `--- frame f1 (url) ---` sections with `f1:e3` refs — act on them like any
  25. ref. A click that opens a new tab/popup reports it and returns the new
  26. tab's snapshot; it becomes the active tab (`tab_id` to go back).
  27. 6. **Dialogs.** An action that opens alert/confirm/prompt says so in its
  28. result; answer with `ax_dialog` (accept|dismiss, `prompt_text`). A
  29. confirm→prompt chain reports the next dialog after each answer.
  30. 7. **Waiting.** `ax_wait_for {text|selector|expression}` is one bounded
  31. in-page wait and returns the diff when the condition holds.
  32. 8. **Pixels only when pixels matter** (canvas, visual checks): the snapshot
  33. shows `canvas (box=x,y,WxH)`; take one `ax_screenshot`, then click
  34. `{x,y}` in page coordinates (the box tells you where the canvas is).
  35. 9. **App shortcuts follow the OS.** `ax_press {combo:"ctrl+alt+1",
  36. platform:true}` translates Windows-convention application shortcuts to
  37. the host OS (macOS: cmd+alt+1, cmd+end, …). Omit `platform` for combos
  38. that must be literal Control (terminals, devtools). Plain `enter`,
  39. `escape`, arrows are identical everywhere; select-all is automatic.
  40. For a bounded source list, use `ax_open_many` with `urls: [...]`. It returns
  41. compact title/status/excerpt rows and closes its temporary tabs by default;
  42. reserve repeated `ax_open` calls for pages that need interactive follow-up.
  43. ## CAPTCHA solving (only with a loaded solver plugin)
  44. `ax_solve_captcha { provider, url? }` solves a CAPTCHA on the active tab (or
  45. opens `url` first) using a **loaded solver plugin** — the open core ships none;
  46. solvers are separate (paid) packages. `ax_plugins` lists what is loaded and any
  47. that failed to load. Results are facts: `solved`, `token`, `attempts`,
  48. `rounds`, `elapsedMs`, `notes` — a null token is reported as unsolved, never
  49. fabricated. Without any solver loaded the call fails typed `NO_SOLVER` with the
  50. install hint.
  51. Loaded plugins may also contribute **capability tools** (`ax_vault`,
  52. `ax_net`, `ax_capture`, `ax_guard`, `ax_extract`, …) through a validated
  53. `tools()` export — capability surfaces, not solves. `ax_plugins` lists each
  54. loaded package together with its contributed tool names.
  55. **Servers (no desktop):** on a display-less Linux host axuse auto-boots a
  56. private Xvfb and keeps Chrome fully **headed** on it — never pass
  57. `headless: true` for challenge/anti-bot pages (they check headless first).
  58. The `ax_open` result reports the fact: `display: :N (virtual …)`. Persona
  59. coherence with the egress persona is configured server-side
  60. (`AXUSE_TIMEZONE`, `AXUSE_LANG`, `AXUSE_PROXY`; `navigator.webdriver`
  61. stays `false` via the `AutomationControlled` launch flag). For
  62. dynamics-sensitive targets pass `human: true` on `ax_click` (jittered
  63. curved mouse approach) and `ax_type` (per-key cadence) — solver plugins
  64. already use it on challenge widgets. Solvers work unchanged; vision needs
  65. `<PREFIX>_VISION_KEY` (any OpenAI-compatible endpoint) or the owner-only
  66. `~/.config/axuse/secrets.json`. Full lane: `docs/server-mode.md`.
  67. ## Error vocabulary (typed — read `code`, `hint`, `retry`)
  68. - `REF_NOT_FOUND` / `STALE_STATE` → the page changed under you: `retry:
  69. "reobserve"` — call `ax_observe`, then re-plan from fresh refs.
  70. - `DIALOG_OPEN` → answer with `ax_dialog`, then retry the action.
  71. - `ELEMENT_HIDDEN` → scroll or re-observe.
  72. - `SCREENSHOT_BUDGET_EXCEEDED` → crop a region instead.
  73. - `DESKTOP_PERMISSION_REQUIRED` → grant macOS Accessibility/Screen Recording
  74. to the host, then retry after `axuse doctor` confirms the grant.
  75. - Actions report `action_sent`: `sent` | `possibly_sent` (a timeout after
  76. dispatch — verify state before retrying, the input may have landed).
  77. ## Desktop apps on macOS — structure-first too
  78. The macOS desktop layer (needs `AXUSE_ENABLE_MACOS_CUA=1` + the
  79. Accessibility grant) mirrors the browser discipline:
  80. 1. **`ax_desktop_windows`** → pick the target app (rows carry pid + bounds;
  81. frontmost first).
  82. 2. **`ax_desktop_ax_snapshot {pid|name|front}`** → the app's accessibility
  83. tree with roles, titles, values, bounds, and live refs — the desktop twin
  84. of `ax_observe`. Budgeted (truncation is reported as facts).
  85. 3. **Act on refs, not pixels**: `ax_desktop_ax_click {ref}` (trusted AXPress;
  86. automatic coordinate fallback when an app refuses), `ax_desktop_ax_set
  87. {ref, value}` (writable text fields, read-back verified), and
  88. `ax_desktop_ax_read {ref}` for fresh facts. Stale refs fail typed
  89. `REF_NOT_FOUND` with a reobserve hint — same vocabulary as the browser.
  90. 4. Coordinates + screenshots (`ax_desktop_click`, `ax_desktop_screenshot`)
  91. remain the fallback for canvas/games — the same "structure before
  92. pixels" rule as the browser side.
  93. ## Attach to the real browser (two ways)
  94. **1. Raw CDP attach.** Relaunch Chrome with `--remote-debugging-port=9222
  95. --remote-allow-origins=*`, then `ax_open { url, cdp_url: "http://127.0.0.1:9222" }`.
  96. Full feature set (including OOPIF frame entry).
  97. **2. The axuse bridge extension (Codex-style).** Drive your daily Chrome/Edge
  98. through an extension — your real profile, cookies and logged-in sessions, with
  99. the debugger infobar keeping you honestly informed:
  100. ```
  101. axuse setup --browser chrome --harness codex # guided one-time setup
  102. axuse bridge verify # check the live connection
  103. axuse bridge install # registers the native messaging host
  104. axuse bridge # start the hub (or it starts on demand)
  105. ```
  106. Then load `bridge/extension` as an UNPACKED extension in Chrome
  107. (chrome://extensions → Developer mode → Load unpacked), and
  108. `ax_open { url, backend: "extension" }`. The service worker dials the hub via
  109. native messaging; tabs it opens are visible in your browser; `ax_close` closes
  110. the tabs axuse created — never your browser.
  111. **Extension backend notes (all discovered live):**
  112. - Branded Chrome 137+ ignores `--load-extension` (silent no-op), so automated
  113. installs need Edge / Chromium / Chrome-for-Testing. Loading unpacked by hand
  114. works in every browser.
  115. - Edge's native-messaging host lookup requires `allowed_origins`
  116. (`chrome-extension://ID/`) in the host manifest — `allowed_extensions` alone
  117. reports "host not found". The installer writes both fields.
  118. - On macOS, `--browser` may also be `chromium` or `chrome-for-testing`; the
  119. installer uses the browser-specific `NativeMessagingHosts` directory.
  120. - On Windows the native host is a `.bat` launcher (cmd.exe relays the stdio
  121. pipes faithfully; a compiled C# shim lost Chrome's stdin writes).
  122. ## Desktop (computer use)
  123. `ax_desktop_*` tools exist on **Windows** (SendInput + GDI capture, physical
  124. pixels — get bounds from `ax_desktop_screen_info`/`ax_desktop_windows`).
  125. Verified live.
  126. **Linux** mounts the same tools automatically on X11/XWayland sessions
  127. (`DISPLAY` set): XTEST input, GetImage JPEG capture, EWMH window list and
  128. activation, RandR per-monitor layout in `ax_desktop_screen_info` — and the
  129. **structure-first layer through AT-SPI** (zero-dependency D-Bus client):
  130. `ax_desktop_ax_snapshot/ax_click/ax_set/ax_read` with refs, roles, names,
  131. bounds and states, like macOS System Events. No extra packages; headless
  132. hosts are covered by server mode's auto-booted Xvfb (`xvfb-run` also works).
  133. Coordinates are root-window pixels. Screenshots honor
  134. the byte budget with a quality → quality → downscale ladder (vision keeps a
  135. usable image instead of a dead end), and `ax_desktop_type` pastes long or
  136. non-keymap text through the clipboard when xclip is present (reported
  137. honestly as `via: "clipboard-paste"`).
  138. **macOS** ships an opt-in driver (System Events keystroke, corrected
  139. CoreGraphics mouse/scroll events via JXA, and `screencapture` + `sips` pixels)
  140. that loads with `AXUSE_ENABLE_MACOS_CUA=1`. It reports TCC permission failures
  141. as typed desktop errors. Grant the host Accessibility + Screen Recording in
  142. System Settings → Privacy & Security, then use `node src/cli.js doctor` before
  143. desktop actions. Coordinates are macOS global display points; desktop JPEGs
  144. are resized to those point dimensions for direct coordinate mapping.
  145. v1 desktop is coordinate+vision based (no OS accessibility tree): use it only
  146. for apps the browser tools cannot reach. Prefer `ax_*` for anything web.
  147. ## Anti-patterns
  148. - Screenshot loops as the primary observation (the CUA trap — expensive,
  149. lossy, coordinate-drifting).
  150. - Clicking coordinates remembered from BEFORE another action or navigation.
  151. - Polling with repeated full observations when a `diff` would do.
  152. - Ignoring `covered: true` notes (an overlay ate the click) — verify in the
  153. diff and re-plan.
README.md · README · 454 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # axuse — browser & computer use, done right
  2. **Accessibility-first browser and computer use for agent harnesses.** One MCP
  3. server plugs into Codex, Claude Code, OpenCode, Antigravity, or any
  4. MCP-capable host. Zero dependencies (Node ≥ 22). Windows desktop driver is
  5. verified; macOS desktop control is available as an explicit opt-in with
  6. native TCC permission checks; Linux desktop control runs over X11/XWayland
  7. (XTEST input + capture + EWMH windows, zero extra packages).
  8. The design is distilled from the two best implementations of this problem —
  9. OpenAI's Codex Chrome extension and the ZCode harness — and exists to prove a
  10. point: **a screenshot loop is not the right interface for a machine**. The
  11. DOM tree, the accessibility tree, and CDP are. axuse instruments the target
  12. instead of photographing it.
  13. **Not a developer?** You don't need anything below. In the Claude desktop
  14. app, download `axuse.mcpb` from [ax-use.com](https://ax-use.com) and
  15. double-click it. In ChatGPT's Codex app, Claude Code, Cursor or any other
  16. agent, paste the [install prompt](https://ax-use.com/install.md) and the
  17. agent sets axuse up for you. See the
  18. [non-developer guide](docs/non-developer-onboarding.md).
  19. ## Three ways to let an agent work for you
  20. One install, three surfaces. In all three the agent reads the screen as **text from the accessibility tree**, not as a screenshot it has to interpret, so it is faster, cheaper and less likely to misclick.
  21. ### Best way to get started
  22. Let your agent do it. Paste one line into Codex, Claude Code, ChatGPT Work, Cursor, OpenCode, Devin, Antigravity or any MCP-capable agent:
  23. - Browser use and computer use: *Install axuse by following https://ax-use.com/install.md*
  24. - Phone use (Android): *Install axdroid by following https://ax-use.com/install-phone.md*
  25. Claude desktop users can instead download `axuse.mcpb` from [ax-use.com](https://ax-use.com) and double-click it. Developers: see the [Quick start](#quick-start) below.
  26. ### Browser use
  27. Your agent drives Chrome, Edge or Chromium: your signed-in daily browser through the extension bridge, or a clean one.
  28. - **Why it is better:** it reads the DOM and accessibility tree over CDP as a compact text snapshot with stable element refs, and every action returns only what changed. Input is trusted CDP input, and failures are typed errors with retry hints. See [why this beats screenshot CUA](#why-this-is-faster-and-more-reliable-than-screenshot-cua).
  29. - **Benchmark:** on 16 server-scored tasks in the same harness and model, axuse's median task takes **17 s** against 26 s for Codex's own Chrome extension and 56 s for browser-use, using about half the tokens, with the same accuracy. [Details](#benchmark).
  30. - **Get started:** [Quick start](#quick-start).
  31. ### Computer use
  32. Your agent drives desktop apps: Windows is verified, macOS is an explicit opt-in with native permission checks, Linux runs over X11 (and on display-less Ubuntu servers, see [server mode](#servers-no-desktop-ubuntu-server-lane)).
  33. - **Why it is better:** the same design as browser use. The agent reads each app's operating-system accessibility tree as text and acts on refs; a screenshot is only a fallback for canvases and other screens with no tree.
  34. - **Benchmark:** we have not published a head-to-head desktop benchmark yet. The browser numbers above are the measured evidence for the shared design.
  35. - **Get started:** the install line above, then the [tool surface](#tool-surface) for the desktop tools.
  36. ### Phone use (Android)
  37. Your agent uses an Android phone through [axdroid](phone-use/android/README.md), an MCP server plus a phone app. axphone for iPhone is coming soon.
  38. - **Why it is better:** it reads the phone's own accessibility tree as text, so no vision model is needed. A guarded screenshot is a fallback for screens with no readable text, and the phone's owner can switch it off.
  39. - **Benchmark:** a Codex agent (gpt-6.1-terra) using axdroid completed **116 of 116** tasks of the official AndroidWorld suite, repeated three times. Median 66 s and 223k input tokens per task, and no screenshots needed. [Results](phone-use/android/androidworld/README.md).
  40. - **Get started:** the `install-phone.md` line above. You need an Android 8+ phone, a USB cable and an MCP-capable agent.
  41. ## Proven in production
  42. axuse is maintained by **CTOL Digital Solutions**
  43. ([ctol.digital](https://ctol.digital)), a leading AI, Strategy and IT
  44. consultancy. This work has been tested and used at large scale across many
  45. client engagements. CTOL Digital Solutions holds the copyright of all the
  46. code.
  47. ## Quick start
  48. You need **Node.js 22+** and a Chromium-family browser (Chrome, Edge,
  49. Chromium, or Chrome for Testing; found automatically, or set
  50. `AXUSE_BROWSER=/path/to/chrome`). On a bare server with no browser and no
  51. sudo: `node src/cli.js install-browser` provisions a pinned Chrome for
  52. Testing under `~/.cache/axuse` — nothing else: axuse has no npm dependencies.
  53. **1. Get it and check the environment**
  54. ```bash
  55. git clone https://github.com/ctoldigital/axuse.git
  56. cd axuse
  57. node src/cli.js doctor # finds a browser, launches it, takes a snapshot
  58. ```
  59. For a daily Chrome/Edge session, use the guided bridge setup. It registers the
  60. native host, opens the extensions page, prints the exact folder to load, and
  61. can print your harness configuration in the same command:
  62. ```bash
  63. node src/cli.js setup --browser chrome --harness codex
  64. ```
  65. After clicking **Load unpacked** once, verify the connection with:
  66. ```bash
  67. node src/cli.js bridge verify
  68. ```
  69. **2. Add it to your harness** (use your absolute path to `src/cli.js`)
  70. ```bash
  71. # Codex
  72. codex mcp add axuse -- node /abs/path/axuse/src/cli.js mcp
  73. # Claude Code
  74. claude mcp add --scope user axuse -- node /abs/path/axuse/src/cli.js mcp
  75. # OpenCode, Antigravity, any other MCP host: print the config block to paste
  76. node src/cli.js print-config --harness opencode # or agy | generic
  77. ```
  78. **3. Ask for something in the browser**
  79. > Open https://example.com, find the "More information" link and tell me
  80. > where it goes.
  81. The agent calls `ax_open`, reads the accessibility snapshot, and acts on
  82. element refs like `[ref=e7]`. Each action returns only what changed.
  83. **Optional settings** (environment variables for the MCP server)
  84. | variable | effect |
  85. |---|---|
  86. | `AXUSE_SERVER_MODE` | `auto` (default): on a display-less Linux server, boot a private Xvfb and keep Chrome fully **headed** on it (anti-bot-safe; see [Servers, no desktop](docs/server-mode.md)) |
  87. | `AXUSE_DISPLAY_SIZE` | virtual display geometry for server mode (default `1920x1080x24`; `random` = a stable geometry per machine, so a fleet is not all 1920x1080) |
  88. | `AXUSE_TIMEZONE` | IANA `TZ` for the Chrome process — `Date`/`Intl` in pages and workers match the egress persona (validated by `doctor`) |
  89. | `AXUSE_LANG` | `--accept-lang` for Chrome — `navigator.language(s)` and the `Accept-Language` header match the egress persona |
  90. | `AXUSE_PROXY` | route every launched browser through a proxy (`http://`·`socks5://`, credentials supported — answered over CDP); `ax_open {proxy}` sets it per launch, and `ax_open {timezone, lang}` completes the per-instance persona |
  91. | `AXUSE_AMBIENT_MOTION` | `0` disables idle pointer park-fuzz (default: automatic in server mode — the agent's "thinking" pauses still show a live hand) |
  92. | `AXUSE_MAX_BROWSERS` / `AXUSE_MAX_TABS` | governance caps for massive operations (defaults 8 / 48) — typed failures instead of OOM spirals |
  93. | `AXUSE_NO_SANDBOX=1` | force `--no-sandbox` (auto for root/container server runs) |
  94. | `AXUSE_NEUTRALIZE_WEBDRIVER=0` | disable the `navigator.webdriver` posture-fix fallback (the `AutomationControlled` launch flag already fixes it at the source) |
  95. | `AXUSE_HEADLESS=1` | launch the browser without a window — **trusted internal pages only**: challenge/anti-bot pages detect headless first; servers get server mode instead |
  96. | `AXUSE_BROWSER` | path to the browser binary |
  97. | `AXUSE_PROFILE_DIR` | where temporary browser profiles go (default: OS temp; they are deleted on exit) |
  98. | `AXUSE_BRIDGE_DIR` | override the local bridge state directory (default: `~/Library/Application Support/axuse` on macOS) |
  99. | `AXUSE_ENABLE_MACOS_CUA=1` | enable macOS desktop tools (Accessibility + Screen Recording grants are still required) |
  100. | `AXUSE_BRIDGE_DEBUG=1` | trace the extension bridge |
  101. To drive your own logged-in Chrome instead of a fresh browser, see
  102. [Driving your daily browser](#driving-your-daily-browser-the-codex-pattern).
  103. ## Servers, no desktop (Ubuntu server lane)
  104. axuse runs under agent CLIs (codex, claude code, opencode) on headless
  105. Ubuntu servers. Challenge/anti-bot pages check *is this a headless browser?*
  106. first, so server mode never uses `--headless`: it boots a private Xvfb and
  107. runs Chrome **fully headed** on it — real headed build, no headless
  108. fingerprint (verified live against Turnstile: `ax_solve_captcha` solves on a
  109. server with no desktop, and the slider engine's human-curve drags land
  110. 10/10). Beyond headed-vs-headless, server mode covers **persona
  111. coherence** (`AXUSE_TIMEZONE`/`AXUSE_LANG` matching the egress persona,
  112. `AXUSE_DISPLAY_SIZE=random` fleet diversity, `--disable-blink-features=
  113. AutomationControlled` so `navigator.webdriver` stays `false` at the source)
  114. and fleet hygiene (zombie-Xvfb sweep). Solver plugins that need vision call
  115. a multimodal model through any OpenAI-compatible API; keys come from env
  116. vars or an owner-only secrets file. One-time setup, systemd wiring, env
  117. reference and scale notes: [`docs/server-mode.md`](docs/server-mode.md) —
  118. and `node src/cli.js doctor` verifies the whole lane (it prints the measured
  119. anti-bot posture and persona).
  120. ## Benchmark
  121. On 16 server-scored browser tasks, run in the same harness (Codex CLI) with
  122. the same model against Codex's own Chrome extension and
  123. [browser-use](https://github.com/browser-use/browser-use) 0.13:
  124. | | axuse | Codex + Chrome extension | browser-use |
  125. |---|---|---|---|
  126. | tasks passed | 28/28 | 27/27 | 27/27 |
  127. | median time per task | **17 s** | 26 s | 56 s |
  128. | tokens, all runs | **4.7M** | 9.8M | 8.8M |
  129. | median tool calls | **5** | 7 | 12 |
  130. Accuracy tied on those tasks. A harder 7-task extension (checkout flow,
  131. virtualized grid, date pickers, signup errors, nested iframes, file upload,
  132. local dev app) has been run once so far: axuse 7/7; Codex + Chrome failed 2
  133. by stopping to ask for confirmation; browser-use failed 1. Method, anti-cheat
  134. rules and how to reproduce: [`bench/suite/README.md`](bench/suite/README.md).
  135. Install footprint and setup comparison: [`bench/suite/EASE.md`](bench/suite/EASE.md).
  136. **Phone use (Android).** A Codex agent (gpt-6.1-terra) using axdroid completed
  137. all 116 tasks of the official AndroidWorld suite, repeated three times: median
  138. 66 s and 223k input tokens per task, 1,576 tool calls, no screenshots needed.
  139. Per-task numbers and method: [`phone-use/android/androidworld/`](phone-use/android/androidworld/README.md).
  140. ## Why this is faster and more reliable than screenshot CUA
  141. | | Screenshot CUA loop | axuse |
  142. |---|---|---|
  143. | Observation | ~1–2MB base64 raster per step, ~1k+ image tokens, vision-encoder latency | YAML-like AX snapshot in KBs; **diff by default** (only removed/added/changed lines) |
  144. | Targeting | pixel coordinates guessed from an image; drifts with resolution/DPR | stable element refs bound to **element identity** (insertion-safe) + Playwright-style semantic lines |
  145. | Input | synthetic DOM events (rejected by sites) or fragile OS injection | trusted CDP `Input.dispatch*` events (`isTrusted`) |
  146. | Failure mode | silent misclick, discovered 2–3 screenshots later | typed errors: `REF_NOT_FOUND` / `STALE_STATE` / `DIALOG_OPEN` with `retry` hints and `action_sent` facts |
  147. | Change detection | re-photograph everything | sequenced event ring buffer + snapshot diffing |
  148. | Cost control | none | budgets enforced in code: snapshot char/node caps, screenshot byte budget with quality fallback, one raster per call |
  149. ## Install into a harness
  150. ```bash
  151. node src/cli.js print-config --harness codex # or claude-code | opencode | agy | generic
  152. ```
  153. - **Codex** — append the printed `[mcp_servers.axuse]` block to `~/.codex/config.toml`
  154. - **Claude Code** — `claude mcp add --scope user axuse -- node <path>/src/cli.js mcp`
  155. - **OpenCode** — merge the printed `mcp` block into `opencode.json`
  156. - **Antigravity / any MCP host** — the printed `mcpServers` block
  157. Verify the environment: `node src/cli.js doctor`.
  158. The model-facing usage contract lives in [`SKILL.md`](SKILL.md) — point your
  159. harness's skill system (or system prompt) at it.
  160. ## Solver plugins (open core ↔ paid solvers)
  161. axuse's core is CAPTCHA-agnostic and ships **no solvers**. Turnstile / hCaptcha /
  162. reCAPTCHA solvers are **separate, commercially licensed packages** that plug in
  163. through a thin, frozen seam (`src/plugins/`):
  164. ```bash
  165. # install: drop a solver package into plugins/ (or list dirs in AXUSE_PLUGINS)
  166. node src/cli.js plugins # → ✓ captcha-pro@1.0.0 — providers: turnstile, recaptcha-v2, hcaptcha
  167. node src/cli.js solve turnstile --url https://demo.turnstile.workers.dev/
  168. ```
  169. From any MCP harness: `ax_solve_captcha { provider, url? }` and `ax_plugins`.
  170. Plugins that own capabilities (session vaults, network governors, guarded
  171. actions…) can also contribute their **own MCP tools** through a validated
  172. `tools()` export (`ax_vault`, `ax_net`, `ax_guard`, …) — same seam, same
  173. trust boundary. The ABI, the solver context, the `tools()` contract, and
  174. packaging/licensing guidance:
  175. [`docs/plugins.md`](docs/plugins.md). A complete free (PolyForm Noncommercial
  176. 1.0.0) reference solver
  177. lives in [`examples/plugin-turnstile-demo/`](examples/plugin-turnstile-demo/).
  178. ## Architecture
  179. ```
  180. harness (Codex / Claude Code / OpenCode / Antigravity / …)
  181. │ MCP over stdio (JSON-RPC 2.0) — the universal integration layer
  182. ▼
  183. src/mcp/ — thin, stateless-per-call adapter (hand-rolled, zero deps)
  184. │ one frozen command contract (src/core/commands.js)
  185. ▼
  186. src/core/ — the state owner (Codex native-host / ZCode host pattern)
  187. ├─ controller.js sessions, tabs, state_id, dialogs, auto-observe diffs,
  188. │ OOPIF frame registry (f-slot namespaced refs)
  189. ├─ cdp.js launch/attach Chromium, browser-level WebSocket, flat sessions,
  190. │ Target.setAutoAttach for OOPIF auto-entry
  191. ├─ serialize.js injected AX serializer: element-identity refs, shadow DOM,
  192. │ same-origin iframes (coordinate-translated), budgets
  193. ├─ diff.js per-ref diff + text-line multiset diff + cost-based full fallback
  194. ├─ actuate.js trusted Input.dispatch*, live ref resolution (frame-chain
  195. │ coordinate translation), combos, waits
  196. ├─ screenshot.js one raster, byte budget, quality fallback, crop hints
  197. └─ errors.js typed codes, retry hints, action_sent, side_effect
  198. │
  199. ├─ src/bridge/ — Codex-style extension backend for the user's daily
  200. │ Chrome/Edge: hub (TCP rendezvous, token auth, NDJSON) ←
  201. │ native-messaging host (framed stdio relay) ← MV3 extension
  202. │ (chrome.debugger = real CDP over the user's real browser)
  203. │ + installer (registry/HKCU, host manifest, launcher) + backend
  204. │ duck-typed onto the same command contract
  205. ▼
  206. src/desktop/ — computer use (driver matrix)
  207. ├─ Windows: resident PowerShell + SendInput/GDI, DPI-aware — VERIFIED
  208. ├─ macOS: JXA/CoreGraphics + System Events + screencapture/sips,
  209. │ display-point coordinates, env-gated with TCC diagnostics
  210. └─ (the driver interface is the seam for a real AX/UIA tree driver)
  211. ```
  212. ### The patterns, and where each one comes from
  213. | Pattern | Codex lineage | ZCode lineage | axuse implementation |
  214. |---|---|---|---|
  215. | Diff-first observation | page-state diffing, default-on | "the tree comes back as a diff only… unchanged rows are omitted and their indices stay valid" | per-ref diff; refs bound to element identity so unchanged rows stay valid |
  216. | Element addressing | axState `element_index`/locators | refs `e1,e2…` in page + live resolution | `window.__axuseRefRegistry` (identity-stable across insertions), resolved live at action time |
  217. | Trusted input | `chrome.debugger` → `Input.dispatch*` | `webContents.debugger` | raw CDP `Input.dispatch*` / `Input.insertText` |
  218. | Event buffer | `tab_cdp_events` sequenced ring (limit ≤1000) | event routing with teardown discipline | ring buffer, sequence cursor, filtered methods |
  219. | Context budgets | 64k DOM text cap | 2×200KB images at protocol boundary, one-raster rule | snapshot node/char caps; screenshot 200KB budget + quality fallback + crop hint |
  220. | Error vocabulary | structured locator errors | `STALE_STATE`, `sideEffect:none\|uncertain`, `actionSent` | typed codes + `retry: reobserve\|retry\|never` + `action_sent` |
  221. | Frozen contract | command envelopes w/ zod | one discriminated union for all backends | `Contract` → validation AND MCP schemas generated from it |
  222. | Real user session | attaches to the user's real Chrome profile | `extension` backend via user Chrome | **extension bridge**: MV3 extension + native messaging + hub; `ax_open {backend:"extension"}` |
  223. | Frame entry (OOPIF) | webmcp/extension content merging | IAB merges frames; OOPIF via CDP | `Target.setAutoAttach` per tab; frames serialized in their own sessions, merged under `f<slot>:` identity-stable refs, input dispatched frame-locally |
  224. ## Tool surface
  225. **Browser** (always): `ax_open`, `ax_open_many`, `ax_navigate`, `ax_go`, `ax_tabs`, `ax_observe`,
  226. `ax_click`, `ax_fill`, `ax_upload`, `ax_hover`, `ax_type`, `ax_press`,
  227. `ax_scroll`, `ax_select`, `ax_drag`, `ax_screenshot`, `ax_element_info`,
  228. `ax_evaluate`, `ax_wait_for`, `ax_events`, `ax_dialog`, `ax_close`.
  229. The ones that save the most steps:
  230. - `ax_fill` sets a whole form in one call (text, selects, checkboxes, radios,
  231. switches, custom dropdowns, autocomplete) and can click submit.
  232. - `ax_observe {find:"text"}` returns only the matching lines of a big page.
  233. - `ax_scroll {until:"text"}` scrolls an infinite feed or virtualized list,
  234. in the page or inside a scrolling panel, until the text appears.
  235. - `ax_upload` sets files on a file input; a click that opens a file chooser
  236. is reported instead of hanging.
  237. - `ax_open_many` opens a bounded URL list sequentially, returns compact titles
  238. and accessibility excerpts, and closes temporary tabs by default. It is the
  239. batch equivalent of `ax_open` and avoids writing a custom MCP client.
  240. - JS dialogs and new tabs opened by an action are reported in its result.
  241. **Desktop** (Windows: verified. macOS: opt-in with
  242. `AXUSE_ENABLE_MACOS_CUA=1` and host Accessibility/Screen Recording grants.
  243. Linux: automatic on X11/XWayland sessions with `DISPLAY` set):
  244. `ax_desktop_screenshot`, `ax_desktop_screen_info`,
  245. `ax_desktop_click`, `ax_desktop_type`, `ax_desktop_press`, `ax_desktop_scroll`,
  246. `ax_desktop_windows`, `ax_desktop_activate`, `ax_desktop_drag`.
  247. ## The loop the model is guided into
  248. ```
  249. ax_open → FULL snapshot with refs (e.g. button "Sign in" [ref=e7])
  250. ax_click {e7} → returns the DIFF (removed/added/changed + text changes)
  251. ax_type {e9, text} → returns the diff (value="…" appears)
  252. … → screenshots ONLY for visual/canvas checks (budget-capped)
  253. ```
  254. Cross-origin iframes appear in the snapshot as `--- frame f1 (url) ---`
  255. sections with `f1:eN` refs — same loop, same diffs, frame-stable refs.
  256. Errors are facts, not strings:
  257. ```json
  258. { "code": "REF_NOT_FOUND",
  259. "hint": "Refs are per-observation. Call ax_observe, then act on fresh refs.",
  260. "retry": "reobserve" }
  261. ```
  262. ## Driving your daily browser (the Codex pattern)
  263. ```bash
  264. axuse bridge install # registers the native messaging host (Chrome + Edge hives)
  265. axuse bridge # hub (or it starts on demand with the MCP server)
  266. ```
  267. Load `bridge/extension` unpacked in Chrome/Edge (Developer mode → Load
  268. unpacked), then `ax_open { url }`: while the extension is connected, an
  269. omitted `backend` picks it (the result reports `backend`; pin it with
  270. `AXUSE_BACKEND=cdp|extension`). Your browser, your
  271. logged-in sessions, tabs visible to you; `ax_close` closes only axuse's tabs.
  272. Input and navigation bring a hidden axuse tab forward first (background tabs
  273. get no requestAnimationFrame, so SPAs never finish wiring their handlers;
  274. `AXUSE_VISIBLE_GUARD=0` opts out). Tabs are shared by every caller of one
  275. server, so `tab_id` is required once more than one tab is controlled
  276. (`AXUSE_REQUIRE_TAB_ID=0` restores the implicit active tab). Tabs no call has
  277. touched for 30 minutes are closed (`AXUSE_TAB_IDLE_MS`, `0` = off), and the
  278. hub closes the tabs of a client process that disconnects. Top-level HTML/text
  279. `data:` URLs load through `Page.setDocumentContent` on this backend (Chrome
  280. refuses the navigation); other `data:` types need localhost. For long
  281. articles whose snapshot is cut or shortened, `ax_observe {mode:"text"}`
  282. returns the full readable text.
  283. The whole chain — native messaging framing, hub auth, CDP-over-chrome.debugger,
  284. diffed snapshots, trusted clicks — is covered by `test/bridge.test.js`.
  285. Field notes (all established live against Edge 155 / Chrome 155):
  286. - Branded Chrome silently ignores `--load-extension` (137+). Automated
  287. extension loads need Edge / Chromium / Chrome-for-Testing; manual unpacked
  288. loads work everywhere.
  289. - Edge requires `allowed_origins` in the host manifest — with only
  290. `allowed_extensions` it reports "native messaging host not found". The
  291. installer writes both.
  292. - On macOS, `setup` and `bridge install` also accept `chromium` and
  293. `chrome-for-testing`, and register the host in each browser's own
  294. `NativeMessagingHosts` directory.
  295. - The Windows native host is a `.bat` launcher: cmd.exe relays the framed
  296. stdio pipes faithfully, where a compiled C# shim (Console.OpenStandardInput)
  297. dropped Chrome's stdin writes. Debug any install with
  298. `AXUSE_BRIDGE_DEBUG=1` (hub + nm-host message tracing) and
  299. `self.__axuseDiag` inside the extension's service worker (readable via an
  300. attached DevTools/CDP session).
  301. ## Honest scope notes
  302. - **macOS desktop control is opt-in** because it can synthesize OS input. Set
  303. `AXUSE_ENABLE_MACOS_CUA=1`, grant the host Accessibility and Screen Recording
  304. in System Settings → Privacy & Security, then run `node src/cli.js doctor`.
  305. Coordinates are macOS global display points and returned desktop JPEGs are
  306. resized to the same point dimensions for direct click mapping. Linux desktop
  307. control mounts automatically on X11/XWayland sessions (needs `DISPLAY`;
  308. Xvfb works for headless hosts) — and it is structure-first too: the AT-SPI
  309. tree (via axuse's zero-dependency D-Bus client) mounts
  310. ax_desktop_ax_snapshot/ax_click/ax_set/ax_read like macOS, plus EWMH
  311. windows, RandR monitor layout, a screenshot budget ladder that downscales
  312. instead of erroring, and clipboard-paste typing when xclip is installed.
  313. - **The extension backend has no Target domain** (chrome.debugger limits), so
  314. OOPIF frames are listed but not entered there; the CDP backend enters them.
  315. - **Anti-bot reality**: axuse runs a real Chromium with a real profile, but
  316. it does not implement Codex's `x-browser-agent` traffic labeling or
  317. bot-detection handoff — sites that fingerprint automation may still block
  318. it. The bridge backend (your daily browser) is the most human-looking path.
  319. For a non-MCP batch workflow, the same bounded operation is available from the
  320. CLI and accepts plain URL lists or Markdown containing links:
  321. ```bash
  322. node src/cli.js open-urls sources.md --backend extension --output report.md
  323. ```
  324. The generated report is intentionally honest: it contains page titles, access
  325. status, and bounded accessibility-text excerpts, not fabricated semantic
  326. summaries.
  327. ## Development
  328. ```bash
  329. node --test test/diff.test.js test/browser.test.js test/mcp.test.js test/desktop.test.js test/desktop-linux.test.js test/oopif.test.js test/bridge.test.js test/plugin.test.js
  330. ```
  331. On macOS, enable the passive desktop capability gates explicitly:
  332. ```bash
  333. AXUSE_ENABLE_MACOS_CUA=1 AXUSE_TEST_MACOS_DESKTOP=1 npm test
  334. ```
  335. Tests are gated: browser/MCP tests skip when no Chromium-family browser is
  336. found; the OOPIF test spins up two loopback origins (127.0.0.1 + localhost);
  337. the bridge test registers + uninstalls a native messaging host and uses an
  338. extension-capable browser when one is installed (branded Chrome on current
  339. macOS builds is honestly skipped because it ignores automated
  340. `--load-extension`); desktop tests exercise passive screen/window discovery on
  341. macOS with `AXUSE_TEST_MACOS_DESKTOP=1`. A real macOS screenshot test is
  342. explicit with `AXUSE_TEST_MACOS_SCREENSHOT=1` because Screen Recording is
  343. TCC-protected. The Linux desktop tests boot their own private Xvfb displays
  344. (and skip typed when Xvfb is not installed); they never touch your session.
  345. No test moves your real mouse or sends keys.
  346. Debugging: `AXUSE_BRIDGE_DEBUG=1` traces hub + native-host messages;
  347. `self.__axuseDiag` in the extension SW reports connect/disconnect history.
  348. Every `# SKIP` in a suite run is explicit, typed, and audited: run TAP output
  349. through `node tools/skip-audit.mjs` and it fails when a skip is missing from
  350. `test/skip-allowlist.json`. CI enforces this on every lane.
  351. macOS production reference — support matrix (browsers × macOS versions ×
  352. arch), TCC permission model, failure taxonomy, the manual TCC validation
  353. lane, benchmarks, and known limitations with evidence:
  354. [docs/macos-support.md](docs/macos-support.md).
  355. ## License
  356. Copyright © CTOL Digital Solutions ([ctol.digital](https://ctol.digital)).
  357. All rights reserved.
  358. Licensed under the **PolyForm Noncommercial License 1.0.0** — see
  359. [`LICENSE`](LICENSE). Use, distribution, and modification are free for any
  360. **noncommercial** purpose (personal use, research, education, charities,
  361. public institutions). **Commercial use requires a separate commercial
  362. license** from CTOL Digital Solutions ([ctol.digital](https://ctol.digital)).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.