Skip to content
Back to search

phone-harness

SkillWorks with Claude CodeGitHub

"Control the user's phone - an iPhone through the Mac's iPhone Mirroring window, an Android over adb, a rented cloud Android, or a cloud iPhone over HTTPS: open apps, tap, type, swipe, read the screen."

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/shawnpana--phone-harness/badge.svg)](/listings/shawnpana--phone-harness)

HTML

<a href="/listings/shawnpana--phone-harness"><img src="/api/listings/shawnpana--phone-harness/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

For the CLI, send this setup prompt to your agent (Claude Code, Codex, Grok Bot, and more):

Install or upgrade phone-harness to the latest stable version with uv using Python 3.12, register the skill from `phone-harness skill`, then read https://github.com/phone-harness/phone-harness/blob/main/onboarding.md and walk me through it. Follow https://github.com/phone-harness/phone-harness/blob/main/install.md if setup or connection fails.

The agent installs the CLI, asks which phone to use, and walks you through the steps that need your hands. What each phone needs:

| Phone | Needs | | --- | --- | | **iPhone** | macOS 15+, iPhone Mirroring paired to an iOS 18+ iPhone, Accessibility and Screen Recording permission | | **Android** | adb and USB debugging, on macOS, Linux or Windows | | **Cloud Android / iPhone** | [Phone Harness Cloud](https://phone-harness.com/dashboard?utm_source=github&utm_medium=readme&utm_campaign=setup-table) account; cloud Android also needs adb |

`phone-harness doctor` checks the setup. Full details: [install.md](install.md) and [setup/](setup/).

**MCP:** add `https://mcp.phone-harness.com/mcp` to any MCP client ([setup](https://github.com/phone-harness/mcp)). Cloud Android and iPhone only for now.

**Hermes Agent:** `hermes plugins install phone-harness/phone-harness/integrations/hermes`. See [integrations/hermes](integrations/hermes/README.md).

**OpenClaw:** `openclaw plugins install clawhub:phone-harness`. See [integrations/openclaw](integrations/openclaw/README.md).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 547 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: phone-harness
  3. description: "Control the user's phone - an iPhone through the Mac's iPhone Mirroring window, an Android over adb, a rented cloud Android, or a cloud iPhone over HTTPS: open apps, tap, type, swipe, read the screen."
  4. ---
  5. # phone-harness
  6. The main helpers are listed in [Helpers](#helpers).
  7. Direct control of a phone from Python scripts. The same helpers drive every
  8. kind of phone below; what differs is how each one sees and touches the screen,
  9. and that is what the per-phone sections below are for. Read **Which phone**, the
  10. shared **Working method**, and then only the section for the phone in front
  11. of you.
  12. ## Setup
  13. Check what this machine can do (OS, shell, install rights, network,
  14. attached phones; `phone-harness doctor` does the phone part), tell the user
  15. in a sentence or two which phone you can drive with what is here, and use
  16. the best one that actually runs. A phone on the desk wins; the cloud needs
  17. no hardware and is always the fallback.
  18. | Mode | Needs |
  19. | --- | --- |
  20. | Personal iPhone | macOS 15+, iPhone Mirroring paired to an iOS 18+ iPhone, Accessibility + Screen Recording for the app that runs Python |
  21. | Android on the desk | macOS, Linux or Windows; adb; USB debugging (or Wireless debugging, Android 11+) |
  22. | Cloud iPhone | HTTPS to phone-harness.com hosts; a signed-in account. No adb, any OS |
  23. | Cloud Android | the cloud iPhone list, plus adb and outbound TCP to the phone's adb port |
  24. | No install possible | HTTPS and an HTTP client: the API directly |
  25. - If something is missing, name it and the one step the user can take.
  26. - Ask the user only to approve sign-in and to pick a phone.
  27. - Detail: [iPhone](https://github.com/phone-harness/phone-harness/blob/main/setup/ios.md), [Android](https://github.com/phone-harness/phone-harness/blob/main/setup/android.md),
  28. [Cloud](https://github.com/phone-harness/phone-harness/blob/main/setup/cloud.md), [API](https://github.com/phone-harness/phone-harness/blob/main/setup/api.md).
  29. ## Helpers
  30. - `ensure_device()` - Screen bounds once the device is usable, or raises with what the user has to do.
  31. - `screen(min_confidence=0.3)` - Visible text as `{text, confidence, source, x, y, w, h}` boxes. On Android the boxes come from the accessibility tree; on iPhone Mirroring they come from recognizing text.
  32. - `screenshot(path=None)` - Capture the screen to a PNG, print its path, and return it.
  33. - `find_text(query, exact=False)` - Visible text boxes matching `query`.
  34. - `tap_text(query, index=0, exact=False)` - Tap the center of visible text matching `query`.
  35. - `tap(x, y)` - Tap one point.
  36. - `long_press(x, y, duration=0.8)` - Press and hold a point.
  37. - `type_text(text, delay=0.03, keystrokes=False)` - Type into the focused field.
  38. - `press(combo)` - Press a key, such as `press("enter")`.
  39. - `open_app(name)` - Launch an app and return the app id that launched.
  40. - `open_url(url)` - Open an https link in the phone's browser, whole. Cloud phones only for now.
  41. - `push_file(path)` - Send a file to the phone; it opens in the phone's browser under its own name. Cloud phones only for now.
  42. - `swipe(direction, distance=0.4, at=None)` - Flick. `direction` is finger motion: `"up"`, `"down"`, `"left"`, or `"right"`.
  43. - `scroll(direction="down", amount=0.3, at=None)` - Scroll. `direction` is the content to reveal: `"down"`, `"up"`, `"left"`, or `"right"`.
  44. - `scroll_screen(direction="down", amount=0.6, settle=2.5, moved_thresh=None, at=None)` - One scroll, then the text on screen before and after.
  45. - `scroll_until(done, direction="down", amount=0.6, max_scrolls=60, settle=2.5, at=None)` - Scroll until `done(boxes)` is truthy or the screen stops changing.
  46. - `scroll_collect(extract=None, key=None, direction="down", amount=0.6, max_scrolls=400, end_after=3, settle=2.5, on_progress=None, at=None)` - Scroll a list and return `{items, stop, scrolls}`.
  47. - `home()` - Go to the Home Screen.
  48. - `back()` - System Back, where the phone has one.
  49. - `app_switcher()` - Open the app switcher.
  50. - `wait_stable(timeout=6.0, interval=0.5, settle=2)` - Wait until the screen stops changing.
  51. - `wait_for_text(query, timeout=10.0, exact=False, interval=0.5)` - Poll until `query` is visible; return its box, or None.
  52. - `wait_for_app(app_id, timeout=10.0, interval=0.3)` - Poll until `app_id` is the foreground app.
  53. - `ui()` - The accessibility tree, where the device has one.
  54. - `tap_ui(query, index=0, exact=False, clickable_only=False)` - Tap an element in the accessibility tree.
  55. - `current_app()` - Foreground app id, where the device exposes one.
  56. - `list_apps(include_system=False)` - Installed app ids.
  57. - `screen_info()` - `{window, frontmost, img_px}`: bounds, focus, and capture size.
  58. - `find_window()` - Screen bounds `{x, y, w, h, id}`, or None if there is no device.
  59. - `image_point(x, y, image_size=None)` - Convert a screenshot pixel to a screen point.
  60. - `tap_image_point(x, y, image_size=None)` - Tap a point taken from a screenshot.
  61. - `connection_state()` - Backend state string; `"ready"` means usable.
  62. - `shell(cmd, binary=False, timeout=60)` - Run a shell command on the phone, where the device has a shell.
  63. ## Which phone?
  64. | Phone | How it is reached | Eyes | Hands | Read |
  65. | --- | --- | --- | --- | --- |
  66. | **Cloud Android** - rented from Phone Harness Cloud, the user's own saved phone | `phone-harness cloud start` connects it; nothing to export | accessibility tree (exact), Vision OCR fallback on a Mac | adb `input` | Working method, then **Cloud phones** and **Android** |
  67. | **Cloud iPhone** - a second iPhone hosted by Phone Harness Cloud, not the one in the user's pocket; an iOS entry in the account catalog | `phone-harness cloud start` with its id, label, or `ios` when that is the only one | server-side OCR (`source: "pixels"`), from any OS | ops over HTTPS | Working method, then **Cloud iPhone** |
  68. | **Android on the desk** - USB or paired Wi-Fi | the harness finds it | accessibility tree (exact), Vision OCR fallback on a Mac | adb `input` | Working method, then **Android** |
  69. | **Personal iPhone** - the user's own phone, through the Mac's iPhone Mirroring window | the harness opens Mirroring and presses Connect; the user locks the phone | screenshots + Vision OCR | HID-level CGEvents into the window | Working method, then **iPhone** |
  70. **"My phone" means the personal one.** The personal iPhone and the cloud
  71. iPhone are two different devices with different apps and logins: the first is
  72. in the user's pocket and mirrored onto the Mac, the second lives in a data
  73. centre and is reached over HTTPS. When the user says "my phone", "my iPhone",
  74. or names an app they use, drive the personal iPhone. Use a cloud phone only
  75. when the user names it ("the cloud phone", `cloud start`) or the task is
  76. clearly about it. If the personal phone cannot be reached, say so and ask;
  77. never fall through to a cloud session that happens to be running, even one
  78. the user started - it is theirs, and it is a different phone.
  79. `phone-harness config` shows the default platform (`ios` on a Mac, `android`
  80. elsewhere) and every other setting. `phone-harness cloud` shows whether a cloud
  81. phone is attached; while one is, scripts drive it regardless of the config
  82. default. `PHONE_HARNESS_PLATFORM=android` overrides per call. On a cloud
  83. iPhone any explicit platform selects a local backend; see **Cloud iPhone**.
  84. **When not to use any of them:** if the task is doable on the Mac or the web -
  85. a website, an API, an app with a web equivalent - do it there and leave the
  86. phone alone. Use a phone when the task genuinely needs one: phone-only apps,
  87. things tied to a phone number or 2FA, checking how something looks on a phone.
  88. ## Working method (every phone)
  89. ```bash
  90. phone-harness <<'PY'
  91. # task: report the OS version and model name from Settings
  92. # step: open Settings, find About, read the screen
  93. open_app("Settings")
  94. print([o["text"] for o in screen()][:10])
  95. PY
  96. ```
  97. - Invoke as `phone-harness` with a heredoc. Helpers are pre-imported. Start
  98. every script with `# task:` (the user's request in one sentence, identical
  99. across the scripts of one request) and `# step:` (what this script does).
  100. - **Tell the user what you are doing as you go.** A phone task is many short
  101. scripts and the user sees none of them: one line before each script (what
  102. you are about to do), one line after (what you saw). Never run two scripts
  103. in a row in silence. The `# task:`/`# step:` comments are not this - the user
  104. cannot see them.
  105. - **Read with `screen()`, not by eyeballing screenshots.** Every visible string
  106. comes back with a tap-ready centre: `[{text, confidence, source, x, y, w, h}]`.
  107. Filter in Python before printing. `tap_text("Weather")` taps by label and,
  108. on failure, raises with what IS visible - read the exception before retrying.
  109. `screenshot()` costs more but shows what OCR cannot: icons, images, state.
  110. - **Act, verify, adapt.** There is no DOM to assert against and no return value
  111. that means "it worked", so:
  112. 1. Name what should change before you act - a title, a row, a field's
  113. contents. Most phone failures are silent no-ops; if you cannot name the
  114. expected change you cannot tell success from one.
  115. 2. Do one thing, then check that one thing: `wait_for_text("Got it")`,
  116. `wait_for_app("com.android.chrome")` or `wait_stable()`, then one `screen()`.
  117. Do not `wait(n)` for a screen to load.
  118. 3. Once a sequence is proven, batch it: a whole sub-task in one script is far
  119. faster than a call per turn. Keep one cheap check at the end.
  120. 4. When a check fails, isolate: re-run that one action, look at the screen,
  121. form one guess, test it. Do not re-run the batch hoping it lands.
  122. 5. Keep what you learn in the agent workspace's `agent_helpers.py`.
  123. A pip or uv install uses `~/.phone-harness/agent-workspace`, or
  124. `~/.local/share/phone-harness/agent-workspace` when
  125. `~/.phone-harness` is a git checkout, so an upgrade does not wipe
  126. it. A checkout of this repo uses its own `agent-workspace/`.
  127. `PH_AGENT_WORKSPACE` overrides that path. The directory is
  128. created from the defaults only when it is missing. An existing
  129. workspace is never overwritten. To get a fresh default, delete the
  130. workspace folder; it is recreated on the next run.
  131. - **The harness reports, you decide.** Helpers return observations, never a
  132. verdict on your intent. Diff two `screen()` sets, watch one label, count rows,
  133. poll until something appears - whatever matches what you asked for.
  134. - **Navigation:** `home()`, `back()`, `app_switcher()`, `open_app("Notes")`,
  135. `type_text("...")`, `press("enter")`, `long_press(x, y)`, `tap(x, y)`.
  136. - **`scroll` says what you want to SEE; `swipe` says which way the finger
  137. goes.** English disagrees the same way - "scroll down the page" and "swipe
  138. up for the next video" describe the same motion. `scroll("down")` reveals
  139. what is further down; `swipe("up")` is a thumb flick, the phrasing everyone
  140. uses for "next". `scroll`, `scroll_screen`, `scroll_until` and
  141. `scroll_collect` take the content direction; only `swipe` takes finger
  142. motion. `"left"`/`"right"` work on both. Which one moves what differs per
  143. phone - see each section.
  144. - **Walking a list:** `scroll_until(done)` stops when your predicate on the
  145. visible boxes is met; `scroll_collect(extract, key=...)` walks and de-dupes,
  146. returning `{items, stop, scrolls}` with `stop` of `'reached-end'` or
  147. `'max-scrolls'`. Both end on YOUR check, so an extractor that misses rows
  148. ends the walk early. `scroll_screen()` is the single step and returns
  149. `{before, after, boxes}` for you to judge. `at=` aims the gesture at an
  150. inner list or strip; only the scroll view under that point moves.
  151. - **`type_text` needs a focused field** on every phone. Tap the field, wait for
  152. the keyboard, then type; verify with `screen()`, because unfocused text goes
  153. elsewhere or nowhere.
  154. - **Consent.** This is the user's real phone and real accounts. Stop and ask
  155. before anything outward-facing or hard to reverse: sending, posting,
  156. following, purchasing, deleting, changing settings. Never type a PIN, a
  157. password or a 2FA code; on a cloud phone, hand the controls over instead
  158. (`phone-harness cloud open`, below).
  159. - **The harness reconnects what it can; the rest is the user's.** On a
  160. personal iPhone, start the task with `ensure_device()`: it opens iPhone
  161. Mirroring if it is closed, presses the window's Connect once, and brings
  162. the window to the front so the user can watch. Pairing, unlocking, tapping Allow,
  163. locking an iPhone that says "iPhone in Use" or "Timed Out", approving a
  164. cloud sign-in in the browser: when the harness still says the phone is not
  165. reachable, relay its message and ask - never loop-poll. Retry once after
  166. the user says it is done; that retry presses Connect again.
  167. ## Cloud phones
  168. There are two things. A **phone** is a row in the account catalog (`GET /me`
  169. -> `available`): a label, a platform, an id, and whether it is a temporary
  170. phone. The service keeps no default phone; a preference is the chooser's. A **session** is one of those phones running right now.
  171. `cloud ls` prints one row per catalog phone and writes any live session on
  172. that row (`*` is the session this process targets). A session lines up with
  173. the row that has the same device_id or profile_id; a session with neither
  174. lines up with the temporary row. A cloud iPhone session does not echo that
  175. id: its `device` is the word "iPhone", and its `profile` (`iphone-...`) matches
  176. no catalog row. The catalog row's `state` is `running` while a session holds
  177. it. A session whose ids match no row counts as unmatched. When exactly one
  178. row of that platform and kind is `running` and exactly one unmatched session
  179. has that same platform and kind, the session is written on that row. A
  180. session that still matches nothing is listed on its own.
  181. `cloud start` with no name starts this machine's `cloud.phone` setting
  182. (`config set cloud.phone "Desk phone"`) when one is set; otherwise it sends
  183. no `platform` or `kind` and gets the service's rule for an unnamed start
  184. (`GET /me` -> `unnamed`): a temporary Android. A name is a catalog id
  185. or label, or `ios` / `android` when only one entry has that platform. Several
  186. matches print the catalog and stop (a terminal may ask which one). The CLI
  187. turns the chosen row into `platform` and `kind`, plus `device_id` for an
  188. iPhone or `profile_id` for the saved Android. The temporary row sends neither
  189. id.
  190. What the helpers do next follows the session's link, not the platform name.
  191. An adb link is the Android section. A control link (`{url, token}`) is
  192. **Cloud iPhone**.
  193. ```bash
  194. phone-harness cloud # signed in? which session is attached? time left?
  195. phone-harness cloud ls # catalog, live sessions inlined; --json for the rows
  196. phone-harness cloud start # cloud.phone if set, else a temporary Android; opens the live view
  197. phone-harness cloud start ios --for 15m # one iPhone, fifteen minutes
  198. phone-harness cloud start "Temporary phone" --for 90s
  199. phone-harness --session SID <<'PY' # this process only; or PHONE_HARNESS_SESSION
  200. print(screen_info())
  201. PY
  202. phone-harness cloud stop # ends the session; returns at once
  203. phone-harness cloud phone # the saved Android's stored state
  204. phone-harness cloud watch # reopen the read-only live view (--print to share)
  205. phone-harness cloud open # the dashboard: the user takes the controls
  206. ```
  207. - **Pin a session when more than one can be up.** `--session SID` or
  208. `PHONE_HARNESS_SESSION` selects one phone for this process, and every
  209. `cloud` command and helper honours it. With neither, a person at a terminal
  210. gets the attached session: the last `cloud start`, or `cloud use SID`.
  211. `cloud use` rewrites that shared fallback. Parallel agents should pass
  212. `--session` and leave `cloud use` alone, or they will steal the fallback
  213. from each other.
  214. - **A cloud Android bills by the minute while it is up; a cloud iPhone is a
  215. flat subscription and does not.** Start it once and keep it for the whole
  216. conversation. Stop an Android when the user is done, and say that you did.
  217. An iPhone can stay running.
  218. `--for` is the length: `90s`, `15m` / `15min`, or a bare number of minutes.
  219. Default 15, cap 30 (`cloud.minutes`, `cloud.max_minutes`).
  220. - **Stop before the deadline; a session cannot be extended.** One that simply
  221. runs out keeps a saved Android's data but not its running state. The harness
  222. warns on stderr under two minutes; `phone-harness cloud` shows the time left.
  223. `cloud stop` returns at once. Stopping an adb phone that has a profile saves
  224. it; `cloud start` of that phone waits out a save still in progress. A
  225. control link keeps its data on the phone, so there is nothing to wait for.
  226. - **Already running.** Starting a phone this machine already holds for the
  227. same catalog choice reuses that session. If the service says it is held
  228. elsewhere (`409` `profile_running`), the message names the SID and says to
  229. attach with `--session SID`. Do that; do not start a second one. `409`
  230. `device_unavailable` means the phone is granted but not on its host.
  231. `409` `session_limit` names how many sessions are in use. `503`
  232. `phones_busy` includes `Retry-After`. `403` `account_blocked` means the account
  233. is blocked from starting phones; a `403` about credit or balance means add credit. A `400` that
  234. names no such phone reprints the catalog.
  235. - **`Not signed in`** means the user runs `phone-harness cloud login` and
  236. approves it in a browser. Relay that; you cannot do it for them.
  237. - **Watching versus controlling.** `cloud start` opens the live view so the
  238. user can watch. `cloud watch` reopens it. For a password, a 2FA prompt, or
  239. the user taking over, run `phone-harness cloud open` and wait until they say
  240. they are done.
  241. - **`cloud phone` is the saved Android only** (`GET /me/profile`).
  242. `cloud phone reset --yes` wipes that phone's stored state: apps, logins,
  243. and data. The CLI has no iPhone reset; that is the operator's.
  244. - **A cloud session that ends does not hand you another phone.** If the
  245. session this command is using expires, the error says `session <sid> expired`.
  246. If the service loses it, the error says the cloud phone failed and is
  247. unreachable. That command stops. It does not drive another phone, and the
  248. CLI does not start a replacement. The saved note of that dead session is
  249. removed once the error has been reported, or once `phone-harness cloud` has
  250. shown it once, so later status output does not repeat it. A command that
  251. hits the dead session while that note is still there fails with the same
  252. message. Start the same catalog phone again with a longer duration and
  253. continue the task: `phone-harness cloud start <same phone> --for <longer>`.
  254. `<same phone>` is that phone's id or label (or `ios` / `android` when only
  255. one entry has that platform). `<longer>` is a duration longer than the
  256. session that just ended (`90s`, `15m`, or a number of minutes). A
  257. `--session` pin whose session expired is the same error, naming that sid.
  258. A missing cloud state file still means no cloud phone was attached, and the
  259. personal phone is used as today. An unreadable state file is an error, not
  260. a cue to switch phones. `cloud stop` and a new `cloud start` also drop a
  261. note that has not been reported yet.
  262. - The adb address the CLI shows is not a secret; the unlock code is. Do not
  263. look for it, print it, or ask the user for it.
  264. ## Cloud iPhone
  265. Start it from the catalog: `phone-harness cloud start ios` when that is
  266. unambiguous, or the entry's id or label (`cloud ls`). The create body is
  267. `platform: "ios"`, `kind: "device"`, and `device_id` set to that entry's id.
  268. A ready session carries `control: {url, token, expires_at}` and no `adb`.
  269. From then on the helpers drive it over HTTPS. This machine needs no adb, no
  270. shell, and no Mac. It is a phone granted to the user's account and kept
  271. between sessions - not the personal iPhone mirrored on the Mac, which is a
  272. different device with its own apps and logins. It is a flat subscription and
  273. does not bill by the minute, so there is no need to stop it to save credit, or
  274. to ask before starting it. `cloud stop` returns at once; the phone keeps its own data. There is no Android-style
  275. disk save, and nothing on this phone for `cloud phone` to wipe.
  276. An Android session can stay up beside it. Target one with `--session SID`.
  277. `cloud use SID` only changes the terminal fallback.
  278. Length, `ls`, `watch`, and `stop` are the **Cloud phones** commands
  279. above. An iPhone session reports `billing_mode: disabled`: the subscription covers it, with no per-minute charge. Run scripts as
  280. plain `phone-harness` (with `--session` when you must pin one).
  281. `PHONE_HARNESS_PLATFORM=...` selects a local backend and skips the attached
  282. cloud phone.
  283. The mirroring section describes a Mac window. This phone is the HTTPS path.
  284. The client posts `{"op", "kw"}` to `<control url>/op` and reads the host's
  285. list from `GET <control url>/ops` (`remote.py`). `supports(op)` is that list.
  286. `send` of an op the host did not list raises `Unsupported` before the POST
  287. (`this cloud phone cannot '...'`). The op names and the shapes below are the
  288. client vocabulary in `transport.py`. Result shapes are what this repo
  289. documents or decodes.
  290. | Op | Arguments | Result this client uses |
  291. | --- | --- | --- |
  292. | `screen.capture` | `path` optional | `(png_path, bounds)`. On the wire the host returns `png_b64` and `bounds`; the client writes the PNG. Timeout 60s. |
  293. | `screen.bounds` | | `{x, y, w, h, id}` or None |
  294. | `screen.require` | | bounds, or raise |
  295. | `screen.text` | `min_confidence` | `[{text, confidence, source, x, y, w, h}]`. `source` is `"pixels"` or `"tree"`. |
  296. | `screen.text_pixels` | `min_confidence` | same, forced through pixel OCR |
  297. | `input.tap` | `x, y` | host result, passed through |
  298. | `input.press` | `x, y, duration` | host result. Long press. |
  299. | `input.drag` | `x1, y1, x2, y2, duration, steps` | host result |
  300. | `input.scroll` | `x, y, dy, dx, steps` | host result. Content-space deltas. |
  301. | `input.keys` | `combo` | host result |
  302. | `input.text` | `s, delay, keystrokes` | host result. Timeout `max(60, 0.2 * len(s) + 30)` seconds. |
  303. | `nav.home` / `nav.back` / `nav.recents` | | host result, or `Unsupported` when the host says `unsupported` |
  304. | `apps.launch` | `name`, `fresh` | app id. Timeout 90s. |
  305. | `apps.current` | | app id or None |
  306. | `apps.list` | `include_system` | `[app id]`. Timeout 90s. |
  307. | `session.state` | | backend-defined string; `'ready'` when usable |
  308. | `session.require` | | bounds, or raise telling the user what to do |
  309. | `session.refocus` | | `None` here. The client does not call the host. |
  310. | `session.detail` | | a string. Local. |
  311. | `focus.probe` | | an opaque snapshot. Local: `(True,)`. |
  312. | `focus.diff` | `before, after` | `{raised, stole_focus}`. Local: both false. |
  313. | `url.open` | `url` | app id. Opens an https link in the phone's browser, whole. Timeout 60s. |
  314. | `tree` | | `[node]` where the device has one |
  315. | `raw` | backend-specific | host result |
  316. Trust `GET /ops` when the host's list is shorter than this table.
  317. ### Files on a cloud iPhone
  318. There is no shell, so bytes move through storage links the service signs and
  319. the phone's `url.open` op. Two helpers wrap it, and they follow the
  320. attached session (or `--session`) like every other helper. Files go to the
  321. phone only; nothing comes back this way.
  322. - `push_file(path)` uploads the file and opens it in the phone's browser
  323. under its own name (a JPEG shows as an image, an MP4 plays). From there
  324. the phone's own UI decides what to do with it; look at the screen and act
  325. on what is there, as with any other task.
  326. - `open_url(url)` is the op behind both: any https link, opened whole, with
  327. no typing and no clipboard. Check `supports('url.open')` first; an older
  328. host lists no such op.
  329. On a local phone both raise `Unsupported` ("cloud phones only for
  330. now"). Links expire within fifteen minutes and with the session; the files
  331. are removed when the session ends. A link is a credential: do not paste one
  332. into notes or output.
  333. ## Android
  334. The phone is reached over adb - a USB phone if plugged in, else the paired
  335. Wi-Fi phone, else the attached cloud Android - so there is nothing to select.
  336. `phone-harness android` shows known phones and what is attached.
  337. ```bash
  338. PHONE_HARNESS_PLATFORM=android phone-harness <<'PY'
  339. open_app("chrome"); wait_for_app("com.android.chrome")
  340. tap_ui("Got it") # exact label from the accessibility tree
  341. PY
  342. ```
  343. - **Coordinates are device pixels;** the screenshot is 1:1 with `tap(x, y)`.
  344. - **`screen()` is the accessibility tree** (`source: "tree"`) - exact, no
  345. misreads. Prefer `ui()` / `find_nodes()` / `tap_ui()`: they also see
  346. elements with no visible text (icons with a content-description, fields by
  347. resource-id like `tap_ui("url_bar")`).
  348. - **Some screens never give up their tree** - a playing video, some Settings
  349. pages. On a Mac, `screen()` then reads the screenshot with Vision instead
  350. (`source: "pixels"`, fuzzier, still tap-ready; the first read costs ~12s
  351. while uiautomator gives up, later reads are fast). Elsewhere it raises
  352. saying so; take a `screenshot()` and look at it instead of retrying.
  353. `ui()` / `tap_ui()` need the real tree and keep raising on such screens.
  354. - **adb is the native language here, and it is first-class.** `shell(cmd)`
  355. runs `adb shell cmd` on whichever phone the harness chose: `shell("input tap
  356. 360 640")`, `shell("input keyevent KEYCODE_BACK")`, `shell("am start -n
  357. pkg/.Activity")`, `shell("dumpsys notification --noredact")`, `shell("pm list
  358. packages -3")`. The input helpers are one-line wrappers over the same
  359. commands - use whichever you think in. What the harness adds: finding and
  360. reconnecting the phone, and the screen as a short list instead of XML.
  361. - **`scroll` and `swipe` are different gestures here.** `scroll` moves the
  362. content and stops: no momentum, the same distance every time, so use it (and
  363. `scroll_until` / `scroll_collect`) to walk a list without skipping rows.
  364. `swipe` is a flick and coasts past whatever was next - right for "next
  365. video" or changing pages, wrong for reading a list.
  366. - `open_app("TikTok")` takes the name a person would say, a package id, or a
  367. fragment of one, and returns the package it launched; when nothing matches,
  368. the error lists what is installed. `back()`, `current_app()`, `list_apps()`
  369. exist.
  370. - `press()` takes single keys only (`"enter"`, `"back"`, `"tab"`); chords
  371. raise Unsupported. `type_text` types ASCII: adb cannot type emoji or
  372. accented letters.
  373. - **Verify cheaply, then read.** adb reports nothing about outcomes - a tap on
  374. empty space "succeeds". After an action: `wait_for_app(...)` (~0.1s a poll)
  375. or `wait_for_text(...)` (returns the box or None), then `ui()`/`screen()` once.
  376. The tree costs ~2-3s a call on a slow phone and a screenshot ~0.5s, so
  377. batching a proven sub-task is worth a lot; a batch of unverified steps fails
  378. silently and tells you nothing about which one broke.
  379. - No focus to keep: nothing on the Mac has to be frontmost, and
  380. `interruption(before, after)` always reports nothing disturbed.
  381. - **A desk phone locks itself** after its screen timeout. `connection_state()`
  382. reports `locked`; taps and `screen()` refuse with the same message. Ask the user
  383. to unlock - never type a PIN. `screenshot()` still works locked, so you can
  384. show them what you see. For a task longer than a minute, ask the user, then
  385. `phone-harness android awake --bg` keeps it awake without changing any phone
  386. setting; `phone-harness android rest` ends that. Cloud phones do not lock.
  387. - Connecting a desk phone is the user's job (USB debugging + Allow, or
  388. Wireless debugging + `phone-harness android pair CODE`); on `no-device` the
  389. error names the missing step - relay it, don't retry-loop.
  390. - **A saved primary that is unreachable is a failure, not a different phone.**
  391. If `phone-harness android use` has named a primary and that phone cannot
  392. be reached, the error says the primary is unreachable. Do not expect the
  393. harness to connect some other paired or mDNS phone instead. When no
  394. primary is configured, a USB phone that is plugged in is still used.
  395. ## iPhone (iPhone Mirroring)
  396. The Mac's iPhone Mirroring app renders the phone as a window; the harness
  397. captures that window and OCRs it with Vision for eyes, and posts HID-level
  398. events into it for hands. All coordinates are global macOS screen points.
  399. - **Start every task with `ensure_device()`.** It is the first call on every
  400. phone, including iPhone Mirroring. It opens iPhone Mirroring
  401. if it is not running, brings the window to the front so the user sees what
  402. you see, presses the interstitial's Connect once through accessibility,
  403. and waits for the live stream. A phone that is in use shows up within
  404. about three seconds and it raises then, with the window's own words,
  405. rather than sitting out a timeout. It is the cheapest first line a script
  406. can have: a phone that cannot be driven fails there, not five taps in.
  407. Call it once in the first script of a task, not before every action.
  408. After that, the default build works the phone **without taking the user's focus**: capture
  409. is by window id and taps and keystrokes are event records delivered straight
  410. to the app. Scrolling is the exception - macOS routes a scroll to whichever
  411. window sits under the pointer, so a scroll raises the mirroring window for
  412. the length of the gesture and hands focus straight back. Expect a brief
  413. flicker on scrolls and nothing on anything else. `PHONE_HARNESS_BACKGROUND=0`
  414. forces the classic path, which focuses before every action.
  415. - **Icons without labels:** `screenshot()`, view the image, and use
  416. `tap_image_point(x, y, image_size=...)` with coordinates measured in the
  417. screenshot. Do **not** pass screenshot pixel coordinates to `tap()`: it
  418. expects global screen points. If using `tap()`, convert with `image_point()`
  419. from the current `screen_info()`; never estimate the window offset.
  420. - **Use `scroll` for anything scrollable.** On macOS 26 a vertical touch-drag
  421. is dropped, so `swipe("up")`/`swipe("down")` move nothing in a list or a
  422. feed - measured on Settings and on TikTok. Horizontal still works, so
  423. `swipe("left")` / `swipe("right")` remain the way to flip Home Screen pages
  424. and carousels, which a scroll cannot do. (Breaking change: `scroll` used to
  425. take finger motion too, so the old `scroll("up")` is today's
  426. `scroll("down")`. `swipe` is unchanged.)
  427. - `open_app("Notes")` goes through Spotlight.
  428. - **Home-Screen labels are not tap targets.** `tap_text("Weather")` hits the
  429. label and nothing happens; the icon is ~35 points above it. Use
  430. `tap_icon("Weather")` (agent helper) on the Home Screen; `tap_text` works
  431. for in-app buttons and list rows.
  432. - **`type_text` pastes; it does not type.** The keystroke path runs through
  433. iOS autocorrect, which rewrites words as they land ("Thu" becomes "thru").
  434. Pass `keystrokes=True` for fields that need real key events. The typed text
  435. stays on the Mac clipboard afterwards. If a tap will not take focus,
  436. `press("tab")` moves between fields.
  437. - **What the harness cannot clear is physical.** Connect only works while
  438. the iPhone is locked; "iPhone in Use" and "Timed Out ... due to iPhone use"
  439. mean it was not. `ensure_device()` presses Connect once and then raises
  440. quoting the screen (`connection_state()` is `ready` / `blocked` /
  441. `no-window` / `not-running`). `connection_detail()` returns the labels and
  442. button titles in the window, with no screenshot, so you can see why it is
  443. blocked. **STOP and relay the error.** It names the next step: lock the
  444. iPhone when the window says in use, timed out, or lock your iPhone; approve
  445. or cancel when the Mac asks for Touch ID or a password (never type one);
  446. otherwise clear the window on the Mac. Retry once when the user says it is
  447. done. When the phone was in use, the app reconnects by itself the moment
  448. the phone is locked, so that retry usually finds it live; if not, it
  449. presses Connect again. Do not press it yourself in a loop and do
  450. not tap the window (an interstitial is a Mac view; taps meant for the
  451. phone go nowhere). **Unlocking the physical phone pauses the session**
  452. ("iPhone in Use") - the same rule applies mid-task. Taps, scrolls, swipes,
  453. typing, key presses, `open_app`, and `home` raise that same
  454. message and do not press Connect. `screenshot()` and `screen_info()` still
  455. work while the mirroring window is open, including when the state is
  456. blocked. Call `screenshot()` to see what the window shows. Ask the user
  457. to do what the error says, then call `ensure_device()` and continue from
  458. the step that raised.
  459. The Mac login prompt that
  460. sometimes appears in the window is never typed into.
  461. - **Unfocused input is swallowed silently - for events you post yourself.**
  462. The helpers are immune in the background build, but raw CGEvents and the
  463. `PHONE_HARNESS_BACKGROUND=0` path need the window frontmost: `activate()`
  464. before posting, and re-activate if a click steals focus. The failure looks
  465. exactly like "scrolling is broken" or "the list already ended" - when a
  466. gesture changes nothing on screen, check focus before inventing another
  467. theory.
  468. - **The window is a video stream.** macOS accessibility sees nothing inside
  469. it; AppleScript `click at` fails silently. Only HID-level CGEvents work.
  470. - **The window moves.** Never cache coordinates across calls; `screen()` and
  471. `swipe()` re-query bounds every time.
  472. - Mouse taps map to touches 1:1, but there is no multi-touch: no pinch, no
  473. two-finger gestures.
  474. - Raw Quartz is the escape hatch: `import Quartz` in your script for anything
  475. the helpers don't cover - but raw CGEvents don't ride the helpers' delivery
  476. path, and where they land is its own question per event type. Check what
  477. actually happened on screen rather than assuming the event arrived.
  478. For task-specific edits, use the agent workspace's `agent_helpers.py`:
  479. `~/.phone-harness/agent-workspace` for a pip or uv install (or
  480. `~/.local/share/phone-harness/agent-workspace` when `~/.phone-harness`
  481. is a git checkout). A checkout of this repo uses its own
  482. `agent-workspace/`. `PH_AGENT_WORKSPACE` overrides that. Installs seed
  483. that directory from the packaged defaults when it does not exist yet;
  484. an existing workspace is left as it is. To get a fresh default, delete
  485. the workspace folder; it is recreated on the next run. For setup or
  486. permission problems, read
  487. https://github.com/phone-harness/phone-harness/blob/main/install.md.
  488. ## Updates
  489. `phone-harness --version` prints `phone-harness <version>`.
  490. `phone-harness update` brings this install up to date. A uv tool runs
  491. `uv tool upgrade phone-harness`. A git checkout runs `git pull --ff-only`
  492. in that checkout. Anything else prints `pip install -U phone-harness` and
  493. does not run it. The CLI never updates itself. Once a day, on a terminal,
  494. it may print one stderr line when PyPI has a newer version:
  495. ```
  496. phone-harness X.Y.Z is available (you have A.B.C): run phone-harness update
  497. ```
  498. Relay that line to the user. `PHONE_HARNESS_NO_UPDATE_CHECK=1` turns the
  499. notice off. A network problem never fails the command.
README.md · README · 134 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. <a href="https://phone-harness.com?utm_source=github&utm_medium=readme&utm_campaign=header">
  3. <picture>
  4. <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/logo-animated-dark.svg?v=6f742c0">
  5. <img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/logo-animated.svg?v=6f742c0" alt="phone-harness" width="560">
  6. </picture>
  7. </a>
  8. <br>
  9. <picture>
  10. <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/tagline-dark.svg?v=6f742c0">
  11. <img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/tagline.svg?v=6f742c0" width="600" height="48" alt="Let your agent use your phone.">
  12. </picture>
  13. </div>
  14. ---
  15. <p>
  16. <a href="https://phone-harness.com/docs?utm_source=github&utm_medium=readme&utm_campaign=button-docs"><picture><source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/button-docs-dark.svg?v=6f742c0"><img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/button-docs.svg?v=6f742c0" width="98" height="40" align="top" alt="Docs"></picture></a>
  17. <a href="https://phone-harness.com/dashboard?utm_source=github&utm_medium=readme&utm_campaign=button-cloud"><picture><source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/button-cloud-dark.svg?v=6f742c0"><img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/button-cloud.svg?v=6f742c0" width="104" height="40" align="top" alt="Phone Harness Cloud"></picture></a>
  18. <a href="https://phone-harness.com/dashboard?utm_source=github&utm_medium=readme&utm_campaign=callout"><picture><source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/callout-dark.svg?v=6f742c0"><img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/callout.svg?v=6f742c0" width="230" height="40" align="top" alt="Try your own cloud Android phone for free!"></picture></a>
  19. <a href="https://discord.gg/7FwGsjvjS"><picture><source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/icon-discord-dark.svg?v=6f742c0"><img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/icon-discord.svg?v=6f742c0" width="32" height="40" align="right" alt="Discord"></picture></a><a href="https://www.linkedin.com/company/phone-harness"><picture><source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/icon-linkedin-dark.svg?v=6f742c0"><img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/icon-linkedin.svg?v=6f742c0" width="32" height="40" align="right" alt="LinkedIn"></picture></a><a href="https://github.com/phone-harness/phone-harness"><picture><source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/icon-github-dark.svg?v=6f742c0"><img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/icon-github.svg?v=6f742c0" width="32" height="40" align="right" alt="GitHub"></picture></a><a href="https://x.com/phone_harness"><picture><source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/icon-x-dark.svg?v=6f742c0"><img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/icon-x.svg?v=6f742c0" width="32" height="40" align="right" alt="X"></picture></a>
  20. </p>
  21. ---
  22. Enable Claude Code, Codex, or any agent to use your real phone. The agent can
  23. scroll, tap, type, and read your phone with no jailbreak, no Xcode, and nothing
  24. installed on the phone.
  25. > **iPhone** through the Mac's iPhone Mirroring window <br/>
  26. > **Android** over adb from macOS, Linux or Windows <br/>
  27. > **Phone Harness Cloud** phone: an Android over adb, or an iPhone over HTTPS
  28. ```
  29. ● agent: wants to open Weather
  30. │
  31. ● find_text("Weather") → (400, 468)
  32. │
  33. ● tap(400, 468) → reads the screen → forecast is up
  34. ✓ done
  35. ```
  36. Try [Phone Harness Cloud](https://phone-harness.com/dashboard?utm_source=github&utm_medium=readme&utm_campaign=cloud) → Hosted iPhones and Androids with stealth, real numbers, 2FA, and unlimited devices
  37. Get started by sending the [setup prompt](https://phone-harness.com?utm_source=github&utm_medium=readme&utm_campaign=setup-prompt) to your
  38. coding agent.
  39. ## Demo
  40. **Task:** "Buy me a Waymo to Delah Coffee from my current location."
  41. <img src="https://raw.githubusercontent.com/phone-harness/media/main/phone-harness/demo-waymo.gif?v=6f742c0" width="100%" alt="An agent orders a Waymo to Delah Coffee on a real iPhone through phone-harness">
  42. ## Setup
  43. For the CLI, send this setup prompt to your agent (Claude Code, Codex, Grok Bot, and more):
  44. ```text
  45. Install or upgrade phone-harness to the latest stable version with uv using Python 3.12, register the skill from `phone-harness skill`, then read https://github.com/phone-harness/phone-harness/blob/main/onboarding.md and walk me through it. Follow https://github.com/phone-harness/phone-harness/blob/main/install.md if setup or connection fails.
  46. ```
  47. The agent installs the CLI, asks which phone to use, and walks you through the steps that need your hands. What each phone needs:
  48. | Phone | Needs |
  49. | --- | --- |
  50. | **iPhone** | macOS 15+, iPhone Mirroring paired to an iOS 18+ iPhone, Accessibility and Screen Recording permission |
  51. | **Android** | adb and USB debugging, on macOS, Linux or Windows |
  52. | **Cloud Android / iPhone** | [Phone Harness Cloud](https://phone-harness.com/dashboard?utm_source=github&utm_medium=readme&utm_campaign=setup-table) account; cloud Android also needs adb |
  53. `phone-harness doctor` checks the setup. Full details: [install.md](install.md) and [setup/](setup/).
  54. **MCP:** add `https://mcp.phone-harness.com/mcp` to any MCP client ([setup](https://github.com/phone-harness/mcp)). Cloud Android and iPhone only for now.
  55. **Hermes Agent:** `hermes plugins install phone-harness/phone-harness/integrations/hermes`. See [integrations/hermes](integrations/hermes/README.md).
  56. **OpenClaw:** `openclaw plugins install clawhub:phone-harness`. See [integrations/openclaw](integrations/openclaw/README.md).
  57. ## Usage
  58. ```bash
  59. phone-harness <<'PY'
  60. open_app("Notes")
  61. tap_text("New Note")
  62. type_text("hello from the harness")
  63. print([o["text"] for o in screen()][:10])
  64. PY
  65. ```
  66. Helpers are pre-imported; [helpers.py](src/phone_harness/helpers.py) has the full list. The agent saves its own helpers in `~/.phone-harness/agent-workspace`.
  67. ## Phone Harness Cloud
  68. No phone on your desk, or need more than one? [Phone Harness Cloud](https://phone-harness.com/dashboard?utm_source=github&utm_medium=readme&utm_campaign=cloud-section) hosts them for you:
  69. - **Android phones** on demand: a fresh one per session, or a saved phone that keeps its apps and logins.
  70. - **Dedicated real iPhones**, driven over HTTPS from any OS.
  71. - **A live view** of the phone in your browser while the agent works.
  72. ```bash
  73. phone-harness cloud login # once: approve in your browser
  74. phone-harness cloud ls # phones on the account, and any live session
  75. phone-harness cloud start # cloud.phone if set, else a temporary Android; 15 min (--for 90s, 15m)
  76. phone-harness cloud start ios # the account's iPhone, when only one is listed
  77. phone-harness <<'PY'
  78. print(screenshot())
  79. PY
  80. phone-harness cloud stop # ends the session; a saved Android is written back
  81. ```
  82. The same helpers drive a cloud phone: an Android over adb, an iPhone over HTTPS. `cloud start` takes a phone's name or id from `cloud ls`, or `ios` / `android`; with none it starts a temporary Android. `phone-harness cloud` lists every command.
  83. ## How it works
  84. Every phone sits behind the same helpers, so a script written for one runs on the others.
  85. - **iPhone:** captures the iPhone Mirroring window, reads it with Apple's Vision OCR, and sends taps, swipes and typing as input events.
  86. - **Android:** adb for screenshots and input, the accessibility tree for text. USB or Wi-Fi.
  87. - **Cloud Android:** the same as Android, over adb to a hosted phone.
  88. - **Cloud iPhone:** ops over HTTPS.
  89. `phone-harness config set platform ios|android` picks the default phone. `phone-harness update` upgrades the CLI, and it prints one line when a new version is out (`PHONE_HARNESS_NO_UPDATE_CHECK=1` silences it).
  90. ## Limits
  91. - Unlocking the iPhone pauses mirroring; a PIN-locked Android needs the user.
  92. - OCR sees text, not icons. Unlabeled controls need a screenshot and a
  93. vision-capable model.
  94. - No multi-touch, no camera or Face ID flows. DRM video renders black.
  95. - Connecting the phone is always the user's job.
  96. ## Sponsor
  97. phone-harness is free and maintained in my own time.
  98. [Sponsoring](https://github.com/sponsors/ShawnPana) keeps it that way.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.