Skip to content
Back to search

pocketpilot

Let an AI agent drive a real iPhone from your Mac. Local MCP server, open source.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/vibiz-ai--pocketpilot/badge.svg)](/listings/vibiz-ai--pocketpilot)

HTML

<a href="/listings/vibiz-ai--pocketpilot"><img src="/api/listings/vibiz-ai--pocketpilot/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/Vibiz-ai/pocketpilot.git && cd pocketpilot
scripts/build-app.sh          # release build -> build/PocketPilot.app, ad-hoc signed
open build/PocketPilot.app        # or drag it into /Applications first

`build-app.sh` writes an Info.plist with bundle id `ai.vibiz.pocketpilot`, the Camera and Bluetooth usage strings and `LSMinimumSystemVersion 14.0`, then ad-hoc signs the bundle. Because it is ad-hoc signed, the first launch may need right-click > Open.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 200 lines
  1. # PocketPilot
  2. Let an AI agent drive a real iPhone from your Mac. PocketPilot is a small macOS app that shows the iPhone's screen to an agent,
  3. reads the text on it, and taps, swipes and types for it, all exposed as a local MCP server.
  4. Open source (MIT), written in Swift, no third-party dependencies, nothing leaves your Mac.
  5. ## How it works
  6. ```
  7. USB cable (screen + lockdown)
  8. +--------+ -----------------------------------> +--------------------------------------------+
  9. | | | PocketPilot.app (your Mac) |
  10. | iPhone | Bluetooth HID (keyboard, mouse, | |
  11. | | <----------------------------------- | Capture -> latest frame -> screenshot |
  12. +--------+ consumer control) | Vision -> OCR text + centre x,y |
  13. | Device -> Trust, AssistiveTouch toggle |
  14. | Bluetooth -> HID reports (0xA1 packets) |
  15. | Control -> calibration, taps, swipes |
  16. | MCP -> http://127.0.0.1:47900/mcp |
  17. +----------------------^---------------------+
  18. | JSON-RPC over HTTP
  19. Claude Code / Cursor / Codex
  20. ```
  21. - **Screen.** An iPhone plugged in by USB appears as a capture device once CoreMediaIO's
  22. `kCMIOHardwarePropertyAllowScreenCaptureDevices` is turned on (the same thing QuickTime does). PocketPilot keeps the latest
  23. frame and serves screenshots downscaled to a 590-pixel-wide "screenshot space".
  24. - **Text.** Vision's accurate text recognizer turns each screenshot into a list of `{text, x, y}` in screenshot coordinates,
  25. so agents can `tap_text("Settings")`.
  26. - **Input.** The Mac publishes a classic Bluetooth HID service (keyboard + relative mouse + consumer control). You pair it
  27. once from the iPhone's Settings > Bluetooth. With AssistiveTouch on, iOS shows a pointer for the mouse.
  28. - **Pointer.** The mouse is relative and iOS accelerates it, so PocketPilot calibrates: it parks the pointer in a corner,
  29. measures how many pixels each mouse step moves it, and stores the curve in
  30. `~/Library/Application Support/PocketPilot/calibration.json`. A tap is: park, move along the curve, find the pointer dot in
  31. the frame, correct, click.
  32. - **USB lockdown.** If Apple's MobileDevice framework is present, PocketPilot uses it to see the iPhone attach, read its name and
  33. iOS version, ask it to Trust this Mac, and turn AssistiveTouch on. If it is missing, those steps fall back to manual
  34. instructions.
  35. - **MCP.** A Streamable-HTTP MCP server (protocol `2025-06-18`) bound to `127.0.0.1:47900`. Every action returns a fresh
  36. screenshot plus the OCR text.
  37. ## Requirements
  38. - A Mac with macOS 14 Sonoma or later and Bluetooth.
  39. - An iPhone and a USB cable that carries data.
  40. - Swift 5.9 or later to build (Xcode or `xcode-select --install`).
  41. ## Install and build
  42. ```sh
  43. git clone https://github.com/Vibiz-ai/pocketpilot.git && cd pocketpilot
  44. scripts/build-app.sh # release build -> build/PocketPilot.app, ad-hoc signed
  45. open build/PocketPilot.app # or drag it into /Applications first
  46. ```
  47. `build-app.sh` writes an Info.plist with bundle id `ai.vibiz.pocketpilot`, the Camera and Bluetooth usage strings and
  48. `LSMinimumSystemVersion 14.0`, then ad-hoc signs the bundle. Because it is ad-hoc signed, the first launch may need
  49. right-click > Open.
  50. ## Onboarding
  51. The setup wizard opens on first launch, and again from the menu with **Run Setup Again…** (Cmd+Shift+R). Every step
  52. checks itself live and turns green; when one fails it says what failed, why, and how to fix it right there.
  53. 1. **Permissions.** Allow Camera (macOS delivers the iPhone screen as a camera-like device) and Bluetooth. Buttons ask
  54. for each permission or open the right pane of System Settings.
  55. 2. **Plug in your iPhone.** Connect it by USB and unlock it. PocketPilot asks it to Trust this Mac; tap Trust and enter your
  56. passcode. The step shows the iPhone's name once it is trusted.
  57. 3. **Screen.** The live preview of the iPhone screen appears.
  58. 4. **Bluetooth.** The Mac becomes discoverable. On the iPhone open Settings > Bluetooth and tap the name the wizard shows
  59. (your Mac's Bluetooth name). The step completes when both HID channels are open.
  60. 5. **AssistiveTouch.** PocketPilot tries to switch it on over USB. If that is not possible it shows the path:
  61. Settings > Accessibility > Touch > AssistiveTouch. It then wiggles the mouse and confirms the pointer is visible.
  62. 6. **Calibration.** About 30 seconds with a progress bar. Keep the iPhone unlocked and do not touch it.
  63. 7. **Connect your agent.** Copy the command or click **Add to Claude Code**, then **Test it** calls `status` over MCP.
  64. After setup the main window shows the live screen, status pills (USB, Screen, Bluetooth, AssistiveTouch, Calibration,
  65. MCP), the first problem with its fix, and a log.
  66. ## Connect an agent
  67. The server lives at `http://127.0.0.1:47900/mcp` while PocketPilot is running.
  68. **Claude Code** (all projects):
  69. ```sh
  70. claude mcp add --transport http --scope user pocketpilot http://127.0.0.1:47900/mcp
  71. ```
  72. or for one project, `.mcp.json`:
  73. ```json
  74. { "mcpServers": { "pocketpilot": { "type": "http", "url": "http://127.0.0.1:47900/mcp" } } }
  75. ```
  76. **Cursor**, `~/.cursor/mcp.json`:
  77. ```json
  78. { "mcpServers": { "pocketpilot": { "url": "http://127.0.0.1:47900/mcp" } } }
  79. ```
  80. **Codex**:
  81. ```sh
  82. codex mcp add pocketpilot --url http://127.0.0.1:47900/mcp
  83. ```
  84. or `~/.codex/config.toml`:
  85. ```toml
  86. [mcp_servers.pocketpilot]
  87. url = "http://127.0.0.1:47900/mcp"
  88. ```
  89. Any other Streamable-HTTP MCP client works with the same URL.
  90. ## Tools
  91. Coordinates are pixels in the most recent screenshot (origin top-left, about 590 wide). Every action returns a fresh
  92. screenshot and the on-screen text with centre coordinates.
  93. | Tool | Arguments | What it does |
  94. |---|---|---|
  95. | `status` | | Health of USB, Screen, Bluetooth, AssistiveTouch, Calibration and MCP, each with its fix. Call first when something fails. |
  96. | `screenshot` | `include_text` (default true) | Current screen as an image plus OCR text. |
  97. | `tap` | `x`, `y` | Tap a point. |
  98. | `tap_text` | `text`, `occurrence` (default 1) | Tap visible text (case-insensitive; exact beats starts-with beats contains, in reading order). |
  99. | `long_press` | `x`, `y`, `duration_ms` (default 800) | Press and hold. |
  100. | `swipe` | `from_x`, `from_y`, `to_x`, `to_y`, `duration_ms` (default 400) | Drag. 500 ms or more does not coast; shorter flings. Up from the bottom edge goes Home, in from the left edge goes Back. |
  101. | `scroll` | `direction` (up/down/left/right), `amount` (0.1 to 1.0, default 0.5) | Scroll content, starting away from the edges. |
  102. | `type_text` | `text`, `submit` (default false) | Type ASCII with a US layout into the focused field; `submit` presses Return. |
  103. | `press_key` | `key` | `return`, `backspace`, `tab`, `escape`, `space`, arrows, `home`, volume keys, letters, digits, combos like `cmd+space`. |
  104. | `open_app` | `name` | Home, Spotlight, type the name, Return. |
  105. | `calibrate` | | Re-measure the pointer curve (about 30 s). |
  106. | `batch` | `actions` | Run steps in order, e.g. `[{"action":"tap_text","text":"Notes"},{"action":"wait","ms":500}]`. Stops at the first failure and names the step. |
  107. ## Troubleshooting
  108. | Symptom | Why | Fix |
  109. |---|---|---|
  110. | Screen step stays on "Waiting for the iPhone screen" | The iPhone is not visible as a capture device: charge-only cable, locked phone, or Trust not accepted | Use a data cable, unlock the iPhone, tap Trust, then unplug and replug |
  111. | "Camera access is off for PocketPilot" | macOS treats the iPhone screen as a camera | System Settings > Privacy & Security > Camera, turn on PocketPilot, relaunch it |
  112. | "No frames" after 5 s | The iPhone is locked, untrusted, or another app is holding the capture device | Unlock the iPhone, tap Trust, stop other recordings of it, replug |
  113. | The Mac does not appear in iPhone Settings > Bluetooth | Bluetooth permission is off, Bluetooth is off, or the Mac stopped being discoverable | Allow Bluetooth for PocketPilot in Privacy & Security, turn Bluetooth on, then Phone menu > Restart Bluetooth |
  114. | Taps fail with `hidNotConnected` | The iPhone is paired but not connected right now | iPhone Settings > Bluetooth, tap your Mac's name so it says Connected |
  115. | "Couldn't see the pointer" / `assistiveTouchOff` | AssistiveTouch is off, so iOS draws no pointer | Settings > Accessibility > Touch > AssistiveTouch, turn it on |
  116. | Taps land in the wrong place | Pointer speed changed (AssistiveTouch > Tracking Speed) or it is a different iPhone | Run `calibrate` again, or Phone menu > Recalibrate |
  117. | `notCalibrated` | No calibration for this iPhone and screen size | Run the Calibration step or the `calibrate` tool |
  118. | `tap_text` says the text is not on screen | OCR did not see it, or it is spelled differently | Take a `screenshot`, use one of the listed texts, or `tap(x, y)` |
  119. | USB step shows manual instructions only | Apple's MobileDevice framework could not be loaded on this Mac | Tap Trust and turn AssistiveTouch on by hand; everything else works |
  120. | "Port 47900 is already in use" | Another PocketPilot (or another program) is listening there | Quit the other copy; `lsof -nP -iTCP:47900 -sTCP:LISTEN` shows who has it |
  121. | Agent gets HTTP 404 "unknown session" | PocketPilot was restarted; sessions live in memory | Reconnect the server in your agent (in Claude Code: `/mcp`) |
  122. | `type_text` rejects a character | Only US-layout ASCII can be typed through a HID keyboard | Type only the ASCII parts |
  123. ## Privacy
  124. Everything runs on your Mac. The screen, OCR text, calibration and logs stay local. The MCP server listens only on
  125. `127.0.0.1` and rejects requests from non-local web origins. PocketPilot has no analytics and no accounts, and makes no
  126. network requests of its own.
  127. ## Development
  128. ```sh
  129. swift build # all targets
  130. swift test # unit + integration tests (no hardware)
  131. scripts/qa.sh # build, tests, MCP end-to-end against the simulated iPhone
  132. swift run pocketpilot-mock --port 47999 # headless MCP server backed by the simulated iPhone
  133. scripts/hardware-smoke.sh # guided checklist with a real iPhone
  134. ```
  135. `pocketpilot-mock` runs the real controller, pointer locator and MCP server against a simulated iPhone that applies pointer
  136. acceleration and reacts to the HID reports, so most behaviour can be tested without a phone. See
  137. [docs/TESTING.md](docs/TESTING.md) for the mock, its fault scenarios and the QA scripts.
  138. | Path | What |
  139. |---|---|
  140. | `Sources/PocketPilotCore/Shared` | Protocols and models every module codes against |
  141. | `Sources/PocketPilotCore/Capture` | USB screen capture, screenshot rendering |
  142. | `Sources/PocketPilotCore/Vision` | OCR |
  143. | `Sources/PocketPilotCore/Bluetooth` | Bluetooth HID keyboard + mouse + consumer control |
  144. | `Sources/PocketPilotCore/Device` | USB device detection, Trust pairing, AssistiveTouch toggle |
  145. | `Sources/PocketPilotCore/Control` | Calibration, pointer detection, actions |
  146. | `Sources/PocketPilotCore/MCP` | Streamable-HTTP MCP server |
  147. | `Sources/PocketPilotCore/Mock` | Simulated iPhone for tests and `pocketpilot-mock` |
  148. | `Sources/PocketPilot` | SwiftUI app |
  149. | `Sources/pocketpilot-mock` | Headless MCP server against the simulated iPhone |
  150. ## Contributing
  151. Issues and pull requests are welcome. Please:
  152. - keep it dependency-free and macOS 14 compatible;
  153. - put logic in `PocketPilotCore` behind the protocols in `Shared/` so it can be tested with the mocks;
  154. - make every error say what failed, why, and how to fix it;
  155. - run `scripts/qa.sh` before opening a PR.
  156. ## License
  157. MIT, see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.