Skip to content
Back to search

android-agent

Open-source framework to drive a real phone with AI agents: Android over ADB, iPhone over WebDriverAgent, 62 MCP tools, Python skills, a Vue dashboard, and swappable local or cloud LLMs

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/ghost-in-the-droid/android-agent with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/ghost-in-the-droid/android-agent'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ghost-in-the-droid--android-agent/badge.svg)](/listings/ghost-in-the-droid--android-agent)

HTML

<a href="/listings/ghost-in-the-droid--android-agent"><img src="/api/listings/ghost-in-the-droid--android-agent/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

Zero-install with [`uvx`](https://docs.astral.sh/uv/) (or `pipx`):

uvx ghost-in-the-droid doctor   # green/red preflight: python, adb on PATH, devices, ports, LLM keys
uvx ghost-in-the-droid login    # sign in with your Claude subscription, no API key needed
uvx ghost-in-the-droid up       # server + dashboard at http://localhost:5055  (API docs at /docs)

`doctor` prints a checklist with fix hints instead of a stack trace when something is missing (e.g. `adb` not on PATH). Prefer `pipx`? `pipx install ghost-in-the-droid` gives you the `ghost-in-the-droid` / `android-agent` commands.

Sign in with your Claude subscription (no API key)

If you have a **Claude Max/Pro** subscription, you don't need an API key. `android-agent login` signs you in through the `claude` CLI's own Anthropic OAuth flow and points Ghost at the `claude-code` provider:

android-agent login       # opens Anthropic sign-in via the claude CLI

Ghost never handles or stores your token; the `claude` CLI owns it, refresh included. `doctor` shows a green **Claude subscription** check once you are signed in. To use an API key instead, set `ANTHROPIC_API_KEY` (or `OPENAI_API_KEY` / `OPENROUTER_API_KEY`) and pick that provider.

<details> <summary>Run fully local with Ollama (no keys, no cloud)</summary>

Select Ollama in the Phone Agent tab. Runs entirely on your machine with [Ollama](https://ollama.com):

brew install ollama       # or: curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama pull llama3.2:3b   # 2GB, fast, good tool-use

</details>

<details> <summary>From a clone (for development)</summary>

git clone https://github.com/ghost-in-the-droid/android-agent.git
cd android-agent
pip install -e ".[all]"

android-agent doctor        # preflight
android-agent up            # start server + dashboard on :5055

# Frontend (separate terminal)
cd frontend && npm install && npx vite --host 0.0.0.0 --port 6175
# Dashboard at http://localhost:6175

</details>

iOS Quick Start

iOS support requires a Mac because Appium uses Xcode's XCUITest/WebDriverAgent stack. Real iPhones also require trust, Developer Mode, UI Automation permission when prompted, and WDA signing with an Apple development team.

# 1. Install and run Appium XCUITest
npm install -g appium
appium driver install xcuitest
appium --base-path /

# 2. Find your iPhone or booted simulator UDID
xcrun xctrace list devices
xcrun simctl list devices booted

# 3. Configure the backend for iOS
export IOS_DEVICE_UDID="<udid>"
export IOS_APPIUM_URL="http://127.0.0.1:4723"
export IOS_BUNDLE_ID="com.google.chrome.ios"       # or com.apple.mobilesafari
export IOS_MJPEG_SERVER_PORT="9100"                # use unique ports per iOS device

# 4. Run a product-path smoke workflow
uv run python scripts/ios_chrome_news_smoke.py \
  --device "ios:<udid>" \
  --bundle-id "$IOS_BUNDLE_ID" \
  --url https://text.npr.org/ \
  --max-headlines 5 \
  --max-articles 3 \
  --fix-health \
  --out-dir data/ios_chrome_news_smoke

For full real-device signing, simulator, WDA MJPEG, health recovery, scheduler, and MCP setup details, see [`docs/SETUP_IOS.md`](docs/SETUP_IOS.md).

Environment Variables

Copy `.env.example` to `.env` (if provided) or create a `.env` file in the project root. The server reads configuration via Pydantic Settings. Optional variables include:

| Variable | Purpose | |----------|---------| | `OPENAI_API_KEY` | LLM features (Skill Creator, Agent Chat) | | `ANTHROPIC_API_KEY` | Alternative LLM provider | | `OPENROUTER_API_KEY` | OpenRouter LLM provider | | `DEFAULT_DEVICE` | ADB serial (auto-detected if empty) | | `IOS_DEVICE_UDID` | iPhone or simulator UDID; devices are addressed as `ios:<udid>` | | `IOS_APPIUM_URL` | Appium server URL, default `http://127.0.0.1:4723` | | `IOS_BUNDLE_ID` | Default iOS app/browser bundle, e.g. `com.google.chrome.ios` or `com.apple.mobilesafari` |

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 488 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <!-- mcp-name: io.github.ghost-in-the-droid/android-agent -->
  2. <p align="center">
  3. <img src="docs/assets/mascot/12-the-tap.png" alt="Ghost tapping a phone" width="200" />
  4. </p>
  5. <h1 align="center">Ghost in the Droid</h1>
  6. <p align="center">
  7. <strong>Summon a ghost into your phone.</strong><br/>
  8. It sees the screen. It taps the buttons. It never sleeps.
  9. </p>
  10. <p align="center">
  11. <a href="https://ghostinthedroid.com">Website</a> &middot;
  12. <a href="https://ghostinthedroid.com/getting-started/installation/">Docs</a> &middot;
  13. <a href="https://ghostinthedroid.com/skills/">Skill Hub</a> &middot;
  14. <a href="https://github.com/ghost-in-the-droid/android-agent/releases/latest">Releases</a> &middot;
  15. <a href="https://discord.gg/gmvDX4Juk">Discord</a>
  16. </p>
  17. <p align="center">
  18. <img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT" />
  19. <img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+" />
  20. <img src="https://img.shields.io/badge/MCP-66%20tools-6366f1" alt="66 MCP tools" />
  21. <img src="https://img.shields.io/badge/platforms-Android%20%2B%20iOS-a78bfa" alt="Android + iOS" />
  22. <img src="https://img.shields.io/badge/inference-cloud%20or%20on--device-22c55e" alt="cloud or on-device" />
  23. <a href="https://mcptoplist.com/server/io.github.ghost-in-the-droid%2Fandroid-agent"><img src="https://mcptoplist.com/badge/io.github.ghost-in-the-droid%2Fandroid-agent.svg" alt="MCP Toplist: Top 1% of 95K" /></a>
  24. </p>
  25. ---
  26. <p align="center">
  27. <img src="docs/assets/hero-reel.webp" alt="Ghost driving nine real phones" width="820" />
  28. </p>
  29. <p align="center"><sub>▶ <a href="https://youtu.be/fdaXdFQo61o">Watch full HD with sound on YouTube</a> · <a href="docs/assets/hero-reel.mp4">Download mp4</a></sub></p>
  30. <p align="center"><em>Nine agents. Nine real devices. One ghost. (Click to watch.)</em></p>
  31. ---
  32. ## The pitch, in one breath
  33. Every AI agent can think. Almost none can touch a phone.
  34. **Ghost is the body.** It gives any LLM agent a real Android phone or iPhone: it reads the screen, taps, swipes, and types through a clean tool surface, and it scales from one device on your desk to a whole phone farm. Point Claude Code, Codex, Antigravity, Cursor, your LangChain app, or a model running *inside the phone itself* at it, and your agent grows hands.
  35. Bring your own brain. Keep the same body. MIT, forever.
  36. ---
  37. ## Mix and match: the Ghost matrix
  38. Ghost 1.3 is the only Android + iOS agent framework where **platform**, **brain**, and **driver** are all swappable. Pick one from each column. They all compose.
  39. | Pick a platform | Pick a brain | Pick a driver |
  40. |---|---|---|
  41. | **Android** over USB ADB | **Claude Code** (free with your Max/Pro sub) | **MCP client** (Claude Code, Codex, Antigravity, Cursor, Opencode) |
  42. | **Android** over Wi-Fi ADB | **Anthropic API** | **LangChain** toolkit |
  43. | **Android** in a **Docker + KVM** emulator pool | **OpenRouter** (any model, one key) | **LlamaIndex** tool spec |
  44. | **iPhone** over Appium + WebDriverAgent | **Ollama** (fully local) | **Ghost CLI** (`ghost "book a table" --device pixel`) |
  45. | **iPhone** over Tailscale + WebDriverAgent (wireless) | **vLLM** (your own GPU) | **Web dashboard** chat with a live phone stream |
  46. | | **On-device** — llama.cpp (`.gguf`, any quant), MediaPipe (`.task`, Gemma mobile bundles), MLX (Apple Silicon, opt-in). The model runs *in the phone*, airplane-mode works | **REST API** (`/docs` OpenAPI) |
  47. <sub>The Docker + KVM emulator pool is Android-only; iOS uses Apple's own simulator and Xcode signing, which are macOS-native.</sub>
  48. The body stays the same 66 MCP tools no matter what you plug in. Every new model release is a free upgrade to your phone agent.
  49. ---
  50. ## See it in action
  51. Nine demos, one per feature, each recorded on real devices. Click any clip for the full mp4.
  52. <table width="100%">
  53. <tr><th colspan="3" align="left">MCP clients: point your agent at a phone</th></tr>
  54. <tr>
  55. <td width="33%" valign="top"><a href="docs/assets/videos/mcp-claude-code-tui.mp4"><img src="docs/assets/videos/mcp-claude-code-tui.webp" width="100%" alt="Claude Code drives the phone"/></a><br/><sub><b>Claude Code</b>: type in the TUI, watch it drive the phone.</sub></td>
  56. <td width="33%" valign="top"><a href="docs/assets/videos/mcp-codex-tui.mp4"><img src="docs/assets/videos/mcp-codex-tui.webp" width="100%" alt="Codex drives the phone"/></a><br/><sub><b>Codex</b>: OpenAI's CLI operating a real phone over MCP.</sub></td>
  57. <td width="33%" valign="top"><a href="docs/assets/videos/mcp-agy-tui.mp4"><img src="docs/assets/videos/mcp-agy-tui.webp" width="100%" alt="Antigravity drives the phone"/></a><br/><sub><b>Antigravity</b>: Google's agent, same 62-tool body.</sub></td>
  58. </tr>
  59. <tr><th colspan="3" align="left">Interfaces: CLI, framework, browser</th></tr>
  60. <tr>
  61. <td width="33%" valign="top"><a href="docs/assets/videos/ghost-cli.mp4"><img src="docs/assets/videos/ghost-cli.webp" width="100%" alt="Ghost CLI one-liner"/></a><br/><sub><b>Ghost CLI</b>: <code>ghost "check reddit" --device asus</code>.</sub></td>
  62. <td width="33%" valign="top"><a href="docs/assets/videos/langchain.mp4"><img src="docs/assets/videos/langchain.webp" width="100%" alt="LangChain adapter"/></a><br/><sub><b>LangChain</b>: Ghost's tools in a framework agent.</sub></td>
  63. <td width="33%" valign="top"><a href="docs/assets/videos/web-chat-gui.mp4"><img src="docs/assets/videos/web-chat-gui.webp" width="100%" alt="Web chat drives phone"/></a><br/><sub><b>Web chat</b>: chat in your browser, it drives a real phone.</sub></td>
  64. </tr>
  65. <tr><th colspan="3" align="left">On-device and iOS: the phone as body and brain</th></tr>
  66. <tr>
  67. <td width="33%" valign="top"><a href="docs/assets/videos/on-device-android.mp4"><img src="docs/assets/videos/on-device-android.webp" width="100%" alt="On-device Android"/></a><br/><sub><b>On-device Android</b>: model runs in the app, airplane mode.</sub></td>
  68. <td width="33%" valign="top"><a href="docs/assets/videos/on-device-ios.mp4"><img src="docs/assets/videos/on-device-ios.webp" width="100%" alt="On-device iPhone"/></a><br/><sub><b>On-device iPhone</b>: your iPhone talks to itself.</sub></td>
  69. <td width="33%" valign="top"><a href="docs/assets/videos/iphone-duolingo.mp4"><img src="docs/assets/videos/iphone-duolingo.webp" width="100%" alt="iPhone real-app control"/></a><br/><sub><b>iPhone, real app</b>: Ghost does the Duolingo lesson.</sub></td>
  70. </tr>
  71. </table>
  72. ---
  73. ## What It Does
  74. Open-source Python framework for controlling Android and iOS devices from one agent harness. Android runs through ADB. iOS runs through Appium XCUITest and WebDriverAgent, with real iPhones as the target path and simulators for development and CI.
  75. Define **skills** for any app, run them from the dashboard or API, scale across a phone farm.
  76. **The ghost taps what you'd tap**
  77. - Android control through ADB: tap, swipe, type, clipboard, shell, intents, and stealth variants
  78. - iOS control through WebDriverAgent: screenshot, accessibility tree, tap, swipe, type, app launch, clipboard, browser actions
  79. - Live phone screen streaming: Android MJPEG/WebRTC, iOS WDA MJPEG with screenshot fallback
  80. - Interactive touch-to-tap on the streamed screen
  81. - Multi-device phone farm with per-device job queues
  82. **Forge reusable skills for any app**
  83. - YAML-based UI element definitions per app
  84. - Platform-specific selectors with `elements.yaml` for Android and `elements_ios.yaml` for iOS
  85. - Python action classes with precondition checks
  86. - Multi-step workflows that chain actions together
  87. - Built-in skills for TikTok and Play Store
  88. - iOS browser/news demo skill and smoke-level TikTok iOS workflows
  89. - **Skill Hub**: browse, search, and install skills from the community registry
  90. - Install from CLI: `android-agent skill install tiktok`
  91. **Teach the ghost new tricks**
  92. - BFS-based auto app explorer: discovers every screen and transition
  93. - LLM-assisted Skill Creator: chat with AI while viewing the live device stream
  94. - The AI identifies UI elements and generates action/workflow code
  95. **Scale the haunting**
  96. - Multi-device phone farm with per-device job queues
  97. - Bot runner: queue, schedule, and monitor automation jobs
  98. - Per-device integration tests with Android screen recording or iOS WDA MJPEG recording
  99. ### Platform Support
  100. | Surface | Android | iOS |
  101. |---------|---------|-----|
  102. | Device ref | ADB serial, e.g. `emulator-5554` | `ios:<udid>` |
  103. | Backend | ADB + optional Portal companion app | Appium XCUITest + WebDriverAgent |
  104. | Real device support | Yes | Yes, with Mac/Xcode/WDA signing |
  105. | Simulator/emulator support | Android emulator tooling | Booted iOS simulators via Appium/WDA |
  106. | Live stream | Portal WebRTC, MJPEG, screencap | WDA MJPEG, screenshot polling fallback |
  107. | Screen tree | Android UIAutomator XML | Normalized XCTest accessibility tree |
  108. | Skills | `elements.yaml`, Android packages | `elements_ios.yaml`, iOS bundle IDs |
  109. | Android-only today | ADB shell, intents, wireless ADB, Play Store helpers, Portal overlay | Unsupported with stable platform errors |
  110. ---
  111. ## Why it is built this way
  112. **Skills cost $0. Thinking costs tokens.** An agent should not pay an LLM to tap a button it has tapped a thousand times. Known workflows compile to **skills**: deterministic YAML + Python recipes that replay in seconds with zero LLM calls. Use AI for the unknown task, replay for the known one.
  113. **The phone is the easiest sandbox there is.** Give the ghost an old Android with its own SIM and its own accounts, physically separate from your life. You already know how to set up a phone.
  114. **Zero-app by default.** A fresh install touches nothing on the Android device: screen reads go through `uiautomator`, actions through `adb shell input`, no root, no accessibility service. Want it faster? The optional Portal companion app gives roughly 30x quicker UI reads. Want it private? On-device mode runs the whole loop inside the phone, nothing leaves it. All opt-in.
  115. **Local-first, cloud-optional.** The server and dashboard run on your machine. With a local or on-device brain, your screenshots never leave your network. Pick a cloud brain and prompts go to that provider, same as any tool. Your call, always.
  116. ---
  117. ## Requirements
  118. - **Python 3.10+**
  119. - **Android**: Android phone with USB debugging enabled and **ADB** on PATH (`adb devices` should list your phone)
  120. - **iOS**: macOS with Xcode, Appium 2 + XCUITest driver, and a trusted iPhone or booted simulator
  121. - **Node.js 18+** (for the frontend dev server)
  122. ---
  123. ## iOS Support
  124. Ghost drives iPhones through Appium/WebDriverAgent with the same tool
  125. surface: `ios:<udid>` device refs route tap/swipe/type/screenshot to WDA, and
  126. iOS-aware browser primitives (`open_url`, `read_news`, `extract_visible_text`)
  127. cover web tasks. Android-only tools (`shell`, `launch_intent`, Portal overlay)
  128. return a clear platform error instead of failing silently.
  129. iOS is **opt-in**: enable it with `GITD_ENABLE_IOS=1` (or `ios_platform_enabled=true`
  130. in `.env`). Wireless drive rides your Tailscale tailnet, so treat the tailnet as
  131. the trust boundary. See [docs/SETUP_IOS.md](docs/SETUP_IOS.md) for Appium/WDA
  132. setup, and [on-device on iPhone](https://ghostinthedroid.com/features/on-device-llm/)
  133. to run the model on the iPhone itself.
  134. ---
  135. ## Quick Start
  136. Zero-install with [`uvx`](https://docs.astral.sh/uv/) (or `pipx`):
  137. ```bash
  138. uvx ghost-in-the-droid doctor # green/red preflight: python, adb on PATH, devices, ports, LLM keys
  139. uvx ghost-in-the-droid login # sign in with your Claude subscription, no API key needed
  140. uvx ghost-in-the-droid up # server + dashboard at http://localhost:5055 (API docs at /docs)
  141. ```
  142. `doctor` prints a checklist with fix hints instead of a stack trace when something is missing (e.g. `adb` not on PATH). Prefer `pipx`? `pipx install ghost-in-the-droid` gives you the `ghost-in-the-droid` / `android-agent` commands.
  143. ### Sign in with your Claude subscription (no API key)
  144. If you have a **Claude Max/Pro** subscription, you don't need an API key. `android-agent login` signs you in through the `claude` CLI's own Anthropic OAuth flow and points Ghost at the `claude-code` provider:
  145. ```bash
  146. android-agent login # opens Anthropic sign-in via the claude CLI
  147. ```
  148. Ghost never handles or stores your token; the `claude` CLI owns it, refresh included. `doctor` shows a green **Claude subscription** check once you are signed in. To use an API key instead, set `ANTHROPIC_API_KEY` (or `OPENAI_API_KEY` / `OPENROUTER_API_KEY`) and pick that provider.
  149. <details>
  150. <summary>Run fully local with Ollama (no keys, no cloud)</summary>
  151. Select Ollama in the Phone Agent tab. Runs entirely on your machine with [Ollama](https://ollama.com):
  152. ```bash
  153. brew install ollama # or: curl -fsSL https://ollama.com/install.sh | sh
  154. ollama serve &
  155. ollama pull llama3.2:3b # 2GB, fast, good tool-use
  156. ```
  157. </details>
  158. <details>
  159. <summary>From a clone (for development)</summary>
  160. ```bash
  161. git clone https://github.com/ghost-in-the-droid/android-agent.git
  162. cd android-agent
  163. pip install -e ".[all]"
  164. android-agent doctor # preflight
  165. android-agent up # start server + dashboard on :5055
  166. # Frontend (separate terminal)
  167. cd frontend && npm install && npx vite --host 0.0.0.0 --port 6175
  168. # Dashboard at http://localhost:6175
  169. ```
  170. </details>
  171. ### iOS Quick Start
  172. iOS support requires a Mac because Appium uses Xcode's XCUITest/WebDriverAgent stack. Real iPhones also require trust, Developer Mode, UI Automation permission when prompted, and WDA signing with an Apple development team.
  173. ```bash
  174. # 1. Install and run Appium XCUITest
  175. npm install -g appium
  176. appium driver install xcuitest
  177. appium --base-path /
  178. # 2. Find your iPhone or booted simulator UDID
  179. xcrun xctrace list devices
  180. xcrun simctl list devices booted
  181. # 3. Configure the backend for iOS
  182. export IOS_DEVICE_UDID="<udid>"
  183. export IOS_APPIUM_URL="http://127.0.0.1:4723"
  184. export IOS_BUNDLE_ID="com.google.chrome.ios" # or com.apple.mobilesafari
  185. export IOS_MJPEG_SERVER_PORT="9100" # use unique ports per iOS device
  186. # 4. Run a product-path smoke workflow
  187. uv run python scripts/ios_chrome_news_smoke.py \
  188. --device "ios:<udid>" \
  189. --bundle-id "$IOS_BUNDLE_ID" \
  190. --url https://text.npr.org/ \
  191. --max-headlines 5 \
  192. --max-articles 3 \
  193. --fix-health \
  194. --out-dir data/ios_chrome_news_smoke
  195. ```
  196. For full real-device signing, simulator, WDA MJPEG, health recovery, scheduler, and MCP setup details, see [`docs/SETUP_IOS.md`](docs/SETUP_IOS.md).
  197. ### Environment Variables
  198. Copy `.env.example` to `.env` (if provided) or create a `.env` file in the project root. The server reads configuration via Pydantic Settings. Optional variables include:
  199. | Variable | Purpose |
  200. |----------|---------|
  201. | `OPENAI_API_KEY` | LLM features (Skill Creator, Agent Chat) |
  202. | `ANTHROPIC_API_KEY` | Alternative LLM provider |
  203. | `OPENROUTER_API_KEY` | OpenRouter LLM provider |
  204. | `DEFAULT_DEVICE` | ADB serial (auto-detected if empty) |
  205. | `IOS_DEVICE_UDID` | iPhone or simulator UDID; devices are addressed as `ios:<udid>` |
  206. | `IOS_APPIUM_URL` | Appium server URL, default `http://127.0.0.1:4723` |
  207. | `IOS_BUNDLE_ID` | Default iOS app/browser bundle, e.g. `com.google.chrome.ios` or `com.apple.mobilesafari` |
  208. | `IOS_DEVICES_JSON` | Per-device iOS config for multiple phones/simulators, WDA ports, bundle IDs, and MJPEG ports |
  209. | `IOS_MJPEG_SERVER_PORT` | WDA MJPEG stream port; use one unique port per iOS device |
  210. ---
  211. ## Give any AI agent a mobile body (MCP)
  212. Ghost ships an [MCP](https://modelcontextprotocol.io) server with **66 tools** for real-device control. Any MCP-compatible client can use them. Android serials receive the Android implementation; `ios:<udid>` refs route to the iOS backend where supported and return stable unsupported-platform errors for Android-only tools. One command wires it up:
  213. ```bash
  214. # Claude Code (same shape for Codex, Cursor, VS Code Copilot, Windsurf)
  215. claude mcp add android-agent -- uvx --from ghost-in-the-droid android-agent-mcp
  216. ```
  217. `uvx` installs the package, builds an isolated env, and runs the server. No clone, no venv.
  218. <details>
  219. <summary>Other clients (Codex, Claude Desktop, Cursor, VS Code, Windsurf)</summary>
  220. ```bash
  221. # Codex (OpenAI)
  222. codex mcp add android-agent -- uvx --from ghost-in-the-droid android-agent-mcp
  223. ```
  224. **Claude Desktop** (`claude_desktop_config.json`), **Cursor** (`.cursor/mcp.json`), **Windsurf** (`mcp_config.json`):
  225. ```json
  226. {
  227. "mcpServers": {
  228. "android-agent": {
  229. "command": "uvx",
  230. "args": ["--from", "ghost-in-the-droid", "android-agent-mcp"]
  231. }
  232. }
  233. }
  234. ```
  235. **VS Code Copilot** (`.vscode/mcp.json`) uses the same block under a `"servers"` key. Contributors who clone the repo get a ready `.mcp.json`, all 66 tools live on first `claude` launch.
  236. </details>
  237. **The 66 tools, by category:**
  238. | Category | What the ghost can do |
  239. |----------|-----------------------|
  240. | **See** | `screenshot`, `get_elements`, `get_screen_tree`, `get_screen_xml`, `screenshot_annotated`, `screenshot_cropped` |
  241. | **Touch** | `tap`, `tap_element`, `swipe`, `long_press`, `type_text`, `type_unicode`, `press_back`, `press_home`, `press_key` |
  242. | **Apps** | `launch_app`, `search_apps`, `list_apps`, `launch_intent`, `force_stop`, `list_packages` |
  243. | **Understand** | `get_phone_state`, `classify_screen`, `find_on_screen`, `ocr_screen`, `ocr_region`, `extract_visible_text` |
  244. | **Browser / iOS** | `open_url`, `browser_back`, `get_current_url`, `read_news`, `extract_articles`, `wait_for_text` |
  245. | **Device** | `list_devices`, `clipboard_get`, `clipboard_set`, `get_notifications`, `open_notifications`, `toggle_overlay`, `device_health` |
  246. | **Skills** | `list_skills`, `run_workflow`, `run_action`, `create_skill`, `explore_app` |
  247. | **Batch** | `run_flow` / `chain`: one call runs a whole multi-step recipe server-side, N round-trips collapse to 1 |
  248. | **Diagnostics** | `list_crashes`, `get_crash`, `web_search`, screen recording, camera + TTS |
  249. `toggle_overlay`, `launch_intent`, Android shell helpers, wireless ADB, and Play Store helpers remain Android-only and return stable platform errors on iOS.
  250. ---
  251. ## Run the brain inside the phone
  252. New in 1.3: the model can live *on the device*. The Ghost companion app embeds real inference engines, so the phone is both the body and the brain. Airplane mode on, agent still working, nothing leaving the device.
  253. | Platform | Engine | Model format | Best for |
  254. |----------|--------|-------------|----------|
  255. | Android | **llama.cpp** (JNI) | `.gguf` | Any GGUF: Gemma, Llama, Mistral, Qwen, DeepSeek |
  256. | Android | **MediaPipe** | `.task` | Small Gemma models, fastest Android integration |
  257. | iPhone | **llama.cpp** (Metal) | `.gguf` | Same GGUF as Android, Qwen2.5 1.5B on Metal |
  258. | iPhone | **MLX** | Apple Silicon | Faster decode on newer chips, opt-in |
  259. On iPhone, Qwen2.5 1.5B runs through llama.cpp on Metal and drives the phone's own UI, with an opt-in MLX engine for faster decode on Apple Silicon. Small models are kept honest with grammar-constrained decoding so tool calls always parse. Full detail on the [on-device LLM page](https://ghostinthedroid.com/features/on-device-llm/).
  260. ---
  261. ## Skill Hub CLI
  262. Path: [install](https://ghostinthedroid.com/getting-started/installation/), [connect a phone](https://ghostinthedroid.com/getting-started/connect-phone/), then your first hub skill:
  263. ```bash
  264. ghost skill search calculator # search the hub registry
  265. ghost skill info ghost/calculator # trust, quality tier, actions, tested-on device
  266. ghost skill install ghost/calculator # hash-checked install into ~/.ghost/skills
  267. ghost skill run ghost/calculator calculate --param expression='1234*5' --device <serial>
  268. ghost skill list # installed skills
  269. ghost skill outdated # installed vs latest
  270. ghost skill validate ./my-skill/ # before publishing
  271. ghost skill submit ./skillhub/<you>/<slug> --push # fork, push, open the PR
  272. ```
  273. How a skill earns its tier and points is on [How skills are scored](https://ghostinthedroid.com/skills/scoring/); the submission flow is on [Publishing Skills](https://ghostinthedroid.com/skills/publishing/). The hub registry is `skillhub/registry.json`, regenerated by CI after merge.
  274. ## Teach the Ghost a New App
  275. Two ways to forge a skill:
  276. **Community skill** (your own repo): create a repo with `skill.yaml`, `elements.yaml`, actions, and workflows, tag it `android-agent-skill`, and it appears on the [Skill Hub](https://ghostinthedroid.com/skills/) automatically (nightly scraper).
  277. **Official skill** (PR to this repo): build and test as a community skill first, then open a PR adding it to [`registry/`](registry/). CI validates, a maintainer reviews, and it gets the "Official" badge.
  278. Each skill needs:
  279. - `skill.yaml`: metadata (name, version, app package or iOS bundle ID, supported platforms, actions, workflows)
  280. - `elements.yaml`: Android UI element resource IDs and descriptions
  281. - `elements_ios.yaml`: optional iOS selectors for XCTest accessibility trees
  282. - `actions/`: Python classes extending `Action` with `precondition()` and `execute()`
  283. - `workflows/`: Python classes extending `Workflow` with `steps()`
  284. Prefer to let AI do it? The **Skill Creator** watches a live device stream and generates the action and workflow code while you narrate. Full guide in [CONTRIBUTING.md](CONTRIBUTING.md).
  285. ---
  286. ## The dashboard
  287. A local-first control room at `http://localhost:5055` (or `:6175` in dev). WebRTC and MJPEG live streams, tap-on-screen, tracing, and every tab below.
  288. | Tab | What it does |
  289. |-----|-------------|
  290. | Phone Agent | Live device stream (MJPEG/WebRTC), tap/swipe on screen, multi-device view |
  291. | Scheduler | Cron-style job scheduling with queue management |
  292. | Skill Hub | Browse installed skills, run actions and workflows, export/delete |
  293. | Skill Creator | LLM-assisted skill builder with live device stream |
  294. | Skill Miner | Auto app explorer: BFS state discovery with screenshots |
  295. | Manual Run | Start/stop bot jobs, queue management, logs |
  296. | Tests | Per-device test runner with screen recording playback |
  297. | Emulators | Create, boot, snapshot, and manage Android emulators |
  298. | Tracing | Per-turn traces, token accounting, tool-call visibility |
  299. ---
  300. ## Architecture
  301. ```
  302. android-agent/
  303. run.py # Uvicorn entry point on :5055
  304. gitd/
  305. app.py # FastAPI app factory + plugin hook
  306. models/ schemas/ # SQLAlchemy 2.0 ORM + Pydantic v2
  307. routers/ services/ # Route handlers + business logic
  308. skills/ # Skill packages (tiktok, play_store, safari [iOS], tiktok_ios)
  309. bots/common/adb.py # Android Device class: tap, swipe, dump tree, wait_for
  310. bots/common/ios.py # iOS Device class: Appium/WDA session, UI tree, gestures
  311. mcp_server.py # MCP server: 66 tools for any LLM agent
  312. frontend/ # Vue 3 + Vite + TypeScript + Tailwind
  313. portal/ # Kotlin companion app (WebRTC, on-device inference)
  314. site/ # Docs site (Astro + Starlight)
  315. ```
  316. **The flow:** FastAPI backend on `:5055` exposes device control, skills, bots, scheduling, and streaming. The Vue SPA talks to it over `/api/*`. Skills define per-app interaction (elements + actions + workflows). Device backends route by ref: bare serials use `bots/common/adb.py`, `ios:<udid>` refs use `bots/common/ios.py` and Appium/WDA. State lives in SQLite via SQLAlchemy 2.0 and Alembic.
  317. ## Benchmark (sneak peek)
  318. Early result, with the full methodology writeup still cooking: driven by Claude Code, Ghost completes **115 of 116 tasks (99.1%)** on [AndroidWorld](https://github.com/google-research/android_world), Google Research's benchmark for Android agents, on the unmodified upstream harness. Treat this as a preview rather than a citable number. The detailed writeup and trajectories are coming. Watch the [releases](https://github.com/ghost-in-the-droid/android-agent/releases/latest).
  319. ---
  320. ## Tech Stack
  321. | Layer | Technology |
  322. |-------|-----------|
  323. | Backend | FastAPI, Uvicorn, Python 3.10+ |
  324. | Data | SQLAlchemy 2.0, Alembic, Pydantic v2, SQLite (WAL) |
  325. | Frontend | Vue 3, TypeScript, Vite, Tailwind CSS 4 |
  326. | Device control | ADB for Android; Appium XCUITest/WebDriverAgent for iOS |
  327. | Streaming | Android MJPEG/WebRTC via Portal; iOS WDA MJPEG plus screenshot fallback |
  328. | On-device | llama.cpp, MediaPipe (Android); llama.cpp on Metal, MLX (iOS) |
  329. | Quality | Ruff, pytest, Playwright |
  330. ---
  331. ## Running Tests
  332. Most tests are unit/API tests and run without a live device. Live Android and iOS integration tests require the relevant device stack.
  333. ```bash
  334. # Run all tests on a specific device
  335. DEVICE=<serial> python3 -m pytest tests/ -v --tb=short
  336. ```
  337. For iOS live smoke tests:
  338. ```bash
  339. IOS_LIVE_NEWS_TEST=1 \
  340. IOS_DEVICE_UDID="<udid>" \
  341. IOS_APPIUM_URL="http://127.0.0.1:4723" \
  342. IOS_BUNDLE_ID="com.google.chrome.ios" \
  343. uv run --extra test python -m pytest tests/test_browser_news.py::test_live_ios_chrome_news_workflow
  344. ```
  345. Get your Android device serial from `adb devices`.
  346. ---
  347. ## Database Migrations
  348. The project uses Alembic for schema migrations:
  349. ```bash
  350. alembic revision --autogenerate -m "add new_field to my_table" # after editing a model
  351. alembic upgrade head # apply pending
  352. alembic downgrade -1 # rollback one
  353. ```
  354. ---
  355. ## Contributing
  356. The ghost gets stronger with every skill. See [CONTRIBUTING.md](CONTRIBUTING.md) for adding app skills (highest impact), writing actions and workflows, backend architecture, and the PR process. Join the community on the [Skill Hub](https://ghostinthedroid.com/skills/) and on [Discord](https://discord.gg/gmvDX4Juk), where you can share the skills and things you build with Ghost and get help.
  357. ---
  358. ## License
  359. [MIT](LICENSE). The ghost is free. The ghost is open source. The ghost is yours.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.