Skip to content
Back to search

SwiftAutoGUI

A Swift library for macOS automation — mouse, keyboard, screenshots, image recognition, and AI-powered agents.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nakaokarei--swiftautogui/badge.svg)](/listings/nakaokarei--swiftautogui)

HTML

<a href="/listings/nakaokarei--swiftautogui"><img src="/api/listings/nakaokarei--swiftautogui/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Swift Package Manager

SwiftAutoGUI is available through [Swift Package Manager](https://www.swift.org/package-manager/).

in `Package.swift` add the following:

dependencies: [
    // Dependencies declare other packages that this package depends on.
    .package(url: "https://github.com/NakaokaRei/SwiftAutoGUI", branch: "master")
],
targets: [
    .target(
        name: "MyProject",
        dependencies: [..., "SwiftAutoGUI"]
    )
    ...
]

Homebrew (sagui CLI)

The `sagui` command-line tool is available via the [Homebrew tap](https://github.com/NakaokaRei/homebrew-tap):

brew install NakaokaRei/tap/sagui

After install, grant **Accessibility** permission to your terminal in System Settings → Privacy & Security → Accessibility.

sagui --version
sagui key list
sagui key shortcut return
sagui mouse click --x 100 --y 200
sagui mouse click --right --x 100 --y 200

Optional Chromium CDP backend

> Quick start: [Browser Agent guide](Documentation/BrowserAgent.md)

`SwiftAutoGUIBrowser` adds semantic automation for an existing Chrome, Edge, or other Chromium debugging session. It is a separate product, so applications that use only native macOS automation do not acquire browser-specific code.

Add the optional product to the client target:

.target(
    name: "MyProject",
    dependencies: [
        .product(name: "SwiftAutoGUI", package: "SwiftAutoGUI"),
        .product(name: "SwiftAutoGUIBrowser", package: "SwiftAutoGUI")
    ]
)

Start Chromium with a dedicated profile and a loopback debugging port. Chrome 136 and later do not honor remote debugging against the default profile.

"/Applications/Google Chrome for Testing.app/Contents/MacOS/Google Chrome for Testing" \
  --remote-debugging-port=9222 \
  --user-data-dir=/tmp/swift-auto-gui-browser-profile

Connect with an explicit navigation allowlist:

import SwiftAutoGUI
import SwiftAutoGUIBrowser

let browser = try await BrowserSession.connect(
    endpoint: URL(string: "http://127.0.0.1:9222")!,
    securityPolicy: BrowserSecurityPolicy(
        allowedDomains: ["example.com", "*.example.org"]
    )
)

let observation = try await browser.observe()
print(observation.formattedContext)

let apiKey = ProcessInfo.processInfo.environment["OPENAI_API_KEY"]!
let browserAgent = Agent(
    model: AutomationModels.openAI(apiKey: apiKey),
    automationBackend: browser
)

The browser Agent is intentionally browser-only: native app, window, AX-label, and coordinate mouse actions fail as unsupported instead of falling back to Accessibility or CGEvent. Stale DOM elements also fail safely without clicking a saved coordinate. Cross-origin navigation and downloads require a `BrowserActionAuthorizing` implementation; without one they are denied.

The `sagui` CLI supports deterministic CDP commands as well as the browser-only Agent:

sagui browser tabs
sagui browser observe --tab-id TARGET_ID
sagui browser click --tab-id TARGET_ID --role link --name "Issues" --domain github.com
sagui browser agent "Open issue 118" --domain github.com --allow-cross-origin

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 19 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "swift-auto-gui",
  3. "owner": {
  4. "name": "NakaokaRei"
  5. },
  6. "description": "SwiftAutoGUI macOS automation plugin for Claude Code.",
  7. "plugins": [
  8. {
  9. "name": "swift-auto-gui",
  10. "source": "./plugins/swift-auto-gui",
  11. "description": "Control macOS GUI applications via mouse, keyboard, screenshots, image recognition, and AppleScript using the sagui CLI.",
  12. "homepage": "https://github.com/NakaokaRei/SwiftAutoGUI",
  13. "repository": "https://github.com/NakaokaRei/SwiftAutoGUI",
  14. "license": "MIT",
  15. "keywords": ["macos", "automation", "gui", "mouse", "keyboard", "screenshot"],
  16. "category": "automation"
  17. }
  18. ]
  19. }
README.md · README · 250 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # SwiftAutoGUI
  2. <!-- # Badges -->
  3. ![SPM is supported](https://img.shields.io/badge/SPM-Supported-orange)
  4. [![Github issues](https://img.shields.io/github/issues/NakaokaRei/SwiftAutoGUI)](https://github.com/NakaokaRei/SwiftAutoGUI/issues)
  5. [![Github forks](https://img.shields.io/github/forks/NakaokaRei/SwiftAutoGUI)](https://github.com/NakaokaRei/SwiftAutoGUI/network/members)
  6. [![Github stars](https://img.shields.io/github/stars/NakaokaRei/SwiftAutoGUI)](https://github.com/NakaokaRei/SwiftAutoGUI/stargazers)
  7. [![Github top language](https://img.shields.io/github/languages/top/NakaokaRei/SwiftAutoGUI)](https://github.com/NakaokaRei/SwiftAutoGUI/)
  8. [![Github license](https://img.shields.io/github/license/NakaokaRei/SwiftAutoGUI)](https://github.com/NakaokaRei/SwiftAutoGUI/)
  9. <!-- # Short Description -->
  10. A Swift library for macOS automation — mouse, keyboard, screenshots, image recognition, and AI-powered agents.
  11. This repository is inspired by [pyautogui](https://github.com/asweigart/pyautogui).
  12. ## Demo
  13. AI Agent that autonomously observes the screen and executes actions to achieve a goal.
  14. `sagui agent "Open Safari and search for Swift"`
  15. <img src="https://github.com/user-attachments/assets/0b501faa-ff8a-4fee-8bf7-9db49bbd6766" alt="Demo: sagui agent" width="900">
  16. # Requirements
  17. - macOS 27.0+
  18. - Swift 6.2+
  19. # Installation
  20. ## Swift Package Manager
  21. SwiftAutoGUI is available through [Swift Package Manager](https://www.swift.org/package-manager/).
  22. in `Package.swift` add the following:
  23. ```swift
  24. dependencies: [
  25. // Dependencies declare other packages that this package depends on.
  26. .package(url: "https://github.com/NakaokaRei/SwiftAutoGUI", branch: "master")
  27. ],
  28. targets: [
  29. .target(
  30. name: "MyProject",
  31. dependencies: [..., "SwiftAutoGUI"]
  32. )
  33. ...
  34. ]
  35. ```
  36. ## Homebrew (sagui CLI)
  37. The `sagui` command-line tool is available via the [Homebrew tap](https://github.com/NakaokaRei/homebrew-tap):
  38. ```bash
  39. brew install NakaokaRei/tap/sagui
  40. ```
  41. After install, grant **Accessibility** permission to your terminal in
  42. System Settings → Privacy & Security → Accessibility.
  43. ```bash
  44. sagui --version
  45. sagui key list
  46. sagui key shortcut return
  47. sagui mouse click --x 100 --y 200
  48. sagui mouse click --right --x 100 --y 200
  49. ```
  50. ## Optional Chromium CDP backend
  51. > Quick start: [Browser Agent guide](Documentation/BrowserAgent.md)
  52. `SwiftAutoGUIBrowser` adds semantic automation for an existing Chrome, Edge,
  53. or other Chromium debugging session. It is a separate product, so applications
  54. that use only native macOS automation do not acquire browser-specific code.
  55. Add the optional product to the client target:
  56. ```swift
  57. .target(
  58. name: "MyProject",
  59. dependencies: [
  60. .product(name: "SwiftAutoGUI", package: "SwiftAutoGUI"),
  61. .product(name: "SwiftAutoGUIBrowser", package: "SwiftAutoGUI")
  62. ]
  63. )
  64. ```
  65. Start Chromium with a dedicated profile and a loopback debugging port. Chrome
  66. 136 and later do not honor remote debugging against the default profile.
  67. ```bash
  68. "/Applications/Google Chrome for Testing.app/Contents/MacOS/Google Chrome for Testing" \
  69. --remote-debugging-port=9222 \
  70. --user-data-dir=/tmp/swift-auto-gui-browser-profile
  71. ```
  72. Connect with an explicit navigation allowlist:
  73. ```swift
  74. import SwiftAutoGUI
  75. import SwiftAutoGUIBrowser
  76. let browser = try await BrowserSession.connect(
  77. endpoint: URL(string: "http://127.0.0.1:9222")!,
  78. securityPolicy: BrowserSecurityPolicy(
  79. allowedDomains: ["example.com", "*.example.org"]
  80. )
  81. )
  82. let observation = try await browser.observe()
  83. print(observation.formattedContext)
  84. let apiKey = ProcessInfo.processInfo.environment["OPENAI_API_KEY"]!
  85. let browserAgent = Agent(
  86. model: AutomationModels.openAI(apiKey: apiKey),
  87. automationBackend: browser
  88. )
  89. ```
  90. The browser Agent is intentionally browser-only: native app, window, AX-label,
  91. and coordinate mouse actions fail as unsupported instead of falling back to
  92. Accessibility or CGEvent. Stale DOM elements also fail safely without clicking
  93. a saved coordinate. Cross-origin navigation and downloads require a
  94. `BrowserActionAuthorizing` implementation; without one they are denied.
  95. The `sagui` CLI supports deterministic CDP commands as well as the browser-only Agent:
  96. ```bash
  97. sagui browser tabs
  98. sagui browser observe --tab-id TARGET_ID
  99. sagui browser click --tab-id TARGET_ID --role link --name "Issues" --domain github.com
  100. sagui browser agent "Open issue 118" --domain github.com --allow-cross-origin
  101. ```
  102. # Example Usage
  103. For complete API and module documentation, see the [SwiftAutoGUI DocC site](https://nakaokarei.github.io/SwiftAutoGUI/documentation/).
  104. ## AI Agent
  105. SwiftAutoGUI includes an Agent that can autonomously observe the screen, reason about what it sees, and execute actions in a loop until a goal is achieved. This follows the **ReAct** (Observe → Think → Act) pattern using a vision-capable LLM.
  106. ```swift
  107. import SwiftAutoGUI
  108. let backend = AutomationModels.openAI(apiKey: "sk-...", model: "gpt-5.6-sol")
  109. let agent = Agent(
  110. model: backend,
  111. maxIterations: 15,
  112. visionMode: .automatic
  113. )
  114. let result = try await agent.run(goal: "Open Safari and search for Swift")
  115. print("Completed: \(result.completed), Steps: \(result.iterationsUsed)")
  116. ```
  117. The CLI defaults to the on-device Apple model. Choose a cloud provider explicitly:
  118. ```bash
  119. sagui agent "Press the Save button" --vision-mode automatic
  120. sagui agent "Open Safari and search for Swift" --provider openai
  121. sagui browser agent "Open Issues" --domain github.com --provider openai
  122. ```
  123. OpenAI reads `OPENAI_API_KEY` (or `--api-key`). `--provider pcc` requires Apple's
  124. managed entitlement in an eligible consuming app/executable; ordinary standalone
  125. CLI distribution is not assumed to qualify. `--fallback-on-device` opts into a
  126. local fallback. No cloud fallback is automatic.
  127. The old `--reasoning-effort` option has been removed: the pinned Apple Chat
  128. Completions adapter does not forward it. Per-step `Reasoning:` output is a short
  129. user-facing explanation, not hidden chain-of-thought.
  130. See the [macOS 27 migration guide](Sources/SwiftAutoGUI/Documentation.docc/ModelMigration.md)
  131. for API changes, context limits, provider compatibility, and PCC configuration.
  132. When screen context is enabled, actionable Accessibility elements receive step-local
  133. identifiers such as `[#12]`. The agent can target these identifiers directly, resolves
  134. them again immediately before execution, and rejects stale elements safely. Each action
  135. returns a structured result describing the execution method, failure, UI change, and
  136. focus change. If an action changes the UI, the remaining batch is stopped and the agent
  137. observes the new state before continuing.
  138. `--vision-mode` accepts `always`, `automatic`, or `never`. `automatic` omits the
  139. screenshot when actionable Accessibility elements are available; `always` preserves
  140. the original behavior and remains the default.
  141. ## Basic Usage
  142. ```swift
  143. import SwiftAutoGUI
  144. // Execute single actions
  145. await Action.leftClick.execute()
  146. await Action.write("Hello, World!").execute()
  147. await Action.keyShortcut([.command, .a]).execute() // Select all
  148. // Build and execute action sequences
  149. let actions: [Action] = [
  150. .move(to: CGPoint(x: 100, y: 100)),
  151. .wait(0.5),
  152. .leftClick,
  153. .write("Hello, SwiftAutoGUI!"),
  154. .keyShortcut([.returnKey])
  155. ]
  156. await actions.execute()
  157. ```
  158. # Claude Code Plugin
  159. SwiftAutoGUI ships as a [Claude Code](https://docs.anthropic.com/en/docs/claude-code) plugin so Claude can control native macOS applications and Chromium pages through the `sagui` CLI.
  160. ## Install from the marketplace
  161. Inside Claude Code:
  162. ```text
  163. /plugin marketplace add NakaokaRei/SwiftAutoGUI
  164. /plugin install swift-auto-gui@swift-auto-gui
  165. ```
  166. This installs two skills:
  167. - `macos-control`, invoked as `/swift-auto-gui:macos-control`, controls native macOS UI.
  168. - `browser-control`, invoked as `/swift-auto-gui:browser-control`, controls Chromium pages through CDP without native input fallback.
  169. The skills walk Claude through installing or updating the `sagui` binary when needed.
  170. ## Permissions
  171. Grant the application running Claude Code (Terminal.app, iTerm, etc.) both:
  172. - **Accessibility** — System Settings → Privacy & Security → Accessibility
  173. - **Screen Recording** — System Settings → Privacy & Security → Screen Recording
  174. These permissions are required for `macos-control`; browser-only CDP actions do not require them.
  175. For full details, see the [`macos-control`](plugins/swift-auto-gui/skills/macos-control/SKILL.md) and [`browser-control`](plugins/swift-auto-gui/skills/browser-control/SKILL.md) skill definitions.
  176. # Contributors
  177. - [NakaokaRei](https://github.com/NakaokaRei)
  178. <!-- CREATED_BY_LEADYOU_README_GENERATOR -->
  179. # License
  180. MIT license. See the [LICENSE file](/LICENSE) for details.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.