Skip to content
Back to search

scrawl

Show an agent a problem: mark up, record, send

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/kitlangton--scrawl/badge.svg)](/listings/kitlangton--scrawl)

HTML

<a href="/listings/kitlangton--scrawl"><img src="/api/listings/kitlangton--scrawl/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Scrawl needs an Apple silicon Mac on macOS 15 or later (live transcription and Liquid Glass need macOS 26). Download the DMG from [scrawl.kitlangton.com](https://scrawl.kitlangton.com), or use Homebrew:

brew install --cask kitlangton/tap/scrawl

The app updates itself (Sparkle), so Homebrew leaves updates to it.

On Windows 10 (2004) or later (x64), download the zip from the same page or the [latest GitHub release](https://github.com/kitlangton/scrawl/releases/latest), unzip it and run `scrawl.exe`. The Windows build is a preview and isn't signed yet, so Windows may say "Windows protected your PC": click **More info**, then **Run anyway**. It doesn't update itself yet; see [Windows](#windows).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 940 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Scrawl
  2. Show an agent a problem instead of describing it.
  3. Press a shortcut (`⌥⇧S`) and Scrawl starts recording the screen. Talk, click around, draw on it, and press `⌥⇧S` again to finish. Scrawl records the screen, your voice, and your marks, then packs them into a compact bundle: a contact sheet of annotated keyframes (`contact-sheet.png`), individual full-resolution keyframes (`frames/frame-XX.png`), a time-aligned transcript and interaction timeline (`summary.md`), and a structured manifest (`scrawl.json`).
  4. ## Install
  5. Scrawl needs an Apple silicon Mac on macOS 15 or later (live transcription and Liquid Glass need
  6. macOS 26). Download the DMG from [scrawl.kitlangton.com](https://scrawl.kitlangton.com), or use
  7. Homebrew:
  8. ```sh
  9. brew install --cask kitlangton/tap/scrawl
  10. ```
  11. The app updates itself (Sparkle), so Homebrew leaves updates to it.
  12. On Windows 10 (2004) or later (x64), download the zip from the same page or the
  13. [latest GitHub release](https://github.com/kitlangton/scrawl/releases/latest), unzip it and run
  14. `scrawl.exe`. The Windows build is a preview and isn't signed yet, so Windows may say "Windows
  15. protected your PC": click **More info**, then **Run anyway**. It doesn't update itself yet; see
  16. [Windows](#windows).
  17. ## Agents
  18. Install the agent skill for Claude Code, OpenCode, Codex and other agents:
  19. ```sh
  20. bunx skills add kitlangton/scrawl
  21. ```
  22. The `scrawl` command is the binary inside Scrawl.app. The Homebrew cask links it onto your PATH, or click
  23. **Install** beside Command-line tool in Settings › Agents, which links it into `/opt/homebrew/bin`
  24. or `/usr/local/bin` when you can write there, else `~/.local/bin`. The link always runs the app's
  25. current copy, so it updates with the app.
  26. With the skill, an agent that gets a Scrawl paste reads the contact sheet and transcript, opens the
  27. keyframes the narration points at, and uses the command to look closer: `scrawl zoom <bundle> --at
  28. 0:12` (or `--at "a phrase you said"`, or `--mark 2` for the mark the text shows as ✎②; `scrawl
  29. inspect --marks` lists them) makes a dense sheet of frames around a moment, `scrawl
  30. frames` pulls a range, `scrawl send` queues a capture into an OpenCode session, and `scrawl export`
  31. makes a shareable `.mp4`. `scrawl skill` prints the skill without installing anything.
  32. ## Usage
  33. ```sh
  34. # Launch the GPUI menu-bar and review app (press ⌥⇧S to record)
  35. cargo run -- app
  36. # Generate a realistic multi-frame demo bundle and print its agent summary
  37. cargo run -- demo --out /tmp/demo.scrawl
  38. # Open a bundle (or `latest`) in the native GPUI review window
  39. cargo run -- preview /tmp/demo.scrawl
  40. # Print the agent summary or JSON manifest for a bundle
  41. cargo run -- inspect /tmp/demo.scrawl
  42. cargo run -- inspect /tmp/demo.scrawl --json
  43. # Extract frames in a time window for an agent to look closer
  44. cargo run -- frames /tmp/demo.scrawl --from 0:02 --to 0:07
  45. # Copy contact-sheet.png + summary.md to the clipboard or send to OpenCode V2
  46. cargo run -- send /tmp/demo.scrawl --clipboard
  47. cargo run -- send /tmp/demo.scrawl --session ses_...
  48. # Which session a capture of a dev site would go to (without sending)
  49. cargo run -- route --url http://localhost:5173/
  50. # Capture a cabled iPhone for 8 seconds from the terminal (mirror window + native video)
  51. cargo run -- record --iphone --duration 8
  52. cargo run -- devices
  53. # Build a bundle from a ScrawlKit in-app recording (newest pair in ~/Downloads by default)
  54. cargo run -- import
  55. cargo run -- import ~/Downloads/20261013T164000.mp4 --timeline ~/Downloads/20261013T164000.json
  56. # Rebuild an existing bundle with the current transcriber and keyframe rules
  57. cargo run -- reprocess latest --out /tmp/again.scrawl
  58. # Build a capture whose build never ran (the app died first), in place; the app does this at launch
  59. cargo run -- recover ~/Library/Application\ Support/scrawl/captures/scrawl-….scrawl
  60. # Tune auto-crop on a Screen capture, live
  61. cargo run -- autocrop latest --serve --open
  62. # Export a bundle as a shareable .mp4 (crop to the activity, ink burned in, audio; --zoom for auto-zoom)
  63. cargo run -- export latest --zoom --out /tmp/demo.mp4
  64. # Clean up old captures now (the app does it daily), or keep one forever
  65. cargo run -- clean --dry-run
  66. cargo run -- pin latest
  67. ```
  68. ## Welcome and permissions
  69. On first launch Scrawl opens a welcome window with four screens: a welcome, how ⌥⇧S works, one
  70. screen with every permission Scrawl needs, and a first capture. Each permission is a row with a
  71. button that shows the macOS prompt (or, once asked, opens System Settings) and a check that appears
  72. by itself when macOS grants it (the window re-reads the grants every 0.5 s). Screen Recording is
  73. first and its button is filled in; after that the next row's button is. Grant them in any order;
  74. Continue unlocks once all three are on, and the screen is skipped when they already are.
  75. | Permission | Used for | When it's asked for |
  76. |---|---|---|
  77. | Screen & System Audio Recording | Recording the screen (ScreenCaptureKit) | onboarding, required |
  78. | Microphone | Your narration | onboarding, required |
  79. | Accessibility ("Device Control and Data Access" on macOS 27) | The key event tap (⌘Z, Esc, ⌘-shortcuts in the window log), and `⌥⇧V` and the Paste chip, which post ⌘V into the app in front | onboarding, required |
  80. | Camera | The cabled iPhone mirror (macOS treats it as a camera) | the first time you pick iPhone |
  81. Scrawl never asks for Input Monitoring. Clicks, drags and scrolls in other apps come from an AppKit
  82. global mouse monitor (`NSEvent.addGlobalMonitorForEvents`), which needs no permission; ⇧ for ink
  83. and whether ⌥⇧ is still held after ⌥⇧S come from polled modifier flags, which need none either; key presses
  84. come from an active event tap, which Accessibility allows. macOS applies an Accessibility grant to
  85. a running app at once (System Settings shows no "Quit & Reopen" for it), and Scrawl makes the tap
  86. within half a second of it.
  87. Screen Recording and Accessibility are lists you drag an app into, so their button skips Apple's
  88. prompt: it opens the pane in System Settings, and a card flies from the button to just below the
  89. Settings window (or, with no room on screen there, over the window's empty lower area), with its
  90. Scrawl row lined up under the list. Drag that row into the list. The card follows the window,
  91. steps aside when another app is in front, and shows "Scrawl is allowed" once a fresh process sees
  92. the grant (a drop alone proves nothing). The row drags the `Scrawl.app` that contains the helper,
  93. the bundle macOS grants. With Reduce Motion on, the card fades in place. Microphone and Camera
  94. can't be dragged in, so they use the macOS prompt.
  95. Camera is asked for in the same window, with just its row, when first needed. Speech recognition
  96. isn't asked for: SpeechAnalyzer transcribes without it. The ⌥⇧S and ⌥⇧V hotkeys themselves need
  97. no permission (Carbon `RegisterEventHotKey`).
  98. Until the required permissions are on, ⌥⇧S starts nothing (no picker or overlay that
  99. could sit over a macOS prompt): once the keys are released it brings the permissions window
  100. forward.
  101. A Screen Recording grant may not reach a running app, and macOS offers **Quit & Reopen** or
  102. **Later** as soon as it's granted. Either way nobody reopens Scrawl by hand. If you pick **Quit &
  103. Reopen**, macOS restarts Scrawl and the tour opens on the permissions screen again, Screen
  104. Recording checked. If you pick **Later** (or nothing), Scrawl checks whether it can capture anyway
  105. (a one-frame ScreenCaptureKit screenshot, tried only once a fresh helper reports the grant), and if
  106. it can't, the row shows "Reopening Scrawl" for a second and Scrawl relaunches itself into the same
  107. screen. It only does that once in two minutes (`relaunchedAt` in the config); if a fresh copy
  108. still can't capture, the row's button becomes **Reopen Scrawl**. The relaunch waits for this copy
  109. to exit and opens the app without `-n`, so if macOS's Quit & Reopen got there first there's still
  110. one Scrawl. `scrawl sck-probe [--watch SECONDS]` and `scrawl-helper sck-probe` try the same
  111. capture from the command line.
  112. Progress is `"onboarding"` in `~/.config/scrawl/config.json`, including the screen the tour is on,
  113. so it picks up there whatever quits Scrawl. Once seen, the tour only comes back from **Welcome
  114. tour** in Settings › About, but a required permission that was granted and is gone again reopens the
  115. permissions screen at launch. `SCRAWL_NO_ONBOARDING=1` skips it.
  116. To replay it from scratch:
  117. ```sh
  118. tccutil reset All dev.kitlangton.scrawl
  119. jq 'del(.onboarding)' ~/.config/scrawl/config.json > /tmp/c.json && mv /tmp/c.json ~/.config/scrawl/config.json
  120. # then quit and reopen Scrawl
  121. ```
  122. Preview every screen without the app, including the permissions screen with none, some and all
  123. granted and each on-demand sheet: `target/debug/scrawl-helper onboarding --render /tmp/onboarding
  124. --icon assets/app-icon.png`.
  125. ## Settings
  126. **Settings…** in the menu bar (⌘, while the menu is open) opens a small window with six panes:
  127. - **General**: Launch at login, Send dev-site captures to OpenCode, and the two shortcuts, Record
  128. (`⌥⇧S`) and Paste last capture (`⌥⇧V`, which can be turned off).
  129. - **Recording**: the microphone, with its input level, the transcriber (Apple Speech with its
  130. language, Hex while it runs, with its model and language, or Command when it's configured) and,
  131. under Experimental, Video export.
  132. - **Storage**: how long recordings are kept, where new captures go (Change…/Reset on macOS, `"capturesDir"` in
  133. `~/.config/scrawl/config.json` everywhere; captures already made stay put, and an unusable folder falls back to the built-in one), how much space they take, and Show in Finder.
  134. - **Permissions**: the onboarding checklist, so a permission can be turned on later.
  135. - **Agents**: Install or Uninstall the `scrawl` command (see [Agents](#agents)), and the skill's
  136. install command with Copy. A `scrawl` that belongs to another program is left alone.
  137. - **About**: the version, Check for Updates…, links, and the welcome tour.
  138. To change a shortcut, click its keys and press the new combination; `Esc` cancels. It needs ⌘, ⌥
  139. or ⌃ (a function key can stand alone), and Scrawl turns down ⌘ with one key, combinations macOS
  140. uses, the other Scrawl shortcut, and anything another app has already taken. The arrow beside the
  141. heading puts the default back. While the recorder listens, Scrawl's own shortcuts are paused. They
  142. are saved as `"shortcuts"` in `~/.config/scrawl/config.json`. The menu keeps Record Screen, Choose
  143. What to Record…, Open Latest Capture, Export Last Capture as Video… (when on), Settings…, Check
  144. for Updates… and Quit.
  145. **Microphone** lists the connected inputs, with System default first, which follows whatever macOS
  146. uses as it changes. The list updates as devices are plugged in or out. A chosen device is saved by
  147. its CoreAudio UID as `"microphone": {"id", "name"}`; if it isn't connected when you record, Scrawl
  148. records from the default instead and says so under the popup (and once in the log). The level
  149. meter runs only while the Recording pane is on screen and Microphone is allowed. Cabled-iPhone
  150. narration uses the same microphone. From the terminal, `scrawl mic list` shows the inputs and the
  151. one recordings will use, `scrawl mic set <id|default>` chooses, and `scrawl mic watch` prints the
  152. list again on every change.
  153. Preview every pane, dark and light, with the recorder listening and a combination turned down:
  154. `target/debug/scrawl-helper settings --render /tmp/settings --icon assets/app-icon.png`.
  155. ## Capture Controls
  156. - **Press `⌥⇧S`**: record the display under the pointer as soon as you let go, however long you
  157. held it, auto-cropped (see below). Press it again to finish; a press within 0.4 s of the start is
  158. ignored, so a double press doesn't leave an empty capture. `⌃⌥⇧S` also works (only while `⌥⇧S` is the
  159. shortcut; see [Settings](#settings) to change it). As it starts, the island opens onto the modes (Area · Window ·
  160. **Screen** · iPhone when one is cabled) for about 3.5 s, or until you click, scroll or draw;
  161. hovering keeps them. Click Window or Area to retarget (the first seconds are dropped and the
  162. picker takes over: click a window or drag an area), iPhone to record the phone instead, or just
  163. carry on.
  164. - **Choose What to Record…** in the menu bar: the picker (Area, Window, Screen, iPhone) opens on
  165. the display under the pointer. Click a mode, click a window or drag an area; `Esc` cancels.
  166. - `⌥⇧V`: Paste both `contact-sheet.png` and `summary.md` into the active app or terminal (`⌃⌥⇧V` also works while it's the default)
  167. - **Normal pointer**: Clicks and scrolls pass straight through to the app underneath
  168. - **Hold `⇧` (Shift) + move**: Draw a persistent red marker stroke (`#ff3b30`) on screen that fades out after release
  169. - **`⇧`-click**: Drop a numbered pin where you click (a press that moves under 12 pt and lets go within 1 s; longer is ink). It shows as a brief red pulse, not a lasting dot, and lands on the keyframes. A click after you've drawn in the same `⇧` hold is part of the drawing and drops no pin
  170. - **`⌘Z`**: Undo the last stroke
  171. - **`Esc`**: Cancel the active selection or capture
  172. While picking a target, the island shows the modes **Area**, **Window**, **Screen**, and **iPhone**
  173. (when one is cabled). It opens on the mode you used last:
  174. - `1`–`4` jump to a mode; `Tab`/`→` and `⇧Tab`/`←` step through them; `Space` toggles Area and Window
  175. - Area: drag a region. Window: click a window. Screen and iPhone: `⏎`, `S`, `⌥⇧S`, **Record**, or
  176. clicking the selected mode again starts recording (`⏎`/`⌥⇧S` record the full screen from Area and
  177. Window too)
  178. ## Sending captures of a dev site to its session
  179. A capture that ends with a browser in front showing a project's dev site goes straight to that
  180. project's OpenCode session, the same way iPhone recordings do. It is still copied to the clipboard.
  181. The island says "Sending to SCRAWL…", then "Sent to SCRAWL", or "Copied · couldn't reach SCRAWL"
  182. when OpenCode didn't take it (the app retries it, as for iPhone recordings). Window captures use the
  183. captured window's app instead of the one in front. Brave, Chrome, Arc, Edge, Vivaldi, Chromium, and
  184. Safari work; macOS asks once whether Scrawl may control the browser, so it can read the tab's
  185. address. If you decline, captures stay on the clipboard.
  186. Scrawl picks the session in this order:
  187. 1. **The page registered itself.** Add one dev-only line to the page:
  188. ```html
  189. <script src="http://127.0.0.1:47777/sdk.js" data-session="ses_…"></script>
  190. ```
  191. or `data-directory="/abs/path/to/project"` to use that project's most recently active session.
  192. The script comes from Scrawl itself. It registers the tab, sends a heartbeat every 30 s while the
  193. tab is visible, and unregisters when the tab closes (a registration lasts 2 minutes after its
  194. last heartbeat). With Vite, inject it only in dev:
  195. ```ts
  196. // vite.config.ts
  197. plugins: [{
  198. name: "scrawl",
  199. apply: "serve",
  200. transformIndexHtml: () => [{
  201. tag: "script",
  202. attrs: { src: "http://127.0.0.1:47777/sdk.js", "data-directory": process.cwd() },
  203. injectTo: "head",
  204. }],
  205. }]
  206. ```
  207. The script does nothing unless the page is on `localhost`, `*.localhost`, `127.x`, or `[::1]`.
  208. Scrawl accepts a registration without a token only from a loopback origin calling from this Mac,
  209. and only for that origin. A page anywhere else (a `*.ts.net` dev host, say) must also set
  210. `data-token` to the ingest token (`ingestToken` in `~/.config/scrawl/config.json`).
  211. 2. **Zero setup for localhost.** If the tab is on a loopback host and no page registered, Scrawl
  212. finds the process listening on that port (`lsof`), its working directory and that directory's
  213. git root, and sends to the most recently active open root session in the repository (any folder
  214. in it). Outside a git repository only a session in that exact folder counts.
  215. 3. Otherwise the capture is copied to the clipboard only, as before.
  216. The destination is recorded in `scrawl.json` (`destination: {session, title, via: "sdk" |
  217. "localhost", url, directory}`) and on the second line of `summary.md`
  218. (`For session: SCRAWL (ses_…) · page http://localhost:5173/`). `scrawl send` without `--session`
  219. sends to it again, and refuses a different `--session` unless you pass `--reroute`.
  220. To see where a capture of a URL would go, without sending anything:
  221. ```sh
  222. scrawl route --url http://localhost:5173/
  223. scrawl route --browser com.brave.Browser # the browser's front tab
  224. ```
  225. Turn it off with **Send dev-site captures to OpenCode** in Settings › General, or `"autoSend": false` in
  226. `~/.config/scrawl/config.json`. `SCRAWL_NO_OPENCODE=1` also turns it off.
  227. ## Auto-crop (Screen captures)
  228. Instead of picking an area up front, record the whole screen: tap `⌥⇧S`, or choose **Screen**.
  229. Area and Window captures aren't cropped.
  230. Scrawl works out where you were gesturing and crops each keyframe to it. While a Screen capture
  231. records, Scrawl logs:
  232. - **`windows`** in `scrawl.json`: the on-screen windows, front to back (id, app, bundle ID, title,
  233. bounds in capture points, layer, frontmost), including the menu bar and the Dock's strip. A
  234. snapshot is taken at the start, when a click, ink stroke or shortcut starts, when the frontmost
  235. app changes, at most every 500 ms while the cursor moves and every 2 s otherwise, and stored only
  236. when the list changed. Each snapshot costs about 0.5 ms. Titles stay in `scrawl.json`; they never
  237. go into `summary.md` or the clipboard.
  238. - **`cursor`**: the cursor at up to 15 Hz while it moves, as `[t, x, y]` (normalized), so dwell
  239. counts.
  240. When the capture finishes, each keyframe gets the slice of the timeline halfway to its neighbours,
  241. and `src/autocrop.rs` weighs what happened in it: ink, clicks, drags and scrolls first, then where
  242. the cursor rested and what changed on screen (a cursor wandering far from the ink is ignored). The
  243. crop is then:
  244. - **the window** holding the activity (two at most, for an arrow from one to the other), including
  245. a menu it opened and the touched part of the menu bar;
  246. - or, when that window is most of the screen (maximized, full screen), **the activity inside it**
  247. with some context, its edges moved into nearby white space and past the ends of any lines of
  248. text they cross (up to the window's edge), so they don't cut words in half;
  249. - or the previous crop, for a keyframe with only talk.
  250. Neighbouring crops that mostly overlap become one, so the sheet doesn't jitter. A crop covering
  251. most of the screen isn't worth it, and the keyframe stays whole.
  252. The bundle then has:
  253. - `frames/frame-XX.png` and `contact-sheet.png`: cropped from the full-resolution frames (Screen
  254. keyframes are decoded from the 2560 px wide recording), with the ink and keyframe numbers redrawn at the crop's scale;
  255. - `frames-full/frame-XX.png` and `contact-sheet-full.png`: the uncropped versions;
  256. - one line in `summary.md`, e.g. `Cropped to: Ghostty + Safari (full frames in frames-full/)`;
  257. - `autocrop` in `scrawl.json`: each keyframe's crop in points, its kind and reason, and the
  258. changed regions that fed it.
  259. `scrawl zoom` crops its cells like the nearest keyframe (`--full` for the whole screen).
  260. `scrawl reprocess <bundle> --out <dir>` re-runs it on an existing Screen capture from its video.
  261. To turn it off, or change a setting, use `~/.config/scrawl/config.json`:
  262. ```json
  263. { "autocrop": false }
  264. { "autocrop": { "padding": 24, "minW": 480 } }
  265. ```
  266. ### Tuning
  267. `scrawl autocrop <bundle|latest> --serve --open` opens a playground that runs the same Rust crop as
  268. you move its sliders: each keyframe with its windows, activity, cursor heat, outliers and crop, the
  269. cropped frame, and the cropped contact sheet. **Copy settings** gives the JSON for the config.
  270. Without `--serve` it writes `autocrop/index.html` with the presets only.
  271. The settings were tuned on simulated captures with hand-labelled ideal crops: one maximized
  272. terminal, two windows side by side, an arrow from one window to another, a tiny badge in a huge
  273. window, a menu-bar menu, talk-only keyframes, a dragged window, overlapping windows, thirty seconds
  274. hopping between three windows, a stray cursor, a full-screen app, and a scrolling page.
  275. `scripts/autocrop-corpus.mjs` renders them with Playwright (from `site/node_modules`), `scrawl
  276. fixture` builds each into a real bundle through the capture pipeline, `scrawl autocrop-eval` scores
  277. the crops (coverage of the ink and clicks, overlap with the ideal, context, legibility in the sheet,
  278. stability; `--sweep` searches the settings), and `scripts/autocrop-gallery.mjs` renders a gallery to
  279. judge them by eye:
  280. ```sh
  281. bun scripts/autocrop-corpus.mjs /tmp/corpus
  282. for s in /tmp/corpus/*/scenario.json; do
  283. scrawl fixture "$s" --out "/tmp/bundles/$(basename "$(dirname "$s")").scrawl"
  284. done
  285. scrawl autocrop-eval /tmp/bundles --json /tmp/scores.json # add --sweep to search
  286. bun scripts/autocrop-gallery.mjs /tmp/gallery /tmp/bundles --scores /tmp/scores.json
  287. ```
  288. `autocrop-eval --export src/autocrop-corpus.json` refreshes the corpus the unit tests score.
  289. ## iPhone
  290. Plug an iPhone in with a cable (unlocked, trusted) and the picker shows **iPhone** next to Area,
  291. Window, and Screen. Choosing it opens a floating mirror of the
  292. phone on the right of the screen and starts recording:
  293. - The phone's own video is recorded natively (1320×2868 at up to 60 fps on an iPhone Pro Max) with
  294. your narration from the Mac's microphone, into the bundle as `recording.mov`. Keyframes are
  295. decoded from that video, never from a screenshot of the mirror. A minute after the capture the
  296. video is re-encoded to HEVC at the keyframes' 1600 px (see [Keeping captures small](#keeping-captures-small)).
  297. - The laser, ink, pins, and clicks work over the mirror exactly as over a window. The mirror's
  298. on-screen rect is the capture target, so everything you draw is stored in phone-screen
  299. coordinates and lands in the right place on the phone keyframes. Move or resize the mirror
  300. freely; the mapping follows it.
  301. - Press **M** with the mirror focused to flag a moment ("look here"); flags always get a keyframe
  302. and show as `⚑` in the summary.
  303. - Finish from the island as usual. Unplugging the phone finishes the capture with what was recorded.
  304. Phone bundles are ordinary bundles: `scrawl.json` adds `"source": "iphone"` and a `device` object
  305. (name, capture `mirror` or `scrawlkit`, id, native size, screen points and scale, app and version
  306. for SDK recordings), and the target's `mode` is `device`. Keyframes fit a 1600 px long edge
  307. (736×1600 for a 1320×2868 phone); `scrawl frames` and `scrawl zoom` extract more frames from the
  308. video when you need them.
  309. ### ScrawlKit (iOS)
  310. `ScrawlKit` (`Sources/ScrawlKit`) is a Swift package (iOS 18+) for recording inside your own iOS app:
  311. video of the app with narration, touches and freehand red strokes, markers, and app events on the
  312. same clock, sent straight to Scrawl on your Mac.
  313. Add this repo as a Swift package dependency (product `ScrawlKit`) and attach `.scrawl()` to your root
  314. view:
  315. ```swift
  316. import ScrawlKit
  317. WindowGroup { ContentView().scrawl() } // three-finger long press starts and stops
  318. Scrawl.event("Done tapped") // optional: what the app did, on the timeline
  319. ```
  320. The host app needs `NSMicrophoneUsageDescription` in its Info.plist for narration (both capture
  321. methods use the microphone). Set `Scrawl.shared.narration = false` to record without it.
  322. `Scrawl.shared.captureMethod` picks how the video is made:
  323. - **`.appFrames`** (default): ScrawlKit draws the app's own windows into an H.264 `.mp4` at 30 fps
  324. and 2× (880×1912 on a 440×956-point phone), with the microphone recorded alongside. There is no
  325. screen-recording prompt; the microphone asks once and iOS remembers the answer. If it's denied, the
  326. recording is video only and the timeline says `"audio": false`. The strip and ink are left out of
  327. the video (Scrawl on the Mac draws touches and strokes from the timeline), but so is
  328. anything the app doesn't draw itself: system alerts and permission prompts, and some layers (video
  329. players, camera previews, some Metal or backdrop effects) may be blank or look different. The
  330. system keyboard is drawn by another process, so its keys can't be captured: a plain panel labelled
  331. "Keyboard" stands in for it, following it as it slides (taps on the keyboard never reach the app,
  332. so they aren't on the timeline either). Sending the app to the background finishes and sends the
  333. recording.
  334. - **`.replayKit`**: ReplayKit's in-app screen recording, the exact pixels on screen with the strip
  335. and the keyboard in them. iOS asks "Record screen and microphone?" unless you allowed it in the
  336. last ~8 minutes, and again after every reinstall.
  337. The timeline records which was used (`"capture": "appFrames"` or `"replayKit"`). With either, the
  338. context records the keyboard: `keyboard: visible` if it was up at the start, and
  339. `keyboard → visible` / `keyboard → hidden` when it comes and goes.
  340. On paired devices, the three-finger trigger turns off iOS's three-finger text-editing gestures in
  341. the app (UIKit's `editingInteractionConfiguration` answers `.none` for the app's windows), because
  342. the same press would otherwise open the Undo · Cut · Copy · Redo bar when a text field is focused.
  343. The cost is that three-finger undo, redo, cut, copy, and paste don't work in that app on a paired
  344. phone. Set `Scrawl.shared.suppressesSystemEditingGestures = false` to keep them. Unpaired devices
  345. are unaffected.
  346. While recording, a slim strip sits on the right edge (drag its dot to move it up or down or over to
  347. the left edge; it remembers where you left it). From top to bottom: the recording dot (hold it to see
  348. the elapsed time), Draw (tap to turn drawing red strokes on or off), Discard (with a few seconds to
  349. Undo), and Send. Doing the three-finger trigger again also sends. The strip moves like liquid (the
  350. "Liquid morph" variant of `site/public/prototype/strip`): it stretches out of a droplet, pours into
  351. the button you tap to become a status circle, buds the "Sent to …", "Discarded · Undo" or "Mac didn't
  352. answer · Share" bubble off on a neck, and stretches as you throw it. On iOS 26 it's Liquid Glass and
  353. the neck is the glass itself blending; Reduce Motion cross-fades instead.
  354. Drawn strokes are smooth and fade on their own:
  355. - **Sampling.** Every coalesced touch is a sample, at the touch's own timestamp, so a busy main
  356. thread doesn't make strokes jagged. A one-euro filter (one cutoff from the 2D speed) takes out
  357. jitter as they come in, and a centered smoothing takes out the rest behind the tip without lag.
  358. - **Look.** The line is a centripetal Catmull-Rom curve, 4.6 pt wide when drawn slowly and down to
  359. 55% when fast (from speed, never pressure, like the Mac's ink), with tapered ends.
  360. - **Drawing.** Finished stretches go into layers that never change, so each touch redraws only the
  361. live tip, which runs ahead through iOS's predicted touches. Predicted points are dropped on the
  362. next touch and never recorded.
  363. - **Fading.** A stroke fades out (300 ms, ease-out) 1.5 s after the pen last lifted. Putting the pen
  364. down again before then holds every stroke on screen, so a circle and then an arrow fade together.
  365. - **Timeline.** It gets the smoothed, unpredicted points, plus `fadeAt` (when the stroke began
  366. fading) on each stroke that faded during the recording. On keyframes the Mac draws a stroke from
  367. when it began until 1.2 s after it ended, or with the rest of its mark.
  368. Touches aren't drawn on screen in either capture method. Each touch goes on the timeline (position,
  369. phase, and the label of the element under it) and Scrawl on the Mac draws taps and swipes from it.
  370. ScrawlKit stays completely inert on devices that haven't been configured (no gesture recognizers,
  371. overlay window, network calls, files, or background `URLSession`). To pair your iPhone with this Mac, print a setup link (and
  372. terminal QR code) and open it on the phone:
  373. ```sh
  374. scrawl link --scheme updown [--session ses_…]
  375. # updown://scrawl?url=http%3A%2F%2F100.64.0.12%3A47777&url=http%3A%2F%2F192.168.1.20%3A47777&token=…
  376. ```
  377. `scrawl link` reads (or generates with mode `0600`) `ingestToken` in
  378. `~/.config/scrawl/config.json` and defaults to this Mac's Tailscale and LAN IPs on port `47777`.
  379. Opening `<scheme>://scrawl?reset=1` on the phone clears its Keychain config (`dev.kitlangton.scrawl`).
  380. For cabled USB fallback via `devicectl` (`Documents/Scrawl`), list the app bundle IDs in
  381. `~/.config/scrawl/config.json` under `"usbApps": ["com.kitlangton.UpDown"]`.
  382. The Mac answers an upload once it has built the bundle and queued it into the OpenCode session (or
  383. after up to 45 s), so the strip says what happened:
  384. | On the Mac | Reply | The strip |
  385. | --- | --- | --- |
  386. | Queued into the session | `delivered: true, state: "sent"` | Sent to UP DOWN |
  387. | Bundle saved, OpenCode didn't take it (retried from the Mac) | `delivered: false, state: "queued"` | On your Mac · Sends to UP DOWN when it can |
  388. | Still building or sending when the wait ran out | `delivered: false, state: "processing"` | On your Mac · Sending to UP DOWN… |
  389. | Bundle saved, no session for this app (or routing off) | `delivered: false, state: "saved"` | Saved on your Mac |
  390. | Bundle couldn't be built | HTTP 500 | Mac didn't answer · Share |
  391. A queued bundle gets `.deliver.json` and the Mac retries it every 15 s, backing off to 30 min, for
  392. a day (then it's renamed `.deliver-failed.json`; both are logged to `~/Library/Logs/Scrawl/scrawl.log`).
  393. The phone doesn't upload it again. Over USB the Mac writes the same reply to `<stamp>.delivery`
  394. before `<stamp>.sent`. Replies without `delivered` (Macs from before this) read as sent.
  395. You can also AirDrop the `.mp4` and `.json` pair to the Mac and run `scrawl import`:
  396. - Touches become taps and swipes, drawn on keyframes as ripples and finger trails and listed in the
  397. review timeline. Each tap or swipe is a keyframe candidate, like a click.
  398. - Markers become flags.
  399. - App events (`Scrawl.event("Done tapped")`) appear in `summary.md` under the keyframe that was on
  400. screen when they happened (` · 0:12 app: Done tapped`).
  401. #### Destination and recovery
  402. The original `timeline.json` stores the paired destination in its top-level `session` field.
  403. `context.session`, when the app supplies it, describes the session being viewed and does not choose
  404. the recipient. For example, recording OpenCode Settings while viewing another project's session
  405. can still send to the paired OpenCode iOS coordinator. Automatic Mac ingest resolves the top-level
  406. destination first; without one, it uses the app/session fallback in `resolve_opencode_session`.
  407. When recovering a failed delivery, read that original top-level field and check whether the capture
  408. already arrived before sending again. `scrawl send --session` explicitly overrides the recipient;
  409. do not fill it from `context.session`. If there is no recorded destination and ownership is unclear,
  410. ask before forwarding. Keep private-project endpoints and recipients unchanged.
  411. #### Sending to an HTTPS endpoint
  412. A setup link can point at a hosted endpoint instead of a Mac, for phones that aren't on your LAN or
  413. tailnet (for example `ledger://scrawl?url=https%3A%2F%2Fproblems.example.com%2Fapi%2Fscrawl%2Fledger&token=pbcap_…`).
  414. It speaks the same protocol as the Mac: `GET {url}/health` and `POST {url}/ingest` with a bearer
  415. token, where the body is a 4-byte big-endian timeline length, the timeline JSON, then the MP4. When
  416. any configured URL is `https`, recordings go through a durable outbox instead of the LAN path
  417. (plain `http` URLs keep the direct POST plus USB fallback):
  418. - **Saved before sending.** On Send, the framed payload and a small metadata file (target,
  419. `recordedAt`, attempts, last error) are written to `Application Support/ScrawlOutbox`. They're
  420. protected until the phone's first unlock, so uploads can run while it's locked, and excluded from
  421. backup. The recording never goes to `Documents/Scrawl` (it's staged in `tmp`) and isn't added to
  422. `Scrawl.shared.recordings`. An entry is deleted only after a 2xx.
  423. - **Background upload.** Uploads are file uploads on a background `URLSession`
  424. (`dev.kitlangton.scrawl.outbox.<bundle ID>`), so they survive the app being suspended or killed
  425. and never load the movie into memory. The session is recreated whenever the app is configured, and
  426. it picks up uploads still in flight from an earlier launch.
  427. - **Retries.** A 401 or 403 (the link was revoked), a 413 (too large) and any other 4xx except 408
  428. and 429 (the server rejected these exact bytes) are permanent: the entry stays on disk, marked
  429. failed, with Share. Anything else (offline, timeouts, 408, 429, 5xx) is retried
  430. with backoff (15 s, doubling up to 30 min), and immediately when the network comes back, the app
  431. comes to the foreground, or it launches. An entry never uploads twice at once. The server dedupes
  432. on person and `recordedAt`, so resending after a lost reply is safe. Opening a new setup link
  433. retries the entries that failed with the old one.
  434. - **Strip.** While the upload runs on a working network (`NWPathMonitor` says `.satisfied`), the
  435. strip stays up as a small status circle with a progress ring, saying "Sending…" however long it
  436. takes. Touches on it reach the app. With no bytes going out for 30 seconds it says "Still
  437. sending…", never "Queued". When the upload lands it turns into "Sent to …" with a success haptic.
  438. "Queued · sends when online" (with a success haptic, since the recording is safe on the phone) is
  439. only for no network or a failed attempt waiting to retry; it leaves after a few seconds, and if a
  440. retry starts while it's up it goes back to "Sending…". If the upload lands after the strip has
  441. gone (a new recording took over, it was queued, or the app was in the background), the strip comes
  442. back briefly with "Sent to …", as soon as the app is in front. Permanent failures say "Scrawl link
  443. expired · Ask Kit for a new one", "Too large to send" or "Couldn't send", with Share, which shares
  444. copies of the recording and leaves the entry queued.
  445. - **Revoked links.** The `/health` check at the start of a recording also goes to the HTTPS
  446. endpoint. A 401 there shows "Scrawl link expired" beside the strip, but the recording carries on
  447. and is kept in the outbox, marked failed, until the link is renewed.
  448. The endpoint rejects bodies over 95 MB, so HTTPS recordings are smaller and capped:
  449. | Capture | Video | Audio | Per second | Cap | At the cap |
  450. | --- | --- | --- | --- | --- | --- |
  451. | `.appFrames` to a Mac | 6 Mbps | 96 kbps | ≈ 0.76 MB | none | — |
  452. | `.appFrames` to HTTPS | 3 Mbps | 96 kbps | ≈ 0.39 MB | 3:00 | ≈ 70 MB |
  453. | `.replayKit` to HTTPS | not settable (est. 8–12 Mbps) | | ≈ 1–1.5 MB | 0:45 | ≈ 45–68 MB |
  454. 3 Mbps at 880×1912 and 30 fps is about 0.06 bits per pixel. That's plenty for app UI, which is mostly
  455. static between scrolls and transitions, and the 25 MB of headroom covers the encoder overshooting its
  456. average. Ten seconds before the cap, the strip shows "Length limit · Sending in 10 s" with a warning
  457. haptic, then sends. A payload that's still over the limit is never uploaded: it's marked "Too large
  458. to send" and kept for Share.
  459. Apps with an app delegate can optionally forward background session events, so iOS can relaunch a
  460. killed app to finish reporting uploads. Without this, finished uploads are picked up at the next
  461. launch:
  462. ```swift
  463. func application(_: UIApplication, handleEventsForBackgroundURLSession identifier: String,
  464. completionHandler: @escaping () -> Void) {
  465. Scrawl.handleBackgroundURLSessionEvents(identifier: identifier, completionHandler: completionHandler)
  466. }
  467. ```
  468. #### App context
  469. The app can say which screen it's on and add its own key/values, so the agent doesn't have to guess
  470. from pixels. ScrawlKit sends only what the app passes in, plus its bundle ID, name, version, and build.
  471. ```swift
  472. SettingsView().scrawlScreen("Settings › Experimental") // while visible; restores the previous on disappear
  473. SessionView().scrawlContext("session", session.id) // follows the value; nil leaves it to enclosing views
  474. Scrawl.screen("Onboarding") // or set either one directly (nil clears)
  475. Scrawl.context("server", "studio")
  476. ```
  477. The most recently appeared screen that is still visible wins, so pushed screens, sheets, and tabs
  478. work. The timeline gets `shortVersion`, `context` (values at the start), `contextChanges`
  479. (`[{"t", "key", "value"}]`, `null` when cleared), and `screens` (`[{"t", "name"}]`, the first at `0`).
  480. The bundle keeps them as `context`, `contextChanges`, and `screens` spans (`{t0, t1, name}`), and
  481. `summary.md` (and the prompt queued into OpenCode) shows them:
  482. ```
  483. [scrawl 0:25 · iPhone · OpenCode 1.4 (354) · /…/20261014-101500.scrawl]
  484. Open contact-sheet.png and frames/ in that folder before changing anything: the marks and words only make sense on the images, which can show more than the text does.
  485. Screen: Sessions · server: studio · session: ses_3f…
  486. - #1 (0:01) “So here.”
  487. - #3 (0:08) ⟨Settings › Experimental⟩ [tapped “Back”] “Go back.”
  488. · 0:09 server → mini
  489. ```
  490. ### Headless ingest server (`scrawl serve`)
  491. On an always-on Mac (such as a Mac Studio reachable over Tailscale), run `scrawl serve` to accept
  492. authenticated ScrawlKit uploads, build `.scrawl` bundles, and route them straight into local
  493. OpenCode sessions without running the GUI app:
  494. ```sh
  495. scrawl serve [--port 47777] [--usb]
  496. ```
  497. To run `scrawl serve` under `launchd`, copy `scripts/scrawl-serve.plist` into `~/Library/LaunchAgents`
  498. and load it:
  499. ```sh
  500. cp scripts/scrawl-serve.plist ~/Library/LaunchAgents/dev.kitlangton.scrawl.serve.plist
  501. launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/dev.kitlangton.scrawl.serve.plist
  502. ```
  503. ## Export as video
  504. `scrawl export` turns a bundle into one shareable `.mp4`: H.264 (`--hevc` for HEVC) with AAC audio in
  505. sync, the red ink, pins, laser and click ripples burned in at the times they happened, cropped to
  506. where the activity was. It runs in `scrawl-helper` on AVFoundation (hardware decode and encode, Core
  507. Image for the crop, Core Graphics for the ink), so nothing is decoded into Rust and ffmpeg isn't
  508. needed.
  509. ```sh
  510. scrawl export latest # ~/Movies/Scrawl/<bundle>.mp4
  511. scrawl export latest --out demo.mp4 --zoom # with Screen Studio-style auto-zoom
  512. scrawl export latest --crop full --size 1280 # whole capture, 1280 px long edge
  513. scrawl export latest --crop 0.1,0.2,0.6,0.5 # x,y,w,h as fractions, or pixels of the video
  514. ```
  515. - **Crop** (`--crop auto|full|x,y,w,h`): `auto`, the default, is auto-crop's whole-recording crop for
  516. Screen captures and everything for window, area and iPhone captures. The output keeps the crop's
  517. aspect with even sides, is capped at `--size` (1920) on its long edge, and is never scaled up.
  518. - **Zoom** (`--zoom`): the camera eases to ~1.5–1.8× toward clicks, drags, marks and laser points,
  519. holds while activity stays nearby (2.5 s after the last), follows the cursor once it leaves a
  520. central safe zone, and eases back out. It never starts a zoom while the cursor is travelling fast.
  521. The path is a pure function of the bundle (`src/export/camera.rs`, critically damped springs,
  522. no bounce); the helper only renders each frame's rectangle.
  523. - **Audio**: `audio.wav` starts where the video does (`videoOffsetMs`); a cabled iPhone's movie
  524. uses its own audio track. `--no-ink` and `--no-clicks` leave the annotations out, `--fps` sets the
  525. rate (default 60 for a 60 fps recording).
  526. - **Menu**: with **Video export** on in Settings › Recording (`"videoExport": true` in
  527. `~/.config/scrawl/config.json`, default off), the menu shows **Export Last Capture as Video…**. It saves to `~/Movies/Scrawl/<name>.mp4`, shows progress
  528. beside the menu bar icon, and reveals the file in Finder. The command works without the flag.
  529. ## Keeping captures small
  530. Captures would otherwise pile up (mostly video). The app cleans up shortly after launch and then
  531. hourly, at low priority, and logs what it did:
  532. - **Video, audio and full-size frames** (`recording.mp4`/`.mov`, `audio.wav`, `frames-full/`),
  533. plus the `zoom/` sheets made from them, go to the Trash after 7 days, as one `<bundle> media`
  534. folder. The prompt (`summary.md`, `scrawl.json`, `contact-sheet.png`, `frames/`) stays, so
  535. pasting and reading still work; `scrawl zoom` says the video was cleaned up.
  536. - **Whole bundles** go to the Trash after 30 days.
  537. - **Keep recordings** in Settings › Storage picks Video for 7 Days (bundles 30), Video for 30 Days
  538. (bundles 90), or Forever. It's stored as `"retention": {"mediaDays": 7, "bundleDays": 30}` in
  539. `~/.config/scrawl/config.json`; `null` keeps forever. The same pane shows how much the captures
  540. take and opens the folder in Finder.
  541. - Never touched: a bundle under a day old or changed in the last day (a copy just arrived from
  542. another Mac, a zoom), one whose media is still copying (`.media-pending` there, `.media-sending`
  543. here; a marker over a day old is ignored), one being recorded or re-encoded, and one with a
  544. `.pinned` file (`scrawl pin`, `scrawl pin --off`).
  545. - **iPhone video** from the cable is ~18 Mbps at 1320×2868. A minute after the capture (after any
  546. copy to another Mac finishes, and never during a recording) `scrawl-helper shrink` re-encodes it
  547. to HEVC at a 1600 px long edge, audio copied as is and every frame at its original time, and
  548. swaps it in place: a 4.5-minute capture goes from 750 MB to 72 MB in about 20 s. One pass a day
  549. shrinks any native iPhone video that was missed.
  550. `scrawl clean --dry-run` prints what a pass would do now.
  551. ## Transcription
  552. Narration is transcribed on device by Apple's SpeechAnalyzer (macOS 26), or on macOS 15 by the
  553. older on-device `SFSpeechRecognizer`. On macOS 15 onboarding asks for Speech Recognition, and
  554. Dictation's on-device model has to be on the Mac (System Settings › Keyboard › Dictation); Scrawl
  555. never falls back to Apple's servers. Both report when every word was spoken. The word timings
  556. drive two things:
  557. - **Text per keyframe.** Each keyframe's text runs from the start of the clause in progress when
  558. its moment began (the action, the transition, or the spoken cue) until the next keyframe, so the
  559. transcript is split at sentence and clause boundaries, in order, with nothing repeated or lost.
  560. - **Spoken cues.** Saying "now", "here", "look", or "watch" is a deliberate cue: if the screen
  561. changes around the word, that settled view becomes a keyframe; otherwise the frame as you said it.
  562. A cue only cuts a keyframe if its image differs from every keyframe already chosen, so talking
  563. over an unchanged screen still produces one frame. "This", "that", "these", "those", "see", and
  564. "there" count only when they coincide with a change, because they are usually filler. Speech
  565. pauses never cut keyframes.
  566. ### Apple or Hex
  567. Apple's SpeechAnalyzer is the default; Scrawl prepares its language model automatically. If you use Hex
  568. (2.2 or later, running), Scrawl can transcribe with it instead and with whichever model Hex has
  569. ready: Settings › Recording › **Transcription** → **Hex**, then pick a model under **Hex model**.
  570. Or in
  571. `~/.config/scrawl/config.json`:
  572. ```json
  573. { "transcriber": "hex", "hexModel": "whisper_large_v3_turbo", "hexLanguage": "nl" }
  574. ```
  575. `"transcriber"` is `"apple"` (default), `"hex"` or `"command"` (below); `hexModel` defaults to the
  576. first ready one of `parakeet_unified_en`, `parakeet_v3`, `parakeet_v2`, `whisper_large_v3_turbo`.
  577. Scrawl never downloads or prepares a Hex model: install it in Hex first.
  578. **Language** in Settings › Recording picks the language Apple Speech transcribes; **Automatic**
  579. (the default) follows the Mac's language. In config it's `"appleLocale"`, e.g. `"zh-TW"` for
  580. Traditional Chinese (Taiwan). Scrawl downloads that language's model in the background, and a
  581. language this Mac doesn't support falls back to Automatic (Settings names the one it uses).
  582. **Hex language** in Settings › Recording (or `"hexLanguage"` in config) sets the language
  583. code sent to Hex. It defaults to `en` to preserve existing behavior; use `nl` for Dutch or
  584. `auto` for automatic detection when the model supports it. Press Return or Apply to save;
  585. a blank field resets to `en`. Codes are not restricted to a list in Scrawl: Hex validates
  586. model/language support. If Hex rejects a language/model combination, Scrawl uses the existing
  587. Apple Speech fallback. This setting does not inherit Hex's dictation language. Whisper uses the selected
  588. language to guide recognition; models such as Parakeet v3 determine language from audio
  589. and ignore the hint.
  590. - Hex is optional. When it isn't running, **Hex** is greyed out as "Hex (not running)" and Scrawl
  591. uses Apple, whatever the saved preference. If Hex fails on a recording, that recording falls back
  592. to Apple Speech (see `~/Library/Logs/Scrawl`).
  593. - Hex returns text without word times, so Scrawl sends each stretch of speech separately and
  594. spreads its words across it by length. The transcript says `"source": "hex:<model>"` and
  595. `"estimated": true`; because those times are approximate, spoken cues ("now", "here") don't cut
  596. keyframes from a Hex transcript. Marks, clicks, laser and scene changes still do.
  597. - `scrawl hex status` shows whether Hex is connected, its ready models and what the next recording
  598. will use. `scrawl reprocess <bundle> --out … --transcriber hex|apple|command` (also
  599. `scrawl import`) picks an engine for one run.
  600. ### Your own model (command)
  601. Any other local model (whisper.cpp, MLX Whisper, faster-whisper, Parakeet, a Core ML model…) can
  602. transcribe through a small wrapper program. Scrawl runs it on the recording's audio and reads a
  603. JSON transcript from its stdout; it knows nothing about the model, adds no network requests and
  604. doesn't sandbox the program. It runs on macOS only for now; on Windows recordings keep the local
  605. engine. In `~/.config/scrawl/config.json` (Settings shows **Command** once it is set; editing it
  606. stays in the file):
  607. ```json
  608. {
  609. "transcriber": "command",
  610. "transcribeCommand": ["/Users/me/bin/whisper-scrawl", "--model", "large-v3-turbo", "{audio}"],
  611. "transcribeTimeoutSeconds": 600
  612. }
  613. ```
  614. - `transcribeCommand` is an argv array run directly: no shell, quoting, `~`, variables or globbing.
  615. `argv[0]` must be an absolute path. Every `{audio}` in the other arguments becomes the audio path;
  616. without one, the path is the last argument. Put your model's language in it too (`--language nl`):
  617. Settings' **Language** is Apple Speech's alone. These keys are read on their own, so a mistake in
  618. them never resets the rest of the config.
  619. - `transcribeTimeoutSeconds` is optional, a positive number of seconds: by default the limit is
  620. `max(300, 60 + 10 × audio seconds)`, wall clock from the start, model loading included.
  621. - The program gets the whole recording as one 16 kHz mono 16-bit WAV, `SCRAWL_TRANSCRIBE_VERSION=1`
  622. and `SCRAWL_AUDIO=<path>`, stdin `/dev/null`, and its own process group. Its stderr is discarded.
  623. It must not change the audio, daemonize or leave processes behind, and must cope with two runs at
  624. once. On timeout or more than 8 MiB of output Scrawl kills the group.
  625. - It exits 0 and prints exactly one JSON object:
  626. ```json
  627. {
  628. "version": 1,
  629. "model": "whisper-large-v3-turbo",
  630. "text": "Click here.",
  631. "segments": [{ "t0": 300, "t1": 980, "text": "Click here." }],
  632. "words": [{ "t0": 300, "t1": 610, "text": "Click" }, { "t0": 640, "t1": 980, "text": "here." }],
  633. "estimated": false
  634. }
  635. ```
  636. `version` (the integer 1) and `text` (empty means no speech) are required; the rest may be
  637. omitted or `null`, and unknown fields (or a `model` or `estimated` of another type) are ignored. Times are integer milliseconds from the first
  638. sample. `segments` and `words`, when given, cover the whole text. `estimated` is `false` only if
  639. the word times were measured (by the recognizer or an aligner); it defaults to `true`, and spoken
  640. cues only cut keyframes from measured words. Without `words`, Scrawl spreads the text over the
  641. segments (or the whole recording) and marks those times estimated, as it does for Hex.
  642. - Spans may overlap or be zero-length; Scrawl sorts them by `t0`. Ends up to 500 ms past the audio
  643. are clamped. A negative, fractional or reversed span, or one further past the end, makes Scrawl
  644. drop that array and keep the text, spreading it as above when `words` are dropped.
  645. - The transcript says `"source": "command:<file name of argv[0]>"`. Anything else going wrong (a
  646. malformed `transcribeCommand` or timeout, the program missing, a non-zero exit, a timeout, invalid
  647. JSON, another version) means Apple Speech transcribes that recording. Each problem is one line in
  648. `~/Library/Logs/Scrawl` (a malformed config only once), never with the program's output,
  649. arguments or what was said.
  650. ### Transcribing while recording (experiment)
  651. ```json
  652. { "experiments": { "liveTranscribe": true } }
  653. ```
  654. With this in `~/.config/scrawl/config.json`, a Mac recording transcribed by Apple Speech streams the
  655. microphone to `scrawl-helper transcribe-stream` as it records, so finishing only waits for the last
  656. words (tens of milliseconds) instead of transcribing the whole recording. The streamed samples are
  657. exactly the ones `audio.wav` holds, so word times are on the same clock. If the stream fails, the
  658. speech model isn't installed yet, or it takes too long, the recording is transcribed afterwards as
  659. usual; the log says which (`Live transcription: …`). Hex and iPhone recordings are unaffected.
  660. `scrawl reprocess <bundle> --out … --bench 1 --live-speed 1` replays a bundle's audio through it.
  661. ## Helper
  662. `scrawl-helper` is a small Swift executable built by `build.rs` from `helper/` and shipped inside
  663. `Scrawl.app` next to `scrawl`. It owns the Swift-only or unbound Apple frameworks: SpeechAnalyzer,
  664. CoreMediaIO/AVFoundation iPhone capture with the mirror window, AVFoundation movie decoding, and the
  665. SwiftUI welcome window (`onboarding`).
  666. Scrawl runs it as a subprocess (JSON on stdout; `stop`/`cancel` on stdin for the mirror). Run it
  667. directly to test a piece in isolation:
  668. ```sh
  669. target/debug/scrawl-helper devices
  670. target/debug/scrawl-helper permissions
  671. target/debug/scrawl-helper transcribe audio.wav
  672. target/debug/scrawl-helper frames recording.mov --out-dir /tmp/f --at 1000,2000 --max-edge 1600
  673. target/debug/scrawl-helper shrink recording.mov --out /tmp/small.mov --max-edge 1600
  674. target/debug/scrawl-helper export plan.json # the plan `scrawl export` writes (src/export)
  675. target/debug/scrawl-helper permission-panel --permission screen-recording --preview # the drag card, no grant polling
  676. ```
  677. `permission-panel --permission screen-recording|accessibility [--from X,Y,W,H]`
  678. (`--from` is the button's frame in global top-left points) prints JSON lines: `docked`, `dropped`,
  679. then `granted` or `cancelled` with a `reason` (`back`, `settings-closed`, `no-settings-window`,
  680. `parent`, `timeout`), and exits. On stdin it takes `cancel`, `back` and `granted`.
  681. `permission-panel --check P` prints `{"granted":…}` from a fresh process. `permission-panel --plan --settings
  682. X,Y,W,H --visible X,Y,W,H` prints where the card would dock (`below`, `inside`, `right`, `left` or
  683. `clamped`); `scripts/check-permission-panel-dock.sh` checks the cases.
  684. ## Windows
  685. Scrawl runs on Windows 10 (2004) and later as the same app: the same capsule, picker, overlay,
  686. review window, welcome tour and Settings, from the same GPUI code. Only the parts the operating
  687. system owns differ:
  688. | | macOS | Windows |
  689. |---|---|---|
  690. | Shortcut | `⌥⇧S`, `⌥⇧V` | `Alt+Shift+S`, `Alt+Shift+V` |
  691. | Menu | Menu bar item | Notification area icon (left or right click) |
  692. | Screen and window capture | ScreenCaptureKit | Windows.Graphics.Capture, H.264 through Media Foundation |
  693. | Clicks, keys, Shift ink | Mouse monitor, Accessibility key tap, modifier state | Low-level hooks and key state |
  694. | Ink overlay | Metal panel | Click-through GPUI window, left out of recordings |
  695. | Transcription | SpeechAnalyzer, or Hex | Parakeet on device (`transcribe.cpp`), Hex's model or Scrawl's own copy |
  696. | Movie decoding | `scrawl-helper` (AVFoundation) | Media Foundation, in process |
  697. | Paste | `⌘V` | `Ctrl+V` |
  698. | Settings and welcome tour | SwiftUI (`scrawl-helper`) | GPUI windows in process, same panes and protocol |
  699. | Recording capsule | Native AppKit + Metal pill (or GPUI) | GPUI capsule |
  700. | Launch at login | `SMAppService` | The user's `Run` key (Settings › Apps › Startup) |
  701. | Captures | `~/Library/Application Support/scrawl/captures` | `%LOCALAPPDATA%\scrawl\captures` |
  702. | Log | `~/Library/Logs/Scrawl/scrawl.log` | `%LOCALAPPDATA%\Scrawl\Logs\scrawl.log` |
  703. - **Permissions:** none, apart from the microphone switch in Settings › Privacy & security, which
  704. the welcome tour and Scrawl's Settings › Permissions check. Screen capture, input hooks and
  705. pasting need no grant on Windows.
  706. - **Shortcuts:** both are recorded in Settings › General like on macOS, as `Ctrl`, `Alt`, `Win` and
  707. `Shift` combinations. Scrawl turns down `Ctrl` with one key (apps use those), `Win` with a key
  708. (Windows uses those), and `Alt+Tab`, `Alt+F4` and `Alt+Space`.
  709. - **Transcription:** Parakeet runs on device with measured word timings, so spoken cues ("look
  710. here") cut keyframes as on macOS. If [Hex](https://github.com/anomalyco/hex) has installed a
  711. model (`%APPDATA%\hex\models`, or `%APPDATA%\voice-control\models` from older Hex), Scrawl
  712. uses it. Otherwise the first launch downloads Hex's Parakeet Unified English (730 MB, checked
  713. against its SHA-256) into `%LOCALAPPDATA%\Scrawl\models` in the background; Settings › Recording
  714. shows the progress, and captures made before it finishes get their transcript once it lands.
  715. - **Command-line tool:** **Install** in Settings › Agents writes a `scrawl.cmd` shim into
  716. `%LOCALAPPDATA%\Microsoft\WindowsApps`, which is on every user's PATH, so agents and terminals
  717. can run `scrawl`.
  718. - **agentContext:** what clicks and marks land on comes from UI Automation, as it does from the
  719. Accessibility tree on macOS.
  720. - **Not on Windows:** the cabled iPhone (it needs AVFoundation's screen capture devices), video
  721. export, Sparkle updates (**Check for Updates…** opens the releases page) and the dev-site tab URL
  722. (`/sdk.js` registrations still route).
  723. - **Releases:** a `v*` tag builds `Scrawl-<version>-windows-x64.zip` on GitHub Actions
  724. (`.github/workflows/release-windows.yml`) and attaches it to the release. It isn't code-signed
  725. yet; signing and automatic updates are planned.
  726. Build with the MSVC toolchain, CMake and the Windows SDK (Visual Studio Build Tools with the C++
  727. workload). Release builds also need GPUI's shader compiler: set `GPUI_FXC_PATH` to the SDK's
  728. `fxc.exe` (for example `C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\fxc.exe`).
  729. ```powershell
  730. cargo build --release
  731. .\target\release\scrawl.exe app
  732. ```
  733. ## Testing
  734. ```sh
  735. taskpolicy -c utility cargo test -j 3 --offline -- --test-threads=2
  736. ./scripts/install-app.sh
  737. ```
  738. On Windows, `cargo test`; then run `scrawl.exe app` and record with `Alt+Shift+S`.
  739. `SCRAWL_SHOW_OWN_WINDOWS=1` keeps Scrawl's own windows (capsule, overlay, picker) in screenshots,
  740. which normally leave them out.
  741. - **iPhone:** connect a phone, press `⌥⇧S`, click the iPhone chip, talk, draw over the mirror, press M,
  742. Finish. Inspect the bundle's `summary.md`, `scrawl.json` (`source`, `device`), contact sheet, and
  743. frames.
  744. - **ScrawlKit:** `scrawl import` on a recording and its timeline (or hand-write a timeline for any
  745. phone video), then check the ripples and trails on frames and the app-event lines in the summary.
  746. `swift test` runs the outbox tests on the Mac against a `URLProtocol` stub of the ingest endpoint
  747. (persisting, retry classification, resends, deletion only on 2xx, and what the strip says through
  748. slow uploads, no network, and retries).
  749. - **Transcription:** `scrawl-helper transcribe <wav or movie>` prints words with timings;
  750. `scrawl reprocess <bundle> --out …` reruns transcription and keyframe selection on a real bundle.
  751. ## License
  752. MIT. See [LICENSE](LICENSE).
  753. ### THIRD_PARTY
  754. `helper/PermissionPanel.swift` adapts code and tuning from two MIT-licensed projects:
  755. - [AskForPermission](https://github.com/riko2chen/AskForPermission), Copyright (c) 2026 Riko: the
  756. flight along a quadratic Bézier, waiting for the System Settings window to settle, following it
  757. through the window server, and the race that resolves a drop.
  758. - [PermissionFlow](https://github.com/jaywcjlove/PermissionFlow), Copyright (c) 2026 小弟调调: the
  759. Privacy pane deep links and macOS 27's "Device Control and Data Access" name for Accessibility.
  760. Both are under the MIT License: permission is hereby granted, free of charge, to any person
  761. obtaining a copy of this software and associated documentation files (the "Software"), to deal in
  762. the Software without restriction, including without limitation the rights to use, copy, modify,
  763. merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to
  764. whom the Software is furnished to do so, subject to the following conditions: The above copyright
  765. notice and this permission notice shall be included in all copies or substantial portions of the
  766. Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
  767. INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
  768. NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES
  769. OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
  770. CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.