Skip to content
Back to search

blurt

SkillGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/AGIHunt/blurt with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/AGIHunt/blurt'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/agihunt--blurt/badge.svg)](/listings/agihunt--blurt)

HTML

<a href="/listings/agihunt--blurt"><img src="/api/listings/agihunt--blurt/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

You need macOS 13 or later. Dictation and the recommended speech models need Apple silicon.

**[Download Blurt for Mac](https://github.com/AGIHunt/blurt/releases/latest/download/Blurt.dmg)** (signed and notarized), open the disk image and drag Blurt into Applications. Updates arrive inside the app.

Or let your agent install it through the Blurt skill:

npx skills add AGIHunt/blurt

<sub>In Claude Code you can also run <code>/plugin marketplace add AGIHunt/blurt</code>, or paste this repository's URL to your agent and ask it to install the skill.</sub>

Then tell your agent **"start blurt"**. The first run checks your machine, picks a speech model and builds the Blurt app into `~/Applications`. It needs the Xcode Command Line Tools and [uv](https://docs.astral.sh/uv/); the agent offers to install what's missing.

**First minute.** The setup assistant asks for Microphone and Accessibility (and Input Monitoring if you use another key than `Fn`), then downloads the speech models: about 5.9 GB on a Mac with 32 GB of memory, 4.4 GB on 16–24 GB. Nothing is downloaded until you click Download. Then press `Fn` in any text field. Screen Recording is asked for the first time you snap or record.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 7 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "blurt",
  3. "owner": { "name": "AGI Hunt", "url": "https://github.com/AGIHunt" },
  4. "plugins": [
  5. { "name": "blurt", "source": "./", "description": "Screen + voice → bugs, ideas, notes and todos (review page; Markdown/CSV and hand-off .zip; your agent files them anywhere). 口喷鸡" }
  6. ]
  7. }
README.md · README · 261 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <p align="center"><picture><source media="(prefers-color-scheme: dark)" srcset="assets/peep-readme-dark.png"><img src="assets/peep-readme.png" width="132" alt="Blurt"></picture></p>
  2. <h1 align="center">Blurt</h1>
  3. <p align="center"><b>The keyboard for the agent era.</b><br>
  4. The fastest way to work with your agents, by text, voice, screenshot or screen recording. Runs locally. Free and open source.<br>
  5. <a href="https://blurt.work/en/">Website</a> · <a href="README.zh-CN.md">中文</a> · <a href="docs/guide.md">User guide</a> · <a href="#community">WeChat group</a></p>
  6. <p align="center"><a href="https://blurt.work/en/#film"><img src="assets/film-poster.jpg" width="720" alt="Watch the 2-minute film"></a></p>
  7. <p align="center"><sub>Watch the 2-minute film, sound on · <a href="https://github.com/AGIHunt/blurt/releases/latest/download/Blurt.dmg">Download for Mac</a></sub></p>
  8. ---
  9. Blurt is a menu-bar app for the Mac, with [test builds for Windows and Linux](https://github.com/AGIHunt/blurt/releases). It works next to the agents you already use, such as Claude Code, Codex, Cursor,
  10. Claude and ChatGPT, and handles the part before they start: getting what you mean into the agent's input box. It has
  11. no agent of its own and doesn't replace yours.
  12. ## What it does
  13. **Dictation (`Fn`).** Tap `Fn` in any app, talk, and tap it again. The words are typed where your cursor is. Finish
  14. with a double tap instead and it's sent too (Return). It works the moment Blurt is installed, with macOS's own speech
  15. recognition (macOS 26) and nothing to download. Open-source local models are an optional upgrade: more accurate, they
  16. learn your words and need no internet; after a minute of talking the text is in about 0.4 s. A local model removes
  17. fillers and stutters and adds punctuation; it doesn't rewrite or summarize. The words show under the notch as you
  18. talk. Double-tap `⇧` while talking to switch to Structured (a list, when you listed things) and again to switch back;
  19. double-tap `⌘` to translate (speak Chinese, type English, or the other way round); `⌥L` reformats the last dictation. Holding `Fn`
  20. while you talk works too.
  21. **Snap (`⌥A`) and Voice Note (`⌥V`).** Drag over part of the screen and say what's wrong; the note opens right
  22. beside the selection. Draw on it as you talk: drag for a freehand line, `⇧` for a box, `⌥` for an ellipse, `⌘Z` to
  23. undo. The snap becomes a card under the notch. Drag the card into Claude, ChatGPT, Slack or a terminal running Claude Code or Codex: the picture and
  24. your note go in together. Or click it to copy, then `⌘V`. A voice note is the same, with your words and no picture.
  25. **Copy text from screen (`⌥C`).** Drag over any text you can't select: a screenshot, a video, a locked page, a
  26. terminal. Let go and it's on the clipboard in about 0.2 s, laid out as if you had copied it: the half line the edge
  27. cut is left out, wrapped list items are one line again, tables keep their tabs and code its indentation. It reads with
  28. macOS's own recognizer, so there's nothing to download. Optionally a second reading by a small local OCR model
  29. (720 MB) or your own cloud model fixes misread symbols in code and paths.
  30. **Record (`⌥R`).** Record your screen while you talk through a product, a page or an idea. Your agent turns
  31. the recording into items (issues, ideas, notes, tasks), each with a screenshot, the spot marked, steps and the code
  32. that is probably involved. It's transcribed while you record, so it's done about a minute after you stop; one
  33. 5-minute walkthrough became 52 items, each with what was said and when. You go through them with the keyboard, then
  34. copy an instruction for your agent (`⌘↩`), or pack them into one `.zip` for a teammate.
  35. **Blurt on your phone.** When you sit back from a big screen, your phone becomes the computer's trackpad, keys and
  36. microphone, and can show its screen. It works with a Mac, Windows or Linux computer, from an iPhone, iPad (including the
  37. iPhone Duo) or Android phone or tablet. The talk key is Blurt's dictation key, and the keys follow the computer you're
  38. driving (⌘ on a Mac, Ctrl and Win elsewhere). With several displays the picture follows the pointer, or stays on the one
  39. you pick. Away from home it goes through a relay you run yourself; everything on the way is end-to-end encrypted. Pair
  40. in Settings → Phone on any computer. The phone apps aren't in the stores yet: build them
  41. yourself ([iPhone and iPad](apps/ios/README.md#build-it-yourself), [Android](apps/android/README.md#build-it-yourself)).
  42. Snaps, voice notes and recordings go into the current project (a folder you open in Blurt). Without one, recordings
  43. go to Unassigned and snaps stay as cards until you pick a project. Every shortcut can be changed in Settings.
  44. ## Install
  45. You need macOS 13 or later. The local models need Apple silicon; an Intel Mac dictates with macOS 26's own speech
  46. recognition or a cloud key. Windows and Linux: see the [FAQ](#faq).
  47. **[Download Blurt for Mac](https://github.com/AGIHunt/blurt/releases/latest/download/Blurt.dmg)** (signed and
  48. notarized), open the disk image and drag Blurt into Applications. Updates arrive inside the app. There is nothing
  49. else to install: the app fetches the Python runtime its tools use (about 150 MB) by itself the first time.
  50. Or give this to your agent (Claude Code, Codex…), or paste it into a terminal:
  51. ```bash
  52. curl -fsSL https://blurt.work/install.sh | sh
  53. ```
  54. It downloads the latest release, checks that it is signed and notarized, puts Blurt in Applications and opens it.
  55. **Use it from your agent.** In Blurt, *Settings → Agents → Connect…* adds the Blurt skill and MCP server to Claude
  56. Code, Codex and the other agents it finds (Gemini CLI, Kimi Code, OpenCode, Grok, CodeBuddy Code, WorkBuddy,
  57. Cursor). Start a new session of the agent, then tell it **"start blurt"** (`/blurt` in Claude Code, `$blurt` in
  58. Codex), or ask it about your items. Your agent can also do the connecting: [llms.txt](https://blurt.work/llms.txt) has
  59. the steps.
  60. <sub>Without the app, from source: <code>npx skills add AGIHunt/blurt -g</code>, or in Claude Code <code>/plugin
  61. marketplace add AGIHunt/blurt</code> and then <code>/plugin install blurt@blurt</code>. Then "start blurt". The first run
  62. builds Blurt into <code>~/Applications</code>; it needs the Xcode Command Line Tools and
  63. <a href="https://docs.astral.sh/uv/">uv</a>, and the agent offers to install what's missing.</sub>
  64. **First minute.** The setup assistant asks for Microphone and Accessibility (and Input Monitoring if you use another
  65. key than `Fn`). Then press `Fn` in any text field: on macOS 26 dictation already works with macOS's own recognizer,
  66. while your Mac's own local model downloads in the background, once (1.6–5.9 GB depending on the Mac). On macOS 13–15
  67. you download a model first or add a cloud key. One optional upgrade: an AI-service key (fixes misheard names and
  68. punctuation, translates with the cloud model, and gives copied text a cloud second reading). Screen Recording is asked
  69. for the first time you snap or record.
  70. ## How it works with your agents
  71. Blurt gets your words, pictures and recordings into an agent. The agent does the work.
  72. | | What it does |
  73. |---|---|
  74. | **Dictation and cards** | Text and pictures go into any chat box or terminal, so they work with every agent. |
  75. | **Skill** | The `blurt` skill lets your agent record, transcribe, write items, open the review page and export. |
  76. | **MCP server** | `blurt` gives agents `search_items`, `get_item` (with screenshots), `get_transcript`, `update_item`, `mark_fixed` and `bundle_items`. *Settings → Agents → Connect…* installs the skill and the MCP server into an agent, after showing what it changes. |
  77. | **Copy Instruction for Agent** (`⌘↩`) | One line on the clipboard to paste into the agent you use (Claude Code, Codex, WorkBuddy, Cursor…), pointing to a hand-off note with every detail and screenshot. The items show as *Handed off*. Blurt doesn't start an agent for this. |
  78. | **Hand-off bundle** | One `.zip` with a summary, every item and its screenshots, for a person rather than an agent. |
  79. Agents report back through the MCP server: an item moves to *In progress*, then *To verify*. "Fixed" from an agent
  80. always means *to verify* until you mark it done.
  81. Any agent with the Blurt skill can organize recordings into items, GUI apps like WorkBuddy, Cursor and CodeBuddy
  82. included: say "process my Blurt recordings" in its chat. The app can also do it in the background as soon as you stop,
  83. with the first of these that is installed and signed in: Claude Code, Codex, Kimi Code, OpenCode, Gemini CLI, Grok.
  84. You can change the order in *Settings → Agents*.
  85. **No connectors, by design.** Your agent already has tools for Slack, Linear, GitHub, Notion, Jira and Feishu/Lark.
  86. Blurt writes files (Markdown, CSV and the hand-off bundle) and leaves delivery to the agent. To send items somewhere,
  87. ask it, for example: "Bundle the checkout issues and post them to #web in Slack."
  88. ## How Blurt compares
  89. Checked on **2026-10-25**. Prices and policies change.
  90. Blurt's numbers come from its own tests on one machine, an Apple M5 Max with 128 GB, with speech generated by macOS
  91. `say`. Synthetic speech is cleaner than a real microphone, so error rates are optimistic; smaller Macs, real
  92. microphones and accents were not measured. Competitor numbers come from other people and methods, some from a
  93. competing vendor, and are not directly comparable.
  94. | | **Blurt** | Typeless | Wispr Flow | Superwhisper | Aqua Voice |
  95. |---|---|---|---|---|---|
  96. | Price | Free, MIT licence, no word limit | Free with a weekly limit; Pro $12/month yearly, $30 monthly [1] | Free 2,000 words/week on desktop; Pro $15/month, $12/month yearly [2] | Free with local models; Pro $8.49/month [3] | Free 1,000 words; Pro $12/month, $10/month yearly [4] |
  97. | Where speech is processed | On your Mac | Cloud [5] | Cloud [6] | Local or cloud, your choice [7] | Cloud [8] |
  98. | End of speech → text | 0.29–0.54 s after 3–60 s of speech | about 3.0 s [9] | about 1.5 s [9] | not published | "about 450ms" (vendor) [8] |
  99. | English accuracy | WER 1.9 % (speech model alone) | WER about 7.9 % [9] | WER about 3.9 % [9] | depends on the model; not published | OpenASR average WER 6.24 (vendor) [10] |
  100. | Chinese accuracy | CER 1.3 % (speech model alone) | not published | not published | not published | not published |
  101. | Platforms | macOS; Windows and Linux test builds | macOS, Windows, Linux, iOS, Android | Mac, Windows, iOS, Android | Mac, Windows, iOS, Android | Mac, Windows, iOS, Android |
  102. | Offline | Dictation needs no network once the models are downloaded (not yet tested with the network off) | No | No | Yes, with local models | No |
  103. | Snap and voice-note cards for agents | Yes | No | No | No | No |
  104. | Screen recording → items with screenshots and code pointers | Yes | No | No | No | No |
  105. | Hand-off to coding agents, status back through MCP | Yes | No | No | No | Lists "MCP/CLI" on its free plan; not examined |
  106. <sub>Blurt sources: latency `docs/design/dictation.md` (median of 3 takes, warm models); accuracy
  107. `docs/design/asr-benchmark.md` (Qwen3-ASR 1.7B, 7 sentences × 2 voices per language). Full detail and the numbers we
  108. could not verify: [docs/positioning/facts.md](docs/positioning/facts.md).
  109. [1] [typeless.com/pricing](https://www.typeless.com/pricing) ·
  110. [2] [wisprflow.ai/pricing](https://wisprflow.ai/pricing) ·
  111. [3] [superwhisper.com](https://superwhisper.com/) ·
  112. [4] [aquavoice.com/pricing](https://aquavoice.com/pricing) ·
  113. [5] [Typeless data controls](https://typeless.com/data-controls), updated 2026-10-13 ·
  114. [6] [Wispr Flow data controls](https://wisprflow.ai/data-controls), updated 2026-08-18 ·
  115. [7] [Superwhisper docs](https://superwhisper.com/docs/llms-full.txt) ·
  116. [8] [Aqua blog, 2026-08-17](https://aquavoice.com/blog/wispr-flow-alternatives) ·
  117. [9] [VoiceDash](https://voicedash.ai/wispr-flow-vs-typeless/), a competing vendor; the same article also gives about 20 % for Typeless ·
  118. [10] [Aqua blog, 2025-10-08](https://aquavoice.com/blog/avalon-openasr-leaderboard).</sub>
  119. ## What users complain about in other dictation apps, and what Blurt does
  120. From low-rated App Store reviews of Typeless and Wispr Flow (US store, read on 2026-10-24) and posts on r/macapps,
  121. V2EX and Product Hunt. Counts and quotes are in [facts.md §3](docs/positioning/facts.md#3-what-users-complain-about-and-what-blurt-does).
  122. | Complaint | What Blurt does |
  123. |---|---|
  124. | It rewrites what I said | The default style only removes fillers and adds punctuation. If polish drops or adds words, the raw transcript is typed instead. Lists happen only when you double-tap `⇧`. |
  125. | Words get lost or cut off | The words show while you talk. Every dictation stays in Dictation History for 30 days. The microphone opens on the key press, so the first words aren't cut. |
  126. | Word limits and subscriptions | Free, no word limit, no account. |
  127. | Names and mixed languages come out wrong | Words, a list of names and terms the speech model listens for, learned from your fixes (as hints, never as replacements), plus the text around the cursor. |
  128. | The microphone stays on; privacy | Speech is processed on your Mac. The microphone is on only while you dictate. No screenshots are taken for dictation. |
  129. | Slow | About 0.3–0.5 s from releasing the key to the text on the test Mac. |
  130. | The hotkey clashes with Fn | Blurt keeps `Fn` for dictation and tells you when macOS or another dictation app also reacts to it. Right Option or either Option can be used instead. |
  131. | Long dictations collapse into "[Pasted N lines]" in Claude Code | Terminals and code editors get the text on one line. |
  132. Not built yet: keeping a dictation's audio until the paste is confirmed, and a setting that limits dictation to your
  133. languages.
  134. ## Privacy
  135. - Speech recognition and polish run on your Mac. Dictation History keeps each dictation's text, its audio and the text
  136. around the cursor for 30 days (you can change how long, or stop keeping the audio).
  137. - When you start talking, Blurt reads about 500 characters around the cursor so names come out right. It never reads
  138. password fields. You can turn this off.
  139. - Snaps, voice notes, recordings and items stay in your folders: `~/Blurt`, or `.blurt/` inside a project, which Blurt
  140. keeps out of git.
  141. - When your agent organizes a recording, the transcript text and a few screenshots go to that agent's provider, like
  142. anything else you give it.
  143. - No account. Network use is limited to downloads of models and the Python runtime, update checks, a cloud speech key if you add one (Groq, OpenAI
  144. or DashScope, set in Settings → Models), cloud polish if you turn it on (the dictated text, never the recording, goes
  145. to DeepSeek or the API you set up), translation with the cloud model if you turn it on (the text), the cloud second
  146. reading in Screen Text if you choose it (the picture of the area you dragged over), what your agent sends to its provider, your phone's connection away from home (through your own relay, end-to-end
  147. encrypted), and Send Feedback: when you click it on the
  148. Dictation page, the dictations you check (their audio, every step's text, the text around the cursor if you leave it
  149. on, your Mac's model and settings) go to Blurt's private storage on Cloudflare, readable only by the Blurt developers
  150. and deleted within 14 days.
  151. *Settings → Privacy → Speech recognition: On Your Mac Only* keeps speech on your Mac even when a key is set.
  152. ## FAQ
  153. **Does Blurt replace my agent?** No. It feeds the agents you already use: Claude Code, Codex, Cursor, Claude, ChatGPT
  154. or any chat box.
  155. **What does it cost?** Nothing. Turning a recording into items uses your own agent's tokens. The video is never sent
  156. to the model: speech is transcribed locally, the agent reads the text and looks at a few frames. Three real sessions
  157. (Claude Code, Opus 5.5, a production web app):
  158. | Recording | Speech | Items | Transcription (local) | New input / output tokens | At API prices |
  159. |---|---|---|---|---|---|
  160. | 7.0 min | 2.8 min | 9 | 4 s | ~142k / ~19k | ~$2 |
  161. | 14.3 min | 12.3 min | 23 | 12 s | ~138k / ~27k | ~$2.6 |
  162. | 20.0 min | 17.7 min | 34 | 15 s | ~122k / ~27k | ~$2.1 |
  163. <sub>Cache reads of the ongoing conversation (about 3–5M tokens at 1/20 of the input price) are included above.</sub>
  164. **How long until I can review a recording?** About a minute after you click Finish; screenshots fill in while you
  165. review. A real 5-minute, 12-item session was ready in 51 s.
  166. **Which models does it use, and how big are they?**
  167. | Use | Model | Download |
  168. |---|---|---|
  169. | Speech on Macs with 16 GB and up | Qwen3-ASR 1.7B, 8-bit, MLX | 2.5 GB |
  170. | Speech on 8 GB Macs and base M1 / M2, for Chinese, English and Japanese speakers | Fun-ASR-Nano | 1.6 GB |
  171. | Speech there for other languages | Qwen3-ASR 0.6B, 8-bit, MLX | 1.0 GB |
  172. | Polish, 16 GB and up (8 GB Macs don't polish locally) | Gemma 4 E2B, 4-bit | 3.4 GB |
  173. | Voice notes on CPU-only machines; Japanese, Korean, Cantonese | SenseVoice Small | 228 MB |
  174. | Recordings in other languages | Whisper large-v3-turbo, MLX | 1.5 GB |
  175. In use, the 1.7B and Gemma take about 3.1 and 2.3 GB of memory. Idle models unload after 10 minutes, and sooner when
  176. macOS runs short of memory. On macOS 26, macOS's own recognizer dictates until your Mac's model is downloaded.
  177. **Which languages?** Dictation was tested in Chinese, English and mixed Chinese–English. Recordings: SenseVoice covers
  178. Chinese, English, Japanese, Korean and Cantonese; Whisper covers 99 languages. Items come back in the language you
  179. spoke. The app is in English and Simplified Chinese.
  180. **Does it run on Windows and Linux?** Yes, as test builds on [GitHub Releases](https://github.com/AGIHunt/blurt/releases): Windows 10 (1903 or later)
  181. and 11, and Linux (Ubuntu 24.04 or later, Debian 13 or later; tested on Ubuntu with GNOME on Wayland). They dictate
  182. (Right Alt on Windows; on Linux a shortcut you pick the first time), take voice notes and record, with speech
  183. recognized locally on the CPU (about 1 GB to download). Windows also has snaps and Screen Text; on Linux
  184. they aren't done yet. Polish needs a cloud key, the notch is a small capsule, and you update by downloading the next
  185. build. The Windows installer isn't signed yet: SmartScreen asks first (More info → Run anyway).
  186. Without the app, the skill's recording scripts (recording, local transcription, the review page, exports) also run
  187. there, Linux with an X11 desktop: on Windows `winget install astral-sh.uv` and
  188. `winget install Gyan.FFmpeg`; on Ubuntu/Debian `sudo apt install ffmpeg libportaudio2 python3-tk` plus
  189. [uv](https://docs.astral.sh/uv/). `doctor.py` tells you what's missing.
  190. **What permissions does it need?** Microphone; Accessibility, to paste and to hear `Fn`; Input Monitoring, for other
  191. keys; Screen Recording, for snaps and recordings.
  192. **Can teammates without the code use it?** Yes. They record with the app and send you the video; your agent processes
  193. it with the code at hand. Items keep the name of the person who recorded them.
  194. **Can I change the shortcuts?** Yes, in *Settings → Shortcuts*. Blurt warns when a shortcut is taken by another
  195. action, another app or macOS. The same setting works in `~/.blurt/config.json`, also on Windows:
  196. ```json
  197. {"record": {"hotkeys": {"record": "ctrl+opt+R", "snap": "opt+shift+A", "pause": "off"}, "draw_key": "control"}}
  198. ```
  199. **Where is the full manual?** The [user guide](docs/guide.md) covers every feature, and
  200. [SKILL.md](skills/blurt/SKILL.md) is what your agent reads.
  201. ## Community
  202. Questions, feedback and new releases go to the Blurt users' group on WeChat. To join, reply 口喷 to the WeChat official
  203. account **AGI Hunt**.
  204. Bugs and feature requests are also welcome as [GitHub issues](https://github.com/AGIHunt/blurt/issues).
  205. <p align="center"><sub>MIT · made by <a href="https://github.com/AGIHunt">AGI Hunt</a></sub></p>
site/install.sh · Bundled script · 71 lines · 1 finding marked
  1. #!/bin/sh
  2. # Install Blurt for Mac: download the latest release, put Blurt.app in Applications and open it.
  3. #
  4. # curl -fsSL https://blurt.work/install.sh | sh
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 4 on GitHub
  5. #
  6. # Made to be run by you or by your agent (Claude Code, Codex…). It replaces an older Blurt.app where it finds one
  7. # (/Applications, else ~/Applications) and checks that the download is signed and notarized before copying it. The
  8. # permissions (microphone, Accessibility) can only be turned on by you: the setup window that opens walks you through
  9. # them. Served from the website (site/install.sh in github.com/AGIHunt/blurt).
  10. #
  11. # BLURT_INSTALL_DIR=/some/folder sh install.sh installs there instead, and quits or opens nothing (tests)
  12. set -eu
  13. URL=https://github.com/AGIHunt/blurt/releases/latest/download/Blurt.dmg
  14. if [ "$(uname -s)" != Darwin ]; then
  15. echo "This installer is for macOS. Windows and Linux test builds: https://github.com/AGIHunt/blurt/releases" >&2
  16. exit 1
  17. fi
  18. major=$(sw_vers -productVersion | cut -d. -f1)
  19. if [ "$major" -lt 13 ]; then
  20. echo "Blurt needs macOS 13 or later (this Mac has $(sw_vers -productVersion))." >&2
  21. exit 1
  22. fi
  23. TMP=$(mktemp -d "${TMPDIR:-/tmp}/blurt-install.XXXXXX")
  24. MNT=$TMP/mnt
  25. cleanup() {
  26. hdiutil detach -quiet "$MNT" 2>/dev/null || true
  27. rm -rf "$TMP"
  28. }
  29. trap cleanup EXIT INT TERM
  30. echo "Downloading Blurt (about 20 MB)..."
  31. curl -fsSL -o "$TMP/Blurt.dmg" "$URL"
  32. mkdir "$MNT"
  33. hdiutil attach -quiet -nobrowse -readonly -mountpoint "$MNT" "$TMP/Blurt.dmg"
  34. NEW=$MNT/Blurt.app
  35. if ! spctl --assess --type execute "$NEW" 2>/dev/null; then
  36. echo "The download isn't a signed and notarized Blurt.app; nothing was installed." >&2
  37. exit 1
  38. fi
  39. # where it goes: over an existing copy, else /Applications (~/Applications when that isn't writable);
  40. # BLURT_INSTALL_DIR puts it elsewhere and doesn't quit or open anything (tests)
  41. if [ -n "${BLURT_INSTALL_DIR:-}" ]; then DEST=$BLURT_INSTALL_DIR
  42. elif [ -d /Applications/Blurt.app ]; then DEST=/Applications
  43. elif [ -d "$HOME/Applications/Blurt.app" ]; then DEST=$HOME/Applications
  44. elif [ -w /Applications ]; then DEST=/Applications
  45. else DEST=$HOME/Applications
  46. fi
  47. mkdir -p "$DEST"
  48. if [ -z "${BLURT_INSTALL_DIR:-}" ] && pgrep -xq blurt-recorder; then
  49. osascript -e 'tell application id "dev.blurt.recorder" to quit' >/dev/null 2>&1 || true
  50. i=0
  51. while pgrep -xq blurt-recorder && [ $i -lt 20 ]; do sleep 0.5; i=$((i + 1)); done
  52. fi
  53. rm -rf "$DEST/Blurt.app"
  54. ditto "$NEW" "$DEST/Blurt.app"
  55. version=$(/usr/libexec/PlistBuddy -c "Print :CFBundleShortVersionString" "$DEST/Blurt.app/Contents/Info.plist" 2>/dev/null || echo "")
  56. if [ -n "${BLURT_INSTALL_DIR:-}" ]; then
  57. echo "Blurt $version is in $DEST."
  58. exit 0
  59. fi
  60. open "$DEST/Blurt.app"
  61. echo "Blurt $version is in $DEST and open."
  62. echo "Finish in the setup window: turn on the microphone and Accessibility. Nothing else to download."
  63. echo "To use Blurt from an agent (\"start blurt\"), connect it in Blurt's Settings > Agents, then start a new session"
  64. echo "of that agent. Agents can connect themselves: the steps are at https://blurt.work/llms.txt (Install)."
skills/blurt/scripts/transcribe.py · Bundled script · 4,751 lines
  1. # /// script
  2. # requires-python = ">=3.10"
  3. # dependencies = ["numpy==2.3.4; python_version >= '3.11'", "numpy==2.2.6; python_version < '3.11'", "sherpa-onnx==1.13.8", "truststore==0.10.5", "opencc-python-reimplemented==0.1.7"]
  4. # ///
  5. """Speech-to-text with timestamps, adapted to the machine it runs on (docs/design/next-round.md K9).
  6. Always VAD-first: silence is dropped, speech is chunked, and every chunk keeps its offset in the original
  7. video, so any backend (even ones without timestamps) works.
  8. transcribe.py run <video|audio> [--out DIR] transcribe; writes transcript.json + transcript.txt
  9. instant when live.py already transcribed the recording
  10. while it was made (<session>/live/); --force redoes it
  11. transcribe.py run <session dir> a shots session (snap mode): transcribe each voice note
  12. shots/NNN.wav into its shots.jsonl record ("said")
  13. transcribe.py note <wav> [--language auto] [--dictation] [--vocab] [--no-server] [--no-download]
  14. one voice note, fast path → {"ok","text","language","ms","model"}
  15. (uses a running `serve` when there is one). --dictation: the
  16. dictation model (record.dictation.asr_model); --vocab: bias it
  17. toward the words (dictation_vocab.py) where the model can be
  18. biased; they are hints, the text is as heard → + "asr_text","biased"
  19. transcribe.py serve [--preload] [--model-id ID | --purpose dictation] [--no-download] [--idle 600] [--parent PID] keep the
  20. voice-note (or dictation) model warm; ends after --idle s unused
  21. (the memory goes back), with the --parent app, or on "exit".
  22. A request with "interim": true (live words while talking) is
  23. stopped as soon as another request waits → code "interrupted"
  24. (a quick model, SenseVoice, decodes it whole)
  25. transcribe.py dictation-models the models dictation can use here: selected, recommended, ready (runs with nothing downloaded, VAD included), missing
  26. transcribe.py probe hardware → tier (pro / standard / light / minimal)
  27. transcribe.py recommend [--languages zh,en] [--purpose batch|live|note]
  28. transcribe.py auto [--apply] pick a model per purpose for this machine (asr.models);
  29. lists what still has to be downloaded
  30. transcribe.py models the registry (asr_models.json) + installed, measured, selected
  31. transcribe.py setup --model ID [--use all|batch,live,note] [--parent PID] download a model (progress on
  32. stderr): one at a time per model (a second one waits), a
  33. partial file continued (HTTP Range), a mirror folder installed
  34. only once every file of its manifest is in; ends with --parent
  35. transcribe.py bench [AUDIO] [--model ID] [--languages zh,en] real-time factor, peak memory, CPU load and (on the
  36. built-in `say` samples) the error rate per language
  37. → $BLURT_HOME/asr-bench.json
  38. transcribe.py backends backends and whether they are ready (legacy)
  39. transcribe.py voiceprint prepare|enroll WAV|check WAV|status|clear only my voice (docs/design/voiceprint.md):
  40. the speaker model, the user's voice profile, a piece's verdict
  41. transcribe.py check-cloud [--model-id qwen3-asr-flash|volcengine-seedasr|openai-compatible] Settings' Test: one
  42. small spoken clip (macOS `say`) to the cloud service →
  43. {"ok","ms","text"} or {"ok": false, "code"}; a key / base_url /
  44. model not saved yet comes as JSON on stdin ({"key": …};
  45. Volcengine: {"app_id": …, "token": …}), never as an argument
  46. Which model: --model ID / --backend B win; else config asr (auto: per purpose from `recommend`, only what is
  47. installed; asr.auto false: asr.models / the legacy asr.backend + asr.model). Resource caps (asr.limits): threads,
  48. one batch job at a time ($BLURT_HOME/run/asr.lock, the rest queue), low priority for batch work, an MLX memory cap.
  49. Chinese comes back in the user's script: for zh-Hans users (or asr.chinese_script "hans") traditional characters
  50. become simplified (OpenCC t2s, `to_user_script`); zh-Hant users keep what the model wrote.
  51. Dependencies are pinned (here and in every `--with`, PACKAGE_PINS): a new release is taken on purpose, per release.
  52. Backends (models: asr_models.json; measured: docs/design/asr-benchmark.md)
  53. sensevoice local CPU, sherpa-onnx + SenseVoice-Small int8 (~230 MB). zh/en/ja/ko/yue.
  54. qwen3-asr-onnx local CPU (any OS), sherpa-onnx + Qwen3-ASR 0.6B int8, 30 languages. Hotwords per stream.
  55. fun-asr-onnx local CPU (any OS), sherpa-onnx + Fun-ASR-Nano int8, zh / en / ja. Hotwords; pieces ≤ 20 s.
  56. qwen3-asr-mlx local, Apple Silicon (MLX), Qwen3-ASR 0.6B 8-bit, 30 languages, `context` biasing per request
  57. (dictation's default: docs/design/dictation.md §Recognition model). needs mlx-qwen3-asr
  58. fun-asr-mlx local, Apple Silicon (MLX), Fun-ASR-Nano 0.8B, zh / en / ja, the Words as hotwords
  59. (dictation on 8 GB and slower Macs: docs/design/dictation.md §Defaults by Mac). needs mlx-audio
  60. mlx-whisper local, Apple Silicon (MLX), Whisper large-v3-turbo (fp16 / 4-bit). needs mlx-whisper
  61. faster-whisper local, NVIDIA CUDA (or CPU), Whisper large-v3-turbo (CTranslate2). needs faster-whisper
  62. openai any OpenAI-compatible /audio/transcriptions API (OpenAI, Groq, SiliconFlow, ...)
  63. env: BLURT_ASR_API_KEY (or OPENAI_API_KEY / GROQ_API_KEY), BLURT_ASR_BASE_URL, BLURT_ASR_MODEL
  64. dashscope Alibaba Qwen3-ASR (qwen3-asr-flash) via DashScope OpenAI-compatible chat API. env: DASHSCOPE_API_KEY
  65. volcengine Volcengine 豆包 Seed-ASR 2.0 (bigmodel_nostream), streamed over a websocket while a dictation is said
  66. (serve "stream"). env: VOLC_APP_ID, VOLC_ACCESS_TOKEN
  67. Backends that need a package re-run themselves with `uv run --with <package>` when it is missing.
  68. """
  69. from __future__ import annotations
  70. import argparse
  71. import contextlib
  72. import base64
  73. import gzip
  74. import hashlib
  75. import io
  76. import json
  77. import os
  78. import platform
  79. import re
  80. import select as _select # `select` is this module's model pick
  81. import shutil
  82. import socket
  83. import struct
  84. import subprocess
  85. import tempfile
  86. import sys
  87. import threading
  88. import time
  89. import unicodedata
  90. import http.client
  91. import urllib.error
  92. import urllib.parse
  93. import urllib.request
  94. import uuid
  95. import wave
  96. from concurrent.futures import ThreadPoolExecutor
  97. from datetime import datetime
  98. from pathlib import Path
  99. import numpy as np
  100. sys.path.insert(0, str(Path(__file__).parent))
  101. import dictation_vocab # noqa: E402
  102. from _common import (BLURT_HOME, CPU_SERVE_IDLE, IS_MAC, IS_WIN, MODELS_DIR, SERVE_IDLE, die, ffmpeg_bin, fmt_ts, global_config, # noqa: E402
  103. landed_in, load_json, probe_duration, progress_line, save_config_if_readable, save_json,
  104. server_exit, serve_exit_reason, serve_said, use_system_certs)
  105. from _common import (begin_download, download_lock, download_path, end_download, fetch, folder_complete, # noqa: E402
  106. safetensors_complete, watch_parent, write_text_atomic)
  107. use_system_certs()
  108. SR = 16000
  109. HERE = Path(__file__).parent
  110. VAD_URLS = [
  111. "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/silero_vad.onnx",
  112. "https://huggingface.co/csukuangfj/vad/resolve/main/silero_vad.onnx",
  113. "https://hf-mirror.com/csukuangfj/vad/resolve/main/silero_vad.onnx",
  114. ]
  115. PURPOSES = ("batch", "live", "note")
  116. # The VAD ships with the skill (models/silero_vad.onnx, 0.6 MB; Silero VAD, MIT, as k2-fsa/sherpa-onnx publishes it —
  117. # the same bytes as VAD_URLS[0]). v1.0.6: on a fresh Mac it was all a local model lacked, and the island and the setup
  118. # said "no speech model · 1 MB" for it. bundled_vad() puts it where the models live before anything looks for it.
  119. BUNDLED_VAD = HERE.parent / "models" / "silero_vad.onnx"
  120. def bundled_vad() -> None:
  121. dest = MODELS_DIR / "silero_vad.onnx"
  122. if dest.exists() or not BUNDLED_VAD.is_file():
  123. return
  124. try:
  125. MODELS_DIR.mkdir(parents=True, exist_ok=True)
  126. tmp = dest.with_suffix(".part")
  127. shutil.copyfile(BUNDLED_VAD, tmp)
  128. tmp.replace(dest)
  129. except OSError:
  130. pass # the download still works
  131. BENCH_FILE = BLURT_HOME / "asr-bench.json"
  132. RUN_DIR = BLURT_HOME / "run"
  133. class Fail(Exception):
  134. """A CLI failure with a machine code (next-round.md: {"ok": false, "error", "code"}, exit 1 / 3 / 4)."""
  135. def __init__(self, message: str, code: str = "error", exit_code: int = 1):
  136. super().__init__(message)
  137. self.code, self.exit_code = code, exit_code
  138. class Interrupted(Exception):
  139. """An interim decode (dictation's live text) stopped because another request is waiting (docs/design/dictation.md
  140. §Live text): the final pieces never wait for one."""
  141. def emit(obj: dict) -> None:
  142. print(json.dumps(obj, ensure_ascii=False), flush=True)
  143. # ------------------------------------------------------------------ the model registry (asr_models.json)
  144. _REGISTRY: dict | None = None
  145. def registry() -> dict:
  146. global _REGISTRY
  147. if _REGISTRY is None:
  148. _REGISTRY = json.loads((HERE / "asr_models.json").read_text(encoding="utf-8"))
  149. return _REGISTRY
  150. def entries(local_only: bool = False) -> list[dict]:
  151. return [m for m in registry()["models"] if not local_only or m.get("local", True)]
  152. def entry(model_id: str | None) -> dict | None:
  153. return next((m for m in registry()["models"] if m["id"] == model_id), None)
  154. def entry_for(backend: str, model: str | None = None) -> dict | None:
  155. """The registry entry a legacy (backend, model) pair means; the backend's first entry when model is None."""
  156. cands = [m for m in registry()["models"] if m["backend"] == backend]
  157. if model:
  158. for m in cands:
  159. if model in (m["model"], m["id"]) or (m["model"] and m["model"].split("/")[-1] == model):
  160. return m
  161. if backend == "faster-whisper" and model == "large-v3-turbo":
  162. return entry("whisper-large-v3-turbo-ct2")
  163. return None
  164. return cands[0] if cands else None
  165. def speaks(m: dict, lang: str) -> bool:
  166. langs = m.get("languages")
  167. return langs == "multi" or lang in (langs or [])
  168. def platform_key() -> str:
  169. arch = platform.machine().lower()
  170. arch = {"amd64": "x86_64", "x64": "x86_64", "aarch64": "arm64"}.get(arch, arch)
  171. osname = "darwin" if IS_MAC else "windows" if IS_WIN else "linux"
  172. return f"{osname}-{arch}"
  173. def runs_here(m: dict, plat: str | None = None) -> bool:
  174. plats = m.get("platforms") or ["*"]
  175. return "*" in plats or (plat or platform_key()) in plats
  176. # ------------------------------------------------------------------ hardware probe and tiers
  177. def _sysctl(name: str) -> str | None:
  178. try:
  179. return subprocess.run(["sysctl", "-n", name], capture_output=True, text=True, timeout=5).stdout.strip() or None
  180. except (OSError, subprocess.TimeoutExpired):
  181. return None
  182. def _ram_bytes() -> int:
  183. try:
  184. if IS_MAC:
  185. return int(_sysctl("hw.memsize") or 0)
  186. if IS_WIN:
  187. import ctypes
  188. class MEMSTAT(ctypes.Structure):
  189. _fields_ = [("dwLength", ctypes.c_ulong), ("dwMemoryLoad", ctypes.c_ulong),
  190. ("ullTotalPhys", ctypes.c_ulonglong), ("ullAvailPhys", ctypes.c_ulonglong),
  191. ("ullTotalPageFile", ctypes.c_ulonglong), ("ullAvailPageFile", ctypes.c_ulonglong),
  192. ("ullTotalVirtual", ctypes.c_ulonglong), ("ullAvailVirtual", ctypes.c_ulonglong),
  193. ("sullAvailExtendedVirtual", ctypes.c_ulonglong)]
  194. st = MEMSTAT()
  195. st.dwLength = ctypes.sizeof(MEMSTAT)
  196. ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(st))
  197. return int(st.ullTotalPhys)
  198. return os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES")
  199. except (OSError, ValueError, AttributeError):
  200. return 0
  201. def _cores() -> tuple[int, int, list[dict]]:
  202. """(performance cores, efficiency cores, per-level detail). Apple Silicon reports its core clusters as perf
  203. levels (M1–M4: Performance + Efficiency; M5 Pro / Max: Super + Performance); level 0 is the fastest."""
  204. total = os.cpu_count() or 1
  205. if IS_MAC:
  206. levels = []
  207. for i in range(int(_sysctl("hw.nperflevels") or 0)):
  208. n = int(_sysctl(f"hw.perflevel{i}.physicalcpu") or 0)
  209. levels.append({"name": _sysctl(f"hw.perflevel{i}.name") or f"level{i}", "cores": n})
  210. if levels:
  211. return levels[0]["cores"], sum(lv["cores"] for lv in levels[1:]), levels
  212. phys = int(_sysctl("hw.physicalcpu") or total)
  213. return phys, 0, []
  214. try: # physical cores where the OS tells us (Linux: unique core ids)
  215. if not IS_WIN and Path("/proc/cpuinfo").exists():
  216. ids = set(re.findall(r"physical id\s*:\s*(\d+)[\s\S]*?core id\s*:\s*(\d+)", Path("/proc/cpuinfo").read_text()))
  217. if ids:
  218. return len(ids), 0, []
  219. except OSError:
  220. pass
  221. return max(1, total // 2) if total > 2 else total, 0, []
  222. def _apple_gpu() -> dict | None:
  223. try:
  224. out = subprocess.run(["ioreg", "-r", "-c", "AGXAccelerator", "-d", "1"], capture_output=True, text=True,
  225. timeout=5).stdout
  226. except (OSError, subprocess.TimeoutExpired):
  227. return None
  228. cores = re.search(r'"gpu-core-count"\s*=\s*(\d+)', out)
  229. name = re.search(r'"model"\s*=\s*"([^"]+)"', out)
  230. return {"kind": "apple", "name": name.group(1) if name else "Apple GPU", **({"cores": int(cores.group(1))} if cores else {})}
  231. def _nvidia_gpu() -> dict | None:
  232. if not shutil.which("nvidia-smi"):
  233. return None
  234. try:
  235. out = subprocess.run(["nvidia-smi", "--query-gpu=name,memory.total", "--format=csv,noheader,nounits"],
  236. capture_output=True, text=True, timeout=10).stdout.strip().splitlines()
  237. except (OSError, subprocess.TimeoutExpired):
  238. return None
  239. if not out:
  240. return None
  241. name, _, mem = out[0].rpartition(",")
  242. try:
  243. vram = round(float(mem) / 1024, 1)
  244. except ValueError:
  245. vram = None
  246. return {"kind": "nvidia", "name": name.strip(), **({"vram_gb": vram} if vram else {})}
  247. def on_battery() -> bool:
  248. try:
  249. if IS_MAC:
  250. out = subprocess.run(["pmset", "-g", "batt"], capture_output=True, text=True, timeout=5).stdout
  251. return "Battery Power" in out
  252. if IS_WIN:
  253. import ctypes
  254. class SPS(ctypes.Structure):
  255. _fields_ = [("ACLineStatus", ctypes.c_byte), ("BatteryFlag", ctypes.c_byte),
  256. ("BatteryLifePercent", ctypes.c_byte), ("SystemStatusFlag", ctypes.c_byte),
  257. ("BatteryLifeTime", ctypes.c_ulong), ("BatteryFullLifeTime", ctypes.c_ulong)]
  258. s = SPS()
  259. return bool(ctypes.windll.kernel32.GetSystemPowerStatus(ctypes.byref(s))) and s.ACLineStatus == 0
  260. mains = [p for p in Path("/sys/class/power_supply").glob("*") if (p / "online").exists()]
  261. return bool(mains) and all((p / "online").read_text().strip() == "0" for p in mains)
  262. except (OSError, ValueError, AttributeError, subprocess.TimeoutExpired):
  263. return False
  264. def tier_for(hw: dict) -> str:
  265. """K9 tiers. Apple Silicon by unified memory; NVIDIA by VRAM; other machines by cores and RAM."""
  266. ram, cores, gpu = hw["ram_gb"], hw["perf_cores"] + hw["eff_cores"], hw["gpu"]
  267. if ram < 7.5 or cores <= 2:
  268. return "minimal"
  269. if hw["arch"] == "arm64" and hw["os"] == "darwin":
  270. return "pro" if ram >= 30 else "standard" if ram >= 15 else "light"
  271. vram = gpu.get("vram_gb") or 0
  272. if gpu.get("kind") == "nvidia" and vram >= 8:
  273. return "pro"
  274. if gpu.get("kind") == "nvidia" and vram >= 4:
  275. return "standard"
  276. if cores >= 8 and ram >= 15:
  277. return "standard"
  278. return "light"
  279. _HW: dict | None = None
  280. def probe(refresh: bool = False) -> dict:
  281. global _HW
  282. if _HW is not None and not refresh:
  283. return _HW
  284. perf, eff, levels = _cores()
  285. osname = "darwin" if IS_MAC else "windows" if IS_WIN else "linux"
  286. arch = platform_key().split("-", 1)[1]
  287. gpu = (_apple_gpu() if IS_MAC and arch == "arm64" else None) or _nvidia_gpu() or {"kind": "none", "name": None}
  288. chip = (_sysctl("machdep.cpu.brand_string") if IS_MAC else None) or platform.processor() or platform.machine()
  289. hw = {"os": osname, "arch": arch, "chip": chip, "perf_cores": perf, "eff_cores": eff,
  290. "ram_gb": round(_ram_bytes() / 2**30, 1), "gpu": gpu, "ane": IS_MAC and arch == "arm64",
  291. "on_battery": on_battery()}
  292. if levels:
  293. hw["core_levels"] = levels
  294. hw["tier"] = tier_for(hw)
  295. hw["fingerprint"] = f"{chip}|{hw['ram_gb']:.0f}GB|{gpu.get('name')}|{gpu.get('cores') or gpu.get('vram_gb') or ''}"
  296. _HW = hw
  297. return hw
  298. # ------------------------------------------------------------------ languages
  299. def system_languages() -> list[str]:
  300. """The OS's preferred languages as ISO 639-1 codes (zh-Hant-HK → zh, yue stays yue), first = most preferred."""
  301. out: list[str] = []
  302. for tag in system_language_tags():
  303. code = re.split(r"[-_.@]", tag)[0].lower()
  304. if code in ("c", "posix", ""):
  305. continue
  306. if tag.lower().startswith(("zh-hk", "zh_hk", "yue")):
  307. code = "yue" if tag.lower().startswith("yue") else "zh"
  308. if code not in out:
  309. out.append(code)
  310. return out
  311. def system_language_tags() -> list[str]:
  312. """The OS's preferred languages as it names them (zh-Hans-CN, en-US …), first = most preferred."""
  313. raw: list[str] = []
  314. try:
  315. if IS_MAC:
  316. out = subprocess.run(["defaults", "read", "-g", "AppleLanguages"], capture_output=True, text=True, timeout=5).stdout
  317. raw = re.findall(r'"?([A-Za-z]{2,3}(?:-[A-Za-z0-9]+)*)"?\s*,?\s*\n', out)
  318. elif IS_WIN:
  319. import ctypes
  320. n, size = ctypes.c_ulong(), ctypes.c_ulong()
  321. if ctypes.windll.kernel32.GetUserPreferredUILanguages(8, ctypes.byref(n), None, ctypes.byref(size)):
  322. buf = ctypes.create_unicode_buffer(size.value)
  323. ctypes.windll.kernel32.GetUserPreferredUILanguages(8, ctypes.byref(n), buf, ctypes.byref(size))
  324. raw = [s for s in buf[:size.value].split("\0") if s]
  325. else:
  326. raw = [p for p in (os.environ.get("LANGUAGE") or os.environ.get("LC_ALL") or os.environ.get("LANG") or "").split(":") if p]
  327. except (OSError, subprocess.TimeoutExpired, AttributeError):
  328. pass
  329. return raw
  330. # The Chinese script the user reads (desi065): Whisper large-v3-turbo writes Mandarin in traditional characters, or
  331. # a mix ("登录頁面周五之前要上線"), for a zh-Hans user. Whisper has no setting for the script (its language token is
  332. # just "zh"), and the usual trick — an initial prompt in simplified Chinese — was measured and dropped: it fixed the
  333. # characters, but on an English note it once sent the model into a loop ("段段段…") and it glued English names to
  334. # the Chinese around them. So text heard as Chinese is converted character by character to the user's script
  335. # (OpenCC's t2s tables: a mechanical script mapping that reads no meaning). zh-Hant users (zh-TW / zh-HK / zh-MO, or
  336. # asr.chinese_script "hant") keep what the model wrote: nothing is converted for them. Qwen3-ASR (dictation) and
  337. # SenseVoice write simplified already; the conversion changes nothing there.
  338. _HANT_TAGS = ("zh-hant", "zh_hant", "zh-tw", "zh_tw", "zh-hk", "zh_hk", "zh-mo", "zh_mo", "yue")
  339. _script: dict = {}
  340. def chinese_script(cfg: dict | None = None) -> str | None:
  341. """"hans" | "hant" | None: asr.chinese_script, else the first Chinese entry of the OS's preferred languages
  342. (zh-Hans-CN, zh-CN, zh → hans; zh-Hant-*, zh-TW, zh-HK, zh-MO, yue → hant); None without Chinese. Read once a
  343. process."""
  344. cfg = asr_config() if cfg is None else cfg
  345. if cfg.get("chinese_script") in ("hans", "hant"):
  346. return cfg["chinese_script"]
  347. if "system" not in _script:
  348. tag = next((t.lower() for t in system_language_tags() if t.lower().startswith(("zh", "yue"))), None)
  349. _script["system"] = None if tag is None else "hant" if tag.startswith(_HANT_TAGS) else "hans"
  350. return _script["system"]
  351. _KANA = re.compile(r"[぀-ヿ]")
  352. def to_user_script(text: str, lang: str | None, script: str | None = None) -> str:
  353. """Text the speech model heard as Chinese (or with no language said), in a zh-Hans user's script: traditional
  354. characters → simplified, one by one. Japanese (kana, or tagged ja) and everyone else's text are left alone."""
  355. script = script if script is not None else chinese_script()
  356. if script != "hans" or not text or (lang and lang not in ("zh", "yue")) or _KANA.search(text):
  357. return text
  358. if not re.search(r"[㐀-鿿]", text):
  359. return text
  360. conv = _script.get("t2s")
  361. if conv is None:
  362. try:
  363. from opencc import OpenCC
  364. conv = OpenCC("t2s").convert
  365. except Exception as e: # noqa: BLE001 — the package missing must not cost the words
  366. print(f"script conversion unavailable: {type(e).__name__}: {e}", file=sys.stderr, flush=True)
  367. conv = False
  368. _script["t2s"] = conv
  369. return conv(text) if conv else text
  370. SWITCH_LANGUAGES = {"zh": "Chinese", "yue": "Cantonese", "ja": "Japanese", "ko": "Korean"}
  371. def switch_language(langs: list[str]) -> str | None:
  372. """The language a user mixes English names into, as Qwen3-ASR names it: the first of their languages that people
  373. commonly speak with English words inside it (zh → "Chinese"). None for English-only users."""
  374. return next((SWITCH_LANGUAGES[c] for c in langs if c in SWITCH_LANGUAGES), None)
  375. def user_languages(cfg: dict | None = None) -> list[str]:
  376. """asr.languages, else the system's preferred languages ∪ {en}."""
  377. cfg = asr_config() if cfg is None else cfg
  378. langs = cfg.get("languages")
  379. if isinstance(langs, str):
  380. langs = [x.strip() for x in langs.split(",") if x.strip()]
  381. if not langs:
  382. langs = system_languages()
  383. if cfg.get("language") and cfg["language"] not in langs: # the spoken language the user picked
  384. langs.insert(0, cfg["language"])
  385. if "en" not in langs:
  386. langs.append("en")
  387. return list(dict.fromkeys(langs))
  388. # ------------------------------------------------------------------ configuration
  389. def asr_config() -> dict:
  390. c = global_config().get("asr")
  391. return c if isinstance(c, dict) else {}
  392. def is_auto(cfg: dict) -> bool:
  393. """asr.auto, default true — except for a legacy explicit choice of a non-default backend (kept as the user's)."""
  394. if "auto" in cfg:
  395. return cfg["auto"] is not False
  396. return not (cfg.get("backend") and cfg.get("backend") != "sensevoice")
  397. def local_only() -> bool:
  398. return bool((global_config().get("privacy") or {}).get("local_asr"))
  399. def limits(hw: dict | None = None, cfg: dict | None = None) -> dict:
  400. """asr.limits with K9 defaults: threads = max(1, min(4, perf_cores // 2)); one batch job at a time;
  401. batch at utility / nice 10, live and voice notes at user-initiated; MLX may use a quarter of the memory."""
  402. hw = hw or probe()
  403. lim = dict((cfg if cfg is not None else asr_config()).get("limits") or {})
  404. prio = {"batch": "utility", "live": "user-initiated", "note": "user-initiated", "bench": "utility"}
  405. if isinstance(lim.get("priority"), dict):
  406. prio.update(lim["priority"])
  407. elif isinstance(lim.get("priority"), str):
  408. prio["batch"] = prio["bench"] = lim["priority"]
  409. return {"threads": max(1, int(lim.get("threads") or max(1, min(4, hw["perf_cores"] // 2)))),
  410. "max_concurrent": max(1, int(lim.get("max_concurrent") or 1)),
  411. "priority": prio,
  412. "max_memory_fraction": min(0.9, max(0.05, float(lim.get("max_memory_fraction") or 0.25)))}
  413. # ------------------------------------------------------------------ resource caps
  414. QOS = {"user-interactive": 0x21, "user-initiated": 0x19, "default": 0x15, "utility": 0x11, "background": 0x09}
  415. def apply_priority(level: str, reexec: bool = True) -> dict:
  416. """Lower this process's priority before any model or worker thread exists. macOS: worker threads that runtimes
  417. create (onnxruntime, CTranslate2) do not inherit the creating thread's QoS, so utility / background work re-runs
  418. itself once under `taskpolicy -c <level>` (a QoS clamp on every thread of the process) — plus nice; the calling
  419. thread's QoS is set either way. Windows: priority class. Linux: nice. Returns what was applied."""
  420. done: dict = {"level": level}
  421. nice = {"background": 15, "utility": 10}.get(level, 0)
  422. if IS_MAC and reexec and level in ("utility", "background") and os.environ.get("BLURT_QOS_CLAMP") != level \
  423. and Path("/usr/sbin/taskpolicy").exists() and sys.argv and sys.argv[0].endswith(".py"):
  424. os.environ["BLURT_QOS_CLAMP"] = level
  425. sys.stdout.flush()
  426. sys.stderr.flush()
  427. os.execv("/usr/sbin/taskpolicy", ["taskpolicy", "-c", level, sys.executable, os.path.abspath(sys.argv[0]),
  428. *sys.argv[1:]])
  429. if IS_MAC and os.environ.get("BLURT_QOS_CLAMP") == level:
  430. done["clamp"] = level
  431. if IS_MAC:
  432. try:
  433. import ctypes
  434. lib = ctypes.CDLL("/usr/lib/libSystem.B.dylib")
  435. if lib.pthread_set_qos_class_self_np(ctypes.c_uint(QOS.get(level, 0x15)), 0) == 0:
  436. done["qos"] = level
  437. except (OSError, AttributeError):
  438. pass
  439. if IS_WIN:
  440. try:
  441. import ctypes
  442. cls = {"background": 0x40, "utility": 0x4000}.get(level, 0x20) # IDLE / BELOW_NORMAL / NORMAL
  443. if ctypes.windll.kernel32.SetPriorityClass(ctypes.windll.kernel32.GetCurrentProcess(), cls):
  444. done["priority_class"] = cls
  445. except (OSError, AttributeError):
  446. pass
  447. elif nice:
  448. try:
  449. cur = os.getpriority(os.PRIO_PROCESS, 0)
  450. if cur < nice:
  451. os.setpriority(os.PRIO_PROCESS, 0, nice)
  452. done["nice"] = os.getpriority(os.PRIO_PROCESS, 0)
  453. except (OSError, AttributeError):
  454. pass
  455. return done
  456. def cap_threads(n: int) -> None:
  457. """BLAS / OpenMP pools size themselves at import: set the caps first."""
  458. for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS",
  459. "NUMEXPR_NUM_THREADS", "RAYON_NUM_THREADS"):
  460. os.environ[k] = str(n)
  461. # MLX's cache of freed buffers, kept for reuse. It used to be min(limit / 4, 2 GB): the warm servers then held what
  462. # their warm-up had freed (polish's two prompt prefills: ~1.5 GB) until the first request — 4.6 GB instead of 3.1 GB
  463. # for Gemma 4 E2B. 64 MB costs 2.4 ms a polish request with the same output (0 MB: +68 ms); docs/design/dictation.md
  464. # §Memory.
  465. MLX_CACHE_BYTES = 64 << 20
  466. def cap_mlx_memory(fraction: float) -> int | None:
  467. """Keep MLX (which otherwise grows its buffer cache to most of unified memory) inside fraction × RAM, and its cache
  468. of freed buffers small (MLX_CACHE_BYTES)."""
  469. try:
  470. import mlx.core as mx
  471. except ImportError:
  472. return None
  473. total = _ram_bytes()
  474. if not total:
  475. return None
  476. limit = int(total * fraction)
  477. for fn, val in (("set_memory_limit", limit), ("set_cache_limit", min(limit // 4, MLX_CACHE_BYTES))):
  478. f = getattr(mx, fn, None) or getattr(getattr(mx, "metal", None), fn, None)
  479. if f:
  480. try:
  481. f(val)
  482. except Exception: # noqa: BLE001 — an older MLX without the call
  483. pass
  484. return limit
  485. class AsrSlot:
  486. """At most N concurrent batch transcriptions machine-wide: byte-range locks on $BLURT_HOME/run/asr.lock
  487. (released by the OS when a process dies). Waiting = queued: said on stderr and, under a job ($BLURT_JOB),
  488. as events in jobs/<id>.events.jsonl (K4). A run that can't write $BLURT_HOME (an organizing agent in its
  489. sandbox) transcribes without a slot."""
  490. _held: set[int] = set() # slots this process holds (record locks are per process, not per thread)
  491. _guard = threading.Lock()
  492. _file = None # one descriptor per process: closing any descriptor drops all of its locks
  493. def __init__(self, n: int, timeout: float = 6 * 3600, poll: float = 0.25):
  494. self.n, self.timeout, self.poll = n, timeout, poll
  495. self.fh = None
  496. self.slot: int | None = None
  497. self.waited = 0.0
  498. def _try(self, i: int) -> bool:
  499. with AsrSlot._guard:
  500. if i in AsrSlot._held or not self._lock(i):
  501. return False
  502. AsrSlot._held.add(i)
  503. return True
  504. def _lock(self, i: int) -> bool:
  505. try:
  506. if IS_WIN:
  507. import msvcrt
  508. self.fh.seek(i)
  509. msvcrt.locking(self.fh.fileno(), msvcrt.LK_NBLCK, 1)
  510. else:
  511. import fcntl
  512. fcntl.lockf(self.fh, fcntl.LOCK_EX | fcntl.LOCK_NB, 1, i, 0)
  513. return True
  514. except OSError:
  515. return False
  516. def __enter__(self):
  517. with AsrSlot._guard:
  518. if AsrSlot._file is None:
  519. try:
  520. RUN_DIR.mkdir(parents=True, exist_ok=True)
  521. AsrSlot._file = open(RUN_DIR / "asr.lock", "a+b")
  522. except PermissionError:
  523. print("asr: $BLURT_HOME/run is read-only here (a sandboxed run): not queued", file=sys.stderr, flush=True)
  524. return self
  525. self.fh = AsrSlot._file
  526. t0 = time.time()
  527. queued = False
  528. while True:
  529. for i in range(self.n):
  530. if self._try(i):
  531. self.slot = i
  532. self.waited = time.time() - t0
  533. if queued:
  534. print("asr: slot free, starting", file=sys.stderr, flush=True)
  535. job_event("asr.started", {"id": "transcribe", "state": "running"}, {"waited_seconds": round(self.waited, 1)})
  536. return self
  537. if not queued:
  538. queued = True
  539. print(f"asr: queued (max_concurrent={self.n}; another transcription is running)", file=sys.stderr, flush=True)
  540. job_event("asr.queued", {"id": "transcribe", "state": "pending"}, {"queued": True, "max_concurrent": self.n})
  541. if time.time() - t0 > self.timeout:
  542. raise Fail(f"waited {self.timeout:.0f}s for a free ASR slot", "lock_timeout", 3)
  543. time.sleep(self.poll)
  544. def __exit__(self, *exc):
  545. if self.slot is None:
  546. return False
  547. with AsrSlot._guard:
  548. try:
  549. if IS_WIN:
  550. import msvcrt
  551. self.fh.seek(self.slot)
  552. msvcrt.locking(self.fh.fileno(), msvcrt.LK_UNLCK, 1)
  553. else:
  554. import fcntl
  555. fcntl.lockf(self.fh, fcntl.LOCK_UN, 1, self.slot, 0)
  556. except OSError:
  557. pass
  558. AsrSlot._held.discard(self.slot)
  559. return False
  560. def job_event(event: str, step: dict | None = None, detail: dict | None = None) -> None:
  561. job = os.environ.get("BLURT_JOB")
  562. if not job or not re.fullmatch(r"[A-Za-z0-9._-]+", job):
  563. return
  564. line = {"t": datetime.now().astimezone().isoformat(timespec="seconds"), "source": "asr", "event": event}
  565. if step:
  566. line["step"] = step
  567. if detail:
  568. line["detail"] = detail
  569. try:
  570. (BLURT_HOME / "jobs").mkdir(parents=True, exist_ok=True)
  571. fd = os.open(BLURT_HOME / "jobs" / f"{job}.events.jsonl", os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o644)
  572. try:
  573. os.write(fd, (json.dumps(line, ensure_ascii=False) + "\n").encode())
  574. finally:
  575. os.close(fd)
  576. except OSError:
  577. pass
  578. def peak_memory_mb() -> float | None:
  579. """This process's peak memory: macOS lifetime max physical footprint (counts GPU buffers in unified memory),
  580. Linux max RSS, Windows peak working set."""
  581. try:
  582. if IS_MAC:
  583. import ctypes
  584. buf = (ctypes.c_uint64 * 64)()
  585. lib = ctypes.CDLL("/usr/lib/libSystem.B.dylib")
  586. if lib.proc_pid_rusage(os.getpid(), 4, ctypes.byref(buf)) == 0: # RUSAGE_INFO_V4
  587. return round(buf[2 + 28] / 2**20, 1) # after the 16-byte uuid: ri_lifetime_max_phys_footprint
  588. return None
  589. if IS_WIN:
  590. import ctypes
  591. from ctypes import wintypes
  592. class PMC(ctypes.Structure):
  593. _fields_ = [("cb", wintypes.DWORD), ("PageFaultCount", wintypes.DWORD),
  594. ("PeakWorkingSetSize", ctypes.c_size_t), ("WorkingSetSize", ctypes.c_size_t),
  595. ("QuotaPeakPagedPoolUsage", ctypes.c_size_t), ("QuotaPagedPoolUsage", ctypes.c_size_t),
  596. ("QuotaPeakNonPagedPoolUsage", ctypes.c_size_t), ("QuotaNonPagedPoolUsage", ctypes.c_size_t),
  597. ("PagefileUsage", ctypes.c_size_t), ("PeakPagefileUsage", ctypes.c_size_t)]
  598. c = PMC()
  599. c.cb = ctypes.sizeof(PMC)
  600. ctypes.windll.psapi.GetProcessMemoryInfo(ctypes.windll.kernel32.GetCurrentProcess(), ctypes.byref(c), c.cb)
  601. return round(c.PeakWorkingSetSize / 2**20, 1)
  602. import resource
  603. return round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 1)
  604. except Exception: # noqa: BLE001 — measurement only
  605. return None
  606. def qos_cpu_seconds() -> dict | None:
  607. """macOS: CPU seconds this process spent per QoS class (proves the priority took effect)."""
  608. if not IS_MAC:
  609. return None
  610. try:
  611. import ctypes
  612. lib = ctypes.CDLL("/usr/lib/libSystem.B.dylib")
  613. buf = (ctypes.c_uint64 * 64)()
  614. if lib.proc_pid_rusage(os.getpid(), 4, ctypes.byref(buf)) != 0:
  615. return None
  616. tb = (ctypes.c_uint32 * 2)() # mach_timebase_info: the counters are in mach ticks (41.67 ns on Apple Silicon)
  617. lib.mach_timebase_info(ctypes.byref(tb))
  618. ns = tb[0] / tb[1] if tb[1] else 1.0
  619. names = ("default", "maintenance", "background", "utility", "legacy", "user_initiated", "user_interactive")
  620. return {n: round(buf[2 + 18 + i] * ns / 1e9, 2) for i, n in enumerate(names) if buf[2 + 18 + i]}
  621. except (OSError, AttributeError):
  622. return None
  623. # ------------------------------------------------------------------ downloads
  624. def _download_url(urls: list[str], dest: Path, label: str, aux: str | None = None) -> Path:
  625. if os.environ.get("BLURT_HF_MIRROR"): # the China-friendly mirrors first (GitHub releases don't load from there)
  626. urls = sorted(urls, key=lambda u: not ("hf-mirror" in u or "modelscope.cn" in u))
  627. dest.parent.mkdir(parents=True, exist_ok=True)
  628. last = None
  629. for url in urls:
  630. try:
  631. serve_said(f"downloading {label} <- {url}")
  632. tmp = dest.with_suffix(dest.suffix + ".part") # a partial one from last time is continued (Range)
  633. t0 = [time.time()]
  634. def progress(done: int, total: int) -> None:
  635. if total and time.time() - t0[0] > 1:
  636. serve_said(progress_line(done, total, aux))
  637. t0[0] = time.time()
  638. fetch(url, tmp, progress=progress, timeout=30)
  639. os.replace(tmp, dest)
  640. return dest
  641. except Exception as e: # noqa: BLE001 — try the next mirror
  642. last = e
  643. print(f" failed: {e}", file=sys.stderr)
  644. raise Fail(f"could not download {label}: {last}. Set BLURT_HF_MIRROR=1 or download manually to {dest}", "network")
  645. def download(rel: str, aux: str | None = None) -> Path:
  646. """A single model file under MODELS_DIR (the VAD, SenseVoice's files). `aux`: fetched on the way to another
  647. model (its progress lines are named so: progress_line)."""
  648. dest = MODELS_DIR / rel
  649. if rel == "silero_vad.onnx":
  650. bundled_vad()
  651. if dest.exists() and dest.stat().st_size > 0:
  652. return dest
  653. if os.environ.get("BLURT_NO_DOWNLOAD"): # live.py: never start a download in the middle of a recording
  654. raise Fail(f"{rel} is not downloaded", "model_missing")
  655. urls = VAD_URLS if rel == "silero_vad.onnx" else VP_URLS if rel == VP_FILE else VP_SEG_URLS if rel == VP_SEG_FILE else next(
  656. (u for m in entries() for f, u in ((m.get("install") or {}).get("files") or {}).items() if f == rel), None)
  657. if not urls:
  658. raise Fail(f"no download source for {rel}", "model_missing")
  659. return _download_url(urls, dest, rel, aux)
  660. def hf_cache() -> Path:
  661. if os.environ.get("HF_HUB_CACHE"):
  662. return Path(os.environ["HF_HUB_CACHE"])
  663. return Path(os.environ.get("HF_HOME") or Path.home() / ".cache" / "huggingface") / "hub"
  664. def _hf_record(repo: str) -> Path:
  665. """What a finished `snapshot_download` of `repo` left: MODELS_DIR/asr/.hf/<repo>.json (its snapshot and files)."""
  666. return MODELS_DIR / "asr" / ".hf" / (repo.replace("/", "--") + ".json")
  667. def _snapshot_complete(s: Path, repo: str) -> bool:
  668. """A Hugging Face snapshot is a whole model. One `setup` saw finish: every file it listed is there. One from before
  669. that record (or fetched by something else): its weights, its config, every shard its index names, and the
  670. tokenizer its tokenizer_config.json asks for — a setup cut short between files leaves one of them out."""
  671. rec = load_json(_hf_record(repo), None)
  672. if isinstance(rec, dict) and rec.get("snapshot") == s.name and isinstance(rec.get("files"), list):
  673. return all((s / f).is_file() for f in rec["files"])
  674. names = {f.name for f in s.iterdir()} if s.is_dir() else set()
  675. if not any(f.endswith((".safetensors", ".npz", ".bin")) and not f.startswith("training_args") for f in names):
  676. return False
  677. if "config.json" not in names or any(f.endswith((".part", ".incomplete")) for f in names):
  678. return False
  679. if any(f.endswith(".safetensors") for f in names) and not safetensors_complete(s):
  680. return False
  681. if "tokenizer_config.json" in names and not names & {"tokenizer.json", "vocab.json", "tokenizer.model"}:
  682. return False
  683. if "tokenizer_config.json" in names and "vocab.json" in names and "tokenizer.json" not in names \
  684. and "merges.txt" not in names: # a BPE tokenizer in two files: both
  685. return False
  686. return True
  687. def hf_snapshot(repo: str) -> Path | None:
  688. """The newest complete snapshot of `repo` in the Hugging Face cache (_snapshot_complete), else None."""
  689. root = hf_cache() / ("models--" + repo.replace("/", "--"))
  690. snaps = sorted((root / "snapshots").glob("*"), key=lambda p: p.stat().st_mtime, reverse=True) if root.is_dir() else []
  691. if any((root / "blobs").glob("*.incomplete")):
  692. return None
  693. for s in snaps:
  694. if _snapshot_complete(s, repo):
  695. return s
  696. return None
  697. def model_path(m: dict) -> Path | None:
  698. """Where an installed model lives, or None when it is not installed."""
  699. inst = m.get("install") or {}
  700. if inst.get("files"):
  701. paths = [MODELS_DIR / f for f in inst["files"]]
  702. return paths[0].parent if all(p.exists() and p.stat().st_size > 0 for p in paths) else None
  703. if inst.get("archive"):
  704. d = MODELS_DIR / inst["dir"]
  705. need = [d / k.rstrip("/") for k in inst.get("keep", [])]
  706. return d if d.is_dir() and all(p.exists() for p in need) else None
  707. if inst.get("hf"):
  708. own = MODELS_DIR / "asr" / inst["hf"].split("/")[-1] # a ModelScope download (the mirror)
  709. if folder_complete(own, legacy=_own_legacy_complete):
  710. return own
  711. snap = hf_snapshot(inst["hf"])
  712. if snap is None and m["backend"] == "faster-whisper": # older Blurt downloaded it under models/faster-whisper
  713. legacy = MODELS_DIR / "faster-whisper"
  714. if legacy.is_dir():
  715. snap = next((p for p in legacy.rglob("model.bin") if "turbo" in str(p)), None)
  716. snap = snap.parent if snap else None
  717. return snap
  718. return None
  719. def _own_legacy_complete(d: Path) -> bool:
  720. """A mirror download from before the manifest: its config, every weight shard, and no file still partial."""
  721. return (d / "config.json").exists() and safetensors_complete(d) and not any(d.glob("*.part"))
  722. def installed(m: dict) -> bool:
  723. if not m.get("local", True):
  724. # every one of its variables when it needs a pair (Volcengine: App ID + Access Token), else any one
  725. return (all if m.get("keys_all") else any)(os.environ.get(k) for k in m.get("keys", []))
  726. if m.get("builtin"):
  727. return apple_speech_ok()
  728. return model_path(m) is not None
  729. # macOS's own speech recognition (docs/design/first-minute-speech.md): SpeechTranscriber, macOS 26+, run by the Blurt
  730. # app's executable (`blurt-recorder --speech WAV`: Capture/AppleSpeech.swift). Nothing to download from Blurt; the
  731. # language's files are macOS's own (the app asks for them; a run without them fails "model_missing").
  732. APPLE = "apple-speech"
  733. APPLE_LOCALES = {"zh": "zh-CN", "zh-hant": "zh-TW", "yue": "yue-CN", "en": "en-US", "ja": "ja-JP", "ko": "ko-KR",
  734. "fr": "fr-FR", "de": "de-DE", "es": "es-ES", "it": "it-IT", "pt": "pt-BR"}
  735. def apple_speech_helper() -> Path | None:
  736. """The Blurt executable that answers `--speech` (BLURT_APP_BIN from the app, else an installed Blurt.app)."""
  737. cands = [os.environ.get("BLURT_APP_BIN") or "", str(BLURT_HOME / "Blurt.app/Contents/MacOS/blurt-recorder"),
  738. "/Applications/Blurt.app/Contents/MacOS/blurt-recorder",
  739. str(Path.home() / "Applications/Blurt.app/Contents/MacOS/blurt-recorder")]
  740. return next((Path(c) for c in cands if c and os.access(c, os.X_OK)), None)
  741. _APPLE_OK: bool | None = None
  742. def apple_speech_ok() -> bool:
  743. """macOS 26 or later, and a Blurt that can run it."""
  744. global _APPLE_OK
  745. if os.environ.get("BLURT_NO_APPLE_SPEECH"): # tests: a Mac before macOS 26
  746. return False
  747. if _APPLE_OK is None:
  748. try:
  749. major = int((platform.mac_ver()[0] or "0").split(".")[0]) if IS_MAC else 0
  750. except ValueError:
  751. major = 0
  752. _APPLE_OK = major >= 26 and apple_speech_helper() is not None
  753. return _APPLE_OK
  754. def apple_locale(language: str | None) -> str | None:
  755. """The locale for an explicit language (or asr.language), else the app's "Automatic" (AppleSpeech.automatic, the
  756. same rule so recordings and dictation hear the same language): of the user's languages the first one people mix
  757. English into (SWITCH_LANGUAGES: its recogniser hears the English words too), else the first one macOS's
  758. recogniser speaks. An English Mac that lists Chinese too listens for Chinese, English included."""
  759. picked = asr_config().get("language")
  760. if language and language not in ("auto", ""):
  761. langs = [language]
  762. elif picked and picked != "auto":
  763. langs = [picked]
  764. else:
  765. spoken = [x for x in user_languages() if x in APPLE_LOCALES]
  766. langs = [x for x in spoken if x in SWITCH_LANGUAGES] + spoken
  767. for lang in langs:
  768. if lang == "zh" and chinese_script() == "hant":
  769. lang = "zh-hant"
  770. if lang in APPLE_LOCALES:
  771. return APPLE_LOCALES[lang]
  772. return None
  773. def _du_mb(p: Path) -> float:
  774. if p.is_file():
  775. return round(p.stat().st_size / 2**20, 1)
  776. total = 0
  777. for f in p.rglob("*"):
  778. try:
  779. total += f.stat().st_size if f.is_file() else 0 # stat follows the HF cache's symlinks into blobs/
  780. except OSError:
  781. pass
  782. return round(total / 2**20, 1)
  783. def _progress_poller(target: Path, total_mb: float, stop: threading.Event) -> None:
  784. """HF downloads: report ' N/M MB' from what has landed on disk (incl. *.incomplete blobs: _common.landed_in)."""
  785. while not stop.wait(1.0):
  786. if target.exists():
  787. serve_said(progress_line(min(landed_in(target), total_mb * 2**20), total_mb * 2**20))
  788. def install_model(m: dict) -> Path:
  789. """Download a registry model (and the VAD every local backend needs). Progress on stderr. One download of a
  790. model at a time (download_lock): a second caller waits, then finds it installed."""
  791. download("silero_vad.onnx", aux="vad") # its own named progress: not the model's (progress_line)
  792. if not m.get("local", True):
  793. return MODELS_DIR
  794. have = model_path(m)
  795. if have:
  796. return have
  797. if os.environ.get("BLURT_NO_DOWNLOAD"):
  798. raise Fail(f"{m['id']} is not downloaded", "model_missing")
  799. if not runs_here(m):
  800. raise Fail(f"{m['id']} does not run on {platform_key()}", "unsupported")
  801. with download_lock(m["id"]):
  802. have = model_path(m) # the other download finished while this one waited
  803. if have:
  804. return have
  805. return _install_unlocked(m)
  806. def _install_unlocked(m: dict) -> Path:
  807. inst = m["install"]
  808. if inst.get("files"):
  809. for rel in inst["files"]:
  810. download(rel)
  811. elif inst.get("archive"):
  812. _install_archive(m)
  813. elif inst.get("hf"):
  814. ensure_package(m)
  815. os.environ.setdefault("HF_HUB_DISABLE_PROGRESS_BARS", "1") # our " N/M MB" lines are the progress
  816. from huggingface_hub import snapshot_download
  817. if os.environ.get("BLURT_HF_MIRROR"):
  818. os.environ.setdefault("HF_ENDPOINT", "https://hf-mirror.com")
  819. mirrored = inst["hf"].startswith("mlx-community/") # ModelScope mirrors mlx-community (fast from China)
  820. if mirrored and os.environ.get("BLURT_HF_MIRROR"):
  821. _modelscope(inst["hf"], float(m["size_mb"]))
  822. else:
  823. print(f"downloading {inst['hf']}", file=sys.stderr, flush=True)
  824. stop = threading.Event()
  825. target = hf_cache() / ("models--" + inst["hf"].replace("/", "--"))
  826. th = threading.Thread(target=_progress_poller, args=(target, float(m["size_mb"]), stop), daemon=True)
  827. th.start()
  828. try:
  829. snap = Path(snapshot_download(inst["hf"]))
  830. # it finished: what it brought is the model (hf_snapshot checks against this list from now on)
  831. write_text_atomic(_hf_record(inst["hf"]), json.dumps(
  832. {"snapshot": snap.name, "files": sorted(str(f.relative_to(snap)) for f in snap.rglob("*") if f.is_file())}))
  833. except Exception as e: # noqa: BLE001
  834. if not mirrored:
  835. raise Fail(f"could not download {inst['hf']}: {e}", "network")
  836. print(f" Hugging Face failed ({e}); trying ModelScope", file=sys.stderr, flush=True)
  837. _modelscope(inst["hf"], float(m["size_mb"]))
  838. finally:
  839. stop.set()
  840. path = model_path(m)
  841. if not path:
  842. raise Fail(f"{m['id']}: download finished but the files are not complete", "model_missing")
  843. return path
  844. def _modelscope(repo: str, total_mb: float) -> Path:
  845. """An mlx-community repo from ModelScope's mirror → MODELS_DIR/asr/<name>: the manifest and the download marker
  846. first (the folder is a model only once every file is in: folder_complete), each file through <name>.part,
  847. continued where a cut-short download left it."""
  848. dest = MODELS_DIR / "asr" / repo.split("/")[-1]
  849. api = f"https://modelscope.cn/api/v1/models/{repo}/repo/files?Recursive=true"
  850. try:
  851. with urllib.request.urlopen(urllib.request.Request(api, headers={"User-Agent": "blurt"}), timeout=30) as r:
  852. files = json.load(r)["Data"]["Files"]
  853. except Exception as e: # noqa: BLE001
  854. raise Fail(f"could not download {repo} from ModelScope: {e}", "network")
  855. want = [x for x in files if x.get("Type") == "blob" and not x["Path"].endswith((".md", ".gitattributes"))]
  856. try:
  857. begin_download(dest, [{"path": x["Path"], "size": x.get("Size")} for x in want])
  858. except ValueError as e:
  859. raise Fail(f"invalid ModelScope manifest: {e}", "network") from None
  860. done = 0.0
  861. serve_said(f"downloading {repo} (ModelScope)")
  862. for x in want:
  863. p = download_path(dest, x["Path"])
  864. if p.exists() and p.stat().st_size == x.get("Size"):
  865. done += p.stat().st_size / 2**20
  866. continue
  867. p.parent.mkdir(parents=True, exist_ok=True)
  868. tmp = p.with_name(p.name + ".part")
  869. url = f"https://modelscope.cn/models/{repo}/resolve/master/{x['Path']}"
  870. t0 = [time.time()]
  871. def progress(have: int, _total: int) -> None:
  872. if time.time() - t0[0] > 1:
  873. serve_said(f" {int(done + have / 2**20)}/{int(total_mb)} MB")
  874. t0[0] = time.time()
  875. got = 0
  876. for attempt in range(3):
  877. try:
  878. got = fetch(url, tmp, x.get("Size"), progress) # each try continues the last one's bytes
  879. except Exception as e: # noqa: BLE001
  880. if attempt == 2:
  881. raise Fail(f"could not download {repo} from ModelScope: {e}", "network")
  882. continue
  883. if not x.get("Size") or got == x["Size"]: # a dropped connection can end the read quietly
  884. break
  885. if attempt == 2:
  886. raise Fail(f"could not download {repo} from ModelScope: {x['Path']} came short", "network")
  887. done += got / 2**20
  888. os.replace(tmp, p)
  889. end_download(dest)
  890. return dest
  891. def _install_archive(m: dict) -> None:
  892. import tarfile
  893. inst = m["install"]
  894. dest = MODELS_DIR / inst["dir"]
  895. tmp = MODELS_DIR / f".{inst['dir']}.tar.bz2"
  896. urls = inst["archive"] if isinstance(inst["archive"], list) else [inst["archive"]]
  897. _download_url(urls, tmp, m["id"])
  898. staging = MODELS_DIR / f".{inst['dir']}.extract"
  899. shutil.rmtree(staging, ignore_errors=True)
  900. staging.mkdir(parents=True)
  901. keep = inst.get("keep") or []
  902. with tarfile.open(tmp, "r:bz2") as tar:
  903. members = []
  904. for mem in tar.getmembers():
  905. parts = Path(mem.name).parts
  906. rel = "/".join(parts[1:]) # drop the archive's top folder
  907. if not rel or ".." in parts or mem.name.startswith("/") or not (mem.isfile() or mem.isdir()):
  908. continue
  909. if keep and not any(rel == k or (k.endswith("/") and rel.startswith(k)) for k in keep):
  910. continue
  911. mem.name = rel
  912. members.append(mem)
  913. print(f"unpacking {m['id']}", file=sys.stderr, flush=True)
  914. tar.extractall(staging, members=members, **({"filter": "data"} if sys.version_info >= (3, 12) else {}))
  915. shutil.rmtree(dest, ignore_errors=True)
  916. os.replace(staging, dest)
  917. tmp.unlink(missing_ok=True)
  918. # ------------------------------------------------------------------ optional packages
  919. class NeedsRestart(Exception):
  920. """A running `serve` was asked for a model whose package isn't in its environment: it answers "restarting",
  921. takes its run file away and starts again with the package (`_reexec_with`) — never mid-request."""
  922. def __init__(self, pkg: str):
  923. super().__init__(f"the speech server restarts to load {pkg}")
  924. self.pkg = pkg
  925. # set by `serve` once its run file is written: from then on a missing package is a clean restart (NeedsRestart),
  926. # not an exec in the middle of a request
  927. SERVING = False
  928. def _has_package(pkg: str) -> bool:
  929. import importlib.util
  930. return importlib.util.find_spec(pkg.replace("-", "_")) is not None
  931. def _reexec_with(pkgs: list[str]) -> None:
  932. """Run this very command again under `uv run --with` these packages (and the ones already added)."""
  933. uv = shutil.which("uv") or str(BLURT_HOME / "bin" / ("uv.exe" if IS_WIN else "uv"))
  934. tried = [p for p in os.environ.get("BLURT_ASR_REEXEC", "").split(",") if p]
  935. new = [p for p in dict.fromkeys(pkgs) if p not in tried]
  936. if not new or not Path(uv).exists() or not sys.argv[0].endswith(".py"):
  937. pkg = (new or pkgs or ["?"])[0]
  938. raise Fail(f"{pkg} is not installed (run with: uv run --with {pkg} {Path(sys.argv[0]).name} …)", "deps", 4)
  939. os.environ["BLURT_ASR_REEXEC"] = ",".join(tried + new)
  940. sys.stdout.flush()
  941. sys.stderr.flush()
  942. withs = [x for p in tried + new for x in ("--with", PACKAGE_PINS.get(p, p))]
  943. os.execvp(uv, [uv, "run", "-q", *withs, os.path.abspath(sys.argv[0]), *sys.argv[1:]])
  944. def ensure_package(m: dict | str) -> None:
  945. """Backends that need a Python package (mlx-whisper, parakeet-mlx, …): re-run this very command under
  946. `uv run --with <package>` when it is missing, so callers never need to know. Inside a running server:
  947. NeedsRestart (the server restarts itself between requests)."""
  948. pkg = m if isinstance(m, str) else m.get("package")
  949. if not pkg or _has_package(pkg):
  950. return
  951. tried = [p for p in os.environ.get("BLURT_ASR_REEXEC", "").split(",") if p]
  952. if SERVING and pkg not in tried:
  953. raise NeedsRestart(pkg)
  954. _reexec_with([pkg])
  955. def serve_packages() -> list[str]:
  956. """The packages of every local model on this Mac that a server may be asked for (any of them can become
  957. dictation's in Settings): a server starts with all of them, so a model switch never restarts it. Before, it
  958. re-ran itself under `uv run --with` in the middle of the request that needed one (2026-10-31: dictation switched
  959. to Whisper) — the request got no answer, and until the new process was up its run file named a live pid with a
  960. closed port, so every piece of the take went to a cold CLI run instead (a 30 s Whisper load each)."""
  961. plat = platform_key()
  962. out = []
  963. for m in entries(local_only=True):
  964. pkg = EXTRA_PACKAGES.get(m.get("backend"))
  965. if pkg and runs_here(m, plat) and installed(m):
  966. out.append(pkg)
  967. return sorted(set(out))
  968. # ------------------------------------------------------------------ audio + VAD
  969. def audio_tracks(path: Path) -> int:
  970. info = subprocess.run([ffmpeg_bin(), "-nostdin", "-hide_banner", "-i", str(path)], capture_output=True, text=True,
  971. errors="replace").stderr
  972. return sum(1 for line in info.splitlines() if "Stream #" in line and "Audio:" in line)
  973. def load_audio(path: Path, track: int = 0) -> np.ndarray:
  974. raw = subprocess.run([ffmpeg_bin(), "-nostdin", "-loglevel", "error", "-i", str(path), "-map", f"0:a:{track}", "-ac", "1",
  975. "-af", "aresample=async=1000:first_pts=0", "-ar", str(SR), "-f", "s16le", "-"], capture_output=True).stdout
  976. if not raw:
  977. die(f"No audio track decoded from {path}")
  978. return np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768
  979. def read_wav(path: Path) -> np.ndarray:
  980. """A 16 kHz mono 16-bit WAV without ffmpeg (voice notes); anything else goes through load_audio."""
  981. try:
  982. with wave.open(str(path)) as w:
  983. if w.getframerate() == SR and w.getnchannels() == 1 and w.getsampwidth() == 2:
  984. return np.frombuffer(w.readframes(w.getnframes()), np.int16).astype(np.float32) / 32768
  985. except (wave.Error, EOFError, OSError):
  986. pass
  987. return load_audio(path)
  988. def make_vad(max_speech: float = 25):
  989. """Silero VAD as used everywhere (whole-file runs and live.py's streaming run share these settings)."""
  990. import sherpa_onnx
  991. cfg = sherpa_onnx.VadModelConfig()
  992. cfg.silero_vad.model = str(download("silero_vad.onnx"))
  993. cfg.silero_vad.threshold = 0.45
  994. cfg.silero_vad.min_silence_duration = 0.3
  995. cfg.silero_vad.min_speech_duration = 0.25
  996. cfg.silero_vad.max_speech_duration = max_speech
  997. cfg.sample_rate = SR
  998. cfg.num_threads = 1
  999. return sherpa_onnx.VoiceActivityDetector(cfg, buffer_size_in_seconds=60), cfg.silero_vad.window_size
  1000. def pad_segment(s: float, e: float, dur: float = float("inf")) -> tuple[float, float]:
  1001. """Pad a little so words at the edges are not clipped."""
  1002. return max(0.0, s - 0.15), min(dur, e + 0.2)
  1003. def vad_segments(samples: np.ndarray, max_speech: float = 25) -> list[tuple[float, float]]:
  1004. vad, win = make_vad(max_speech)
  1005. segs = []
  1006. def drain():
  1007. while not vad.empty():
  1008. seg = vad.front
  1009. segs.append((seg.start / SR, (seg.start + len(seg.samples)) / SR))
  1010. vad.pop()
  1011. for i in range(0, len(samples), win):
  1012. vad.accept_waveform(samples[i:i + win])
  1013. drain()
  1014. vad.flush()
  1015. drain()
  1016. dur = len(samples) / SR
  1017. return [pad_segment(s, e, dur) for s, e in segs]
  1018. def group_chunks(segs: list[tuple[float, float]], max_len: float, max_gap: float = 1.0) -> list[tuple[float, float]]:
  1019. """Merge neighbouring speech segments into contiguous windows (original timeline, no remapping needed)."""
  1020. chunks: list[list[float]] = []
  1021. for s, e in segs:
  1022. if chunks and s - chunks[-1][1] <= max_gap and e - chunks[-1][0] <= max_len:
  1023. chunks[-1][1] = e
  1024. else:
  1025. chunks.append([s, e])
  1026. return [(s, e) for s, e in chunks]
  1027. def pause_spans(samples: np.ndarray, max_len: float) -> list[tuple[float, float]]:
  1028. """The audio cut into contiguous spans of at most `max_len` s, each cut in the middle of a pause the VAD heard (a
  1029. hard cut only inside speech longer than that); spans with no speech in them are left out. Nothing at a span's edges
  1030. is trimmed: for a model that hears an utterance whole (Fun-ASR-Nano: its piece's first 0.18 s of silence cut off
  1031. turned 语音输入 into 语音水软, a 1.1 s VAD chunk of a 5.9 s piece came back as "Puritan")."""
  1032. dur = len(samples) / SR
  1033. segs = vad_segments(samples, max_speech=min(25, max_len))
  1034. if not segs:
  1035. return []
  1036. cuts = [(a[1] + b[0]) / 2 for a, b in zip(segs, segs[1:]) if b[0] > a[1]]
  1037. spans, start = [], 0.0
  1038. for i, c in enumerate(cuts + [dur]):
  1039. nxt = cuts[i + 1] if i + 1 < len(cuts) else dur
  1040. while c - start > max_len: # one stretch of speech longer than a span
  1041. spans.append((start, start + max_len))
  1042. start += max_len
  1043. if nxt - start > max_len and c > start:
  1044. spans.append((start, c))
  1045. start = c
  1046. if dur > start:
  1047. spans.append((start, dur))
  1048. return [(a, b) for a, b in spans if any(s < b and e > a for s, e in segs)]
  1049. def wav_bytes(samples: np.ndarray) -> bytes:
  1050. buf = io.BytesIO()
  1051. with wave.open(buf, "wb") as w:
  1052. w.setnchannels(1)
  1053. w.setsampwidth(2)
  1054. w.setframerate(SR)
  1055. w.writeframes((np.clip(samples, -1, 1) * 32767).astype(np.int16).tobytes())
  1056. return buf.getvalue()
  1057. # ------------------------------------------------------------------ backends
  1058. # each backend: transcribe(samples) -> list[(rel_start, rel_end, text)] or [(…, text, language)]
  1059. LANG_NAMES = {"zh": "Chinese", "en": "English", "yue": "Cantonese", "ja": "Japanese", "ko": "Korean", "es": "Spanish",
  1060. "fr": "French", "de": "German", "it": "Italian", "pt": "Portuguese", "ru": "Russian", "ar": "Arabic",
  1061. "hi": "Hindi", "th": "Thai", "vi": "Vietnamese", "id": "Indonesian", "ms": "Malay", "tr": "Turkish",
  1062. "nl": "Dutch", "sv": "Swedish", "da": "Danish", "fi": "Finnish", "pl": "Polish", "cs": "Czech",
  1063. "el": "Greek", "hu": "Hungarian", "ro": "Romanian", "fa": "Persian", "fil": "Filipino", "mk": "Macedonian"}
  1064. LANG_CODES = {v.lower(): k for k, v in LANG_NAMES.items()}
  1065. def lang_code(v) -> str | None:
  1066. """'<|zh|>' / 'Chinese' / 'zh' → 'zh'."""
  1067. if not v or not isinstance(v, str):
  1068. return None
  1069. v = v.strip().strip("<|>").strip()
  1070. return LANG_CODES.get(v.lower()) or (v.lower() if re.fullmatch(r"[a-z]{2,3}", v.lower()) else None)
  1071. _CJK = r"぀-ヿ㐀-鿿가-힯"
  1072. def syllables(text: str) -> int:
  1073. """Rough syllable count in any script: a CJK character, kana or Hangul block is one; elsewhere a vowel group."""
  1074. n = 0
  1075. for w in re.findall(r"\w+", unicodedata.normalize("NFKC", text)):
  1076. n += len(re.findall(f"[{_CJK}]", w))
  1077. n += sum(max(1, len(re.findall(r"[aeiouyàáâãäåæèéêëìíîïòóôõöøœùúûüýÿаеёиоуыэюя]+", p.lower())))
  1078. for p in re.sub(f"[{_CJK}]", " ", w).split())
  1079. return n
  1080. _FULLWIDTH = {",": ",", "?": "?", "!": "!", ":": ":", ";": ";"}
  1081. def cjk_punctuation(text: str) -> str:
  1082. """Whisper writes Chinese / Japanese with ASCII commas ("页面,输入"): use the full-width marks readers expect."""
  1083. return re.sub(f"(?<=[{_CJK}])([,?!:;])\\s*", lambda m: _FULLWIDTH[m.group(1)], text)
  1084. def _threads(args) -> int:
  1085. return getattr(args, "threads", None) or limits()["threads"]
  1086. def _model_dir(args, default: str) -> Path:
  1087. m = getattr(args, "model", None) or default
  1088. return Path(m) if os.path.isabs(m) else MODELS_DIR / m
  1089. class SenseVoice:
  1090. # SenseVoice degrades on long, mixed-language chunks (drops whole clauses); keep utterances short
  1091. max_chunk = 10
  1092. model = "sense-voice-small-int8"
  1093. # Non-autoregressive (one CTC pass): a piece decodes in one call of about 60–120 ms on the CPU (a 4 s piece:
  1094. # 59–84 ms, M-series), so there is nothing to stop halfway — an interim request (the live words of a voice note,
  1095. # a snap's note, dictation) is served whole after one look for a waiting request (NoteEngine.note).
  1096. quick_decode = True
  1097. def __init__(self, args):
  1098. import sherpa_onnx
  1099. lang = args.language if args.language in ("zh", "en", "ja", "ko", "yue") else "auto"
  1100. self.rec = sherpa_onnx.OfflineRecognizer.from_sense_voice(
  1101. model=str(download("sensevoice/model.int8.onnx")), tokens=str(download("sensevoice/tokens.txt")),
  1102. num_threads=_threads(args), use_itn=True, language=lang)
  1103. def transcribe(self, x):
  1104. st = self.rec.create_stream()
  1105. st.accept_waveform(SR, x)
  1106. self.rec.decode_stream(st)
  1107. return [(0.0, len(x) / SR, st.result.text.strip(), lang_code(getattr(st.result, "lang", None)))]
  1108. # No set_vocab: sherpa-onnx can't bias SenseVoice (greedy CTC only; create_stream(hotwords=…) aborts the process
  1109. # with "Only transducer models support contextual biasing"). Its words go to polish.
  1110. class Qwen3Onnx:
  1111. """Qwen3-ASR 0.6B int8 on the CPU through sherpa-onnx: 30 languages on any OS."""
  1112. max_chunk = 20
  1113. max_terms = dictation_vocab.MAX_TERMS # its hotwords
  1114. def __init__(self, args):
  1115. import sherpa_onnx
  1116. d = _model_dir(args, "qwen3-asr-0.6b-int8")
  1117. if not (d / "decoder.int8.onnx").exists():
  1118. raise Fail("Qwen3-ASR (CPU) is not downloaded: transcribe.py setup --model-id qwen3-asr-0.6b-onnx", "model_missing")
  1119. self.model = "qwen3-asr-0.6b-int8"
  1120. self.rec = sherpa_onnx.OfflineRecognizer.from_qwen3_asr(
  1121. conv_frontend=str(d / "conv_frontend.onnx"), encoder=str(d / "encoder.int8.onnx"),
  1122. decoder=str(d / "decoder.int8.onnx"), tokenizer=str(d / "tokenizer"), num_threads=_threads(args),
  1123. max_total_len=1024, max_new_tokens=400, hotwords=args.prompt or "")
  1124. self.hotwords = ""
  1125. def set_vocab(self, terms: list[dict]) -> bool:
  1126. """Per stream (no rebuild): "; "-joined — sherpa splits ASCII commas and re-joins them with spaces."""
  1127. self.hotwords = dictation_vocab.hotwords(terms)
  1128. return bool(self.hotwords)
  1129. def transcribe(self, x):
  1130. st = self.rec.create_stream()
  1131. if self.hotwords:
  1132. try:
  1133. st.set_option("hotwords", self.hotwords)
  1134. except Exception: # noqa: BLE001 — an older sherpa-onnx: no per-stream hotwords
  1135. pass
  1136. st.accept_waveform(SR, x)
  1137. self.rec.decode_stream(st)
  1138. return [(0.0, len(x) / SR, st.result.text.strip(), lang_code(getattr(st.result, "lang", None)))]
  1139. class FunAsrOnnx:
  1140. """Fun-ASR-Nano int8 on the CPU through sherpa-onnx: Windows' and Linux's dictation model for zh / en / ja
  1141. (docs/design/windows-models.md: 3.69 % on the founder's takes, RTF 0.12 on 4 Zen 4 threads). The export caps prompt
  1142. + audio at 512 tokens — about 25 s without hotwords — so pieces stay ≤ 20 s and the hotwords short; a longer
  1143. piece comes back empty. Hotwords are fixed when the recognizer is built, so a new Words list rebuilds it."""
  1144. max_chunk = 20
  1145. whole_spans = True # like FunAsrMlx: a piece is heard whole, never trimmed by the VAD
  1146. vad_gate = True # …but a piece with no speech in it isn't decoded (transcribe_track)
  1147. max_terms = 24 # each term costs prompt tokens out of the 512
  1148. app_hint = False # its hotwords are fixed when the recognizer is built (bias)
  1149. def __init__(self, args):
  1150. d = _model_dir(args, "fun-asr-nano-int8")
  1151. if not (d / "llm.int8.onnx").exists():
  1152. raise Fail("Fun-ASR-Nano (CPU) is not downloaded: transcribe.py setup --model-id fun-asr-nano-onnx",
  1153. "model_missing")
  1154. self.model = "fun-asr-nano-int8"
  1155. self.dir, self.threads, self.hotwords = d, _threads(args), ""
  1156. self.rec = self._build("")
  1157. # The decoder's own cap is 512 new tokens. Speech is under ~8 tokens a second (the Mac's FunAsrMlx says the same),
  1158. # so a piece of at most 20 s needs ~160: a repetition loop (it happens: the dogfood's 6 s of English, replayed on
  1159. # the VM, came back as 2,123 characters of one clause after 14.2 s of decoding) stops at this instead of running on
  1160. # to 512. The Mac caps each decode at 32 + 20 × its seconds; here the cap is fixed when the recognizer is built.
  1161. MAX_NEW_TOKENS = 32 + 8 * max_chunk
  1162. looped = False # the last transcribe cut a loop (the server's answer says so: the app won't send it)
  1163. # A transcript longer than this many characters a second of audio (+ LOOP_SLACK) is a loop, not speech: the
  1164. # fastest real dictation is ~5 CJK characters or ~18 Latin characters a second.
  1165. LOOP_CPS, LOOP_SLACK = 30, 40
  1166. def _build(self, hotwords: str):
  1167. import inspect
  1168. import sherpa_onnx
  1169. d = self.dir
  1170. kw = {}
  1171. if "max_new_tokens" in inspect.signature(sherpa_onnx.OfflineRecognizer.from_funasr_nano).parameters:
  1172. kw["max_new_tokens"] = self.MAX_NEW_TOKENS
  1173. return sherpa_onnx.OfflineRecognizer.from_funasr_nano(
  1174. encoder_adaptor=str(d / "encoder_adaptor.int8.onnx"), llm=str(d / "llm.int8.onnx"),
  1175. embedding=str(d / "embedding.int8.onnx"), tokenizer=str(d / "Qwen3-0.6B"), num_threads=self.threads,
  1176. hotwords=hotwords, **kw)
  1177. def set_vocab(self, terms: list[dict]) -> bool:
  1178. hw = dictation_vocab.hotwords(terms[: self.max_terms])
  1179. if hw != self.hotwords:
  1180. self.rec, self.hotwords = self._build(hw), hw
  1181. return bool(hw)
  1182. def _decode(self, x) -> tuple[str, str | None]:
  1183. st = self.rec.create_stream()
  1184. st.accept_waveform(SR, x)
  1185. self.rec.decode_stream(st)
  1186. return st.result.text.strip(), lang_code(getattr(st.result, "lang", None))
  1187. def transcribe(self, x):
  1188. dur = len(x) / SR
  1189. text, lang = self._decode(x)
  1190. cap = self.LOOP_CPS * dur + self.LOOP_SLACK
  1191. if len(text) > cap:
  1192. # a loop: the run repeated over and over is kept once; still too long → the piece heard again in two halves
  1193. # cut at a pause (a loop rarely starts again on a shorter piece), each half kept once in the same way
  1194. text = collapse_repeats(text)
  1195. if len(text) > cap and dur >= 2:
  1196. halves = pause_spans(x, max(1.0, dur / 2 + 0.5))
  1197. if len(halves) > 1:
  1198. parts = [collapse_repeats(self._decode(x[int(s * SR):int(e * SR)])[0]) for s, e in halves]
  1199. joined = ("" if lang in ("zh", "ja", "yue") else " ").join(p for p in parts if p).strip()
  1200. if len(joined) < len(text):
  1201. text = joined
  1202. self.looped = True
  1203. print(f"funasr: a repetition loop in {dur:.1f} s cut to {len(text)} characters", file=sys.stderr, flush=True)
  1204. return [(0.0, dur, text, lang)]
  1205. def collapse_repeats(text: str, min_unit: int = 4, times: int = 4, max_unit: int = 200) -> str:
  1206. """A decoder's repetition loop kept once: any run of the same span (≥ `min_unit` characters) said `times` or more
  1207. times in a row becomes one span; a cut-off copy at the very end goes too. Structural — it compares characters, it
  1208. reads no meaning — and only asked of a transcript already too long for its audio (FunAsrOnnx.transcribe)."""
  1209. n = len(text)
  1210. i = 0
  1211. out = []
  1212. while i < n:
  1213. best = None
  1214. for u in range(min_unit, min(max_unit, (n - i) // times) + 1):
  1215. unit = text[i:i + u]
  1216. k = 1
  1217. while text.startswith(unit, i + k * u):
  1218. k += 1
  1219. if k >= times:
  1220. best = (u, k)
  1221. break
  1222. if best:
  1223. u, k = best
  1224. out.append(text[i:i + u])
  1225. j = i + k * u
  1226. # the loop's last copy cut off by the token cap: a prefix of the span, at the end
  1227. if j < n and text[i:i + u].startswith(text[j:]):
  1228. j = n
  1229. i = j
  1230. else:
  1231. out.append(text[i])
  1232. i += 1
  1233. return "".join(out).strip()
  1234. class MlxBackend:
  1235. """MLX models. MLX streams belong to the thread that made them and some libraries keep the loading thread's
  1236. stream, while callers decode on worker threads (transcribe_track's pool, live.py): so the model is loaded and
  1237. every call runs on one thread of its own. The MLX memory cap is set before loading."""
  1238. max_chunk = 30
  1239. def __init__(self, args):
  1240. self._ex = ThreadPoolExecutor(1, thread_name_prefix="mlx")
  1241. self.args = args
  1242. self._ex.submit(self._init, args).result()
  1243. def _init(self, args):
  1244. cap_mlx_memory(limits()["max_memory_fraction"])
  1245. self.load(args)
  1246. def transcribe(self, x):
  1247. return self._ex.submit(self.decode, np.asarray(x, np.float32)).result()
  1248. def close(self):
  1249. try:
  1250. self._ex.submit(self.release).result(timeout=10) # before the next model loads, not racing it
  1251. except Exception: # noqa: BLE001 — a decode still running: the process ending frees it all the same
  1252. pass
  1253. self._ex.shutdown(wait=False)
  1254. def release(self):
  1255. """Drop this backend's model and MLX's buffer cache (on the MLX thread), so a replacement doesn't sit beside it."""
  1256. try:
  1257. cache = getattr(getattr(self, "holder", None), "_cache", None)
  1258. if cache is not None and hasattr(cache, "_data"):
  1259. cache._data.clear()
  1260. import mlx.core as mx
  1261. getattr(mx, "clear_cache", None) and mx.clear_cache()
  1262. except Exception: # noqa: BLE001 — best effort: the process ending frees it all the same
  1263. pass
  1264. class MlxWhisper(MlxBackend):
  1265. max_terms = dictation_vocab.MAX_TERMS # its prompt (~224 tokens)
  1266. def load(self, args):
  1267. try:
  1268. import mlx.core as mx
  1269. import mlx_whisper
  1270. from mlx_whisper.transcribe import ModelHolder
  1271. except ImportError:
  1272. raise Fail("mlx-whisper not installed (uv run --with mlx-whisper …)", "deps", 4)
  1273. self.m, self.mx = mlx_whisper, mx
  1274. self.repo = self.model = args.model or "mlx-community/whisper-large-v3-turbo"
  1275. self.net = ModelHolder.get_model(self.repo, mx.float16) # load now, not inside the first chunk
  1276. self.langs = [] if args.language else user_languages()
  1277. self.base_prompt = args.prompt
  1278. def set_vocab(self, terms: list[dict]) -> bool:
  1279. """initial_prompt per call: a plain comma list (dictation eval: no insertions, no loops)."""
  1280. self.args.prompt = dictation_vocab.whisper_prompt(terms) or self.base_prompt
  1281. return bool(terms)
  1282. def decode(self, x):
  1283. out = self.whisper(x, self.args.language)
  1284. # Too little text for the speech (normal speech is 4–7 syllables a second in every language we measured):
  1285. # for someone who speaks several languages, the chunk probably switched language and Whisper dropped a part.
  1286. if len(self.langs) > 1 and len(x) > 4 * SR and syllables(" ".join(r[2] for r in out)) < 2.5 * len(x) / SR:
  1287. try:
  1288. runs = self.language_runs(x)
  1289. except Exception as e: # noqa: BLE001 — an MLX / library change must not cost the chunk
  1290. print(f"language split skipped: {type(e).__name__}: {e}", file=sys.stderr, flush=True)
  1291. runs = []
  1292. if len(runs) > 1:
  1293. return [(s + a, s + b, t, lg) for s, e, lang in runs
  1294. for a, b, t, lg in self.whisper(x[int(s * SR):int(e * SR)], lang)]
  1295. return out
  1296. def whisper(self, x, language):
  1297. r = self.m.transcribe(x, path_or_hf_repo=self.repo, language=language, initial_prompt=self.args.prompt,
  1298. condition_on_previous_text=False, verbose=None)
  1299. lang = r.get("language")
  1300. return [(s["start"], s["end"], cjk_punctuation(s["text"].strip()), lang) for s in r.get("segments", [])]
  1301. def language_runs(self, x) -> list[tuple[float, float, str]]:
  1302. """Whisper keeps one language per window: a chunk where one sentence is English and the next Chinese comes
  1303. back as Chinese only (asr-benchmark.md). Find the language of each speech segment in the chunk and decode runs
  1304. of one language apart, each with its language set."""
  1305. segs = vad_segments(x)
  1306. if len(segs) < 2:
  1307. return []
  1308. from mlx_whisper.audio import N_FRAMES, log_mel_spectrogram, pad_or_trim
  1309. from mlx_whisper.decoding import detect_language
  1310. runs: list[list] = []
  1311. for s, e in segs: # one at a time: a batch of 30 s mel windows costs a GB per segment
  1312. mel = pad_or_trim(log_mel_spectrogram(x[int(s * SR):int(e * SR)], n_mels=self.net.dims.n_mels), N_FRAMES,
  1313. axis=-2).astype(self.mx.float16)
  1314. p = detect_language(self.net, mel)[1] # one 2-D mel in → one dict out
  1315. p = p[0] if isinstance(p, list) else p
  1316. lang = max(p, key=p.get)
  1317. if runs and runs[-1][2] == lang:
  1318. runs[-1][1] = e
  1319. else:
  1320. runs.append([s, e, lang])
  1321. return [(s, e, lang) for s, e, lang in runs]
  1322. class Qwen3Mlx(MlxBackend):
  1323. """Qwen3-ASR 0.6B on MLX (mlx-qwen3-asr): ~0.18 s for a 6 s dictation on an M5 Max, and the only fast local
  1324. model here that takes a vocabulary per request (`context`) — dictation's default on Apple Silicon.
  1325. Interruptible: with `stop_check` set (an interim request), the decode looks at it before each stage and each
  1326. token and raises Interrupted — the tag-aware decoder below is a loop of ours, so it can stop between steps."""
  1327. stop_check = None
  1328. @property
  1329. def interruptible(self) -> bool:
  1330. return self.tagged
  1331. def _check(self):
  1332. if self.stop_check is not None and self.stop_check():
  1333. raise Interrupted("another request is waiting")
  1334. def load(self, args):
  1335. try:
  1336. import mlx.core as mx
  1337. import mlx_qwen3_asr
  1338. from mlx_qwen3_asr.load_models import _ModelHolder
  1339. except ImportError:
  1340. raise Fail("mlx-qwen3-asr not installed (uv run --with mlx-qwen3-asr …)", "deps", 4)
  1341. self.m, self.mx = mlx_qwen3_asr, mx
  1342. self.repo = self.model = args.model or "mlx-community/Qwen3-ASR-0.6B-8bit"
  1343. # the package keeps up to 4 loaded models in its own LRU: a server that switched models (dictation ↔ a voice
  1344. # note's pick) held both — about 1 GB more than the one it uses. One is all a backend ever needs.
  1345. cache = getattr(_ModelHolder, "_cache", None)
  1346. if cache is not None and hasattr(cache, "set_max_entries"):
  1347. cache.set_max_entries(1)
  1348. self.holder = _ModelHolder
  1349. _ModelHolder.get(self.repo, dtype=mx.float16) # load now, not inside the first chunk
  1350. self.context = args.prompt or ""
  1351. self.base_context = self.context
  1352. self.names: list[str] = []
  1353. self.prior = self.field = ""
  1354. # The language the user mixes English into (docs/design/dictation.md §Code-switching): Qwen3-ASR writes its
  1355. # whole answer in the language of the tag it decodes first, so a Chinese sentence full of English names that
  1356. # it tags English comes back translated (和 → "and", Musk 的 → "Musk's"). None = leave the tag to the model.
  1357. self.prefer = None if args.language else switch_language(user_languages())
  1358. self.tagged = True # the tag-aware decoder below works with this mlx-qwen3-asr; False after it failed
  1359. self.last: dict = {}
  1360. def set_vocab(self, terms: list[dict]) -> bool:
  1361. self.context = dictation_vocab.qwen_context(terms) or self.base_context
  1362. self.names = [t["term"] for t in terms[:dictation_vocab.QWEN_MAX_TERMS]]
  1363. return bool(terms)
  1364. def set_prior(self, text: str) -> None:
  1365. """A streamed dictation's text so far: after the names, in the same system prompt (docs/design/dictation.md
  1366. §Streaming), so a name is spelled the same way in every piece."""
  1367. self.prior = (text or "").strip()
  1368. def set_field(self, text: str) -> None:
  1369. """The text before the cursor in the field the dictation goes into (docs/design/dictation.md §Field context):
  1370. after the names and before the text so far, in the same system prompt — what the speech is likely about
  1371. (截了 when the field said 我刚截了; blurt, Typeless spelled as the conversation spells them). Never in the
  1372. language tag's prompt (that stays the text so far: tag_context)."""
  1373. self.field = (text or "").strip()
  1374. def decode(self, x):
  1375. prior = getattr(self, "prior", "")
  1376. # the field's text, then the text so far: both after the names
  1377. said = "\n".join(t for t in (getattr(self, "field", ""), prior) if t)
  1378. ctx = (self.context + "\n" + said).strip() if said else self.context
  1379. text, lang = self._decode(x, ctx, prior)
  1380. # The model now and then answers with the word list itself (seen on a noisy piece it couldn't make out:
  1381. # "Claude Code、Codex、blurt、…"): once more with only the text so far (and the field's) as context.
  1382. dump = False
  1383. if self.names and ctx != said and dictation_vocab.context_dump(text, self.names):
  1384. text, lang = self._decode(x, said, prior)
  1385. dump = True
  1386. # The text so far (or the field's) written out again as this piece's words (dict023): once more without it,
  1387. # and the shorter answer is the piece when it is clearly shorter — a speaker who really repeats themself gets
  1388. # the same text.
  1389. echo = False
  1390. if said and echoes_prior(text, said, len(x) / SR):
  1391. redo, lang2 = self._decode(x, self.context, prior)
  1392. if redo.strip() and len(redo) < 0.8 * len(text):
  1393. text, lang, echo = redo, lang2, True
  1394. if dump:
  1395. self.last["context_dump"] = True
  1396. if echo:
  1397. self.last["echo"] = True
  1398. return [(0.0, len(x) / SR, text, lang)]
  1399. def _decode(self, x, ctx: str, prior: str) -> tuple[str, str | None]:
  1400. prefer = self.prefer or ("Chinese" if not self.args.language and dictation_vocab.cjk_share(prior) >= 0.3 else None)
  1401. self.last = {}
  1402. # the language tag is read from the prompt without the Words (only the text so far): see _decode_tagged
  1403. self.tag_context = prior
  1404. # an interim request goes through our own loop even without a preferred tag: it has to be able to stop
  1405. if (prefer or self.stop_check is not None) and self.tagged:
  1406. try:
  1407. return self._decode_tagged(x, ctx, prefer)
  1408. except Interrupted:
  1409. raise
  1410. except Exception as e: # noqa: BLE001 — another mlx-qwen3-asr: its own transcribe, as before
  1411. self.tagged = False
  1412. print(f"qwen3-asr: tag-aware decode unavailable ({type(e).__name__}: {e}); plain decode", file=sys.stderr, flush=True)
  1413. r = self.m.transcribe(x, model=self.repo, context=ctx, language=self.args.language)
  1414. return (r.text or "").strip(), lang_code(getattr(r, "language", None))
  1415. def _decode_tagged(self, x, ctx: str, prefer: str | None) -> tuple[str, str | None]:
  1416. """mlx-qwen3-asr's decode, with the language tag chosen here (docs/design/dictation.md §Code-switching).
  1417. The tag comes from a prompt **without the Words** (`tag_context`: only the dictation's text so far): the
  1418. prompt is prefilled up to `language`, the model's probabilities for English and `prefer` are read there, and
  1419. `prefer` wins unless English is all but certain (dictation_vocab.choose_tag, whose threshold was measured
  1420. without a word list). The text is then decoded with the full context (the Words, which bias the spelling) under
  1421. that tag. The Words framed as an English sentence pushed the tag toward English and made it depend on their
  1422. order: one app-captured mixed sentence went from p_en 0.948 (no Words) to 0.9956 — over the 0.995 line, a clean
  1423. English translation typed — and back under it with the same words in another order (v0.9 dogfood).
  1424. English that still has Chinese characters in it (translated around a name) is decoded once more under
  1425. `prefer`. Cost: one more prefill of the prompt (tens of ms), only when the Words are in the context.
  1426. prefer None (an interim request with no preferred language): the model's own tag under the full context, as
  1427. the plain decode does. `self.last`: the tag chosen, its probabilities with and without the Words, and the
  1428. margin to the threshold (the history and the Lab show it)."""
  1429. mx = self.mx
  1430. from mlx_qwen3_asr.audio import compute_features
  1431. from mlx_qwen3_asr.chunking import split_audio_into_chunks
  1432. from mlx_qwen3_asr.generate import GenerationConfig, detect_repetition, resolve_max_new_tokens
  1433. from mlx_qwen3_asr.load_models import _ModelHolder
  1434. from mlx_qwen3_asr.tokenizer import _TokenizerHolder, parse_asr_output
  1435. model, _ = _ModelHolder.get(self.repo, dtype=mx.float16)
  1436. tok = _TokenizerHolder.get(_ModelHolder.get_resolved_path(self.repo, dtype=mx.float16))
  1437. eos = set(GenerationConfig().eos_token_ids)
  1438. head = tok.encode("language")
  1439. tag_ids = {name: tok.encode(" " + name)[0] for name in (prefer, "English") if name}
  1440. tag_ctx = (getattr(self, "tag_context", "") or "").strip()
  1441. def positions(a: int, b: int):
  1442. p = mx.arange(a, b)[None, :]
  1443. return mx.stack([p, p, p], axis=1)
  1444. def prefill(af, budget: int, context: str):
  1445. ids = tok.build_prompt_tokens(n_audio_tokens=af.shape[1], language=None, context=context) + head
  1446. self._check()
  1447. cache = model.create_cache(max_seq_len=len(ids) + budget + 8)
  1448. logits = model.prefill(input_ids=mx.array([ids]), audio_features=af, position_ids=positions(0, len(ids)), cache=cache)
  1449. return logits, cache, len(ids)
  1450. def tag_probs(logits) -> tuple[float, float, bool]:
  1451. lp = logits.reshape(-1).astype(mx.float32)
  1452. lp = lp - mx.logsumexp(lp)
  1453. top = int(mx.argmax(lp))
  1454. return float(mx.exp(lp[tag_ids["English"]])), float(mx.exp(lp[tag_ids[prefer]])), top in tag_ids.values()
  1455. def run(af, budget: int, tag: str | None, state=None) -> tuple[str, str | None, dict]:
  1456. """The text under `tag` (None: the model's own tag), from the prompt with the full context."""
  1457. logits, cache, pos = state or prefill(af, budget, ctx)
  1458. info: dict = {}
  1459. if prefer and state is None and ctx.strip() != tag_ctx:
  1460. # what the Words alone would have tagged (only shown: the tag was picked without them)
  1461. p_en, p_pref, _ = tag_probs(logits)
  1462. info["p_en_words"] = round(p_en, 4)
  1463. if tag is None: # the model's own tag (no speech, another language): decode on from it, as before
  1464. suffix = [int(mx.argmax(logits.reshape(-1)))]
  1465. else:
  1466. suffix = tok.encode(f" {tag}<asr_text>")
  1467. logits = model.step_many(input_ids=mx.array([suffix]), position_ids=positions(pos, pos + len(suffix)), cache=cache)
  1468. pos += len(suffix)
  1469. token = int(mx.argmax(logits[0, -1]))
  1470. out = [token]
  1471. for _ in range(1, budget):
  1472. if token in eos or detect_repetition(out):
  1473. break
  1474. self._check()
  1475. logits = model.step(input_ids=mx.array([[token]]), position_ids=positions(pos, pos + 1), cache=cache,
  1476. validate_input_ids=False)
  1477. pos += 1
  1478. token = int(mx.argmax(logits.reshape(-1)))
  1479. out.append(token)
  1480. out = [t for t in out if t not in eos]
  1481. raw = tok.decode((head + suffix + out) if tag is None else out)
  1482. if tag is None:
  1483. lang, text = parse_asr_output(raw)
  1484. else:
  1485. lang, text = tag, parse_asr_output(raw, user_language=tag)[1]
  1486. return (text or "").strip(), lang, info
  1487. def pick(af, budget: int):
  1488. """The tag from the prompt without the Words → (tag or None, info, the prefill when it is the decode's own)."""
  1489. state = prefill(af, budget, tag_ctx)
  1490. p_en, p_pref, top_is_tag = tag_probs(state[0])
  1491. info = {"p_en": round(p_en, 4), f"p_{lang_code(prefer) or prefer}": round(p_pref, 4),
  1492. "margin": dictation_vocab.tag_margin(p_en, p_pref)}
  1493. tag = dictation_vocab.choose_tag(p_en, p_pref, prefer) if top_is_tag else None
  1494. return tag, info, (state if ctx.strip() == tag_ctx else None)
  1495. texts, langs, last = [], [], {}
  1496. for chunk, _off in split_audio_into_chunks(np.asarray(x, np.float32), sr=SR):
  1497. self._check()
  1498. mel, fl = compute_features(chunk)
  1499. af, _ = model.audio_tower(mel.astype(mx.float16), fl)
  1500. budget = resolve_max_new_tokens(None, audio_duration_sec=len(chunk) / SR)
  1501. if prefer:
  1502. tag, info, state = pick(af, budget)
  1503. text, lang, more = run(af, budget, tag, state)
  1504. info.update(more)
  1505. else:
  1506. (text, lang, info), tag = run(af, budget, None), None
  1507. if prefer and lang == "English" and dictation_vocab.translated_glue(text, self.names):
  1508. text2, _, _ = run(af, budget, prefer)
  1509. if dictation_vocab.cjk_share(text2) > 0:
  1510. text, lang = text2, prefer
  1511. info["redone"] = True
  1512. info["tag"] = lang
  1513. last = info
  1514. texts.append(text)
  1515. langs.append(lang)
  1516. self.last = last
  1517. lang = next((l for l in langs if l), None)
  1518. return ("" if lang in ("Chinese", "Japanese", "Cantonese") else " ").join(t for t in texts if t).strip(), lang_code(lang)
  1519. class FunAsrMlx(MlxBackend):
  1520. """Fun-ASR-Nano (Alibaba, 0.8B: SenseVoice's encoder + Qwen3-0.6B) on MLX (mlx-audio). On the founder's 95 takes
  1521. 2.84 % errors against the 0.6B's 3.34 % (2.92 % through the app's path), the best local model on Chinese with
  1522. English inside it (ASCEND 6.6 %), and ~0.1 s after key-up (docs/design/dictation-eval-2026-10-31.md §2.1, §3, §5;
  1523. docs/design/dictation.md §Defaults by Mac). Chinese, English and Japanese only
  1524. (asr_models.json `languages`). The Words go in as hotwords; it takes no other context (no text so far, no field
  1525. text: it wasn't measured with any). One whole utterance per call, ≤ 30 s (max_chunk). Interruptible: the decode is
  1526. our loop over mlx-audio's tokens, so an interim request stops between tokens."""
  1527. stop_check = None
  1528. interruptible = True
  1529. whole_spans = True # never VAD-trimmed (transcribe_track: pause_spans)
  1530. max_terms = dictation_vocab.QWEN_MAX_TERMS # its hotword list goes into the prompt like Qwen3-ASR's context
  1531. LANGS = {"zh": "zh", "yue": "zh", "en": "en", "ja": "ja"} # what its prompt can name; anything else: automatic
  1532. EOS = (151643, 151645)
  1533. def _check(self):
  1534. if self.stop_check is not None and self.stop_check():
  1535. raise Interrupted("another request is waiting")
  1536. def load(self, args):
  1537. try:
  1538. import mlx.core as mx
  1539. from mlx_audio.stt import load as load_stt
  1540. except ImportError:
  1541. raise Fail("mlx-audio not installed (uv run --with mlx-audio …)", "deps", 4)
  1542. self.mx = mx
  1543. self.repo = self.model = args.model or "mlx-community/Fun-ASR-Nano-2512"
  1544. self.net = load_stt(self.repo)
  1545. self.language = self.LANGS.get((args.language or "").lower())
  1546. self.hotwords: list[str] = []
  1547. self.last: dict = {}
  1548. def release(self):
  1549. self.net = None
  1550. super().release()
  1551. def set_vocab(self, terms: list[dict]) -> bool:
  1552. self.hotwords = [t["term"] for t in terms[:self.max_terms] if str(t.get("term") or "").strip()]
  1553. return bool(self.hotwords)
  1554. def decode(self, x):
  1555. self.last = {}
  1556. text = self._decode(x, self.hotwords)
  1557. # the hotword list written out as the answer (a noisy piece it couldn't make out): once more without it
  1558. if self.hotwords and dictation_vocab.context_dump(text, self.hotwords):
  1559. text = self._decode(x, [])
  1560. self.last["context_dump"] = True
  1561. return [(0.0, len(x) / SR, text, self._lang(text))]
  1562. def _decode(self, x, hotwords: list[str]) -> str:
  1563. mx, net = self.mx, self.net
  1564. from mlx_audio.lm.generate import generate_step
  1565. self._check()
  1566. ids, embeds = net._build_inputs_embeds(mx.array(np.asarray(x, np.float32)), language=self.language,
  1567. hotwords=hotwords or None, itn=True)
  1568. # a loop, if it ever starts, stops well before the model's 512: speech is under ~8 tokens a second
  1569. budget = min(512, 32 + int(20 * len(x) / SR))
  1570. out: list[int] = []
  1571. for token, _ in generate_step(prompt=ids[0], input_embeddings=embeds[0], model=net, max_tokens=budget):
  1572. t = int(token)
  1573. if t in self.EOS:
  1574. break
  1575. out.append(t)
  1576. self._check()
  1577. text = net._tokenizer.decode(out, skip_special_tokens=True)
  1578. mx.clear_cache()
  1579. return cjk_punctuation(re.sub(r"\s+", " ", text.replace("/sil", " ")).strip())
  1580. @staticmethod
  1581. def _lang(text: str) -> str | None:
  1582. """It names no language: kana → ja, any Chinese character → zh, Latin letters only → en."""
  1583. if not text:
  1584. return None
  1585. if _KANA.search(text):
  1586. return "ja"
  1587. return "zh" if re.search(r"[㐀-鿿]", text) else "en" if re.search(r"[A-Za-z]", text) else None
  1588. class FasterWhisper:
  1589. max_chunk = 30
  1590. def __init__(self, args):
  1591. try:
  1592. from faster_whisper import WhisperModel
  1593. except ImportError:
  1594. raise Fail("faster-whisper not installed (uv run --with faster-whisper …)", "deps", 4)
  1595. self.args = args
  1596. name = args.model or "mobiuslabsgmbh/faster-whisper-large-v3-turbo"
  1597. if name == "large-v3-turbo":
  1598. name = "mobiuslabsgmbh/faster-whisper-large-v3-turbo"
  1599. m = entry_for("faster-whisper", name)
  1600. local = model_path(m) if m else None
  1601. self.model = name
  1602. self.net = WhisperModel(str(local) if local else name, device="auto", compute_type="auto", cpu_threads=_threads(args))
  1603. def transcribe(self, x):
  1604. segs, info = self.net.transcribe(x, language=self.args.language, initial_prompt=self.args.prompt,
  1605. condition_on_previous_text=False, vad_filter=False)
  1606. return [(s.start, s.end, cjk_punctuation(s.text.strip()), info.language) for s in segs]
  1607. class AppleSpeech:
  1608. """macOS's recogniser through the Blurt executable: the whole audio at once (it finds its own pauses and gives
  1609. each phrase its time), so no VAD — nothing for a first run to download."""
  1610. max_chunk = 3600
  1611. no_vad = True
  1612. def __init__(self, args):
  1613. self.helper = apple_speech_helper()
  1614. if not self.helper or not apple_speech_ok():
  1615. raise Fail("macOS speech recognition needs macOS 26 and the Blurt app", "model_missing")
  1616. self.args = args
  1617. self.terms: list[str] = []
  1618. self.model = "SpeechTranscriber"
  1619. def set_vocab(self, terms: list[dict]) -> bool:
  1620. self.terms = [t["term"] for t in terms if t.get("term")]
  1621. return bool(self.terms)
  1622. def transcribe(self, x):
  1623. loc = apple_locale(self.args.language)
  1624. if not loc:
  1625. raise Fail("macOS speech recognition doesn't speak this language", "unsupported_language")
  1626. with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
  1627. f.write(wav_bytes(x))
  1628. path = f.name
  1629. try:
  1630. cmd = [str(self.helper), "--speech", path, "--locale", loc]
  1631. if self.terms:
  1632. cmd += ["--words", "\n".join(self.terms)]
  1633. r = subprocess.run(cmd, capture_output=True, text=True, timeout=120 + len(x) / SR)
  1634. finally:
  1635. os.unlink(path)
  1636. line = next((ln for ln in reversed(r.stdout.splitlines()) if ln.startswith("{")), "{}")
  1637. data = json.loads(line)
  1638. if not data.get("ok"):
  1639. raise Fail(data.get("error") or "macOS speech recognition failed", data.get("code") or "error")
  1640. lang = loc.split("-")[0]
  1641. return [(s["start"], s["end"], s["text"], lang) for s in data.get("segments") or []]
  1642. # Cloud recognition can hear the audio sped up (asr.cloud.speed, every provider; docs/design/first-minute-speech.md
  1643. # §2.4): billing is per second of audio sent. Measured with DashScope qwen3-asr-flash on 270 kept takes: 1x 6.8 % (full
  1644. # cost), 1.5x 8.0 % for 64 % of the cost, 2x 11.6 % for 47 % — and twice it answered with the word list instead of the
  1645. # speech. Since dictation sends the whole take (2026-10-31, docs/design/dictation.md §Cloud recognition) the default is
  1646. # 1x: on the founder's 95 takes, whole with the words and the field's text, 2.02 % at 1x against 3.32 % at 1.5x.
  1647. # Pitch is kept (ffmpeg atempo); the times that come back are put back on the original timeline.
  1648. CLOUD_SPEEDS = (1.0, 1.5, 2.0)
  1649. CLOUD_SPEED_DEFAULT = 1.0
  1650. def cloud_speed(cfg: dict | None = None) -> float:
  1651. v = os.environ.get("BLURT_ASR_SPEED") or ((cfg if cfg is not None else asr_config()).get("cloud") or {}).get("speed")
  1652. try:
  1653. v = float(v) if v is not None else CLOUD_SPEED_DEFAULT
  1654. except (TypeError, ValueError):
  1655. v = CLOUD_SPEED_DEFAULT
  1656. return min(CLOUD_SPEEDS, key=lambda s: abs(s - v))
  1657. def speed_up(x: np.ndarray, speed: float) -> np.ndarray:
  1658. """x played `speed` times as fast, same pitch (ffmpeg atempo); x itself when 1x or ffmpeg isn't there."""
  1659. if speed == 1.0 or not len(x) or not shutil.which("ffmpeg"):
  1660. return x
  1661. pcm = (np.clip(x, -1, 1) * 32767).astype(np.int16).tobytes()
  1662. r = subprocess.run([shutil.which("ffmpeg"), "-nostdin", "-loglevel", "error", "-f", "s16le", "-ar", str(SR), "-ac", "1",
  1663. "-i", "-", "-filter:a", f"atempo={speed}", "-f", "s16le", "-ar", str(SR), "-ac", "1", "-"],
  1664. input=pcm, capture_output=True)
  1665. if r.returncode or not r.stdout:
  1666. return x
  1667. return np.frombuffer(r.stdout, dtype=np.int16).astype(np.float32) / 32768
  1668. def api_key(*names: str) -> str | None:
  1669. """The first of these variables that is set, trimmed. A key with a character no API key has (2026-10-31: two
  1670. "ç" — what ⌥C types — saved after a Model Studio key; the HTTP header can't carry them, and every take failed as
  1671. a bare "error") is an "auth" failure that names the characters, never the key."""
  1672. for n in names:
  1673. k = (os.environ.get(n) or "").strip()
  1674. if not k:
  1675. continue
  1676. bad = [c for c in dict.fromkeys(k) if not (c.isascii() and c.isprintable()) or c.isspace()]
  1677. if bad:
  1678. raise Fail(f"the API key in {n} has characters a key can't have "
  1679. f"({', '.join(f'U+{ord(c):04X}' for c in bad)}): paste the key again", "auth")
  1680. return k
  1681. return None
  1682. class OpenAICompat:
  1683. """POST {base}/audio/transcriptions. verbose_json gives segment timestamps where supported."""
  1684. retries, timeout = 4, 120.0
  1685. def __init__(self, args):
  1686. for name, default_base in (("BLURT_ASR_API_KEY", "https://api.openai.com/v1"),
  1687. ("OPENAI_API_KEY", "https://api.openai.com/v1"),
  1688. ("GROQ_API_KEY", "https://api.groq.com/openai/v1")):
  1689. self.key = api_key(name)
  1690. if self.key:
  1691. break
  1692. else:
  1693. raise Fail("no API key: set BLURT_ASR_API_KEY (or OPENAI_API_KEY / GROQ_API_KEY)", "auth")
  1694. self.base = (os.environ.get("BLURT_ASR_BASE_URL") or default_base).rstrip("/")
  1695. self.model = args.model or os.environ.get("BLURT_ASR_MODEL") or (
  1696. "whisper-large-v3-turbo" if "groq" in self.base else "whisper-1")
  1697. self.verbose = not self.model.startswith("gpt-4o") # gpt-4o-*-transcribe has no segment timestamps
  1698. # Longer chunks = fewer requests (Groq bills >=10s per request); shorter when no timestamps come back.
  1699. self.max_chunk = 60 if self.verbose else 20
  1700. self.args = args
  1701. self.speed = cloud_speed()
  1702. self.terms: list[str] = []
  1703. def set_vocab(self, terms: list[dict]) -> bool:
  1704. """The dictation's words, in the prompt (Whisper-style services read it as preceding text)."""
  1705. self.terms = [t["term"] for t in terms if t.get("term")]
  1706. return bool(self.terms)
  1707. def transcribe(self, x):
  1708. fields = {"model": self.model, "response_format": "verbose_json" if self.verbose else "json"}
  1709. if self.args.language:
  1710. fields["language"] = self.args.language
  1711. prompt = self.args.prompt or (", ".join(self.terms) if self.terms else None)
  1712. if prompt:
  1713. fields["prompt"] = prompt
  1714. boundary = uuid.uuid4().hex
  1715. body = io.BytesIO()
  1716. for k, v in fields.items():
  1717. body.write(f'--{boundary}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
  1718. body.write(f'--{boundary}\r\nContent-Disposition: form-data; name="file"; filename="a.wav"\r\n'
  1719. f'Content-Type: audio/wav\r\n\r\n'.encode())
  1720. body.write(wav_bytes(speed_up(x, self.speed)))
  1721. body.write(f"\r\n--{boundary}--\r\n".encode())
  1722. data = _post(f"{self.base}/audio/transcriptions", body.getvalue(),
  1723. {"Authorization": f"Bearer {self.key}", "Content-Type": f"multipart/form-data; boundary={boundary}"},
  1724. retries=self.retries, timeout=self.timeout)
  1725. lang = lang_code(data.get("language"))
  1726. if data.get("segments"):
  1727. k = self.speed
  1728. return [(s["start"] * k, min(s["end"] * k, len(x) / SR), s["text"].strip(), lang) for s in data["segments"]]
  1729. return [(0.0, len(x) / SR, (data.get("text") or "").strip(), lang)]
  1730. # Model Studio (阿里云百炼) speech models take one of two HTTP APIs, and Blurt learns which from the service's answers,
  1731. # not from the id (measured 2026-11-11 with a spoken clip: docs/design/dictation.md §Cloud recognition). Qwen3-ASR Flash
  1732. # takes the OpenAI-compatible chat/completions; qwen-audio 3.x ASR Flash and Fun-ASR Flash answer 404
  1733. # model_not_supported or 400 there and work on Model Studio's own API. A model that only streams (…-realtime) or reads a
  1734. # file from a URL (…-filetrans, Paraformer, Fun-ASR) gives text on neither: a `model` failure, with both answers.
  1735. # The words as hot words: weight 1–5, or 50 — a "super" hot word, at most 50 of them. On the founder's 95 takes
  1736. # (2026-11-11, qwen-audio-3.1-asr-flash, words + field) 4 or 5 changed almost nothing (Blurt 1 of 9 right, 1.89 %); 50
  1737. # got Blurt 8 of 9 and 1.60 % (Qwen3-ASR Flash the same day: 2.15 %). The words past the 50th go at 5.
  1738. DASHSCOPE_HOTWORD_WEIGHT = 50
  1739. DASHSCOPE_SUPER_WORDS = 50
  1740. # models known to take only the own API — measured, so the first request of a server doesn't go to the compatible API
  1741. # for its 404 (2.5–31 s that day); any other model the server learns once (_ask)
  1742. _DASHSCOPE_OWN_API: set[str] = {"qwen-audio-3.1-asr-flash"}
  1743. def dashscope_native_url(base: str) -> str:
  1744. """The OpenAI-compatible base (…/compatible-mode/v1, the one Settings keeps) → the same host's own API."""
  1745. b = base.rstrip("/")
  1746. if b.endswith("/compatible-mode/v1"):
  1747. b = b[: -len("/compatible-mode/v1")]
  1748. else:
  1749. u = urllib.parse.urlsplit(b)
  1750. b = f"{u.scheme}://{u.netloc}"
  1751. return b + "/api/v1/services/aigc/multimodal-generation/generation"
  1752. class DashScope:
  1753. # A dictation goes here whole at key-up (docs/design/dictation.md §Cloud recognition): one request up to 50 s (the
  1754. # eval's whole-take requests, dictation-eval-2026-10-31.md §2.1), a longer take cut at its pauses into ≤ 50 s parts
  1755. # sent side by side (transcribe_track, CLOUD_WORKERS).
  1756. max_chunk = 50
  1757. retries, timeout = 4, 120.0
  1758. def __init__(self, args):
  1759. self.key = api_key("DASHSCOPE_API_KEY")
  1760. if not self.key:
  1761. raise Fail("no API key: set DASHSCOPE_API_KEY", "auth")
  1762. # its own variable: BLURT_ASR_BASE_URL is the OpenAI-compatible service's (both can be set in the app)
  1763. self.base = (os.environ.get("DASHSCOPE_BASE_URL") or "https://dashscope.aliyuncs.com/compatible-mode/v1").rstrip("/")
  1764. # the model picked in Settings (CloudSpeech: DASHSCOPE_ASR_MODEL), else the registry's
  1765. self.model = os.environ.get("DASHSCOPE_ASR_MODEL") or args.model or "qwen3-asr-flash"
  1766. self.args = args
  1767. self.speed = cloud_speed()
  1768. self.terms: list[str] = []
  1769. self.field = ""
  1770. self.last: dict = {}
  1771. def set_vocab(self, terms: list[dict]) -> bool:
  1772. """The dictation's words as Qwen3-ASR's context (measured: 27 of 35 terms with them, §2.4)."""
  1773. self.terms = [t["term"] for t in terms if t.get("term")]
  1774. return bool(self.terms)
  1775. def set_field(self, text: str) -> None:
  1776. """The field's text before the cursor, after the words in the same context (Qwen3-ASR Flash reads background
  1777. text there as well as a word list: Model Studio's "context enhancement"; docs/design/dictation.md §Field
  1778. context)."""
  1779. self.field = (text or "").strip()
  1780. def context_text(self, words: bool = True, field: bool = True) -> str:
  1781. w = (self.args.prompt or ", ".join(self.terms)) if words else ""
  1782. return "\n".join(t for t in (w, getattr(self, "field", "") if field else "") if t)
  1783. def transcribe(self, x):
  1784. """The same nets as the MLX backend's decode (Qwen3Mlx.decode), for the same model family: the field's text
  1785. before the cursor written out as the speech (13 of 95 of the founder's takes when the app sent short pieces
  1786. with it: dictation-eval-2026-10-31.md §4.1) and the word list answered instead of the speech (3 of 95). An
  1787. answer that repeats the field (`echoes_prior`) is asked once more with the words only, and the shorter answer is
  1788. kept when it is clearly shorter; one that is the word list, once more with the field only. Costs a second
  1789. request only when a net fires (`last`: echo / context_dump, the answer's `tag`; NoteEngine empties it per note,
  1790. not per part: a long take's parts run side by side)."""
  1791. field = getattr(self, "field", "")
  1792. text = self._ask(x)
  1793. if self.terms and not self.args.prompt and dictation_vocab.context_dump(text, self.terms):
  1794. text = self._ask(x, words=False)
  1795. self.last["context_dump"] = True
  1796. if field and echoes_prior(text, field, len(x) / SR):
  1797. redo = self._ask(x, field=False)
  1798. # nothing heard without the field, and the answer was the field itself (≥ 80 % of it one run of the field):
  1799. # there was no speech to write — a short piece of breath came back as the whole field (6 of the founder's
  1800. # takes in the old piece path, re-measured 2026-10-31)
  1801. nothing = not redo.strip() and longest_shared_run(text, field) >= 0.8 * len(_bare(text))
  1802. if (redo.strip() or nothing) and len(redo) < 0.8 * len(text):
  1803. text = redo
  1804. self.last["echo"] = True
  1805. return [(0.0, len(x) / SR, text)]
  1806. def _ask(self, x, words: bool = True, field: bool = True) -> str:
  1807. """The compatible API first; a refusal there (4xx) or a broken answer (5xx: the realtime models answer 500) is
  1808. asked once more on the own API, which then serves this model from here on. Neither giving text: `model`."""
  1809. model = getattr(self, "model", "")
  1810. if model in _DASHSCOPE_OWN_API:
  1811. return self._request_native(x, words, field)
  1812. try:
  1813. return self._request(x, self.context_text(words, field))
  1814. except Fail as first:
  1815. if first.code not in ("bad_response", "network") or not model:
  1816. raise # a key or quota answer is the same on both APIs
  1817. try:
  1818. text = self._request_native(x, words, field)
  1819. except Fail as second:
  1820. if second.code == "bad_response":
  1821. raise Fail(f"Model Studio gave no text for a recording from {model}: compatible API: {first}; "
  1822. f"own API: {second}", "model")
  1823. raise first if first.code == "network" else second
  1824. _DASHSCOPE_OWN_API.add(model)
  1825. return text
  1826. def _request_native(self, x, words: bool, field: bool) -> str:
  1827. """Model Studio's own HTTP API (qwen-audio-…-asr-flash takes no other). The words go as its
  1828. hot words (`vocabulary`: DASHSCOPE_HOTWORD_WEIGHT) instead of context text; a typed prompt and the field's text go as one
  1829. context turn, which the service keeps to 400 characters (the field's end kept: it is next to the cursor)."""
  1830. audio = "data:audio/wav;base64," + base64.b64encode(wav_bytes(speed_up(x, self.speed))).decode()
  1831. turn = "\n".join(t for t in (self.args.prompt or "", getattr(self, "field", "") if field else "") if t)[-400:]
  1832. messages = [{"role": "user", "content": [{"type": "input_text", "text": turn}]}] if turn else []
  1833. messages.append({"role": "user", "content": [{"type": "input_audio", "input_audio": {"data": audio}}]})
  1834. params: dict = {"format": "wav", "sample_rate": str(SR)}
  1835. if self.args.language:
  1836. params["language_hints"] = [self.args.language]
  1837. if words and self.terms and not self.args.prompt:
  1838. params["vocabulary"] = {t: DASHSCOPE_HOTWORD_WEIGHT if i < DASHSCOPE_SUPER_WORDS else 5
  1839. for i, t in enumerate(self.terms[:2000])}
  1840. data = _post(dashscope_native_url(self.base), json.dumps({"model": self.model, "input": {"messages": messages},
  1841. "parameters": params}).encode(),
  1842. {"Authorization": f"Bearer {self.key}", "Content-Type": "application/json"},
  1843. retries=self.retries, timeout=self.timeout)
  1844. text = (data.get("output") or {}).get("text") if isinstance(data, dict) else None
  1845. if text is None:
  1846. raise Fail(f"ASR API answered without a transcript: {json.dumps(data)[:300]}", "bad_response")
  1847. return text.strip()
  1848. def _request(self, x, context: str) -> str:
  1849. audio = "data:audio/wav;base64," + base64.b64encode(wav_bytes(speed_up(x, self.speed))).decode()
  1850. payload = {"model": self.model, "stream": False,
  1851. "messages": [{"role": "system", "content": [{"type": "text", "text": context}]},
  1852. {"role": "user", "content": [{"type": "input_audio", "input_audio": {"data": audio}}]}],
  1853. "asr_options": {"enable_itn": True, **({"language": self.args.language} if self.args.language else {})}}
  1854. data = _post(f"{self.base}/chat/completions", json.dumps(payload).encode(),
  1855. {"Authorization": f"Bearer {self.key}", "Content-Type": "application/json"},
  1856. retries=self.retries, timeout=self.timeout)
  1857. try:
  1858. text = data["choices"][0]["message"]["content"]
  1859. except (KeyError, IndexError, TypeError):
  1860. raise Fail(f"ASR API answered without a transcript: {json.dumps(data)[:300]}", "bad_response")
  1861. if isinstance(text, list):
  1862. text = "".join(p.get("text", "") for p in text)
  1863. return (text or "").strip()
  1864. # ------------------------------------------------------------------ Volcengine 豆包 Seed-ASR 2.0 (docs/design/dictation.md §Cloud recognition)
  1865. # The streamed `bigmodel_nostream` endpoint (dictation-eval-2026-10-31.md §2.1, §5, §6.3 B′): on the founder's 95 takes
  1866. # 2.18 % with the words as hotwords and the field's text as a dialogue turn, ~0.45–0.52 s after key-up however long the
  1867. # take — it hears the audio while it is said and re-recognises it in ~15 s blocks, so at key-up only the last block is
  1868. # left. The app streams a take's microphone packets through the warm server (`serve` cmd "stream", VolcStream); Retry,
  1869. # the Lab, Settings' Test and a voice note send a whole recording through the same session as fast as it goes (the
  1870. # result doesn't depend on the pace: §2.1 note ⁴). Credentials are an App ID and an Access Token (VOLC_APP_ID,
  1871. # VOLC_ACCESS_TOKEN: the app's Keychain item for the "volcengine-speech" service), resource volc.seedasr.sauc.duration.
  1872. # The protocol (the docs' binary frames): a 4-byte header (version 1, header size 1; message type, flags; JSON / raw
  1873. # serialisation, gzip), a signed sequence where the flags say so, a 4-byte size, the payload. The first frame is the
  1874. # request (gzip JSON), then raw 16 kHz s16 mono PCM packets; the last packet's sequence is negative. The final result
  1875. # comes back flagged last; an error is message type 0b1111 with a code and a message.
  1876. VOLC_URL = "wss://openspeech.bytedance.com/api/v3/sauc/bigmodel_nostream"
  1877. VOLC_RESOURCE = "volc.seedasr.sauc.duration"
  1878. VOLC_PACKET = 6400 # 200 ms of 16 kHz s16 mono: the microphone's pace and the docs' recommended size
  1879. # a language set by the user (asr.language) → the service's code; "zh" and anything else unlisted: the zh-en default
  1880. VOLC_LANGS = {"en": "en-US", "ja": "ja-JP", "ko": "ko-KR", "de": "de-DE", "fr": "fr-FR", "es": "es-MX", "pt": "pt-BR",
  1881. "id": "id-ID", "ms": "ms-MY", "th": "th-TH", "ar": "ar-SA", "fil": "fil-PH"}
  1882. def volc_frame(kind: int, flags: int, serial: int, payload: bytes, seq: int | None = None, gz: bool = True) -> bytes:
  1883. p = gzip.compress(payload) if gz else payload
  1884. return (bytes([0x11, (kind << 4) | flags, (serial << 4) | (1 if gz else 0), 0])
  1885. + (struct.pack(">i", seq) if seq is not None else b"") + struct.pack(">I", len(p)) + p)
  1886. def volc_parse(b: bytes) -> tuple[dict, bool]:
  1887. """One frame from the service → (its JSON, or {"error": (code, message)}; whether it is the last)."""
  1888. hs = (b[0] & 0x0F) * 4
  1889. kind, fl, comp = b[1] >> 4, b[1] & 0x0F, b[2] & 0x0F
  1890. p = b[hs:]
  1891. seq = None
  1892. if fl & 1:
  1893. seq = struct.unpack(">i", p[:4])[0]
  1894. p = p[4:]
  1895. if kind == 0b1111:
  1896. code = struct.unpack(">I", p[:4])[0]
  1897. n = struct.unpack(">I", p[4:8])[0]
  1898. msg = p[8:8 + n]
  1899. try:
  1900. msg = gzip.decompress(msg) if comp == 1 else msg
  1901. except OSError:
  1902. pass
  1903. return {"error": (code, msg.decode(errors="replace")[:300])}, True
  1904. n = struct.unpack(">I", p[:4])[0]
  1905. body = p[4:4 + n]
  1906. if comp == 1 and body:
  1907. body = gzip.decompress(body)
  1908. return (json.loads(body) if body else {}), bool(fl & 2) or (seq is not None and seq < 0)
  1909. def volc_fail(code: int, msg: str) -> Fail:
  1910. """The service's error frame → a Fail the app words plainly. 45000292 "quota exceeded for types: concurrency" (the
  1911. trial allows only a few sessions at once; the eval met it at 6) and 55000031 (busy) are "quota"; a token or app the
  1912. service doesn't accept, "auth"; its own time-outs and server errors, "network"."""
  1913. low = msg.lower()
  1914. if code == 45000292 or code == 55000031 or "quota" in low or "concurrency" in low:
  1915. return Fail(f"Volcengine refused ({code}): {msg}", "quota")
  1916. if any(w in low for w in ("auth", "token", "permission", "grant", "app key", "appkey", "access key")):
  1917. return Fail(f"Volcengine refused the App ID or Access Token ({code}): {msg}", "auth")
  1918. if code == 45000081 or code >= 55000000:
  1919. return Fail(f"Volcengine answered {code}: {msg}", "network")
  1920. return Fail(f"Volcengine answered {code}: {msg}", "bad_response")
  1921. class _WebSocket:
  1922. """The least of a websocket client (RFC 6455) a recognition session needs — binary frames out (masked), frames in,
  1923. ping / pong, close — on the standard library (transcribe.py's own dependencies stay as they are). The handshake's
  1924. refusal is a Fail: 401 / 403 auth, 429 quota, anything else network or bad_response."""
  1925. def __init__(self, url: str, headers: dict, timeout: float = 10):
  1926. u = urllib.parse.urlsplit(url)
  1927. host, port = u.hostname or "", u.port or (443 if u.scheme == "wss" else 80)
  1928. try:
  1929. raw = socket.create_connection((host, port), timeout=timeout)
  1930. if u.scheme == "wss":
  1931. import ssl
  1932. raw = ssl.create_default_context().wrap_socket(raw, server_hostname=host)
  1933. except (OSError, ValueError) as e:
  1934. raise Fail(f"can't reach {host}: {e}", "network")
  1935. self.sock = raw
  1936. self.lock = threading.Lock()
  1937. self.closed = False
  1938. key = base64.b64encode(os.urandom(16)).decode()
  1939. lines = [f"GET {u.path or '/'}{'?' + u.query if u.query else ''} HTTP/1.1", f"Host: {host}", "Upgrade: websocket",
  1940. "Connection: Upgrade", f"Sec-WebSocket-Key: {key}", "Sec-WebSocket-Version: 13"]
  1941. lines += [f"{k}: {v}" for k, v in headers.items()]
  1942. try:
  1943. self.sock.sendall(("\r\n".join(lines) + "\r\n\r\n").encode("latin-1"))
  1944. head = b""
  1945. while b"\r\n\r\n" not in head and len(head) < 65536:
  1946. chunk = self.sock.recv(4096)
  1947. if not chunk:
  1948. break
  1949. head += chunk
  1950. except UnicodeEncodeError:
  1951. self.close()
  1952. raise Fail("the Access Token has characters a token can't have: paste it again", "auth")
  1953. except OSError as e:
  1954. self.close()
  1955. raise Fail(f"{host} didn't answer: {e}", "network")
  1956. top, _, rest = head.partition(b"\r\n\r\n")
  1957. status_line, *hdr = top.decode("latin-1").split("\r\n")
  1958. try:
  1959. status = int(status_line.split()[1])
  1960. except (IndexError, ValueError):
  1961. self.close()
  1962. raise Fail(f"{host} answered something that isn't HTTP: {status_line[:80]!r}", "network")
  1963. h = {k.strip().lower(): v.strip() for k, _, v in (x.partition(":") for x in hdr)}
  1964. if status != 101:
  1965. msg = (h.get("x-api-message") or rest.decode(errors="replace"))[:300]
  1966. self.close()
  1967. code = "auth" if status in (401, 403) else "quota" if status == 429 else \
  1968. "network" if status >= 500 else "bad_response"
  1969. # a resource the app wasn't granted (the streaming model not enabled for this App ID) is the key's setup
  1970. if status == 400 and any(w in msg.lower() for w in ("resource", "grant", "app key", "appid", "app id", "token")):
  1971. code = "auth"
  1972. raise Fail(f"Volcengine refused the session (HTTP {status}): {msg}", code)
  1973. self.logid = h.get("x-tt-logid")
  1974. self.buf = rest
  1975. self.sock.settimeout(None)
  1976. def _read(self, n: int) -> bytes:
  1977. while len(self.buf) < n:
  1978. chunk = self.sock.recv(max(65536, n - len(self.buf)))
  1979. if not chunk:
  1980. raise ConnectionError("closed")
  1981. self.buf += chunk
  1982. out, self.buf = self.buf[:n], self.buf[n:]
  1983. return out
  1984. def _send(self, opcode: int, payload: bytes) -> None:
  1985. n = len(payload)
  1986. head = bytes([0x80 | opcode])
  1987. head += bytes([0x80 | n]) if n < 126 else bytes([0x80 | 126]) + struct.pack(">H", n) if n < 65536 \
  1988. else bytes([0x80 | 127]) + struct.pack(">Q", n)
  1989. mask = os.urandom(4)
  1990. if n:
  1991. m = int.from_bytes((mask * (n // 4 + 1))[:n], "big")
  1992. payload = (int.from_bytes(payload, "big") ^ m).to_bytes(n, "big")
  1993. with self.lock:
  1994. self.sock.sendall(head + mask + payload)
  1995. def send(self, payload: bytes) -> None:
  1996. self._send(0x2, payload)
  1997. def recv(self) -> bytes | None:
  1998. """The next whole data message; None when the connection closed."""
  1999. data = b""
  2000. while True:
  2001. b0, b1 = self._read(2)
  2002. op, n = b0 & 0x0F, b1 & 0x7F
  2003. if n == 126:
  2004. n = struct.unpack(">H", self._read(2))[0]
  2005. elif n == 127:
  2006. n = struct.unpack(">Q", self._read(8))[0]
  2007. mask = self._read(4) if b1 & 0x80 else b""
  2008. p = self._read(n)
  2009. if mask:
  2010. p = bytes(c ^ mask[i % 4] for i, c in enumerate(p))
  2011. if op == 0x8:
  2012. return None
  2013. if op == 0x9:
  2014. self._send(0xA, p)
  2015. continue
  2016. if op == 0xA:
  2017. continue
  2018. data += p
  2019. if b0 & 0x80:
  2020. return data
  2021. def close(self) -> None:
  2022. if self.closed:
  2023. return
  2024. self.closed = True
  2025. with contextlib.suppress(OSError):
  2026. self._send(0x8, struct.pack(">H", 1000))
  2027. with contextlib.suppress(OSError):
  2028. self.sock.shutdown(socket.SHUT_RDWR)
  2029. with contextlib.suppress(OSError):
  2030. self.sock.close()
  2031. class VolcStream:
  2032. """One recognition session (one take): open → feed PCM as it comes (sent in 200 ms packets) → finish (the last
  2033. packet, then the final text within `wait` s). Thread-safe between the feeder and the reader it starts."""
  2034. def __init__(self, app_id: str, token: str, request: dict, timeout: float = 10):
  2035. hdr = {"X-Api-App-Key": app_id, "X-Api-Access-Key": token, "X-Api-Resource-Id": VOLC_RESOURCE,
  2036. "X-Api-Request-Id": str(uuid.uuid4()), "X-Api-Connect-Id": str(uuid.uuid4())}
  2037. self.ws = _WebSocket(VOLC_URL, hdr, timeout)
  2038. self.seq = 1
  2039. self.pending = b""
  2040. self.sent = 0 # bytes of PCM sent
  2041. self.text: str | None = None
  2042. self.utterances: list | None = None # with show_utterances: [{"text", "start_time", "end_time"} (ms)]
  2043. self.error: Fail | None = None
  2044. self.final = threading.Event()
  2045. self.t_final: float | None = None
  2046. self.t_last: float | None = None
  2047. try:
  2048. self.ws.send(volc_frame(0b0001, 0b0001, 1, json.dumps(request, ensure_ascii=False).encode(), self.seq))
  2049. except OSError as e:
  2050. self.ws.close()
  2051. raise Fail(f"Volcengine dropped the session: {e}", "network")
  2052. self.reader = threading.Thread(target=self._read, daemon=True)
  2053. self.reader.start()
  2054. def _read(self) -> None:
  2055. try:
  2056. while True:
  2057. m = self.ws.recv()
  2058. if m is None:
  2059. break
  2060. d, last = volc_parse(m)
  2061. if "error" in d:
  2062. code, msg = d["error"]
  2063. # 45000002: no audio at all (an empty take) — nothing was said
  2064. if code == 45000002:
  2065. self.text = ""
  2066. else:
  2067. self.error = volc_fail(code, msg)
  2068. break
  2069. r = d.get("result") or {}
  2070. if isinstance(r, list):
  2071. r = r[0] if r else {}
  2072. if r.get("text") is not None:
  2073. self.text = r.get("text")
  2074. if isinstance(r.get("utterances"), list):
  2075. self.utterances = r["utterances"]
  2076. if last:
  2077. break
  2078. except (OSError, ConnectionError, ValueError, struct.error) as e:
  2079. if not self.ws.closed and self.error is None:
  2080. self.error = Fail(f"Volcengine dropped the session: {e}", "network")
  2081. finally:
  2082. self.t_final = time.perf_counter()
  2083. self.final.set()
  2084. @property
  2085. def failed(self) -> bool:
  2086. return self.error is not None
  2087. def feed(self, pcm: bytes) -> None:
  2088. """PCM (16 kHz s16 mono) as it comes; whole 200 ms packets go out, the rest waits for more. A session the
  2089. service already ended takes nothing more (finish says why)."""
  2090. if self.final.is_set():
  2091. return
  2092. self.pending += pcm
  2093. while len(self.pending) >= 2 * VOLC_PACKET: # one packet always held back: the last one is flagged
  2094. self._packet(self.pending[:VOLC_PACKET], last=False)
  2095. self.pending = self.pending[VOLC_PACKET:]
  2096. def _packet(self, p: bytes, last: bool) -> None:
  2097. self.seq += 1
  2098. try:
  2099. self.ws.send(volc_frame(0b0010, 0b0011 if last else 0b0001, 0, p, -self.seq if last else self.seq))
  2100. self.sent += len(p)
  2101. except OSError as e:
  2102. if self.error is None and not self.final.is_set():
  2103. self.error = Fail(f"Volcengine dropped the session: {e}", "network")
  2104. self.final.set()
  2105. def finish(self, wait: float = 10) -> str:
  2106. """The rest of the audio as the last packet(s), then the final text — waited for at most `wait` s (then
  2107. "network": the service didn't answer)."""
  2108. try:
  2109. if not self.final.is_set():
  2110. rest = self.pending
  2111. self.pending = b""
  2112. while len(rest) > VOLC_PACKET:
  2113. self._packet(rest[:VOLC_PACKET], last=False)
  2114. rest = rest[VOLC_PACKET:]
  2115. self._packet(rest, last=True)
  2116. self.t_last = time.perf_counter()
  2117. if not self.final.wait(wait):
  2118. raise Fail(f"Volcengine gave no final text within {wait:.0f} s", "network")
  2119. if self.error is not None:
  2120. raise self.error
  2121. if self.text is None:
  2122. raise Fail("Volcengine closed the session without a final text", "network")
  2123. return self.text.strip()
  2124. finally:
  2125. self.close()
  2126. def close(self) -> None:
  2127. self.ws.close()
  2128. def volc_pcm(x: np.ndarray) -> bytes:
  2129. return (np.clip(x, -1, 1) * 32767).astype("<i2").tobytes()
  2130. class Volcengine:
  2131. """Volcengine Seed-ASR 2.0 (`bigmodel_nostream`): the words as inline hotwords, the field's text before the cursor
  2132. as one dialogue turn; no language set for a Chinese speaker (its zh-en default: the founder's takes 2.18 % against
  2133. 2.84 % with automatic language), automatic language for everyone else (best measured on English: §3), or the
  2134. language the user set. `transcribe` sends a whole recording through one session; `open` starts one a take streams
  2135. into (serve "stream")."""
  2136. max_chunk = 600 # one session per recording up to 10 min (a dictation stops at 5)
  2137. max_terms = 100
  2138. retries, timeout = 2, 60.0
  2139. streams = True
  2140. def __init__(self, args):
  2141. self.app_id = api_key("VOLC_APP_ID")
  2142. self.token = api_key("VOLC_ACCESS_TOKEN")
  2143. if not self.app_id or not self.token:
  2144. raise Fail("no Volcengine App ID and Access Token: set VOLC_APP_ID and VOLC_ACCESS_TOKEN", "auth")
  2145. self.model = "seed-asr-2.0 (bigmodel_nostream)"
  2146. self.args = args
  2147. self.terms: list[str] = []
  2148. self.field = ""
  2149. self.last: dict = {}
  2150. self.utterances = False # only my voice on a stream: each sentence's time, to drop another speaker's
  2151. def set_vocab(self, terms: list[dict]) -> bool:
  2152. self.terms = [str(t["term"]).strip() for t in terms if str(t.get("term") or "").strip()][:self.max_terms]
  2153. return bool(self.terms)
  2154. def set_field(self, text: str) -> None:
  2155. self.field = (text or "").strip()
  2156. def request(self) -> dict:
  2157. req = {"model_name": "bigmodel", "enable_itn": True, "enable_punc": True, "enable_ddc": False,
  2158. "show_utterances": self.utterances, "result_type": "full"}
  2159. ctx: dict = {}
  2160. if self.terms:
  2161. ctx["hotwords"] = [{"word": w} for w in self.terms]
  2162. if self.field:
  2163. ctx["context_type"] = "dialog_ctx"
  2164. ctx["context_data"] = [{"text": self.field}]
  2165. if ctx:
  2166. req["corpus"] = {"context": json.dumps(ctx, ensure_ascii=False)}
  2167. audio = {"format": "pcm", "codec": "raw", "rate": SR, "bits": 16, "channel": 1}
  2168. lang = (self.args.language or "").lower().split("-")[0] or None
  2169. if lang in VOLC_LANGS:
  2170. audio["language"] = VOLC_LANGS[lang]
  2171. elif lang is None and not any(x.split("-")[0] in ("zh", "yue") for x in user_languages()):
  2172. req["enable_auto_lang"] = True
  2173. return {"user": {"uid": "blurt"}, "audio": audio, "request": req}
  2174. def open(self, timeout: float = 10) -> VolcStream:
  2175. return VolcStream(self.app_id, self.token, self.request(), timeout)
  2176. def transcribe(self, x):
  2177. pcm = volc_pcm(x)
  2178. for attempt in range(self.retries):
  2179. try:
  2180. s = self.open(timeout=min(10, self.timeout))
  2181. s.feed(pcm)
  2182. # the whole recording at once: the service works through it at ~4× real time
  2183. text = s.finish(wait=max(self.timeout, final_wait(len(x) / SR, 0)))
  2184. return [(0.0, len(x) / SR, text)]
  2185. except Fail as e:
  2186. # a dropped connection is tried once more; a refusal (key, quota) is final
  2187. if e.code != "network" or attempt == self.retries - 1:
  2188. raise
  2189. time.sleep(1)
  2190. return [(0.0, len(x) / SR, "")]
  2191. def _post(url: str, body: bytes, headers: dict, retries: int = 4, timeout: float = 120) -> dict:
  2192. """One cloud request → its JSON. Every way it can fail is a Fail with a code the app words plainly (auth, quota,
  2193. network, bad_response): an exception of any other kind came back from the speech server as a bare "error"."""
  2194. for attempt in range(retries):
  2195. try:
  2196. req = urllib.request.Request(url, data=body, headers=headers, method="POST")
  2197. with urllib.request.urlopen(req, timeout=timeout) as r:
  2198. raw = r.read()
  2199. try:
  2200. return json.loads(raw)
  2201. except ValueError:
  2202. raise Fail(f"ASR API answered something that isn't JSON: {raw[:200]!r}", "bad_response")
  2203. except urllib.error.HTTPError as e:
  2204. msg = e.read().decode(errors="replace")[:500]
  2205. if e.code in (429, 500, 502, 503) and attempt < retries - 1:
  2206. time.sleep(2 ** attempt * 2)
  2207. continue
  2208. # 4xx: the service answered and refused the request (a model it doesn't serve this way, a wrong address):
  2209. # not a connection problem (2026-11-11: qwen-audio-3.1-asr-flash on chat/completions read "can't connect")
  2210. raise Fail(f"ASR API error {e.code}: {msg}", "auth" if e.code in (401, 403) else "quota" if e.code == 429
  2211. else "network" if e.code >= 500 or e.code == 408 else "bad_response")
  2212. except UnicodeEncodeError:
  2213. # a header (the key) with a character HTTP can't carry: api_key catches it first; this is the safety net
  2214. raise Fail("the API key has characters a key can't have: paste the key again", "auth")
  2215. except (OSError, http.client.HTTPException) as e: # URLError, timeouts, TLS, a dropped connection
  2216. if attempt < retries - 1:
  2217. time.sleep(2 ** attempt)
  2218. continue
  2219. raise Fail(f"ASR API unreachable: {e}", "network")
  2220. BACKENDS = {"sensevoice": SenseVoice, "qwen3-asr-onnx": Qwen3Onnx, "fun-asr-onnx": FunAsrOnnx, "mlx-whisper": MlxWhisper,
  2221. "qwen3-asr-mlx": Qwen3Mlx, "fun-asr-mlx": FunAsrMlx, "faster-whisper": FasterWhisper, "apple": AppleSpeech,
  2222. "openai": OpenAICompat, "dashscope": DashScope, "volcengine": Volcengine}
  2223. LOCAL = {"sensevoice", "qwen3-asr-onnx", "fun-asr-onnx", "mlx-whisper", "qwen3-asr-mlx", "fun-asr-mlx", "faster-whisper", "apple"}
  2224. EXTRA_PACKAGES = {"mlx-whisper": "mlx-whisper", "faster-whisper": "faster-whisper",
  2225. "qwen3-asr-mlx": "mlx-qwen3-asr", "fun-asr-mlx": "mlx-audio"} # backend → `uv run --with` package
  2226. # The exact version each `--with` installs (arch101: no release is picked up by itself; bumped on purpose, per Blurt
  2227. # release, like the script header's). faster-whisper (NVIDIA, Windows / Linux) is pinned when it is next verified.
  2228. # mlx-audio 0.6.3 pins mlx 0.32.4, the version the other two resolve to as well: one server holds all three
  2229. # (serve_packages).
  2230. PACKAGE_PINS = {"mlx-whisper": "mlx-whisper==0.4.3", "mlx-qwen3-asr": "mlx-qwen3-asr==0.4.4",
  2231. "mlx-audio": "mlx-audio==0.6.3"}
  2232. API_KEYS = {"openai": ("BLURT_ASR_API_KEY", "OPENAI_API_KEY", "GROQ_API_KEY"), "dashscope": ("DASHSCOPE_API_KEY",),
  2233. "volcengine": ("VOLC_APP_ID", "VOLC_ACCESS_TOKEN")}
  2234. KEYS_ALL = {"volcengine"} # needs every variable of its API_KEYS (an App ID and an Access Token), not any one
  2235. CLOUD_WORKERS = 4 # a cloud service's parts of one take in flight at once (NoteEngine.note)
  2236. # ------------------------------------------------------------------ recommendation
  2237. # The rules come from docs/design/asr-benchmark.md (2026-10-22, Apple M5 Max): Whisper large-v3-turbo on MLX was the
  2238. # most accurate in all seven test languages and the fastest multilingual model; its 4-bit build keeps nearly all of
  2239. # that in 2.6 GB; SenseVoice is the fastest and lightest for zh / en / ja / ko / yue; on a CPU without a usable GPU,
  2240. # Qwen3-ASR 0.6B (sherpa-onnx) was 9x faster than Whisper (CTranslate2) at a usable accuracy.
  2241. SV, TURBO, TURBO_Q4, CT2, QWEN_CPU = ("sensevoice-small", "whisper-large-v3-turbo-mlx", "whisper-large-v3-turbo-q4-mlx",
  2242. "whisper-large-v3-turbo-ct2", "qwen3-asr-0.6b-onnx")
  2243. def hw_kind(hw: dict) -> str:
  2244. """apple (Apple Silicon: MLX on the GPU) | nvidia (CUDA) | cpu"""
  2245. if hw["os"] == "darwin" and hw["arch"] == "arm64":
  2246. return "apple"
  2247. return "nvidia" if hw["gpu"].get("kind") == "nvidia" else "cpu"
  2248. QWEN_FIRST_LANGS = ("zh", "en") # where Qwen3-ASR measured best on real speech; Cantonese (36–39 % on the `say`
  2249. # samples), Japanese, Korean and French keep the table's choice
  2250. def preference(kind: str, tier: str, purpose: str, sv_ok: bool, sv_good: bool, qwen_ok: bool = False) -> list[str]:
  2251. """Models to try, best first (the first that fits and speaks every language wins). sv_ok: SenseVoice speaks
  2252. every language; sv_good: and its reference error rate is ≤ 5 % in each (not Korean: 10 % vs Whisper's 0.5 %);
  2253. qwen_ok: every language is one of QWEN_FIRST_LANGS."""
  2254. multi = {"apple": {"pro": [TURBO, TURBO_Q4], "standard": [TURBO, TURBO_Q4], "light": [TURBO_Q4], "minimal": []},
  2255. "nvidia": {"pro": [CT2, QWEN_CPU], "standard": [CT2, QWEN_CPU], "light": [QWEN_CPU], "minimal": []},
  2256. "cpu": {"pro": [QWEN_CPU], "standard": [QWEN_CPU], "light": [QWEN_CPU], "minimal": []}}[kind][tier]
  2257. # Apple Silicon with 16 GB or more: one model for every job — dictation, voice notes, live captions, recordings
  2258. # (v1.0.2). Qwen3-ASR on MLX was the most accurate on real speech in every set measured (AISHELL-1 NE terms, ASCEND
  2259. # zh-en, FLEURS zh, LibriSpeech, contextual earnings calls: docs/design/asr-benchmark.md §Real speech) and the only
  2260. # one that takes the user's words; one model also means one warm copy instead of three (the 24 GB Mac that
  2261. # swapped held a Qwen, a SenseVoice and a Whisper).
  2262. # v1.0.6: the 0.6B on pro Macs too. On the founder's real takes the two are close — the 0.6B ahead on the Lab's
  2263. # corrected takes (4.6 vs 6.4 %), the 1.7B ahead on English (2.7 vs 3.2 %) and custom words (28 vs 25 of 35) —
  2264. # while the 1.7B is 2.5x the download and holds 1 GB more (docs/design/first-minute-speech.md §2.1). The 1.7B
  2265. # stays a choice, and a Mac that already has it keeps using it.
  2266. if kind == "apple" and tier in ("pro", "standard") and qwen_ok:
  2267. return [QWEN_MLX] + ([QWEN_MLX_17] if tier == "pro" else []) + multi + ([SV] if sv_ok else [])
  2268. # on a CPU the multilingual model is 10x slower and no more accurate in SenseVoice's languages
  2269. prefer_sv = sv_ok if kind == "cpu" else sv_good
  2270. fallback = [SV] if sv_ok else []
  2271. if purpose == "batch" or (purpose == "live" and kind == "apple" and tier == "pro"):
  2272. # recordings: accuracy first. Live on a pro Mac runs the recordings model itself, so the live transcript is
  2273. # the final one (transcribe.py run reuses it).
  2274. if prefer_sv and (kind == "cpu" or tier in ("light", "minimal")):
  2275. return [SV] + multi
  2276. return multi + fallback
  2277. # live captions and voice notes: latency first. Their model stays warm next to dictation's: on a 16–24 GB Mac the
  2278. # 4-bit Whisper (2.6 GB, a 442 MB download) comes before the full one (3.7 GB) — arch098; recordings above keep
  2279. # accuracy first (a one-off process that ends with the job)
  2280. if kind == "apple" and tier == "standard":
  2281. multi = [TURBO_Q4, TURBO]
  2282. return ([SV] if prefer_sv else []) + multi + fallback
  2283. def _bench_results() -> dict:
  2284. d = load_json(BENCH_FILE, {}) or {}
  2285. return d.get("results", {}) if isinstance(d, dict) else {}
  2286. def _fits(m: dict, hw: dict, lim: dict | None = None) -> bool:
  2287. """Runs on this platform and its measured peak memory stays within 40 % of RAM (the rest is the user's)."""
  2288. if not runs_here(m, f"{hw['os']}-{hw['arch']}"):
  2289. return False
  2290. if "cuda" in (m.get("requires") or []) and hw["gpu"].get("kind") != "nvidia":
  2291. return False
  2292. return m.get("ram_mb", 0) <= max(hw["ram_gb"] * 1024 * 0.4, 700)
  2293. def candidate_row(m: dict, hw: dict, lim: dict, bench: dict, langs: list[str]) -> dict:
  2294. b = bench.get(m["id"]) or {}
  2295. ref = m.get("rtf_ref")
  2296. scale = {"pro": 1, "standard": 1.8, "light": 3.5, "minimal": 6}[hw["tier"]]
  2297. row = {"id": m["id"], "name": m.get("name"), "backend": m["backend"], "languages": m["languages"],
  2298. "size_mb": m["size_mb"], "ram_mb": b.get("peak_ram_mb") or m.get("ram_mb"),
  2299. "rtf_est": b.get("rtf") if b.get("rtf") is not None else (round(ref * scale, 3) if ref else None),
  2300. "measured": bool(b.get("rtf") is not None), "quality_note_key": m.get("quality_note_key"),
  2301. "installed": installed(m), "fits": _fits(m, hw, lim), "speaks": [x for x in langs if speaks(m, x)],
  2302. "local": m.get("local", True), "verified": m.get("verified")}
  2303. if b.get("error"):
  2304. row["error_rate"] = b["error"]
  2305. elif m.get("error_ref"):
  2306. row["error_rate_ref"] = {k: v for k, v in m["error_ref"].items() if k in langs}
  2307. return row
  2308. def recommend(languages: list[str] | None = None, purpose: str = "batch", hw: dict | None = None,
  2309. cfg: dict | None = None) -> dict:
  2310. """K9 + docs/design/asr-benchmark.md (preference() has the order):
  2311. - recordings: Apple Silicon pro / standard → Whisper large-v3-turbo (MLX); light → its 4-bit build; NVIDIA → turbo
  2312. on CUDA; CPU only → SenseVoice for zh / en / ja / ko / yue, Qwen3-ASR 0.6B (CPU) for anything else;
  2313. - live captions and voice notes: SenseVoice when it speaks every language, else the tier's multilingual model;
  2314. a pro Mac runs live captions with the recordings model;
  2315. - on battery (asr.on_battery = lighter, the default) recordings drop one tier;
  2316. - minimal tier with a language SenseVoice doesn't speak → cloud_recommended (SenseVoice is still the local pick)."""
  2317. hw = hw or probe()
  2318. cfg = asr_config() if cfg is None else cfg
  2319. lim = limits(hw, cfg)
  2320. langs = languages or user_languages(cfg)
  2321. tier = hw["tier"]
  2322. if hw.get("on_battery") and purpose == "batch" and cfg.get("on_battery", "lighter") == "lighter":
  2323. tier = {"pro": "standard", "standard": "light"}.get(tier, tier)
  2324. sv = entry(SV)
  2325. sv_ok = all(speaks(sv, x) for x in langs)
  2326. sv_good = sv_ok and all((sv.get("error_ref") or {}).get(x, 0) <= 0.05 for x in langs)
  2327. qwen_ok = all(x in QWEN_FIRST_LANGS for x in langs)
  2328. order = preference(hw_kind(hw), tier, purpose, sv_ok, sv_good, qwen_ok)
  2329. bench = _bench_results()
  2330. # v1.0.7: where dictation runs on the 1.7B (dictation_tier: 16 GB+ and fast enough), every job does — one model,
  2331. # one warm copy (v1.0.2), and Settings never recommends the 0.6B next to a 1.7B that is already in use. By the
  2332. # Mac's own tier: a recording on battery doesn't trade the model dictation already holds for a smaller one.
  2333. if hw_kind(hw) == "apple" and qwen_ok and hw["tier"] in ("pro", "standard"):
  2334. plan = dictation_tier(hw["ram_gb"], (bench.get(QWEN_MLX_17) or {}).get("rtf"), slow_chip(hw.get("chip")),
  2335. languages=langs)
  2336. if plan["recognizer"] == QWEN_MLX_17:
  2337. order = [QWEN_MLX_17] + order
  2338. pick = None
  2339. for mid in dict.fromkeys(order):
  2340. m = entry(mid)
  2341. if m and _fits(m, hw, lim) and all(speaks(m, x) for x in langs):
  2342. pick = m
  2343. break
  2344. cloud = tier == "minimal" and not sv_ok
  2345. if pick is None:
  2346. pick = sv # nothing local covers it: SenseVoice is still the fastest thing that runs
  2347. rows = [candidate_row(m, hw, lim, bench, langs) for m in entries(local_only=False)
  2348. if m.get("local", True) or not local_only()]
  2349. rank = {mid: i for i, mid in enumerate(dict.fromkeys(order + [m["id"] for m in entries()]))}
  2350. rows.sort(key=lambda r: (not r["fits"], len(r["speaks"]) < len(langs), rank.get(r["id"], 99)))
  2351. out = {"ok": True, "tier": hw["tier"], "effective_tier": tier, "purpose": purpose, "languages": langs,
  2352. "pick": {"model": pick["id"], "backend": pick["backend"]}, "candidates": rows}
  2353. if cloud:
  2354. out["cloud_recommended"] = True
  2355. return out
  2356. # ------------------------------------------------------------------ which model to run
  2357. def select(purpose: str, a=None) -> dict:
  2358. """The registry entry (or a synthetic one for a legacy backend / model) this run uses."""
  2359. cfg = asr_config()
  2360. if a is not None and getattr(a, "model_id", None):
  2361. m = entry(a.model_id)
  2362. if not m:
  2363. raise Fail(f"unknown model {a.model_id}; see transcribe.py models", "unknown_model")
  2364. return m
  2365. if a is not None and getattr(a, "backend", None):
  2366. name = a.backend
  2367. if name not in BACKENDS:
  2368. raise Fail(f"unknown backend {name}; choose from {list(BACKENDS)}", "unknown_model")
  2369. model = getattr(a, "model", None) or (cfg.get("model") if name == cfg.get("backend") else None)
  2370. return entry_for(name, model) or {"id": f"{name}:{model or 'default'}", "backend": name, "model": model,
  2371. "languages": "multi", "local": name in LOCAL}
  2372. chosen = (cfg.get("models") or {}).get(purpose) if isinstance(cfg.get("models"), dict) else None
  2373. if not is_auto(cfg):
  2374. if chosen and entry(chosen):
  2375. return entry(chosen)
  2376. name = cfg.get("backend") or "sensevoice"
  2377. model = cfg.get("model")
  2378. return entry_for(name, model) or {"id": f"{name}:{model or 'default'}", "backend": name, "model": model,
  2379. "languages": "multi", "local": name in LOCAL}
  2380. rec = recommend(purpose=purpose, cfg=cfg)
  2381. order = ([chosen] if chosen else []) + [rec["pick"]["model"]] + [c["id"] for c in rec["candidates"]
  2382. if c["fits"] and len(c["speaks"]) == len(rec["languages"])]
  2383. apple = entry(APPLE)
  2384. apple_ok = bool(apple and runs_here(apple) and installed(apple) and all(speaks(apple, x) for x in rec["languages"]))
  2385. # a local model hears through the VAD (0.6 MB) too: one whose own files are already here from elsewhere (a Whisper
  2386. # in a shared Hugging Face cache) doesn't beat macOS's recogniser, which needs nothing, until the VAD is in
  2387. bundled_vad()
  2388. vad = (MODELS_DIR / "silero_vad.onnx").is_file()
  2389. for mid in dict.fromkeys(order):
  2390. m = entry(mid)
  2391. if m and m.get("local", True) and runs_here(m) and installed(m) and (vad or m.get("builtin") or not apple_ok):
  2392. return m
  2393. if m and not m.get("local", True) and not local_only() and installed(m) and mid == chosen:
  2394. return m
  2395. # nothing downloaded: macOS's own recogniser when it speaks the user's languages (first-minute-speech.md)
  2396. if apple_ok:
  2397. return apple
  2398. # no macOS recogniser (macOS 13–15, ~23 % of Macs): a cloud service whose key the user set — the setup assistant's
  2399. # one key (Model Studio in China) says it recognises speech too — before a model that would have to download
  2400. if not local_only():
  2401. for mid in DICTATION_CLOUD:
  2402. m = entry(mid)
  2403. if m and installed(m):
  2404. return m
  2405. return entry("sensevoice-small") # downloads on first use (never during a recording: BLURT_NO_DOWNLOAD)
  2406. def resolve_backend(a, purpose: str = "batch") -> str:
  2407. """Backend for this run; fills a.model / a.model_id / a.language / a.prompt (explicit options win)."""
  2408. cfg = asr_config()
  2409. m = select(purpose, a)
  2410. a.model = None if m["backend"] == "sensevoice" else m.get("model")
  2411. if m["backend"] in ("qwen3-asr-mlx", "mlx-whisper", "fun-asr-mlx"):
  2412. # the folder it is in (HF cache snapshot or a ModelScope copy): loaded offline. Whisper got its repo id, and
  2413. # mlx-whisper asked huggingface.co about it on every load (`Fetching 4 files`: +2 s here, longer where the
  2414. # site is slow or blocked) before reading the files it already had
  2415. p = model_path(m)
  2416. if p:
  2417. a.model = str(p)
  2418. a.model_id = m["id"]
  2419. lang = getattr(a, "language", None) or cfg.get("language")
  2420. a.language = None if lang in (None, "", "auto") else lang
  2421. a.prompt = getattr(a, "prompt", None)
  2422. if not m.get("local", True) and local_only():
  2423. raise Fail("local-only recognition is on (privacy.local_asr): cloud ASR is not allowed", "privacy")
  2424. return m["backend"]
  2425. def backend_ready(name: str, model_id: str | None = None) -> str | None:
  2426. """None when `name` (or registry model `model_id`) can run right now without downloading anything or asking for
  2427. a key, else why not."""
  2428. if name not in BACKENDS:
  2429. return f"unknown backend {name}"
  2430. if name == "apple":
  2431. return None if apple_speech_ok() else "macOS speech recognition needs macOS 26 and the Blurt app"
  2432. bundled_vad()
  2433. if not (MODELS_DIR / "silero_vad.onnx").exists():
  2434. return "VAD model not downloaded (transcribe.py setup)"
  2435. if name in API_KEYS:
  2436. have = (all if name in KEYS_ALL else any)(os.environ.get(k) for k in API_KEYS[name])
  2437. return None if have else \
  2438. f"no API key in the environment ({' / '.join(API_KEYS[name])})"
  2439. m = entry(model_id) if model_id else entry_for(name)
  2440. if m and m.get("install") and not installed(m):
  2441. return f"{m['id']} is not downloaded (transcribe.py setup --model {m['id']})"
  2442. return None
  2443. def throttle(purpose: str, threads: int | None = None) -> dict:
  2444. """The process-wide caps for a CLI run (before any model or thread pool exists): thread pools and priority.
  2445. In-process callers (tests, the app's helpers) don't call this, so importing transcribe never renices anyone."""
  2446. lim = limits()
  2447. cap_threads(threads or lim["threads"])
  2448. return apply_priority(lim["priority"].get(purpose, "utility"))
  2449. def make_backend(name: str, a, purpose: str = "batch"):
  2450. """Load the backend with the thread cap, after making sure its package and files are there."""
  2451. a.threads = getattr(a, "threads", None) or limits()["threads"]
  2452. m = entry(getattr(a, "model_id", None)) or entry_for(name)
  2453. if name in EXTRA_PACKAGES:
  2454. ensure_package(EXTRA_PACKAGES[name])
  2455. if m and m.get("install") and m.get("local", True) and not m["install"].get("files"):
  2456. if not installed(m):
  2457. if os.environ.get("BLURT_NO_DOWNLOAD"):
  2458. raise Fail(f"{m['id']} is not downloaded", "model_missing")
  2459. install_model(m)
  2460. b = BACKENDS[name](a)
  2461. if name in ("qwen3-asr-mlx", "fun-asr-mlx") and purpose in ("batch", "live"):
  2462. b.max_chunk = 15 # a segment per chunk: a recording's timestamps (frames, items) stay this fine
  2463. return b
  2464. def model_label(name: str, backend, a) -> str | None:
  2465. if name == "sensevoice":
  2466. return "sense-voice-small-int8"
  2467. if name == "apple":
  2468. return "apple-speech-transcriber"
  2469. m = getattr(backend, "model", None)
  2470. return m if isinstance(m, str) else getattr(backend, "repo", None) or a.model
  2471. # ------------------------------------------------------------------ shared pipeline (also used by live.py)
  2472. def decode_slice(backend, x: np.ndarray, s: float, e: float) -> list[tuple]:
  2473. """Transcribe x (= the samples of chunk s..e seconds) → segments on the original timeline:
  2474. (start, end, text) or (start, end, text, language) when the backend says which language it heard. Chinese comes
  2475. back in the user's script (to_user_script) whichever model wrote it: voice notes, recordings, live captions and
  2476. dictation all pass here."""
  2477. out = []
  2478. for r in backend.transcribe(x):
  2479. rs, re_, t, *lg = r
  2480. seg = (s + rs, s + min(re_, e - s), to_user_script(t, lang_code(lg[0]) if lg and lg[0] else None))
  2481. out.append(seg + (lg[0],) if lg and lg[0] else seg)
  2482. return out
  2483. def decode_chunk(backend, samples: np.ndarray, s: float, e: float) -> list[tuple]:
  2484. return decode_slice(backend, samples[int(s * SR):int(e * SR)], s, e)
  2485. def is_blank(results) -> bool:
  2486. return not any(ch.isalnum() for r in results for ch in r[2])
  2487. def build_transcript(source: str, name: str, backend, a, duration: float, speech: float, blank: float,
  2488. results: list, elapsed: float) -> dict:
  2489. """results: [(start, end, text, speaker|None[, language])] → the transcript.json document."""
  2490. results = sorted(results, key=lambda r: r[0])
  2491. segments = []
  2492. for r in results:
  2493. s, e, t, lb, *lg = r
  2494. if t:
  2495. segments.append({"start": round(s, 2), "end": round(e, 2), "text": t, **({"speaker": lb} if lb else {}),
  2496. **({"language": lg[0]} if lg and lg[0] else {})})
  2497. data = {"source": source, "backend": name, "model": model_label(name, backend, a),
  2498. "duration": round(duration, 2), "speech_seconds": round(speech, 2),
  2499. "elapsed_seconds": round(elapsed, 1), "segments": segments}
  2500. langs = [sg["language"] for sg in segments if sg.get("language")]
  2501. if langs:
  2502. data["languages"] = sorted(set(langs), key=lambda x: -langs.count(x))
  2503. # SenseVoice knows only zh/en/ja/ko/yue; other languages (German…) mostly come back as "." tagged <|en|>, so
  2504. # the language tag can't catch it. Supported speech that VAD kept practically never decodes to nothing.
  2505. # Mic track only: system audio may be music or effects.
  2506. if name == "sensevoice" and speech >= 5 and blank / speech > 0.3:
  2507. data["warning"] = (f"{blank:.0f}s of {speech:.0f}s of speech decoded to nothing: the language is probably "
  2508. "not zh/en/ja/ko/yue, which is all SenseVoice supports. Re-run with a multilingual model "
  2509. "(transcribe.py recommend; reference/asr.md) and pass --language.")
  2510. return data
  2511. def write_transcript(out: Path, data: dict) -> list[str]:
  2512. save_json(out / "transcript.json", data)
  2513. lines = [f"[{fmt_ts(s['start'])}-{fmt_ts(s['end'])}] {'(system) ' if s.get('speaker') == 'system' else ''}{s['text']}"
  2514. for s in data["segments"]]
  2515. (out / "transcript.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")
  2516. return [str(out / "transcript.json"), str(out / "transcript.txt")]
  2517. def print_summary(data: dict, files: list[str], **extra) -> None:
  2518. print(json.dumps({k: v for k, v in data.items() if k != "segments"} | {"segments": len(data["segments"]),
  2519. "files": files} | extra,
  2520. ensure_ascii=False, indent=2))
  2521. def transcribe_track(samples: np.ndarray, backend, name: str, workers: int, label: str | None = None,
  2522. quiet: bool = False):
  2523. """VAD → chunks → backend. Returns ([(start, end, text, label[, language])], speech s, s that decoded to nothing).
  2524. A backend that finds its own pauses (`no_vad`: macOS's recogniser) gets the whole audio."""
  2525. # …and so does a cloud service for audio it takes in one request (a dictation's or voice note's piece, which the
  2526. # app has already cut at a pause): no VAD file needed for it either
  2527. whole = getattr(backend, "no_vad", False) or (name in API_KEYS and len(samples) / SR <= backend.max_chunk)
  2528. if whole:
  2529. dur = len(samples) / SR
  2530. r = decode_slice(backend, samples, 0.0, dur) if dur > 0 else []
  2531. return [(s_, e_, t, label, *lg) for s_, e_, t, *lg in r], dur, dur if is_blank(r) else 0.0
  2532. if getattr(backend, "whole_spans", False):
  2533. # a model that hears utterances whole (Fun-ASR-Nano): a piece or window as it is, a longer one cut in pauses
  2534. dur = len(samples) / SR
  2535. # …but only with speech in it: a quiet room's hiss handed whole to an LLM-decoder model comes back as an invented
  2536. # sentence (Windows dogfood 2026-11-02: −55 dBFS noise typed 我感觉我这个头发有点长。). The VAD says no speech →
  2537. # nothing to decode, and the app says "No speech detected".
  2538. # Only the ONNX build (Windows / Linux) is gated: the Mac's MLX model keeps its measured path, with no VAD pass
  2539. if 0 < dur <= backend.max_chunk:
  2540. gate = getattr(backend, "vad_gate", False)
  2541. chunks = [(0.0, dur)] if not gate or vad_segments(samples, max_speech=min(25, backend.max_chunk)) else []
  2542. else:
  2543. chunks = pause_spans(samples, backend.max_chunk)
  2544. else:
  2545. segs = vad_segments(samples, max_speech=min(25, backend.max_chunk))
  2546. chunks = group_chunks(segs, backend.max_chunk)
  2547. sp = sum(e - s for s, e in chunks)
  2548. if not quiet:
  2549. print(f"audio {fmt_ts(len(samples) / SR)}{f' ({label})' if label else ''}, speech {fmt_ts(sp)} "
  2550. f"in {len(chunks)} chunks → {name}", file=sys.stderr, flush=True)
  2551. results, blank = [], 0.0
  2552. with ThreadPoolExecutor(workers) as pool:
  2553. decoded = pool.map(lambda ch: decode_chunk(backend, samples, *ch), chunks)
  2554. for i, ((s, e), r) in enumerate(zip(chunks, decoded)):
  2555. results.extend((s_, e_, t, label, *lg) for s_, e_, t, *lg in r)
  2556. if is_blank(r):
  2557. blank += e - s
  2558. if (i + 1) % 20 == 0 and not quiet:
  2559. print(f" {i + 1}/{len(chunks)} chunks", file=sys.stderr, flush=True)
  2560. return results, sp, blank
  2561. # ------------------------------------------------------------------ reuse live.py's transcript (docs/design/snap-and-live.md)
  2562. def live_transcript(src: Path, out: Path, a, explicit: dict, wait: float = 180) -> dict | None:
  2563. """The transcript live.py finished during this very recording (same session, same length), else None.
  2564. If live.py is still draining after the stop, wait for it — its status.json stays fresh while it works."""
  2565. if getattr(a, "force", False) or out.resolve() != src.parent.resolve():
  2566. return None
  2567. live = src.parent / "live"
  2568. deadline = time.time() + wait
  2569. while True:
  2570. st = load_json(live / "status.json") or {}
  2571. if st.get("asr") == "done":
  2572. break
  2573. fresh = time.time() - float(st.get("updated") or 0) < 15
  2574. if st.get("asr") != "running" or not (live / "stop").exists() or not fresh or time.time() > deadline:
  2575. return None
  2576. time.sleep(0.5)
  2577. try:
  2578. data = load_json(out / "transcript.json")
  2579. except (ValueError, OSError):
  2580. return None
  2581. if not isinstance(data, dict) or data.get("source") != src.name or not isinstance(data.get("segments"), list):
  2582. return None
  2583. if explicit.get("backend") and explicit["backend"] != data.get("backend"):
  2584. return None # explicitly asked for another backend / model
  2585. if explicit.get("model") and explicit["model"] != data.get("model"):
  2586. return None
  2587. if explicit.get("prompt") and data.get("backend") != "sensevoice":
  2588. return None # a glossary helps Whisper / API backends (live.py has none): redo with it
  2589. try:
  2590. dur = probe_duration(src)
  2591. except Exception:
  2592. return None
  2593. if abs(dur - float(data.get("duration") or 0)) > max(2.0, 0.02 * dur):
  2594. return None # another take (restart) or a cut-short live run
  2595. return data
  2596. # ------------------------------------------------------------------ shots (snap mode): voice notes → "said"
  2597. def read_shots(session: Path) -> dict:
  2598. """shots.jsonl → {n: merged record}; later lines win."""
  2599. latest: dict = {}
  2600. log = session / "shots.jsonl"
  2601. if log.exists():
  2602. for line in log.read_text(encoding="utf-8").splitlines():
  2603. try:
  2604. rec = json.loads(line)
  2605. except json.JSONDecodeError:
  2606. continue
  2607. if isinstance(rec, dict) and rec.get("n") is not None:
  2608. latest[rec["n"]] = {**latest.get(rec["n"], {}), **rec}
  2609. return latest
  2610. def shot_voice(session: Path, n, rec: dict) -> Path | None:
  2611. voice = rec.get("voice")
  2612. if isinstance(voice, str) and voice:
  2613. p = session / voice
  2614. if not p.exists() and (session / "shots" / Path(voice).name).exists():
  2615. p = session / "shots" / Path(voice).name
  2616. return p if p.exists() else None
  2617. if voice is False:
  2618. return None
  2619. try:
  2620. p = session / "shots" / f"{int(n):03d}.wav"
  2621. except (TypeError, ValueError):
  2622. return None
  2623. return p if p.exists() else None
  2624. def structure_note(text: str) -> str | None:
  2625. """A Structured voice note's text through the warm polish model (polish.py serve) — the same `structure` request
  2626. the app sends for dictation and voice notes (model-based: lists laid out, self-corrections resolved). None when no
  2627. polish server runs, the text is empty, or the model fell back."""
  2628. if not text.strip():
  2629. return None
  2630. try:
  2631. import polish
  2632. timeout = polish.structure_timeout_s()
  2633. r = polish._server_call({"cmd": "structure", "text": text, "timeout": timeout}, timeout + 5)
  2634. except Exception: # noqa: BLE001 — no polish: the transcript stays the note
  2635. return None
  2636. if not isinstance(r, dict) or not r.get("ok") or r.get("fell_back"):
  2637. return None
  2638. out = (r.get("text") or "").strip()
  2639. return out or None
  2640. def transcribe_shots(session: Path, a) -> None:
  2641. latest = read_shots(session)
  2642. todo = [(n, rec, wav) for n, rec in latest.items()
  2643. if (wav := shot_voice(session, n, rec)) and (a.force or "said" not in rec)]
  2644. name = resolve_backend(a, "note")
  2645. t0 = time.time()
  2646. done = []
  2647. if todo:
  2648. with AsrSlot(limits()["max_concurrent"]):
  2649. backend = make_backend(name, a, "batch")
  2650. workers = 1 if name in LOCAL else a.concurrency
  2651. for n, rec, wav in todo:
  2652. results, _, _ = transcribe_track(load_audio(wav), backend, name, workers, quiet=True)
  2653. said = " ".join(r[2] for r in sorted(results, key=lambda r: r[0]) if r[2]).strip()
  2654. upd = {"said": said}
  2655. if rec.get("structure"):
  2656. # a Structured note (⇧ while it listened): its note is the structured text and the transcript stays
  2657. # as said_raw, as the app keeps it. No polish model: the transcript, and said_raw cleared — a
  2658. # stale one from an earlier pass would sit next to a note it no longer matches
  2659. s = structure_note(said)
  2660. upd = {"said": s, "said_raw": said} if s and s != said else {"said": said, "said_raw": None}
  2661. # an update line with the same n (full record + "said"); later lines win — docs/design/snap-and-live.md
  2662. with open(session / "shots.jsonl", "a", encoding="utf-8") as fh:
  2663. fh.write(json.dumps({**rec, **upd}, ensure_ascii=False) + "\n")
  2664. done.append({"n": n, **upd})
  2665. print(json.dumps({"session": str(session), "mode": "shots", "shots": len(latest), "transcribed": len(done),
  2666. "backend": name, "elapsed_seconds": round(time.time() - t0, 1), "said": done},
  2667. ensure_ascii=False, indent=2))
  2668. # ------------------------------------------------------------------ commands
  2669. def cmd_run(a):
  2670. src = Path(a.input)
  2671. if not src.exists():
  2672. die(f"{src} not found")
  2673. if src.is_dir(): # a session dir: a shots session, or the recording inside it
  2674. meta = load_json(src / "meta.json", {}) or {}
  2675. recs = sorted(src.glob("recording.*"))
  2676. if meta.get("mode") == "shots" or (not recs and meta.get("mode") != "video" and (src / "shots.jsonl").exists()):
  2677. throttle("batch", a.threads)
  2678. return transcribe_shots(src, a)
  2679. if not recs:
  2680. die(f"no recording.* in {src}")
  2681. src = recs[0]
  2682. out = Path(a.out or src.parent)
  2683. explicit = {"backend": a.backend, "model": a.model, "prompt": a.prompt}
  2684. data = live_transcript(src, out, a, explicit)
  2685. n_tracks = max(1, audio_tracks(src))
  2686. if data is not None and (n_tracks == 1 or any(s.get("speaker") == "system" for s in data["segments"])):
  2687. files = [str(out / "transcript.json"), str(out / "transcript.txt")]
  2688. if not (out / "transcript.txt").exists():
  2689. files = write_transcript(out, data)
  2690. print("transcript was already made during the recording (live/); --force redoes it", file=sys.stderr, flush=True)
  2691. return print_summary(data, files, live=True)
  2692. name = resolve_backend(a, "batch")
  2693. throttle("batch", a.threads)
  2694. with AsrSlot(limits()["max_concurrent"]):
  2695. t0 = time.time()
  2696. backend = make_backend(name, a, "batch")
  2697. workers = 1 if name in LOCAL else a.concurrency
  2698. # Blurt's native recorder writes the mic as track 0 and, with --system-audio, what the Mac played as track 1.
  2699. # Transcribing them apart labels who spoke: the user ("mic") or the call / demo ("system").
  2700. duration = speech = blank = 0.0
  2701. results = []
  2702. for track in range(min(n_tracks, 2)):
  2703. label = ("mic", "system")[track] if n_tracks > 1 else None
  2704. if track == 0 and data is not None: # live.py already did the mic; only the system track is left
  2705. results += [(s["start"], s["end"], s["text"], label) for s in data["segments"]]
  2706. duration, speech = float(data["duration"]), float(data["speech_seconds"])
  2707. print("mic track: reusing the live transcript", file=sys.stderr, flush=True)
  2708. continue
  2709. samples = load_audio(src, track)
  2710. duration = max(duration, len(samples) / SR)
  2711. r, sp, bl = transcribe_track(samples, backend, name, workers, label)
  2712. results += r
  2713. if track == 0:
  2714. speech, blank = sp, bl
  2715. data = build_transcript(src.name, name, backend, a, duration, speech, blank, results, time.time() - t0)
  2716. if data.get("warning"):
  2717. print("warning: " + data["warning"], file=sys.stderr, flush=True)
  2718. print_summary(data, write_transcript(out, data))
  2719. # --- voice notes: the fast path and the warm server
  2720. # A streamed dictation's (or voice note's) text so far goes to the speech model as context, so a name is spelled the
  2721. # same way in every piece. Long, it gets echoed: the model answers with the earlier text instead of, or before, the
  2722. # new piece (dict023, measured on a 43 s take: a 5.5 s last piece with 200 characters of prior came back as 78–165
  2723. # characters, with 80 or 40 as the right 24, every run). So only the tail goes, and with it only the most relevant
  2724. # words of the list; `echoes_prior` is the meaning-free net behind that.
  2725. PRIOR_CHARS = 80
  2726. PRIOR_TERMS = 32
  2727. # The field's text before the cursor goes to a model that takes it (asr_models.json field_context_chars: Qwen3-ASR
  2728. # 1.7B, the cloud Qwen3-ASR Flash) as context, ahead of the text so far: on 43 of the founder's takes the 1.7B went from
  2729. # 3.04 to 2.63 % on whole takes (docs/design/dictation.md §Field context). The app sends at most this many.
  2730. FIELD_CHARS = 300
  2731. def field_context_chars(model_id: str | None) -> int:
  2732. """How much of the field's text before the cursor this model hears as context (0: none — the 0.6B was mixed)."""
  2733. m = entry(model_id) or {}
  2734. n = m.get("field_context_chars")
  2735. return min(FIELD_CHARS, int(n)) if isinstance(n, (int, float)) and n > 0 else 0
  2736. # More than this many syllables a second is no one's speech (normal speech is 4–7 in every language measured).
  2737. MAX_SYLLABLES_PER_SECOND = 10
  2738. # A run this long (characters, spaces and punctuation aside) shared with the text so far is worth one more look.
  2739. ECHO_RUN = 12
  2740. def prior_tail(prior: str, n: int | None = None) -> str:
  2741. """The last `n` (PRIOR_CHARS) characters of the text so far, not starting inside a Latin word."""
  2742. n = n or PRIOR_CHARS
  2743. p = (prior or "").strip()
  2744. if len(p) <= n:
  2745. return p
  2746. tail = p[-n:]
  2747. if tail[0].isascii() and tail[0].isalnum() and p[-n - 1].isascii() and p[-n - 1].isalnum() and " " in tail:
  2748. tail = tail[tail.index(" ") + 1:]
  2749. return tail.strip()
  2750. def _bare(text: str) -> str:
  2751. return "".join(c for c in unicodedata.normalize("NFKC", text or "").lower()
  2752. if not unicodedata.category(c).startswith(("P", "Z", "C", "S")))
  2753. def longest_shared_run(a: str, b: str) -> int:
  2754. """The longest run of characters `a` and `b` share (spaces and punctuation aside)."""
  2755. a, b = _bare(a), _bare(b)
  2756. best, prev = 0, [0] * (len(b) + 1)
  2757. for ca in a:
  2758. cur = [0] * (len(b) + 1)
  2759. for j, cb in enumerate(b, 1):
  2760. if ca == cb:
  2761. cur[j] = prev[j - 1] + 1
  2762. best = max(best, cur[j])
  2763. prev = cur
  2764. return best
  2765. def echoes_prior(text: str, prior: str, seconds: float) -> bool:
  2766. """A piece's answer that may be the context written out again, judged without reading it: more syllables than
  2767. `seconds` of speech can hold, or a long run of the prior in it. Only a reason to decode once more without the
  2768. prior (the caller keeps the shorter answer only when it is clearly shorter)."""
  2769. if not text or not prior:
  2770. return False
  2771. if seconds >= 1 and syllables(text) > MAX_SYLLABLES_PER_SECOND * seconds:
  2772. return True
  2773. return longest_shared_run(text, prior) >= ECHO_RUN
  2774. def note_text(results: list) -> tuple[str, str | None]:
  2775. results = sorted(results, key=lambda r: r[0])
  2776. texts = [r[2] for r in results if r[2]]
  2777. langs = [r[4] for r in results if len(r) > 4 and r[4]]
  2778. sep = "" if langs and all(x in ("zh", "ja", "yue") for x in langs) else " "
  2779. return sep.join(texts).strip(), (max(set(langs), key=langs.count) if langs else None)
  2780. # --- voiceprint: only my voice (issue #26; docs/design/voiceprint.md) -----------------------------------------
  2781. # A speaker-embedding gate in front of a dictation's recognition. With record.dictation.only_my_voice on (Settings →
  2782. # Dictation) and a voice enrolled, each piece is cut where silero hears pauses (cuts ≤ 10 s, gaps ≤ 0.6 s), each cut is
  2783. # embedded (3D-Speaker CAM++ zh-en, ~3.5 ms per second of audio on the CPU, in this process) and compared with the
  2784. # user's voice (cosine with the mean of the enrolled and learned embeddings). A cut under VP_THRESHOLD is someone
  2785. # else: silenced out of what recognition hears; a piece that is all someone else answers an empty text before the
  2786. # model is even loaded. The verdict comes back as `speaker` (with `drop`, the silenced spans) so the app can mask the
  2787. # same audio for whatever else hears it (a rehear window, macOS's second transcript). Interim requests (the live
  2788. # words) are gated too, but only cuts long enough to judge (VP_INTERIM_MIN). Only dictation is gated: a voice note or
  2789. # a recording is meant to keep everyone. Without an enrolled voice, or without the model, a piece goes through as it
  2790. # is (fail-open). Measured on the founder's real takes against AISHELL / ASCEND / LibriSpeech / Common Voice speakers
  2791. # (docs/design/voiceprint.md): the old ERes2Net + 0.5 dropped 17 % of the user's own speech; CAM++ at 0.28 with
  2792. # learning drops none of it and lets ~0.6 % of other people's through.
  2793. VP_FILE = "3dspeaker_speech_campplus_sv_zh_en_16k-common_advanced.onnx"
  2794. VP_URLS = [
  2795. f"https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/{VP_FILE}",
  2796. f"https://huggingface.co/csukuangfj/speaker-embedding-models/resolve/main/{VP_FILE}",
  2797. f"https://hf-mirror.com/csukuangfj/speaker-embedding-models/resolve/main/{VP_FILE}",
  2798. ]
  2799. VP_MB = 30 # both models' download
  2800. # speaker turns inside a piece (pyannote segmentation 3.0, MIT, as sherpa-onnx publishes it; int8, 1.5 MB): in a
  2801. # conversation the next person starts within a fraction of a second, too soon for a pause to cut at — on the founder's
  2802. # own tests every reply of a friend sat in one cut with his own words and was typed with them
  2803. VP_SEG_FILE = "pyannote-segmentation-3-0.int8.onnx"
  2804. VP_SEG_URLS = [
  2805. "https://huggingface.co/csukuangfj/sherpa-onnx-pyannote-segmentation-3-0/resolve/main/model.int8.onnx",
  2806. "https://hf-mirror.com/csukuangfj/sherpa-onnx-pyannote-segmentation-3-0/resolve/main/model.int8.onnx",
  2807. ]
  2808. # under this cosine a cut is someone else (record.dictation.voiceprint_threshold overrides). The founder's own cuts:
  2809. # 0.16 % dropped at 0.275 / 0.57 % at 0.30 with a 20 s enrollment, none once learning has a few takes; other people's
  2810. # speech: the highest of 1,150 cuts scored 0.38
  2811. VP_THRESHOLD = 0.28
  2812. # learning: a cut of ≥ 3 s that scores ≥ 0.5 (far above anyone else measured) joins the profile, the latest 40 —
  2813. # the user's other microphones, rooms and moods, which one enrollment take can't hold
  2814. VP_LEARN = 0.5
  2815. VP_LEARN_MIN = 3.0
  2816. VP_LEARNED_MAX = 40
  2817. VP_LEARN_EVERY = 20.0 # s between two learned cuts (one take doesn't fill the profile)
  2818. # with two voices in a piece, the one far below the user's is someone else even above VP_THRESHOLD (a voice close to
  2819. # the user's — the founder's friend scored 0.24–0.27 against his 0.68–0.84): dropped when it is VP_MARGIN below the
  2820. # best, unless it scores VP_SURE or more
  2821. VP_MARGIN = 0.3
  2822. VP_SURE = 0.45
  2823. VP_MARGIN_MIN = 1.5 # …a shorter stretch scores too unevenly for that (the founder's own 0.8 s "丫头": 0.36)
  2824. VP_CLUSTER = 0.5 # sherpa-onnx's clustering threshold: turns closer than this are one speaker
  2825. # an interim request judges only cuts with this much speech: a shorter one is too little to tell (the final piece
  2826. # decides it)
  2827. VP_INTERIM_MIN = 1.5
  2828. def voiceprint_settings(flag: bool | None) -> tuple[bool, float]:
  2829. """(on, threshold): the request's "voiceprint" wins; else record.dictation.only_my_voice / .voiceprint_threshold."""
  2830. d = (global_config().get("record") or {}).get("dictation") or {}
  2831. on = bool(d.get("only_my_voice", False)) if flag is None else bool(flag)
  2832. try:
  2833. threshold = float(d.get("voiceprint_threshold", VP_THRESHOLD))
  2834. except (TypeError, ValueError):
  2835. threshold = VP_THRESHOLD
  2836. return on, threshold
  2837. class Voiceprint:
  2838. """The speaker-embedding extractor, loaded once per process; the user's voice in $BLURT_HOME/voiceprint.json:
  2839. {"model", "embeddings" (enrollment windows, the latest 4 takes'), "learned" (cuts from dictation), "updated"}."""
  2840. PROFILE = BLURT_HOME / "voiceprint.json"
  2841. TAKES = 4
  2842. _ext = None
  2843. _lock = threading.Lock()
  2844. _cache: dict = {} # sha1 of a cut's samples → its embedding (interim requests repeat the same cuts)
  2845. _target: tuple | None = None # (profile mtime, target, the enrollment's own)
  2846. _diar = None
  2847. _learned_at = 0.0
  2848. @classmethod
  2849. def extractor(cls):
  2850. with cls._lock:
  2851. if cls._ext is None:
  2852. import sherpa_onnx
  2853. cfg = sherpa_onnx.SpeakerEmbeddingExtractorConfig(model=str(download(VP_FILE)), num_threads=2,
  2854. provider="cpu")
  2855. cls._ext = sherpa_onnx.SpeakerEmbeddingExtractor(cfg)
  2856. return cls._ext
  2857. @classmethod
  2858. def diarizer(cls):
  2859. """Speaker turns (pyannote segmentation + this same embedding model, clustered), loaded once."""
  2860. with cls._lock:
  2861. if cls._diar is None:
  2862. import sherpa_onnx
  2863. seg = sherpa_onnx.OfflineSpeakerSegmentationModelConfig(
  2864. pyannote=sherpa_onnx.OfflineSpeakerSegmentationPyannoteModelConfig(model=str(download(VP_SEG_FILE))),
  2865. num_threads=2)
  2866. emb = sherpa_onnx.SpeakerEmbeddingExtractorConfig(model=str(download(VP_FILE)), num_threads=2)
  2867. cls._diar = sherpa_onnx.OfflineSpeakerDiarization(sherpa_onnx.OfflineSpeakerDiarizationConfig(
  2868. segmentation=seg, embedding=emb,
  2869. clustering=sherpa_onnx.FastClusteringConfig(num_clusters=-1, threshold=VP_CLUSTER),
  2870. min_duration_on=0.2, min_duration_off=0.3))
  2871. return cls._diar
  2872. @classmethod
  2873. def speakers(cls, x: np.ndarray, t: np.ndarray) -> list[dict]:
  2874. """Who spoke when: the audio in ≤ 10 s spans cut at pauses (segmentation's window; a longer take is judged
  2875. span by span — each speaker is compared with the user, not with the others), each span's speaker turns, and
  2876. each speaker's similarity to the user (their turns' audio together, up to 10 s). [{"turns": [(s, e)], "sim",
  2877. "seconds"}]."""
  2878. out = []
  2879. for a, b in pause_spans(x, 10.0) or [(0.0, len(x) / SR)]:
  2880. part = x[int(a * SR):int(b * SR)]
  2881. spk: dict[int, list] = {}
  2882. for s in cls.diarizer().process(part).sort_by_start_time():
  2883. spk.setdefault(s.speaker, []).append((a + s.start, a + s.end))
  2884. for turns in spk.values():
  2885. audio = np.concatenate([x[int(s * SR):int(e * SR)] for s, e in turns])
  2886. secs = len(audio) / SR
  2887. sim = float(np.dot(cls.embed(audio[:10 * SR]), t)) if secs >= 0.5 else None
  2888. out.append({"turns": turns, "sim": sim, "seconds": secs, "audio": audio})
  2889. # speech the turns leave out (segmentation hears a quiet voice — a phone's video across the room — as
  2890. # nothing, silero as speech): each stretch of it judged on its own, as one more speaker
  2891. covered = [tr for s in out for tr in s["turns"]]
  2892. for s, e in vad_segments(x):
  2893. rest = [(s, e)]
  2894. for a, b in covered:
  2895. rest = [p for r in rest for p in ((r[0], min(r[1], a)), (max(r[0], b), r[1])) if p[1] - p[0] > 0.05]
  2896. for a, b in rest:
  2897. if b - a >= 0.5:
  2898. audio = x[int(a * SR):int(b * SR)]
  2899. out.append({"turns": [(a, b)], "sim": float(np.dot(cls.embed(audio[:10 * SR]), t)),
  2900. "seconds": b - a, "audio": audio})
  2901. return out
  2902. @classmethod
  2903. def embed(cls, x: np.ndarray) -> np.ndarray:
  2904. key = hashlib.sha1(np.ascontiguousarray(x, dtype=np.float32).tobytes()).digest()
  2905. if (e := cls._cache.get(key)) is not None:
  2906. return e
  2907. ext = cls.extractor()
  2908. stream = ext.create_stream()
  2909. stream.accept_waveform(SR, x)
  2910. stream.input_finished()
  2911. e = np.asarray(ext.compute(stream), dtype=np.float32)
  2912. e = e / (np.linalg.norm(e) + 1e-9)
  2913. if len(cls._cache) > 256:
  2914. cls._cache.clear()
  2915. cls._cache[key] = e
  2916. return e
  2917. @staticmethod
  2918. def speech_seconds(x: np.ndarray) -> float:
  2919. """Loud-frame seconds (20 ms frames, above −45 dBFS: speech is far louder, a quiet room far quieter). An
  2920. enrollment of room tone is refused, never learned; an all-speech clip counts in full (no adaptive floor:
  2921. one made of speech alone would call nothing loud)."""
  2922. f = SR // 50
  2923. n = len(x) // f
  2924. if n == 0:
  2925. return 0.0
  2926. rms = np.sqrt((x[: n * f].reshape(n, f) ** 2).mean(axis=1))
  2927. loud = 20 * np.log10(rms + 1e-9) > -45
  2928. return float(loud.sum() * f / SR)
  2929. @classmethod
  2930. def embed_windows(cls, x: np.ndarray, win: float = 5.0) -> list[np.ndarray]:
  2931. """One embedding per 5 s window that is mostly speech: a long enrollment is several looks at the voice."""
  2932. w = int(win * SR)
  2933. return [cls.embed(part) for off in range(0, len(x), w)
  2934. if len(part := x[off:off + w]) >= SR and cls.speech_seconds(part) >= 0.4 * len(part) / SR]
  2935. @classmethod
  2936. def profile(cls) -> dict:
  2937. """The profile when it was made with this model ({} without one; `stale` when another model made it: its
  2938. embeddings can't be compared with this one's)."""
  2939. p = load_json(cls.PROFILE, None)
  2940. if not isinstance(p, dict) or not isinstance(p.get("embeddings"), list) or not p["embeddings"]:
  2941. return {}
  2942. if p.get("model") != VP_FILE:
  2943. return {"stale": True}
  2944. if p.get("version") != 2: # v1 learned whole cuts, a friend's reply inside the user's cut included
  2945. p["learned"] = []
  2946. return p
  2947. @classmethod
  2948. def target(cls) -> np.ndarray | None:
  2949. """The user's voice: the mean of the enrolled and learned embeddings (re-read when the file changes — the CLI
  2950. enrolls beside a running server)."""
  2951. try:
  2952. mtime = cls.PROFILE.stat().st_mtime_ns
  2953. except OSError:
  2954. cls._target = None
  2955. return None
  2956. if cls._target and cls._target[0] == mtime:
  2957. return cls._target[1]
  2958. p = cls.profile()
  2959. if not p.get("embeddings"):
  2960. cls._target = (mtime, None, None)
  2961. return None
  2962. def unit(v):
  2963. e = np.asarray(v, dtype=np.float32).mean(axis=0)
  2964. return e / (np.linalg.norm(e) + 1e-9)
  2965. cls._target = (mtime, unit(p["embeddings"] + (p.get("learned") or [])), unit(p["embeddings"]))
  2966. return cls._target[1]
  2967. @classmethod
  2968. def enrolled(cls) -> np.ndarray | None:
  2969. """The enrollment take alone: what learning is measured against, so it can't drift toward whoever it once
  2970. let through."""
  2971. return cls._target[2] if cls.target() is not None and cls._target else None
  2972. @classmethod
  2973. def similarity(cls, x: np.ndarray) -> float | None:
  2974. """Cosine of this audio against the user's voice; None without one."""
  2975. t = cls.target()
  2976. return None if t is None else float(np.dot(cls.embed(x), t))
  2977. @classmethod
  2978. def gate(cls, x: np.ndarray, threshold: float, interim: bool = False) -> tuple[dict, np.ndarray]:
  2979. """The gate, per piece: who spoke when (`speakers`: speaker turns, not pauses — a reply right after the user
  2980. is found too), each speaker compared with the user's voice, and another speaker's turns silenced out of what
  2981. recognition hears (10 ms ramps: a hard zero is a click; where two talk at once, the user's turn wins). A
  2982. speaker is someone else under `threshold`, or VP_MARGIN below the best speaker of the piece unless they score
  2983. VP_SURE. ({"match", "similarity", "threshold", "dropped", "drop": [[s, e], …], "speakers"}, the audio to
  2984. recognise). match false: nobody in it is the user — the caller answers blank without recognising. Fail
  2985. "not_enrolled" / "model_missing": the caller lets the piece through. Interim requests judge only speakers
  2986. with VP_INTERIM_MIN s or more."""
  2987. t = cls.target()
  2988. if t is None:
  2989. raise Fail("no voice enrolled", "not_enrolled")
  2990. if not vad_segments(x): # no speech heard: as ungated (recognition answers blank as ever)
  2991. return {"match": True, "similarity": None, "threshold": threshold, "dropped": 0, "drop": []}, x
  2992. spk = cls.speakers(x, t)
  2993. judged = [s["sim"] for s in spk if s["sim"] is not None and (not interim or s["seconds"] >= VP_INTERIM_MIN)]
  2994. best = max(judged, default=None)
  2995. # a short stretch scores unevenly (a sub-second clip of anyone can land above the threshold): it is kept only
  2996. # beside a long stretch that is surely the user — a piece of someone else alone can't slip through on it
  2997. long_mine = any(s["sim"] is not None and s["seconds"] >= VP_MARGIN_MIN and s["sim"] >= threshold for s in spk)
  2998. any_long = any(s["seconds"] >= VP_MARGIN_MIN for s in spk)
  2999. for s in spk:
  3000. v = s["sim"]
  3001. short = s["seconds"] < VP_MARGIN_MIN
  3002. s["keep"] = (interim and s["seconds"] < VP_INTERIM_MIN) or (v is None and (long_mine or not judged)) or (v is not None and
  3003. v >= threshold and (v >= best - VP_MARGIN or v >= VP_SURE or short)
  3004. and (not short or long_mine or not any_long))
  3005. # a fragment too short to score (< 0.5 s) goes with the piece: kept beside the user, dropped beside others
  3006. kept = [s for s in spk if s["keep"]]
  3007. dropped = [s for s in spk if not s["keep"]]
  3008. # learning: a piece of one voice only, long and surely the user's against the enrollment itself
  3009. e0 = cls.enrolled()
  3010. if not interim and len(spk) == 1 and spk[0]["seconds"] >= VP_LEARN_MIN and e0 is not None:
  3011. e_ = cls.embed(spk[0]["audio"][:10 * SR])
  3012. if float(np.dot(e_, e0)) >= VP_LEARN:
  3013. cls.learn(e_)
  3014. sims = [s["sim"] for s in kept if s["sim"] is not None]
  3015. out = {"match": bool(kept) or not spk, "threshold": threshold, "dropped": len(dropped),
  3016. "similarity": round(min(sims), 3) if sims else round(best, 3) if best is not None else None,
  3017. "speakers": [round(s["sim"], 3) if s["sim"] is not None else None for s in spk],
  3018. "drop": [[round(a, 3), round(b, 3)] for s in dropped for a, b in s["turns"]]}
  3019. if not dropped or not kept:
  3020. return out, x
  3021. mask = np.ones(len(x), dtype=np.float32)
  3022. r = int(0.01 * SR)
  3023. for s in dropped:
  3024. for a_, b_ in s["turns"]:
  3025. a, b = int(a_ * SR), min(len(x), int(b_ * SR))
  3026. mask[a:b] = 0
  3027. n = min(r, a)
  3028. mask[a - n:a] = np.minimum(mask[a - n:a], np.linspace(1, 0, n, dtype=np.float32))
  3029. n = min(r, len(x) - b)
  3030. mask[b:b + n] = np.minimum(mask[b:b + n], np.linspace(0, 1, n, dtype=np.float32))
  3031. for s in kept: # two at once: the user's turn wins
  3032. for a_, b_ in s["turns"]:
  3033. mask[int(a_ * SR):int(b_ * SR)] = 1
  3034. return out, x * mask
  3035. @classmethod
  3036. def learn(cls, e: np.ndarray) -> None:
  3037. """A cut that is surely the user's joins the profile (at most one per VP_LEARN_EVERY s, the latest
  3038. VP_LEARNED_MAX kept)."""
  3039. now = time.time()
  3040. if now - cls._learned_at < VP_LEARN_EVERY:
  3041. return
  3042. p = cls.profile()
  3043. if not p.get("embeddings"):
  3044. return
  3045. cls._learned_at = now
  3046. p["version"] = 2
  3047. p["learned"] = ((p.get("learned") or []) + [[round(float(v), 5) for v in e]])[-VP_LEARNED_MAX:]
  3048. p["updated"] = datetime.now().astimezone().isoformat(timespec="seconds")
  3049. save_json(cls.PROFILE, p)
  3050. @classmethod
  3051. def enroll(cls, wav: Path) -> dict:
  3052. """One enrollment take (Settings' "Only my voice": a passage read aloud): its windows join the profile (the
  3053. latest TAKES takes; what dictation taught it stays). Refused when there is too little speech in it — room tone
  3054. is never learned. A profile another model made is replaced."""
  3055. t0 = time.perf_counter()
  3056. if not wav.is_file():
  3057. return {"ok": False, "code": "not_found", "error": f"{wav} not found"}
  3058. x = read_wav(wav)
  3059. es = cls.embed_windows(x)
  3060. if len(es) < 2: # under ~8 s of speech
  3061. return {"ok": False, "code": "too_short", "audio_seconds": round(len(x) / SR, 2),
  3062. "error": "too little speech — read the passage aloud"}
  3063. p = cls.profile()
  3064. takes = [t for t in (p.get("takes") or []) if isinstance(t, list)][-(cls.TAKES - 1):] + [
  3065. [[round(float(v), 5) for v in e] for e in es]]
  3066. save_json(cls.PROFILE, {"model": VP_FILE, "version": 2, "takes": takes, "embeddings": [e for t in takes for e in t],
  3067. "learned": p.get("learned") or [],
  3068. "updated": datetime.now().astimezone().isoformat(timespec="seconds")})
  3069. return {**cls.status(), "audio_seconds": round(len(x) / SR, 2), "ms": int((time.perf_counter() - t0) * 1000)}
  3070. @classmethod
  3071. def prepare(cls) -> dict:
  3072. """The model on this Mac (downloaded when missing: GitHub, then Hugging Face and its mirror), then the status."""
  3073. download(VP_FILE)
  3074. download(VP_SEG_FILE)
  3075. return cls.status()
  3076. @classmethod
  3077. def status(cls) -> dict:
  3078. p = cls.profile()
  3079. return {"ok": True, "enrolled": bool(p.get("embeddings")), "stale": bool(p.get("stale")),
  3080. "takes": len(p.get("takes") or []), "learned": len(p.get("learned") or []), "model": VP_FILE,
  3081. "model_ready": (MODELS_DIR / VP_FILE).is_file() and (MODELS_DIR / VP_SEG_FILE).is_file(), "model_mb": VP_MB, "updated": p.get("updated")}
  3082. @classmethod
  3083. def clear(cls) -> dict:
  3084. cls.PROFILE.unlink(missing_ok=True)
  3085. cls._target = None
  3086. return cls.status()
  3087. class NoteEngine:
  3088. """The voice-note model, loaded once (in-process `note`, or kept warm by `serve`)."""
  3089. def __init__(self):
  3090. self.backend = self.name = self.model_id = None
  3091. self.last = time.time()
  3092. def load(self, language: str | None = None, model_id: str | None = None):
  3093. a = argparse.Namespace(backend=None, model=None, model_id=model_id, language=language, prompt=None)
  3094. name = resolve_backend(a, "note")
  3095. if self.backend is None or a.model_id != self.model_id or (language or None) != getattr(self, "language", None):
  3096. if self.backend is not None and a.model_id != self.model_id and hasattr(self.backend, "close"):
  3097. self.backend.close() # its model goes before the next one loads (a language change keeps it)
  3098. elif self.backend is not None and hasattr(self.backend, "_ex"):
  3099. self.backend._ex.shutdown(wait=False) # the same model: only its thread goes
  3100. self.backend = make_backend(name, a, "note")
  3101. self.name, self.model_id, self.language = name, a.model_id, language or None
  3102. return self
  3103. def note(self, wav: Path, language: str | None = None, model_id: str | None = None, vocab=None,
  3104. purpose: str | None = None, prior: str | None = None, hint: str | None = None, stop=None,
  3105. app: str | None = None, before: str | None = None, voiceprint: bool | None = None) -> dict:
  3106. """vocab: None = no words; True = the file's; a list = these (strings or term dicts). The model is biased toward
  3107. them where it can be (Whisper's prompt, Qwen3-ASR's context / hotwords); they are hints, so `text` (and
  3108. `asr_text`, the same) is what the model wrote. A misheard name is polish's to fix: it gets the words and sees
  3109. the sentence.
  3110. hint: the target app's window title and the text around the cursor — words seen there are listed to the model
  3111. first.
  3112. app: the name of the app the text goes into ("飞书"), named to the model before the words — what the speech is
  3113. likely about where two words sound the same (飞书会话 vs 绘画: 4 → 8 of 12 synthetic takes right). Only for a
  3114. backend that takes words (Qwen3-ASR's context); nothing else is said about the app.
  3115. before: the field's text before the cursor (the app sends it only with "Use the text around the cursor" on, never
  3116. from a password field). Heard as context only by a model whose registry entry has field_context_chars (its
  3117. last that many characters, ahead of the text so far); never logged or answered back.
  3118. stop: an interim request (the live words of dictation and of notes) — a callable that says another request is
  3119. waiting; the decode then raises Interrupted. An interruptible backend looks at it between steps; a quick one
  3120. (`quick_decode`: SenseVoice, one ~0.1 s pass) is asked once before and then decodes whole — a waiting final
  3121. piece waits that long at most. Any other backend (a slow one that can't stop: Whisper on CT2, Qwen3-ASR on the
  3122. CPU) answers Fail "interim_unsupported"."""
  3123. t0 = time.perf_counter()
  3124. language = None if language in (None, "", "auto") else language
  3125. if purpose == "dictation" and not model_id:
  3126. model_id = dictation_models()["selected"]
  3127. x = read_wav(wav)
  3128. speaker = None
  3129. vp_on, vp_threshold = voiceprint_settings(voiceprint)
  3130. if vp_on and purpose == "dictation":
  3131. # only my voice (issue #26): someone else's speech is silenced out of what recognition hears, a piece that
  3132. # is all someone else answers blank right here, before the model is even loaded. An interim request (stop
  3133. # set) judges only cuts long enough to tell. No profile, no model, the gate failing: as if it were off.
  3134. with no_download(True): # a dictation never waits for the voiceprint model either
  3135. try:
  3136. speaker, x = Voiceprint.gate(x, vp_threshold, interim=stop is not None)
  3137. except (Fail, ImportError, RuntimeError, ValueError):
  3138. speaker = None
  3139. if speaker is not None and not speaker["match"]:
  3140. return {"ok": True, "text": "", "language": language, "ms": int((time.perf_counter() - t0) * 1000),
  3141. "model": model_id or self.model_id, "audio_seconds": round(len(x) / SR, 2), "load_ms": 0,
  3142. "speaker": speaker}
  3143. was = self.backend
  3144. t_load = time.perf_counter()
  3145. self.load(language, model_id)
  3146. load_ms = int((time.perf_counter() - t_load) * 1000) if self.backend is not was else 0
  3147. if stop is not None and not getattr(self.backend, "interruptible", False):
  3148. if not getattr(self.backend, "quick_decode", False):
  3149. raise Fail(f"{self.name} can't stop a decode halfway: no interim requests", "interim_unsupported")
  3150. if stop():
  3151. raise Interrupted("another request is waiting")
  3152. stop = None # one short pass: nothing to look at halfway
  3153. biased = bias(self.backend, vocab, prior, hint, app)
  3154. if prior and hasattr(self.backend, "set_prior"):
  3155. self.backend.set_prior(prior_tail(prior))
  3156. field = ""
  3157. if before and hasattr(self.backend, "set_field") and (n := field_context_chars(self.model_id)):
  3158. field = prior_tail(before, n)
  3159. self.backend.set_field(field)
  3160. retried = False
  3161. if stop is not None:
  3162. self.backend.stop_check = stop
  3163. # a cloud service hears a long take's parts side by side (a take at key-up: dictation-eval §5, Flash grows
  3164. # from 0.5 to 1.5 s with the take when sent as one request); a local model has one decoder
  3165. workers = CLOUD_WORKERS if self.name in API_KEYS else 1
  3166. if self.name in API_KEYS:
  3167. self.backend.last = {}
  3168. if hasattr(self.backend, "looped"):
  3169. self.backend.looped = False
  3170. try:
  3171. results, _speech, _blank = transcribe_track(x, self.backend, self.name, workers, quiet=True)
  3172. # Qwen3-ASR now and then answers nothing for a short piece that follows a whole sentence of context (a
  3173. # dictation's last words after a long pause: measured 2–5 of 6 takes of one synthetic clip, never when the
  3174. # same request was replayed). The VAD heard speech, so once more without the context: ~0.13 s, only then.
  3175. if (prior or field) and _speech >= 0.4 and not note_text(results)[0].strip() \
  3176. and hasattr(self.backend, "set_prior"):
  3177. self.backend.set_prior("")
  3178. if field:
  3179. self.backend.set_field("")
  3180. results, _speech, _blank = transcribe_track(x, self.backend, self.name, workers, quiet=True)
  3181. retried = True
  3182. finally:
  3183. if stop is not None:
  3184. self.backend.stop_check = None
  3185. if biased:
  3186. self.backend.set_vocab([]) # a voice note after this gets no dictation words
  3187. if prior and hasattr(self.backend, "set_prior"):
  3188. self.backend.set_prior("")
  3189. if field:
  3190. self.backend.set_field("")
  3191. text, lang = note_text(results)
  3192. out = {"ok": True, "text": text, "language": lang or language, "ms": int((time.perf_counter() - t0) * 1000),
  3193. "model": self.model_id, "audio_seconds": round(len(x) / SR, 2), "load_ms": load_ms}
  3194. if speaker:
  3195. out["speaker"] = speaker
  3196. if retried:
  3197. out["retried"] = True
  3198. if getattr(self.backend, "looped", False):
  3199. out["looped"] = True # a repetition loop was cut out of it: words may be missing (never sent)
  3200. if field:
  3201. out["field_context"] = len(field) # how much of it was heard (never the text)
  3202. tag = getattr(self.backend, "last", None)
  3203. if tag:
  3204. out["tag"] = tag # the language tag chosen and the model's English / preferred probabilities
  3205. if vocab is not None:
  3206. out.update(asr_text=text, biased=biased)
  3207. self.last = time.time()
  3208. return out
  3209. def bias(backend, vocab, prior: str | None = None, hint: str | None = None, app: str | None = None) -> bool:
  3210. """The request's words (None: none; True: the file's; a list: these) given to a backend that takes them: the most
  3211. relevant `max_terms` of them — ranked() cuts by relevance, then names them in a fixed order (a backend that took
  3212. fewer than it was given would cut alphabetically: Whisper's 40 of 64, a third of the right words gone,
  3213. contextual-earnings22) — with the app's name first. True when the backend was biased."""
  3214. if not hasattr(backend, "set_vocab"):
  3215. return False
  3216. terms = dictation_vocab.load() if vocab is True else dictation_vocab.normalize(vocab) if vocab else []
  3217. cap = getattr(backend, "max_terms", dictation_vocab.QWEN_MAX_TERMS)
  3218. limit = min(PRIOR_TERMS, cap) if prior else cap
  3219. use = dictation_vocab.ranked(terms, prior or "", hint or "", limit)
  3220. # the app's name goes first — not to a model whose words are built into it (FunAsrOnnx: a new word list is a new
  3221. # recognizer, ~3 s on a CPU; an app switch must not cost that)
  3222. name = " ".join(str(app or "").split())[:40] if getattr(backend, "app_hint", True) else ""
  3223. if name and not any(t["term"] == name for t in use):
  3224. use = [{"term": name, "sounds_like": [], "source": "app", "hits": 0}] + use[:max(0, limit - 1)]
  3225. return backend.set_vocab(use)
  3226. # --- a take streamed while it is said (serve "stream"; docs/design/dictation.md §Cloud recognition)
  3227. # A recogniser that hears audio as it comes (Volcengine's `streams`) gets a dictation's microphone packets during the
  3228. # take, so at key-up only its last ~15 s block is left (dictation-eval-2026-10-31.md §5: ~0.5 s however long the take,
  3229. # against 1.5 s for a minute-long take sent whole). The app opens one connection per take: one JSON line
  3230. # ({"token", "cmd": "stream", "model_id", "language", "vocab", "app", "hint", "before"}), then raw 16 kHz s16 mono PCM
  3231. # as the microphone gives it, then shuts its side for writing at key-up. The answer is one JSON line, a note's answer
  3232. # (+ "streamed", "after_keyup_ms"), after the end of the audio — a refusal (no key, quota: "quota exceeded for types:
  3233. # concurrency", a model that can't take a stream) too: the PCM is read to its end first, so the answer is never lost
  3234. # to a reset. A connection the app resets (a cancelled take) ends the session unanswered. Each stream runs on its own
  3235. # thread beside the server's loop; the server doesn't end while one is open.
  3236. STREAM_FINAL_WAIT = 10.0 # s after key-up for the final text (measured p90: 0.58–0.70 s)
  3237. # …plus this much per second of audio that came faster than it is spoken (a server that came up late gets the take so
  3238. # far at once): the service works through a backlog at ~4× real time (the eval's unpaced takes: ~0.23 s per second)
  3239. STREAM_BACKLOG_WAIT = 0.4
  3240. def voiceprint_utterances(text: str, utterances: list | None, speaker: dict) -> str:
  3241. """A stream's text without another speaker's sentences: those whose time lies mostly in the gate's dropped spans.
  3242. All someone else → ""; nothing dropped, or no sentence times to go by → the text as it came."""
  3243. if not speaker.get("match"):
  3244. return ""
  3245. drop = speaker.get("drop") or []
  3246. if not drop or not utterances:
  3247. return text
  3248. kept = []
  3249. for u in utterances:
  3250. s, e = (u.get("start_time") or 0) / 1000, (u.get("end_time") or 0) / 1000
  3251. gone = sum(max(0.0, min(e, b) - max(s, a)) for a, b in drop)
  3252. if e <= s or gone < 0.5 * (e - s):
  3253. kept.append(str(u.get("text") or "").strip())
  3254. out = ""
  3255. for k in filter(None, kept):
  3256. cjk = re.match(f"[{_CJK},。?!、]", out[-1:] or " ") or re.match(f"[{_CJK}]", k[0])
  3257. out += k if not out or cjk else " " + k
  3258. return out
  3259. def final_wait(audio_s: float, streamed_s: float) -> float:
  3260. """How long a take's final text is waited for after key-up: `audio_s` of audio arrived over `streamed_s` s."""
  3261. return STREAM_FINAL_WAIT + STREAM_BACKLOG_WAIT * max(0.0, audio_s - streamed_s)
  3262. _streams = 0
  3263. _streams_lock = threading.Lock()
  3264. def streams_open() -> int:
  3265. with _streams_lock:
  3266. return _streams
  3267. def serve_stream(conn: socket.socket, req: dict, extra: bytes) -> None:
  3268. global _streams
  3269. with _streams_lock:
  3270. _streams += 1
  3271. sess = None
  3272. fail: Fail | None = None
  3273. got = 0
  3274. odd = b""
  3275. t0 = time.perf_counter()
  3276. model_id = req.get("model_id")
  3277. biased = False
  3278. field = ""
  3279. gate = False
  3280. pcm = bytearray()
  3281. speaker = None
  3282. try:
  3283. conn.settimeout(120) # a take sends a packet every ~0.1 s: two minutes of nothing is an app gone
  3284. try:
  3285. if local_only():
  3286. raise Fail("local-only recognition is on (privacy.local_asr): cloud ASR is not allowed", "privacy")
  3287. model_id = model_id or dictation_models()["selected"]
  3288. m = entry(model_id)
  3289. cls = BACKENDS.get((m or {}).get("backend"))
  3290. if not m or not getattr(cls, "streams", False):
  3291. raise Fail(f"{model_id} doesn't take a stream", "usage")
  3292. lang = req.get("language")
  3293. a = argparse.Namespace(model=None, model_id=model_id, prompt=None,
  3294. language=None if lang in (None, "", "auto") else lang)
  3295. backend = cls(a)
  3296. # only my voice: the PCM is kept, judged at key-up while the service finishes, and another speaker's
  3297. # sentences are left out of the text by their times (the audio itself has gone out as it was said)
  3298. vp_on, vp_threshold = voiceprint_settings(req.get("voiceprint"))
  3299. gate = vp_on and req.get("purpose", "dictation") == "dictation" and Voiceprint.target() is not None \
  3300. and (MODELS_DIR / VP_FILE).is_file() and (MODELS_DIR / VP_SEG_FILE).is_file()
  3301. backend.utterances = gate
  3302. biased = bias(backend, req.get("vocab"), None, req.get("hint"), req.get("app"))
  3303. if req.get("before") and (n := field_context_chars(model_id)):
  3304. field = prior_tail(req["before"], n)
  3305. backend.set_field(field)
  3306. sess = backend.open()
  3307. except Fail as e:
  3308. fail = e
  3309. data = extra
  3310. t_first = None
  3311. while True:
  3312. if data:
  3313. t_first = t_first or time.perf_counter()
  3314. got += len(data)
  3315. data = odd + data
  3316. odd = data[len(data) & ~1:] # whole samples only
  3317. if sess is not None:
  3318. sess.feed(data[:len(data) & ~1])
  3319. if gate:
  3320. pcm += data[:len(data) & ~1]
  3321. data = conn.recv(65536)
  3322. if not data:
  3323. break
  3324. t_up = time.perf_counter()
  3325. text = ""
  3326. if fail is None and sess is not None:
  3327. judged: dict = {}
  3328. judge = None
  3329. if gate and pcm:
  3330. def run_gate():
  3331. with contextlib.suppress(Fail, ImportError, RuntimeError, ValueError):
  3332. x = np.frombuffer(bytes(pcm), dtype="<i2").astype(np.float32) / 32768
  3333. judged["speaker"] = Voiceprint.gate(x, vp_threshold)[0]
  3334. judge = threading.Thread(target=run_gate, daemon=True)
  3335. judge.start()
  3336. try:
  3337. text = sess.finish(wait=final_wait(got / 2 / SR, t_up - (t_first or t_up)))
  3338. if judge is not None:
  3339. judge.join(timeout=5)
  3340. speaker = judged.get("speaker")
  3341. if speaker is not None:
  3342. text = voiceprint_utterances(text, sess.utterances, speaker)
  3343. text = to_user_script(text, None)
  3344. except Fail as e:
  3345. fail = e
  3346. if fail is not None:
  3347. resp = {"ok": False, "error": str(fail), "code": fail.code, "streamed": True}
  3348. else:
  3349. resp = {"ok": True, "text": text, "asr_text": text, "language": a.language, "model": model_id,
  3350. "audio_seconds": round(got / 2 / SR, 2), "ms": int((time.perf_counter() - t_up) * 1000),
  3351. "after_keyup_ms": int((time.perf_counter() - t_up) * 1000), "load_ms": 0, "biased": biased,
  3352. "streamed": True, "stream_s": round(t_up - t0, 2)}
  3353. if field:
  3354. resp["field_context"] = len(field)
  3355. if speaker is not None:
  3356. resp["speaker"] = speaker
  3357. with contextlib.suppress(OSError):
  3358. conn.sendall((json.dumps(resp, ensure_ascii=False) + "\n").encode())
  3359. except (OSError, ValueError) as e:
  3360. # the app reset the connection (a cancelled take) or went away: the session ends unanswered
  3361. serve_said(f"serve: stream ended ({type(e).__name__})")
  3362. finally:
  3363. if sess is not None:
  3364. sess.close()
  3365. with contextlib.suppress(OSError):
  3366. conn.close()
  3367. with _streams_lock:
  3368. _streams -= 1
  3369. SERVE_FILE = RUN_DIR / "asr-serve.json"
  3370. def _server_call(req: dict, timeout: float = 120) -> dict | None:
  3371. info = load_json(SERVE_FILE, None)
  3372. if not isinstance(info, dict) or not info.get("port"):
  3373. return None
  3374. try:
  3375. with socket.create_connection(("127.0.0.1", int(info["port"])), timeout=0.3) as s:
  3376. s.settimeout(timeout)
  3377. s.sendall((json.dumps({**req, "token": info.get("token")}) + "\n").encode())
  3378. buf = b""
  3379. while not buf.endswith(b"\n"):
  3380. chunk = s.recv(65536)
  3381. if not chunk:
  3382. break
  3383. buf += chunk
  3384. return json.loads(buf) if buf else None
  3385. except (OSError, ValueError):
  3386. return None
  3387. @contextlib.contextmanager
  3388. def no_download(on: bool):
  3389. """While on: a model that isn't on this Mac is an error (model_missing), never a download — a dictation never waits
  3390. for one (`note --no-download`, `serve --no-download` for its preload, a request's "no_download")."""
  3391. if not on or os.environ.get("BLURT_NO_DOWNLOAD"):
  3392. yield
  3393. return
  3394. os.environ["BLURT_NO_DOWNLOAD"] = "1"
  3395. try:
  3396. yield
  3397. finally:
  3398. os.environ.pop("BLURT_NO_DOWNLOAD", None)
  3399. def cmd_note(a):
  3400. if getattr(a, "no_download", False):
  3401. os.environ["BLURT_NO_DOWNLOAD"] = "1"
  3402. wav = Path(a.wav).resolve()
  3403. if not wav.exists():
  3404. raise Fail(f"{wav} not found", "not_found")
  3405. vocab = True if a.vocab else None
  3406. purpose = "dictation" if a.dictation else None
  3407. if not a.no_server:
  3408. r = _server_call({"cmd": "note", "wav": str(wav), "language": a.language, "model_id": a.model_id,
  3409. "vocab": vocab, "purpose": purpose, "no_download": bool(getattr(a, "no_download", False))})
  3410. if r is not None:
  3411. if not r.get("ok"):
  3412. raise Fail(r.get("error") or "server error", r.get("code") or "error")
  3413. return emit({**r, "server": True})
  3414. throttle("note")
  3415. emit(NoteEngine().note(wav, a.language, a.model_id, vocab, purpose))
  3416. def voiceprint_verdict(wav: Path) -> dict:
  3417. """The gate's verdict on a dictation piece without recognising it (serve "vp_gate", `voiceprint check`): {"ok",
  3418. "speaker"}; "speaker" is null when the gate can't judge (no voice enrolled, no model: as if it were off)."""
  3419. x = read_wav(wav)
  3420. _, threshold = voiceprint_settings(None)
  3421. with no_download(True):
  3422. try:
  3423. speaker, _ = Voiceprint.gate(x, threshold)
  3424. except (Fail, ImportError, RuntimeError, ValueError) as e:
  3425. return {"ok": True, "speaker": None, "why": getattr(e, "code", type(e).__name__)}
  3426. return {"ok": True, "speaker": speaker, "audio_seconds": round(len(x) / SR, 2)}
  3427. def cmd_voiceprint(a):
  3428. """`voiceprint prepare` (the model on this Mac, downloaded when missing), `enroll <wav>` (learn the user's voice
  3429. from a passage read aloud), `check <wav>` (the gate's verdict on it), `status`, `clear`. The warm server answers
  3430. vp_gate / vp_status / vp_clear; enrolling is only here, so the server never waits for a download."""
  3431. if a.vp_cmd in ("enroll", "check") and not a.wav:
  3432. raise Fail(f"voiceprint {a.vp_cmd} needs a wav", "usage")
  3433. if a.vp_cmd == "prepare":
  3434. emit(Voiceprint.prepare())
  3435. elif a.vp_cmd == "enroll":
  3436. Voiceprint.prepare()
  3437. emit(Voiceprint.enroll(Path(a.wav).resolve()))
  3438. elif a.vp_cmd == "check":
  3439. emit(voiceprint_verdict(Path(a.wav).resolve()))
  3440. elif a.vp_cmd == "status":
  3441. emit(Voiceprint.status())
  3442. elif a.vp_cmd == "clear":
  3443. emit(Voiceprint.clear())
  3444. def cmd_serve(a):
  3445. """One warm voice-note model per machine. Protocol: one JSON line in ({"token","cmd":"note"|"load"|"ping"|"exit",
  3446. …}), one JSON line out. A "note" with "interim": true (dictation's live text: the piece still being said) is
  3447. stopped as soon as another connection waits — a final piece never waits for it — and answers
  3448. {"ok": false, "code": "interrupted"}; a quick model (SenseVoice) decodes it whole; a slow model that can't stop
  3449. halfway answers "interim_unsupported". A "stream" (a dictation's audio while it is said, for a recogniser that hears it
  3450. as it comes: serve_stream) runs on its own thread; the server doesn't end while one is open. Ends
  3451. (serve_exit_reason) when its model went unused for --idle s — an unload left most of
  3452. the memory resident, so the process goes and the app starts it again on the next key-down — when the app that
  3453. started it (--parent) is gone, on "exit", or after --exit-after s without a request while no model is loaded."""
  3454. global SERVING
  3455. if (r := _server_call({"cmd": "ping"}, timeout=2)) and r.get("ok"):
  3456. return emit({"ok": True, "already": True, "pid": r.get("pid")})
  3457. # every installed model's package now, before there is a run file (serve_packages); one that can't be added
  3458. # (offline with nothing cached) leaves it to a restart when that model is asked for
  3459. missing = [p for p in serve_packages() if not _has_package(p)
  3460. and p not in os.environ.get("BLURT_ASR_REEXEC", "").split(",")]
  3461. if missing:
  3462. try:
  3463. _reexec_with(missing)
  3464. except Fail as e:
  3465. serve_said(f"serve: packages not added ({e})")
  3466. throttle("note")
  3467. RUN_DIR.mkdir(parents=True, exist_ok=True)
  3468. token = uuid.uuid4().hex
  3469. srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  3470. srv.bind(("127.0.0.1", 0))
  3471. srv.listen(4)
  3472. srv.settimeout(1.0)
  3473. port = srv.getsockname()[1]
  3474. save_json(SERVE_FILE, {"pid": os.getpid(), "port": port, "token": token, "parent": a.parent,
  3475. "started": datetime.now().astimezone().isoformat(timespec="seconds")})
  3476. eng = NoteEngine()
  3477. why = None
  3478. restart: str | None = None # a package this server must restart with (NeedsRestart)
  3479. SERVING = True
  3480. # from here on the run file is ours to remove, whatever happens — a preload that fails included (a run file left
  3481. # pointing at a dead or stuck process made the app wait for a server that never answered)
  3482. try:
  3483. if a.preload:
  3484. mid = getattr(a, "model_id", None)
  3485. if not mid and getattr(a, "purpose", None) == "dictation":
  3486. mid = dictation_models()["selected"]
  3487. # --no-download: a model that isn't here stays unloaded (the server still answers: model_missing)
  3488. try:
  3489. with no_download(getattr(a, "no_download", False)):
  3490. eng.load(None, mid)
  3491. except Fail as e:
  3492. if e.code != "model_missing":
  3493. raise
  3494. serve_said(f"serve: preload skipped ({e})")
  3495. except NeedsRestart as e:
  3496. restart, why = e.pkg, "restart"
  3497. if voiceprint_settings(None)[0] and Voiceprint.target() is not None and (MODELS_DIR / VP_FILE).is_file():
  3498. # only my voice: its models too (~0.3 s), so the first piece doesn't wait for them
  3499. def warm_voiceprint():
  3500. with contextlib.suppress(Exception):
  3501. Voiceprint.extractor()
  3502. Voiceprint.diarizer()
  3503. threading.Thread(target=warm_voiceprint, daemon=True).start()
  3504. emit({"ok": True, "pid": os.getpid(), "port": port, "model": eng.model_id})
  3505. last = time.time()
  3506. uses: list[float] = [] # dictation requests: kept warm longer while dictation is used often
  3507. tier = probe()["tier"]
  3508. # a CPU model left at the default idle: an hour where memory allows (_common.CPU_SERVE_IDLE)
  3509. idle = CPU_SERVE_IDLE if not IS_MAC and a.idle == SERVE_IDLE and tier in ("pro", "standard") else a.idle
  3510. while not why:
  3511. now = time.time()
  3512. if streams_open():
  3513. last = now # a take is streaming: never idle
  3514. why = serve_exit_reason(eng.backend is not None, now - eng.last, now - last, uses, tier, idle,
  3515. a.exit_after, a.parent)
  3516. if why:
  3517. break
  3518. try:
  3519. conn, _ = srv.accept()
  3520. except socket.timeout:
  3521. continue
  3522. with conn:
  3523. conn.settimeout(10)
  3524. try:
  3525. buf = b""
  3526. while b"\n" not in buf and len(buf) < 1 << 20: # a vocabulary can be long
  3527. chunk = conn.recv(65536)
  3528. if not chunk:
  3529. break
  3530. buf += chunk
  3531. line, _, extra = buf.partition(b"\n") # a stream's PCM may follow its line
  3532. req = json.loads(line or b"{}")
  3533. if req.get("token") != token:
  3534. resp = {"ok": False, "error": "bad token", "code": "auth"}
  3535. elif req.get("cmd") == "stream":
  3536. # its own thread, its own copy of the connection (this one closes with the block)
  3537. last = time.time()
  3538. if req.get("purpose", "dictation") == "dictation":
  3539. uses.append(last)
  3540. del uses[:-50]
  3541. threading.Thread(target=serve_stream, args=(conn.dup(), req, extra), daemon=True).start()
  3542. continue
  3543. elif req.get("cmd") == "ping":
  3544. resp = {"ok": True, "pid": os.getpid(), "model": eng.model_id, "loaded": eng.backend is not None}
  3545. elif req.get("cmd") == "note":
  3546. last = time.time()
  3547. interim = bool(req.get("interim"))
  3548. if req.get("purpose") == "dictation" and not interim:
  3549. uses.append(last)
  3550. del uses[:-50]
  3551. # an interim decode looks for a waiting connection (a final piece, anything) between steps
  3552. stop = (lambda: bool(_select.select([srv], [], [], 0)[0])) if interim else None
  3553. # a dictation (final or interim) never waits for a download, whatever the client sent
  3554. with no_download(bool(req.get("no_download")) or req.get("purpose") == "dictation"):
  3555. resp = eng.note(Path(req["wav"]), req.get("language"), req.get("model_id"), req.get("vocab"),
  3556. req.get("purpose"), req.get("prior"), req.get("hint"), stop=stop,
  3557. app=req.get("app"), before=req.get("before"),
  3558. voiceprint=req.get("voiceprint"))
  3559. if interim:
  3560. resp["interim"] = True
  3561. elif req.get("cmd") == "vp_gate":
  3562. # only my voice for a piece macOS's recogniser hears (dictation on Apple Speech, or a cloud
  3563. # recogniser's live text): the verdict alone, the app silences the dropped spans itself
  3564. last = time.time()
  3565. resp = voiceprint_verdict(Path(req["wav"]))
  3566. elif req.get("cmd") == "vp_status": # enrolling (it may download) is the CLI's: never here
  3567. resp = Voiceprint.status()
  3568. elif req.get("cmd") == "vp_clear":
  3569. resp = Voiceprint.clear()
  3570. elif req.get("cmd") == "load": # warm a model now (dictation's key went down)
  3571. last = time.time()
  3572. mid = req.get("model_id") or (dictation_models()["selected"] if req.get("purpose") == "dictation" else None)
  3573. with no_download(bool(req.get("no_download")) or req.get("purpose") == "dictation"):
  3574. eng.load(None, mid)
  3575. eng.last = time.time()
  3576. resp = {"ok": True, "model": eng.model_id}
  3577. elif req.get("cmd") == "exit": # the app is quitting (or someone wants the memory back now)
  3578. resp = {"ok": True, "pid": os.getpid()}
  3579. why = "asked"
  3580. else:
  3581. resp = {"ok": False, "error": "unknown cmd", "code": "usage"}
  3582. except Fail as e:
  3583. resp = {"ok": False, "error": str(e), "code": e.code}
  3584. except NeedsRestart as e:
  3585. # answered, then gone: the app waits for the new server (it comes back with the model loaded when
  3586. # it was started with --preload) instead of sending the take to a cold CLI run
  3587. resp = {"ok": False, "error": str(e), "code": "restarting"}
  3588. restart, why = e.pkg, "restart"
  3589. except Interrupted:
  3590. resp = {"ok": False, "error": "interrupted", "code": "interrupted", "interim": True}
  3591. except Exception as e: # noqa: BLE001 — keep serving
  3592. resp = {"ok": False, "error": f"{type(e).__name__}: {e}"[:300], "code": "error"}
  3593. try:
  3594. conn.sendall((json.dumps(resp, ensure_ascii=False) + "\n").encode())
  3595. except OSError:
  3596. pass
  3597. finally:
  3598. SERVING = False
  3599. info = load_json(SERVE_FILE, None)
  3600. if isinstance(info, dict) and info.get("pid") == os.getpid():
  3601. SERVE_FILE.unlink(missing_ok=True)
  3602. srv.close()
  3603. serve_said(f"serve: exiting ({why or 'error'})")
  3604. if restart:
  3605. _reexec_with([restart])
  3606. # --- probe / recommend / auto / models / setup
  3607. def cmd_probe(_a):
  3608. emit({"ok": True, **probe(), "platform": platform_key(), "limits": limits()})
  3609. def _langs_arg(v) -> list[str] | None:
  3610. return [x.strip() for x in v.split(",") if x.strip()] if v else None
  3611. def cmd_recommend(a):
  3612. emit(recommend(_langs_arg(a.languages), a.purpose))
  3613. def auto_plan(apply: bool) -> dict:
  3614. """The per-purpose pick for this machine and the user's languages. --apply writes asr.models (only models that
  3615. are installed — the rest are listed under needs_download for the host to ask about) and the hardware fingerprint."""
  3616. cfg = asr_config()
  3617. hw = probe()
  3618. picks, needs, keep = {}, [], {}
  3619. for p in PURPOSES:
  3620. r = recommend(purpose=p, hw=hw, cfg=cfg)
  3621. mid = r["pick"]["model"]
  3622. m = entry(mid)
  3623. if installed(m):
  3624. picks[p] = keep[p] = mid
  3625. else:
  3626. fallback = select(p) # what runs today
  3627. picks[p] = fallback["id"]
  3628. needs.append({"purpose": p, "model": mid, "size_mb": m["size_mb"]})
  3629. by_model: dict = {}
  3630. for n in needs:
  3631. by_model.setdefault(n["model"], {"model": n["model"], "size_mb": n["size_mb"], "purposes": []})["purposes"].append(n["purpose"])
  3632. out = {"ok": True, "auto": is_auto(cfg), "tier": hw["tier"], "languages": user_languages(cfg), "models": picks,
  3633. "recommended": {p: recommend(purpose=p, hw=hw, cfg=cfg)["pick"]["model"] for p in PURPOSES},
  3634. "needs_download": list(by_model.values()),
  3635. "hardware_changed": cfg.get("hardware") != hw["fingerprint"]}
  3636. if apply and is_auto(cfg):
  3637. # a config.json that doesn't parse: nothing saved, said (not a traceback) — the picks still run as defaults
  3638. # only the recommended picks are kept: a stand-in (what runs until the recommended one is in) stays unpinned,
  3639. # so select() takes the best one installed each time — a Whisper found in a shared cache on the first launch
  3640. # no longer outlasts the 1.7B downloaded after it (v1.0.8)
  3641. out["applied"] = save_config_if_readable({"asr": {"models": keep, "hardware": hw["fingerprint"]}})
  3642. if not out["applied"]:
  3643. out["saved"] = False
  3644. return out
  3645. def cmd_auto(a):
  3646. emit(auto_plan(a.apply))
  3647. # --- dictation's recognition model (record.dictation.asr_model; docs/design/dictation.md §Recognition model)
  3648. QWEN_MLX = "qwen3-asr-0.6b-mlx"
  3649. QWEN_MLX_17 = "qwen3-asr-1.7b-mlx"
  3650. FUN_MLX = "fun-asr-nano-mlx" # Chinese, English and Japanese only: offered and picked only for those (speaks_all)
  3651. FUN_CPU = "fun-asr-nano-onnx" # the same model on the CPU (sherpa-onnx int8): Windows and Linux
  3652. DICTATION_MODELS = (QWEN_MLX, QWEN_MLX_17, FUN_MLX, FUN_CPU, SV, TURBO, TURBO_Q4, QWEN_CPU, CT2, APPLE) # the local ones dictation offers, best
  3653. # first; macOS's own last (nothing to download)
  3654. # offered once their key is set (first-minute-speech.md §2.4; Volcengine: dictation-eval-2026-10-31.md §6.3 B′)
  3655. DICTATION_CLOUD = ("qwen3-asr-flash", "volcengine-seedasr", "openai-compatible")
  3656. def polish_ram_mb(hw: dict, cfg: dict | None = None) -> float:
  3657. """The memory the polish model dictation runs next to takes (polish_models.json ram_mb): the configured one, else
  3658. the tier's default — 0 where polish doesn't run (MLX: Apple Silicon only) or is turned off. Read here rather than
  3659. from polish.py, which imports this module."""
  3660. cfg = ((global_config().get("record") or {}).get("dictation") or {}) if cfg is None else cfg
  3661. if hw_kind(hw) != "apple" or (isinstance(cfg, dict) and cfg.get("polish", "light") == "off"):
  3662. return 0.0
  3663. try:
  3664. reg = json.loads((HERE / "polish_models.json").read_text(encoding="utf-8"))
  3665. except (OSError, ValueError):
  3666. return 0.0
  3667. want = (cfg.get("polish_model") if isinstance(cfg, dict) else None) or (reg.get("defaults") or {}).get(hw["tier"])
  3668. m = next((x for x in reg.get("models") or [] if x.get("id") == want), None)
  3669. return float((m or {}).get("ram_mb") or 0)
  3670. def fits_with_polish(m: dict, hw: dict, polish_mb: float) -> bool:
  3671. """`_fits`, counting the polish model too: a dictation holds both warm at once (ASR + polish), so the pair has to
  3672. stay within the share of RAM one model may take (on a 16 GB Mac: 0.6B 2.1 GB + Gemma 4 E2B 3.9 GB fits; 1.7B
  3673. 3.1 GB + 3.9 GB doesn't)."""
  3674. return _fits({**m, "ram_mb": (m.get("ram_mb") or 0) + polish_mb}, hw)
  3675. # Defaults by Mac (docs/design/dictation-eval-2026-10-31.md §6, docs/design/dictation.md §Defaults by Mac): on the
  3676. # founder's 95 takes the 1.7B (30 s windows, the field's text as context) makes 2.50 % errors, the 0.6B (20 s windows)
  3677. # 3.34 %; Gemma 4 E2B as local polish is within ±0.2 points (fillers and punctuation, not words). The 1.7B needs ~3.1 GB
  3678. # and its last 30 s window decoded at key-up: on a slow chip (a base M1 / M2, 7–10 GPU cores against the M5 Max's 40)
  3679. # that is over a second, so it goes where its measured rtf (`transcribe.py bench`, asr-bench.json) is at most
  3680. # TIER_RTF_LIMIT — or, not measured, where the chip isn't a base M1 / M2. Where the 1.7B doesn't go (8 GB, slow chips),
  3681. # Fun-ASR-Nano for users who speak only Chinese, English and Japanese (2.92 % through the app's path, 0.07 / 0.17 s
  3682. # after key-up, 2.6 GB at peak: it fits 8 GB as the 0.6B does), else the 0.6B (small_recognizer). BlurtCore
  3683. # DictationTier.recommended is the same rule for the app (Settings shows it); tests/fixtures/dictation_tier.jsonl checks
  3684. # both.
  3685. TIER_RTF_LIMIT = 0.05
  3686. SLOW_CHIPS = ("Apple M1", "Apple M2")
  3687. TIER_POLISH = "gemma4-e2b"
  3688. def speaks_all(m: dict | None, langs: list[str] | None) -> bool:
  3689. """Every one of `langs` is a language of model `m` (asr_models.json `languages`); False for no languages (nothing
  3690. known about the user is no reason for a model that speaks three)."""
  3691. return bool(m) and bool(langs) and all(speaks(m, x) for x in langs)
  3692. def small_recognizer(langs: list[str] | None) -> str:
  3693. """The recogniser for Macs the 1.7B doesn't suit (8 GB, or too slow for it): Fun-ASR-Nano when it speaks every
  3694. language of the user's (Chinese, English, Japanese) — fewer errors than the 0.6B (2.92 vs 3.34 % on the founder's
  3695. takes through the app's path; Chinese with English inside it ASCEND 6.6 vs 8.9 %, English level, Japanese 16.0 vs
  3696. 20.2 %) and as fast after key-up (dictation-eval-2026-10-31.md §2.1, §3, §5; docs/design/dictation.md §Defaults by
  3697. Mac) — else the 0.6B (30 languages)."""
  3698. return FUN_MLX if speaks_all(entry(FUN_MLX), langs) else QWEN_MLX
  3699. def cpu_recognizer(hw: dict, langs: list[str] | None) -> str:
  3700. """Dictation's recogniser on a PC without Apple Silicon (docs/design/windows-models.md): Fun-ASR-Nano int8 for
  3701. Chinese / English / Japanese speakers (3.69 % on the founder's takes, ~1 s after key-up for a 10 s piece on 4 Zen 4
  3702. threads), else Qwen3-ASR 0.6B int8 (30 languages, 3.51 %); SenseVoice on a minimal PC (< 8 GB or ≤ 2 cores:
  3703. 0.2 s, 5.2 %)."""
  3704. if hw.get("ram_gb", 16) < 7.5 or (hw.get("perf_cores") or 4) + (hw.get("eff_cores") or 0) <= 2:
  3705. return SV
  3706. return FUN_CPU if speaks_all(entry(FUN_CPU), langs) else QWEN_CPU
  3707. def slow_chip(chip: str | None) -> bool:
  3708. """A base M1 / M2 (not Pro, Max or Ultra): the chips the 1.7B is too slow on when nothing was measured."""
  3709. return " ".join(str(chip or "").split()) in SLOW_CHIPS
  3710. def dictation_tier(ram_gb: float, rtf: float | None = None, slow: bool = False, cloud_key: bool = False,
  3711. languages: list[str] | None = None) -> dict:
  3712. """What dictation runs on for a Mac with `ram_gb` of memory (Apple Silicon): the recogniser (its window and field
  3713. context come with it from asr_models.json), the local polish model (None: off), and — with a cloud key —
  3714. cloud polish with macOS's transcript as the second opinion (§6.3 A: the recogniser stays local). `rtf`: the 1.7B's
  3715. measured real-time factor here (None: not measured — then `slow`, a base M1 / M2, decides). `languages`: the
  3716. user's (user_languages): where the 1.7B doesn't go, Fun-ASR-Nano for Chinese / English / Japanese speakers, else
  3717. the 0.6B (small_recognizer). Pure."""
  3718. tier = "minimal" if ram_gb < 7.5 else "light" if ram_gb < 15 else "standard" if ram_gb < 30 else "pro"
  3719. fast = (rtf <= TIER_RTF_LIMIT) if rtf is not None else not slow
  3720. big = tier in ("standard", "pro") and fast
  3721. rec = QWEN_MLX_17 if big else small_recognizer(languages) if tier != "minimal" else None
  3722. m = entry(rec) or {}
  3723. return {"tier": tier, "recognizer": rec, "window_s": m.get("rehear_window_s") or (30 if big else 20 if rec else None),
  3724. "field_context": bool(m.get("field_context_chars")) if m else False,
  3725. "local_polish": TIER_POLISH if tier in ("standard", "pro") else None,
  3726. "cloud_polish": bool(cloud_key), "second_transcript": bool(cloud_key), "first": APPLE,
  3727. "why": "ram" if tier in ("minimal", "light") else "fast" if big else "rtf" if rtf is not None else "chip"}
  3728. def dictation_models(hw: dict | None = None) -> dict:
  3729. """configured (record.dictation.asr_model or None = automatic), recommended (by the dictation eval: Qwen3-ASR MLX
  3730. on Apple Silicon pro / standard, else the voice-note pick), selected (what runs: the configured one when it is
  3731. installed, else the recommended one when installed, else the best installed Qwen3-ASR MLX that runs here next to
  3732. the polish model, else the voice-note model) and the rows for Settings. Memory is judged for the pair dictation
  3733. holds — the recogniser plus the polish model (fits_with_polish)."""
  3734. hw = hw or probe()
  3735. cfg = (global_config().get("record") or {}).get("dictation") or {}
  3736. configured = cfg.get("asr_model") if isinstance(cfg, dict) else None
  3737. note_pick = select("note")
  3738. apple = hw_kind(hw) == "apple"
  3739. pmb = polish_ram_mb(hw, cfg if isinstance(cfg, dict) else {})
  3740. def fits(mid: str) -> bool:
  3741. m = entry(mid)
  3742. return bool(m) and fits_with_polish(m, hw, pmb)
  3743. # by Mac (dictation_tier: the 2026-10-31 eval): the 1.7B on 16 GB+ Macs fast enough for it, else Fun-ASR-Nano for
  3744. # Chinese / English / Japanese speakers, else the 0.6B — each only where it fits beside the polish model; else the
  3745. # voice-note pick (the voice-note pick is macOS's own recogniser while nothing is downloaded: the download to offer
  3746. # is then the one the voice-note rules would choose)
  3747. note_rec = note_pick["id"] if note_pick["id"] != APPLE else recommend(purpose="note", hw=hw)["pick"]["model"]
  3748. langs = user_languages()
  3749. small = small_recognizer(langs)
  3750. plan = dictation_tier(hw["ram_gb"], (_bench_results().get(QWEN_MLX_17) or {}).get("rtf"), slow_chip(hw.get("chip")),
  3751. languages=langs)
  3752. rec = next((mid for mid in ((plan["recognizer"], small, QWEN_MLX) if plan["recognizer"] else ())
  3753. if apple and fits(mid)), note_rec)
  3754. if not apple:
  3755. rec = cpu_recognizer(hw, langs)
  3756. # what runs: the configured one (the user's choice, as it is), else the recommended one, else any of the 1.7B,
  3757. # Fun-ASR-Nano (for its languages) or the 0.6B already here that runs next to polish, best first for this Mac (the
  3758. # 1.7B last where it is too slow or too big) — they take the Words list; the voice-note model doesn't — a 1.7B
  3759. # brought from a bigger Mac, or a 0.6B until Fun-ASR-Nano is in — all only when installed; else the voice-note model
  3760. order = (QWEN_MLX_17, small, QWEN_MLX) if plan["recognizer"] == QWEN_MLX_17 else (small, QWEN_MLX, QWEN_MLX_17)
  3761. usable = [mid for mid in dict.fromkeys(order) if mid != rec and apple and fits(mid)]
  3762. if not apple: # any CPU model already here, best first: Fun-ASR-Nano (its languages), the 0.6B, SenseVoice
  3763. usable = [mid for mid in ((FUN_CPU,) if speaks_all(entry(FUN_CPU), langs) else ()) + (QWEN_CPU, SV) if mid != rec]
  3764. # every local model hears through the VAD (silero_vad.onnx, 0.6 MB, which `setup` fetches first): without it a
  3765. # dictation can't run ("model_missing" under BLURT_NO_DOWNLOAD), so a model counts as installed only with it;
  3766. # `files` = the model's own files are in (then `setup` of it only fetches the VAD)
  3767. bundled_vad()
  3768. vad = (MODELS_DIR / "silero_vad.onnx").is_file()
  3769. # without the VAD, macOS's recogniser (the voice-note pick then) runs before a model whose files only are here —
  3770. # unless the user chose that model
  3771. pick = (configured,) if not vad and note_pick["id"] == APPLE else (configured, rec, *usable)
  3772. selected = next((mid for mid in pick if mid and entry(mid) and runs_here(entry(mid))
  3773. and installed(entry(mid))), note_pick["id"])
  3774. rows = []
  3775. for mid in DICTATION_MODELS + DICTATION_CLOUD:
  3776. m = entry(mid)
  3777. if not m or not runs_here(m):
  3778. continue
  3779. if not m.get("local", True) and (local_only() or not installed(m)):
  3780. continue # a cloud service: only with its key, never when recognition stays on this Mac
  3781. if mid in (FUN_MLX, FUN_CPU) and not speaks_all(m, langs) and mid not in (configured, selected):
  3782. continue # three languages: not offered to someone who speaks another (unless already chosen)
  3783. have = installed(m)
  3784. rows.append({"id": mid, "name": m.get("name"), "backend": m["backend"], "size_mb": m["size_mb"],
  3785. "ram_mb": m.get("ram_mb"), "installed": have and (vad or not m.get("local", True) or bool(m.get("builtin"))),
  3786. "files": have,
  3787. "builtin": bool(m.get("builtin")),
  3788. "vocab": m.get("vocab") or "none", "recommended": mid == rec, "fits": fits(mid)})
  3789. # ready: a dictation can run right now with nothing downloaded (it never downloads: `no_download`) — the
  3790. # selected model (local and on this Mac, or a cloud one with its key: judged by itself, not by being a row) and
  3791. # the VAD every take goes through. missing: what a download would have to bring for that ("silero-vad" and/or
  3792. # the selected model's id)
  3793. sel = entry(selected)
  3794. # macOS's recogniser hears without the VAD, and so does a cloud service for a dictation's pieces (transcribe_track)
  3795. missing = [] if vad or selected == APPLE or bool(sel and not sel.get("local", True)) else ["silero-vad"]
  3796. selected_installed = bool(sel and installed(sel))
  3797. if not selected_installed:
  3798. missing.append(selected)
  3799. return {"configured": configured, "configured_installed": bool(configured and entry(configured)
  3800. and installed(entry(configured))),
  3801. "recommended": rec, "selected": selected, "selected_installed": selected_installed,
  3802. # the dictation goes to a cloud service: the app shows live text from macOS's recogniser and doesn't hear it
  3803. # again (each pass is billed)
  3804. "selected_cloud": bool(sel and not sel.get("local", True)),
  3805. # a PC the CPU rules give the fast, less accurate model (SenseVoice: < 8 GB or ≤ 2 cores): the app offers
  3806. # cloud recognition there first (windows-models.md §Cloud recognition on a weak PC) — never switches to it
  3807. "weak": not apple and rec == SV,
  3808. # cloud recognition may be offered at all (privacy.local_asr off)
  3809. "cloud_allowed": not local_only(),
  3810. "ready": not missing, "missing": missing, "vad": vad, "models": rows, "plan": plan}
  3811. def cmd_dictation_models(_a):
  3812. emit({"ok": True, **dictation_models()})
  3813. def selected_models() -> dict:
  3814. out = {}
  3815. for p in PURPOSES:
  3816. try:
  3817. out[p] = select(p)["id"]
  3818. except Fail:
  3819. out[p] = None
  3820. return out
  3821. def cmd_models(_a):
  3822. hw = probe()
  3823. lim = limits(hw)
  3824. bench = _bench_results()
  3825. sel = selected_models()
  3826. langs = user_languages()
  3827. rows = []
  3828. for m in entries():
  3829. r = {k: v for k, v in m.items() if k not in ("install", "rtf_ref", "error_ref")}
  3830. r["local"] = m.get("local", True)
  3831. r["role"] = "asr"
  3832. r["installed"] = installed(m)
  3833. p = model_path(m) if r["local"] else None
  3834. if p:
  3835. r["path"] = str(p)
  3836. r["size_mb"], r["approx"] = _du_mb(p), False
  3837. else:
  3838. r["approx"] = r["local"]
  3839. r["runs_here"] = runs_here(m)
  3840. r["fits"] = _fits(m, hw, lim)
  3841. r["purposes"] = [k for k, v in sel.items() if v == m["id"]]
  3842. r["configured"] = bool(r["purposes"])
  3843. r["speaks"] = [x for x in langs if speaks(m, x)]
  3844. if m["id"] in bench:
  3845. r["bench"] = bench[m["id"]]
  3846. for k in ("rtf_ref", "error_ref"):
  3847. if m.get(k) is not None:
  3848. r[k] = m[k]
  3849. rows.append(r)
  3850. bundled_vad()
  3851. vad = MODELS_DIR / "silero_vad.onnx" # every local model needs it; listed so hosts can show the whole download
  3852. rows.append({"id": "silero-vad", "name": "Silero VAD", "backend": "vad", "role": "vad", "local": True,
  3853. "languages": ["*"], "installed": vad.exists(), "size_mb": _du_mb(vad) if vad.exists() else 0.6,
  3854. "approx": not vad.exists(), "path": str(vad), "purposes": [], "configured": False})
  3855. cfg = asr_config()
  3856. emit({"ok": True, "models_dir": str(MODELS_DIR), "configured": cfg, "auto": is_auto(cfg), "tier": hw["tier"],
  3857. "platform": platform_key(), "hardware": hw, "limits": lim, "languages": langs,
  3858. "languages_set": bool(cfg.get("languages")), "selected": sel,
  3859. "recommended": {p: recommend(langs, p, hw, cfg)["pick"]["model"] for p in PURPOSES},
  3860. "local_only": local_only(), "models": rows})
  3861. def cmd_setup(a):
  3862. watch_parent(getattr(a, "parent", None)) # started by the app: ends with it (the next setup continues the files)
  3863. if a.model_id:
  3864. m = entry(a.model_id)
  3865. if not m:
  3866. raise Fail(f"unknown model {a.model_id}; see transcribe.py models", "unknown_model")
  3867. else: # legacy: setup --backend B [--model M]
  3868. m = entry_for(a.backend, a.model) or entry_for(a.backend)
  3869. if m is None:
  3870. raise Fail(f"unknown backend {a.backend}", "unknown_model")
  3871. path = install_model(m)
  3872. patch: dict = {}
  3873. if a.use:
  3874. purposes = PURPOSES if a.use == "all" else [p for p in a.use.split(",") if p in PURPOSES]
  3875. cur = selected_models()
  3876. patch["models"] = {**cur, **{p: m["id"] for p in purposes}}
  3877. patch["auto"] = False # a hand-picked model is the user's choice: not replaced on hardware changes
  3878. elif not a.model_id: # legacy behaviour: setup --backend selects it
  3879. patch["backend"] = a.backend
  3880. if a.model:
  3881. patch["model"] = a.model
  3882. if a.language:
  3883. patch["language"] = a.language
  3884. # the download is done either way: a config.json that doesn't parse only leaves the choice unsaved ("saved": false)
  3885. saved = save_config_if_readable({"asr": patch}) if patch else True
  3886. emit({"ok": True, "model": m["id"], "path": str(path), "asr": asr_config(), **({} if saved else {"saved": False})})
  3887. # --- check-cloud: Settings' Test for a cloud speech service (CloudSpeech.check in the app)
  3888. CHECK_PHRASE = "Testing, one, two, three."
  3889. # the variables each cloud backend reads, for a key / address / model typed in Settings but not saved yet
  3890. CLOUD_ENV = {"dashscope": {"key": "DASHSCOPE_API_KEY", "base_url": "DASHSCOPE_BASE_URL", "model": "DASHSCOPE_ASR_MODEL"},
  3891. "volcengine": {"app_id": "VOLC_APP_ID", "token": "VOLC_ACCESS_TOKEN"},
  3892. "openai": {"key": "BLURT_ASR_API_KEY", "base_url": "BLURT_ASR_BASE_URL", "model": "BLURT_ASR_MODEL"}}
  3893. def check_clip() -> tuple[np.ndarray, str]:
  3894. """A short spoken clip (macOS `say`, offline, ~1.5 s) and what it says. Where there is no `say`: 0.6 s of
  3895. silence and "" — still the whole request (key, address, model), only nothing to hear."""
  3896. if shutil.which("say"):
  3897. with tempfile.TemporaryDirectory(prefix="blurt-check-") as d:
  3898. out = Path(d) / "check.wav"
  3899. try:
  3900. r = subprocess.run(["say", "-o", str(out), f"--data-format=LEI16@{SR}", CHECK_PHRASE],
  3901. capture_output=True, timeout=20)
  3902. if r.returncode == 0 and out.exists():
  3903. x = read_wav(out)
  3904. if len(x) > SR // 4:
  3905. return x, CHECK_PHRASE
  3906. except (OSError, subprocess.TimeoutExpired):
  3907. pass
  3908. return np.zeros(int(0.6 * SR), np.float32), ""
  3909. def cloud_check(model_id: str, given: dict | None = None, timeout: float = 20) -> dict:
  3910. """One small request to a cloud speech service, as dictation sends it → {"ok", "ms", "text", "heard", "model"}
  3911. or {"ok": False, "code", "error"} (codes: auth, quota, network, bad_response, privacy, usage). `given`: a key /
  3912. base_url / model typed but not saved (the app passes them on stdin, never on the command line)."""
  3913. m = entry(model_id)
  3914. if not m or m.get("local", True) or m.get("backend") not in CLOUD_ENV:
  3915. return {"ok": False, "code": "usage", "error": f"{model_id} is not a cloud speech service"}
  3916. try:
  3917. if local_only():
  3918. raise Fail("local-only recognition is on (privacy.local_asr): cloud ASR is not allowed", "privacy")
  3919. names = CLOUD_ENV[m["backend"]]
  3920. for k, v in (given or {}).items():
  3921. if k in names and isinstance(v, str) and v.strip():
  3922. os.environ[names[k]] = v.strip()
  3923. a = argparse.Namespace(model=None, model_id=model_id, language=None, prompt=None)
  3924. backend = BACKENDS[m["backend"]](a)
  3925. backend.retries, backend.timeout = 1, timeout # Settings waits for one try, not four
  3926. x, said = check_clip()
  3927. t0 = time.perf_counter()
  3928. r = backend.transcribe(x)
  3929. ms = int((time.perf_counter() - t0) * 1000)
  3930. text = " ".join(seg[2] for seg in r if seg[2]).strip()
  3931. return {"ok": True, "ms": ms, "text": text, "heard": bool(text) or not said, "spoken": said,
  3932. "model": getattr(backend, "model", None)}
  3933. except Fail as e:
  3934. return {"ok": False, "code": e.code, "error": str(e)}
  3935. def cmd_check_cloud(a):
  3936. given = {}
  3937. if not sys.stdin.isatty():
  3938. try:
  3939. given = json.loads(sys.stdin.read() or "{}")
  3940. except ValueError:
  3941. given = {}
  3942. r = cloud_check(a.model_id, given if isinstance(given, dict) else {})
  3943. emit(r)
  3944. if not r.get("ok"):
  3945. sys.exit(1)
  3946. def cmd_backends(_a):
  3947. bundled_vad()
  3948. have = lambda rel: (MODELS_DIR / rel).exists() # noqa: E731
  3949. emit({
  3950. "ok": True,
  3951. "configured": global_config().get("asr"),
  3952. "sensevoice_ready": have("sensevoice/model.int8.onnx") and have("sensevoice/tokens.txt") and have("silero_vad.onnx"),
  3953. "selected": selected_models(),
  3954. "api_keys": {k: bool(os.environ.get(k)) for k in ("BLURT_ASR_API_KEY", "OPENAI_API_KEY", "GROQ_API_KEY", "DASHSCOPE_API_KEY")},
  3955. })
  3956. # --- bench: speed, memory, CPU / GPU load, and accuracy on spoken samples
  3957. CER_LANGS = {"zh", "yue", "ja", "ko", "th"}
  3958. def normalize(text: str, lang: str) -> list[str]:
  3959. t = unicodedata.normalize("NFKC", text).lower()
  3960. t = "".join(" " if unicodedata.category(c)[0] in "PSZ" else c for c in t)
  3961. if lang in CER_LANGS:
  3962. return [c for c in t if not c.isspace()]
  3963. return t.split()
  3964. def error_rate(ref: str, hyp: str, lang: str) -> float:
  3965. """CER for zh / yue / ja / ko / th, WER otherwise (punctuation and case ignored)."""
  3966. r, h = normalize(ref, lang), normalize(hyp, lang)
  3967. if not r:
  3968. return 0.0 if not h else 1.0
  3969. prev = list(range(len(h) + 1))
  3970. for i, rc in enumerate(r, 1):
  3971. cur = [i] + [0] * len(h)
  3972. for j, hc in enumerate(h, 1):
  3973. cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (rc != hc))
  3974. prev = cur
  3975. return round(prev[-1] / len(r), 4)
  3976. def say_voices() -> dict[str, list[str]]:
  3977. """locale → voice names, from `say -v ?`."""
  3978. try:
  3979. out = subprocess.run(["say", "-v", "?"], capture_output=True, text=True, timeout=20).stdout
  3980. except (OSError, subprocess.TimeoutExpired):
  3981. return {}
  3982. voices: dict[str, list[str]] = {}
  3983. for line in out.splitlines():
  3984. m = re.match(r"^(.+?)\s{2,}([a-z]{2,3}_[A-Z0-9]{2,3})\s", line)
  3985. if m:
  3986. voices.setdefault(m.group(2), []).append(m.group(1).strip())
  3987. return voices
  3988. def bench_samples(langs: list[str]) -> list[dict]:
  3989. """Spoken samples for `langs` made with macOS `say`, cached under $BLURT_HOME/bench/: [{lang, path, text}]."""
  3990. if not shutil.which("say"):
  3991. raise Fail("no AUDIO given and no `say` to make samples: pass a recording, e.g. transcribe.py bench clip.mp4",
  3992. "usage")
  3993. defs = registry()["bench"]["samples"]
  3994. voices = None
  3995. out = []
  3996. for lang in langs:
  3997. d = defs.get(lang)
  3998. if not d:
  3999. continue
  4000. path = BLURT_HOME / "bench" / f"say-{lang}.wav"
  4001. text = " ".join(d["text"])
  4002. if not path.exists():
  4003. voices = voices if voices is not None else say_voices()
  4004. have = {v for vs in voices.values() for v in vs}
  4005. voice = next((v for v in d["voices"] if v in have), None) or next(iter(voices.get(d["locale"], [])), None)
  4006. if not voice:
  4007. continue
  4008. path.parent.mkdir(parents=True, exist_ok=True)
  4009. aiff = path.with_suffix(".aiff")
  4010. subprocess.run(["say", "-v", voice, "-o", str(aiff), " [[slnc 900]] ".join(d["text"])], check=True,
  4011. capture_output=True)
  4012. subprocess.run([ffmpeg_bin(), "-nostdin", "-loglevel", "error", "-y", "-i", str(aiff), "-ar", str(SR), "-ac", "1",
  4013. str(path)], check=True)
  4014. aiff.unlink(missing_ok=True)
  4015. out.append({"lang": lang, "path": path, "text": text})
  4016. return out
  4017. class GpuSampler:
  4018. """Apple GPU utilisation (system-wide, from IOAccelerator's PerformanceStatistics) sampled while decoding."""
  4019. def __init__(self, every: float = 0.2):
  4020. self.every, self.samples = every, []
  4021. self.stop = threading.Event()
  4022. self.th = threading.Thread(target=self._run, daemon=True) if IS_MAC else None
  4023. def _run(self):
  4024. while not self.stop.wait(self.every):
  4025. try:
  4026. out = subprocess.run(["ioreg", "-r", "-d", "1", "-c", "IOAccelerator"], capture_output=True, text=True,
  4027. timeout=2).stdout
  4028. m = re.search(r'"Device Utilization %"=(\d+)', out)
  4029. if m:
  4030. self.samples.append(int(m.group(1)))
  4031. except (OSError, subprocess.TimeoutExpired):
  4032. pass
  4033. def __enter__(self):
  4034. if self.th:
  4035. self.th.start()
  4036. return self
  4037. def __exit__(self, *exc):
  4038. self.stop.set()
  4039. if self.th:
  4040. self.th.join(1)
  4041. return False
  4042. def summary(self) -> dict | None:
  4043. if not self.samples:
  4044. return None
  4045. return {"mean": round(sum(self.samples) / len(self.samples)), "max": max(self.samples)}
  4046. def run_bench(m: dict, a) -> dict:
  4047. """Measure one model on this machine (in this process): load time, decode RTF, peak memory, CPU cores busy,
  4048. GPU utilisation, and the error rate per language on the `say` samples (or just speed on a given AUDIO)."""
  4049. langs = _langs_arg(a.languages) or (list(registry()["bench"]["samples"]) if a.all_languages else user_languages())
  4050. if a.input:
  4051. samples = [{"lang": None, "path": Path(a.input), "text": None}]
  4052. else:
  4053. samples = bench_samples([x for x in langs if speaks(m, x)] or langs[:1])
  4054. if not samples:
  4055. raise Fail("no bench sample for these languages", "usage")
  4056. audio = [(s, (load_audio(s["path"]) if a.input else read_wav(s["path"]))[: int(a.seconds * SR)]) for s in samples]
  4057. ns = argparse.Namespace(backend=None, model=None, model_id=m["id"], language=None, prompt=None,
  4058. threads=a.threads, concurrency=4)
  4059. name = resolve_backend(ns, "batch")
  4060. base_mem = peak_memory_mb()
  4061. t0 = time.perf_counter()
  4062. backend = make_backend(name, ns, "bench")
  4063. load_s = time.perf_counter() - t0
  4064. # warm-up (first-call kernel compilation on MLX), not timed
  4065. transcribe_track(audio[0][1][: SR * 5], backend, name, 1, quiet=True)
  4066. errors, texts, audio_s, dec_s, cpu_s = {}, {}, 0.0, 0.0, 0.0
  4067. with GpuSampler() as gpu:
  4068. for s, x in audio:
  4069. c0, t1 = os.times(), time.perf_counter()
  4070. results, _sp, _bl = transcribe_track(x, backend, name, 1 if name in LOCAL else 4, quiet=True)
  4071. dt = time.perf_counter() - t1
  4072. c1 = os.times()
  4073. dec_s += dt
  4074. cpu_s += (c1.user - c0.user) + (c1.system - c0.system)
  4075. audio_s += len(x) / SR
  4076. hyp, _lang = note_text(results)
  4077. if s["lang"]:
  4078. texts[s["lang"]] = hyp
  4079. if s["text"] and len(x) / SR >= 0.95 * float(probe_wav_seconds(s["path"])):
  4080. errors[s["lang"]] = error_rate(s["text"], hyp, s["lang"])
  4081. out = {"ok": True, "model": m["id"], "backend": name, "threads": ns.threads, "audio_seconds": round(audio_s, 1),
  4082. "load_seconds": round(load_s, 2), "decode_seconds": round(dec_s, 2),
  4083. "rtf": round(dec_s / audio_s, 4) if audio_s else None,
  4084. "cpu_cores": round(cpu_s / dec_s, 2) if dec_s else None,
  4085. "peak_ram_mb": peak_memory_mb(), "base_ram_mb": base_mem,
  4086. "at": datetime.now().astimezone().isoformat(timespec="seconds"), "machine": probe()["fingerprint"]}
  4087. if gpu.summary():
  4088. out["gpu_util"] = gpu.summary()
  4089. if qos_cpu_seconds():
  4090. out["qos_cpu_seconds"] = qos_cpu_seconds()
  4091. if errors:
  4092. out["error"] = errors
  4093. if a.show_text:
  4094. out["text"] = texts
  4095. return out
  4096. def probe_wav_seconds(p: Path) -> float:
  4097. try:
  4098. with wave.open(str(p)) as w:
  4099. return w.getnframes() / w.getframerate()
  4100. except (wave.Error, OSError):
  4101. return 0.0
  4102. def cmd_bench(a):
  4103. if a.model_id:
  4104. m = entry(a.model_id)
  4105. if not m:
  4106. raise Fail(f"unknown model {a.model_id}; see transcribe.py models", "unknown_model")
  4107. elif a.backend:
  4108. m = entry_for(a.backend, a.model) or entry_for(a.backend)
  4109. else:
  4110. m = select("batch")
  4111. if m is None:
  4112. raise Fail("no such model", "unknown_model")
  4113. throttle("bench", a.threads)
  4114. with AsrSlot(limits()["max_concurrent"]):
  4115. out = run_bench(m, a)
  4116. if not a.no_save and not a.input:
  4117. d = load_json(BENCH_FILE, {}) or {}
  4118. d = d if isinstance(d, dict) else {}
  4119. d["schema"] = 1
  4120. d.setdefault("results", {})[m["id"]] = {k: v for k, v in out.items() if k not in ("ok", "text")}
  4121. save_json(BENCH_FILE, d)
  4122. emit(out)
  4123. def main():
  4124. p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
  4125. sub = p.add_subparsers(dest="cmd", required=True)
  4126. def model_opts(sp, backend_default=None):
  4127. sp.add_argument("--backend", choices=list(BACKENDS), default=backend_default)
  4128. sp.add_argument("--model", help="the backend's model name / repo (legacy); --model-id picks from the registry")
  4129. sp.add_argument("--model-id", dest="model_id", metavar="ID", help="a model id from asr_models.json")
  4130. sp.add_argument("--language", help="e.g. zh, en, ja; default auto-detect")
  4131. r = sub.add_parser("run")
  4132. r.add_argument("input")
  4133. r.add_argument("--out", help="output dir (default: next to input)")
  4134. model_opts(r)
  4135. r.add_argument("--prompt", help="glossary / context: product & module names, jargon (Whisper / Qwen3 / API backends)")
  4136. r.add_argument("--concurrency", type=int, default=4)
  4137. r.add_argument("--threads", type=int)
  4138. r.add_argument("--force", action="store_true", help="transcribe again even if live.py already did")
  4139. n = sub.add_parser("note")
  4140. n.add_argument("wav")
  4141. n.add_argument("--language", default="auto")
  4142. n.add_argument("--model-id", dest="model_id", metavar="ID")
  4143. n.add_argument("--no-server", action="store_true", help="always transcribe in this process")
  4144. n.add_argument("--dictation", action="store_true", help="dictation's model (record.dictation.asr_model)")
  4145. n.add_argument("--vocab", action="store_true", help="the dictation words: bias the model toward them (hints only)")
  4146. n.add_argument("--no-download", action="store_true", help="a model that isn't here is an error (model_missing), never a download")
  4147. sv = sub.add_parser("serve")
  4148. sv.add_argument("--preload", action="store_true", help="load the voice-note model now (e.g. when recording starts)")
  4149. sv.add_argument("--model-id", dest="model_id", metavar="ID", help="preload this model instead")
  4150. sv.add_argument("--purpose", choices=["note", "dictation"], help="dictation: preload dictation's model")
  4151. sv.add_argument("--no-download", action="store_true", help="the preload never downloads (a missing model stays unloaded)")
  4152. sv.add_argument("--idle", type=float, default=SERVE_IDLE, help="end after the model went unused this many seconds")
  4153. sv.add_argument("--exit-after", type=float, default=1800, help="with no model loaded: end after this many seconds without a request")
  4154. sv.add_argument("--parent", type=int, help="end when this process (the app that started the server) is gone")
  4155. sub.add_parser("probe")
  4156. rc = sub.add_parser("recommend")
  4157. rc.add_argument("--languages", help="comma-separated, e.g. zh,en,ja (default: asr.languages / the system's)")
  4158. rc.add_argument("--purpose", choices=PURPOSES, default="batch")
  4159. au = sub.add_parser("auto")
  4160. au.add_argument("--apply", action="store_true", help="write asr.models for what is installed")
  4161. s = sub.add_parser("setup")
  4162. s.add_argument("--model-id", "--id", dest="model_id", metavar="ID")
  4163. s.add_argument("--backend", choices=list(BACKENDS), default="sensevoice")
  4164. s.add_argument("--model")
  4165. s.add_argument("--language")
  4166. s.add_argument("--use", help="also select it: all, or a comma list of batch,live,note")
  4167. s.add_argument("--parent", type=int, help="end when this process (the app that started the download) is gone")
  4168. sub.add_parser("backends")
  4169. cc = sub.add_parser("check-cloud", help="Settings' Test: one small request to a cloud speech service")
  4170. cc.add_argument("--model-id", dest="model_id", metavar="ID", default="qwen3-asr-flash")
  4171. sub.add_parser("models")
  4172. sub.add_parser("dictation-models")
  4173. vp = sub.add_parser("voiceprint", help="only my voice: prepare the model, enroll / check a voice, status, clear")
  4174. vp.add_argument("vp_cmd", choices=["prepare", "enroll", "check", "status", "clear"])
  4175. vp.add_argument("wav", nargs="?", help="enroll / check: the wav")
  4176. b = sub.add_parser("bench")
  4177. b.add_argument("input", nargs="?", help="audio / video file (default: spoken samples made with `say`)")
  4178. model_opts(b)
  4179. b.add_argument("--prompt")
  4180. b.add_argument("--languages", help="comma-separated (default: your languages that the model speaks)")
  4181. b.add_argument("--all-languages", action="store_true", help="every bench language the registry has")
  4182. b.add_argument("--seconds", type=float, default=60, help="at most this much of each sample")
  4183. b.add_argument("--threads", type=int)
  4184. b.add_argument("--show-text", action="store_true")
  4185. b.add_argument("--no-save", action="store_true")
  4186. a = p.parse_args()
  4187. # `--model ID` with a registry id means --model-id (K9 writes it that way); other values stay backend model names
  4188. if a.cmd in ("setup", "bench", "run") and not a.model_id and a.model and entry(a.model):
  4189. a.model_id, a.model = a.model, None
  4190. try:
  4191. {"run": cmd_run, "note": cmd_note, "serve": cmd_serve, "probe": cmd_probe, "recommend": cmd_recommend,
  4192. "auto": cmd_auto, "setup": cmd_setup, "backends": cmd_backends, "models": cmd_models, "bench": cmd_bench,
  4193. "dictation-models": cmd_dictation_models, "check-cloud": cmd_check_cloud, "voiceprint": cmd_voiceprint}[a.cmd](a)
  4194. except Fail as e:
  4195. print(f"ERROR: {e}", file=sys.stderr)
  4196. emit({"ok": False, "error": str(e), "code": e.code})
  4197. if a.cmd == "serve":
  4198. server_exit(e.exit_code)
  4199. sys.exit(e.exit_code)
  4200. except Exception as e: # noqa: BLE001 — a server never goes through the interpreter's teardown (server_exit)
  4201. if a.cmd in ("note", "check-cloud"):
  4202. # the app reads the last JSON line: a crash still says what it was (it read a bare "error" before)
  4203. import traceback
  4204. traceback.print_exc()
  4205. emit({"ok": False, "error": f"{type(e).__name__}: {e}"[:300], "code": "error"})
  4206. sys.exit(1)
  4207. if a.cmd != "serve":
  4208. raise
  4209. import traceback
  4210. traceback.print_exc()
  4211. server_exit(1)
  4212. if a.cmd == "serve":
  4213. server_exit(0)
  4214. if __name__ == "__main__":
  4215. main()

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.