blurt
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/AGIHunt/blurt with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/AGIHunt/blurt'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/agihunt--blurt)HTML
<a href="/listings/agihunt--blurt"><img src="/api/listings/agihunt--blurt/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
You need macOS 13 or later. Dictation and the recommended speech models need Apple silicon.
**[Download Blurt for Mac](https://github.com/AGIHunt/blurt/releases/latest/download/Blurt.dmg)** (signed and notarized), open the disk image and drag Blurt into Applications. Updates arrive inside the app.
Or let your agent install it through the Blurt skill:
npx skills add AGIHunt/blurt
<sub>In Claude Code you can also run <code>/plugin marketplace add AGIHunt/blurt</code>, or paste this repository's URL to your agent and ask it to install the skill.</sub>
Then tell your agent **"start blurt"**. The first run checks your machine, picks a speech model and builds the Blurt app into `~/Applications`. It needs the Xcode Command Line Tools and [uv](https://docs.astral.sh/uv/); the agent offers to install what's missing.
**First minute.** The setup assistant asks for Microphone and Accessibility (and Input Monitoring if you use another key than `Fn`), then downloads the speech models: about 5.9 GB on a Mac with 32 GB of memory, 4.4 GB on 16–24 GB. Nothing is downloaded until you click Download. Then press `Fn` in any text field. Screen Recording is asked for the first time you snap or record.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 7 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- {
- "name": "blurt",
- "owner": { "name": "AGI Hunt", "url": "https://github.com/AGIHunt" },
- "plugins": [
- { "name": "blurt", "source": "./", "description": "Screen + voice → bugs, ideas, notes and todos (review page; Markdown/CSV and hand-off .zip; your agent files them anywhere). 口喷鸡" }
- ]
- }
README.md · README · 261 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <p align="center"><picture><source media="(prefers-color-scheme: dark)" srcset="assets/peep-readme-dark.png"><img src="assets/peep-readme.png" width="132" alt="Blurt"></picture></p>
- <h1 align="center">Blurt</h1>
- <p align="center"><b>The keyboard for the agent era.</b><br>
- The fastest way to work with your agents, by text, voice, screenshot or screen recording. Runs locally. Free and open source.<br>
- <a href="https://blurt.work/en/">Website</a> · <a href="README.zh-CN.md">中文</a> · <a href="docs/guide.md">User guide</a> · <a href="#community">WeChat group</a></p>
- <p align="center"><a href="https://blurt.work/en/#film"><img src="assets/film-poster.jpg" width="720" alt="Watch the 2-minute film"></a></p>
- <p align="center"><sub>Watch the 2-minute film, sound on · <a href="https://github.com/AGIHunt/blurt/releases/latest/download/Blurt.dmg">Download for Mac</a></sub></p>
- ---
- Blurt is a menu-bar app for the Mac, with [test builds for Windows and Linux](https://github.com/AGIHunt/blurt/releases). It works next to the agents you already use, such as Claude Code, Codex, Cursor,
- Claude and ChatGPT, and handles the part before they start: getting what you mean into the agent's input box. It has
- no agent of its own and doesn't replace yours.
- ## What it does
- **Dictation (`Fn`).** Tap `Fn` in any app, talk, and tap it again. The words are typed where your cursor is. Finish
- with a double tap instead and it's sent too (Return). It works the moment Blurt is installed, with macOS's own speech
- recognition (macOS 26) and nothing to download. Open-source local models are an optional upgrade: more accurate, they
- learn your words and need no internet; after a minute of talking the text is in about 0.4 s. A local model removes
- fillers and stutters and adds punctuation; it doesn't rewrite or summarize. The words show under the notch as you
- talk. Double-tap `⇧` while talking to switch to Structured (a list, when you listed things) and again to switch back;
- double-tap `⌘` to translate (speak Chinese, type English, or the other way round); `⌥L` reformats the last dictation. Holding `Fn`
- while you talk works too.
- **Snap (`⌥A`) and Voice Note (`⌥V`).** Drag over part of the screen and say what's wrong; the note opens right
- beside the selection. Draw on it as you talk: drag for a freehand line, `⇧` for a box, `⌥` for an ellipse, `⌘Z` to
- undo. The snap becomes a card under the notch. Drag the card into Claude, ChatGPT, Slack or a terminal running Claude Code or Codex: the picture and
- your note go in together. Or click it to copy, then `⌘V`. A voice note is the same, with your words and no picture.
- **Copy text from screen (`⌥C`).** Drag over any text you can't select: a screenshot, a video, a locked page, a
- terminal. Let go and it's on the clipboard in about 0.2 s, laid out as if you had copied it: the half line the edge
- cut is left out, wrapped list items are one line again, tables keep their tabs and code its indentation. It reads with
- macOS's own recognizer, so there's nothing to download. Optionally a second reading by a small local OCR model
- (720 MB) or your own cloud model fixes misread symbols in code and paths.
- **Record (`⌥R`).** Record your screen while you talk through a product, a page or an idea. Your agent turns
- the recording into items (issues, ideas, notes, tasks), each with a screenshot, the spot marked, steps and the code
- that is probably involved. It's transcribed while you record, so it's done about a minute after you stop; one
- 5-minute walkthrough became 52 items, each with what was said and when. You go through them with the keyboard, then
- copy an instruction for your agent (`⌘↩`), or pack them into one `.zip` for a teammate.
- **Blurt on your phone.** When you sit back from a big screen, your phone becomes the computer's trackpad, keys and
- microphone, and can show its screen. It works with a Mac, Windows or Linux computer, from an iPhone, iPad (including the
- iPhone Duo) or Android phone or tablet. The talk key is Blurt's dictation key, and the keys follow the computer you're
- driving (⌘ on a Mac, Ctrl and Win elsewhere). With several displays the picture follows the pointer, or stays on the one
- you pick. Away from home it goes through a relay you run yourself; everything on the way is end-to-end encrypted. Pair
- in Settings → Phone on any computer. The phone apps aren't in the stores yet: build them
- yourself ([iPhone and iPad](apps/ios/README.md#build-it-yourself), [Android](apps/android/README.md#build-it-yourself)).
- Snaps, voice notes and recordings go into the current project (a folder you open in Blurt). Without one, recordings
- go to Unassigned and snaps stay as cards until you pick a project. Every shortcut can be changed in Settings.
- ## Install
- You need macOS 13 or later. The local models need Apple silicon; an Intel Mac dictates with macOS 26's own speech
- recognition or a cloud key. Windows and Linux: see the [FAQ](#faq).
- **[Download Blurt for Mac](https://github.com/AGIHunt/blurt/releases/latest/download/Blurt.dmg)** (signed and
- notarized), open the disk image and drag Blurt into Applications. Updates arrive inside the app. There is nothing
- else to install: the app fetches the Python runtime its tools use (about 150 MB) by itself the first time.
- Or give this to your agent (Claude Code, Codex…), or paste it into a terminal:
- ```bash
- curl -fsSL https://blurt.work/install.sh | sh
- ```
- It downloads the latest release, checks that it is signed and notarized, puts Blurt in Applications and opens it.
- **Use it from your agent.** In Blurt, *Settings → Agents → Connect…* adds the Blurt skill and MCP server to Claude
- Code, Codex and the other agents it finds (Gemini CLI, Kimi Code, OpenCode, Grok, CodeBuddy Code, WorkBuddy,
- Cursor). Start a new session of the agent, then tell it **"start blurt"** (`/blurt` in Claude Code, `$blurt` in
- Codex), or ask it about your items. Your agent can also do the connecting: [llms.txt](https://blurt.work/llms.txt) has
- the steps.
- <sub>Without the app, from source: <code>npx skills add AGIHunt/blurt -g</code>, or in Claude Code <code>/plugin
- marketplace add AGIHunt/blurt</code> and then <code>/plugin install blurt@blurt</code>. Then "start blurt". The first run
- builds Blurt into <code>~/Applications</code>; it needs the Xcode Command Line Tools and
- <a href="https://docs.astral.sh/uv/">uv</a>, and the agent offers to install what's missing.</sub>
- **First minute.** The setup assistant asks for Microphone and Accessibility (and Input Monitoring if you use another
- key than `Fn`). Then press `Fn` in any text field: on macOS 26 dictation already works with macOS's own recognizer,
- while your Mac's own local model downloads in the background, once (1.6–5.9 GB depending on the Mac). On macOS 13–15
- you download a model first or add a cloud key. One optional upgrade: an AI-service key (fixes misheard names and
- punctuation, translates with the cloud model, and gives copied text a cloud second reading). Screen Recording is asked
- for the first time you snap or record.
- ## How it works with your agents
- Blurt gets your words, pictures and recordings into an agent. The agent does the work.
- | | What it does |
- |---|---|
- | **Dictation and cards** | Text and pictures go into any chat box or terminal, so they work with every agent. |
- | **Skill** | The `blurt` skill lets your agent record, transcribe, write items, open the review page and export. |
- | **MCP server** | `blurt` gives agents `search_items`, `get_item` (with screenshots), `get_transcript`, `update_item`, `mark_fixed` and `bundle_items`. *Settings → Agents → Connect…* installs the skill and the MCP server into an agent, after showing what it changes. |
- | **Copy Instruction for Agent** (`⌘↩`) | One line on the clipboard to paste into the agent you use (Claude Code, Codex, WorkBuddy, Cursor…), pointing to a hand-off note with every detail and screenshot. The items show as *Handed off*. Blurt doesn't start an agent for this. |
- | **Hand-off bundle** | One `.zip` with a summary, every item and its screenshots, for a person rather than an agent. |
- Agents report back through the MCP server: an item moves to *In progress*, then *To verify*. "Fixed" from an agent
- always means *to verify* until you mark it done.
- Any agent with the Blurt skill can organize recordings into items, GUI apps like WorkBuddy, Cursor and CodeBuddy
- included: say "process my Blurt recordings" in its chat. The app can also do it in the background as soon as you stop,
- with the first of these that is installed and signed in: Claude Code, Codex, Kimi Code, OpenCode, Gemini CLI, Grok.
- You can change the order in *Settings → Agents*.
- **No connectors, by design.** Your agent already has tools for Slack, Linear, GitHub, Notion, Jira and Feishu/Lark.
- Blurt writes files (Markdown, CSV and the hand-off bundle) and leaves delivery to the agent. To send items somewhere,
- ask it, for example: "Bundle the checkout issues and post them to #web in Slack."
- ## How Blurt compares
- Checked on **2026-10-25**. Prices and policies change.
- Blurt's numbers come from its own tests on one machine, an Apple M5 Max with 128 GB, with speech generated by macOS
- `say`. Synthetic speech is cleaner than a real microphone, so error rates are optimistic; smaller Macs, real
- microphones and accents were not measured. Competitor numbers come from other people and methods, some from a
- competing vendor, and are not directly comparable.
- | | **Blurt** | Typeless | Wispr Flow | Superwhisper | Aqua Voice |
- |---|---|---|---|---|---|
- | Price | Free, MIT licence, no word limit | Free with a weekly limit; Pro $12/month yearly, $30 monthly [1] | Free 2,000 words/week on desktop; Pro $15/month, $12/month yearly [2] | Free with local models; Pro $8.49/month [3] | Free 1,000 words; Pro $12/month, $10/month yearly [4] |
- | Where speech is processed | On your Mac | Cloud [5] | Cloud [6] | Local or cloud, your choice [7] | Cloud [8] |
- | End of speech → text | 0.29–0.54 s after 3–60 s of speech | about 3.0 s [9] | about 1.5 s [9] | not published | "about 450ms" (vendor) [8] |
- | English accuracy | WER 1.9 % (speech model alone) | WER about 7.9 % [9] | WER about 3.9 % [9] | depends on the model; not published | OpenASR average WER 6.24 (vendor) [10] |
- | Chinese accuracy | CER 1.3 % (speech model alone) | not published | not published | not published | not published |
- | Platforms | macOS; Windows and Linux test builds | macOS, Windows, Linux, iOS, Android | Mac, Windows, iOS, Android | Mac, Windows, iOS, Android | Mac, Windows, iOS, Android |
- | Offline | Dictation needs no network once the models are downloaded (not yet tested with the network off) | No | No | Yes, with local models | No |
- | Snap and voice-note cards for agents | Yes | No | No | No | No |
- | Screen recording → items with screenshots and code pointers | Yes | No | No | No | No |
- | Hand-off to coding agents, status back through MCP | Yes | No | No | No | Lists "MCP/CLI" on its free plan; not examined |
- <sub>Blurt sources: latency `docs/design/dictation.md` (median of 3 takes, warm models); accuracy
- `docs/design/asr-benchmark.md` (Qwen3-ASR 1.7B, 7 sentences × 2 voices per language). Full detail and the numbers we
- could not verify: [docs/positioning/facts.md](docs/positioning/facts.md).
- [1] [typeless.com/pricing](https://www.typeless.com/pricing) ·
- [2] [wisprflow.ai/pricing](https://wisprflow.ai/pricing) ·
- [3] [superwhisper.com](https://superwhisper.com/) ·
- [4] [aquavoice.com/pricing](https://aquavoice.com/pricing) ·
- [5] [Typeless data controls](https://typeless.com/data-controls), updated 2026-10-13 ·
- [6] [Wispr Flow data controls](https://wisprflow.ai/data-controls), updated 2026-08-18 ·
- [7] [Superwhisper docs](https://superwhisper.com/docs/llms-full.txt) ·
- [8] [Aqua blog, 2026-08-17](https://aquavoice.com/blog/wispr-flow-alternatives) ·
- [9] [VoiceDash](https://voicedash.ai/wispr-flow-vs-typeless/), a competing vendor; the same article also gives about 20 % for Typeless ·
- [10] [Aqua blog, 2025-10-08](https://aquavoice.com/blog/avalon-openasr-leaderboard).</sub>
- ## What users complain about in other dictation apps, and what Blurt does
- From low-rated App Store reviews of Typeless and Wispr Flow (US store, read on 2026-10-24) and posts on r/macapps,
- V2EX and Product Hunt. Counts and quotes are in [facts.md §3](docs/positioning/facts.md#3-what-users-complain-about-and-what-blurt-does).
- | Complaint | What Blurt does |
- |---|---|
- | It rewrites what I said | The default style only removes fillers and adds punctuation. If polish drops or adds words, the raw transcript is typed instead. Lists happen only when you double-tap `⇧`. |
- | Words get lost or cut off | The words show while you talk. Every dictation stays in Dictation History for 30 days. The microphone opens on the key press, so the first words aren't cut. |
- | Word limits and subscriptions | Free, no word limit, no account. |
- | Names and mixed languages come out wrong | Words, a list of names and terms the speech model listens for, learned from your fixes (as hints, never as replacements), plus the text around the cursor. |
- | The microphone stays on; privacy | Speech is processed on your Mac. The microphone is on only while you dictate. No screenshots are taken for dictation. |
- | Slow | About 0.3–0.5 s from releasing the key to the text on the test Mac. |
- | The hotkey clashes with Fn | Blurt keeps `Fn` for dictation and tells you when macOS or another dictation app also reacts to it. Right Option or either Option can be used instead. |
- | Long dictations collapse into "[Pasted N lines]" in Claude Code | Terminals and code editors get the text on one line. |
- Not built yet: keeping a dictation's audio until the paste is confirmed, and a setting that limits dictation to your
- languages.
- ## Privacy
- - Speech recognition and polish run on your Mac. Dictation History keeps each dictation's text, its audio and the text
- around the cursor for 30 days (you can change how long, or stop keeping the audio).
- - When you start talking, Blurt reads about 500 characters around the cursor so names come out right. It never reads
- password fields. You can turn this off.
- - Snaps, voice notes, recordings and items stay in your folders: `~/Blurt`, or `.blurt/` inside a project, which Blurt
- keeps out of git.
- - When your agent organizes a recording, the transcript text and a few screenshots go to that agent's provider, like
- anything else you give it.
- - No account. Network use is limited to downloads of models and the Python runtime, update checks, a cloud speech key if you add one (Groq, OpenAI
- or DashScope, set in Settings → Models), cloud polish if you turn it on (the dictated text, never the recording, goes
- to DeepSeek or the API you set up), translation with the cloud model if you turn it on (the text), the cloud second
- reading in Screen Text if you choose it (the picture of the area you dragged over), what your agent sends to its provider, your phone's connection away from home (through your own relay, end-to-end
- encrypted), and Send Feedback: when you click it on the
- Dictation page, the dictations you check (their audio, every step's text, the text around the cursor if you leave it
- on, your Mac's model and settings) go to Blurt's private storage on Cloudflare, readable only by the Blurt developers
- and deleted within 14 days.
- *Settings → Privacy → Speech recognition: On Your Mac Only* keeps speech on your Mac even when a key is set.
- ## FAQ
- **Does Blurt replace my agent?** No. It feeds the agents you already use: Claude Code, Codex, Cursor, Claude, ChatGPT
- or any chat box.
- **What does it cost?** Nothing. Turning a recording into items uses your own agent's tokens. The video is never sent
- to the model: speech is transcribed locally, the agent reads the text and looks at a few frames. Three real sessions
- (Claude Code, Opus 5.5, a production web app):
- | Recording | Speech | Items | Transcription (local) | New input / output tokens | At API prices |
- |---|---|---|---|---|---|
- | 7.0 min | 2.8 min | 9 | 4 s | ~142k / ~19k | ~$2 |
- | 14.3 min | 12.3 min | 23 | 12 s | ~138k / ~27k | ~$2.6 |
- | 20.0 min | 17.7 min | 34 | 15 s | ~122k / ~27k | ~$2.1 |
- <sub>Cache reads of the ongoing conversation (about 3–5M tokens at 1/20 of the input price) are included above.</sub>
- **How long until I can review a recording?** About a minute after you click Finish; screenshots fill in while you
- review. A real 5-minute, 12-item session was ready in 51 s.
- **Which models does it use, and how big are they?**
- | Use | Model | Download |
- |---|---|---|
- | Speech on Macs with 16 GB and up | Qwen3-ASR 1.7B, 8-bit, MLX | 2.5 GB |
- | Speech on 8 GB Macs and base M1 / M2, for Chinese, English and Japanese speakers | Fun-ASR-Nano | 1.6 GB |
- | Speech there for other languages | Qwen3-ASR 0.6B, 8-bit, MLX | 1.0 GB |
- | Polish, 16 GB and up (8 GB Macs don't polish locally) | Gemma 4 E2B, 4-bit | 3.4 GB |
- | Voice notes on CPU-only machines; Japanese, Korean, Cantonese | SenseVoice Small | 228 MB |
- | Recordings in other languages | Whisper large-v3-turbo, MLX | 1.5 GB |
- In use, the 1.7B and Gemma take about 3.1 and 2.3 GB of memory. Idle models unload after 10 minutes, and sooner when
- macOS runs short of memory. On macOS 26, macOS's own recognizer dictates until your Mac's model is downloaded.
- **Which languages?** Dictation was tested in Chinese, English and mixed Chinese–English. Recordings: SenseVoice covers
- Chinese, English, Japanese, Korean and Cantonese; Whisper covers 99 languages. Items come back in the language you
- spoke. The app is in English and Simplified Chinese.
- **Does it run on Windows and Linux?** Yes, as test builds on [GitHub Releases](https://github.com/AGIHunt/blurt/releases): Windows 10 (1903 or later)
- and 11, and Linux (Ubuntu 24.04 or later, Debian 13 or later; tested on Ubuntu with GNOME on Wayland). They dictate
- (Right Alt on Windows; on Linux a shortcut you pick the first time), take voice notes and record, with speech
- recognized locally on the CPU (about 1 GB to download). Windows also has snaps and Screen Text; on Linux
- they aren't done yet. Polish needs a cloud key, the notch is a small capsule, and you update by downloading the next
- build. The Windows installer isn't signed yet: SmartScreen asks first (More info → Run anyway).
- Without the app, the skill's recording scripts (recording, local transcription, the review page, exports) also run
- there, Linux with an X11 desktop: on Windows `winget install astral-sh.uv` and
- `winget install Gyan.FFmpeg`; on Ubuntu/Debian `sudo apt install ffmpeg libportaudio2 python3-tk` plus
- [uv](https://docs.astral.sh/uv/). `doctor.py` tells you what's missing.
- **What permissions does it need?** Microphone; Accessibility, to paste and to hear `Fn`; Input Monitoring, for other
- keys; Screen Recording, for snaps and recordings.
- **Can teammates without the code use it?** Yes. They record with the app and send you the video; your agent processes
- it with the code at hand. Items keep the name of the person who recorded them.
- **Can I change the shortcuts?** Yes, in *Settings → Shortcuts*. Blurt warns when a shortcut is taken by another
- action, another app or macOS. The same setting works in `~/.blurt/config.json`, also on Windows:
- ```json
- {"record": {"hotkeys": {"record": "ctrl+opt+R", "snap": "opt+shift+A", "pause": "off"}, "draw_key": "control"}}
- ```
- **Where is the full manual?** The [user guide](docs/guide.md) covers every feature, and
- [SKILL.md](skills/blurt/SKILL.md) is what your agent reads.
- ## Community
- Questions, feedback and new releases go to the Blurt users' group on WeChat. To join, reply 口喷 to the WeChat official
- account **AGI Hunt**.
- Bugs and feature requests are also welcome as [GitHub issues](https://github.com/AGIHunt/blurt/issues).
- <p align="center"><sub>MIT · made by <a href="https://github.com/AGIHunt">AGI Hunt</a></sub></p>
site/install.sh · Bundled script · 71 lines · 1 finding marked
- #!/bin/sh
- # Install Blurt for Mac: download the latest release, put Blurt.app in Applications and open it.
- #
- # curl -fsSL https://blurt.work/install.sh | sh
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 4 on GitHub
- #
- # Made to be run by you or by your agent (Claude Code, Codex…). It replaces an older Blurt.app where it finds one
- # (/Applications, else ~/Applications) and checks that the download is signed and notarized before copying it. The
- # permissions (microphone, Accessibility) can only be turned on by you: the setup window that opens walks you through
- # them. Served from the website (site/install.sh in github.com/AGIHunt/blurt).
- #
- # BLURT_INSTALL_DIR=/some/folder sh install.sh installs there instead, and quits or opens nothing (tests)
- set -eu
- URL=https://github.com/AGIHunt/blurt/releases/latest/download/Blurt.dmg
- if [ "$(uname -s)" != Darwin ]; then
- echo "This installer is for macOS. Windows and Linux test builds: https://github.com/AGIHunt/blurt/releases" >&2
- exit 1
- fi
- major=$(sw_vers -productVersion | cut -d. -f1)
- if [ "$major" -lt 13 ]; then
- echo "Blurt needs macOS 13 or later (this Mac has $(sw_vers -productVersion))." >&2
- exit 1
- fi
- TMP=$(mktemp -d "${TMPDIR:-/tmp}/blurt-install.XXXXXX")
- MNT=$TMP/mnt
- cleanup() {
- hdiutil detach -quiet "$MNT" 2>/dev/null || true
- rm -rf "$TMP"
- }
- trap cleanup EXIT INT TERM
- echo "Downloading Blurt (about 20 MB)..."
- curl -fsSL -o "$TMP/Blurt.dmg" "$URL"
- mkdir "$MNT"
- hdiutil attach -quiet -nobrowse -readonly -mountpoint "$MNT" "$TMP/Blurt.dmg"
- NEW=$MNT/Blurt.app
- if ! spctl --assess --type execute "$NEW" 2>/dev/null; then
- echo "The download isn't a signed and notarized Blurt.app; nothing was installed." >&2
- exit 1
- fi
- # where it goes: over an existing copy, else /Applications (~/Applications when that isn't writable);
- # BLURT_INSTALL_DIR puts it elsewhere and doesn't quit or open anything (tests)
- if [ -n "${BLURT_INSTALL_DIR:-}" ]; then DEST=$BLURT_INSTALL_DIR
- elif [ -d /Applications/Blurt.app ]; then DEST=/Applications
- elif [ -d "$HOME/Applications/Blurt.app" ]; then DEST=$HOME/Applications
- elif [ -w /Applications ]; then DEST=/Applications
- else DEST=$HOME/Applications
- fi
- mkdir -p "$DEST"
- if [ -z "${BLURT_INSTALL_DIR:-}" ] && pgrep -xq blurt-recorder; then
- osascript -e 'tell application id "dev.blurt.recorder" to quit' >/dev/null 2>&1 || true
- i=0
- while pgrep -xq blurt-recorder && [ $i -lt 20 ]; do sleep 0.5; i=$((i + 1)); done
- fi
- rm -rf "$DEST/Blurt.app"
- ditto "$NEW" "$DEST/Blurt.app"
- version=$(/usr/libexec/PlistBuddy -c "Print :CFBundleShortVersionString" "$DEST/Blurt.app/Contents/Info.plist" 2>/dev/null || echo "")
- if [ -n "${BLURT_INSTALL_DIR:-}" ]; then
- echo "Blurt $version is in $DEST."
- exit 0
- fi
- open "$DEST/Blurt.app"
- echo "Blurt $version is in $DEST and open."
- echo "Finish in the setup window: turn on the microphone and Accessibility. Nothing else to download."
- echo "To use Blurt from an agent (\"start blurt\"), connect it in Blurt's Settings > Agents, then start a new session"
- echo "of that agent. Agents can connect themselves: the steps are at https://blurt.work/llms.txt (Install)."
skills/blurt/scripts/transcribe.py · Bundled script · 4,751 lines
- # /// script
- # requires-python = ">=3.10"
- # dependencies = ["numpy==2.3.4; python_version >= '3.11'", "numpy==2.2.6; python_version < '3.11'", "sherpa-onnx==1.13.8", "truststore==0.10.5", "opencc-python-reimplemented==0.1.7"]
- # ///
- """Speech-to-text with timestamps, adapted to the machine it runs on (docs/design/next-round.md K9).
- Always VAD-first: silence is dropped, speech is chunked, and every chunk keeps its offset in the original
- video, so any backend (even ones without timestamps) works.
- transcribe.py run <video|audio> [--out DIR] transcribe; writes transcript.json + transcript.txt
- instant when live.py already transcribed the recording
- while it was made (<session>/live/); --force redoes it
- transcribe.py run <session dir> a shots session (snap mode): transcribe each voice note
- shots/NNN.wav into its shots.jsonl record ("said")
- transcribe.py note <wav> [--language auto] [--dictation] [--vocab] [--no-server] [--no-download]
- one voice note, fast path → {"ok","text","language","ms","model"}
- (uses a running `serve` when there is one). --dictation: the
- dictation model (record.dictation.asr_model); --vocab: bias it
- toward the words (dictation_vocab.py) where the model can be
- biased; they are hints, the text is as heard → + "asr_text","biased"
- transcribe.py serve [--preload] [--model-id ID | --purpose dictation] [--no-download] [--idle 600] [--parent PID] keep the
- voice-note (or dictation) model warm; ends after --idle s unused
- (the memory goes back), with the --parent app, or on "exit".
- A request with "interim": true (live words while talking) is
- stopped as soon as another request waits → code "interrupted"
- (a quick model, SenseVoice, decodes it whole)
- transcribe.py dictation-models the models dictation can use here: selected, recommended, ready (runs with nothing downloaded, VAD included), missing
- transcribe.py probe hardware → tier (pro / standard / light / minimal)
- transcribe.py recommend [--languages zh,en] [--purpose batch|live|note]
- transcribe.py auto [--apply] pick a model per purpose for this machine (asr.models);
- lists what still has to be downloaded
- transcribe.py models the registry (asr_models.json) + installed, measured, selected
- transcribe.py setup --model ID [--use all|batch,live,note] [--parent PID] download a model (progress on
- stderr): one at a time per model (a second one waits), a
- partial file continued (HTTP Range), a mirror folder installed
- only once every file of its manifest is in; ends with --parent
- transcribe.py bench [AUDIO] [--model ID] [--languages zh,en] real-time factor, peak memory, CPU load and (on the
- built-in `say` samples) the error rate per language
- → $BLURT_HOME/asr-bench.json
- transcribe.py backends backends and whether they are ready (legacy)
- transcribe.py voiceprint prepare|enroll WAV|check WAV|status|clear only my voice (docs/design/voiceprint.md):
- the speaker model, the user's voice profile, a piece's verdict
- transcribe.py check-cloud [--model-id qwen3-asr-flash|volcengine-seedasr|openai-compatible] Settings' Test: one
- small spoken clip (macOS `say`) to the cloud service →
- {"ok","ms","text"} or {"ok": false, "code"}; a key / base_url /
- model not saved yet comes as JSON on stdin ({"key": …};
- Volcengine: {"app_id": …, "token": …}), never as an argument
- Which model: --model ID / --backend B win; else config asr (auto: per purpose from `recommend`, only what is
- installed; asr.auto false: asr.models / the legacy asr.backend + asr.model). Resource caps (asr.limits): threads,
- one batch job at a time ($BLURT_HOME/run/asr.lock, the rest queue), low priority for batch work, an MLX memory cap.
- Chinese comes back in the user's script: for zh-Hans users (or asr.chinese_script "hans") traditional characters
- become simplified (OpenCC t2s, `to_user_script`); zh-Hant users keep what the model wrote.
- Dependencies are pinned (here and in every `--with`, PACKAGE_PINS): a new release is taken on purpose, per release.
- Backends (models: asr_models.json; measured: docs/design/asr-benchmark.md)
- sensevoice local CPU, sherpa-onnx + SenseVoice-Small int8 (~230 MB). zh/en/ja/ko/yue.
- qwen3-asr-onnx local CPU (any OS), sherpa-onnx + Qwen3-ASR 0.6B int8, 30 languages. Hotwords per stream.
- fun-asr-onnx local CPU (any OS), sherpa-onnx + Fun-ASR-Nano int8, zh / en / ja. Hotwords; pieces ≤ 20 s.
- qwen3-asr-mlx local, Apple Silicon (MLX), Qwen3-ASR 0.6B 8-bit, 30 languages, `context` biasing per request
- (dictation's default: docs/design/dictation.md §Recognition model). needs mlx-qwen3-asr
- fun-asr-mlx local, Apple Silicon (MLX), Fun-ASR-Nano 0.8B, zh / en / ja, the Words as hotwords
- (dictation on 8 GB and slower Macs: docs/design/dictation.md §Defaults by Mac). needs mlx-audio
- mlx-whisper local, Apple Silicon (MLX), Whisper large-v3-turbo (fp16 / 4-bit). needs mlx-whisper
- faster-whisper local, NVIDIA CUDA (or CPU), Whisper large-v3-turbo (CTranslate2). needs faster-whisper
- openai any OpenAI-compatible /audio/transcriptions API (OpenAI, Groq, SiliconFlow, ...)
- env: BLURT_ASR_API_KEY (or OPENAI_API_KEY / GROQ_API_KEY), BLURT_ASR_BASE_URL, BLURT_ASR_MODEL
- dashscope Alibaba Qwen3-ASR (qwen3-asr-flash) via DashScope OpenAI-compatible chat API. env: DASHSCOPE_API_KEY
- volcengine Volcengine 豆包 Seed-ASR 2.0 (bigmodel_nostream), streamed over a websocket while a dictation is said
- (serve "stream"). env: VOLC_APP_ID, VOLC_ACCESS_TOKEN
- Backends that need a package re-run themselves with `uv run --with <package>` when it is missing.
- """
- from __future__ import annotations
- import argparse
- import contextlib
- import base64
- import gzip
- import hashlib
- import io
- import json
- import os
- import platform
- import re
- import select as _select # `select` is this module's model pick
- import shutil
- import socket
- import struct
- import subprocess
- import tempfile
- import sys
- import threading
- import time
- import unicodedata
- import http.client
- import urllib.error
- import urllib.parse
- import urllib.request
- import uuid
- import wave
- from concurrent.futures import ThreadPoolExecutor
- from datetime import datetime
- from pathlib import Path
- import numpy as np
- sys.path.insert(0, str(Path(__file__).parent))
- import dictation_vocab # noqa: E402
- from _common import (BLURT_HOME, CPU_SERVE_IDLE, IS_MAC, IS_WIN, MODELS_DIR, SERVE_IDLE, die, ffmpeg_bin, fmt_ts, global_config, # noqa: E402
- landed_in, load_json, probe_duration, progress_line, save_config_if_readable, save_json,
- server_exit, serve_exit_reason, serve_said, use_system_certs)
- from _common import (begin_download, download_lock, download_path, end_download, fetch, folder_complete, # noqa: E402
- safetensors_complete, watch_parent, write_text_atomic)
- use_system_certs()
- SR = 16000
- HERE = Path(__file__).parent
- VAD_URLS = [
- "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/silero_vad.onnx",
- "https://huggingface.co/csukuangfj/vad/resolve/main/silero_vad.onnx",
- "https://hf-mirror.com/csukuangfj/vad/resolve/main/silero_vad.onnx",
- ]
- PURPOSES = ("batch", "live", "note")
- # The VAD ships with the skill (models/silero_vad.onnx, 0.6 MB; Silero VAD, MIT, as k2-fsa/sherpa-onnx publishes it —
- # the same bytes as VAD_URLS[0]). v1.0.6: on a fresh Mac it was all a local model lacked, and the island and the setup
- # said "no speech model · 1 MB" for it. bundled_vad() puts it where the models live before anything looks for it.
- BUNDLED_VAD = HERE.parent / "models" / "silero_vad.onnx"
- def bundled_vad() -> None:
- dest = MODELS_DIR / "silero_vad.onnx"
- if dest.exists() or not BUNDLED_VAD.is_file():
- return
- try:
- MODELS_DIR.mkdir(parents=True, exist_ok=True)
- tmp = dest.with_suffix(".part")
- shutil.copyfile(BUNDLED_VAD, tmp)
- tmp.replace(dest)
- except OSError:
- pass # the download still works
- BENCH_FILE = BLURT_HOME / "asr-bench.json"
- RUN_DIR = BLURT_HOME / "run"
- class Fail(Exception):
- """A CLI failure with a machine code (next-round.md: {"ok": false, "error", "code"}, exit 1 / 3 / 4)."""
- def __init__(self, message: str, code: str = "error", exit_code: int = 1):
- super().__init__(message)
- self.code, self.exit_code = code, exit_code
- class Interrupted(Exception):
- """An interim decode (dictation's live text) stopped because another request is waiting (docs/design/dictation.md
- §Live text): the final pieces never wait for one."""
- def emit(obj: dict) -> None:
- print(json.dumps(obj, ensure_ascii=False), flush=True)
- # ------------------------------------------------------------------ the model registry (asr_models.json)
- _REGISTRY: dict | None = None
- def registry() -> dict:
- global _REGISTRY
- if _REGISTRY is None:
- _REGISTRY = json.loads((HERE / "asr_models.json").read_text(encoding="utf-8"))
- return _REGISTRY
- def entries(local_only: bool = False) -> list[dict]:
- return [m for m in registry()["models"] if not local_only or m.get("local", True)]
- def entry(model_id: str | None) -> dict | None:
- return next((m for m in registry()["models"] if m["id"] == model_id), None)
- def entry_for(backend: str, model: str | None = None) -> dict | None:
- """The registry entry a legacy (backend, model) pair means; the backend's first entry when model is None."""
- cands = [m for m in registry()["models"] if m["backend"] == backend]
- if model:
- for m in cands:
- if model in (m["model"], m["id"]) or (m["model"] and m["model"].split("/")[-1] == model):
- return m
- if backend == "faster-whisper" and model == "large-v3-turbo":
- return entry("whisper-large-v3-turbo-ct2")
- return None
- return cands[0] if cands else None
- def speaks(m: dict, lang: str) -> bool:
- langs = m.get("languages")
- return langs == "multi" or lang in (langs or [])
- def platform_key() -> str:
- arch = platform.machine().lower()
- arch = {"amd64": "x86_64", "x64": "x86_64", "aarch64": "arm64"}.get(arch, arch)
- osname = "darwin" if IS_MAC else "windows" if IS_WIN else "linux"
- return f"{osname}-{arch}"
- def runs_here(m: dict, plat: str | None = None) -> bool:
- plats = m.get("platforms") or ["*"]
- return "*" in plats or (plat or platform_key()) in plats
- # ------------------------------------------------------------------ hardware probe and tiers
- def _sysctl(name: str) -> str | None:
- try:
- return subprocess.run(["sysctl", "-n", name], capture_output=True, text=True, timeout=5).stdout.strip() or None
- except (OSError, subprocess.TimeoutExpired):
- return None
- def _ram_bytes() -> int:
- try:
- if IS_MAC:
- return int(_sysctl("hw.memsize") or 0)
- if IS_WIN:
- import ctypes
- class MEMSTAT(ctypes.Structure):
- _fields_ = [("dwLength", ctypes.c_ulong), ("dwMemoryLoad", ctypes.c_ulong),
- ("ullTotalPhys", ctypes.c_ulonglong), ("ullAvailPhys", ctypes.c_ulonglong),
- ("ullTotalPageFile", ctypes.c_ulonglong), ("ullAvailPageFile", ctypes.c_ulonglong),
- ("ullTotalVirtual", ctypes.c_ulonglong), ("ullAvailVirtual", ctypes.c_ulonglong),
- ("sullAvailExtendedVirtual", ctypes.c_ulonglong)]
- st = MEMSTAT()
- st.dwLength = ctypes.sizeof(MEMSTAT)
- ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(st))
- return int(st.ullTotalPhys)
- return os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES")
- except (OSError, ValueError, AttributeError):
- return 0
- def _cores() -> tuple[int, int, list[dict]]:
- """(performance cores, efficiency cores, per-level detail). Apple Silicon reports its core clusters as perf
- levels (M1–M4: Performance + Efficiency; M5 Pro / Max: Super + Performance); level 0 is the fastest."""
- total = os.cpu_count() or 1
- if IS_MAC:
- levels = []
- for i in range(int(_sysctl("hw.nperflevels") or 0)):
- n = int(_sysctl(f"hw.perflevel{i}.physicalcpu") or 0)
- levels.append({"name": _sysctl(f"hw.perflevel{i}.name") or f"level{i}", "cores": n})
- if levels:
- return levels[0]["cores"], sum(lv["cores"] for lv in levels[1:]), levels
- phys = int(_sysctl("hw.physicalcpu") or total)
- return phys, 0, []
- try: # physical cores where the OS tells us (Linux: unique core ids)
- if not IS_WIN and Path("/proc/cpuinfo").exists():
- ids = set(re.findall(r"physical id\s*:\s*(\d+)[\s\S]*?core id\s*:\s*(\d+)", Path("/proc/cpuinfo").read_text()))
- if ids:
- return len(ids), 0, []
- except OSError:
- pass
- return max(1, total // 2) if total > 2 else total, 0, []
- def _apple_gpu() -> dict | None:
- try:
- out = subprocess.run(["ioreg", "-r", "-c", "AGXAccelerator", "-d", "1"], capture_output=True, text=True,
- timeout=5).stdout
- except (OSError, subprocess.TimeoutExpired):
- return None
- cores = re.search(r'"gpu-core-count"\s*=\s*(\d+)', out)
- name = re.search(r'"model"\s*=\s*"([^"]+)"', out)
- return {"kind": "apple", "name": name.group(1) if name else "Apple GPU", **({"cores": int(cores.group(1))} if cores else {})}
- def _nvidia_gpu() -> dict | None:
- if not shutil.which("nvidia-smi"):
- return None
- try:
- out = subprocess.run(["nvidia-smi", "--query-gpu=name,memory.total", "--format=csv,noheader,nounits"],
- capture_output=True, text=True, timeout=10).stdout.strip().splitlines()
- except (OSError, subprocess.TimeoutExpired):
- return None
- if not out:
- return None
- name, _, mem = out[0].rpartition(",")
- try:
- vram = round(float(mem) / 1024, 1)
- except ValueError:
- vram = None
- return {"kind": "nvidia", "name": name.strip(), **({"vram_gb": vram} if vram else {})}
- def on_battery() -> bool:
- try:
- if IS_MAC:
- out = subprocess.run(["pmset", "-g", "batt"], capture_output=True, text=True, timeout=5).stdout
- return "Battery Power" in out
- if IS_WIN:
- import ctypes
- class SPS(ctypes.Structure):
- _fields_ = [("ACLineStatus", ctypes.c_byte), ("BatteryFlag", ctypes.c_byte),
- ("BatteryLifePercent", ctypes.c_byte), ("SystemStatusFlag", ctypes.c_byte),
- ("BatteryLifeTime", ctypes.c_ulong), ("BatteryFullLifeTime", ctypes.c_ulong)]
- s = SPS()
- return bool(ctypes.windll.kernel32.GetSystemPowerStatus(ctypes.byref(s))) and s.ACLineStatus == 0
- mains = [p for p in Path("/sys/class/power_supply").glob("*") if (p / "online").exists()]
- return bool(mains) and all((p / "online").read_text().strip() == "0" for p in mains)
- except (OSError, ValueError, AttributeError, subprocess.TimeoutExpired):
- return False
- def tier_for(hw: dict) -> str:
- """K9 tiers. Apple Silicon by unified memory; NVIDIA by VRAM; other machines by cores and RAM."""
- ram, cores, gpu = hw["ram_gb"], hw["perf_cores"] + hw["eff_cores"], hw["gpu"]
- if ram < 7.5 or cores <= 2:
- return "minimal"
- if hw["arch"] == "arm64" and hw["os"] == "darwin":
- return "pro" if ram >= 30 else "standard" if ram >= 15 else "light"
- vram = gpu.get("vram_gb") or 0
- if gpu.get("kind") == "nvidia" and vram >= 8:
- return "pro"
- if gpu.get("kind") == "nvidia" and vram >= 4:
- return "standard"
- if cores >= 8 and ram >= 15:
- return "standard"
- return "light"
- _HW: dict | None = None
- def probe(refresh: bool = False) -> dict:
- global _HW
- if _HW is not None and not refresh:
- return _HW
- perf, eff, levels = _cores()
- osname = "darwin" if IS_MAC else "windows" if IS_WIN else "linux"
- arch = platform_key().split("-", 1)[1]
- gpu = (_apple_gpu() if IS_MAC and arch == "arm64" else None) or _nvidia_gpu() or {"kind": "none", "name": None}
- chip = (_sysctl("machdep.cpu.brand_string") if IS_MAC else None) or platform.processor() or platform.machine()
- hw = {"os": osname, "arch": arch, "chip": chip, "perf_cores": perf, "eff_cores": eff,
- "ram_gb": round(_ram_bytes() / 2**30, 1), "gpu": gpu, "ane": IS_MAC and arch == "arm64",
- "on_battery": on_battery()}
- if levels:
- hw["core_levels"] = levels
- hw["tier"] = tier_for(hw)
- hw["fingerprint"] = f"{chip}|{hw['ram_gb']:.0f}GB|{gpu.get('name')}|{gpu.get('cores') or gpu.get('vram_gb') or ''}"
- _HW = hw
- return hw
- # ------------------------------------------------------------------ languages
- def system_languages() -> list[str]:
- """The OS's preferred languages as ISO 639-1 codes (zh-Hant-HK → zh, yue stays yue), first = most preferred."""
- out: list[str] = []
- for tag in system_language_tags():
- code = re.split(r"[-_.@]", tag)[0].lower()
- if code in ("c", "posix", ""):
- continue
- if tag.lower().startswith(("zh-hk", "zh_hk", "yue")):
- code = "yue" if tag.lower().startswith("yue") else "zh"
- if code not in out:
- out.append(code)
- return out
- def system_language_tags() -> list[str]:
- """The OS's preferred languages as it names them (zh-Hans-CN, en-US …), first = most preferred."""
- raw: list[str] = []
- try:
- if IS_MAC:
- out = subprocess.run(["defaults", "read", "-g", "AppleLanguages"], capture_output=True, text=True, timeout=5).stdout
- raw = re.findall(r'"?([A-Za-z]{2,3}(?:-[A-Za-z0-9]+)*)"?\s*,?\s*\n', out)
- elif IS_WIN:
- import ctypes
- n, size = ctypes.c_ulong(), ctypes.c_ulong()
- if ctypes.windll.kernel32.GetUserPreferredUILanguages(8, ctypes.byref(n), None, ctypes.byref(size)):
- buf = ctypes.create_unicode_buffer(size.value)
- ctypes.windll.kernel32.GetUserPreferredUILanguages(8, ctypes.byref(n), buf, ctypes.byref(size))
- raw = [s for s in buf[:size.value].split("\0") if s]
- else:
- raw = [p for p in (os.environ.get("LANGUAGE") or os.environ.get("LC_ALL") or os.environ.get("LANG") or "").split(":") if p]
- except (OSError, subprocess.TimeoutExpired, AttributeError):
- pass
- return raw
- # The Chinese script the user reads (desi065): Whisper large-v3-turbo writes Mandarin in traditional characters, or
- # a mix ("登录頁面周五之前要上線"), for a zh-Hans user. Whisper has no setting for the script (its language token is
- # just "zh"), and the usual trick — an initial prompt in simplified Chinese — was measured and dropped: it fixed the
- # characters, but on an English note it once sent the model into a loop ("段段段…") and it glued English names to
- # the Chinese around them. So text heard as Chinese is converted character by character to the user's script
- # (OpenCC's t2s tables: a mechanical script mapping that reads no meaning). zh-Hant users (zh-TW / zh-HK / zh-MO, or
- # asr.chinese_script "hant") keep what the model wrote: nothing is converted for them. Qwen3-ASR (dictation) and
- # SenseVoice write simplified already; the conversion changes nothing there.
- _HANT_TAGS = ("zh-hant", "zh_hant", "zh-tw", "zh_tw", "zh-hk", "zh_hk", "zh-mo", "zh_mo", "yue")
- _script: dict = {}
- def chinese_script(cfg: dict | None = None) -> str | None:
- """"hans" | "hant" | None: asr.chinese_script, else the first Chinese entry of the OS's preferred languages
- (zh-Hans-CN, zh-CN, zh → hans; zh-Hant-*, zh-TW, zh-HK, zh-MO, yue → hant); None without Chinese. Read once a
- process."""
- cfg = asr_config() if cfg is None else cfg
- if cfg.get("chinese_script") in ("hans", "hant"):
- return cfg["chinese_script"]
- if "system" not in _script:
- tag = next((t.lower() for t in system_language_tags() if t.lower().startswith(("zh", "yue"))), None)
- _script["system"] = None if tag is None else "hant" if tag.startswith(_HANT_TAGS) else "hans"
- return _script["system"]
- _KANA = re.compile(r"[-ヿ]")
- def to_user_script(text: str, lang: str | None, script: str | None = None) -> str:
- """Text the speech model heard as Chinese (or with no language said), in a zh-Hans user's script: traditional
- characters → simplified, one by one. Japanese (kana, or tagged ja) and everyone else's text are left alone."""
- script = script if script is not None else chinese_script()
- if script != "hans" or not text or (lang and lang not in ("zh", "yue")) or _KANA.search(text):
- return text
- if not re.search(r"[㐀-鿿]", text):
- return text
- conv = _script.get("t2s")
- if conv is None:
- try:
- from opencc import OpenCC
- conv = OpenCC("t2s").convert
- except Exception as e: # noqa: BLE001 — the package missing must not cost the words
- print(f"script conversion unavailable: {type(e).__name__}: {e}", file=sys.stderr, flush=True)
- conv = False
- _script["t2s"] = conv
- return conv(text) if conv else text
- SWITCH_LANGUAGES = {"zh": "Chinese", "yue": "Cantonese", "ja": "Japanese", "ko": "Korean"}
- def switch_language(langs: list[str]) -> str | None:
- """The language a user mixes English names into, as Qwen3-ASR names it: the first of their languages that people
- commonly speak with English words inside it (zh → "Chinese"). None for English-only users."""
- return next((SWITCH_LANGUAGES[c] for c in langs if c in SWITCH_LANGUAGES), None)
- def user_languages(cfg: dict | None = None) -> list[str]:
- """asr.languages, else the system's preferred languages ∪ {en}."""
- cfg = asr_config() if cfg is None else cfg
- langs = cfg.get("languages")
- if isinstance(langs, str):
- langs = [x.strip() for x in langs.split(",") if x.strip()]
- if not langs:
- langs = system_languages()
- if cfg.get("language") and cfg["language"] not in langs: # the spoken language the user picked
- langs.insert(0, cfg["language"])
- if "en" not in langs:
- langs.append("en")
- return list(dict.fromkeys(langs))
- # ------------------------------------------------------------------ configuration
- def asr_config() -> dict:
- c = global_config().get("asr")
- return c if isinstance(c, dict) else {}
- def is_auto(cfg: dict) -> bool:
- """asr.auto, default true — except for a legacy explicit choice of a non-default backend (kept as the user's)."""
- if "auto" in cfg:
- return cfg["auto"] is not False
- return not (cfg.get("backend") and cfg.get("backend") != "sensevoice")
- def local_only() -> bool:
- return bool((global_config().get("privacy") or {}).get("local_asr"))
- def limits(hw: dict | None = None, cfg: dict | None = None) -> dict:
- """asr.limits with K9 defaults: threads = max(1, min(4, perf_cores // 2)); one batch job at a time;
- batch at utility / nice 10, live and voice notes at user-initiated; MLX may use a quarter of the memory."""
- hw = hw or probe()
- lim = dict((cfg if cfg is not None else asr_config()).get("limits") or {})
- prio = {"batch": "utility", "live": "user-initiated", "note": "user-initiated", "bench": "utility"}
- if isinstance(lim.get("priority"), dict):
- prio.update(lim["priority"])
- elif isinstance(lim.get("priority"), str):
- prio["batch"] = prio["bench"] = lim["priority"]
- return {"threads": max(1, int(lim.get("threads") or max(1, min(4, hw["perf_cores"] // 2)))),
- "max_concurrent": max(1, int(lim.get("max_concurrent") or 1)),
- "priority": prio,
- "max_memory_fraction": min(0.9, max(0.05, float(lim.get("max_memory_fraction") or 0.25)))}
- # ------------------------------------------------------------------ resource caps
- QOS = {"user-interactive": 0x21, "user-initiated": 0x19, "default": 0x15, "utility": 0x11, "background": 0x09}
- def apply_priority(level: str, reexec: bool = True) -> dict:
- """Lower this process's priority before any model or worker thread exists. macOS: worker threads that runtimes
- create (onnxruntime, CTranslate2) do not inherit the creating thread's QoS, so utility / background work re-runs
- itself once under `taskpolicy -c <level>` (a QoS clamp on every thread of the process) — plus nice; the calling
- thread's QoS is set either way. Windows: priority class. Linux: nice. Returns what was applied."""
- done: dict = {"level": level}
- nice = {"background": 15, "utility": 10}.get(level, 0)
- if IS_MAC and reexec and level in ("utility", "background") and os.environ.get("BLURT_QOS_CLAMP") != level \
- and Path("/usr/sbin/taskpolicy").exists() and sys.argv and sys.argv[0].endswith(".py"):
- os.environ["BLURT_QOS_CLAMP"] = level
- sys.stdout.flush()
- sys.stderr.flush()
- os.execv("/usr/sbin/taskpolicy", ["taskpolicy", "-c", level, sys.executable, os.path.abspath(sys.argv[0]),
- *sys.argv[1:]])
- if IS_MAC and os.environ.get("BLURT_QOS_CLAMP") == level:
- done["clamp"] = level
- if IS_MAC:
- try:
- import ctypes
- lib = ctypes.CDLL("/usr/lib/libSystem.B.dylib")
- if lib.pthread_set_qos_class_self_np(ctypes.c_uint(QOS.get(level, 0x15)), 0) == 0:
- done["qos"] = level
- except (OSError, AttributeError):
- pass
- if IS_WIN:
- try:
- import ctypes
- cls = {"background": 0x40, "utility": 0x4000}.get(level, 0x20) # IDLE / BELOW_NORMAL / NORMAL
- if ctypes.windll.kernel32.SetPriorityClass(ctypes.windll.kernel32.GetCurrentProcess(), cls):
- done["priority_class"] = cls
- except (OSError, AttributeError):
- pass
- elif nice:
- try:
- cur = os.getpriority(os.PRIO_PROCESS, 0)
- if cur < nice:
- os.setpriority(os.PRIO_PROCESS, 0, nice)
- done["nice"] = os.getpriority(os.PRIO_PROCESS, 0)
- except (OSError, AttributeError):
- pass
- return done
- def cap_threads(n: int) -> None:
- """BLAS / OpenMP pools size themselves at import: set the caps first."""
- for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS",
- "NUMEXPR_NUM_THREADS", "RAYON_NUM_THREADS"):
- os.environ[k] = str(n)
- # MLX's cache of freed buffers, kept for reuse. It used to be min(limit / 4, 2 GB): the warm servers then held what
- # their warm-up had freed (polish's two prompt prefills: ~1.5 GB) until the first request — 4.6 GB instead of 3.1 GB
- # for Gemma 4 E2B. 64 MB costs 2.4 ms a polish request with the same output (0 MB: +68 ms); docs/design/dictation.md
- # §Memory.
- MLX_CACHE_BYTES = 64 << 20
- def cap_mlx_memory(fraction: float) -> int | None:
- """Keep MLX (which otherwise grows its buffer cache to most of unified memory) inside fraction × RAM, and its cache
- of freed buffers small (MLX_CACHE_BYTES)."""
- try:
- import mlx.core as mx
- except ImportError:
- return None
- total = _ram_bytes()
- if not total:
- return None
- limit = int(total * fraction)
- for fn, val in (("set_memory_limit", limit), ("set_cache_limit", min(limit // 4, MLX_CACHE_BYTES))):
- f = getattr(mx, fn, None) or getattr(getattr(mx, "metal", None), fn, None)
- if f:
- try:
- f(val)
- except Exception: # noqa: BLE001 — an older MLX without the call
- pass
- return limit
- class AsrSlot:
- """At most N concurrent batch transcriptions machine-wide: byte-range locks on $BLURT_HOME/run/asr.lock
- (released by the OS when a process dies). Waiting = queued: said on stderr and, under a job ($BLURT_JOB),
- as events in jobs/<id>.events.jsonl (K4). A run that can't write $BLURT_HOME (an organizing agent in its
- sandbox) transcribes without a slot."""
- _held: set[int] = set() # slots this process holds (record locks are per process, not per thread)
- _guard = threading.Lock()
- _file = None # one descriptor per process: closing any descriptor drops all of its locks
- def __init__(self, n: int, timeout: float = 6 * 3600, poll: float = 0.25):
- self.n, self.timeout, self.poll = n, timeout, poll
- self.fh = None
- self.slot: int | None = None
- self.waited = 0.0
- def _try(self, i: int) -> bool:
- with AsrSlot._guard:
- if i in AsrSlot._held or not self._lock(i):
- return False
- AsrSlot._held.add(i)
- return True
- def _lock(self, i: int) -> bool:
- try:
- if IS_WIN:
- import msvcrt
- self.fh.seek(i)
- msvcrt.locking(self.fh.fileno(), msvcrt.LK_NBLCK, 1)
- else:
- import fcntl
- fcntl.lockf(self.fh, fcntl.LOCK_EX | fcntl.LOCK_NB, 1, i, 0)
- return True
- except OSError:
- return False
- def __enter__(self):
- with AsrSlot._guard:
- if AsrSlot._file is None:
- try:
- RUN_DIR.mkdir(parents=True, exist_ok=True)
- AsrSlot._file = open(RUN_DIR / "asr.lock", "a+b")
- except PermissionError:
- print("asr: $BLURT_HOME/run is read-only here (a sandboxed run): not queued", file=sys.stderr, flush=True)
- return self
- self.fh = AsrSlot._file
- t0 = time.time()
- queued = False
- while True:
- for i in range(self.n):
- if self._try(i):
- self.slot = i
- self.waited = time.time() - t0
- if queued:
- print("asr: slot free, starting", file=sys.stderr, flush=True)
- job_event("asr.started", {"id": "transcribe", "state": "running"}, {"waited_seconds": round(self.waited, 1)})
- return self
- if not queued:
- queued = True
- print(f"asr: queued (max_concurrent={self.n}; another transcription is running)", file=sys.stderr, flush=True)
- job_event("asr.queued", {"id": "transcribe", "state": "pending"}, {"queued": True, "max_concurrent": self.n})
- if time.time() - t0 > self.timeout:
- raise Fail(f"waited {self.timeout:.0f}s for a free ASR slot", "lock_timeout", 3)
- time.sleep(self.poll)
- def __exit__(self, *exc):
- if self.slot is None:
- return False
- with AsrSlot._guard:
- try:
- if IS_WIN:
- import msvcrt
- self.fh.seek(self.slot)
- msvcrt.locking(self.fh.fileno(), msvcrt.LK_UNLCK, 1)
- else:
- import fcntl
- fcntl.lockf(self.fh, fcntl.LOCK_UN, 1, self.slot, 0)
- except OSError:
- pass
- AsrSlot._held.discard(self.slot)
- return False
- def job_event(event: str, step: dict | None = None, detail: dict | None = None) -> None:
- job = os.environ.get("BLURT_JOB")
- if not job or not re.fullmatch(r"[A-Za-z0-9._-]+", job):
- return
- line = {"t": datetime.now().astimezone().isoformat(timespec="seconds"), "source": "asr", "event": event}
- if step:
- line["step"] = step
- if detail:
- line["detail"] = detail
- try:
- (BLURT_HOME / "jobs").mkdir(parents=True, exist_ok=True)
- fd = os.open(BLURT_HOME / "jobs" / f"{job}.events.jsonl", os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o644)
- try:
- os.write(fd, (json.dumps(line, ensure_ascii=False) + "\n").encode())
- finally:
- os.close(fd)
- except OSError:
- pass
- def peak_memory_mb() -> float | None:
- """This process's peak memory: macOS lifetime max physical footprint (counts GPU buffers in unified memory),
- Linux max RSS, Windows peak working set."""
- try:
- if IS_MAC:
- import ctypes
- buf = (ctypes.c_uint64 * 64)()
- lib = ctypes.CDLL("/usr/lib/libSystem.B.dylib")
- if lib.proc_pid_rusage(os.getpid(), 4, ctypes.byref(buf)) == 0: # RUSAGE_INFO_V4
- return round(buf[2 + 28] / 2**20, 1) # after the 16-byte uuid: ri_lifetime_max_phys_footprint
- return None
- if IS_WIN:
- import ctypes
- from ctypes import wintypes
- class PMC(ctypes.Structure):
- _fields_ = [("cb", wintypes.DWORD), ("PageFaultCount", wintypes.DWORD),
- ("PeakWorkingSetSize", ctypes.c_size_t), ("WorkingSetSize", ctypes.c_size_t),
- ("QuotaPeakPagedPoolUsage", ctypes.c_size_t), ("QuotaPagedPoolUsage", ctypes.c_size_t),
- ("QuotaPeakNonPagedPoolUsage", ctypes.c_size_t), ("QuotaNonPagedPoolUsage", ctypes.c_size_t),
- ("PagefileUsage", ctypes.c_size_t), ("PeakPagefileUsage", ctypes.c_size_t)]
- c = PMC()
- c.cb = ctypes.sizeof(PMC)
- ctypes.windll.psapi.GetProcessMemoryInfo(ctypes.windll.kernel32.GetCurrentProcess(), ctypes.byref(c), c.cb)
- return round(c.PeakWorkingSetSize / 2**20, 1)
- import resource
- return round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 1)
- except Exception: # noqa: BLE001 — measurement only
- return None
- def qos_cpu_seconds() -> dict | None:
- """macOS: CPU seconds this process spent per QoS class (proves the priority took effect)."""
- if not IS_MAC:
- return None
- try:
- import ctypes
- lib = ctypes.CDLL("/usr/lib/libSystem.B.dylib")
- buf = (ctypes.c_uint64 * 64)()
- if lib.proc_pid_rusage(os.getpid(), 4, ctypes.byref(buf)) != 0:
- return None
- tb = (ctypes.c_uint32 * 2)() # mach_timebase_info: the counters are in mach ticks (41.67 ns on Apple Silicon)
- lib.mach_timebase_info(ctypes.byref(tb))
- ns = tb[0] / tb[1] if tb[1] else 1.0
- names = ("default", "maintenance", "background", "utility", "legacy", "user_initiated", "user_interactive")
- return {n: round(buf[2 + 18 + i] * ns / 1e9, 2) for i, n in enumerate(names) if buf[2 + 18 + i]}
- except (OSError, AttributeError):
- return None
- # ------------------------------------------------------------------ downloads
- def _download_url(urls: list[str], dest: Path, label: str, aux: str | None = None) -> Path:
- if os.environ.get("BLURT_HF_MIRROR"): # the China-friendly mirrors first (GitHub releases don't load from there)
- urls = sorted(urls, key=lambda u: not ("hf-mirror" in u or "modelscope.cn" in u))
- dest.parent.mkdir(parents=True, exist_ok=True)
- last = None
- for url in urls:
- try:
- serve_said(f"downloading {label} <- {url}")
- tmp = dest.with_suffix(dest.suffix + ".part") # a partial one from last time is continued (Range)
- t0 = [time.time()]
- def progress(done: int, total: int) -> None:
- if total and time.time() - t0[0] > 1:
- serve_said(progress_line(done, total, aux))
- t0[0] = time.time()
- fetch(url, tmp, progress=progress, timeout=30)
- os.replace(tmp, dest)
- return dest
- except Exception as e: # noqa: BLE001 — try the next mirror
- last = e
- print(f" failed: {e}", file=sys.stderr)
- raise Fail(f"could not download {label}: {last}. Set BLURT_HF_MIRROR=1 or download manually to {dest}", "network")
- def download(rel: str, aux: str | None = None) -> Path:
- """A single model file under MODELS_DIR (the VAD, SenseVoice's files). `aux`: fetched on the way to another
- model (its progress lines are named so: progress_line)."""
- dest = MODELS_DIR / rel
- if rel == "silero_vad.onnx":
- bundled_vad()
- if dest.exists() and dest.stat().st_size > 0:
- return dest
- if os.environ.get("BLURT_NO_DOWNLOAD"): # live.py: never start a download in the middle of a recording
- raise Fail(f"{rel} is not downloaded", "model_missing")
- urls = VAD_URLS if rel == "silero_vad.onnx" else VP_URLS if rel == VP_FILE else VP_SEG_URLS if rel == VP_SEG_FILE else next(
- (u for m in entries() for f, u in ((m.get("install") or {}).get("files") or {}).items() if f == rel), None)
- if not urls:
- raise Fail(f"no download source for {rel}", "model_missing")
- return _download_url(urls, dest, rel, aux)
- def hf_cache() -> Path:
- if os.environ.get("HF_HUB_CACHE"):
- return Path(os.environ["HF_HUB_CACHE"])
- return Path(os.environ.get("HF_HOME") or Path.home() / ".cache" / "huggingface") / "hub"
- def _hf_record(repo: str) -> Path:
- """What a finished `snapshot_download` of `repo` left: MODELS_DIR/asr/.hf/<repo>.json (its snapshot and files)."""
- return MODELS_DIR / "asr" / ".hf" / (repo.replace("/", "--") + ".json")
- def _snapshot_complete(s: Path, repo: str) -> bool:
- """A Hugging Face snapshot is a whole model. One `setup` saw finish: every file it listed is there. One from before
- that record (or fetched by something else): its weights, its config, every shard its index names, and the
- tokenizer its tokenizer_config.json asks for — a setup cut short between files leaves one of them out."""
- rec = load_json(_hf_record(repo), None)
- if isinstance(rec, dict) and rec.get("snapshot") == s.name and isinstance(rec.get("files"), list):
- return all((s / f).is_file() for f in rec["files"])
- names = {f.name for f in s.iterdir()} if s.is_dir() else set()
- if not any(f.endswith((".safetensors", ".npz", ".bin")) and not f.startswith("training_args") for f in names):
- return False
- if "config.json" not in names or any(f.endswith((".part", ".incomplete")) for f in names):
- return False
- if any(f.endswith(".safetensors") for f in names) and not safetensors_complete(s):
- return False
- if "tokenizer_config.json" in names and not names & {"tokenizer.json", "vocab.json", "tokenizer.model"}:
- return False
- if "tokenizer_config.json" in names and "vocab.json" in names and "tokenizer.json" not in names \
- and "merges.txt" not in names: # a BPE tokenizer in two files: both
- return False
- return True
- def hf_snapshot(repo: str) -> Path | None:
- """The newest complete snapshot of `repo` in the Hugging Face cache (_snapshot_complete), else None."""
- root = hf_cache() / ("models--" + repo.replace("/", "--"))
- snaps = sorted((root / "snapshots").glob("*"), key=lambda p: p.stat().st_mtime, reverse=True) if root.is_dir() else []
- if any((root / "blobs").glob("*.incomplete")):
- return None
- for s in snaps:
- if _snapshot_complete(s, repo):
- return s
- return None
- def model_path(m: dict) -> Path | None:
- """Where an installed model lives, or None when it is not installed."""
- inst = m.get("install") or {}
- if inst.get("files"):
- paths = [MODELS_DIR / f for f in inst["files"]]
- return paths[0].parent if all(p.exists() and p.stat().st_size > 0 for p in paths) else None
- if inst.get("archive"):
- d = MODELS_DIR / inst["dir"]
- need = [d / k.rstrip("/") for k in inst.get("keep", [])]
- return d if d.is_dir() and all(p.exists() for p in need) else None
- if inst.get("hf"):
- own = MODELS_DIR / "asr" / inst["hf"].split("/")[-1] # a ModelScope download (the mirror)
- if folder_complete(own, legacy=_own_legacy_complete):
- return own
- snap = hf_snapshot(inst["hf"])
- if snap is None and m["backend"] == "faster-whisper": # older Blurt downloaded it under models/faster-whisper
- legacy = MODELS_DIR / "faster-whisper"
- if legacy.is_dir():
- snap = next((p for p in legacy.rglob("model.bin") if "turbo" in str(p)), None)
- snap = snap.parent if snap else None
- return snap
- return None
- def _own_legacy_complete(d: Path) -> bool:
- """A mirror download from before the manifest: its config, every weight shard, and no file still partial."""
- return (d / "config.json").exists() and safetensors_complete(d) and not any(d.glob("*.part"))
- def installed(m: dict) -> bool:
- if not m.get("local", True):
- # every one of its variables when it needs a pair (Volcengine: App ID + Access Token), else any one
- return (all if m.get("keys_all") else any)(os.environ.get(k) for k in m.get("keys", []))
- if m.get("builtin"):
- return apple_speech_ok()
- return model_path(m) is not None
- # macOS's own speech recognition (docs/design/first-minute-speech.md): SpeechTranscriber, macOS 26+, run by the Blurt
- # app's executable (`blurt-recorder --speech WAV`: Capture/AppleSpeech.swift). Nothing to download from Blurt; the
- # language's files are macOS's own (the app asks for them; a run without them fails "model_missing").
- APPLE = "apple-speech"
- APPLE_LOCALES = {"zh": "zh-CN", "zh-hant": "zh-TW", "yue": "yue-CN", "en": "en-US", "ja": "ja-JP", "ko": "ko-KR",
- "fr": "fr-FR", "de": "de-DE", "es": "es-ES", "it": "it-IT", "pt": "pt-BR"}
- def apple_speech_helper() -> Path | None:
- """The Blurt executable that answers `--speech` (BLURT_APP_BIN from the app, else an installed Blurt.app)."""
- cands = [os.environ.get("BLURT_APP_BIN") or "", str(BLURT_HOME / "Blurt.app/Contents/MacOS/blurt-recorder"),
- "/Applications/Blurt.app/Contents/MacOS/blurt-recorder",
- str(Path.home() / "Applications/Blurt.app/Contents/MacOS/blurt-recorder")]
- return next((Path(c) for c in cands if c and os.access(c, os.X_OK)), None)
- _APPLE_OK: bool | None = None
- def apple_speech_ok() -> bool:
- """macOS 26 or later, and a Blurt that can run it."""
- global _APPLE_OK
- if os.environ.get("BLURT_NO_APPLE_SPEECH"): # tests: a Mac before macOS 26
- return False
- if _APPLE_OK is None:
- try:
- major = int((platform.mac_ver()[0] or "0").split(".")[0]) if IS_MAC else 0
- except ValueError:
- major = 0
- _APPLE_OK = major >= 26 and apple_speech_helper() is not None
- return _APPLE_OK
- def apple_locale(language: str | None) -> str | None:
- """The locale for an explicit language (or asr.language), else the app's "Automatic" (AppleSpeech.automatic, the
- same rule so recordings and dictation hear the same language): of the user's languages the first one people mix
- English into (SWITCH_LANGUAGES: its recogniser hears the English words too), else the first one macOS's
- recogniser speaks. An English Mac that lists Chinese too listens for Chinese, English included."""
- picked = asr_config().get("language")
- if language and language not in ("auto", ""):
- langs = [language]
- elif picked and picked != "auto":
- langs = [picked]
- else:
- spoken = [x for x in user_languages() if x in APPLE_LOCALES]
- langs = [x for x in spoken if x in SWITCH_LANGUAGES] + spoken
- for lang in langs:
- if lang == "zh" and chinese_script() == "hant":
- lang = "zh-hant"
- if lang in APPLE_LOCALES:
- return APPLE_LOCALES[lang]
- return None
- def _du_mb(p: Path) -> float:
- if p.is_file():
- return round(p.stat().st_size / 2**20, 1)
- total = 0
- for f in p.rglob("*"):
- try:
- total += f.stat().st_size if f.is_file() else 0 # stat follows the HF cache's symlinks into blobs/
- except OSError:
- pass
- return round(total / 2**20, 1)
- def _progress_poller(target: Path, total_mb: float, stop: threading.Event) -> None:
- """HF downloads: report ' N/M MB' from what has landed on disk (incl. *.incomplete blobs: _common.landed_in)."""
- while not stop.wait(1.0):
- if target.exists():
- serve_said(progress_line(min(landed_in(target), total_mb * 2**20), total_mb * 2**20))
- def install_model(m: dict) -> Path:
- """Download a registry model (and the VAD every local backend needs). Progress on stderr. One download of a
- model at a time (download_lock): a second caller waits, then finds it installed."""
- download("silero_vad.onnx", aux="vad") # its own named progress: not the model's (progress_line)
- if not m.get("local", True):
- return MODELS_DIR
- have = model_path(m)
- if have:
- return have
- if os.environ.get("BLURT_NO_DOWNLOAD"):
- raise Fail(f"{m['id']} is not downloaded", "model_missing")
- if not runs_here(m):
- raise Fail(f"{m['id']} does not run on {platform_key()}", "unsupported")
- with download_lock(m["id"]):
- have = model_path(m) # the other download finished while this one waited
- if have:
- return have
- return _install_unlocked(m)
- def _install_unlocked(m: dict) -> Path:
- inst = m["install"]
- if inst.get("files"):
- for rel in inst["files"]:
- download(rel)
- elif inst.get("archive"):
- _install_archive(m)
- elif inst.get("hf"):
- ensure_package(m)
- os.environ.setdefault("HF_HUB_DISABLE_PROGRESS_BARS", "1") # our " N/M MB" lines are the progress
- from huggingface_hub import snapshot_download
- if os.environ.get("BLURT_HF_MIRROR"):
- os.environ.setdefault("HF_ENDPOINT", "https://hf-mirror.com")
- mirrored = inst["hf"].startswith("mlx-community/") # ModelScope mirrors mlx-community (fast from China)
- if mirrored and os.environ.get("BLURT_HF_MIRROR"):
- _modelscope(inst["hf"], float(m["size_mb"]))
- else:
- print(f"downloading {inst['hf']}", file=sys.stderr, flush=True)
- stop = threading.Event()
- target = hf_cache() / ("models--" + inst["hf"].replace("/", "--"))
- th = threading.Thread(target=_progress_poller, args=(target, float(m["size_mb"]), stop), daemon=True)
- th.start()
- try:
- snap = Path(snapshot_download(inst["hf"]))
- # it finished: what it brought is the model (hf_snapshot checks against this list from now on)
- write_text_atomic(_hf_record(inst["hf"]), json.dumps(
- {"snapshot": snap.name, "files": sorted(str(f.relative_to(snap)) for f in snap.rglob("*") if f.is_file())}))
- except Exception as e: # noqa: BLE001
- if not mirrored:
- raise Fail(f"could not download {inst['hf']}: {e}", "network")
- print(f" Hugging Face failed ({e}); trying ModelScope", file=sys.stderr, flush=True)
- _modelscope(inst["hf"], float(m["size_mb"]))
- finally:
- stop.set()
- path = model_path(m)
- if not path:
- raise Fail(f"{m['id']}: download finished but the files are not complete", "model_missing")
- return path
- def _modelscope(repo: str, total_mb: float) -> Path:
- """An mlx-community repo from ModelScope's mirror → MODELS_DIR/asr/<name>: the manifest and the download marker
- first (the folder is a model only once every file is in: folder_complete), each file through <name>.part,
- continued where a cut-short download left it."""
- dest = MODELS_DIR / "asr" / repo.split("/")[-1]
- api = f"https://modelscope.cn/api/v1/models/{repo}/repo/files?Recursive=true"
- try:
- with urllib.request.urlopen(urllib.request.Request(api, headers={"User-Agent": "blurt"}), timeout=30) as r:
- files = json.load(r)["Data"]["Files"]
- except Exception as e: # noqa: BLE001
- raise Fail(f"could not download {repo} from ModelScope: {e}", "network")
- want = [x for x in files if x.get("Type") == "blob" and not x["Path"].endswith((".md", ".gitattributes"))]
- try:
- begin_download(dest, [{"path": x["Path"], "size": x.get("Size")} for x in want])
- except ValueError as e:
- raise Fail(f"invalid ModelScope manifest: {e}", "network") from None
- done = 0.0
- serve_said(f"downloading {repo} (ModelScope)")
- for x in want:
- p = download_path(dest, x["Path"])
- if p.exists() and p.stat().st_size == x.get("Size"):
- done += p.stat().st_size / 2**20
- continue
- p.parent.mkdir(parents=True, exist_ok=True)
- tmp = p.with_name(p.name + ".part")
- url = f"https://modelscope.cn/models/{repo}/resolve/master/{x['Path']}"
- t0 = [time.time()]
- def progress(have: int, _total: int) -> None:
- if time.time() - t0[0] > 1:
- serve_said(f" {int(done + have / 2**20)}/{int(total_mb)} MB")
- t0[0] = time.time()
- got = 0
- for attempt in range(3):
- try:
- got = fetch(url, tmp, x.get("Size"), progress) # each try continues the last one's bytes
- except Exception as e: # noqa: BLE001
- if attempt == 2:
- raise Fail(f"could not download {repo} from ModelScope: {e}", "network")
- continue
- if not x.get("Size") or got == x["Size"]: # a dropped connection can end the read quietly
- break
- if attempt == 2:
- raise Fail(f"could not download {repo} from ModelScope: {x['Path']} came short", "network")
- done += got / 2**20
- os.replace(tmp, p)
- end_download(dest)
- return dest
- def _install_archive(m: dict) -> None:
- import tarfile
- inst = m["install"]
- dest = MODELS_DIR / inst["dir"]
- tmp = MODELS_DIR / f".{inst['dir']}.tar.bz2"
- urls = inst["archive"] if isinstance(inst["archive"], list) else [inst["archive"]]
- _download_url(urls, tmp, m["id"])
- staging = MODELS_DIR / f".{inst['dir']}.extract"
- shutil.rmtree(staging, ignore_errors=True)
- staging.mkdir(parents=True)
- keep = inst.get("keep") or []
- with tarfile.open(tmp, "r:bz2") as tar:
- members = []
- for mem in tar.getmembers():
- parts = Path(mem.name).parts
- rel = "/".join(parts[1:]) # drop the archive's top folder
- if not rel or ".." in parts or mem.name.startswith("/") or not (mem.isfile() or mem.isdir()):
- continue
- if keep and not any(rel == k or (k.endswith("/") and rel.startswith(k)) for k in keep):
- continue
- mem.name = rel
- members.append(mem)
- print(f"unpacking {m['id']}", file=sys.stderr, flush=True)
- tar.extractall(staging, members=members, **({"filter": "data"} if sys.version_info >= (3, 12) else {}))
- shutil.rmtree(dest, ignore_errors=True)
- os.replace(staging, dest)
- tmp.unlink(missing_ok=True)
- # ------------------------------------------------------------------ optional packages
- class NeedsRestart(Exception):
- """A running `serve` was asked for a model whose package isn't in its environment: it answers "restarting",
- takes its run file away and starts again with the package (`_reexec_with`) — never mid-request."""
- def __init__(self, pkg: str):
- super().__init__(f"the speech server restarts to load {pkg}")
- self.pkg = pkg
- # set by `serve` once its run file is written: from then on a missing package is a clean restart (NeedsRestart),
- # not an exec in the middle of a request
- SERVING = False
- def _has_package(pkg: str) -> bool:
- import importlib.util
- return importlib.util.find_spec(pkg.replace("-", "_")) is not None
- def _reexec_with(pkgs: list[str]) -> None:
- """Run this very command again under `uv run --with` these packages (and the ones already added)."""
- uv = shutil.which("uv") or str(BLURT_HOME / "bin" / ("uv.exe" if IS_WIN else "uv"))
- tried = [p for p in os.environ.get("BLURT_ASR_REEXEC", "").split(",") if p]
- new = [p for p in dict.fromkeys(pkgs) if p not in tried]
- if not new or not Path(uv).exists() or not sys.argv[0].endswith(".py"):
- pkg = (new or pkgs or ["?"])[0]
- raise Fail(f"{pkg} is not installed (run with: uv run --with {pkg} {Path(sys.argv[0]).name} …)", "deps", 4)
- os.environ["BLURT_ASR_REEXEC"] = ",".join(tried + new)
- sys.stdout.flush()
- sys.stderr.flush()
- withs = [x for p in tried + new for x in ("--with", PACKAGE_PINS.get(p, p))]
- os.execvp(uv, [uv, "run", "-q", *withs, os.path.abspath(sys.argv[0]), *sys.argv[1:]])
- def ensure_package(m: dict | str) -> None:
- """Backends that need a Python package (mlx-whisper, parakeet-mlx, …): re-run this very command under
- `uv run --with <package>` when it is missing, so callers never need to know. Inside a running server:
- NeedsRestart (the server restarts itself between requests)."""
- pkg = m if isinstance(m, str) else m.get("package")
- if not pkg or _has_package(pkg):
- return
- tried = [p for p in os.environ.get("BLURT_ASR_REEXEC", "").split(",") if p]
- if SERVING and pkg not in tried:
- raise NeedsRestart(pkg)
- _reexec_with([pkg])
- def serve_packages() -> list[str]:
- """The packages of every local model on this Mac that a server may be asked for (any of them can become
- dictation's in Settings): a server starts with all of them, so a model switch never restarts it. Before, it
- re-ran itself under `uv run --with` in the middle of the request that needed one (2026-10-31: dictation switched
- to Whisper) — the request got no answer, and until the new process was up its run file named a live pid with a
- closed port, so every piece of the take went to a cold CLI run instead (a 30 s Whisper load each)."""
- plat = platform_key()
- out = []
- for m in entries(local_only=True):
- pkg = EXTRA_PACKAGES.get(m.get("backend"))
- if pkg and runs_here(m, plat) and installed(m):
- out.append(pkg)
- return sorted(set(out))
- # ------------------------------------------------------------------ audio + VAD
- def audio_tracks(path: Path) -> int:
- info = subprocess.run([ffmpeg_bin(), "-nostdin", "-hide_banner", "-i", str(path)], capture_output=True, text=True,
- errors="replace").stderr
- return sum(1 for line in info.splitlines() if "Stream #" in line and "Audio:" in line)
- def load_audio(path: Path, track: int = 0) -> np.ndarray:
- raw = subprocess.run([ffmpeg_bin(), "-nostdin", "-loglevel", "error", "-i", str(path), "-map", f"0:a:{track}", "-ac", "1",
- "-af", "aresample=async=1000:first_pts=0", "-ar", str(SR), "-f", "s16le", "-"], capture_output=True).stdout
- if not raw:
- die(f"No audio track decoded from {path}")
- return np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768
- def read_wav(path: Path) -> np.ndarray:
- """A 16 kHz mono 16-bit WAV without ffmpeg (voice notes); anything else goes through load_audio."""
- try:
- with wave.open(str(path)) as w:
- if w.getframerate() == SR and w.getnchannels() == 1 and w.getsampwidth() == 2:
- return np.frombuffer(w.readframes(w.getnframes()), np.int16).astype(np.float32) / 32768
- except (wave.Error, EOFError, OSError):
- pass
- return load_audio(path)
- def make_vad(max_speech: float = 25):
- """Silero VAD as used everywhere (whole-file runs and live.py's streaming run share these settings)."""
- import sherpa_onnx
- cfg = sherpa_onnx.VadModelConfig()
- cfg.silero_vad.model = str(download("silero_vad.onnx"))
- cfg.silero_vad.threshold = 0.45
- cfg.silero_vad.min_silence_duration = 0.3
- cfg.silero_vad.min_speech_duration = 0.25
- cfg.silero_vad.max_speech_duration = max_speech
- cfg.sample_rate = SR
- cfg.num_threads = 1
- return sherpa_onnx.VoiceActivityDetector(cfg, buffer_size_in_seconds=60), cfg.silero_vad.window_size
- def pad_segment(s: float, e: float, dur: float = float("inf")) -> tuple[float, float]:
- """Pad a little so words at the edges are not clipped."""
- return max(0.0, s - 0.15), min(dur, e + 0.2)
- def vad_segments(samples: np.ndarray, max_speech: float = 25) -> list[tuple[float, float]]:
- vad, win = make_vad(max_speech)
- segs = []
- def drain():
- while not vad.empty():
- seg = vad.front
- segs.append((seg.start / SR, (seg.start + len(seg.samples)) / SR))
- vad.pop()
- for i in range(0, len(samples), win):
- vad.accept_waveform(samples[i:i + win])
- drain()
- vad.flush()
- drain()
- dur = len(samples) / SR
- return [pad_segment(s, e, dur) for s, e in segs]
- def group_chunks(segs: list[tuple[float, float]], max_len: float, max_gap: float = 1.0) -> list[tuple[float, float]]:
- """Merge neighbouring speech segments into contiguous windows (original timeline, no remapping needed)."""
- chunks: list[list[float]] = []
- for s, e in segs:
- if chunks and s - chunks[-1][1] <= max_gap and e - chunks[-1][0] <= max_len:
- chunks[-1][1] = e
- else:
- chunks.append([s, e])
- return [(s, e) for s, e in chunks]
- def pause_spans(samples: np.ndarray, max_len: float) -> list[tuple[float, float]]:
- """The audio cut into contiguous spans of at most `max_len` s, each cut in the middle of a pause the VAD heard (a
- hard cut only inside speech longer than that); spans with no speech in them are left out. Nothing at a span's edges
- is trimmed: for a model that hears an utterance whole (Fun-ASR-Nano: its piece's first 0.18 s of silence cut off
- turned 语音输入 into 语音水软, a 1.1 s VAD chunk of a 5.9 s piece came back as "Puritan")."""
- dur = len(samples) / SR
- segs = vad_segments(samples, max_speech=min(25, max_len))
- if not segs:
- return []
- cuts = [(a[1] + b[0]) / 2 for a, b in zip(segs, segs[1:]) if b[0] > a[1]]
- spans, start = [], 0.0
- for i, c in enumerate(cuts + [dur]):
- nxt = cuts[i + 1] if i + 1 < len(cuts) else dur
- while c - start > max_len: # one stretch of speech longer than a span
- spans.append((start, start + max_len))
- start += max_len
- if nxt - start > max_len and c > start:
- spans.append((start, c))
- start = c
- if dur > start:
- spans.append((start, dur))
- return [(a, b) for a, b in spans if any(s < b and e > a for s, e in segs)]
- def wav_bytes(samples: np.ndarray) -> bytes:
- buf = io.BytesIO()
- with wave.open(buf, "wb") as w:
- w.setnchannels(1)
- w.setsampwidth(2)
- w.setframerate(SR)
- w.writeframes((np.clip(samples, -1, 1) * 32767).astype(np.int16).tobytes())
- return buf.getvalue()
- # ------------------------------------------------------------------ backends
- # each backend: transcribe(samples) -> list[(rel_start, rel_end, text)] or [(…, text, language)]
- LANG_NAMES = {"zh": "Chinese", "en": "English", "yue": "Cantonese", "ja": "Japanese", "ko": "Korean", "es": "Spanish",
- "fr": "French", "de": "German", "it": "Italian", "pt": "Portuguese", "ru": "Russian", "ar": "Arabic",
- "hi": "Hindi", "th": "Thai", "vi": "Vietnamese", "id": "Indonesian", "ms": "Malay", "tr": "Turkish",
- "nl": "Dutch", "sv": "Swedish", "da": "Danish", "fi": "Finnish", "pl": "Polish", "cs": "Czech",
- "el": "Greek", "hu": "Hungarian", "ro": "Romanian", "fa": "Persian", "fil": "Filipino", "mk": "Macedonian"}
- LANG_CODES = {v.lower(): k for k, v in LANG_NAMES.items()}
- def lang_code(v) -> str | None:
- """'<|zh|>' / 'Chinese' / 'zh' → 'zh'."""
- if not v or not isinstance(v, str):
- return None
- v = v.strip().strip("<|>").strip()
- return LANG_CODES.get(v.lower()) or (v.lower() if re.fullmatch(r"[a-z]{2,3}", v.lower()) else None)
- _CJK = r"-ヿ㐀-鿿가-"
- def syllables(text: str) -> int:
- """Rough syllable count in any script: a CJK character, kana or Hangul block is one; elsewhere a vowel group."""
- n = 0
- for w in re.findall(r"\w+", unicodedata.normalize("NFKC", text)):
- n += len(re.findall(f"[{_CJK}]", w))
- n += sum(max(1, len(re.findall(r"[aeiouyàáâãäåæèéêëìíîïòóôõöøœùúûüýÿаеёиоуыэюя]+", p.lower())))
- for p in re.sub(f"[{_CJK}]", " ", w).split())
- return n
- _FULLWIDTH = {",": ",", "?": "?", "!": "!", ":": ":", ";": ";"}
- def cjk_punctuation(text: str) -> str:
- """Whisper writes Chinese / Japanese with ASCII commas ("页面,输入"): use the full-width marks readers expect."""
- return re.sub(f"(?<=[{_CJK}])([,?!:;])\\s*", lambda m: _FULLWIDTH[m.group(1)], text)
- def _threads(args) -> int:
- return getattr(args, "threads", None) or limits()["threads"]
- def _model_dir(args, default: str) -> Path:
- m = getattr(args, "model", None) or default
- return Path(m) if os.path.isabs(m) else MODELS_DIR / m
- class SenseVoice:
- # SenseVoice degrades on long, mixed-language chunks (drops whole clauses); keep utterances short
- max_chunk = 10
- model = "sense-voice-small-int8"
- # Non-autoregressive (one CTC pass): a piece decodes in one call of about 60–120 ms on the CPU (a 4 s piece:
- # 59–84 ms, M-series), so there is nothing to stop halfway — an interim request (the live words of a voice note,
- # a snap's note, dictation) is served whole after one look for a waiting request (NoteEngine.note).
- quick_decode = True
- def __init__(self, args):
- import sherpa_onnx
- lang = args.language if args.language in ("zh", "en", "ja", "ko", "yue") else "auto"
- self.rec = sherpa_onnx.OfflineRecognizer.from_sense_voice(
- model=str(download("sensevoice/model.int8.onnx")), tokens=str(download("sensevoice/tokens.txt")),
- num_threads=_threads(args), use_itn=True, language=lang)
- def transcribe(self, x):
- st = self.rec.create_stream()
- st.accept_waveform(SR, x)
- self.rec.decode_stream(st)
- return [(0.0, len(x) / SR, st.result.text.strip(), lang_code(getattr(st.result, "lang", None)))]
- # No set_vocab: sherpa-onnx can't bias SenseVoice (greedy CTC only; create_stream(hotwords=…) aborts the process
- # with "Only transducer models support contextual biasing"). Its words go to polish.
- class Qwen3Onnx:
- """Qwen3-ASR 0.6B int8 on the CPU through sherpa-onnx: 30 languages on any OS."""
- max_chunk = 20
- max_terms = dictation_vocab.MAX_TERMS # its hotwords
- def __init__(self, args):
- import sherpa_onnx
- d = _model_dir(args, "qwen3-asr-0.6b-int8")
- if not (d / "decoder.int8.onnx").exists():
- raise Fail("Qwen3-ASR (CPU) is not downloaded: transcribe.py setup --model-id qwen3-asr-0.6b-onnx", "model_missing")
- self.model = "qwen3-asr-0.6b-int8"
- self.rec = sherpa_onnx.OfflineRecognizer.from_qwen3_asr(
- conv_frontend=str(d / "conv_frontend.onnx"), encoder=str(d / "encoder.int8.onnx"),
- decoder=str(d / "decoder.int8.onnx"), tokenizer=str(d / "tokenizer"), num_threads=_threads(args),
- max_total_len=1024, max_new_tokens=400, hotwords=args.prompt or "")
- self.hotwords = ""
- def set_vocab(self, terms: list[dict]) -> bool:
- """Per stream (no rebuild): "; "-joined — sherpa splits ASCII commas and re-joins them with spaces."""
- self.hotwords = dictation_vocab.hotwords(terms)
- return bool(self.hotwords)
- def transcribe(self, x):
- st = self.rec.create_stream()
- if self.hotwords:
- try:
- st.set_option("hotwords", self.hotwords)
- except Exception: # noqa: BLE001 — an older sherpa-onnx: no per-stream hotwords
- pass
- st.accept_waveform(SR, x)
- self.rec.decode_stream(st)
- return [(0.0, len(x) / SR, st.result.text.strip(), lang_code(getattr(st.result, "lang", None)))]
- class FunAsrOnnx:
- """Fun-ASR-Nano int8 on the CPU through sherpa-onnx: Windows' and Linux's dictation model for zh / en / ja
- (docs/design/windows-models.md: 3.69 % on the founder's takes, RTF 0.12 on 4 Zen 4 threads). The export caps prompt
- + audio at 512 tokens — about 25 s without hotwords — so pieces stay ≤ 20 s and the hotwords short; a longer
- piece comes back empty. Hotwords are fixed when the recognizer is built, so a new Words list rebuilds it."""
- max_chunk = 20
- whole_spans = True # like FunAsrMlx: a piece is heard whole, never trimmed by the VAD
- vad_gate = True # …but a piece with no speech in it isn't decoded (transcribe_track)
- max_terms = 24 # each term costs prompt tokens out of the 512
- app_hint = False # its hotwords are fixed when the recognizer is built (bias)
- def __init__(self, args):
- d = _model_dir(args, "fun-asr-nano-int8")
- if not (d / "llm.int8.onnx").exists():
- raise Fail("Fun-ASR-Nano (CPU) is not downloaded: transcribe.py setup --model-id fun-asr-nano-onnx",
- "model_missing")
- self.model = "fun-asr-nano-int8"
- self.dir, self.threads, self.hotwords = d, _threads(args), ""
- self.rec = self._build("")
- # The decoder's own cap is 512 new tokens. Speech is under ~8 tokens a second (the Mac's FunAsrMlx says the same),
- # so a piece of at most 20 s needs ~160: a repetition loop (it happens: the dogfood's 6 s of English, replayed on
- # the VM, came back as 2,123 characters of one clause after 14.2 s of decoding) stops at this instead of running on
- # to 512. The Mac caps each decode at 32 + 20 × its seconds; here the cap is fixed when the recognizer is built.
- MAX_NEW_TOKENS = 32 + 8 * max_chunk
- looped = False # the last transcribe cut a loop (the server's answer says so: the app won't send it)
- # A transcript longer than this many characters a second of audio (+ LOOP_SLACK) is a loop, not speech: the
- # fastest real dictation is ~5 CJK characters or ~18 Latin characters a second.
- LOOP_CPS, LOOP_SLACK = 30, 40
- def _build(self, hotwords: str):
- import inspect
- import sherpa_onnx
- d = self.dir
- kw = {}
- if "max_new_tokens" in inspect.signature(sherpa_onnx.OfflineRecognizer.from_funasr_nano).parameters:
- kw["max_new_tokens"] = self.MAX_NEW_TOKENS
- return sherpa_onnx.OfflineRecognizer.from_funasr_nano(
- encoder_adaptor=str(d / "encoder_adaptor.int8.onnx"), llm=str(d / "llm.int8.onnx"),
- embedding=str(d / "embedding.int8.onnx"), tokenizer=str(d / "Qwen3-0.6B"), num_threads=self.threads,
- hotwords=hotwords, **kw)
- def set_vocab(self, terms: list[dict]) -> bool:
- hw = dictation_vocab.hotwords(terms[: self.max_terms])
- if hw != self.hotwords:
- self.rec, self.hotwords = self._build(hw), hw
- return bool(hw)
- def _decode(self, x) -> tuple[str, str | None]:
- st = self.rec.create_stream()
- st.accept_waveform(SR, x)
- self.rec.decode_stream(st)
- return st.result.text.strip(), lang_code(getattr(st.result, "lang", None))
- def transcribe(self, x):
- dur = len(x) / SR
- text, lang = self._decode(x)
- cap = self.LOOP_CPS * dur + self.LOOP_SLACK
- if len(text) > cap:
- # a loop: the run repeated over and over is kept once; still too long → the piece heard again in two halves
- # cut at a pause (a loop rarely starts again on a shorter piece), each half kept once in the same way
- text = collapse_repeats(text)
- if len(text) > cap and dur >= 2:
- halves = pause_spans(x, max(1.0, dur / 2 + 0.5))
- if len(halves) > 1:
- parts = [collapse_repeats(self._decode(x[int(s * SR):int(e * SR)])[0]) for s, e in halves]
- joined = ("" if lang in ("zh", "ja", "yue") else " ").join(p for p in parts if p).strip()
- if len(joined) < len(text):
- text = joined
- self.looped = True
- print(f"funasr: a repetition loop in {dur:.1f} s cut to {len(text)} characters", file=sys.stderr, flush=True)
- return [(0.0, dur, text, lang)]
- def collapse_repeats(text: str, min_unit: int = 4, times: int = 4, max_unit: int = 200) -> str:
- """A decoder's repetition loop kept once: any run of the same span (≥ `min_unit` characters) said `times` or more
- times in a row becomes one span; a cut-off copy at the very end goes too. Structural — it compares characters, it
- reads no meaning — and only asked of a transcript already too long for its audio (FunAsrOnnx.transcribe)."""
- n = len(text)
- i = 0
- out = []
- while i < n:
- best = None
- for u in range(min_unit, min(max_unit, (n - i) // times) + 1):
- unit = text[i:i + u]
- k = 1
- while text.startswith(unit, i + k * u):
- k += 1
- if k >= times:
- best = (u, k)
- break
- if best:
- u, k = best
- out.append(text[i:i + u])
- j = i + k * u
- # the loop's last copy cut off by the token cap: a prefix of the span, at the end
- if j < n and text[i:i + u].startswith(text[j:]):
- j = n
- i = j
- else:
- out.append(text[i])
- i += 1
- return "".join(out).strip()
- class MlxBackend:
- """MLX models. MLX streams belong to the thread that made them and some libraries keep the loading thread's
- stream, while callers decode on worker threads (transcribe_track's pool, live.py): so the model is loaded and
- every call runs on one thread of its own. The MLX memory cap is set before loading."""
- max_chunk = 30
- def __init__(self, args):
- self._ex = ThreadPoolExecutor(1, thread_name_prefix="mlx")
- self.args = args
- self._ex.submit(self._init, args).result()
- def _init(self, args):
- cap_mlx_memory(limits()["max_memory_fraction"])
- self.load(args)
- def transcribe(self, x):
- return self._ex.submit(self.decode, np.asarray(x, np.float32)).result()
- def close(self):
- try:
- self._ex.submit(self.release).result(timeout=10) # before the next model loads, not racing it
- except Exception: # noqa: BLE001 — a decode still running: the process ending frees it all the same
- pass
- self._ex.shutdown(wait=False)
- def release(self):
- """Drop this backend's model and MLX's buffer cache (on the MLX thread), so a replacement doesn't sit beside it."""
- try:
- cache = getattr(getattr(self, "holder", None), "_cache", None)
- if cache is not None and hasattr(cache, "_data"):
- cache._data.clear()
- import mlx.core as mx
- getattr(mx, "clear_cache", None) and mx.clear_cache()
- except Exception: # noqa: BLE001 — best effort: the process ending frees it all the same
- pass
- class MlxWhisper(MlxBackend):
- max_terms = dictation_vocab.MAX_TERMS # its prompt (~224 tokens)
- def load(self, args):
- try:
- import mlx.core as mx
- import mlx_whisper
- from mlx_whisper.transcribe import ModelHolder
- except ImportError:
- raise Fail("mlx-whisper not installed (uv run --with mlx-whisper …)", "deps", 4)
- self.m, self.mx = mlx_whisper, mx
- self.repo = self.model = args.model or "mlx-community/whisper-large-v3-turbo"
- self.net = ModelHolder.get_model(self.repo, mx.float16) # load now, not inside the first chunk
- self.langs = [] if args.language else user_languages()
- self.base_prompt = args.prompt
- def set_vocab(self, terms: list[dict]) -> bool:
- """initial_prompt per call: a plain comma list (dictation eval: no insertions, no loops)."""
- self.args.prompt = dictation_vocab.whisper_prompt(terms) or self.base_prompt
- return bool(terms)
- def decode(self, x):
- out = self.whisper(x, self.args.language)
- # Too little text for the speech (normal speech is 4–7 syllables a second in every language we measured):
- # for someone who speaks several languages, the chunk probably switched language and Whisper dropped a part.
- if len(self.langs) > 1 and len(x) > 4 * SR and syllables(" ".join(r[2] for r in out)) < 2.5 * len(x) / SR:
- try:
- runs = self.language_runs(x)
- except Exception as e: # noqa: BLE001 — an MLX / library change must not cost the chunk
- print(f"language split skipped: {type(e).__name__}: {e}", file=sys.stderr, flush=True)
- runs = []
- if len(runs) > 1:
- return [(s + a, s + b, t, lg) for s, e, lang in runs
- for a, b, t, lg in self.whisper(x[int(s * SR):int(e * SR)], lang)]
- return out
- def whisper(self, x, language):
- r = self.m.transcribe(x, path_or_hf_repo=self.repo, language=language, initial_prompt=self.args.prompt,
- condition_on_previous_text=False, verbose=None)
- lang = r.get("language")
- return [(s["start"], s["end"], cjk_punctuation(s["text"].strip()), lang) for s in r.get("segments", [])]
- def language_runs(self, x) -> list[tuple[float, float, str]]:
- """Whisper keeps one language per window: a chunk where one sentence is English and the next Chinese comes
- back as Chinese only (asr-benchmark.md). Find the language of each speech segment in the chunk and decode runs
- of one language apart, each with its language set."""
- segs = vad_segments(x)
- if len(segs) < 2:
- return []
- from mlx_whisper.audio import N_FRAMES, log_mel_spectrogram, pad_or_trim
- from mlx_whisper.decoding import detect_language
- runs: list[list] = []
- for s, e in segs: # one at a time: a batch of 30 s mel windows costs a GB per segment
- mel = pad_or_trim(log_mel_spectrogram(x[int(s * SR):int(e * SR)], n_mels=self.net.dims.n_mels), N_FRAMES,
- axis=-2).astype(self.mx.float16)
- p = detect_language(self.net, mel)[1] # one 2-D mel in → one dict out
- p = p[0] if isinstance(p, list) else p
- lang = max(p, key=p.get)
- if runs and runs[-1][2] == lang:
- runs[-1][1] = e
- else:
- runs.append([s, e, lang])
- return [(s, e, lang) for s, e, lang in runs]
- class Qwen3Mlx(MlxBackend):
- """Qwen3-ASR 0.6B on MLX (mlx-qwen3-asr): ~0.18 s for a 6 s dictation on an M5 Max, and the only fast local
- model here that takes a vocabulary per request (`context`) — dictation's default on Apple Silicon.
- Interruptible: with `stop_check` set (an interim request), the decode looks at it before each stage and each
- token and raises Interrupted — the tag-aware decoder below is a loop of ours, so it can stop between steps."""
- stop_check = None
- @property
- def interruptible(self) -> bool:
- return self.tagged
- def _check(self):
- if self.stop_check is not None and self.stop_check():
- raise Interrupted("another request is waiting")
- def load(self, args):
- try:
- import mlx.core as mx
- import mlx_qwen3_asr
- from mlx_qwen3_asr.load_models import _ModelHolder
- except ImportError:
- raise Fail("mlx-qwen3-asr not installed (uv run --with mlx-qwen3-asr …)", "deps", 4)
- self.m, self.mx = mlx_qwen3_asr, mx
- self.repo = self.model = args.model or "mlx-community/Qwen3-ASR-0.6B-8bit"
- # the package keeps up to 4 loaded models in its own LRU: a server that switched models (dictation ↔ a voice
- # note's pick) held both — about 1 GB more than the one it uses. One is all a backend ever needs.
- cache = getattr(_ModelHolder, "_cache", None)
- if cache is not None and hasattr(cache, "set_max_entries"):
- cache.set_max_entries(1)
- self.holder = _ModelHolder
- _ModelHolder.get(self.repo, dtype=mx.float16) # load now, not inside the first chunk
- self.context = args.prompt or ""
- self.base_context = self.context
- self.names: list[str] = []
- self.prior = self.field = ""
- # The language the user mixes English into (docs/design/dictation.md §Code-switching): Qwen3-ASR writes its
- # whole answer in the language of the tag it decodes first, so a Chinese sentence full of English names that
- # it tags English comes back translated (和 → "and", Musk 的 → "Musk's"). None = leave the tag to the model.
- self.prefer = None if args.language else switch_language(user_languages())
- self.tagged = True # the tag-aware decoder below works with this mlx-qwen3-asr; False after it failed
- self.last: dict = {}
- def set_vocab(self, terms: list[dict]) -> bool:
- self.context = dictation_vocab.qwen_context(terms) or self.base_context
- self.names = [t["term"] for t in terms[:dictation_vocab.QWEN_MAX_TERMS]]
- return bool(terms)
- def set_prior(self, text: str) -> None:
- """A streamed dictation's text so far: after the names, in the same system prompt (docs/design/dictation.md
- §Streaming), so a name is spelled the same way in every piece."""
- self.prior = (text or "").strip()
- def set_field(self, text: str) -> None:
- """The text before the cursor in the field the dictation goes into (docs/design/dictation.md §Field context):
- after the names and before the text so far, in the same system prompt — what the speech is likely about
- (截了 when the field said 我刚截了; blurt, Typeless spelled as the conversation spells them). Never in the
- language tag's prompt (that stays the text so far: tag_context)."""
- self.field = (text or "").strip()
- def decode(self, x):
- prior = getattr(self, "prior", "")
- # the field's text, then the text so far: both after the names
- said = "\n".join(t for t in (getattr(self, "field", ""), prior) if t)
- ctx = (self.context + "\n" + said).strip() if said else self.context
- text, lang = self._decode(x, ctx, prior)
- # The model now and then answers with the word list itself (seen on a noisy piece it couldn't make out:
- # "Claude Code、Codex、blurt、…"): once more with only the text so far (and the field's) as context.
- dump = False
- if self.names and ctx != said and dictation_vocab.context_dump(text, self.names):
- text, lang = self._decode(x, said, prior)
- dump = True
- # The text so far (or the field's) written out again as this piece's words (dict023): once more without it,
- # and the shorter answer is the piece when it is clearly shorter — a speaker who really repeats themself gets
- # the same text.
- echo = False
- if said and echoes_prior(text, said, len(x) / SR):
- redo, lang2 = self._decode(x, self.context, prior)
- if redo.strip() and len(redo) < 0.8 * len(text):
- text, lang, echo = redo, lang2, True
- if dump:
- self.last["context_dump"] = True
- if echo:
- self.last["echo"] = True
- return [(0.0, len(x) / SR, text, lang)]
- def _decode(self, x, ctx: str, prior: str) -> tuple[str, str | None]:
- prefer = self.prefer or ("Chinese" if not self.args.language and dictation_vocab.cjk_share(prior) >= 0.3 else None)
- self.last = {}
- # the language tag is read from the prompt without the Words (only the text so far): see _decode_tagged
- self.tag_context = prior
- # an interim request goes through our own loop even without a preferred tag: it has to be able to stop
- if (prefer or self.stop_check is not None) and self.tagged:
- try:
- return self._decode_tagged(x, ctx, prefer)
- except Interrupted:
- raise
- except Exception as e: # noqa: BLE001 — another mlx-qwen3-asr: its own transcribe, as before
- self.tagged = False
- print(f"qwen3-asr: tag-aware decode unavailable ({type(e).__name__}: {e}); plain decode", file=sys.stderr, flush=True)
- r = self.m.transcribe(x, model=self.repo, context=ctx, language=self.args.language)
- return (r.text or "").strip(), lang_code(getattr(r, "language", None))
- def _decode_tagged(self, x, ctx: str, prefer: str | None) -> tuple[str, str | None]:
- """mlx-qwen3-asr's decode, with the language tag chosen here (docs/design/dictation.md §Code-switching).
- The tag comes from a prompt **without the Words** (`tag_context`: only the dictation's text so far): the
- prompt is prefilled up to `language`, the model's probabilities for English and `prefer` are read there, and
- `prefer` wins unless English is all but certain (dictation_vocab.choose_tag, whose threshold was measured
- without a word list). The text is then decoded with the full context (the Words, which bias the spelling) under
- that tag. The Words framed as an English sentence pushed the tag toward English and made it depend on their
- order: one app-captured mixed sentence went from p_en 0.948 (no Words) to 0.9956 — over the 0.995 line, a clean
- English translation typed — and back under it with the same words in another order (v0.9 dogfood).
- English that still has Chinese characters in it (translated around a name) is decoded once more under
- `prefer`. Cost: one more prefill of the prompt (tens of ms), only when the Words are in the context.
- prefer None (an interim request with no preferred language): the model's own tag under the full context, as
- the plain decode does. `self.last`: the tag chosen, its probabilities with and without the Words, and the
- margin to the threshold (the history and the Lab show it)."""
- mx = self.mx
- from mlx_qwen3_asr.audio import compute_features
- from mlx_qwen3_asr.chunking import split_audio_into_chunks
- from mlx_qwen3_asr.generate import GenerationConfig, detect_repetition, resolve_max_new_tokens
- from mlx_qwen3_asr.load_models import _ModelHolder
- from mlx_qwen3_asr.tokenizer import _TokenizerHolder, parse_asr_output
- model, _ = _ModelHolder.get(self.repo, dtype=mx.float16)
- tok = _TokenizerHolder.get(_ModelHolder.get_resolved_path(self.repo, dtype=mx.float16))
- eos = set(GenerationConfig().eos_token_ids)
- head = tok.encode("language")
- tag_ids = {name: tok.encode(" " + name)[0] for name in (prefer, "English") if name}
- tag_ctx = (getattr(self, "tag_context", "") or "").strip()
- def positions(a: int, b: int):
- p = mx.arange(a, b)[None, :]
- return mx.stack([p, p, p], axis=1)
- def prefill(af, budget: int, context: str):
- ids = tok.build_prompt_tokens(n_audio_tokens=af.shape[1], language=None, context=context) + head
- self._check()
- cache = model.create_cache(max_seq_len=len(ids) + budget + 8)
- logits = model.prefill(input_ids=mx.array([ids]), audio_features=af, position_ids=positions(0, len(ids)), cache=cache)
- return logits, cache, len(ids)
- def tag_probs(logits) -> tuple[float, float, bool]:
- lp = logits.reshape(-1).astype(mx.float32)
- lp = lp - mx.logsumexp(lp)
- top = int(mx.argmax(lp))
- return float(mx.exp(lp[tag_ids["English"]])), float(mx.exp(lp[tag_ids[prefer]])), top in tag_ids.values()
- def run(af, budget: int, tag: str | None, state=None) -> tuple[str, str | None, dict]:
- """The text under `tag` (None: the model's own tag), from the prompt with the full context."""
- logits, cache, pos = state or prefill(af, budget, ctx)
- info: dict = {}
- if prefer and state is None and ctx.strip() != tag_ctx:
- # what the Words alone would have tagged (only shown: the tag was picked without them)
- p_en, p_pref, _ = tag_probs(logits)
- info["p_en_words"] = round(p_en, 4)
- if tag is None: # the model's own tag (no speech, another language): decode on from it, as before
- suffix = [int(mx.argmax(logits.reshape(-1)))]
- else:
- suffix = tok.encode(f" {tag}<asr_text>")
- logits = model.step_many(input_ids=mx.array([suffix]), position_ids=positions(pos, pos + len(suffix)), cache=cache)
- pos += len(suffix)
- token = int(mx.argmax(logits[0, -1]))
- out = [token]
- for _ in range(1, budget):
- if token in eos or detect_repetition(out):
- break
- self._check()
- logits = model.step(input_ids=mx.array([[token]]), position_ids=positions(pos, pos + 1), cache=cache,
- validate_input_ids=False)
- pos += 1
- token = int(mx.argmax(logits.reshape(-1)))
- out.append(token)
- out = [t for t in out if t not in eos]
- raw = tok.decode((head + suffix + out) if tag is None else out)
- if tag is None:
- lang, text = parse_asr_output(raw)
- else:
- lang, text = tag, parse_asr_output(raw, user_language=tag)[1]
- return (text or "").strip(), lang, info
- def pick(af, budget: int):
- """The tag from the prompt without the Words → (tag or None, info, the prefill when it is the decode's own)."""
- state = prefill(af, budget, tag_ctx)
- p_en, p_pref, top_is_tag = tag_probs(state[0])
- info = {"p_en": round(p_en, 4), f"p_{lang_code(prefer) or prefer}": round(p_pref, 4),
- "margin": dictation_vocab.tag_margin(p_en, p_pref)}
- tag = dictation_vocab.choose_tag(p_en, p_pref, prefer) if top_is_tag else None
- return tag, info, (state if ctx.strip() == tag_ctx else None)
- texts, langs, last = [], [], {}
- for chunk, _off in split_audio_into_chunks(np.asarray(x, np.float32), sr=SR):
- self._check()
- mel, fl = compute_features(chunk)
- af, _ = model.audio_tower(mel.astype(mx.float16), fl)
- budget = resolve_max_new_tokens(None, audio_duration_sec=len(chunk) / SR)
- if prefer:
- tag, info, state = pick(af, budget)
- text, lang, more = run(af, budget, tag, state)
- info.update(more)
- else:
- (text, lang, info), tag = run(af, budget, None), None
- if prefer and lang == "English" and dictation_vocab.translated_glue(text, self.names):
- text2, _, _ = run(af, budget, prefer)
- if dictation_vocab.cjk_share(text2) > 0:
- text, lang = text2, prefer
- info["redone"] = True
- info["tag"] = lang
- last = info
- texts.append(text)
- langs.append(lang)
- self.last = last
- lang = next((l for l in langs if l), None)
- return ("" if lang in ("Chinese", "Japanese", "Cantonese") else " ").join(t for t in texts if t).strip(), lang_code(lang)
- class FunAsrMlx(MlxBackend):
- """Fun-ASR-Nano (Alibaba, 0.8B: SenseVoice's encoder + Qwen3-0.6B) on MLX (mlx-audio). On the founder's 95 takes
- 2.84 % errors against the 0.6B's 3.34 % (2.92 % through the app's path), the best local model on Chinese with
- English inside it (ASCEND 6.6 %), and ~0.1 s after key-up (docs/design/dictation-eval-2026-10-31.md §2.1, §3, §5;
- docs/design/dictation.md §Defaults by Mac). Chinese, English and Japanese only
- (asr_models.json `languages`). The Words go in as hotwords; it takes no other context (no text so far, no field
- text: it wasn't measured with any). One whole utterance per call, ≤ 30 s (max_chunk). Interruptible: the decode is
- our loop over mlx-audio's tokens, so an interim request stops between tokens."""
- stop_check = None
- interruptible = True
- whole_spans = True # never VAD-trimmed (transcribe_track: pause_spans)
- max_terms = dictation_vocab.QWEN_MAX_TERMS # its hotword list goes into the prompt like Qwen3-ASR's context
- LANGS = {"zh": "zh", "yue": "zh", "en": "en", "ja": "ja"} # what its prompt can name; anything else: automatic
- EOS = (151643, 151645)
- def _check(self):
- if self.stop_check is not None and self.stop_check():
- raise Interrupted("another request is waiting")
- def load(self, args):
- try:
- import mlx.core as mx
- from mlx_audio.stt import load as load_stt
- except ImportError:
- raise Fail("mlx-audio not installed (uv run --with mlx-audio …)", "deps", 4)
- self.mx = mx
- self.repo = self.model = args.model or "mlx-community/Fun-ASR-Nano-2512"
- self.net = load_stt(self.repo)
- self.language = self.LANGS.get((args.language or "").lower())
- self.hotwords: list[str] = []
- self.last: dict = {}
- def release(self):
- self.net = None
- super().release()
- def set_vocab(self, terms: list[dict]) -> bool:
- self.hotwords = [t["term"] for t in terms[:self.max_terms] if str(t.get("term") or "").strip()]
- return bool(self.hotwords)
- def decode(self, x):
- self.last = {}
- text = self._decode(x, self.hotwords)
- # the hotword list written out as the answer (a noisy piece it couldn't make out): once more without it
- if self.hotwords and dictation_vocab.context_dump(text, self.hotwords):
- text = self._decode(x, [])
- self.last["context_dump"] = True
- return [(0.0, len(x) / SR, text, self._lang(text))]
- def _decode(self, x, hotwords: list[str]) -> str:
- mx, net = self.mx, self.net
- from mlx_audio.lm.generate import generate_step
- self._check()
- ids, embeds = net._build_inputs_embeds(mx.array(np.asarray(x, np.float32)), language=self.language,
- hotwords=hotwords or None, itn=True)
- # a loop, if it ever starts, stops well before the model's 512: speech is under ~8 tokens a second
- budget = min(512, 32 + int(20 * len(x) / SR))
- out: list[int] = []
- for token, _ in generate_step(prompt=ids[0], input_embeddings=embeds[0], model=net, max_tokens=budget):
- t = int(token)
- if t in self.EOS:
- break
- out.append(t)
- self._check()
- text = net._tokenizer.decode(out, skip_special_tokens=True)
- mx.clear_cache()
- return cjk_punctuation(re.sub(r"\s+", " ", text.replace("/sil", " ")).strip())
- @staticmethod
- def _lang(text: str) -> str | None:
- """It names no language: kana → ja, any Chinese character → zh, Latin letters only → en."""
- if not text:
- return None
- if _KANA.search(text):
- return "ja"
- return "zh" if re.search(r"[㐀-鿿]", text) else "en" if re.search(r"[A-Za-z]", text) else None
- class FasterWhisper:
- max_chunk = 30
- def __init__(self, args):
- try:
- from faster_whisper import WhisperModel
- except ImportError:
- raise Fail("faster-whisper not installed (uv run --with faster-whisper …)", "deps", 4)
- self.args = args
- name = args.model or "mobiuslabsgmbh/faster-whisper-large-v3-turbo"
- if name == "large-v3-turbo":
- name = "mobiuslabsgmbh/faster-whisper-large-v3-turbo"
- m = entry_for("faster-whisper", name)
- local = model_path(m) if m else None
- self.model = name
- self.net = WhisperModel(str(local) if local else name, device="auto", compute_type="auto", cpu_threads=_threads(args))
- def transcribe(self, x):
- segs, info = self.net.transcribe(x, language=self.args.language, initial_prompt=self.args.prompt,
- condition_on_previous_text=False, vad_filter=False)
- return [(s.start, s.end, cjk_punctuation(s.text.strip()), info.language) for s in segs]
- class AppleSpeech:
- """macOS's recogniser through the Blurt executable: the whole audio at once (it finds its own pauses and gives
- each phrase its time), so no VAD — nothing for a first run to download."""
- max_chunk = 3600
- no_vad = True
- def __init__(self, args):
- self.helper = apple_speech_helper()
- if not self.helper or not apple_speech_ok():
- raise Fail("macOS speech recognition needs macOS 26 and the Blurt app", "model_missing")
- self.args = args
- self.terms: list[str] = []
- self.model = "SpeechTranscriber"
- def set_vocab(self, terms: list[dict]) -> bool:
- self.terms = [t["term"] for t in terms if t.get("term")]
- return bool(self.terms)
- def transcribe(self, x):
- loc = apple_locale(self.args.language)
- if not loc:
- raise Fail("macOS speech recognition doesn't speak this language", "unsupported_language")
- with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
- f.write(wav_bytes(x))
- path = f.name
- try:
- cmd = [str(self.helper), "--speech", path, "--locale", loc]
- if self.terms:
- cmd += ["--words", "\n".join(self.terms)]
- r = subprocess.run(cmd, capture_output=True, text=True, timeout=120 + len(x) / SR)
- finally:
- os.unlink(path)
- line = next((ln for ln in reversed(r.stdout.splitlines()) if ln.startswith("{")), "{}")
- data = json.loads(line)
- if not data.get("ok"):
- raise Fail(data.get("error") or "macOS speech recognition failed", data.get("code") or "error")
- lang = loc.split("-")[0]
- return [(s["start"], s["end"], s["text"], lang) for s in data.get("segments") or []]
- # Cloud recognition can hear the audio sped up (asr.cloud.speed, every provider; docs/design/first-minute-speech.md
- # §2.4): billing is per second of audio sent. Measured with DashScope qwen3-asr-flash on 270 kept takes: 1x 6.8 % (full
- # cost), 1.5x 8.0 % for 64 % of the cost, 2x 11.6 % for 47 % — and twice it answered with the word list instead of the
- # speech. Since dictation sends the whole take (2026-10-31, docs/design/dictation.md §Cloud recognition) the default is
- # 1x: on the founder's 95 takes, whole with the words and the field's text, 2.02 % at 1x against 3.32 % at 1.5x.
- # Pitch is kept (ffmpeg atempo); the times that come back are put back on the original timeline.
- CLOUD_SPEEDS = (1.0, 1.5, 2.0)
- CLOUD_SPEED_DEFAULT = 1.0
- def cloud_speed(cfg: dict | None = None) -> float:
- v = os.environ.get("BLURT_ASR_SPEED") or ((cfg if cfg is not None else asr_config()).get("cloud") or {}).get("speed")
- try:
- v = float(v) if v is not None else CLOUD_SPEED_DEFAULT
- except (TypeError, ValueError):
- v = CLOUD_SPEED_DEFAULT
- return min(CLOUD_SPEEDS, key=lambda s: abs(s - v))
- def speed_up(x: np.ndarray, speed: float) -> np.ndarray:
- """x played `speed` times as fast, same pitch (ffmpeg atempo); x itself when 1x or ffmpeg isn't there."""
- if speed == 1.0 or not len(x) or not shutil.which("ffmpeg"):
- return x
- pcm = (np.clip(x, -1, 1) * 32767).astype(np.int16).tobytes()
- r = subprocess.run([shutil.which("ffmpeg"), "-nostdin", "-loglevel", "error", "-f", "s16le", "-ar", str(SR), "-ac", "1",
- "-i", "-", "-filter:a", f"atempo={speed}", "-f", "s16le", "-ar", str(SR), "-ac", "1", "-"],
- input=pcm, capture_output=True)
- if r.returncode or not r.stdout:
- return x
- return np.frombuffer(r.stdout, dtype=np.int16).astype(np.float32) / 32768
- def api_key(*names: str) -> str | None:
- """The first of these variables that is set, trimmed. A key with a character no API key has (2026-10-31: two
- "ç" — what ⌥C types — saved after a Model Studio key; the HTTP header can't carry them, and every take failed as
- a bare "error") is an "auth" failure that names the characters, never the key."""
- for n in names:
- k = (os.environ.get(n) or "").strip()
- if not k:
- continue
- bad = [c for c in dict.fromkeys(k) if not (c.isascii() and c.isprintable()) or c.isspace()]
- if bad:
- raise Fail(f"the API key in {n} has characters a key can't have "
- f"({', '.join(f'U+{ord(c):04X}' for c in bad)}): paste the key again", "auth")
- return k
- return None
- class OpenAICompat:
- """POST {base}/audio/transcriptions. verbose_json gives segment timestamps where supported."""
- retries, timeout = 4, 120.0
- def __init__(self, args):
- for name, default_base in (("BLURT_ASR_API_KEY", "https://api.openai.com/v1"),
- ("OPENAI_API_KEY", "https://api.openai.com/v1"),
- ("GROQ_API_KEY", "https://api.groq.com/openai/v1")):
- self.key = api_key(name)
- if self.key:
- break
- else:
- raise Fail("no API key: set BLURT_ASR_API_KEY (or OPENAI_API_KEY / GROQ_API_KEY)", "auth")
- self.base = (os.environ.get("BLURT_ASR_BASE_URL") or default_base).rstrip("/")
- self.model = args.model or os.environ.get("BLURT_ASR_MODEL") or (
- "whisper-large-v3-turbo" if "groq" in self.base else "whisper-1")
- self.verbose = not self.model.startswith("gpt-4o") # gpt-4o-*-transcribe has no segment timestamps
- # Longer chunks = fewer requests (Groq bills >=10s per request); shorter when no timestamps come back.
- self.max_chunk = 60 if self.verbose else 20
- self.args = args
- self.speed = cloud_speed()
- self.terms: list[str] = []
- def set_vocab(self, terms: list[dict]) -> bool:
- """The dictation's words, in the prompt (Whisper-style services read it as preceding text)."""
- self.terms = [t["term"] for t in terms if t.get("term")]
- return bool(self.terms)
- def transcribe(self, x):
- fields = {"model": self.model, "response_format": "verbose_json" if self.verbose else "json"}
- if self.args.language:
- fields["language"] = self.args.language
- prompt = self.args.prompt or (", ".join(self.terms) if self.terms else None)
- if prompt:
- fields["prompt"] = prompt
- boundary = uuid.uuid4().hex
- body = io.BytesIO()
- for k, v in fields.items():
- body.write(f'--{boundary}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
- body.write(f'--{boundary}\r\nContent-Disposition: form-data; name="file"; filename="a.wav"\r\n'
- f'Content-Type: audio/wav\r\n\r\n'.encode())
- body.write(wav_bytes(speed_up(x, self.speed)))
- body.write(f"\r\n--{boundary}--\r\n".encode())
- data = _post(f"{self.base}/audio/transcriptions", body.getvalue(),
- {"Authorization": f"Bearer {self.key}", "Content-Type": f"multipart/form-data; boundary={boundary}"},
- retries=self.retries, timeout=self.timeout)
- lang = lang_code(data.get("language"))
- if data.get("segments"):
- k = self.speed
- return [(s["start"] * k, min(s["end"] * k, len(x) / SR), s["text"].strip(), lang) for s in data["segments"]]
- return [(0.0, len(x) / SR, (data.get("text") or "").strip(), lang)]
- # Model Studio (阿里云百炼) speech models take one of two HTTP APIs, and Blurt learns which from the service's answers,
- # not from the id (measured 2026-11-11 with a spoken clip: docs/design/dictation.md §Cloud recognition). Qwen3-ASR Flash
- # takes the OpenAI-compatible chat/completions; qwen-audio 3.x ASR Flash and Fun-ASR Flash answer 404
- # model_not_supported or 400 there and work on Model Studio's own API. A model that only streams (…-realtime) or reads a
- # file from a URL (…-filetrans, Paraformer, Fun-ASR) gives text on neither: a `model` failure, with both answers.
- # The words as hot words: weight 1–5, or 50 — a "super" hot word, at most 50 of them. On the founder's 95 takes
- # (2026-11-11, qwen-audio-3.1-asr-flash, words + field) 4 or 5 changed almost nothing (Blurt 1 of 9 right, 1.89 %); 50
- # got Blurt 8 of 9 and 1.60 % (Qwen3-ASR Flash the same day: 2.15 %). The words past the 50th go at 5.
- DASHSCOPE_HOTWORD_WEIGHT = 50
- DASHSCOPE_SUPER_WORDS = 50
- # models known to take only the own API — measured, so the first request of a server doesn't go to the compatible API
- # for its 404 (2.5–31 s that day); any other model the server learns once (_ask)
- _DASHSCOPE_OWN_API: set[str] = {"qwen-audio-3.1-asr-flash"}
- def dashscope_native_url(base: str) -> str:
- """The OpenAI-compatible base (…/compatible-mode/v1, the one Settings keeps) → the same host's own API."""
- b = base.rstrip("/")
- if b.endswith("/compatible-mode/v1"):
- b = b[: -len("/compatible-mode/v1")]
- else:
- u = urllib.parse.urlsplit(b)
- b = f"{u.scheme}://{u.netloc}"
- return b + "/api/v1/services/aigc/multimodal-generation/generation"
- class DashScope:
- # A dictation goes here whole at key-up (docs/design/dictation.md §Cloud recognition): one request up to 50 s (the
- # eval's whole-take requests, dictation-eval-2026-10-31.md §2.1), a longer take cut at its pauses into ≤ 50 s parts
- # sent side by side (transcribe_track, CLOUD_WORKERS).
- max_chunk = 50
- retries, timeout = 4, 120.0
- def __init__(self, args):
- self.key = api_key("DASHSCOPE_API_KEY")
- if not self.key:
- raise Fail("no API key: set DASHSCOPE_API_KEY", "auth")
- # its own variable: BLURT_ASR_BASE_URL is the OpenAI-compatible service's (both can be set in the app)
- self.base = (os.environ.get("DASHSCOPE_BASE_URL") or "https://dashscope.aliyuncs.com/compatible-mode/v1").rstrip("/")
- # the model picked in Settings (CloudSpeech: DASHSCOPE_ASR_MODEL), else the registry's
- self.model = os.environ.get("DASHSCOPE_ASR_MODEL") or args.model or "qwen3-asr-flash"
- self.args = args
- self.speed = cloud_speed()
- self.terms: list[str] = []
- self.field = ""
- self.last: dict = {}
- def set_vocab(self, terms: list[dict]) -> bool:
- """The dictation's words as Qwen3-ASR's context (measured: 27 of 35 terms with them, §2.4)."""
- self.terms = [t["term"] for t in terms if t.get("term")]
- return bool(self.terms)
- def set_field(self, text: str) -> None:
- """The field's text before the cursor, after the words in the same context (Qwen3-ASR Flash reads background
- text there as well as a word list: Model Studio's "context enhancement"; docs/design/dictation.md §Field
- context)."""
- self.field = (text or "").strip()
- def context_text(self, words: bool = True, field: bool = True) -> str:
- w = (self.args.prompt or ", ".join(self.terms)) if words else ""
- return "\n".join(t for t in (w, getattr(self, "field", "") if field else "") if t)
- def transcribe(self, x):
- """The same nets as the MLX backend's decode (Qwen3Mlx.decode), for the same model family: the field's text
- before the cursor written out as the speech (13 of 95 of the founder's takes when the app sent short pieces
- with it: dictation-eval-2026-10-31.md §4.1) and the word list answered instead of the speech (3 of 95). An
- answer that repeats the field (`echoes_prior`) is asked once more with the words only, and the shorter answer is
- kept when it is clearly shorter; one that is the word list, once more with the field only. Costs a second
- request only when a net fires (`last`: echo / context_dump, the answer's `tag`; NoteEngine empties it per note,
- not per part: a long take's parts run side by side)."""
- field = getattr(self, "field", "")
- text = self._ask(x)
- if self.terms and not self.args.prompt and dictation_vocab.context_dump(text, self.terms):
- text = self._ask(x, words=False)
- self.last["context_dump"] = True
- if field and echoes_prior(text, field, len(x) / SR):
- redo = self._ask(x, field=False)
- # nothing heard without the field, and the answer was the field itself (≥ 80 % of it one run of the field):
- # there was no speech to write — a short piece of breath came back as the whole field (6 of the founder's
- # takes in the old piece path, re-measured 2026-10-31)
- nothing = not redo.strip() and longest_shared_run(text, field) >= 0.8 * len(_bare(text))
- if (redo.strip() or nothing) and len(redo) < 0.8 * len(text):
- text = redo
- self.last["echo"] = True
- return [(0.0, len(x) / SR, text)]
- def _ask(self, x, words: bool = True, field: bool = True) -> str:
- """The compatible API first; a refusal there (4xx) or a broken answer (5xx: the realtime models answer 500) is
- asked once more on the own API, which then serves this model from here on. Neither giving text: `model`."""
- model = getattr(self, "model", "")
- if model in _DASHSCOPE_OWN_API:
- return self._request_native(x, words, field)
- try:
- return self._request(x, self.context_text(words, field))
- except Fail as first:
- if first.code not in ("bad_response", "network") or not model:
- raise # a key or quota answer is the same on both APIs
- try:
- text = self._request_native(x, words, field)
- except Fail as second:
- if second.code == "bad_response":
- raise Fail(f"Model Studio gave no text for a recording from {model}: compatible API: {first}; "
- f"own API: {second}", "model")
- raise first if first.code == "network" else second
- _DASHSCOPE_OWN_API.add(model)
- return text
- def _request_native(self, x, words: bool, field: bool) -> str:
- """Model Studio's own HTTP API (qwen-audio-…-asr-flash takes no other). The words go as its
- hot words (`vocabulary`: DASHSCOPE_HOTWORD_WEIGHT) instead of context text; a typed prompt and the field's text go as one
- context turn, which the service keeps to 400 characters (the field's end kept: it is next to the cursor)."""
- audio = "data:audio/wav;base64," + base64.b64encode(wav_bytes(speed_up(x, self.speed))).decode()
- turn = "\n".join(t for t in (self.args.prompt or "", getattr(self, "field", "") if field else "") if t)[-400:]
- messages = [{"role": "user", "content": [{"type": "input_text", "text": turn}]}] if turn else []
- messages.append({"role": "user", "content": [{"type": "input_audio", "input_audio": {"data": audio}}]})
- params: dict = {"format": "wav", "sample_rate": str(SR)}
- if self.args.language:
- params["language_hints"] = [self.args.language]
- if words and self.terms and not self.args.prompt:
- params["vocabulary"] = {t: DASHSCOPE_HOTWORD_WEIGHT if i < DASHSCOPE_SUPER_WORDS else 5
- for i, t in enumerate(self.terms[:2000])}
- data = _post(dashscope_native_url(self.base), json.dumps({"model": self.model, "input": {"messages": messages},
- "parameters": params}).encode(),
- {"Authorization": f"Bearer {self.key}", "Content-Type": "application/json"},
- retries=self.retries, timeout=self.timeout)
- text = (data.get("output") or {}).get("text") if isinstance(data, dict) else None
- if text is None:
- raise Fail(f"ASR API answered without a transcript: {json.dumps(data)[:300]}", "bad_response")
- return text.strip()
- def _request(self, x, context: str) -> str:
- audio = "data:audio/wav;base64," + base64.b64encode(wav_bytes(speed_up(x, self.speed))).decode()
- payload = {"model": self.model, "stream": False,
- "messages": [{"role": "system", "content": [{"type": "text", "text": context}]},
- {"role": "user", "content": [{"type": "input_audio", "input_audio": {"data": audio}}]}],
- "asr_options": {"enable_itn": True, **({"language": self.args.language} if self.args.language else {})}}
- data = _post(f"{self.base}/chat/completions", json.dumps(payload).encode(),
- {"Authorization": f"Bearer {self.key}", "Content-Type": "application/json"},
- retries=self.retries, timeout=self.timeout)
- try:
- text = data["choices"][0]["message"]["content"]
- except (KeyError, IndexError, TypeError):
- raise Fail(f"ASR API answered without a transcript: {json.dumps(data)[:300]}", "bad_response")
- if isinstance(text, list):
- text = "".join(p.get("text", "") for p in text)
- return (text or "").strip()
- # ------------------------------------------------------------------ Volcengine 豆包 Seed-ASR 2.0 (docs/design/dictation.md §Cloud recognition)
- # The streamed `bigmodel_nostream` endpoint (dictation-eval-2026-10-31.md §2.1, §5, §6.3 B′): on the founder's 95 takes
- # 2.18 % with the words as hotwords and the field's text as a dialogue turn, ~0.45–0.52 s after key-up however long the
- # take — it hears the audio while it is said and re-recognises it in ~15 s blocks, so at key-up only the last block is
- # left. The app streams a take's microphone packets through the warm server (`serve` cmd "stream", VolcStream); Retry,
- # the Lab, Settings' Test and a voice note send a whole recording through the same session as fast as it goes (the
- # result doesn't depend on the pace: §2.1 note ⁴). Credentials are an App ID and an Access Token (VOLC_APP_ID,
- # VOLC_ACCESS_TOKEN: the app's Keychain item for the "volcengine-speech" service), resource volc.seedasr.sauc.duration.
- # The protocol (the docs' binary frames): a 4-byte header (version 1, header size 1; message type, flags; JSON / raw
- # serialisation, gzip), a signed sequence where the flags say so, a 4-byte size, the payload. The first frame is the
- # request (gzip JSON), then raw 16 kHz s16 mono PCM packets; the last packet's sequence is negative. The final result
- # comes back flagged last; an error is message type 0b1111 with a code and a message.
- VOLC_URL = "wss://openspeech.bytedance.com/api/v3/sauc/bigmodel_nostream"
- VOLC_RESOURCE = "volc.seedasr.sauc.duration"
- VOLC_PACKET = 6400 # 200 ms of 16 kHz s16 mono: the microphone's pace and the docs' recommended size
- # a language set by the user (asr.language) → the service's code; "zh" and anything else unlisted: the zh-en default
- VOLC_LANGS = {"en": "en-US", "ja": "ja-JP", "ko": "ko-KR", "de": "de-DE", "fr": "fr-FR", "es": "es-MX", "pt": "pt-BR",
- "id": "id-ID", "ms": "ms-MY", "th": "th-TH", "ar": "ar-SA", "fil": "fil-PH"}
- def volc_frame(kind: int, flags: int, serial: int, payload: bytes, seq: int | None = None, gz: bool = True) -> bytes:
- p = gzip.compress(payload) if gz else payload
- return (bytes([0x11, (kind << 4) | flags, (serial << 4) | (1 if gz else 0), 0])
- + (struct.pack(">i", seq) if seq is not None else b"") + struct.pack(">I", len(p)) + p)
- def volc_parse(b: bytes) -> tuple[dict, bool]:
- """One frame from the service → (its JSON, or {"error": (code, message)}; whether it is the last)."""
- hs = (b[0] & 0x0F) * 4
- kind, fl, comp = b[1] >> 4, b[1] & 0x0F, b[2] & 0x0F
- p = b[hs:]
- seq = None
- if fl & 1:
- seq = struct.unpack(">i", p[:4])[0]
- p = p[4:]
- if kind == 0b1111:
- code = struct.unpack(">I", p[:4])[0]
- n = struct.unpack(">I", p[4:8])[0]
- msg = p[8:8 + n]
- try:
- msg = gzip.decompress(msg) if comp == 1 else msg
- except OSError:
- pass
- return {"error": (code, msg.decode(errors="replace")[:300])}, True
- n = struct.unpack(">I", p[:4])[0]
- body = p[4:4 + n]
- if comp == 1 and body:
- body = gzip.decompress(body)
- return (json.loads(body) if body else {}), bool(fl & 2) or (seq is not None and seq < 0)
- def volc_fail(code: int, msg: str) -> Fail:
- """The service's error frame → a Fail the app words plainly. 45000292 "quota exceeded for types: concurrency" (the
- trial allows only a few sessions at once; the eval met it at 6) and 55000031 (busy) are "quota"; a token or app the
- service doesn't accept, "auth"; its own time-outs and server errors, "network"."""
- low = msg.lower()
- if code == 45000292 or code == 55000031 or "quota" in low or "concurrency" in low:
- return Fail(f"Volcengine refused ({code}): {msg}", "quota")
- if any(w in low for w in ("auth", "token", "permission", "grant", "app key", "appkey", "access key")):
- return Fail(f"Volcengine refused the App ID or Access Token ({code}): {msg}", "auth")
- if code == 45000081 or code >= 55000000:
- return Fail(f"Volcengine answered {code}: {msg}", "network")
- return Fail(f"Volcengine answered {code}: {msg}", "bad_response")
- class _WebSocket:
- """The least of a websocket client (RFC 6455) a recognition session needs — binary frames out (masked), frames in,
- ping / pong, close — on the standard library (transcribe.py's own dependencies stay as they are). The handshake's
- refusal is a Fail: 401 / 403 auth, 429 quota, anything else network or bad_response."""
- def __init__(self, url: str, headers: dict, timeout: float = 10):
- u = urllib.parse.urlsplit(url)
- host, port = u.hostname or "", u.port or (443 if u.scheme == "wss" else 80)
- try:
- raw = socket.create_connection((host, port), timeout=timeout)
- if u.scheme == "wss":
- import ssl
- raw = ssl.create_default_context().wrap_socket(raw, server_hostname=host)
- except (OSError, ValueError) as e:
- raise Fail(f"can't reach {host}: {e}", "network")
- self.sock = raw
- self.lock = threading.Lock()
- self.closed = False
- key = base64.b64encode(os.urandom(16)).decode()
- lines = [f"GET {u.path or '/'}{'?' + u.query if u.query else ''} HTTP/1.1", f"Host: {host}", "Upgrade: websocket",
- "Connection: Upgrade", f"Sec-WebSocket-Key: {key}", "Sec-WebSocket-Version: 13"]
- lines += [f"{k}: {v}" for k, v in headers.items()]
- try:
- self.sock.sendall(("\r\n".join(lines) + "\r\n\r\n").encode("latin-1"))
- head = b""
- while b"\r\n\r\n" not in head and len(head) < 65536:
- chunk = self.sock.recv(4096)
- if not chunk:
- break
- head += chunk
- except UnicodeEncodeError:
- self.close()
- raise Fail("the Access Token has characters a token can't have: paste it again", "auth")
- except OSError as e:
- self.close()
- raise Fail(f"{host} didn't answer: {e}", "network")
- top, _, rest = head.partition(b"\r\n\r\n")
- status_line, *hdr = top.decode("latin-1").split("\r\n")
- try:
- status = int(status_line.split()[1])
- except (IndexError, ValueError):
- self.close()
- raise Fail(f"{host} answered something that isn't HTTP: {status_line[:80]!r}", "network")
- h = {k.strip().lower(): v.strip() for k, _, v in (x.partition(":") for x in hdr)}
- if status != 101:
- msg = (h.get("x-api-message") or rest.decode(errors="replace"))[:300]
- self.close()
- code = "auth" if status in (401, 403) else "quota" if status == 429 else \
- "network" if status >= 500 else "bad_response"
- # a resource the app wasn't granted (the streaming model not enabled for this App ID) is the key's setup
- if status == 400 and any(w in msg.lower() for w in ("resource", "grant", "app key", "appid", "app id", "token")):
- code = "auth"
- raise Fail(f"Volcengine refused the session (HTTP {status}): {msg}", code)
- self.logid = h.get("x-tt-logid")
- self.buf = rest
- self.sock.settimeout(None)
- def _read(self, n: int) -> bytes:
- while len(self.buf) < n:
- chunk = self.sock.recv(max(65536, n - len(self.buf)))
- if not chunk:
- raise ConnectionError("closed")
- self.buf += chunk
- out, self.buf = self.buf[:n], self.buf[n:]
- return out
- def _send(self, opcode: int, payload: bytes) -> None:
- n = len(payload)
- head = bytes([0x80 | opcode])
- head += bytes([0x80 | n]) if n < 126 else bytes([0x80 | 126]) + struct.pack(">H", n) if n < 65536 \
- else bytes([0x80 | 127]) + struct.pack(">Q", n)
- mask = os.urandom(4)
- if n:
- m = int.from_bytes((mask * (n // 4 + 1))[:n], "big")
- payload = (int.from_bytes(payload, "big") ^ m).to_bytes(n, "big")
- with self.lock:
- self.sock.sendall(head + mask + payload)
- def send(self, payload: bytes) -> None:
- self._send(0x2, payload)
- def recv(self) -> bytes | None:
- """The next whole data message; None when the connection closed."""
- data = b""
- while True:
- b0, b1 = self._read(2)
- op, n = b0 & 0x0F, b1 & 0x7F
- if n == 126:
- n = struct.unpack(">H", self._read(2))[0]
- elif n == 127:
- n = struct.unpack(">Q", self._read(8))[0]
- mask = self._read(4) if b1 & 0x80 else b""
- p = self._read(n)
- if mask:
- p = bytes(c ^ mask[i % 4] for i, c in enumerate(p))
- if op == 0x8:
- return None
- if op == 0x9:
- self._send(0xA, p)
- continue
- if op == 0xA:
- continue
- data += p
- if b0 & 0x80:
- return data
- def close(self) -> None:
- if self.closed:
- return
- self.closed = True
- with contextlib.suppress(OSError):
- self._send(0x8, struct.pack(">H", 1000))
- with contextlib.suppress(OSError):
- self.sock.shutdown(socket.SHUT_RDWR)
- with contextlib.suppress(OSError):
- self.sock.close()
- class VolcStream:
- """One recognition session (one take): open → feed PCM as it comes (sent in 200 ms packets) → finish (the last
- packet, then the final text within `wait` s). Thread-safe between the feeder and the reader it starts."""
- def __init__(self, app_id: str, token: str, request: dict, timeout: float = 10):
- hdr = {"X-Api-App-Key": app_id, "X-Api-Access-Key": token, "X-Api-Resource-Id": VOLC_RESOURCE,
- "X-Api-Request-Id": str(uuid.uuid4()), "X-Api-Connect-Id": str(uuid.uuid4())}
- self.ws = _WebSocket(VOLC_URL, hdr, timeout)
- self.seq = 1
- self.pending = b""
- self.sent = 0 # bytes of PCM sent
- self.text: str | None = None
- self.utterances: list | None = None # with show_utterances: [{"text", "start_time", "end_time"} (ms)]
- self.error: Fail | None = None
- self.final = threading.Event()
- self.t_final: float | None = None
- self.t_last: float | None = None
- try:
- self.ws.send(volc_frame(0b0001, 0b0001, 1, json.dumps(request, ensure_ascii=False).encode(), self.seq))
- except OSError as e:
- self.ws.close()
- raise Fail(f"Volcengine dropped the session: {e}", "network")
- self.reader = threading.Thread(target=self._read, daemon=True)
- self.reader.start()
- def _read(self) -> None:
- try:
- while True:
- m = self.ws.recv()
- if m is None:
- break
- d, last = volc_parse(m)
- if "error" in d:
- code, msg = d["error"]
- # 45000002: no audio at all (an empty take) — nothing was said
- if code == 45000002:
- self.text = ""
- else:
- self.error = volc_fail(code, msg)
- break
- r = d.get("result") or {}
- if isinstance(r, list):
- r = r[0] if r else {}
- if r.get("text") is not None:
- self.text = r.get("text")
- if isinstance(r.get("utterances"), list):
- self.utterances = r["utterances"]
- if last:
- break
- except (OSError, ConnectionError, ValueError, struct.error) as e:
- if not self.ws.closed and self.error is None:
- self.error = Fail(f"Volcengine dropped the session: {e}", "network")
- finally:
- self.t_final = time.perf_counter()
- self.final.set()
- @property
- def failed(self) -> bool:
- return self.error is not None
- def feed(self, pcm: bytes) -> None:
- """PCM (16 kHz s16 mono) as it comes; whole 200 ms packets go out, the rest waits for more. A session the
- service already ended takes nothing more (finish says why)."""
- if self.final.is_set():
- return
- self.pending += pcm
- while len(self.pending) >= 2 * VOLC_PACKET: # one packet always held back: the last one is flagged
- self._packet(self.pending[:VOLC_PACKET], last=False)
- self.pending = self.pending[VOLC_PACKET:]
- def _packet(self, p: bytes, last: bool) -> None:
- self.seq += 1
- try:
- self.ws.send(volc_frame(0b0010, 0b0011 if last else 0b0001, 0, p, -self.seq if last else self.seq))
- self.sent += len(p)
- except OSError as e:
- if self.error is None and not self.final.is_set():
- self.error = Fail(f"Volcengine dropped the session: {e}", "network")
- self.final.set()
- def finish(self, wait: float = 10) -> str:
- """The rest of the audio as the last packet(s), then the final text — waited for at most `wait` s (then
- "network": the service didn't answer)."""
- try:
- if not self.final.is_set():
- rest = self.pending
- self.pending = b""
- while len(rest) > VOLC_PACKET:
- self._packet(rest[:VOLC_PACKET], last=False)
- rest = rest[VOLC_PACKET:]
- self._packet(rest, last=True)
- self.t_last = time.perf_counter()
- if not self.final.wait(wait):
- raise Fail(f"Volcengine gave no final text within {wait:.0f} s", "network")
- if self.error is not None:
- raise self.error
- if self.text is None:
- raise Fail("Volcengine closed the session without a final text", "network")
- return self.text.strip()
- finally:
- self.close()
- def close(self) -> None:
- self.ws.close()
- def volc_pcm(x: np.ndarray) -> bytes:
- return (np.clip(x, -1, 1) * 32767).astype("<i2").tobytes()
- class Volcengine:
- """Volcengine Seed-ASR 2.0 (`bigmodel_nostream`): the words as inline hotwords, the field's text before the cursor
- as one dialogue turn; no language set for a Chinese speaker (its zh-en default: the founder's takes 2.18 % against
- 2.84 % with automatic language), automatic language for everyone else (best measured on English: §3), or the
- language the user set. `transcribe` sends a whole recording through one session; `open` starts one a take streams
- into (serve "stream")."""
- max_chunk = 600 # one session per recording up to 10 min (a dictation stops at 5)
- max_terms = 100
- retries, timeout = 2, 60.0
- streams = True
- def __init__(self, args):
- self.app_id = api_key("VOLC_APP_ID")
- self.token = api_key("VOLC_ACCESS_TOKEN")
- if not self.app_id or not self.token:
- raise Fail("no Volcengine App ID and Access Token: set VOLC_APP_ID and VOLC_ACCESS_TOKEN", "auth")
- self.model = "seed-asr-2.0 (bigmodel_nostream)"
- self.args = args
- self.terms: list[str] = []
- self.field = ""
- self.last: dict = {}
- self.utterances = False # only my voice on a stream: each sentence's time, to drop another speaker's
- def set_vocab(self, terms: list[dict]) -> bool:
- self.terms = [str(t["term"]).strip() for t in terms if str(t.get("term") or "").strip()][:self.max_terms]
- return bool(self.terms)
- def set_field(self, text: str) -> None:
- self.field = (text or "").strip()
- def request(self) -> dict:
- req = {"model_name": "bigmodel", "enable_itn": True, "enable_punc": True, "enable_ddc": False,
- "show_utterances": self.utterances, "result_type": "full"}
- ctx: dict = {}
- if self.terms:
- ctx["hotwords"] = [{"word": w} for w in self.terms]
- if self.field:
- ctx["context_type"] = "dialog_ctx"
- ctx["context_data"] = [{"text": self.field}]
- if ctx:
- req["corpus"] = {"context": json.dumps(ctx, ensure_ascii=False)}
- audio = {"format": "pcm", "codec": "raw", "rate": SR, "bits": 16, "channel": 1}
- lang = (self.args.language or "").lower().split("-")[0] or None
- if lang in VOLC_LANGS:
- audio["language"] = VOLC_LANGS[lang]
- elif lang is None and not any(x.split("-")[0] in ("zh", "yue") for x in user_languages()):
- req["enable_auto_lang"] = True
- return {"user": {"uid": "blurt"}, "audio": audio, "request": req}
- def open(self, timeout: float = 10) -> VolcStream:
- return VolcStream(self.app_id, self.token, self.request(), timeout)
- def transcribe(self, x):
- pcm = volc_pcm(x)
- for attempt in range(self.retries):
- try:
- s = self.open(timeout=min(10, self.timeout))
- s.feed(pcm)
- # the whole recording at once: the service works through it at ~4× real time
- text = s.finish(wait=max(self.timeout, final_wait(len(x) / SR, 0)))
- return [(0.0, len(x) / SR, text)]
- except Fail as e:
- # a dropped connection is tried once more; a refusal (key, quota) is final
- if e.code != "network" or attempt == self.retries - 1:
- raise
- time.sleep(1)
- return [(0.0, len(x) / SR, "")]
- def _post(url: str, body: bytes, headers: dict, retries: int = 4, timeout: float = 120) -> dict:
- """One cloud request → its JSON. Every way it can fail is a Fail with a code the app words plainly (auth, quota,
- network, bad_response): an exception of any other kind came back from the speech server as a bare "error"."""
- for attempt in range(retries):
- try:
- req = urllib.request.Request(url, data=body, headers=headers, method="POST")
- with urllib.request.urlopen(req, timeout=timeout) as r:
- raw = r.read()
- try:
- return json.loads(raw)
- except ValueError:
- raise Fail(f"ASR API answered something that isn't JSON: {raw[:200]!r}", "bad_response")
- except urllib.error.HTTPError as e:
- msg = e.read().decode(errors="replace")[:500]
- if e.code in (429, 500, 502, 503) and attempt < retries - 1:
- time.sleep(2 ** attempt * 2)
- continue
- # 4xx: the service answered and refused the request (a model it doesn't serve this way, a wrong address):
- # not a connection problem (2026-11-11: qwen-audio-3.1-asr-flash on chat/completions read "can't connect")
- raise Fail(f"ASR API error {e.code}: {msg}", "auth" if e.code in (401, 403) else "quota" if e.code == 429
- else "network" if e.code >= 500 or e.code == 408 else "bad_response")
- except UnicodeEncodeError:
- # a header (the key) with a character HTTP can't carry: api_key catches it first; this is the safety net
- raise Fail("the API key has characters a key can't have: paste the key again", "auth")
- except (OSError, http.client.HTTPException) as e: # URLError, timeouts, TLS, a dropped connection
- if attempt < retries - 1:
- time.sleep(2 ** attempt)
- continue
- raise Fail(f"ASR API unreachable: {e}", "network")
- BACKENDS = {"sensevoice": SenseVoice, "qwen3-asr-onnx": Qwen3Onnx, "fun-asr-onnx": FunAsrOnnx, "mlx-whisper": MlxWhisper,
- "qwen3-asr-mlx": Qwen3Mlx, "fun-asr-mlx": FunAsrMlx, "faster-whisper": FasterWhisper, "apple": AppleSpeech,
- "openai": OpenAICompat, "dashscope": DashScope, "volcengine": Volcengine}
- LOCAL = {"sensevoice", "qwen3-asr-onnx", "fun-asr-onnx", "mlx-whisper", "qwen3-asr-mlx", "fun-asr-mlx", "faster-whisper", "apple"}
- EXTRA_PACKAGES = {"mlx-whisper": "mlx-whisper", "faster-whisper": "faster-whisper",
- "qwen3-asr-mlx": "mlx-qwen3-asr", "fun-asr-mlx": "mlx-audio"} # backend → `uv run --with` package
- # The exact version each `--with` installs (arch101: no release is picked up by itself; bumped on purpose, per Blurt
- # release, like the script header's). faster-whisper (NVIDIA, Windows / Linux) is pinned when it is next verified.
- # mlx-audio 0.6.3 pins mlx 0.32.4, the version the other two resolve to as well: one server holds all three
- # (serve_packages).
- PACKAGE_PINS = {"mlx-whisper": "mlx-whisper==0.4.3", "mlx-qwen3-asr": "mlx-qwen3-asr==0.4.4",
- "mlx-audio": "mlx-audio==0.6.3"}
- API_KEYS = {"openai": ("BLURT_ASR_API_KEY", "OPENAI_API_KEY", "GROQ_API_KEY"), "dashscope": ("DASHSCOPE_API_KEY",),
- "volcengine": ("VOLC_APP_ID", "VOLC_ACCESS_TOKEN")}
- KEYS_ALL = {"volcengine"} # needs every variable of its API_KEYS (an App ID and an Access Token), not any one
- CLOUD_WORKERS = 4 # a cloud service's parts of one take in flight at once (NoteEngine.note)
- # ------------------------------------------------------------------ recommendation
- # The rules come from docs/design/asr-benchmark.md (2026-10-22, Apple M5 Max): Whisper large-v3-turbo on MLX was the
- # most accurate in all seven test languages and the fastest multilingual model; its 4-bit build keeps nearly all of
- # that in 2.6 GB; SenseVoice is the fastest and lightest for zh / en / ja / ko / yue; on a CPU without a usable GPU,
- # Qwen3-ASR 0.6B (sherpa-onnx) was 9x faster than Whisper (CTranslate2) at a usable accuracy.
- SV, TURBO, TURBO_Q4, CT2, QWEN_CPU = ("sensevoice-small", "whisper-large-v3-turbo-mlx", "whisper-large-v3-turbo-q4-mlx",
- "whisper-large-v3-turbo-ct2", "qwen3-asr-0.6b-onnx")
- def hw_kind(hw: dict) -> str:
- """apple (Apple Silicon: MLX on the GPU) | nvidia (CUDA) | cpu"""
- if hw["os"] == "darwin" and hw["arch"] == "arm64":
- return "apple"
- return "nvidia" if hw["gpu"].get("kind") == "nvidia" else "cpu"
- QWEN_FIRST_LANGS = ("zh", "en") # where Qwen3-ASR measured best on real speech; Cantonese (36–39 % on the `say`
- # samples), Japanese, Korean and French keep the table's choice
- def preference(kind: str, tier: str, purpose: str, sv_ok: bool, sv_good: bool, qwen_ok: bool = False) -> list[str]:
- """Models to try, best first (the first that fits and speaks every language wins). sv_ok: SenseVoice speaks
- every language; sv_good: and its reference error rate is ≤ 5 % in each (not Korean: 10 % vs Whisper's 0.5 %);
- qwen_ok: every language is one of QWEN_FIRST_LANGS."""
- multi = {"apple": {"pro": [TURBO, TURBO_Q4], "standard": [TURBO, TURBO_Q4], "light": [TURBO_Q4], "minimal": []},
- "nvidia": {"pro": [CT2, QWEN_CPU], "standard": [CT2, QWEN_CPU], "light": [QWEN_CPU], "minimal": []},
- "cpu": {"pro": [QWEN_CPU], "standard": [QWEN_CPU], "light": [QWEN_CPU], "minimal": []}}[kind][tier]
- # Apple Silicon with 16 GB or more: one model for every job — dictation, voice notes, live captions, recordings
- # (v1.0.2). Qwen3-ASR on MLX was the most accurate on real speech in every set measured (AISHELL-1 NE terms, ASCEND
- # zh-en, FLEURS zh, LibriSpeech, contextual earnings calls: docs/design/asr-benchmark.md §Real speech) and the only
- # one that takes the user's words; one model also means one warm copy instead of three (the 24 GB Mac that
- # swapped held a Qwen, a SenseVoice and a Whisper).
- # v1.0.6: the 0.6B on pro Macs too. On the founder's real takes the two are close — the 0.6B ahead on the Lab's
- # corrected takes (4.6 vs 6.4 %), the 1.7B ahead on English (2.7 vs 3.2 %) and custom words (28 vs 25 of 35) —
- # while the 1.7B is 2.5x the download and holds 1 GB more (docs/design/first-minute-speech.md §2.1). The 1.7B
- # stays a choice, and a Mac that already has it keeps using it.
- if kind == "apple" and tier in ("pro", "standard") and qwen_ok:
- return [QWEN_MLX] + ([QWEN_MLX_17] if tier == "pro" else []) + multi + ([SV] if sv_ok else [])
- # on a CPU the multilingual model is 10x slower and no more accurate in SenseVoice's languages
- prefer_sv = sv_ok if kind == "cpu" else sv_good
- fallback = [SV] if sv_ok else []
- if purpose == "batch" or (purpose == "live" and kind == "apple" and tier == "pro"):
- # recordings: accuracy first. Live on a pro Mac runs the recordings model itself, so the live transcript is
- # the final one (transcribe.py run reuses it).
- if prefer_sv and (kind == "cpu" or tier in ("light", "minimal")):
- return [SV] + multi
- return multi + fallback
- # live captions and voice notes: latency first. Their model stays warm next to dictation's: on a 16–24 GB Mac the
- # 4-bit Whisper (2.6 GB, a 442 MB download) comes before the full one (3.7 GB) — arch098; recordings above keep
- # accuracy first (a one-off process that ends with the job)
- if kind == "apple" and tier == "standard":
- multi = [TURBO_Q4, TURBO]
- return ([SV] if prefer_sv else []) + multi + fallback
- def _bench_results() -> dict:
- d = load_json(BENCH_FILE, {}) or {}
- return d.get("results", {}) if isinstance(d, dict) else {}
- def _fits(m: dict, hw: dict, lim: dict | None = None) -> bool:
- """Runs on this platform and its measured peak memory stays within 40 % of RAM (the rest is the user's)."""
- if not runs_here(m, f"{hw['os']}-{hw['arch']}"):
- return False
- if "cuda" in (m.get("requires") or []) and hw["gpu"].get("kind") != "nvidia":
- return False
- return m.get("ram_mb", 0) <= max(hw["ram_gb"] * 1024 * 0.4, 700)
- def candidate_row(m: dict, hw: dict, lim: dict, bench: dict, langs: list[str]) -> dict:
- b = bench.get(m["id"]) or {}
- ref = m.get("rtf_ref")
- scale = {"pro": 1, "standard": 1.8, "light": 3.5, "minimal": 6}[hw["tier"]]
- row = {"id": m["id"], "name": m.get("name"), "backend": m["backend"], "languages": m["languages"],
- "size_mb": m["size_mb"], "ram_mb": b.get("peak_ram_mb") or m.get("ram_mb"),
- "rtf_est": b.get("rtf") if b.get("rtf") is not None else (round(ref * scale, 3) if ref else None),
- "measured": bool(b.get("rtf") is not None), "quality_note_key": m.get("quality_note_key"),
- "installed": installed(m), "fits": _fits(m, hw, lim), "speaks": [x for x in langs if speaks(m, x)],
- "local": m.get("local", True), "verified": m.get("verified")}
- if b.get("error"):
- row["error_rate"] = b["error"]
- elif m.get("error_ref"):
- row["error_rate_ref"] = {k: v for k, v in m["error_ref"].items() if k in langs}
- return row
- def recommend(languages: list[str] | None = None, purpose: str = "batch", hw: dict | None = None,
- cfg: dict | None = None) -> dict:
- """K9 + docs/design/asr-benchmark.md (preference() has the order):
- - recordings: Apple Silicon pro / standard → Whisper large-v3-turbo (MLX); light → its 4-bit build; NVIDIA → turbo
- on CUDA; CPU only → SenseVoice for zh / en / ja / ko / yue, Qwen3-ASR 0.6B (CPU) for anything else;
- - live captions and voice notes: SenseVoice when it speaks every language, else the tier's multilingual model;
- a pro Mac runs live captions with the recordings model;
- - on battery (asr.on_battery = lighter, the default) recordings drop one tier;
- - minimal tier with a language SenseVoice doesn't speak → cloud_recommended (SenseVoice is still the local pick)."""
- hw = hw or probe()
- cfg = asr_config() if cfg is None else cfg
- lim = limits(hw, cfg)
- langs = languages or user_languages(cfg)
- tier = hw["tier"]
- if hw.get("on_battery") and purpose == "batch" and cfg.get("on_battery", "lighter") == "lighter":
- tier = {"pro": "standard", "standard": "light"}.get(tier, tier)
- sv = entry(SV)
- sv_ok = all(speaks(sv, x) for x in langs)
- sv_good = sv_ok and all((sv.get("error_ref") or {}).get(x, 0) <= 0.05 for x in langs)
- qwen_ok = all(x in QWEN_FIRST_LANGS for x in langs)
- order = preference(hw_kind(hw), tier, purpose, sv_ok, sv_good, qwen_ok)
- bench = _bench_results()
- # v1.0.7: where dictation runs on the 1.7B (dictation_tier: 16 GB+ and fast enough), every job does — one model,
- # one warm copy (v1.0.2), and Settings never recommends the 0.6B next to a 1.7B that is already in use. By the
- # Mac's own tier: a recording on battery doesn't trade the model dictation already holds for a smaller one.
- if hw_kind(hw) == "apple" and qwen_ok and hw["tier"] in ("pro", "standard"):
- plan = dictation_tier(hw["ram_gb"], (bench.get(QWEN_MLX_17) or {}).get("rtf"), slow_chip(hw.get("chip")),
- languages=langs)
- if plan["recognizer"] == QWEN_MLX_17:
- order = [QWEN_MLX_17] + order
- pick = None
- for mid in dict.fromkeys(order):
- m = entry(mid)
- if m and _fits(m, hw, lim) and all(speaks(m, x) for x in langs):
- pick = m
- break
- cloud = tier == "minimal" and not sv_ok
- if pick is None:
- pick = sv # nothing local covers it: SenseVoice is still the fastest thing that runs
- rows = [candidate_row(m, hw, lim, bench, langs) for m in entries(local_only=False)
- if m.get("local", True) or not local_only()]
- rank = {mid: i for i, mid in enumerate(dict.fromkeys(order + [m["id"] for m in entries()]))}
- rows.sort(key=lambda r: (not r["fits"], len(r["speaks"]) < len(langs), rank.get(r["id"], 99)))
- out = {"ok": True, "tier": hw["tier"], "effective_tier": tier, "purpose": purpose, "languages": langs,
- "pick": {"model": pick["id"], "backend": pick["backend"]}, "candidates": rows}
- if cloud:
- out["cloud_recommended"] = True
- return out
- # ------------------------------------------------------------------ which model to run
- def select(purpose: str, a=None) -> dict:
- """The registry entry (or a synthetic one for a legacy backend / model) this run uses."""
- cfg = asr_config()
- if a is not None and getattr(a, "model_id", None):
- m = entry(a.model_id)
- if not m:
- raise Fail(f"unknown model {a.model_id}; see transcribe.py models", "unknown_model")
- return m
- if a is not None and getattr(a, "backend", None):
- name = a.backend
- if name not in BACKENDS:
- raise Fail(f"unknown backend {name}; choose from {list(BACKENDS)}", "unknown_model")
- model = getattr(a, "model", None) or (cfg.get("model") if name == cfg.get("backend") else None)
- return entry_for(name, model) or {"id": f"{name}:{model or 'default'}", "backend": name, "model": model,
- "languages": "multi", "local": name in LOCAL}
- chosen = (cfg.get("models") or {}).get(purpose) if isinstance(cfg.get("models"), dict) else None
- if not is_auto(cfg):
- if chosen and entry(chosen):
- return entry(chosen)
- name = cfg.get("backend") or "sensevoice"
- model = cfg.get("model")
- return entry_for(name, model) or {"id": f"{name}:{model or 'default'}", "backend": name, "model": model,
- "languages": "multi", "local": name in LOCAL}
- rec = recommend(purpose=purpose, cfg=cfg)
- order = ([chosen] if chosen else []) + [rec["pick"]["model"]] + [c["id"] for c in rec["candidates"]
- if c["fits"] and len(c["speaks"]) == len(rec["languages"])]
- apple = entry(APPLE)
- apple_ok = bool(apple and runs_here(apple) and installed(apple) and all(speaks(apple, x) for x in rec["languages"]))
- # a local model hears through the VAD (0.6 MB) too: one whose own files are already here from elsewhere (a Whisper
- # in a shared Hugging Face cache) doesn't beat macOS's recogniser, which needs nothing, until the VAD is in
- bundled_vad()
- vad = (MODELS_DIR / "silero_vad.onnx").is_file()
- for mid in dict.fromkeys(order):
- m = entry(mid)
- if m and m.get("local", True) and runs_here(m) and installed(m) and (vad or m.get("builtin") or not apple_ok):
- return m
- if m and not m.get("local", True) and not local_only() and installed(m) and mid == chosen:
- return m
- # nothing downloaded: macOS's own recogniser when it speaks the user's languages (first-minute-speech.md)
- if apple_ok:
- return apple
- # no macOS recogniser (macOS 13–15, ~23 % of Macs): a cloud service whose key the user set — the setup assistant's
- # one key (Model Studio in China) says it recognises speech too — before a model that would have to download
- if not local_only():
- for mid in DICTATION_CLOUD:
- m = entry(mid)
- if m and installed(m):
- return m
- return entry("sensevoice-small") # downloads on first use (never during a recording: BLURT_NO_DOWNLOAD)
- def resolve_backend(a, purpose: str = "batch") -> str:
- """Backend for this run; fills a.model / a.model_id / a.language / a.prompt (explicit options win)."""
- cfg = asr_config()
- m = select(purpose, a)
- a.model = None if m["backend"] == "sensevoice" else m.get("model")
- if m["backend"] in ("qwen3-asr-mlx", "mlx-whisper", "fun-asr-mlx"):
- # the folder it is in (HF cache snapshot or a ModelScope copy): loaded offline. Whisper got its repo id, and
- # mlx-whisper asked huggingface.co about it on every load (`Fetching 4 files`: +2 s here, longer where the
- # site is slow or blocked) before reading the files it already had
- p = model_path(m)
- if p:
- a.model = str(p)
- a.model_id = m["id"]
- lang = getattr(a, "language", None) or cfg.get("language")
- a.language = None if lang in (None, "", "auto") else lang
- a.prompt = getattr(a, "prompt", None)
- if not m.get("local", True) and local_only():
- raise Fail("local-only recognition is on (privacy.local_asr): cloud ASR is not allowed", "privacy")
- return m["backend"]
- def backend_ready(name: str, model_id: str | None = None) -> str | None:
- """None when `name` (or registry model `model_id`) can run right now without downloading anything or asking for
- a key, else why not."""
- if name not in BACKENDS:
- return f"unknown backend {name}"
- if name == "apple":
- return None if apple_speech_ok() else "macOS speech recognition needs macOS 26 and the Blurt app"
- bundled_vad()
- if not (MODELS_DIR / "silero_vad.onnx").exists():
- return "VAD model not downloaded (transcribe.py setup)"
- if name in API_KEYS:
- have = (all if name in KEYS_ALL else any)(os.environ.get(k) for k in API_KEYS[name])
- return None if have else \
- f"no API key in the environment ({' / '.join(API_KEYS[name])})"
- m = entry(model_id) if model_id else entry_for(name)
- if m and m.get("install") and not installed(m):
- return f"{m['id']} is not downloaded (transcribe.py setup --model {m['id']})"
- return None
- def throttle(purpose: str, threads: int | None = None) -> dict:
- """The process-wide caps for a CLI run (before any model or thread pool exists): thread pools and priority.
- In-process callers (tests, the app's helpers) don't call this, so importing transcribe never renices anyone."""
- lim = limits()
- cap_threads(threads or lim["threads"])
- return apply_priority(lim["priority"].get(purpose, "utility"))
- def make_backend(name: str, a, purpose: str = "batch"):
- """Load the backend with the thread cap, after making sure its package and files are there."""
- a.threads = getattr(a, "threads", None) or limits()["threads"]
- m = entry(getattr(a, "model_id", None)) or entry_for(name)
- if name in EXTRA_PACKAGES:
- ensure_package(EXTRA_PACKAGES[name])
- if m and m.get("install") and m.get("local", True) and not m["install"].get("files"):
- if not installed(m):
- if os.environ.get("BLURT_NO_DOWNLOAD"):
- raise Fail(f"{m['id']} is not downloaded", "model_missing")
- install_model(m)
- b = BACKENDS[name](a)
- if name in ("qwen3-asr-mlx", "fun-asr-mlx") and purpose in ("batch", "live"):
- b.max_chunk = 15 # a segment per chunk: a recording's timestamps (frames, items) stay this fine
- return b
- def model_label(name: str, backend, a) -> str | None:
- if name == "sensevoice":
- return "sense-voice-small-int8"
- if name == "apple":
- return "apple-speech-transcriber"
- m = getattr(backend, "model", None)
- return m if isinstance(m, str) else getattr(backend, "repo", None) or a.model
- # ------------------------------------------------------------------ shared pipeline (also used by live.py)
- def decode_slice(backend, x: np.ndarray, s: float, e: float) -> list[tuple]:
- """Transcribe x (= the samples of chunk s..e seconds) → segments on the original timeline:
- (start, end, text) or (start, end, text, language) when the backend says which language it heard. Chinese comes
- back in the user's script (to_user_script) whichever model wrote it: voice notes, recordings, live captions and
- dictation all pass here."""
- out = []
- for r in backend.transcribe(x):
- rs, re_, t, *lg = r
- seg = (s + rs, s + min(re_, e - s), to_user_script(t, lang_code(lg[0]) if lg and lg[0] else None))
- out.append(seg + (lg[0],) if lg and lg[0] else seg)
- return out
- def decode_chunk(backend, samples: np.ndarray, s: float, e: float) -> list[tuple]:
- return decode_slice(backend, samples[int(s * SR):int(e * SR)], s, e)
- def is_blank(results) -> bool:
- return not any(ch.isalnum() for r in results for ch in r[2])
- def build_transcript(source: str, name: str, backend, a, duration: float, speech: float, blank: float,
- results: list, elapsed: float) -> dict:
- """results: [(start, end, text, speaker|None[, language])] → the transcript.json document."""
- results = sorted(results, key=lambda r: r[0])
- segments = []
- for r in results:
- s, e, t, lb, *lg = r
- if t:
- segments.append({"start": round(s, 2), "end": round(e, 2), "text": t, **({"speaker": lb} if lb else {}),
- **({"language": lg[0]} if lg and lg[0] else {})})
- data = {"source": source, "backend": name, "model": model_label(name, backend, a),
- "duration": round(duration, 2), "speech_seconds": round(speech, 2),
- "elapsed_seconds": round(elapsed, 1), "segments": segments}
- langs = [sg["language"] for sg in segments if sg.get("language")]
- if langs:
- data["languages"] = sorted(set(langs), key=lambda x: -langs.count(x))
- # SenseVoice knows only zh/en/ja/ko/yue; other languages (German…) mostly come back as "." tagged <|en|>, so
- # the language tag can't catch it. Supported speech that VAD kept practically never decodes to nothing.
- # Mic track only: system audio may be music or effects.
- if name == "sensevoice" and speech >= 5 and blank / speech > 0.3:
- data["warning"] = (f"{blank:.0f}s of {speech:.0f}s of speech decoded to nothing: the language is probably "
- "not zh/en/ja/ko/yue, which is all SenseVoice supports. Re-run with a multilingual model "
- "(transcribe.py recommend; reference/asr.md) and pass --language.")
- return data
- def write_transcript(out: Path, data: dict) -> list[str]:
- save_json(out / "transcript.json", data)
- lines = [f"[{fmt_ts(s['start'])}-{fmt_ts(s['end'])}] {'(system) ' if s.get('speaker') == 'system' else ''}{s['text']}"
- for s in data["segments"]]
- (out / "transcript.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")
- return [str(out / "transcript.json"), str(out / "transcript.txt")]
- def print_summary(data: dict, files: list[str], **extra) -> None:
- print(json.dumps({k: v for k, v in data.items() if k != "segments"} | {"segments": len(data["segments"]),
- "files": files} | extra,
- ensure_ascii=False, indent=2))
- def transcribe_track(samples: np.ndarray, backend, name: str, workers: int, label: str | None = None,
- quiet: bool = False):
- """VAD → chunks → backend. Returns ([(start, end, text, label[, language])], speech s, s that decoded to nothing).
- A backend that finds its own pauses (`no_vad`: macOS's recogniser) gets the whole audio."""
- # …and so does a cloud service for audio it takes in one request (a dictation's or voice note's piece, which the
- # app has already cut at a pause): no VAD file needed for it either
- whole = getattr(backend, "no_vad", False) or (name in API_KEYS and len(samples) / SR <= backend.max_chunk)
- if whole:
- dur = len(samples) / SR
- r = decode_slice(backend, samples, 0.0, dur) if dur > 0 else []
- return [(s_, e_, t, label, *lg) for s_, e_, t, *lg in r], dur, dur if is_blank(r) else 0.0
- if getattr(backend, "whole_spans", False):
- # a model that hears utterances whole (Fun-ASR-Nano): a piece or window as it is, a longer one cut in pauses
- dur = len(samples) / SR
- # …but only with speech in it: a quiet room's hiss handed whole to an LLM-decoder model comes back as an invented
- # sentence (Windows dogfood 2026-11-02: −55 dBFS noise typed 我感觉我这个头发有点长。). The VAD says no speech →
- # nothing to decode, and the app says "No speech detected".
- # Only the ONNX build (Windows / Linux) is gated: the Mac's MLX model keeps its measured path, with no VAD pass
- if 0 < dur <= backend.max_chunk:
- gate = getattr(backend, "vad_gate", False)
- chunks = [(0.0, dur)] if not gate or vad_segments(samples, max_speech=min(25, backend.max_chunk)) else []
- else:
- chunks = pause_spans(samples, backend.max_chunk)
- else:
- segs = vad_segments(samples, max_speech=min(25, backend.max_chunk))
- chunks = group_chunks(segs, backend.max_chunk)
- sp = sum(e - s for s, e in chunks)
- if not quiet:
- print(f"audio {fmt_ts(len(samples) / SR)}{f' ({label})' if label else ''}, speech {fmt_ts(sp)} "
- f"in {len(chunks)} chunks → {name}", file=sys.stderr, flush=True)
- results, blank = [], 0.0
- with ThreadPoolExecutor(workers) as pool:
- decoded = pool.map(lambda ch: decode_chunk(backend, samples, *ch), chunks)
- for i, ((s, e), r) in enumerate(zip(chunks, decoded)):
- results.extend((s_, e_, t, label, *lg) for s_, e_, t, *lg in r)
- if is_blank(r):
- blank += e - s
- if (i + 1) % 20 == 0 and not quiet:
- print(f" {i + 1}/{len(chunks)} chunks", file=sys.stderr, flush=True)
- return results, sp, blank
- # ------------------------------------------------------------------ reuse live.py's transcript (docs/design/snap-and-live.md)
- def live_transcript(src: Path, out: Path, a, explicit: dict, wait: float = 180) -> dict | None:
- """The transcript live.py finished during this very recording (same session, same length), else None.
- If live.py is still draining after the stop, wait for it — its status.json stays fresh while it works."""
- if getattr(a, "force", False) or out.resolve() != src.parent.resolve():
- return None
- live = src.parent / "live"
- deadline = time.time() + wait
- while True:
- st = load_json(live / "status.json") or {}
- if st.get("asr") == "done":
- break
- fresh = time.time() - float(st.get("updated") or 0) < 15
- if st.get("asr") != "running" or not (live / "stop").exists() or not fresh or time.time() > deadline:
- return None
- time.sleep(0.5)
- try:
- data = load_json(out / "transcript.json")
- except (ValueError, OSError):
- return None
- if not isinstance(data, dict) or data.get("source") != src.name or not isinstance(data.get("segments"), list):
- return None
- if explicit.get("backend") and explicit["backend"] != data.get("backend"):
- return None # explicitly asked for another backend / model
- if explicit.get("model") and explicit["model"] != data.get("model"):
- return None
- if explicit.get("prompt") and data.get("backend") != "sensevoice":
- return None # a glossary helps Whisper / API backends (live.py has none): redo with it
- try:
- dur = probe_duration(src)
- except Exception:
- return None
- if abs(dur - float(data.get("duration") or 0)) > max(2.0, 0.02 * dur):
- return None # another take (restart) or a cut-short live run
- return data
- # ------------------------------------------------------------------ shots (snap mode): voice notes → "said"
- def read_shots(session: Path) -> dict:
- """shots.jsonl → {n: merged record}; later lines win."""
- latest: dict = {}
- log = session / "shots.jsonl"
- if log.exists():
- for line in log.read_text(encoding="utf-8").splitlines():
- try:
- rec = json.loads(line)
- except json.JSONDecodeError:
- continue
- if isinstance(rec, dict) and rec.get("n") is not None:
- latest[rec["n"]] = {**latest.get(rec["n"], {}), **rec}
- return latest
- def shot_voice(session: Path, n, rec: dict) -> Path | None:
- voice = rec.get("voice")
- if isinstance(voice, str) and voice:
- p = session / voice
- if not p.exists() and (session / "shots" / Path(voice).name).exists():
- p = session / "shots" / Path(voice).name
- return p if p.exists() else None
- if voice is False:
- return None
- try:
- p = session / "shots" / f"{int(n):03d}.wav"
- except (TypeError, ValueError):
- return None
- return p if p.exists() else None
- def structure_note(text: str) -> str | None:
- """A Structured voice note's text through the warm polish model (polish.py serve) — the same `structure` request
- the app sends for dictation and voice notes (model-based: lists laid out, self-corrections resolved). None when no
- polish server runs, the text is empty, or the model fell back."""
- if not text.strip():
- return None
- try:
- import polish
- timeout = polish.structure_timeout_s()
- r = polish._server_call({"cmd": "structure", "text": text, "timeout": timeout}, timeout + 5)
- except Exception: # noqa: BLE001 — no polish: the transcript stays the note
- return None
- if not isinstance(r, dict) or not r.get("ok") or r.get("fell_back"):
- return None
- out = (r.get("text") or "").strip()
- return out or None
- def transcribe_shots(session: Path, a) -> None:
- latest = read_shots(session)
- todo = [(n, rec, wav) for n, rec in latest.items()
- if (wav := shot_voice(session, n, rec)) and (a.force or "said" not in rec)]
- name = resolve_backend(a, "note")
- t0 = time.time()
- done = []
- if todo:
- with AsrSlot(limits()["max_concurrent"]):
- backend = make_backend(name, a, "batch")
- workers = 1 if name in LOCAL else a.concurrency
- for n, rec, wav in todo:
- results, _, _ = transcribe_track(load_audio(wav), backend, name, workers, quiet=True)
- said = " ".join(r[2] for r in sorted(results, key=lambda r: r[0]) if r[2]).strip()
- upd = {"said": said}
- if rec.get("structure"):
- # a Structured note (⇧ while it listened): its note is the structured text and the transcript stays
- # as said_raw, as the app keeps it. No polish model: the transcript, and said_raw cleared — a
- # stale one from an earlier pass would sit next to a note it no longer matches
- s = structure_note(said)
- upd = {"said": s, "said_raw": said} if s and s != said else {"said": said, "said_raw": None}
- # an update line with the same n (full record + "said"); later lines win — docs/design/snap-and-live.md
- with open(session / "shots.jsonl", "a", encoding="utf-8") as fh:
- fh.write(json.dumps({**rec, **upd}, ensure_ascii=False) + "\n")
- done.append({"n": n, **upd})
- print(json.dumps({"session": str(session), "mode": "shots", "shots": len(latest), "transcribed": len(done),
- "backend": name, "elapsed_seconds": round(time.time() - t0, 1), "said": done},
- ensure_ascii=False, indent=2))
- # ------------------------------------------------------------------ commands
- def cmd_run(a):
- src = Path(a.input)
- if not src.exists():
- die(f"{src} not found")
- if src.is_dir(): # a session dir: a shots session, or the recording inside it
- meta = load_json(src / "meta.json", {}) or {}
- recs = sorted(src.glob("recording.*"))
- if meta.get("mode") == "shots" or (not recs and meta.get("mode") != "video" and (src / "shots.jsonl").exists()):
- throttle("batch", a.threads)
- return transcribe_shots(src, a)
- if not recs:
- die(f"no recording.* in {src}")
- src = recs[0]
- out = Path(a.out or src.parent)
- explicit = {"backend": a.backend, "model": a.model, "prompt": a.prompt}
- data = live_transcript(src, out, a, explicit)
- n_tracks = max(1, audio_tracks(src))
- if data is not None and (n_tracks == 1 or any(s.get("speaker") == "system" for s in data["segments"])):
- files = [str(out / "transcript.json"), str(out / "transcript.txt")]
- if not (out / "transcript.txt").exists():
- files = write_transcript(out, data)
- print("transcript was already made during the recording (live/); --force redoes it", file=sys.stderr, flush=True)
- return print_summary(data, files, live=True)
- name = resolve_backend(a, "batch")
- throttle("batch", a.threads)
- with AsrSlot(limits()["max_concurrent"]):
- t0 = time.time()
- backend = make_backend(name, a, "batch")
- workers = 1 if name in LOCAL else a.concurrency
- # Blurt's native recorder writes the mic as track 0 and, with --system-audio, what the Mac played as track 1.
- # Transcribing them apart labels who spoke: the user ("mic") or the call / demo ("system").
- duration = speech = blank = 0.0
- results = []
- for track in range(min(n_tracks, 2)):
- label = ("mic", "system")[track] if n_tracks > 1 else None
- if track == 0 and data is not None: # live.py already did the mic; only the system track is left
- results += [(s["start"], s["end"], s["text"], label) for s in data["segments"]]
- duration, speech = float(data["duration"]), float(data["speech_seconds"])
- print("mic track: reusing the live transcript", file=sys.stderr, flush=True)
- continue
- samples = load_audio(src, track)
- duration = max(duration, len(samples) / SR)
- r, sp, bl = transcribe_track(samples, backend, name, workers, label)
- results += r
- if track == 0:
- speech, blank = sp, bl
- data = build_transcript(src.name, name, backend, a, duration, speech, blank, results, time.time() - t0)
- if data.get("warning"):
- print("warning: " + data["warning"], file=sys.stderr, flush=True)
- print_summary(data, write_transcript(out, data))
- # --- voice notes: the fast path and the warm server
- # A streamed dictation's (or voice note's) text so far goes to the speech model as context, so a name is spelled the
- # same way in every piece. Long, it gets echoed: the model answers with the earlier text instead of, or before, the
- # new piece (dict023, measured on a 43 s take: a 5.5 s last piece with 200 characters of prior came back as 78–165
- # characters, with 80 or 40 as the right 24, every run). So only the tail goes, and with it only the most relevant
- # words of the list; `echoes_prior` is the meaning-free net behind that.
- PRIOR_CHARS = 80
- PRIOR_TERMS = 32
- # The field's text before the cursor goes to a model that takes it (asr_models.json field_context_chars: Qwen3-ASR
- # 1.7B, the cloud Qwen3-ASR Flash) as context, ahead of the text so far: on 43 of the founder's takes the 1.7B went from
- # 3.04 to 2.63 % on whole takes (docs/design/dictation.md §Field context). The app sends at most this many.
- FIELD_CHARS = 300
- def field_context_chars(model_id: str | None) -> int:
- """How much of the field's text before the cursor this model hears as context (0: none — the 0.6B was mixed)."""
- m = entry(model_id) or {}
- n = m.get("field_context_chars")
- return min(FIELD_CHARS, int(n)) if isinstance(n, (int, float)) and n > 0 else 0
- # More than this many syllables a second is no one's speech (normal speech is 4–7 in every language measured).
- MAX_SYLLABLES_PER_SECOND = 10
- # A run this long (characters, spaces and punctuation aside) shared with the text so far is worth one more look.
- ECHO_RUN = 12
- def prior_tail(prior: str, n: int | None = None) -> str:
- """The last `n` (PRIOR_CHARS) characters of the text so far, not starting inside a Latin word."""
- n = n or PRIOR_CHARS
- p = (prior or "").strip()
- if len(p) <= n:
- return p
- tail = p[-n:]
- if tail[0].isascii() and tail[0].isalnum() and p[-n - 1].isascii() and p[-n - 1].isalnum() and " " in tail:
- tail = tail[tail.index(" ") + 1:]
- return tail.strip()
- def _bare(text: str) -> str:
- return "".join(c for c in unicodedata.normalize("NFKC", text or "").lower()
- if not unicodedata.category(c).startswith(("P", "Z", "C", "S")))
- def longest_shared_run(a: str, b: str) -> int:
- """The longest run of characters `a` and `b` share (spaces and punctuation aside)."""
- a, b = _bare(a), _bare(b)
- best, prev = 0, [0] * (len(b) + 1)
- for ca in a:
- cur = [0] * (len(b) + 1)
- for j, cb in enumerate(b, 1):
- if ca == cb:
- cur[j] = prev[j - 1] + 1
- best = max(best, cur[j])
- prev = cur
- return best
- def echoes_prior(text: str, prior: str, seconds: float) -> bool:
- """A piece's answer that may be the context written out again, judged without reading it: more syllables than
- `seconds` of speech can hold, or a long run of the prior in it. Only a reason to decode once more without the
- prior (the caller keeps the shorter answer only when it is clearly shorter)."""
- if not text or not prior:
- return False
- if seconds >= 1 and syllables(text) > MAX_SYLLABLES_PER_SECOND * seconds:
- return True
- return longest_shared_run(text, prior) >= ECHO_RUN
- def note_text(results: list) -> tuple[str, str | None]:
- results = sorted(results, key=lambda r: r[0])
- texts = [r[2] for r in results if r[2]]
- langs = [r[4] for r in results if len(r) > 4 and r[4]]
- sep = "" if langs and all(x in ("zh", "ja", "yue") for x in langs) else " "
- return sep.join(texts).strip(), (max(set(langs), key=langs.count) if langs else None)
- # --- voiceprint: only my voice (issue #26; docs/design/voiceprint.md) -----------------------------------------
- # A speaker-embedding gate in front of a dictation's recognition. With record.dictation.only_my_voice on (Settings →
- # Dictation) and a voice enrolled, each piece is cut where silero hears pauses (cuts ≤ 10 s, gaps ≤ 0.6 s), each cut is
- # embedded (3D-Speaker CAM++ zh-en, ~3.5 ms per second of audio on the CPU, in this process) and compared with the
- # user's voice (cosine with the mean of the enrolled and learned embeddings). A cut under VP_THRESHOLD is someone
- # else: silenced out of what recognition hears; a piece that is all someone else answers an empty text before the
- # model is even loaded. The verdict comes back as `speaker` (with `drop`, the silenced spans) so the app can mask the
- # same audio for whatever else hears it (a rehear window, macOS's second transcript). Interim requests (the live
- # words) are gated too, but only cuts long enough to judge (VP_INTERIM_MIN). Only dictation is gated: a voice note or
- # a recording is meant to keep everyone. Without an enrolled voice, or without the model, a piece goes through as it
- # is (fail-open). Measured on the founder's real takes against AISHELL / ASCEND / LibriSpeech / Common Voice speakers
- # (docs/design/voiceprint.md): the old ERes2Net + 0.5 dropped 17 % of the user's own speech; CAM++ at 0.28 with
- # learning drops none of it and lets ~0.6 % of other people's through.
- VP_FILE = "3dspeaker_speech_campplus_sv_zh_en_16k-common_advanced.onnx"
- VP_URLS = [
- f"https://github.com/k2-fsa/sherpa-onnx/releases/download/speaker-recongition-models/{VP_FILE}",
- f"https://huggingface.co/csukuangfj/speaker-embedding-models/resolve/main/{VP_FILE}",
- f"https://hf-mirror.com/csukuangfj/speaker-embedding-models/resolve/main/{VP_FILE}",
- ]
- VP_MB = 30 # both models' download
- # speaker turns inside a piece (pyannote segmentation 3.0, MIT, as sherpa-onnx publishes it; int8, 1.5 MB): in a
- # conversation the next person starts within a fraction of a second, too soon for a pause to cut at — on the founder's
- # own tests every reply of a friend sat in one cut with his own words and was typed with them
- VP_SEG_FILE = "pyannote-segmentation-3-0.int8.onnx"
- VP_SEG_URLS = [
- "https://huggingface.co/csukuangfj/sherpa-onnx-pyannote-segmentation-3-0/resolve/main/model.int8.onnx",
- "https://hf-mirror.com/csukuangfj/sherpa-onnx-pyannote-segmentation-3-0/resolve/main/model.int8.onnx",
- ]
- # under this cosine a cut is someone else (record.dictation.voiceprint_threshold overrides). The founder's own cuts:
- # 0.16 % dropped at 0.275 / 0.57 % at 0.30 with a 20 s enrollment, none once learning has a few takes; other people's
- # speech: the highest of 1,150 cuts scored 0.38
- VP_THRESHOLD = 0.28
- # learning: a cut of ≥ 3 s that scores ≥ 0.5 (far above anyone else measured) joins the profile, the latest 40 —
- # the user's other microphones, rooms and moods, which one enrollment take can't hold
- VP_LEARN = 0.5
- VP_LEARN_MIN = 3.0
- VP_LEARNED_MAX = 40
- VP_LEARN_EVERY = 20.0 # s between two learned cuts (one take doesn't fill the profile)
- # with two voices in a piece, the one far below the user's is someone else even above VP_THRESHOLD (a voice close to
- # the user's — the founder's friend scored 0.24–0.27 against his 0.68–0.84): dropped when it is VP_MARGIN below the
- # best, unless it scores VP_SURE or more
- VP_MARGIN = 0.3
- VP_SURE = 0.45
- VP_MARGIN_MIN = 1.5 # …a shorter stretch scores too unevenly for that (the founder's own 0.8 s "丫头": 0.36)
- VP_CLUSTER = 0.5 # sherpa-onnx's clustering threshold: turns closer than this are one speaker
- # an interim request judges only cuts with this much speech: a shorter one is too little to tell (the final piece
- # decides it)
- VP_INTERIM_MIN = 1.5
- def voiceprint_settings(flag: bool | None) -> tuple[bool, float]:
- """(on, threshold): the request's "voiceprint" wins; else record.dictation.only_my_voice / .voiceprint_threshold."""
- d = (global_config().get("record") or {}).get("dictation") or {}
- on = bool(d.get("only_my_voice", False)) if flag is None else bool(flag)
- try:
- threshold = float(d.get("voiceprint_threshold", VP_THRESHOLD))
- except (TypeError, ValueError):
- threshold = VP_THRESHOLD
- return on, threshold
- class Voiceprint:
- """The speaker-embedding extractor, loaded once per process; the user's voice in $BLURT_HOME/voiceprint.json:
- {"model", "embeddings" (enrollment windows, the latest 4 takes'), "learned" (cuts from dictation), "updated"}."""
- PROFILE = BLURT_HOME / "voiceprint.json"
- TAKES = 4
- _ext = None
- _lock = threading.Lock()
- _cache: dict = {} # sha1 of a cut's samples → its embedding (interim requests repeat the same cuts)
- _target: tuple | None = None # (profile mtime, target, the enrollment's own)
- _diar = None
- _learned_at = 0.0
- @classmethod
- def extractor(cls):
- with cls._lock:
- if cls._ext is None:
- import sherpa_onnx
- cfg = sherpa_onnx.SpeakerEmbeddingExtractorConfig(model=str(download(VP_FILE)), num_threads=2,
- provider="cpu")
- cls._ext = sherpa_onnx.SpeakerEmbeddingExtractor(cfg)
- return cls._ext
- @classmethod
- def diarizer(cls):
- """Speaker turns (pyannote segmentation + this same embedding model, clustered), loaded once."""
- with cls._lock:
- if cls._diar is None:
- import sherpa_onnx
- seg = sherpa_onnx.OfflineSpeakerSegmentationModelConfig(
- pyannote=sherpa_onnx.OfflineSpeakerSegmentationPyannoteModelConfig(model=str(download(VP_SEG_FILE))),
- num_threads=2)
- emb = sherpa_onnx.SpeakerEmbeddingExtractorConfig(model=str(download(VP_FILE)), num_threads=2)
- cls._diar = sherpa_onnx.OfflineSpeakerDiarization(sherpa_onnx.OfflineSpeakerDiarizationConfig(
- segmentation=seg, embedding=emb,
- clustering=sherpa_onnx.FastClusteringConfig(num_clusters=-1, threshold=VP_CLUSTER),
- min_duration_on=0.2, min_duration_off=0.3))
- return cls._diar
- @classmethod
- def speakers(cls, x: np.ndarray, t: np.ndarray) -> list[dict]:
- """Who spoke when: the audio in ≤ 10 s spans cut at pauses (segmentation's window; a longer take is judged
- span by span — each speaker is compared with the user, not with the others), each span's speaker turns, and
- each speaker's similarity to the user (their turns' audio together, up to 10 s). [{"turns": [(s, e)], "sim",
- "seconds"}]."""
- out = []
- for a, b in pause_spans(x, 10.0) or [(0.0, len(x) / SR)]:
- part = x[int(a * SR):int(b * SR)]
- spk: dict[int, list] = {}
- for s in cls.diarizer().process(part).sort_by_start_time():
- spk.setdefault(s.speaker, []).append((a + s.start, a + s.end))
- for turns in spk.values():
- audio = np.concatenate([x[int(s * SR):int(e * SR)] for s, e in turns])
- secs = len(audio) / SR
- sim = float(np.dot(cls.embed(audio[:10 * SR]), t)) if secs >= 0.5 else None
- out.append({"turns": turns, "sim": sim, "seconds": secs, "audio": audio})
- # speech the turns leave out (segmentation hears a quiet voice — a phone's video across the room — as
- # nothing, silero as speech): each stretch of it judged on its own, as one more speaker
- covered = [tr for s in out for tr in s["turns"]]
- for s, e in vad_segments(x):
- rest = [(s, e)]
- for a, b in covered:
- rest = [p for r in rest for p in ((r[0], min(r[1], a)), (max(r[0], b), r[1])) if p[1] - p[0] > 0.05]
- for a, b in rest:
- if b - a >= 0.5:
- audio = x[int(a * SR):int(b * SR)]
- out.append({"turns": [(a, b)], "sim": float(np.dot(cls.embed(audio[:10 * SR]), t)),
- "seconds": b - a, "audio": audio})
- return out
- @classmethod
- def embed(cls, x: np.ndarray) -> np.ndarray:
- key = hashlib.sha1(np.ascontiguousarray(x, dtype=np.float32).tobytes()).digest()
- if (e := cls._cache.get(key)) is not None:
- return e
- ext = cls.extractor()
- stream = ext.create_stream()
- stream.accept_waveform(SR, x)
- stream.input_finished()
- e = np.asarray(ext.compute(stream), dtype=np.float32)
- e = e / (np.linalg.norm(e) + 1e-9)
- if len(cls._cache) > 256:
- cls._cache.clear()
- cls._cache[key] = e
- return e
- @staticmethod
- def speech_seconds(x: np.ndarray) -> float:
- """Loud-frame seconds (20 ms frames, above −45 dBFS: speech is far louder, a quiet room far quieter). An
- enrollment of room tone is refused, never learned; an all-speech clip counts in full (no adaptive floor:
- one made of speech alone would call nothing loud)."""
- f = SR // 50
- n = len(x) // f
- if n == 0:
- return 0.0
- rms = np.sqrt((x[: n * f].reshape(n, f) ** 2).mean(axis=1))
- loud = 20 * np.log10(rms + 1e-9) > -45
- return float(loud.sum() * f / SR)
- @classmethod
- def embed_windows(cls, x: np.ndarray, win: float = 5.0) -> list[np.ndarray]:
- """One embedding per 5 s window that is mostly speech: a long enrollment is several looks at the voice."""
- w = int(win * SR)
- return [cls.embed(part) for off in range(0, len(x), w)
- if len(part := x[off:off + w]) >= SR and cls.speech_seconds(part) >= 0.4 * len(part) / SR]
- @classmethod
- def profile(cls) -> dict:
- """The profile when it was made with this model ({} without one; `stale` when another model made it: its
- embeddings can't be compared with this one's)."""
- p = load_json(cls.PROFILE, None)
- if not isinstance(p, dict) or not isinstance(p.get("embeddings"), list) or not p["embeddings"]:
- return {}
- if p.get("model") != VP_FILE:
- return {"stale": True}
- if p.get("version") != 2: # v1 learned whole cuts, a friend's reply inside the user's cut included
- p["learned"] = []
- return p
- @classmethod
- def target(cls) -> np.ndarray | None:
- """The user's voice: the mean of the enrolled and learned embeddings (re-read when the file changes — the CLI
- enrolls beside a running server)."""
- try:
- mtime = cls.PROFILE.stat().st_mtime_ns
- except OSError:
- cls._target = None
- return None
- if cls._target and cls._target[0] == mtime:
- return cls._target[1]
- p = cls.profile()
- if not p.get("embeddings"):
- cls._target = (mtime, None, None)
- return None
- def unit(v):
- e = np.asarray(v, dtype=np.float32).mean(axis=0)
- return e / (np.linalg.norm(e) + 1e-9)
- cls._target = (mtime, unit(p["embeddings"] + (p.get("learned") or [])), unit(p["embeddings"]))
- return cls._target[1]
- @classmethod
- def enrolled(cls) -> np.ndarray | None:
- """The enrollment take alone: what learning is measured against, so it can't drift toward whoever it once
- let through."""
- return cls._target[2] if cls.target() is not None and cls._target else None
- @classmethod
- def similarity(cls, x: np.ndarray) -> float | None:
- """Cosine of this audio against the user's voice; None without one."""
- t = cls.target()
- return None if t is None else float(np.dot(cls.embed(x), t))
- @classmethod
- def gate(cls, x: np.ndarray, threshold: float, interim: bool = False) -> tuple[dict, np.ndarray]:
- """The gate, per piece: who spoke when (`speakers`: speaker turns, not pauses — a reply right after the user
- is found too), each speaker compared with the user's voice, and another speaker's turns silenced out of what
- recognition hears (10 ms ramps: a hard zero is a click; where two talk at once, the user's turn wins). A
- speaker is someone else under `threshold`, or VP_MARGIN below the best speaker of the piece unless they score
- VP_SURE. ({"match", "similarity", "threshold", "dropped", "drop": [[s, e], …], "speakers"}, the audio to
- recognise). match false: nobody in it is the user — the caller answers blank without recognising. Fail
- "not_enrolled" / "model_missing": the caller lets the piece through. Interim requests judge only speakers
- with VP_INTERIM_MIN s or more."""
- t = cls.target()
- if t is None:
- raise Fail("no voice enrolled", "not_enrolled")
- if not vad_segments(x): # no speech heard: as ungated (recognition answers blank as ever)
- return {"match": True, "similarity": None, "threshold": threshold, "dropped": 0, "drop": []}, x
- spk = cls.speakers(x, t)
- judged = [s["sim"] for s in spk if s["sim"] is not None and (not interim or s["seconds"] >= VP_INTERIM_MIN)]
- best = max(judged, default=None)
- # a short stretch scores unevenly (a sub-second clip of anyone can land above the threshold): it is kept only
- # beside a long stretch that is surely the user — a piece of someone else alone can't slip through on it
- long_mine = any(s["sim"] is not None and s["seconds"] >= VP_MARGIN_MIN and s["sim"] >= threshold for s in spk)
- any_long = any(s["seconds"] >= VP_MARGIN_MIN for s in spk)
- for s in spk:
- v = s["sim"]
- short = s["seconds"] < VP_MARGIN_MIN
- s["keep"] = (interim and s["seconds"] < VP_INTERIM_MIN) or (v is None and (long_mine or not judged)) or (v is not None and
- v >= threshold and (v >= best - VP_MARGIN or v >= VP_SURE or short)
- and (not short or long_mine or not any_long))
- # a fragment too short to score (< 0.5 s) goes with the piece: kept beside the user, dropped beside others
- kept = [s for s in spk if s["keep"]]
- dropped = [s for s in spk if not s["keep"]]
- # learning: a piece of one voice only, long and surely the user's against the enrollment itself
- e0 = cls.enrolled()
- if not interim and len(spk) == 1 and spk[0]["seconds"] >= VP_LEARN_MIN and e0 is not None:
- e_ = cls.embed(spk[0]["audio"][:10 * SR])
- if float(np.dot(e_, e0)) >= VP_LEARN:
- cls.learn(e_)
- sims = [s["sim"] for s in kept if s["sim"] is not None]
- out = {"match": bool(kept) or not spk, "threshold": threshold, "dropped": len(dropped),
- "similarity": round(min(sims), 3) if sims else round(best, 3) if best is not None else None,
- "speakers": [round(s["sim"], 3) if s["sim"] is not None else None for s in spk],
- "drop": [[round(a, 3), round(b, 3)] for s in dropped for a, b in s["turns"]]}
- if not dropped or not kept:
- return out, x
- mask = np.ones(len(x), dtype=np.float32)
- r = int(0.01 * SR)
- for s in dropped:
- for a_, b_ in s["turns"]:
- a, b = int(a_ * SR), min(len(x), int(b_ * SR))
- mask[a:b] = 0
- n = min(r, a)
- mask[a - n:a] = np.minimum(mask[a - n:a], np.linspace(1, 0, n, dtype=np.float32))
- n = min(r, len(x) - b)
- mask[b:b + n] = np.minimum(mask[b:b + n], np.linspace(0, 1, n, dtype=np.float32))
- for s in kept: # two at once: the user's turn wins
- for a_, b_ in s["turns"]:
- mask[int(a_ * SR):int(b_ * SR)] = 1
- return out, x * mask
- @classmethod
- def learn(cls, e: np.ndarray) -> None:
- """A cut that is surely the user's joins the profile (at most one per VP_LEARN_EVERY s, the latest
- VP_LEARNED_MAX kept)."""
- now = time.time()
- if now - cls._learned_at < VP_LEARN_EVERY:
- return
- p = cls.profile()
- if not p.get("embeddings"):
- return
- cls._learned_at = now
- p["version"] = 2
- p["learned"] = ((p.get("learned") or []) + [[round(float(v), 5) for v in e]])[-VP_LEARNED_MAX:]
- p["updated"] = datetime.now().astimezone().isoformat(timespec="seconds")
- save_json(cls.PROFILE, p)
- @classmethod
- def enroll(cls, wav: Path) -> dict:
- """One enrollment take (Settings' "Only my voice": a passage read aloud): its windows join the profile (the
- latest TAKES takes; what dictation taught it stays). Refused when there is too little speech in it — room tone
- is never learned. A profile another model made is replaced."""
- t0 = time.perf_counter()
- if not wav.is_file():
- return {"ok": False, "code": "not_found", "error": f"{wav} not found"}
- x = read_wav(wav)
- es = cls.embed_windows(x)
- if len(es) < 2: # under ~8 s of speech
- return {"ok": False, "code": "too_short", "audio_seconds": round(len(x) / SR, 2),
- "error": "too little speech — read the passage aloud"}
- p = cls.profile()
- takes = [t for t in (p.get("takes") or []) if isinstance(t, list)][-(cls.TAKES - 1):] + [
- [[round(float(v), 5) for v in e] for e in es]]
- save_json(cls.PROFILE, {"model": VP_FILE, "version": 2, "takes": takes, "embeddings": [e for t in takes for e in t],
- "learned": p.get("learned") or [],
- "updated": datetime.now().astimezone().isoformat(timespec="seconds")})
- return {**cls.status(), "audio_seconds": round(len(x) / SR, 2), "ms": int((time.perf_counter() - t0) * 1000)}
- @classmethod
- def prepare(cls) -> dict:
- """The model on this Mac (downloaded when missing: GitHub, then Hugging Face and its mirror), then the status."""
- download(VP_FILE)
- download(VP_SEG_FILE)
- return cls.status()
- @classmethod
- def status(cls) -> dict:
- p = cls.profile()
- return {"ok": True, "enrolled": bool(p.get("embeddings")), "stale": bool(p.get("stale")),
- "takes": len(p.get("takes") or []), "learned": len(p.get("learned") or []), "model": VP_FILE,
- "model_ready": (MODELS_DIR / VP_FILE).is_file() and (MODELS_DIR / VP_SEG_FILE).is_file(), "model_mb": VP_MB, "updated": p.get("updated")}
- @classmethod
- def clear(cls) -> dict:
- cls.PROFILE.unlink(missing_ok=True)
- cls._target = None
- return cls.status()
- class NoteEngine:
- """The voice-note model, loaded once (in-process `note`, or kept warm by `serve`)."""
- def __init__(self):
- self.backend = self.name = self.model_id = None
- self.last = time.time()
- def load(self, language: str | None = None, model_id: str | None = None):
- a = argparse.Namespace(backend=None, model=None, model_id=model_id, language=language, prompt=None)
- name = resolve_backend(a, "note")
- if self.backend is None or a.model_id != self.model_id or (language or None) != getattr(self, "language", None):
- if self.backend is not None and a.model_id != self.model_id and hasattr(self.backend, "close"):
- self.backend.close() # its model goes before the next one loads (a language change keeps it)
- elif self.backend is not None and hasattr(self.backend, "_ex"):
- self.backend._ex.shutdown(wait=False) # the same model: only its thread goes
- self.backend = make_backend(name, a, "note")
- self.name, self.model_id, self.language = name, a.model_id, language or None
- return self
- def note(self, wav: Path, language: str | None = None, model_id: str | None = None, vocab=None,
- purpose: str | None = None, prior: str | None = None, hint: str | None = None, stop=None,
- app: str | None = None, before: str | None = None, voiceprint: bool | None = None) -> dict:
- """vocab: None = no words; True = the file's; a list = these (strings or term dicts). The model is biased toward
- them where it can be (Whisper's prompt, Qwen3-ASR's context / hotwords); they are hints, so `text` (and
- `asr_text`, the same) is what the model wrote. A misheard name is polish's to fix: it gets the words and sees
- the sentence.
- hint: the target app's window title and the text around the cursor — words seen there are listed to the model
- first.
- app: the name of the app the text goes into ("飞书"), named to the model before the words — what the speech is
- likely about where two words sound the same (飞书会话 vs 绘画: 4 → 8 of 12 synthetic takes right). Only for a
- backend that takes words (Qwen3-ASR's context); nothing else is said about the app.
- before: the field's text before the cursor (the app sends it only with "Use the text around the cursor" on, never
- from a password field). Heard as context only by a model whose registry entry has field_context_chars (its
- last that many characters, ahead of the text so far); never logged or answered back.
- stop: an interim request (the live words of dictation and of notes) — a callable that says another request is
- waiting; the decode then raises Interrupted. An interruptible backend looks at it between steps; a quick one
- (`quick_decode`: SenseVoice, one ~0.1 s pass) is asked once before and then decodes whole — a waiting final
- piece waits that long at most. Any other backend (a slow one that can't stop: Whisper on CT2, Qwen3-ASR on the
- CPU) answers Fail "interim_unsupported"."""
- t0 = time.perf_counter()
- language = None if language in (None, "", "auto") else language
- if purpose == "dictation" and not model_id:
- model_id = dictation_models()["selected"]
- x = read_wav(wav)
- speaker = None
- vp_on, vp_threshold = voiceprint_settings(voiceprint)
- if vp_on and purpose == "dictation":
- # only my voice (issue #26): someone else's speech is silenced out of what recognition hears, a piece that
- # is all someone else answers blank right here, before the model is even loaded. An interim request (stop
- # set) judges only cuts long enough to tell. No profile, no model, the gate failing: as if it were off.
- with no_download(True): # a dictation never waits for the voiceprint model either
- try:
- speaker, x = Voiceprint.gate(x, vp_threshold, interim=stop is not None)
- except (Fail, ImportError, RuntimeError, ValueError):
- speaker = None
- if speaker is not None and not speaker["match"]:
- return {"ok": True, "text": "", "language": language, "ms": int((time.perf_counter() - t0) * 1000),
- "model": model_id or self.model_id, "audio_seconds": round(len(x) / SR, 2), "load_ms": 0,
- "speaker": speaker}
- was = self.backend
- t_load = time.perf_counter()
- self.load(language, model_id)
- load_ms = int((time.perf_counter() - t_load) * 1000) if self.backend is not was else 0
- if stop is not None and not getattr(self.backend, "interruptible", False):
- if not getattr(self.backend, "quick_decode", False):
- raise Fail(f"{self.name} can't stop a decode halfway: no interim requests", "interim_unsupported")
- if stop():
- raise Interrupted("another request is waiting")
- stop = None # one short pass: nothing to look at halfway
- biased = bias(self.backend, vocab, prior, hint, app)
- if prior and hasattr(self.backend, "set_prior"):
- self.backend.set_prior(prior_tail(prior))
- field = ""
- if before and hasattr(self.backend, "set_field") and (n := field_context_chars(self.model_id)):
- field = prior_tail(before, n)
- self.backend.set_field(field)
- retried = False
- if stop is not None:
- self.backend.stop_check = stop
- # a cloud service hears a long take's parts side by side (a take at key-up: dictation-eval §5, Flash grows
- # from 0.5 to 1.5 s with the take when sent as one request); a local model has one decoder
- workers = CLOUD_WORKERS if self.name in API_KEYS else 1
- if self.name in API_KEYS:
- self.backend.last = {}
- if hasattr(self.backend, "looped"):
- self.backend.looped = False
- try:
- results, _speech, _blank = transcribe_track(x, self.backend, self.name, workers, quiet=True)
- # Qwen3-ASR now and then answers nothing for a short piece that follows a whole sentence of context (a
- # dictation's last words after a long pause: measured 2–5 of 6 takes of one synthetic clip, never when the
- # same request was replayed). The VAD heard speech, so once more without the context: ~0.13 s, only then.
- if (prior or field) and _speech >= 0.4 and not note_text(results)[0].strip() \
- and hasattr(self.backend, "set_prior"):
- self.backend.set_prior("")
- if field:
- self.backend.set_field("")
- results, _speech, _blank = transcribe_track(x, self.backend, self.name, workers, quiet=True)
- retried = True
- finally:
- if stop is not None:
- self.backend.stop_check = None
- if biased:
- self.backend.set_vocab([]) # a voice note after this gets no dictation words
- if prior and hasattr(self.backend, "set_prior"):
- self.backend.set_prior("")
- if field:
- self.backend.set_field("")
- text, lang = note_text(results)
- out = {"ok": True, "text": text, "language": lang or language, "ms": int((time.perf_counter() - t0) * 1000),
- "model": self.model_id, "audio_seconds": round(len(x) / SR, 2), "load_ms": load_ms}
- if speaker:
- out["speaker"] = speaker
- if retried:
- out["retried"] = True
- if getattr(self.backend, "looped", False):
- out["looped"] = True # a repetition loop was cut out of it: words may be missing (never sent)
- if field:
- out["field_context"] = len(field) # how much of it was heard (never the text)
- tag = getattr(self.backend, "last", None)
- if tag:
- out["tag"] = tag # the language tag chosen and the model's English / preferred probabilities
- if vocab is not None:
- out.update(asr_text=text, biased=biased)
- self.last = time.time()
- return out
- def bias(backend, vocab, prior: str | None = None, hint: str | None = None, app: str | None = None) -> bool:
- """The request's words (None: none; True: the file's; a list: these) given to a backend that takes them: the most
- relevant `max_terms` of them — ranked() cuts by relevance, then names them in a fixed order (a backend that took
- fewer than it was given would cut alphabetically: Whisper's 40 of 64, a third of the right words gone,
- contextual-earnings22) — with the app's name first. True when the backend was biased."""
- if not hasattr(backend, "set_vocab"):
- return False
- terms = dictation_vocab.load() if vocab is True else dictation_vocab.normalize(vocab) if vocab else []
- cap = getattr(backend, "max_terms", dictation_vocab.QWEN_MAX_TERMS)
- limit = min(PRIOR_TERMS, cap) if prior else cap
- use = dictation_vocab.ranked(terms, prior or "", hint or "", limit)
- # the app's name goes first — not to a model whose words are built into it (FunAsrOnnx: a new word list is a new
- # recognizer, ~3 s on a CPU; an app switch must not cost that)
- name = " ".join(str(app or "").split())[:40] if getattr(backend, "app_hint", True) else ""
- if name and not any(t["term"] == name for t in use):
- use = [{"term": name, "sounds_like": [], "source": "app", "hits": 0}] + use[:max(0, limit - 1)]
- return backend.set_vocab(use)
- # --- a take streamed while it is said (serve "stream"; docs/design/dictation.md §Cloud recognition)
- # A recogniser that hears audio as it comes (Volcengine's `streams`) gets a dictation's microphone packets during the
- # take, so at key-up only its last ~15 s block is left (dictation-eval-2026-10-31.md §5: ~0.5 s however long the take,
- # against 1.5 s for a minute-long take sent whole). The app opens one connection per take: one JSON line
- # ({"token", "cmd": "stream", "model_id", "language", "vocab", "app", "hint", "before"}), then raw 16 kHz s16 mono PCM
- # as the microphone gives it, then shuts its side for writing at key-up. The answer is one JSON line, a note's answer
- # (+ "streamed", "after_keyup_ms"), after the end of the audio — a refusal (no key, quota: "quota exceeded for types:
- # concurrency", a model that can't take a stream) too: the PCM is read to its end first, so the answer is never lost
- # to a reset. A connection the app resets (a cancelled take) ends the session unanswered. Each stream runs on its own
- # thread beside the server's loop; the server doesn't end while one is open.
- STREAM_FINAL_WAIT = 10.0 # s after key-up for the final text (measured p90: 0.58–0.70 s)
- # …plus this much per second of audio that came faster than it is spoken (a server that came up late gets the take so
- # far at once): the service works through a backlog at ~4× real time (the eval's unpaced takes: ~0.23 s per second)
- STREAM_BACKLOG_WAIT = 0.4
- def voiceprint_utterances(text: str, utterances: list | None, speaker: dict) -> str:
- """A stream's text without another speaker's sentences: those whose time lies mostly in the gate's dropped spans.
- All someone else → ""; nothing dropped, or no sentence times to go by → the text as it came."""
- if not speaker.get("match"):
- return ""
- drop = speaker.get("drop") or []
- if not drop or not utterances:
- return text
- kept = []
- for u in utterances:
- s, e = (u.get("start_time") or 0) / 1000, (u.get("end_time") or 0) / 1000
- gone = sum(max(0.0, min(e, b) - max(s, a)) for a, b in drop)
- if e <= s or gone < 0.5 * (e - s):
- kept.append(str(u.get("text") or "").strip())
- out = ""
- for k in filter(None, kept):
- cjk = re.match(f"[{_CJK},。?!、]", out[-1:] or " ") or re.match(f"[{_CJK}]", k[0])
- out += k if not out or cjk else " " + k
- return out
- def final_wait(audio_s: float, streamed_s: float) -> float:
- """How long a take's final text is waited for after key-up: `audio_s` of audio arrived over `streamed_s` s."""
- return STREAM_FINAL_WAIT + STREAM_BACKLOG_WAIT * max(0.0, audio_s - streamed_s)
- _streams = 0
- _streams_lock = threading.Lock()
- def streams_open() -> int:
- with _streams_lock:
- return _streams
- def serve_stream(conn: socket.socket, req: dict, extra: bytes) -> None:
- global _streams
- with _streams_lock:
- _streams += 1
- sess = None
- fail: Fail | None = None
- got = 0
- odd = b""
- t0 = time.perf_counter()
- model_id = req.get("model_id")
- biased = False
- field = ""
- gate = False
- pcm = bytearray()
- speaker = None
- try:
- conn.settimeout(120) # a take sends a packet every ~0.1 s: two minutes of nothing is an app gone
- try:
- if local_only():
- raise Fail("local-only recognition is on (privacy.local_asr): cloud ASR is not allowed", "privacy")
- model_id = model_id or dictation_models()["selected"]
- m = entry(model_id)
- cls = BACKENDS.get((m or {}).get("backend"))
- if not m or not getattr(cls, "streams", False):
- raise Fail(f"{model_id} doesn't take a stream", "usage")
- lang = req.get("language")
- a = argparse.Namespace(model=None, model_id=model_id, prompt=None,
- language=None if lang in (None, "", "auto") else lang)
- backend = cls(a)
- # only my voice: the PCM is kept, judged at key-up while the service finishes, and another speaker's
- # sentences are left out of the text by their times (the audio itself has gone out as it was said)
- vp_on, vp_threshold = voiceprint_settings(req.get("voiceprint"))
- gate = vp_on and req.get("purpose", "dictation") == "dictation" and Voiceprint.target() is not None \
- and (MODELS_DIR / VP_FILE).is_file() and (MODELS_DIR / VP_SEG_FILE).is_file()
- backend.utterances = gate
- biased = bias(backend, req.get("vocab"), None, req.get("hint"), req.get("app"))
- if req.get("before") and (n := field_context_chars(model_id)):
- field = prior_tail(req["before"], n)
- backend.set_field(field)
- sess = backend.open()
- except Fail as e:
- fail = e
- data = extra
- t_first = None
- while True:
- if data:
- t_first = t_first or time.perf_counter()
- got += len(data)
- data = odd + data
- odd = data[len(data) & ~1:] # whole samples only
- if sess is not None:
- sess.feed(data[:len(data) & ~1])
- if gate:
- pcm += data[:len(data) & ~1]
- data = conn.recv(65536)
- if not data:
- break
- t_up = time.perf_counter()
- text = ""
- if fail is None and sess is not None:
- judged: dict = {}
- judge = None
- if gate and pcm:
- def run_gate():
- with contextlib.suppress(Fail, ImportError, RuntimeError, ValueError):
- x = np.frombuffer(bytes(pcm), dtype="<i2").astype(np.float32) / 32768
- judged["speaker"] = Voiceprint.gate(x, vp_threshold)[0]
- judge = threading.Thread(target=run_gate, daemon=True)
- judge.start()
- try:
- text = sess.finish(wait=final_wait(got / 2 / SR, t_up - (t_first or t_up)))
- if judge is not None:
- judge.join(timeout=5)
- speaker = judged.get("speaker")
- if speaker is not None:
- text = voiceprint_utterances(text, sess.utterances, speaker)
- text = to_user_script(text, None)
- except Fail as e:
- fail = e
- if fail is not None:
- resp = {"ok": False, "error": str(fail), "code": fail.code, "streamed": True}
- else:
- resp = {"ok": True, "text": text, "asr_text": text, "language": a.language, "model": model_id,
- "audio_seconds": round(got / 2 / SR, 2), "ms": int((time.perf_counter() - t_up) * 1000),
- "after_keyup_ms": int((time.perf_counter() - t_up) * 1000), "load_ms": 0, "biased": biased,
- "streamed": True, "stream_s": round(t_up - t0, 2)}
- if field:
- resp["field_context"] = len(field)
- if speaker is not None:
- resp["speaker"] = speaker
- with contextlib.suppress(OSError):
- conn.sendall((json.dumps(resp, ensure_ascii=False) + "\n").encode())
- except (OSError, ValueError) as e:
- # the app reset the connection (a cancelled take) or went away: the session ends unanswered
- serve_said(f"serve: stream ended ({type(e).__name__})")
- finally:
- if sess is not None:
- sess.close()
- with contextlib.suppress(OSError):
- conn.close()
- with _streams_lock:
- _streams -= 1
- SERVE_FILE = RUN_DIR / "asr-serve.json"
- def _server_call(req: dict, timeout: float = 120) -> dict | None:
- info = load_json(SERVE_FILE, None)
- if not isinstance(info, dict) or not info.get("port"):
- return None
- try:
- with socket.create_connection(("127.0.0.1", int(info["port"])), timeout=0.3) as s:
- s.settimeout(timeout)
- s.sendall((json.dumps({**req, "token": info.get("token")}) + "\n").encode())
- buf = b""
- while not buf.endswith(b"\n"):
- chunk = s.recv(65536)
- if not chunk:
- break
- buf += chunk
- return json.loads(buf) if buf else None
- except (OSError, ValueError):
- return None
- @contextlib.contextmanager
- def no_download(on: bool):
- """While on: a model that isn't on this Mac is an error (model_missing), never a download — a dictation never waits
- for one (`note --no-download`, `serve --no-download` for its preload, a request's "no_download")."""
- if not on or os.environ.get("BLURT_NO_DOWNLOAD"):
- yield
- return
- os.environ["BLURT_NO_DOWNLOAD"] = "1"
- try:
- yield
- finally:
- os.environ.pop("BLURT_NO_DOWNLOAD", None)
- def cmd_note(a):
- if getattr(a, "no_download", False):
- os.environ["BLURT_NO_DOWNLOAD"] = "1"
- wav = Path(a.wav).resolve()
- if not wav.exists():
- raise Fail(f"{wav} not found", "not_found")
- vocab = True if a.vocab else None
- purpose = "dictation" if a.dictation else None
- if not a.no_server:
- r = _server_call({"cmd": "note", "wav": str(wav), "language": a.language, "model_id": a.model_id,
- "vocab": vocab, "purpose": purpose, "no_download": bool(getattr(a, "no_download", False))})
- if r is not None:
- if not r.get("ok"):
- raise Fail(r.get("error") or "server error", r.get("code") or "error")
- return emit({**r, "server": True})
- throttle("note")
- emit(NoteEngine().note(wav, a.language, a.model_id, vocab, purpose))
- def voiceprint_verdict(wav: Path) -> dict:
- """The gate's verdict on a dictation piece without recognising it (serve "vp_gate", `voiceprint check`): {"ok",
- "speaker"}; "speaker" is null when the gate can't judge (no voice enrolled, no model: as if it were off)."""
- x = read_wav(wav)
- _, threshold = voiceprint_settings(None)
- with no_download(True):
- try:
- speaker, _ = Voiceprint.gate(x, threshold)
- except (Fail, ImportError, RuntimeError, ValueError) as e:
- return {"ok": True, "speaker": None, "why": getattr(e, "code", type(e).__name__)}
- return {"ok": True, "speaker": speaker, "audio_seconds": round(len(x) / SR, 2)}
- def cmd_voiceprint(a):
- """`voiceprint prepare` (the model on this Mac, downloaded when missing), `enroll <wav>` (learn the user's voice
- from a passage read aloud), `check <wav>` (the gate's verdict on it), `status`, `clear`. The warm server answers
- vp_gate / vp_status / vp_clear; enrolling is only here, so the server never waits for a download."""
- if a.vp_cmd in ("enroll", "check") and not a.wav:
- raise Fail(f"voiceprint {a.vp_cmd} needs a wav", "usage")
- if a.vp_cmd == "prepare":
- emit(Voiceprint.prepare())
- elif a.vp_cmd == "enroll":
- Voiceprint.prepare()
- emit(Voiceprint.enroll(Path(a.wav).resolve()))
- elif a.vp_cmd == "check":
- emit(voiceprint_verdict(Path(a.wav).resolve()))
- elif a.vp_cmd == "status":
- emit(Voiceprint.status())
- elif a.vp_cmd == "clear":
- emit(Voiceprint.clear())
- def cmd_serve(a):
- """One warm voice-note model per machine. Protocol: one JSON line in ({"token","cmd":"note"|"load"|"ping"|"exit",
- …}), one JSON line out. A "note" with "interim": true (dictation's live text: the piece still being said) is
- stopped as soon as another connection waits — a final piece never waits for it — and answers
- {"ok": false, "code": "interrupted"}; a quick model (SenseVoice) decodes it whole; a slow model that can't stop
- halfway answers "interim_unsupported". A "stream" (a dictation's audio while it is said, for a recogniser that hears it
- as it comes: serve_stream) runs on its own thread; the server doesn't end while one is open. Ends
- (serve_exit_reason) when its model went unused for --idle s — an unload left most of
- the memory resident, so the process goes and the app starts it again on the next key-down — when the app that
- started it (--parent) is gone, on "exit", or after --exit-after s without a request while no model is loaded."""
- global SERVING
- if (r := _server_call({"cmd": "ping"}, timeout=2)) and r.get("ok"):
- return emit({"ok": True, "already": True, "pid": r.get("pid")})
- # every installed model's package now, before there is a run file (serve_packages); one that can't be added
- # (offline with nothing cached) leaves it to a restart when that model is asked for
- missing = [p for p in serve_packages() if not _has_package(p)
- and p not in os.environ.get("BLURT_ASR_REEXEC", "").split(",")]
- if missing:
- try:
- _reexec_with(missing)
- except Fail as e:
- serve_said(f"serve: packages not added ({e})")
- throttle("note")
- RUN_DIR.mkdir(parents=True, exist_ok=True)
- token = uuid.uuid4().hex
- srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- srv.bind(("127.0.0.1", 0))
- srv.listen(4)
- srv.settimeout(1.0)
- port = srv.getsockname()[1]
- save_json(SERVE_FILE, {"pid": os.getpid(), "port": port, "token": token, "parent": a.parent,
- "started": datetime.now().astimezone().isoformat(timespec="seconds")})
- eng = NoteEngine()
- why = None
- restart: str | None = None # a package this server must restart with (NeedsRestart)
- SERVING = True
- # from here on the run file is ours to remove, whatever happens — a preload that fails included (a run file left
- # pointing at a dead or stuck process made the app wait for a server that never answered)
- try:
- if a.preload:
- mid = getattr(a, "model_id", None)
- if not mid and getattr(a, "purpose", None) == "dictation":
- mid = dictation_models()["selected"]
- # --no-download: a model that isn't here stays unloaded (the server still answers: model_missing)
- try:
- with no_download(getattr(a, "no_download", False)):
- eng.load(None, mid)
- except Fail as e:
- if e.code != "model_missing":
- raise
- serve_said(f"serve: preload skipped ({e})")
- except NeedsRestart as e:
- restart, why = e.pkg, "restart"
- if voiceprint_settings(None)[0] and Voiceprint.target() is not None and (MODELS_DIR / VP_FILE).is_file():
- # only my voice: its models too (~0.3 s), so the first piece doesn't wait for them
- def warm_voiceprint():
- with contextlib.suppress(Exception):
- Voiceprint.extractor()
- Voiceprint.diarizer()
- threading.Thread(target=warm_voiceprint, daemon=True).start()
- emit({"ok": True, "pid": os.getpid(), "port": port, "model": eng.model_id})
- last = time.time()
- uses: list[float] = [] # dictation requests: kept warm longer while dictation is used often
- tier = probe()["tier"]
- # a CPU model left at the default idle: an hour where memory allows (_common.CPU_SERVE_IDLE)
- idle = CPU_SERVE_IDLE if not IS_MAC and a.idle == SERVE_IDLE and tier in ("pro", "standard") else a.idle
- while not why:
- now = time.time()
- if streams_open():
- last = now # a take is streaming: never idle
- why = serve_exit_reason(eng.backend is not None, now - eng.last, now - last, uses, tier, idle,
- a.exit_after, a.parent)
- if why:
- break
- try:
- conn, _ = srv.accept()
- except socket.timeout:
- continue
- with conn:
- conn.settimeout(10)
- try:
- buf = b""
- while b"\n" not in buf and len(buf) < 1 << 20: # a vocabulary can be long
- chunk = conn.recv(65536)
- if not chunk:
- break
- buf += chunk
- line, _, extra = buf.partition(b"\n") # a stream's PCM may follow its line
- req = json.loads(line or b"{}")
- if req.get("token") != token:
- resp = {"ok": False, "error": "bad token", "code": "auth"}
- elif req.get("cmd") == "stream":
- # its own thread, its own copy of the connection (this one closes with the block)
- last = time.time()
- if req.get("purpose", "dictation") == "dictation":
- uses.append(last)
- del uses[:-50]
- threading.Thread(target=serve_stream, args=(conn.dup(), req, extra), daemon=True).start()
- continue
- elif req.get("cmd") == "ping":
- resp = {"ok": True, "pid": os.getpid(), "model": eng.model_id, "loaded": eng.backend is not None}
- elif req.get("cmd") == "note":
- last = time.time()
- interim = bool(req.get("interim"))
- if req.get("purpose") == "dictation" and not interim:
- uses.append(last)
- del uses[:-50]
- # an interim decode looks for a waiting connection (a final piece, anything) between steps
- stop = (lambda: bool(_select.select([srv], [], [], 0)[0])) if interim else None
- # a dictation (final or interim) never waits for a download, whatever the client sent
- with no_download(bool(req.get("no_download")) or req.get("purpose") == "dictation"):
- resp = eng.note(Path(req["wav"]), req.get("language"), req.get("model_id"), req.get("vocab"),
- req.get("purpose"), req.get("prior"), req.get("hint"), stop=stop,
- app=req.get("app"), before=req.get("before"),
- voiceprint=req.get("voiceprint"))
- if interim:
- resp["interim"] = True
- elif req.get("cmd") == "vp_gate":
- # only my voice for a piece macOS's recogniser hears (dictation on Apple Speech, or a cloud
- # recogniser's live text): the verdict alone, the app silences the dropped spans itself
- last = time.time()
- resp = voiceprint_verdict(Path(req["wav"]))
- elif req.get("cmd") == "vp_status": # enrolling (it may download) is the CLI's: never here
- resp = Voiceprint.status()
- elif req.get("cmd") == "vp_clear":
- resp = Voiceprint.clear()
- elif req.get("cmd") == "load": # warm a model now (dictation's key went down)
- last = time.time()
- mid = req.get("model_id") or (dictation_models()["selected"] if req.get("purpose") == "dictation" else None)
- with no_download(bool(req.get("no_download")) or req.get("purpose") == "dictation"):
- eng.load(None, mid)
- eng.last = time.time()
- resp = {"ok": True, "model": eng.model_id}
- elif req.get("cmd") == "exit": # the app is quitting (or someone wants the memory back now)
- resp = {"ok": True, "pid": os.getpid()}
- why = "asked"
- else:
- resp = {"ok": False, "error": "unknown cmd", "code": "usage"}
- except Fail as e:
- resp = {"ok": False, "error": str(e), "code": e.code}
- except NeedsRestart as e:
- # answered, then gone: the app waits for the new server (it comes back with the model loaded when
- # it was started with --preload) instead of sending the take to a cold CLI run
- resp = {"ok": False, "error": str(e), "code": "restarting"}
- restart, why = e.pkg, "restart"
- except Interrupted:
- resp = {"ok": False, "error": "interrupted", "code": "interrupted", "interim": True}
- except Exception as e: # noqa: BLE001 — keep serving
- resp = {"ok": False, "error": f"{type(e).__name__}: {e}"[:300], "code": "error"}
- try:
- conn.sendall((json.dumps(resp, ensure_ascii=False) + "\n").encode())
- except OSError:
- pass
- finally:
- SERVING = False
- info = load_json(SERVE_FILE, None)
- if isinstance(info, dict) and info.get("pid") == os.getpid():
- SERVE_FILE.unlink(missing_ok=True)
- srv.close()
- serve_said(f"serve: exiting ({why or 'error'})")
- if restart:
- _reexec_with([restart])
- # --- probe / recommend / auto / models / setup
- def cmd_probe(_a):
- emit({"ok": True, **probe(), "platform": platform_key(), "limits": limits()})
- def _langs_arg(v) -> list[str] | None:
- return [x.strip() for x in v.split(",") if x.strip()] if v else None
- def cmd_recommend(a):
- emit(recommend(_langs_arg(a.languages), a.purpose))
- def auto_plan(apply: bool) -> dict:
- """The per-purpose pick for this machine and the user's languages. --apply writes asr.models (only models that
- are installed — the rest are listed under needs_download for the host to ask about) and the hardware fingerprint."""
- cfg = asr_config()
- hw = probe()
- picks, needs, keep = {}, [], {}
- for p in PURPOSES:
- r = recommend(purpose=p, hw=hw, cfg=cfg)
- mid = r["pick"]["model"]
- m = entry(mid)
- if installed(m):
- picks[p] = keep[p] = mid
- else:
- fallback = select(p) # what runs today
- picks[p] = fallback["id"]
- needs.append({"purpose": p, "model": mid, "size_mb": m["size_mb"]})
- by_model: dict = {}
- for n in needs:
- by_model.setdefault(n["model"], {"model": n["model"], "size_mb": n["size_mb"], "purposes": []})["purposes"].append(n["purpose"])
- out = {"ok": True, "auto": is_auto(cfg), "tier": hw["tier"], "languages": user_languages(cfg), "models": picks,
- "recommended": {p: recommend(purpose=p, hw=hw, cfg=cfg)["pick"]["model"] for p in PURPOSES},
- "needs_download": list(by_model.values()),
- "hardware_changed": cfg.get("hardware") != hw["fingerprint"]}
- if apply and is_auto(cfg):
- # a config.json that doesn't parse: nothing saved, said (not a traceback) — the picks still run as defaults
- # only the recommended picks are kept: a stand-in (what runs until the recommended one is in) stays unpinned,
- # so select() takes the best one installed each time — a Whisper found in a shared cache on the first launch
- # no longer outlasts the 1.7B downloaded after it (v1.0.8)
- out["applied"] = save_config_if_readable({"asr": {"models": keep, "hardware": hw["fingerprint"]}})
- if not out["applied"]:
- out["saved"] = False
- return out
- def cmd_auto(a):
- emit(auto_plan(a.apply))
- # --- dictation's recognition model (record.dictation.asr_model; docs/design/dictation.md §Recognition model)
- QWEN_MLX = "qwen3-asr-0.6b-mlx"
- QWEN_MLX_17 = "qwen3-asr-1.7b-mlx"
- FUN_MLX = "fun-asr-nano-mlx" # Chinese, English and Japanese only: offered and picked only for those (speaks_all)
- FUN_CPU = "fun-asr-nano-onnx" # the same model on the CPU (sherpa-onnx int8): Windows and Linux
- DICTATION_MODELS = (QWEN_MLX, QWEN_MLX_17, FUN_MLX, FUN_CPU, SV, TURBO, TURBO_Q4, QWEN_CPU, CT2, APPLE) # the local ones dictation offers, best
- # first; macOS's own last (nothing to download)
- # offered once their key is set (first-minute-speech.md §2.4; Volcengine: dictation-eval-2026-10-31.md §6.3 B′)
- DICTATION_CLOUD = ("qwen3-asr-flash", "volcengine-seedasr", "openai-compatible")
- def polish_ram_mb(hw: dict, cfg: dict | None = None) -> float:
- """The memory the polish model dictation runs next to takes (polish_models.json ram_mb): the configured one, else
- the tier's default — 0 where polish doesn't run (MLX: Apple Silicon only) or is turned off. Read here rather than
- from polish.py, which imports this module."""
- cfg = ((global_config().get("record") or {}).get("dictation") or {}) if cfg is None else cfg
- if hw_kind(hw) != "apple" or (isinstance(cfg, dict) and cfg.get("polish", "light") == "off"):
- return 0.0
- try:
- reg = json.loads((HERE / "polish_models.json").read_text(encoding="utf-8"))
- except (OSError, ValueError):
- return 0.0
- want = (cfg.get("polish_model") if isinstance(cfg, dict) else None) or (reg.get("defaults") or {}).get(hw["tier"])
- m = next((x for x in reg.get("models") or [] if x.get("id") == want), None)
- return float((m or {}).get("ram_mb") or 0)
- def fits_with_polish(m: dict, hw: dict, polish_mb: float) -> bool:
- """`_fits`, counting the polish model too: a dictation holds both warm at once (ASR + polish), so the pair has to
- stay within the share of RAM one model may take (on a 16 GB Mac: 0.6B 2.1 GB + Gemma 4 E2B 3.9 GB fits; 1.7B
- 3.1 GB + 3.9 GB doesn't)."""
- return _fits({**m, "ram_mb": (m.get("ram_mb") or 0) + polish_mb}, hw)
- # Defaults by Mac (docs/design/dictation-eval-2026-10-31.md §6, docs/design/dictation.md §Defaults by Mac): on the
- # founder's 95 takes the 1.7B (30 s windows, the field's text as context) makes 2.50 % errors, the 0.6B (20 s windows)
- # 3.34 %; Gemma 4 E2B as local polish is within ±0.2 points (fillers and punctuation, not words). The 1.7B needs ~3.1 GB
- # and its last 30 s window decoded at key-up: on a slow chip (a base M1 / M2, 7–10 GPU cores against the M5 Max's 40)
- # that is over a second, so it goes where its measured rtf (`transcribe.py bench`, asr-bench.json) is at most
- # TIER_RTF_LIMIT — or, not measured, where the chip isn't a base M1 / M2. Where the 1.7B doesn't go (8 GB, slow chips),
- # Fun-ASR-Nano for users who speak only Chinese, English and Japanese (2.92 % through the app's path, 0.07 / 0.17 s
- # after key-up, 2.6 GB at peak: it fits 8 GB as the 0.6B does), else the 0.6B (small_recognizer). BlurtCore
- # DictationTier.recommended is the same rule for the app (Settings shows it); tests/fixtures/dictation_tier.jsonl checks
- # both.
- TIER_RTF_LIMIT = 0.05
- SLOW_CHIPS = ("Apple M1", "Apple M2")
- TIER_POLISH = "gemma4-e2b"
- def speaks_all(m: dict | None, langs: list[str] | None) -> bool:
- """Every one of `langs` is a language of model `m` (asr_models.json `languages`); False for no languages (nothing
- known about the user is no reason for a model that speaks three)."""
- return bool(m) and bool(langs) and all(speaks(m, x) for x in langs)
- def small_recognizer(langs: list[str] | None) -> str:
- """The recogniser for Macs the 1.7B doesn't suit (8 GB, or too slow for it): Fun-ASR-Nano when it speaks every
- language of the user's (Chinese, English, Japanese) — fewer errors than the 0.6B (2.92 vs 3.34 % on the founder's
- takes through the app's path; Chinese with English inside it ASCEND 6.6 vs 8.9 %, English level, Japanese 16.0 vs
- 20.2 %) and as fast after key-up (dictation-eval-2026-10-31.md §2.1, §3, §5; docs/design/dictation.md §Defaults by
- Mac) — else the 0.6B (30 languages)."""
- return FUN_MLX if speaks_all(entry(FUN_MLX), langs) else QWEN_MLX
- def cpu_recognizer(hw: dict, langs: list[str] | None) -> str:
- """Dictation's recogniser on a PC without Apple Silicon (docs/design/windows-models.md): Fun-ASR-Nano int8 for
- Chinese / English / Japanese speakers (3.69 % on the founder's takes, ~1 s after key-up for a 10 s piece on 4 Zen 4
- threads), else Qwen3-ASR 0.6B int8 (30 languages, 3.51 %); SenseVoice on a minimal PC (< 8 GB or ≤ 2 cores:
- 0.2 s, 5.2 %)."""
- if hw.get("ram_gb", 16) < 7.5 or (hw.get("perf_cores") or 4) + (hw.get("eff_cores") or 0) <= 2:
- return SV
- return FUN_CPU if speaks_all(entry(FUN_CPU), langs) else QWEN_CPU
- def slow_chip(chip: str | None) -> bool:
- """A base M1 / M2 (not Pro, Max or Ultra): the chips the 1.7B is too slow on when nothing was measured."""
- return " ".join(str(chip or "").split()) in SLOW_CHIPS
- def dictation_tier(ram_gb: float, rtf: float | None = None, slow: bool = False, cloud_key: bool = False,
- languages: list[str] | None = None) -> dict:
- """What dictation runs on for a Mac with `ram_gb` of memory (Apple Silicon): the recogniser (its window and field
- context come with it from asr_models.json), the local polish model (None: off), and — with a cloud key —
- cloud polish with macOS's transcript as the second opinion (§6.3 A: the recogniser stays local). `rtf`: the 1.7B's
- measured real-time factor here (None: not measured — then `slow`, a base M1 / M2, decides). `languages`: the
- user's (user_languages): where the 1.7B doesn't go, Fun-ASR-Nano for Chinese / English / Japanese speakers, else
- the 0.6B (small_recognizer). Pure."""
- tier = "minimal" if ram_gb < 7.5 else "light" if ram_gb < 15 else "standard" if ram_gb < 30 else "pro"
- fast = (rtf <= TIER_RTF_LIMIT) if rtf is not None else not slow
- big = tier in ("standard", "pro") and fast
- rec = QWEN_MLX_17 if big else small_recognizer(languages) if tier != "minimal" else None
- m = entry(rec) or {}
- return {"tier": tier, "recognizer": rec, "window_s": m.get("rehear_window_s") or (30 if big else 20 if rec else None),
- "field_context": bool(m.get("field_context_chars")) if m else False,
- "local_polish": TIER_POLISH if tier in ("standard", "pro") else None,
- "cloud_polish": bool(cloud_key), "second_transcript": bool(cloud_key), "first": APPLE,
- "why": "ram" if tier in ("minimal", "light") else "fast" if big else "rtf" if rtf is not None else "chip"}
- def dictation_models(hw: dict | None = None) -> dict:
- """configured (record.dictation.asr_model or None = automatic), recommended (by the dictation eval: Qwen3-ASR MLX
- on Apple Silicon pro / standard, else the voice-note pick), selected (what runs: the configured one when it is
- installed, else the recommended one when installed, else the best installed Qwen3-ASR MLX that runs here next to
- the polish model, else the voice-note model) and the rows for Settings. Memory is judged for the pair dictation
- holds — the recogniser plus the polish model (fits_with_polish)."""
- hw = hw or probe()
- cfg = (global_config().get("record") or {}).get("dictation") or {}
- configured = cfg.get("asr_model") if isinstance(cfg, dict) else None
- note_pick = select("note")
- apple = hw_kind(hw) == "apple"
- pmb = polish_ram_mb(hw, cfg if isinstance(cfg, dict) else {})
- def fits(mid: str) -> bool:
- m = entry(mid)
- return bool(m) and fits_with_polish(m, hw, pmb)
- # by Mac (dictation_tier: the 2026-10-31 eval): the 1.7B on 16 GB+ Macs fast enough for it, else Fun-ASR-Nano for
- # Chinese / English / Japanese speakers, else the 0.6B — each only where it fits beside the polish model; else the
- # voice-note pick (the voice-note pick is macOS's own recogniser while nothing is downloaded: the download to offer
- # is then the one the voice-note rules would choose)
- note_rec = note_pick["id"] if note_pick["id"] != APPLE else recommend(purpose="note", hw=hw)["pick"]["model"]
- langs = user_languages()
- small = small_recognizer(langs)
- plan = dictation_tier(hw["ram_gb"], (_bench_results().get(QWEN_MLX_17) or {}).get("rtf"), slow_chip(hw.get("chip")),
- languages=langs)
- rec = next((mid for mid in ((plan["recognizer"], small, QWEN_MLX) if plan["recognizer"] else ())
- if apple and fits(mid)), note_rec)
- if not apple:
- rec = cpu_recognizer(hw, langs)
- # what runs: the configured one (the user's choice, as it is), else the recommended one, else any of the 1.7B,
- # Fun-ASR-Nano (for its languages) or the 0.6B already here that runs next to polish, best first for this Mac (the
- # 1.7B last where it is too slow or too big) — they take the Words list; the voice-note model doesn't — a 1.7B
- # brought from a bigger Mac, or a 0.6B until Fun-ASR-Nano is in — all only when installed; else the voice-note model
- order = (QWEN_MLX_17, small, QWEN_MLX) if plan["recognizer"] == QWEN_MLX_17 else (small, QWEN_MLX, QWEN_MLX_17)
- usable = [mid for mid in dict.fromkeys(order) if mid != rec and apple and fits(mid)]
- if not apple: # any CPU model already here, best first: Fun-ASR-Nano (its languages), the 0.6B, SenseVoice
- usable = [mid for mid in ((FUN_CPU,) if speaks_all(entry(FUN_CPU), langs) else ()) + (QWEN_CPU, SV) if mid != rec]
- # every local model hears through the VAD (silero_vad.onnx, 0.6 MB, which `setup` fetches first): without it a
- # dictation can't run ("model_missing" under BLURT_NO_DOWNLOAD), so a model counts as installed only with it;
- # `files` = the model's own files are in (then `setup` of it only fetches the VAD)
- bundled_vad()
- vad = (MODELS_DIR / "silero_vad.onnx").is_file()
- # without the VAD, macOS's recogniser (the voice-note pick then) runs before a model whose files only are here —
- # unless the user chose that model
- pick = (configured,) if not vad and note_pick["id"] == APPLE else (configured, rec, *usable)
- selected = next((mid for mid in pick if mid and entry(mid) and runs_here(entry(mid))
- and installed(entry(mid))), note_pick["id"])
- rows = []
- for mid in DICTATION_MODELS + DICTATION_CLOUD:
- m = entry(mid)
- if not m or not runs_here(m):
- continue
- if not m.get("local", True) and (local_only() or not installed(m)):
- continue # a cloud service: only with its key, never when recognition stays on this Mac
- if mid in (FUN_MLX, FUN_CPU) and not speaks_all(m, langs) and mid not in (configured, selected):
- continue # three languages: not offered to someone who speaks another (unless already chosen)
- have = installed(m)
- rows.append({"id": mid, "name": m.get("name"), "backend": m["backend"], "size_mb": m["size_mb"],
- "ram_mb": m.get("ram_mb"), "installed": have and (vad or not m.get("local", True) or bool(m.get("builtin"))),
- "files": have,
- "builtin": bool(m.get("builtin")),
- "vocab": m.get("vocab") or "none", "recommended": mid == rec, "fits": fits(mid)})
- # ready: a dictation can run right now with nothing downloaded (it never downloads: `no_download`) — the
- # selected model (local and on this Mac, or a cloud one with its key: judged by itself, not by being a row) and
- # the VAD every take goes through. missing: what a download would have to bring for that ("silero-vad" and/or
- # the selected model's id)
- sel = entry(selected)
- # macOS's recogniser hears without the VAD, and so does a cloud service for a dictation's pieces (transcribe_track)
- missing = [] if vad or selected == APPLE or bool(sel and not sel.get("local", True)) else ["silero-vad"]
- selected_installed = bool(sel and installed(sel))
- if not selected_installed:
- missing.append(selected)
- return {"configured": configured, "configured_installed": bool(configured and entry(configured)
- and installed(entry(configured))),
- "recommended": rec, "selected": selected, "selected_installed": selected_installed,
- # the dictation goes to a cloud service: the app shows live text from macOS's recogniser and doesn't hear it
- # again (each pass is billed)
- "selected_cloud": bool(sel and not sel.get("local", True)),
- # a PC the CPU rules give the fast, less accurate model (SenseVoice: < 8 GB or ≤ 2 cores): the app offers
- # cloud recognition there first (windows-models.md §Cloud recognition on a weak PC) — never switches to it
- "weak": not apple and rec == SV,
- # cloud recognition may be offered at all (privacy.local_asr off)
- "cloud_allowed": not local_only(),
- "ready": not missing, "missing": missing, "vad": vad, "models": rows, "plan": plan}
- def cmd_dictation_models(_a):
- emit({"ok": True, **dictation_models()})
- def selected_models() -> dict:
- out = {}
- for p in PURPOSES:
- try:
- out[p] = select(p)["id"]
- except Fail:
- out[p] = None
- return out
- def cmd_models(_a):
- hw = probe()
- lim = limits(hw)
- bench = _bench_results()
- sel = selected_models()
- langs = user_languages()
- rows = []
- for m in entries():
- r = {k: v for k, v in m.items() if k not in ("install", "rtf_ref", "error_ref")}
- r["local"] = m.get("local", True)
- r["role"] = "asr"
- r["installed"] = installed(m)
- p = model_path(m) if r["local"] else None
- if p:
- r["path"] = str(p)
- r["size_mb"], r["approx"] = _du_mb(p), False
- else:
- r["approx"] = r["local"]
- r["runs_here"] = runs_here(m)
- r["fits"] = _fits(m, hw, lim)
- r["purposes"] = [k for k, v in sel.items() if v == m["id"]]
- r["configured"] = bool(r["purposes"])
- r["speaks"] = [x for x in langs if speaks(m, x)]
- if m["id"] in bench:
- r["bench"] = bench[m["id"]]
- for k in ("rtf_ref", "error_ref"):
- if m.get(k) is not None:
- r[k] = m[k]
- rows.append(r)
- bundled_vad()
- vad = MODELS_DIR / "silero_vad.onnx" # every local model needs it; listed so hosts can show the whole download
- rows.append({"id": "silero-vad", "name": "Silero VAD", "backend": "vad", "role": "vad", "local": True,
- "languages": ["*"], "installed": vad.exists(), "size_mb": _du_mb(vad) if vad.exists() else 0.6,
- "approx": not vad.exists(), "path": str(vad), "purposes": [], "configured": False})
- cfg = asr_config()
- emit({"ok": True, "models_dir": str(MODELS_DIR), "configured": cfg, "auto": is_auto(cfg), "tier": hw["tier"],
- "platform": platform_key(), "hardware": hw, "limits": lim, "languages": langs,
- "languages_set": bool(cfg.get("languages")), "selected": sel,
- "recommended": {p: recommend(langs, p, hw, cfg)["pick"]["model"] for p in PURPOSES},
- "local_only": local_only(), "models": rows})
- def cmd_setup(a):
- watch_parent(getattr(a, "parent", None)) # started by the app: ends with it (the next setup continues the files)
- if a.model_id:
- m = entry(a.model_id)
- if not m:
- raise Fail(f"unknown model {a.model_id}; see transcribe.py models", "unknown_model")
- else: # legacy: setup --backend B [--model M]
- m = entry_for(a.backend, a.model) or entry_for(a.backend)
- if m is None:
- raise Fail(f"unknown backend {a.backend}", "unknown_model")
- path = install_model(m)
- patch: dict = {}
- if a.use:
- purposes = PURPOSES if a.use == "all" else [p for p in a.use.split(",") if p in PURPOSES]
- cur = selected_models()
- patch["models"] = {**cur, **{p: m["id"] for p in purposes}}
- patch["auto"] = False # a hand-picked model is the user's choice: not replaced on hardware changes
- elif not a.model_id: # legacy behaviour: setup --backend selects it
- patch["backend"] = a.backend
- if a.model:
- patch["model"] = a.model
- if a.language:
- patch["language"] = a.language
- # the download is done either way: a config.json that doesn't parse only leaves the choice unsaved ("saved": false)
- saved = save_config_if_readable({"asr": patch}) if patch else True
- emit({"ok": True, "model": m["id"], "path": str(path), "asr": asr_config(), **({} if saved else {"saved": False})})
- # --- check-cloud: Settings' Test for a cloud speech service (CloudSpeech.check in the app)
- CHECK_PHRASE = "Testing, one, two, three."
- # the variables each cloud backend reads, for a key / address / model typed in Settings but not saved yet
- CLOUD_ENV = {"dashscope": {"key": "DASHSCOPE_API_KEY", "base_url": "DASHSCOPE_BASE_URL", "model": "DASHSCOPE_ASR_MODEL"},
- "volcengine": {"app_id": "VOLC_APP_ID", "token": "VOLC_ACCESS_TOKEN"},
- "openai": {"key": "BLURT_ASR_API_KEY", "base_url": "BLURT_ASR_BASE_URL", "model": "BLURT_ASR_MODEL"}}
- def check_clip() -> tuple[np.ndarray, str]:
- """A short spoken clip (macOS `say`, offline, ~1.5 s) and what it says. Where there is no `say`: 0.6 s of
- silence and "" — still the whole request (key, address, model), only nothing to hear."""
- if shutil.which("say"):
- with tempfile.TemporaryDirectory(prefix="blurt-check-") as d:
- out = Path(d) / "check.wav"
- try:
- r = subprocess.run(["say", "-o", str(out), f"--data-format=LEI16@{SR}", CHECK_PHRASE],
- capture_output=True, timeout=20)
- if r.returncode == 0 and out.exists():
- x = read_wav(out)
- if len(x) > SR // 4:
- return x, CHECK_PHRASE
- except (OSError, subprocess.TimeoutExpired):
- pass
- return np.zeros(int(0.6 * SR), np.float32), ""
- def cloud_check(model_id: str, given: dict | None = None, timeout: float = 20) -> dict:
- """One small request to a cloud speech service, as dictation sends it → {"ok", "ms", "text", "heard", "model"}
- or {"ok": False, "code", "error"} (codes: auth, quota, network, bad_response, privacy, usage). `given`: a key /
- base_url / model typed but not saved (the app passes them on stdin, never on the command line)."""
- m = entry(model_id)
- if not m or m.get("local", True) or m.get("backend") not in CLOUD_ENV:
- return {"ok": False, "code": "usage", "error": f"{model_id} is not a cloud speech service"}
- try:
- if local_only():
- raise Fail("local-only recognition is on (privacy.local_asr): cloud ASR is not allowed", "privacy")
- names = CLOUD_ENV[m["backend"]]
- for k, v in (given or {}).items():
- if k in names and isinstance(v, str) and v.strip():
- os.environ[names[k]] = v.strip()
- a = argparse.Namespace(model=None, model_id=model_id, language=None, prompt=None)
- backend = BACKENDS[m["backend"]](a)
- backend.retries, backend.timeout = 1, timeout # Settings waits for one try, not four
- x, said = check_clip()
- t0 = time.perf_counter()
- r = backend.transcribe(x)
- ms = int((time.perf_counter() - t0) * 1000)
- text = " ".join(seg[2] for seg in r if seg[2]).strip()
- return {"ok": True, "ms": ms, "text": text, "heard": bool(text) or not said, "spoken": said,
- "model": getattr(backend, "model", None)}
- except Fail as e:
- return {"ok": False, "code": e.code, "error": str(e)}
- def cmd_check_cloud(a):
- given = {}
- if not sys.stdin.isatty():
- try:
- given = json.loads(sys.stdin.read() or "{}")
- except ValueError:
- given = {}
- r = cloud_check(a.model_id, given if isinstance(given, dict) else {})
- emit(r)
- if not r.get("ok"):
- sys.exit(1)
- def cmd_backends(_a):
- bundled_vad()
- have = lambda rel: (MODELS_DIR / rel).exists() # noqa: E731
- emit({
- "ok": True,
- "configured": global_config().get("asr"),
- "sensevoice_ready": have("sensevoice/model.int8.onnx") and have("sensevoice/tokens.txt") and have("silero_vad.onnx"),
- "selected": selected_models(),
- "api_keys": {k: bool(os.environ.get(k)) for k in ("BLURT_ASR_API_KEY", "OPENAI_API_KEY", "GROQ_API_KEY", "DASHSCOPE_API_KEY")},
- })
- # --- bench: speed, memory, CPU / GPU load, and accuracy on spoken samples
- CER_LANGS = {"zh", "yue", "ja", "ko", "th"}
- def normalize(text: str, lang: str) -> list[str]:
- t = unicodedata.normalize("NFKC", text).lower()
- t = "".join(" " if unicodedata.category(c)[0] in "PSZ" else c for c in t)
- if lang in CER_LANGS:
- return [c for c in t if not c.isspace()]
- return t.split()
- def error_rate(ref: str, hyp: str, lang: str) -> float:
- """CER for zh / yue / ja / ko / th, WER otherwise (punctuation and case ignored)."""
- r, h = normalize(ref, lang), normalize(hyp, lang)
- if not r:
- return 0.0 if not h else 1.0
- prev = list(range(len(h) + 1))
- for i, rc in enumerate(r, 1):
- cur = [i] + [0] * len(h)
- for j, hc in enumerate(h, 1):
- cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (rc != hc))
- prev = cur
- return round(prev[-1] / len(r), 4)
- def say_voices() -> dict[str, list[str]]:
- """locale → voice names, from `say -v ?`."""
- try:
- out = subprocess.run(["say", "-v", "?"], capture_output=True, text=True, timeout=20).stdout
- except (OSError, subprocess.TimeoutExpired):
- return {}
- voices: dict[str, list[str]] = {}
- for line in out.splitlines():
- m = re.match(r"^(.+?)\s{2,}([a-z]{2,3}_[A-Z0-9]{2,3})\s", line)
- if m:
- voices.setdefault(m.group(2), []).append(m.group(1).strip())
- return voices
- def bench_samples(langs: list[str]) -> list[dict]:
- """Spoken samples for `langs` made with macOS `say`, cached under $BLURT_HOME/bench/: [{lang, path, text}]."""
- if not shutil.which("say"):
- raise Fail("no AUDIO given and no `say` to make samples: pass a recording, e.g. transcribe.py bench clip.mp4",
- "usage")
- defs = registry()["bench"]["samples"]
- voices = None
- out = []
- for lang in langs:
- d = defs.get(lang)
- if not d:
- continue
- path = BLURT_HOME / "bench" / f"say-{lang}.wav"
- text = " ".join(d["text"])
- if not path.exists():
- voices = voices if voices is not None else say_voices()
- have = {v for vs in voices.values() for v in vs}
- voice = next((v for v in d["voices"] if v in have), None) or next(iter(voices.get(d["locale"], [])), None)
- if not voice:
- continue
- path.parent.mkdir(parents=True, exist_ok=True)
- aiff = path.with_suffix(".aiff")
- subprocess.run(["say", "-v", voice, "-o", str(aiff), " [[slnc 900]] ".join(d["text"])], check=True,
- capture_output=True)
- subprocess.run([ffmpeg_bin(), "-nostdin", "-loglevel", "error", "-y", "-i", str(aiff), "-ar", str(SR), "-ac", "1",
- str(path)], check=True)
- aiff.unlink(missing_ok=True)
- out.append({"lang": lang, "path": path, "text": text})
- return out
- class GpuSampler:
- """Apple GPU utilisation (system-wide, from IOAccelerator's PerformanceStatistics) sampled while decoding."""
- def __init__(self, every: float = 0.2):
- self.every, self.samples = every, []
- self.stop = threading.Event()
- self.th = threading.Thread(target=self._run, daemon=True) if IS_MAC else None
- def _run(self):
- while not self.stop.wait(self.every):
- try:
- out = subprocess.run(["ioreg", "-r", "-d", "1", "-c", "IOAccelerator"], capture_output=True, text=True,
- timeout=2).stdout
- m = re.search(r'"Device Utilization %"=(\d+)', out)
- if m:
- self.samples.append(int(m.group(1)))
- except (OSError, subprocess.TimeoutExpired):
- pass
- def __enter__(self):
- if self.th:
- self.th.start()
- return self
- def __exit__(self, *exc):
- self.stop.set()
- if self.th:
- self.th.join(1)
- return False
- def summary(self) -> dict | None:
- if not self.samples:
- return None
- return {"mean": round(sum(self.samples) / len(self.samples)), "max": max(self.samples)}
- def run_bench(m: dict, a) -> dict:
- """Measure one model on this machine (in this process): load time, decode RTF, peak memory, CPU cores busy,
- GPU utilisation, and the error rate per language on the `say` samples (or just speed on a given AUDIO)."""
- langs = _langs_arg(a.languages) or (list(registry()["bench"]["samples"]) if a.all_languages else user_languages())
- if a.input:
- samples = [{"lang": None, "path": Path(a.input), "text": None}]
- else:
- samples = bench_samples([x for x in langs if speaks(m, x)] or langs[:1])
- if not samples:
- raise Fail("no bench sample for these languages", "usage")
- audio = [(s, (load_audio(s["path"]) if a.input else read_wav(s["path"]))[: int(a.seconds * SR)]) for s in samples]
- ns = argparse.Namespace(backend=None, model=None, model_id=m["id"], language=None, prompt=None,
- threads=a.threads, concurrency=4)
- name = resolve_backend(ns, "batch")
- base_mem = peak_memory_mb()
- t0 = time.perf_counter()
- backend = make_backend(name, ns, "bench")
- load_s = time.perf_counter() - t0
- # warm-up (first-call kernel compilation on MLX), not timed
- transcribe_track(audio[0][1][: SR * 5], backend, name, 1, quiet=True)
- errors, texts, audio_s, dec_s, cpu_s = {}, {}, 0.0, 0.0, 0.0
- with GpuSampler() as gpu:
- for s, x in audio:
- c0, t1 = os.times(), time.perf_counter()
- results, _sp, _bl = transcribe_track(x, backend, name, 1 if name in LOCAL else 4, quiet=True)
- dt = time.perf_counter() - t1
- c1 = os.times()
- dec_s += dt
- cpu_s += (c1.user - c0.user) + (c1.system - c0.system)
- audio_s += len(x) / SR
- hyp, _lang = note_text(results)
- if s["lang"]:
- texts[s["lang"]] = hyp
- if s["text"] and len(x) / SR >= 0.95 * float(probe_wav_seconds(s["path"])):
- errors[s["lang"]] = error_rate(s["text"], hyp, s["lang"])
- out = {"ok": True, "model": m["id"], "backend": name, "threads": ns.threads, "audio_seconds": round(audio_s, 1),
- "load_seconds": round(load_s, 2), "decode_seconds": round(dec_s, 2),
- "rtf": round(dec_s / audio_s, 4) if audio_s else None,
- "cpu_cores": round(cpu_s / dec_s, 2) if dec_s else None,
- "peak_ram_mb": peak_memory_mb(), "base_ram_mb": base_mem,
- "at": datetime.now().astimezone().isoformat(timespec="seconds"), "machine": probe()["fingerprint"]}
- if gpu.summary():
- out["gpu_util"] = gpu.summary()
- if qos_cpu_seconds():
- out["qos_cpu_seconds"] = qos_cpu_seconds()
- if errors:
- out["error"] = errors
- if a.show_text:
- out["text"] = texts
- return out
- def probe_wav_seconds(p: Path) -> float:
- try:
- with wave.open(str(p)) as w:
- return w.getnframes() / w.getframerate()
- except (wave.Error, OSError):
- return 0.0
- def cmd_bench(a):
- if a.model_id:
- m = entry(a.model_id)
- if not m:
- raise Fail(f"unknown model {a.model_id}; see transcribe.py models", "unknown_model")
- elif a.backend:
- m = entry_for(a.backend, a.model) or entry_for(a.backend)
- else:
- m = select("batch")
- if m is None:
- raise Fail("no such model", "unknown_model")
- throttle("bench", a.threads)
- with AsrSlot(limits()["max_concurrent"]):
- out = run_bench(m, a)
- if not a.no_save and not a.input:
- d = load_json(BENCH_FILE, {}) or {}
- d = d if isinstance(d, dict) else {}
- d["schema"] = 1
- d.setdefault("results", {})[m["id"]] = {k: v for k, v in out.items() if k not in ("ok", "text")}
- save_json(BENCH_FILE, d)
- emit(out)
- def main():
- p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
- sub = p.add_subparsers(dest="cmd", required=True)
- def model_opts(sp, backend_default=None):
- sp.add_argument("--backend", choices=list(BACKENDS), default=backend_default)
- sp.add_argument("--model", help="the backend's model name / repo (legacy); --model-id picks from the registry")
- sp.add_argument("--model-id", dest="model_id", metavar="ID", help="a model id from asr_models.json")
- sp.add_argument("--language", help="e.g. zh, en, ja; default auto-detect")
- r = sub.add_parser("run")
- r.add_argument("input")
- r.add_argument("--out", help="output dir (default: next to input)")
- model_opts(r)
- r.add_argument("--prompt", help="glossary / context: product & module names, jargon (Whisper / Qwen3 / API backends)")
- r.add_argument("--concurrency", type=int, default=4)
- r.add_argument("--threads", type=int)
- r.add_argument("--force", action="store_true", help="transcribe again even if live.py already did")
- n = sub.add_parser("note")
- n.add_argument("wav")
- n.add_argument("--language", default="auto")
- n.add_argument("--model-id", dest="model_id", metavar="ID")
- n.add_argument("--no-server", action="store_true", help="always transcribe in this process")
- n.add_argument("--dictation", action="store_true", help="dictation's model (record.dictation.asr_model)")
- n.add_argument("--vocab", action="store_true", help="the dictation words: bias the model toward them (hints only)")
- n.add_argument("--no-download", action="store_true", help="a model that isn't here is an error (model_missing), never a download")
- sv = sub.add_parser("serve")
- sv.add_argument("--preload", action="store_true", help="load the voice-note model now (e.g. when recording starts)")
- sv.add_argument("--model-id", dest="model_id", metavar="ID", help="preload this model instead")
- sv.add_argument("--purpose", choices=["note", "dictation"], help="dictation: preload dictation's model")
- sv.add_argument("--no-download", action="store_true", help="the preload never downloads (a missing model stays unloaded)")
- sv.add_argument("--idle", type=float, default=SERVE_IDLE, help="end after the model went unused this many seconds")
- sv.add_argument("--exit-after", type=float, default=1800, help="with no model loaded: end after this many seconds without a request")
- sv.add_argument("--parent", type=int, help="end when this process (the app that started the server) is gone")
- sub.add_parser("probe")
- rc = sub.add_parser("recommend")
- rc.add_argument("--languages", help="comma-separated, e.g. zh,en,ja (default: asr.languages / the system's)")
- rc.add_argument("--purpose", choices=PURPOSES, default="batch")
- au = sub.add_parser("auto")
- au.add_argument("--apply", action="store_true", help="write asr.models for what is installed")
- s = sub.add_parser("setup")
- s.add_argument("--model-id", "--id", dest="model_id", metavar="ID")
- s.add_argument("--backend", choices=list(BACKENDS), default="sensevoice")
- s.add_argument("--model")
- s.add_argument("--language")
- s.add_argument("--use", help="also select it: all, or a comma list of batch,live,note")
- s.add_argument("--parent", type=int, help="end when this process (the app that started the download) is gone")
- sub.add_parser("backends")
- cc = sub.add_parser("check-cloud", help="Settings' Test: one small request to a cloud speech service")
- cc.add_argument("--model-id", dest="model_id", metavar="ID", default="qwen3-asr-flash")
- sub.add_parser("models")
- sub.add_parser("dictation-models")
- vp = sub.add_parser("voiceprint", help="only my voice: prepare the model, enroll / check a voice, status, clear")
- vp.add_argument("vp_cmd", choices=["prepare", "enroll", "check", "status", "clear"])
- vp.add_argument("wav", nargs="?", help="enroll / check: the wav")
- b = sub.add_parser("bench")
- b.add_argument("input", nargs="?", help="audio / video file (default: spoken samples made with `say`)")
- model_opts(b)
- b.add_argument("--prompt")
- b.add_argument("--languages", help="comma-separated (default: your languages that the model speaks)")
- b.add_argument("--all-languages", action="store_true", help="every bench language the registry has")
- b.add_argument("--seconds", type=float, default=60, help="at most this much of each sample")
- b.add_argument("--threads", type=int)
- b.add_argument("--show-text", action="store_true")
- b.add_argument("--no-save", action="store_true")
- a = p.parse_args()
- # `--model ID` with a registry id means --model-id (K9 writes it that way); other values stay backend model names
- if a.cmd in ("setup", "bench", "run") and not a.model_id and a.model and entry(a.model):
- a.model_id, a.model = a.model, None
- try:
- {"run": cmd_run, "note": cmd_note, "serve": cmd_serve, "probe": cmd_probe, "recommend": cmd_recommend,
- "auto": cmd_auto, "setup": cmd_setup, "backends": cmd_backends, "models": cmd_models, "bench": cmd_bench,
- "dictation-models": cmd_dictation_models, "check-cloud": cmd_check_cloud, "voiceprint": cmd_voiceprint}[a.cmd](a)
- except Fail as e:
- print(f"ERROR: {e}", file=sys.stderr)
- emit({"ok": False, "error": str(e), "code": e.code})
- if a.cmd == "serve":
- server_exit(e.exit_code)
- sys.exit(e.exit_code)
- except Exception as e: # noqa: BLE001 — a server never goes through the interpreter's teardown (server_exit)
- if a.cmd in ("note", "check-cloud"):
- # the app reads the last JSON line: a crash still says what it was (it read a bare "error" before)
- import traceback
- traceback.print_exc()
- emit({"ok": False, "error": f"{type(e).__name__}: {e}"[:300], "code": "error"})
- sys.exit(1)
- if a.cmd != "serve":
- raise
- import traceback
- traceback.print_exc()
- server_exit(1)
- if a.cmd == "serve":
- server_exit(0)
- if __name__ == "__main__":
- main()
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.