Back to search
dsh-voice
SkillGitHub
Full-duplex voice plugin for DeepSeek Harness: mic → SenseVoice ASR (sherpa-onnx) → LLM → Edge TTS with true barge-in. 全双工语音插件:麦克风语音输入,SenseVoice 本地识别(简体中文+标点+ITN),流式 TTS 朗读,支持语音打断。Zero API key.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Checked today
Badge for your README
Markdown
[](/listings/haoku123--dsh-voice)HTML
<a href="/listings/haoku123--dsh-voice"><img src="/api/listings/haoku123--dsh-voice/badge.svg" alt="findsafeskills safety rating" /></a>Install
i
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
dsh plugin --profile web add <repo-url-or-path> dsh --profile web
Note: needs Node ≥ 22.19 or ≥ 24 (`node:zlib` zstd APIs). Works with dsh 0.1.0-rc.6 and 0.2.x — the `@deepseek-ai/dsh-web` peer range covers both.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 154 lines
- # dsh-voice
- English | [简体中文](./README.zh-CN.md)
- Full-duplex voice mode for DeepSeek Harness: streamed ASR → LLM → TTS with barge-in.
- ## Status
- **v0.8.2 — press-to-talk with a live caption.**
- Speak into the composer mic: the assistant silences itself (playback stops,
- host synthesis queue drops), the running turn is cancelled (the stop-button
- route), and your speech is transcribed by the host (SenseVoice, CN-native
- simplified-Chinese ASR with punctuation + ITN) and submitted. The reply
- streams back as spoken audio with live captions.
- Three ways to dictate:
- | gesture | behaviour |
- | --- | --- |
- | tap the mic | continuous dictation; the VAD segments on trailing silence |
- | **hold the send key** (or the mic) | records until release, slide up to discard |
- | **hold `Ctrl`** (configurable `asr.hotkey`) | same, without leaving the keyboard; `Esc` discards |
- While a hold is open the overlay shows a **live caption** — the interim
- transcript of what has been said so far — and keeps a spinner up after
- release until the authoritative transcript lands.
- Barge-in detection is triggered by the mic's leading speech edge. The ASR
- engine runs an **NLMS acoustic echo canceller** (see `src/aec.ts`) using the
- page's own TTS playback as the echo reference, so loud assistant audio is
- subtracted from the mic before the VAD — the browser-level
- `echoCancellation` constraint is kept only as a fallback when no echo
- reference is available.
- Live caption interims are **incremental**: each pass sends only the audio
- recorded since the last pass (correlated by session header), and the host
- decodes a bounded sliding window per session instead of re-decoding the
- whole hold. Preview cost therefore stops growing with hold length.
- ## Demo
- 
- The loop: hold the composer's send key (its arrow is covered by a mic glyph),
- watch the live caption fill in while you speak, release into a spinner until
- the final transcript lands, then the reply streams back as spoken audio
- sentence-by-sentence — until the user's voice interrupts playback and stops
- the running turn mid-line (true barge-in). `Ctrl` does the same without
- leaving the keyboard.
- ## How it works
- ```
- input: mic ──RMS endpoint detection──▶ POST /asr (raw f32 PCM)
- │ text (SenseVoice)
- ▼
- composer draft ──submit──▶ model stream ──llm/stream tap──▶ SentenceSegmenter
- │
- browser ◀── SSE /dsh-voice-api/stream ── TtsQueue (msedge-tts) ◀──┘
- (base64 MP3 frames + caption text)
- barge-in: speech edge ──▶ engine.skip() + POST /cancel (epoch bump)
- + session.cancel() when a turn is running
- ```
- - The `llm/stream` tap is **lossless**: every chunk is yielded unchanged, the
- segmenter only observes. The model stream is never blocked by synthesis.
- - ASR runs **host-side** with [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
- (Apache-2.0) running **SenseVoice** — the CN-native speech model that
- outperforms whisper on Chinese: native simplified output, punctuation,
- inverse text normalization (ITN) and 50+ language auto-detection. The
- browser only records and posts raw little-endian f32 PCM.
- - Model files stream through a **cache-through proxy** at
- `/dsh-voice-api/hf` and are mirrored to disk
- (`~/.cache/dsh-voice/models/`, configurable via `cacheDir`), so every
- browser/recognizer load after the first is served from local disk.
- Downloads resume from partial `.part` files when interrupted. Use
- `npm run prefetch` to warm the cache once.
- - RMS endpoint detection: 16kHz getUserMedia, 2s trailing-silence cutoff,
- max 30s segment, pre/post padding. Zero dependencies.
- - **Press-to-talk bypasses the VAD entirely.** Holding the key is already the
- intent, so every buffer between press and release is kept — gating on
- loudness there only drops quiet speech, which is indistinguishable from a
- broken button. Only captures below 250ms are discarded (mis-taps).
- - **Live caption**: while a hold is open the engine re-decodes the buffer
- every ~900ms and shows the interim text. SenseVoice is not a streaming
- model, so this is only requested while the overlay is actually on screen,
- and stops past 12s of audio. Interims are strictly previews: they never
- reach the composer draft, and an interim that lands after the release is
- dropped (epoch check) so it can never overwrite the final transcript.
- - Barge-in is three-layered: local playback queue cleared, host `TtsQueue`
- epoch bumped (queued AND in-flight synthesis dropped), and the running
- turn cancelled when `session.running` is true. An aborted turn never
- flushes its trailing half-sentence — exactly what the user interrupted.
- - `modelHost` accepts any HF-compatible mirror (e.g. `https://hf-mirror.com`
- for CN networks).
- ## API
- | Route | Purpose |
- |-------|---------|
- | `GET /dsh-voice-api/stream` | SSE; `event: audio` frames `{sessionId, seq, text, audio(base64 MP3)}` |
- | `POST /dsh-voice-api/asr` | raw little-endian f32 PCM body → `{text}` via SenseVoice |
- | `POST /dsh-voice-api/cancel` | `{sessionId}` drops queued + in-flight synthesis (epoch bump) |
- | `GET /dsh-voice-api/config` | ASR runtime config `{asr, basePath}`. Always served at this fixed path, so a client can discover a custom `basePath` |
- | `GET /dsh-voice-api/hf/*` | cache-through HF model proxy (mirrors to `cacheDir`) |
- | `GET /dsh-voice-api/*` | ping: `{ok, name, enabled}` |
- Config (bundle patch row):
- ```yaml
- - id: voice
- name: '@haoku123/dsh-voice'
- config:
- voice: zh-CN-XiaoxiaoNeural
- cacheDir: ~/.cache/dsh-voice/models # optional, on-disk model cache
- asr:
- model: csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17
- modelHost: https://huggingface.co # or https://hf-mirror.com
- language: auto # auto | zh | en | ja | ko | yue
- useItn: true # inverse text normalization
- autoSend: false
- mode: toggle # toggle | hold
- hotkey: Control # keyboard press-to-talk; '' disables
- ```
- Model files are fetched through the proxy on first use; warm the cache once
- with the dsh host running:
- ```sh
- npm run prefetch # uses http://127.0.0.1:3080 by default
- ```
- ## Install
- ```sh
- dsh plugin --profile web add <repo-url-or-path>
- dsh --profile web
- ```
- Note: needs Node ≥ 22.19 or ≥ 24 (`node:zlib` zstd APIs). Works with dsh
- 0.1.0-rc.6 and 0.2.x — the `@deepseek-ai/dsh-web` peer range covers both.
- ## Tests
- ```sh
- npm test # host suites + client component tests
- npm run test:host # segmenter / TTS backoff / barge-in / integration
- npm run test:client # vitest + jsdom
- node test/host.integration.test.mjs # llm/stream tap + real Edge TTS + SSE + /config
- node test/tts-retry.test.mjs # failed synthesis backs off instead of hot-looping
- node verify-client.mjs # client bundle registration/exports/slots/dynamic-import
- ```
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.