Skip to content
Back to search

lecture-to-notes

"Turn a lecture/conference recording (video or audio: MOV/MP4/M4A/MP3/WAV) into structured vault notes via local GPU transcription + slide extraction — 演講影片, 演講音檔, 上課錄影, '整理演講', '影片轉筆記', '音檔轉筆記', or a dropped media file. Handles batch runs."

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/drpwchen--lecture-to-notes/badge.svg)](/listings/drpwchen--lecture-to-notes)

HTML

<a href="/listings/drpwchen--lecture-to-notes"><img src="/api/listings/drpwchen--lecture-to-notes/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/drpwchen/lecture-to-notes
cd lecture-to-notes

pip install -r requirements.txt
pip install -r requirements-optional.txt     # recommended
cp config.example.yaml config.yaml           # every value is blank/optional by default

ollama pull minicpm-v:8b                     # optional, Stage D

# Put your material in one folder, then ask what to run:
python scripts/route_inputs.py /path/to/material_folder

`route_inputs.py` is the front door. It classifies the folder, prints the ordered commands for the right path, and lists the questions a human has to answer first. **It is plan-only** — it never runs anything and never writes a file. For agents and workflow engines, `--json` gives each step as an argv list (no shell parsing) with `network`, `gpu`, `reads`/`writes` and the `<LANG>`-style `placeholders` a human must fill before the step is `runnable`.

Then run the printed commands. The first one is transcription, and it will not start without `--lang`:

python scripts/transcribe_video.py "lecture.mp4" \
    --output-dir "$OUT_DIR" --lang <zh|en|bilingual|auto>

That is deliberate. There is no default language, because guessing wrong makes Whisper hallucinate fluent Chinese out of accented English and the transcript is unusable in a way that is not obvious until you read it.

> 🧋 Saved you some time? [Buy me a drink](https://drpwchen.com/en/support/).

**Related tools:** [textbook-to-note](https://github.com/drpwchen/textbook-to-note) (the same idea for textbooks) · [ask-your-library](https://github.com/drpwchen/ask-your-library) (search the finished notes) · [note-supplement](https://github.com/drpwchen/note-supplement) (merge new material into existing notes)

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 455 lines
  1. ---
  2. name: lecture-to-notes
  3. description: "Turn a lecture/conference recording (video or audio: MOV/MP4/M4A/MP3/WAV) into structured vault notes via local GPU transcription + slide extraction — 演講影片, 演講音檔, 上課錄影, '整理演講', '影片轉筆記', '音檔轉筆記', or a dropped media file. Handles batch runs."
  4. allowed-tools: Read Write Edit Bash Glob Grep Agent
  5. ---
  6. # Lecture-to-Notes
  7. Turn a lecture recording (video or audio-only) into structured notes. Every heavy
  8. stage runs locally at 0 Claude tokens; Claude only does the final synthesis. This
  9. page is the map; detail lives in `reference/`, one topic per file.
  10. | File | What is in it |
  11. |---|---|
  12. | `reference/pipeline.md` | Per-stage flags, thresholds, JSON schemas, timeouts, observability |
  13. | `reference/note-spec.md` | Note quality spec, tier scoring, width table, synthesis prompt requirements |
  14. | `reference/segmented-mode.md` | Multi-talk workshop folders → per-segment L2/L3 + Hub + web viewer |
  15. | `reference/multi-camera.md` | One long recording + many phone clips/photos → one timeline |
  16. | `reference/decisions.md` | Post-mortems, benchmarks, wrong turns, VRAM measurements |
  17. ## HARD RULES
  18. 1. ==ASK the user what language the speaker(s) used== (English / Mandarin /
  19. bilingual code-switching) before transcribing. There is no default and
  20. `transcribe_video.py` exits without `--lang`. A wrong guess makes Whisper
  21. hallucinate Chinese from accented English and the transcript is unusable.
  22. 2. ==Never skip Stage D (VLM) or Stage E (grounding)== for speed or for a
  23. deadline. ==The user has not set a deadline; do not invent one.== If a stage
  24. really is too slow (>2 h ETA), report the ETA and ask.
  25. 3. ==Never auto-correct the transcript.== Flag suspects, let synthesis resolve
  26. them. Both auto-correction passes ever built were measured and retired — see
  27. `reference/decisions.md#asr-auto-correction`. Do not add an auto-apply mode.
  28. 4. ==Do not bypass the collapse auto-retry.== `transcribe_video.py` auto-runs
  29. `retranscribe_segment.py --auto` on detected token-collapse. If collapses
  30. survive that, escalate (wider beam + `--no-repeat-ngram-size` + a glossary),
  31. never skip.
  32. 5. ==The VLM does not do OCR.== Stage D asks only for semantic signals. Text
  33. comes from Stage B `quick_text`, Stage B2 `clean_text`, or `pdf_text.json`.
  34. `ocr.vlm_text` is an always-empty compatibility field.
  35. 6. ==Serialize all GPU work.== Whisper and the VLM may not run concurrently on an
  36. 8 GB card, and frame extraction must not run alongside transcription.
  37. 7. ==On an 8 GB card, never exceed `--batch-size 4` with `--beam-size 10`==, and
  38. never combine `--beam-size 15` with sequential mode — that crashes
  39. (`0xC0000005`). These are memory ceilings for the anti-collapse retry, ==not a
  40. recommended first pass== — first pass is the bare defaults (Step 4).
  41. 8. ==Director batch dispatch==: for a multi-lecture batch, dispatch Steps 1–9 as
  42. one subagent and Step 10 synthesis as a separate fresh subagent, spawned only
  43. after `slides_grounded.json` exists. A single subagent bounces during the long
  44. GPU waits and burns 30+ min of wall time per lecture.
  45. 9. ==Order material by REAL CAPTURE TIME — never by filename, never by the
  46. printed agenda.== Filenames are labels, not clocks: a camcorder counter
  47. restarts across days (a two-day shoot has two `00000`), a recorder's
  48. `240526_1119.mp3` sorts into the middle of the video files, and on-site
  49. `-1-`/`-2-` labels get stuck on the wrong file. Run
  50. `scripts/batch/course_timeline.py` BEFORE segmenting any multi-source course;
  51. `manifest.json` clip order and `L1_coarse.md` section order must both be
  52. built from it. An agenda is not a clock either — the 2024-05 Conference-Y
  53. conference ran ~25 min early on day 1 and ~35 min late on day 2, while its
  54. break gaps matched to the minute.
  55. 10. ==🚫 PHI red line==: if the recording contains patient-identifiable content
  56. (case discussion, ward rounds, named patients), transcribe LOCAL ONLY — drop
  57. `--engine groq`. When unsure, ask; default to local. `LECTURE_LOCAL_ONLY=1`
  58. (or config `privacy.local_only: true`) makes the Groq and Gemini routes exit 2
  59. before uploading anything.
  60. ## Input types and routing
  61. ==Start here. `route_inputs.py` is the front door== — it classifies a folder and
  62. prints the ordered commands plus the questions a human must answer. It is
  63. plan-only: it never runs anything and never writes a file.
  64. ```bash
  65. python <skill-dir>/scripts/route_inputs.py <material_dir> [--recursive] [--out-dir DIR] [--json]
  66. ```
  67. | What is in the folder | Slide source | Route |
  68. |---|---|---|
  69. | Video, no deck | frames from the video | Path A — Steps 5–7 |
  70. | Audio/video **+ PDF deck** (==preferred==) | PDF text + page renders | Path B — `build_slides_from_pdf.py` |
  71. | Audio/video **+ loose slide images** (≥3) | the images themselves | Path B-images — `build_slides_from_images.py` |
  72. | Audio only, no deck | none | Path C — transcript-only note |
  73. | N-up handout PDF | cropped tiles | Path B-multi — `crop_multiup_pdf.py` first |
  74. | **Multi-talk workshop folder** | per segment | `reference/segmented-mode.md` |
  75. | One long recording + many phone clips/photos | per source | `reference/multi-camera.md` |
  76. | `.pptx` / `.docx` / `.key` | — | convert to PDF yourself first; there is no conversion step here |
  77. ==Multi-source contract==: when two or more independent sources are present,
  78. establish the timeline BEFORE anything else — `course_timeline.py <course_dir>`
  79. for a course folder with a manifest (it writes `_seg/real_timeline.json`, maps
  80. photos onto the recordings, and with `--reorder-manifest` fixes clip order at
  81. the root), or `media_capture_index.py --emit-alignment alignment.json` for a
  82. loose material folder.
  83. ==Capture timestamps are HYPOTHESES; transcript cross-correlation
  84. (`xcorr_media_offsets.py`) is EVIDENCE.== A source whose `reliable` flag is false
  85. got its start from mtime or has none — it must not be aligned on. Nothing is ever
  86. auto-corrected: a claimed-vs-measured disagreement >5 s is flagged
  87. `"conflict": true` for a human to judge. Details in `reference/multi-camera.md`.
  88. ## Pipeline
  89. One command plus its purpose per step; flags, thresholds and outputs are in
  90. `reference/pipeline.md`.
  91. ### Step 1 — Ask the language (mandatory, no command)
  92. English / Mandarin / bilingual? Accented speakers? Code-switching mid-sentence?
  93. Use AskUserQuestion if the user has not said. HARD RULE 1.
  94. ### Step 2 — Set up the lecture directory
  95. One directory per lecture holds every intermediate; name it
  96. `{date}_{speaker}_{topic}`, the shape `finalize_to_vault.py` parses.
  97. ### Step 3 — GPU pre-flight
  98. ```bash
  99. python <skill-dir>/scripts/gpu_check.py --out-dir "$OUT_DIR" --min-free-mb 6000
  100. ```
  101. Gate before transcription and again before Stage D. Exit `0` proceed, `1` warn
  102. and proceed, `2` blocked — surface it, ==do not retry in a loop==. A card whose
  103. *total* VRAM is under the threshold (2–4 GB laptops) is `GPU_TOO_SMALL`, exit
  104. `0`: not contention, nothing will free up — proceed with the CPU path
  105. (`transcribe_video.py --device cpu --model small`, or `--engine groq`).
  106. → `reference/pipeline.md#gpu-check`
  107. ### Step 4 — Transcribe
  108. ```bash
  109. python <skill-dir>/scripts/transcribe_video.py "<media>" \
  110. --output-dir "$OUT_DIR" --lang <zh|en|bilingual|auto>
  111. ```
  112. ==Bare defaults ARE the production config== (asr-benchmark 2026-07-12 n=15,
  113. `breeze25 / novad_nocond`: breeze25, beam 5, VAD off, conditioning off) — ==for
  114. English too==: breeze25 ties large-v3 on English audio, so do not switch models.
  115. Do not add `--beam-size 10` (loses recall, 1.7× time), `--initial-prompt` (invents
  116. near-miss words) or `--model large-v3` on a first pass, and ==do not copy flags
  117. from an older batch's run script==; the script prints a WARNING when you do.
  118. Those flags belong only in the anti-collapse retry recipe.
  119. Local faster-whisper by default; `--engine groq` is an optional offload (HARD
  120. RULE 9). Default model alias is `breeze25` (needs a local model dir); on a
  121. machine without one, pass `--model large-v3`, which faster-whisper downloads.
  122. Recordings over ~30 min go through the chunked runner instead.
  123. → `reference/pipeline.md#transcription`
  124. **Transcript already exists elsewhere** (MLX Whisper, a hosted STT, captions, a human
  125. transcript): import it instead of transcribing —
  126. `python <skill-dir>/scripts/import_transcript.py <file.srt|.vtt|.json> --out-dir "$OUT_DIR"
  127. [--engine NAME] [--model NAME]`. It writes the same `transcript.json`/`.txt`, never
  128. touches the source, refuses to overwrite without `--force`; the next step is Step 5.
  129. **Gemini as the primary transcript** (measured 2026-10-24: Gemini 97–100 % vs best local 64–85 %
  130. on 15 course clips in 3 categories, 82 % vs 54–60 % on far-field hall audio). Allowed only where the
  131. recording is non-PHI (HARD RULE 10) and ==the user has said yes to sending it out== — a project may
  132. record a standing yes in its own AGENTS.md; everywhere else ask per recording, and without a yes run
  133. the bare default above.
  134. ```bash
  135. python <skill-dir>/scripts/asr_gemini_primary.py "<media>" -o "$OUT_DIR" --lang <zh|en|bilingual> \
  136. [--start <ISO wall clock>] [--extra <args for the breeze25 fallback>]
  137. ```
  138. 1. `--model breeze25` (bare default, ~10× realtime) → `transcript_anchor.{txt,json}` — the clock ==and the
  139. cross-check==: Gemini swaps names/terms and contradicts slides now and then, so a writer checks names
  140. and numbers against this file. `--anchor-model turbo` (beam 1, ~56×) clocks as well but its words
  141. are too poor to check against — use it only when speed matters more.
  142. 2. `scripts/gemini_asr.py` sends 10-min chunks to Gemini (`--via api` = REST API key; `--via agy` =
  143. the Antigravity CLI in headless mode — ==read the terms-of-service note in its docstring first==)
  144. and re-anchors every line to the local clock (`ts_anchor.py`; a chunk < 50 % anchored is linearly
  145. rescaled instead). It also catches Gemini's silent failures: a chunk cut short (gap check against
  146. the local clock) and a repetition loop (resend, then drop the looping lines).
  147. 3. Chunks Gemini's filter refuses on every route ("blocked by Gemini's filters" — happens on medical
  148. talks) are transcribed by ==breeze25 for that chunk's range only== (`--fill-blocked breeze25`,
  149. cached in `*.part.fill`).
  150. 4. The result becomes `transcript.json`/`transcript.txt` (each JSON line has `source`: `gemini` |
  151. `breeze25`), so split_segments/build_L1/audit_note run unchanged; `transcript_source.json` records
  152. the path. Gemini failing outright (quota, network) → plain breeze25 for the whole file (old path).
  153. HARD RULE 3 still holds: Gemini is a different ASR source, not a correction pass — never edit its
  154. lines. Gemini is an LLM, so a number only it has (no slide, no second source) stays unconfirmed in
  155. the note. `gemini_asr.py` alone (with `--anchor`, no fill) remains the way to add Gemini as a
  156. ==second== source next to an existing breeze25 transcript (e.g. old courses, far-field halls).
  157. → `reference/decisions.md#gemini-primary`, `#far-field-hall`.
  158. **Far-field hall recordings** (audience seat, PA system, non-native speakers):
  159. - Where sending audio out is allowed: add Gemini as a source (above) — 82 % vs 54–60 % term hits.
  160. - Otherwise (offline): also run `--model large-v3` and Qwen3-ASR-1.7B, and give the writer
  161. all versions (65 % hits, mishearings 22 → 3).
  162. - Either way, ==a number found in only one source is unconfirmed== until a slide or a second
  163. source agrees — the same model gave 58 % vs 150 % for one figure across two runs.
  164. Audio pre-processing (band-pass, denoise, loudnorm, WPE) measured flat — don't.
  165. → `reference/decisions.md#far-field-hall`
  166. ### Step 5 — Stage A: frame extraction (Path A only)
  167. ```bash
  168. python <skill-dir>/scripts/extract_slides.py "<video>" --output-dir "$OUT_DIR" --interval 15
  169. ```
  170. Writes `slides/frame_NNNN.jpg` + `slides/timestamps.json`, phash-deduping adjacent
  171. near-identical frames. Path B/B-images skip this. → `reference/pipeline.md#stage-a`
  172. ### Step 6 — Stage B: quick OCR + entropy (Path A only)
  173. ```bash
  174. python <skill-dir>/scripts/quick_ocr.py "$OUT_DIR"
  175. ```
  176. RapidOCR on every frame → `slides_raw.json`. ==Required==: without it every slide
  177. looks decorative to the Stage D gate. → `reference/pipeline.md#stage-b`
  178. ### Step 6-alt — Path B / B-images bridge
  179. ```bash
  180. python <skill-dir>/scripts/build_slides_from_pdf.py "$OUT_DIR" [--audio-duration-sec N]
  181. python <skill-dir>/scripts/build_slides_from_images.py "<img_dir>" -o "$OUT_DIR" [--audio-duration-sec N]
  182. ```
  183. Either bridge emits `slides_raw.json` + `slides_dedup.json` directly, replacing
  184. Steps 5–7. → `reference/pipeline.md#path-b`
  185. ### Step 7 — Stage C: semantic dedup (Path A only)
  186. ```bash
  187. python <skill-dir>/scripts/dedup_semantic.py "$OUT_DIR"
  188. ```
  189. Merges adjacent frames by text-subset or layout similarity, marks
  190. `dedup.is_canonical`. Output `slides_dedup.json`.
  191. → `reference/pipeline.md#stage-c`
  192. ### Step 8 — Stage B2: high-quality OCR (Surya)
  193. ```bash
  194. python <skill-dir>/scripts/ocr_surya.py "$OUT_DIR" [--resume]
  195. ```
  196. Surya in its own venv on canonical text-bearing slides, RapidOCR as the shallow
  197. fallback. Adds `ocr.clean_text` / `ocr_engine` / `ocr_confidence`. ==Updates
  198. `slides_dedup.json` in place== (one-time backup `slides_dedup.pre_b2.json`) and
  199. writes `slides_ocr.json`. Path B skips it — `pdf_text` is already clean. Without
  200. a Surya venv it warns and routes everything to RapidOCR rather than failing.
  201. → `reference/pipeline.md#stage-b2`
  202. ### Step 9 — Stage D: VLM signals
  203. ```bash
  204. python <skill-dir>/scripts/vlm_signals.py "$OUT_DIR" --model minicpm-v:8b --num-ctx 4096
  205. ```
  206. Semantic signals per canonical slide, behind a 4-condition pre-skip gate for
  207. decorative frames. Re-check the GPU first (Step 3). Output `slides_vlm.json`.
  208. `scripts/ocr_slides.py` is a deprecated shim forwarding here, same argv and
  209. outputs. → `reference/pipeline.md#stage-d`
  210. ### Step 10 — Stage E: transcript grounding
  211. ```bash
  212. python <skill-dir>/scripts/ground_slides.py "$OUT_DIR"
  213. ```
  214. Pure Python, 0 LLM calls. Ties each canonical slide to the words spoken over it.
  215. Output `slides_grounded.json` — the input to synthesis.
  216. → `reference/pipeline.md#stage-e`
  217. ### Step 11 — Flag suspect ASR tokens
  218. ```bash
  219. python <skill-dir>/scripts/flag_asr_suspects.py --dir "$OUT_DIR"
  220. ```
  221. Runs HERE, after Stage E: the slide glossary it needs comes from
  222. `slides_grounded.json`. Writes `asr_suspects.txt`; ==the transcript is left
  223. byte-identical==. Treat each line as a question, never a substitution.
  224. → `reference/pipeline.md#asr-suspects`
  225. ### Step 12 — Chunked pre-summarization (long lectures only)
  226. Over ~30 min / 25 k tokens of transcript, offload chunk summaries to a Sonnet
  227. subagent instead of reading the whole transcript into main context. Coverage
  228. guards (`[CHUNK_END]`, `[CONTINUE_NEEDED]`, expected-chunk count) are mandatory.
  229. → `reference/pipeline.md#chunked-summarization`
  230. ### Step 13 — Stage F: synthesis (Claude)
  231. Two passes for batches and long lectures — ==Tier-pass then Write-pass==:
  232. - **Tier-pass subagent** reads `slides_grounded.json` + `transcript.txt` +
  233. `pdf_text.json`, applies the tier scoring rules, writes **only**
  234. `slides_final.json` (integer `tier`, `attachment_name`, `embed_width`,
  235. `section_suggestion`). This file is the frozen tier authority.
  236. - **Write-pass subagent** reads the frozen `slides_final.json` + transcript +
  237. slide text, writes `note_draft.md` with `[[EMBED sN]]` placeholders only — no
  238. paths, widths or callouts.
  239. One pass is fine for one short lecture; splitting them stops the writer from
  240. simplifying structure to make its own embed audit pass. → `reference/note-spec.md`
  241. (mandatory: quality spec, tier rules, prompt requirements)
  242. ### Step 14 — Render, finalize, audit
  243. ```bash
  244. python <skill-dir>/scripts/render_embeds.py "$OUT_DIR" --note note_draft.md --in-place
  245. python <skill-dir>/scripts/finalize_to_vault.py "$OUT_DIR" [--vault-root PATH]
  246. python <skill-dir>/scripts/audit_note.py "<note path>" --mode lecture --grounding "$OUT_DIR"
  247. ```
  248. `render_embeds.py` expands placeholders to col-0 callouts with path + width and
  249. audits Tier-1/2 coverage; `finalize_to_vault.py` copies cited slides + the note
  250. into the vault; the auditor is the gate. ==Always pass `--grounding`== — without
  251. it the caption↔frame check only warns. → `reference/note-spec.md`
  252. **Draft review exemption**: this output is machine-transcribed and synthesized —
  253. write to the inbox without showing a draft; the user reviews in Obsidian.
  254. ### Step 15 — Web viewer export (default whenever there is video)
  255. ```bash
  256. python <skill-dir>/scripts/build_single_talk_web.py "$OUT_DIR" --plan seg_plan.json
  257. # … Stage F writes the L3/ files … then:
  258. python <skill-dir>/scripts/build_single_talk_web.py "$OUT_DIR" --plan seg_plan.json --export
  259. ```
  260. A single talk gets the synced HTML viewer by default, not just workshops.
  261. ==Stage F authors the segment plan itself== (JSON list:
  262. seg/start/end/slug/title_zh, derived from the transcript + slide topics — no
  263. human segmentation needed); ==content sections = more segments, not more
  264. headings==, the viewer builds exactly two chapters per segment. The script assembles manifest + `segments.json` + L2 slices + HUB
  265. skeleton from `transcript.json`, and ==auto-appends an L3-only 全場總整理
  266. overview segment== (sorted first — many segments still need one place listing
  267. everything; the HUB never renders in the viewer). Stage F then writes one
  268. `L3_segNN_<slug>.md` per segment (==image embeds by basename only, timecodes
  269. `` `(V1 MM:SS)` ``==) plus the overview note — content shape picked by ==one
  270. question: is this a single body of knowledge?== (single talk → pearls +
  271. per-segment one-liners; same-lecturer arc → thematic reorganization with
  272. sources; multi-speaker workshop → ==catalog only, never forced synthesis== —
  273. spec in `reference/pipeline.md#web-export`); `--export` refuses while any L3
  274. is missing.
  275. Compression default is ==H.265 CRF 24== (self-use; `--codec h264` when sharing
  276. to machines you can't verify). Multi-talk workshops keep their own flow →
  277. `reference/segmented-mode.md`. → `reference/pipeline.md#web-export`
  278. ### Last step — Searchable corpus (optional, off by default)
  279. ```bash
  280. python <skill-dir>/scripts/export_corpus.py "<note path>" [more notes of the same course] \
  281. --course-dir "<course folder name>" [--replace]
  282. ```
  283. Runs only when `rag.corpus_root` (config.yaml) or `LECTURE_CORPUS_ROOT` is set or
  284. `--corpus-root` is given; otherwise it prints "RAG export is off" and writes nothing.
  285. Rule-based (no model): each 總整理 `##` → one entry with an H1 + `Source:` line,
  286. slide notes minus repeats, timecodes → `[cNN MM:SS]`, bookkeeping → `_meta.md`.
  287. Pass every note of one course in one call; a re-run needs `--replace`. Indexing is
  288. ask-your-library's job (`server/corpus.py add <root>`), not this skill's. → README
  289. "Make your lectures searchable".
  290. ## Edge cases
  291. - **Audio only, no deck** → transcript-only note using `# 逐段筆記` instead of
  292. `# 逐投影片筆記`.
  293. - **N-up handout PDF** → render one mid page and ==look at it== before deciding
  294. the grid; heuristics are unreliable on slide-heavy PDFs. Then
  295. `crop_multiup_pdf.py <pdf> <out_dir> --expected-rows R --expected-cols C`.
  296. Pages that are genuinely 1-up (title pages) are handled per page, not forced
  297. into the consensus grid.
  298. - **Very long lecture (>90 min)** → chunked runner for transcription, Step 12 for
  299. reading it. **Batch of recordings** → transcribe strictly sequentially; ~4.5 GB
  300. RAM per faster-whisper instance.
  301. - **One talk split across several files** → one note, not several.
  302. - **Slides English, speaker Mandarin** → keep both; the deck gives terms, the
  303. transcript gives the explanation.
  304. - **Speaker asked not to be recorded** → exclude that content.
  305. - **Dense text handout, not a slide deck** → primary source, but drop the
  306. slide-by-slide structure.
  307. - **CJK path failures** (exit 127 / 3221226505) → extract audio with ffmpeg
  308. separately first; the script reuses a validated `audio.wav`.
  309. ## Optional infrastructure
  310. Everything below is ==this machine's setup, not a requirement== — nothing in the
  311. pipeline depends on any of it, and the generic alternative is inline.
  312. | Used for | Generic alternative |
  313. |---|---|
  314. | `job_runner.py` wrapping long GPU jobs (tree-kills children on timeout) | plain `timeout <n> <cmd>`, or run in the foreground |
  315. | `gpu_lease.py` / a pause-flag file between concurrent batches | run GPU stages one at a time; leave `paths.pause_flag` empty in config |
  316. | `vault-search` / OpenEvidence / Zotero lookups during synthesis | skip; cite only what the lecture itself provided |
  317. | `ntfy` completion pings | skip |
  318. | external batch control plane (`run_queue`, `rerun_batch`, `clip_order`, dashboard) | course-specific, not shipped — see `reference/segmented-mode.md` |
  319. Vault paths (`99Attachment/lecture_{slug}`, the inbox folder) are ==a private vault
  320. convention== and configurable: `render_embeds.py --attach-root` / `--attach-dir`,
  321. `finalize_to_vault.py --vault-root`.
  322. ## Dependencies
  323. ```bash
  324. ollama pull minicpm-v:8b # ~5.5 GB Q4_K_M, Stage D
  325. pip install rapidfuzz rapidocr-onnxruntime scikit-image pyyaml json_repair pillow numpy
  326. ```
  327. `ffmpeg` + `ffprobe` on PATH. Surya (Stage B2) lives in its own venv; point
  328. `ocr_engine.surya_python` at it in `config.yaml`, or leave it blank to fall back
  329. to RapidOCR. Copy `config.example.yaml` → `config.yaml` on a new machine; every
  330. machine-specific value there is blank by default and env-overridable.
  331. ==Optional dependencies degrade loudly, not silently== — a missing scikit-image or
  332. RapidOCR is reported and gated, because a silent degrade produced *wrong* output
  333. rather than less output (`reference/decisions.md#optional-dependency-degradation`).
  334. ## File locations
  335. ```
  336. <skill-dir>/
  337. ├── SKILL.md, config.yaml, config.example.yaml
  338. ├── reference/ pipeline.md · note-spec.md · segmented-mode.md · multi-camera.md · decisions.md
  339. ├── data/ real_words.txt, real_acronyms.txt (regenerated, not committed)
  340. ├── ocr_bench/ engine benchmark harness (bring your own fixtures)
  341. └── scripts/
  342. ├── route_inputs.py front door — classifies material, prints the plan
  343. ├── transcribe_video.py retranscribe_segment.py gpu_check.py groq_asr.py gemini_asr.py
  344. ├── import_transcript.py external SRT / VTT / JSON -> transcript.json
  345. ├── provenance.py provenance.json record + `verify <dir>`
  346. ├── extract_slides.py quick_ocr.py dedup_semantic.py ocr_surya.py
  347. ├── vlm_signals.py (ocr_slides.py = deprecated shim → here)
  348. ├── build_slides_from_pdf.py build_slides_from_images.py crop_multiup_pdf.py
  349. ├── ground_slides.py flag_asr_suspects.py make_glossary.py build_real_words.py
  350. ├── render_embeds.py finalize_to_vault.py audit_note.py export_web.py
  351. ├── export_corpus.py optional: notes -> t2n-corpus/1 lecture corpus (last step, optional)
  352. ├── media_capture_index.py xcorr_media_offsets.py query_near_field.py
  353. ├── _mdpm.py AVCHD .MTS recording clock (no container tag)
  354. ├── adapters/ surya_adapter.py (production OCR adapters)
  355. ├── batch/ course_timeline (real-time ordering authority) ·
  356. │ audit_segmentation (is the proposal still valid?) ·
  357. │ build_L1 · split_segments · split_L1_by_segment · add_dhash ·
  358. │ vlm_cache · detect_language · detect_language_audio ·
  359. │ phi_mask · process_slide_deck (generic batch layer)
  360. ├── layout2/ viewer.css, viewer.js (web viewer assets, edited verbatim)
  361. └── _common.py _log.py _paths.py
  362. ```
  363. Per-lecture output directory:
  364. ```
  365. {lecture}/
  366. ├── metadata.json run_id + media fingerprint + per-stage status
  367. ├── provenance.json per artifact: stage, tool, engine, model, params, network, input hashes
  368. ├── transcript.json/.txt timestamped segments; .txt is [MM:SS] text, H:MM:SS past an hour
  369. ├── asr_suspects.txt flagged tokens — flags only, never a rewrite
  370. ├── alignment.json multi-source capture-start hypotheses (when applicable)
  371. ├── slides/ frame_NNNN.jpg | page_NN.jpg | original photo names
  372. ├── slides_raw.json Stage B quick text + density + entropy
  373. ├── slides_dedup.json Stage C canonical markers (Stage B2 updates in place)
  374. ├── slides_dedup.pre_b2.json one-time pre-Stage-B2 snapshot
  375. ├── slides_ocr.json Stage B2 Surya result, for inspection
  376. ├── slides_vlm.json Stage D VLM signals + vlm_skip + skip_metrics
  377. ├── slides_grounded.json Stage E transcript grounding + retrieval fields
  378. ├── slides_final.json Stage F tier + score + attachment_name + width
  379. └── logs/progress_*.jsonl per-stage event streams (not every stage emits one)
  380. ```
  381. Each stage output is a superset of the previous, so you can re-run one stage
  382. without redoing transcription or frame extraction. `python <skill-dir>/scripts/provenance.py
  383. verify "$OUT_DIR"` lists artifacts that are `stale` (an input changed after they were
  384. built — re-run that stage) or `modified` (edited after their stage wrote them). Stage F
  385. records itself with `provenance.py record` (see the public repo's `docs/stage-f-contract.md`).
  386. Names taken from data files (plan slugs, `attachment_name`, embed targets, slide
  387. filenames) must stay inside the directory given on the command line — a stage exits 2
  388. on one that does not; fix the data, never the check. `runs.jsonl` one level up
  389. carries one summary line per stage run, joined by `run_id`. Keep the intermediates
  390. — they are how tier decisions get debugged.
README.md · README · 382 lines
  1. # lecture-to-notes
  2. [![CI](https://github.com/drpwchen/lecture-to-notes/actions/workflows/secret-scan.yml/badge.svg)](https://github.com/drpwchen/lecture-to-notes/actions/workflows/secret-scan.yml)
  3. **English** · [繁體中文](README.zh-TW.md)
  4. ![lecture-to-notes — turn any mix of video, audio, slides and photos into structured, grounded notes](docs/assets/hero.png)
  5. Turn a lecture or conference recording into structured, slide-illustrated notes —
  6. and into a **synced HTML viewer** where the video, the timestamped transcript,
  7. and the curated summary sit on one page: the video highlights the matching note
  8. as it plays, and clicking any note timestamp seeks the video.
  9. Every expensive stage runs **locally**: Whisper ASR on your GPU, frame
  10. extraction, OCR, and a local vision model for slide semantics. An LLM is used
  11. only at the end, to write prose from evidence the pipeline already assembled.
  12. The design goal is not "summarize a video". It is **traceability**: every claim
  13. in the finished note should be attachable to a moment in the transcript and to
  14. the slide that was on screen at that moment. Most of the machinery here exists
  15. to make that link trustworthy rather than plausible.
  16. ## Why this exists
  17. Rehabilitation-medicine courses are notoriously hard to take notes on. Manual
  18. therapy, ultrasound scanning — the knowledge is in the *motion*, so at every
  19. course you see a forest of tripods: everyone records everything, planning to
  20. rewatch over dinner. Nobody actually rewatches hours of video.
  21. With AI the first instinct was to teach it precise screenshotting, to squeeze
  22. the video back into a traditional text-plus-figures note. The turn was realizing
  23. a note doesn't have to be that shape at all: build a webpage that ties the video
  24. to the transcript, and you can jump straight to the moment you care about. What
  25. you actually want to know is *how the maneuver is performed* — and that only
  26. lives in motion. So the summary is for studying, the transcript is for
  27. verifying, and the video is one click away from both.
  28. ## Requirements
  29. - **Tested on Windows 11** with an NVIDIA RTX 3070 Ti (8 GB). Linux and macOS are untested.
  30. - **Python 3.12** and **`ffmpeg` / `ffprobe` on PATH** (not optional).
  31. - **An NVIDIA GPU (8 GB+) is the fast path, not a requirement.** Without one, transcription falls back to CPU
  32. automatically and prints an ETA first; a 60-minute lecture then takes hours instead of minutes.
  33. - **Optional:** [ollama](https://ollama.com) with `minicpm-v:8b` (Stage D slide semantics), a separate venv with
  34. [Surya](https://github.com/VikParuchuri/surya) for OCR, `pandoc` for web/PDF export.
  35. - **Cost:** Stages A-E run locally and cost nothing beyond electricity. The final note-writing step (Stage F)
  36. uses an LLM through your own agent or API. Hosted ASR (Groq, Gemini) is opt-in, billed to your key, and sends
  37. audio off your machine.
  38. Small GPUs, hosted-ASR offload, importing an external transcript and Apple Silicon: see
  39. [docs/requirements.md](docs/requirements.md).
  40. ## Quickstart
  41. ```bash
  42. git clone https://github.com/drpwchen/lecture-to-notes
  43. cd lecture-to-notes
  44. pip install -r requirements.txt
  45. pip install -r requirements-optional.txt # recommended
  46. cp config.example.yaml config.yaml # every value is blank/optional by default
  47. ollama pull minicpm-v:8b # optional, Stage D
  48. # Put your material in one folder, then ask what to run:
  49. python scripts/route_inputs.py /path/to/material_folder
  50. ```
  51. `route_inputs.py` is the front door. It classifies the folder, prints the ordered
  52. commands for the right path, and lists the questions a human has to answer
  53. first. **It is plan-only** — it never runs anything and never writes a file.
  54. For agents and workflow engines, `--json` gives each step as an argv list (no
  55. shell parsing) with `network`, `gpu`, `reads`/`writes` and the `<LANG>`-style
  56. `placeholders` a human must fill before the step is `runnable`.
  57. Then run the printed commands. The first one is transcription, and it will not
  58. start without `--lang`:
  59. ```bash
  60. python scripts/transcribe_video.py "lecture.mp4" \
  61. --output-dir "$OUT_DIR" --lang <zh|en|bilingual|auto>
  62. ```
  63. That is deliberate. There is no default language, because guessing wrong makes
  64. Whisper hallucinate fluent Chinese out of accented English and the transcript is
  65. unusable in a way that is not obvious until you read it.
  66. > 🧋 Saved you some time? [Buy me a drink](https://drpwchen.com/en/support/).
  67. **Related tools:** [textbook-to-note](https://github.com/drpwchen/textbook-to-note) (the same idea for textbooks) ·
  68. [ask-your-library](https://github.com/drpwchen/ask-your-library) (search the finished notes) ·
  69. [note-supplement](https://github.com/drpwchen/note-supplement) (merge new material into existing notes)
  70. ## Bring whatever you captured
  71. Real courses never produce one tidy recording. Some talks you filmed, some you
  72. only audio-recorded, for some you photographed the slides with your phone, and
  73. sometimes the organizer hands out a PDF deck afterwards. This pipeline takes
  74. the folder **as-is**: `route_inputs.py` figures out what role each file plays,
  75. and the alignment stage stitches every source onto **one shared timeline** —
  76. so the finished note cites the right slide at the right moment even when that
  77. slide never appeared inside the video.
  78. ---
  79. ## What it does
  80. ```mermaid
  81. flowchart TD
  82. IN["material folder<br/>(video / audio / PDF deck / slide images)"]
  83. IN --> R["route_inputs.py<br/><i>classifies the folder, prints the plan</i>"]
  84. R --> T["transcribe_video.py<br/>faster-whisper, local GPU"]
  85. R --> S{"slide source"}
  86. S -->|video only| A["extract_slides.py<br/>frames + perceptual dedup"]
  87. S -->|PDF deck| P["build_slides_from_pdf.py<br/>page render + embedded text"]
  88. S -->|loose images| I["build_slides_from_images.py"]
  89. A --> B["quick_ocr.py<br/>Stage B — cheap OCR triage"]
  90. B --> C["dedup_semantic.py<br/>Stage C — canonical slides"]
  91. C --> B2["ocr_surya.py<br/>Stage B2 — high-quality OCR"]
  92. P --> D
  93. I --> D
  94. B2 --> D["vlm_signals.py<br/>Stage D — slide semantics (local VLM)"]
  95. D --> E["ground_slides.py<br/>Stage E — tie slides to spoken words"]
  96. T --> E
  97. E --> F["Stage F — synthesis (LLM)<br/>tier pass, then write pass"]
  98. F --> O["render_embeds.py → finalize_to_vault.py → audit_note.py"]
  99. F --> W["export_web.py<br/><b>synced HTML viewer</b> — video + transcript + summary on one page"]
  100. ```
  101. Stages A–E are plain Python and cost zero LLM tokens. They produce
  102. `slides_grounded.json`: for each canonical slide, its OCR text, its semantic
  103. signals, and the transcript segments spoken while it was on screen. That file is
  104. the input to synthesis, and it is also readable on its own — if you never run
  105. Stage F you still have a transcript, a deduplicated slide set, and the mapping
  106. between them.
  107. ## The HTML viewer: video, transcript and summary on one page
  108. The signature output. `export_web.py` builds a **single self-contained HTML
  109. page** per course where the video, the timestamped transcript, and the curated
  110. summary notes are presented **together and kept in sync both ways**
  111. (it also moves each referenced clip into the page's `媒體/影片/` folder — a rename
  112. on the same volume — and rewrites `manifest.json` to match, keeping a `.bak`; pass
  113. `--media-in-place` to leave clips where they are):
  114. - **Video → notes**: as the video plays, the matching note bullet
  115. auto-highlights and scrolls into view.
  116. - **Notes → video**: every bullet carries a `(Vn MM:SS)` timestamp — click it
  117. and the video seeks to that moment.
  118. - **Three read modes**: summary only, transcript only, or both side by side —
  119. the summary layer (curated, workflow-ordered) and the transcript layer
  120. (verbatim, time-ordered) are separate chapters of the same timeline, so you
  121. can study top-down and verify bottom-up without leaving the page.
  122. - **Resizable panes, sidebar segment navigation**, slide images inline where
  123. they were shown.
  124. - **Offline and shareable**: one `.html` plus one support folder (browser-ready
  125. clips, friendly-named slides, markdown + PDF copies). No server, no build
  126. step for the reader — send the folder, they double-click the page.
  127. `--compress` produces a smaller H.264 set for handing around.
  128. - **Notes are data, not markup**: transcript, slide OCR and synthesis text is
  129. re-serialised through an HTML allowlist (a transcript that says `<script>`
  130. shows those characters), the embedded timeline JSON cannot end its `<script>`,
  131. and a Content-Security-Policy pins the page's one inline script by hash and
  132. allows no network fetches. Tested against a hostile fixture course in a real
  133. browser (`tests/test_viewer_hostile.py`).
  134. Real output from a cervical-ultrasound workshop (dates and speaker names
  135. blurred):
  136. ![Synced view — summary and transcript index with a draggable floating player; the highlighted bullet follows the video](docs/assets/viewer-sync.png)
  137. | Course hub — every segment with duration and a one-line hook | Summary view — per-video source table and key pearls |
  138. |---|---|
  139. | ![Course hub page](docs/assets/viewer-home.png) | ![Summary view](docs/assets/viewer-summary.png) |
  140. The viewer UI lives in `scripts/layout2/` (`viewer.css`, `viewer.js`);
  141. `export_web.py` only generates the per-course timeline manifest and the synced
  142. note HTML, so a UI tweak is an asset edit, not a generator change.
  143. ## Markdown notes for your vault
  144. The HTML viewer is the reading surface; **plain markdown is the storage
  145. format**, so everything also lands in your note vault:
  146. - `finalize_to_vault.py` ships the finished note plus its cited slide images
  147. into an Obsidian-style vault (attachment folder and inbox are configurable
  148. flags — any folder of markdown works).
  149. - The viewer's support folder contains a `markdown/` copy of every transcript
  150. and summary, wikilink-rewritten so the folder itself **opens directly as an
  151. Obsidian vault** — same content as the webpage, readable and searchable in
  152. your normal note workflow, plus PDF copies for people who want neither.
  153. So you get three durable forms of the same lecture: the synced webpage for
  154. studying, markdown for your knowledge base, PDF for handing to anyone.
  155. ## Make your lectures searchable (optional)
  156. Off by default. If you never want search, skip this section — nothing in the
  157. pipeline changes.
  158. A note written for reading is the wrong shape for a retrieval index (one long
  159. file, frontmatter, callouts, embedded slides). `scripts/export_corpus.py` turns
  160. one course's finished notes into a **lecture corpus folder** — one plain
  161. Markdown file per topic, each starting with an H1 and a `Source:` line — by
  162. rules alone: no model, no API key, same notes in → same bytes out.
  163. ```bash
  164. python scripts/export_corpus.py "<vault>/00Inbox/20250301_Example_Shoulder US basics.md" \
  165. --corpus-root ~/lecture-corpus --course-dir 20250301_example_shoulder-US
  166. ```
  167. ```
  168. ~/lecture-corpus/
  169. ├── corpus.json {"format": "t2n-corpus/1", "kind": "lecture"}
  170. └── 20250301_example_shoulder-US/
  171. ├── _meta.md speakers, recordings (c01, c02…), note → entry map, abbreviations
  172. ├── 01_Biceps_groove_landmark.md
  173. └── 02_….md
  174. ```
  175. ```markdown
  176. # Biceps groove landmark
  177. Source: 2025-03-01 Dr. Example《Shoulder ultrasound basics》· 20250301_example_shoulder-US · c01
  178. - Start in short axis over the bicipital groove [c01 03:21]
  179. ```
  180. - Each `##` section of `# 總整理` becomes an entry (short ones merged, long ones
  181. split at `###`); slide notes follow, minus every line 總整理 already has.
  182. `--summary-only` exports 總整理 alone.
  183. - Timecodes become `[cNN MM:SS]`, `cNN` being a recording listed in `_meta.md`.
  184. A line without a timecode stays without one — nothing is invented.
  185. - Several notes of one course (day 1, day 2, several speakers) go in one call;
  186. each keeps its own date and speaker in its `Source:` line.
  187. - Re-running for the same course needs `--replace`. `--dry-run` prints the plan.
  188. To make it the last step of every run, set `rag.corpus_root` in `config.yaml`
  189. (or `LECTURE_CORPUS_ROOT`); without either the script prints "RAG export is off"
  190. and writes nothing.
  191. **Indexing is not part of this repo.** The folder follows the
  192. [`t2n-corpus/1` format](https://github.com/drpwchen/ask-your-library/blob/main/docs/corpus-format.md);
  193. [ask-your-library](https://github.com/drpwchen/ask-your-library) (formerly *vault-search*) indexes it in one command:
  194. ```bash
  195. python <ask-your-library>/server/corpus.py check ~/lecture-corpus
  196. python <ask-your-library>/server/corpus.py add ~/lecture-corpus
  197. ```
  198. What a rule-based export cannot do: regroup content into new topics, spell
  199. out abbreviations in the text (they are listed in `_meta.md`), convert
  200. Simplified Chinese ASR leftovers, or add a timecode the note never had.
  201. Speaker and title come from the note's frontmatter and title line; for notes
  202. that lack them, pass `--speaker` / `--title` / `--date`.
  203. ## Alignment: capture time is a hypothesis, cross-correlation is evidence
  204. The part worth stealing, if you take nothing else.
  205. When a talk is captured by more than one device — a room recording plus phone
  206. clips plus photos — you need a shared timeline. The obvious approach is to trust
  207. each file's capture timestamp. That approach is wrong often enough to matter:
  208. phone clocks drift, some files only have an mtime, and a recording that was
  209. stopped and restarted lies about its own start.
  210. So this pipeline separates the two:
  211. - **`media_capture_index.py`** reads capture times and emits them as *claims*,
  212. each with a `reliable` flag. A source whose start came from mtime, or has no
  213. start at all, is marked unreliable and must not be aligned on.
  214. - **`xcorr_media_offsets.py`** measures the actual offset by cross-correlating
  215. transcripts of the overlapping audio. That is evidence.
  216. - When claim and measurement disagree by more than 5 seconds, the pipeline sets
  217. `"conflict": true` and **stops**. It never auto-corrects.
  218. The failure this prevents is a 44-minute misalignment that looks completely
  219. normal in the output, because every downstream stage faithfully processes the
  220. wrong pairing. A pipeline that silently reconciles conflicting evidence produces
  221. confident garbage; one that flags the conflict produces a question.
  222. ## Is this a Claude Code skill or a set of scripts?
  223. Both, and the distinction matters for what you get.
  224. The repo is laid out as a [Claude Code](https://claude.com/claude-code) skill:
  225. `SKILL.md` is the agent-facing map, and `reference/` holds the detailed specs the
  226. agent reads on demand. Drop the tree into `~/.claude/skills/lecture-to-notes/`
  227. and an agent can drive the whole thing.
  228. **As plain CLI scripts, Stages 0–E work standalone** and give you the transcript,
  229. the canonical slide set, the OCR text, the VLM signals, and the grounding map.
  230. That is most of the value and all of the local compute.
  231. **Stage F — synthesis — is prompt-driven.** It is a specification (in
  232. `reference/note-spec.md`) for what a competent writer should do with
  233. `slides_grounded.json`, not a script that calls an API. There is no
  234. `synthesize.py` you can run. If you are not using an agent driver, treat
  235. `slides_grounded.json` as the handoff point and write your own Stage F against
  236. whatever model you prefer — the spec tells you what the output has to satisfy,
  237. and `audit_note.py` mechanically checks it. [`docs/stage-f-contract.md`](docs/stage-f-contract.md)
  238. is that contract without any provider in it: inputs, the output files and their
  239. keys, and which script checks what afterwards.
  240. ## Limitations
  241. - Only tested on Windows 11 with one 8 GB NVIDIA card; Linux, macOS and Apple Silicon are untested.
  242. - Without a GPU, transcription runs but is slow (hours per lecture).
  243. - On an 8 GB card, GPU stages must run one at a time (Whisper and the VLM cannot overlap).
  244. - The transcript is never auto-corrected; suspect tokens are only flagged, so names and numbers still need a human check.
  245. - Hosted ASR (Groq, Gemini) sends audio off your machine and is for non-confidential recordings only; no stage
  246. falls back from local to cloud by itself.
  247. - Defaults still show their origin (Mandarin Zoom UI dedup list, Traditional Chinese note headings, medical word lists).
  248. They are config, not code.
  249. ## Repo layout
  250. | Path | What is in it |
  251. |---|---|
  252. | `SKILL.md` | The map: hard rules, the numbered pipeline, edge cases |
  253. | `reference/pipeline.md` | Per-stage flags, thresholds, JSON schemas, timeouts |
  254. | `reference/note-spec.md` | Output note spec, slide tier scoring, synthesis prompt requirements |
  255. | `reference/segmented-mode.md` | Multi-talk workshop folders → per-segment notes + hub |
  256. | `reference/multi-camera.md` | One recording + many clips/photos → one timeline |
  257. | `reference/decisions.md` | Post-mortems, benchmarks, wrong turns, VRAM measurements |
  258. | `scripts/` | The pipeline (see `SKILL.md` for what each stage does) |
  259. | `scripts/batch/` | Generic batch layer: segment splitting, language detection, caching |
  260. | `scripts/layout2/` | Web viewer assets for `export_web.py` |
  261. | `tests/` | Dependency-light tests (`pytest`), incl. a hostile-content viewer fixture |
  262. | `pyproject.toml` | Metadata + test config; dependency lists mirror `requirements*.txt` |
  263. | `ocr_bench/` | OCR engine A/B/C harness — bring your own fixtures |
  264. | `data/` | Word/acronym frequency lists used to flag suspect ASR tokens |
  265. | `docs/AUDIT_SUMMARY.md` | What the pre-release audit found, and two things it got wrong |
  266. | `docs/transcript-contract.md` | What `transcript.json` must contain; `import_transcript.py` for an external ASR |
  267. | `docs/stage-f-contract.md` | Stage F (synthesis) inputs, outputs and gates, for any LLM |
  268. | `scripts/export_corpus.py` | Optional: finished notes → a searchable lecture corpus folder (`t2n-corpus/1`) |
  269. | `SECURITY.md` | How to report a vulnerability privately |
  270. | `tools/sync_from_skill.py` | Maintenance: re-export from the upstream skill tree |
  271. ## Design notes
  272. The invariants to know before changing anything (transcript never auto-corrected, VLM does no OCR, optional dependencies degrade loudly, provenance on every artifact, no data-derived paths) are in [docs/design-notes.md](docs/design-notes.md).
  273. ## Provenance and defaults
  274. This was extracted from a personal note-taking pipeline built for physical
  275. medicine and rehabilitation lectures, and some defaults still show it: the
  276. example config's dedup token list targets Mandarin-language Zoom UI chrome,
  277. the note templates use Traditional Chinese section headings, and the shipped
  278. word lists lean medical. All of it is config, not code — see
  279. `config.example.yaml` and `reference/note-spec.md`.
  280. The two files in `data/` are frequency lists of ordinary dictionary words and
  281. acronyms compiled from a local reference corpus, used only to decide whether an
  282. ASR token looks like a real word. Regenerate them from your own corpus with
  283. `scripts/build_real_words.py` if you want lists tuned to your domain.
  284. ## 🌱 Start here if you're new to AI agents
  285. This pipeline is one piece of my personal AI workflow. If you want to learn how
  286. to use AI agents like Claude Code from zero (no programming background needed),
  287. I wrote a beginner series (in Traditional Chinese):
  288. 1. [從零開始:安裝、看懂 GitHub、跑起你的第一個工具](https://drpwchen.com/posts/getting-started/)
  289. 2. [怎麼跟 AI agent 講話:心法、元技能與規則檔](https://drpwchen.com/posts/talking-to-agents/)
  290. 3. [自動化流程不是設計出來的,是長出來的](https://drpwchen.com/posts/growing-your-workflow/)
  291. The story behind this particular tool →
  292. [演講影片變成筆記:本機 GPU 轉錄 + 投影片對位](https://drpwchen.com/posts/lecture-to-notes/)
  293. Full map of my tools and posts → [drpwchen.com/map](https://drpwchen.com/map/)
  294. ## License
  295. MIT — see [LICENSE](LICENSE).
  296. ## Support 支持
  297. 覺得這個工具有幫助嗎?歡迎[請我喝飲料](https://drpwchen.com/support/) 🧋
  298. If this tool helped you, you can [buy me a drink](https://drpwchen.com/en/support/).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.