lecture-to-notes
"Turn a lecture/conference recording (video or audio: MOV/MP4/M4A/MP3/WAV) into structured vault notes via local GPU transcription + slide extraction — 演講影片, 演講音檔, 上課錄影, '整理演講', '影片轉筆記', '音檔轉筆記', or a dropped media file. Handles batch runs."
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/drpwchen--lecture-to-notes)HTML
<a href="/listings/drpwchen--lecture-to-notes"><img src="/api/listings/drpwchen--lecture-to-notes/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
git clone https://github.com/drpwchen/lecture-to-notes cd lecture-to-notes pip install -r requirements.txt pip install -r requirements-optional.txt # recommended cp config.example.yaml config.yaml # every value is blank/optional by default ollama pull minicpm-v:8b # optional, Stage D # Put your material in one folder, then ask what to run: python scripts/route_inputs.py /path/to/material_folder
`route_inputs.py` is the front door. It classifies the folder, prints the ordered commands for the right path, and lists the questions a human has to answer first. **It is plan-only** — it never runs anything and never writes a file. For agents and workflow engines, `--json` gives each step as an argv list (no shell parsing) with `network`, `gpu`, `reads`/`writes` and the `<LANG>`-style `placeholders` a human must fill before the step is `runnable`.
Then run the printed commands. The first one is transcription, and it will not start without `--lang`:
python scripts/transcribe_video.py "lecture.mp4" \
--output-dir "$OUT_DIR" --lang <zh|en|bilingual|auto>That is deliberate. There is no default language, because guessing wrong makes Whisper hallucinate fluent Chinese out of accented English and the transcript is unusable in a way that is not obvious until you read it.
> 🧋 Saved you some time? [Buy me a drink](https://drpwchen.com/en/support/).
**Related tools:** [textbook-to-note](https://github.com/drpwchen/textbook-to-note) (the same idea for textbooks) · [ask-your-library](https://github.com/drpwchen/ask-your-library) (search the finished notes) · [note-supplement](https://github.com/drpwchen/note-supplement) (merge new material into existing notes)
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 455 lines
- ---
- name: lecture-to-notes
- description: "Turn a lecture/conference recording (video or audio: MOV/MP4/M4A/MP3/WAV) into structured vault notes via local GPU transcription + slide extraction — 演講影片, 演講音檔, 上課錄影, '整理演講', '影片轉筆記', '音檔轉筆記', or a dropped media file. Handles batch runs."
- allowed-tools: Read Write Edit Bash Glob Grep Agent
- ---
- # Lecture-to-Notes
- Turn a lecture recording (video or audio-only) into structured notes. Every heavy
- stage runs locally at 0 Claude tokens; Claude only does the final synthesis. This
- page is the map; detail lives in `reference/`, one topic per file.
- | File | What is in it |
- |---|---|
- | `reference/pipeline.md` | Per-stage flags, thresholds, JSON schemas, timeouts, observability |
- | `reference/note-spec.md` | Note quality spec, tier scoring, width table, synthesis prompt requirements |
- | `reference/segmented-mode.md` | Multi-talk workshop folders → per-segment L2/L3 + Hub + web viewer |
- | `reference/multi-camera.md` | One long recording + many phone clips/photos → one timeline |
- | `reference/decisions.md` | Post-mortems, benchmarks, wrong turns, VRAM measurements |
- ## HARD RULES
- 1. ==ASK the user what language the speaker(s) used== (English / Mandarin /
- bilingual code-switching) before transcribing. There is no default and
- `transcribe_video.py` exits without `--lang`. A wrong guess makes Whisper
- hallucinate Chinese from accented English and the transcript is unusable.
- 2. ==Never skip Stage D (VLM) or Stage E (grounding)== for speed or for a
- deadline. ==The user has not set a deadline; do not invent one.== If a stage
- really is too slow (>2 h ETA), report the ETA and ask.
- 3. ==Never auto-correct the transcript.== Flag suspects, let synthesis resolve
- them. Both auto-correction passes ever built were measured and retired — see
- `reference/decisions.md#asr-auto-correction`. Do not add an auto-apply mode.
- 4. ==Do not bypass the collapse auto-retry.== `transcribe_video.py` auto-runs
- `retranscribe_segment.py --auto` on detected token-collapse. If collapses
- survive that, escalate (wider beam + `--no-repeat-ngram-size` + a glossary),
- never skip.
- 5. ==The VLM does not do OCR.== Stage D asks only for semantic signals. Text
- comes from Stage B `quick_text`, Stage B2 `clean_text`, or `pdf_text.json`.
- `ocr.vlm_text` is an always-empty compatibility field.
- 6. ==Serialize all GPU work.== Whisper and the VLM may not run concurrently on an
- 8 GB card, and frame extraction must not run alongside transcription.
- 7. ==On an 8 GB card, never exceed `--batch-size 4` with `--beam-size 10`==, and
- never combine `--beam-size 15` with sequential mode — that crashes
- (`0xC0000005`). These are memory ceilings for the anti-collapse retry, ==not a
- recommended first pass== — first pass is the bare defaults (Step 4).
- 8. ==Director batch dispatch==: for a multi-lecture batch, dispatch Steps 1–9 as
- one subagent and Step 10 synthesis as a separate fresh subagent, spawned only
- after `slides_grounded.json` exists. A single subagent bounces during the long
- GPU waits and burns 30+ min of wall time per lecture.
- 9. ==Order material by REAL CAPTURE TIME — never by filename, never by the
- printed agenda.== Filenames are labels, not clocks: a camcorder counter
- restarts across days (a two-day shoot has two `00000`), a recorder's
- `240526_1119.mp3` sorts into the middle of the video files, and on-site
- `-1-`/`-2-` labels get stuck on the wrong file. Run
- `scripts/batch/course_timeline.py` BEFORE segmenting any multi-source course;
- `manifest.json` clip order and `L1_coarse.md` section order must both be
- built from it. An agenda is not a clock either — the 2024-05 Conference-Y
- conference ran ~25 min early on day 1 and ~35 min late on day 2, while its
- break gaps matched to the minute.
- 10. ==🚫 PHI red line==: if the recording contains patient-identifiable content
- (case discussion, ward rounds, named patients), transcribe LOCAL ONLY — drop
- `--engine groq`. When unsure, ask; default to local. `LECTURE_LOCAL_ONLY=1`
- (or config `privacy.local_only: true`) makes the Groq and Gemini routes exit 2
- before uploading anything.
- ## Input types and routing
- ==Start here. `route_inputs.py` is the front door== — it classifies a folder and
- prints the ordered commands plus the questions a human must answer. It is
- plan-only: it never runs anything and never writes a file.
- ```bash
- python <skill-dir>/scripts/route_inputs.py <material_dir> [--recursive] [--out-dir DIR] [--json]
- ```
- | What is in the folder | Slide source | Route |
- |---|---|---|
- | Video, no deck | frames from the video | Path A — Steps 5–7 |
- | Audio/video **+ PDF deck** (==preferred==) | PDF text + page renders | Path B — `build_slides_from_pdf.py` |
- | Audio/video **+ loose slide images** (≥3) | the images themselves | Path B-images — `build_slides_from_images.py` |
- | Audio only, no deck | none | Path C — transcript-only note |
- | N-up handout PDF | cropped tiles | Path B-multi — `crop_multiup_pdf.py` first |
- | **Multi-talk workshop folder** | per segment | `reference/segmented-mode.md` |
- | One long recording + many phone clips/photos | per source | `reference/multi-camera.md` |
- | `.pptx` / `.docx` / `.key` | — | convert to PDF yourself first; there is no conversion step here |
- ==Multi-source contract==: when two or more independent sources are present,
- establish the timeline BEFORE anything else — `course_timeline.py <course_dir>`
- for a course folder with a manifest (it writes `_seg/real_timeline.json`, maps
- photos onto the recordings, and with `--reorder-manifest` fixes clip order at
- the root), or `media_capture_index.py --emit-alignment alignment.json` for a
- loose material folder.
- ==Capture timestamps are HYPOTHESES; transcript cross-correlation
- (`xcorr_media_offsets.py`) is EVIDENCE.== A source whose `reliable` flag is false
- got its start from mtime or has none — it must not be aligned on. Nothing is ever
- auto-corrected: a claimed-vs-measured disagreement >5 s is flagged
- `"conflict": true` for a human to judge. Details in `reference/multi-camera.md`.
- ## Pipeline
- One command plus its purpose per step; flags, thresholds and outputs are in
- `reference/pipeline.md`.
- ### Step 1 — Ask the language (mandatory, no command)
- English / Mandarin / bilingual? Accented speakers? Code-switching mid-sentence?
- Use AskUserQuestion if the user has not said. HARD RULE 1.
- ### Step 2 — Set up the lecture directory
- One directory per lecture holds every intermediate; name it
- `{date}_{speaker}_{topic}`, the shape `finalize_to_vault.py` parses.
- ### Step 3 — GPU pre-flight
- ```bash
- python <skill-dir>/scripts/gpu_check.py --out-dir "$OUT_DIR" --min-free-mb 6000
- ```
- Gate before transcription and again before Stage D. Exit `0` proceed, `1` warn
- and proceed, `2` blocked — surface it, ==do not retry in a loop==. A card whose
- *total* VRAM is under the threshold (2–4 GB laptops) is `GPU_TOO_SMALL`, exit
- `0`: not contention, nothing will free up — proceed with the CPU path
- (`transcribe_video.py --device cpu --model small`, or `--engine groq`).
- → `reference/pipeline.md#gpu-check`
- ### Step 4 — Transcribe
- ```bash
- python <skill-dir>/scripts/transcribe_video.py "<media>" \
- --output-dir "$OUT_DIR" --lang <zh|en|bilingual|auto>
- ```
- ==Bare defaults ARE the production config== (asr-benchmark 2026-07-12 n=15,
- `breeze25 / novad_nocond`: breeze25, beam 5, VAD off, conditioning off) — ==for
- English too==: breeze25 ties large-v3 on English audio, so do not switch models.
- Do not add `--beam-size 10` (loses recall, 1.7× time), `--initial-prompt` (invents
- near-miss words) or `--model large-v3` on a first pass, and ==do not copy flags
- from an older batch's run script==; the script prints a WARNING when you do.
- Those flags belong only in the anti-collapse retry recipe.
- Local faster-whisper by default; `--engine groq` is an optional offload (HARD
- RULE 9). Default model alias is `breeze25` (needs a local model dir); on a
- machine without one, pass `--model large-v3`, which faster-whisper downloads.
- Recordings over ~30 min go through the chunked runner instead.
- → `reference/pipeline.md#transcription`
- **Transcript already exists elsewhere** (MLX Whisper, a hosted STT, captions, a human
- transcript): import it instead of transcribing —
- `python <skill-dir>/scripts/import_transcript.py <file.srt|.vtt|.json> --out-dir "$OUT_DIR"
- [--engine NAME] [--model NAME]`. It writes the same `transcript.json`/`.txt`, never
- touches the source, refuses to overwrite without `--force`; the next step is Step 5.
- **Gemini as the primary transcript** (measured 2026-10-24: Gemini 97–100 % vs best local 64–85 %
- on 15 course clips in 3 categories, 82 % vs 54–60 % on far-field hall audio). Allowed only where the
- recording is non-PHI (HARD RULE 10) and ==the user has said yes to sending it out== — a project may
- record a standing yes in its own AGENTS.md; everywhere else ask per recording, and without a yes run
- the bare default above.
- ```bash
- python <skill-dir>/scripts/asr_gemini_primary.py "<media>" -o "$OUT_DIR" --lang <zh|en|bilingual> \
- [--start <ISO wall clock>] [--extra <args for the breeze25 fallback>]
- ```
- 1. `--model breeze25` (bare default, ~10× realtime) → `transcript_anchor.{txt,json}` — the clock ==and the
- cross-check==: Gemini swaps names/terms and contradicts slides now and then, so a writer checks names
- and numbers against this file. `--anchor-model turbo` (beam 1, ~56×) clocks as well but its words
- are too poor to check against — use it only when speed matters more.
- 2. `scripts/gemini_asr.py` sends 10-min chunks to Gemini (`--via api` = REST API key; `--via agy` =
- the Antigravity CLI in headless mode — ==read the terms-of-service note in its docstring first==)
- and re-anchors every line to the local clock (`ts_anchor.py`; a chunk < 50 % anchored is linearly
- rescaled instead). It also catches Gemini's silent failures: a chunk cut short (gap check against
- the local clock) and a repetition loop (resend, then drop the looping lines).
- 3. Chunks Gemini's filter refuses on every route ("blocked by Gemini's filters" — happens on medical
- talks) are transcribed by ==breeze25 for that chunk's range only== (`--fill-blocked breeze25`,
- cached in `*.part.fill`).
- 4. The result becomes `transcript.json`/`transcript.txt` (each JSON line has `source`: `gemini` |
- `breeze25`), so split_segments/build_L1/audit_note run unchanged; `transcript_source.json` records
- the path. Gemini failing outright (quota, network) → plain breeze25 for the whole file (old path).
- HARD RULE 3 still holds: Gemini is a different ASR source, not a correction pass — never edit its
- lines. Gemini is an LLM, so a number only it has (no slide, no second source) stays unconfirmed in
- the note. `gemini_asr.py` alone (with `--anchor`, no fill) remains the way to add Gemini as a
- ==second== source next to an existing breeze25 transcript (e.g. old courses, far-field halls).
- → `reference/decisions.md#gemini-primary`, `#far-field-hall`.
- **Far-field hall recordings** (audience seat, PA system, non-native speakers):
- - Where sending audio out is allowed: add Gemini as a source (above) — 82 % vs 54–60 % term hits.
- - Otherwise (offline): also run `--model large-v3` and Qwen3-ASR-1.7B, and give the writer
- all versions (65 % hits, mishearings 22 → 3).
- - Either way, ==a number found in only one source is unconfirmed== until a slide or a second
- source agrees — the same model gave 58 % vs 150 % for one figure across two runs.
- Audio pre-processing (band-pass, denoise, loudnorm, WPE) measured flat — don't.
- → `reference/decisions.md#far-field-hall`
- ### Step 5 — Stage A: frame extraction (Path A only)
- ```bash
- python <skill-dir>/scripts/extract_slides.py "<video>" --output-dir "$OUT_DIR" --interval 15
- ```
- Writes `slides/frame_NNNN.jpg` + `slides/timestamps.json`, phash-deduping adjacent
- near-identical frames. Path B/B-images skip this. → `reference/pipeline.md#stage-a`
- ### Step 6 — Stage B: quick OCR + entropy (Path A only)
- ```bash
- python <skill-dir>/scripts/quick_ocr.py "$OUT_DIR"
- ```
- RapidOCR on every frame → `slides_raw.json`. ==Required==: without it every slide
- looks decorative to the Stage D gate. → `reference/pipeline.md#stage-b`
- ### Step 6-alt — Path B / B-images bridge
- ```bash
- python <skill-dir>/scripts/build_slides_from_pdf.py "$OUT_DIR" [--audio-duration-sec N]
- python <skill-dir>/scripts/build_slides_from_images.py "<img_dir>" -o "$OUT_DIR" [--audio-duration-sec N]
- ```
- Either bridge emits `slides_raw.json` + `slides_dedup.json` directly, replacing
- Steps 5–7. → `reference/pipeline.md#path-b`
- ### Step 7 — Stage C: semantic dedup (Path A only)
- ```bash
- python <skill-dir>/scripts/dedup_semantic.py "$OUT_DIR"
- ```
- Merges adjacent frames by text-subset or layout similarity, marks
- `dedup.is_canonical`. Output `slides_dedup.json`.
- → `reference/pipeline.md#stage-c`
- ### Step 8 — Stage B2: high-quality OCR (Surya)
- ```bash
- python <skill-dir>/scripts/ocr_surya.py "$OUT_DIR" [--resume]
- ```
- Surya in its own venv on canonical text-bearing slides, RapidOCR as the shallow
- fallback. Adds `ocr.clean_text` / `ocr_engine` / `ocr_confidence`. ==Updates
- `slides_dedup.json` in place== (one-time backup `slides_dedup.pre_b2.json`) and
- writes `slides_ocr.json`. Path B skips it — `pdf_text` is already clean. Without
- a Surya venv it warns and routes everything to RapidOCR rather than failing.
- → `reference/pipeline.md#stage-b2`
- ### Step 9 — Stage D: VLM signals
- ```bash
- python <skill-dir>/scripts/vlm_signals.py "$OUT_DIR" --model minicpm-v:8b --num-ctx 4096
- ```
- Semantic signals per canonical slide, behind a 4-condition pre-skip gate for
- decorative frames. Re-check the GPU first (Step 3). Output `slides_vlm.json`.
- `scripts/ocr_slides.py` is a deprecated shim forwarding here, same argv and
- outputs. → `reference/pipeline.md#stage-d`
- ### Step 10 — Stage E: transcript grounding
- ```bash
- python <skill-dir>/scripts/ground_slides.py "$OUT_DIR"
- ```
- Pure Python, 0 LLM calls. Ties each canonical slide to the words spoken over it.
- Output `slides_grounded.json` — the input to synthesis.
- → `reference/pipeline.md#stage-e`
- ### Step 11 — Flag suspect ASR tokens
- ```bash
- python <skill-dir>/scripts/flag_asr_suspects.py --dir "$OUT_DIR"
- ```
- Runs HERE, after Stage E: the slide glossary it needs comes from
- `slides_grounded.json`. Writes `asr_suspects.txt`; ==the transcript is left
- byte-identical==. Treat each line as a question, never a substitution.
- → `reference/pipeline.md#asr-suspects`
- ### Step 12 — Chunked pre-summarization (long lectures only)
- Over ~30 min / 25 k tokens of transcript, offload chunk summaries to a Sonnet
- subagent instead of reading the whole transcript into main context. Coverage
- guards (`[CHUNK_END]`, `[CONTINUE_NEEDED]`, expected-chunk count) are mandatory.
- → `reference/pipeline.md#chunked-summarization`
- ### Step 13 — Stage F: synthesis (Claude)
- Two passes for batches and long lectures — ==Tier-pass then Write-pass==:
- - **Tier-pass subagent** reads `slides_grounded.json` + `transcript.txt` +
- `pdf_text.json`, applies the tier scoring rules, writes **only**
- `slides_final.json` (integer `tier`, `attachment_name`, `embed_width`,
- `section_suggestion`). This file is the frozen tier authority.
- - **Write-pass subagent** reads the frozen `slides_final.json` + transcript +
- slide text, writes `note_draft.md` with `[[EMBED sN]]` placeholders only — no
- paths, widths or callouts.
- One pass is fine for one short lecture; splitting them stops the writer from
- simplifying structure to make its own embed audit pass. → `reference/note-spec.md`
- (mandatory: quality spec, tier rules, prompt requirements)
- ### Step 14 — Render, finalize, audit
- ```bash
- python <skill-dir>/scripts/render_embeds.py "$OUT_DIR" --note note_draft.md --in-place
- python <skill-dir>/scripts/finalize_to_vault.py "$OUT_DIR" [--vault-root PATH]
- python <skill-dir>/scripts/audit_note.py "<note path>" --mode lecture --grounding "$OUT_DIR"
- ```
- `render_embeds.py` expands placeholders to col-0 callouts with path + width and
- audits Tier-1/2 coverage; `finalize_to_vault.py` copies cited slides + the note
- into the vault; the auditor is the gate. ==Always pass `--grounding`== — without
- it the caption↔frame check only warns. → `reference/note-spec.md`
- **Draft review exemption**: this output is machine-transcribed and synthesized —
- write to the inbox without showing a draft; the user reviews in Obsidian.
- ### Step 15 — Web viewer export (default whenever there is video)
- ```bash
- python <skill-dir>/scripts/build_single_talk_web.py "$OUT_DIR" --plan seg_plan.json
- # … Stage F writes the L3/ files … then:
- python <skill-dir>/scripts/build_single_talk_web.py "$OUT_DIR" --plan seg_plan.json --export
- ```
- A single talk gets the synced HTML viewer by default, not just workshops.
- ==Stage F authors the segment plan itself== (JSON list:
- seg/start/end/slug/title_zh, derived from the transcript + slide topics — no
- human segmentation needed); ==content sections = more segments, not more
- headings==, the viewer builds exactly two chapters per segment. The script assembles manifest + `segments.json` + L2 slices + HUB
- skeleton from `transcript.json`, and ==auto-appends an L3-only 全場總整理
- overview segment== (sorted first — many segments still need one place listing
- everything; the HUB never renders in the viewer). Stage F then writes one
- `L3_segNN_<slug>.md` per segment (==image embeds by basename only, timecodes
- `` `(V1 MM:SS)` ``==) plus the overview note — content shape picked by ==one
- question: is this a single body of knowledge?== (single talk → pearls +
- per-segment one-liners; same-lecturer arc → thematic reorganization with
- sources; multi-speaker workshop → ==catalog only, never forced synthesis== —
- spec in `reference/pipeline.md#web-export`); `--export` refuses while any L3
- is missing.
- Compression default is ==H.265 CRF 24== (self-use; `--codec h264` when sharing
- to machines you can't verify). Multi-talk workshops keep their own flow →
- `reference/segmented-mode.md`. → `reference/pipeline.md#web-export`
- ### Last step — Searchable corpus (optional, off by default)
- ```bash
- python <skill-dir>/scripts/export_corpus.py "<note path>" [more notes of the same course] \
- --course-dir "<course folder name>" [--replace]
- ```
- Runs only when `rag.corpus_root` (config.yaml) or `LECTURE_CORPUS_ROOT` is set or
- `--corpus-root` is given; otherwise it prints "RAG export is off" and writes nothing.
- Rule-based (no model): each 總整理 `##` → one entry with an H1 + `Source:` line,
- slide notes minus repeats, timecodes → `[cNN MM:SS]`, bookkeeping → `_meta.md`.
- Pass every note of one course in one call; a re-run needs `--replace`. Indexing is
- ask-your-library's job (`server/corpus.py add <root>`), not this skill's. → README
- "Make your lectures searchable".
- ## Edge cases
- - **Audio only, no deck** → transcript-only note using `# 逐段筆記` instead of
- `# 逐投影片筆記`.
- - **N-up handout PDF** → render one mid page and ==look at it== before deciding
- the grid; heuristics are unreliable on slide-heavy PDFs. Then
- `crop_multiup_pdf.py <pdf> <out_dir> --expected-rows R --expected-cols C`.
- Pages that are genuinely 1-up (title pages) are handled per page, not forced
- into the consensus grid.
- - **Very long lecture (>90 min)** → chunked runner for transcription, Step 12 for
- reading it. **Batch of recordings** → transcribe strictly sequentially; ~4.5 GB
- RAM per faster-whisper instance.
- - **One talk split across several files** → one note, not several.
- - **Slides English, speaker Mandarin** → keep both; the deck gives terms, the
- transcript gives the explanation.
- - **Speaker asked not to be recorded** → exclude that content.
- - **Dense text handout, not a slide deck** → primary source, but drop the
- slide-by-slide structure.
- - **CJK path failures** (exit 127 / 3221226505) → extract audio with ffmpeg
- separately first; the script reuses a validated `audio.wav`.
- ## Optional infrastructure
- Everything below is ==this machine's setup, not a requirement== — nothing in the
- pipeline depends on any of it, and the generic alternative is inline.
- | Used for | Generic alternative |
- |---|---|
- | `job_runner.py` wrapping long GPU jobs (tree-kills children on timeout) | plain `timeout <n> <cmd>`, or run in the foreground |
- | `gpu_lease.py` / a pause-flag file between concurrent batches | run GPU stages one at a time; leave `paths.pause_flag` empty in config |
- | `vault-search` / OpenEvidence / Zotero lookups during synthesis | skip; cite only what the lecture itself provided |
- | `ntfy` completion pings | skip |
- | external batch control plane (`run_queue`, `rerun_batch`, `clip_order`, dashboard) | course-specific, not shipped — see `reference/segmented-mode.md` |
- Vault paths (`99Attachment/lecture_{slug}`, the inbox folder) are ==a private vault
- convention== and configurable: `render_embeds.py --attach-root` / `--attach-dir`,
- `finalize_to_vault.py --vault-root`.
- ## Dependencies
- ```bash
- ollama pull minicpm-v:8b # ~5.5 GB Q4_K_M, Stage D
- pip install rapidfuzz rapidocr-onnxruntime scikit-image pyyaml json_repair pillow numpy
- ```
- `ffmpeg` + `ffprobe` on PATH. Surya (Stage B2) lives in its own venv; point
- `ocr_engine.surya_python` at it in `config.yaml`, or leave it blank to fall back
- to RapidOCR. Copy `config.example.yaml` → `config.yaml` on a new machine; every
- machine-specific value there is blank by default and env-overridable.
- ==Optional dependencies degrade loudly, not silently== — a missing scikit-image or
- RapidOCR is reported and gated, because a silent degrade produced *wrong* output
- rather than less output (`reference/decisions.md#optional-dependency-degradation`).
- ## File locations
- ```
- <skill-dir>/
- ├── SKILL.md, config.yaml, config.example.yaml
- ├── reference/ pipeline.md · note-spec.md · segmented-mode.md · multi-camera.md · decisions.md
- ├── data/ real_words.txt, real_acronyms.txt (regenerated, not committed)
- ├── ocr_bench/ engine benchmark harness (bring your own fixtures)
- └── scripts/
- ├── route_inputs.py front door — classifies material, prints the plan
- ├── transcribe_video.py retranscribe_segment.py gpu_check.py groq_asr.py gemini_asr.py
- ├── import_transcript.py external SRT / VTT / JSON -> transcript.json
- ├── provenance.py provenance.json record + `verify <dir>`
- ├── extract_slides.py quick_ocr.py dedup_semantic.py ocr_surya.py
- ├── vlm_signals.py (ocr_slides.py = deprecated shim → here)
- ├── build_slides_from_pdf.py build_slides_from_images.py crop_multiup_pdf.py
- ├── ground_slides.py flag_asr_suspects.py make_glossary.py build_real_words.py
- ├── render_embeds.py finalize_to_vault.py audit_note.py export_web.py
- ├── export_corpus.py optional: notes -> t2n-corpus/1 lecture corpus (last step, optional)
- ├── media_capture_index.py xcorr_media_offsets.py query_near_field.py
- ├── _mdpm.py AVCHD .MTS recording clock (no container tag)
- ├── adapters/ surya_adapter.py (production OCR adapters)
- ├── batch/ course_timeline (real-time ordering authority) ·
- │ audit_segmentation (is the proposal still valid?) ·
- │ build_L1 · split_segments · split_L1_by_segment · add_dhash ·
- │ vlm_cache · detect_language · detect_language_audio ·
- │ phi_mask · process_slide_deck (generic batch layer)
- ├── layout2/ viewer.css, viewer.js (web viewer assets, edited verbatim)
- └── _common.py _log.py _paths.py
- ```
- Per-lecture output directory:
- ```
- {lecture}/
- ├── metadata.json run_id + media fingerprint + per-stage status
- ├── provenance.json per artifact: stage, tool, engine, model, params, network, input hashes
- ├── transcript.json/.txt timestamped segments; .txt is [MM:SS] text, H:MM:SS past an hour
- ├── asr_suspects.txt flagged tokens — flags only, never a rewrite
- ├── alignment.json multi-source capture-start hypotheses (when applicable)
- ├── slides/ frame_NNNN.jpg | page_NN.jpg | original photo names
- ├── slides_raw.json Stage B quick text + density + entropy
- ├── slides_dedup.json Stage C canonical markers (Stage B2 updates in place)
- ├── slides_dedup.pre_b2.json one-time pre-Stage-B2 snapshot
- ├── slides_ocr.json Stage B2 Surya result, for inspection
- ├── slides_vlm.json Stage D VLM signals + vlm_skip + skip_metrics
- ├── slides_grounded.json Stage E transcript grounding + retrieval fields
- ├── slides_final.json Stage F tier + score + attachment_name + width
- └── logs/progress_*.jsonl per-stage event streams (not every stage emits one)
- ```
- Each stage output is a superset of the previous, so you can re-run one stage
- without redoing transcription or frame extraction. `python <skill-dir>/scripts/provenance.py
- verify "$OUT_DIR"` lists artifacts that are `stale` (an input changed after they were
- built — re-run that stage) or `modified` (edited after their stage wrote them). Stage F
- records itself with `provenance.py record` (see the public repo's `docs/stage-f-contract.md`).
- Names taken from data files (plan slugs, `attachment_name`, embed targets, slide
- filenames) must stay inside the directory given on the command line — a stage exits 2
- on one that does not; fix the data, never the check. `runs.jsonl` one level up
- carries one summary line per stage run, joined by `run_id`. Keep the intermediates
- — they are how tier decisions get debugged.
README.md · README · 382 lines
- # lecture-to-notes
- [](https://github.com/drpwchen/lecture-to-notes/actions/workflows/secret-scan.yml)
- **English** · [繁體中文](README.zh-TW.md)
- 
- Turn a lecture or conference recording into structured, slide-illustrated notes —
- and into a **synced HTML viewer** where the video, the timestamped transcript,
- and the curated summary sit on one page: the video highlights the matching note
- as it plays, and clicking any note timestamp seeks the video.
- Every expensive stage runs **locally**: Whisper ASR on your GPU, frame
- extraction, OCR, and a local vision model for slide semantics. An LLM is used
- only at the end, to write prose from evidence the pipeline already assembled.
- The design goal is not "summarize a video". It is **traceability**: every claim
- in the finished note should be attachable to a moment in the transcript and to
- the slide that was on screen at that moment. Most of the machinery here exists
- to make that link trustworthy rather than plausible.
- ## Why this exists
- Rehabilitation-medicine courses are notoriously hard to take notes on. Manual
- therapy, ultrasound scanning — the knowledge is in the *motion*, so at every
- course you see a forest of tripods: everyone records everything, planning to
- rewatch over dinner. Nobody actually rewatches hours of video.
- With AI the first instinct was to teach it precise screenshotting, to squeeze
- the video back into a traditional text-plus-figures note. The turn was realizing
- a note doesn't have to be that shape at all: build a webpage that ties the video
- to the transcript, and you can jump straight to the moment you care about. What
- you actually want to know is *how the maneuver is performed* — and that only
- lives in motion. So the summary is for studying, the transcript is for
- verifying, and the video is one click away from both.
- ## Requirements
- - **Tested on Windows 11** with an NVIDIA RTX 3070 Ti (8 GB). Linux and macOS are untested.
- - **Python 3.12** and **`ffmpeg` / `ffprobe` on PATH** (not optional).
- - **An NVIDIA GPU (8 GB+) is the fast path, not a requirement.** Without one, transcription falls back to CPU
- automatically and prints an ETA first; a 60-minute lecture then takes hours instead of minutes.
- - **Optional:** [ollama](https://ollama.com) with `minicpm-v:8b` (Stage D slide semantics), a separate venv with
- [Surya](https://github.com/VikParuchuri/surya) for OCR, `pandoc` for web/PDF export.
- - **Cost:** Stages A-E run locally and cost nothing beyond electricity. The final note-writing step (Stage F)
- uses an LLM through your own agent or API. Hosted ASR (Groq, Gemini) is opt-in, billed to your key, and sends
- audio off your machine.
- Small GPUs, hosted-ASR offload, importing an external transcript and Apple Silicon: see
- [docs/requirements.md](docs/requirements.md).
- ## Quickstart
- ```bash
- git clone https://github.com/drpwchen/lecture-to-notes
- cd lecture-to-notes
- pip install -r requirements.txt
- pip install -r requirements-optional.txt # recommended
- cp config.example.yaml config.yaml # every value is blank/optional by default
- ollama pull minicpm-v:8b # optional, Stage D
- # Put your material in one folder, then ask what to run:
- python scripts/route_inputs.py /path/to/material_folder
- ```
- `route_inputs.py` is the front door. It classifies the folder, prints the ordered
- commands for the right path, and lists the questions a human has to answer
- first. **It is plan-only** — it never runs anything and never writes a file.
- For agents and workflow engines, `--json` gives each step as an argv list (no
- shell parsing) with `network`, `gpu`, `reads`/`writes` and the `<LANG>`-style
- `placeholders` a human must fill before the step is `runnable`.
- Then run the printed commands. The first one is transcription, and it will not
- start without `--lang`:
- ```bash
- python scripts/transcribe_video.py "lecture.mp4" \
- --output-dir "$OUT_DIR" --lang <zh|en|bilingual|auto>
- ```
- That is deliberate. There is no default language, because guessing wrong makes
- Whisper hallucinate fluent Chinese out of accented English and the transcript is
- unusable in a way that is not obvious until you read it.
- > 🧋 Saved you some time? [Buy me a drink](https://drpwchen.com/en/support/).
- **Related tools:** [textbook-to-note](https://github.com/drpwchen/textbook-to-note) (the same idea for textbooks) ·
- [ask-your-library](https://github.com/drpwchen/ask-your-library) (search the finished notes) ·
- [note-supplement](https://github.com/drpwchen/note-supplement) (merge new material into existing notes)
- ## Bring whatever you captured
- Real courses never produce one tidy recording. Some talks you filmed, some you
- only audio-recorded, for some you photographed the slides with your phone, and
- sometimes the organizer hands out a PDF deck afterwards. This pipeline takes
- the folder **as-is**: `route_inputs.py` figures out what role each file plays,
- and the alignment stage stitches every source onto **one shared timeline** —
- so the finished note cites the right slide at the right moment even when that
- slide never appeared inside the video.
- ---
- ## What it does
- ```mermaid
- flowchart TD
- IN["material folder<br/>(video / audio / PDF deck / slide images)"]
- IN --> R["route_inputs.py<br/><i>classifies the folder, prints the plan</i>"]
- R --> T["transcribe_video.py<br/>faster-whisper, local GPU"]
- R --> S{"slide source"}
- S -->|video only| A["extract_slides.py<br/>frames + perceptual dedup"]
- S -->|PDF deck| P["build_slides_from_pdf.py<br/>page render + embedded text"]
- S -->|loose images| I["build_slides_from_images.py"]
- A --> B["quick_ocr.py<br/>Stage B — cheap OCR triage"]
- B --> C["dedup_semantic.py<br/>Stage C — canonical slides"]
- C --> B2["ocr_surya.py<br/>Stage B2 — high-quality OCR"]
- P --> D
- I --> D
- B2 --> D["vlm_signals.py<br/>Stage D — slide semantics (local VLM)"]
- D --> E["ground_slides.py<br/>Stage E — tie slides to spoken words"]
- T --> E
- E --> F["Stage F — synthesis (LLM)<br/>tier pass, then write pass"]
- F --> O["render_embeds.py → finalize_to_vault.py → audit_note.py"]
- F --> W["export_web.py<br/><b>synced HTML viewer</b> — video + transcript + summary on one page"]
- ```
- Stages A–E are plain Python and cost zero LLM tokens. They produce
- `slides_grounded.json`: for each canonical slide, its OCR text, its semantic
- signals, and the transcript segments spoken while it was on screen. That file is
- the input to synthesis, and it is also readable on its own — if you never run
- Stage F you still have a transcript, a deduplicated slide set, and the mapping
- between them.
- ## The HTML viewer: video, transcript and summary on one page
- The signature output. `export_web.py` builds a **single self-contained HTML
- page** per course where the video, the timestamped transcript, and the curated
- summary notes are presented **together and kept in sync both ways**
- (it also moves each referenced clip into the page's `媒體/影片/` folder — a rename
- on the same volume — and rewrites `manifest.json` to match, keeping a `.bak`; pass
- `--media-in-place` to leave clips where they are):
- - **Video → notes**: as the video plays, the matching note bullet
- auto-highlights and scrolls into view.
- - **Notes → video**: every bullet carries a `(Vn MM:SS)` timestamp — click it
- and the video seeks to that moment.
- - **Three read modes**: summary only, transcript only, or both side by side —
- the summary layer (curated, workflow-ordered) and the transcript layer
- (verbatim, time-ordered) are separate chapters of the same timeline, so you
- can study top-down and verify bottom-up without leaving the page.
- - **Resizable panes, sidebar segment navigation**, slide images inline where
- they were shown.
- - **Offline and shareable**: one `.html` plus one support folder (browser-ready
- clips, friendly-named slides, markdown + PDF copies). No server, no build
- step for the reader — send the folder, they double-click the page.
- `--compress` produces a smaller H.264 set for handing around.
- - **Notes are data, not markup**: transcript, slide OCR and synthesis text is
- re-serialised through an HTML allowlist (a transcript that says `<script>`
- shows those characters), the embedded timeline JSON cannot end its `<script>`,
- and a Content-Security-Policy pins the page's one inline script by hash and
- allows no network fetches. Tested against a hostile fixture course in a real
- browser (`tests/test_viewer_hostile.py`).
- Real output from a cervical-ultrasound workshop (dates and speaker names
- blurred):
- 
- | Course hub — every segment with duration and a one-line hook | Summary view — per-video source table and key pearls |
- |---|---|
- |  |  |
- The viewer UI lives in `scripts/layout2/` (`viewer.css`, `viewer.js`);
- `export_web.py` only generates the per-course timeline manifest and the synced
- note HTML, so a UI tweak is an asset edit, not a generator change.
- ## Markdown notes for your vault
- The HTML viewer is the reading surface; **plain markdown is the storage
- format**, so everything also lands in your note vault:
- - `finalize_to_vault.py` ships the finished note plus its cited slide images
- into an Obsidian-style vault (attachment folder and inbox are configurable
- flags — any folder of markdown works).
- - The viewer's support folder contains a `markdown/` copy of every transcript
- and summary, wikilink-rewritten so the folder itself **opens directly as an
- Obsidian vault** — same content as the webpage, readable and searchable in
- your normal note workflow, plus PDF copies for people who want neither.
- So you get three durable forms of the same lecture: the synced webpage for
- studying, markdown for your knowledge base, PDF for handing to anyone.
- ## Make your lectures searchable (optional)
- Off by default. If you never want search, skip this section — nothing in the
- pipeline changes.
- A note written for reading is the wrong shape for a retrieval index (one long
- file, frontmatter, callouts, embedded slides). `scripts/export_corpus.py` turns
- one course's finished notes into a **lecture corpus folder** — one plain
- Markdown file per topic, each starting with an H1 and a `Source:` line — by
- rules alone: no model, no API key, same notes in → same bytes out.
- ```bash
- python scripts/export_corpus.py "<vault>/00Inbox/20250301_Example_Shoulder US basics.md" \
- --corpus-root ~/lecture-corpus --course-dir 20250301_example_shoulder-US
- ```
- ```
- ~/lecture-corpus/
- ├── corpus.json {"format": "t2n-corpus/1", "kind": "lecture"}
- └── 20250301_example_shoulder-US/
- ├── _meta.md speakers, recordings (c01, c02…), note → entry map, abbreviations
- ├── 01_Biceps_groove_landmark.md
- └── 02_….md
- ```
- ```markdown
- # Biceps groove landmark
- Source: 2025-03-01 Dr. Example《Shoulder ultrasound basics》· 20250301_example_shoulder-US · c01
- - Start in short axis over the bicipital groove [c01 03:21]
- ```
- - Each `##` section of `# 總整理` becomes an entry (short ones merged, long ones
- split at `###`); slide notes follow, minus every line 總整理 already has.
- `--summary-only` exports 總整理 alone.
- - Timecodes become `[cNN MM:SS]`, `cNN` being a recording listed in `_meta.md`.
- A line without a timecode stays without one — nothing is invented.
- - Several notes of one course (day 1, day 2, several speakers) go in one call;
- each keeps its own date and speaker in its `Source:` line.
- - Re-running for the same course needs `--replace`. `--dry-run` prints the plan.
- To make it the last step of every run, set `rag.corpus_root` in `config.yaml`
- (or `LECTURE_CORPUS_ROOT`); without either the script prints "RAG export is off"
- and writes nothing.
- **Indexing is not part of this repo.** The folder follows the
- [`t2n-corpus/1` format](https://github.com/drpwchen/ask-your-library/blob/main/docs/corpus-format.md);
- [ask-your-library](https://github.com/drpwchen/ask-your-library) (formerly *vault-search*) indexes it in one command:
- ```bash
- python <ask-your-library>/server/corpus.py check ~/lecture-corpus
- python <ask-your-library>/server/corpus.py add ~/lecture-corpus
- ```
- What a rule-based export cannot do: regroup content into new topics, spell
- out abbreviations in the text (they are listed in `_meta.md`), convert
- Simplified Chinese ASR leftovers, or add a timecode the note never had.
- Speaker and title come from the note's frontmatter and title line; for notes
- that lack them, pass `--speaker` / `--title` / `--date`.
- ## Alignment: capture time is a hypothesis, cross-correlation is evidence
- The part worth stealing, if you take nothing else.
- When a talk is captured by more than one device — a room recording plus phone
- clips plus photos — you need a shared timeline. The obvious approach is to trust
- each file's capture timestamp. That approach is wrong often enough to matter:
- phone clocks drift, some files only have an mtime, and a recording that was
- stopped and restarted lies about its own start.
- So this pipeline separates the two:
- - **`media_capture_index.py`** reads capture times and emits them as *claims*,
- each with a `reliable` flag. A source whose start came from mtime, or has no
- start at all, is marked unreliable and must not be aligned on.
- - **`xcorr_media_offsets.py`** measures the actual offset by cross-correlating
- transcripts of the overlapping audio. That is evidence.
- - When claim and measurement disagree by more than 5 seconds, the pipeline sets
- `"conflict": true` and **stops**. It never auto-corrects.
- The failure this prevents is a 44-minute misalignment that looks completely
- normal in the output, because every downstream stage faithfully processes the
- wrong pairing. A pipeline that silently reconciles conflicting evidence produces
- confident garbage; one that flags the conflict produces a question.
- ## Is this a Claude Code skill or a set of scripts?
- Both, and the distinction matters for what you get.
- The repo is laid out as a [Claude Code](https://claude.com/claude-code) skill:
- `SKILL.md` is the agent-facing map, and `reference/` holds the detailed specs the
- agent reads on demand. Drop the tree into `~/.claude/skills/lecture-to-notes/`
- and an agent can drive the whole thing.
- **As plain CLI scripts, Stages 0–E work standalone** and give you the transcript,
- the canonical slide set, the OCR text, the VLM signals, and the grounding map.
- That is most of the value and all of the local compute.
- **Stage F — synthesis — is prompt-driven.** It is a specification (in
- `reference/note-spec.md`) for what a competent writer should do with
- `slides_grounded.json`, not a script that calls an API. There is no
- `synthesize.py` you can run. If you are not using an agent driver, treat
- `slides_grounded.json` as the handoff point and write your own Stage F against
- whatever model you prefer — the spec tells you what the output has to satisfy,
- and `audit_note.py` mechanically checks it. [`docs/stage-f-contract.md`](docs/stage-f-contract.md)
- is that contract without any provider in it: inputs, the output files and their
- keys, and which script checks what afterwards.
- ## Limitations
- - Only tested on Windows 11 with one 8 GB NVIDIA card; Linux, macOS and Apple Silicon are untested.
- - Without a GPU, transcription runs but is slow (hours per lecture).
- - On an 8 GB card, GPU stages must run one at a time (Whisper and the VLM cannot overlap).
- - The transcript is never auto-corrected; suspect tokens are only flagged, so names and numbers still need a human check.
- - Hosted ASR (Groq, Gemini) sends audio off your machine and is for non-confidential recordings only; no stage
- falls back from local to cloud by itself.
- - Defaults still show their origin (Mandarin Zoom UI dedup list, Traditional Chinese note headings, medical word lists).
- They are config, not code.
- ## Repo layout
- | Path | What is in it |
- |---|---|
- | `SKILL.md` | The map: hard rules, the numbered pipeline, edge cases |
- | `reference/pipeline.md` | Per-stage flags, thresholds, JSON schemas, timeouts |
- | `reference/note-spec.md` | Output note spec, slide tier scoring, synthesis prompt requirements |
- | `reference/segmented-mode.md` | Multi-talk workshop folders → per-segment notes + hub |
- | `reference/multi-camera.md` | One recording + many clips/photos → one timeline |
- | `reference/decisions.md` | Post-mortems, benchmarks, wrong turns, VRAM measurements |
- | `scripts/` | The pipeline (see `SKILL.md` for what each stage does) |
- | `scripts/batch/` | Generic batch layer: segment splitting, language detection, caching |
- | `scripts/layout2/` | Web viewer assets for `export_web.py` |
- | `tests/` | Dependency-light tests (`pytest`), incl. a hostile-content viewer fixture |
- | `pyproject.toml` | Metadata + test config; dependency lists mirror `requirements*.txt` |
- | `ocr_bench/` | OCR engine A/B/C harness — bring your own fixtures |
- | `data/` | Word/acronym frequency lists used to flag suspect ASR tokens |
- | `docs/AUDIT_SUMMARY.md` | What the pre-release audit found, and two things it got wrong |
- | `docs/transcript-contract.md` | What `transcript.json` must contain; `import_transcript.py` for an external ASR |
- | `docs/stage-f-contract.md` | Stage F (synthesis) inputs, outputs and gates, for any LLM |
- | `scripts/export_corpus.py` | Optional: finished notes → a searchable lecture corpus folder (`t2n-corpus/1`) |
- | `SECURITY.md` | How to report a vulnerability privately |
- | `tools/sync_from_skill.py` | Maintenance: re-export from the upstream skill tree |
- ## Design notes
- The invariants to know before changing anything (transcript never auto-corrected, VLM does no OCR, optional dependencies degrade loudly, provenance on every artifact, no data-derived paths) are in [docs/design-notes.md](docs/design-notes.md).
- ## Provenance and defaults
- This was extracted from a personal note-taking pipeline built for physical
- medicine and rehabilitation lectures, and some defaults still show it: the
- example config's dedup token list targets Mandarin-language Zoom UI chrome,
- the note templates use Traditional Chinese section headings, and the shipped
- word lists lean medical. All of it is config, not code — see
- `config.example.yaml` and `reference/note-spec.md`.
- The two files in `data/` are frequency lists of ordinary dictionary words and
- acronyms compiled from a local reference corpus, used only to decide whether an
- ASR token looks like a real word. Regenerate them from your own corpus with
- `scripts/build_real_words.py` if you want lists tuned to your domain.
- ## 🌱 Start here if you're new to AI agents
- This pipeline is one piece of my personal AI workflow. If you want to learn how
- to use AI agents like Claude Code from zero (no programming background needed),
- I wrote a beginner series (in Traditional Chinese):
- 1. [從零開始:安裝、看懂 GitHub、跑起你的第一個工具](https://drpwchen.com/posts/getting-started/)
- 2. [怎麼跟 AI agent 講話:心法、元技能與規則檔](https://drpwchen.com/posts/talking-to-agents/)
- 3. [自動化流程不是設計出來的,是長出來的](https://drpwchen.com/posts/growing-your-workflow/)
- The story behind this particular tool →
- [演講影片變成筆記:本機 GPU 轉錄 + 投影片對位](https://drpwchen.com/posts/lecture-to-notes/)
- Full map of my tools and posts → [drpwchen.com/map](https://drpwchen.com/map/)
- ## License
- MIT — see [LICENSE](LICENSE).
- ## Support 支持
- 覺得這個工具有幫助嗎?歡迎[請我喝飲料](https://drpwchen.com/support/) 🧋
- If this tool helped you, you can [buy me a drink](https://drpwchen.com/en/support/).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.