Skip to content
Back to search

video-content-reader

Read the content of a video file or a folder of videos and get a usable transcript. Extracts burned-in (hard) subtitles by cropping the subtitle band and running PP-OCRv4, falls back to faster-whisper speech-to-text when there are no burned-in subs, and pulls embedded subtitle tracks and keyframes alongside. Use when asked to 读取视频 / 视频内容 / 视频转文字 / 提取字幕 / transcribe a video / summarize what a video says / 把这个视频讲了什么整理出来. Built for social-platform clips where the subtitle is printed into the picture, which is far more accurate than speech recognition on a Japanese or English dub over Chinese subs.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/zyu0184-stack--video-content-reader/badge.svg)](/listings/zyu0184-stack--video-content-reader)

HTML

<a href="/listings/zyu0184-stack--video-content-reader"><img src="/api/listings/zyu0184-stack--video-content-reader/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/zyu0184-stack/video-content-reader
cd video-content-reader
pip install ".[ocr]"        # subtitle band reader
pip install ".[all]"        # adds faster-whisper for the audio path

That puts four commands on PATH:

| Command | Replaces | |---|---| | `vidprobe` | `python scripts/probe_video.py` | | `subocr` | `python scripts/subocr.py` | | `vidread` | `python scripts/vidread.py` | | `cleansubs` | `python scripts/clean_subs.py` |

The files under `scripts/` stay as thin shims, so a checkout works with no install at all — every command below has the same flags either way.

ffmpeg 7 or newer on PATH. `-vsync` is gone from 7, and the scripts use `-fps_mode`.

Tagged releases carry a wheel and an sdist: [releases](https://github.com/zyu0184-stack/video-content-reader/releases).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 130 lines
  1. ---
  2. name: video-content-reader
  3. description: Read the content of a video file or a folder of videos and get a usable transcript. Extracts burned-in (hard) subtitles by cropping the subtitle band and running PP-OCRv4, falls back to faster-whisper speech-to-text when there are no burned-in subs, and pulls embedded subtitle tracks and keyframes alongside. Use when asked to 读取视频 / 视频内容 / 视频转文字 / 提取字幕 / transcribe a video / summarize what a video says / 把这个视频讲了什么整理出来. Built for social-platform clips where the subtitle is printed into the picture, which is far more accurate than speech recognition on a Japanese or English dub over Chinese subs.
  4. ---
  5. # Video Content Reader
  6. A video is three independent channels: picture,声音, and printed text. Read them separately, then align.
  7. Order of preference — this is the whole point of the skill:
  8. | Priority | Source | When | Why |
  9. |---|---|---|---|
  10. | 1 | Burned-in subtitles (OCR) | A subtitle is printed into the picture | Authoritative. It is what the uploader intended the audience to read. |
  11. | 2 | Embedded subtitle track | `ffprobe` shows a `subtitle` stream | Exact text, zero recognition error |
  12. | 3 | Speech-to-text | Audio is human speech and nothing is printed | Needed when the clip is a talking head or a raw dub |
  13. | 4 | Keyframes | Always, as a supplement | Says what is on screen when nobody is talking |
  14. A clip ripped to a short-video platform almost always falls into case 1 with a Japanese or Korean original audio track. Running speech-to-text there produces garbage; OCR of the printed subtitle produces the actual script. Check for burned-in subtitles first.
  15. Measured on one line of dialogue, same clip:
  16. ```
  17. Whisper : 这一切也算是和天宇宙富有关,因为残怨家一直觉得有天宇宙富的人,无配成为宙术师。
  18. OCR : 这一切也算是和"天与咒缚"有养 因为禅院家一直觉得 有"天与咒缚"的人 不配成为咒术师
  19. ```
  20. Three proper nouns (天与咒缚, 禅院家, 咒术师): OCR gets all three, transcription none. Quote this when a user asks why the OCR path is the default.
  21. ## Step 0 — Inspect before doing anything
  22. Commands below use `python scripts/<name>.py`, which runs straight from a checkout with no install. When the package is installed the same flags work as `vidprobe` / `subocr` / `vidread` / `cleansubs`.
  23. ```bash
  24. python scripts/probe_video.py VIDEO
  25. ```
  26. Reports duration, resolution, codec, audio channels, embedded subtitle tracks, and — the part that matters — whether the bottom band of the picture contains a stable high-contrast text row across sampled frames. Exit code 0 when burned-in subtitles are detected, 1 when not.
  27. Pick the branch from that output, not from the file extension.
  28. ## Step 1 — Burned-in subtitles
  29. ```bash
  30. python scripts/subocr.py VIDEO --out DIR
  31. ```
  32. Pipeline: sample frames at a fixed interval (default 0.8 s) → crop the subtitle band → OCR each band → drop watermark and platform-UI boxes → merge consecutive frames holding the same sentence → write `subs.srt` + `subs.txt`.
  33. Key parameters, tuned on real footage:
  34. | Flag | Default | Effect |
  35. |---|---|---|
  36. | `--interval` | 0.8 | Seconds between sampled frames. A subtitle line can be on screen for under a second; 1.2 already drops lines. Lower costs time linearly. |
  37. | `--top` | 0.70 | Where the subtitle band starts, as a fraction of frame height |
  38. | `--bottom` | 1.00 | Where it ends |
  39. | `--scale` | 1.0 | Resize factor for the cropped band. 0.5 is faster with almost no accuracy loss on 1080p source |
  40. | `--jobs` | 4 | OCR worker processes |
  41. | `--sim` | 0.72 | Character-overlap ratio above which two frames count as the same sentence |
  42. | `--keep-frames` | off | Keep the cropped JPEGs for inspection |
  43. Two settings decide whether this runs in 5 minutes or 30, and neither is obvious from the docs:
  44. 1. **`det_limit_type='max'`** (set inside the script; do not remove). RapidOCR's default is `min`, which scales the *short* side of the input to `limit_side_len`. A cropped subtitle band is extremely wide and short — 960×163 becomes 4360×736 — and detection time explodes. Measured on the same 10 frames: **1.41 s/frame with `min`, 0.32 s/frame with `max`**. 4.4× for one config key.
  45. 2. **Explicit `intra_op_num_threads`**. Left at `-1` each worker grabs every core; four workers then fight over 8 hardware threads. Measured: `-1` → 3.11 fps, `4` → 3.82 fps, `1` → 2.41 fps. Multi-process speedup only materialises when per-process threads are capped.
  46. Full numbers and the sweep that produced them: [references/ocr-tuning.md](references/ocr-tuning.md).
  47. ## Step 2 — Clean the transcript
  48. OCR on a subtitle band picks up more than the subtitle. Three contamination classes, all handled by:
  49. ```bash
  50. python scripts/clean_subs.py DIR/subs.txt
  51. ```
  52. - **Platform UI** — 抖音号:, 截图保存到相册, 愛別離-style account names. Removed by a word list plus a position rule: boxes whose centre sits in the right 26% and are under 14 characters.
  53. - **Foreign-language credits** — an anime ending's Japanese staff roll reads as a dense汉-name string with no punctuation. Kana lines are dropped outright.
  54. - **Scroll repeats** — the same frame sampled while a graphic stays on screen yields six near-identical lines. Adjacent duplicates and any line repeated within a 12-line window are collapsed.
  55. Then read the cleaned text. Expect residual character-substitution errors even after cleaning: OCR reproduces what the uploader typed, and uploaders of ripped anime routinely mistype character names. `宿滩`/`宿雄`/`宿怒` all mean 宿傩; `慈` is 茈; `星宿` is 新宿. Build a correction table by sampling the actual frames and comparing.
  56. ## Step 3 — Speech-to-text, only if Step 1 found nothing
  57. ```bash
  58. python scripts/vidread.py VIDEO --lang auto
  59. ```
  60. Extracts 16 kHz mono audio, transcribes with faster-whisper, writes `transcript.txt` + `transcript.srt`, and also dumps keyframes and any embedded subtitle track in one pass.
  61. Three things the script already handles:
  62. - **Simplified-Chinese output.** Whisper's `small` model returns Traditional for Mandarin input. `initial_prompt="以下是普通话的句子,请使用简体中文转写。"` fixes it completely — measured on the same clip: without prompt `今天下午3點開會,討論第四季度的預算分配問題`, with prompt `今天下午三点开会,讨论第四季度的预算分配问题。`
  63. - **Thread count.** `cpu_threads=8` on an 8-core/16-thread CPU gives 1.76× realtime; 16 gives 1.51×. Hyperthreads hurt ctranslate2.
  64. - **Model location.** Local directory wins over hub name. See [references/troubleshooting.md](references/troubleshooting.md) for the China-network model download path.
  65. ## Step 4 — Write the deliverable
  66. For each clip, produce:
  67. ```
  68. NAME.txt one sentence per line, time order
  69. NAME.srt timecoded
  70. NAME_已清洗.txt after clean_subs.py
  71. NAME_ocr原始.jsonl per-frame boxes with coordinates and confidence
  72. ```
  73. Then a single `README.md` covering: a clip table (file, duration, resolution, topic, line count), a per-clip skeleton of the argument or plot in sections, and the correction table. A transcript dump is not a deliverable on its own — the reader wants to know what the video says, not to re-read OCR output.
  74. Report figures with units: frames sampled, wall-clock seconds, per-frame cost. "Fast" is not a measurement.
  75. ## Dependencies
  76. ```bash
  77. pip install ".[ocr]" # OCR path only
  78. pip install ".[all]" # OCR + speech-to-text
  79. # no install wanted? the two libraries on their own:
  80. pip install rapidocr-onnxruntime faster-whisper
  81. # plus ffmpeg + ffprobe on PATH
  82. ```
  83. `faster-whisper` and `rapidocr-onnxruntime` are only needed for their respective steps. OCR alone needs `rapidocr-onnxruntime`; the audio path needs `faster-whisper`.
  84. ## Known edges
  85. - **HEVC sources** decode fine; nothing special needed.
  86. - **Vertical video** (1080×1920): the subtitle band still sits in the lower third, but `--top 0.70` may clip a subtitle block that is stacked across two lines. Inspect one frame first.
  87. - **Subtitle inside a decorative frame** (a white rounded box, a chat bubble): the box edge is high-contrast and may be detected as text. If `subs.txt` shows a line of stray punctuation, narrow `--top`.
  88. - **Multiple text rows in the band** — bilingual subtitles, or a title card over the dialogue: both get OCR'd and joined with a space, ordered left to right. Split them by inspecting `_ocr原始.jsonl`, which keeps each box separately.
  89. - **No audio track** — `vidread.py` skips transcription and still produces frames and subtitle tracks.
README.md · README · 150 lines
  1. # video-content-reader
  2. [![ci](https://github.com/zyu0184-stack/video-content-reader/actions/workflows/ci.yml/badge.svg)](https://github.com/zyu0184-stack/video-content-reader/actions/workflows/ci.yml)
  3. [![release](https://github.com/zyu0184-stack/video-content-reader/actions/workflows/release.yml/badge.svg)](https://github.com/zyu0184-stack/video-content-reader/actions/workflows/release.yml)
  4. Read what a video actually says, without watching it.
  5. ![One video splits into three channels: picture, sound, and printed subtitle. The printed subtitle is read first.](docs/pipeline.svg)
  6. Built for the case that breaks speech recognition: a Chinese-subtitled clip over a Japanese or Korean original audio track — the standard shape of a short-video reupload. The printed subtitle is the script the uploader intended; the audio is a dub nobody is listening to. This reads the printed subtitle.
  7. ## The decision that matters
  8. | Source | When to trust it | Error rate |
  9. |---|---|---|
  10. | Burned-in subtitle | A subtitle is printed into the picture | Character-level OCR only |
  11. | Embedded subtitle track | `ffprobe` reports a `subtitle` stream | Zero |
  12. | Speech-to-text | Nothing is printed, audio is human speech | Word-level, and total on a dub |
  13. | Keyframes | Always | — |
  14. Running Whisper on a Japanese dub of a Chinese-subtitled anime explains nothing. OCR of the band does. Check the picture before choosing a tool.
  15. Same sentence from the same clip, two methods:
  16. ```
  17. Whisper : 这一切也算是和天宇宙富有关,因为残怨家一直觉得有天宇宙富的人,无配成为宙术师。
  18. OCR : 这一切也算是和"天与咒缚"有养 因为禅院家一直觉得 有"天与咒缚"的人 不配成为咒术师
  19. ```
  20. Three proper nouns — 天与咒缚, 禅院家, 咒术师 — the OCR got all three, the transcription got none. That is the whole argument in one line. OCR still carries its own errors (`有养` for `有关`), but they are character-level and readable; speech recognition loses the terms the video is about.
  21. ## Install
  22. ```bash
  23. git clone https://github.com/zyu0184-stack/video-content-reader
  24. cd video-content-reader
  25. pip install ".[ocr]" # subtitle band reader
  26. pip install ".[all]" # adds faster-whisper for the audio path
  27. ```
  28. That puts four commands on PATH:
  29. | Command | Replaces |
  30. |---|---|
  31. | `vidprobe` | `python scripts/probe_video.py` |
  32. | `subocr` | `python scripts/subocr.py` |
  33. | `vidread` | `python scripts/vidread.py` |
  34. | `cleansubs` | `python scripts/clean_subs.py` |
  35. The files under `scripts/` stay as thin shims, so a checkout works with no install at all — every command below has the same flags either way.
  36. ffmpeg 7 or newer on PATH. `-vsync` is gone from 7, and the scripts use `-fps_mode`.
  37. Tagged releases carry a wheel and an sdist: [releases](https://github.com/zyu0184-stack/video-content-reader/releases).
  38. ## Use
  39. **1. Inspect — decide the branch before spending minutes**
  40. ```bash
  41. vidprobe clip.mp4
  42. ```
  43. Prints codec, duration, audio language, embedded subtitle tracks, and whether the bottom band carries printed text across sampled frames. Exit 0 means burned-in subtitles are present, and the last line tells you which command to run next.
  44. **2. Extract**
  45. ```bash
  46. subocr clip.mp4 --interval 0.8 --jobs 4
  47. ```
  48. ```
  49. clip/subs.srt timecoded
  50. clip/subs.txt one sentence per line
  51. clip/ocr_raw.jsonl per-frame boxes with coordinates and confidence
  52. ```
  53. **3. Clean, then read**
  54. ```bash
  55. cleansubs clip/subs.txt
  56. ```
  57. Drops kana lines (anime staff rolls), platform UI (`抖音号:`, `截图保存到相册`), two-character fragments, and the repeats a static graphic produces.
  58. **No printed subtitle?** Take the audio path:
  59. ```bash
  60. vidread clip.mp4 --lang auto
  61. ```
  62. Keyframes, 16 kHz mono audio, every embedded track, and a transcript in one pass.
  63. ## Measured
  64. Ryzen 7 8845HS, 8 cores / 16 threads, CPU only. 19 minutes of footage across four clips:
  65. | Step | Result |
  66. |---|---|
  67. | Frame sampling + OCR, 1445 frames | 522 s wall clock, 2.77 frames/s |
  68. | Subtitle lines recovered | 647 |
  69. | Whisper `small`, int8, 70.8 s audio, 8 threads | 40.2 s, 1.76× realtime |
  70. Two settings carry most of the speed. Both are in the modules, both are documented with the sweep that produced them in [references/ocr-tuning.md](references/ocr-tuning.md):
  71. - `det_limit_type='max'` instead of the library default — **4.4× faster**, 1.41 → 0.32 s/frame. The default enlarges the short side of a wide subtitle band by up to 6.8×.
  72. - Explicit `intra_op_num_threads` per worker — the default `-1` gives every worker every core, and four workers then fight over eight hardware threads.
  73. ## What breaks
  74. Character substitution in OCR is not a bug to fix, it is the input. Reuploads of Japanese animation routinely mistype character names themselves, so the OCR faithfully reproduces `宿滩` for `宿傩` and `慈` for `茈`. Sample the actual frames, build a correction table, publish it next to the transcript.
  75. Watermarks, staff rolls, and platform buttons share the band with the subtitle. The scripts filter by position, by script (kana), and by word list — all three are needed, none is sufficient.
  76. Failure modes with exact error strings: [references/troubleshooting.md](references/troubleshooting.md).
  77. ## Files
  78. ```
  79. src/video_content_reader/
  80. probe_video.py probe + burned-in subtitle detection
  81. subocr.py band crop, OCR, watermark filter, sentence merge
  82. vidread.py keyframes + audio + embedded subs + whisper
  83. clean_subs.py noise removal
  84. scripts/
  85. *.py shims, so a checkout runs without pip install
  86. check_wheel.py verifies a built wheel before release
  87. fetch_model.sh whisper model download bypassing the hub API
  88. SKILL.md agent-facing instructions and decision table
  89. references/ocr-tuning.md measured parameter sweeps
  90. references/troubleshooting.md real errors and fixes
  91. docs/pipeline.svg the diagram above
  92. .github/workflows/ci.yml lint, build+install the wheel, end-to-end OCR
  93. .github/workflows/release.yml tag -> sdist + wheel -> GitHub release
  94. ```
  95. ## Model download
  96. `WhisperModel("small")` calls the hub API for validation. On networks where that endpoint is blocked or returning 500 while the file CDN is healthy, the download fails with `LocalEntryNotFoundError`. Resolve the files directly:
  97. ```bash
  98. ./scripts/fetch_model.sh small
  99. vidread clip.mp4 --model-path models/faster-whisper-small
  100. ```
  101. `vidread` looks for `models/faster-whisper-<name>` relative to the working directory, the repo root, and `~/.cache/faster-whisper`, so an installed wheel finds a model the same way a checkout does.
  102. ## License
  103. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.