Skip to content
Back to search

claude-real-video

MarketplaceWorks with Claude CodeGitHub

Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/huangchihhungleo--claude-real-video/badge.svg)](/listings/huangchihhungleo--claude-real-video)

HTML

<a href="/listings/huangchihhungleo--claude-real-video"><img src="/api/listings/huangchihhungleo--claude-real-video/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pip install "claude-real-video[whisper]"   # recommended: frames + dedup + audio transcription
pip install claude-real-video              # core only (frames + dedup)

pip extras never install themselves — without `[whisper]` there is **no speech-to-text** (videos that ship their own subtitles still get a transcript).

System requirement: ffmpeg

`ffmpeg` / `ffprobe` are used for frame extraction and audio, and aren't pip-installable. Install them once:

| OS | command | |---|---| | **macOS** | `brew install ffmpeg` | | **Linux** | `sudo apt install ffmpeg` (or your distro's package manager) | | **Windows** | `winget install Gyan.FFmpeg` — or `choco install ffmpeg` — or [download a build](https://www.gyan.dev/ffmpeg/builds/) and add its `bin\` folder to your `PATH` |

Verify it's on your `PATH`:

ffmpeg -version

Transcription uses the `whisper` CLI (installed by the `[whisper]` extra, or `pip install openai-whisper`). Whisper also relies on ffmpeg.

**Faster + hallucination-proof transcripts (recommended):** install the `[fast]` extra and crv automatically switches to [faster-whisper](https://github.com/SYSTRAN/faster-whisper) — same models, same output files, several times faster, and gated by Silero VAD (voice-activity detection): music-only or silent audio yields an honest "no speech" note instead of whisper's classic invented caption. No new flags to learn:

pip install 'claude-real-video[fast]'

If both are installed, faster-whisper wins; if it ever fails, crv falls back to the `whisper` CLI on its own.

**Apple Silicon (M1–M4): GPU transcription.** Install the `[mlx]` extra and crv runs Whisper on the Mac's GPU through [mlx-whisper](https://github.com/ml-explore/mlx-examples/tree/main/whisper) — a 21-minute talk that takes ~6 minutes on faster-whisper finishes in about a minute on an M4, with the same transcript. The Silero VAD gate still runs first, so music or silence never turns into invented captions; if the gate or mlx cannot run, crv drops back to faster-whisper, then the CLI. Contributed by [@blazejp83](https://github.com/blazejp83) (#28).

pip install 'claude-real-video[mlx]'

Works on **macOS, Windows, and Linux** — Python 3.10+.

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 14 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "claude-real-video",
  3. "owner": {
  4. "name": "Leo Huang",
  5. "url": "https://leoaido.com"
  6. },
  7. "plugins": [
  8. {
  9. "name": "claude-real-video",
  10. "source": "./plugins/claude-real-video",
  11. "description": "Let Claude actually watch a video — scene-aware, deduplicated keyframes + timestamped transcript from a URL or local file. Runs 100% locally, no API key."
  12. }
  13. ]
  14. }
README.md · README · 444 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # claude-real-video
  2. [![PyPI](https://img.shields.io/pypi/v/claude-real-video)](https://pypi.org/project/claude-real-video/) [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/claude-real-video/) [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![HN front page](https://img.shields.io/badge/Hacker%20News-front%20page-orange)](https://news.ycombinator.com/item?id=48766005)
  3. [![Can Claude watch a video? Same question to ChatGPT, Gemini and Claude](https://img.youtube.com/vi/QRuh1oByT3M/hqdefault.jpg)](https://youtu.be/QRuh1oByT3M)
  4. **▶ [100-second demo](https://youtu.be/QRuh1oByT3M):** ChatGPT and Gemini can watch video, Claude can't. Here's what Claude answers once crv turns the video into frames plus a timestamped transcript.
  5. [![LLM Real Video — Give Your LLM Eyes (60-second film)](https://img.youtube.com/vi/sw6_8E-57w4/maxresdefault.jpg)](https://youtu.be/sw6_8E-57w4)
  6. **▶ [The 60-second pixel film — sound on](https://youtu.be/sw6_8E-57w4)** ([mp4 on GitHub](https://github.com/HUANGCHIHHUNGLeo/claude-real-video/releases/download/v0.7.16/crv-999-film-60s.mp4)) · an AI agent searches "how can an LLM truly understand video?", finds a key, and unlocks vision.
  7. [![crv 60s demo](https://raw.githubusercontent.com/HUANGCHIHHUNGLeo/claude-real-video/master/docs/crv-demo-poster.jpg)](https://github.com/HUANGCHIHHUNGLeo/claude-real-video/releases/download/v0.7.15/crv-demo-60s.mp4)
  8. 60-second real demo — real install, real run, real viewer.
  9. **Let Claude — or any LLM — actually watch a video.**
  10. ```bash
  11. pip install "claude-real-video[whisper]"
  12. npx skills add HUANGCHIHHUNGLeo/claude-real-video # one command, installs the skill into Claude Code, Cursor, Codex, Copilot, Gemini CLI & 50+ agent hosts
  13. ```
  14. Claude Code plugin marketplace (enable auto-update in /plugin → Marketplaces if you want it):
  15. ```
  16. /plugin marketplace add HUANGCHIHHUNGLeo/claude-real-video
  17. /plugin install claude-real-video@claude-real-video
  18. ```
  19. Then paste a video link into your agent and ask about it. (CLI-only use? `crv "<url>"` works with just the pip install.)
  20. > **Naming:** crv is the short name for claude-real-video (the PyPI package). The paid add-on, **crv Pro**, is sold on Capafy under the listing name "llm-real-video Pro".
  21. ![demo](https://raw.githubusercontent.com/HUANGCHIHHUNGLeo/claude-real-video/master/docs/demo.gif)
  22. > ▶ **New: the 40-second film** — [*my AI agent learned to watch videos (and stopped working)*](https://youtu.be/xFqPtcju_xo)
  23. > Same 58-second clip: fixed 1 fps sampling = **58 frames**. crv keeps the **26 that actually differ** — and `--grid` packs them into **3 contact sheets**. Fewer tokens, nothing missed.
  24. > **This free version lets your AI *see* the video.** [crv Pro](https://leoaido.com/crv-pro/) lets it *understand* it — how it was shot (cut rhythm, camera moves) plus a timestamped timeline of what frames can't show: gestures, expressions, voice pitch shifts, emotion, sound events. One-time price $29 — [get it on Capafy](https://capafy.ai/agent/llm-real-video-pro-let-any-llm-watch-videos/5451082151) or [buy with card via Lemon Squeezy](https://leoaido.lemonsqueezy.com/checkout/buy/ff552000-adc0-49f1-8eec-5e8ada1905a1).
  25. > **On Codex CLI?** ▶ [36-second demo](https://youtu.be/XLtBXlcBRzE) · [Real Video for Codex](https://capafy.ai/agent/real-video-for-codex/3594524971) is a self-contained build for it. You ask Codex about a video, it runs one command, then reads a `SUMMARY.md` with chapters, a timecoded transcript and **the text that was on screen**, which this free edition leaves out. So it can quote the config block on the slide at 12:40, not only what the speaker said about it, and it answers you in `hh:mm:ss`. Download, frames, transcription and text recognition all run on your machine, and the tool itself needs no API key and no account. Ships with the Codex skill and a `crv-easy doctor` self-check. Python 3.10 to 3.14, macOS or Windows 11 native. One-time $9.90: [get it on Capafy](https://capafy.ai/agent/real-video-for-codex/3594524971) or [buy with card via Lemon Squeezy](https://leoaido.lemonsqueezy.com/buy/cbbead56-5838-43fd-a00c-62b53b0de13e?checkout%5Bcustom%5D%5Bref%5D=github-readme).
  26. Most AI tools don't really *see* a video. Paste a YouTube link into ChatGPT and it
  27. reads the **transcript**, not the picture. Claude won't take a video file at all.
  28. Even Gemini, which *can* read video natively, has to send it up to Google and
  29. samples frames at a **fixed interval** (1 fps by default), so fast cuts slip past.
  30. `claude-real-video` does it differently, and **the processing runs locally**: point it at a URL or a
  31. file, and it pulls the frames that *actually matter* (every scene change, not a
  32. fixed quota), throws away the near-duplicates, transcribes the audio, and hands
  33. you a clean folder any LLM can read. All the processing happens on your own machine — what gets sent anywhere is only the frames/text *you* choose to paste into an LLM afterwards.
  34. ```bash
  35. crv "https://www.youtube.com/watch?v=..."
  36. # → crv-out/frames/*.jpg + frames.json (per-frame timestamps) + transcript.txt/.json + MANIFEST.txt
  37. ```
  38. Then drop the frames + `MANIFEST.txt` into Claude / ChatGPT / Gemini and ask away.
  39. **No terminal needed** — run `crv-web` and a local page opens (Traditional Chinese / Simplified Chinese / English): paste a YouTube or Reels link or a file path, click Analyze, open the result viewer. Video analysis and output generation run on your machine — the source video never gets uploaded. (If you then paste the extracted frames or transcript into a cloud LLM, that data goes to that provider.)
  40. Want to eyeball what the model will see first? Add `--viewer` — it writes a local `viewer.html` (video + keyframe grid + transcript) you can double-click open. No network, no extra installs.
  41. **Only part of a video matters** (a 10-minute screen share inside a 90-minute call): `--from 28:00 --to 43:00`. ffmpeg seeks instead of decoding the whole file, Whisper only hears the window, and the frame budget is spent inside it — but every timestamp crv reports is still a source timecode you can quote to a colleague.
  42. **The meaning is small text** (a terminal, a spreadsheet, an IDE): `--frame-width 1600`. Frame *selection* is the hard part and crv already does it; at 640px on a 1920-wide screen recording the right moment gets found and then the detail that made it worth finding is thrown away.
  43. **Slow-changing content** (animation tutorials, gradual morphs, slow pans): add `--adaptive` — frames are picked against their rolling neighbourhood instead of a fixed threshold, so a 2-3s squash-and-stretch that never spikes any single frame still gets captured.
  44. **Text-heavy content** (lecture slides, screen recordings, talking-head explainers): add `--text-anchors` — extra frames are forced at subtitle-cue timestamps, so each spoken segment gets a matching visual even when the scene barely changes. Needs a sidecar `.srt`/`.vtt` or an embedded subtitle track — captions burned into the pixels can't be detected. At most one forced frame per second; scene detection is untouched.
  45. **Multi-speaker content** (interviews, podcasts, meetings): add `--speakers` — every transcript line gets a speaker label (`[SPEAKER_00]`, `[SPEAKER_01]`, …) so the model can follow who said what. Runs a local diarization model (45 MB, downloads once, no account or token needed). Install with `pip install "claude-real-video[speakers]"`.
  46. Not doing LLM work? It also works as a **general-purpose video keyframe extractor** —
  47. scene-change detection + dedup, no ML models to download.
  48. **Using Claude Code — or any coding agent?** One command installs the skill
  49. (works with Claude Code, Cursor, Codex, Copilot, Gemini CLI and other
  50. [agentskills.io](https://agentskills.io)-compatible hosts):
  51. ```bash
  52. pip install "claude-real-video[whisper]"
  53. npx skills add HUANGCHIHHUNGLeo/claude-real-video
  54. ```
  55. Then just paste a video link into your agent and ask about it.
  56. <details>
  57. <summary>Manual install (clone + copy)</summary>
  58. ```bash
  59. git clone https://github.com/HUANGCHIHHUNGLeo/claude-real-video.git
  60. mkdir -p ~/.claude/skills && cp -r claude-real-video/skills/claude-real-video ~/.claude/skills/
  61. ```
  62. </details>
  63. **Tell it *why* you're watching, and keep what it finds:**
  64. ```bash
  65. crv "https://youtu.be/..." --why "find the pricing strategy" --kb ~/notes
  66. ```
  67. `--why` makes the analysis focus on what you care about instead of a generic summary;
  68. `--kb` saves the result as a dated note in your own notes folder, so it doesn't die in `crv-out`.
  69. **New in 0.10.x** — analyse only the part that matters:
  70. ```bash
  71. crv long-meeting.mp4 --from 28:00 --to 43:00
  72. ```
  73. `--from` / `--to` cut a window out of a long video: ffmpeg seeks instead of decoding
  74. the whole file, the transcript and frame budget follow the window, and every reported
  75. timestamp is still a source timecode you can quote back to the original.
  76. ---
  77. ## Measured numbers
  78. Real run on a 3-minute 640x360 video (benchmark/jfk-rice.mp4), Mac mini M4, local CPU, frames + dedup only (`--no-transcribe`). Image tokens estimated with Anthropic's `(width x height) / 750` — 307 tokens/frame at 640x360.
  79. | Mode | Frames kept | Wall time | Est. image tokens |
  80. |------|------------|-----------|-------------------|
  81. | default (scene-change + 1s floor) | 170 (from 180 extracted) | 23.5 s | ~52k |
  82. | `--max-frames 80` | 80 | 23.4 s | ~25k |
  83. | `--adaptive` (catches slow morphs) | 270 | 36.8 s | ~83k |
  84. **Dedup v0.7.16 — small-subject fast action no longer disappears.** A percentage comparator is structurally blind to a subject that covers <1% of the frame (it can never change 8% of the pixels). Found in a user's 2,181-video batch run; fixed with a third "action channel". Synthetic repro — static 1280x720 shot, a 40x90 px subject (0.4% of frame) moves fast only in the last 10 of 65 frames:
  85. | | Frames kept | Action frames survived |
  86. |---|---|---|
  87. | v0.7.15 | 2 | 1 / 10 |
  88. | v0.7.16 | 11 | **10 / 10** — full trajectory |
  89. ## Why not just sample frames?
  90. Most "let an LLM watch a video" scripts (and Gemini's own pipeline) grab frames
  91. at a **fixed interval** — e.g. one per second. That over-samples a static
  92. screencast and under-samples a fast-cut reel. `claude-real-video` is smarter:
  93. | | fixed-interval sampling | **claude-real-video** |
  94. |---|---|---|
  95. | Frame selection | every N seconds | **scene-change detection** + density floor |
  96. | Repeated shots (A-B-A cuts) | sent again every time | **sliding-window dedup** sends each shot once |
  97. | Static slide (10 min) | ~600 near-identical frames | **collapses to 1** (dedup) |
  98. | Fast-cut reel | misses frames between samples | catches each visual change |
  99. | Audio | often ignored | Whisper transcript w/ language detect |
  100. | Where the processing happens | often in someone's cloud | **on your machine** (you choose what to share with an LLM afterwards) |
  101. | Input | usually local file only | **URL (yt-dlp) or local file** |
  102. You feed the model *fewer, more meaningful* frames — cheaper context, better
  103. understanding.
  104. ---
  105. ## Install
  106. ```bash
  107. pip install "claude-real-video[whisper]" # recommended: frames + dedup + audio transcription
  108. pip install claude-real-video # core only (frames + dedup)
  109. ```
  110. pip extras never install themselves — without `[whisper]` there is **no speech-to-text**
  111. (videos that ship their own subtitles still get a transcript).
  112. ### System requirement: ffmpeg
  113. `ffmpeg` / `ffprobe` are used for frame extraction and audio, and aren't
  114. pip-installable. Install them once:
  115. | OS | command |
  116. |---|---|
  117. | **macOS** | `brew install ffmpeg` |
  118. | **Linux** | `sudo apt install ffmpeg` (or your distro's package manager) |
  119. | **Windows** | `winget install Gyan.FFmpeg` — or `choco install ffmpeg` — or [download a build](https://www.gyan.dev/ffmpeg/builds/) and add its `bin\` folder to your `PATH` |
  120. Verify it's on your `PATH`:
  121. ```bash
  122. ffmpeg -version
  123. ```
  124. Transcription uses the `whisper` CLI (installed by the `[whisper]` extra, or
  125. `pip install openai-whisper`). Whisper also relies on ffmpeg.
  126. **Faster + hallucination-proof transcripts (recommended):** install the `[fast]`
  127. extra and crv automatically switches to
  128. [faster-whisper](https://github.com/SYSTRAN/faster-whisper) — same models, same
  129. output files, several times faster, and gated by Silero VAD (voice-activity
  130. detection): music-only or silent audio yields an honest "no speech" note instead
  131. of whisper's classic invented caption. No new flags to learn:
  132. ```bash
  133. pip install 'claude-real-video[fast]'
  134. ```
  135. If both are installed, faster-whisper wins; if it ever fails, crv falls back
  136. to the `whisper` CLI on its own.
  137. **Apple Silicon (M1–M4): GPU transcription.** Install the `[mlx]` extra and crv
  138. runs Whisper on the Mac's GPU through
  139. [mlx-whisper](https://github.com/ml-explore/mlx-examples/tree/main/whisper) —
  140. a 21-minute talk that takes ~6 minutes on faster-whisper finishes in about a
  141. minute on an M4, with the same transcript. The Silero VAD gate still runs
  142. first, so music or silence never turns into invented captions; if the gate or
  143. mlx cannot run, crv drops back to faster-whisper, then the CLI. Contributed by
  144. [@blazejp83](https://github.com/blazejp83) (#28).
  145. ```bash
  146. pip install 'claude-real-video[mlx]'
  147. ```
  148. Works on **macOS, Windows, and Linux** — Python 3.10+.
  149. ---
  150. ## Usage
  151. ```bash
  152. # A YouTube / Instagram / TikTok / ... link
  153. crv "https://www.instagram.com/reel/XXXX/"
  154. # A local file, English transcript, output to ./out
  155. crv lecture.mp4 -o out --lang en
  156. # Frames only, no transcription
  157. crv clip.mp4 --no-transcribe
  158. # A login-gated video (your own / authorised use): pass a Netscape cookie file
  159. crv "https://..." --cookies cookies.txt
  160. ```
  161. `python -m claude_real_video ...` works as an alias for `crv` too.
  162. ### Options
  163. | flag | default | meaning |
  164. |---|---|---|
  165. | `-o, --out` | `crv-out` | output directory |
  166. | `--overwrite` | off | replace a previous analysis living in the output directory (without this, a non-empty output dir is refused to avoid mixing videos) |
  167. | `--scene` | `0.30` | scene-change sensitivity (lower = more frames) |
  168. | `--fps-floor` | `1.0` | at least one frame every N seconds |
  169. | `--from` / `--to` | whole file | analyse only part of a video (`90`, `1:30`, `0:01:30.5`). Reported timestamps stay **source** timecodes — a window shifts the analysis, not the clock — and the frame budget plus the transcript follow the window instead of the whole file |
  170. | `--frame-width` | `640` | width of extracted frames, aspect kept. Raise it when the meaning *is* small text (terminals, spreadsheets, dense dashboards); larger frames multiply output size and model cost |
  171. | `--max-frames` | auto: `clamp(150, window×1.5, 600)` | hard cap on total frames (explicit value always wins) |
  172. | `--adaptive` | off | adaptive scene detection: catches slow morphs (2-3s squash/stretch, gradual pans) a fixed threshold misses, by comparing each frame against its rolling neighbourhood |
  173. | `--text-anchors` | off | force extra frames at subtitle-cue timestamps (sidecar `.srt`/`.vtt` or embedded track) — for videos where meaning changes faster than pixels; at most one forced frame per second |
  174. | `--speakers` | off | label every transcript line with the speaker (`[SPEAKER_00]` …) via local diarization — needs `pip install "claude-real-video[speakers]"`, 45 MB model downloads once |
  175. | `--lang` | `auto` | Whisper language (`en`, `zh`, `auto`, ...) |
  176. | `--whisper-model` | `base` | Whisper model for transcription (`tiny`/`base`/`small`/`medium`/`large`/`turbo` — base is fast; **want sharper transcripts? `--whisper-model turbo` is one flag away**: a pruned large-v3 — much faster than `large` with a minor quality trade-off, one-time 1.6GB download, ~6GB memory) |
  177. | `--dedup-threshold` | `8` | % of pixels that must change for a frame to count as new; higher = fewer frames (the settled-local detector's gate scales with it too) |
  178. | `--dedup-window` | `4` | compare against the last N kept frames — a shot the model already saw doesn't come back after a cutaway (`1` = consecutive-only) |
  179. | `--report` | off | keep dropped frames in `./dropped` + write `report.html` visualising every keep/drop decision |
  180. | `--no-transcribe` | off | skip audio |
  181. | `--keep-audio` | off | also save the **full soundtrack** (`audio.m4a`) so audio models can *hear* it |
  182. | `--viewer` | off | also write `viewer.html` — browse the video, keyframes and transcript in one local page (double-click to open) |
  183. | `--grid` | off | also tile the kept frames into 3x3 contact sheets (`./grids`) — consecutive frames side by side help the model follow motion and progression |
  184. | `--why` | – | why you're watching, e.g. `--why "find the pricing strategy"` — written into `MANIFEST.txt` so the model analyses with that lens instead of a generic summary |
  185. | `--kb` | – | also save the analysis as a dated markdown note into this folder (your Obsidian vault, notes dir, ...) — so it joins your knowledge base instead of dying in `crv-out` |
  186. | `--cookies` | – | Netscape cookie file for login-gated sources |
  187. | `--cookies-from-browser` | – | read login cookies straight from your own browser — `chrome`, `safari`, `firefox` or `edge` (your own account only) |
  188. ---
  189. ### What `--grid` output looks like
  190. One contact sheet = nine consecutive keyframes, in order, filenames on each cell — the model reads a sequence, not scattered stills:
  191. ![contact sheet example](https://raw.githubusercontent.com/HUANGCHIHHUNGLeo/claude-real-video/master/docs/grid_example.jpg)
  192. ## Memory — ask across everything you've watched
  193. Every analysis is indexed locally (transcript lines + on-screen text, with timestamps),
  194. so a question can span your whole library instead of one output folder:
  195. ```bash
  196. crv-ask "pricing strategy" # → which video, which second, the exact line
  197. crv-ask 定價 # CJK works — trigram FTS with a substring fallback
  198. crv-ask --list # everything you've watched, newest first
  199. crv-ask --stats # where the index lives, how big it is
  200. crv-ask --prune 200 # keep the newest 200 videos, reclaim the space
  201. ```
  202. Re-running the same source with the same options doesn't re-process — crv says
  203. "already watched" and points at the existing analysis (0.04s vs several seconds
  204. measured). Different options, or `--overwrite`, re-analyse as usual.
  205. Everything stays local: one SQLite file at `~/.crv/memory.db` (override with
  206. `CRV_MEMORY_DB`), user-only permissions, no embeddings, no network. The first time
  207. anything is indexed, crv prints one line saying so. Don't want it at all?
  208. `CRV_NO_MEMORY=1`.
  209. ## MCP server (Claude Desktop / Cursor / any MCP client)
  210. crv also ships as an MCP server, so MCP clients can ask for a video to be
  211. watched directly — same local pipeline, zero cloud. Five tools: `watch_video`,
  212. `get_frames`, `search_memory` (ask across every watched video), `list_watched`
  213. (check before re-watching), and `get_transcript` (words only — no frames, so a
  214. long talk doesn't cost image tokens).
  215. ```bash
  216. pip install 'claude-real-video[mcp]'
  217. ```
  218. Claude Code:
  219. ```bash
  220. claude mcp add crv -- crv-mcp
  221. ```
  222. Claude Desktop — add to `claude_desktop_config.json`:
  223. ```json
  224. { "mcpServers": { "crv": { "command": "crv-mcp" } } }
  225. ```
  226. Tools: `watch_video(source, max_frames, language, transcribe)` returns the
  227. timestamped transcript plus the first batch of keyframes as images;
  228. `get_frames(source, start_index, count)` pages through the rest. Analyses are
  229. cached under `~/.cache/crv-mcp`, so follow-up questions about the same video
  230. are instant.
  231. mcp-name: io.github.HUANGCHIHHUNGLeo/claude-real-video
  232. Verified end-to-end on Claude Code (the model described a test video's frames
  233. correctly through the tool). Claude Desktop and Cursor speak the same MCP
  234. stdio protocol — config above; open an issue if anything misbehaves.
  235. ## Use it from Python
  236. ```python
  237. from claude_real_video import process
  238. r = process("https://youtu.be/...", "out", lang="en")
  239. print(r.frame_count, r.transcript_path)
  240. ```
  241. ---
  242. ## How it works
  243. 1. **Fetch** — `yt-dlp` for URLs (optional cookies), or copy a local file.
  244. 2. **Extract** — one chronological `ffmpeg select` pass grabs every scene change
  245. *plus* a density floor (at least one frame every `--fps-floor` seconds), so
  246. fast cuts and slow screencasts are both covered.
  247. 3. **Dedup** — three channels against a **sliding window** of the last
  248. `--dedup-window` kept frames, so an A-B-A cutaway doesn't re-send a shot the
  249. model has already seen. A *global* channel measures real pixel difference
  250. (downscaled RGB, not a perceptual hash — hashes go blind on flat colours and
  251. equal-luma hue changes); `--dedup-threshold` is the % of it that must change.
  252. A *settled-local* channel (v0.7.4) catches what the global one can't see:
  253. thin pen strokes, caption/text-card swaps and small UI updates that average
  254. out to ~0% globally. It looks, on a finer signature, for a region that
  255. differs strongly from every recent kept frame (with 1px shift tolerance, so
  256. film grain and frame jitter don't trigger) *and* is no longer changing — a
  257. settled new state, not motion mid-flight — with a cooldown so continuous
  258. motion that pauses every second (a waving flag, drifting smoke) can't keep
  259. re-firing. The final frame is evaluated even if still in motion (so a video's closing state is never lost), but it must clear both contrast gates like any other frame. `--report` writes `report.html` showing every keep/drop decision
  260. with its diff % (settled-local keeps are labelled), for tuning.
  261. 4. **Text** — if the video **already has subtitles** (a sidecar `.srt`/`.vtt` next to a
  262. local file, or an embedded subtitle track), those are used as the transcript —
  263. faster and more accurate than re-transcribing. Only when there are no subtitles
  264. does it fall back to **Whisper** on the audio (skipped cleanly if there's no audio).
  265. 5. **Audio** *(optional, `--keep-audio`)* — save the **full original soundtrack**
  266. (`audio.m4a`: music + speech + effects, copied losslessly when possible). The
  267. transcript only has the *words*; the audio file lets a model that can listen
  268. (Gemini, GPT-4o, …) actually *hear* the music and tone.
  269. 6. **Timestamps** — every kept frame's source-video time survives the whole
  270. pipeline (extraction → dedup → `--max-frames` thinning → renaming) and is
  271. written to `frames.json` (`file` / `timestamp_sec` / `timestamp` /
  272. `selection_reason`). Cite visual evidence as `frame_012 @ 00:03:41`, align
  273. frames with `transcript.json` segments, or feed the map to a video-RAG
  274. pipeline. In `viewer.html`, click any keyframe → "play video from here".
  275. 7. **Manifest** — `MANIFEST.txt` summarises everything for the model.
  276. So the model can **see** (key frames), **read** (transcript) and — with `--keep-audio` —
  277. **hear** (full soundtrack) the video. The transcript is plain text any model can read;
  278. the tool **doesn't burn subtitles into the video** — burning is a presentation choice,
  279. not something needed to make a video AI-readable.
  280. ---
  281. ## Notes
  282. - Only download content you have the right to. The `--cookies` option is for
  283. your own, authorised access — don't ship credentials in a repo.
  284. - Use one output folder per video. Re-running into a folder that already holds
  285. an analysis is refused (so two videos never mix); pass `--overwrite` to replace it.
  286. ## crv Pro — understand *how* a video was shot
  287. The free tool gives your AI keyframes and a transcript — enough to know **what** a video is about. **crv Pro adds everything else: how it's shot, how it's cut, how it's spoken, what it feels like.** All computed on your machine, written as plain text any LLM can read.
  288. - **Camera & pacing (`--motion`)** — every shot auto-labelled: static, pan, tilt, zoom, handheld. Full shot table: per-shot duration, cuts per minute, pacing across open/middle/close. High-motion shots get 0.2s-apart burst frames.
  289. - **Sound & emotion (`--senses`)** — voice emotion, tone curves and audio events (laughter, SFX, ambience) timestamped segment by segment. Vocals and music auto-separated: emotion reads the clean voice, music gets its own BPM + energy track. No-dialogue footage (MVs, film) falls back to reading mood from color and light.
  290. - **Interactive viewer (`--viewer`)** — one self-contained web page per analysis: the video, a clickable event timeline that jumps to the second, a transcript that highlights along with playback. EN / 繁中 / 简中.
  291. - **Two reports, one flag (`--ai-report`)** — with your own API key: one report on how it's shot, one on what it says.
  292. - **Breakdown report (`--breakdown`)** — hook analysis, pacing curve, camera language, and a rubric your own LLM completes into a full teardown.
  293. - **Memory across your library (`crv-pro-ask`)** — search everything Pro has watched by what the camera and the voice did: `--camera zoom` (every zoom you've ever watched), `--track emotion --label angry`, `--rhythm` (cuts/min ranked). The free `crv-ask` searches words; these search measurements the free edition never takes.
  294. One-time price **$29**:
  295. - **Buy on Capafy** (instant download, license key included): https://capafy.ai/agent/llm-real-video-pro-let-any-llm-watch-videos/5451082151
  296. - **Buy with credit card** (Lemon Squeezy checkout, instant download): https://leoaido.lemonsqueezy.com/checkout/buy/ff552000-adc0-49f1-8eec-5e8ada1905a1
  297. - Product page & demo: https://leoaido.com/crv-pro/
  298. ---
  299. **Following the build?** I'm documenting the road from open-source tool to first paying customer, in public — [@LeoAidoAI on X](https://x.com/LeoAidoAI).
  300. ## FAQ
  301. ### What is the best tool to let an LLM watch or analyze a video?
  302. It depends on what "watch" means for you. If you just want one answer about one clip and don't mind uploading it, a hosted multimodal model (e.g. Gemini) is the shortest path. If you want **any** LLM — Claude, GPT, Gemini or a local model — to analyze video **reproducibly and locally**, you want a preprocessing pipeline: scene-aware keyframes + a timestamped transcript, handed to the model as evidence it can cite. That pipeline is exactly what `claude-real-video` does, in one command, with nothing leaving your machine. Uniform frame sampling (1 fps) either misses cuts or floods the context window; scene-aware extraction keeps the frames that carry information.
  303. ### How can I make Claude understand a video?
  304. Claude cannot ingest video files directly. The working approach:
  305. ```bash
  306. pip install "claude-real-video[fast]"
  307. npx skills add HUANGCHIHHUNGLeo/claude-real-video # or install via the Claude Code plugin marketplace
  308. ```
  309. Then in Claude Code: `Analyze this video: /path/to/video.mp4`. The skill extracts scene-aware keyframes, a timestamped transcript (`transcript.json`), a frame→timestamp map (`frames.json`) and a `MANIFEST.txt` that tells the model how to read the folder — so Claude can cite `frame_012 @ 00:03:41` instead of guessing.
  310. ### What is claude-real-video?
  311. An MIT-licensed Python CLI (`crv`) that turns a video into what an LLM can actually read: scene-aware keyframes (with real source timestamps that survive dedup and renaming), sliding-window deduplication so small-subject motion isn't thrown away, and local Whisper transcription with optional speaker labels. Works with YouTube URLs or local files, runs 100% locally. It exists because subtitles alone are not watching — models that only read the transcript hallucinate everything visual.
  312. ## Who makes this
  313. Built by Leo Huang — a one-person company running on AI.
  314. I post what actually breaks and what works while building tools like this:
  315. https://x.com/LeoAidoAI
  316. ## License
  317. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.