Skip to content
Back to search

claude-video-vision

MarketplaceWorks with Claude CodeGitHub

Give Claude the ability to watch and understand videos

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/jordanrendric-claude-video-vision/badge.svg)](/listings/jordanrendric-claude-video-vision)

HTML

<a href="/listings/jordanrendric-claude-video-vision"><img src="/api/listings/jordanrendric-claude-video-vision/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Install the plugin

Inside Claude Code, run these commands **one at a time**:

/plugin marketplace add https://github.com/jordanrendric/claude-video-vision

Then:

/plugin install claude-video-vision

The MCP server will auto-install via `npx` from [npm](https://www.npmjs.com/package/claude-video-vision) on first use — no build step required.

Alternative: local development

git clone https://github.com/jordanrendric/claude-video-vision.git
claude --plugin-dir /path/to/claude-video-vision

2. Configure

Inside Claude Code, run the interactive wizard:

/claude-video-vision:setup-video-vision

It will walk you through backend selection, whisper configuration (if local), frame options, and dependency verification.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 33 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
  3. "name": "claude-video-vision",
  4. "description": "Give Claude the ability to watch and understand videos",
  5. "version": "1.4.0",
  6. "owner": {
  7. "name": "Jordan Vasconcelos",
  8. "url": "https://github.com/jordanrendric"
  9. },
  10. "plugins": [
  11. {
  12. "name": "claude-video-vision",
  13. "description": "Give Claude the ability to watch and understand videos — extracts frames via ffmpeg and processes audio via Gemini API, local Whisper, or OpenAI API. Multimodal perception layer for Claude Code.",
  14. "version": "1.4.0",
  15. "source": "./",
  16. "category": "productivity",
  17. "homepage": "https://github.com/jordanrendric/claude-video-vision",
  18. "author": {
  19. "name": "Jordan Vasconcelos",
  20. "url": "https://github.com/jordanrendric"
  21. },
  22. "tags": [
  23. "video",
  24. "vision",
  25. "multimodal",
  26. "ffmpeg",
  27. "whisper",
  28. "gemini",
  29. "perception"
  30. ]
  31. }
  32. ]
  33. }
README.md · README · 236 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src="./assets/hero.avif" alt="claude-video-vision" width="100%" />
  3. </p>
  4. # Claude Code Video Vision
  5. Give Claude the ability to **watch and understand videos**.
  6. A Claude Code plugin that extracts frames via ffmpeg and processes audio via multiple backends (Gemini API, local Whisper, or OpenAI API). Claude receives frames as images and audio transcription with timestamps — the plugin is a **perception layer**, not an interpretation layer.
  7. ## Features
  8. - **Multimodal perception** — Claude sees video frames directly and reads audio transcriptions with timestamps
  9. - **YouTube URL support** — Pass a YouTube URL directly; the MCP server downloads it with `yt-dlp`, preserving source metadata and captions for context
  10. - **Flexible backends** — Choose between cloud APIs or fully local processing
  11. - **Adaptive extraction** — Claude adjusts fps, time range, and resolution based on your question
  12. - **Auto-installation** — Whisper models download automatically on first use
  13. - **Interactive setup wizard** — `/setup-video-vision` walks you through configuration
  14. ## Quick Start
  15. ### 1. Install the plugin
  16. Inside Claude Code, run these commands **one at a time**:
  17. ```
  18. /plugin marketplace add https://github.com/jordanrendric/claude-video-vision
  19. ```
  20. Then:
  21. ```
  22. /plugin install claude-video-vision
  23. ```
  24. The MCP server will auto-install via `npx` from [npm](https://www.npmjs.com/package/claude-video-vision) on first use — no build step required.
  25. Alternative: local development
  26. ```bash
  27. git clone https://github.com/jordanrendric/claude-video-vision.git
  28. claude --plugin-dir /path/to/claude-video-vision
  29. ```
  30. ### 2. Configure
  31. Inside Claude Code, run the interactive wizard:
  32. ```
  33. /claude-video-vision:setup-video-vision
  34. ```
  35. It will walk you through backend selection, whisper configuration (if local), frame options, and dependency verification.
  36. ## Usage
  37. ### Slash command
  38. ```
  39. /watch-video path/to/video.mp4
  40. /watch-video tutorial.mp4 "what language is used in this tutorial?"
  41. /watch-video https://www.youtube.com/watch?v=... "summarize this video"
  42. ```
  43. ### Conversational
  44. Just mention a video file or YouTube URL — Claude will detect it:
  45. > "analyze this video for me: ~/Downloads/demo.mp4"
  46. >
  47. > "take a look at the first second of ~/videos/bug-report.mov"
  48. >
  49. > "summarize this YouTube Short: https://www.youtube.com/shorts/..."
  50. Claude adapts parameters automatically:
  51. - "the first second" → extracts at original fps from `00:00:00` to `00:00:01`
  52. - "summarize this 1h lecture" → low fps, full duration
  53. - "what text is on screen at 1:30?" → high resolution, narrow time window
  54. ## Backends
  55. | Backend | Audio processing | Cost | Setup |
  56. |---------|------------------|------|-------|
  57. | **Gemini API** | Native (speech + non-speech events) | Free tier: 1500 req/day | `GEMINI_API_KEY` env var |
  58. | **Local (Whisper)** | `whisper.cpp` or Python `openai-whisper` | Free, fully offline | `brew install whisper-cpp` + auto model download |
  59. | **OpenAI API** | OpenAI Whisper API | Paid per usage | `OPENAI_API_KEY` env var |
  60. **All backends** extract video frames via ffmpeg — Claude always has direct visual access.
  61. ## Architecture
  62. ```
  63. ┌───────────────────────────────────────────────────────┐
  64. │ Claude Code (your session) │
  65. │ │
  66. │ /watch-video ──→ Skill: video-perception │
  67. │ │ │
  68. │ ▼ │
  69. │ MCP tool: video_watch │
  70. │ │ │
  71. └────────────────────────┼──────────────────────────────┘
  72. │
  73. ▼
  74. ┌────────────────────────────────────┐
  75. │ MCP Server (Node.js) │
  76. │ │
  77. │ ┌──────────┐ ┌──────────────┐ │
  78. │ │ ffmpeg │ │ Audio backend│ │
  79. │ │ frames │ ║ │ (parallel) │ │
  80. │ └──────────┘ └──────────────┘ │
  81. │ │ │ │
  82. └───────┼─────────────────┼──────────┘
  83. ▼ ▼
  84. base64 images transcription
  85. + timestamps + audio events
  86. │ │
  87. └────────┬────────┘
  88. ▼
  89. Claude receives both
  90. ```
  91. ## Requirements
  92. - **Node.js 24+** (for the MCP server)
  93. - **ffmpeg** (auto-detected, install instructions provided by setup wizard)
  94. - **yt-dlp** (optional, required only for YouTube URLs; `brew install yt-dlp` on macOS)
  95. - **Backend-specific**:
  96. - Gemini API: free API key from [ai.google.dev](https://ai.google.dev/gemini-api/docs/api-key)
  97. - Local: `brew install whisper-cpp` (macOS) or equivalent
  98. - OpenAI: API key from OpenAI
  99. ## MCP Tools
  100. The plugin exposes 6 MCP tools:
  101. - `video_watch` — Extract frames + process audio (main tool)
  102. - `video_analyze` — Analyze video structure with ffmpeg filters before extraction
  103. - `video_detail` — Drill into specific cached or newly extracted moments
  104. - `video_info` — Get video metadata without processing
  105. - `video_configure` — Change settings
  106. - `video_setup` — Check and guide dependency installation
  107. ## Slash Commands
  108. - `/watch-video <path> [question]` — Analyze a video
  109. - `/setup-video-vision` — Interactive configuration wizard
  110. ## Configuration
  111. Settings are stored in `~/.claude-video-vision/config.json`:
  112. ```json
  113. {
  114. "backend": "local",
  115. "whisper_engine": "cpp",
  116. "whisper_model": "auto",
  117. "whisper_at": false,
  118. "frame_mode": "images",
  119. "frame_format": "jpeg",
  120. "frame_resolution": 512,
  121. "default_fps": "auto",
  122. "max_frames": 100,
  123. "frame_describer_model": "sonnet",
  124. "enable_index": false,
  125. "session_max_age_days": 7,
  126. "downloads_max_age_days": 7
  127. }
  128. ```
  129. `frame_format` can be `jpeg`, `png`, or `webp`. `jpeg` remains the default for backwards compatibility; `png` is useful for screen recordings where text and sharp UI edges should stay lossless.
  130. **Whisper models** auto-download to `~/.claude-video-vision/models/` on first use. Available: `tiny`, `base`, `small`, `medium`, `large-v3-turbo`, `large-v3`, `auto` (picks best for your RAM).
  131. ## YouTube Transcripts
  132. For YouTube URLs, the server uses this transcript order:
  133. 1. Manual YouTube subtitles when an English track is available.
  134. 2. YouTube automatic captions when manual subtitles are not available.
  135. 3. The configured audio backend when captions are missing, empty, or cover too little of a longer video.
  136. Audio results label provenance with `transcription_source`, for example `youtube_subtitles` or `youtube_auto_captions`, so Claude can treat manual subtitles as stronger evidence than auto-captions.
  137. ## Troubleshooting
  138. ### Windows: MCP server fails to connect with `Unknown command: "claude-video-vision@latest"`
  139. The plugin starts its MCP server with `npx -y claude-video-vision@latest`. If `claude mcp list` shows the server as failed and running that command yourself prints `Unknown command: "claude-video-vision@latest"` followed by a hint about `npm help`, then `npm` received the package name as if it were a subcommand. That points to a broken or outdated `npx` on your machine, not to the plugin.
  140. 1. **Check your npm install** in PowerShell:
  141. ```powershell
  142. npx --version
  143. npm --version
  144. where.exe npx
  145. ```
  146. Both versions should match (7 or newer), and `where.exe npx` should point to `npx.cmd` inside your Node.js install directory.
  147. 2. **Fix it** by updating npm (`npm install -g npm@latest`) or reinstalling Node.js 24+ LTS, then restart Claude Code.
  148. 3. **Workaround** if it still fails: install the server globally and register it yourself.
  149. ```powershell
  150. npm install -g claude-video-vision
  151. claude mcp add --scope user claude-video-vision -- claude-video-vision
  152. ```
  153. The plugin's own server entry will keep showing as failed; you can ignore it. A global install does not update itself, so re-run `npm install -g claude-video-vision` to pick up new releases.
  154. ## Status
  155. **v1.4.0** — see the [CHANGELOG](./CHANGELOG.md) for release history. CI runs on Ubuntu and macOS with Node.js 24 and 26.
  156. ## License
  157. MIT — see [LICENSE](./LICENSE).
  158. ## Author
  159. **Jordan Vasconcelos**
  160. - GitHub: [@jordanrendric](https://github.com/jordanrendric)
  161. - LinkedIn: [jordanvasconcelos](https://www.linkedin.com/in/jordanvasconcelos/)
  162. - Instagram: [@jordanvasconcelos__](https://www.instagram.com/jordanvasconcelos__/)
  163. - X/Twitter: [@jordanrendric](https://x.com/jordanrendric)
  164. ## Star History
  165. <a href="https://www.star-history.com/?repos=jordanrendric%2Fclaude-video-vision&type=date&legend=top-left">
  166. <picture>
  167. <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=jordanrendric/claude-video-vision&type=date&theme=dark&legend=top-left" />
  168. <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=jordanrendric/claude-video-vision&type=date&legend=top-left" />
  169. <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=jordanrendric/claude-video-vision&type=date&legend=top-left" />
  170. </picture>
  171. </a>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.