Skip to content
Back to search

dsh-video-lens

Video understanding for DeepSeek Harness — give text-only agents eyes and ears on video.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/dundunhan--dsh-video-lens/badge.svg)](/listings/dundunhan--dsh-video-lens)

HTML

<a href="/listings/dundunhan--dsh-video-lens"><img src="/api/listings/dundunhan--dsh-video-lens/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Prerequisites: Node.js ≥ 20, `ffmpeg` ≥ 6.0 (recommended) with `ffprobe` on PATH (`brew install ffmpeg` / `apt install ffmpeg`).

Option A — npm (recommended)

# in your DSH profile directory (the one containing package.json)
pnpm add dsh-video-lens

Option B — from source (development)

Clone the repo, then mount it into your DSH profile via a local link:

git clone https://github.com/dundunhan/dsh-video-lens.git

Either way, register the bundle in your profile's `package.json` — **this exact block is the full profile configuration**:

{
  "dependencies": {
    "dsh-video-lens": "^0.3"
  },
  "dsh": {
    "profile": {
      "bundles": [
        "@deepseek-ai/dsh-base",
        "@deepseek-ai/dsh-web-app",
        "dsh-video-lens"
      ]
    }
  }
}

Then export the keys and restart the profile:

export VIDEO_LENS_API_KEY=sk-...        # vision
export VIDEO_LENS_ASR_KEY=sk-...        # optional, ASR

> **Do not install the host runtime yourself.** `@deepseek-ai/dsh-tools` is declared as an **optional peer**: the plugin always uses the `dsh-tools` that already ships with your DSH installation / DSH Desktop. Adding it to your profile as a dependency — or pinning one exact `-rc` version, which is what 0.3.1 did — installs a second, older runtime next to the host's, makes the Loader entry fail to import, and takes the whole plugin tree (and the app) down with it.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 187 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # dsh-video-lens
  2. **Video understanding for DeepSeek Harness — give text-only agents eyes and ears on video.**
  3. A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plugin that lets text-only LLM agents understand local video files. It provides two tools:
  4. | Tool | What it does |
  5. |---|---|
  6. | `video_probe` | Cheap, instant metadata via `ffprobe`: container, duration, resolution, fps, codecs, audio tracks, subtitles. |
  7. | `video_analyze` | Content understanding: **scene-change-aware frame sampling** (`ffmpeg scdet`), **optional ASR transcript** (speech with timestamps), fused with any OpenAI-compatible vision model into structured evidence JSON. |
  8. | `video_ask` | **Time-anchored Q&A**: parses explicit time references ("at 3:20", "第2分钟") or locates relevant speech via transcript keyword matching, re-samples frames from the matched windows, and answers with grounded evidence (answer + confidence + supporting timestamps). |
  9. > v0.3.2. The plugin never locks you into a provider: vision and ASR are both OpenAI-compatible endpoints configured via `baseUrl` + `model` + key env var.
  10. ## How it works
  11. ```
  12. video file ──► video_probe ──► ffprobe ──► compact metadata JSON
  13. └─► video_analyze ──► scdet scene detection ──► shot boundaries
  14. ├─► ffmpeg frame sampling (one representative frame per shot, capped)
  15. ├─► ffmpeg audio extract ──► ASR transcript (timestamped) [optional]
  16. └─► OpenAI-compatible vision API ──► evidence JSON
  17. ```
  18. - Scene changes are detected with ffmpeg's `scdet` filter (ffmpeg ≥ 6.0). Videos without detectable cuts fall back to uniform midpoint sampling.
  19. - ASR is **strictly additive**: if `asrApiKeyEnv` is unset or the provider fails, the visual analysis still completes and `transcript` is `null`.
  20. - All media work is delegated to `ffmpeg`/`ffprobe` on `PATH` — no native decoding in the agent.
  21. ## Install
  22. Prerequisites: Node.js ≥ 20, `ffmpeg` ≥ 6.0 (recommended) with `ffprobe` on PATH (`brew install ffmpeg` / `apt install ffmpeg`).
  23. ### Option A — npm (recommended)
  24. ```bash
  25. # in your DSH profile directory (the one containing package.json)
  26. pnpm add dsh-video-lens
  27. ```
  28. ### Option B — from source (development)
  29. Clone the repo, then mount it into your DSH profile via a local link:
  30. ```bash
  31. git clone https://github.com/dundunhan/dsh-video-lens.git
  32. ```
  33. Either way, register the bundle in your profile's `package.json` — **this exact block is the full profile configuration**:
  34. ```json
  35. {
  36. "dependencies": {
  37. "dsh-video-lens": "^0.3"
  38. },
  39. "dsh": {
  40. "profile": {
  41. "bundles": [
  42. "@deepseek-ai/dsh-base",
  43. "@deepseek-ai/dsh-web-app",
  44. "dsh-video-lens"
  45. ]
  46. }
  47. }
  48. }
  49. ```
  50. Then export the keys and restart the profile:
  51. ```bash
  52. export VIDEO_LENS_API_KEY=sk-... # vision
  53. export VIDEO_LENS_ASR_KEY=sk-... # optional, ASR
  54. ```
  55. > **Do not install the host runtime yourself.** `@deepseek-ai/dsh-tools` is declared as an **optional peer**: the plugin always uses the `dsh-tools` that already ships with your DSH installation / DSH Desktop. Adding it to your profile as a dependency — or pinning one exact `-rc` version, which is what 0.3.1 did — installs a second, older runtime next to the host's, makes the Loader entry fail to import, and takes the whole plugin tree (and the app) down with it.
  56. ## Configuration
  57. All options are DSH config values:
  58. | Key | Default | Meaning |
  59. |---|---|---|
  60. | `visionBaseUrl` | `https://api.siliconflow.cn/v1` | Vision endpoint (OpenAI-compatible) |
  61. | `visionModel` | `Qwen/Qwen3-VL-8B-Instruct` | Vision model name |
  62. | `visionApiKeyEnv` | `VIDEO_LENS_API_KEY` | Env var holding the vision key |
  63. | `asrBaseUrl` | `https://api.siliconflow.cn/v1` | ASR endpoint (OpenAI-compatible `/audio/transcriptions`) |
  64. | `asrModel` | `FunAudioLLM/SenseVoiceSmall` | ASR model name |
  65. | `asrApiKeyEnv` | `VIDEO_LENS_ASR_KEY` | Env var holding the ASR key |
  66. | `maxFrames` | `12` | Frame budget cap (1–max); actual count is duration-adaptive (~1 frame per 30s, denser for short videos) |
  67. | `frameMaxWidth` | `768` | Max frame width; keeps payloads small |
  68. | `frameQuality` | `4` | JPEG quality (ffmpeg `-q:v`) |
  69. | `sceneThreshold` | `10` | `scdet` threshold (0–100); higher = fewer cuts |
  70. | `askPaddingSec` | `2` | `video_ask` window padding around matched transcript segments |
  71. | `vlmMaxTokens` | `1500` | Vision model max output tokens |
  72. | `vlmTimeoutMs` | `90000` | Vision call timeout |
  73. | `asrTimeoutMs` | `120000` | ASR call timeout |
  74. ## Usage
  75. Ask the agent:
  76. > "What's in /tmp/demo.mp4?"
  77. The agent calls `video_probe` first, then `video_analyze`. Evidence includes:
  78. ```json
  79. {
  80. "metadata": { "container": "mov,mp4,m4a,3gp,3g2,mj2", "durationSec": 268.4, "...": "..." },
  81. "shots": [{ "timeSec": 12.3, "score": 45.2 }],
  82. "framesSampled": [{ "timestampSec": 5.5, "jpegBytes": 12345 }],
  83. "transcript": {
  84. "text": "…",
  85. "segments": [{ "start": 0.0, "end": 2.4, "text": "…" }],
  86. "language": "zh"
  87. },
  88. "visionModel": "Qwen/Qwen3-VL-8B-Instruct",
  89. "analysis": { "overall_summary": "…", "timeline": [{"timestamp_sec": 5.5, "description": "…"}], "on_screen_text": "…", "visual_style": "…", "notable_moments": "…" }
  90. }
  91. ```
  92. ## Permissions & security
  93. > Read this before using or redistributing. DSH plugins run in the **host process as trusted code** and there is **no official plugin review** — self-review is on the author. See [SECURITY.md](SECURITY.md).
  94. **What this plugin does**
  95. - **Reads**: any local file path the agent passes to its tools (via `ffprobe`/`ffmpeg`).
  96. - **Executes**: `ffprobe` and `ffmpeg` from `PATH` (never a shell — argv arrays only).
  97. - **Network**: one outbound call per `video_analyze` to the configured `visionBaseUrl` (frames + vision key), and optionally one to `asrBaseUrl` (audio + ASR key).
  98. - **Does not**: execute shells, eval code, phone home, auto-update, or read files on its own.
  99. **Operator responsibilities**
  100. - Keys are only as safe as the endpoints they are sent to — configure only endpoints you trust.
  101. - The real access boundary is the DSH host sandbox; the plugin's readability check is a UX guard, not a security boundary.
  102. - Payload sizes are bounded: `maxFrames` × ~100–300 KB (768px JPEG) per analysis call.
  103. ## Compatibility
  104. - Tested with DSH profile bundles `@deepseek-ai/dsh-base` + `@deepseek-ai/dsh-web-app`.
  105. - Host runtime is **not pinned**: `@deepseek-ai/dsh-tools` is an optional peer resolved from the host installation, so the plugin follows the core it is loaded by (verified against core `0.1.0-rc.7` and `0.1.5-rc.2`, the upstream version DSH Desktop 2.0.5 pins).
  106. - Node ≥ 20 (uses `AbortSignal.any` / built-in `fetch` / `FormData`).
  107. - ffmpeg ≥ 6.0 for `scdet`; older versions degrade to uniform sampling.
  108. - macOS verified. Windows: the 0.3.1 boot failure reported on Windows was **not** platform-specific — it was the pinned old `dsh-tools` runtime (see Troubleshooting); the code paths themselves are OS-neutral (`ffmpeg`/`ffprobe` are spawned via argv, no shell).
  109. ## Troubleshooting
  110. **`dsh-plugin-desktop: plugin tree failed to load: failed to apply loader entry include (cordis:include): AggregateError: loader entries failed to apply` — the client no longer starts.**
  111. 0.3.1 hit this on DSH Desktop 2.0.5. The full error underneath is an import failure of the plugin (or of the host `tools` entry):
  112. ```
  113. failed to import loader entry video-lens (dsh-video-lens): The requested module '@deepseek-ai/dsh-llm' does not provide an export named 'CallId'
  114. [cause]: profiles/<name>/node_modules/@deepseek-ai/dsh-tools/lib/index.js:4
  115. ```
  116. Cause: 0.3.1 pinned `@deepseek-ai/dsh-tools@0.1.0-rc.7`, so the profile got a second, older `dsh-tools` while the host ran a newer core (`0.1.5-rc.2`). Any failing Loader entry fails the whole tree, so the app cannot boot until the plugin is removed.
  117. Recovery (0.3.1 installed and the app will not start):
  118. 1. Use the client's Recovery page to return to the last healthy profile, **or** remove the plugin from the profile: `dsh plugin --profile <name> remove dsh-video-lens` (Desktop: run that in its terminal).
  119. 2. Install `dsh-video-lens@^0.3.2`, where the host runtime is an optional peer and nothing is installed into the profile.
  120. ## Testing
  121. ```bash
  122. npm test # packaging + unit + network + e2e (needs ffmpeg)
  123. node test/host-boot.test.mjs latest # boot a real host with the plugin mounted
  124. node test/host-boot.test.mjs 0.1.0-rc.7 # exactly one core version
  125. node test/host-boot.test.mjs desktop-stable # the upstream core DSH Desktop currently pins
  126. ```
  127. - `test/packaging.test.mjs` — manifest contract: official host runtime packages (`@deepseek-ai/dsh-*`) must stay **optional peers**, never installed dependencies, and the published bundle patch / entry must resolve.
  128. - `test/host-boot.test.mjs` — end-to-end guard: packs the plugin, installs it into a throwaway profile under the DSH Desktop profile contract (`nodeLinker: hoisted`, `autoInstallPeers: false`), then boots that profile with a real DSH host and requires the Web surface to come up — while asserting the profile installed **no** host runtime package. Needs `pnpm` on PATH (or `PNPM_BIN`) and network access; no API keys and no ffmpeg. CI runs it for `latest`, `0.1.0-rc.7`, and the core currently pinned by DSH Desktop.
  129. ## Uninstall
  130. 1. Remove `dsh-video-lens` from `dsh.profile.bundles` in your profile `package.json`.
  131. 2. Remove the dependency: `pnpm remove dsh-video-lens` (npm install) — or delete the `link:` entry if you installed from source — then reinstall the profile.
  132. ## Roadmap
  133. - v1.0: frame caching by file hash, evaluation table in README (5 video types × metrics), publish to npm (in progress).
  134. - Beyond: native video-input models as an optional fast path when the configured VLM supports them.
  135. ## License
  136. MIT — see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.