Skip to content
Back to search

ffmpeg-skill

SkillWorks with Claude CodeGitHub

ffmpeg-skill as a Claude Code plugin: local FFmpeg video and audio editing with a machine-readable contract.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/kajisho5/ffmpeg-skill with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/kajisho5/ffmpeg-skill'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/kajisho5-ffmpeg-skill/badge.svg)](/listings/kajisho5-ffmpeg-skill)

HTML

<a href="/listings/kajisho5-ffmpeg-skill"><img src="/api/listings/kajisho5-ffmpeg-skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# 0. FFmpeg itself, once (Ubuntu/Debian shown; macOS and Windows are in the Install table below)
sudo apt install ffmpeg

# 1. install the skill for Claude Code (Cursor: --cursor, Codex: --codex, all three: --all)
npx ffmpeg-skill

# 2. check the machine: ffmpeg, ffprobe and every FFmpeg component the tools need
npx ffmpeg-skill doctor

# 3. (for agent frameworks) read the machine-readable contract
npx ffmpeg-skill contract --json | head -40

Captions from speech (`caption.py --transcribe`) need one local speech engine, installed separately. `pip install faster-whisper` is the simplest; the first run downloads its model, and the audio never leaves the machine. Without an engine, everything else works, `doctor` lists the engine as missing optional, and `--transcribe` refuses with the install line for each engine it supports.

Already installed? re-run `npx ffmpeg-skill` to refresh `~/.claude/skills/ffmpeg-skill`. Copies are not updated automatically.

`doctor`'s overall `ok` and a single tool's `usable` are different signals: `ok` is false when anything *some* tool requires is missing, so one missing filter (a plain Homebrew `ffmpeg` without `subtitles`, or a build without libvidstab) turns it false even though most tools still run. Check `doctor --json`'s `tools` field for the per-tool answer, not just `ok`.

Then talk to your agent:

> "Take `interview.mp4`, keep 0:45–3:10 and 5:00–6:30, and make it exactly 60 seconds for Reels."

The agent runs `probe.py`, `cut.py --segments 0:45-3:10,5:00-6:30`, `fit.py --duration 60 --aspect 9:16 --fit crop`, `export.py --preset reels`, `check.py --platform reels` and `look.py`, then reports "final.mp4: 59.98 s, 1080×1920, 30 fps, AAC stereo" with the contact sheet it inspected.

Deliver to a platform

One command per destination, with the app's own UI taken into account:

python3 $S/render.py talk.mp4 --template tiktok --cues cues.txt

That fills the shipped `templates/tiktok.json`: 9:16 crop, captions popped word by word *above* TikTok's description bar and clear of its like column, −14 LUFS, the `tiktok` export preset, and a `check.py --platform tiktok` on the file it wrote. The caption size comes from the delivery table, so since 1.17.1 the filled project also states `"fit_size": "on"`: a size nobody asked for shrinks to fit the cue instead of splitting the sentence across two cues. A template that states its own `fit_size`, and a `--brand` that states a caption size, still win, and a project with `"fit_size": "off"` renders the captions 1.17.0 rendered. Templates ship for `tiktok`, `reels`, `shorts`, `youtube-shorts`, `youtube`, `x`, `linkedin`, `facebook`, `podcast` and `audiogram`; `--template all` (or a comma-separated list) renders the seven video destinations (`tiktok`, `reels`, `shorts`, `youtube`, `x`, `linkedin`, `facebook`) from the same edit and writes a `<name>_pack.md` table of what each one produced. Files land next to the input unless `-o` says otherwise, and `--dry-run` shows every planned command rather than a result. `render.py --list-templates` prints them with their frames, limits and safe zones. Alias spellings work everywhere a platform is named (`youtube-shorts` = `shorts`, `ig` = `reels`, `twitter` = `x`, `fb` = `facebook`).

The tools also work on their own, from any shell:

S=~/.claude/skills/ffmpeg-skill/scripts
python3 $S/probe.py input.mp4 --compact
python3 $S/fit.py input.mp4 --duration 60 --aspect 9:16 --dry-run    # print the plan, run nothing
python3 $S/export.py input.mp4 --preset reels --json                 # structured result with a probe of the output

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 20 lines
  1. {
  2. "name": "ffmpeg-skill",
  3. "description": "ffmpeg-skill as a Claude Code plugin: local FFmpeg video and audio editing with a machine-readable contract.",
  4. "owner": {
  5. "name": "kajisho5",
  6. "url": "https://github.com/kajisho5"
  7. },
  8. "plugins": [
  9. {
  10. "name": "ffmpeg-skill",
  11. "source": {
  12. "source": "git-subdir",
  13. "url": "https://github.com/kajisho5/ffmpeg-skill.git",
  14. "path": "ffmpeg-skill",
  15. "ref": "plugin"
  16. },
  17. "description": "Local FFmpeg video/audio editing for coding agents: a machine-readable contract, an MCP server and a probe-first / verify-last workflow."
  18. }
  19. ]
  20. }
README.md · README · 705 lines
  1. <p align="center">
  2. <picture>
  3. <source media="(prefers-color-scheme: dark)" srcset="brand/logo/ffmpeg-skill-horizontal-on-dark.svg">
  4. <img src="brand/logo/ffmpeg-skill-horizontal.svg" height="72" alt="ffmpeg-skill">
  5. </picture>
  6. </p>
  7. <p align="center"><b>Give your coding agent a video editor.</b><br>
  8. Claude Code, Cursor, Codex or any MCP client edits your footage with local FFmpeg, then checks its own output.<br>
  9. No cloud · No API keys · Python standard library</p>
  10. <p align="center">
  11. <a href="https://kajisho5.github.io/ffmpeg-skill/"><b>Website</b></a> ·
  12. <a href="#quick-start"><b>Quick start</b></a> ·
  13. <a href="docs/demos.md">Demos</a> ·
  14. <a href="#tools">Tools</a> ·
  15. <a href="#mcp">MCP</a> ·
  16. <a href="https://www.npmjs.com/package/ffmpeg-skill">npm</a>
  17. </p>
  18. <p align="center">
  19. <a href="https://github.com/kajisho5/ffmpeg-skill/actions/workflows/ci.yml"><img src="https://github.com/kajisho5/ffmpeg-skill/actions/workflows/ci.yml/badge.svg" alt="tests"></a>
  20. <a href="https://scorecard.dev/viewer/?uri=github.com/kajisho5/ffmpeg-skill"><img src="https://api.scorecard.dev/projects/github.com/kajisho5/ffmpeg-skill/badge" alt="OpenSSF Scorecard"></a>
  21. <a href="https://www.npmjs.com/package/ffmpeg-skill"><img src="https://img.shields.io/npm/v/ffmpeg-skill" alt="npm"></a>
  22. <a href="#ffmpeg-compatibility"><img src="https://img.shields.io/badge/ffmpeg-6.1%20%7C%207.1%20%7C%208%20%7C%209%20tested-orange" alt="FFmpeg 6.1, 7.1, 8 and 9 tested in CI"></a>
  23. <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="MIT"></a>
  24. <a href="https://github.com/sponsors/kajisho5"><img src="https://img.shields.io/badge/sponsor-%E2%9D%A4-ea4aaa?logo=githubsponsors" alt="Sponsor"></a>
  25. </p>
  26. <p align="center">
  27. <a href="https://kajisho5.github.io/ffmpeg-skill/"><img src="brand/og.png" width="720" alt="ffmpeg-skill v3: give your coding agent a video editor. An agent runs probe, cut, fit, export and check, and the final file is measured: 59.98 s, 1080x1920, 30 fps."></a>
  28. </p>
  29. ```bash
  30. npx ffmpeg-skill
  31. ```
  32. <table>
  33. <tr>
  34. <td width="50%"><img src="docs/demos/captions_pop_karaoke.gif" alt="animated captions with a karaoke highlight"><br><sub><code>caption.py --animate pop --karaoke</code></sub></td>
  35. <td width="50%"><img src="docs/demos/reframe_crop.gif" alt="16:9 reframed to 9:16"><br><sub><code>fit.py --aspect 9:16 --fit crop</code></sub></td>
  36. </tr>
  37. <tr>
  38. <td width="50%"><img src="docs/demos/silence_removal.gif" alt="silence removed, timeline shorter"><br><sub><code>silence.py --threshold -35</code></sub></td>
  39. <td width="50%"><img src="docs/demos/loudness.gif" alt="waveform before and after loudness normalisation"><br><sub><code>loudness.py -I -14 --tp -1</code></sub></td>
  40. </tr>
  41. </table>
  42. Left half is the input, right half is what the command produced. **[Every before/after demo, with the exact command under each one →](docs/demos.md)** — all of it generated from synthetic footage by `python3 demos/build.py`, so you can rebuild every frame of it yourself.
  43. `ffmpeg-skill` is an [Agent Skill](https://docs.anthropic.com/en/docs/agents-and-tools/agent-skills) for Claude Code, Cursor, Codex and any agent that reads `SKILL.md`. It teaches the agent a fixed workflow (probe → edit losslessly where possible → check → verify) and ships **42 tools** that do the actual work with `ffmpeg` / `ffprobe`: cut, join, silence removal, fit to duration and aspect, captions and karaoke, overlays and motion graphics, HDR → SDR and LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, whole-edit project rendering, batch folders. Every tool is also callable as an MCP tool (`tools/list` advertises a core 12 by default to keep client context small, with `FFMPEG_SKILL_MCP_FULL=1` listing all 42; every tool is reachable by name through `tools/call` either way), and the whole set is described by a machine-readable contract.
  44. If `ffmpeg` and `python3` are on your PATH, it works: offline, on footage you would rather not upload.
  45. > **SPEC** (Self-Producing Execution Contract): each tool's `input_schema` — the part of its
  46. > contract and MCP tool definition that has to track the CLI flag-for-flag — is never
  47. > hand-authored beside the code. It is derived, at run time, from the same `argparse` parser that
  48. > already defines the CLI, and CI fails the build if any of it drifts.
  49. > → [full explanation](#what-is-spec)
  50. ---
  51. ## Standalone, and in an ecosystem
  52. **Standalone**, this is a local FFmpeg engine: probe → edit → verify, `npx ffmpeg-skill` and nothing else. No API key, no account, no other repo required. Everything above and below this section describes that standalone tool, and none of it changes if you never read the rest of this one.
  53. **In [kajisho5](https://github.com/kajisho5)'s wider video-production ecosystem**, this repo is the *hands*: it cuts, measures and exports files, and reports back in structured JSON. It does not decide what to cut, whether a deliverable is approvable, what makes a highlight interesting, or what a caption should say (the user's cue text is burned as written, never rewritten to fit) — those are a *brain*'s job, sitting in front of this engine, not inside it.
  54. | You want to... | Use |
  55. |---|---|
  56. | Cut / join / measure / export a file right now | **this repo** (`ffmpeg-skill`), standalone |
  57. | Decide cut points, approve a deliverable, plan a whole edit | [`video-production-agent`](https://github.com/kajisho5/video-production-agent) / [`AI-video-production-OS`](https://github.com/kajisho5/AI-video-production-OS) |
  58. | Build a typed editing graph across a workspace, without writing raw `ffmpeg` | [`video-editing-skill`](https://github.com/kajisho5/video-editing-skill) / [`audio-production-skill`](https://github.com/kajisho5/audio-production-skill) |
  59. | Put Remotion motion graphics (animated captions, silence jump cuts, audio-peak motion) on a real recording | [`remotion-ffmpeg-skill`](https://github.com/kajisho5/remotion-ffmpeg-skill): its compositions take their length, fps, frame, silences, scene cuts, audio peaks and beats from this repo's `probe.py`, `silence.py` and `scenes.py`. It verifies its render with `check.py` and `look.py`, composites a graphics layer with `overlay.py` (`--place` for its `--sparse` mode), and delivers through `render.py --template` |
  60. Other repos in the ecosystem — [`media-analysis-skill`](https://github.com/kajisho5/media-analysis-skill), [`transcription-skill`](https://github.com/kajisho5/transcription-skill), [`subtitle-skill`](https://github.com/kajisho5/subtitle-skill), [`thumbnail-skill`](https://github.com/kajisho5/thumbnail-skill), [`color-grading-skill`](https://github.com/kajisho5/color-grading-skill), [`motion-graphics-skill`](https://github.com/kajisho5/motion-graphics-skill), [`qc-skill`](https://github.com/kajisho5/qc-skill) — read this repo's `contract --json`, its tools' `--json` output and `doctor`, the same way any agent framework would; this repo does not call into any of them. The dependency runs one way.
  61. ---
  62. **Contents**
  63. [Standalone, and in an ecosystem](#standalone-and-in-an-ecosystem) · [Why](#why) · [Quick start](#quick-start) · [How it works](#how-it-works) · [Design principles](#design-principles) · [Tools](#tools) · [Audio](#audio-is-a-first-class-input) · [Built for agents](#built-for-agents) · [FAQ](#faq) · [FFmpeg compatibility](#ffmpeg-compatibility) · [Tested on real footage](#tested-on-real-footage) · [Install](#install) · [Requirements](#requirements) · [Development](#development) · [Docs](#docs)
  64. ---
  65. ## Why
  66. An agent that "knows FFmpeg" still guesses: it assumes a frame rate, picks a codec the container cannot hold, re-encodes a file that only needed a stream copy, and reports "done" without opening the result. This skill takes the guessing out:
  67. - **Real files first.** Every job starts with `probe.py`; the agent decides from the measured duration, fps, resolution, colour and audio layout, not from the file name.
  68. - **Structured tools, not shell strings.** Each operation is a script with typed arguments. Nothing runs through a shell; no filter graph is accepted from the caller.
  69. - **A contract the agent can read.** `contract --json` states, for every tool, what it takes, what it writes, which FFmpeg components it needs and how the result is verified. The MCP surface is derived from it.
  70. - **Verification after execution.** The result is probed, checked against the destination's spec and, when the picture changed, looked at as a contact sheet.
  71. - **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a speech engine (Parakeet or a whisper) is installed, never required — by `caption.py --transcribe` and, since 1.17, by `silence.py --filler --transcribe`; both take a transcript you already have instead, and both refuse with the install lines rather than guessing. Both take `--engine` and `--model`; `FFMPEG_SKILL_ASR_ENGINE` and `FFMPEG_SKILL_ASR_MODEL` set this machine's defaults (e.g. `large-v3-turbo` where that whisper.cpp model is installed; without either, auto and `base`). The engine runs on your machine and the audio is never uploaded; parakeet-mlx, faster-whisper and openai-whisper download their model weights the first time they run (parakeet-mlx's default from Hugging Face).
  72. ## Paper
  73. The design discipline behind this skill — per-operation verification for AI agents — is written up as a position paper. The paper scores ffmpeg-skill itself as only partially conformant with the discipline it defines.
  74. Kajiyama, S. (2026). *What Is Verification-Driven Coding? — A Discipline for Not Trusting Generation.* Zenodo. https://doi.org/10.5281/zenodo.23720216
  75. ## Quick start
  76. ```bash
  77. # 0. FFmpeg itself, once (Ubuntu/Debian shown; macOS and Windows are in the Install table below)
  78. sudo apt install ffmpeg
  79. # 1. install the skill for Claude Code (Cursor: --cursor, Codex: --codex, all three: --all)
  80. npx ffmpeg-skill
  81. # 2. check the machine: ffmpeg, ffprobe and every FFmpeg component the tools need
  82. npx ffmpeg-skill doctor
  83. # 3. (for agent frameworks) read the machine-readable contract
  84. npx ffmpeg-skill contract --json | head -40
  85. ```
  86. Captions from speech (`caption.py --transcribe`) need one local speech engine, installed separately. `pip install faster-whisper` is the simplest; the first run downloads its model, and the audio never leaves the machine. Without an engine, everything else works, `doctor` lists the engine as missing optional, and `--transcribe` refuses with the install line for each engine it supports.
  87. Already installed? re-run `npx ffmpeg-skill` to refresh `~/.claude/skills/ffmpeg-skill`. Copies are not updated automatically.
  88. `doctor`'s overall `ok` and a single tool's `usable` are different signals: `ok` is false when anything *some* tool requires is missing, so one missing filter (a plain Homebrew `ffmpeg` without `subtitles`, or a build without libvidstab) turns it false even though most tools still run. Check `doctor --json`'s `tools` field for the per-tool answer, not just `ok`.
  89. Then talk to your agent:
  90. > "Take `interview.mp4`, keep 0:45–3:10 and 5:00–6:30, and make it exactly 60 seconds for Reels."
  91. The agent runs `probe.py`, `cut.py --segments 0:45-3:10,5:00-6:30`, `fit.py --duration 60 --aspect 9:16 --fit crop`, `export.py --preset reels`, `check.py --platform reels` and `look.py`, then reports "final.mp4: 59.98 s, 1080×1920, 30 fps, AAC stereo" with the contact sheet it inspected.
  92. ### Deliver to a platform
  93. One command per destination, with the app's own UI taken into account:
  94. ```bash
  95. python3 $S/render.py talk.mp4 --template tiktok --cues cues.txt
  96. ```
  97. That fills the shipped `templates/tiktok.json`: 9:16 crop, captions popped word by word *above*
  98. TikTok's description bar and clear of its like column, −14 LUFS, the `tiktok` export preset, and
  99. a `check.py --platform tiktok` on the file it wrote. The caption size comes from the delivery
  100. table, so since 1.17.1 the filled project also states `"fit_size": "on"`: a size nobody asked for
  101. shrinks to fit the cue instead of splitting the sentence across two cues. A template that states
  102. its own `fit_size`, and a `--brand` that states a caption size, still win, and a project with
  103. `"fit_size": "off"` renders the captions 1.17.0 rendered. Templates ship for `tiktok`, `reels`,
  104. `shorts`, `youtube-shorts`, `youtube`, `x`, `linkedin`, `facebook`, `podcast` and `audiogram`;
  105. `--template all` (or a comma-separated list) renders the seven video destinations (`tiktok`, `reels`, `shorts`, `youtube`, `x`, `linkedin`, `facebook`) from the same edit and
  106. writes a `<name>_pack.md` table of what each one produced. Files land next to the input unless
  107. `-o` says otherwise, and `--dry-run` shows every planned command rather than a result.
  108. `render.py --list-templates` prints them with their frames, limits and safe zones. Alias
  109. spellings work everywhere a platform is named (`youtube-shorts` = `shorts`, `ig` = `reels`,
  110. `twitter` = `x`, `fb` = `facebook`).
  111. The tools also work on their own, from any shell:
  112. ```bash
  113. S=~/.claude/skills/ffmpeg-skill/scripts
  114. python3 $S/probe.py input.mp4 --compact
  115. python3 $S/fit.py input.mp4 --duration 60 --aspect 9:16 --dry-run # print the plan, run nothing
  116. python3 $S/export.py input.mp4 --preset reels --json # structured result with a probe of the output
  117. ```
  118. On Windows in Git Bash, `python3` is only on PATH if Python was installed from the Microsoft Store; a python.org install exposes `python` (or the `py` launcher) instead — replace `python3` with `python` above if you see a "command not found". `bin/install.js` and `doctor`/`contract` already handle this for you; only the raw script examples above need it spelled out manually.
  119. More requests and the commands behind them: [examples/README.md](examples/README.md). To see everything run end-to-end on generated footage: `npm run demo` (the gallery in [docs/demos.md](docs/demos.md)).
  120. ## How it works
  121. ```mermaid
  122. flowchart TD
  123. U[User request] --> A[AI agent<br/>Claude Code · Cursor · Codex]
  124. A -->|reads| S[SKILL.md<br/>workflow, request → tool map, report format]
  125. A -->|runs| T[Structured tool<br/>scripts/&lt;name&gt;.py, typed argparse flags]
  126. T --> C[Contract<br/>input schema · role · capabilities · verification policy]
  127. C --> D[Capability detection<br/>doctor: available / missing / unknown]
  128. D --> F[FFmpeg execution<br/>no shell, stream copy when possible]
  129. F --> V[Verification<br/>probe · check · look.py contact sheet]
  130. V --> R[Structured result<br/>--json: status, output, commands, probe]
  131. R --> A
  132. ```
  133. Over MCP the same tools are reached through a transport that holds no tool table of its own:
  134. ```mermaid
  135. flowchart LR
  136. M[MCP client<br/>Claude Desktop · Cursor · any client] --> P[mcp/server.py<br/>stdio JSON-RPC]
  137. P -->|tools/list| C[Contract-derived ToolSpecs<br/>names · order · inputSchema]
  138. P -->|tools/call| T[scripts/&lt;name&gt;.py]
  139. C -.derived from.-> K[scripts/_contract.py]
  140. T -.described by.-> K
  141. ```
  142. Names, order and `inputSchema` in `tools/list` are translated from each tool's argparse parser at start-up, so a new flag or a new script appears in MCP with no edit to `mcp/`. A test copies the skill, adds, removes and edits a script, and reads `tools/list` again to prove it.
  143. ## Design principles
  144. These are the rules the skill file gives the agent and the code enforces.
  145. 1. **Probe first.** No tool decides from the file name. `probe.py` measures duration, fps (with variable-frame-rate detection), resolution, rotation, bit depth, HDR format including Dolby Vision, colour tags and every audio stream before anything is cut.
  146. 2. **Lossless when possible.** `cut.py` and `loudness.py` stream-copy what they do not need to touch (`join.py` re-encodes so it can crossfade and conform mismatched clips; `--copy --transition none` joins clips that already match by stream copy). Re-encoding happens only when it must: frame-accurate cuts, filters, format changes, or a keyframe farther than the tolerance.
  147. 3. **Plan before render.** Every tool takes `--dry-run` (print the ffmpeg command lines, write nothing), `--plan FILE` (the dry run saved as a plan with fingerprinted inputs that `render.py FILE` executes later, refusing if an input changed), `--json` (structured result with a probe of the output), `--fast` (preview quality), `--progress` (percent and ETA), `--timeout` (a hung ffmpeg is killed and reported, never waited on forever; Ctrl-C or SIGTERM likewise stops the running ffmpeg, removes its partial output and reports `kind: interrupted`) and `--overwrite` (explicit consent before an existing output is replaced). A test runs every tool under `--dry-run` behind a fake ffmpeg and asserts that no ffmpeg call happened and no file appeared.
  148. 4. **Machine-readable contract.** `contract --json` describes all 42 tools: input schema generated from the parser, output schema, role, required and conditional FFmpeg capabilities, dry-run support, the verification tools to run afterwards, whether a visual check is required, `mutates_input: false`. `provides` lists all 42 by a cross-repository Capability id (`ffmpeg-skill.cut`, `ffmpeg-skill.loudness`, ...) for [`kajisho5/AI-video-production-OS`](https://github.com/kajisho5/AI-video-production-OS)'s `CapabilityContract.provides` — see `docs/contract.md`.
  149. 5. **Contract-derived MCP.** `mcp/server.py` builds its `tools/list` from the contract. Tool names, order and `inputSchema` cannot drift from the scripts; a test keeps the two byte-identical.
  150. 6. **Capability detection.** `doctor` reads `ffmpeg -encoders / -decoders / -filters / -bsfs` and reports which of the components the tools need are present on this build (libx264, libass, zscale, loudnorm, xfade, …), before a job fails inside ffmpeg.
  151. 7. **Unknown is not missing.** When a listing cannot be read (a layout the parser does not know, ffmpeg exiting non-zero) the affected capabilities are `unknown`: never `missing`, never silently `available`. An installed filter is not reported absent; a failed detection is not a pass.
  152. 8. **Verify the result.** The output is probed, and when the picture changed (captions, overlays, crops, colour, transitions) the agent runs `look.py` and inspects the PNG. The report is not finished until its `Look:` line names that image; audio-only jobs say `Look: not needed`. **"Inspects" means the calling agent's own vision, not a feature of this skill:** `look.py` only renders a PNG; nothing in this repository detects faces, products, subjects, or "the interesting part" of a frame or a scene. When a crop or reframe needs to keep a specific part of the frame (`fit.py --fit crop --crop-x/-y`, see [Tools](#tools)), it is the multimodal agent looking at that PNG and choosing the anchor — a non-visual caller (a script, a CLI user without eyes on the sheet) has to supply that decision itself, and the default is a plain centre crop. Likewise `scenes.py --highlights` ranks candidate scenes by a measured proxy (`--rank-by audio` or `--rank-by duration`), never by content; it is the agent that turns a look at the sheet into a judgement.
  153. 9. **One label per report.** A finished job is `Done:`, a failure or a refusal is `Failed:`, and a partial result is `Done:` with the shortfall named in `Notes:` — never a third label such as `Done (partially):`.
  154. 10. **Keep originals.** No tool overwrites its input. Outputs are new files named `<input>_<operation>.<ext>` unless told otherwise, and a test hashes every input after the run.
  155. ## Tools
  156. 42 public tools, all Python 3.10+ standard library, all with `--help`, `--dry-run`, `--json`, `--plan FILE` (a dry run written as a plan `render.py` executes later), non-zero exit and a reason on stderr on failure. `--json-brief` (1.11.0) prints the same result document trimmed to what a caller acts on — status, output, `verified`, a compact `summary` of the output probe, the tool's own keys and the command count instead of the command lines — for roughly a third of the bytes; `--json` itself is unchanged. Every re-encoding tool except `export.py`, `render.py` and `batch.py` (a preset or the project decides there) takes `--codec h264|hevc|av1|prores` and `--quality N` (1.8) and, with `export.py`, `--hw`/`--no-hw` (Apple VideoToolbox on Apple Silicon, opt-in; see `doctor` below), and every time flag takes seconds, `mm:ss`, `hh:mm:ss.fff` or SMPTE `hh:mm:ss:ff` with an optional `@fps` suffix (1.9).
  157. **Analysis and inspection**
  158. | Tool | What it does |
  159. |---|---|
  160. | `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, bit depth, HDR format incl. Dolby Vision (`hdr` for a real PQ/HLG/DV transfer only, `bt2020_or_hdr` also for BT.2020 SDR), colour space, rotation, every audio stream; `--analyze` flags Log footage |
  161. | `scenes.py` | Scene changes, audio peaks, highlight proposals (`--rank-by audio` loudest, or `--rank-by duration` longest — both proxies, not "best") and a per-scene sheet; cut list for `cut.py --segments`; `--beats` measures the music's beat grid (tempo, beat times, confidence); `--shots` classifies each shot static/pan/motion by measured optical flow; `--audio-peaks` lists per-second dBFS; `--speech` reports a speech-vs-music ratio, not a classification |
  162. | `look.py` | Contact sheet, single frames, side-by-side comparison as PNG so the agent can see what it made; `--safe NAME` shades the zones a platform's own UI covers; `--guides title,action,4:3` outlines the broadcast safe areas (on the PNG only) |
  163. **Editing**
  164. | Tool | What it does |
  165. |---|---|
  166. | `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact video and sample-exact audio, `--smart` for a frame-exact H.264 cut that re-encodes only the partial GOPs at the cut points (matched to the source, verified against its packets and frames) and copies the rest, an SDR HEVC source re-encodes as HEVC (`--no-keep-hevc` for x264), a single `.mp4`/`.mov` copy keeps its edit list, so the picture starts at `--start` unless a keyframe lands later (`start_snapped` then says so; `--no-edit-list` for 2.x's shifted start), `--vfr-guard` to choose the VFR check (`sampled` by default); reports `precision`; `--snap beats` moves the in/out points onto a measured beat, or refuses when there is no measurable pulse ; `--remove A-B,...` drops ranges and keeps the rest |
  167. | `join.py` | Concatenate clips with xfade transitions, normalising size, fps, sample rate and channel layout (the widest clip's, or `--channels`); audio-only inputs are joined as audio; `--list parts.txt` reads generated segments (TTS lines) and checks every one first — missing, empty or unreadable parts are all named in one refusal, or skipped and reported with `--on-missing skip` ; `--copy --transition none` joins clips with matching streams by stream copy, measured afterwards, and refuses a mismatch by name |
  168. | `silence.py` | Detect and remove dead air (jump cuts) with a margin around speech (`--margin`, 0.2 s each side); list or export the cut list, or a source-to-output time map (`--timemap`, JSON or CSV); reports the level at every cut and warns where one may land in a breath or a word's tail; `--filler` also removes filler words, but only where a speech engine timed them; `--speech-aware` keeps a breath inside a sentence and cuts only at sentence-boundary pauses, composing with `--filler` through the same `keep_ranges()` |
  169. | `fit.py` | Fit to a duration (pitch-preserving speed change or trim, smooth slow-mo) and/or aspect ratio (pad, crop or `--fit blur`'s blurred, darkened fill, with `--crop-x`/`--crop-y` to keep an off-centre subject) and/or exact `--width`/`--height`; rotate 90/180/270, flip h/v; force constant fps |
  170. | `crop.py` | Crop to an exact pixel rectangle (`--x --y --width --height`) — distinct from `fit.py --fit crop`, which crops to an aspect ratio it computes itself |
  171. | `cropdetect.py` | Measure existing black letterbox/pillarbox bars and report the `crop.py`-ready rectangle that removes them — analysis only, writes no file; `--motion-centre` reports the per-second motion centroid instead, for a 9:16 reframe — report only, the calling agent picks the crop, no auto-reframing |
  172. | `deinterlace.py` | Deinterlace interlaced source footage (`yadif`), `--mode frame`/`field`, `--parity` |
  173. | `denoise.py` | Reduce video noise/grain (`hqdn3d`), `--strength low/medium/high` or individual spatial/temporal overrides |
  174. | `redact.py` | Blur, pixelate, or (`--mode delogo`) interpolate away an exact pixel rectangle for the whole clip. The box is given: nothing here finds a face, a plate or a logo, and `delogo` is a border blend, not inpainting |
  175. | `sphere.py` | Extract a flat rectilinear viewport from a 360/spherical video (`--yaw --pitch --roll --h-fov --v-fov`); no subject tracking, only the aim you give it |
  176. | `straighten.py` | Rotate by an arbitrary angle for horizon correction (`--degrees`, `--fit crop/pad`) — distinct from `fit.py --rotate`'s exact 90-degree turns |
  177. | `insert.py` | Turn a still image into a silent, fixed-duration video clip (title card, end slate) at an exact frame size / fps, with an optional Ken Burns zoom/pan |
  178. | `background.py` | Generate a solid-colour or two-colour gradient clip at an exact size/duration — no input file; `--transparent` writes a fully transparent canvas (ProRes 4444, QuickTime Animation, PNG or VP9 with alpha) to draw graphics on |
  179. | `reverse.py` | Reverse playback (video and, unless `--no-audio`, audio) |
  180. | `stabilize.py` | Two-pass motion stabilisation (`vidstabdetect`/`vidstabtransform`) |
  181. | `sequence.py` | Numbered (`frame_%04d.png`) or glob-matched still images into a video |
  182. | `waveform.py` | Render an audio track as a waveform or spectrum visualization video (`showwaves`/`showspectrum`) — for audio-only inputs with no picture worth showing; `--image` puts the visualisation over a still plate (an audiogram), with `--platform`, `--title` and burnt-in captions |
  183. | `freeze.py` | Hold a frame for N seconds (`--at`, `--hold`, `--mode insert/extend`) — an end-card hold or a comedic beat |
  184. | `pad.py` | Add black/silent padding at the start and/or end of the timeline (`--start`, `--end`) — distinct from `fit.py --fit pad`'s per-frame letterbox bars |
  185. | `speedramp.py` | Step through different constant speeds across a clip via `--segment START-END:FACTOR` (repeatable) — distinct from `fit.py`'s single whole-clip speed factor |
  186. | `loop.py` | Repeat a clip `--times` N or to a target `--duration` — for background loops and filling a fixed slot length; `--crossfade` smooths the seam, `--boomerang` plays it forward then backward |
  187. | `broll.py` | Cut away to a B-roll clip over the A-roll for a window (`--insert B --at T --duration D`, repeatable) and come back; A's length and audio untouched by default |
  188. | `metadata.py` | Write container chapter markers from a `TIME TITLE` text file and title/artist/comment tags, every stream copied bit for bit; `--auto-chapters` proposes the markers from measured pauses and scene cuts and names them `Chapter N` for you to rename |
  189. | `grid.py` | Composite `--cols`x`--rows` clips into one grid, each cell letterboxed and labelled with its filename by default (`--label none` to skip) |
  190. **Audio**
  191. | Tool | What it does |
  192. |---|---|
  193. | `audio.py` | Voice clean-up chain at three strengths (`--voice light\|medium\|strong`), FFT denoise, typed compressor / limiter / gate, music bed with sidechain ducking (`--duck-amount/-threshold/-attack/-release`), a never-ducked effects bed (`--effects`), `--stereo-widen`, fades, 5.1 → stereo, track replacement, extraction (`-o out.wav`), `--audio-stream N` |
  194. | `sync.py` | Offset between two recordings by audio cross-correlation (1 ms, pure Python), clock-drift correction; aligned video or audio out (audio-to-audio only — no lip-sync/face detection); a third or later camera is an additional `more_sources` positional (`sync.py REF A B ...`), the `second` positional itself keeps its name |
  195. | `loudness.py` | Two-pass EBU R128 `loudnorm` to −14 LUFS / −1 dBTP or any target (`--lra` for the range), video stream-copied; the written file is measured again and re-encoded until it meets `--tp` (lossy encoders overshoot); `--measure-only`; `--versions streaming,ebu-r128,broadcast-jp` (or `LUFS/TP` pairs) writes one file per target from one master, each measured after and listed in `versions` (`output` is null), `verified` only when every version met its own target |
  196. **Picture**
  197. | Tool | What it does |
  198. |---|---|
  199. | `caption.py` | Burn SRT/ASS with font, size, colour, outline, position; build SRT from timed plain text; wraps to the safe area — with `--platform`, the destination's own left/right safe zone, which since 1.17.2 is also what the burnt ASS states as its side margins — with a phrase-aware breaker (`--wrap phrase\|measured`) and `--max-lines`/`--min-duration`/`--offset`; `--mode mux` takes a repeated `--srt file:lang` for several language-tagged, toggleable tracks in one file; picks a font by script for non-Latin text (`--lang`); animated and word-by-word karaoke timed to the speech energy or real word timings; `--fit-size` shrinks the size until a cue fits `--max-lines` instead of splitting the sentence (on by default on the delivery-template path since 1.17.1, where the size comes from the platform table); says `caption text unchanged` when it burned the cues exactly as given; optional local transcription |
  200. | `overlay.py` | Logos, watermarks and titles with position, time range, opacity, fades; `--platform NAME` keeps them clear of that destination's UI; `--video` for picture-in-picture, `--chromakey` for green-screen compositing, `--place` to lay ranges of a layer at chosen times; `--keep-alpha` composites onto a transparent canvas or alpha clip and writes the result with its alpha; a VP9 layer with alpha keeps its transparency |
  201. | `graphics.py` | Lower-thirds, title cards, chapter chips, progress bars, countdowns, corner bugs, social stickers, opening hook cards and meme captions drawn by FFmpeg from a brand kit; `--platform NAME` keeps them inside that destination's safe zone; `--text-render` routes shaping scripts through libass and `--emoji-assets` composites colour emoji; `--keep-alpha` renders a template on a transparent canvas as a per-element alpha layer |
  202. | `color.py` | HDR10 / HLG / Dolby Vision → SDR BT.709 tone mapping, DV layer stripping, 3D LUT (.cube), colour-tag rewriting, typed primary correction (exposure/contrast/saturation/gamma/white balance/lift-gain/levels/curves); `--keep-alpha` grades an alpha clip and keeps its alpha, checked frame by frame; `--legalize bt709\|ebu-r103`: a legal-range clamp after the 4:2:0 conversion, measured on the written file |
  203. **Delivery**
  204. | Tool | What it does |
  205. |---|---|
  206. | `export.py` | Presets `youtube`, `youtube4k`, `reels`, `tiktok`, `shorts`, `linkedin`, `facebook`, `x`, `youtube-hdr` (HEVC Main10, source HDR tags kept), `youtube-av1`, `prores`, `dnxhr-lb`/`sq`/`hq`/`hqx`/`444`, the alpha masters `prores4444`/`prores4444xq`/`qtrle`/`vp9-alpha`/`hap-alpha` (alpha read back), `key-fill` (a switcher's fill + key pair), `h265`, `gif`, `copy`, all tagged BT.709 unless they carry HDR or are an editing master (`prores`, `dnxhr-*`: the source's tags kept, codec, profile and pixel format read back) (a BT.601-matrix or RGB source is converted to BT.709, not relabelled); `--normalize` meets the platform's loudness in the same call (`render.py` turns it on by default for platform presets) ; `--max-mb MB` fits the file under a size limit by a two-pass encode and measures it ; `--preset copy --repair` remuxes a damaged file (timestamps regenerated, corrupt packets dropped) and counts the errors before and after |
  207. | `proxy.py` | Small, low-bitrate proxy for downstream AI analysis/preview/editing decisions — resize by `--width`/`--scale`, proxy-grade `--quality` (default 30), `--fps`, `--no-audio`; not a delivery preset |
  208. | `check.py` | PASS / WARN / FAIL against YouTube, Shorts, Reels, TikTok, X, LinkedIn, Facebook, broadcast and podcast specs, from the same delivery table the export presets and templates read (podcast also reports chapter markers and channel count), with the fix for each failure and a `format` / `judgement` kind per row; `--content` adds black-frame, frozen-frame, silence, short audio-dropout, single-frame-flash and flat-first-frame rows ; `--compare REF` measures SSIM/PSNR against a reference, and with `--expect-changed SPAN\|none` fails when the picture changed outside the spans the caller declared; `--levels bt709\|ebu-r103` counts the luma and chroma samples outside the broadcast legal range, per clip and per window, with min/max Y/U/V; `--expect-identity` checks that the file is the one a run reported writing (`output_identity`) |
  209. | `report.py` | Single-file HTML delivery report: before/after sheets, media facts, loudness, compliance, the commands run; `--pack` renders a social pack table |
  210. **Orchestration**
  211. | Tool | What it does |
  212. |---|---|
  213. | `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays including the social sticker/hook/meme graphics and a picture-in-picture or chroma-keyed `video` layer sized for the final frame, music and stem levels, loudness, export, chapter markers, check); `--init`, `--dry-run`, `--stop-after`, `--cache DIR`/`--from STAGE` (reuse the stages that did not change); the captions block takes the fit-size policy (`fit_size`, `min_size`, `fit_size_scope`) and the run reports the caption stage's counts as `caption`; `--template NAME INPUT` renders a shipped delivery template (`--template all` writes the whole social pack plus its table); `--export-timeline edit.fcpxml\|.edl\|.otio` (2.1) writes the cut as an editor timeline for Final Cut, DaVinci Resolve or Premiere instead of rendering it — clips, speed, dissolves, the music bed and chapter markers carry over, and `not_exported` names what stays behind |
  214. | `batch.py` | Apply a step recipe or a project to a folder with a content-hash cache; `--watch`; `--jobs N` processes several files at once under one shared `--timeout` |
  215. | `multicam.py` | Align any number of cameras and recorders by audio (with drift correction) and cut between them from a switch list; `--switch energy` auto-cuts to whichever camera is loudest at that moment instead of a hand-built list (audio level as a stand-in for who is talking: there is no speaker identification, so it needs each speaker on their own camera's mic), with `--min-shot` (minimum shot length) and `--edl` (write a plain START-END cut list, not a CMX EDL; for Premiere/Resolve use `--write-project` then `render.py --export-timeline`) |
  216. | `verify.py` | Run the toolchain on real device files and report PASS / FAIL per step |
  217. Not tools, but part of the surface: `mcp/server.py` (the MCP transport) and `scripts/_contract.py` (`contract --json`, `doctor`). Per-flag reference for every tool: [references/scripts.md](references/scripts.md).
  218. ## Audio is a first-class input
  219. WAV, FLAC, MP3, M4A/AAC, OGG and Opus go through `probe`, `cut`, `join`, `silence`, `loudness`, `audio`, `sync` and `check --platform podcast` with the same commands as video. The output extension picks the codec: `-o out.wav` writes PCM, `-o out.flac` FLAC, `-o out.mp3` MP3, `-o out.m4a` AAC.
  220. - **Extraction.** An audio extension on a video input drops the picture: `audio.py talk.mp4 -o talk.wav`, or `--voice -o talk.m4a` to clean it on the way. `--audio-stream N` picks a track; `probe` lists them under `audio_streams`.
  221. - **Join.** `join.py intro.wav episode.m4a outro.wav -o full.flac` resamples every clip to one rate and channel layout and crossfades them (`--transition none` for a butt join). Audio and video inputs cannot be mixed in one join.
  222. - **Sample-accurate trims.** `cut.py talk.wav --start 1.2345 --end 2.3456 --accurate` trims at the sample; the JSON reports `precision` (`packet` for a stream copy, `sample` for PCM / FLAC, `codec_frame` when a lossy encoder frames the audio again, `frame` for video) and the measured `duration_error_ms`. A `.wav` never receives compressed packets.
  223. - **Typed dynamics.** `audio.py --compress --comp-threshold -20 --comp-ratio 4`, `--limit --limit-ceiling -1`, `--gate --gate-threshold -45`. Each flag is one documented option of FFmpeg's `acompressor`, `alimiter` or `agate`, range-checked before ffmpeg runs; no filter string is accepted from the caller.
  224. - **Loudness.** `loudness.py talk.wav -I -16 --tp -1.5 -o talk.m4a` for podcast levels; `check.py talk.m4a --platform podcast` measures LUFS and true peak and reports the chapter markers and channel count.
  225. Picture tools (`fit`, `caption`, `overlay`, `graphics`, `color`, `export`, `look`) refuse an audio file with "input has no video stream" instead of inventing a picture. `scenes.py` measures an audio file (`audio_only: true`, one scene): `--beats`, `--audio-peaks`, `--speech` and `--highlights` run on the sound, and `--sheet` / `--shots` are refused.
  226. ## Built for agents
  227. ### What is SPEC?
  228. **SPEC** (Self-Producing Execution Contract) is the name this project's author,
  229. [kajisho5](https://github.com/kajisho5), gave the pattern the tool layer is built on: each tool's
  230. `input_schema` — the part of its contract that has to track the CLI exactly, flag for flag — is
  231. never hand-authored side by side with the code. It is derived, at run time, from the one thing
  232. that has to be correct for the CLI to work at all: the script's own `argparse` parser.
  233. Concretely, `scripts/_contract.py`'s `_capture_parser()` imports every tool script and
  234. intercepts its `parse_args()` call to get the live, fully-built parser object — flags, types,
  235. choices, defaults, required/positional, mutually exclusive groups, all of it. `input_schema` is
  236. built straight from that object. (The rest of a `ToolSpec` — `role`, `capabilities`, `inputs`,
  237. `outputs`, `output_schema` — comes from a hand-authored table, `TOOL_META`, since those facts
  238. aren't things a parser can express; only `input_schema` is parser-derived.)
  239. - **The contract**'s `input_schema` for every tool is generated from the live parser directly.
  240. - **SKILL.md is two-tier** (1.11.0): the file the agent loads every session keeps the workflow, the
  241. request→script table and one line per gotcha; the long-form detail lives in `references/gotchas.md`
  242. and the other `references/` files, read only when a job needs it.
  243. - **The MCP server** (`mcp/server.py`) carries no schema of its own; `tools/list` is translated
  244. straight from the contract, `input_schema` included.
  245. - **The docs** (`docs/contract.md`'s field reference, this README's tool table) describe the same
  246. shape. `tests/test_contract.py` runs on every CI run and fails the build if any of them drift
  247. out of sync with what the code actually does — it catches drift, it doesn't fix it for you.
  248. So adding a flag to a script's `argparse` block updates `input_schema` and the MCP tool
  249. definition with no second edit, and a docs page or `TOOL_META` entry that falls behind fails CI
  250. rather than drifting silently. There is no separate `input_schema` file to forget to update.
  251. ### Machine-readable contract
  252. ```bash
  253. npx ffmpeg-skill contract --json # or: python3 scripts/_contract.py --json
  254. npx ffmpeg-skill contract --json --static # without environment detection
  255. ```
  256. The contract is generated from the code that runs, not maintained beside it. For each of the 42 tools (`ffmpeg-skill/<name>`) it states:
  257. | Field | Meaning |
  258. |---|---|
  259. | `input_schema` | generated from the tool's argparse parser: properties, types, enums, defaults, required, positional order, mutually exclusive groups |
  260. | `output_schema` | what `--json` prints: `status`, `output`, `commands`, `probe`, `output_identity` (`{size, mtime_ns}` of the file written, on every tool but `probe`), plus tool-specific fields (`precision`, `checks`, `offset_seconds`, …) |
  261. | `role` | `analysis`, `analysis_and_execution`, `execution` or `verification` |
  262. | `capabilities` | the FFmpeg encoders, filters and bitstream filters the tool always needs, and the ones needed only for a flag or input |
  263. | `supports_dry_run`, `supports_json`, `supports_json_brief` | measured by the tests, not declared |
  264. | `verification` | which tools to run on the output afterwards (`probe`, `check`, `look`) |
  265. | `requires_visual_verification` | the picture changed; inspect the contact sheet |
  266. | `audio_only`, `video_required` | whether an audio-only input is accepted or refused |
  267. | `mutates_input` | always `false` |
  268. | `idempotency_hint` | `bit_exact`, `content_equivalent`, `cached` or `environment_dependent` |
  269. Next to the tool list the document carries a top-level `deprecated` list (since 1.10): what the next major removes, since when, the replacement and the surface it lives on. It is empty in 3.0.0: Python 3.9 and FFmpeg 5.x, the two 2.x entries, moved to `removed` with what 2.0 removed, and `docs/contract.md` "What 2.0 changed" and "What 3.0 changed" are written from that.
  270. `contract_version` (1.0) is separate from the skill version, so a consumer can pin the shape and read the version for provenance. The document also states the invocation mapping (structured arguments → argv), the JSON shapes for success and failure (`{"status": "failed", "error": {"kind": "input | ffmpeg | output | missing_tool | timeout | verification | interrupted", "message": …}}`; a refusal adds `error.reason`, a stable code such as `output_exists` or `plan_inputs_changed`, and `error.details`, the facts behind it; warnings, fallbacks and skipped steps a run decided on its own are in `diagnostics`), and that no tool runs a shell or executes anything other than the named script, `ffmpeg` and `ffprobe`. Field-by-field reference: [docs/contract.md](docs/contract.md).
  271. ### MCP
  272. ```json
  273. {"mcpServers": {"ffmpeg-skill": {"command": "python3", "args": ["/Users/you/.claude/skills/ffmpeg-skill/mcp/server.py"]}}}
  274. ```
  275. On Windows, `python3` is only on PATH if Python was installed from the Microsoft Store; a python.org install exposes `python` (or the `py` launcher) instead — if your MCP client reports the server failed to start, change `"command"` above to `"python"` (or the full path from `where python`).
  276. `mcp/server.py` is a stdio JSON-RPC transport with no tool table of its own. `tools/list` is derived from the contract at start-up, in contract order, with `inputSchema` translated from each tool's `input_schema`. By default it lists only the core 12 (`render`, `look`, `caption`, `export`, `check`, `fit`, `cut`, `audio`, `loudness`, `graphics`, `silence`, `probe`) so a client doesn't pay context for 30 schemas it rarely calls directly; set `FFMPEG_SKILL_MCP_FULL=1` to list all 42. Every tool, listed or not, is callable through `tools/call`, which maps structured arguments to argv and runs the named script; a raw `argv` form is accepted for compatibility and marked non-canonical. `python3 mcp/server.py --list` prints the tools; `--call probe '{"inputs": ["a.mp4"]}'` runs one from the shell.
  277. No `inputSchema` lists `json` or `progress`: they are transport flags the server sets itself, not tool arguments (2.0; 1.x listed them unless `FFMPEG_SKILL_MCP_LEAN=1`). A client that still sends them is not refused.
  278. The server also advertises a `prompts` capability: five canned workflow recipes an agent can ask for by name instead of composing the individual tool calls itself — `reel`, `podcast`, `multicam`, `delivery_check`, `hdr`. `prompts/list` returns each one's name, description and arguments; `prompts/get {name, arguments}` fills the matching template and returns one text message naming the actual command lines to run and what to check afterward. These are recipes, not new tool calls — every command line a prompt names is one `tools/call` (or the CLI) can already run.
  279. ### Capability detection
  280. ```bash
  281. npx ffmpeg-skill doctor # human-readable
  282. npx ffmpeg-skill doctor --json # available / missing / missing_optional / unknown / detection / errors / tools / gpu_encoders / ffmpeg_security
  283. ```
  284. `doctor` reads `ffmpeg -encoders`, `-decoders`, `-filters` and `-bsfs` and resolves every capability the contract declares against this machine's build. Three states per capability: `available`, `missing`, `unknown`. Exit 0 when everything required is available, 1 when something required is missing, 2 when nothing is proven missing but a required capability is unknown. With detection on (the default), `contract --json` carries the same lists under `capabilities`. `doctor` also reports fonts: the default drawtext family, and `fonts.scripts` — one `available`/`missing`/`unknown` per writing system (ja, zh, ko, ar, he, hi, bn, ta, th, lo, ru, el, latin) with the file it would use — so "can this machine render Korean captions" is answered before the job, not after. `doctor --json`'s `tools` field folds that down to one answer per tool — `{"caption": {"usable": "no", "missing": ["filter:subtitles"], "fix": "..."}, ...}` — so "is `doctor` overall `ok`" and "can I run `caption.py` on this machine" are answered separately: a plain Homebrew `ffmpeg` is `ok` for tools that don't need `subtitles`/`drawtext`/`zscale`, while `caption`'s own `usable` is `"no"`.
  285. `doctor --json`'s `gpu_encoders` reports which GPU-backed encoders (`nvenc`, `videotoolbox`, `qsv`, `vaapi`, `amf`) this ffmpeg *build* was compiled with — read from `-encoders` alone, so it proves the capability shipped, not that the GPU/driver on this machine will actually accept a job (that needs a real encode, which `doctor`'s introspection never runs). It never affects `ok` or any tool's `usable`. The one GPU path a tool uses is Apple VideoToolbox, opt-in: `--hw` on every re-encoding tool (h264/hevc/prores on the GPU, AV1 stays on SVT-AV1), `FFMPEG_SKILL_HW=1` to make it this machine's default for everything except `export.py`'s delivery presets (those take `--hw` explicitly, or `render.py --hw` for a whole project). It runs only on Apple Silicon and never by default, except on a build with no `libx264`/`libx265` (an LGPL ffmpeg that has only `h264_videotoolbox`/`hevc_videotoolbox`): there it is the only encoder, so every re-encoding tool and `export.py`'s presets use it without `--hw`, the result says `hw.source: "auto"`, and `doctor` lists the missing x264/x265 under `covered_by` instead of calling the tools unusable (`--no-hw` still asks for the CPU). It falls back to the CPU encoder — reported in the result's `hw.fallback`/`hw.notes` — only when VideoToolbox itself refuses a job (any other ffmpeg failure is reported as it happened, not retried); every result names the `encoder` that actually ran (`batch.py`, which encodes nothing itself: each `results` row), and a GPU encode adds a `notes` line that its quality is not CRF-equivalent. Measured on an M4 Max, 2–7× faster than x264/x265 `medium`; at matched SSIM (`tests/bench_vt.py`) 1.1–2.9× the bytes on SDR and 1.9–3.5× on HDR, which is why it is a flag and not the default (`docs/design-decisions.md`). `doctor --json`'s `hw` says whether this machine can run it (`platform_ok`) and whether it is the default here (`default_on`).
  286. `doctor` also compares the FFmpeg version with a small, dated table of high-severity FFmpeg CVEs (`doctor --json`'s `ffmpeg_security`; source [ffmpeg.org/security.html](https://ffmpeg.org/security.html), checked 2026-10-22). ffmpeg-skill does not ship FFmpeg, so a decoder bug in the build you have stays yours until you upgrade it. When the version is older than the first upstream release on its branch that carries a fix (today: CVE-2026-8461, the MagicYUV decoder heap write, fixed in 8.0.3, 8.1.2 and 9.0), `doctor` prints one `warning:` line on stderr with the update command and lists it under `ffmpeg_security.warnings`. It is advice, never a failure: `ok`, each tool's `usable` and the exit code do not change. A distribution package (Ubuntu `8.0.1-3ubuntu2`, Debian `+deb13u1`) may carry the fix as a backport under the old version number, so the line says the build is older than the fix, not that it is vulnerable. A version it cannot read (a git snapshot such as `N-12345-g…`) gets no warning, and nothing is fetched: the table ships with the skill.
  287. ## Gotchas and best practices
  288. The short list for humans. The agent-facing version, with the reasoning, is the "Things that look right but are wrong" and "Gotchas" sections of [SKILL.md](SKILL.md).
  289. - **Variable frame rate (phone and screen recordings).** `probe.py` flags it; every re-encoding tool conforms to a constant rate automatically. `cut.py` samples the frame timing before a copy and switches to frame-accurate mode when it is variable or too short to judge, because copy-cuts on VFR land on the wrong frame (`--vfr-guard average` restores 2.x's rule, re-encoding whenever the nominal and average rates differ; `--vfr-guard off` keeps the copy). Choose the rate yourself with `fit.py input.mp4 --fps 30` when the measured average is odd.
  290. - **Lossless cuts snap to keyframes.** A stream-copy cut can start up to one GOP earlier than asked. `cut.py` re-encodes when the snap exceeds 0.5 s (`--tolerance` changes the limit). For a strictly lossless file pass `--tolerance -1` (plus `--vfr-guard off` on a variable-frame-rate source), and expect the cut to land on the nearest earlier keyframe; the JSON result lists them under `nearest_keyframes`. A `--segments` join can still be re-cut: open-GOP video, a later segment of B-frame `.mp4`/`.mov` video that starts between keyframes (start it on a keyframe, or write `.mkv`), or a join that fails its check. A single `.mp4`/`.mov` copy keeps its edit list by default (since 3.0.0), so the picture starts where asked: the keyframe's pre-roll is stored but hidden by an MP4 edit list, which a player that ignores edit lists shows (`--no-edit-list` starts the file at the keyframe instead).
  291. - **Frame-exact without re-encoding the whole cut (H.264).** `cut.py --smart` re-encodes only from each cut point to the next keyframe and copies the GOPs between, matched to the source's profile, level, pixel format, B-frame depth and colour tags. It checks the result against the source (frame grid, copied packets, decoded frames). A source it cannot match (HEVC, 10-bit, interlaced, VFR, open GOPs) is re-encoded in full, and `smart.reason` says why. `probe.py --keyframes-near T` shows where the keyframes are.
  292. - **HDR stays HDR.** When the probe reports HDR (HDR10, HLG, Dolby Vision, BT.2020), the tools keep it rather than flatten it. Convert deliberately with `color.py --to-sdr` before H.264 deliverables or LUT work. `export.py` platform presets are SDR and warn on HDR input.
  293. - **Loudness targets.** −14 LUFS / −1 dBTP for YouTube and social platforms (the `loudness.py` default), `-I -16 --tp -1.5` for podcasts, `-I -23` for broadcast. A clip measured at −40 LUFS or below is room tone, not content; raising it raises the noise. Check true peak as well as LUFS: `check.py file --platform podcast` measures both.
  294. - **Alpha is kept only when asked, and a lost alpha is not verified.** The re-encoding tools write opaque H.264 by default. On a source whose alpha is used (ProRes 4444, QuickTime Animation, PNG, VP9 with alpha), `color.py`, `graphics.py` and `overlay.py` take `--keep-alpha` (ProRes 4444 by default, `--alpha-codec qtrle|png|vp9`, `--alpha-mode straight|premultiplied`) and read the alpha back; without it the result says `verified: false` with a failing `alpha` step, and `--drop-alpha` records that an opaque output is wanted. `background.py --transparent` makes the canvas to draw graphics on.
  295. - **Broadcast legal levels are measured, not assumed.** `check.py final.mxf --levels ebu-r103` (or `bt709`) counts every luma and chroma sample outside the legal range and FAILs when one frame has more than 1 % out (`--levels-tolerance`); the JSON `levels` adds min/max Y/U/V per `--levels-window`. `color.py --correct --legalize ebu-r103` clamps after the 4:2:0 conversion and measures what the encode left: a lossy encoder can push a few samples around a clamped edge back out, so the result says how many.
  296. - **Frame changes first, text second.** Captions and overlays burned before a crop or resize end up off-frame. Reframe, then caption.
  297. - **Cropping 16:9 to 9:16 discards 70 % of the width.** `fit.py --fit crop` centres by default; pass `--crop-x`/`--crop-y` toward the subject, or pad with `--fit pad --pad-fill blur`. Look at the contact sheet before deciding.
  298. - **Phrase-aware caption breaking (1.16).** `caption.py`/`graphics.py --wrap phrase` (the default) never breaks inside a word or on the wrong side of a hyphen, never leaves a lone digit, kana or punctuation pair on a line, prefers Japanese sentence ends and particles over a mid-word break, and never ends a line on an article or preposition. All four are penalties over break positions that already fit, so no line is widened and the line count never changes; `--wrap measured` restores 1.15's width-only wrap. The text itself is never rewritten or shortened. Since 1.16.1 a Thai run and a katakana word are never broken inside (Thai writes no space inside a phrase and there is no dictionary: the break goes where you put a space or `|`), and a line wider than the safe width is reported as `overlong` with the fix named.
  299. - **Caption size fitted to the cue (1.17, reachable from the templates since 1.17.1).** At a platform caption size a line holds about six em, so an ordinary sentence needs four lines and `--max-lines 2` used to cut it into consecutive cues — half the sentence arriving late. `caption.py --fit-size` (default `auto`) walks the size down until every cue fits, *before* laying the cues out, with a legibility floor of 4.5 % of the frame height (`--min-size`, default 13 ASS units). An explicit `--size` or a `brand.json` size is a statement about the look and is never overridden — which in 1.17.0 also silenced the fitter on every `render.py --template` run, since a template fills the size from the platform table; 1.17.1 marks that size as the default it is (`"fit_size": "on"` in the filled project), so the type shrinks and no cue is split. `--fit-size off` restores that earlier behaviour byte for byte, and the caption text is still never rewritten to make it fit.
  300. - **Beat-synced cuts (1.17).** `scenes.py --beats` reports the measured grid — tempo, beat times, and a confidence built from how far the winning autocorrelation lag stands above the others and how many onsets land on it. `cut.py --snap beats` (and a `"snap"` block in a `render.py` project) moves in/out points to the nearest beat within `--snap-tolerance` — and only onto the grid points a measured onset actually marks, never onto the regular grid's continuation through a passage with no music in it. Below `--min-confidence` it **refuses**: a cut point may move to a measured beat and may not appear from one, so speech and ambience get an honest "no steady pulse here" instead of an invented grid. A music bed is measured as it is delivered, WAV/FLAC/MP3 with no picture; on a synthetic click track at 90 and 120 BPM every beat is within 1 ms (WAV/FLAC) or 9 ms (MP3) of its click, `tests/test_analysis.py` holds it to 20 ms.
  301. - **Filler words (1.17).** `silence.py --filler --words transcript.json` removes "um" and "uh" through the same cut graph the silences use. Never without measured word timings — there is no heuristic that finds an "um" without them that would not also cut real speech — and `like`, `tipo` and `cioè` are deliberately not in the built-in lists, because a discourse marker is a content word. Built-in lists since 2.19: English, Japanese, Korean, Chinese, Spanish, German, French, Portuguese, Italian, Dutch, Polish, Russian and Turkish; any other language takes `--filler-words FILE`. Whisper stays optional: `--transcribe` with no engine installed refuses and names the five installs.
  302. - **Throughput (1.17).** `batch.py --jobs N` runs several files at once, capped at `min(N, cpu_count, 8)` and sharing one `--timeout` budget rather than one per item; the per-item table keeps its order. `render.py --cache DIR` reuses stages whose inputs and arguments did not change, so swapping an export preset re-runs export only. The cache is opt-in with no default directory, and the ffmpeg, skill and contract versions are part of every key, so a cache is never reused across them.
  303. - **Audiogram (1.16).** `waveform.py --image cover.png` (or `render.py --template audiogram`) puts the waveform over a still plate for an episode that has no picture, with `--platform` for the frame, `--title` and burnt-in captions. The image is a local file you give: nothing is fetched and no cover art is ever invented.
  304. - **Emoji in captions and titles (1.15).** `caption.py`/`graphics.py --emoji-assets DIR` composites a PNG per emoji (Twemoji/Noto naming, `1f389.png`) on top of the text, because drawtext cannot load a colour emoji font at all and an installed one does not prove libass will draw it in colour — `doctor --json`'s `fonts.emoji` answers that from a render probe. Without assets the run still succeeds and reports `mode: mono`. Nothing is ever downloaded.
  305. - **Indic and Thai text shaped correctly in titles and lower-thirds (1.15).** `graphics.py` renders Devanagari, Bengali, Tamil, Thai and Lao through libass automatically (`text_renderer: "ass"`), because drawtext never reorders matras or re-clusters marks; Arabic and Hebrew were already correct on a fribidi build. `--text-render drawtext` with such a script is refused, never rendered wrongly.
  306. - **Non-Latin text picks a font by script (1.12).** Japanese, Chinese, Korean, Arabic, Hebrew, Devanagari, Thai, Cyrillic and Greek cues, titles and overlays resolve a font file that covers them automatically, and a machine with no such font fails the job (`kind: input`) instead of rendering boxes. `doctor --json`'s `fonts.scripts` says which languages this machine can render; `--lang ja|ko` disambiguates Han-only text, and `--lang zh-TW` (or `zh-HK`, `zh-Hant`) picks a Traditional Chinese face (2.19); an explicit `--font`/`--font-file` is always kept.
  307. - **Silence detection finds nothing?** The default threshold is −35 dBFS. The tool prints a hint with the track's measured level; raise the threshold (`silence.py --threshold -25`) or shorten `--min-silence`.
  308. - **Sync results carry a confidence.** Below 0.3, or an offset near the edge of the analysis window, is probably wrong: enlarge `--analyze-seconds` or find a clap. Recordings over ten minutes from separate devices need `sync.py --fix-drift`.
  309. - **Outputs are never overwritten silently.** An existing output path is refused (`kind: input`, `error.reason: output_exists`, before anything runs); pass `--overwrite` where a replacement is intended.
  310. - **Nothing happens silently.** A refusal names its `error.reason` and `error.details`; a stale output lock taken over, a run without a lock, a plan input nobody could fingerprint, a fallback (an odd-sized frame scaled to even, a substituted font, libaom for SVT-AV1), a skipped step and every warning are recorded in `diagnostics` and on stderr (codes: `json_output.diagnostic_codes`); tools that draw text name the font they drew with (`font_used`). A refused plan and a lock takeover also append a line to an audit log (`<plan>.audit.jsonl`, or `render.py --audit-log FILE`). A degraded or partial result says so: `render.py`'s `verified` is every stage's, an unmeasured loudness or a report section that could not be built is a failing verification step, `export.py` reports `trimmed`, `multicam.py`'s `cuts` name the camera rendered, and `batch.py` rows carry the failing step's `error`, `verified`, and a row for every input after Ctrl-C.
  311. - **Long masters need a longer `--timeout`.** The default is 1800 s per ffmpeg call, which is there to catch a hang. A full re-encode of an hour of 1080p at the default preset ran past it on a 4-core machine (see [A one-hour recording](#a-one-hour-recording)). Pass `--timeout 0` or a larger value, or `--fast`. Analysis and stream-copy cuts are far below it.
  312. - **Long chains belong in a plan.** Three hand-chained re-encodes lose quality and are hard to change; `render.py` runs the whole edit from one JSON file, and `--dry-run` shows every ffmpeg command before anything is written.
  313. ## FAQ
  314. ### Can I pass raw ffmpeg arguments, or a filter_complex?
  315. Each operation is a script whose flags are its argparse parser, and that parser is what `contract --json` and the MCP `inputSchema` are built from. CI fails the build when those drift from the scripts. A caller-supplied filter graph, `-vf` string, or other ffmpeg option is not an argument any script accepts. `doctor` reads `ffmpeg -filters` (and `-encoders`, `-bsfs`) and reports a filter a tool needs before a job runs into it. MCP does accept an `argv` array, and that array is the named script's own CLI flags, still bound to that script: the script's flags, never an ffmpeg command line and never a shell.
  316. ### Why 42 tools? Isn't that overkill?
  317. One tool per job, because each job promises something different, and the tool measures that promise after the run. `loudness.py` re-measures integrated loudness and true peak. `cut.py` reports how far a cut snapped to a keyframe. `silence.py` returns the ranges it kept. `export.py` checks the preset's maximum length. A value a tool cannot read fails that step as `not_measured`; it does not pass. One tool with a mode flag would need as large a schema, and its checks would be vaguer.
  318. An agent does not read 42 schemas. Over MCP, `tools/list` advertises the core 12 (bold below). The other 30 are called by name through `tools/call` when a job needs one, and `FFMPEG_SKILL_MCP_FULL=1` lists all 42. The groups are the ones in [Tools](#tools).
  319. <!-- why42:start -->
  320. | Group | MCP default / all | Tools |
  321. |---|---|---|
  322. | Analysis and inspection | 2 / 3 | **probe**, **look**, scenes |
  323. | Editing | 3 / 24 | **cut**, **fit**, **silence**, join, crop, cropdetect, deinterlace, denoise, redact, sphere, straighten, insert, background, reverse, stabilize, sequence, waveform, freeze, pad, speedramp, loop, broll, metadata, grid |
  324. | Audio | 2 / 3 | **audio**, **loudness**, sync |
  325. | Picture | 2 / 4 | **caption**, **graphics**, overlay, color |
  326. | Delivery | 2 / 4 | **export**, **check**, proxy, report |
  327. | Orchestration | 1 / 4 | **render**, batch, multicam, verify |
  328. | **Total** | **12 / 42** | |
  329. <!-- why42:end -->
  330. Without the other 30, an agent writes raw ffmpeg for those jobs, and a mistake there often still exits 0:
  331. - `stabilize.py` runs vidstab in two passes, and the second pass needs the first pass's transforms file.
  332. - `color.py` builds the HDR-to-SDR chain. With a wrong colour tag the run still finishes, and the picture comes out washed out.
  333. - `join.py` puts transitions between clips that differ in size, frame rate or audio layout. Done by hand, those clips have to be normalised first or xfade fails.
  334. - `sync.py` and `multicam.py` find the offset between two recordings by audio cross-correlation. No ffmpeg filter measures it.
  335. Every command a tool ran is in its `--json` `commands`, shell-quoted so it can be pasted and run again, and `--dry-run` prints the commands without running anything. In the 100-prompt agent eval (1.17.0, [Tested on real footage](#tested-on-real-footage)) the agent made no raw ffmpeg calls and reported no false successes.
  336. ### What does `verified` check, and what can it not tell me?
  337. `verified` covers what can be measured from the file. Each tool measures its output again: the probe (the file opens, its duration, codecs, size and frame rate) and the tool's own promise. For example, loudness and true peak after `loudness.py` or `export.py`, the length against the asked one, and for `cut.py`, `join.py` and `export.py` the stream count, where picture and sound end, and whether both ends of every stream decode. `check.py` grades a file against a platform's limits. A value that cannot be read fails its step as `not_measured`; it does not pass.
  338. It cannot tell you whether the result looks or sounds right, and the result says so: every writing tool lists the properties it did not measure in `unverified`. Examples are `picture_content` (what is in the frame), `subject_framing` after a reframe, `rendered_placement` and `legibility` of captions and titles, `grade_look` after a colour conversion, `removed_span_content` after a silence cut (whether a removed span held speech), `mix_balance` and `lip_sync`. For those, look at frames with `look.py` (a contact sheet, or `--safe` for a platform's UI zones) and listen where the result points you. `silence.py` lists the level at every cut, and `cut_warnings` names the cuts that may have landed in a breath or a word's tail.
  339. ### Are silence removal and captions stable?
  340. They are the most used tools here, and they are tested on every pull request and every push to `main` on Linux, macOS and Windows, with FFmpeg 6.1 to 9. On 20 test files with known gaps, `silence.py` missed none and left at most 1 ms of silence (see [Tested on real footage](#tested-on-real-footage)). A cut keeps `--margin` seconds of air on each side, 0.2 s by default; raise it to 0.3 s for a speaker who trails off. `--speech-aware` keeps the short breaths inside a sentence. Filler words are removed only from measured word timings (`--words` or a local `--transcribe`), never guessed. `--timemap` writes where each kept and removed range sits in the source and the output, for re-timing subtitles. If `--threshold` is set high enough to cut into a breath or a weak word ending, the result warns at that cut.
  341. `caption.py` burns SRT/ASS or muxes them as soft tracks, fits the size to the platform's safe area, and picks a font that covers the script (Japanese, Chinese, Korean, Arabic, Thai, Hindi and others). It needs FFmpeg's `subtitles` filter (libass); `doctor` reports whether this build has it. Placement and legibility are listed as `unverified`, so look at a contact sheet before delivering. Speech-to-text runs on a local engine you install; its first run may download a model.
  342. ### How well does HDR to SDR work, and what should I watch for?
  343. `color.py --to-sdr` converts PQ (HDR10), HLG and iPhone Dolby Vision 8.4 to SDR BT.709 with real tone mapping (`zscale` and `tonemap`, `hable` by default; `mobius`, `reinhard`, `clip`, `linear` and `gamma` with `--tonemap`). BT.2020 footage on an SDR transfer gets a gamut conversion only. Watch for four things:
  344. - It needs `zscale` (libzimg). The plain Homebrew `ffmpeg` lacks it, so use `ffmpeg-full` (keg-only: see [Install](#install) for putting it on `PATH`); `doctor` reports whether it is there.
  345. - `--peak` (default 1000 nits) should match how bright the source was mastered. A wrong value makes highlights dull or clipped.
  346. - Tone mapping is a judgement, and the result says so (`grade_look` in `unverified`). Compare frames with `look.py`, and try another `--tonemap` if skin or skies look off.
  347. - An HDR source that only needs to play everywhere may not need converting at all. `--strip-dovi` removes the Dolby Vision layer from an iPhone clip, keeping its HLG picture, with a stream copy.
  348. ### How does ffmpeg-skill compare with Remotion?
  349. ffmpeg-skill edits footage you already have, on this machine, with FFmpeg tools whose flags are a machine-readable contract (`contract --json`). Remotion renders a video from React components. They complement each other: this repository does not call Remotion, and a file Remotion writes is an ordinary input (`probe.py`, `cut.py`, `fit.py`, `export.py`, `check.py`). [`remotion-ffmpeg-skill`](https://github.com/kajisho5/remotion-ffmpeg-skill) joins the two. It measures the footage with this repo, draws the graphics in Remotion from those measurements, and checks and delivers the result with this repo again. Install both (`npx ffmpeg-skill`, then `npx remotion-ffmpeg-skill`). As with the other skills, the dependency runs one way: it calls these tools, and nothing here calls it. HyperFrames renders a video from HTML and is the same kind of neighbour: a file it writes is an ordinary input too.
  350. ### Does it validate the filter graph before running?
  351. No. `--dry-run` prints the ffmpeg command a writing tool would run and does not start ffmpeg, so ffmpeg never parses that filter graph. ffprobe still runs when the tool probes an input that exists. Unless `--overwrite` is passed, an existing output is refused (`kind: input`) before ffmpeg. An output that is the same file as an input is refused either way. No media file is written, and `verified` is false; `--plan FILE` does write the plan JSON. A path that is not on disk yet is not measured: the probe is a stub (0×0, duration 0), so a number taken from it is not a measurement. The measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) still run their measurements; only the artifact is skipped, and `silence` with `--filler --transcribe` also runs the speech engine, which may download a model. `overlay` measures the lines of a multi-line `--text` with one drawtext on a generated frame on FFmpeg 8.0, 8.0.1 and 8.1.0, where it draws them one by one. `verify` accepts `--dry-run` and ignores it: its steps run. `--plan FILE` is that dry run written down — the planned argv, the command lines, and a fingerprint of each existing input (size, plus a sha256 of the first and last 8 MiB). `render.py` and `verify.py` refuse `--plan`. `render.py FILE` runs a plan only when those fingerprints still match (a mismatch is refused with each input's expected and actual fingerprint in `error.details`, and logged to `<plan>.audit.jsonl`), by running the tool again with the planned argv, not by executing the recorded ffmpeg lines; `--dry-run` on that run only prints them. `doctor` reports whether the filters, encoders and bitstream filters the contract names are in this build (`ffmpeg -filters`, `-encoders`, `-bsfs`) and which fonts are installed. A name in that listing is not a check of the graph this command would build, and a listed GPU encoder is not a successful encode.
  352. ### Can it remove burned-in subtitles or logos?
  353. A rectangle you already know, sitting still for the whole clip, can be filled from the picture around it with `redact.py --mode delogo` (ffmpeg's `delogo` filter). That is a 1 px border blend. It is not inpainting: a logo that moves, or burned-in subtitles whose position changes, is out of scope. The same known box can be obscured instead, with `--mode blur` or `--mode pixelate`. None of the three finds the box. `delogo` also needs the box at least 1 px inside the frame, and it only interpolates 8-bit YUV, so an HDR or 10-bit source is refused until `color.py --to-sdr`.
  354. ### Does it auto-track faces or subjects for reframing?
  355. `fit.py --fit crop` centres the crop unless you pass `--crop-x` and `--crop-y`. `crop.py` cuts the rectangle you pass. `cropdetect.py` measures letterbox bars, and `--motion-centre` reports where motion sat each second; it writes no file and picks no crop. `sphere.py` points a 360 view where you aim it. `look.py` writes a contact sheet. Nothing in this repository detects a face, a product or a subject. Choosing what to keep in frame is the caller's.
  356. ### Does it encode on the GPU?
  357. The encode path a tool will actually take is Apple VideoToolbox, for h264, hevc and prores, and only on Apple Silicon. `--hw` asks for it on that call. `FFMPEG_SKILL_HW=1` makes it the default for every re-encoding tool except `export.py`'s delivery presets, which take `--hw` on their own (or `render.py --hw` for a whole project). AV1 stays on SVT-AV1. nvenc, QSV, VAAPI and AMF are reported by `doctor --json`'s `gpu_encoders` when the ffmpeg build was compiled with them, and no tool encodes with them. One build is the exception to opt-in: with no `libx264` and no `libx265`, and with `h264_videotoolbox` or `hevc_videotoolbox`, those encodes use VideoToolbox without `--hw` and the result says `hw.source` is `"auto"`. `--no-hw` still asks for the CPU encoder. A VideoToolbox job the hardware refuses is re-encoded on the CPU and the result says so (`hw.fallback`).
  358. ## FFmpeg compatibility
  359. The tools need FFmpeg 6.1 or later and Python 3.10 or later (standard library only). 3.0.0 removed FFmpeg 5.x and Python 3.9: every tool refuses an `ffmpeg`/`ffprobe` whose version reads as older than 6.1 (`kind: missing_tool`, `error.reason: ffmpeg_unsupported`; a git build whose version is not a release number is not refused), and `doctor` lists an unsupported FFmpeg or Python under `unsupported` with `ok: false`. What CI actually exercises on every pull request is FFmpeg 6.1 (Ubuntu 24.04 apt), 7.1 (Debian trixie apt), 8.x (macOS Homebrew), 9.x (Windows gyan.dev) and 8.0.1 (Ubuntu 26.04 apt), on Python 3.10 and 3.13 (the two ends of the supported range). The capability parser has been run against the listings of these builds:
  360. | FFmpeg | `-filters` row layout | Source |
  361. |---|---|---|
  362. | 6.1.1 | three flag characters: `..C acompressor A->A` | Ubuntu 24.04 apt, captured |
  363. | 7.1.x | same as 6.x | Debian trixie apt in a CI container (plus a constructed fixture in tests/) |
  364. | 8.1.2 | two flag characters: `TS aap AA->A`, three-character legend, `------` separator | Homebrew on the macOS CI runner, captured |
  365. | 9.0.1 | same as 8.x, CRLF | gyan.dev build on the Windows CI runner, captured |
  366. FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on the old width matches nothing on FFmpeg 8 and, if "nothing matched" is read as "nothing installed", reports every filter missing; that is what 0.9.0 did on macOS and Windows. Since 0.9.1 rows are recognised by their io-spec token (`A->A`, `AA->A`, `|->V`, `N->N`), so the flag width, the legend and the separator do not matter, and a listing that still cannot be read yields `unknown` rather than `missing`. The captured listings live in [tests/fixtures/](tests/fixtures/README.md) with their provenance; CI uploads each runner's listing and `doctor --json` as an artifact so a new layout is visible before it bites.
  367. ## Tested on real footage
  368. **What is tested where.** The contract and the test suite (`tests/test_contract.py`,
  369. `tests/test_all.py`, which aggregates one module per tool group — `test_analysis.py`,
  370. `test_editing.py`, `test_audio.py`, `test_picture.py`, `test_delivery.py`,
  371. `test_orchestration.py`, `test_accel.py`, `test_asr.py`, `test_cut_copy.py` — over the shared footage in `tests/_fixtures.py`) run on Linux, macOS
  372. and Windows on every pull request, minus the handful of POSIX-shim tests listed under
  373. [Development](#development). The real-device media corpus
  374. (`tests/corpus.py`) has been run on Linux and macOS; the full corpus has **not** been run on
  375. Windows yet, and neither has an install by someone other than the maintainer been reproduced
  376. there — [issue #143](https://github.com/kajisho5/ffmpeg-skill/issues/143) tracks both. Treat the
  377. numbers below as measured on Linux (and, where stated, macOS), not as a claim about every file
  378. type on every OS.
  379. | Result | Measurement |
  380. |---|---|
  381. | **92 / 92** | verification steps on a 10-file real-device corpus (GoPro, DJI, iPhone incl. Dolby Vision, Android screen recordings, HDR10, 24p, Tears of Steel), 0.8.0, local ffmpeg 6.1 |
  382. | **40 / 40 within 10 ms** | `sync.py` offset detection, ±30 s offsets with gain, noise and EQ changes on real dialogue and music, 120 s windows (max error 1.1 ms); 60 s stress windows 95 % within 10 ms, 4 of 5 misses flagged by confidence |
  383. | **0 missed gaps** | `silence.py`, 20 cases with known gaps, ≤ 1 ms leftover silence |
  384. | **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
  385. | **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
  386. | **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
  387. | **7 / 8 routed** | eval 22 at 1.18.3 (2026-09-17, eight new symptom-only prompts for the five 1.18.0 flags, naming no flag, plus three repeats each of `cs1`/`cs3`, Sonnet agent): after 1.18.1's SKILL.md routing rows and 1.18.2's matching README rows, `scenes.py --shots`, `cropdetect.py --motion-centre`, `silence.py --speech-aware`, `sync.py`'s N-source form and `multicam.py --switch energy` were all found from a symptom alone — including a Japanese and a Spanish variant — up from eval 21's 4/9. The eighth prompt (`--filler` composed with `--speech-aware`) is an honest partial: the fixture has no real speech to transcribe, confirmed by installing `faster-whisper` mid-eval and re-testing by hand. `cs3` no longer rewrites the user's captions in any of 3 runs — the 1.17.3 fix holds through 1.18.1-1.18.3. Written up in `evals/results/iteration-22.json` |
  388. | **8 / 12 routed** | eval 21 at 1.18.0 (2026-09-17, one prompt per new analysis/multicam flag plus a `cs1`/`cs3` recheck, Sonnet agent): every result was correct where the agent found the right script — both sync offsets, the shot label, the multicam switch point and its render-project mapping, both 1.17.2/1.17.3 rechecks — but 4 of 12 prompts hit a script the agent could not find, because SKILL.md named none of the five new flags (confirmed by grep). Three refused honestly rather than fabricate a number; one reached a correct answer without the intended flag, by luck of one fixture's silence durations. 1.18.1 (2026-09-17) is the SKILL.md fix, since evaluated by eval 22. Written up in `evals/results/iteration-21.json` |
  389. | **20 / 20** | 1.17.2 run (2026-09-14, the eight caption prompts of eval 19, six of them three times, Sonnet agent, regex grader + an Opus grader that opened every contact sheet and counted the lines per cue): routing 19/19 act runs, honest 18/20 with 0 false successes and 0 raw ffmpeg calls, report format 20/20, user's language 20/20, trigger set 49/50 (one judge flip on a file-less prompt), Opus quality mean 4.25 (3.65 at eval 19). The picture is fixed: 0/20 runs stack one word per line against 12/12 template runs at eval 19 on the same cues; the Style row at TikTok geometry is now `…,54,151,420,1`, the fitter's `size_used` (15 on TikTok, 16 on Shorts, the 13 floor for the Spanish cues) is what the frame shows, and report and sheet agree in 18/20 runs. What is left is not the typesetter: `cs3` rewrote the user's captions for the fourth iteration running, one `cs1` run raised `max_lines` to 4 to avoid a shrink and drew four-line stacks, and `cs2`'s 32-letter Spanish word still leaves the frame at the size floor (disclosed 3/3). Written up in `evals/results/iteration-20.json` |
  390. | **26 / 26** | 1.17.1 run (2026-09-14, targeted re-run of the 18 prompts eval 18's follow-up named, plus three repeats each of the four caption-size prompts, Sonnet agent, regex grader + a full Opus grader over all 26 runs, every PNG opened and every written output re-probed): routing 23/26, honest refusals and failures 24/26 with 0 false successes and 0 raw ffmpeg calls, report format 26/26 with the third label gone, user's language 26/26, trigger set 50/50, Opus quality mean 3.65. 1.17.1's fix holds — `--fit-size` now fires on the `render.py --template` path in 12/12 caption runs (24 → 16, `dl4` to the 13-unit floor, `split` 0, `text_unchanged` true, identical across repeats), the beat, filler and `--jobs` prompts route on the first try, and `bt2` quotes its measured 0.184 confidence instead of denying the capability exists. The honest part: the picture is unchanged. `caption.py`'s `write_ass` writes the platform's *vertical* safe margin into `MarginL`, `MarginR` and `MarginV` alike (tiktok 63 ASS units → 420 px), so at `PlayResX` 1080 the text column is 240 px and libass wraps every word — the fitter budgets `play_w × 0.9`, which is why `split: 0` is true of the ASS text and false of the frame. It is the `--animate`/`--karaoke` path only, present since 1.14, and it explains eval 17's and eval 18's "one word per line" too; 1.17.2 is the patch and the finding is written up in `evals/results/iteration-19.json` |
  391. | **100 / 100** | 1.17.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, every written output re-probed, `check.py` re-run on every delivery output) on the set grown to 100 prompts (caption size fitting, beat-synced cuts, filler removal, batch `--jobs`, render `--cache`): routing 95% over the 64 act prompts, honest refusals and failures 22/25 with 0 false successes and 0 raw ffmpeg calls, report format 98/100 (two runs label an honest partial result with a third label), user's language 100/100 across seventeen languages, visual check 24/24, real execution 6/6 with honest failure 5/5, trigger set 50/50 including all five new 1.17 prompts, Opus quality mean 3.71. The honest part: `--fit-size` is unreachable on the template path (`render.py` forwards the platform table's caption size as an explicit `--size`, so the fitter declines to shrink a size it thinks the user chose, and the project schema rejects `fit_size` outright — only the one run that called `caption.py` by hand got 24 → 16, `split` 0), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most — 1.17.1 is the patch and the finding is written up in `evals/results/iteration-18.json` |
  392. | **90 / 90** | 1.16.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 30 runs, chapters and subtitle streams re-probed, check.py re-run on every delivery output) on the set grown to 90 prompts (audiogram, auto chapters, multi-language tracks, caption breaking): routing 90/90, honest refusals and failures 90/90 with 0 false successes and 0 raw ffmpeg calls, report format 89/90 (one `Done (partially):`), user's language 90/90 by regex (89/90 by Opus), audiogram 2/2 with the cover behind the waveform and nothing fetched, auto chapters 2/2 with `Chapter N` titles only, delivery 16/16 platform pass, trigger set 45/45, Opus quality mean 4.17. The honest part: the phrase breaker never gets to act at the platform caption sizes (a five-word cue does not fit two lines at TikTok size, so the split is byte-identical to 1.15.1), Thai still breaks inside words, and a katakana word was split — 1.16.1 is the patch and the finding is written up in `evals/results/iteration-17.json` |
  393. | **82 / 82** | 1.15.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, stills extracted inside the emoji window, check.py re-run on every delivery output) on the set grown to 82 prompts (emoji captions and title cards, a Hindi and a Thai lower-third): routing 82/82, honest refusals and failures 82/82 with 0 false successes and 0 raw ffmpeg calls, report format 82/82 (both iteration-15 label defects closed: `dl8` and `he2` now carry one `Failed:`), user's language 82/82 by regex (81/82 by Opus: one Spanish report with three English labels), non-Latin glyphs 11/11 (Devanagari through `graphics.py` is fixed; Thai lower-third and captions correct), emoji visible in colour in 3/3 runs given PNG assets and reported monochrome in the one that was not, visual check 23/24, delivery 12/13 one encode and 13/13 platform pass, trigger set 40/40, Opus quality mean 4.68. Still open: the caption breaker splits phrases (`dl1`, `dl4` unchanged) — queued for 1.16.0. Tokens per run flat at 73.3k on the same 76. Details in `evals/results/iteration-16.json` |
  394. | **76 / 76** | 1.14.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 26 runs, check.py re-run on every delivery output): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 by regex (75/76 by Opus: one Spanish report with three English labels), visual check 18/18, trigger set 38/38, Opus quality mean 4.58. The delivery templates did their job: 12 of 13 delivery requests went through `render.py --template`, finished in one encode (was 3 of 7) and all 13 pass their platform check (was 7 of 8). Tokens per run flat at 73.4k. Details in `evals/results/iteration-15.json` |
  395. | **76 / 76** | 1.13.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 76 prompts: 18 in Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish and Italian, and 8 delivery requests (TikTok, Reels, Shorts, LinkedIn, Douyin, podcast): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 across seventeen languages, visual check 18/18, trigger set 38/38, Opus quality mean 4.65 over the 26 new runs. One real defect found: Hindi through `graphics.py` (drawtext) comes out wrong-shaped even though the font covers Devanagari; captions through libass are fine (queued for 1.15.0). Four delivery runs spent a second encode for loudness, which 1.14.0's templates address. Tokens per run flat at 72.3k. Details in `evals/results/iteration-14.json` |
  396. | **50 / 50** | 1.12.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 50 prompts with two each in Chinese, Korean, Spanish, Portuguese, French, German and Arabic: routing 50/50, honest refusals and failures 50/50 with 0 false successes and 0 raw ffmpeg calls, report format 50/50, user's language 50/50 across nine languages, visual check 13/13, trigger set 29/29, Opus quality mean 4.83. Every non-Latin caption and lower-third picked a covering font by itself and rendered real glyphs (Arabic shaped and right-to-left); tokens per run unchanged at 72.3k. Details in `evals/results/iteration-13.json` |
  397. | **36 / 36** | 1.11.1 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22, Opus quality mean 4.75. The 1.11.1 wording did what it said (`doctor` before a job 23 of 36 runs → 0, `--json-brief` 4 → 23) and tokens per run stayed flat at 71.8k, because about 64k of every run is the host's own context; the token-diet theme closes here. Details in `evals/results/iteration-12.json` |
  398. | **36 / 36** | 1.11.0 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22. First iteration to measure tokens per run: mean 72.2k against 68.7k at 1.10.0, mostly a fixed per-run floor the skill does not control (a refusal run that only reads SKILL.md costs about 64k), plus `doctor` on 23 of 36 runs; 1.11.1 rewords step 0 and iteration 12 re-measures. Details in `evals/results/iteration-11.json` |
  399. | **108 / 108** | 1.10.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + focused Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, report format 108/108 by both graders (the harness now names the five labels), user's language 105/108 (every Japanese request in Japanese; 3 English requests drifted to Spanish or Portuguese), visual check 21/24, trigger set 22/22; real-device corpus 101/101 steps PASS. Details in `evals/results/iteration-10.json` |
  400. | **108 / 108** | 1.9.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + independent Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 23/26, user's language 105/108 (every Japanese request answered in Japanese; 3 English requests drifted to Spanish), report format 108/108 by regex (65/108 by the stricter grader, which now counts any missing label), trigger set 22/22; 12 of 15 platform jobs were one encode and `render.py` rendered once in 3/3 (was 1/3). The 1.9.0 time grammar was not used by any agent. Details in `evals/results/iteration-9.json` |
  401. | **108 / 108** | 1.8.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 22 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 22/24, report format 108/108 by regex (91/108 by the stricter grader: 'What/how:' in place of Steps:), user's language 98/108 by the stricter grader (Japanese labels-only reports counted), trigger set 22/22; 13 of 14 platform exports used `--normalize` and platform jobs went from three encodes to one; r04/f01 now answered in the request's language 5/6 (was 0/6). Details in `evals/results/iteration-8.json` |
  402. | **108 / 108** | 1.7.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 24 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 25/25, report format 108/108 by regex (104/108 by the stricter grader), user's language 101/108 (six English refusals answered in Spanish or Portuguese, one Japanese request in English); trigger set 22/22. Iteration-6 fixes held (fade-in only, Japanese audio trims, music no longer shortens the video). Details in `evals/results/iteration-7.json` |
  403. | **36 / 36** | 1.4.15 re-run (2026-09-12, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, report format 25/25, user's language 9/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3, one Japanese report with English labels; trigger set 22/22. Both iteration-5 defects gone (no raw ffmpeg fallback, music no longer shortens the video). Details in `evals/results/iteration-6.json` |
  404. | **36 / 36** | 1.4.0 re-run (2026-09-11, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, user's language 8/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3; trigger set 22/22. Details and the six findings in `evals/results/iteration-5.json` |
  405. | **6 / 6** | 0.9.1 audio evals (audio join, extraction, track selection, sample-accurate trim, typed dynamics; 2 in Japanese): routing, report format and audio-as-audio handling 6/6 |
  406. ```bash
  407. python3 tests/corpus.py --fetch --verify # ~1.4 GB download, then verify (slow on 4K)
  408. python3 tests/bench_sync.py --cases 100
  409. python3 tests/bench_silence.py
  410. python3 tests/bench_scenes.py
  411. ```
  412. Benchmarks live in `tests/bench_*.py`, agent evals in [evals/](evals/), results by iteration in `evals/results/`.
  413. ### A one-hour recording
  414. Measured at 2.17.0 (2026-10-25) on one 60-minute 1080p30 H.264 file (4.3 GB, AAC mono 48 kHz,
  415. synthetic `testsrc2` picture and a tone). The machine was a 4-vCPU Xeon at 2.1 GHz with FFmpeg
  416. 6.1.1 on Linux, and each tool ran alone. The analysis rows read the whole hour
  417. (`silence.py`, `scenes.py` and `check.py` report 3600 s). The edit and export rows used 10-minute segments cut from it with
  418. stream copy, and the last row is the whole hour.
  419. | Step | Wall time | Note |
  420. |---|---|---|
  421. | `probe.py` | 0.3 s | |
  422. | `silence.py --list` | 14 s | one audio decode |
  423. | `loudness.py --measure-only` | 66 s | |
  424. | `look.py` (contact sheet) | 97 s | |
  425. | `scenes.py` | 241 s | |
  426. | `check.py --platform youtube --content` | 308 s | the rows, `dropouts` included, cover the full hour |
  427. | `cut.py`, a 10-minute stream-copy segment | 2.4–4.4 s | no re-encode |
  428. | `join.py`, two 10-minute segments | 624 s | `join.py` re-encodes by default, so it can crossfade and conform (`--copy` does not, for clips that match) |
  429. | `export.py --preset youtube`, a 10-minute segment | 514 s | about 1.2× real time at the default `medium` preset |
  430. | `export.py --preset youtube --fast`, the same segment | 162 s | about 3.7× real time at `veryfast` |
  431. | `export.py --preset youtube`, the whole hour | killed at 1802 s | `kind: timeout` at the default `--timeout` of 1800 s (below) |
  432. **A full-length re-encode of an hour does not fit in the default `--timeout`.** The limit is
  433. 1800 s per ffmpeg call. On this machine `export.py --preset youtube` on the whole hour was
  434. killed at 1802 s with `kind: timeout` (exit 124), and nothing was written. At the 10-minute
  435. segment's rate it would have needed about 50 minutes. A whole-hour `join.py` would land at the
  436. limit. The error message names the fix. For a long master, pass `--timeout 0` (no limit) or a
  437. larger value, or `--fast`, which at the measured rate takes about 16 minutes for the hour.
  438. Analysis tools and stream-copy cuts stay far below the limit.
  439. ## Install
  440. Where it can be installed from today. Each route is spelled out below the table.
  441. | Route | For | Command or link |
  442. |---|---|---|
  443. | npm | Claude Code, Cursor, Codex skills folders | `npx ffmpeg-skill` ([npmjs.com/package/ffmpeg-skill](https://www.npmjs.com/package/ffmpeg-skill)) |
  444. | Claude Code plugin | Claude Code 2.1.275 or later (2.1.69 to 2.1.274: the two commands below) | `/plugin install ffmpeg-skill --marketplace kajisho5/ffmpeg-skill` |
  445. | Codex plugin | Codex CLI | `codex plugin marketplace add kajisho5/ffmpeg-skill --sparse .claude-plugin`, then `codex plugin add ffmpeg-skill@ffmpeg-skill` |
  446. | Gemini CLI extension | Gemini CLI | `gemini extensions install https://github.com/kajisho5/ffmpeg-skill` |
  447. | MCP server | clients without a shell | `npx -y ffmpeg-skill mcp` (see [MCP](#mcp)) |
  448. | MCP Registry | clients that install from the registry | `io.github.kajisho5/ffmpeg-skill` ([entry](https://registry.modelcontextprotocol.io/v0/servers/io.github.kajisho5%2Fffmpeg-skill/versions/latest)); the release workflow publishes a new version there after npm, and a failed registry step leaves the entry on the previous version |
  449. | GitHub release | a pinned copy | [releases](https://github.com/kajisho5/ffmpeg-skill/releases) |
  450. Not listed in a directory yet: the Claude plugin directory, the Cursor Marketplace, the Gemini CLI extension gallery and Glama (each is submitted and waiting on its review or crawl). This table gains a row when a listing goes live; until then the routes above are the whole list.
  451. ```bash
  452. npx ffmpeg-skill # Claude Code → ~/.claude/skills/ffmpeg-skill
  453. npx ffmpeg-skill --cursor # Cursor → ~/.cursor/skills/ffmpeg-skill
  454. npx ffmpeg-skill --codex # Codex → ~/.agents/skills/ffmpeg-skill (Cursor reads this location too)
  455. npx ffmpeg-skill --all # all three
  456. npx ffmpeg-skill --project # this project → ./.claude/skills/ffmpeg-skill
  457. npx ffmpeg-skill --dir ./my-skills
  458. npx ffmpeg-skill --uninstall # remove from the selected targets (--codex also clears the older ~/.codex/skills location)
  459. ```
  460. As a Claude Code plugin (no Node needed, updates with `claude plugin update`). This repository is its own plugin marketplace:
  461. ```bash
  462. /plugin install ffmpeg-skill --marketplace kajisho5/ffmpeg-skill # in Claude Code 2.1.275 or later
  463. claude plugin marketplace add kajisho5/ffmpeg-skill --sparse .claude-plugin # 2.1.69 to 2.1.274: add the marketplace,
  464. claude plugin install ffmpeg-skill@ffmpeg-skill # then install from it
  465. ```
  466. The same repository installs as a plugin in two more hosts (checked against Codex CLI 0.167 and Gemini CLI 0.68):
  467. ```bash
  468. codex plugin marketplace add kajisho5/ffmpeg-skill --sparse .claude-plugin # Codex: add the marketplace,
  469. codex plugin add ffmpeg-skill@ffmpeg-skill # then the plugin
  470. gemini extensions install https://github.com/kajisho5/ffmpeg-skill # Gemini CLI
  471. ```
  472. Those hosts find a plugin's skill only at `skills/ffmpeg-skill/SKILL.md`, a generated copy of the root `SKILL.md` (`.github/scripts/build_plugin_skill.py`; a test fails when it is stale). A Cursor plugin manifest (`.cursor-plugin/plugin.json`) is in the repository for the Cursor Marketplace; it follows Cursor's reference and has not been through their review yet.
  473. The plugin namespaces the skill as `ffmpeg-skill:ffmpeg-skill`. The manifests are [.claude-plugin/plugin.json](.claude-plugin/plugin.json), whose version follows every release automatically, and [.claude-plugin/marketplace.json](.claude-plugin/marketplace.json), which names the `ffmpeg-skill/` directory of the `plugin` branch: the 2.7 MB a running skill reads, rebuilt by every release, and not the 28 MB repository. `--sparse .claude-plugin` keeps the marketplace checkout small as well (measured at 3.4.0: 1.1 MB instead of 50 MB in Claude Code 2.1.322, 2.2 MB instead of 66 MB in Codex 0.167); without it the install is the same and only that checkout is larger. A `git-subdir` source needs Claude Code 2.1.69 or later (2.1.288 or later where git is older than 2.39, such as Ubuntu 22.04, by Claude Code's changelog; not run here); on anything older, update Claude Code or use `npx ffmpeg-skill`. Before 2.17.1 the README's one-line `claude plugin install kajisho5/ffmpeg-skill` could not work, because there was no marketplace file.
  474. Without Node: clone this repository and copy `SKILL.md`, `package.json`, `scripts/`, `templates/`, `references/`, `docs/` and `mcp/` into your agent's skills directory (the same set `npx ffmpeg-skill` installs; without `templates/` no `--template` renders, and without `package.json` the contract reports its version as unknown).
  475. After installing:
  476. ```bash
  477. npx ffmpeg-skill doctor # every required FFmpeg component present?
  478. npx ffmpeg-skill contract --json # what the agent framework will see
  479. ```
  480. FFmpeg itself:
  481. | OS | Command |
  482. |----|---------|
  483. | macOS | `brew install ffmpeg-full`, then put it on `PATH` (below). The plain `ffmpeg` formula lacks the subtitles, drawtext and zscale filters |
  484. | Ubuntu / Debian | `sudo apt install ffmpeg` |
  485. | Windows | `winget install Gyan.FFmpeg` |
  486. **On macOS, `ffmpeg-full` is keg-only:** Homebrew installs it without linking it, so after `brew install ffmpeg-full` the `ffmpeg` on `PATH` is still the lean `ffmpeg` formula (often present as a dependency of another formula), or no `ffmpeg` at all. **If the `ffmpeg` on `PATH` is Homebrew's lean formula, the skill uses the `ffmpeg-full` of the same Homebrew prefix by itself** (`ffmpeg` and `ffprobe` both, from `opt/ffmpeg-full/bin`; `doctor` says so, and `doctor --json` gives `ffmpeg_path` / `ffmpeg_source`); any other `ffmpeg` on `PATH` (a tap build, MacPorts, a static download) is used as it is. With no `ffmpeg` on `PATH` at all, or to make the whole shell use it, do one of two things:
  487. - **Put it first on `PATH`.** Add `export PATH="$(brew --prefix ffmpeg-full)/bin:$PATH"` to your shell profile. `brew --prefix ffmpeg-full` is the stable `opt/` path, so this survives upgrades, but it only reaches processes started from a shell that read the profile.
  488. - **Link it over the lean formula:** `brew unlink ffmpeg 2>/dev/null; brew link --overwrite --force ffmpeg-full`. Every process finds this one, but `brew doctor` warns about a linked keg-only formula, and an upgrade of the lean `ffmpeg` has been seen to link the lean build back on top without a message. After upgrading, check that `command -v ffmpeg` still resolves into `ffmpeg-full`, and re-run the link if it doesn't.
  489. ## Requirements
  490. - FFmpeg 6.1+ (3.0.0 removed 5.x; an older build is refused). Required by at least one tool (so `doctor`'s `ok` needs them all): `libx264`, `aac`, and the `drawtext`, `subtitles` (libass), `loudnorm`, `xfade`, `acrossfade`, `amix`, `overlay`, `scdet`, `silencedetect`, `cropdetect`, `tile`, `tpad`, `boxblur`, `hqdn3d`, `yadif`, `reverse`, `rotate`, `showwaves`, `v360`, `xstack` and `vidstabdetect`/`vidstabtransform` (libvidstab, `stabilize.py` only) filters; `doctor --json` lists them as `capabilities.required`. Needed only by the flags that use them: `libx265`, `prores_ks`, `libzimg` / `zscale`, `libmp3lame`, `libopus`, `libvorbis`, the `ass` filter. `doctor` tells you which are present. The apt and gyan.dev builds carry all of them; some Homebrew bottles lack `libass` / `libfreetype` / `libzimg`, which `doctor` reports as missing.
  491. - Python 3.10+, standard library only (3.0.0 removed 3.9)
  492. - Node 16+ only for the `npx` installer
  493. `doctor`'s own introspection calls (`ffmpeg -filters`/`-encoders`/`-bsfs`/`-version`) time out after 10s and report `failed` rather than hanging forever — those are meant to be fast. Every tool's actual media-processing `ffmpeg` invocation (cut, fit, caption, ...) runs under `--timeout` (default 1800 s, `FFMPEG_SKILL_TIMEOUT`, `0` = none): past the limit the process is killed, its partial output removed, and the failure reported as `kind: timeout` (exit 124) — a legitimately long `--accurate` re-encode should raise the limit rather than run unbounded. `-nostdin` is always passed, so a hung ffmpeg process waiting on stdin cannot happen.
  494. ## Stability
  495. 3.x keeps every tool name, CLI argument, JSON output key and exit code working: nothing is removed or renamed, and nothing optional becomes required, until 4.0. The full list of what is promised and what is not, and the three-step deprecation policy, is in [docs/contract.md](docs/contract.md#stability-guarantee-3x). It is enforced by a test that pins every tool's argument names against a snapshot, so a breaking change fails CI instead of slipping into a patch. Releases are batched: related fixes and features ship together as one version rather than one version per merged pull request, so a dependent project sees fewer, larger steps. Pin a tagged version, not `main`. 2.0.0 made the five changes announced since 1.10.0 -- `--crf` removed in favour of `--quality` (except on `export.py`, whose preset chooses the encoder), an existing output refused without `--overwrite`, `probe`'s `hdr` narrowed to a real HDR signal, `json`/`progress` out of the MCP schema, the `result_v2` preview withdrawn -- listed with what to change in [docs/contract.md](docs/contract.md#what-20-changed) and under `removed` in `contract --json`. 3.0.0 is the second break: it removed FFmpeg 5.x and Python 3.9 (announced in 2.14.0 and 2.10.9), made `cut.py` keep an `.mp4`/`.mov` copy's edit list, keep an SDR HEVC source HEVC and use `--vfr-guard sampled` by default (`--no-edit-list`, `--no-keep-hevc` and `--vfr-guard average` restore 2.x), and changed what `cut.py`'s `precision` (now the least exact segment's) and `keyframe_snapped` (now equal to `start_snapped`) mean (`contract_version` 2.0) -- see [docs/contract.md](docs/contract.md#what-30-changed).
  496. ## Development
  497. ```bash
  498. npm test # tests/run_parallel.py (test_all.py's end-to-end modules, sharded across the cores) + tests/test_contract.py
  499. python3 tests/test_picture.py # one tool group on its own (analysis, editing, cut_copy, audio, picture, delivery, orchestration, accel, asr)
  500. npm run release-check # pack, install, contract from the installed copy, MCP == contract, doctor, tests, contract evals
  501. npm run demo # python3 demos/build.py: synthetic footage -> every before/after demo + docs/demos/*.gif
  502. npm run demo:pipeline # examples/make_demo.sh: the older single end-to-end run of every script
  503. python3 evals/run.py --list # agent eval prompts (see evals/); regex-only grading, runnable from any agent harness -- see evals/README.md "Running this from Cursor, Codex, or another harness"
  504. node bin/install.js --dir /tmp/skills # try the installer without touching ~/.claude
  505. ```
  506. CI (`.github/workflows/ci.yml`) runs on every pull request and on pushes to `main`, on Ubuntu (FFmpeg 6.1 with Python 3.10 on 24.04, FFmpeg 8.0.1 with Python 3.13 on 26.04), macOS (Homebrew FFmpeg 8.x) and Windows (gyan.dev FFmpeg 9.x), plus a Linux job on FFmpeg 7.1 (Debian trixie container), and uploads each runner's FFmpeg listings as an artifact.
  507. `tests/test_contract.py` runs on all three OSes, but a handful of its tests build a fake `ffmpeg` as a `#!/bin/sh` script on a PATH shim to force specific FFmpeg 6/7/8/9 fixture layouts through `doctor`'s parser — that technique isn't portable to Windows, so `test_dry_run_never_runs_ffmpeg_and_writes_nothing` and the whole `DoctorDetectionTests` class (fixture-driven layout parsing) are individually `skipIf`'d there and show as `skipped`, not silently absent, in the Windows job's log. Everything else — contract schema, `reencodes_*`, `doctor.tools`, MCP derivation, and every tool exercised through the contract, including `cut.py`'s provenance fields — runs against the real Windows `ffmpeg` on every PR. See [references/ci-platform-pitfalls.md](references/ci-platform-pitfalls.md) for this and other per-OS behaviour differences already diagnosed, before spending a CI cycle re-diagnosing a platform-only failure.
  508. **Releasing** is fully automated end to end, including the version number itself — a PR doesn't need to touch `package.json`, `docs/contract.md`, or `CHANGELOG.md` at all. Once a PR merges to `main`, `.github/workflows/release.yml` takes it from there: if nobody bumped the version by hand, it resolves the next version ([.github/scripts/resolve_version.py](.github/scripts/resolve_version.py)) from the labels on every PR merged since the last tag (`minor`/`feature`/`enhancement` → minor, `fix`/`bug`/`patch` → patch, an unlabeled PR defaults to `patch`; a PR whose only labels are `chore`, `ci`, `docs` or `dependencies` is not releasable, so a merge that changes no shipped file releases nothing). Most PRs don't need a label added by hand: `release-drafter.yml`'s `autolabel` job applies one automatically from the PR's title/changed files (`Fix ...` → `fix`, `Add ...`/`feat ...` → `feature`, `docs`/`.md`/`.github/`/`build(deps)` changes → `chore`) as soon as it's opened — add a label yourself only to override that. **A major version is never chosen automatically**: no label rule produces `major`, and the workflow refuses to auto-bump across a major boundary even if someone applies that label — a real major release is a deliberate `package.json` bump in a PR, which the manual path below already handles. It bumps `package.json`, `.claude-plugin/plugin.json`, `docs/contract.md` and `docs/roadmap.md`'s released-version sentence, writes a `CHANGELOG.md` section listing those PRs (and any issues they closed), and pushes that commit to `main` itself. Either way — auto-bumped or hand-bumped in the PR — it then creates the `vX.Y.Z` tag, publishes a GitHub Release with notes extracted from `CHANGELOG.md`'s matching section, and publishes the package to npm. A PR that still wants to write its own version bump and `CHANGELOG.md` prose (e.g. to explain the "why" of a release by hand) can — the automation only fills in when nobody made that call already. A push to `main` with nothing new to release is a no-op. npm publishing needs an `NPM_TOKEN` repo secret (an npm access token with publish rights on this package) — without it the tag and GitHub Release still happen, only the npm step is skipped. A repo that depends on this one (an editing skill, an agent) should pin an `ffmpeg-skill` version by tag or npm version, not by tracking `main` — a merged-but-not-yet-released commit on `main` can be ahead of the last published npm version for the few minutes between merge and this workflow completing.
  509. Contributing a change: see [CONTRIBUTING.md](CONTRIBUTING.md).
  510. ## Docs
  511. | | |
  512. |---|---|
  513. | [CONTRIBUTING.md](CONTRIBUTING.md) | scope, dev setup, tests, PR expectations |
  514. | [docs/roadmap.md](docs/roadmap.md) | the release history from 1.8.0, one theme per minor, each marked shipped + evaluated, shipped with eval pending, or planned; and what 2.0.0 then removes |
  515. | [docs/design-decisions.md](docs/design-decisions.md) | behaviours that look like bugs but are decisions, with rationale and the pinning test; read before filing a bug |
  516. | [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) | Contributor Covenant 2.1; reports go through the SECURITY.md channel |
  517. | [SECURITY.md](SECURITY.md) | how to report a vulnerability privately |
  518. | [SKILL.md](SKILL.md) | what the agent reads: workflow, request → tool map, audio-only rules, report format, pitfalls |
  519. | [references/scripts.md](references/scripts.md) | per-flag reference for every tool |
  520. | [references/scripts.md § Verdict thresholds](references/scripts.md#verdict-thresholds-what-passes) | every number behind `verified` and `check.py`'s rows (tolerances, silence levels, per-platform loudness and length), each with the constant or flag it comes from |
  521. | [references/devices.md](references/devices.md) | real-device notes (iPhone HDR, GoPro, DJI, screen recordings) |
  522. | [references/ci-platform-pitfalls.md](references/ci-platform-pitfalls.md) | per-OS ffmpeg/CI behaviour differences already diagnosed once — read before re-diagnosing a Windows/macOS-only test failure |
  523. | [references/process-pitfalls.md](references/process-pitfalls.md) | process mistakes already made once (breaking a pinned test by narrowing a capability list, retrying a git/GitHub operation this environment can't do, re-designing a fixture instead of recognising a real platform difference) — a living record, add to it whenever one recurs |
  524. | [docs/contract.md](docs/contract.md) | the execution contract field by field, MCP relationship, how a planner consumes it |
  525. | [examples/README.md](examples/README.md) | natural-language requests and the commands behind them, `brand.json`, `project.json`, batch recipes |
  526. | [tests/fixtures/README.md](tests/fixtures/README.md) | captured and constructed FFmpeg listings, which is which |
  527. | [CHANGELOG.md](CHANGELOG.md) | what changed in each release |
  528. ## Support
  529. If this skill saves you time, you can help keep it maintained through [GitHub Sponsors](https://github.com/sponsors/kajisho5). Issues and pull requests are just as welcome.
  530. ## License
  531. [MIT](LICENSE)
scripts/_common/emoji.py · Bundled script · 512 lines
  1. """Emoji: cluster detection (ZWJ sequences, keycaps, flags, skin tones), the PNG asset lookup,
  2. the per-machine support probe and the overlay filter chain that composites the assets.
  3. Split out of _common.text in the refactor after 1.17.3; every body is byte-identical.
  4. """
  5. from __future__ import annotations
  6. import os
  7. import subprocess
  8. from typing import Any, Dict, List, Tuple, Optional
  9. from _common.decision import escape_filter_path
  10. from _common.emit import die
  11. from _common.runner import ffmpeg_exe
  12. from _common.runner import safe_which as _safe_which
  13. # --------------------------------------------------------------------------- emoji (1.15)
  14. # Emoji are orthogonal to the writing system: "やった 🎉" is Japanese AND emoji. They are detected
  15. # separately from detect_script() so a cue's font resolution is still decided by its letters.
  16. EMOJI_RANGES = (
  17. (0x1F300, 0x1FAFF), # symbols & pictographs, supplemental, extended-A
  18. (0x1F000, 0x1F0FF), # mahjong/domino/playing cards (only 🀄 and 🃏 are emoji)
  19. (0x2600, 0x27BF), # misc symbols + dingbats (Emoji_Presentation only: _is_emoji_char)
  20. (0x2B00, 0x2BFF), # misc symbols and arrows (likewise)
  21. (0xFE0F, 0xFE0F), # VS16 (emoji presentation selector)
  22. (0x1F1E6, 0x1F1FF), # regional indicators (flags)
  23. (0x20E3, 0x20E3), # combining enclosing keycap
  24. (0x1F3FB, 0x1F3FF), # skin-tone modifiers
  25. )
  26. # U+200D ZWJ is deliberately NOT in EMOJI_RANGES: it is ordinary Indic/Persian orthography
  27. # (क्‍ष is ka + virama + ZWJ + ssa) and only becomes emoji glue *between two emoji bases*.
  28. # Characters that never START a cluster: they bind to whatever stands before them.
  29. _EMOJI_TAIL = frozenset({0x200D, 0xFE0F, 0x20E3} | set(range(0x1F3FB, 0x1F400)))
  30. _EMOJI_REGIONAL = range(0x1F1E6, 0x1F200)
  31. _ZWJ = 0x200D
  32. _VS15 = 0xFE0E # text-presentation selector: "draw this as a character, not as an emoji"
  33. _VS16 = 0xFE0F
  34. _KEYCAP = 0x20E3
  35. _KEYCAP_BASES = frozenset("0123456789#*")
  36. def _cps(*spans) -> "frozenset":
  37. out = set()
  38. for span in spans:
  39. lo, hi = span if isinstance(span, tuple) else (span, span)
  40. out.update(range(lo, hi + 1))
  41. return frozenset(out)
  42. # The text-symbol blocks in EMOJI_RANGES (U+2600-27BF, U+2B00-2BFF, U+1F000-1F0FF) are mostly
  43. # *text*: ♪ U+266A, ✓ U+2713 and ★ U+2605 are not emoji at all in Unicode's emoji-data.txt, and
  44. # ♥ U+2665, ✔ U+2714, ☺ U+263A are emoji only when VS16 asks for the picture -- their default
  45. # presentation is the character, which the caption font draws. Counting the whole block made
  46. # SDH music notes and check marks "emoji", and `--emoji none` (what macOS and a Linux box
  47. # without a colour face get under auto) cut them out of the burned text. In those blocks only
  48. # Emoji_Presentation=Yes characters start a cluster on their own; an Emoji=Yes one does when
  49. # VS16 or a skin-tone modifier follows it. Pinned from emoji-data.txt (Unicode 15.1); the
  50. # pictograph planes keep their block rule.
  51. _EMOJI_PRESENTATION_SYMBOLS = _cps(
  52. (0x2614, 0x2615), (0x2648, 0x2653), 0x267F, 0x2693, 0x26A1, (0x26AA, 0x26AB),
  53. (0x26BD, 0x26BE), (0x26C4, 0x26C5), 0x26CE, 0x26D4, 0x26EA, (0x26F2, 0x26F3), 0x26F5,
  54. 0x26FA, 0x26FD, 0x2705, (0x270A, 0x270B), 0x2728, 0x274C, 0x274E, (0x2753, 0x2755), 0x2757,
  55. (0x2795, 0x2797), 0x27B0, 0x27BF, (0x2B1B, 0x2B1C), 0x2B50, 0x2B55, 0x1F004, 0x1F0CF)
  56. _EMOJI_TEXT_DEFAULT_SYMBOLS = _cps(
  57. (0x2600, 0x2604), 0x260E, 0x2611, 0x2618, 0x261D, 0x2620, (0x2622, 0x2623), 0x2626, 0x262A,
  58. (0x262E, 0x262F), (0x2638, 0x263A), 0x2640, 0x2642, (0x265F, 0x2660), 0x2663,
  59. (0x2665, 0x2666), 0x2668, 0x267B, 0x267E, (0x2692, 0x2697), 0x2699, (0x269B, 0x269C),
  60. 0x26A0, 0x26A7, (0x26B0, 0x26B1), 0x26C8, 0x26CF, 0x26D1, 0x26D3, 0x26E9, (0x26F0, 0x26F1),
  61. 0x26F4, (0x26F7, 0x26F9), 0x2702, (0x2708, 0x2709), (0x270C, 0x270D), 0x270F, 0x2712, 0x2714,
  62. 0x2716, 0x271D, 0x2721, (0x2733, 0x2734), 0x2744, 0x2747, (0x2763, 0x2764), 0x27A1,
  63. (0x2B05, 0x2B07))
  64. _SYMBOL_BLOCKS = ((0x2600, 0x27BF), (0x2B00, 0x2BFF), (0x1F000, 0x1F0FF))
  65. def _in_symbol_block(cp: int) -> bool:
  66. return any(lo <= cp <= hi for lo, hi in _SYMBOL_BLOCKS)
  67. def _is_emoji_char(ch: str) -> bool:
  68. """Is this character drawn as an emoji with nothing after it? In the text-symbol blocks only
  69. the Emoji_Presentation characters are; ♪ ✓ ★ ♥ are text (see _EMOJI_PRESENTATION_SYMBOLS)."""
  70. cp = ord(ch)
  71. if _in_symbol_block(cp):
  72. return cp in _EMOJI_PRESENTATION_SYMBOLS
  73. return any(lo <= cp <= hi for lo, hi in EMOJI_RANGES)
  74. def _can_be_emoji(ch: str) -> bool:
  75. """Emoji=Yes: an emoji by default, or a text-default symbol VS16 can turn into one (♥ ✔)."""
  76. return _is_emoji_char(ch) or ord(ch) in _EMOJI_TEXT_DEFAULT_SYMBOLS
  77. def _is_emoji_base(ch: str) -> bool:
  78. """Can this character START an emoji cluster? Pictographs and regional indicators can;
  79. the joiners and modifiers (ZWJ, VS16, keycap, skin tone) never can -- they only bind to an
  80. emoji base that already stands before them. Without this, a ZWJ or a VS16 sitting after an
  81. ordinary letter turned that letter into "an emoji" and the PNG route replaced it with a gap.
  82. A text-default symbol (♥ ✔ ☝) is a base only with the VS16 or modifier that follows it:
  83. emoji_clusters() checks that, this function answers for the character alone."""
  84. return ord(ch) not in _EMOJI_TAIL and _is_emoji_char(ch)
  85. def _starts_emoji(text: str, i: int) -> bool:
  86. """Does an emoji cluster start at text[i]? A default-emoji base does; a text-default symbol
  87. does only when VS16 or a skin-tone modifier follows it (♥️, ☝🏽), the sequences Unicode
  88. defines as emoji; ♥ alone, and ♪ ✓ ★ always, are characters the font draws."""
  89. ch = text[i]
  90. if _is_emoji_base(ch):
  91. return True
  92. if ord(ch) in _EMOJI_TEXT_DEFAULT_SYMBOLS and i + 1 < len(text):
  93. nxt = ord(text[i + 1])
  94. return nxt == _VS16 or 0x1F3FB <= nxt <= 0x1F3FF
  95. return False
  96. def emoji_clusters(text: str) -> "List[Tuple[int, str]]":
  97. """(index in `text`, cluster) for every emoji in it, ZWJ sequences, VS16, keycaps, flag pairs
  98. and skin-tone modifiers kept together -- 👩‍💻 is one cluster, not three, and 1️⃣ starts at the
  99. digit even though the digit is not itself an emoji character.
  100. A cluster can only START at an emoji base (a pictograph, a regional indicator) or at a keycap
  101. base (`0-9 # *`) that is actually followed by U+20E3. A ZWJ is glue *inside* a cluster, never
  102. a starter and never a tail on its own: `क्‍ष` (Hindi ka + virama + ZWJ + ssa) and `abc‍def`
  103. contain no emoji. A base explicitly marked with U+FE0E (VS15, text presentation) is likewise
  104. not an emoji -- the author asked for the character, not the picture. A text-default symbol
  105. (♥ ✔ ☝) starts one only with the VS16 or skin-tone modifier after it, and a symbol that is
  106. not emoji at all (♪ ✓ ★) never does: the caption font draws those.
  107. """
  108. out: "List[Tuple[int, str]]" = []
  109. i = 0
  110. n = len(text or "")
  111. while i < n:
  112. ch = text[i]
  113. start = i
  114. if _starts_emoji(text, i):
  115. j = i + 1
  116. if j < n and ord(text[j]) == _VS15: # text presentation requested: not an emoji
  117. i = j + 1
  118. continue
  119. elif ch in _KEYCAP_BASES:
  120. j = i + 1
  121. if j < n and ord(text[j]) == _VS16:
  122. j += 1
  123. if not (j < n and ord(text[j]) == _KEYCAP):
  124. i += 1
  125. continue
  126. j += 1
  127. else:
  128. i += 1
  129. continue
  130. # extend: modifiers bind rightwards, a ZWJ only when a real emoji base follows it
  131. while j < n:
  132. cp = ord(text[j])
  133. if cp in (_VS16, _KEYCAP) or 0x1F3FB <= cp <= 0x1F3FF:
  134. j += 1
  135. continue
  136. if cp == _ZWJ and j + 1 < n and ord(text[j + 1]) not in _EMOJI_TAIL \
  137. and _can_be_emoji(text[j + 1]):
  138. # inside a ZWJ sequence a text-default symbol is emoji glue too: 🏃‍♀ ends in ♀
  139. j += 2
  140. continue
  141. if (j == start + 1 and ord(ch) in _EMOJI_REGIONAL and cp in _EMOJI_REGIONAL):
  142. j += 1
  143. continue
  144. break
  145. out.append((start, text[start:j]))
  146. i = j
  147. return out
  148. def has_emoji(text: str) -> bool:
  149. return bool(emoji_clusters(text or ""))
  150. def emoji_codepoint_name(cluster: str) -> str:
  151. """The asset filename stem for a cluster: lowercase hex code points joined by '-', the
  152. Twemoji convention (1f389, 1f469-200d-1f4bb, 1f1ef-1f1f5); emoji_asset_for() also tries Noto's."""
  153. return "-".join(f"{ord(c):x}" for c in cluster)
  154. def _emoji_name_candidates(cluster: str) -> "List[str]":
  155. """Asset stems to try, most specific first: exact, without VS16, without skin tone, the ZWJ
  156. sequence reduced to its first code point, the bare base."""
  157. cps = [ord(c) for c in cluster]
  158. names = [emoji_codepoint_name(cluster)]
  159. def add(seq):
  160. name = "-".join(f"{c:x}" for c in seq)
  161. if name and name not in names:
  162. names.append(name)
  163. add([c for c in cps if c != 0xFE0F])
  164. add([c for c in cps if c != 0xFE0F and not (0x1F3FB <= c <= 0x1F3FF)])
  165. if 0x200D in cps:
  166. add([cps[0]])
  167. add([cps[0]])
  168. return names
  169. def emoji_asset_for(cluster: str, assets_dir: "Optional[str]") -> "Optional[str]":
  170. """The PNG for `cluster` under `assets_dir`, or None when nothing matches."""
  171. if not assets_dir or not os.path.isdir(assets_dir):
  172. return None
  173. for name in _emoji_name_candidates(cluster):
  174. # Twemoji's name (1f469-200d-1f4bb.png, 31-20e3.png), then Noto Emoji's, whose code points
  175. # are at least four hex digits (emoji_u1f469_200d_1f4bb.png, emoji_u0031_20e3.png): the hint
  176. # names noto-emoji/png/128, whose files were never found (#404)
  177. for stem in (name, "emoji_u" + "_".join(cp.zfill(4) for cp in name.split("-"))):
  178. for ext in (".png", ".PNG"):
  179. candidate = os.path.join(assets_dir, stem + ext)
  180. if os.path.isfile(candidate):
  181. return candidate
  182. return None
  183. def emoji_mono_on_colour_note(support: "Dict[str, Any]") -> str:
  184. """--emoji mono on a build whose libass draws the emoji face in colour: nothing here makes that
  185. glyph monochrome (mono and color build the same text), so the result says color (#403: the
  186. warning said "no colour path" beside a detail saying it renders in colour)."""
  187. return ("--emoji mono: libass draws the emoji face in colour on this build and nothing here makes it "
  188. "monochrome, so the emoji are drawn in colour (mode: color). " + str(support.get("detail") or ""))
  189. def emoji_missing_asset_note(missing: "List[str]", support: "Dict[str, Any]") -> str:
  190. """What happens to the clusters --emoji-assets has no PNG for: they stay in the text for libass,
  191. which is said as measured (libass_glyph), not as "drawn by the text font" (#403)."""
  192. names = ", ".join(missing)
  193. if support.get("libass_glyph") is False:
  194. return (f"no PNG in the assets directory for {names} -- those clusters stay in the text, and libass draws no "
  195. "emoji glyph on this build: an empty box at best")
  196. return (f"no PNG in the assets directory for {names} -- those clusters stay in the text for libass to draw "
  197. "with an installed face" + ("" if support.get("libass_glyph") else " (whether one has the glyph was not measured)"))
  198. EMOJI_ASSET_HINT = (
  199. "point --emoji-assets at a directory of PNGs named by code point (1f389.png): "
  200. "twemoji/assets/72x72 (Twemoji, CC-BY 4.0) or noto-emoji/png/128 (Noto Emoji, OFL/Apache-2.0) "
  201. "are the two people already have. The skill has no network at runtime, so the assets must "
  202. "already exist on this machine -- nothing is ever downloaded")
  203. _EMOJI_COLOR_FAMILIES = ("Noto Color Emoji", "Apple Color Emoji", "Segoe UI Emoji")
  204. def _fc_list_path(line: str) -> str:
  205. """The file from one `fc-list ... file` line.
  206. fontconfig prints `<file>: <families>`. The separator is a colon followed by a space.
  207. A Windows drive letter is `C:` followed by a backslash, so splitting on the first colon
  208. yields `C` and the font is reported missing.
  209. """
  210. if ": " in line:
  211. return line.partition(": ")[0].strip()
  212. return line.strip().rstrip(":")
  213. _EMOJI_SUPPORT_CACHE: "Dict[Tuple[Optional[str], bool], Dict[str, Any]]" = {}
  214. def _emoji_color_font() -> "Tuple[Optional[str], Optional[str], bool]":
  215. """(family, file, fontconfig_answered) for the first installed colour emoji family."""
  216. exe = _safe_which("fc-list")
  217. if not exe:
  218. return None, None, False
  219. for family in _EMOJI_COLOR_FAMILIES:
  220. try:
  221. proc = subprocess.run([exe, f":family={family}", "file"], stdout=subprocess.PIPE,
  222. stderr=subprocess.DEVNULL, text=True, encoding="utf-8", errors="replace", timeout=10)
  223. except (subprocess.TimeoutExpired, OSError):
  224. return None, None, False
  225. if proc.returncode != 0:
  226. return None, None, False
  227. for line in proc.stdout.splitlines():
  228. path = _fc_list_path(line)
  229. if path and os.path.exists(path):
  230. return family, path, True
  231. return None, None, True
  232. def _libass_color_probe() -> "Optional[bool]":
  233. """Does THIS ffmpeg render an emoji in colour through libass? Answered by a render, never by
  234. the font listing: Noto Color Emoji installs happily on builds whose freetype/libass has no
  235. colour-bitmap path at all, and those render a monochrome outline instead (measured). ~80 ms.
  236. None means the probe could not be run (no ffmpeg, a failure) -- unknown, not false."""
  237. exe = ffmpeg_exe("ffmpeg")[0]
  238. if not exe:
  239. return None
  240. import tempfile
  241. with tempfile.TemporaryDirectory() as td:
  242. srt = os.path.join(td, "e.srt")
  243. with open(srt, "w", encoding="utf-8") as fh:
  244. fh.write("1\n00:00:00,000 --> 00:00:01,000\n\U0001F389\n")
  245. try:
  246. proc = subprocess.run(
  247. [exe, "-hide_banner", "-loglevel", "error", "-f", "lavfi",
  248. "-i", "color=c=black:s=64x64:d=0.04",
  249. "-vf", "subtitles=" + escape_filter_path(srt, bind=False), "-frames:v", "1",
  250. "-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
  251. stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=20)
  252. except (subprocess.TimeoutExpired, OSError):
  253. return None
  254. if proc.returncode != 0 or len(proc.stdout) < 64 * 64 * 3:
  255. return None
  256. data = proc.stdout
  257. for i in range(0, 64 * 64 * 3, 3):
  258. r, g, b = data[i], data[i + 1], data[i + 2]
  259. if max(r, g, b) - min(r, g, b) > 40:
  260. return True
  261. return False
  262. def _libass_render(exe: str, char: str, fonts_dir: "Optional[str]" = None) -> "Optional[bytes]":
  263. """One 96x96 rgb24 frame of `char` drawn through libass (subtitles=), or None when it failed."""
  264. import tempfile
  265. with tempfile.TemporaryDirectory() as td:
  266. srt = os.path.join(td, "g.srt")
  267. with open(srt, "w", encoding="utf-8") as fh:
  268. fh.write(f"1\n00:00:00,000 --> 00:00:01,000\n{char}\n")
  269. try:
  270. proc = subprocess.run(
  271. [exe, "-hide_banner", "-loglevel", "error", "-f", "lavfi",
  272. "-i", "color=c=black:s=96x96:d=0.04",
  273. "-vf", "subtitles=" + escape_filter_path(srt, bind=False) + ":force_style='FontSize=64,Alignment=5'"
  274. + (":fontsdir=" + escape_filter_path(fonts_dir, bind=False) if fonts_dir else ""),
  275. "-frames:v", "1", "-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
  276. stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=20)
  277. except (subprocess.TimeoutExpired, OSError):
  278. return None
  279. if proc.returncode != 0 or len(proc.stdout) < 96 * 96 * 3:
  280. return None
  281. return proc.stdout
  282. def _libass_glyph_probe(fonts_dir: "Optional[str]" = None) -> "Optional[bool]":
  283. """Does libass draw an emoji GLYPH at all, or only what it draws for a character no face has?
  284. "No colour pixels" (_libass_color_probe False) covers both a monochrome glyph and the empty
  285. .notdef box: Apple Color Emoji (an sbix bitmap face) gives libass no usable glyph, and the box
  286. is white (#395). So the emoji is compared with U+10FFFD, a code point no font carries: the same
  287. frame, or nothing lit, is no glyph. A caller's --fonts-dir is searched too, as the burn searches it.
  288. None: the probe could not run."""
  289. exe = ffmpeg_exe("ffmpeg")[0]
  290. if not exe:
  291. return None
  292. emoji, nothing = _libass_render(exe, "\U0001F389", fonts_dir), _libass_render(exe, "\U0010FFFD", fonts_dir)
  293. if emoji is None or nothing is None:
  294. return None
  295. lit = any(emoji[i] > 60 for i in range(0, len(emoji)))
  296. return bool(lit and emoji != nothing)
  297. def emoji_support(assets: "Optional[str]" = None, probe: bool = True, fonts_dir: "Optional[str]" = None) -> "Dict[str, Any]":
  298. """What this machine can actually do with emoji, cached per process.
  299. `mode` is `color` when a render probe proves libass draws colour, else `png` when an assets
  300. directory resolves, else `mono` when some installed face has a glyph at all, else `none`.
  301. An installed colour emoji font proves nothing on its own -- that is why `libass_color` comes
  302. from a render (see references/gotchas.md#emoji). `probe=False` (`contract --json --static`, and every
  303. static/JSON-only path) skips the render entirely and leaves `libass_color` unknown.
  304. """
  305. key = (assets or None, bool(probe), fonts_dir or None)
  306. if key in _EMOJI_SUPPORT_CACHE:
  307. return dict(_EMOJI_SUPPORT_CACHE[key])
  308. family, file, fc_answered = _emoji_color_font()
  309. libass_color = _libass_color_probe() if probe else None
  310. # a colour render is a glyph; without one, ask whether libass draws one at all (#395)
  311. libass_glyph = True if libass_color else (_libass_glyph_probe(fonts_dir) if probe and libass_color is False else None)
  312. assets_dir = assets if (assets and os.path.isdir(assets)) else None
  313. if libass_color:
  314. mode = "color"
  315. elif assets_dir:
  316. mode = "png"
  317. elif family and libass_glyph is False:
  318. mode = "none"
  319. elif family or libass_glyph:
  320. # a face libass draws the emoji from, colour family listed or not (A64, #457: a glyph from
  321. # a monochrome face was answered `none`, against this docstring and libass_glyph: true)
  322. mode = "mono"
  323. elif not fc_answered:
  324. # No fontconfig to ask (a static ffmpeg build, a bare container): the PNG path needs none,
  325. # so the honest answer is png-or-none, never "none because fc-list is missing".
  326. mode = "none"
  327. else:
  328. mode = "none"
  329. if not fc_answered:
  330. detail = "no fontconfig on this machine; the PNG overlay path needs none"
  331. elif libass_color:
  332. detail = f"{family or 'an installed face'} renders in colour through libass on this ffmpeg"
  333. elif family and libass_glyph is False:
  334. detail = (f"{family} installed but libass draws no emoji glyph from it on this build (the render "
  335. "matches a code point no font has: an empty box or nothing)")
  336. elif family and libass_color is False:
  337. detail = f"{family} installed but libass renders it monochrome on this build"
  338. elif family and libass_color is None:
  339. detail = f"{family} installed; the colour render probe was not run"
  340. elif assets_dir:
  341. detail = "no colour emoji family installed; using the PNG assets directory"
  342. elif libass_glyph:
  343. detail = "no colour emoji family installed; libass draws the emoji monochrome from an installed face"
  344. else:
  345. detail = "no colour emoji family installed and no --emoji-assets directory"
  346. result = {"mode": mode, "color_font": family, "color_font_file": file,
  347. "libass_color": libass_color, "libass_glyph": libass_glyph, "assets": assets_dir,
  348. "detail": detail, "fix": EMOJI_ASSET_HINT}
  349. _EMOJI_SUPPORT_CACHE[key] = result
  350. return dict(result)
  351. def resolve_emoji_assets(flag: "Optional[str]" = None, project: "Optional[str]" = None,
  352. brand: "Optional[dict]" = None) -> "Optional[str]":
  353. """--emoji-assets DIR, else the project key, else brand.json, else FFMPEG_SKILL_EMOJI_ASSETS.
  354. A directory that was named but does not exist is a failed job, never a silent downgrade."""
  355. brand = brand or {}
  356. styles = (brand.get("styles") or {}).get("caption") or {}
  357. for value, where in ((flag, "--emoji-assets"), (project, "the project's text.emoji_assets"),
  358. (styles.get("emoji_assets"), "brand.json styles.caption.emoji_assets"),
  359. (brand.get("emoji_assets"), "brand.json emoji_assets"),
  360. (os.environ.get("FFMPEG_SKILL_EMOJI_ASSETS"), "FFMPEG_SKILL_EMOJI_ASSETS")):
  361. if not value:
  362. continue
  363. if not os.path.isdir(str(value)):
  364. die(f"{where}: {value} is not a readable directory -- {EMOJI_ASSET_HINT}", kind="input")
  365. return str(value)
  366. return None
  367. def emoji_filter_chain(plan, base_label, out_label, first_input=1, to_yuv="", base_chain="", overlay_opts=""):
  368. """(chains, inputs) that composite the planned PNGs on top of `base_label`.
  369. `inputs` is a list of argv fragments, each ending in the asset path, to be appended to the
  370. ffmpeg command in order (an overlay that fades needs `-loop 1` on its input so the still has
  371. a timeline the fade filter can move along; one that does not is a plain `-i`).
  372. `to_yuv` ends each PNG's chain: the caller's rgb_chain_to_bt709(meta), so the RGBA asset is
  373. converted to the BT.709 the output is tagged with instead of by overlay's BT.601 default;
  374. `base_chain` (label_chain(meta, meta): an untagged source's BT.709 label) goes in front of
  375. the first overlay, so 7.1+ does not convert the two pictures to one another.
  376. `overlay_opts` ends each overlay's options: the caller's overlay_format(meta), so a 10-bit
  377. picture is not composited at overlay's 8-bit default.
  378. """
  379. overlays = plan.get("overlays") or []
  380. if not overlays:
  381. return [], []
  382. # Group by everything that makes two uses of the same PNG a different STREAM: the fade is
  383. # expressed in the cue's own timeline, so two cues cannot share one faded input.
  384. def _key(o):
  385. fades = (round(float(o.get("fade_in") or 0.0), 3), round(float(o.get("fade_out") or 0.0), 3))
  386. window = (round(float(o["start"]), 3), round(float(o["end"]), 3)) if any(fades) else (None, None)
  387. return (o["asset"], o["box"]) + fades + window
  388. groups: "List[Tuple]" = []
  389. for o in overlays:
  390. if _key(o) not in groups:
  391. groups.append(_key(o))
  392. chains: List[str] = []
  393. inputs: "List[List[str]]" = []
  394. pads: "Dict[Tuple, List[str]]" = {}
  395. for k, key in enumerate(groups):
  396. asset, box, fin, fout, gstart, gend = key
  397. uses = [o for o in overlays if _key(o) == key]
  398. idx = first_input + k
  399. labels = [f"e{k}_{j}" for j in range(len(uses))]
  400. chain = f"[{idx}:v]format=rgba,scale={box}:{box}"
  401. if fin or fout:
  402. # -loop 1 gives the still an advancing timeline on the SAME clock as the main video,
  403. # so the fade times below are the cue's own seconds. The emoji then appears and
  404. # leaves with the text instead of popping in against a fading line.
  405. # -t bounds the loop at the cue's end: an unbounded looped still never EOFs and the
  406. # whole encode hangs (overlay keeps pulling from it after the main video is done).
  407. inputs.append(["-loop", "1", "-t", f"{gend:.3f}", "-i", asset])
  408. if fin:
  409. chain += f",fade=t=in:st={gstart:.3f}:d={fin:.3f}:alpha=1"
  410. if fout:
  411. chain += f",fade=t=out:st={max(gstart, gend - fout):.3f}:d={fout:.3f}:alpha=1"
  412. else:
  413. inputs.append(["-i", asset])
  414. chain += to_yuv
  415. if len(labels) > 1:
  416. chain += f",split={len(labels)}"
  417. chains.append(chain + "".join(f"[{l}]" for l in labels))
  418. pads[key] = labels
  419. cur = base_label
  420. if base_chain:
  421. chains.append(f"[{cur}]{base_chain}null[ebase]")
  422. cur = "ebase"
  423. remaining = {key: list(v) for key, v in pads.items()}
  424. for j, o in enumerate(overlays):
  425. label = remaining[_key(o)].pop(0)
  426. nxt = out_label if j == len(overlays) - 1 else f"eov{j}"
  427. x = o["x"]
  428. x = f"'{x}'" if isinstance(x, str) else x
  429. # No eof_action=pass here: a PNG input is a SINGLE frame at pts 0, and eof_action=pass
  430. # switches off overlay's default "hold the last frame of the secondary input", so the
  431. # asset would be composited on frame 0 only and vanish for the rest of the cue (that is
  432. # exactly what shipped first). eof_action=repeat (the default) holds the still for the
  433. # whole timeline; enable= is what confines it to the cue's window.
  434. chains.append(f"[{cur}][{label}]overlay=x={x}:y={o['y']}:"
  435. f"enable='between(t,{o['start']:.3f},{o['end']:.3f})'{overlay_opts}[{nxt}]")
  436. cur = nxt
  437. return chains, inputs

Version history

  • v2.16.110/25/2026

  • v2.12.010/24/2026

  • v2.11.010/23/2026

  • v2.10.710/21/2026

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.