Skip to content
Back to search

vex

SkillWorks with Claude CodeGitHub

claude code for video editing

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/akmessi--vex/badge.svg)](/listings/akmessi--vex)

HTML

<a href="/listings/akmessi--vex"><img src="/api/listings/akmessi--vex/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requirements

- Python 3.11+ - FFmpeg installed and available on `PATH` - `yt-dlp` available through the Python environment for YouTube downloads - `openai-whisper` is optional and only needed for local transcription - Node.js 22+ and npm are required for HyperFrames- or Remotion-powered generated visuals - `manim` is optional for specialist math, geometry, and axes-heavy generated visuals - `blender` is optional if you want typed 3D titles, transparent overlays, product/model spins, logo reveals, object orbits, and cinematic replacement shots; set `BLENDER_PATH` to `blender`, a full executable path, or a Blender install directory if it is not already on `PATH`

FFmpeg install:

- macOS: `brew install ffmpeg` - Ubuntu/Debian: `sudo apt install ffmpeg` - Windows: install from `https://ffmpeg.org/download.html` and add `ffmpeg/bin` to `PATH`

Install Vex

`pipx` is the recommended installation method because it gives Vex an isolated Python environment while keeping the `vex` command available globally:

pipx install vex-video

Standard `pip` installation is also supported:

python -m pip install vex-video

Optional feature sets:

pipx install "vex-video[transcription]"
pipx install "vex-video[manim]"
pipx install "vex-video[all]"

For an existing pipx installation, install Whisper into Vex's isolated environment with:

vex setup transcription

Vex also discovers a `python` or `python3` executable on `PATH` that already contains Whisper. Set `WHISPER_PYTHON_PATH` when the intended interpreter is not the first one on `PATH`.

For a source checkout used for development:

git clone https://github.com/AKMessi/vex.git
cd vex
python -m pip install -e ".[dev]"

Verify the installation:

vex --version
vex renderers doctor

For HyperFrames or Remotion visuals, install Vex's shared version-locked renderer runtime explicitly:

vex renderers install hyperframes
vex renderers install remotion

This runs `npm ci` from the lockfile shipped with the installed Vex version. Vex does not silently install Node packages during `pip install` or first launch. For a source checkout, Remotion uses the repository `package-lock.json`; run `npm ci` before `renderer=remotion` jobs.

See [docs/installing.md](docs/installing.md) for upgrades, release candidates, rollbacks, optional features, and uninstall behavior.

Windows PATH note

Run `pipx ensurepath`, restart the terminal, and retry. For a standard `pip` installation, add your Python Scripts directory to `PATH`.

Example:

C:\Users\aarya\AppData\Roaming\Python\Python314\Scripts

Your path may differ depending on your Python installation.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 1,119 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Vex
  2. <p align="center">
  3. <strong>Terminal-first AI video editing, driven by plain English.</strong>
  4. </p>
  5. <p align="center">
  6. <a href="https://github.com/AKMessi/vex"><img alt="Source available, non-commercial" src="https://img.shields.io/badge/source--available-PolyForm--NC--1.0.0-111111?style=for-the-badge"></a>
  7. <img alt="Python 3.11+" src="https://img.shields.io/badge/python-3.11%2B-3776AB?style=for-the-badge&logo=python&logoColor=white">
  8. <img alt="FFmpeg powered" src="https://img.shields.io/badge/FFmpeg-powered-007808?style=for-the-badge&logo=ffmpeg&logoColor=white">
  9. <img alt="Terminal native" src="https://img.shields.io/badge/terminal-native-1F2937?style=for-the-badge">
  10. </p>
  11. <p align="center">
  12. <img src="assets/terminal.png" alt="Vex running in a terminal" width="900">
  13. </p>
  14. Vex is a source-available, non-commercial AI video editing agent for the terminal. Launch `vex`, talk to it in plain English, point it at a video file, and it edits a safe working copy of your footage using FFmpeg, MoviePy, transcript intelligence, generated visuals, and an LLM-driven tool loop.
  15. It is built for creators and builders who want CLI speed without memorizing editing commands, FFmpeg flags, subtitle workflows, color-correction knobs, or export presets.
  16. ## Highlights
  17. | You ask for | Vex handles |
  18. |---|---|
  19. | "Trim the awkward intro and remove pauses." | Timestamp parsing, silence detection, timeline-safe edits, and rebuildable project state |
  20. | "Auto color grade this properly." | Shot-aware exposure, contrast, saturation, white balance, look selection, preview scoring, and output validation |
  21. | "Add subtitle-aware zoom effects." | Caption timing, emphasis scoring, punch-ins, pans, pulses, freeze accents, vignette, flash, focus, and subtitle highlights in one replayable pass |
  22. | "Add visuals where the explanation needs them." | Transcript beat mining, renderer selection, Hyperframes, Manim, or Remotion generation, QA, and compositing |
  23. | "Turn this podcast into shorts." | Shorts Director planning, highlight selection, vertical reframing, captions, hooks, scoring, metadata, and bundle manifests |
  24. | "Convert this MOV to MP4 and compress it." | Metadata inspection, encode planning, safe FFmpeg command generation, confirmation, and validation |
  25. ## Why Vex
  26. - Natural language first: describe the edit you want instead of memorizing tool syntax
  27. - Safe by design: original footage stays untouched while Vex edits a project working copy
  28. - Stateful projects: resume later with timeline history, undo, redo, and rebuild support intact
  29. - Real editing operations: trims, overlays, audio edits, subtitle burn-in, silence cleanup, exports, and more
  30. - Production-minded auto color grading: sampled-frame analysis builds reusable FFmpeg grades with shot-aware candidate scoring and validation
  31. - Subtitle-aware auto effects: caption beats can trigger punch-ins, punch-outs, slow pushes, pans, pulses, freeze accents, subtle shake, vignette, flash, focus, and subtitle-safe highlights
  32. - Transcript-aware visuals: Vex can plan explanatory inserts from narration, generate them with Hyperframes, Manim, Remotion, or typed Blender 3D asset templates, and composite them back into the cut
  33. - Plain-English encoding: messy export requests become inspected, validated FFmpeg plans before anything runs
  34. - Multi-provider ready: Gemini by default, Claude, Groq Qwen native vision, or local OpenAI-compatible providers when configured
  35. - Live run status: see the active tool, progress, and optional trace artifacts while the agent works
  36. - Terminal-native: fast, scriptable, and easy to integrate into existing workflows
  37. Generated visuals share source-grounded meaning checks, actual browser typography measurements, bounded frame-based repairs, resumable rendering, and final asset receipts. See [native vision setup](docs/native-vision.md), the [implementation ledger](docs/visual-harness-implementation.md), and the [rendered benchmark](docs/rendered-visual-benchmark.md) for configuration, validation, and measurement limits.
  38. ## Contents
  39. - [What Vex Can Do](#what-vex-can-do)
  40. - [What's New: Auto Visuals](#whats-new-auto-visuals)
  41. - [Auto Color Grading](#auto-color-grading)
  42. - [Installation](#installation)
  43. - [Configuration](#configuration)
  44. - [Quick Start](#quick-start)
  45. - [Vex Studio](#vex-studio)
  46. - [Natural-Language Examples](#natural-language-examples)
  47. - [Full Tool Surface](#full-tool-surface)
  48. - [CLI Commands](#cli-commands)
  49. - [Architecture](#architecture)
  50. - [Troubleshooting](#troubleshooting)
  51. ## What Vex Can Do
  52. ### Core editing
  53. - Inspect video metadata
  54. - Trim clips by timestamp
  55. - Merge multiple clips
  56. - Adjust playback speed for a full clip or a selected segment
  57. - Add fade in, fade out, and fade-through-black transitions
  58. - Add timed text overlays
  59. - Remove silent gaps from raw footage
  60. - Extract selected highlight segments into a shorter cut
  61. - Auto color grade with natural, vibrant, cinematic, warm, cool, documentary, or punchy looks
  62. ### Audio
  63. - Extract audio as `mp3`, `wav`, or `aac`
  64. - Replace a video's audio track
  65. - Mix external audio with original audio
  66. - Mute a selected time range
  67. ### Captions and transcript workflows
  68. - Transcribe video locally with Whisper
  69. - Generate `transcript.txt` and `transcript.srt`
  70. - Burn subtitles directly into the video from an SRT file
  71. - Auto-summarize long clips into highlight cuts using transcript-aware segment selection
  72. - Auto-create multiple vertical shorts with a typed Shorts Director v3 program, graph-searched multi-source edit plans, captions, ranking, hooks, metadata, and a bundle manifest
  73. - Score candidate clips with a shared local Video Understanding Graph for retention, visual opportunity, topic alignment, hook strength, payoff, novelty, clarity, shareability, standalone story completeness, pacing, and topic diversity before final selection
  74. - Run pre-render edit-plan validation plus final transcript quality gates so abrupt, incoherent, invalid, or weak-payoff cuts are rejected before the final manifest
  75. - Generate timestamped B-roll suggestions for each short
  76. - Fetch and splice subtitle-aligned, transcript-aware stock B-roll from configured providers such as Pexels, Pixabay, and Coverr into the working video
  77. - Add context-aware auto emphasis effects from full-video rhythm, transcript timing, scene stability, pacing, pauses, questions, numeric claims, contrast turns, and payoff lines
  78. - Generate transcript-aligned custom visuals and animations with Hyperframes-first HTML motion slides, typed Blender 3D assets, and Manim retained for specialist math/geometry scenes
  79. - Generate brand-new audio-synced HyperFrames videos from a prompt or script without loading a source video
  80. - Add transcript-driven punch-in moments for emphasis inside generated shorts
  81. - Record local creative-run history with graph versions, quality scores, manifest paths, and output artifacts for shorts, visuals, and color grading
  82. - Optimize Auto Visuals as a coherent semantic portfolio, compare compatible renderers by measured output quality, learn bounded project-local quality priors, and verify the final encoded composite before changing project state
  83. - Author Auto Visuals as signed, evidence-bound Open Visual Programs so the model
  84. can control primitives, layout, metaphor, and choreography without receiving
  85. unrestricted renderer-code execution
  86. ## Local Creative Intelligence
  87. Vex now builds a shared local `VideoUnderstandingGraph` for the highest-level creative automation paths. Auto shorts, auto visuals, and auto color grading no longer depend only on isolated feature heuristics; they share local evidence about transcript beats, retention moments, topic weights, scene cuts, quality contracts, and graph-backed scoring.
  88. Each successful creative run records:
  89. - a feature manifest with the creative graph or graph summary
  90. - a creative QA report with score, pass/review status, issues, warnings, metrics, and evidence
  91. - a local `creative_runs.json` registry entry in the project working directory
  92. - dashboard and CLI visibility through `/creative-runs` or `vex creative-runs --project <project-id>`
  93. ## What's New: Auto Visuals
  94. `add_auto_visuals` is the biggest recent addition to Vex.
  95. Instead of only fetching stock footage, Vex can now:
  96. - transcribe a talking-head or explainer video
  97. - segment long transcripts into semantic episodes before scoring which spoken
  98. windows deserve a custom visual
  99. - preflight source-grounded opportunity contracts through the HyperFrames
  100. semantic compiler before expensive scene authoring
  101. - schedule a globally coherent visual set with episode-aware reserves instead
  102. of independently selecting the highest-scoring subtitle lines
  103. - build a video-level visual narrative program with chapters, concept memory, continuity groups, and transition intent
  104. - plan where a full-screen replacement is safe versus where picture-in-picture is smarter
  105. - compile transcript evidence into typed facts, explanation objects, semantic beats, and explicit rejection reasons
  106. - compile Remotion opportunities into signed semantic scene programs with
  107. hard evidence constraints, signed SceneGraph v2 capability plans, responsive
  108. constraint solving, measured text, and frame-driven motion before starting
  109. Chromium
  110. - compile both HyperFrames and Remotion through one signed creative-direction
  111. harness for semantic medium choice, focal hierarchy, art direction,
  112. choreography, negative space, and video-level anti-repetition
  113. - author multiple complete Open Visual Program scene graphs, reject invented or
  114. unbound content, and preserve deterministic candidates when model authoring is
  115. unavailable
  116. - compile narration into signed atomic propositions and viewer questions, then
  117. search six distinct visual concept lanes with four-frame reference boards
  118. - inspect final rendered frames with a blind multimodal verifier, retry through
  119. independent providers, and keep unavailable, degraded, verified, and rejected
  120. states explicit
  121. - render a bounded alternate concept and use order-reversed pairwise judging to
  122. choose the clearer and more beautiful silent explanation
  123. - apply typed counterexample-guided repairs, re-render them, and accept only
  124. candidates that improve without semantic or technical regression
  125. - evaluate the complete published set for repeated concept lanes, media, motion
  126. grammar, composition, palette, and rendered appearance
  127. - select a role-constrained semantic blueprint and sign a production contract before Hyperframes receives a render spec
  128. - generate transcript-aligned custom visuals with a deterministic Hyperframes HTML renderer
  129. - compile every Hyperframes scene through a typed design IR for art direction, density, theme, safe areas, and motion intensity
  130. - carry episode context, visual beats, recurring concepts, and transition contracts into Hyperframes metadata
  131. - ground interface walkthroughs in real captured source frames when the source contains a screen or slide
  132. - render multiple art-directed variants, inspect semantic state at four times, score extracted frames for contrast, occupancy, dead space, edge safety, and motion, then promote only a passing version
  133. - score rendered candidates with a shared aesthetic critic for balance,
  134. hierarchy, depth, palette vitality, edge safety, whitespace, and settled
  135. motion before promotion
  136. - lint, validate, and render those scenes before the final composite
  137. - optimize the complete visual set for quality, semantic coverage, diversity, and clean timing instead of truncating candidates by timestamp
  138. - substitute a preflighted reserve when a primary scene fails compilation,
  139. rendering, or final timeline QA
  140. - remember failed source moments inside the project so retries do not repeat
  141. the same rejected visual concept
  142. - compare compatible renderer outputs through a bounded quality tournament when renderer routing is flexible
  143. - verify the final encoded replacement frames, duration, resolution, file integrity, and audio before project-state promotion
  144. - learn conservative project-local renderer and intent priors from repeated render QA outcomes without bypassing hard gates
  145. - fall back to specialist renderers only when a scene needs a different engine
  146. - render deterministic Blender 3D assets from typed specs for titles, labels, pointers, product spins, object orbits, logo reveals, and data tunnels without allowing raw Blender Python from the agent
  147. The current renderer stack is:
  148. - `hyperframes` for premium HTML/CSS motion slides, process diagrams, product UI scenes, comparisons, timelines, data-driven explainers, causal chains, flywheels, decision matrices, anatomy cutaways, rankings, contrast ladders, proof sequences, and narrative arcs with built-in variant QA
  149. - `remotion` for source-grounded React motion graphics compiled into typed
  150. metric, mechanism, contrast, timeline, interface, and emphasis programs. It
  151. renders and scores six distinct treatments in one browser session, gates
  152. solved geometry and dense temporal samples, and emits verified ProRes masters
  153. through Remotion's local SSR APIs
  154. - `manim` for formula-heavy math, geometry, axes, and visuals that genuinely need Manim's object model
  155. - `ffmpeg` for fast editorial overlays and clean picture-in-picture support graphics
  156. - `blender` for optional deterministic 3D assets when Blender is installed. Vex owns timing, project state, undo/rebuild, and final FFmpeg compositing; Blender only renders the visual asset.
  157. Best results today:
  158. - short and long-form explainers, tutorials, essays, and talking-head videos
  159. - beats with clear process, contrast, numbers, or "how it works" structure
  160. - multi-sentence windows with enough local evidence and screen time for a
  161. custom visual to actually add intuition
  162. Vex deliberately tries to skip weak beats instead of forcing generic filler.
  163. The complete implementation report is in
  164. [docs/feedback-driven-automation-report.md](docs/feedback-driven-automation-report.md).
  165. The Remotion semantic architecture is in
  166. [docs/remotion-auto-visuals-architecture.md](docs/remotion-auto-visuals-architecture.md).
  167. The shared HyperFrames and Remotion direction harness is in
  168. [docs/creative-direction-harness.md](docs/creative-direction-harness.md).
  169. The shared generative scene-graph architecture is in
  170. [docs/open-visual-program.md](docs/open-visual-program.md).
  171. The final-render search, verification, repair, and portfolio architecture is in
  172. [docs/visual-director-v2.md](docs/visual-director-v2.md).
  173. The repository-wide quality architecture continuation is in
  174. [docs/creative-quality-architecture-report.md](docs/creative-quality-architecture-report.md).
  175. The HyperFrames director skill pack is in
  176. [skills/vex-hyperframes-director](skills/vex-hyperframes-director).
  177. The counterexample-guided repair skill pack is in
  178. [skills/vex-hyperframes-visual-repair](skills/vex-hyperframes-visual-repair).
  179. ## Auto Color Grading
  180. `auto_color_grade` analyzes sampled frames from the current working cut, builds a deterministic Color Grade Director plan, and applies a reusable FFmpeg filter graph. The timeline stores the exact filter graph, so undo, redo, and project rebuilds do not depend on re-running analysis.
  181. It corrects:
  182. - exposure and gamma drift
  183. - low contrast or overly harsh contrast
  184. - muted saturation
  185. - red, green, or blue color casts through bounded white balance
  186. - weak black/white points with guarded level normalization
  187. - flat footage with subtle look-aware contrast curves
  188. - screen recordings, talking heads, product shots, low-light shots, and already-graded footage with subject-aware protection budgets
  189. The analyzer weights reliable frames, skips flat transition frames, estimates white balance from neutral midtones instead of saturated image regions, and guards saturation/channel changes when skin-tone-like pixels are present. The Color Grade Director classifies each shot, picks an auto-look policy, caps risky saturation/contrast/gamma changes for UI and skin, scores multiple preview candidates, and smooths similar neighboring shots so the grade does not flicker between scenes.
  190. After rendering, Vex samples the graded output again and records a validation score with warnings for remaining clipping, low contrast, extreme brightness, or risky saturation. The color-grade tool also attaches a local creative QA report that combines output validation, shot validation, director metadata, candidate scoring, preview coverage, correction strength, and warnings into one publishability signal.
  191. Supported looks are `auto`, `natural`, `vibrant`, `cinematic`, `warm`, `cool`, `documentary`, and `punchy`.
  192. ### Export and delivery
  193. - Export with built-in presets for YouTube, Instagram, TikTok, X, and podcast audio
  194. - Plan plain-English FFmpeg encode, conversion, compression, and target-size commands before running them
  195. - Validate encoded files with `ffprobe` plus a full FFmpeg decode pass before reporting success
  196. - Export directly from the REPL or from the CLI
  197. - Estimate output size and check disk space before export
  198. ### Project system
  199. - Persistent saved projects
  200. - Auto-load a project when you mention a known video path
  201. - Deterministic project loading
  202. - Undo and redo through timeline rebuild
  203. - Timeline inspection and project summaries
  204. ## The Intended Experience
  205. This is the default workflow:
  206. ```text
  207. > vex
  208. [Vex banner]
  209. Vex > trim the first 30 seconds of "D:\videos\clip.mp4"
  210. Vex > remove awkward pauses
  211. Vex > burn subtitles
  212. Vex > export it for instagram
  213. Vex > /quit
  214. ```
  215. You do not need to type subcommands inside the session.
  216. If no project is loaded, include a video path in your first message and Vex handles the rest.
  217. During each turn, Vex shows a live status spinner with the active tool name. If you want the deeper activity log afterward, use `/trace`.
  218. ## Installation
  219. ### Requirements
  220. - Python 3.11+
  221. - FFmpeg installed and available on `PATH`
  222. - `yt-dlp` available through the Python environment for YouTube downloads
  223. - `openai-whisper` is optional and only needed for local transcription
  224. - Node.js 22+ and npm are required for HyperFrames- or Remotion-powered generated visuals
  225. - `manim` is optional for specialist math, geometry, and axes-heavy generated visuals
  226. - `blender` is optional if you want typed 3D titles, transparent overlays, product/model spins, logo reveals, object orbits, and cinematic replacement shots; set `BLENDER_PATH` to `blender`, a full executable path, or a Blender install directory if it is not already on `PATH`
  227. FFmpeg install:
  228. - macOS: `brew install ffmpeg`
  229. - Ubuntu/Debian: `sudo apt install ffmpeg`
  230. - Windows: install from `https://ffmpeg.org/download.html` and add `ffmpeg/bin` to `PATH`
  231. ### Install Vex
  232. `pipx` is the recommended installation method because it gives Vex an isolated
  233. Python environment while keeping the `vex` command available globally:
  234. ```bash
  235. pipx install vex-video
  236. ```
  237. Standard `pip` installation is also supported:
  238. ```bash
  239. python -m pip install vex-video
  240. ```
  241. Optional feature sets:
  242. ```bash
  243. pipx install "vex-video[transcription]"
  244. pipx install "vex-video[manim]"
  245. pipx install "vex-video[all]"
  246. ```
  247. For an existing pipx installation, install Whisper into Vex's isolated
  248. environment with:
  249. ```bash
  250. vex setup transcription
  251. ```
  252. Vex also discovers a `python` or `python3` executable on `PATH` that already
  253. contains Whisper. Set `WHISPER_PYTHON_PATH` when the intended interpreter is
  254. not the first one on `PATH`.
  255. For a source checkout used for development:
  256. ```bash
  257. git clone https://github.com/AKMessi/vex.git
  258. cd vex
  259. python -m pip install -e ".[dev]"
  260. ```
  261. Verify the installation:
  262. ```bash
  263. vex --version
  264. vex renderers doctor
  265. ```
  266. For HyperFrames or Remotion visuals, install Vex's shared version-locked renderer runtime explicitly:
  267. ```bash
  268. vex renderers install hyperframes
  269. vex renderers install remotion
  270. ```
  271. This runs `npm ci` from the lockfile shipped with the installed Vex version. Vex
  272. does not silently install Node packages during `pip install` or first launch.
  273. For a source checkout, Remotion uses the repository `package-lock.json`; run
  274. `npm ci` before `renderer=remotion` jobs.
  275. See [docs/installing.md](docs/installing.md) for upgrades, release candidates,
  276. rollbacks, optional features, and uninstall behavior.
  277. ### Windows PATH note
  278. Run `pipx ensurepath`, restart the terminal, and retry. For a standard `pip`
  279. installation, add your Python Scripts directory to `PATH`.
  280. Example:
  281. ```text
  282. C:\Users\aarya\AppData\Roaming\Python\Python314\Scripts
  283. ```
  284. Your path may differ depending on your Python installation.
  285. ## Configuration
  286. Create a local configuration template without needing a source checkout:
  287. ```bash
  288. vex setup config
  289. ```
  290. Vex refuses to overwrite an existing `.env` unless `--force` is supplied.
  291. Then configure your provider in that file.
  292. ### Default provider
  293. Vex defaults to Gemini.
  294. ```env
  295. PROVIDER=gemini
  296. GEMINI_API_KEY=your_key_here
  297. GEMINI_MODEL=gemma-4-31b-it
  298. ```
  299. ### Claude support
  300. If you want Claude instead:
  301. ```env
  302. PROVIDER=claude
  303. ANTHROPIC_API_KEY=your_key_here
  304. ```
  305. ### Local LLM support
  306. Vex can also use local OpenAI-compatible servers such as Ollama, LM Studio, and llama.cpp:
  307. ```env
  308. PROVIDER=ollama
  309. OLLAMA_BASE_URL=http://localhost:11434/v1
  310. OLLAMA_MODEL=qwen2.5-coder:14b
  311. ```
  312. See [docs/local-llms.md](docs/local-llms.md) for the full setup guide and troubleshooting notes.
  313. ### Important settings
  314. - `PROVIDER`
  315. - `GEMINI_API_KEY`
  316. - `GEMINI_MODEL`
  317. - `ANTHROPIC_API_KEY`
  318. - `CLAUDE_MODEL`
  319. - `OPENAI_COMPAT_BASE_URL`
  320. - `OPENAI_COMPAT_MODEL`
  321. - `OLLAMA_BASE_URL`
  322. - `OLLAMA_MODEL`
  323. - `LM_STUDIO_BASE_URL`
  324. - `LM_STUDIO_MODEL`
  325. - `LLAMA_CPP_BASE_URL`
  326. - `LLAMA_CPP_MODEL`
  327. - `PEXELS_API_KEY`
  328. - `PIXABAY_API_KEY`
  329. - `COVERR_API_KEY`
  330. - `AUTO_BROLL_PROVIDERS` defaults to `auto`; use `pexels`, `pixabay`, `coverr`, or comma-separated names to control stock B-roll provider order
  331. - `AUTO_BROLL_MAX_OVERLAYS` defaults to `24`
  332. - `AUTO_VISUALS_MAX_VISUALS` defaults to `32`
  333. - `AGENT_PROJECTS_DIR`
  334. - `FFMPEG_PATH`
  335. - `ENCODE_VALIDATION_TIMEOUT_SEC`
  336. - `BLENDER_PATH`
  337. - `BLENDER_RENDER_TIMEOUT_SEC` defaults to one hour; set it to `0` to disable the Blender process timeout
  338. - `HYPERFRAMES_RENDER_TIMEOUT_SEC` defaults to `0`, which disables the Hyperframes process timeout; set a positive value to enforce one
  339. - `HYPERFRAMES_VARIANT_COUNT` controls legacy/manual art-direction variants, capped at `5`
  340. - `HYPERFRAMES_PROOF_CANDIDATE_COUNT` controls automatic structural proof candidates, capped at `8`
  341. - `HYPERFRAMES_MIN_QUALITY_SCORE` controls variant promotion
  342. - `HYPERFRAMES_QA_MODE` accepts `local`, `hybrid`, or `vision`
  343. - `HYPERFRAMES_ENABLE_VISION_QA` enables blind inverse-decoder QA
  344. - `HYPERFRAMES_ENABLE_COUNTERFACTUAL_QA` enables relation-ablation and temporal-scramble probes
  345. - `HYPERFRAMES_BLIND_DECODER_MIN_SCORE` sets the independent semantic decode floor
  346. - `HYPERFRAMES_VISION_MODEL` optionally selects the model used for blind decoding
  347. - `HYPERFRAMES_ENABLE_CEGIS` enables typed counterexample-guided visual repair
  348. - `HYPERFRAMES_MAX_CRITIC_FRAMES` controls adaptive semantic captures, default `8`, capped at `12`
  349. - `HYPERFRAMES_MAX_REPAIR_ROUNDS` controls monotonic repair attempts, default `3`, capped at `5`
  350. - `HYPERFRAMES_MIN_REPAIR_DELTA` sets the minimum composite improvement when hard failures are unchanged
  351. - `WHISPER_MODEL`
  352. ## Quick Start
  353. ### First run
  354. ```bash
  355. vex
  356. ```
  357. Then type a normal sentence with a video path:
  358. ```text
  359. Vex > trim the first 30 seconds of "D:\videos\clip.mp4"
  360. ```
  361. Then continue naturally:
  362. ```text
  363. Vex > remove the silent pauses
  364. Vex > add "DealScout" at the bottom for the first 4 seconds
  365. Vex > export it for youtube
  366. Vex > /quit
  367. ```
  368. ### If a project already exists
  369. - If exactly one saved project exists, Vex resumes it automatically
  370. - If multiple saved projects exist, Vex starts clean and waits for a file path or an explicit project command
  371. - If you mention a video path that already belongs to a saved project, Vex reuses that project instead of creating a duplicate
  372. ## Vex Studio
  373. Vex also ships with a local web app for the same project and agent architecture:
  374. ```bash
  375. vex web
  376. ```
  377. Open [http://127.0.0.1:5173](http://127.0.0.1:5173) in a browser. To open it automatically, use:
  378. ```bash
  379. vex web --open
  380. ```
  381. Studio v1 provides local video upload or local-path project creation, a working-cut preview backed by the project file on disk, plain-English editing through the existing intent compiler and agent loop, live plan/tool trace updates, project metadata, timeline history, creative runs, and a project library.
  382. The Studio server binds to `127.0.0.1` by default and does not add a cloud service or database. Because the API can open local media paths, Vex deliberately rejects non-loopback interfaces. Use `--host` only to select another loopback address such as `localhost` or `::1`, and use `--port` to choose a different local port.
  383. Uploads are streamed into a private temporary file and then copied into the project workspace; the browser never needs to hold the complete video in memory. Studio also resumes active task progress after a refresh and rejects cross-site API writes.
  384. ## Natural-Language Examples
  385. ### Basic trim
  386. ```text
  387. Vex > trim the first 15 seconds of "D:\videos\intro.mp4"
  388. ```
  389. ### Speed up a section
  390. ```text
  391. Vex > speed up the section from 1:10 to 1:35 by 1.25x
  392. ```
  393. ### Remove pauses
  394. ```text
  395. Vex > remove the silent gaps
  396. ```
  397. ### Add subtitles
  398. ```text
  399. Vex > transcribe this video
  400. Vex > burn subtitles in yellow at the bottom
  401. Vex > add cinematic subtitles
  402. Vex > add creator bold captions for shorts
  403. ```
  404. ### Add a song or soundtrack
  405. ```text
  406. Vex > add background music assets/beat.mp3
  407. Vex > add song assets/intro.wav as intro and outro music
  408. Vex > add assets/theme.mp3 from 00:12 to 00:28 at 20% volume
  409. Vex > replace the soundtrack with assets/final-track.m4a
  410. ```
  411. `add_song` uses Music Director v1 instead of a raw fixed-ratio audio mix. It
  412. classifies the request into a song-mix skill, trims or loops the song to the
  413. video or selected segment, adds fades, ducks music under existing speech when
  414. needed, normalizes loudness, renders a deterministic FFmpeg filter graph, and
  415. promotes the result only after audio QA passes. Each run writes a
  416. `song_mix_bundles/.../manifest.json`, `mix_plan.json`, `filtergraph.txt`,
  417. `audio_qa.json`, and notes file so the mix can be audited and replayed.
  418. ### Summarize a long video
  419. ```text
  420. Vex > make this a 60 second highlight reel
  421. ```
  422. ### Create viral-style shorts
  423. ```text
  424. Vex > turn this podcast into 4 YouTube Shorts with captions
  425. ```
  426. Auto shorts can build either one continuous clip or a stitched edit plan from multiple source ranges. Vex keeps the timeline and FFmpeg render path in control; the Shorts Director chooses typed source ranges, hard cuts, punch-ins, caption emphasis, and contextual visual insert intent, then validates the plan before rendering.
  427. ### Create shorts from a YouTube link
  428. ```text
  429. Vex > make 3 shorts from https://www.youtube.com/watch?v=example123
  430. ```
  431. ### Add stock B-roll automatically
  432. ```text
  433. Vex > add auto b-roll from Pexels to this video
  434. Vex > add auto b-roll using Pixabay and Coverr
  435. Vex > add 4 stock cutaways that match the narration
  436. ```
  437. `add_auto_broll` uses B-roll Director v2. It builds a video-level context graph, turns transcript beats into typed visual intents, searches all configured stock providers by default, verifies candidate visual fit, and runs a final abruptness/semantic QA gate before rendering. Set one or more of `PEXELS_API_KEY`, `PIXABAY_API_KEY`, and `COVERR_API_KEY`, then optionally control provider order with `AUTO_BROLL_PROVIDERS=auto` or a comma-separated list such as `pexels,pixabay`. Vex normalizes candidates from each provider, reranks them against the active subtitle and nearby transcript context, caches the chosen clip locally, and writes `stock_attribution.md` in the run bundle.
  438. ### Add generated visuals automatically
  439. ```text
  440. Vex > add precise generated visuals wherever the explanation needs them
  441. Vex > create custom animations for the key claims and process steps in this video
  442. Vex > use clean product-style generated visuals for the UI explanations
  443. ```
  444. `add_auto_visuals` now uses a transcript-aware planner, a video-level visual narrative program, context-aware visual budgeting, source-frame visual-need scoring, renderer auto-selection, and a signed Open Visual Program authoring tournament shared by HyperFrames and Remotion. Templates remain deterministic fallbacks instead of limiting automatic composition. If the transcript has many high-signal visual opportunities, Vex can plan a denser sequence instead of stopping after one or two inserts. Today it can choose between:
  445. - `hyperframes` for premium HTML/CSS motion slides, diagrams, flows, comparisons, UI explainers, causal chains, flywheels, decision matrices, anatomy cutaways, rankings, proof sequences, narrative arcs, concept maps, problem/solution pivots, myth-busters, checklists, radar scans, opportunity maps, scorecards, pipeline X-rays, decision trees, momentum waves, focus rings, filmstrip timelines, quote breakdowns, market maps, mechanism blueprints, data pulses, and data-heavy visual inserts
  446. - `remotion` for signed SceneGraph v2 visual programs, specialized SVG/DOM
  447. primitives, rendered six-treatment search, frame-driven data explanations,
  448. spatial metaphors, UI callouts, timelines, comparisons, and process
  449. explainers. Final inserts use ProRes HQ; transparent overlays use ProRes 4444.
  450. - `manim` for formula-heavy math, geometry, axes, and specialist vector animation
  451. - `ffmpeg` for fast, clean editorial cards and picture-in-picture support graphics
  452. - `blender` for deterministic 3D titles, transparent overlays, object/model shots, logo reveals, product spins, and cinematic data/abstract inserts when Blender is installed
  453. Suggested prompts:
  454. ```text
  455. Vex > add auto visuals
  456. Vex > add auto visuals with hyperframes
  457. Vex > add auto visuals with manim
  458. Vex > add auto visuals with remotion
  459. Vex > add auto visuals with both hyperframes and manim
  460. Vex > add custom animations only where they make the explanation clearer
  461. Vex > use generated visuals for the process beats and keep everything else clean
  462. Vex > add a rotating 3D title saying Matrix Multiplication at 00:18
  463. Vex > add a floating 3D arrow pointing to the chart on the right from 00:12 to 00:16
  464. Vex > add a cinematic glowing chip animation when I say GPU
  465. Vex > add a 3D product spin overlay using assets/model.glb from 5s to 9s
  466. ```
  467. Blender 3D support is opt-in and typed. The agent can choose a template and parameters, but Vex compiles those parameters into internal deterministic Blender Python. It never asks the LLM to emit raw Blender scripts.
  468. Supported Blender templates:
  469. - `three_d_title`
  470. - `floating_3d_label`
  471. - `object_orbit`
  472. - `logo_reveal`
  473. - `screen_pointer_3d`
  474. - `data_tunnel`
  475. - `product_model_spin`
  476. - legacy Blender templates: `quote_focus`, `keyword_stack`, `metric_callout`
  477. Composition modes:
  478. - `replace` renders a normal full-screen MP4 insert that Vex cuts into the timeline.
  479. - `overlay` renders transparent Blender frames, encodes an alpha-capable asset, and lets Vex composite it with FFmpeg over the source video.
  480. Supported local 3D model formats are `.glb`, `.gltf`, `.obj`, and `.blend`. Asset paths must stay inside the project or workspace-safe input roots. Blender remains optional; if it is unavailable, non-3D auto visuals continue to use Hyperframes, Remotion, Manim, or FFmpeg as appropriate.
  481. Hyperframes tuning:
  482. - Plain `add auto visuals` in the agent asks you to choose `hyperframes`, `manim`, `remotion`, or `both` before rendering.
  483. - `renderer=hyperframes` is strict Hyperframes-only and will not fall back to Manim.
  484. - `renderer=manim` is strict Manim-only and will not fall back to Hyperframes.
  485. - `renderer=remotion` is strict Remotion-only and will not fall back to Hyperframes or Manim.
  486. - `renderer=both` lets Vex choose between Hyperframes and Manim per visual.
  487. - Auto Visuals Director v3 samples tiny source frames, scores whether the moment actually needs a generated insert, rejects weak semantic matches, and drops rendered Hyperframes/Remotion/Manim outputs that fail renderer QA before compositing.
  488. - Auto Visuals uses deterministic set optimization before and after rendering, so stronger and more diverse candidates can replace weaker or overlapping candidates.
  489. - Flexible `auto` and `both` runs use bounded renderer quality tournaments; strict renderer requests remain strict.
  490. - Auto Visuals and Auto B-roll run shared encoded-composite QA before state mutation.
  491. - `AUTO_VISUALS_RENDERER_TOURNAMENT_SIZE` controls successful contenders per flexible visual, default `2`, capped at `3`.
  492. - `VISUAL_COMPOSITE_SIMILARITY_FLOOR` controls replacement-frame publish similarity, default `0.72`.
  493. - Full-screen generated replacements receive soft transition handles automatically so the compositor avoids abrupt slide cuts when the plan did not specify transitions.
  494. - `HYPERFRAMES_PROOF_CANDIDATE_COUNT` controls how many structurally distinct proof programs Auto Visuals renders, capped at 8
  495. - `HYPERFRAMES_VARIANT_COUNT` remains the legacy/manual art-direction variant control
  496. - `HYPERFRAMES_MIN_QUALITY_SCORE` sets the promotion threshold used by extracted-frame QA
  497. - `HYPERFRAMES_QA_MODE` records whether the run should be treated as `local`, `hybrid`, or `vision` QA
  498. - `HYPERFRAMES_ENABLE_COUNTERFACTUAL_QA` tests whether relation geometry and time order are necessary for comprehension
  499. - `HYPERFRAMES_BLIND_DECODER_MIN_SCORE` sets the blind semantic recovery threshold
  500. - `HYPERFRAMES_RENDER_QUALITY` can be set to `draft`, `standard`, `high`, or left empty for the Hyperframes default
  501. - `HYPERFRAMES_RENDER_TIMEOUT_SEC` defaults to `0`, which disables the Hyperframes render timeout
  502. - `REMOTION_RENDER_TIMEOUT_SEC` defaults to `0`, which lets Vex use Remotion's internal timeout while disabling the outer process timeout
  503. - `REMOTION_RENDER_CONCURRENCY` optionally forwards a Remotion render concurrency value such as `1`, `2`, or `50%`
  504. - `OPEN_VISUAL_PROGRAM_LLM_AUTHORING` enables evidence-bound model scene-graph authoring; invalid output automatically falls back to the deterministic compiler
  505. - `OPEN_VISUAL_PROGRAM_CANDIDATES` controls the shared concept tournament size, default `6`, capped at `6`
  506. - `OPEN_VISUAL_PROGRAM_AUTHORING_ATTEMPTS` controls the initial authoring call plus one validation-aware retry, capped at `2`
  507. - `OPEN_VISUAL_PROGRAM_MIN_SCORE` sets the pre-render grounding, motion, novelty, and semantic-fitness acceptance floor
  508. - `VISUAL_DIRECTOR_ENABLED` enables the final-render Visual Director boundary for HyperFrames and Remotion
  509. - `VISUAL_DIRECTOR_VERIFICATION_MODE` is `strict`, `balanced`, or `off`; balanced outages can publish only an explicitly degraded candidate above the local floor
  510. - `VISUAL_DIRECTOR_VISION_MODEL` and `VISUAL_DIRECTOR_CLAUDE_VISION_MODEL` select independent Gemini and Claude vision models
  511. - `VISUAL_DIRECTOR_RENDER_CANDIDATES` controls bounded full-render concept search, default `2`, capped at `3`
  512. - `VISUAL_DIRECTOR_MAX_REPAIR_ROUNDS` controls counterexample-guided re-renders, default `2`, capped at `4`
  513. - `VISUAL_DIRECTOR_MIN_REPAIR_DELTA` sets the minimum monotonic score improvement, default `0.025`
  514. - `VISUAL_DIRECTOR_PAIRWISE_TOP_K` controls the blind pairwise finalist count, default `3`, capped at `4`
  515. - `VISUAL_DIRECTOR_VERIFIER_RETRIES`, `VISUAL_DIRECTOR_CIRCUIT_FAILURES`, and `VISUAL_DIRECTOR_CIRCUIT_COOLDOWN_SEC` bound provider reliability behavior
  516. See [HyperFrames Visual Proof Search](docs/hyperframes-visual-proof-search.md) for the
  517. claim graph, structural tournament, blind inverse decoder, counterfactual QA, and
  518. promotion architecture.
  519. See [HyperFrames Counterexample-Guided Visual Synthesis](docs/hyperframes-visual-cegis.md)
  520. for Scene Program V2, typed critics, safe patches, monotonic repair, and independent
  521. final judging.
  522. ### Add subtitle-aware effects automatically
  523. ```text
  524. Vex > add subtitle-aware auto effects
  525. Vex > add strong zoom effects where the captions need emphasis
  526. Vex > add subtle camera-only effects, no style accents
  527. ```
  528. `add_auto_effects` treats subtitle/caption timing as the source of truth. It scores each subtitle beat for hooks, questions, numbers, contrast turns, payoff language, pauses, and nearby scene cuts, then sends the selected beats through Auto Effects Director. The director chooses a taste profile, converts effects into a bounded camera-motion plan, softens unsafe zooms/shakes/flashes, validates the motion path, renders one deterministic FFmpeg pass, and samples tiny rendered preview frames before accepting the output. Effects include `punch_in`, `punch_out`, `slow_push`, `micro_pan`, `snap_reframe`, `impact_pulse`, `freeze_accent`, `subtle_shake`, `vignette`, `flash_accent`, `focus_blur`, and `subtitle_highlight`.
  529. Taste profiles can be requested with `taste_profile` / `effect_taste_profile` when calling the tool, or `--taste-profile` in the CLI: `clean_documentary`, `viral_commentary`, `tutorial_focus`, `cinematic_subtle`, or `high_energy_shorts`. If omitted, Vex infers one from rhythm, density, intensity, duration, and aspect ratio.
  530. ### Export for social
  531. ```text
  532. Vex > export it for instagram
  533. ```
  534. ### Auto color grade
  535. ```text
  536. Vex > auto color grade this video
  537. Vex > give it a subtle cinematic look
  538. Vex > make the colors pop
  539. ```
  540. ## Full Tool Surface
  541. These are the editing tools Vex exposes to the agent loop.
  542. | Tool | What it does |
  543. |---|---|
  544. | `get_video_info` | Reads duration, resolution, fps, codec, audio presence, format, and size |
  545. | `trim_clip` | Trims the current working video to a selected range |
  546. | `merge_clips` | Concatenates the current working clip with one or more external clips |
  547. | `adjust_speed` | Changes playback speed globally or for a selected segment |
  548. | `add_transition` | Adds `fade_in`, `fade_out`, or fade-through-black style transitions |
  549. | `add_text_overlay` | Adds timed text overlays to the video |
  550. | `extract_audio` | Exports audio from the current working video |
  551. | `replace_audio` | Replaces or mixes audio with an external track |
  552. | `add_song` | Adds a song through Music Director v1 with skill routing, loop/trim placement, fades, speech ducking, loudness normalization, manifests, audio QA, and transactional promotion |
  553. | `mute_segment` | Silences audio in a selected time range |
  554. | `trim_silence` | Detects and removes dead-air pauses while preserving natural speech handles by default |
  555. | `auto_color_grade` | Samples frames, builds a Color Grade Director plan with subject protection and scene smoothing, applies a reusable FFmpeg grade, stores the filter for rebuilds, and records creative grade QA |
  556. | `burn_subtitles` | Compiles SRT captions into styled ASS subtitles and burns them into the video |
  557. | `transcribe_video` | Generates `transcript.txt` and `transcript.srt` using Whisper |
  558. | `summarize_clip` | Uses transcript-aware LLM selection to build a shorter highlight cut |
  559. | `create_auto_shorts` | Builds Shorts Director v4 programs from semantic transcript units, plans complete stories chapter by chapter, deterministically compiles exact source ranges, runs cold-viewer and render QA, and promotes prevalidated reserves when a primary short fails |
  560. | `add_auto_broll` | Uses B-roll Director v2 to plan typed, graph-aware stock cutaway intents, searches configured providers, verifies visual fit, runs final QA, writes attribution, and splices approved clips into the current working video |
  561. | `add_auto_visuals` | Compiles transcript evidence into signed semantic render contracts, rejects unsupported filler, generates custom visuals with the best supported renderer, records semantic/render QA, and composites approved assets |
  562. | `add_visual_asset` | Inserts a project-safe local HTML, video, GIF, or image asset at an exact time without transcript scoring |
  563. | `add_auto_effects` | Scores subtitle beats and applies replayable camera and style emphasis effects in a single FFmpeg pass |
  564. | `upscale_video` | Scales and exports with FFmpeg Lanczos using `fit`, `fill`, or explicit `stretch` |
  565. | `renderers_doctor` | Reports HyperFrames, Node.js, FFmpeg, Manim, Remotion, and Blender availability and versions |
  566. | `plan_encode` | Turns plain-English encode, conversion, and compression requests into a pending FFmpeg command |
  567. | `run_pending_encode` | Executes the latest confirmed encode plan after the user replies `yes` |
  568. | `export_video` | Exports the working video with a named preset |
  569. | `undo` | Rebuilds the project without the last operation |
  570. | `redo` | Reapplies the most recently undone operation |
  571. Inside the REPL, you can also use `/trace` to inspect the latest recorded agent trace for the current project.
  572. Use `/creative-runs` to inspect recent local creative automation runs for the active project.
  573. ## CLI Commands
  574. Vex supports both a conversational mode and explicit power-user commands.
  575. ### `vex`
  576. Start the interactive REPL.
  577. ### `vex web`
  578. Start the loopback-only Vex Studio web app. The default address is `http://127.0.0.1:5173`.
  579. ```bash
  580. vex web --open
  581. vex web --port 5174
  582. ```
  583. ### `vex shorts`
  584. Generate a packaged shorts bundle directly from an existing project.
  585. ```bash
  586. vex shorts --project <project-id> --count 4 --target-platform youtube_shorts
  587. ```
  588. Add `--subtitle-style glass` or another subtitle preset to change the burned caption look for the generated shorts.
  589. ### `vex auto-broll`
  590. Plan and apply provider-backed stock footage inserts to an existing project.
  591. ```bash
  592. vex auto-broll --project <project-id> --max-overlays 5
  593. vex auto-broll --project <project-id> --providers pixabay,coverr --max-overlays 5
  594. vex auto-broll --project <project-id> --max-overlays 15 --coverage-policy target_count
  595. ```
  596. ### `vex auto-visuals`
  597. Plan and apply generated supporting visuals to an existing project.
  598. ```bash
  599. vex auto-visuals --project <project-id> --max-visuals 4 --renderer auto --style-pack editorial_clean
  600. vex auto-visuals --project <project-id> --renderer hyperframes
  601. vex auto-visuals --project <project-id> --renderer remotion
  602. vex auto-visuals --project <project-id> --renderer both
  603. vex auto-visuals --project <project-id> --max-visuals 18 --coverage-policy target_count --density chapter_coverage
  604. ```
  605. Generic requests use `quality_only`. Explicit counts use `target_count`; `exact_count`
  606. raises coverage pressure but still cannot bypass path, render, semantic, or QA failures.
  607. ### `vex generate-video`
  608. Generate a new HyperFrames video without a source clip. Vex writes an audio-first native-motion project with a narration script, TTS audio, timed beat graph, director crew contract, video-generation skill graph, per-beat HyperFrames compositions, motion cues, transitions, captions, portfolio QA report, manifest, and final render.
  609. The generator now uses a deterministic Video Generation Skill Graph before
  610. cinematography. It classifies the whole video into a production skill, assigns
  611. every beat a source-grounded visual skill, seeds HyperFrames semantic frames,
  612. steers proof tournaments and native motion, writes `VIDEO_SKILL_GRAPH.json`, and
  613. makes skill coverage part of portfolio and generated-video QA.
  614. ```bash
  615. vex generate-video "explain retrieval augmented generation in 30 seconds" --duration 30
  616. vex generate-video "show why KV cache compression matters" --aspect portrait --voice af_nova
  617. vex generate-video "show sparse attention becoming a focused reasoning path" --quality high --fps 60 --render-resolution 4k
  618. vex generate-video "turn this into a project only" --no-render --no-audio --output-dir D:\renders\vex-tests
  619. ```
  620. The generated project is stored under `~/.video-agent/generated_videos` unless `--output-dir` is provided.
  621. Generated videos use a Director Crew pass before rendering. The director repairs
  622. weak prompt-only scripts, writes `DIRECTOR_CREW.json`, assigns each beat a
  623. source-grounded visual contract, then the skill graph converts those contracts
  624. into explicit beat skills, renderer routes, proof encodings, visual-world
  625. priors, continuity subjects, and motion grammar. Compatible HyperFrames variants
  626. enter a per-beat tournament. The native motion compiler writes
  627. `MOTION_PLAN.json`, `motion_cues.json`, template-backed `compositions/*.html`,
  628. root inline native beat surfaces with traceable `data-external-composition-src`
  629. artifact links, and rendered-frame QA. A portfolio judge writes
  630. `PORTFOLIO_JUDGE.json` and can reject the whole project when the result is too
  631. generic, repetitive, weakly grounded, or below skill-coverage policy.
  632. ### `vex add_visual_asset`
  633. Insert an exact-time local asset without transcript scoring:
  634. ```bash
  635. vex add_visual_asset --project <project-id> --asset assets/overlay.mp4 --start 12 --end 16 --mode overlay
  636. vex add_visual_asset --project <project-id> --asset assets/visual.html --start 72 --end 78 --mode replace
  637. ```
  638. Local HTML cannot reference remote URLs or Node/shell execution hooks.
  639. ### `vex upscale_video`
  640. ```bash
  641. vex upscale_video --project <project-id> --resolution 1920x1080 --scale-mode fit
  642. ```
  643. This is FFmpeg Lanczos resize/export, not AI super-resolution.
  644. ### `vex renderers doctor`
  645. ```bash
  646. vex renderers doctor
  647. ```
  648. Install the shared managed HyperFrames and Remotion runtime:
  649. ```bash
  650. vex renderers install hyperframes
  651. vex renderers install remotion
  652. ```
  653. ### `vex auto-effects`
  654. Plan and apply context-aware emphasis effects to an existing project. The planner reads the whole transcript timeline, local pacing, scene stability, blocked overlay ranges, and subtitle placement before selecting restrained camera/style moves.
  655. ```bash
  656. vex auto-effects --project <project-id> --density medium --intensity high --max-effects 12
  657. ```
  658. ### `vex creative-runs`
  659. Inspect recent local creative automation runs recorded for a project.
  660. ```bash
  661. vex creative-runs --project <project-id>
  662. ```
  663. ### `vex color-grade`
  664. Analyze and apply an automatic color grade to an existing project.
  665. ```bash
  666. vex color-grade --project <project-id> --look cinematic --intensity 0.8
  667. ```
  668. ### `vex youtube-shorts`
  669. Download a YouTube video and immediately run the auto shorts workflow.
  670. ```bash
  671. vex youtube-shorts "https://www.youtube.com/watch?v=example123" --count 4
  672. ```
  673. - resumes the only saved project automatically
  674. - otherwise waits for natural-language input
  675. - if no video is loaded, include a file path in your message
  676. ### `vex start <video_path> [--name TEXT]`
  677. Create a new project explicitly and open the REPL.
  678. ```bash
  679. vex start "D:\videos\clip.mp4" --name "Launch Cut"
  680. ```
  681. ### `vex repl [--project TEXT]`
  682. Open the REPL for an existing project.
  683. ```bash
  684. vex repl
  685. vex repl --project 7e5a4d1c
  686. ```
  687. ### `vex run "<instruction>" --project TEXT`
  688. Run a single instruction against a saved project and exit.
  689. ```bash
  690. vex run "export it for instagram" --project 7e5a4d1c
  691. ```
  692. ### `vex projects`
  693. List saved projects.
  694. ### `vex export <preset_name> --project TEXT [--output TEXT]`
  695. Export without entering the REPL.
  696. ```bash
  697. vex export instagram_reels --project 7e5a4d1c --output "D:\exports\clip.mp4"
  698. ```
  699. ### `vex encode "<request>" --project TEXT [--output TEXT] [--yes]`
  700. Plan a metadata-aware FFmpeg encode from plain English. By default Vex prints the command and stores it as pending; pass `--yes` to run it immediately.
  701. ```bash
  702. vex encode "convert this MOV to MP4 and compress it without losing much quality" --project 7e5a4d1c
  703. ```
  704. After the FFmpeg command finishes, Vex probes the output, checks the expected container, streams, codecs, dimensions, duration, and target-size behavior, then runs a full decode smoke test. If validation finds a fatal issue, the encode is reported as failed instead of silently accepting a bad file.
  705. ### `vex --version`
  706. Show the installed version.
  707. ## REPL Slash Commands
  708. These commands work only inside the interactive session.
  709. | Command | Action |
  710. |---|---|
  711. | `/status` | Show the current project summary |
  712. | `/timeline` | Show the applied timeline |
  713. | `/undo` | Undo the last edit |
  714. | `/redo` | Redo the last undone edit |
  715. | `/export <preset>` | Export immediately with a preset |
  716. | `/encode <request>` | Plan an encode/conversion/compression command and wait for confirmation |
  717. | `/color-grade [look]` | Apply an automatic color grade with an optional look |
  718. | `/provider` | Show the active provider and model |
  719. | `/projects` | List saved projects |
  720. | `/help` | Show available slash commands |
  721. | `/quit` or `/exit` | Save and exit |
  722. ## Export Presets
  723. Built-in presets:
  724. | Preset | Description | Format |
  725. |---|---|---|
  726. | `youtube_1080p` | YouTube 1080p HD | `mp4` |
  727. | `youtube_4k` | YouTube 4K UHD | `mp4` |
  728. | `instagram_reels` | Instagram Reels / Stories vertical | `mp4` |
  729. | `instagram_square` | Instagram square feed | `mp4` |
  730. | `tiktok` | TikTok vertical | `mp4` |
  731. | `twitter_x` | X / Twitter landscape | `mp4` |
  732. | `podcast_audio` | Audio-only podcast export | `mp3` |
  733. | `custom` | Start from your own settings | variable |
  734. ## How Project Loading Works
  735. - Vex never edits the original source file directly
  736. - each project stores:
  737. - original source path
  738. - working copy path
  739. - timeline operations
  740. - provider and model
  741. - session log
  742. - metadata
  743. - when you mention a video path in the REPL:
  744. - Vex checks for an existing project for that exact source file
  745. - if found, it loads that project
  746. - otherwise, it creates a new project automatically
  747. Default project storage:
  748. ```text
  749. ~/.video-agent/projects/
  750. ```
  751. You can override that with `AGENT_PROJECTS_DIR`.
  752. ## Architecture
  753. | Path | Responsibility |
  754. |---|---|
  755. | `main.py` | CLI, REPL, auto-loading, slash commands, and live terminal status UI |
  756. | `agent.py` | Provider-agnostic agent loop and tool orchestration |
  757. | `providers/` | Gemini, Claude, and OpenAI-compatible local provider adapters behind one interface |
  758. | `tools/` | Agent-callable editing tools |
  759. | `video_generation/` | Standalone audio-first HyperFrames video generation, beat graphs, project writing, rendering, and QA |
  760. | `effects/` | Context-aware auto-effects modeling, scoring, planning, FFmpeg compilation, and validation |
  761. | `engine.py` | FFmpeg and MoviePy operations |
  762. | `color_grading.py` | Sampled-frame color analysis and reusable FFmpeg grade planning |
  763. | `state.py` | Persistent project state and timeline history |
  764. | `visual_intelligence.py` | Transcript beat mining, visual planning, and renderer-aware spec normalization |
  765. | `renderers/` | Generated-visual backends for Hyperframes, Remotion, Manim, FFmpeg, and optional Blender |
  766. | `vex_hyperframes/` | Hyperframes design IR, art directions, composition building, production rules, variants, QA, validation, and skill slices |
  767. | `vex_manim/` | Manim scene briefs, blueprinting, runtime helpers, validation, and QA |
  768. | `vex_runtime/` | Canonical version, packaged configuration, and managed HyperFrames/Remotion runtime installation |
  769. | `presets/export_presets.json` | Built-in export presets |
  770. For the larger production architecture roadmap, see [docs/architecture-upgrades.md](docs/architecture-upgrades.md).
  771. For release-system design and operator controls, see
  772. [docs/release-architecture-report.md](docs/release-architecture-report.md) and
  773. [docs/releasing.md](docs/releasing.md).
  774. ## Dependencies and Runtime Notes
  775. ### FFmpeg is mandatory
  776. Vex depends on FFmpeg for:
  777. - metadata probing
  778. - trims and merges
  779. - audio processing
  780. - subtitle burn-in
  781. - silence detection
  782. - exports
  783. ### Whisper is optional
  784. `transcribe_video` requires `openai-whisper` and a local environment capable of running it. Install it with:
  785. ```bash
  786. vex setup transcription
  787. ```
  788. Installing Whisper into a separate system Python does not expose it to a Vex
  789. installation managed by pipx directly. Vex bridges that boundary through a
  790. bounded external worker when the interpreter is discoverable on `PATH` or
  791. configured with `WHISPER_PYTHON_PATH`.
  792. ### Text overlays on Windows may require ImageMagick
  793. MoviePy text rendering on Windows can require ImageMagick.
  794. If timed text overlays fail, install it from:
  795. ```text
  796. https://imagemagick.org/script/download.php#windows
  797. ```
  798. ### Subtitle burning depends on FFmpeg ASS subtitle support
  799. Vex compiles SRT captions into styled ASS files, then burns them with FFmpeg's `ass` filter. Most full FFmpeg builds include libass support. If yours does not, subtitle burn-in may fail until you install a build with subtitle filter support.
  800. ## Troubleshooting
  801. ### `vex` is not recognized
  802. Add your Python Scripts directory to `PATH`, then restart the terminal.
  803. ### `FFmpeg was not found in PATH`
  804. Install FFmpeg and verify `ffmpeg` works in the terminal before launching Vex.
  805. ### Vex says no video is loaded
  806. Start with a normal sentence that includes a file path:
  807. ```text
  808. Vex > trim the first 10 seconds of "D:\videos\clip.mp4"
  809. ```
  810. ### Subtitle burn-in fails
  811. Check that:
  812. - your `transcript.srt` exists
  813. - your FFmpeg build supports the `ass` filter / libass
  814. - the subtitle path is valid on your OS
  815. ### Text overlay fails on Windows
  816. Install ImageMagick and retry.
  817. ### Transcription fails
  818. Run `vex setup transcription`. Vex will reuse a discoverable system Python
  819. that already has Whisper, or install Whisper into its own environment when no
  820. compatible external runtime exists. Set `WHISPER_PYTHON_PATH` to select a
  821. specific interpreter.
  822. ### Summarization does not work
  823. Make sure:
  824. - the active provider API key is configured
  825. - transcription completed successfully
  826. - `transcript.txt` and `transcript.srt` exist in the project working directory
  827. ## Contributing
  828. Issues and PRs are welcome.
  829. If you report a bug, include:
  830. - OS
  831. - Python version
  832. - active provider
  833. - the exact command or REPL input
  834. - the traceback or terminal output
  835. - whether the issue happens on a fresh project or a resumed project
  836. ## License
  837. PolyForm Noncommercial License 1.0.0. Commercial use is not permitted under this license. For commercial licensing inquiries, contact aaryankakad1@gmail.com. See [LICENSE](LICENSE).
  838. Plain English: You may use Vex for personal, educational, research, and other non-commercial projects for free. You may not use Vex to build paid products, paid services, commercial SaaS, internal business tooling, or revenue-generating workflows without a separate commercial license.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.