ClipTalk
ClipTalk —— 通过对话完成视频剪辑的 AI Agent。说句话,片子就剪好了。ClipTalk — An AI agent that edits videos through conversation. Just say it, it's edited.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/gml-mmgroup--cliptalk)HTML
<a href="/listings/gml-mmgroup--cliptalk"><img src="/api/listings/gml-mmgroup--cliptalk/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Prefer AI-assisted setup? Give this repository to Codex, Claude Code, or Gemini and ask it to follow the [AI installation guide](docs/AI_INSTALL.md); Codex discovers the root [AGENTS.md](AGENTS.md) automatically.
Requires Linux x86_64 (or WSL2), Python 3.10–3.11, FFmpeg/ffprobe, curl, and a CJK font. On Debian/Ubuntu:
sudo apt-get update sudo apt-get install -y ffmpeg curl fonts-wqy-zenhei
Install the complete CPU runtime and start ClipTalk:
git clone https://github.com/GML-MMGroup/ClipTalk.git cd ClipTalk python3 -m venv .venv source .venv/bin/activate python3 -m pip install --upgrade pip python3 -m pip install -r requirements-cpu.txt # Recommended before the first person/content recognition task python3 tools/prepare_recognition_models.py --data-root data python3 tools/doctor.py --profile cpu ./start.sh
For an NVIDIA host with a compatible driver, install `requirements-gpu.txt` instead of `requirements-cpu.txt`, then run `python3 tools/doctor.py --profile cuda`.
Models
* **VLM · Required** — understands frames, finds events, and refines shot boundaries. * **LLM · Optional** — plans shot selection, ordering, and alternative edits; it can reuse the VLM. * **Local multimodal stack · Included** — SenseVoice, OCR, visual/audio embeddings, and anonymous-person recognition; model weights are prepared locally or downloaded on first use.
Open the URL printed in the terminal. In **Settings**, configure the VLM and optionally a separate LLM, then upload a video and describe the edit you want.
---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 196 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <div align="center">
- <!-- 👇 Replace this with your cover banner -->
- <img src="./assets/banner.png" alt="ClipTalk Banner" width="100%" />
- # ClipTalk ✂️
- ### An AI Agent That Edits Videos Through Conversation
- **Just say it. It's edited.**
- **English** · [简体中文](./README_zh.md)
- </div>
- ---
- ## 📰 News
- * **[2026-09-11]** 🛠️ Added guided local setup, a standalone Agent service, and Docker Compose deployment.
- * **[2026-09-10]** 🎤 Integrated **TalkNet active-speaker detection** to distinguish speaking from visual presence, with automatic CPU/GPU selection and explicit fallback notices.
- * **[2026-09-09]** 🎨 Redesigned the landscape and portrait editing workspaces, improving preview layouts, timeline space usage, and light/dark theme support across aspect ratios.
- * **[2026-09-08]** 🤖 Added a reviewable conversational editing Agent that presents its editing plan before executing controlled tools and delivering previewable, revisable versions.
- * **[2026-08-28]** 🎬 Upgraded the **multimodal editing workspace** with reusable content-search results, person/speaker workflows, voiceprint-based clipping, and an editable fine-cut timeline; also streamlined CPU/GPU deployment.
- * **[2026-08-20]** 🎯 Added **Face-Matched Editing** and **Topic-Based Editing**, enabling target-person retrieval and topic-driven clip extraction.
- * **[2026-08-12]** 🚀 Optimized the timeline display and added support for switching to **vertical creation mode**.
- * **[2026-07-20]** ✨ Added the **Highlight Editing Agent** for automatic highlight clip extraction.
- * **[2026-07-10]** 🎉 ClipTalk is now **open-source**!
- <!-- Keep adding future updates here -->
- ---
- ## 🗺️ Roadmap
- ClipTalk is actively evolving toward a more powerful conversational video editing experience.
- * [x] 💬 **Conversational Editing Infrastructure**
- * [x] ⚡ **Highlight Editing Agent**
- * [x] 👤 **Face-Matched Editing** — Find a target person and extract the segments where they appear on screen.
- * [x] 🔊 **Voiceprint-Based Editing** — Identify a target speaker and extract the segments where they are speaking.
- * [x] 🧭 **Topic-Based Editing** — Understand the content and extract segments around a specific topic.
- * [x] 🎞️ **Editable Timeline** — Further edit and fine-tune AI-generated results directly on the timeline.
- ---
- ## 🌟 Showcase — One Sentence, One Edit
- > Real editing tasks completed by ClipTalk with a single instruction.
- >
- > <!-- Add GIF comparisons here: source video → instruction → result -->
- <table>
- <tr>
- <td align="center"><b>📰 News & Broadcast Highlights</b><br/><img src="./assets/cases/news-broadcast.gif" width="240"/></td>
- <td align="center"><b>🧵 DIY & Craft Tutorials</b><br/><img src="./assets/cases/diy-craft.gif" width="240"/></td>
- <td align="center"><b>📦 Product Demos</b><br/><img src="./assets/cases/product.gif" width="240"/></td>
- </tr>
- <tr>
- <td align="center"><b>🏢 Lifestyle Vlogs</b><br/><img src="./assets/cases/meeting.gif" width="240"/></td>
- <td align="center"><b>🔥 Live PK Reaction Highlights</b><br/><img src="./assets/cases/music.gif" alt="AI-edited live PK reaction highlights" width="240"/></td>
- <td align="center"><b>⚽ Sports Highlights</b><br/><img src="./assets/cases/sports.gif" width="240"/></td>
- </tr>
- </table>
- ---
- ## 💡 What is ClipTalk?
- ClipTalk is an **AI video-editing agent**. You don't drag clips on a timeline or scrub through hours of footage — you just describe what you want in natural language, and the agent handles the entire process:
- **understanding the footage → locating the target content → planning the edit → executing cuts → delivering the final clips.**
- <div align="center">
- <img src="./assets/showcase/conversational-highlight-editing-preview.gif" alt="ClipTalk conversational highlight editing workflow" width="900" />
- <br />
- <sub><b>“Make a highlight from the best moments.”</b> — ClipTalk analyzes the footage, presents the event timeline, and delivers multiple AI-edited versions.</sub>
- </div>
- <br />
- > Upload a 1-hour video, describe the moments you want, and let ClipTalk understand the footage, plan the edit, and deliver the finished clips.
- ---
- ## ✨ Core Features
- ### 💬 Conversational Editing
- Edit videos simply by describing what you want.
- From *“make a highlight of the best moments”* to *“cut out the part where they introduce Product X”*, ClipTalk turns natural-language instructions directly into editing actions.
- You can also refine the result through follow-up instructions such as *“make it shorter”* or *“start from the part about pricing”*.
- ### 🎯 Four Core Editing Capabilities
- * **⚡ Highlight Extraction** — Automatically identify and extract the most valuable moments from long-form footage to create concise highlight clips.
- * **👤 Face-Matched Editing** — Find a target person in the video and extract the segments where that person appears on screen.
- * **🧭 Topic-Based Editing** — Locate and extract clips around a specific topic, such as *“cut out the parts where they explain Product X.”*
- * **🔊 Voiceprint-Based Editing** — Identify a specific speaker by voiceprint and extract the segments where that person is speaking.
- ---
- ## 🚀 Quick Start
- Prefer AI-assisted setup? Give this repository to Codex, Claude Code, or Gemini and ask it to follow the [AI installation guide](docs/AI_INSTALL.md); Codex discovers the root [AGENTS.md](AGENTS.md) automatically.
- Requires Linux x86_64 (or WSL2), Python 3.10–3.11, FFmpeg/ffprobe, curl, and a CJK font. On Debian/Ubuntu:
- ```bash
- sudo apt-get update
- sudo apt-get install -y ffmpeg curl fonts-wqy-zenhei
- ```
- Install the complete CPU runtime and start ClipTalk:
- ```bash
- git clone https://github.com/GML-MMGroup/ClipTalk.git
- cd ClipTalk
- python3 -m venv .venv
- source .venv/bin/activate
- python3 -m pip install --upgrade pip
- python3 -m pip install -r requirements-cpu.txt
- # Recommended before the first person/content recognition task
- python3 tools/prepare_recognition_models.py --data-root data
- python3 tools/doctor.py --profile cpu
- ./start.sh
- ```
- For an NVIDIA host with a compatible driver, install `requirements-gpu.txt` instead of `requirements-cpu.txt`, then run `python3 tools/doctor.py --profile cuda`.
- ### Models
- * **VLM · Required** — understands frames, finds events, and refines shot boundaries.
- * **LLM · Optional** — plans shot selection, ordering, and alternative edits; it can reuse the VLM.
- * **Local multimodal stack · Included** — SenseVoice, OCR, visual/audio embeddings, and anonymous-person recognition; model weights are prepared locally or downloaded on first use.
- Open the URL printed in the terminal. In **Settings**, configure the VLM and optionally a separate LLM, then upload a video and describe the edit you want.
- ---
- ## ⚙️ Deployment
- ### 🐳 Docker
- ```bash
- git clone https://github.com/GML-MMGroup/ClipTalk.git
- cd ClipTalk
- cp .env.example .env
- docker compose up --build
- ```
- Open the URL printed in the terminal, then configure the vision and planning models in **Settings**.
- ### 🔧 Optional setup
- - **NVIDIA GPU:** install `requirements-gpu.txt` for a native environment. For Docker, use `docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build` with NVIDIA Container Toolkit installed.
- - **Local models:** SenseVoice weights download on first use; run `python3 tools/prepare_recognition_models.py --data-root data` to prepare person/content recognition assets.
- - **Environment variables:** see [`.env.example`](./.env.example). Never commit `.env` or API keys.
- <details>
- <summary><strong>Remote deployment and security</strong></summary>
- For remote access, set `HIGHLIGHT_HOST=0.0.0.0` for a native install or `CLIPTALK_BIND_ADDRESS=0.0.0.0` for Docker. Configure a strong `HIGHLIGHT_ACCESS_TOKEN`, allow the selected port, and use an authenticated HTTPS reverse proxy for internet-facing deployments.
- </details>
- ---
- ## 🤝 Contributing
- Pull Requests are always welcome. Contribute code, new features, bug fixes, or other improvements to AdCraft and become a project contributor.
- ## 💬 Contact
- For questions, feedback, collaboration, or other inquiries, feel free to contact us:
- Ma Fei — mafei@gml.ac.cn
- Xu Hongbo — xuhongbo@gml.ac.cn
- <div align="center">
- ⭐ If you find ClipTalk useful, please give us a star!
- Made with ❤️ by GML-MMGroup
- </div>
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.