Skip to content
Back to search

ClipTalk

ClipTalk —— 通过对话完成视频剪辑的 AI Agent。说句话,片子就剪好了。ClipTalk — An AI agent that edits videos through conversation. Just say it, it's edited.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/gml-mmgroup--cliptalk/badge.svg)](/listings/gml-mmgroup--cliptalk)

HTML

<a href="/listings/gml-mmgroup--cliptalk"><img src="/api/listings/gml-mmgroup--cliptalk/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Prefer AI-assisted setup? Give this repository to Codex, Claude Code, or Gemini and ask it to follow the [AI installation guide](docs/AI_INSTALL.md); Codex discovers the root [AGENTS.md](AGENTS.md) automatically.

Requires Linux x86_64 (or WSL2), Python 3.10–3.11, FFmpeg/ffprobe, curl, and a CJK font. On Debian/Ubuntu:

sudo apt-get update
sudo apt-get install -y ffmpeg curl fonts-wqy-zenhei

Install the complete CPU runtime and start ClipTalk:

git clone https://github.com/GML-MMGroup/ClipTalk.git
cd ClipTalk

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements-cpu.txt

# Recommended before the first person/content recognition task
python3 tools/prepare_recognition_models.py --data-root data
python3 tools/doctor.py --profile cpu

./start.sh

For an NVIDIA host with a compatible driver, install `requirements-gpu.txt` instead of `requirements-cpu.txt`, then run `python3 tools/doctor.py --profile cuda`.

Models

* **VLM · Required** — understands frames, finds events, and refines shot boundaries. * **LLM · Optional** — plans shot selection, ordering, and alternative edits; it can reuse the VLM. * **Local multimodal stack · Included** — SenseVoice, OCR, visual/audio embeddings, and anonymous-person recognition; model weights are prepared locally or downloaded on first use.

Open the URL printed in the terminal. In **Settings**, configure the VLM and optionally a separate LLM, then upload a video and describe the edit you want.

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 196 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. <!-- 👇 Replace this with your cover banner -->
  3. <img src="./assets/banner.png" alt="ClipTalk Banner" width="100%" />
  4. # ClipTalk ✂️
  5. ### An AI Agent That Edits Videos Through Conversation
  6. **Just say it. It's edited.**
  7. **English** · [简体中文](./README_zh.md)
  8. </div>
  9. ---
  10. ## 📰 News
  11. * **[2026-09-11]** 🛠️ Added guided local setup, a standalone Agent service, and Docker Compose deployment.
  12. * **[2026-09-10]** 🎤 Integrated **TalkNet active-speaker detection** to distinguish speaking from visual presence, with automatic CPU/GPU selection and explicit fallback notices.
  13. * **[2026-09-09]** 🎨 Redesigned the landscape and portrait editing workspaces, improving preview layouts, timeline space usage, and light/dark theme support across aspect ratios.
  14. * **[2026-09-08]** 🤖 Added a reviewable conversational editing Agent that presents its editing plan before executing controlled tools and delivering previewable, revisable versions.
  15. * **[2026-08-28]** 🎬 Upgraded the **multimodal editing workspace** with reusable content-search results, person/speaker workflows, voiceprint-based clipping, and an editable fine-cut timeline; also streamlined CPU/GPU deployment.
  16. * **[2026-08-20]** 🎯 Added **Face-Matched Editing** and **Topic-Based Editing**, enabling target-person retrieval and topic-driven clip extraction.
  17. * **[2026-08-12]** 🚀 Optimized the timeline display and added support for switching to **vertical creation mode**.
  18. * **[2026-07-20]** ✨ Added the **Highlight Editing Agent** for automatic highlight clip extraction.
  19. * **[2026-07-10]** 🎉 ClipTalk is now **open-source**!
  20. <!-- Keep adding future updates here -->
  21. ---
  22. ## 🗺️ Roadmap
  23. ClipTalk is actively evolving toward a more powerful conversational video editing experience.
  24. * [x] 💬 **Conversational Editing Infrastructure**
  25. * [x] ⚡ **Highlight Editing Agent**
  26. * [x] 👤 **Face-Matched Editing** — Find a target person and extract the segments where they appear on screen.
  27. * [x] 🔊 **Voiceprint-Based Editing** — Identify a target speaker and extract the segments where they are speaking.
  28. * [x] 🧭 **Topic-Based Editing** — Understand the content and extract segments around a specific topic.
  29. * [x] 🎞️ **Editable Timeline** — Further edit and fine-tune AI-generated results directly on the timeline.
  30. ---
  31. ## 🌟 Showcase — One Sentence, One Edit
  32. > Real editing tasks completed by ClipTalk with a single instruction.
  33. >
  34. > <!-- Add GIF comparisons here: source video → instruction → result -->
  35. <table>
  36. <tr>
  37. <td align="center"><b>📰 News &amp; Broadcast Highlights</b><br/><img src="./assets/cases/news-broadcast.gif" width="240"/></td>
  38. <td align="center"><b>🧵 DIY &amp; Craft Tutorials</b><br/><img src="./assets/cases/diy-craft.gif" width="240"/></td>
  39. <td align="center"><b>📦 Product Demos</b><br/><img src="./assets/cases/product.gif" width="240"/></td>
  40. </tr>
  41. <tr>
  42. <td align="center"><b>🏢 Lifestyle Vlogs</b><br/><img src="./assets/cases/meeting.gif" width="240"/></td>
  43. <td align="center"><b>🔥 Live PK Reaction Highlights</b><br/><img src="./assets/cases/music.gif" alt="AI-edited live PK reaction highlights" width="240"/></td>
  44. <td align="center"><b>⚽ Sports Highlights</b><br/><img src="./assets/cases/sports.gif" width="240"/></td>
  45. </tr>
  46. </table>
  47. ---
  48. ## 💡 What is ClipTalk?
  49. ClipTalk is an **AI video-editing agent**. You don't drag clips on a timeline or scrub through hours of footage — you just describe what you want in natural language, and the agent handles the entire process:
  50. **understanding the footage → locating the target content → planning the edit → executing cuts → delivering the final clips.**
  51. <div align="center">
  52. <img src="./assets/showcase/conversational-highlight-editing-preview.gif" alt="ClipTalk conversational highlight editing workflow" width="900" />
  53. <br />
  54. <sub><b>“Make a highlight from the best moments.”</b> — ClipTalk analyzes the footage, presents the event timeline, and delivers multiple AI-edited versions.</sub>
  55. </div>
  56. <br />
  57. > Upload a 1-hour video, describe the moments you want, and let ClipTalk understand the footage, plan the edit, and deliver the finished clips.
  58. ---
  59. ## ✨ Core Features
  60. ### 💬 Conversational Editing
  61. Edit videos simply by describing what you want.
  62. From *“make a highlight of the best moments”* to *“cut out the part where they introduce Product X”*, ClipTalk turns natural-language instructions directly into editing actions.
  63. You can also refine the result through follow-up instructions such as *“make it shorter”* or *“start from the part about pricing”*.
  64. ### 🎯 Four Core Editing Capabilities
  65. * **⚡ Highlight Extraction** — Automatically identify and extract the most valuable moments from long-form footage to create concise highlight clips.
  66. * **👤 Face-Matched Editing** — Find a target person in the video and extract the segments where that person appears on screen.
  67. * **🧭 Topic-Based Editing** — Locate and extract clips around a specific topic, such as *“cut out the parts where they explain Product X.”*
  68. * **🔊 Voiceprint-Based Editing** — Identify a specific speaker by voiceprint and extract the segments where that person is speaking.
  69. ---
  70. ## 🚀 Quick Start
  71. Prefer AI-assisted setup? Give this repository to Codex, Claude Code, or Gemini and ask it to follow the [AI installation guide](docs/AI_INSTALL.md); Codex discovers the root [AGENTS.md](AGENTS.md) automatically.
  72. Requires Linux x86_64 (or WSL2), Python 3.10–3.11, FFmpeg/ffprobe, curl, and a CJK font. On Debian/Ubuntu:
  73. ```bash
  74. sudo apt-get update
  75. sudo apt-get install -y ffmpeg curl fonts-wqy-zenhei
  76. ```
  77. Install the complete CPU runtime and start ClipTalk:
  78. ```bash
  79. git clone https://github.com/GML-MMGroup/ClipTalk.git
  80. cd ClipTalk
  81. python3 -m venv .venv
  82. source .venv/bin/activate
  83. python3 -m pip install --upgrade pip
  84. python3 -m pip install -r requirements-cpu.txt
  85. # Recommended before the first person/content recognition task
  86. python3 tools/prepare_recognition_models.py --data-root data
  87. python3 tools/doctor.py --profile cpu
  88. ./start.sh
  89. ```
  90. For an NVIDIA host with a compatible driver, install `requirements-gpu.txt` instead of `requirements-cpu.txt`, then run `python3 tools/doctor.py --profile cuda`.
  91. ### Models
  92. * **VLM · Required** — understands frames, finds events, and refines shot boundaries.
  93. * **LLM · Optional** — plans shot selection, ordering, and alternative edits; it can reuse the VLM.
  94. * **Local multimodal stack · Included** — SenseVoice, OCR, visual/audio embeddings, and anonymous-person recognition; model weights are prepared locally or downloaded on first use.
  95. Open the URL printed in the terminal. In **Settings**, configure the VLM and optionally a separate LLM, then upload a video and describe the edit you want.
  96. ---
  97. ## ⚙️ Deployment
  98. ### 🐳 Docker
  99. ```bash
  100. git clone https://github.com/GML-MMGroup/ClipTalk.git
  101. cd ClipTalk
  102. cp .env.example .env
  103. docker compose up --build
  104. ```
  105. Open the URL printed in the terminal, then configure the vision and planning models in **Settings**.
  106. ### 🔧 Optional setup
  107. - **NVIDIA GPU:** install `requirements-gpu.txt` for a native environment. For Docker, use `docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build` with NVIDIA Container Toolkit installed.
  108. - **Local models:** SenseVoice weights download on first use; run `python3 tools/prepare_recognition_models.py --data-root data` to prepare person/content recognition assets.
  109. - **Environment variables:** see [`.env.example`](./.env.example). Never commit `.env` or API keys.
  110. <details>
  111. <summary><strong>Remote deployment and security</strong></summary>
  112. For remote access, set `HIGHLIGHT_HOST=0.0.0.0` for a native install or `CLIPTALK_BIND_ADDRESS=0.0.0.0` for Docker. Configure a strong `HIGHLIGHT_ACCESS_TOKEN`, allow the selected port, and use an authenticated HTTPS reverse proxy for internet-facing deployments.
  113. </details>
  114. ---
  115. ## 🤝 Contributing
  116. Pull Requests are always welcome. Contribute code, new features, bug fixes, or other improvements to AdCraft and become a project contributor.
  117. ## 💬 Contact
  118. For questions, feedback, collaboration, or other inquiries, feel free to contact us:
  119. Ma Fei — mafei@gml.ac.cn
  120. Xu Hongbo — xuhongbo@gml.ac.cn
  121. <div align="center">
  122. ⭐ If you find ClipTalk useful, please give us a star!
  123. Made with ❤️ by GML-MMGroup
  124. </div>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.