Skip to content
Back to search

py-xiaozhi

Open-source AI assistant ecosystem with MCP integrations, multimodal workflows, IoT support, and cross-platform voice interaction.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/huangjunsen0406--py-xiaozhi/badge.svg)](/listings/huangjunsen0406--py-xiaozhi)

HTML

<a href="/listings/huangjunsen0406--py-xiaozhi"><img src="/api/listings/huangjunsen0406--py-xiaozhi/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 313 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # py-xiaozhi
  2. <p align="center" class="trendshift">
  3. <a href="https://trendshift.io/repositories/14130" target="_blank">
  4. <img src="https://trendshift.io/api/badge/repositories/14130" alt="Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
  5. </a>
  6. </p>
  7. <p align="center">
  8. <a href="https://github.com/huangjunsen0406/py-xiaozhi/releases/latest">
  9. <img src="https://img.shields.io/github/v/release/huangjunsen0406/py-xiaozhi?style=flat-square&logo=github&color=blue" alt="Release"/>
  10. </a>
  11. <a href="https://opensource.org/licenses/MIT">
  12. <img src="https://img.shields.io/badge/License-MIT-green.svg?style=flat-square" alt="License: MIT"/>
  13. </a>
  14. <a href="https://github.com/huangjunsen0406/py-xiaozhi/stargazers">
  15. <img src="https://img.shields.io/github/stars/huangjunsen0406/py-xiaozhi?style=flat-square&logo=github" alt="Stars"/>
  16. </a>
  17. <a href="https://github.com/huangjunsen0406/py-xiaozhi/releases/latest">
  18. <img src="https://img.shields.io/github/downloads/huangjunsen0406/py-xiaozhi/total?style=flat-square&logo=github&color=52c41a1&maxAge=86400" alt="Download"/>
  19. </a>
  20. <a href="https://gitee.com/huang-jun-sen/py-xiaozhi">
  21. <img src="https://img.shields.io/badge/Gitee-FF5722?style=flat-square&logo=gitee" alt="Gitee"/>
  22. </a>
  23. <a href="https://huangjunsen0406.github.io/py-xiaozhi/guide/00_%E6%96%87%E6%A1%A3%E7%9B%AE%E5%BD%95.html">
  24. <img alt="Usage Docs" src="https://img.shields.io/badge/Usage Docs-View-blue?labelColor=2d2d2d" />
  25. </a>
  26. <a href="https://atomgit.com/huangjunsen0406/py-xiaozhi">
  27. <img src="./assets/AtomGit.svg" alt="AtomGit" height="20"/>
  28. </a>
  29. </p>
  30. English | [简体中文](README.zh.md)
  31. ## ❤️ Sponsors
  32. > [Want to appear here?](mailto:95134130@qq.com)
  33. <details open>
  34. <summary>Click to collapse</summary>
  35. [**GitDo.net**](https://gitdo.net/)
  36. Thanks to GitDo.net for sponsoring this project! GitDo.net is an AI API aggregation platform — one API Key for **Claude, Gemini, GPT** and other major models. Direct connection, no proxy needed, stable and efficient. An ideal choice for enterprise-grade AI programming. **[Visit GitDo.net](https://gitdo.net/)**
  37. ---
  38. | | |
  39. | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  40. | [**Token能量站**](https://factory.pub/register?aff=3rXc) | Thanks to Token能量站 (Factory.pub) for sponsoring this project! Providing API relay for **GPT, Grok, Claude** and other major models — stable, reliable, and affordable. **[Register here](https://factory.pub/register?aff=3rXc)** |
  41. </details>
  42. ## About
  43. py-xiaozhi is a lightweight, cross-platform multi-modal AI interaction framework built on Python's async architecture. It supports real-time voice streaming, vision-language tasks, and IoT device control. Deployable across Windows, macOS, Linux desktops, and ARM embedded platforms (Raspberry Pi, Horizon Robotics RDK, Jetson Nano), it bridges the gap between Large Language Models and physical hardware — out of the box.
  44. > Evolved from the [xiaozhi-esp32](https://github.com/78/xiaozhi-esp32) firmware project. Officially adopted by [D-Robotics (xiaozhi-in-rdk)](https://github.com/D-Robotics/xiaozhi-in-rdk) as an upstream dependency.
  45. ## Related Projects
  46. - [xiaozhi-desktop](https://xiaozhi.junsen.online) — Electron desktop client with AEC echo cancellation, Live2D, floating window modes, and Windows / macOS installers
  47. ## Demo
  48. - [Bilibili Demo Video](https://www.bilibili.com/video/BV1HmPjeSED2/#reply255921347937)
  49. ![Image](./documents/docs/guide/images/系统界面.png)
  50. ## Key Features
  51. - **Real-time Voice AI** — Opus codec with auto frame detection (RFC 6716 TOC parsing), async streaming, sub-20ms latency
  52. - **Multi-modal Vision** — Camera capture + vision-language model integration for image understanding and scene perception
  53. - **MCP Tool Ecosystem** — Modular JSON-RPC 2.0 tool server: music player, camera, screenshot, app management, weather, volume control
  54. - **Cross-platform Deployment** — Windows 10+ / macOS 10.15+ / Linux (x86_64 & ARM), optimized for Raspberry Pi and edge boards
  55. - **Multiple UI Modes** — PySide6 + QML GUI / TUI / GPIO, adapting to desktop, headless server, and embedded environments
  56. - **Offline Wake Word** — Sherpa-ONNX based on-device keyword spotting with custom wake word support
  57. - **IoT & Embodied AI Ready** — GPIO interface for robotics control, hardware actuation, and sensor integration
  58. - **WebSocket / MQTT** — Dual protocol communication with WSS/TLS encrypted transmission and auto-reconnection
  59. - **Plugin Architecture** — Event-driven async design, clean dependency injection, extensible plugin system
  60. ## System Requirements
  61. ### Basic Requirements
  62. - **Python Version**: 3.10 - 3.12
  63. - **Operating System**: Windows 10+, macOS 10.15+, Linux
  64. - **Audio Devices**: Microphone and speaker devices
  65. - **Network Connection**: Stable internet connection (for AI services and online features)
  66. ### Recommended Configuration
  67. - **Memory**: At least 4GB RAM (8GB+ recommended)
  68. - **Processor**: Modern CPU with AVX instruction set support
  69. - **Storage**: At least 2GB available disk space (for model files and cache)
  70. - **Audio**: Audio devices supporting 16kHz sampling rate
  71. ### Optional Feature Requirements
  72. - **Voice Wake-up**: Requires downloading Sherpa-ONNX speech recognition models
  73. - **Camera Features**: Requires camera device and OpenCV support
  74. ## Read This First
  75. - Carefully read [项目文档](https://huangjunsen0406.github.io/py-xiaozhi/) for startup tutorials and file descriptions
  76. - The main branch has the latest code; manually reinstall pip dependencies after each update to ensure you have new dependencies
  77. [Zero to Xiaozhi Client (Video Tutorial)](https://www.bilibili.com/video/BV1dWQhYEEmq/?vd_source=2065ec11f7577e7107a55bbdc3d12fce)
  78. ## Technical Architecture
  79. ### Core Architecture Design
  80. - **Event-Driven Architecture**: Based on asyncio asynchronous event loop, supporting high-concurrency processing
  81. - **Layered Design**: Clear separation of application layer, protocol layer, and UI layer
  82. - **Dependency Injection**: Component lifecycle managed via bootstrap container
  83. - **Plugin System**: Audio, UI, MCP tools and other components loaded via plugin system
  84. ### Device States
  85. `StateManager` owns `IDLE`, `LISTENING`, `SPEAKING` and `NOTIFYING`;
  86. `ConversationSession` coordinates transitions. A successful wake/button action enters
  87. `LISTENING` after the protocol connects and accepts the listen request. TTS moves the
  88. device to `SPEAKING`; once local playback drains, the session resumes `LISTENING` or
  89. returns to `IDLE`. Server notifications play in `NOTIFYING` while otherwise idle,
  90. then return to `IDLE`. Connection establishment belongs to the protocol lifecycle,
  91. not a separate device state.
  92. ### Key Technical Components
  93. - **Audio Processing**: Opus codec, real-time resampling
  94. - **Speech Recognition**: Sherpa-ONNX offline models, wake word recognition
  95. - **Protocol Communication**: WebSocket/MQTT dual protocol support, encrypted transmission, auto-reconnection
  96. - **Configuration System**: Hierarchical configuration, dot notation access, dynamic updates
  97. ### Performance Optimization
  98. - **Async First**: Full system asynchronous architecture, avoiding blocking operations
  99. - **Memory Management**: Smart caching, garbage collection
  100. - **Audio Optimization**: 5ms low-latency processing, queue management, streaming transmission
  101. - **Concurrency Control**: Task pool management, semaphore control, thread safety
  102. ### Security Mechanisms
  103. - **Encrypted Communication**: WSS/TLS encryption, certificate verification
  104. - **Device Authentication**: Dual protocol activation, device fingerprint recognition
  105. - **Access Control**: Tool permission management, API access control
  106. - **Error Isolation**: Exception isolation, fault recovery, graceful degradation
  107. ## Development Guide
  108. ### Project Structure
  109. ```
  110. py-xiaozhi/
  111. ├── main.py # Application entry point
  112. ├── src/
  113. │ ├── activation/ # Device activation
  114. │ ├── audio_codecs/ # Audio codecs
  115. │ ├── audio_processing/ # Wake word detection
  116. │ ├── bootstrap/ # Application bootstrap & dependency injection
  117. │ ├── constants/ # Constants
  118. │ ├── core/ # Core infrastructure (event bus, state management, task management, etc.)
  119. │ ├── logging/ # Logging subsystem
  120. │ ├── mcp/ # MCP tool system
  121. │ │ ├── mcp_server.py # MCP server
  122. │ │ └── tools/ # Tool modules (music/camera/screenshot/app/weather/volume)
  123. │ ├── plugins/ # Plugin system (audio, UI, MCP, wake word, shortcuts)
  124. │ ├── protocols/ # Communication protocols (WebSocket/MQTT)
  125. │ ├── ui/ # User interface
  126. │ │ ├── gui/ # PySide6 + QML graphical interface
  127. │ │ ├── tui/ # Textual full-screen terminal interface
  128. │ │ └── gpio/ # GPIO embedded interface
  129. │ └── utils/ # Utility functions
  130. ├── libs/ # Third-party native libraries
  131. │ ├── libopus/ # Opus audio codec library
  132. │ └── webrtc_apm/ # WebRTC audio processing module
  133. ├── models/ # Wake word models
  134. ├── assets/ # Static resources
  135. ├── scripts/ # Auxiliary scripts
  136. ├── documents/ # VitePress documentation site
  137. ├── pyproject.toml # Project configuration
  138. └── build.json # Build configuration
  139. ```
  140. ### Development Environment Setup
  141. ```bash
  142. # Clone project
  143. git clone https://github.com/huangjunsen0406/py-xiaozhi.git
  144. cd py-xiaozhi
  145. # Base install (GPIO mode on Linux)
  146. uv sync # Recommended (uv users)
  147. # or: pip install -e . # pip users
  148. # TUI mode (extra: Textual)
  149. uv sync --extra tui
  150. # or: pip install -e '.[tui]'
  151. # GUI mode (extra: PySide6 + qasync)
  152. uv sync --extra gui # Recommended (uv users)
  153. # or: pip install -e '.[gui]' # pip users
  154. # Full development environment (GUI + TUI + test / packaging tools)
  155. uv sync --extra gui --extra tui --group dev
  156. # Code formatting
  157. ./format_code.sh
  158. # Run program - GUI mode (default; requires gui extra)
  159. python main.py
  160. # Run program - TUI mode (requires tui extra)
  161. python main.py --mode tui
  162. # Specify communication protocol
  163. python main.py --protocol websocket # WebSocket (default)
  164. python main.py --protocol mqtt # MQTT protocol
  165. ```
  166. ### Entry points and verification
  167. Source checkouts, installed wheels and frozen builds share `src.cli.main(argv=None)`.
  168. Importing the launcher does not parse arguments, initialize configuration/logging,
  169. or load Qt/audio. `--help` and invalid arguments exit before runtime initialization.
  170. ```bash
  171. # Installed command (GUI still needs the gui extra)
  172. xiaozhi --mode tui
  173. # Equivalent module entry point
  174. python -m src --mode tui
  175. # Full regression suite, including Qt model tests without a display
  176. uv run --extra gui --extra tui python -m pytest -q -W error::RuntimeWarning
  177. # Build sdist, then build the wheel from that sdist
  178. uvx --from build pyproject-build
  179. # Install the wheel into a temporary venv outside the checkout and verify it
  180. python scripts/check_wheel.py dist/*.whl
  181. ```
  182. The wheel contains assets, wake-word models and the supplied native libraries under
  183. `src/_resources`; source and PyInstaller layouts keep their existing resource paths.
  184. FFmpeg is resolved from the system PATH for wheel installs. The wheel check installs
  185. only `platformdirs` alongside the wheel to verify entry points and resources; it does
  186. not exercise audio hardware, GUI rendering or the full runtime dependency set.
  187. PRs and branch pushes run Python tests on 3.10/3.12 and installed-wheel checks on
  188. Linux, macOS and Windows. Lint gates critical syntax/undefined-name rules across the
  189. project and all configured rules for the new entry-point/verification code. Existing
  190. style warnings elsewhere remain a separate cleanup task.
  191. ### Core Development Patterns
  192. - **Async First**: Use `async/await` syntax, avoid blocking operations
  193. - **Error Handling**: Complete exception handling and logging
  194. - **Configuration Management**: Use `ConfigManager` for unified configuration access
  195. - **Test-Driven**: Write unit tests to ensure code quality
  196. ### Extension Development
  197. - **Add MCP Tools**: Create new tool modules in `src/mcp/tools/` directory
  198. - **Add Protocols**: Implement `Protocol` abstract base class
  199. - **Add Plugins**: Extend the plugin system via `src/plugins/`
  200. ### State Transition Diagram
  201. ```text
  202. IDLE -- wake/button + protocol ready --> LISTENING
  203. LISTENING -- TTS starts --> SPEAKING
  204. SPEAKING -- playback drained --> LISTENING (continue) / IDLE (finish)
  205. IDLE -- server notification --> NOTIFYING -- playback drained --> IDLE
  206. ```
  207. ## Contributing
  208. - Start with [CONTRIBUTING.md](./CONTRIBUTING.md) for the repository workflow
  209. - Chinese version: [CONTRIBUTING_ZH.md](./CONTRIBUTING_ZH.md)
  210. - Detailed docs: [Contribution Guide](https://huangjunsen0406.github.io/py-xiaozhi/en/contributing)
  211. ## Maintainer Workflow
  212. - Triage incoming work as `bug`, `feature`, `docs`, `refactor`, or `maintenance`
  213. - Prefer focused pull requests with clear validation steps and linked context
  214. - Require docs updates when behavior, configuration, or public APIs change
  215. - Merge after CI passes and review feedback is resolved
  216. - Release through the normal release flow; merge does not imply immediate shipping
  217. ## Community and Support
  218. ### Thanks to the Following Open Source Contributors
  219. >
  220. > In no particular order
  221. [Xiaoxia](https://github.com/78)
  222. [zhh827](https://github.com/zhh827)
  223. [SmartArduino-Li Honggang](https://github.com/SmartArduino)
  224. [HonestQiao](https://github.com/HonestQiao)
  225. [vonweller](https://github.com/vonweller)
  226. [Sun Weigong](https://space.bilibili.com/416954647)
  227. [isamu2025](https://github.com/isamu2025)
  228. [Rain120](https://github.com/Rain120)
  229. [kejily](https://github.com/kejily)
  230. [Radio bilibili Jun](https://space.bilibili.com/119751)
  231. [Cyber Intelligence](https://shop115087494.m.taobao.com/?refer=https%3A%2F%2Fm.tb.cn%2F&ut_sk=1.WMelxbgDQWkDAJ1Rq9Pn7DCD_21380790_1757337352472.Copy.shop&suid=0E25E948-651D-46E0-8E89-5C8CB03B4F56&shop_navi=shopindex&sourceType=shop&shareUniqueId=33038752403&un=d22c5ceda82844ab8bd7bab98ffeb263&share_crt_v=1&un_site=0&spm=a2159r.13376460.0.0&sp_tk=dkRKUjRKUWo2ZHY%3D&bc_fl_src=share-1041250486811064-2-1&cpp=1&shareurl=true&short_name=h.SaBKVHytsCKIPNS&bxsign=scdGtSe264e_qkFQBh0rXCkF-Mrb_s6t35EnpVBBU5dsrd-J24c-_rn_PhJiXRk0hg2hjGoAm0L7j2UQg27OIH_6gZkbhKDyLziD2cy4pDf8sC3KmqrF55TXP3USZaPTw_-&app=weixin)
  232. ## Project Statistics
  233. [![Star History Chart](https://star-history.dera.page/svg?repos=huangjunsen0406/py-xiaozhi&type=Date)](https://star-history.dera.page/#huangjunsen0406/py-xiaozhi&Date)
  234. ## License
  235. [MIT License](LICENSE)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.