Skip to content
Back to search

telemem

TeleMem is a high-performance drop-in replacement for Mem0, featuring semantic deduplication, long-term dialogue memory, and multimodal video reasoning.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/TeleAI-UAGI/telemem with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/TeleAI-UAGI/telemem'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/teleai-uagi--telemem/badge.svg)](/listings/teleai-uagi--telemem)

HTML

<a href="/listings/teleai-uagi--telemem"><img src="/api/listings/teleai-uagi--telemem/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Installation

pip install telemem            # core (text memory)
pip install "telemem[mcp]"     # + MCP server
pip install "telemem[video]"   # + video/multimodal pipeline
pip install "telemem[all]"     # everything

Development Environment

Using [uv](https://docs.astral.sh/uv/) (recommended — creates `.venv` from the committed `uv.lock` for a reproducible environment):

uv sync --all-extras   # install TeleMem (editable) + all extras, incl. MCP
uv run python examples/quickstart.py

Or with conda + pip:

# Create and activate virtual environment
conda create -n telemem python=3.10
conda activate telemem
# Install from source (editable), with the extras you need
pip install -e ".[all]"

Example

Set your OpenAI API key:

export OPENAI_API_KEY="your-openai-api-key"
# python examples/quickstart.py
import telemem as mem0

memory = mem0.Memory()

messages = [
    {"role": "user", "content": "Jordan, did you take the subway to work again today?"},
    {"role": "assistant", "content": "Yes, James. The subway is much faster than driving. I leave at 7 o'clock and it's just not crowded."},
    {"role": "user", "content": "Jordan, I want to try taking the subway too. Can you tell me which station is closest?"},
    {"role": "assistant", "content": "Of course, James. You take Line 2 to Civic Center Station, exit from Exit A, and walk 5 minutes to the company."}
]

memory.add(messages=messages, user_id="Jordan")
results = memory.search("What transportation did Jordan use to go to work today?", user_id="Jordan")
for hit in results["results"]:   # same result shape as mem0
    print(hit["memory"])

`Memory()` uses the default provider settings inherited from `mem0ai`. To use the repository's local Qwen + FAISS configuration, load `config/config.yaml` explicitly:

from telemem.utils import load_config
import telemem as mem0

config = load_config("config/config.yaml")
memory = mem0.Memory(config=config)

The runnable examples also honor the same configuration through `TELEMEM_CONFIG`:

TELEMEM_CONFIG=config/config.yaml python examples/quickstart.py

Using MiniMax as the LLM Provider

TeleMem supports [MiniMax](https://api.minimax.io) as an LLM backend via its OpenAI-compatible API. A ready-to-use example config is provided at `config/config.minimax.yaml`.

export MINIMAX_API_KEY="your-minimax-api-key"
export OPENAI_API_KEY="your-openai-api-key"  # still needed for embeddings
from telemem.utils import load_config
import telemem as mem0

config = load_config("config/config.minimax.yaml")
memory = mem0.Memory(config=config)

Key points for MiniMax usage: - **LLM**: MiniMax M3 (1M context, default) via `https://api.minimax.io/v1`; MiniMax M2.7 (204,800 context) is also available. MiniMax-M3 accepts text, image and video input and supports adaptive thinking; MiniMax-M2.7 is text-only with always-on thinking - **Regional endpoints**: use `https://api.minimax.io/v1` (global) or `https://api.minimaxi.com/v1` (China) as `openai_base_url` - **Temperature**: must be in **(0.0, 1.0]** — set explicitly (e.g. `0.7`) to avoid out-of-range errors - **Embeddings**: MiniMax does not provide a public embedding API; configure a separate embedder (e.g. `text-embedding-3-small`) in the `embedder` section

More LLM Providers

TeleMem works with **any OpenAI-compatible endpoint**. Ready-to-use config examples ship in `config/`:

| Provider | Config file | LLM | Embeddings | Notes | | -------- | ----------- | --- | ---------- | ----- | | **Ollama** (fully local) | [`config.ollama.yaml`](config/config.ollama.yaml) | any local model (e.g. `qwen3:8b`) | `nomic-embed-text`, local | **No API key, no cloud** — everything runs on your machine | | **DeepSeek** | [`config.deepseek.yaml`](config/config.deepseek.yaml) | `deepseek-chat` / `deepseek-reasoner` | external (e.g. OpenAI) | `DEEPSEEK_API_KEY` + `OPENAI_API_KEY` |

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 822 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <p align="center">
  2. <a href="https://github.com/TeleAI-UAGI/telemem">
  3. <img src="./assets/TeleMem.png" width="40%" />
  4. </a>
  5. </p>
  6. <h1 align="center"> TeleMem: Building Long-Term and Multimodal Memory for Agentic AI </h1>
  7. <p align="center">
  8. <a href="https://arxiv.org/abs/2601.06037">
  9. <img src="https://img.shields.io/badge/arXiv-Paper-red" alt="arXiv">
  10. </a>
  11. <a href="https://github.com/TeleAI-UAGI/telemem/actions/workflows/ci.yml">
  12. <img src="https://github.com/TeleAI-UAGI/telemem/actions/workflows/ci.yml/badge.svg" alt="CI">
  13. </a>
  14. <a href="https://pypi.org/project/telemem/">
  15. <img src="https://img.shields.io/pypi/v/telemem?color=blue" alt="PyPI">
  16. </a>
  17. <a href="https://github.com/TeleAI-UAGI/telemem">
  18. <img src="https://img.shields.io/github/stars/TeleAI-UAGI/TeleMem?style=social" alt="GitHub Stars">
  19. </a>
  20. <a href="https://github.com/TeleAI-UAGI/TeleMem/blob/main/LICENSE">
  21. <img src="https://img.shields.io/badge/license-Apache%20License%202.0-blue" alt="License: Apache 2.0">
  22. </a>
  23. <img src="https://img.shields.io/github/last-commit/TeleAI-UAGI/TeleMem?color=blue" alt="Last Commit">
  24. <img src="https://img.shields.io/badge/PRs-Welcome-red" alt="PRs Welcome">
  25. <a href="https://mcpvault.io/servers/telemem/health?utm_source=external_badge&utm_medium=referral&utm_campaign=mcp_health_report">
  26. <img src="https://mcpvault.io/badge/telemem.svg" alt="MCPVault: verified">
  27. </a>
  28. </p>
  29. <div align="center">
  30. **If you find this project helpful, please give us a ⭐️ on GitHub for the latest update.**
  31. _🤝 Contributions welcome! Feel free to open an issue or submit a pull request._
  32. </div>
  33. ---
  34. <div align="center">
  35. <p>
  36. <a href="README.md">English</a> | <a href="README-ZH.md">简体中文</a>
  37. </p>
  38. <p>
  39. <a href="https://github.com/TeleAI-UAGI/Awesome-Agent-Memory"> <strong>📄 Awesome-Agent-Memory →</strong></a>
  40. </p>
  41. </div>
  42. TeleMem is an agent memory management layer that can be used as <mark>**a high-performance drop-in replacement for [Mem0](https://mem0.ai/)** with one line of code (`import telemem as mem0`)</mark>, deeply optimized for complex scenarios involving **multi-turn dialogues**, **character modeling**, **long-term information storage**, and **semantic retrieval**.
  43. Through its unique **context-aware enhancement mechanism**, TeleMem provides conversational AI with core infrastructure offering **higher accuracy**, **faster performance**, and **stronger character memory capabilities**.
  44. Building upon this foundation, TeleMem implements **video understanding, multimodal reasoning, and visual question answering** capabilities. Through a complete pipeline of video frame extraction, caption generation, and vector database construction, AI Agents can effortlessly **store, retrieve, and reason over video content** just like handling text memories.
  45. The ultimate goal of the TeleMem project is to _use an agent's hindsight to improve its foresight_.
  46. **TeleMem, where memory lives on and intelligence grows strong.**
  47. ### Why TeleMem?
  48. - 🎭 **Character memory done right** — the only open-source memory layer that automatically builds **isolated, per-character memory profiles**, built for role-play, companion AI, NPCs, and multi-persona assistants.
  49. - 🎬 **Memory for video, not just text** — a full video → frames → captions → vector DB pipeline with **ReAct-style multi-step video QA**.
  50. - 🏠 **Fully local by default** — runs end-to-end on your hardware (Qwen + FAISS); no cloud service, no paid tier, no data leaving your machine.
  51. - 🔌 **mem0-compatible API** — `add()` / `search()` accept the same arguments and return the same `{"results": [...]}` shapes, so existing Mem0 code keeps working.
  52. ---
  53. ## 📢 Latest Updates
  54. - **[2026-08-15] 🎉 TeleMem [v1.10.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.10.0) adds first-class [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) support: an opt-in Cordis/MCP patch starts TeleMem with `uvx`, registers all 8 memory tools under `mcp__telemem__*`, and securely forwards provider configuration. See the [MCP server docs](docs/MCP.md#deepseek-harness).**
  55. - **[2026-08-06] 🎉 TeleMem [v1.9.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.9.0) is on the latest MCP standard: migrated to the official [MCP Python SDK v2](https://github.com/modelcontextprotocol/python-sdk) (spec 2026-07-28) — all 8 tools now declare titles, behavior annotations, and structured output, while staying compatible with older MCP clients. See the [MCP server docs](docs/MCP.md).**
  56. - **[2026-07-11] 🎉 TeleMem [v1.8.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.8.0) — the "claims = contracts" release: character-memory extraction fix, `infer=False`/`prompt`/`memory_type` now fully honored, offline contract test suite, telemetry disabled by default, and a [multi-NPC demo](examples/multi_npc.py)!**
  57. - **[2026-06-12] 🎉 TeleMem [v1.7.1](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.7.1) is live on the [official MCP registry](https://registry.modelcontextprotocol.io) — run the memory server with zero install: `uvx telemem`! Also new: [evaluation principles](https://teleai-uagi.github.io/telemem/evaluation/) and a LongMemEval harness with built-in baselines.**
  58. - **[2026-06-12] 🎉 TeleMem is now on PyPI: `pip install telemem`! [v1.6.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.6.0) adds Ollama/DeepSeek/Kimi configs, LangChain & LlamaIndex examples, and a [documentation site](https://teleai-uagi.github.io/telemem/).**
  59. - **[2026-06-12] 🎉 TeleMem [v1.5.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.5.0) has been released: true mem0 drop-in API, lightweight core install, and CI!**
  60. - **[2026-06-11] 🎉 TeleMem [v1.4.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.4.0) has been released with [MCP support](docs/MCP.md)!**
  61. - **[2026-01-28] 🎉 TeleMem [v1.3.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.3.0) has been released!**
  62. - **[2026-01-22] 🎉 TeleMem [Tech Report](https://arxiv.org/abs/2601.06037) has been updated to its 4th version!**
  63. - **[2026-01-13] 🎉 TeleMem [Tech Report](https://arxiv.org/abs/2601.06037) has been released on arXiv!**
  64. - **[2026-01-09] 🎉 TeleMem [v1.2.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.2.0) has been released!**
  65. - **[2025-12-31] 🎉 TeleMem [v1.1.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.1.0) has been released!**
  66. - **[2025-12-05] 🎉 TeleMem [v1.0.0](https://github.com/TeleAI-UAGI/telemem/releases/tag/v1.0.0) has been released!**
  67. ---
  68. ## 🔥 Research Highlights
  69. * **Significantly improved memory accuracy**: Achieved **86.33%** accuracy on the ZH-4O Chinese multi-character long-dialogue benchmark, **19% higher** than Mem0.
  70. * **Doubled speed performance**: Millisecond-level semantic retrieval enabled by efficient buffering and batch writing.
  71. * **Greatly reduced token cost**: Optimized token usage delivers the same performance with significantly lower LLM overhead.
  72. * **Precise character memory preservation**: Automatically builds independent memory profiles for each character, eliminating confusion.
  73. * **Automated Video Processing Pipeline**: From raw video → frame extraction → caption generation → vector database, fully automated
  74. * **ReAct-Style Video QA**: Multi-step reasoning + tool calling for precise video content understanding
  75. ---
  76. ## 📌 Table of Contents
  77. * [Project Introduction](#project-introduction)
  78. * [TeleMem vs Mem0: Core Advantages](#telemem-vs-mem0-core-advantages)
  79. * [Experimental Results](#experimental-results)
  80. * [Quick Start](#quick-start)
  81. * [Project Structure](#project-structure)
  82. * [Core Functions](#core-functions)
  83. * [Multimodal Extensions](#multimodal-extensions)
  84. * [MCP Server](#mcp-server)
  85. * [Framework Integrations](#framework-integrations)
  86. * [Data Storage Explanation](#data-storage)
  87. * [Telemetry & Privacy](#telemetry--privacy)
  88. * [Development and Contribution](#development-and-contribution)
  89. * [Acknowledgements](#acknowledgements)
  90. * [Citation](#-citation)
  91. ---
  92. ## Project Introduction
  93. TeleMem enables conversational AI to maintain stable, natural, and continuous worldviews and character settings during long-term interactions through a deeply optimized pipeline of **character-aware summarization → semantic clustering deduplication → efficient storage → precise retrieval**.
  94. ```mermaid
  95. flowchart LR
  96. A["Dialogue<br/>messages"] --> B["Character-aware<br/>summarization<br/>(global + per-character)"]
  97. B --> C["Embedding +<br/>similar-memory<br/>retrieval"]
  98. C --> D["Write buffer<br/>(batch flush)"]
  99. D --> E["LLM semantic<br/>clustering & fusion"]
  100. E --> F[("FAISS index +<br/>JSON metadata")]
  101. Q["Query"] --> S["Vector search<br/>+ rerank"]
  102. F --> S
  103. S --> R["results"]
  104. ```
  105. ### Features
  106. - **Automatic memory extraction**: Extracts and structures key facts from dialogues.
  107. - **Semantic clustering & deduplication**: Uses LLMs to semantically merge similar memories, reducing conflicts and improving consistency.
  108. - **Character-profiled memory management**: Builds independent memory archives for each character in a dialogue, ensuring precise isolation and personalized management.
  109. - **Efficient asynchronous writing**: Employs a buffer + batch-flush mechanism for high-performance, stable persistence.
  110. - **Precise semantic retrieval**: Combines **FAISS + JSON dual storage** for fast recall and human-readable auditability.
  111. ### Applicable Scenarios
  112. * Multi-character virtual agent systems
  113. * Long-memory AI assistants (e.g., customer service, companionship, creative co-pilots)
  114. * Complex narrative/world-building in virtual environments
  115. * Dialogue scenarios with strong contextual dependencies
  116. * Video content QA and reasoning
  117. * Multimodal agent memory management
  118. * Long video understanding and information retrieval
  119. ![image](assets/text-writing.png)
  120. ---
  121. ## TeleMem vs Mem0: Core Advantages
  122. TeleMem deeply refactors Mem0 to address **characterization**, **long-term memory**, and **high performance**. Key differences:
  123. | Capability Dimension | Mem0 | TeleMem |
  124. | -------------------------- | --------------------------- | ------------------------------------------------------------ |
  125. | Multi-character separation | ❌ Not supported | ✅ Automatically creates **independent memory profiles** per character |
  126. | Summary quality | Basic summarization | ✅ **Context-aware + character-focused prompts** covering key entities, actions, and timestamps |
  127. | Deduplication mechanism | Vector similarity filtering | ✅ **LLM-based semantic clustering**: merges similar memories via LLM |
  128. | Write performance | Streaming, single writes | ✅ **Batch flush + concurrency**: 2–3× faster writes |
  129. | Storage format | SQLite / vector DB | ✅ **FAISS + JSON metadata dual-write**: fast retrieval + human-readable |
  130. | Multimodal Capability | Single image to text only | ✅ **Video Multimodal Memory**: Full video processing pipeline + ReAct multi-step reasoning QA |
  131. ---
  132. ## Experimental Results
  133. ### Dataset
  134. We evaluate the ZH-4O Chinese long-character dialogue dataset constructed in the paper [MOOM: Maintenance, Organization and Optimization of Memory in Ultra-Long Role-Playing Dialogues](https://arxiv.org/abs/2509.11860):
  135. - Average dialogue length: **600 turns per conversation**
  136. - Scenarios: daily interactions, plot progression, evolving character relationships
  137. Memory capability was assessed via QA benchmarks, e.g.:
  138. ```json
  139. {
  140. "question": "What is Zhao Qi's nickname for Bai Yulan? A Xiaobai B Xiaoyu C Lanlan D Yuyu",
  141. "answer": "A"
  142. },
  143. {
  144. "question": "What is the relationship between Zhao Qi and Bai Yulan? A Classmates B Teacher and student C Enemies D Neighbors",
  145. "answer": "B"
  146. }
  147. ```
  148. ### Experimental Configuration
  149. - LLM: [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) (thinking mode disabled)
  150. - Embedding model: [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B)
  151. - Metric: QA accuracy
  152. | Method | Overall(%) |
  153. |:--------------------------------------------------------- |:---------- |
  154. | RAG | 62.45 |
  155. | _[Mem0](https://github.com/mem0ai/mem0)_ | _70.20_ |
  156. | [MOOM](https://github.com/cows21/MOOM-Roleplay-Dialogue) | 72.60 |
  157. | [A-mem](https://github.com/agiresearch/A-mem) | 73.78 |
  158. | [Memobase](https://github.com/memodb-io/memobase) | 76.78 |
  159. | **[TeleMem](https://github.com/TeleAI-UAGI/TeleMem)** | **86.33** |
  160. <!--
  161. | Long-Context LLM (Slow and Expensive) | 84.92 |
  162. -->
  163. ---
  164. ## Quick Start
  165. ### Installation
  166. ```shell
  167. pip install telemem # core (text memory)
  168. pip install "telemem[mcp]" # + MCP server
  169. pip install "telemem[video]" # + video/multimodal pipeline
  170. pip install "telemem[all]" # everything
  171. ```
  172. ### Development Environment
  173. Using [uv](https://docs.astral.sh/uv/) (recommended — creates `.venv` from the committed `uv.lock` for a reproducible environment):
  174. ```shell
  175. uv sync --all-extras # install TeleMem (editable) + all extras, incl. MCP
  176. uv run python examples/quickstart.py
  177. ```
  178. Or with conda + pip:
  179. ```shell
  180. # Create and activate virtual environment
  181. conda create -n telemem python=3.10
  182. conda activate telemem
  183. # Install from source (editable), with the extras you need
  184. pip install -e ".[all]"
  185. ```
  186. ### Example
  187. Set your OpenAI API key:
  188. ```shell
  189. export OPENAI_API_KEY="your-openai-api-key"
  190. ```
  191. ```python
  192. # python examples/quickstart.py
  193. import telemem as mem0
  194. memory = mem0.Memory()
  195. messages = [
  196. {"role": "user", "content": "Jordan, did you take the subway to work again today?"},
  197. {"role": "assistant", "content": "Yes, James. The subway is much faster than driving. I leave at 7 o'clock and it's just not crowded."},
  198. {"role": "user", "content": "Jordan, I want to try taking the subway too. Can you tell me which station is closest?"},
  199. {"role": "assistant", "content": "Of course, James. You take Line 2 to Civic Center Station, exit from Exit A, and walk 5 minutes to the company."}
  200. ]
  201. memory.add(messages=messages, user_id="Jordan")
  202. results = memory.search("What transportation did Jordan use to go to work today?", user_id="Jordan")
  203. for hit in results["results"]: # same result shape as mem0
  204. print(hit["memory"])
  205. ```
  206. `Memory()` uses the default provider settings inherited from `mem0ai`. To use the repository's local Qwen + FAISS configuration, load `config/config.yaml` explicitly:
  207. ```python
  208. from telemem.utils import load_config
  209. import telemem as mem0
  210. config = load_config("config/config.yaml")
  211. memory = mem0.Memory(config=config)
  212. ```
  213. The runnable examples also honor the same configuration through `TELEMEM_CONFIG`:
  214. ```shell
  215. TELEMEM_CONFIG=config/config.yaml python examples/quickstart.py
  216. ```
  217. ### Using MiniMax as the LLM Provider
  218. TeleMem supports [MiniMax](https://api.minimax.io) as an LLM backend via its OpenAI-compatible API.
  219. A ready-to-use example config is provided at `config/config.minimax.yaml`.
  220. ```shell
  221. export MINIMAX_API_KEY="your-minimax-api-key"
  222. export OPENAI_API_KEY="your-openai-api-key" # still needed for embeddings
  223. ```
  224. ```python
  225. from telemem.utils import load_config
  226. import telemem as mem0
  227. config = load_config("config/config.minimax.yaml")
  228. memory = mem0.Memory(config=config)
  229. ```
  230. Key points for MiniMax usage:
  231. - **LLM**: `MiniMax-M3.1-Flash` (default) or `MiniMax-M3`, both with a 1,000,000-token context window and text/image/video input. Existing configs keep their explicit model IDs, including legacy and custom models.
  232. - **Regional endpoints**: use `https://api.minimax.io/v1` (global) or `https://api.minimax.cn/v1` (China) as `openai_base_url`; existing custom endpoint values are preserved.
  233. - **Thinking**: Flash always uses adaptive thinking, with `max` reasoning effort when omitted. The preset sets `is_reasoning_model: true` so mem0 forwards an optional `reasoning_effort` (`low`, `medium`, `high`, `xhigh`, or `max`) and omits its small default token cap. Flash rejects disabled thinking, `reasoning_effort="none"`, `service_tier`, and `reasoning_split=false`. See the [OpenAI-compatible API contract](https://platform.minimax.io/docs/api-reference/text-openai-api) for details.
  234. - **Tool history**: when calling the LLM adapter directly, use mem0's `response_callback` to capture the complete SDK assistant message and append it with tool results on the next call. Preserve both `reasoning_content` and `tool_calls`; the adapter's simplified return value is not a complete assistant message.
  235. - **Temperature**: ignored by Flash and M3, which use a fixed value of 1.
  236. - **Embeddings**: MiniMax does not provide a public embedding API; configure a separate embedder (e.g. `text-embedding-3-small`) in the `embedder` section
  237. ### More LLM Providers
  238. TeleMem works with **any OpenAI-compatible endpoint**. Ready-to-use config examples ship in `config/`:
  239. | Provider | Config file | LLM | Embeddings | Notes |
  240. | -------- | ----------- | --- | ---------- | ----- |
  241. | **Ollama** (fully local) | [`config.ollama.yaml`](config/config.ollama.yaml) | any local model (e.g. `qwen3:8b`) | `nomic-embed-text`, local | **No API key, no cloud** — everything runs on your machine |
  242. | **DeepSeek** | [`config.deepseek.yaml`](config/config.deepseek.yaml) | `deepseek-chat` / `deepseek-reasoner` | external (e.g. OpenAI) | `DEEPSEEK_API_KEY` + `OPENAI_API_KEY` |
  243. | **Moonshot (Kimi)** | [`config.moonshot.yaml`](config/config.moonshot.yaml) | `kimi-k2-0905-preview` | external (e.g. OpenAI) | `.cn` and `.ai` endpoints supported |
  244. | **MiniMax** | [`config.minimax.yaml`](config/config.minimax.yaml) | `MiniMax-M3.1-Flash` / `MiniMax-M3` | external (e.g. OpenAI) | see section above |
  245. ```shell
  246. TELEMEM_CONFIG=config/config.ollama.yaml python examples/quickstart.py # 100% local memory
  247. ```
  248. ---
  249. ## Project Structure
  250. <details>
  251. <summary>Expand/Collapse Directory Structure</summary>
  252. ```
  253. telemem/
  254. ├── assets/ # Documentation assets and figures
  255. ├── baselines/ # Baseline implementations for comparative evaluation
  256. │ ├── RAG # Retrieval-Augmented Generation baseline
  257. │ ├── MemoBase # MemoBase memory management system
  258. │ ├── MOOM # MOOM dual-branch narrative memory framework
  259. │ ├── A-mem # A-mem agent memory baseline
  260. │ └── Mem0 # Mem0 baseline implementation
  261. ├── config/
  262. │ ├── config.yaml # TeleMem default configuration
  263. │ └── config.minimax.yaml # MiniMax provider example configuration
  264. ├── data/ # Small sample datasets for evaluation or demonstration
  265. ├── examples/ # Code examples and tutorial demos
  266. │ ├── quickstart.py         # Quick start
  267. │ ├── quickstart_mm.py # Quick start (multimodal)
  268. │ ├── mcp_client.py # Quick start over MCP (stdio client)
  269. │ ├── mcp_config.json # MCP config snippet for Claude Desktop / Cursor
  270. │ └── deepseek-harness.cordis.yml # DeepSeek Harness memory patch
  271. ├── docs/
  272. │ ├── MCP.md # MCP server reference
  273. │ └── TeleMem_Tech_Report.pdf
  274. ├── telemem/ # Telemem code
  275. │ └── mcp/ # Model Context Protocol server
  276. ├── tests/ # Telemem test
  277. ├── README.md               # English README
  278. ├── README-ZH.md            # Chinese README
  279. └── pyproject.toml # Python environment
  280. ```
  281. </details>
  282. ---
  283. ## Core Functions
  284. ### Add Memory (add)
  285. The `add()` method injects one or more dialogue turns into the memory system.
  286. ```python
  287. def add(
  288. self,
  289. messages,
  290. *,
  291. user_id: Optional[str] = None,
  292. agent_id: Optional[str] = None,
  293. run_id: Optional[str] = None,
  294. metadata: Optional[Dict[str, Any]] = None,
  295. infer: bool = True,
  296. memory_type: Optional[str] = None,
  297. prompt: Optional[str] = None,
  298. batch: bool = False,
  299. )
  300. ```
  301. #### 🔎 Parameter Description
  302. | Parameter | Type | Required | Description |
  303. | ------------- | ------------------------------- | -------- | ------------------------------------------------------------ |
  304. | `messages` | `str` or `List[Dict[str, str]]` | ✅ Yes | A single statement, or a list of dialogue messages with `role` (`user`/`assistant`) and `content` |
  305. | `user_id` | `Optional[str]` | ❌ No | Character/user to attribute the memory to; TeleMem keeps an **independent memory profile per `user_id`**. Omit it to store shared conversation-event memories |
  306. | `agent_id` / `run_id` | `Optional[str]` | ❌ No | Additional mem0-compatible scopes (e.g. one `run_id` per session) |
  307. | `metadata` | `Optional[Dict[str, Any]]` | ❌ No | Arbitrary metadata stored with each memory |
  308. | `infer` | `bool` | ❌ No | Extract salient facts with the LLM (default: `True`); `False` stores message contents verbatim with no LLM call |
  309. | `memory_type` | `Optional[str]` | ❌ No | Pass `"procedural_memory"` to create procedural memories via mem0's pipeline; omit for conversational memories |
  310. | `prompt` | `Optional[str]` | ❌ No | Custom extraction prompt (replaces the optimized default as the system prompt) |
  311. | `batch` | `bool` | ❌ No | Route through the high-throughput batched pipeline (`add_batch`) |
  312. **Returns** the mem0-compatible shape: `{"results": [{"id": "...", "memory": "...", "event": "ADD"}, ...]}`
  313. #### 🔁 Internal Workflow of `add()`
  314. 1. **Message preprocessing**: Merge consecutive messages from the same speaker; normalize turn structure.
  315. 2. **Multi-perspective summarization**:
  316. - Global event summary
  317. - Character 1’s perspective (actions, preferences, relationships)
  318. - Character 2’s perspective
  319. 3. **Vectorization & similarity search**: Generate embeddings and retrieve existing similar memories.
  320. 4. **Batch processing**: When buffer threshold is reached, invoke LLM to **semantically merge** similar memories.
  321. 5. **Persistence**: Dual-write to **FAISS (for retrieval)** and **JSON (for metadata)**.
  322. > 🎭 **Multi-character demo**: [examples/multi_npc.py](examples/multi_npc.py) runs five tavern
  323. > NPCs through one scene — a single `add_batch(scene, user_id=[...])` call gives each NPC a
  324. > private memory profile plus a shared `"events"` world-state, and each NPC then recalls the
  325. > scene from their own perspective.
  326. ---
  327. ### Search Memory (search)
  328. Performs semantic vector-based retrieval of relevant memories with context-aware recall.
  329. ```python
  330. def search(
  331. self,
  332. query: str,
  333. *,
  334. user_id: Optional[str] = None,
  335. agent_id: Optional[str] = None,
  336. run_id: Optional[str] = None,
  337. limit: int = 100,
  338. filters: Optional[Dict[str, Any]] = None,
  339. threshold: Optional[float] = None,
  340. rerank: bool = True,
  341. )
  342. ```
  343. #### 🔎 Parameter Description
  344. | Parameter | Type | Required | Description |
  345. | ----------- | ------------------ | -------- | ------------------------------------------------- |
  346. | `query` | `str` | ✅ Yes | Natural language query |
  347. | `user_id` | `Optional[str]` | ❌ No | Character/user profile to search. The shared event memories (pseudo-user `"events"`) are always searched as well |
  348. | `agent_id` / `run_id` | `Optional[str]` | ❌ No | Additional mem0-compatible scope filters |
  349. | `limit` | `int` | ❌ No | Max number of results (default: 100) |
  350. | `threshold` | `Optional[float]` | ❌ No | Similarity threshold (0–1; auto-tuned if omitted) |
  351. | `filters` | `Dict[str, Any]` | ❌ No | Custom filters (e.g., by character, time range) |
  352. | `rerank` | `bool` | ❌ No | Whether to rerank results (default: `True`) |
  353. **Returns** the mem0-compatible shape: `{"results": [{"id": "...", "memory": "...", "score": ..., ...}, ...]}`
  354. > 🔍 Search is based on FAISS vector retrieval, supporting millisecond-level responses.
  355. ---
  356. ## Multimodal Extensions
  357. Beyond text memory, TeleMem further extends multimodal capabilities. Drawing inspiration from [Deep Video Discovery](https://github.com/microsoft/DeepVideoDiscovery)'s Agentic Search and Tool Use approach, we implemented two core methods in the TeleMemory class to support intelligent storage and semantic retrieval of video content.
  358. | Method | Description |
  359. |------|----------|
  360. | `add_mm()` | Process video into retrievable memory (frame extraction → caption generation → vector database) |
  361. | `search_mm()` | Query video content using natural language, supporting ReAct-style multi-step reasoning |
  362. ### Add Multimodal Memory (add_mm)
  363. ```python
  364. def add_mm(
  365. self,
  366. video_path: str,
  367. output_dir: str,
  368. clip_secs: int | None = None,
  369. emb_dim: int | None = None,
  370. subtitle_path: str | None = None,
  371. )
  372. ```
  373. #### 🔎 Parameter Description
  374. | Parameter | Type | Required | Description |
  375. |--------|------|----------|------|
  376. | video_path | str | ✅ Yes | Source video file path, e.g., `"video/3EQLFHRHpag.mp4"` |
  377. | output_dir | str | ✅ Yes | Root output directory. Artifacts are written under `frames/`, `captions/`, and `vdb/` subdirectories |
  378. | clip_secs | int | ❌ No | Reserved parameter; clip length is currently read from `config.vlm["CLIP_SECS"]` |
  379. | emb_dim | int | ❌ No | Embedding dimension, reads from config by default |
  380. | subtitle_path | str | ❌ No | Subtitle file path (.srt), optional |
  381. #### 🔁 add_mm() Internal Flow
  382. 1. **Frame Extraction**: `decode_video_to_frames` - Decodes video to JPEG frames at configured FPS
  383. 2. **Caption Generation**: `process_video` - Uses VLM (e.g., Qwen3-Omni) to generate detailed descriptions for each clip
  384. 3. **Vector Database Construction**: `init_single_video_db` - Generates embeddings for semantic retrieval
  385. > 💡 **Smart Caching**: If the target file for a stage already exists, that stage is automatically skipped to save computational resources.
  386. #### Return Value Example
  387. ```python
  388. {
  389. "output_dir": "/abs/path/to/output_dir"
  390. }
  391. ```
  392. ---
  393. ### Search Multimodal Memory (search_mm)
  394. ```python
  395. def search_mm(
  396. self,
  397. question: str,
  398. output_dir: str,
  399. max_iterations: int = 15,
  400. )
  401. ```
  402. #### 🔎 Parameter Description
  403. | Parameter | Type | Required | Description |
  404. |--------|------|----------|------|
  405. | question | str | ✅ Yes | Question string (supports A/B/C/D multiple choice format) |
  406. | output_dir | str | ✅ Yes | The same root output directory used by `add_mm`; it must contain exactly one `captions/*/captions.json` and one `vdb/*/*_vdb.json` |
  407. | max_iterations | int | ❌ No | Maximum MMCoreAgent reasoning iterations (default 15) |
  408. #### 🛠️ ReAct-Style Reasoning Tools
  409. `search_mm` internally uses `MMCoreAgent`, employing a THINK → ACTION → OBSERVATION loop with three specialized tools:
  410. | Tool Name | Function |
  411. |--------|------|
  412. | `global_browse_tool` | Get global overview of video events and themes |
  413. | `clip_search_tool` | Search for specific content using semantic queries |
  414. | `frame_inspect_tool` | Inspect frame details within a specific time range |
  415. ---
  416. ### Multimodal Example
  417. Run the multimodal demo:
  418. ```bash
  419. python examples/quickstart_mm.py
  420. ```
  421. On the first run, frames, captions and VDB JSON will be generated under the chosen `output_dir`. The repository ships a small sample video; generating captions and the video database still requires configured VLM and embedding services unless you already have these artifacts locally.
  422. Complete code example:
  423. ```python
  424. import telemem as mem0
  425. from pathlib import Path
  426. from telemem.mm_utils.core import extract_choice_from_msg
  427. # Initialize
  428. memory = mem0.Memory()
  429. # Define paths
  430. repo_root = Path(__file__).resolve().parents[1]
  431. video_path = repo_root / "data" / "samples" / "video" / "3EQLFHRHpag.mp4"
  432. video_name = video_path.stem
  433. output_dir = video_path.parent
  434. # Step 1: Add video to memory (auto-processing)
  435. vdb_json_path = output_dir / "vdb" / video_name / f"{video_name}_vdb.json"
  436. if not vdb_json_path.exists():
  437. result = memory.add_mm(
  438. video_path=str(video_path),
  439. output_dir=str(output_dir),
  440. )
  441. print(f"Video processing complete: {result}")
  442. else:
  443. print(f"VDB already exists: {vdb_json_path}")
  444. # Step 2: Query video content
  445. question = """The problems people encounter in the video are caused by what?
  446. (A) Catastrophic weather.
  447. (B) Global warming.
  448. (C) Financial crisis.
  449. (D) Oil crisis.
  450. """
  451. messages = memory.search_mm(
  452. question=question,
  453. output_dir=str(output_dir),
  454. max_iterations=15,
  455. )
  456. # Extract final answer
  457. answer = extract_choice_from_msg(messages)
  458. print(f"Answer: ({answer})")
  459. ```
  460. ---
  461. ## MCP Server
  462. [![MCPVault: verified](https://mcpvault.io/badge/telemem.svg)](https://mcpvault.io/servers/telemem/health?utm_source=external_badge&utm_medium=referral&utm_campaign=mcp_health_report)
  463. TeleMem ships a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server, so any MCP-compatible client — Claude Desktop, Claude Code, Cursor, custom agents — can use TeleMem as its long-term memory.
  464. ```shell
  465. pip install telemem
  466. telemem-mcp # stdio (default)
  467. telemem-mcp --transport streamable-http # Streamable HTTP on :8421
  468. TELEMEM_CONFIG=config/config.yaml telemem-mcp # custom TeleMem config
  469. uvx telemem # zero-install run (stdio)
  470. ```
  471. Built on the official MCP Python SDK v2, the server implements the current MCP
  472. specification (2026-07-28) while remaining compatible with older clients; every tool
  473. declares titles, behavior annotations (read-only/destructive hints), and structured output.
  474. The server exposes eight tools: `add_memory`, `search_memories`, `get_memories`, `get_memory`, `update_memory`, `delete_memory`, `delete_all_memories`, and `memory_history`. Calls without an explicit scope default to `TELEMEM_DEFAULT_USER_ID` (`telemem-mcp`); destructive bulk deletion always requires an explicit scope.
  475. Claude Desktop / Cursor configuration ([examples/mcp_config.json](examples/mcp_config.json)):
  476. ```json
  477. {
  478. "mcpServers": {
  479. "telemem": {
  480. "command": "telemem-mcp",
  481. "env": {
  482. "TELEMEM_CONFIG": "/absolute/path/to/config/config.yaml",
  483. "OPENAI_API_KEY": "sk-..."
  484. }
  485. }
  486. }
  487. }
  488. ```
  489. Or drive it programmatically over stdio — the quickstart flow as MCP tool calls:
  490. ```shell
  491. python examples/mcp_client.py
  492. ```
  493. See [docs/MCP.md](docs/MCP.md) for the full tool reference, transports, and client setup.
  494. ---
  495. ## Framework Integrations
  496. TeleMem drops into any agent framework with the same two calls — `search()` before answering, `add()` after each exchange:
  497. | Framework | Example | Install |
  498. | --------- | ------- | ------- |
  499. | **LangChain** | [examples/langchain_memory.py](examples/langchain_memory.py) | `pip install langchain-core langchain-openai` |
  500. | **LlamaIndex** | [examples/llamaindex_memory.py](examples/llamaindex_memory.py) | `pip install llama-index-llms-openai` |
  501. | **Claude Desktop / Cursor / any MCP client** | [MCP Server](#mcp-server) | `pip install "telemem[mcp]"` |
  502. Because TeleMem is mem0 API-compatible, any framework adapter written for Mem0's OSS client also works — point it at `telemem.Memory` instead.
  503. ---
  504. ## Data Storage
  505. ### Text Memory Storage
  506. TeleMem automatically creates a structured storage layout under `./faiss_db/`, organized by session and character:
  507. ```
  508. faiss_db/
  509. ├── session_001_events.index
  510. ├── session_001_events_meta.json
  511. ├── session_001_person_1.index
  512. ├── session_001_person_1_meta.json
  513. ├── session_001_person_2.index
  514. └── session_001_person_2_meta.json
  515. ```
  516. ### 📄 Metadata Example (_meta.json)
  517. ```json
  518. {
  519. "summary": "Characters discussed the upcoming action plan.",
  520. "sample_id": "session_001",
  521. "round_index": 3,
  522. "timestamp": "2024-01-01T00:00:00Z",
  523. "user": "Jordan" // Only present in person_*.json
  524. }
  525. ```
  526. > All memories include summary, round number, timestamp, and character, facilitating auditing and debugging.
  527. ------
  528. ### Multimodal Memory Storage
  529. TeleMem generates video-related storage files in the `.data/samples/video/` directory:
  530. ```
  531. video/
  532. ├── frames/
  533. │ └── <video_name>/
  534. │ └── frames/
  535. │ ├── frame_000001_n0.00.jpg
  536. │ ├── frame_000002_n0.50.jpg
  537. │ └── ...
  538. ├── captions/
  539. │ └── <video_name>/
  540. │ ├── captions.json # Clip descriptions + subject registry
  541. │ └── ckpt/ # Checkpoint for resume
  542. │ ├── 0_10.json
  543. │ └── 10_20.json
  544. └── vdb/
  545. └── <video_name>/
  546. └── <video_name>_vdb.json # Semantic retrieval vector database
  547. ```
  548. #### 📄 captions.json Structure
  549. ```json
  550. {
  551. "0_10": {
  552. "caption": "The narrator discusses climate data, showing melting glaciers..."
  553. },
  554. "10_20": {
  555. "caption": "Scene shifts to coastal communities affected by rising sea levels..."
  556. },
  557. "subject_registry": {
  558. "narrator": {
  559. "name": "narrator",
  560. "appearance": ["professional attire"],
  561. "identity": ["climate scientist"],
  562. "first_seen": "00:00:00"
  563. }
  564. }
  565. }
  566. ```
  567. ------
  568. ## Telemetry & Privacy
  569. TeleMem itself collects no telemetry. The underlying `mem0ai` library ships
  570. anonymized PostHog usage telemetry, which TeleMem **disables by default**
  571. (`import telemem` sets `MEM0_TELEMETRY=False` unless you have already set it).
  572. To opt back in:
  573. ```bash
  574. export MEM0_TELEMETRY=true
  575. ```
  576. ------
  577. ## Development and Contribution
  578. * Issues and pull requests are welcome — see the [Contributing Guide](CONTRIBUTING.md) to get started.
  579. * Changes between releases are tracked in the [Changelog](CHANGELOG.md).
  580. * CI runs the offline test suite (`uv run pytest tests/ -q`) on Python 3.10–3.12 for every PR.
  581. * Chinese documentation: [README-ZH.md](README-ZH.md)
  582. ---
  583. ## License
  584. [Apache 2.0 License](LICENSE)
  585. ---
  586. ## Acknowledgements
  587. TeleMem’s development has been deeply inspired by open-source communities and cutting-edge research. We extend our sincere gratitude to the following projects and teams:
  588. - **[Mem0](https://github.com/mem0ai/mem0)**
  589. - **[Memobase](https://github.com/memodb-io/memobase)**
  590. - **[MOOM](https://github.com/cows21/MOOM-Roleplay-Dialogue)**
  591. - **[DVD](https://github.com/microsoft/DeepVideoDiscovery)**
  592. - **[Memento](https://github.com/Agent-on-the-Fly/Memento)**
  593. - **[Momento-Skills](https://github.com/Memento-Teams/Memento-Skills)**
  594. ---
  595. ## 📑 Citation
  596. If you find TeleMem useful to your research or development, please cite our arXiv paper:
  597. > Chunliang Chen, Ming Guan, Xiao Lin, Jiaxu Li, Luxi Lin, Qiyi Wang, Xiangyu Chen, Jixiang Luo, Changzhi Sun, Dell Zhang, Xuelong Li. **TeleMem: Building Long-Term and Multimodal Memory for Agentic AI.** arXiv:2601.06037, 2026. https://arxiv.org/abs/2601.06037
  598. ```bibtex
  599. @article{chen2026telemem,
  600. title = {TeleMem: Building Long-Term and Multimodal Memory for Agentic AI},
  601. author = {Chen, Chunliang and Guan, Ming and Lin, Xiao and Li, Jiaxu and Lin, Luxi and Wang, Qiyi and Chen, Xiangyu and Luo, Jixiang and Sun, Changzhi and Zhang, Dell and Li, Xuelong},
  602. journal = {arXiv preprint arXiv:2601.06037},
  603. year = {2026},
  604. url = {https://arxiv.org/abs/2601.06037}
  605. }
  606. ```
  607. Citation metadata is also available in [CITATION.cff](CITATION.cff) (GitHub's "Cite this repository" button).
  608. ---
  609. <!-- ## Star History
  610. <a href="https://github.com/TeleAI-UAGI/telemem/stargazers"><picture>
  611. <source media="(prefers-color-scheme: dark)" srcset="assets/star-history-dark.svg">
  612. <img alt="Star history chart of TeleAI-UAGI/telemem" src="assets/star-history-light.svg">
  613. </picture></a>
  614. <sub>Regenerated weekly by [a scheduled workflow](.github/workflows/star-history.yml); the previous [star-history.com](https://www.star-history.com/) live chart broke when GitHub restricted the stargazers API to repo admins and collaborators in June 2026.</sub>
  615. --- -->
  616. <div align="center">
  617. **If you find this project helpful, please give it a ⭐️ — starring also keeps updates in your GitHub feed.**
  618. Made with ❤️ by [Bloo-Mind AI Ltd](https://www.bloo-mind.ai/) and the Ubiquitous AGI team at TeleAI.
  619. </div>
  620. <div align="center" style="margin-top: 10px;">
  621. <a href="https://www.bloo-mind.ai/"><img src="assets/bloo-mind.png" alt="Bloo-Mind Logo" width="120px" /></a>
  622. &nbsp;&nbsp;&nbsp;
  623. <img src="assets/TeleAI.png" alt="TeleAI Logo" width="120px" />
  624. </div>
  625. <sub>mcp-name: io.github.TeleAI-UAGI/telemem</sub>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.