Skip to content
Back to search

deskmate

Local-first PC activity recorder for Windows. Captures screenshots, accessibility text, UI events, clipboard activity, and optional audio transcription into a local SQLite database — then exposes everything through a REST API, browser UI, and MCP server.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/zhaohb--deskmate/badge.svg)](/listings/zhaohb--deskmate)

HTML

<a href="/listings/zhaohb--deskmate"><img src="/api/listings/zhaohb--deskmate/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

If you just want to try it out, follow these 4 steps. **You don't need to understand anything below this section.**

Step 1 — Install Python (if you don't have it)

You need **Python 3.10 or newer**. - If it's not installed, get it from <https://www.python.org/downloads/> and **check "Add Python to PATH"** during installation. - Verify: open PowerShell and run `python --version` — you should see `Python 3.10.x` or higher.

Step 2 — Get the code and create an isolated environment

git clone <repo-url>
cd deskmate

# Create an isolated Python environment (keeps your system clean)
python -m venv .venv
.\.venv\Scripts\Activate.ps1

> Every time you open a new terminal to use DeskMate, `cd` into the folder and run > `.\.venv\Scripts\Activate.ps1` again to activate the environment. You'll know it > worked when you see `(.venv)` at the start of your prompt.

Step 3 — Install (recommended default combo)

pip install -e ".[ocr-winrt,audio,vad,mcp]"

This installs: **core features + built-in Windows OCR + audio transcription + voice activity detection + MCP**. That's enough for most people. (What each `[...]` means and what else you can add → see [Install features as needed](#-install-features-as-needed-submodule-dependencies).)

Step 4 — Launch

deskmate ui

Your browser opens at **http://127.0.0.1:3030/deskmate** and DeskMate starts recording.

On first run it auto-creates a config and data folder under `C:\Users\<your-username>\.deskmate\`. **You're up and running.**

> To stop: press `Ctrl + C` in the terminal where `deskmate ui` is running.

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 620 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # DeskMate
  2. <div style="text-align:center;">
  3. <img src="./imgs/deskmate_overview.gif" alt="DeskMate" width="1200" height="800">
  4. </div>
  5. DeskMate is a **local-first** desktop activity recorder for **Windows**. It captures
  6. your screenshots, on-screen text (OCR / accessibility tree), keyboard/mouse/clipboard
  7. events, and optional audio transcription into a SQLite database **on your own machine**,
  8. then lets you use that data through a browser UI, a REST API, and an **MCP server**
  9. that plugs straight into AI coding agents like **Claude Code** and **Cursor** —
  10. so your assistant can search your activity, ask natural-language questions, and
  11. run auto-generated reports (see [MCP server](#-mcp-server-use-deskmate-from-ai-agents)).
  12. DeskMate can also read what is on your screen and **suggest the next step** — start a
  13. learning session, start a meeting recording, reply to an email, book a meeting room —
  14. which you confirm with one click (see
  15. [Decision routing](#-decision-routing-suggestions-you-confirm)).
  16. > **All your data stays on your own computer — nothing is uploaded to the cloud.**
  17. ---
  18. ## 🚀 Quick Start (5 minutes — just get it running)
  19. If you just want to try it out, follow these 4 steps. **You don't need to understand
  20. anything below this section.**
  21. ### Step 1 — Install Python (if you don't have it)
  22. You need **Python 3.10 or newer**.
  23. - If it's not installed, get it from <https://www.python.org/downloads/> and **check
  24. "Add Python to PATH"** during installation.
  25. - Verify: open PowerShell and run `python --version` — you should see `Python 3.10.x`
  26. or higher.
  27. ### Step 2 — Get the code and create an isolated environment
  28. ```powershell
  29. git clone <repo-url>
  30. cd deskmate
  31. # Create an isolated Python environment (keeps your system clean)
  32. python -m venv .venv
  33. .\.venv\Scripts\Activate.ps1
  34. ```
  35. > Every time you open a new terminal to use DeskMate, `cd` into the folder and run
  36. > `.\.venv\Scripts\Activate.ps1` again to activate the environment. You'll know it
  37. > worked when you see `(.venv)` at the start of your prompt.
  38. ### Step 3 — Install (recommended default combo)
  39. ```powershell
  40. pip install -e ".[ocr-winrt,audio,vad,mcp]"
  41. ```
  42. This installs: **core features + built-in Windows OCR + audio transcription + voice
  43. activity detection + MCP**. That's enough for most people.
  44. (What each `[...]` means and what else you can add → see [Install features as needed](#-install-features-as-needed-submodule-dependencies).)
  45. ### Step 4 — Launch
  46. ```powershell
  47. deskmate ui
  48. ```
  49. Your browser opens at **http://127.0.0.1:3030/deskmate** and DeskMate starts recording.
  50. On first run it auto-creates a config and data folder under
  51. `C:\Users\<your-username>\.deskmate\`. **You're up and running.**
  52. > To stop: press `Ctrl + C` in the terminal where `deskmate ui` is running.
  53. ---
  54. ## 🧩 Install features as needed (submodule dependencies)
  55. DeskMate splits the "heavy" features into **optional modules (extras)** — install only
  56. what you need, not everything at once. The install format is always:
  57. ```powershell
  58. pip install -e ".[module-name]" # install one
  59. pip install -e ".[module1,module2]" # install several (comma-separated, NO spaces)
  60. ```
  61. | Module | What it adds | When you need it / notes |
  62. |--------|--------------|--------------------------|
  63. | `ocr-winrt` | Built-in Windows OCR | **Recommended default**, no extra downloads |
  64. | `ocr-rapidocr` | PP-OCR (via OpenVINO) | **Best for Chinese / small UI text**, models bundled |
  65. | `ocr-tesseract` | Tesseract OCR | Requires [Tesseract](https://github.com/tesseract-ocr/tesseract) installed and on PATH |
  66. | `audio` | Recording + Whisper transcription | Needs a mic or system audio (WASAPI). Default transcription backend |
  67. | `audio-openvino` | Whisper on Intel NPU/GPU/CPU | Accelerated on Intel Core Ultra (NPU) or Arc GPU, see below |
  68. | `vad` | Voice activity detection (Silero VAD) | Skips silent segments to save compute; install alongside `audio` |
  69. | `speaker` | Speaker diarization | Tells "who said what" |
  70. | `redact-onnx` | PII redaction | Optional privacy-protection model |
  71. | `semantic` | Semantic (vector) search | Search by meaning, not just keywords |
  72. | `pipes` | Scheduled tasks (YAML) | Run reports automatically on a schedule |
  73. | `mcp` | MCP server | Integrate with AI agents / Claude, etc. |
  74. | `route` | Decision routing (local kev-0.8b model) | Suggests starting a learning session / meeting, replying to an email, or booking a room; see [Decision routing](#-decision-routing-suggestions-you-confirm) |
  75. | `training` | Local LoRA fine-tuning | Fine-tune a small model on your own data, see [training docs](docs/16-learning-training.md) |
  76. | `full` | Most common extras at once | = tesseract + audio + vad + redact + semantic + pipes. **Excludes** OpenVINO & training |
  77. | `dev` | Dev tools (pytest, ruff) | Only needed to modify code / run tests |
  78. > Beginner tip: start with the default combo from Step 3. Add a module later when you
  79. > decide you want a feature (e.g. Chinese OCR, semantic search) — re-running
  80. > `pip install -e ".[...]"` is safe.
  81. >
  82. > Battery Saver / Power Manager is built in and uses Windows APIs through the
  83. > standard library; it does **not** need an extra package.
  84. ### Install from a built wheel (instead of the source checkout)
  85. If you have the built `.whl` (see [Building a wheel](#-building-a-wheel-for-distribution) below)
  86. instead of the source tree, use the same `[extras]` syntax on the wheel file:
  87. ```powershell
  88. :: core only
  89. pip install deskmate-0.1.0-py3-none-any.whl
  90. :: recommended combo (same as Step 3)
  91. pip install "deskmate-0.1.0-py3-none-any.whl[ocr-winrt,audio,vad,mcp]"
  92. ```
  93. **Install (almost) everything in one go** — every optional feature at once. Note
  94. there is no single "all" extra, so list them explicitly. This pulls in the heavy
  95. deps (PyTorch, OpenVINO, ONNX Runtime, Whisper, RapidOCR…), so expect a large download:
  96. ```powershell
  97. pip install "deskmate-0.1.0-py3-none-any.whl[ocr-tesseract,ocr-rapidocr,ocr-winrt,audio,audio-openvino,vad,speaker,redact-onnx,semantic,mcp,pipes]"
  98. ```
  99. > This deliberately omits `training` (LoRA fine-tuning — only needed if you train
  100. > models; see [training docs](docs/16-learning-training.md)), `redact-onnx-dml` (the
  101. > DirectML build of `redact-onnx` — pick one, not both), and `dev` (test tooling).
  102. > The same works on the source checkout — just swap the wheel filename for `-e "."`,
  103. > e.g. `pip install -e ".[ocr-winrt,audio,vad,speaker,semantic,mcp,pipes]"`.
  104. >
  105. > ⚠️ Most extras pull Windows-only / GPU-specific wheels and large model runtimes.
  106. > Don't install everything unless you actually need it — the [recommended combo](#step-3--install-recommended-default-combo)
  107. > is enough for most users.
  108. ### 📦 Building a wheel (for distribution)
  109. To produce a distributable wheel (e.g. to install on another machine):
  110. ```powershell
  111. pip install build
  112. python -m build --wheel :: output: dist\deskmate-0.1.0-py3-none-any.whl
  113. ```
  114. The wheel bundles all of DeskMate's own code **and** the built-in apps
  115. (`deskmate/apps/`), but **not** third-party dependencies — those are recorded as
  116. requirements and fetched by `pip install` from PyPI. For an **offline** install,
  117. pre-download everything into a folder first:
  118. ```powershell
  119. pip download "deskmate-0.1.0-py3-none-any.whl[ocr-winrt,audio,vad,mcp]" -d wheelhouse
  120. :: then on the offline machine:
  121. pip install --no-index --find-links wheelhouse "deskmate[ocr-winrt,audio,vad,mcp]"
  122. ```
  123. ### 🧠 Installing the `training` extra (LoRA fine-tuning)
  124. The `training` extra (`torch` + `unsloth` + `transformers` + `peft` + `accelerate`)
  125. is **deliberately left out** of the "everything" command above, because the right
  126. `torch` build depends on your accelerator. A pip `[extra]` can't pin a
  127. hardware-specific wheel (the Intel XPU `torch` lives on PyTorch's own index, not
  128. PyPI). Below, `WHL` = `deskmate-0.1.0-py3-none-any.whl`; on a source checkout use
  129. `-e "."` instead.
  130. **Intel Arc / Core-Ultra iGPU (XPU):** the XPU `torch` wheel **and** an
  131. oneAPI/Level-Zero/MSVC toolchain are required — there are several non-obvious
  132. gotchas, so this path is scripted. Install the extra, then run the setup script:
  133. ```powershell
  134. pip install "WHL[training]"
  135. scripts\setup-intel-xpu.bat :: pulls torch 2.10.0+xpu + Level-Zero SDK headers
  136. ```
  137. Full walkthrough (oneAPI, MSVC, the toolchain gotchas) → [training docs](docs/16-learning-training.md).
  138. **CPU only** (no GPU — slow, for validating the pipeline on a tiny base like
  139. `Qwen/Qwen3-0.6B`, not real ≥1B runs):
  140. ```powershell
  141. pip install "WHL[training]" :: the default torch is already CPU-only
  142. ```
  143. > Verify XPU is live after setup:
  144. > ```powershell
  145. > python -c "import torch; print(torch.__version__, getattr(torch, 'xpu', None) is not None and torch.xpu.is_available())"
  146. > :: expect: 2.10.0+xpu True
  147. > ```
  148. ---
  149. ## 🤖 Want Q&A / auto-reports? Install Ollama first (optional but recommended)
  150. DeskMate's **natural-language Q&A (Ask)** and the **LLM apps under My Apps** (daily
  151. recap, todo extraction, meeting summaries…) need a local LLM server, **Ollama**. If you
  152. don't use these, you can **skip this section** — recording and search work fine without it.
  153. > **💡 Recommended: use the Ollama OpenVINO build on Intel hardware**
  154. > If you're on an Intel CPU / Arc GPU / Core Ultra (NPU), we strongly recommend
  155. > **Ollama-OV (OpenVINO backend)** — it runs models on Intel CPU/GPU/NPU, faster and
  156. > more power-efficient:
  157. > 👉 <https://github.com/zhaohb/ollama_openvino>
  158. > . On regular GPUs / other platforms, the official
  159. > Ollama works fine.
  160. **Option A — Ollama OpenVINO build (recommended on Intel)**
  161. 1. Follow <https://github.com/zhaohb/ollama_openvino> to get `ollama.exe` and start the
  162. service (see that repo's README).
  163. 2. Import a model in **OpenVINO IR format** as described there (get it from HuggingFace /
  164. ModelScope → write a `Modelfile` with `ModelBackend "OpenVINO"` → `ollama create <name> -f Modelfile`).
  165. 3. Put the model name into `config.toml` (below).
  166. **Option B — Official Ollama (other platforms / simplest)**
  167. 1. Install Ollama: <https://ollama.com/download> (it runs a local service in the
  168. background at `http://127.0.0.1:11434`).
  169. 2. Pull a model: `ollama pull qwen3`.
  170. **Final step for both options** — in `C:\Users\<your-username>\.deskmate\config.toml`,
  171. set the model name to the one you actually have:
  172. ```toml
  173. [ollama]
  174. base = "http://127.0.0.1:11434"
  175. model = "qwen3_8b_ov:v1" # OpenVINO build: the name you imported it as.
  176. # Official build: the name you pulled (e.g. qwen3).
  177. ```
  178. Then restart `deskmate ui` — you can now use Ask on the Home page and run reports under **My Apps**.
  179. ---
  180. ## ⚡ Advanced features (read when you need them)
  181. These are more specialized capabilities - **beginners can ignore them for now** and
  182. come back when needed:
  183. | Feature | One-liner | Docs |
  184. |---------|-----------|------|
  185. | OpenVINO-accelerated transcription | Run Whisper faster on Intel NPU/GPU | [docs/04-audio.md](docs/04-audio.md#whisper-backends) |
  186. | Live speech translation | Translate as you speak, shown in the UI | [docs/18-live-translation.md](docs/18-live-translation.md) |
  187. | Decision routing | A local model reads what is on screen and suggests the next step (learn / meeting / reply / book a room); you confirm in one click | [below](#-decision-routing-suggestions-you-confirm) |
  188. | Learning sessions | Start/end a study block (by hand, or confirm a suggestion), recap, review queue, knowledge graph | — |
  189. | Meetings | Sessions you start by hand or confirm from a suggestion: transcript, screen text, recap video, summary | [docs/09-meeting-workflow.md](docs/09-meeting-workflow.md) |
  190. | Gmail / Outlook integration | Search real mailboxes in Ask and apps | [docs/11-connections.md](docs/11-connections.md) |
  191. | Battery Saver | On battery, move background AI work and user-selected apps onto efficient cores; the SPA view and live status text are localized in Chinese/English | [docs/22-power-manager.md](docs/22-power-manager.md) |
  192. | Local LoRA fine-tuning | Train a small model on your data (incl. Intel iGPU) | [docs/16-learning-training.md](docs/16-learning-training.md) |
  193. | All technical design docs | Architecture of every module | [docs/README.md](docs/README.md) |
  194. **OpenVINO Whisper quick setup** (Intel devices):
  195. ```powershell
  196. pip install -e ".[audio-openvino,vad]"
  197. ```
  198. Then in `config.toml`:
  199. ```toml
  200. [audio]
  201. whisper_backend = "openvino_genai" # default is "onnx_cpu"
  202. openvino_device = "NPU" # NPU | GPU | CPU | AUTO
  203. ```
  204. The model is auto-downloaded from ModelScope on first use; if the chosen device fails to
  205. load, it automatically falls back to CPU.
  206. **LoRA training (Intel GPU)** has a one-click setup script — see
  207. [scripts/setup-intel-xpu.bat](scripts/setup-intel-xpu.bat) and [docs/16](docs/16-learning-training.md).
  208. ---
  209. ## 🧭 Decision routing (suggestions you confirm)
  210. DeskMate can watch what is happening — the foreground window, on-screen text, audio
  211. transcripts, key text, and whether you are in a meeting or a study session — and
  212. suggest what to do next. A **small local model** (`kev-0.8b`, run in-process; no
  213. cloud, no extra service) judges four things:
  214. | Suggestion | When it shows up | What *Confirm* does |
  215. |------------|------------------|---------------------|
  216. | **Start learning** | A course / lecture / tutorial is in the foreground | Opens a learning session. You end it yourself |
  217. | **Start meeting** | A live call UI is in the foreground (Meet, Teams, Zoom, Tencent Meeting, Feishu, DingTalk, …) | Opens a meeting recording. You end it yourself from the Meetings page |
  218. | **Reply to email** | The email you are reading asks you a question or a favour | Records your confirmation only (nothing is sent) |
  219. | **Book a meeting room** | A mail, chat, or call asks you to book a room | Records your confirmation only (nothing is booked); keeps a time / size hint |
  220. > **Nothing runs without your click.** Booking a room and replying to an email are not
  221. > automated yet — the confirmation is stored so a future executor can pick it up
  222. > (`GET /route/executions?status=confirmed`).
  223. ### How it works
  224. 1. You open a course page → a Windows notification appears (*"Looks like you are
  225. studying — start a learning session?"*). Email / room notifications show the mail
  226. subject (basic redaction applied; see `toast_show_subject`).
  227. 2. Click **Open** → the **Suggestions** page highlights the card.
  228. 3. Click **Confirm** (or **Not this**). Your answer is saved as feedback for later
  229. threshold tuning. Sessions you start this way are marked *From Suggestions*.
  230. It stays quiet on purpose: quiet hours (`[habits] quiet_hours`) are honoured, the same
  231. page or mail is only suggested once (2 h; 8 h after you said *Not this*), nothing is
  232. suggested while a learning session or meeting is already open, and terminals, editors,
  233. and the desktop / taskbar are never scored. Outside a meeting, audio is **not** used
  234. to decide on room booking, so a video playing in the background cannot trigger it.
  235. ### Turn it on
  236. Off by default. Install the extra, then enable it and restart DeskMate:
  237. ```powershell
  238. pip install -e ".[route]"
  239. ```
  240. If you already run a tuned PyTorch (for example an Intel XPU build), install `kev`
  241. without dependencies so it does not replace it — see the note above the `route`
  242. extra in `pyproject.toml`.
  243. ```toml
  244. [route]
  245. enabled = true # default: false
  246. # kev_device = "xpu" # Intel GPU; falls back to "cpu" automatically
  247. # executor_actions = ["start_learning", "start_meeting", "reply_email", "book_room"]
  248. # toast_show_subject = true # false: notifications stay generic, subjects only on the page
  249. # screen_excluded_apps = [...] # windows that are never scored (terminals, desktop, ...)
  250. [meeting]
  251. nudge_minutes = 15 # remind you that a meeting you started is still open (0 = off)
  252. ```
  253. The model loads on the first judgment (about 12 s, ~6 GB RAM on XPU; roughly 0.5 s per
  254. judgment afterwards; on CPU it is several times slower). Model files are read from
  255. your Hugging Face cache — set `HF_HUB_OFFLINE=1` to forbid downloads.
  256. ### Privacy
  257. Everything stays on your machine. Routing reads the same text DeskMate already stores
  258. (the redacted version when `[redact]` is enabled), and mail subjects are additionally
  259. masked for e-mail addresses and phone numbers before they are shown — also only when
  260. `[redact]` is enabled; names and project names are never masked. Pausing capture stops
  261. routing, and `/capture/forget` also removes routing data (and any session or meeting
  262. that was started from a suggestion in the forgotten span).
  263. ### Inspect it
  264. ```powershell
  265. deskmate route tail --follow # watch suggestions arrive
  266. ```
  267. ```
  268. GET /route/live POST /route/suggestions/{id}/execute
  269. GET /route/suggestions POST /route/suggestions/{id}/dismiss
  270. GET /route/executions?status=pending POST /route/suggestions/{id}/ack
  271. GET /route/intents GET /route/scores
  272. POST /route/feedback GET /route/segments
  273. ```
  274. ### Known limits
  275. - A **live webinar or livestream** can look exactly like a meeting on screen; you get
  276. one extra notification, never an automatic meeting.
  277. - The *reply to email* threshold is not calibrated yet and can miss a plain question.
  278. - Tested end to end on Chrome / Gmail and Google Meet; other call apps are covered by
  279. offline samples only.
  280. - Pages whose title keeps changing (live counters, progress in the title) may be
  281. suggested more than once.
  282. - Windows toasts appear under the PowerShell app name.
  283. Run the tests with `pytest tests/test_route_*.py`; the real-model contract tests are
  284. opt-in (`RUN_KEV_TESTS=1`).
  285. ---
  286. ## Requirements
  287. - **Windows 10 / 11**
  288. - **Python 3.10+**
  289. - Optional, by feature:
  290. - **OCR**: `ocr-winrt` needs nothing extra; `ocr-tesseract` needs Tesseract on PATH;
  291. `ocr-rapidocr` just needs the extra (best for Chinese)
  292. - **Audio**: a microphone, or a system-audio loopback (WASAPI) device
  293. - **OpenVINO acceleration**: an Intel CPU; an Intel Core Ultra (NPU) or Arc/iGPU
  294. unlocks NPU/GPU inference
  295. - **Q&A & apps**: a local [Ollama](http://127.0.0.1:11434) server
  296. - **LoRA training**: a GPU is recommended (CPU works but is slow)
  297. - **Decision routing**: the `route` extra; an Intel GPU (XPU) is recommended, CPU
  298. works but is several times slower; the local model needs about 6 GB of RAM
  299. ---
  300. ## Everyday usage
  301. ### Browser UI
  302. ```powershell
  303. deskmate ui # record + API + auto-open /deskmate
  304. deskmate ui --no-run-daemon # view existing data only, don't start new recording
  305. ```
  306. | Page | What it does |
  307. |------|--------------|
  308. | Home | Health status, recent activity, **Ask** (natural-language queries) |
  309. | Apps | Run built-in and plugin LLM apps; configure schedules and time ranges |
  310. | Email | Connect Gmail / Outlook and review mailbox integration status |
  311. | Timeline | Browse the screenshot timeline |
  312. | Transcripts | Audio transcriptions |
  313. | Translate | Live speech translation controls and translated transcript stream |
  314. | Events | Keyboard / mouse / clipboard / window-focus events |
  315. | Suggestions | Decision routing: live status, items waiting for your confirmation, intents, scores, correct / not-this feedback |
  316. | Learning | Start/end a study session (or confirm one from Suggestions); recap, review queue, knowledge graph |
  317. | Todos | Action items extracted from email + meetings |
  318. | Meetings | Start/end a session (or confirm one from Suggestions); audio + screen text, recap video, summary |
  319. | Capture | Capture-control view for recording sources and runtime toggles |
  320. | Training | Local LoRA fine-tuning |
  321. | Model Service | Start/stop local model backends and inspect service logs |
  322. | Diagnostics | Environment self-checks for common setup issues |
  323. | Reminders | Proactive habit reminders, acknowledgement, and history |
  324. | Battery Saver | EcoQoS battery saver for background AI workers and selected apps; UI text is localized in Chinese/English |
  325. | Settings | Config and monitors |
  326. ### CLI
  327. ```powershell
  328. deskmate serve # HTTP API only (records by default)
  329. deskmate record # recorder only
  330. deskmate capture-once # capture one frame manually
  331. deskmate route tail # latest decision-routing suggestions (add --follow / --intents)
  332. deskmate search "query" # keyword search via the API
  333. deskmate mcp # MCP server for AI agents (see "MCP server" below; API must be running)
  334. ```
  335. Split API and recorder into two processes:
  336. ```powershell
  337. deskmate record
  338. deskmate serve --no-run-daemon
  339. ```
  340. ### LLM apps (My Apps)
  341. Built-in apps live in `deskmate/apps/` (and ship inside the installed package);
  342. your own apps go in `~/.deskmate/apps/plugins/`. Run them from the **My Apps**
  343. page or the CLI. Full list, examples, and how to write your own in
  344. [deskmate/apps/README.md](deskmate/apps/README.md).
  345. | App | Purpose |
  346. |-----|---------|
  347. | `todo-list` | Unified checkbox todos from **email + meetings** |
  348. | `meeting-summary` | Summarize a meeting you started/stopped |
  349. | `user-learning` | Recap a study session (concepts + notes) |
  350. | `email-digest` | Inbox overview |
  351. | `email-compose` | Draft / reply via Gmail or Outlook |
  352. | `day-recap` / `time-breakdown` / `ai-habits` | Daily recap / time split / AI-tool usage habits |
  353. | `user-profile` / `habit-report` | Multi-day user profile / routine (default: last 7 days) |
  354. > These apps need both **Ollama and the DeskMate API** running.
  355. ---
  356. ## Configuration
  357. Config file: `C:\Users\<your-username>\.deskmate\config.toml` (auto-created on first run).
  358. Main sections: `[capture]`, `[a11y]`, `[ocr]`, `[audio]`, `[ollama]`, `[redact]`,
  359. `[filters]`, `[server]`, `[route]`, `[learning]`, `[meeting]`
  360. Override with environment variables (prefix `DESKMATE_`):
  361. ```powershell
  362. $env:DESKMATE_SERVER__PORT = "4040"
  363. $env:DESKMATE_AUDIO__ENABLED = "true"
  364. ```
  365. ## Where your data lives
  366. ```
  367. C:\Users\<your-username>\.deskmate\
  368. ├── config.toml # configuration
  369. ├── data.db # SQLite database (full-text search; also routing suggestions and feedback)
  370. ├── frames\ # screenshots
  371. ├── audio\ # audio chunks (when enabled)
  372. ├── videos\ # video chunks
  373. ├── checkpoints\ # LoRA training artifacts (when training)
  374. ├── apps\ # LLM app output reports; apps\plugins\ for your own apps
  375. └── logs\ # logs
  376. ```
  377. **To wipe and start over**: stop DeskMate, then delete `data.db` and the folders above.
  378. ## API
  379. Default base URL: **http://127.0.0.1:3030**. Full endpoint list is in the running
  380. server's `/docs` (OpenAPI). Common ones:
  381. ```
  382. GET /health GET /search?q=... POST /ask
  383. GET /frames POST /capture GET /meetings
  384. GET /todos POST /todos PATCH /todos/{id}
  385. GET /route/live GET /route/suggestions POST /route/suggestions/{id}/execute
  386. ```
  387. Decision-routing endpoints are listed in [Decision routing](#-decision-routing-suggestions-you-confirm).
  388. ---
  389. ## 🔌 MCP server (use DeskMate from AI agents)
  390. DeskMate ships an [MCP](https://modelcontextprotocol.io) stdio server so other AI
  391. agents (Claude Desktop, IDE assistants, custom clients) can query your local
  392. activity **and** drive DeskMate's Q&A and report apps. It's a thin proxy over the
  393. local HTTP API, so **the DeskMate API must already be running** (`deskmate serve`
  394. or `deskmate ui`).
  395. **Install & launch**
  396. ```powershell
  397. pip install -e ".[mcp]" # or add `mcp` to your extras combo
  398. deskmate serve # in one terminal — the API on :3030
  399. deskmate mcp # in another — the MCP stdio server
  400. ```
  401. **Tools exposed** (all prefixed `deskmate_` so they don't collide with other MCP servers):
  402. | Tool | Kind | What it does |
  403. |------|------|--------------|
  404. | `deskmate_search` | read | Full-text search over frames (OCR + a11y text), UI events, audio transcripts |
  405. | `deskmate_recent_frames` | read | List the latest captured frames |
  406. | `deskmate_recent_events` | read | List latest UI events (clicks, focus, key text, clipboard) |
  407. | `deskmate_health` | read | Daemon liveness + counters |
  408. | `deskmate_capture_once` | action | Trigger a paired capture now |
  409. | `deskmate_ask` | action | Natural-language Q&A: an LLM searches your local context and answers (slow) |
  410. | `deskmate_list_apps` | read | List the report apps (day-recap, standup-update, todo-list, meeting-summary, email-compose, …) |
  411. | `deskmate_run_app` | action | Run a report app and return its result (slow; params vary per app) |
  412. | `deskmate_learning_sessions` | read | List recent study sessions |
  413. | `deskmate_learning_reviews` | read | Due review cards (spaced repetition) |
  414. | `deskmate_list_app_outputs` | read | List an app's past run outputs |
  415. | `deskmate_get_app_output` | read | Fetch one output file (markdown/json) of a past run |
  416. > `deskmate_ask` and `deskmate_run_app` invoke a local LLM, so they need **Ollama +
  417. > the API** running and can take tens of seconds to minutes.
  418. ### 💡 Why connect it to Claude Code / Cursor?
  419. Once DeskMate is registered, **your coding assistant knows what you've been doing
  420. locally** — no copy-pasting context. While you work in Claude Code or Cursor you can ask:
  421. **deskmate_ask** — natural-language Q&A over your activity
  422. - "Use deskmate to recap the last hour: which apps I used, how long on each, and what I was doing — as a timeline."
  423. - "Ask deskmate: between 2pm and 4pm today, what task was I mainly working on, and which files/windows were involved?"
  424. - "From my local activity record (deskmate), what was I last reading about in the browser?"
  425. **deskmate_search** — full-text search over OCR / UI / audio
  426. - "Use deskmate to search my screen OCR, UI text and audio transcripts for 'recursive doubling', list the hits by time with the app/window they appeared in."
  427. - "Use deskmate to find frames in the last 3 hours whose OCR text contains 'error' or 'exception', and tell me which app each was in."
  428. - "Use deskmate to find which .py files I opened in VS Code today."
  429. **deskmate_list_apps / run_app** — report apps
  430. - "Use deskmate to list all available report apps and what each one does."
  431. - "Use deskmate to run the time-breakdown report for the last hour."
  432. - "Use deskmate to run day-recap and summarize what I did today."
  433. - "Use deskmate to run standup-update: what I did, what's next, and any blockers."
  434. - "Use deskmate to run user-learning for my latest study session."
  435. **deskmate_learning_sessions / reviews** — study recap
  436. - "Use deskmate to list my recent study sessions and what's due for review."
  437. **deskmate_recent_events / recent_frames** — raw activity stream
  438. - "Use deskmate to show my last 50 UI events (clicks/focus/typing/clipboard) and summarize what I was just doing."
  439. - "Use deskmate to list the last 20 captured frames and tell me which apps/windows I've been switching between."
  440. **deskmate_health / capture_once** — status & instant capture
  441. - "Use deskmate to check whether the recorder daemon is healthy and how many frames/transcripts it has captured."
  442. - "Use deskmate to capture the screen right now and tell me what's on it."
  443. **deskmate_list_app_outputs / get_app_output** — past reports
  444. - "Use deskmate to list day-recap's past report runs and show me the contents of the latest one."
  445. The assistant reads your **local activity record** on demand and answers in-context,
  446. entirely on your machine.
  447. ### Register the server
  448. Point `command` at the Python env that has the `mcp` extra installed.
  449. **Claude Code** (CLI):
  450. ```bash
  451. claude mcp add deskmate -e DESKMATE_API=http://127.0.0.1:3030 \
  452. -- C:/path/to/env/python.exe -m deskmate mcp
  453. # verify: claude mcp list (should show deskmate ✓ connected)
  454. ```
  455. **Cursor** — add to `~/.cursor/mcp.json` (global) or `<project>/.cursor/mcp.json`:
  456. ```json
  457. {
  458. "mcpServers": {
  459. "deskmate": {
  460. "command": "C:/path/to/env/python.exe",
  461. "args": ["-m", "deskmate", "mcp"],
  462. "env": { "DESKMATE_API": "http://127.0.0.1:3030" }
  463. }
  464. }
  465. }
  466. ```
  467. **Claude Desktop** — same block in `claude_desktop_config.json`.
  468. > After editing config, **fully restart** the client — MCP servers are only loaded
  469. > at startup.
  470. ### Notes
  471. - **`DESKMATE_API`** (optional) overrides the API base URL; defaults to
  472. `http://127.0.0.1:3030`.
  473. - **Slow tools / timeouts:** `deskmate_ask` and `deskmate_run_app` invoke a local
  474. LLM and can take minutes. The server-side budget defaults to **30 min** and is
  475. configurable via `DESKMATE_MCP_ASK_TIMEOUT` / `DESKMATE_MCP_RUN_APP_TIMEOUT`
  476. (seconds, in the server's `env`). These tools also stream MCP progress every
  477. ~15s, which keeps most clients from cutting the call off; if your client still
  478. times out, raise its tool timeout with `MCP_TOOL_TIMEOUT` (milliseconds; e.g.
  479. `1800000` for 30 min).
  480. - **Behind a corporate proxy?** No action needed — the MCP server talks only to
  481. the local API and deliberately ignores `HTTP(S)_PROXY`, so `127.0.0.1` traffic
  482. is never routed through a proxy.
  483. - The MCP server and the daemon are **independent processes** coupled only by the
  484. API, so the MCP server can run from a different environment than the daemon —
  485. it just needs the `mcp` extra.
  486. ---
  487. ## FAQ
  488. - **Install error / a feature is missing?** Most likely the matching extra isn't
  489. installed. Go back to [Install features as needed](#-install-features-as-needed-submodule-dependencies)
  490. and add that one.
  491. - **Ask / My Apps does nothing or errors?** Check that Ollama is running
  492. (`ollama list` should list models) and that `[ollama] model` in `config.toml` matches
  493. a model you actually have. On Intel hardware, use the
  494. [Ollama OpenVINO build](https://github.com/zhaohb/ollama_openvino) (the default model
  495. name `qwen3_8b_ov:v1` is meant for it).
  496. - **`deskmate` not found in a new terminal?** You didn't activate the virtual
  497. environment. `cd` into the project folder, then run `.\.venv\Scripts\Activate.ps1`.
  498. - **Decision routing never suggests anything?** Check that the `route` extra is
  499. installed, that `[route] enabled = true` is set in `config.toml`, and that you
  500. restarted DeskMate. `GET /route/live` shows whether the model loaded, and
  501. `deskmate route tail --follow` shows suggestions as they arrive. It stays silent in
  502. quiet hours, in terminals / editors / the desktop, and while a session or meeting is
  503. already open.
  504. - **Want the internals / finer config of a feature?** It's all in [docs/](docs/README.md),
  505. numbered by module.
  506. ## License
  507. **MIT** — see [LICENSE](LICENSE).
  508. DeskMate is free to use, copy, modify, merge, publish, distribute, sublicense,
  509. and/or sell copies of the Software, subject to the MIT license terms.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.