Skip to content
Back to search

qmd

MarketplaceWorks with Claude CodeGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/tobi-qmd/badge.svg)](/listings/tobi-qmd)

HTML

<a href="/listings/tobi-qmd"><img src="/api/listings/tobi-qmd/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# Install globally (Node or Bun)
npm install -g @tobilu/qmd
# or
bun install -g @tobilu/qmd

# Or run directly
npx @tobilu/qmd ...
bunx @tobilu/qmd ...

# Create collections for your notes, docs, and meeting transcripts
qmd collection add ~/notes --name notes
qmd collection add ~/Documents/meetings --name meetings
qmd collection add ~/work/docs --name docs

# Add context to help with search results, each piece of context will be returned when matching sub documents are returned. This works as a tree. This is the key feature of QMD as it allows LLMs to make much better contextual choices when selecting documents. Don't sleep on it!
qmd context add qmd://notes "Personal notes and ideas"
qmd context add qmd://meetings "Meeting transcripts and notes"
qmd context add qmd://docs "Work documentation"

# Generate embeddings for semantic search
qmd embed

# Search across everything
qmd search "project timeline"           # Fast keyword search
qmd vsearch "how to deploy"             # Semantic search
qmd query "quarterly planning process"  # Hybrid + reranking (best quality)

# Get a specific document
qmd get "meetings/2024-01-15.md"

# Get a document by docid (shown in search results)
qmd get "#abc123"

# Get multiple documents by glob pattern
qmd multi-get "journals/2025-05*.md"

# Search within a specific collection
qmd search "API" -c notes

# Export all matches for an agent
qmd search "API" --all --files --min-score 0.3

Using with AI Agents

QMD's `--json` and `--files` output formats are designed for agentic workflows:

# Get structured results for an LLM
qmd search "authentication" --json -n 10

# List all relevant files above a threshold
qmd query "error handling" --all --files --min-score 0.4

# Retrieve full document content
qmd get "docs/api-reference.md" --full

MCP Server

Although the tool works perfectly fine when you just tell your agent to use it on the command line, it also exposes an MCP (Model Context Protocol) server for tighter integration.

**Tools exposed:** - `query` — Search with typed sub-queries (`lex`/`vec`/`hyde`), combined via RRF + reranking - `get` — Retrieve a document by path or docid (with fuzzy matching suggestions) - `multi_get` — Batch retrieve by glob pattern, comma-separated list, or docids - `status` — Index health and collection info, including each collection's top metadata keys - `metadata` — Discover metadata keys, types, and value counts to filter on

**Claude Desktop configuration** (`~/Library/Application Support/Claude/claude_desktop_config.json`):

{
  "mcpServers": {
    "qmd": {
      "command": "qmd",
      "args": ["mcp"]
    }
  }
}

**Claude Code** — Install the plugin (recommended):

claude plugin marketplace add tobi/qmd
claude plugin install qmd@qmd

Or configure MCP manually in `~/.claude/settings.json`:

{
  "mcpServers": {
    "qmd": {
      "command": "qmd",
      "args": ["mcp"]
    }
  }
}

HTTP Transport

By default, QMD's MCP server uses stdio (launched as a subprocess by each client). For a shared, long-lived server that avoids repeated model loading, use the HTTP transport:

# Foreground (Ctrl-C to stop)
qmd mcp --http                    # localhost:8181
qmd mcp --http --port 8080        # custom port
qmd mcp --http --host 0.0.0.0     # bind all interfaces (e.g. container probes)

# Background daemon
qmd mcp --http --daemon           # start, writes PID to ~/.cache/qmd/mcp.pid
qmd mcp stop                      # stop via PID file
qmd status                        # shows "MCP: running (PID ...)" when active

MCP shutdown allows active requests five seconds to finish. After that it cancels them; a standalone server exits nonzero if work remains. Embedded callers receive a timeout error while resource cleanup continues safely. An external supervisor is still needed if native code blocks the event loop.

The server binds to `localhost` by default. Pass `--host` (or set the `QMD_HOST`

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 29 lines
  1. {
  2. "name": "qmd",
  3. "owner": {
  4. "name": "tobi",
  5. "email": "tobi@lutke.com"
  6. },
  7. "plugins": [
  8. {
  9. "name": "qmd",
  10. "source": "./skills",
  11. "description": "Search and retrieve documents from local markdown files.",
  12. "version": "2.9.0",
  13. "author": {
  14. "name": "tobi",
  15. "email": "tobi@lutke.com"
  16. },
  17. "repository": "https://github.com/tobi/qmd",
  18. "license": "MIT",
  19. "keywords": ["markdown", "search", "qmd"],
  20. "skills": ["./qmd", "./release"],
  21. "mcpServers": {
  22. "qmd": {
  23. "command": "qmd",
  24. "args": ["mcp"]
  25. }
  26. }
  27. }
  28. ]
  29. }
README.md · README · 1,794 lines
  1. # QMD - Query Markup Documents
  2. An on-device search engine for everything you need to remember. Index your markdown notes, meeting transcripts, documentation, and knowledge bases. Search with keywords or natural language. Ideal for your agentic flows.
  3. QMD combines BM25 full-text search, vector semantic search, and LLM re-ranking—all running locally via node-llama-cpp with GGUF models.
  4. ```mermaid
  5. flowchart LR
  6. Q[User Query] --> X[Query Expansion]
  7. Q --> FTS[BM25 Search]
  8. Q --> VS[Vector Search]
  9. X --> HYDE[HyDE]
  10. X --> VEC[Vec dense sentences]
  11. X --> LEX[Lex BM25 keywords]
  12. HYDE --> VS
  13. VEC --> VS
  14. LEX --> FTS
  15. VS --> RRF[Reciprocal Rank Fusion]
  16. FTS --> RRF
  17. RRF --> RR[LLM Reranker]
  18. RR --> OUT[Final ranked results]
  19. ```
  20. Typed expansions are routed exclusively: `lex` → BM25/FTS, `vec` and `hyde` → vector search. The original query is sent to both backends, then fused with RRF and reranked.
  21. You can read more about QMD's progress in the [CHANGELOG](CHANGELOG.md).
  22. ## Quick Start
  23. ```sh
  24. # Install globally (Node or Bun)
  25. npm install -g @tobilu/qmd
  26. # or
  27. bun install -g @tobilu/qmd
  28. # Or run directly
  29. npx @tobilu/qmd ...
  30. bunx @tobilu/qmd ...
  31. # Create collections for your notes, docs, and meeting transcripts
  32. qmd collection add ~/notes --name notes
  33. qmd collection add ~/Documents/meetings --name meetings
  34. qmd collection add ~/work/docs --name docs
  35. # Add context to help with search results, each piece of context will be returned when matching sub documents are returned. This works as a tree. This is the key feature of QMD as it allows LLMs to make much better contextual choices when selecting documents. Don't sleep on it!
  36. qmd context add qmd://notes "Personal notes and ideas"
  37. qmd context add qmd://meetings "Meeting transcripts and notes"
  38. qmd context add qmd://docs "Work documentation"
  39. # Generate embeddings for semantic search
  40. qmd embed
  41. # Search across everything
  42. qmd search "project timeline" # Fast keyword search
  43. qmd vsearch "how to deploy" # Semantic search
  44. qmd query "quarterly planning process" # Hybrid + reranking (best quality)
  45. # Get a specific document
  46. qmd get "meetings/2024-01-15.md"
  47. # Get a document by docid (shown in search results)
  48. qmd get "#abc123"
  49. # Get multiple documents by glob pattern
  50. qmd multi-get "journals/2025-05*.md"
  51. # Search within a specific collection
  52. qmd search "API" -c notes
  53. # Export all matches for an agent
  54. qmd search "API" --all --files --min-score 0.3
  55. ```
  56. ### Using with AI Agents
  57. QMD's `--json` and `--files` output formats are designed for agentic workflows:
  58. ```sh
  59. # Get structured results for an LLM
  60. qmd search "authentication" --json -n 10
  61. # List all relevant files above a threshold
  62. qmd query "error handling" --all --files --min-score 0.4
  63. # Retrieve full document content
  64. qmd get "docs/api-reference.md" --full
  65. ```
  66. ### MCP Server
  67. Although the tool works perfectly fine when you just tell your agent to use it on the command line, it also exposes an MCP (Model Context Protocol) server for tighter integration.
  68. **Tools exposed:**
  69. - `query` — Search with typed sub-queries (`lex`/`vec`/`hyde`), combined via RRF + reranking
  70. - `get` — Retrieve a document by path or docid (with fuzzy matching suggestions)
  71. - `multi_get` — Batch retrieve by glob pattern, comma-separated list, or docids
  72. - `status` — Index health and collection info, including each collection's top metadata keys
  73. - `metadata` — Discover metadata keys, types, and value counts to filter on
  74. **Claude Desktop configuration** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
  75. ```json
  76. {
  77. "mcpServers": {
  78. "qmd": {
  79. "command": "qmd",
  80. "args": ["mcp"]
  81. }
  82. }
  83. }
  84. ```
  85. **Claude Code** — Install the plugin (recommended):
  86. ```bash
  87. claude plugin marketplace add tobi/qmd
  88. claude plugin install qmd@qmd
  89. ```
  90. Or configure MCP manually in `~/.claude/settings.json`:
  91. ```json
  92. {
  93. "mcpServers": {
  94. "qmd": {
  95. "command": "qmd",
  96. "args": ["mcp"]
  97. }
  98. }
  99. }
  100. ```
  101. #### HTTP Transport
  102. By default, QMD's MCP server uses stdio (launched as a subprocess by each client). For a shared, long-lived server that avoids repeated model loading, use the HTTP transport:
  103. ```sh
  104. # Foreground (Ctrl-C to stop)
  105. qmd mcp --http # localhost:8181
  106. qmd mcp --http --port 8080 # custom port
  107. qmd mcp --http --host 0.0.0.0 # bind all interfaces (e.g. container probes)
  108. # Background daemon
  109. qmd mcp --http --daemon # start, writes PID to ~/.cache/qmd/mcp.pid
  110. qmd mcp stop # stop via PID file
  111. qmd status # shows "MCP: running (PID ...)" when active
  112. ```
  113. MCP shutdown allows active requests five seconds to finish. After that it
  114. cancels them; a standalone server exits nonzero if work remains. Embedded
  115. callers receive a timeout error while resource cleanup continues safely.
  116. An external supervisor is still needed if native code blocks the event loop.
  117. The server binds to `localhost` by default. Pass `--host` (or set the `QMD_HOST`
  118. environment variable) to override — `--host 0.0.0.0` is useful when the server
  119. runs in a container and a liveness probe connects from a non-loopback address.
  120. The HTTP server exposes two endpoints:
  121. - `POST /mcp` — MCP Streamable HTTP (stateless JSON and SSE responses)
  122. - `POST /query` (alias `/search`) — structured search without the MCP protocol. Accepts the same optional `filter` object as the `query` tool (invalid filters return `400`); see [Metadata Filtering](#metadata-filtering)
  123. - `POST /metadata` — metadata discovery without the MCP protocol. Same body as the `metadata` tool (invalid filters return `400`); see [Metadata Discovery](#metadata-discovery)
  124. - `GET /health` — liveness check with uptime
  125. ##### Origin and Host validation
  126. Every request is screened before routing: a request carrying an `Origin` header
  127. that does not name a loopback address is rejected with `403`, as is a `Host`
  128. header naming something other than the address the server is bound to. This is
  129. what stops a web page you visit from reading your index through DNS rebinding —
  130. loopback binding alone does not, since the browser makes the request from your
  131. own machine.
  132. Requests without an `Origin` header — curl, MCP clients, editors — are
  133. unaffected, which covers every normal local client.
  134. | Variable | Effect |
  135. |----------|--------|
  136. | `QMD_ALLOWED_ORIGINS` | Comma-separated origins to accept in addition to loopback, e.g. `https://notes.internal`. Set to `*` to disable the check entirely. |
  137. | `QMD_ALLOWED_HOSTS` | Comma-separated `Host` values to accept in addition to loopback and the bind address. |
  138. `--host 0.0.0.0` cannot know which `Host` values are legitimate, so it skips the
  139. host check and warns at startup. Set `QMD_ALLOWED_HOSTS` to re-enable it, and
  140. remember the endpoints are unauthenticated — put your own auth in front of a
  141. server that is reachable off-host.
  142. LLM models stay loaded in VRAM across requests. Embedding/reranking contexts are disposed after 5 min idle and transparently recreated on the next request (~1s penalty, models remain loaded).
  143. Point any MCP client at `http://localhost:8181/mcp` to connect.
  144. #### MCP Tool Parameters
  145. | Tool | Parameter | Type | Notes |
  146. |------|-----------|------|-------|
  147. | `query` | `searches` | array | Typed sub-queries (`lex`/`vec`/`hyde`), 1–10. **Required.** First gets 2x weight. |
  148. | `query` | `collections` | string[] | Filter by collection names (OR). **Array only** — singular `collection` is silently ignored. |
  149. | `query` | `filter` | object | Metadata filter (recursive `operator`-discriminated JSON AST; see [Metadata Filtering](#metadata-filtering)) |
  150. | `query` | `intent` | string | Disambiguation context (does not search on its own) |
  151. | `query` | `limit` | number | Max results (default 10) |
  152. | `query` | `minScore` | number | Minimum relevance 0–1 (default 0) |
  153. | `query` | `candidateLimit` | number | Max candidates to rerank (default 40) |
  154. | `query` | `rerank` | boolean | Run LLM reranking (default **true**); set false for RRF-only |
  155. | `get` | `file` | string | Path, docid (`#abc123`), or `path:from:count` (e.g. `#abc123:120:40`) |
  156. | `get` | `fromLine` | number | Start line (1-indexed); overrides the `:from` suffix |
  157. | `get` | `maxLines` | number | Limit returned lines |
  158. | `get` | `lineNumbers` | boolean | Prefix lines with numbers (default **true**) |
  159. | `multi_get` | `pattern` | string | Glob pattern or comma-separated list |
  160. | `multi_get` | `maxBytes` | number | Skip files larger than N (default 10240) |
  161. | `multi_get` | `maxLines` | number | Limit lines per file |
  162. | `multi_get` | `lineNumbers` | boolean | Prefix lines with numbers (default **true**) |
  163. | `metadata` | `collections` | string[] | Restrict discovery to collection names (default: the collections `query` searches) |
  164. | `metadata` | `match` | object | Report only metadata entries matching this condition (same AST as `filter`, with `field` naming the entry's `key` or `value`) |
  165. | `metadata` | `filter` | object | Count only documents matching this filter (same AST as `query`) |
  166. | `metadata` | `keyLimit` | number | Keys reported (default 50). `totalKeys` and `remainingKeys` describe the rest |
  167. | `metadata` | `keyOffset` | number | Keys skipped before the window, in report order (default 0) |
  168. | `metadata` | `valueLimit` | number | Values reported per key and type (default 10). `remainingValues` reports the rest |
  169. | `metadata` | `valueOffset` | number | Values skipped per key and type before the window, in `sort` order (default 0) |
  170. | `metadata` | `sort` | string | `count` (default) or `value` |
  171. | `metadata` | `minCount` | number | Hide values held by fewer documents (default 1) |
  172. Unknown parameters are silently ignored (not rejected) — double-check names if
  173. results seem unscoped. The HTTP `/query` and `/search` endpoints return
  174. `qmd://collection/path` URIs in the `file` field, matching the CLI and MCP output.
  175. ### SDK / Library Usage
  176. Use QMD as a library in your own Node.js or Bun applications.
  177. #### Installation
  178. ```sh
  179. npm install @tobilu/qmd
  180. ```
  181. #### Quick Start
  182. ```typescript
  183. import { createStore } from '@tobilu/qmd'
  184. const store = await createStore({
  185. dbPath: './my-index.sqlite',
  186. config: {
  187. collections: {
  188. docs: { path: '/path/to/docs', pattern: '**/*.md' },
  189. },
  190. },
  191. })
  192. const results = await store.search({ query: "authentication flow" })
  193. console.log(results.map(r => `${r.title} (${Math.round(r.score * 100)}%)`))
  194. await store.close()
  195. ```
  196. Each store owns its configuration source. `close()` rejects new SDK operations
  197. and waits for admitted operations before closing the database; a stuck operation
  198. can therefore delay shutdown indefinitely. Access through `store.internal`
  199. bypasses this lifecycle.
  200. SDK `embed()` and the CLI share an exclusive per-index lock. A concurrent SDK
  201. call rejects with `EmbedLockBusyError`; the CLI reports that embedding is busy.
  202. Older QMD versions do not use this lock and must not embed concurrently.
  203. #### Store Creation
  204. `createStore()` accepts three modes:
  205. ```typescript
  206. import { createStore } from '@tobilu/qmd'
  207. // 1. Inline config — no files needed besides the DB
  208. const store = await createStore({
  209. dbPath: './index.sqlite',
  210. config: {
  211. collections: {
  212. docs: { path: '/path/to/docs', pattern: '**/*.md' },
  213. notes: { path: '/path/to/notes' },
  214. },
  215. },
  216. })
  217. // 2. YAML config file — collections defined in a file
  218. const store2 = await createStore({
  219. dbPath: './index.sqlite',
  220. configPath: './qmd.yml',
  221. })
  222. // 3. DB-only — reopen a previously configured store
  223. const store3 = await createStore({ dbPath: './index.sqlite' })
  224. ```
  225. #### Search
  226. The unified `search()` method handles both simple queries and pre-expanded structured queries:
  227. ```typescript
  228. // Simple query — auto-expanded via LLM, then BM25 + vector + reranking
  229. const results = await store.search({ query: "authentication flow" })
  230. // With options
  231. const results2 = await store.search({
  232. query: "rate limiting",
  233. intent: "API throttling and abuse prevention",
  234. collection: "docs",
  235. limit: 5,
  236. minScore: 0.3,
  237. explain: true,
  238. })
  239. // Pre-expanded queries — skip auto-expansion, control each sub-query
  240. const results3 = await store.search({
  241. queries: [
  242. { type: 'lex', query: '"connection pool" timeout -redis' },
  243. { type: 'vec', query: 'why do database connections time out under load' },
  244. ],
  245. collections: ["docs", "notes"],
  246. })
  247. // Skip reranking for faster results
  248. const fast = await store.search({ query: "auth", rerank: false })
  249. // Metadata filter — every returned result satisfies it (also available on
  250. // searchLex() and searchVector()); results expose indexed metadata via
  251. // r.metadata. See "Metadata Filtering" for the full grammar.
  252. const published = await store.search({
  253. query: "authentication flow",
  254. filter: {
  255. operator: "and",
  256. operands: [
  257. { field: "topics", operator: "all", value: ["typescript"] },
  258. { field: "status", operator: "ne", value: "draft" },
  259. ],
  260. },
  261. })
  262. ```
  263. For direct backend access:
  264. ```typescript
  265. // BM25 keyword search (fast, no LLM)
  266. const lexResults = await store.searchLex("auth middleware", { limit: 10 })
  267. // Vector similarity search (embedding model, no reranking)
  268. const vecResults = await store.searchVector("how users log in", { limit: 10 })
  269. // Manual query expansion for full control
  270. const expanded = await store.expandQuery("auth flow", { intent: "user login" })
  271. const results4 = await store.search({ queries: expanded })
  272. ```
  273. #### Retrieval
  274. ```typescript
  275. // Get a document by path or docid
  276. const doc = await store.get("docs/readme.md")
  277. const byId = await store.get("#abc123")
  278. if (!("error" in doc)) {
  279. console.log(doc.title, doc.displayPath, doc.context)
  280. }
  281. // Get document body with line range
  282. const body = await store.getDocumentBody("docs/readme.md", {
  283. fromLine: 50,
  284. maxLines: 100,
  285. })
  286. // Batch retrieve by glob or comma-separated list
  287. const { docs, errors } = await store.multiGet("docs/**/*.md", {
  288. maxBytes: 20480,
  289. })
  290. ```
  291. #### Collections
  292. ```typescript
  293. // Add a collection
  294. await store.addCollection("myapp", {
  295. path: "/src/myapp",
  296. pattern: "**/*.ts",
  297. ignore: ["node_modules/**", "*.test.ts"],
  298. })
  299. // List collections with document stats
  300. const collections = await store.listCollections()
  301. // => [{ name, pwd, glob_pattern, doc_count, active_count, last_modified, includeByDefault }]
  302. // Get names of collections included in queries by default
  303. const defaults = await store.getDefaultCollectionNames()
  304. // Remove / rename
  305. await store.removeCollection("myapp")
  306. await store.renameCollection("old-name", "new-name")
  307. ```
  308. #### Context
  309. Context adds descriptive metadata that improves search relevance and is returned alongside results:
  310. ```typescript
  311. // Add context for a path within a collection
  312. await store.addContext("docs", "/api", "REST API reference documentation")
  313. // Set global context (applies to all collections)
  314. await store.setGlobalContext("Internal engineering documentation")
  315. // List all contexts
  316. const contexts = await store.listContexts()
  317. // => [{ collection, path, context }]
  318. // Remove context
  319. await store.removeContext("docs", "/api")
  320. await store.setGlobalContext(undefined) // clear global
  321. ```
  322. #### Indexing
  323. [Collection processors](docs/PROCESSORS.md) optionally turn each source file into multiple stable-key Markdown index documents, through explicit SDK callbacks or trusted CLI commands.
  324. ```typescript
  325. // Re-index collections by scanning the filesystem
  326. const result = await store.update({
  327. collections: ["docs"], // optional — defaults to all
  328. onProgress: ({ collection, file, current, total }) => {
  329. console.log(`[${collection}] ${current}/${total} ${file}`)
  330. },
  331. })
  332. // => { collections, indexed, updated, unchanged, removed, needsEmbedding }
  333. // Generate vector embeddings
  334. const embedResult = await store.embed({
  335. force: false, // true to re-embed everything
  336. chunkStrategy: "auto", // "regex" (default) or "auto" (AST for code files)
  337. onProgress: ({ current, total, collection }) => {
  338. console.log(`Embedding ${current}/${total}`)
  339. },
  340. })
  341. ```
  342. #### Types
  343. Key types exported for SDK consumers:
  344. ```typescript
  345. import type {
  346. QMDStore, // The store interface
  347. SearchOptions, // Options for search()
  348. LexSearchOptions, // Options for searchLex()
  349. VectorSearchOptions, // Options for searchVector()
  350. HybridQueryResult, // Search result with score, snippet, context
  351. SearchResult, // Result from searchLex/searchVector
  352. ExpandedQuery, // Typed sub-query { type: 'lex'|'vec'|'hyde', query }
  353. DocumentResult, // Document metadata + body
  354. DocumentNotFound, // Error with similarFiles suggestions
  355. MultiGetResult, // Batch retrieval result
  356. UpdateProgress, // Progress callback info for update()
  357. UpdateResult, // Aggregated update result
  358. EmbedProgress, // Progress callback info for embed()
  359. EmbedResult, // Embedding result
  360. StoreOptions, // createStore() options
  361. CollectionConfig, // Inline config shape
  362. IndexStatus, // From getStatus()
  363. IndexHealthInfo, // From getIndexHealth()
  364. } from '@tobilu/qmd'
  365. ```
  366. Utility exports:
  367. ```typescript
  368. import {
  369. extractSnippet, // Extract a relevant snippet from text
  370. addLineNumbers, // Add line numbers to text
  371. DEFAULT_MULTI_GET_MAX_BYTES, // Default max file size for multiGet (64KB)
  372. Maintenance, // Database maintenance operations
  373. } from '@tobilu/qmd'
  374. ```
  375. #### Lifecycle
  376. ```typescript
  377. // Close the store — disposes LLM models and DB connection
  378. await store.close()
  379. ```
  380. The SDK requires explicit `dbPath` — no defaults are assumed. This makes it safe to embed in any application without side effects.
  381. ## Architecture
  382. ```
  383. ┌─────────────────────────────────────────────────────────────────────────────┐
  384. │ QMD Hybrid Search Pipeline │
  385. └─────────────────────────────────────────────────────────────────────────────┘
  386. ┌─────────────────┐
  387. │ User Query │
  388. └────────┬────────┘
  389. │
  390. ┌──────────────┴──────────────┐
  391. ▼ ▼
  392. ┌────────────────┐ ┌────────────────┐
  393. │ Query Expansion│ │ Original Query│
  394. │ (fine-tuned) │ │ (×2 weight) │
  395. └───────┬────────┘ └───────┬────────┘
  396. │ │
  397. │ 2 alternative queries │
  398. └──────────────┬──────────────┘
  399. │
  400. ┌───────────────────────┼───────────────────────┐
  401. ▼ ▼ ▼
  402. ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
  403. │ Original Query │ │ Expanded Query 1│ │ Expanded Query 2│
  404. └────────┬────────┘ └────────┬────────┘ └────────┬────────┘
  405. │ │ │
  406. ┌───────┴───────┐ ┌───────┴───────┐ ┌───────┴───────┐
  407. ▼ ▼ ▼ ▼ ▼ ▼
  408. ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐
  409. │ BM25 │ │Vector │ │ BM25 │ │Vector │ │ BM25 │ │Vector │
  410. │(FTS5) │ │Search │ │(FTS5) │ │Search │ │(FTS5) │ │Search │
  411. └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘
  412. │ │ │ │ │ │
  413. └───────┬───────┘ └──────┬──────┘ └──────┬──────┘
  414. │ │ │
  415. └────────────────────────┼───────────────────────┘
  416. │
  417. ▼
  418. ┌───────────────────────┐
  419. │ RRF Fusion + Bonus │
  420. │ Original query: ×2 │
  421. │ Top-rank bonus: +0.05│
  422. │ Top 30 Kept │
  423. └───────────┬───────────┘
  424. │
  425. ▼
  426. ┌───────────────────────┐
  427. │ LLM Re-ranking │
  428. │ (qwen3-reranker) │
  429. │ Yes/No + logprobs │
  430. └───────────┬───────────┘
  431. │
  432. ▼
  433. ┌───────────────────────┐
  434. │ Position-Aware Blend │
  435. │ Top 1-3: 75% RRF │
  436. │ Top 4-10: 60% RRF │
  437. │ Top 11+: 40% RRF │
  438. └───────────────────────┘
  439. ```
  440. ## Score Normalization & Fusion
  441. ### Search Backends
  442. | Backend | Raw Score | Conversion | Range |
  443. |---------|-----------|------------|-------|
  444. | **FTS (BM25)** | SQLite FTS5 BM25 | `Math.abs(score)` | 0 to ~25+ |
  445. | **Vector** | Cosine distance | `1 / (1 + distance)` | 0.0 to 1.0 |
  446. | **Reranker** | LLM 0-10 rating | `score / 10` | 0.0 to 1.0 |
  447. ### Fusion Strategy
  448. The `query` command uses **Reciprocal Rank Fusion (RRF)** with position-aware blending:
  449. 1. **Query Expansion**: Original query (×2 for weighting) + 1 LLM variation
  450. 2. **Parallel Retrieval**: Each query searches both FTS and vector indexes
  451. 3. **RRF Fusion**: Combine all result lists using `score = Σ(1/(k+rank+1))` where k=60
  452. 4. **Top-Rank Bonus**: Documents ranking #1 in any list get +0.05, #2-3 get +0.02
  453. 5. **Top-K Selection**: Take top 30 candidates for reranking
  454. 6. **Re-ranking**: LLM scores each document (yes/no with logprobs confidence)
  455. 7. **Position-Aware Blending**:
  456. - RRF rank 1-3: 75% retrieval, 25% reranker (preserves exact matches)
  457. - RRF rank 4-10: 60% retrieval, 40% reranker
  458. - RRF rank 11+: 40% retrieval, 60% reranker (trust reranker more)
  459. **Why this approach**: Pure RRF can dilute exact matches when expanded queries don't match. The top-rank bonus preserves documents that score #1 for the original query. Position-aware blending prevents the reranker from destroying high-confidence retrieval results.
  460. ### Score Interpretation
  461. | Score | Meaning |
  462. |-------|---------|
  463. | 0.8 - 1.0 | Highly relevant |
  464. | 0.5 - 0.8 | Moderately relevant |
  465. | 0.2 - 0.5 | Somewhat relevant |
  466. | 0.0 - 0.2 | Low relevance |
  467. ## Requirements
  468. ### System Requirements
  469. - **Node.js** >= 22
  470. - **Bun** >= 1.0.0
  471. - **macOS**: Homebrew SQLite (for extension support)
  472. ```sh
  473. brew install sqlite
  474. ```
  475. ### GGUF Models (via node-llama-cpp)
  476. QMD uses three local GGUF models (auto-downloaded on first use):
  477. | Model | Purpose | Size |
  478. |-------|---------|------|
  479. | `embeddinggemma-300M-Q8_0` | Vector embeddings (default) | ~300MB |
  480. | `qwen3-reranker-0.6b-q8_0` | Re-ranking | ~640MB |
  481. | `qmd-query-expansion-1.7B-q4_k_m` | Query expansion (fine-tuned) | ~1.1GB |
  482. Models are downloaded from HuggingFace and cached in `~/.cache/qmd/models/`.
  483. ### Custom Embedding Model
  484. Override the default embedding model via the `QMD_EMBED_MODEL` environment variable.
  485. This is useful for multilingual corpora (e.g. Chinese, Japanese, Korean) where
  486. `embeddinggemma-300M` has limited coverage.
  487. ```sh
  488. # Use Qwen3-Embedding-0.6B for better multilingual (CJK) support
  489. export QMD_EMBED_MODEL="hf:Qwen/Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf"
  490. # After changing the model, re-embed all collections:
  491. qmd embed -f
  492. ```
  493. Supported model families:
  494. - **embeddinggemma** (default) — English-optimized, small footprint
  495. - **Qwen3-Embedding** — Multilingual (119 languages including CJK), MTEB top-ranked
  496. > **Note:** When switching embedding models, you must re-index with `qmd embed -f`
  497. > since vectors are not cross-compatible between models. The prompt format is
  498. > automatically adjusted for each model family.
  499. ### Remote Embeddings
  500. Local GGUF defaults are unchanged. To explicitly use an OpenAI-compatible
  501. embedding server instead, set one API-base URI with a `#model-id` fragment:
  502. ```yaml
  503. models:
  504. embed: "http://localhost:1234/v1#my-embedding-model"
  505. embedOptions:
  506. dimensions: 768 # optional, integer 1..4096
  507. revision: "deployment-v1" # optional, nonempty string, at most 256 characters
  508. apiKeyEnv: EMBEDDING_KEY # optional environment variable NAME, not a key
  509. ```
  510. QMD sends `POST /v1/embeddings` with `model`, `input`, `encoding_format: "float"`,
  511. and `dimensions` when configured. The base may have a different path; QMD appends
  512. `/embeddings`. HTTP and HTTPS are intentional choices; use HTTPS for credentials
  513. outside a trusted local connection. URL credentials, query strings, control
  514. characters, endpoint lists, plaintext-key options, and unknown options are rejected.
  515. `apiKeyEnv` accepts `[A-Za-z_][A-Za-z0-9_]{0,127}`. Its value is read at each
  516. request, never saved in the index or shown in errors. Without it, requests are
  517. keyless: there is **no** `OPENAI_API_KEY` or other ambient auth fallback. A declared
  518. but missing/invalid credential fails before HTTP. Changing these options, including
  519. the credential variable name, re-arms a project-local config's `qmd trust` approval.
  520. ```sh
  521. qmd update
  522. qmd embed
  523. # Embedding-only retrieval: no local expansion or reranking model needed.
  524. qmd query 'vec: your semantic query' --no-rerank
  525. ```
  526. `qmd search` remains lexical. Ordinary free-text `query` and `vsearch` still use
  527. local query expansion. Reranking uses its separately selected model (local by
  528. default; see [TypeSafe System One](#remote-reranking-with-typesafe-system-one)).
  529. Remote embeddings do not replace those models. `qmd pull` skips the remote embedding role but still pulls
  530. local generation/reranking models. `status` and `doctor` describe remote embedding
  531. as configured, **not verified**; they do not probe the server or download/load an
  532. embedding model. Doctor also skips its native device probe in this mode.
  533. The SDK uses the same config, captured per store (later input mutation does not
  534. change that store's embedding selection):
  535. ```typescript
  536. import { createStore, RemoteEmbedder } from '@tobilu/qmd'
  537. const controller = new AbortController()
  538. const store = await createStore({
  539. dbPath: './index.sqlite',
  540. config: {
  541. collections: { docs: { path: '/path/to/docs', pattern: '**/*.md' } },
  542. models: {
  543. embed: 'http://localhost:1234/v1#my-embedding-model',
  544. embedOptions: { dimensions: 768, revision: 'deployment-v1', apiKeyEnv: 'EMBEDDING_KEY' }
  545. }
  546. }
  547. })
  548. await store.update()
  549. await store.embed({ signal: controller.signal })
  550. const results = await store.searchVector('your semantic query', { signal: controller.signal })
  551. await store.close()
  552. // Standalone embedding client; pass already-formatted text.
  553. const client = new RemoteEmbedder('http://localhost:1234/v1#my-embedding-model')
  554. const vectors = await client.embedBatch(['document text'], controller.signal)
  555. ```
  556. Remote input formatting is pinned: raw queries, and `title + "\n" + document`
  557. when a title exists. Documents use the existing surrogate-safe character chunker
  558. (3600 UTF-16 units, 540 overlap), never a local tokenizer or token truncation pass.
  559. The opaque `remote-openai:<sha256>` space identity includes the canonical endpoint,
  560. model ID, requested dimensions (or server-default marker), revision, and format/
  561. chunk version. Two hosts advertising the same model ID are **different spaces**.
  562. Credentials are excluded: change `revision` when an account or deployment changes
  563. embedding semantics. Re-embed after changing spaces; use `qmd embed --force` when
  564. switching chunk formats or dimensions. Dimensions never automatically clear a table.
  565. QMD has one vector table and content-scoped chunk metadata, not independent model
  566. indexes per collection; a shared content hash is embedded for all its collections.
  567. Vector searches enforce a remote embedding boundary in their selected collection/
  568. metadata scope before requesting a query embedding: a remote query requires exact
  569. model/fingerprint matches on every selected vector row; a local query refuses
  570. selected remote vector rows before loading a local model. Existing local-to-local
  571. behavior, including legacy empty fingerprints and differing local model labels,
  572. is unchanged. After HTTP, QMD rechecks the same boundary and reads KNN matches,
  573. documents, metadata and bodies in one SQLite read snapshot. No read transaction
  574. is held over the network.
  575. Each HTTP call is one attempt (no redirects, retries, fallback or normalization),
  576. with a 30-second timeout combined with the caller/session signal. Limits: 64 inputs,
  577. 64 KiB UTF-8 per input, 2 MiB encoded request, 8 MiB streamed response, 4096 dimensions.
  578. Oversized input is refused, never truncated. Responses must be valid UTF-8 JSON,
  579. echo the exact model
  580. and contain exactly one uniquely indexed, uniformly dimensioned, finite nonzero
  581. Float32-representable vector per input. An invalid/partial batch returns no vectors;
  582. indexing does not rescue it with individual remote calls. Errors never include
  583. response bodies, credentials, headers or provider error messages.
  584. ## Installation
  585. ```sh
  586. npm install -g @tobilu/qmd
  587. # or
  588. bun install -g @tobilu/qmd
  589. ```
  590. ### Development
  591. ```sh
  592. git clone https://github.com/tobi/qmd
  593. cd qmd
  594. bun install
  595. bun link
  596. ```
  597. On Linux, `nix develop` supplies the C++ runtime needed by Bun's llama backend.
  598. ## Usage
  599. ### Collection Management
  600. ```sh
  601. # Create a collection from current directory
  602. qmd collection add . --name myproject
  603. # Create a collection with explicit path and custom glob mask
  604. qmd collection add ~/Documents/notes --name notes --mask "**/*.md"
  605. # Comma-separated masks are a union (brace form `{a,b}` also works)
  606. qmd collection add ~/notes --name notes --mask "sources/**/*.md,CO - *.md"
  607. # List all collections
  608. qmd collection list
  609. # Remove a collection
  610. qmd collection remove myproject
  611. # Rename a collection
  612. qmd collection rename myproject my-project
  613. # List files in a collection
  614. qmd ls notes
  615. qmd ls notes/subfolder
  616. # Show collection details (path, glob mask, include status, context count, top metadata keys)
  617. qmd collection show notes
  618. # Discover metadata keys, types, and value counts (see Metadata Discovery)
  619. qmd collection metadata notes
  620. qmd collection metadata notes --match '{"field":"key","operator":"eq","value":"topics"}'
  621. qmd collection metadata notes --match '{"field":"value","operator":"eq","value":"docs-team"}'
  622. # Include or exclude a collection from default (unscoped) queries
  623. qmd collection include notes
  624. qmd collection exclude notes
  625. # Run a command before every `qmd update` (e.g. git pull); empty arg clears it
  626. qmd collection update-cmd notes 'git pull --rebase'
  627. qmd collection update-cmd notes
  628. ```
  629. ### Generate Vector Embeddings
  630. ```sh
  631. # Embed all indexed documents (900 tokens/chunk, 15% overlap)
  632. qmd embed
  633. # Force re-embed everything
  634. qmd embed -f
  635. # Enable AST-aware chunking for code files (TS, JS, Python, Go, Rust)
  636. qmd embed --chunk-strategy auto
  637. # Also works with query for consistent chunk selection
  638. qmd query "auth flow" --chunk-strategy auto
  639. # Memory control for large corpora / constrained systems
  640. qmd embed --max-docs-per-batch 50 # cap docs per embedding batch
  641. qmd embed --max-batch-mb 64 # cap batch size in MB
  642. ```
  643. **AST-aware chunking** (`--chunk-strategy auto`) uses tree-sitter to chunk code
  644. files at function, class, and import boundaries instead of arbitrary text
  645. positions. This produces higher-quality chunks and better search results for
  646. codebases. Markdown and other file types always use regex-based chunking
  647. regardless of strategy.
  648. The default is `regex` (existing behavior). Use `--chunk-strategy auto` to
  649. opt in. Run `qmd status` to verify which grammars are available.
  650. > **Note:** Tree-sitter grammars are optional dependencies. If they are not
  651. > installed, `--chunk-strategy auto` falls back to regex-only chunking
  652. > automatically. Tested on both Node.js and Bun.
  653. ### Context Management
  654. Context adds descriptive metadata to collections and paths, helping search understand your content.
  655. ```sh
  656. # Add context to a collection (using qmd:// virtual paths)
  657. qmd context add qmd://notes "Personal notes and ideas"
  658. qmd context add qmd://docs/api "API documentation"
  659. # Add context from within a collection directory
  660. cd ~/notes && qmd context add "Personal notes and ideas"
  661. cd ~/notes/work && qmd context add "Work-related notes"
  662. # Add global context (applies to all collections)
  663. qmd context add / "Knowledge base for my projects"
  664. # List all contexts
  665. qmd context list
  666. # Remove context
  667. qmd context rm qmd://notes/old
  668. ```
  669. ### Configuring `index.yml`
  670. The `collection` and `context` commands above all read and write a single YAML
  671. config file — you can also edit it directly. Everything QMD knows about your
  672. collections (paths, masks, exclusions, per-collection update hooks, contexts, and
  673. optional model overrides) lives here. A fully-commented starter template ships as
  674. [`example-index.yml`](example-index.yml) in this repo.
  675. **Location:** `~/.config/qmd/index.yml` by default. The directory honors
  676. `XDG_CONFIG_HOME` (→ `$XDG_CONFIG_HOME/qmd/index.yml`) and `QMD_CONFIG_DIR`. A
  677. named index uses `{name}.yml` — `qmd --index work …` reads/writes `work.yml`.
  678. A **project-local** index created with `qmd init` lives at `.qmd/index.yml`
  679. (`.qmd/index.yaml` is also accepted) alongside a project-local `index.sqlite`,
  680. so config and index stay inside the project instead of `~/.config` / `~/.cache`.
  681. Keep generated `*.lock.sqlite` and `*.pending-rename.json` sidecars out of
  682. version control. If a collection rename is interrupted, leave its pending
  683. marker in place and repeat the same rename to finish index recovery.
  684. ```yaml
  685. # ~/.config/qmd/index.yml
  686. # Context applied to every collection (system-message style). Optional.
  687. global_context: "Knowledge base for my projects"
  688. # Terminal hyperlink template for search results. Optional.
  689. # Overridden by the QMD_EDITOR_URI env var. See "Editor Links" below.
  690. editor_uri: "vscode://file{path}:{line}:{col}"
  691. # Override the default GGUF models per role. Optional — omit to use the
  692. # built-in defaults. `qmd init` writes this block pre-filled with the
  693. # resolved defaults. See "Model Configuration" for the default URIs.
  694. models:
  695. embed: "hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf"
  696. rerank: "hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf"
  697. generate: "hf:tobil/qmd-query-expansion-1.7B-gguf/qmd-query-expansion-1.7B-q4_k_m.gguf"
  698. # One entry per collection. The key is the collection name.
  699. collections:
  700. notes:
  701. path: /Users/me/notes # absolute path to index (required)
  702. pattern: "**/*.md" # glob mask (default: **/*.md)
  703. ignore: # glob patterns to exclude from indexing
  704. - "Archive/**"
  705. - "**/drafts/**"
  706. update: "git pull --rebase" # bash command run before each `qmd update`
  707. includeByDefault: true # include in unscoped queries (default: true)
  708. context: # path prefix → description; longest match wins
  709. "/": "Personal notes and ideas"
  710. "/work": "Work-related notes"
  711. ```
  712. | Key | Scope | Purpose |
  713. |-----|-------|---------|
  714. | `global_context` | top-level | Context prepended for every collection. Set via `qmd context add /`. |
  715. | `editor_uri` (alias `editor_uri_template`) | top-level | Hyperlink template for clickable result paths; `QMD_EDITOR_URI` overrides. |
  716. | `models.embed` / `.rerank` / `.generate` | top-level | GGUF URIs overriding the built-in defaults per role. `embed` also accepts one HTTP API-base URI with `#model-id`; see [Remote Embeddings](#remote-embeddings). `rerank` also accepts `typesafe:<model>`; see [TypeSafe System One](#remote-reranking-with-typesafe-system-one). |
  717. | `models.embedOptions` | top-level | Remote-only `dimensions`, `revision`, and `apiKeyEnv` (environment variable name, never a key). |
  718. | `models.rerankOptions` | top-level | TypeSafe-only `revision` and `apiKeyEnv` (default variable name `TYPESAFE_API_KEY`). |
  719. | `collections.<name>.path` | per-collection | Absolute directory to index. |
  720. | `collections.<name>.pattern` | per-collection | Glob mask. Set via `qmd collection add --mask`. Default `**/*.md`. Comma-separated lists and brace groups (`{a,b}`) are a union of patterns. |
  721. | `collections.<name>.ignore` | per-collection | Glob patterns excluded from indexing — useful to stop nested collections double-indexing. **YAML-only — no CLI command sets this.** Additive with QMD's built-in exclusions (`node_modules`, `.git`, `.cache`, `vendor`, `dist`, `build`), which you cannot un-ignore. |
  722. | `collections.<name>.update` | per-collection | Bash command run before `qmd update` re-indexes this collection. Set via `qmd collection update-cmd`. |
  723. | `collections.<name>.includeByDefault` | per-collection | Whether unscoped queries search it. Toggle with `qmd collection include`/`exclude`. Default `true`. |
  724. | `collections.<name>.context` | per-collection | Path-prefix → description map; the most specific (longest) matching prefix wins. Set via `qmd context add`. |
  725. > **Note:** Editing `index.yml` changes which directories and models QMD *uses*,
  726. > but does not re-index on its own. Run `qmd update` after changing `path`,
  727. > `pattern`, or `ignore`, and `qmd embed` after changing `models.embed`.
  728. #### Automatic update commands
  729. A collection's `update` field is QMD's built-in refresh hook: when you run
  730. `qmd update`, each collection's `update` command runs **first**, then the
  731. collection is re-indexed. This keeps a collection in sync with an upstream source
  732. (a git remote, a sync script) without wrapping `qmd` yourself.
  733. ```yaml
  734. collections:
  735. wiki:
  736. path: ~/reference/wiki
  737. update: "git pull --ff-only"
  738. ```
  739. $ qmd update
  740. [1/3] wiki (**/*.md)
  741. Running update command: git pull --ff-only
  742. Already up to date.
  743. Collection: ~/reference/wiki (**/*.md)
  744. Indexed: 0 new, 2 updated, 340 unchanged, 0 removed
  745. The command runs via `bash -c` in the collection's own directory (its `path`), not
  746. your current working directory. If it exits non-zero, `qmd update` prints the
  747. failure and **aborts the entire run** — collections after the failing one are not
  748. re-indexed. Set or clear it from the CLI instead of editing YAML by hand:
  749. ```sh
  750. qmd collection update-cmd wiki 'git pull --ff-only' # set
  751. qmd collection update-cmd wiki # clear
  752. ```
  753. ##### Checked-in `.qmd` config is not trusted by default
  754. A project-local `.qmd/index.yml` travels with a `git clone`, and QMD adopts it
  755. automatically for any command run inside the tree. Three fields in that file can
  756. reach outside the project, and QMD will not use them unattended:
  757. - `update` commands — somebody else's shell script, run by `qmd update`
  758. - `collections.*.path` pointing **outside** the project directory
  759. - `models.embed` / `models.rerank` / `models.generate` other than the built-in
  760. defaults (any `hf:` repo or local GGUF path)
  761. In-project collection paths (for example `./docs`) still index. On a terminal
  762. `qmd update` (and `qmd embed` / `qmd pull` / `qmd query`) lists the gated
  763. fields and asks. Approving records the approval in `~/.config/qmd/trusted.json`.
  764. With no terminal to ask — agents, CI, MCP — those fields are **skipped** and
  765. in-project indexing continues.
  766. Approvals cover the exact gated set you saw. Editing a command, pointing a
  767. collection outside the project, or changing a custom model URI asks again.
  768. ```sh
  769. qmd trust # review and approve this project's gated fields
  770. qmd trust list # show every approved project config
  771. qmd trust revoke # drop the approval for this project
  772. ```
  773. Set `QMD_TRUST_LOCAL_CONFIG=1` (or `QMD_TRUST_UPDATE_HOOKS=1`) for CI that
  774. should allow them unattended. Your own `~/.config/qmd/*.yml` — including
  775. anything `qmd collection update-cmd` or `qmd collection add` writes — is
  776. never gated.
  777. ### Search Commands
  778. ```
  779. ┌──────────────────────────────────────────────────────────────────┐
  780. │ Search Modes │
  781. ├──────────┬───────────────────────────────────────────────────────┤
  782. │ search │ BM25 full-text search only │
  783. │ vsearch │ Vector semantic search only │
  784. │ query │ Hybrid: FTS + Vector + Query Expansion + Re-ranking │
  785. └──────────┴───────────────────────────────────────────────────────┘
  786. ```
  787. ```sh
  788. # Full-text search (fast, keyword-based)
  789. qmd search "authentication flow"
  790. # Vector search (semantic similarity)
  791. qmd vsearch "how to login"
  792. # Hybrid search with re-ranking (best quality)
  793. qmd query "user authentication"
  794. ```
  795. Two aliases exist for the semantic/hybrid modes: `vector-search` (→ `vsearch`)
  796. and `deep-search` (→ `query`).
  797. ### Options
  798. ```sh
  799. # Search options
  800. -n <num> # Number of results (default: 5, or 20 for --files/--json)
  801. -c, --collection # Restrict search to a specific collection
  802. --all # Return all matches (use with --min-score to filter)
  803. --min-score <num> # Minimum score threshold (default: 0)
  804. --full # Show full document content
  805. --line-numbers # Add line numbers to output
  806. --explain # Include retrieval score traces (query, JSON/CLI output)
  807. --filter <json> # Metadata filter (recursive JSON AST; see Metadata Filtering)
  808. --index <name> # Use named index
  809. --intent "<text>" # Disambiguation context (e.g. "web page load times")
  810. --no-rerank # Skip LLM reranking (RRF scores only; faster on CPU)
  811. -C, --candidate-limit <n> # Max candidates to rerank (default: 40)
  812. --full-path # Emit on-disk filesystem paths instead of qmd:// URIs
  813. # (a result whose file has moved or been deleted since
  814. # indexing keeps its qmd:// URI + docid, and a notice is
  815. # printed to stderr — run `qmd update` to refresh)
  816. # Output formats (for search and multi-get)
  817. --format <kind> # cli (default) | json | csv | md | xml | files
  818. # (--json, --csv, --md, --xml, --files are legacy aliases)
  819. # Get options
  820. qmd get <file>[:from[:count]] # Get document; optional start line and count
  821. -l <num> # Maximum lines to return
  822. --from <num> # Start line (overrides the :from suffix)
  823. --no-line-numbers # Disable line numbering (on by default)
  824. # Multi-get options
  825. -l <num> # Maximum lines per file
  826. --max-bytes <num> # Skip files larger than N bytes (default: 64KB)
  827. ```
  828. ### Collection Filtering
  829. The `-c`/`--collection` flag filters results by collection **name** (as shown by
  830. `qmd collection list`). Collections are a global registry — you can search any
  831. collection from any directory:
  832. ```sh
  833. qmd search "auth" -c notes # single collection
  834. qmd search "auth" -c notes -c docs # multiple collections (OR)
  835. ```
  836. With no `-c` flag, all default-included collections are searched. Collections
  837. marked excluded (`qmd collection exclude <name>`) are skipped unless named
  838. explicitly with `-c`.
  839. > **Note:** With multiple `-c` flags, results come from a global top-K pool and are
  840. > then filtered. If one collection dominates the rankings, matches from smaller
  841. > collections may not appear at the default limit — raise `-n` or use `--all`.
  842. ### Metadata Filtering
  843. Documents can opt into typed metadata through a namespaced frontmatter block. A document without `qmd.metadata` behaves exactly as before, and the frontmatter stays ordinary searchable content (no chunking, embedding, or line-number changes):
  844. ```markdown
  845. ---
  846. qmd:
  847. metadata:
  848. topics:
  849. - typescript
  850. - programming
  851. status: published
  852. priority: 3
  853. reviewed: true
  854. ---
  855. # Document body starts here
  856. ```
  857. Supported values are strings, numbers, booleans, and flat homogeneous arrays of one of those. Nested objects, nulls, empty arrays, and mixed-type arrays are rejected (the document still indexes; it is excluded from filtered search until corrected). Metadata keys are user-defined data — `tags`, `topics`, and `labels` are all ordinary keys with no special semantics.
  858. Every search surface (CLI, SDK, MCP, HTTP) accepts the same recursive filter, a JSON AST discriminated by `operator`. A condition is a predicate over one field of the document's metadata: `field` names the metadata key, `operator` says how to compare, and `value` is what to compare against:
  859. ```sh
  860. # One condition
  861. qmd search "authentication" \
  862. --filter '{"field":"status","operator":"eq","value":"published"}'
  863. # Composed conditions — works with search, vsearch, and query
  864. qmd query "dependency injection" --filter '{
  865. "operator": "and",
  866. "operands": [
  867. { "field": "topics", "operator": "all", "value": ["typescript", "programming"] },
  868. { "field": "status", "operator": "nin", "value": ["draft", "archived"] },
  869. { "operator": "or", "operands": [
  870. { "field": "priority", "operator": "gte", "value": 3 },
  871. { "field": "reviewed", "operator": "eq", "value": true }
  872. ] },
  873. { "field": "topics", "operator": "prefix", "value": "sql", "caseInsensitive": true },
  874. { "operator": "not", "operand": { "field": "audience", "operator": "eq", "value": "internal" } }
  875. ]
  876. }'
  877. ```
  878. | Node | Shape |
  879. |------|-------|
  880. | Logical group | `{ "operator": "and" \| "or", "operands": […] }` |
  881. | Negation | `{ "operator": "not", "operand": {…} }` |
  882. | Comparison | `{ "field", "operator": "eq" \| "ne" \| "gt" \| "gte" \| "lt" \| "lte", "value" }` |
  883. | Membership | `{ "field", "operator": "in" \| "nin" \| "all", "value": […] }` |
  884. | Text | `{ "field", "operator": "contains" \| "prefix" \| "suffix", "value": "…" }` |
  885. | Type | `{ "field", "operator": "type", "value": "string" \| "number" \| "boolean" }` |
  886. | Presence | `{ "field", "operator": "exists", "value": true \| false }` |
  887. Any condition whose value is a string or an array of strings may add `"caseInsensitive": true`.
  888. Semantics:
  889. - Matching is typed and exact — no string/number/boolean coercion, and a type mismatch never matches (including `ne` and `nin`). Text operators match string values only. `type` matches the stored type of a key's values, which is how a filter reaches one side of a key whose documents disagree on type.
  890. - Array-valued metadata is a set: a condition matches when any element satisfies it, `all` requires every filter value to be present, and `ne`/`nin` require that no element equals the operand (with at least one element of the operand's type present).
  891. - Missing keys do not match `ne`/`nin`; combine with `{ "operator": "exists", "value": false }` in an `or` group to include them.
  892. - Matching is case-sensitive unless a condition sets `caseInsensitive`, which folds ASCII letters on both sides. Non-ASCII letters compare exactly.
  893. - Multiple conditions require an explicit `and` group — there is no implicit AND, and no `$`-prefixed shorthand.
  894. Guarantees and limits:
  895. - Every returned result satisfies the filter, before RRF fusion and reranking.
  896. - Highly selective filters can return fewer than `limit` results. Lexical search filters a bounded over-fetch window; vector search scans the filter-eligible set exactly when it holds at most 20,000 chunk vectors, and above that over-fetches and post-filters.
  897. - Filtered search only considers documents whose metadata has been extracted (run `qmd update` after upgrading; `qmd status` shows the pending count).
  898. JSON output (`--format json`), the SDK, MCP structured results, and the HTTP endpoints include each result's indexed metadata.
  899. ### Metadata Discovery
  900. Filtering is only useful if you know what to filter on. Discovery reports the metadata keys, types, and value counts already in the index, so a filter can be written from what is indexed instead of guessed. It reads the same tables filtering reads: no re-indexing, and every value it reports is one an `eq` filter can match.
  901. The command is one sentence: show metadata matching X for documents filtered by Y. `--filter` narrows **documents** by their metadata (same AST as search) and decides which are counted. `--match` narrows **the metadata itself** and decides which entries are reported. It takes the same AST, evaluated against each metadata entry instead of each document: a condition's `field` is `"key"` (the entry's key name) or `"value"` (its value). Every operator applies, including `type`, the text operators, `caseInsensitive`, and `and`/`or`/`not`. Only `exists` and `all` are rejected, as they have no meaning for a single entry.
  902. | `--match` | Question answered |
  903. |-----------|-------------------|
  904. | | Which keys exist, with a window of values each |
  905. | `{"field":"key","operator":"eq","value":"topics"}` | Everything about one key |
  906. | `{"field":"key","operator":"prefix","value":"mem-"}` | A family of keys |
  907. | `{"field":"key","operator":"in","value":["tags","topics","labels"]}` | Which of these key names exist |
  908. | `{"field":"value","operator":"eq","value":"docs-team"}` | Which keys hold this value |
  909. | `{"field":"value","operator":"prefix","value":"2025-"}` | Which keys hold values shaped like this |
  910. | `{"field":"value","operator":"type","value":"boolean"}` | Which keys hold booleans |
  911. | `and` of `key eq priority` and `value gte 3` | Values of one key above a threshold |
  912. | `and` of `key eq priority` and `value type number` | The numeric side of a key whose documents disagree on type |
  913. Start wide and narrow:
  914. ```sh
  915. # Which keys does this collection use? (`qmd collection show notes` previews the top five)
  916. qmd collection metadata notes
  917. # Everything about one key: coverage, distinct count, top values
  918. qmd collection metadata notes --match '{"field":"key","operator":"eq","value":"topics"}'
  919. # Reverse lookup: which keys hold this value
  920. qmd collection metadata notes --match '{"field":"value","operator":"eq","value":"docs-team"}'
  921. # Values of one key matching a pattern
  922. qmd collection metadata notes --match '{
  923. "operator": "and",
  924. "operands": [
  925. { "field": "key", "operator": "eq", "value": "topics" },
  926. { "field": "value", "operator": "prefix", "value": "sql" }
  927. ]
  928. }'
  929. # What remains after a filter, before committing to it in a query
  930. qmd collection metadata notes \
  931. --match '{"field":"key","operator":"eq","value":"topics"}' \
  932. --filter '{"field":"status","operator":"eq","value":"published"}'
  933. # Then search with the filter you just validated
  934. qmd query "dependency injection" -c notes --filter '{
  935. "operator": "and",
  936. "operands": [
  937. { "field": "status", "operator": "eq", "value": "published" },
  938. { "field": "topics", "operator": "all", "value": ["typescript"] }
  939. ]
  940. }'
  941. ```
  942. The drill-down prints one block per key, in coverage order:
  943. ```sh
  944. qmd collection metadata notes
  945. ```
  946. ```
  947. topics string[] 388 of 480 documents 1,204 distinct
  948. typescript 140
  949. sqlite 92
  950. search 77
  951. architecture 61
  952. sqlite-vec 44
  953. mcp 39
  954. embeddings 35
  955. agents 31
  956. cli 28
  957. testing 26
  958. 1,194 more values, use --value-limit <n>, --value-offset <n>, or --all-values
  959. priority number 205 of 480 documents 5 distinct
  960. min 1 median 3 max 5
  961. 1 (12) 2 (40) 3 (88) 4 (50) 5 (15)
  962. reviewed boolean 480 of 480 documents
  963. true 61 false 419
  964. ```
  965. With `--filter`, the output opens with a `filter:` line stating how many documents pass, and every coverage count is measured against that population rather than the whole collection — the numbers a filtered search would see:
  966. ```sh
  967. qmd collection metadata notes \
  968. --match '{"field":"key","operator":"eq","value":"topics"}' \
  969. --filter '{"field":"status","operator":"eq","value":"published"}' \
  970. --value-limit 5
  971. ```
  972. ```
  973. filter: 312 of 480 documents
  974. topics string[] 260 of 312 documents 811 distinct
  975. typescript 104
  976. sqlite 70
  977. search 58
  978. architecture 40
  979. mcp 31
  980. 806 more values, use --value-limit <n>, --value-offset <n>, or --all-values
  981. ```
  982. Two windows page the key and value lists. Each windowed list ends with the exact remainder and the flags that reach it:
  983. ```sh
  984. --key-limit <n> # Keys reported, in coverage order (default 50)
  985. --key-offset <n> # Keys skipped before the window
  986. --all-keys # Remove the key window
  987. --value-limit <n> # Values reported per key and type (default 10)
  988. --value-offset <n> # Values skipped per key and type, in --sort order
  989. --all-values # Remove the value window
  990. --sort count|value # Order values by document count (default) or by value
  991. --min-count <n> # Drop values held by fewer documents
  992. ```
  993. Omitting the collection name covers the default collections, exactly as an unscoped search does.
  994. Reading the output:
  995. - **Counts are documents, not values.** A document with `topics: [a, b]` contributes one to each. Coverage is "documents declaring this key", out of the documents the filter admits when there is one.
  996. - **A bare string is exactly the value.** A string whose bare form could be read as something else (`"42"`, `""`, `"a, b"`) prints as a JSON string, so paste it into a filter as the JSON string it is.
  997. - **Numbers report min, median, and max**, plus the enumerated values when they fit, which is enough to write a `gt`/`lt` threshold in one call.
  998. - **Discovery sees exactly what filtering sees.** Same extraction gate, same active-document rule, same collection scope, and every count in a result comes from one database snapshot. Documents still pending extraction are reported on stderr and excluded until `qmd update` runs.
  999. - **Type conflicts are reported, not resolved.** Metadata is validated one document at a time, so `priority: 3` in one file and `priority: high` in another both index, within one collection or across several. Discovery splits such a key by type, each with its own document count and (across collections) its contributing collections:
  1000. ```sh
  1001. qmd collection metadata --match '{"field":"key","operator":"eq","value":"priority"}'
  1002. ```
  1003. ```
  1004. priority number | string 1,427 of 2,100 documents
  1005. number 223 docs min 1 median 3 max 5 notes, work
  1006. string 1,204 docs high (700), medium (380), low (124) work
  1007. ```
  1008. To report one side only, add a `type` condition on the value field. The filter that reaches exactly those documents is the same condition with the metadata key in `field`:
  1009. ```sh
  1010. qmd collection metadata --match '{
  1011. "operator": "and",
  1012. "operands": [
  1013. { "field": "key", "operator": "eq", "value": "priority" },
  1014. { "field": "value", "operator": "type", "value": "number" }
  1015. ]
  1016. }'
  1017. qmd query "release checklist" --filter '{"field":"priority","operator":"type","value":"number"}'
  1018. ```
  1019. `qmd collection list` names each collection's top keys, `qmd collection show <name>` details the top five with a value preview, and `qmd status` summarizes how many keys and files carry metadata.
  1020. The CLI prints text and rejects unsupported format flags. Structured discovery is available through the SDK, MCP, and HTTP with the same options (`collection`, `match`, `filter`, `keyLimit`, `keyOffset`, `valueLimit`, `valueOffset`, `sort`, `minCount`) and the same result shape:
  1021. ```typescript
  1022. // SDK: one flat result, keys in coverage order, each split per type
  1023. const discovery = await store.listMetadata({
  1024. collection: "notes",
  1025. match: { field: "key", operator: "eq", value: "topics" },
  1026. valueLimit: 5,
  1027. })
  1028. discovery.documents // active documents in scope
  1029. discovery.totalKeys // keys with a matching entry, before the key window
  1030. discovery.remainingKeys // keys after the window
  1031. discovery.keys[0].types[0] // { type, multiValued, documents, distinctValues, values, remainingValues, range?, collections }
  1032. // Narrowed by a filter: how many documents pass, and what is left to filter on
  1033. const published = await store.listMetadata({
  1034. collection: "notes",
  1035. filter: { field: "status", operator: "eq", value: "published" },
  1036. })
  1037. published.filteredDocuments // the denominator for every coverage count in this result
  1038. // Page through a wide vocabulary
  1039. const nextPage = await store.listMetadata({ keyLimit: 50, keyOffset: 50 })
  1040. ```
  1041. `filter` is a `MetadataFilter` and `match` is a `MetadataMatch`. Both are `MetadataPredicate<Condition>`, the one recursive grammar over the conditions each record admits, so a document-only condition (`exists`, `all`, or a metadata key as the `field`) is a type error in a match as well as a runtime one. An option outside its domain throws `MetadataOptionError`, and a `filter` and `match` that together bind more SQL parameters than one statement allows throw `MetadataBindingBudgetError`.
  1042. The MCP `metadata` tool takes the same options with `collections` spelled as on `query`, returns the CLI text plus the result as `structuredContent`, and the MCP `status` tool lists each collection's most covered keys so an agent's first call reveals that metadata exists. `POST /metadata` accepts the same body as the tool and returns the same result (`400` on an invalid match, filter, or option).
  1043. ### Output Format
  1044. Default output is colorized CLI format (respects `NO_COLOR` env).
  1045. When stdout is a TTY, result paths are emitted as clickable terminal hyperlinks (OSC 8). Clicking a path opens the file in your editor using an editor URI template.
  1046. When stdout is not a TTY (for example piped to another command or redirected to a file), QMD emits plain text paths with no escape sequences.
  1047. TTY example:
  1048. ```
  1049. docs/guide.md:42 #a1b2c3
  1050. Title: Software Craftsmanship
  1051. Context: Work documentation
  1052. Score: 93%
  1053. This section covers the **craftsmanship** of building
  1054. quality software with attention to detail.
  1055. See also: engineering principles
  1056. notes/meeting.md:15 #d4e5f6
  1057. Title: Q4 Planning
  1058. Context: Personal notes and ideas
  1059. Score: 67%
  1060. Discussion about code quality and craftsmanship
  1061. in the development process.
  1062. ```
  1063. Configure the editor link target with `QMD_EDITOR_URI` (or `editor_uri` in config):
  1064. ```sh
  1065. # VS Code (default)
  1066. export QMD_EDITOR_URI="vscode://file/{path}:{line}:{col}"
  1067. # Cursor
  1068. export QMD_EDITOR_URI="cursor://file/{path}:{line}:{col}"
  1069. # Zed
  1070. export QMD_EDITOR_URI="zed://file/{path}:{line}:{col}"
  1071. # Sublime Text
  1072. export QMD_EDITOR_URI="subl://open?url=file://{path}&line={line}"
  1073. ```
  1074. Template placeholders:
  1075. - `{path}` absolute filesystem path (URI-encoded)
  1076. - `{line}` 1-based line number
  1077. - `{col}` or `{column}` 1-based column number
  1078. - **Path**: Collection-relative path (e.g., `docs/guide.md`)
  1079. - **Docid**: Short hash identifier (e.g., `#a1b2c3`) - use with `qmd get #a1b2c3`
  1080. - **Title**: Extracted from document (first heading or filename)
  1081. - **Context**: Path context if configured via `qmd context add`
  1082. - **Score**: Color-coded (green >70%, yellow >40%, dim otherwise)
  1083. - **Snippet**: Context around match with query terms highlighted
  1084. ### Examples
  1085. ```sh
  1086. # Get 10 results with minimum score 0.3
  1087. qmd query -n 10 --min-score 0.3 "API design patterns"
  1088. # Output as markdown for LLM context
  1089. qmd search --md --full "error handling"
  1090. # JSON output for scripting
  1091. qmd query --json "quarterly reports"
  1092. # Inspect how each result was scored (RRF + rerank blend)
  1093. qmd query --json --explain "quarterly reports"
  1094. # Use separate index for different knowledge base
  1095. qmd --index work search "quarterly reports"
  1096. ```
  1097. The `--explain` flag attaches a score breakdown to each result: the FTS/vector
  1098. backend scores plus the RRF fusion math (rank, weight, top-rank bonus) and every
  1099. sub-query's contribution. Abbreviated:
  1100. ```json
  1101. {
  1102. "docid": "#6c90f0",
  1103. "score": 0.89,
  1104. "file": "qmd://qmd/README.md",
  1105. "explain": {
  1106. "ftsScores": [0.892, 0.907],
  1107. "vectorScores": [0.540, 0.484],
  1108. "rrf": {
  1109. "rank": 1,
  1110. "weight": 0.75,
  1111. "baseScore": 0.123,
  1112. "topRankBonus": 0.05,
  1113. "totalScore": 0.173,
  1114. "contributions": [
  1115. { "source": "fts", "queryType": "original", "query": "reranking",
  1116. "rank": 1, "weight": 2, "backendScore": 0.892, "rrfContribution": 0.0328 }
  1117. ]
  1118. }
  1119. }
  1120. }
  1121. ```
  1122. ### Index Maintenance
  1123. ```sh
  1124. # Show index status and collections with contexts
  1125. qmd status
  1126. # Re-index all collections. If a collection has a configured update command
  1127. # (e.g. `git pull`), it runs first — set one with `qmd collection update-cmd`.
  1128. qmd update
  1129. # Diagnose the install (runtime, sqlite-vec, embedding fingerprints, GPU probe)
  1130. qmd doctor
  1131. # Initialize a project-local index in the current directory
  1132. qmd init
  1133. # Get document by filepath (with fuzzy matching suggestions)
  1134. qmd get notes/meeting.md
  1135. # Get document by docid (from search results)
  1136. qmd get "#abc123"
  1137. # Get document starting at line 50, max 100 lines
  1138. qmd get notes/meeting.md:50 -l 100
  1139. # Read 40 lines starting at line 120 via the :from:count suffix (works with docids)
  1140. qmd get notes/meeting.md:120:40
  1141. qmd get "#abc123:120:40"
  1142. # get / multi-get are line-numbered by default; disable with --no-line-numbers
  1143. qmd get notes/meeting.md --no-line-numbers
  1144. # Get multiple documents by glob pattern
  1145. qmd multi-get "journals/2025-05*.md"
  1146. # Get multiple documents by comma-separated list (supports docids)
  1147. qmd multi-get "doc1.md, doc2.md, #abc123"
  1148. # Limit multi-get to files under 20KB
  1149. qmd multi-get "docs/*.md" --max-bytes 20480
  1150. # Output multi-get as JSON for agent processing
  1151. qmd multi-get "docs/*.md" --json
  1152. # Drop caches and orphans; repack the vector table when it has holes
  1153. qmd cleanup
  1154. ```
  1155. ### Benchmarking
  1156. Measure search quality across all four backends with `qmd bench` and a fixture file
  1157. of queries with known-relevant documents.
  1158. **From a git checkout**, an example fixture and its test corpus ship in the repo:
  1159. ```sh
  1160. # One-time setup (indexes the repo's test corpus into its own collection)
  1161. qmd collection add test/eval-docs --name eval-docs
  1162. qmd embed -c eval-docs
  1163. # Run the benchmark (table output)
  1164. qmd bench src/bench/fixtures/example.json
  1165. # JSON output for programmatic analysis
  1166. qmd bench src/bench/fixtures/example.json --json
  1167. ```
  1168. > The example fixture (`src/bench/fixtures/example.json`) and its test corpus
  1169. > (`test/eval-docs/`) exist only in a git checkout — they are **not** part of the
  1170. > published npm package. If you installed via `npm`/`npx`, write your own fixture
  1171. > (see below) against a collection you have already indexed:
  1172. >
  1173. > ```sh
  1174. > qmd bench my-fixture.json -c my-collection
  1175. > ```
  1176. Each query runs against four backends, reporting precision@k, recall, MRR, and F1:
  1177. | Backend | What it tests | LLM required |
  1178. |---------|---------------|--------------|
  1179. | `bm25` | Keyword search only (FTS5) | No |
  1180. | `vector` | Semantic similarity only | Embedding model |
  1181. | `hybrid` | BM25 + vector fusion (no reranking) | Embedding model |
  1182. | `full` | Full pipeline with LLM reranking | All three models |
  1183. **Score interpretation:** `1.00` = perfect (all expected docs in top results),
  1184. `0.00` = complete miss. The example fixture typically shows bm25 ~0.50, vector
  1185. ~0.70, and hybrid/full ~1.00 — a concrete demonstration of why hybrid search beats
  1186. either backend alone.
  1187. **Custom fixtures** are JSON:
  1188. ```json
  1189. {
  1190. "description": "My benchmark",
  1191. "version": 1,
  1192. "collection": "my-collection",
  1193. "queries": [
  1194. {
  1195. "id": "find-auth",
  1196. "query": "authentication flow",
  1197. "type": "semantic",
  1198. "expected_files": ["docs/auth-design.md"],
  1199. "expected_in_top_k": 3
  1200. }
  1201. ]
  1202. }
  1203. ```
  1204. `expected_files` are collection-relative paths as shown by `qmd ls`. The `type`
  1205. field (`exact`, `semantic`, `topical`, `cross-domain`, `alias`) labels queries for
  1206. grouping — it does not change search behavior.
  1207. > **Heads-up:** if the fixture's collection isn't indexed, bench currently runs to
  1208. > completion and reports all zeros with no warning. Verify setup with
  1209. > `qmd ls <collection>` first.
  1210. ## Data Storage
  1211. Index stored in: `~/.cache/qmd/index.sqlite`
  1212. ### Schema
  1213. ```sql
  1214. collections -- Indexed directories with name and glob patterns
  1215. path_contexts -- Context descriptions by virtual path (qmd://...)
  1216. documents -- Markdown content with metadata and docid (6-char hash)
  1217. documents_fts -- FTS5 full-text index
  1218. content_vectors -- Embedding chunks (hash, seq, pos, 900 tokens each)
  1219. vector_collection_ids -- Integer id per collection name (the vector partition key)
  1220. vector_rows -- Vector rowid to (hash, seq, collection_id)
  1221. vectors_by_collection -- sqlite-vec vector index, one row per chunk and collection
  1222. llm_cache -- Cached LLM responses (query expansion, rerank scores)
  1223. ```
  1224. ## Environment Variables
  1225. | Variable | Default | Description |
  1226. |----------|---------|-------------|
  1227. | `XDG_CACHE_HOME` | `~/.cache` | Cache directory location |
  1228. | `XDG_CONFIG_HOME` | `~/.config` | Config directory location (where `index.yml` lives) |
  1229. | `QMD_CONFIG_DIR` | unset | Override the config directory outright (takes precedence over `XDG_CONFIG_HOME`) |
  1230. | `QMD_LLAMA_GPU` | `auto` | Force llama.cpp GPU backend (`metal`, `vulkan`, `cuda`) or disable GPU with `false` |
  1231. | `QMD_FORCE_CPU` | unset | Set to `1`/`true` to force CPU mode before any CUDA/Vulkan/Metal probing. Equivalent CLI flag: `--no-gpu`. |
  1232. | `QMD_EMBED_PARALLELISM` | automatic | Override embedding/reranking context parallelism (1-8). Windows CUDA defaults to `1` because parallel CUDA contexts can crash with `ggml-cuda.cu:98`; use Vulkan or raise this only if your driver is stable. |
  1233. ## How It Works
  1234. ### Indexing Flow
  1235. ```
  1236. Collection ──► Glob Pattern ──► Markdown Files ──► Parse Title ──► Hash Content
  1237. │ │ │
  1238. │ │ ▼
  1239. │ │ Generate docid
  1240. │ │ (6-char hash)
  1241. │ │ │
  1242. └──────────────────────────────────────────────────►└──► Store in SQLite
  1243. │
  1244. ▼
  1245. FTS5 Index
  1246. ```
  1247. ### Embedding Flow
  1248. Documents are chunked into ~900-token pieces with 15% overlap using smart boundary detection:
  1249. ```
  1250. Document ──► Smart Chunk (~900 tokens) ──► Format each chunk ──► node-llama-cpp ──► Store Vectors
  1251. │ "title | text" embedBatch()
  1252. │
  1253. └─► Chunks stored with:
  1254. - hash: document hash
  1255. - seq: chunk sequence (0, 1, 2...)
  1256. - pos: character position in original
  1257. ```
  1258. ### Smart Chunking
  1259. Instead of cutting at hard token boundaries, QMD uses a scoring algorithm to find natural markdown break points. This keeps semantic units (sections, paragraphs, code blocks) together.
  1260. **Break Point Scores:**
  1261. | Pattern | Score | Description |
  1262. |---------|-------|-------------|
  1263. | `# Heading` | 100 | H1 - major section |
  1264. | `## Heading` | 90 | H2 - subsection |
  1265. | `### Heading` | 80 | H3 |
  1266. | `#### Heading` | 70 | H4 |
  1267. | `##### Heading` | 60 | H5 |
  1268. | `###### Heading` | 50 | H6 |
  1269. | ` ``` ` | 80 | Code block boundary |
  1270. | `---` / `***` | 60 | Horizontal rule |
  1271. | Blank line | 20 | Paragraph boundary |
  1272. | `- item` / `1. item` | 5 | List item |
  1273. | Line break | 1 | Minimal break |
  1274. **Algorithm:**
  1275. 1. Scan document for all break points with scores
  1276. 2. When approaching the 900-token target, search a 200-token window before the cutoff
  1277. 3. Score each break point: `finalScore = baseScore × (1 - (distance/window)² × 0.7)`
  1278. 4. Cut at the highest-scoring break point
  1279. The squared distance decay means a heading 200 tokens back (score ~30) still beats a simple line break at the target (score 1), but a closer heading wins over a distant one.
  1280. **Code Fence Protection:** Break points inside code blocks are ignored—code stays together. If a code block exceeds the chunk size, it's kept whole when possible.
  1281. **AST-Aware Chunking (Code Files):**
  1282. For supported code files, QMD also parses the source with [tree-sitter](https://tree-sitter.github.io/) and adds AST-derived break points that are merged with the regex scores above:
  1283. | AST Node | Score | Languages |
  1284. |----------|-------|-----------|
  1285. | Class / interface / struct / impl / trait | 100 | All |
  1286. | Function / method | 90 | All |
  1287. | Type alias / enum | 80 | All |
  1288. | Import / use declaration | 60 | All |
  1289. Supported for `.ts`, `.tsx`, `.js`, `.jsx`, `.py`, `.go`, and `.rs` files. Enable with `--chunk-strategy auto`. Markdown and other file types always use regex chunking.
  1290. ### Query Flow (Hybrid)
  1291. ```
  1292. Query ──► LLM Expansion ──► [Original, Variant 1, Variant 2]
  1293. │
  1294. ┌─────────┴─────────┐
  1295. ▼ ▼
  1296. For each query: FTS (BM25)
  1297. │ │
  1298. ▼ ▼
  1299. Vector Search Ranked List
  1300. │
  1301. ▼
  1302. Ranked List
  1303. │
  1304. └─────────┬─────────┘
  1305. ▼
  1306. RRF Fusion (k=60)
  1307. Original query ×2 weight
  1308. Top-rank bonus: +0.05/#1, +0.02/#2-3
  1309. │
  1310. ▼
  1311. Top 30 candidates
  1312. │
  1313. ▼
  1314. LLM Re-ranking
  1315. (yes/no + logprob confidence)
  1316. │
  1317. ▼
  1318. Position-Aware Blend
  1319. Rank 1-3: 75% RRF / 25% reranker
  1320. Rank 4-10: 60% RRF / 40% reranker
  1321. Rank 11+: 40% RRF / 60% reranker
  1322. │
  1323. ▼
  1324. Final Results
  1325. ```
  1326. ## Model Configuration
  1327. The default models are defined in `src/llm.ts` as HuggingFace URIs:
  1328. ```typescript
  1329. const DEFAULT_EMBED_MODEL = "hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf";
  1330. const DEFAULT_RERANK_MODEL = "hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf";
  1331. const DEFAULT_GENERATE_MODEL = "hf:tobil/qmd-query-expansion-1.7B-gguf/qmd-query-expansion-1.7B-q4_k_m.gguf";
  1332. ```
  1333. Override them per-role without touching source via the `models:` block in
  1334. `index.yml` (see [Configuring `index.yml`](#configuring-indexyml)) or the
  1335. `QMD_EMBED_MODEL` env var. Re-run `qmd embed` after changing the embedding model.
  1336. ### EmbeddingGemma Prompt Format
  1337. ```
  1338. // For queries
  1339. "task: search result | query: {query}"
  1340. // For documents
  1341. "title: {title} | text: {content}"
  1342. ```
  1343. ### Qwen3-Reranker
  1344. Uses node-llama-cpp's `createRankingContext()` and `rankAndSort()` API for cross-encoder reranking. Returns documents sorted by relevance score (0.0 - 1.0).
  1345. ### Remote reranking with TypeSafe System One
  1346. Select `typesafe:<model>` to score each candidate chunk independently through
  1347. TypeSafe's fixed `https://api.typesafe.ai/v1/systemone` endpoint instead of loading
  1348. a local reranker. There is no configurable endpoint, model discovery, local
  1349. fallback or retry. Embeddings and query expansion keep their own selections.
  1350. ```yaml
  1351. models:
  1352. rerank: "typesafe:jev-latest"
  1353. rerankOptions:
  1354. revision: "deployment-v1" # optional, nonempty string, at most 256 characters
  1355. apiKeyEnv: TYPESAFE_API_KEY # optional variable NAME; this is also the default
  1356. ```
  1357. Set the named credential in the process environment, never in YAML. Explicit
  1358. TypeSafe selection requires a credential: a missing/invalid named variable fails
  1359. before HTTP, with no fallback to another environment variable. `apiKeyEnv` accepts
  1360. `[A-Za-z_][A-Za-z0-9_]{0,127}`. Unknown options are rejected. Model/revision strings
  1361. must be well-formed, bounded and free of control characters. Options are copied
  1362. and frozen per store; changing their revision or credential variable name re-arms
  1363. project-local `qmd trust` approval. Credential values never enter cache identities,
  1364. status, diagnostics or errors.
  1365. The SDK accepts the same model config. For retrieval without local query expansion,
  1366. use pre-expanded queries (also available as `qmd query 'vec: your query'`):
  1367. ```typescript
  1368. import { createStore } from '@tobilu/qmd'
  1369. const store = await createStore({
  1370. dbPath: './index.sqlite',
  1371. config: {
  1372. collections: { docs: { path: '/path/to/docs', pattern: '**/*.md' } },
  1373. models: {
  1374. embed: 'http://localhost:1234/v1#my-embedding-model',
  1375. rerank: 'typesafe:jev-latest',
  1376. rerankOptions: { revision: 'deployment-v1' }
  1377. }
  1378. }
  1379. })
  1380. await store.update()
  1381. await store.embed()
  1382. const controller = new AbortController()
  1383. const results = await store.search({
  1384. queries: [{ type: 'vec', query: 'your semantic query' }],
  1385. signal: controller.signal
  1386. })
  1387. await store.close()
  1388. ```
  1389. `SearchOptions.signal` cancels remote embedding and rerank stages, including queued
  1390. requests and fully cached reranks. Existing local inference may finish before
  1391. cancellation is observed. The standalone `JevReranker` export also accepts these
  1392. model/options and a caller signal on `rerank(query, documents, signal)`.
  1393. Each document produces its own request: `model`, `state: { query, document }`
  1394. (with `document_title` if supplied), and one `questions.relevant` noul question:
  1395. "Does the document contain information that answers or directly addresses the
  1396. search query?" File identifiers are not sent. Responses must be valid UTF-8 JSON
  1397. with a finite `answers.relevant.noul` score in 0..1, including zero; an optional
  1398. returned model must match the requested model. No normalization or score-zero
  1399. rescue is applied. Results preserve original indexes and sort stably by score.
  1400. The opaque `remote-typesafe:<sha256>` rerank identity pins the fixed endpoint,
  1401. model, revision and prompt format. Scores stay independently cached by query and
  1402. chunk text, even for different chunks sharing a file name. Other models/revisions
  1403. cannot reuse them. Credentials are excluded; change `revision` when changing an
  1404. account or deployment's scoring semantics. Local model cache keys are unchanged.
  1405. A call has at most two in-flight requests, 64 documents and 2 MiB of aggregate
  1406. encoded request JSON. Queries are limited to 64 KiB UTF-8, titles to 8 KiB and
  1407. logical file identifiers to 4 KiB; input is refused, never silently truncated.
  1408. The whole call, including its queue, has a 30-second deadline combined with
  1409. caller/session cancellation. Each HTTP request also uses the shared byte/stream
  1410. bounds (2 MiB request, 8 MiB response). Failure stops dispatch, aborts and drains
  1411. in-flight work, and returns no partial scores or new score-cache rows.
  1412. **These are resource bounds, not a spending cap.** The service can bill per sent
  1413. document; cancellation cannot retract requests already sent. `status` and `doctor`
  1414. report configured remote reranking, not a verified working service, without probes
  1415. or local reranker downloads. `pull` skips this role but still downloads selected
  1416. local models. Remote rerank selection alone does not skip local embedding device
  1417. checks. Free-text query expansion remains local unless bypassed with typed queries.
  1418. ### Qwen3 (Query Expansion)
  1419. Used for generating query variations via `LlamaChatSession`.
  1420. ## License
  1421. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.