Skip to content
Back to search

synthadoc

Synthadoc: An open-source LLM knowledge compilation engine that turns raw documents into structured, local-first wikis. A transparent, human-readable alternative to traditional RAG, which can be self-managed and self-improved without the use of any tools.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/axoviq-ai/synthadoc with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/axoviq-ai/synthadoc'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/axoviq-ai--synthadoc/badge.svg)](/listings/axoviq-ai--synthadoc)

HTML

<a href="/listings/axoviq-ai--synthadoc"><img src="/api/listings/axoviq-ai--synthadoc/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Production

**Prerequisites:** Python 3.11+. No Node.js, no Git, no build steps.

pip install synthadoc
synthadoc --version   # confirm it works

The Obsidian plugin is bundled inside the package. New wikis created with `synthadoc install` have it installed automatically. If you are upgrading an existing Synthadoc installation, run `synthadoc plugin upgrade` to push the updated plugin binary to all registered wikis.

---

Development

For developers modifying the Python engine, running the test suite, or developing the Obsidian plugin TypeScript.

**Additional prerequisites:** Git (any). Node.js 18+ only if modifying the plugin TypeScript source.

Step 1 — Clone and install

git clone https://github.com/axoviq-ai/synthadoc.git
cd synthadoc
pip install -e ".[dev]"

`[dev]` adds `pytest`, `respx`, and the other test dependencies. Tests require a source checkout — they are not included in the pip wheel.

Step 2 — Run the test suite

pytest --ignore=tests/performance/ -q

Expected: all tests pass, 0 failures. Performance benchmarks (optional):

pytest tests/performance/ -v --benchmark-disable

Step 3 — Obsidian plugin development (optional)

The compiled plugin is bundled with the package and kept up to date by CI — no build step is needed to work on the Python side or run tests.

If you modify the TypeScript source under `obsidian-plugin/src/`, recompile and sync it into the Python package manually:

cd obsidian-plugin
npm install           # first time only, or after package.json changes
# edit src/main.ts or other source files
npm run build         # compile TypeScript → main.js (also syncs to synthadoc/data/obsidian-plugin/)
npm test              # run Vitest unit tests
cd ..

---

Set your API keys

**At least one LLM API key is required** — unless you use Claude Code or Opencode as your provider (no separate API key needed — see [Coding tool CLI providers](docs/design.md#coding-tool-cli-providers--no-api-key-needed)).

Synthadoc defaults to **Gemini Flash** — free tier, no credit card, 1 million tokens per day. Get a key at **aistudio.google.com/app/apikey** (click "Create API key").

| Provider | Free tier | Vision | Get key | | ---------------- | --------------------------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | **Gemini Flash** | Yes — 15 RPM / 1M tokens/day, no credit card | Yes | [aistudio.google.com](https://aistudio.google.com/app/apikey) | | Groq | Yes — rate-limited | No | [console.groq.com](https://console.groq.com/keys) | | Ollama | Yes — runs locally, no key (**GPU required**) | Model-dependent | [ollama.com](https://ollama.com) | | Qwen | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent | [bailian.console.aliyun.com](https://bailian.console.aliyun.com/) | | MiniMax | No — pay-per-token | Yes | [platform.minimax.io](https://platform.minimax.io/) |

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 1,502 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Synthadoc
  2. ```
  3. .-+###############+-.
  4. .## ##.
  5. ## .----. .----. ##
  6. ## /######\ /######\ ##
  7. ## |######| |######| ##
  8. ## | [SD] | | wiki | ##
  9. ## |######| |######| ##
  10. ## \######/ \######/ ##
  11. ## '----' '----' ##
  12. '## ##'
  13. '-+###############+-'
  14. S Y N T H A D O C
  15. Community Edition v1.5.2
  16. ────────────────────────────────
  17. Domain-agnostic LLM wiki engine
  18. ```
  19. <p>
  20. <a href="https://github.com/axoviq-ai/synthadoc/actions/workflows/ci.yml"><img src="https://github.com/axoviq-ai/synthadoc/actions/workflows/ci.yml/badge.svg" alt="CI"/></a>
  21. <a href="https://github.com/axoviq-ai/synthadoc/actions/workflows/ci.yml"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Faxoviq-ai%2Fsynthadoc%2Fbadges%2Fdocs%2Fbadges.json&query=%24.coverage&label=Coverage&suffix=%25&color=brightgreen" alt="Coverage"/></a>
  22. <a href="https://github.com/axoviq-ai/synthadoc/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-AGPL--3.0-blue.svg" alt="License"/></a>
  23. <a href="https://www.python.org/"><img src="https://img.shields.io/badge/Python-3.11%2B-yellow.svg" alt="Python"/></a>
  24. <a href="https://github.com/axoviq-ai/synthadoc/tree/main/synthadoc/agents"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Faxoviq-ai%2Fsynthadoc%2Fbadges%2Fdocs%2Fbadges.json&query=%24.agents&label=AI%20agents&color=crimson" alt="AI agents"/></a>
  25. <a href="https://github.com/axoviq-ai/synthadoc/tree/main/synthadoc/skills"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Faxoviq-ai%2Fsynthadoc%2Fbadges%2Fdocs%2Fbadges.json&query=%24.skills&label=Skills&color=purple" alt="Skills"/></a>
  26. <a href="https://github.com/axoviq-ai/synthadoc/blob/main/docs/user-quick-start-guide.md#appendix-i--connect-claude-via-mcp"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Faxoviq-ai%2Fsynthadoc%2Fbadges%2Fdocs%2Fbadges.json&query=%24.mcp_tools&label=MCP%20tools&color=orange" alt="MCP tools"/></a>
  27. <a href="https://github.com/axoviq-ai/synthadoc/tree/main/hooks"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Faxoviq-ai%2Fsynthadoc%2Fbadges%2Fdocs%2Fbadges.json&query=%24.hooks&label=Hook%20events&color=teal" alt="Hook events"/></a>
  28. <a href="https://github.com/axoviq-ai/synthadoc"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Faxoviq-ai%2Fsynthadoc%2Fbadges%2Fdocs%2Fbadges.json&query=%24.cli_commands&label=CLI%20commands&color=darkblue" alt="CLI commands"/></a>
  29. <a href="https://github.com/axoviq-ai/synthadoc/tree/main/obsidian-plugin"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Faxoviq-ai%2Fsynthadoc%2Fbadges%2Fdocs%2Fbadges.json&query=%24.obsidian_commands&label=Obsidian%20commands&color=blueviolet" alt="Obsidian commands"/></a>
  30. <a href="https://github.com/axoviq-ai/synthadoc/blob/main/docs/user-quick-start-guide.md#agentic-workflows"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Faxoviq-ai%2Fsynthadoc%2Fbadges%2Fdocs%2Fbadges.json&query=%24.maintenance_workflows&label=Maintenance%20workflows&color=green" alt="Maintenance workflows"/></a>
  31. <a href="https://hub.docker.com/r/chenp/synthadoc"><img src="https://img.shields.io/docker/pulls/chenp/synthadoc?label=Docker+pulls" alt="Docker pulls"/></a>
  32. <a href="https://pepy.tech/projects/synthadoc"><img src="https://static.pepy.tech/personalized-badge/synthadoc?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=PyPI+downloads" alt="PyPI downloads"/></a>
  33. <a href="https://github.com/axoviq-ai/synthadoc"><img src="https://img.shields.io/badge/Community%20Edition-v1.5.2-brightgreen.svg" alt="Version"/></a>
  34. </p>
  35. **Document version: v1.5.2**
  36. **Engineered for solo users and enterprises alike, providing a domain-specific knowledge base that scales seamlessly while maintaining accuracy through autonomous self-optimization.**
  37. > Built for individuals, small teams, and large organizations who need a knowledge base that stays accurate as documents accumulate.
  38. **Documentation, FAQ and articles:** https://axoviq.com/resources/synthadoc
  39. Synthadoc reads your raw source documents — PDFs, spreadsheets, PPTs, web pages, images, videos, Word files, TXTs, and AI session transcripts (.jsonl) — and uses an LLM to synthesize them into a persistent, structured wiki. Cross-references are built automatically, contradictions are detected and surfaced, orphan pages are flagged, and every answer cites its sources. Outputs are stored as local Markdown files, ensuring seamless integration and autonomous management within [Obsidian](https://obsidian.md) or any wiki-compliant ecosystem.
  40. ---
  41. <table>
  42. <tr>
  43. <td align="center" width="33%">
  44. <a href="https://www.youtube.com/watch?v=rIGO6zi9XQE">
  45. <img src="https://img.youtube.com/vi/rIGO6zi9XQE/mqdefault.jpg" width="260" alt="From Documents to Wiki — demo walkthrough"/>
  46. </a>
  47. <br/>
  48. <a href="https://www.youtube.com/watch?v=rIGO6zi9XQE">▶ From Documents to Wiki</a>
  49. </td>
  50. <td align="center" width="33%">
  51. <a href="https://youtu.be/ue_kHhG0iog">
  52. <img src="https://img.youtube.com/vi/ue_kHhG0iog/mqdefault.jpg" width="260" alt="Four Interfaces: CLI, Obsidian, Web UI & MCP"/>
  53. </a>
  54. <br/>
  55. <a href="https://youtu.be/ue_kHhG0iog">▶ Four Interfaces: CLI, Obsidian, Web UI & MCP</a>
  56. </td>
  57. <td align="center" width="33%">
  58. <a href="https://www.youtube.com/watch?v=ojBtNlXVHQk">
  59. <img src="https://img.youtube.com/vi/ojBtNlXVHQk/mqdefault.jpg" width="260" alt="Agentic Maintenance Workflow"/>
  60. </a>
  61. <br/>
  62. <a href="https://www.youtube.com/watch?v=ojBtNlXVHQk">▶ Agentic Maintenance Workflow</a>
  63. </td>
  64. </tr>
  65. </table>
  66. <!-- pypi-strip-start -->
  67. <p align="center">
  68. <a href="docs/media/README.md">📝 Blogs & Media — YouTube · Coderlegion · DEV.to · Medium</a>
  69. </p>
  70. <p align="center">
  71. <a href="docs/example/aquaflow/README.md">📂 End-to-end Example — AquaFlow Capital M&A due diligence walkthrough</a>
  72. </p>
  73. <!-- pypi-strip-end -->
  74. ---
  75. ## Table of Contents
  76. - [Who Is It For?](#who-is-it-for)
  77. - [Inspiration and Vision](#inspiration-and-vision)
  78. - [Problems Addressed](#problems-addressed)
  79. - [Why Synthadoc?](#why-synthadoc)
  80. - [Architecture](#architecture)
  81. - [What's Included](#whats-included)
  82. - [Installation](#installation)
  83. - [Quick-Start Guide](#quick-start-guide)
  84. - [Creating Your Own Wiki](#creating-your-own-wiki)
  85. - [Configuration](#configuration)
  86. - [Command Reference by Use Case](#command-reference-by-use-case)
  87. - [Administrative Reference](#administrative-reference)
  88. - [Understanding Logs and the Audit Trail](#understanding-logs-and-the-audit-trail)
  89. - [Customization](docs/design.md#customization)
  90. - [Links](#links)
  91. ---
  92. ## Who Is It For?
  93. Synthadoc scales from a single researcher to a company-wide knowledge platform:
  94. | Team size | Typical use case |
  95. | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  96. | **Solo / 1–2 people** | Personal research wiki, freelance knowledge base, indie hacker documentation - run it free on Gemini Flash or a local Ollama model with zero ongoing cost |
  97. | **Small team (3–20)** | Centralized internal knowledge base for startups and departments that aggregates diverse individual data sources into a unified, high-integrity wiki. The system automatically resolves contradictions and scales autonomously, ensuring organizational intelligence grows in tandem with your team |
  98. | **Medium / enterprise** | Compliance-sensitive knowledge bases that must stay local; per-department wikis on separate ports; audit trail for every ingest and cost event; hook system for CI/CD integration; OpenTelemetry for ops dashboards |
  99. **Built for large wikis.** Measured on wikis of thousands of pages: keyword search finds the pages for an answer in about 20 ms at 2,000 pages; [semantic search](docs/design.md#semantic-search), on by default for new wikis when installed, puts the right page in the top 5 for 70% of questions asked in your own words, up from 48%; and the [knowledge graph](docs/user-quick-start-guide.md#large-wikis-cluster-overview) opens a 5,000-page wiki as a clickable overview of named topic clusters in about 6 seconds.
  100. ![Knowledge graph of a 5,000-page wiki: one bubble per topic cluster, named after its main pages, with a clickable list of clusters below](docs/png/synthadoc-graph-cluster-overview.png)
  101. *The web UI's Graph tab on a 5,000-page test wiki: 100,801 links summarised as 24 named clusters. Click a cluster to see its pages. [How it works](docs/user-quick-start-guide.md#large-wikis-cluster-overview)*
  102. No cloud account. No vendor lock-in. The wiki is plain Markdown — open it in any editor, back it up with git, sync it with any cloud drive.
  103. ---
  104. ## Inspiration and Vision
  105. > *"The LLM should be able to maintain a wiki for you."*
  106. > — Andrej Karpathy, [LLM Wiki gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
  107. Most knowledge-management tools retrieve and summarize at query time. Synthadoc inverts this: it **compiles knowledge at ingest time**. Every new source enriches and cross-links the entire corpus, not just appends a new chunk. The wiki is the artifact — readable, editable, and browsable without any tool running.
  108. **Long-term alignment:**
  109. | Direction | How Synthadoc moves there |
  110. | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  111. | Agent orchestration | Orchestrator dispatches parallel ingest, query, and lint sub-agents with cost guards and retry backoff |
  112. | Sub-agent skills/plugins | Featuring a 3-tier lazy-load capability system, the platform allows for the injection of custom skills and hooks via a plug-and-play interface, ensuring core stability is never compromised during extension |
  113. | LLM wiki vs. RAG | Pre-compiled structured knowledge beats query-time synthesis for contradiction detection, graph traversal, and offline access |
  114. | CLI / HTTP | A unified interface via CLI and RESTful endpoints, the system streamlines full-spectrum integration: from data ingestion and querying to automated linting, security auditing, and job orchestration |
  115. | Local-first | All data stays on your machine; localhost-only network binding; no cloud dependency except the LLM API itself |
  116. | Provider choice | LLM backends including free-tier Gemini and Groq, paid Anthropic/OpenAI/DeepSeek/MiniMax/Qwen (DashScope), local Ollama and Qwen, and coding-tool CLI providers (Claude Code, Opencode) — no API key required if you already have a subscription |
  117. ---
  118. ## Problems Addressed
  119. RAG retrieves document chunks at query time. Synthadoc **compiles** knowledge at ingest — synthesising sources into a linked, audited wiki graph so contradictions are caught, claims are traced to sources, and the artifact survives outside the tool.
  120. | Problem | Synthadoc approach |
  121. | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
  122. | **Contradictions blended silently** | Ingest-time conflict detection; page flagged`status: contradicted`; auto-resolve or queue for human review |
  123. | **No links between related content** | `[[wikilinks]]` auto-built on every ingest pass; weighted graph (wikilink + co-source signals) with Louvain clustering in web UI |
  124. | **Orphan pages never surfaced** | Lint reports unreferenced pages with ready-to-paste index entries |
  125. | **LLM output can be overconfident** | Adversarial second-LLM pass flags overstated claims, unsupported superlatives, and contestable facts per page |
  126. | **Claims lack source traceability** | `^[file:L-L]` citation on every claim; Source Viewer in Obsidian; PDF page resolution; broken-citation lint |
  127. | **Knowledge lifecycle invisible** | 5-state machine (`draft → active → contradicted / stale → archived`); auto-transitions via lint; immutable event log |
  128. | **Repeat ingest is expensive** | 3-layer cache (embedding, LLM, provider prompt) — repeat lint on unchanged pages costs near-zero tokens |
  129. | **Knowledge locked in proprietary tools** | Plain Markdown + YAML frontmatter; [OKF v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) compatible; fully offline-readable in any editor |
  130. | **Wiki structure drifts with growth** | `scaffold` regenerates index, AGENTS.md, and purpose.md from current wiki state without touching linked pages |
  131. | **Migration requires full re-ingestion** | Single-zip backup + restore with port/domain rewriting; no re-ingestion needed |
  132. | **Cost and compliance exposure** | Localhost-only; per-job token+cost log; configurable soft-warn and hard-gate thresholds |
  133. > **Citation quality:** Generated pages include inline citations linking every claim to its source lines. Pages without citations trigger a model-compatibility warning — use Gemini 2.5 Flash or higher for reliable citation annotation.
  134. >
  135. > **Active page protection:** Pages promoted to `active` status are protected — sources that contradict them are flagged for review rather than overwriting the human-reviewed content.
  136. ---
  137. ## Why Synthadoc?
  138. Every **Yes** below is a built-in feature — no add-ons or upgrades required.
  139. ### Knowledge Quality
  140. | Capability | Synthadoc | Typical RAG | NotebookLM | Notion AI |
  141. | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ----------- | ---------- | --------- |
  142. | **[Ingest-time synthesis](docs/design.md#ingestagent)** — sources compiled into the wiki at ingest; not re-summarised at query time | **Yes** | No | Partial | No |
  143. | **[Contradiction detection & resolution](docs/user-quick-start-guide.md#step-9--resolve-a-contradiction)** — conflicting claims flagged `status: contradicted`; auto-resolve available; full conflict history | **Yes** | No | No | No |
  144. | **[Adversarial claim review + gate](docs/user-quick-start-guide.md#step-11--run-the-adversarial-review)** — concurrent second-LLM pass flags overstated claims, unsupported superlatives, and contestable facts per page; configurable gate auto-demotes pages that exceed the warning threshold to `contradicted` | **Yes** | No | No | No |
  145. | **[Claim-level provenance](docs/user-quick-start-guide.md#step-20--establish-claim-level-provenance)** — `^[file:L-L]` citation on every claim; Source Viewer in Obsidian; PDF page resolution; broken-citation lint | **Yes** | No | Partial | No |
  146. | **[5-state lifecycle machine](docs/user-quick-start-guide.md#step-8--manage-page-lifecycle)** — `draft → active → contradicted / stale → archived`; auto-transitions via lint; immutable event log; cascade link cleanup on archive; only `active` and `stale` pages enter the BM25 search index | **Yes** | No | No | No |
  147. | **[Pre-LLM source sanitizer](docs/design.md#29-pre-llm-source-sanitizer)** — strips zero-width chars, bidi overrides, hidden HTML, and instruction-override phrases before any LLM call | **Yes** | No | No | No |
  148. ### Knowledge Structure
  149. | Capability | Synthadoc | Typical RAG | NotebookLM | Notion AI |
  150. | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------- | ----------- | ---------- | --------- |
  151. | **[Weighted knowledge graph + D3 visualisation](docs/user-quick-start-guide.md#step-24--knowledge-graph)** — `[[wikilinks]]` auto-built at ingest; co-source edges connect pages compiled from the same document; edge thickness reflects combined weight; dashed edges for co-source-only relationships; Louvain cluster colouring; click node to query; **scales to thousands of pages** — wikis over 300 pages open on a cluster overview (named clusters, click to drill in), 5,000 pages in ~6 s at 60 fps; Obsidian graph panel: same graph inside Obsidian via Canvas, type filter, click node to open page | **Yes** | No | Partial | No |
  152. | **[Orphan page detection](docs/user-quick-start-guide.md#step-10--fix-an-orphan-page)** — unreferenced pages surfaced by lint with ready-to-paste index entries | **Yes** | No | No | No |
  153. | **[Query-scoped routing](docs/user-quick-start-guide.md#step-17--set-up-routingmd--scoped-search)** — ROUTING.md maps wiki branches to page slugs; queries auto-select relevant branches; new pages auto-slotted | **Yes** | No | No | No |
  154. | **[Candidates staging](docs/user-quick-start-guide.md#step-18--configure-candidates-staging)** — ingest pages to a staging area first; review, promote, or discard before they enter the live wiki | **Yes** | No | No | No |
  155. | **[Scaffold automation](docs/user-quick-start-guide.md#step-14--enrich-the-wiki-with-scaffold)** — regenerates index categories, AGENTS.md/CLAUDE.md/GEMINI.md, and purpose.md from current wiki state; protected pages never overwritten | **Yes** | No | No | No |
  156. | **[Domain templates](synthadoc/templates/README.md)** — 30 fully-authored templates (finance, technology, healthcare, legal, and more) so you start with the right agent guidelines, routing, and structure from day one | **Yes** | No | No | No |
  157. ### Search & Query
  158. | Capability | Synthadoc | Typical RAG | NotebookLM | Notion AI |
  159. | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------- | ---------- | --------- |
  160. | **[Query decomposition + gap detection](docs/user-quick-start-guide.md#knowledge-gap-detection)** — compound questions split into parallel BM25 sub-queries; thin results trigger a knowledge-gap callout with suggested web searches | **Yes** | Partial | No | No |
  161. | **[BM25 TF fallback + compound identifier search](docs/design.md#queryagent)** — reliable results on small corpora (IDF collapse → TF fallback); underscore identifiers expanded at index and query time so `capex growth` matches `capex_growth` | **Yes** | No | No | No |
  162. | **[Web search → wiki pages](docs/user-quick-start-guide.md#step-12--web-search-ingestion)** — Tavily search fans out into parallel URL ingest jobs; gap callout in web UI suggests searches inline | **Yes** | No | No | No |
  163. | **[Semantic search](docs/design.md#semantic-search)** — on by default for new wikis when installed (`pip install "synthadoc[vectors]"`, built into the Docker image): BM25 fused with local embeddings (`BAAI/bge-small-en-v1.5`); Chinese/Japanese/Korean text stays on keyword search; finds pages asked about in your own words, keeps exact-name precision ([benchmark](benchmarks/retrieval/README.md))| **Yes** (optional) | Varies | No | No |
  164. | **[Streaming output + query cache](docs/user-quick-start-guide.md#step-23--query-caching)** — token-by-token streaming; cache key = question + wiki version; auto-invalidates on ingest or lifecycle change | **Yes** | Partial | Partial | Partial |
  165. | **[Proportional context budget](docs/design.md#31-proportional-context-budget)** — sources allocated proportionally to model context window (60 % wiki / 20 % history / 15 % system / 5 % index); replaces fixed top-N cap | **Yes** | No | No | No |
  166. | **[Cross-wiki federation](docs/user-quick-start-guide.md#appendix-l--cross-wiki-queries)** — `synthadoc query --cross-wiki` fans out a single query across all registered running wikis; LLM auto-routes to relevant wikis; knowledge-gap detection with wiki-aware suggested searches across the federation (v1.5.2); `[[wiki-name::PageTitle]]` citation pills; amber offline banner when a target wiki is unreachable; graceful degradation when wikis are offline | **Yes** | No | No | No |
  167. ### Interfaces & Integration
  168. | Capability | Synthadoc | Typical RAG | NotebookLM | Notion AI |
  169. | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ----------- | ---------- | --------- |
  170. | **[Obsidian integration](docs/user-quick-start-guide.md#step-3--open-the-vault-in-obsidian)** — native plugin: ingest modal, streaming query, lint report, lifecycle controls, context pack builder, provenance viewer, export modal, **knowledge graph panel** (Canvas force graph, type filter, hover tooltip, click-to-open page); **background vault monitoring** (auto-snapshot on every file save, 2 s debounce, dedup so unchanged saves are free); Reading View set as default on install so citation chips are visible immediately | **Yes** | No | No | No |
  171. | **[Web chat UI](docs/user-quick-start-guide.md#step-22--use-the-web-chat-ui)** — `synthadoc web`: streaming answers, session sidebar, multi-turn history, knowledge-gap callouts, knowledge graph tab, **light/dark/system theme toggle** | **Yes** | No | Yes | Yes |
  172. | **[MCP server](docs/design.md#27-mcp-server)** — 12 tools; Claude Desktop (stdio), Claude Code (SSE), n8n/LangGraph (HTTP/SSE); brain+memory architecture; no double-LLM cost for reads | **Yes** | No | No | No |
  173. | **[Context packs](docs/user-quick-start-guide.md#step-19--build-a-context-pack)** — goal → sub-questions → token-budget evidence pack; REST + MCP callable; paste into any LLM chat as grounded context | **Yes** | No | No | No |
  174. | **[Export formats](docs/user-quick-start-guide.md#step-21--export-your-wiki)** — `llms.txt`, `llms-full.txt`, GraphML, JSON (provenance + lifecycle), OKF v0.2 bundle; lifecycle-filtered; zero extra LLM calls | **Yes** | No | Partial | No |
  175. | **[Multi-platform agent skill files](docs/design.md#multi-platform-agent-skill-files)** — `AGENTS.md` (Codex/OpenCode), `CLAUDE.md` (Claude Code), `GEMINI.md` (Gemini CLI); all include full CLI quick-reference, domain guidelines, MCP tool table; regenerated by `scaffold` | **Yes** | No | No | No |
  176. ### Content Sources
  177. | Capability | Synthadoc | Typical RAG | NotebookLM | Notion AI |
  178. | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ----------- | ---------- | --------- |
  179. | **[Multi-format ingest](docs/design.md#built-in-skills)** — PDF, DOCX, PPTX, XLSX/CSV, Markdown, TXT, images (vision), web URLs, YouTube transcripts, AI session transcripts (.jsonl) | **Yes** | Varies | Partial | Partial |
  180. | **[Web search decomposition](docs/user-quick-start-guide.md#step-12--web-search-ingestion)** — broad topics decomposed into focused Tavily keyword searches; results merged and deduplicated | **Yes** | No | No | Partial |
  181. | **[YouTube transcript ingest](docs/user-quick-start-guide.md#step-13--ingest-a-youtube-video)** — timestamped transcript + executive summary; no API key; auto-generated captions supported | **Yes** | No | Yes | No |
  182. | **[Multilingual / CJK queries](docs/design.md#queryagent)** — Chinese, Japanese, Korean — no false knowledge gaps | **Yes** | Limited | No | No |
  183. | **[Multiple LLM providers + coding tools](docs/user-quick-start-guide.md#appendix-c--switching-llm-providers)** — Gemini, Groq, Qwen, MiniMax, DeepSeek, Anthropic, OpenAI, Ollama, Kimi; Claude Code and Opencode (no API key needed) | **Yes** | No | No | No |
  184. ### Operations & Trust
  185. | Capability | Synthadoc | Typical RAG | NotebookLM | Notion AI |
  186. | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------ | ---------------------- | --------------------------------------------------------------------------------------------------- |
  187. | **Local-first + offline artifact** — source documents never leave your machine; compiled wiki is plain Markdown, fully readable offline in any editor without the server | **Yes** | Varies | No | No |
  188. | **[Portable backup / restore](docs/design.md#28-backup--restore)** — single zip: wiki pages + audit/lifecycle DB + config; port and domain rewriting on restore; migrate machines without re-ingesting | **Yes** | No — re-ingest required | No — AI metadata lost | No |
  189. | **[Lifecycle content snapshots + rollback](docs/user-quick-start-guide.md#snapshots-and-content-recovery)** — page body captured at every lifecycle transition (manual CLI/Obsidian, lint-driven auto-transition, or Obsidian vault save); deduplicated so unchanged saves cost nothing; browse per-page version history with `lifecycle history` (newest-first, with reason); restore any prior version with `lifecycle rollback`; rollback saves the current body first so it is always undoable | **Yes** | No | No | Partial — Notion has time-based edit history (plan-gated, no lifecycle tie, no auditable rollback) |
  190. | **[Cost guard + full audit trail](docs/user-quick-start-guide.md#step-15--audit-features)** — per-job token + cost log; soft-warn and hard-gate thresholds; `audit citations` validates every claim citation; immutable event log | **Yes** | No | No | No |
  191. | **[Citation faithfulness audit](docs/user-quick-start-guide.md#checking-citation-faithfulness)** — on-demand LLM verification that each claim's text is actually supported by its cited source lines; per-page result cache with staleness detection; findings surfaced in the Obsidian source viewer | **Yes** | No | No | No |
  192. | **[Resumable job queue + retry](docs/design.md#14-job-queue)** — every ingest/lint job persisted with status and error; batch a hundred documents and resume after a crash | **Yes** | No | No | No |
  193. | **[Custom skills + CI hooks](docs/design.md#11-hook-system)** — subclass `BaseSkill` for new file formats; 2 hook events (`on_ingest_complete` + `on_lint_complete`); example git auto-commit hook included; blocking hooks can gate operations | **Yes** | Limited | No | No |
  194. | **[Per-source truncation flag](docs/design.md#30-per-source-truncation-flag)** — `--max-source-chars` caps any source (PDF, DOCX, web page, plain text) before the LLM call; truncated sources flagged with `truncated: true` in frontmatter and warned in lint output | **Yes** | No | No | No |
  195. | **[Docker deployment](docs/docker-deploy.md)** — official `chenp/synthadoc` image (amd64 + arm64); one container per wiki with HTTP API, Web UI, and MCP on one port; Compose recipes for multi-wiki cross-wiki queries and an Ollama sidecar; non-root, secrets via `--env-file` only; startup log names the exact `-p` port mapping the container needs | **Yes** | Varies — usually DIY | No | No |
  196. | **[Multi-wiki isolation](docs/design.md#wiki-targeting)** — each wiki on its own port with independent config, audit trail, and job queue; switch with `synthadoc use` | **Yes** | No | Partial | No |
  197. | **[Agentic maintenance workflows](docs/user-quick-start-guide.md#agentic-workflows)** — nine conversational workflows (stale-page re-ingest, broken wikilinks fix, lint report, scaffold regeneration, **contradiction resolver**, **orphan resolver**, **[broken citation resolver](docs/user-quick-start-guide.md#demo--broken-citation-resolver)**, **[content health resolver](docs/user-quick-start-guide.md#demo--content-health-resolver)**) via web UI or `synthadoc workflow` CLI; confirm-before-act gate on all writes; graph sidebar chips trigger workflows with one click | **Yes** | No | No | No |
  198. | **[Sensitive Data Retract](docs/user-quick-start-guide.md#step-27--scan-and-retract-sensitive-data)** — scan all wiki pages for API keys, emails, SSNs, credit cards, and custom patterns; apply `[REDACTED]` in place; background scan is incremental (only pages modified since the last cycle are checked); server auto-scans ingested pages immediately after each ingest job completes; `--changed-only` CLI flag for incremental manual runs; `synthadoc retract status` captures the full retraction audit trail (slug, pattern names, counts — never the sensitive values themselves) | **Yes** | No | No | No |
  199. ### Business value
  200. | Value | How |
  201. | --------------------- | ------------------------------------------------------------------------------- |
  202. | **Faster onboarding** | New team members query the wiki instead of digging through documents |
  203. | **Audit trail** | Every ingest recorded in`audit.db` with source hash, token count, and timestamp |
  204. | **Cost control** | Configurable thresholds; 3-layer cache reduces repeat spend |
  205. | **Compliance** | Local-first — source documents and compiled knowledge never leave your machine |
  206. | **Extensibility** | Hooks fire on every event; custom skills load without a server restart |
  207. ---
  208. ## Architecture
  209. ![Synthadoc Architecture](docs/png/architecture-overview.png)
  210. For full architecture details, data models, API reference, and plugin development guide see **[docs/design.md](docs/design.md)**.
  211. ---
  212. ## What's Included
  213. See [docs/design.md — Appendix A: Release Feature Index](docs/design.md#appendix-a--release-feature-index) for a full feature list by version.
  214. ---
  215. ## Installation
  216. Choose one way to run Synthadoc:
  217. | | Install with pip | Run with Docker |
  218. | --- | --- | --- |
  219. | **Host requirements** | Python 3.11+ | A container runtime only, with **no Python or pip on the host** |
  220. | **Best for** | Personal use on one machine | Team or shared servers, CI/CD, isolated or reproducible runtimes |
  221. | **Steps** | [Production](#production), then [Set your API keys](#set-your-api-keys) and [Install a wiki and start the engine](#install-a-wiki-and-start-the-engine) | [Docker](#docker) below, then the [Docker Deployment Guide](docs/docker-deploy.md) |
  222. ### Production
  223. **Prerequisites:** Python 3.11+. No Node.js, no Git, no build steps.
  224. ```bash
  225. pip install "synthadoc[vectors]" # recommended — includes semantic search
  226. pip install synthadoc # lightweight install, no semantic search
  227. synthadoc --version # confirm it works
  228. ```
  229. The Obsidian plugin is bundled inside the package. New wikis created with `synthadoc install` have it installed automatically. If you are upgrading an existing Synthadoc installation, run `synthadoc plugin upgrade` to push the updated plugin binary to all registered wikis.
  230. ### Docker
  231. **Prerequisites:** a container runtime only (Docker Engine or Desktop, Apple `container`, or `wslc`). No Python, pip, or `synthadoc` install on the host: every command, including creating the wiki, runs in a container.
  232. For server deployments, team wikis, CI/CD pipelines, and isolated personal use, Synthadoc ships an OCI image, [`chenp/synthadoc`](https://hub.docker.com/r/chenp/synthadoc), for `linux/amd64` and `linux/arm64`. It works with Docker Engine and Docker Desktop, the Apple `container` tool (macOS 26+ Apple Silicon), and WSL Containers (`wslc`, Windows 11). One container serves one wiki. The HTTP API, Web UI (`/app`), and MCP over HTTP (`/mcp`) all share the wiki's port, and the wiki files stay on the host so Obsidian keeps working unchanged.
  233. Commands are shown **Windows first**, then **Linux / macOS**. Notes on Windows: `~` in `-v` paths and `\` line continuation don't work in `cmd.exe`, so Windows commands use `"%USERPROFILE%\wikis\…"` and are single-line.
  234. **1. Create the wiki** in a one-off container. It lands in `%USERPROFILE%\wikis\history-of-computing` (Windows) or `~/wikis/history-of-computing` (Linux/macOS); note the port it prints, e.g. `7070`. `docker run` downloads the image on first use. Run `docker pull chenp/synthadoc:latest` later to update a cached image.
  235. **Windows:**
  236. ```bat
  237. if not exist "%USERPROFILE%\wikis" mkdir "%USERPROFILE%\wikis"
  238. docker run --rm -v "%USERPROFILE%\wikis:/wikis" chenp/synthadoc:latest synthadoc install history-of-computing --target /wikis --demo
  239. ```
  240. **Linux / macOS:**
  241. ```bash
  242. mkdir -p ~/wikis
  243. docker run --rm -v ~/wikis:/wikis chenp/synthadoc:latest \
  244. synthadoc install history-of-computing --target /wikis --demo
  245. ```
  246. **2. Put your LLM API key in `~/wikis/.env`**, never on the command line. Create it with any text editor (Windows: `notepad "%USERPROFILE%\wikis\.env"`). For example, to use MiniMax:
  247. ```bash
  248. # ~/wikis/.env
  249. MINIMAX_API_KEY=sk-...
  250. # optional — enables web search ingest
  251. TAVILY_API_KEY=tvly-...
  252. ```
  253. Keep comments on their own line; `--env-file` does not strip a trailing `# …` from a value.
  254. Then select the **same provider** in `~/wikis/history-of-computing/.synthadoc/config.toml`. Under `[agents]`, comment out the active `default =` line and uncomment the matching one:
  255. ```toml
  256. [agents]
  257. default = { provider = "minimax", model = "MiniMax-M3", thinking = "disabled" }
  258. ```
  259. **3. Run the server in a container**, using the same port on both sides of `-p`:
  260. **Windows:**
  261. ```bat
  262. docker run -d --name history-of-computing -v "%USERPROFILE%\wikis\history-of-computing:/wiki" -p 7070:7070 --env-file "%USERPROFILE%\wikis\.env" chenp/synthadoc:latest
  263. ```
  264. **Linux / macOS:**
  265. ```bash
  266. docker run -d --name history-of-computing \
  267. -v ~/wikis/history-of-computing:/wiki \
  268. -p 7070:7070 \
  269. --env-file ~/wikis/.env \
  270. chenp/synthadoc:latest
  271. ```
  272. **4. Use it.** Open the Web UI at `http://localhost:7070/app`, or run the CLI inside the container (same on every platform):
  273. ```bash
  274. docker exec history-of-computing synthadoc query "Who was Alan Turing?"
  275. docker exec history-of-computing synthadoc lint run --wait
  276. ```
  277. After editing `config.toml`, run `docker restart history-of-computing`. After changing `.env`, remove and re-create the container (`docker rm -f history-of-computing`, then step 3 again), because `restart` does not re-read `.env`.
  278. See **[docs/docker-deploy.md](docs/docker-deploy.md)** for the full deployment guide: architecture, API key setup, platform-native runtimes, Docker Compose (single wiki, multi-wiki with cross-wiki queries, Ollama sidecar), CI/CD, file permissions, security, and maintenance.
  279. > **Docker users:** the rest of this Installation section (Development, *Set your API keys* with shell `export`s, and *Install a wiki and start the engine*) is for the pip install. With Docker, API keys go in the `.env` file, and any `synthadoc …` command elsewhere in this README runs as `docker exec <container> synthadoc …` (no `-w` needed).
  280. ---
  281. ### Development
  282. For developers modifying the Python engine, running the test suite, or developing the Obsidian plugin TypeScript.
  283. **Additional prerequisites:** Git (any). Node.js 18+ only if modifying the plugin TypeScript source.
  284. #### Step 1 — Clone and install
  285. ```bash
  286. git clone https://github.com/axoviq-ai/synthadoc.git
  287. cd synthadoc
  288. pip install -e ".[dev]"
  289. ```
  290. `[dev]` adds `pytest`, `respx`, and the other test dependencies. Tests require a source checkout — they are not included in the pip wheel.
  291. #### Step 2 — Run the test suite
  292. ```bash
  293. pytest --ignore=tests/performance/ -q
  294. ```
  295. Expected: all tests pass, 0 failures. Performance benchmarks (optional):
  296. ```bash
  297. pytest tests/performance/ -v --benchmark-disable
  298. ```
  299. #### Step 3 — Obsidian plugin development (optional)
  300. The compiled plugin is bundled with the package and kept up to date by CI — no build step is needed to work on the Python side or run tests.
  301. If you modify the TypeScript source under `obsidian-plugin/src/`, recompile and sync it into the Python package manually:
  302. ```bash
  303. cd obsidian-plugin
  304. npm install # first time only, or after package.json changes
  305. # edit src/main.ts or other source files
  306. npm run build # compile TypeScript → main.js (also syncs to synthadoc/data/obsidian-plugin/)
  307. npm test # run Vitest unit tests
  308. cd ..
  309. ```
  310. ---
  311. ### Set your API keys
  312. **At least one LLM API key is required** — unless you use Claude Code or Opencode as your provider (no separate API key needed — see [Coding tool CLI providers](docs/design.md#coding-tool-cli-providers--no-api-key-needed)).
  313. Synthadoc defaults to **Gemini Flash** — free tier, no credit card, 1 million tokens per day. Get a key at **aistudio.google.com/app/apikey** (click "Create API key").
  314. | Provider | Free tier | Vision | Get key |
  315. | ---------------- | --------------------------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
  316. | **Gemini Flash** | Yes — 15 RPM / 1M tokens/day, no credit card | Yes | [aistudio.google.com](https://aistudio.google.com/app/apikey) |
  317. | Groq | Yes — rate-limited | No | [console.groq.com](https://console.groq.com/keys) |
  318. | Ollama | Yes — runs locally, no key (**GPU required**) | Model-dependent | [ollama.com](https://ollama.com) |
  319. | Qwen | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent | [bailian.console.aliyun.com](https://bailian.console.aliyun.com/) |
  320. | MiniMax | No — pay-per-token | Yes | [platform.minimax.io](https://platform.minimax.io/) |
  321. | DeepSeek | No — pay-per-token (very cheap text rates) | No | [platform.deepseek.com](https://platform.deepseek.com/api_keys) |
  322. | Kimi | No — pay-per-token; `kimi-k3` ($3/$15/1M, 1M ctx), `kimi-k2.6` ($0.95/$4/1M) | No | [bailian.console.aliyun.com](https://bailian.console.aliyun.com/) |
  323. | Anthropic | No | Yes | [console.anthropic.com](https://console.anthropic.com/) |
  324. | OpenAI | No | Yes | [platform.openai.com](https://platform.openai.com/api-keys) |
  325. | **Claude Code** | Included with subscription — no API key | No | Set`provider = "claude-code"` in config.toml |
  326. | **Opencode** | Free via Opencode Zen — no API key | No | Set`provider = "opencode", model = "opencode/big-pickle"` in config.toml; connect first: run `opencode` → `/connect` → select Zen |
  327. ```bash
  328. # macOS / Linux — add to ~/.bashrc or ~/.zshrc to persist
  329. export GEMINI_API_KEY=AIza… # default — free tier, 1M tokens/day
  330. export GROQ_API_KEY=gsk_… # alternative free tier — 100K tokens/day
  331. export ANTHROPIC_API_KEY=sk-ant-… # paid — highest quality
  332. export OPENAI_API_KEY=sk-… # paid
  333. export MINIMAX_API_KEY=… # paid — text rates (image support)
  334. export DEEPSEEK_API_KEY=… # paid — text rates (no image support)
  335. export QWEN_API_KEY=… # DashScope cloud — 1M free tokens trial (DASHSCOPE_API_KEY also accepted)
  336. export KIMI_API_KEY=… # Kimi (DashScope) — kimi-k3 flagship, kimi-k2.6 general (QWEN_API_KEY also accepted)
  337. export TAVILY_API_KEY=tvly-… # web search (optional)
  338. # Windows cmd — current session only
  339. set GEMINI_API_KEY=AIza…
  340. set GROQ_API_KEY=gsk_…
  341. set ANTHROPIC_API_KEY=sk-ant-…
  342. set OPENAI_API_KEY=sk-…
  343. set MINIMAX_API_KEY=…
  344. set DEEPSEEK_API_KEY=…
  345. set QWEN_API_KEY=… # DASHSCOPE_API_KEY also accepted
  346. set KIMI_API_KEY=… # QWEN_API_KEY also accepted
  347. set TAVILY_API_KEY=tvly-…
  348. ```
  349. Web search uses **Tavily** (`TAVILY_API_KEY`) — optional, only needed for `synthadoc ingest "search for: …"` jobs. Get a free key at [tavily.com](https://tavily.com).
  350. ---
  351. ### Install a wiki and start the engine
  352. A **wiki** is a self-contained knowledge base — a folder of Markdown pages maintained and cross-referenced automatically by Synthadoc. The fastest way to get started is the **History of Computing** demo (13 pre-built pages, no LLM API key required to browse).
  353. ```bat
  354. REM Windows
  355. synthadoc install history-of-computing --target %USERPROFILE%\wikis --demo
  356. ```
  357. ```bash
  358. # Linux / macOS
  359. synthadoc install history-of-computing --target ~/wikis --demo
  360. ```
  361. Then start the engine:
  362. ```bash
  363. # Foreground — logs stream to the console
  364. synthadoc serve -w history-of-computing
  365. # Background — releases the terminal
  366. synthadoc serve -w history-of-computing --background
  367. ```
  368. The server binds to `http://localhost:7070` (localhost-only). Leave it running while you work — the Obsidian plugin, CLI ingest commands, and query commands all talk to it.
  369. To switch LLM provider, edit `[agents]` in `<wiki-root>/.synthadoc/config.toml` and restart `synthadoc serve`. See [Appendix — Switching LLM providers](docs/user-quick-start-guide.md#appendix-c--switching-llm-providers) for step-by-step instructions.
  370. To stop a background server:
  371. ```bash
  372. synthadoc stop -w history-of-computing
  373. ```
  374. **Upgrading:** after updating synthadoc (via `pip install --upgrade "synthadoc[vectors]"` or `git pull`), restart the server to pick up the new code, then run these to keep registered wikis in sync:
  375. ```bash
  376. synthadoc plugin upgrade # push updated Obsidian plugin binary to all registered wikis
  377. synthadoc templates sync --force # demo and template wikis — refresh seeds, routing, agent files, and stubs
  378. ```
  379. Neither command requires the server to be running.
  380. **Upgrading a Docker deployment:** `docker pull chenp/synthadoc:latest`, then remove and re-create the container with the same `docker run` command; wiki data on the host is untouched. Then run inside the container to sync the plugin and templates:
  381. ```bash
  382. docker exec <container> synthadoc plugin upgrade # push updated plugin binary to the wiki vault
  383. docker exec <container> synthadoc templates sync # refresh seeds, routing, agent files, and stubs
  384. ```
  385. See [Docker Deployment Guide — Updating the image](docs/docker-deploy.md#updating-the-image).
  386. ---
  387. ## Quick-Start Guide
  388. The **History of Computing** demo includes 13 pre-built pages, raw source files covering clean-merge, contradiction, and orphan scenarios, and a full walkthrough of key Synthadoc feature.
  389. **Full step-by-step walkthrough: [docs/user-quick-start-guide.md](docs/user-quick-start-guide.md)**
  390. The guide covers:
  391. 1. Verify the demo server started (banner, health check)
  392. 2. Install the Synthadoc plugin (auto-installs Dataview) and open the vault
  393. 3. Review wiki structure and key files (index, purpose, AGENTS.md, dashboard)
  394. 4. Query the pre-built wiki — including knowledge gap detection
  395. 5. Batch ingest all demo source files
  396. 6. Run lint — auto-promote clean pages to active
  397. 7. Manage page lifecycle — 5-state machine (draft → active → stale/contradicted/archived), manual transitions, immutable audit trail
  398. 8. Resolve a contradiction
  399. 9. Fix an orphan page
  400. 10. Run the adversarial lint pass — flag overstated claims across all pages
  401. 11. Web search ingestion with automatic decomposition
  402. 12. Ingest a YouTube video
  403. 13. Enrich the wiki with scaffold (regenerate/update index, purpose, AGENTS.md)
  404. 14. Audit features (token cost, history, events)
  405. 15. Schedule recurring operations
  406. 16. Set up query-scoped routing with ROUTING.md
  407. 17. Stage and review candidate pages before promoting them
  408. 18. Build a context pack for grounded LLM prompts
  409. 19. Verify claim provenance — source-line citations, broken citation audit, global provenance table
  410. 20. Export your wiki — llms.txt, llms-full.txt, GraphML wikilink graph, agent-ready JSON with provenance and lifecycle history, OKF v0.2 bundle for zero-code agent consumption
  411. 21. Use the web chat UI — streaming answers, session-aware hint chips, citations in-browser
  412. 22. Query caching — understand how answers are cached and how to bypass with `--no-cache`
  413. 23. Backup and restore — create a portable wiki zip, restore on a different machine
  414. 24. Knowledge graph — weighted edges (wikilink + co-source signals), explore clusters in the web UI Graph tab, click a node to query it; large wikis open on a cluster overview
  415. 25. Ingest an AI session transcript — turn Claude Code, Codex, or chat conversations into structured wiki pages
  416. 26. Agentic maintenance workflows — nine conversational workflows via web UI or CLI: re-ingest stale pages (bulk or by slug), scan and fix broken wikilinks, run lint and view the full report, run scaffold and regenerate wiki files, resolve contradicted pages interactively (diff shown before every write, human approval required), resolve orphaned pages by inserting natural wikilinks into related content, fix broken source citation markers, detect and repair content quality issues (duplicate sections, thin/empty sections, broken markdown structure) with LLM-assisted analysis — agent confirms before touching anything, streams inline progress
  417. 27. Scan and retract sensitive data — dry-run to surface API keys, emails, SSNs, credit cards, and custom patterns; apply redactions with a single confirmation; incremental `--changed-only` mode for recurring runs; audit trail records pattern names only (never the values)
  418. ---
  419. ## Creating Your Own Wiki
  420. > **New to building your own wiki?** Work through the [AquaFlow Capital Workshop Walkthrough](https://github.com/axoviq-ai/synthadoc/tree/main/docs/example/aquaflow) first — a complete end-to-end workshop using a real M&A due-diligence wiki with pre-built pages, evaluation queries, and benchmark results. Once you are comfortable with the flow, come back here to build your own wiki from scratch.
  421. ### 1. Pick a template and install
  422. Browse the 30 available templates across 9 categories and pick the one closest to your domain:
  423. ```bash
  424. synthadoc templates list
  425. ```
  426. Install with a template (example: consumer and competitive market research):
  427. ```bash
  428. synthadoc install my-market-wiki --target ~/wikis --template research/market-research
  429. synthadoc use my-market-wiki # set as default — no -w needed from here on
  430. ```
  431. If none of the 30 templates fits, install without one and provide a `--domain` description instead:
  432. ```bash
  433. synthadoc install my-wiki --target ~/wikis --domain "Your domain description"
  434. ```
  435. → Full template catalog and per-category install commands: [synthadoc/templates/README.md](synthadoc/templates/README.md)
  436. ### 2. Configure your LLM provider and start the server
  437. Open `~/.wikis/my-market-wiki/.synthadoc/config.toml` and set your LLM provider, then start:
  438. ```bash
  439. synthadoc serve
  440. synthadoc status # should show 0 pages and the scheduled jobs registered
  441. ```
  442. → Provider list and API key setup: [Quick-Start Guide — Appendix C](docs/user-quick-start-guide.md#appendix-c--switching-llm-providers)
  443. `synthadoc install` copies the Synthadoc and Dataview plugins into the vault and pre-enables them. Open the wiki folder in Obsidian — both plugins are active immediately. A local web UI is also available at `http://localhost:{port}/app` via `synthadoc web`.
  444. ### 3. Add your sources and ingest
  445. > **Start with `seeds.md`.** Your installed wiki includes a `seeds.md` file at the wiki root — open it before anything else. It contains the complete getting-started guide for your domain: pre-vetted copy-paste ingest commands for curated public URLs, a domain-specific first-steps checklist, and topic hints for finding additional sources.
  446. >
  447. > Open it in Obsidian (file pane → `seeds.md`) or in any editor:
  448. >
  449. > ```
  450. > <your-wiki-folder>/seeds.md
  451. > ```
  452. Once you have worked through `seeds.md`, drop your own documents into the `raw_sources/` folder and ingest them. Subfolders are supported — organise by topic, date, or document type:
  453. ```bash
  454. synthadoc ingest raw_sources/ --batch # all local files at once
  455. synthadoc ingest "https://example.com/report" # a web URL
  456. synthadoc ingest "search for: Bank of Canada rate outlook 2025" # web search
  457. synthadoc jobs list # watch progress
  458. ```
  459. → Supported formats, subfolder layout, and web search tips: [templates/README.md — Adding your own content](synthadoc/templates/README.md#adding-your-own-content)
  460. ### 4. Review and promote candidates
  461. Template installs stage every new page in `candidates/` for review before it enters the wiki. Promote what belongs, discard what doesn't:
  462. ```bash
  463. synthadoc candidates list
  464. synthadoc candidates promote --all # or promote individually by slug
  465. synthadoc candidates discard <slug>
  466. ```
  467. → CLI commands and Obsidian plugin walkthrough: [templates/README.md — Reviewing and promoting candidates](synthadoc/templates/README.md#reviewing-and-promoting-candidates)
  468. ### 5. Query, lint, and scaffold
  469. ```bash
  470. synthadoc query "What are the current consumer confidence trends in Canada?"
  471. synthadoc lint run # validates pages, promotes clean drafts to active
  472. synthadoc lint report # view issues and citation violations
  473. synthadoc scaffold # regenerates index.md with richer category links
  474. ```
  475. Lint and scaffold run automatically on a weekly schedule (Sunday 2 AM and 3 AM). Your own content above the `<!-- synthadoc:scaffold -->` marker in `index.md` and `purpose.md` is never overwritten.
  476. → Schedule customisation and scaffold zone rules: [templates/README.md — Scheduled maintenance](synthadoc/templates/README.md#scheduled-maintenance)
  477. ### 6. Context packs and MCP
  478. Template installs include a pre-configured `ROUTING.md` that automatically scopes queries to the relevant topic branch — no setup needed. If you installed without a template, generate it once with `synthadoc routing init`.
  479. Build a cited excerpt pack for use in an external agent prompt:
  480. ```bash
  481. synthadoc context build "Canadian consumer confidence trends" --tokens 4000
  482. ```
  483. Connect Claude Code directly to your wiki's MCP server:
  484. ```bash
  485. # Replace 7070 with the port shown when you ran synthadoc serve
  486. claude mcp add --transport sse synthadoc-my-market-wiki http://localhost:7070/mcp/sse
  487. ```
  488. → Routing performance benchmarks: [Quick-Start Guide — Appendix H](docs/user-quick-start-guide.md#appendix-h--bm25-routing-performance-benchmarks)
  489. → Context packs and MCP integration: [docs/design.md — Context packs](docs/design.md#context-packs)
  490. → Knowledge gap workflow: [docs/design.md — Knowledge gap workflow](docs/design.md#knowledge-gap-workflow)
  491. → Semantic search: [docs/design.md — Semantic search](docs/design.md#semantic-search)
  492. → How retrieval quality and speed are measured, and how to rerun it: [benchmarks/README.md](benchmarks/README.md)
  493. ---
  494. ## Configuration
  495. You do not need to configure anything to run the demo. The demo wiki ships with its own settings and sensible built-in defaults cover everything else. Set your API key env var, run `synthadoc serve`, and go.
  496. For the full configuration reference — layer precedence, global vs. per-project config, all keys and defaults — see [Appendix E — Configuration](docs/user-quick-start-guide.md#appendix-e--configuration) in the Quick-Start Guide, or [docs/design.md — Configuration](docs/design.md#configuration) for the complete technical reference.
  497. ---
  498. ## Command Reference by Use Case
  499. > **Full CLI command tree** (all subcommands, flags, and groupings): [docs/design.md — Command tree](docs/design.md#command-tree)
  500. ### Setting up a wiki
  501. ```bash
  502. # Create a new wiki (LLM scaffold runs automatically; port is auto-assigned to avoid conflicts)
  503. synthadoc install my-wiki --target ~/wikis --domain "Machine Learning"
  504. # Pin a specific port manually
  505. synthadoc install my-wiki --target ~/wikis --domain "Machine Learning" --port 7071
  506. # Install the demo (includes pre-built pages and raw sources — no LLM call needed)
  507. synthadoc install history-of-computing --target ~/wikis --demo
  508. # List available demo templates
  509. synthadoc demo list
  510. # List all 30 domain templates (and demos) — browse before installing
  511. synthadoc templates list
  512. # Install with a domain template — pre-configured guidelines, routing, and starter pages
  513. synthadoc install my-finance-wiki --target ~/wikis --template finance/investment
  514. # Sync a demo or template wiki — refresh seeds, routing, agent files, and intake forms
  515. synthadoc templates sync history-of-computing
  516. # Also refresh stub pages that have never been ingested (sources: [])
  517. synthadoc templates sync history-of-computing --force
  518. # Reinstall the Obsidian plugin into a wiki's vault — normally done automatically by synthadoc install
  519. synthadoc plugin install history-of-computing
  520. # Push the updated plugin binary to every registered wiki after a Synthadoc upgrade
  521. synthadoc plugin upgrade
  522. ```
  523. ### Switching the active wiki
  524. ```bash
  525. # Set a wiki as the default so -w is not required for any subsequent command
  526. synthadoc use my-wiki
  527. # Check which wiki is currently active
  528. synthadoc use
  529. # Clear the saved default (revert to requiring -w on every command)
  530. synthadoc use --clear
  531. ```
  532. ### Refreshing wiki scaffold
  533. After install, you can re-run the LLM scaffold at any time to regenerate domain-specific content (index categories, AGENTS.md guidelines, purpose.md scope). Pages already linked in `index.md` are protected and preserved.
  534. ```bash
  535. # Regenerate scaffold for an existing wiki
  536. synthadoc scaffold -w my-wiki
  537. # Schedule weekly refresh (runs every Sunday at 4 AM)
  538. synthadoc schedule add --op "scaffold" --cron "0 4 * * 0" -w my-wiki
  539. ```
  540. `config.toml` and `dashboard.md` are never modified by `scaffold`.
  541. ### Running the server
  542. ```bash
  543. # Start HTTP API + job worker (foreground — terminal stays attached)
  544. synthadoc serve -w my-wiki
  545. # Detach to background — banner shown, then shell is released
  546. # All logs go to <wiki>/.synthadoc/logs/synthadoc.log
  547. synthadoc serve -w my-wiki --background
  548. # Custom port
  549. synthadoc serve -w my-wiki --port 7071
  550. # Bind to all interfaces (team server, remote access, containers) — default is 127.0.0.1
  551. synthadoc serve -w my-wiki --host 0.0.0.0
  552. # Verbose debug logging to console
  553. synthadoc serve -w my-wiki --verbose
  554. # Start servers for all registered wikis in the background (cross-wiki queries).
  555. # Wikis start one at a time and stop before the machine passes ~60% memory use;
  556. # skipped wikis are listed with the command to start them.
  557. synthadoc serve --all
  558. # Same, but cap the total memory the started wikis may use (a unit is required: GB or MB)
  559. synthadoc serve --all --max-memory 6GB
  560. # Also show each server's PID and log path (hidden by default; only useful when debugging)
  561. synthadoc serve --all --verbose
  562. # Stop the active wiki server
  563. synthadoc stop
  564. # Stop a named wiki server
  565. synthadoc stop -w my-wiki
  566. # Stop all running wiki servers
  567. synthadoc stop --all
  568. ```
  569. ### Ingesting sources
  570. ```bash
  571. # Single file or URL
  572. synthadoc ingest report.pdf -w my-wiki
  573. synthadoc ingest https://example.com/article -w my-wiki
  574. # Entire folder — enqueues each file; the worker processes jobs sequentially
  575. synthadoc ingest --batch raw_sources/ -w my-wiki
  576. # Manifest file — ingest a curated list of sources in one shot.
  577. # sources.txt: one entry per line; each line is either an absolute file path
  578. # (PDF, DOCX, PPTX, MD, …) or a URL. Blank lines and # comments are ignored.
  579. # Each entry becomes a separate job in the queue, processed sequentially.
  580. #
  581. # Example sources.txt:
  582. # /home/user/docs/research-paper.pdf
  583. # /home/user/slides/keynote.pptx
  584. # https://en.wikipedia.org/wiki/Alan_Turing
  585. # # this line is ignored
  586. synthadoc ingest --file sources.txt -w my-wiki
  587. # Force re-ingest (bypass deduplication and cache)
  588. synthadoc ingest --force report.pdf -w my-wiki
  589. # Web search — triggers a Tavily search, then ingests each result URL as a child job.
  590. # Prefix the query with any recognised intent: "search for:", "find on the web:",
  591. # "look up:", or "web search:" (prefix is stripped before the search is sent)
  592. # Requires TAVILY_API_KEY to be set.
  593. #
  594. # Note: web search content is NOT saved to raw_sources/. The flow is direct:
  595. # query → Tavily → URLs → each URL fetched → wiki pages written
  596. # raw_sources/ is for user-provided local files (PDF, DOCX, PPTX, etc.) only.
  597. # The wiki pages themselves are the persistent output of a web search.
  598. synthadoc ingest "search for: Bank of Canada interest rate decisions 2024" -w my-wiki
  599. synthadoc ingest "find on the web: unemployment trends Ontario Q1 2025" -w my-wiki
  600. # Limit how many URLs are enqueued (default 20, overrides [web_search] max_results)
  601. synthadoc ingest "search for: quantum computing basics" --max-results 5 -w my-wiki
  602. # Multiple web searches at once via a manifest file
  603. # web-searches.txt:
  604. # search for: Bank of Canada interest rate decisions 2024
  605. # find on the web: unemployment trends Ontario Q1 2025
  606. # look up: Toronto housing market affordability index
  607. synthadoc ingest --file web-searches.txt -w my-wiki
  608. # YouTube video — transcript extracted automatically, no API key needed.
  609. # The video must have captions (auto-generated or manual).
  610. # Check: open the video on YouTube → ... → Show transcript.
  611. synthadoc ingest "https://www.youtube.com/watch?v=O5nskjZ_GoI" -w my-wiki
  612. synthadoc ingest "https://youtu.be/O5nskjZ_GoI" -w my-wiki
  613. # YouTube URLs returned by web search are also routed automatically:
  614. # if Tavily returns a YouTube URL, the transcript is ingested instead of the page HTML.
  615. synthadoc ingest "search for: history of computing lecture" -w my-wiki
  616. ```
  617. Each YouTube wiki page opens with an **executive summary** — what the video is about,
  618. the main topics covered, and the key takeaway — followed by the full timestamped transcript
  619. for precise citation.
  620. ### Querying
  621. ```bash
  622. # Ask a question — answer streams token-by-token as the LLM generates it
  623. synthadoc query "What is Moore's Law?" -w my-wiki
  624. # Blocking mode (no streaming) — useful in scripts or pipes
  625. synthadoc query "What is Moore's Law?" --no-stream -w my-wiki
  626. # Skip the cache — always call the LLM even if the answer is cached
  627. synthadoc query "What is Moore's Law?" --no-cache -w my-wiki
  628. # Save the answer as a new wiki page
  629. synthadoc query "What is Moore's Law?" --save -w my-wiki
  630. ```
  631. Query answers are **cached automatically** by question content and wiki version. Repeated identical questions return instantly from cache. The cache invalidates automatically when you ingest new content or change a page's lifecycle state.
  632. ### Cross-wiki queries
  633. Fan-out a single question across all registered running wikis and receive one synthesised answer with per-wiki citations.
  634. ```bash
  635. # Query across all running registered wikis
  636. synthadoc query --cross-wiki "What are our M&A covenants and deployment runbooks?" -w my-wiki
  637. # Check which wikis are running
  638. synthadoc status --all
  639. # Manage the optional CROSS_WIKI_ROUTING.md override
  640. synthadoc cross-wiki routing init # generate from registry, then edit
  641. synthadoc cross-wiki routing show # print current contents
  642. synthadoc cross-wiki routing edit # open in $EDITOR
  643. ```
  644. Citations in cross-wiki answers use the format `[[wiki-name::PageTitle]]`. In the web UI these render as two-part pills; clicking opens the page in that wiki's server. When a target wiki is offline, an amber warning appears and the answer is synthesised from the available wikis only.
  645. For full architecture detail see [Cross-Wiki Queries in the design doc](docs/design.md#40-cross-wiki-queries-v140).
  646. ### Web Chat UI
  647. ```bash
  648. # Open the browser-based chat interface for a wiki
  649. synthadoc web -w my-wiki
  650. ```
  651. → Full walkthrough (session modes, multi-turn conversation, session history sidebar): [Quick-Start Guide — Step 22](docs/user-quick-start-guide.md#step-22--use-the-web-chat-ui)
  652. ### Linting
  653. ```bash
  654. # Run a full lint pass (enqueues job)
  655. synthadoc lint run -w my-wiki
  656. # Only contradictions
  657. synthadoc lint run --scope contradictions -w my-wiki
  658. # Auto-apply high-confidence resolutions
  659. synthadoc lint run --auto-resolve -w my-wiki
  660. # Skip adversarial review (structural checks only; also clears existing warnings)
  661. synthadoc lint run --no-adversarial -w my-wiki
  662. # Skip lifecycle checks (structural and adversarial checks only)
  663. synthadoc lint run --no-lifecycle -w my-wiki
  664. # Block until lint finishes, printing phase progress (scripts, CI, docker exec)
  665. # Exits with code 1 if the server stays unreachable
  666. synthadoc lint run --wait -w my-wiki
  667. # Instant report (reads wiki files directly, no server needed)
  668. synthadoc lint report -w my-wiki
  669. ```
  670. ### Agentic workflows
  671. Agentic workflows run as interactive tool-call loops — they stream progress inline, ask for approval before any destructive change, and exit cleanly if you decline.
  672. ```bash
  673. # List all available workflows (with descriptions and accepted parameters)
  674. synthadoc workflow list
  675. # Run a workflow by name
  676. synthadoc workflow run --name lint-report -w my-wiki
  677. synthadoc workflow run --name broken-wikilinks -w my-wiki
  678. synthadoc workflow run --name scaffold -w my-wiki
  679. synthadoc workflow run --name ingest-lint -w my-wiki
  680. # Contradiction resolver — resolve all contradicted pages
  681. synthadoc workflow run --name contradiction-resolver -w my-wiki
  682. # Resolve only one page by slug
  683. synthadoc workflow run --name contradiction-resolver --slug alan-turing -w my-wiki
  684. # Scope to pages demoted by the adversarial gate
  685. synthadoc workflow run --name contradiction-resolver --type adversarial -w my-wiki
  686. # Scope to pages with source-level contradictions
  687. synthadoc workflow run --name contradiction-resolver --type source-conflict -w my-wiki
  688. # Custom timeout (default 3600 s)
  689. synthadoc workflow run --name contradiction-resolver --timeout 7200 -w my-wiki
  690. # Orphan resolver — find pages with no incoming wikilinks and insert natural links
  691. synthadoc workflow run --name orphan-resolver -w my-wiki
  692. # Resolve only one orphaned page by slug
  693. synthadoc workflow run --name orphan-resolver --slug ada-lovelace -w my-wiki
  694. # Broken citation resolver — fix broken ^[file:L-L] source markers (broken_ref / malformed / out_of_range)
  695. synthadoc workflow run --name broken-citation-resolver -w my-wiki
  696. # Fix citations on a single page only
  697. synthadoc workflow run --name broken-citation-resolver --slug my-page -w my-wiki
  698. # Content health resolver — merge duplicate sections, fix empty sections and broken markdown structure
  699. synthadoc workflow run --name content-health -w my-wiki
  700. # Fix content issues on a single page only
  701. synthadoc workflow run --name content-health --slug alan-turing -w my-wiki
  702. # List every content health fix (timestamp + pre-fix snapshot)
  703. synthadoc workflow run --name content-health list -w my-wiki
  704. # Undo one content health fix — restores the page's pre-fix snapshot
  705. synthadoc workflow run --name content-health roll back alan-turing -w my-wiki
  706. ```
  707. The content health resolver works on pages already flagged by a fast heuristic scanner (no LLM) that runs after every ingest, after every workflow write, and on every full lint pass. It detects duplicate sections, empty or thin sections, unclosed code blocks, orphaned headings, and broken list nesting. Duplicate sections are merged in Python with no LLM, keeping every sentence; the other structural issues get one LLM rewrite per page. Each fix is presented as individual per-block edits with per-block include/skip toggles — or switch to a full-page CodeMirror editor via "Edit full page manually" — and written only after you approve. Any rewrite that would drop a sentence is rejected. Thin sections need new facts, so the resolver leaves them to you and the lint report lists them. A snapshot is saved before every write, so any fix can be rolled back later.
  708. → Full walkthrough and web UI path: [Quick-Start Guide — Demo: content health resolver](docs/user-quick-start-guide.md#demo--content-health-resolver)
  709. The contradiction resolver lists contradicted pages, estimates cost, then walks through each page: reads the content and conflicting sources, proposes a rewrite, presents the changes as individual per-block edits with per-block include/skip toggles, and waits for your approval. After an approved write it re-lints the page and promotes it to *active* on pass, or escalates after 3 failed attempts. Any rewrite that mutates a heading is rejected before you are asked, and every write is snapshotted — say "roll back `<slug>`" to undo a fix.
  710. → Full walkthrough and web UI path: [Quick-Start Guide — Step 9](docs/user-quick-start-guide.md#step-9--resolve-a-contradiction)
  711. The orphan resolver finds pages with no inbound wikilinks and searches for the best page to link each one from, using up to four progressively broader strategies (title match, content match, then two LLM-assisted scans). It shows the full diff for each proposed `[[wikilink]]` insertion and waits for your approval. Every approved write is snapshotted the same way — say "roll back `<slug>`" to undo it.
  712. → Full walkthrough and web UI path: [Quick-Start Guide — Step 10](docs/user-quick-start-guide.md#step-10--fix-an-orphan-page)
  713. ### Sensitive data retraction
  714. Scan wiki pages for API keys, emails, SSNs, credit card numbers, and custom patterns. Apply `[REDACTED]` in place after confirmation. The server also scans pages automatically after each ingest job.
  715. ```bash
  716. # Dry-run — show pattern matches without making changes
  717. synthadoc retract scan -w my-wiki
  718. # Dry-run for a single page
  719. synthadoc retract scan --slug ada-lovelace -w my-wiki
  720. # Only scan pages modified since the last scan cycle (incremental)
  721. synthadoc retract scan --changed-only -w my-wiki
  722. # Apply redactions across all pages (prompts for confirmation)
  723. synthadoc retract scan --apply -w my-wiki
  724. # Apply only to recently-modified pages, auto-confirm
  725. synthadoc retract scan --apply --changed-only --yes -w my-wiki
  726. # Apply without confirmation prompt
  727. synthadoc retract scan --apply --yes -w my-wiki
  728. # Show recent scan history from the audit log
  729. synthadoc retract status -w my-wiki
  730. synthadoc retract status --json -w my-wiki
  731. ```
  732. → Full walkthrough: [Quick-Start Guide — Step 27](docs/user-quick-start-guide.md#step-27--scan-and-retract-sensitive-data)
  733. ### Managing page lifecycle
  734. ```bash
  735. # Show page counts by lifecycle state
  736. synthadoc status -w my-wiki
  737. # Show running/stopped table for all registered wikis (for cross-wiki setup)
  738. synthadoc status --all
  739. # Promote a draft page to active after manual review
  740. synthadoc lifecycle activate <slug> -w my-wiki --reason "reviewed and verified"
  741. # Archive a page whose source has been superseded
  742. synthadoc lifecycle archive <slug> -w my-wiki --reason "replaced by updated source"
  743. # After archiving, any [[slug]] references in other pages are cleaned up automatically.
  744. # The output reports which pages were updated:
  745. # Cascade: [[old-page]] removed from 2 page(s): cpu-design, transistor-history
  746. # Restore an archived page back to draft for re-review
  747. synthadoc lifecycle restore <slug> -w my-wiki --reason "source re-added"
  748. # View full state history for a page (or all pages)
  749. synthadoc lifecycle log <slug> -w my-wiki
  750. synthadoc lifecycle log -w my-wiki
  751. # Browse content snapshots captured at each lifecycle transition
  752. synthadoc lifecycle history <slug> -w my-wiki
  753. synthadoc lifecycle history <slug> --index 1 -w my-wiki # inspect newest snapshot
  754. synthadoc lifecycle history <slug> --index 2 --show-content -w my-wiki # print full body
  755. # Restore the page body to a prior snapshot (rollback is itself snapshotted, so it is undoable)
  756. synthadoc lifecycle rollback <slug> --index 2 --reason "restoring original body" -w my-wiki
  757. # Purge old lifecycle events to reclaim audit.db space
  758. synthadoc audit lifecycle purge -w my-wiki --before 2026-01-01
  759. synthadoc audit lifecycle purge -w my-wiki --keep-latest 100
  760. ```
  761. Synthadoc captures the page body in three situations: a manual lifecycle transition
  762. (activate, archive, restore), a lint-driven auto-transition (stale detection, contradiction
  763. auto-resolve, draft promotion), or a file save detected by the Obsidian plugin (2 s debounce,
  764. deduplicated — unchanged saves cost nothing). Use `synthadoc lifecycle history <slug>` to
  765. browse snapshots and `synthadoc lifecycle rollback <slug> --index N --reason "..."` to
  766. restore any prior version — the rollback itself is recorded as an auditable event so it can be undone.
  767. ### Monitoring jobs
  768. ```bash
  769. # List all jobs (oldest first by default)
  770. synthadoc jobs list -w my-wiki
  771. # Sort column — created_at (default) | status | operation
  772. synthadoc jobs list --sort created_at -w my-wiki # oldest first (default)
  773. synthadoc jobs list --sort status -w my-wiki # alphabetical by status
  774. synthadoc jobs list --sort operation -w my-wiki # alphabetical by operation type
  775. # Sort direction — asc (default) | desc
  776. synthadoc jobs list --order desc -w my-wiki # newest first
  777. synthadoc jobs list --sort status --order desc -w my-wiki # status Z→A
  778. # Filter by status — pending | in_progress | completed | failed | skipped | dead | cancelled
  779. synthadoc jobs list --status pending -w my-wiki
  780. synthadoc jobs list --status failed -w my-wiki
  781. synthadoc jobs list --status dead -w my-wiki
  782. # Combine sort, order, and status freely
  783. synthadoc jobs list --status failed --sort created_at --order desc -w my-wiki
  784. # Single job detail
  785. synthadoc jobs status <job-id> -w my-wiki
  786. # Retry a dead job
  787. synthadoc jobs retry <job-id> -w my-wiki
  788. # Cancel all pending jobs at once (e.g. after a bad batch ingest)
  789. synthadoc jobs cancel -w my-wiki # prompts for confirmation
  790. synthadoc jobs cancel --yes -w my-wiki # skip confirmation
  791. # Remove old records
  792. synthadoc jobs purge --older-than 30 -w my-wiki
  793. ```
  794. ### Inspecting ingest results
  795. ```bash
  796. # Preview how a source will be analysed without writing pages
  797. synthadoc ingest report.pdf --analyse-only -w my-wiki
  798. # → {"entities": [...], "tags": [...], "summary": "..."}
  799. ```
  800. ### Audit trail
  801. ```bash
  802. # Ingest history: timestamp, source file, wiki page, tokens, cost
  803. synthadoc audit history -w my-wiki # last 50 records (shows total)
  804. synthadoc audit history -n 100 -w my-wiki # last 100 records
  805. synthadoc audit history -n 50 --offset 50 -w my-wiki # page 2 (records 51-100)
  806. synthadoc audit history --json -w my-wiki # raw JSON for scripting
  807. # Query history: question, sub-questions, tokens, cost
  808. synthadoc audit queries -w my-wiki # last 50 records (shows total)
  809. synthadoc audit queries -n 100 -w my-wiki # last 100 records
  810. synthadoc audit queries -n 50 --offset 50 -w my-wiki # page 2 (records 51-100)
  811. synthadoc audit queries --json -w my-wiki # raw JSON for scripting
  812. # Token usage: totals + daily breakdown
  813. synthadoc audit cost -w my-wiki # last 30 days
  814. synthadoc audit cost --days 7 -w my-wiki # last 7 days
  815. # Audit events: contradictions found, auto-resolutions, cost gate triggers
  816. synthadoc audit events -w my-wiki # last 100 events (shows total)
  817. synthadoc audit events -n 100 --offset 100 -w my-wiki # page 2 (events 101-200)
  818. synthadoc audit events --json -w my-wiki # raw JSON for scripting
  819. # Claim citations: source-line provenance for every annotated claim
  820. synthadoc audit citations -w my-wiki # all citations (last 50)
  821. synthadoc audit citations --page alan-turing -w my-wiki # citations for one page
  822. synthadoc audit citations --source turing.pdf -w my-wiki # citations from one source
  823. synthadoc audit citations --broken -w my-wiki # validation failures only
  824. synthadoc audit citations --json -w my-wiki # raw JSON for scripting
  825. ```
  826. **Audit citation faithfulness**
  827. ```bash
  828. synthadoc audit citations --faithfulness # all active pages
  829. synthadoc audit citations --faithfulness --page SLUG # single page
  830. synthadoc audit citations --faithfulness --force # re-run even if cache is fresh
  831. synthadoc audit citations --faithfulness --yes # skip cost prompt (e.g. in CI)
  832. synthadoc audit citations --faithfulness --json # machine-readable output
  833. # Results cached per-page; Obsidian tab shows cached results instantly on open
  834. ```
  835. ### Scheduling recurring jobs
  836. Relative paths in `--op` (e.g. `raw_sources/`) are resolved against the **wiki root directory**, not the working directory of the shell that runs the schedule. This means they work correctly even when the OS scheduler fires the task with a different working directory (e.g. `C:\Windows\System32` on Windows).
  837. ```bash
  838. # Register a nightly ingest
  839. synthadoc schedule add --op "ingest --batch raw_sources/" --cron "0 2 * * *" -w my-wiki
  840. # Weekly lint
  841. synthadoc schedule add --op "lint run" --cron "0 3 * * 0" -w my-wiki
  842. # Bulk-register all jobs declared in [[schedule.jobs]] in config.toml (alternative to schedule add)
  843. # See docs/design.md § "schedule sub-commands" for the config.toml format
  844. synthadoc schedule apply -w my-wiki
  845. # List scheduled jobs (shows schedule, next run, last run, last result)
  846. synthadoc schedule list -w my-wiki
  847. # Remove a scheduled job
  848. synthadoc schedule remove <id> -w my-wiki
  849. # Run a scheduled operation immediately and record the result in the audit trail
  850. synthadoc schedule run --op "lint run" -w my-wiki
  851. # Show recent scheduled run history
  852. synthadoc schedule history -w my-wiki
  853. synthadoc schedule history --limit 50 -w my-wiki
  854. ```
  855. **Cron expression format:** `minute hour day-of-month month day-of-week`
  856. | Field | Range | Examples |
  857. | ------------ | ----- | -------------------------- |
  858. | minute | 0–59 | `0` = on the hour |
  859. | hour | 0–23 | `2` = 2 AM, `22` = 10 PM |
  860. | day of month | 1–31 | `*` = every day |
  861. | month | 1–12 | `*` = every month |
  862. | day of week | 0–6 | `0` = Sunday, `1` = Monday |
  863. Common schedules:
  864. | Expression | Meaning |
  865. | -------------- | -------------------- |
  866. | `0 2 * * *` | Every day at 2 AM |
  867. | `0 22 * * *` | Every night at 10 PM |
  868. | `0 3 * * 0` | Every Sunday at 3 AM |
  869. | `0 */6 * * *` | Every 6 hours |
  870. | `30 8 * * 1-5` | Weekdays at 8:30 AM |
  871. ### Routing
  872. ROUTING.md maps wiki branches to page slugs so queries and ingest jobs are scoped to the relevant section of the wiki. Create it once from your existing `index.md`, then let Synthadoc maintain it automatically as new pages are added.
  873. ```bash
  874. # Bootstrap ROUTING.md from current index.md branch structure (run once)
  875. synthadoc routing init -w my-wiki
  876. # Report dangling slugs (pages listed in ROUTING.md that no longer exist)
  877. synthadoc routing validate -w my-wiki
  878. # Auto-remove dangling slugs from ROUTING.md
  879. synthadoc routing clean -w my-wiki
  880. ```
  881. ### Candidates staging
  882. When staging is enabled, ingest writes new pages to `wiki/candidates/` for human review instead of the main wiki. Useful when you want to approve AI-generated pages before they become canonical.
  883. ```bash
  884. # Show current staging policy
  885. synthadoc staging policy -w my-wiki
  886. # Route all new pages to staging (review everything)
  887. synthadoc staging policy all -w my-wiki
  888. # Only stage pages below a confidence threshold (auto-promote high-confidence)
  889. synthadoc staging policy threshold --min-confidence high -w my-wiki
  890. # Turn staging off (pages go directly to wiki/)
  891. synthadoc staging policy off -w my-wiki
  892. # List candidate pages awaiting review
  893. synthadoc candidates list -w my-wiki
  894. # Promote a specific page (moves it from candidates/ to wiki/)
  895. synthadoc candidates promote my-page-slug -w my-wiki
  896. # Promote all candidates at once
  897. synthadoc candidates promote --all -w my-wiki
  898. # Discard a specific candidate
  899. synthadoc candidates discard my-page-slug -w my-wiki
  900. # Discard all candidates
  901. synthadoc candidates discard --all -w my-wiki
  902. ```
  903. ### Context packs
  904. A context pack decomposes a goal into sub-questions, runs parallel BM25 searches, and packs the highest-scoring excerpts into a single cited Markdown document within a token budget.
  905. **Typical use cases:**
  906. - Paste into an external LLM chat (Claude.ai, ChatGPT) as grounded context before asking a question
  907. - Save next to a document you are writing as a cited research brief
  908. - Pipe into another CLI tool that reads Markdown
  909. ```bash
  910. # Print to terminal — inspect, copy, or pipe
  911. synthadoc context build "How did transistors change computing?" -w my-wiki
  912. # Copy to clipboard and paste into an LLM chat (macOS)
  913. synthadoc context build "early computing pioneers" -w my-wiki | pbcopy
  914. # Custom token budget (default 4000)
  915. synthadoc context build "Early programming languages" --tokens 8000 -w my-wiki
  916. # Save next to a document you are writing
  917. synthadoc context build "Rise of microprocessors" --output ~/drafts/computing-brief.md -w my-wiki
  918. ```
  919. ### Exporting
  920. Export your wiki in machine-readable formats for RAG pipelines, LLM context windows, graph analysis tools, and OKF-compliant agents. All formats are assembled server-side with zero additional LLM calls. Requires `synthadoc serve` to be running.
  921. ```bash
  922. # Active pages as LLM context (llms.txt spec)
  923. synthadoc export --format llms.txt --status active -w my-wiki
  924. # Full content dump with provenance footnotes preserved
  925. synthadoc export --format llms-full.txt --output exports/wiki-full.txt -w my-wiki
  926. # Export wikilink graph as GraphML — open in yEd, Gephi, or Cytoscape
  927. synthadoc export --format graphml --output exports/wiki.graphml -w my-wiki
  928. # Agent-ready JSON with provenance, lifecycle history, and compilation cost
  929. synthadoc export --format json --output exports/wiki.json -w my-wiki
  930. # OKF v0.2 bundle — consumable by any OKF-aware agent without code changes
  931. # Write outside the wiki folder to avoid Obsidian picking up bundle files as source
  932. synthadoc export --format okf --output ~/exports/my-wiki-okf/ -w my-wiki
  933. ```
  934. **Flags:** `--format/-f` (required: `llms.txt`, `llms-full.txt`, `graphml`, `json`, `okf`), `--output/-o` (file path, or directory for `okf`; omit for stdout), `--status/-s` (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`).
  935. > **[OKF v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) export requires `--output`** — the bundle is a directory tree, not a single file.
  936. > **OKF default pages:** `--format okf` with `--status all` (the default) includes only `active` and `contradicted` pages. Draft and stale are excluded — they carry unverified content. Contradicted pages appear with `status: draft` and `synthadoc_lifecycle: contradicted` in their frontmatter and a `> **Contradiction:** …` blockquote in the body.
  937. > **Tip:** Keep the OKF bundle **outside** your wiki folder. The output path can be any absolute or relative path — `--output ~/exports/my-wiki-okf/` or `--output ../okf-bundles/my-wiki/` both work. Placing it inside the wiki folder risks Obsidian or the ingestor picking up the bundle files as source documents.
  938. In Obsidian: command palette → **Synthadoc: Export Wiki** — choose format and status filter, then click **Export**. For all formats except `okf`, the file is saved to the vault's `exports/` folder and opened automatically. For `okf`, the output path defaults to `~/exports/{vault-name}-okf-{date}/` outside the vault and is written via the filesystem. For GraphML, a **View Graph** button renders an inline preview.
  939. ### Backup & Restore
  940. ```bash
  941. # Backup to the current directory
  942. synthadoc backup -w my-wiki
  943. # Backup to a specific directory, excluding raw source files to reduce size
  944. synthadoc backup -w my-wiki --output ~/backups --no-sources
  945. # Restore to the same folder as the zip (default)
  946. synthadoc restore synthadoc-backup-my-wiki-20260624-103000.zip
  947. # Restore under a different name to a specific location on a specific port
  948. synthadoc restore backup.zip --name my-wiki-staging --target ~/wikis --port 7071
  949. ```
  950. → Flags, use-case table, what is and isn't backed up, and post-restore checklist: [Quick-Start Guide — Appendix J](docs/user-quick-start-guide.md#appendix-j--backup--restore)
  951. ### Removing a wiki
  952. Stop the server for that wiki before uninstalling — the serve process must not be running
  953. when the directory is deleted.
  954. ```bash
  955. # Stop the wiki server (background or foreground)
  956. synthadoc stop -w my-wiki
  957. # Then uninstall — two-step confirmation required, no --yes escape
  958. synthadoc uninstall my-wiki
  959. ```
  960. For Obsidian plugin commands see [Appendix A — Obsidian Plugin Command Reference](docs/user-quick-start-guide.md#appendix-a--obsidian-plugin-commands) in the Quick-Start Guide.
  961. ---
  962. ## Administrative Reference
  963. ### Health and status
  964. ```bash
  965. # Wiki statistics: pages, queue depth, cache hit rate
  966. synthadoc status -w my-wiki
  967. # Liveness probe (useful in scripts and monitoring)
  968. # Port is per-wiki — check [server] port in <wiki-root>/.synthadoc/config.toml
  969. # Default is 7070; each additional wiki uses its own port (7071, 7072, …)
  970. curl http://localhost:7070/health
  971. ```
  972. Expected `status` output:
  973. ```
  974. Wiki: /home/user/wikis/my-wiki
  975. Pages: 34
  976. active 34
  977. draft 0
  978. stale 0
  979. contradicted 0
  980. archived 0
  981. Jobs pending: 0
  982. Jobs total: 12
  983. ```
  984. ### Logs
  985. Synthadoc writes three log artefacts per wiki:
  986. | File | Location | Format | Use |
  987. | --------------- | --------------------------------- | ----------------------- | ------------------------------------------------------------------- |
  988. | `log.md` | `<wiki-root>/log.md` | Human-readable Markdown | Read inside Obsidian; shows every ingest, contradiction, lint event |
  989. | `synthadoc.log` | `<wiki-root>/.synthadoc/logs/` | JSON lines (rotating) | Structured debug/ops log; grep or pipe to jq |
  990. | `audit.db` | `<wiki-root>/.synthadoc/audit.db` | SQLite (append-only) | Source hashes, cost records, job history |
  991. **Tailing the JSON log:**
  992. ```bash
  993. # Tail and pretty-print with jq
  994. tail -f .synthadoc/logs/synthadoc.log | jq .
  995. # Filter to errors only
  996. tail -f .synthadoc/logs/synthadoc.log | jq 'select(.level == "ERROR")'
  997. # Filter to a specific job
  998. # job_id is present only on records logged in job context (ingest/lint workers)
  999. tail -f .synthadoc/logs/synthadoc.log | jq 'select(.job_id == "abc123")'
  1000. ```
  1001. **Log rotation:** When `synthadoc.log` reaches `max_file_mb`, it is renamed to `synthadoc.log.1`; the previous `.1` becomes `.2`; files beyond `backup_count` are deleted. Total disk ≈ `max_file_mb × (backup_count + 1)`.
  1002. **Changing log level at runtime:** Edit `[logs] level` in `.synthadoc/config.toml` and restart `synthadoc serve`. Or pass `--verbose` to get `DEBUG` for one session without editing config.
  1003. ### Audit trail
  1004. ```bash
  1005. synthadoc audit history -w my-wiki # table: timestamp, source file, wiki page, tokens, cost
  1006. synthadoc audit history -n 100 -w my-wiki # last 100 records (default 50)
  1007. synthadoc audit history --json -w my-wiki # raw JSON for scripting
  1008. synthadoc audit cost -w my-wiki # total tokens + daily breakdown, last 30 days
  1009. synthadoc audit cost --days 7 -w my-wiki # weekly view
  1010. synthadoc audit cost --json -w my-wiki # {total_tokens, total_cost_usd, daily: [...]}
  1011. synthadoc audit events -w my-wiki # table: timestamp, job_id, event type, metadata
  1012. synthadoc audit events --json -w my-wiki # raw JSON
  1013. synthadoc audit citations -w my-wiki # all claim citations (last 50)
  1014. synthadoc audit citations --page alan-turing -w my-wiki # citations for one page
  1015. synthadoc audit citations --source turing.pdf -w my-wiki # citations from one source file
  1016. synthadoc audit citations --broken -w my-wiki # validation failures only
  1017. synthadoc audit citations --json -w my-wiki # raw JSON for scripting
  1018. ```
  1019. > **Note:** Per-model cost tracking is live from v0.2.0 — pricing tables cover all 7 API providers. Token counts and USD cost are recorded for every ingest and query operation in `audit.db`.
  1020. > **Lifecycle transitions** are also recorded in `audit.db` — every state change (slug, from/to state, triggered_by, reason, timestamp) is permanently stored as an immutable audit event. Query the log with `synthadoc lifecycle log` or `GET /lifecycle/events`.
  1021. ### Cache management
  1022. ```bash
  1023. # Remove all cached LLM responses
  1024. # Output: "Cache cleared: N entries removed."
  1025. synthadoc cache clear -w my-wiki
  1026. ```
  1027. Cache invalidation happens automatically when:
  1028. - A source file's SHA-256 hash changes (content changed)
  1029. - The internal cache version is bumped (after prompt template edits)
  1030. - `--force` is passed to ingest
  1031. ### OpenTelemetry integration
  1032. By default, traces and metrics are written to `<wiki-root>/.synthadoc/logs/traces.jsonl`. To send to any OTLP backend (Jaeger, Grafana Tempo, Honeycomb, Datadog):
  1033. ```toml
  1034. # ~/.synthadoc/config.toml
  1035. [observability]
  1036. exporter = "otlp"
  1037. otlp_endpoint = "http://localhost:4317"
  1038. ```
  1039. ### Debugging
  1040. ```bash
  1041. # Start server with DEBUG console logging
  1042. synthadoc serve -w my-wiki --verbose
  1043. # Check for configuration problems
  1044. synthadoc status -w my-wiki # prints pre-flight warnings
  1045. # View recent job failures
  1046. synthadoc jobs list --status failed -w my-wiki
  1047. synthadoc jobs status <job-id> -w my-wiki # shows error message + traceback
  1048. # Force a re-ingest to rule out cache issues
  1049. synthadoc ingest --force problem.pdf -w my-wiki
  1050. ```
  1051. ---
  1052. ## Understanding Logs and the Audit Trail
  1053. Synthadoc writes three log artefacts per wiki: `log.md` (human-readable Markdown, open in Obsidian), `synthadoc.log` (JSON lines, rotate-by-size, grep with `jq`), and `audit.db` (append-only SQLite — source hashes, cost records, job history).
  1054. For the full field reference, log levels, rotation config, OTel integration, and audit query examples see [docs/design.md — Logs and Audit Trail](docs/design.md#logs-and-audit-trail).
  1055. ---
  1056. ## Customization
  1057. Custom skills, LLM providers, hooks, cache control, and per-wiki AGENTS.md are documented in [docs/design.md — Customization](docs/design.md#customization).
  1058. ---
  1059. ## Links
  1060. <!-- pypi-strip-start -->
  1061. - Blogs & Media: [docs/media/](docs/media/README.md) — YouTube, Coderlegion, DEV.to, Medium
  1062. <!-- pypi-strip-end -->
  1063. - Design document: [docs/design.md](docs/design.md)
  1064. - Customization: [docs/design.md — Customization](docs/design.md#customization)
  1065. - Quick-Start Guide: [docs/user-quick-start-guide.md](docs/user-quick-start-guide.md)
  1066. <!-- pypi-strip-start -->
  1067. - Contributing: [CONTRIBUTING.md](CONTRIBUTING.md)
  1068. <!-- pypi-strip-end -->
  1069. - GitHub: [axoviq-ai/synthadoc](https://github.com/axoviq-ai/synthadoc)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.