Skip to content
Back to search

llmwiki

MarketplaceWorks with Claude CodeGitHub

Turn your Claude Code session history into a beautiful, searchable, Karpathy-style knowledge base.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/pratiyush--llm-wiki/badge.svg)](/listings/pratiyush--llm-wiki)

HTML

<a href="/listings/pratiyush--llm-wiki"><img src="/api/listings/pratiyush--llm-wiki/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

macOS / Linux

git clone https://github.com/Pratiyush/llm-wiki.git
cd llm-wiki
./setup.sh

Windows

git clone https://github.com/Pratiyush/llm-wiki.git
cd llm-wiki
setup.bat

With pip (v0.3+)

pip install -e .                # basic — everything you need
pip install -e '.[graph]'       # + graphifyy AI-powered graph engine
pip install -e '.[dev]'         # + pytest + ruff
pip install -e '.[e2e]'         # + Playwright + pytest-bdd + pytest-html (E2E)
pip install -e '.[all]'         # graph + everything

Syntax highlighting is now powered by [highlight.js](https://highlightjs.org/), loaded from a CDN at view time — no optional deps required.

What setup does

1. Creates `raw/`, `wiki/`, `site/` data directories 2. Installs the `llmwiki` Python package in-place 3. Detects your coding agents and enables matching adapters 4. Optionally offers to install the `SessionStart` hook into `~/.claude/settings.json` for auto-sync 5. Runs a first sync so you see output immediately

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 17 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "$schema": "https://raw.githubusercontent.com/anthropics/claude-code/main/schemas/marketplace.json",
  3. "name": "llmwiki",
  4. "displayName": "LLM Wiki",
  5. "description": "Turn your Claude Code session history into a beautiful, searchable, Karpathy-style knowledge base.",
  6. "version": "0.1.0",
  7. "author": "Pratiyush",
  8. "homepage": "https://github.com/Pratiyush/llmwiki",
  9. "license": "MIT",
  10. "plugins": [
  11. {
  12. "name": "llmwiki",
  13. "path": ".",
  14. "description": "Core llmwiki plugin — commands, skills, hooks, and MCP server for building a wiki from your agent sessions."
  15. }
  16. ]
  17. }
README.md · README · 641 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # llmwiki
  2. > **LLM-powered knowledge base from your Claude Code, Codex CLI, Cursor, Gemini CLI, and Obsidian sessions.**
  3. > Built on [Andrej Karpathy's LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
  4. ## 👉 Live demo: **[pratiyush.github.io/llm-wiki](https://pratiyush.github.io/llm-wiki/)**
  5. Rebuilt on every `master` push from the synthetic sessions in [`examples/demo-sessions/`](examples/demo-sessions). No personal data. Shows every feature of the real tool (activity heatmap, tool charts, token usage, model info cards, vs-comparisons, project topics) running against safe reference data.
  6. [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
  7. [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/)
  8. [![Version](https://img.shields.io/badge/version-v1.3.82-10B981.svg)](CHANGELOG.md)
  9. [![Tests](https://img.shields.io/badge/tests-2651%20passing-10B981.svg)](tests/)
  10. [![CI](https://github.com/Pratiyush/llm-wiki/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/Pratiyush/llm-wiki/actions/workflows/ci.yml)
  11. [![Link check](https://github.com/Pratiyush/llm-wiki/actions/workflows/link-check.yml/badge.svg?branch=master)](https://github.com/Pratiyush/llm-wiki/actions/workflows/link-check.yml)
  12. [![Wiki checks](https://github.com/Pratiyush/llm-wiki/actions/workflows/wiki-checks.yml/badge.svg?branch=master)](https://github.com/Pratiyush/llm-wiki/actions/workflows/wiki-checks.yml)
  13. [![Docker](https://github.com/Pratiyush/llm-wiki/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/Pratiyush/llm-wiki/pkgs/container/llm-wiki)
  14. [![Works with Claude Code](https://img.shields.io/badge/Claude%20Code-✓-7C3AED.svg)](https://claude.com/claude-code)
  15. [![Works with Codex CLI](https://img.shields.io/badge/Codex%20CLI-✓-7C3AED.svg)](https://github.com/openai/codex)
  16. [![Works with Copilot](https://img.shields.io/badge/GitHub%20Copilot-✓-7C3AED.svg)](https://github.com/features/copilot)
  17. [![Works with Cursor](https://img.shields.io/badge/Cursor-✓-7C3AED.svg)](https://cursor.com)
  18. [![Works with Gemini CLI](https://img.shields.io/badge/Gemini%20CLI-✓-7C3AED.svg)](https://ai.google.dev/gemini-api)
  19. [![Works with Obsidian](https://img.shields.io/badge/Obsidian-✓-7C3AED.svg)](https://obsidian.md)
  20. ---
  21. Every Claude Code, Codex CLI, Copilot, Cursor, and Gemini CLI session writes a full transcript to disk. You already have hundreds of them and never look at them again.
  22. **llmwiki** turns that dormant history into a beautiful, searchable, interlinked knowledge base — locally, in two commands. Plus, it produces AI-consumable exports (`llms.txt`, `llms-full.txt`, JSON-LD graph, per-page `.txt` + `.json` siblings) so other AI agents can query your wiki directly.
  23. ```bash
  24. ./setup.sh # one-time install
  25. ./build.sh && ./serve.sh # build + serve at http://127.0.0.1:8765
  26. ```
  27. **Want LLM-enriched wiki pages from the CLI?** After `./setup.sh`, `llmwiki sync` only fills `raw/`. To populate `wiki/sources/` (semantic summaries), set `synthesis.backend` in `examples/sessions_config.json` (`ollama` for local Ollama, or keep `dummy` for stubs), then run `llmwiki synthesize` or chain it with `llmwiki all --with-synth`. See [docs/modes/api/](docs/modes/api/) and `llmwiki sync --status` for the current backend hint.
  28. ```bash
  29. llmwiki sync
  30. llmwiki all --with-synth --graph-engine builtin # synthesize → build → graph → export → lint
  31. ```
  32. ![llmwiki — 70-second demo](docs/demo.gif)
  33. **Contributing in one line:** read [`CONTRIBUTING.md`](CONTRIBUTING.md), keep PRs focused (one concern each), use `feat:` / `fix:` / `docs:` / `chore:` / `test:` commit prefixes, never commit real session data (`raw/` is gitignored), no new runtime deps. CI must be green to merge.
  34. ## Screenshots
  35. All screenshots below are from the **public demo site** which is built on every `master` push from the dummy example sessions. Your own wiki will look identical — just with your real work.
  36. ### Home — projects overview with activity heatmap
  37. ![llmwiki home page — LLM Wiki header, activity heatmap, and a grid of three demo projects (demo-blog-engine, demo-ml-pipeline, demo-todo-api)](docs/images/home.png)
  38. ### All sessions — filterable table across every project
  39. ![llmwiki sessions index — activity timeline above a table of eight demo sessions with project, model, date, message count, and tool-call columns](docs/images/sessions.png)
  40. ### Session detail — full conversation + tool calls
  41. ![llmwiki session detail — Rust blog engine scaffolding session showing summary, breadcrumbs, a TOML Cargo.toml block and a Rust main.rs block, both highlighted by highlight.js](docs/images/session-rust.png)
  42. ### Changelog — renders `CHANGELOG.md` as a first-class page
  43. ![llmwiki changelog page — keep-a-changelog format with colored headings for Added / Fixed / Changed and auto-linked PR references](docs/images/changelog.png)
  44. ### Projects index — freshness badges + per-project stats
  45. ![llmwiki projects index — three demo project cards with green/yellow/red freshness badges showing how recently each project was touched](docs/images/projects.png)
  46. ## What you get
  47. ### Human-readable
  48. - **All your sessions**, converted from `.jsonl` to clean, redacted markdown
  49. - **A Karpathy-style wiki** — `sources/`, `entities/`, `concepts/`, `syntheses/`, `comparisons/`, `questions/` linked with `[[wikilinks]]`
  50. - **A beautiful static site** you can browse locally or deploy to GitHub Pages
  51. - Global search (Cmd+K command palette with fuzzy match over pre-built index)
  52. - [highlight.js](https://highlightjs.org/) client-side syntax highlighting (light + dark themes)
  53. - Dark mode (system-aware + manual toggle with `data-theme`)
  54. - Keyboard shortcuts: `/` search · `g h/p/s` nav · `j/k` rows · `?` help
  55. - Collapsible tool-result sections (auto-expand > 500 chars)
  56. - Copy-as-markdown + copy-code buttons
  57. - Breadcrumbs + reading progress bar
  58. - Filter bar on sessions table (project/model/date/text)
  59. - Reading time estimates (`X min read`)
  60. - Related pages panel at the bottom of every session
  61. - Activity heatmap on the home page
  62. - Model info cards with structured schema (provider, pricing, benchmarks)
  63. - Auto-generated vs-comparison pages between AI models
  64. - Append-only changelog timeline with pricing sparkline
  65. - Project topic chips (GitHub-style tags on project cards)
  66. - Agent labels (colored badges: Claude/Codex/Copilot/Cursor/Gemini)
  67. - Recently-updated card on the home page
  68. - Dataview-style structured queries in the command palette
  69. - Hover-to-preview wikilinks
  70. - Deep-link icons next to every heading
  71. - Mobile-responsive + print-friendly
  72. ### AI-consumable (v0.4)
  73. Every HTML page has sibling machine-readable files at the same URL:
  74. - `<page>.html` — human HTML with schema.org microdata
  75. - `<page>.txt` — plain text version (no HTML tags)
  76. - `<page>.json` — structured metadata + body
  77. Site-level AI-agent entry points:
  78. | File | What |
  79. |---|---|
  80. | [`/llms.txt`](https://llmstxt.org) | Short index per [llmstxt.org spec](https://llmstxt.org) |
  81. | `/llms-full.txt` | Flattened plain-text dump (~5 MB cap) — paste into any LLM's context |
  82. | `/graph.jsonld` | Schema.org JSON-LD entity/concept/source graph |
  83. | `/sitemap.xml` | Standard sitemap with `lastmod` |
  84. | `/rss.xml` | RSS 2.0 feed of newest sessions |
  85. | `/robots.txt` | AI-friendly robots with llms.txt reference |
  86. | `/ai-readme.md` | AI-specific navigation instructions |
  87. | `/manifest.json` | Build manifest with SHA-256 hashes + perf budget |
  88. Every page also includes an `<!-- llmwiki:metadata -->` HTML comment that AI agents can parse without fetching the separate `.json` sibling.
  89. #### Recipe — query `graph.jsonld` from your terminal
  90. The JSON-LD graph isn't just for crawlers — you can ask quick questions about your wiki without leaving the shell. Example: print every session as a tree, grouped by project:
  91. ```bash
  92. python3 examples/scripts/tree_from_graph.py
  93. ```
  94. Output:
  95. ```
  96. 📚 8 sessions across 3 projects
  97. (site/graph.jsonld v1.3.0)
  98. llmwiki/
  99. ├── demo-blog-engine/ (4 sessions)
  100. │ ├── 2026-03-12 scaffolding-the-rust-blog-engine
  101. │ ├── 2026-03-18 adding-syntax-highlighting
  102. │ ├── 2026-03-25 rss-feed-and-sitemap
  103. │ └── 2026-04-01 dark-mode-toggle
  104. ├── demo-ml-pipeline/ (2 sessions)
  105. │ ├── 2026-01-20 training-data-pipeline
  106. │ └── 2026-02-02 model-training-loop
  107. └── demo-todo-api/ (2 sessions)
  108. ├── 2026-02-08 fastapi-project-bootstrap
  109. └── 2026-02-15 adding-oauth-login
  110. ```
  111. The full script is **stdlib-only** at [`examples/scripts/tree_from_graph.py`](examples/scripts/tree_from_graph.py). Same recipe pattern works for any aggregation question — count sessions per model, find the largest project by token usage, list every entity that appears in 3+ sessions, etc. The graph is yours to slice.
  112. ### Quality & governance (v1.0)
  113. - **4-factor confidence scoring** — source count, source quality, recency, cross-references; with Ebbinghaus-inspired decay per content-type
  114. - **5-state lifecycle machine** — draft → reviewed → verified → stale → archived with 90-day auto-stale
  115. - **16 lint rules** — 8 structural (frontmatter, link integrity, orphans, freshness, duplicates, index sync…) + 3 LLM-powered (contradictions, claim verification, summary accuracy) + stale_candidates (#51) + tags_topics_convention (#302) + stale_reference_detection (#303) + frontmatter_count_consistency (#378) + tools_consistency (#378)
  116. - **Auto Dream** — MEMORY.md consolidation after 24h + 5 sessions: resolve relative dates, prune outdated, 200-line cap
  117. - **9 navigation files** — CLAUDE.md, AGENTS.md, MEMORY.md, SOUL.md, CRITICAL_FACTS.md, hints.md, hot.md + per-project hot caches
  118. ### Obsidian-native experience (v1.0)
  119. - **`link-obsidian` CLI** — symlinks the whole project into an Obsidian vault; graph view + backlinks + full-text search just work
  120. - **Dataview dashboard** — 10 ready-to-use queries (recently updated, by confidence, by lifecycle, by project, by entity type, open questions, stale pages)
  121. - **Templater templates** — 4 templates for source/entity/concept/synthesis pages, seeded with confidence + lifecycle + today's date
  122. - **Category pages** — tag-based index pages in both Dataview (Obsidian) and static markdown (HTML) modes
  123. - **Integration guide** — [`docs/obsidian-integration.md`](docs/obsidian-integration.md) covers 6 recommended plugins with per-plugin configs
  124. ### Automation
  125. - **SessionStart hook** — auto-syncs new sessions in the background on every Claude Code launch
  126. - **Auto-build on sync** — `/wiki-sync` triggers `/wiki-build` (configurable; default on)
  127. - **One-shot pipeline** — `llmwiki all` runs build → graph → export → lint in a single command (`--strict` for CI)
  128. - **MCP server** — 12 production tools (query, search, list, read, lint, sync, export, + confidence, lifecycle, dashboard, entity search, category browse) queryable from any MCP client (Claude Desktop, Cline, Cursor, ChatGPT desktop)
  129. - **Pending ingest queue** — SessionStart hook converts + queues; `/wiki-sync` processes queue
  130. - **No servers, no database, no npm** — Python stdlib + `markdown`. Syntax highlighting loads from a highlight.js CDN at view time.
  131. ## Tutorial — every command in 90 seconds
  132. A guided tour. Run these in order and you'll have a fully working wiki at `http://127.0.0.1:8765/` by the end. Each command is idempotent and prints what it did.
  133. A scripted recording of the same flow ships at [`docs/videos/cli-tutorial.gif`](docs/videos/cli-tutorial.gif) (31 seconds against an 8-session sandbox). The reproducible source is [`docs/videos/cli-tutorial.tape`](docs/videos/cli-tutorial.tape) — re-render anytime with `vhs docs/videos/cli-tutorial.tape`.
  134. ```bash
  135. # 1. One-time scaffold (≈1 sec). Creates raw/, wiki/, site/, seed nav files.
  136. llmwiki init
  137. # 2. Pull in your sessions (≈1 sec / 100 sessions). Walks every adapter
  138. # that's "available" on this machine (Claude Code, Codex CLI, Cursor,
  139. # Gemini, Obsidian, Copilot Chat / CLI), converts new .jsonl files to
  140. # raw/sessions/*.md, then runs build + lint by default.
  141. llmwiki sync
  142. # 3. Compile the static HTML site (≈3 sec on a 500-session corpus).
  143. # Already runs as part of `sync`; call directly when you're iterating
  144. # on a wiki/ page and don't need a fresh sync.
  145. llmwiki build
  146. # 4. Browse it locally. Cmd+K opens the search palette; / focuses the
  147. # filter bar on /sessions/. Press Ctrl+C to stop.
  148. llmwiki serve
  149. # 5. (Optional) Generate the knowledge graph + AI-consumable exports.
  150. # `all` runs build → graph → export → lint in one shot.
  151. llmwiki graph
  152. llmwiki export all
  153. llmwiki all # one-shot equivalent of build + graph + export + lint
  154. ```
  155. That's the entire happy path. Two more commands you'll reach for occasionally:
  156. ```bash
  157. # Inspect what's installed + configured. Prints a per-adapter table:
  158. # (available: yes/no, configured: yes/no, session-store path).
  159. llmwiki adapters
  160. # Lint the wiki. 16 rules — broken wikilinks, orphaned pages, stale
  161. # summaries, duplicate detection, freshness, missing entities, etc.
  162. # Runs as part of `sync` by default; call directly for a one-shot check.
  163. llmwiki lint
  164. ```
  165. Three optional flags you'll discover later:
  166. - `--adapter <name>` — limit `sync` to one adapter (e.g. `--adapter claude_code`)
  167. - `--vault PATH` — write into an Obsidian / Logseq vault overlay instead of `wiki/` (#54)
  168. - `--synthesize` — call out to a local Claude / Ollama backend during `build` for an LLM-generated overview page
  169. Each subcommand has its own `--help` with the rest. The CLI reference table below is the full list.
  170. ## How it works
  171. ```
  172. ┌─────────────────────────────────────┐
  173. │ ~/.claude/projects/*/*.jsonl │ ← Claude Code sessions
  174. │ ~/.codex/sessions/**/*.jsonl │ ← Codex CLI sessions
  175. │ ~/Library/.../Cursor/workspaceS… │ ← Cursor
  176. │ ~/Documents/Obsidian Vault/ │ ← Obsidian
  177. │ ~/.gemini/ │ ← Gemini CLI
  178. └──────────────┬──────────────────────┘
  179. │
  180. ▼ python3 -m llmwiki sync
  181. ┌─────────────────────────────────────┐
  182. │ raw/sessions/<project>/ │ ← immutable markdown (Karpathy layer 1)
  183. │ 2026-04-08-<slug>.md │
  184. └──────────────┬──────────────────────┘
  185. │
  186. ▼ /wiki-ingest (your coding agent)
  187. ┌─────────────────────────────────────┐
  188. │ wiki/sources/<slug>.md │ ← LLM-generated wiki (Karpathy layer 2)
  189. │ wiki/entities/<Name>.md │
  190. │ wiki/concepts/<Name>.md │
  191. │ wiki/syntheses/<Name>.md │
  192. │ wiki/comparisons/<Name>.md │
  193. │ wiki/questions/<Name>.md │
  194. │ wiki/index.md, overview.md, log.md │
  195. └──────────────┬──────────────────────┘
  196. │
  197. ▼ python3 -m llmwiki build
  198. ┌─────────────────────────────────────┐
  199. │ site/ │ ← static HTML + AI exports
  200. │ ├── index.html, style.css, ... │
  201. │ ├── sessions/<project>/<slug>.html │
  202. │ ├── sessions/<project>/<slug>.txt │ (AI sibling)
  203. │ ├── sessions/<project>/<slug>.json │ (AI sibling)
  204. │ ├── llms.txt, llms-full.txt │
  205. │ ├── graph.jsonld │
  206. │ ├── sitemap.xml, rss.xml │
  207. │ ├── robots.txt, ai-readme.md │
  208. │ ├── manifest.json │
  209. │ └── search-index.json │
  210. └─────────────────────────────────────┘
  211. ```
  212. See [docs/architecture.md](docs/architecture.md) for the full 3-layer Karpathy + 8-layer build breakdown.
  213. ## Documentation
  214. Full production documentation lives under [`docs/`](docs/). The editorial
  215. hub is **[`docs/index.md`](docs/index.md)** — tutorials, per-agent guides,
  216. reference, and deployment, all in one place.
  217. **Start here:**
  218. | Goal | Read |
  219. |---|---|
  220. | Install and build your first site in 10 minutes | [Tutorial 01 → 02](docs/tutorials/01-installation.md) |
  221. | Use llmwiki with Claude Code | [Tutorial 03](docs/tutorials/03-use-with-claude-code.md) |
  222. | Use llmwiki with Codex CLI | [Tutorial 04](docs/tutorials/04-use-with-codex-cli.md) |
  223. | Query / lint / review your wiki daily | [Tutorial 05](docs/tutorials/05-querying-your-wiki.md) |
  224. | Point llmwiki at an existing Obsidian / Logseq vault | [Tutorial 06](docs/tutorials/06-bring-your-obsidian-vault.md) |
  225. | See four real end-to-end workflows | [Tutorial 07](docs/tutorials/07-example-workflows.md) |
  226. Contributing to docs? See the **[style guide](docs/style-guide.md)**.
  227. ## Install
  228. ### macOS / Linux
  229. ```bash
  230. git clone https://github.com/Pratiyush/llm-wiki.git
  231. cd llm-wiki
  232. ./setup.sh
  233. ```
  234. ### Windows
  235. ```cmd
  236. git clone https://github.com/Pratiyush/llm-wiki.git
  237. cd llm-wiki
  238. setup.bat
  239. ```
  240. ### With pip (v0.3+)
  241. ```bash
  242. pip install -e . # basic — everything you need
  243. pip install -e '.[graph]' # + graphifyy AI-powered graph engine
  244. pip install -e '.[dev]' # + pytest + ruff
  245. pip install -e '.[e2e]' # + Playwright + pytest-bdd + pytest-html (E2E)
  246. pip install -e '.[all]' # graph + everything
  247. ```
  248. Syntax highlighting is now powered by [highlight.js](https://highlightjs.org/), loaded from a CDN at view time — no optional deps required.
  249. ### What setup does
  250. 1. Creates `raw/`, `wiki/`, `site/` data directories
  251. 2. Installs the `llmwiki` Python package in-place
  252. 3. Detects your coding agents and enables matching adapters
  253. 4. Optionally offers to install the `SessionStart` hook into `~/.claude/settings.json` for auto-sync
  254. 5. Runs a first sync so you see output immediately
  255. ## For maintainers
  256. Running the project? The governance scaffold lives under [`docs/maintainers/`](docs/maintainers) and is loaded by a dedicated skill:
  257. | File | What it's for |
  258. |---|---|
  259. | [`CONTRIBUTING.md`](CONTRIBUTING.md) | Short rules for contributors — read this first |
  260. | [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md) | Contributor Covenant 2.1 |
  261. | [`SECURITY.md`](SECURITY.md) | Disclosure process for redaction bugs, XSS, data leaks |
  262. | [`docs/maintainers/ARCHITECTURE.md`](docs/maintainers/ARCHITECTURE.md) | One-page system diagram + layer boundaries + what NOT to add |
  263. | [`docs/maintainers/REVIEW_CHECKLIST.md`](docs/maintainers/REVIEW_CHECKLIST.md) | Canonical code-review criteria |
  264. | [`docs/maintainers/RELEASE_PROCESS.md`](docs/maintainers/RELEASE_PROCESS.md) | Version bump → CHANGELOG → tag → build → publish |
  265. | [`docs/maintainers/TRIAGE.md`](docs/maintainers/TRIAGE.md) | Label taxonomy + stale-issue policy |
  266. | [`docs/maintainers/ROADMAP.md`](docs/maintainers/ROADMAP.md) | Near-term plan + release themes |
  267. | [`docs/maintainers/DECLINED.md`](docs/maintainers/DECLINED.md) | Graveyard of declined ideas with reasons |
  268. Four Claude Code slash commands automate the common ops:
  269. - `/review-pr <N>` — apply the REVIEW_CHECKLIST to a PR and post findings
  270. - `/triage-issue <N>` — label + milestone + priority a new issue
  271. - `/release <version>` — walk the release process step by step
  272. - `/maintainer` — meta-skill that loads every governance doc as context
  273. ## Running E2E tests
  274. The unit suite (`pytest tests/` — 2,651 tests) runs in seconds and
  275. covers every module. The **end-to-end suite** under `tests/e2e/` is
  276. separate: it builds a minimal demo site, serves it on a random port,
  277. drives a real browser via [Playwright](https://playwright.dev/python),
  278. and runs scenarios written in [Gherkin](https://cucumber.io/docs/gherkin/)
  279. via [pytest-bdd](https://pytest-bdd.readthedocs.io/).
  280. Why both? Unit tests lock the contract at the module boundary;
  281. E2E locks the contract at the **user's browser**. A diff that passes
  282. unit tests but breaks the Cmd+K palette will fail E2E.
  283. Install the extras (one-time, several hundred MB for the Chromium binary):
  284. ```bash
  285. pip install -e '.[e2e]'
  286. python -m playwright install chromium
  287. ```
  288. Run the suite:
  289. ```bash
  290. pytest tests/e2e/ --browser=chromium
  291. ```
  292. Run a single feature:
  293. ```bash
  294. pytest tests/e2e/test_command_palette.py --browser=chromium -v
  295. ```
  296. The E2E suite is **excluded from the default `pytest tests/` run**
  297. (see the `--ignore=tests/e2e` addopt in `pyproject.toml`) so you
  298. can iterate on the unit suite without waiting for browser installs.
  299. CI runs the E2E job as a separate workflow (`.github/workflows/e2e.yml`)
  300. that only fires on PRs touching `build.py`, the viz modules, or
  301. `tests/e2e/**`.
  302. Feature files live under `tests/e2e/features/` — one per UI area
  303. (homepage, session page, command palette, keyboard nav, mobile nav,
  304. theme toggle, copy-as-markdown, **responsive breakpoints**, **edge
  305. cases**, **accessibility**, **visual regression**). Step definitions
  306. are all in `tests/e2e/steps/ui_steps.py`. Adding a new scenario is
  307. usually a 2-line change to a `.feature` file plus maybe one new step.
  308. Run locally with an HTML report:
  309. ```bash
  310. pytest tests/e2e/ --browser=chromium \
  311. --html=e2e-report/index.html --self-contained-html
  312. open e2e-report/index.html # macOS — opens the browseable report
  313. ```
  314. **Where to see test reports:**
  315. | What | Where |
  316. |---|---|
  317. | Unit test results | GitHub Actions → `ci.yml` → latest run → `lint-and-test` job logs |
  318. | E2E HTML report | GitHub Actions → `e2e.yml` → latest run → Artifacts → `e2e-html-report` (14-day retention) |
  319. | Visual regression screenshots | Same run → Artifacts → `e2e-screenshots` |
  320. | Playwright traces (failed runs only) | Same run → Artifacts → `playwright-traces` (open with `playwright show-trace <zip>`) |
  321. | Demo site deploy status | GitHub Actions → `pages.yml` → latest run |
  322. Locally, the HTML report is one file (`e2e-report/index.html`) that
  323. you can open in any browser — pass/fail per scenario, duration,
  324. stdout/stderr, screenshot on failure.
  325. ## Scheduled sync
  326. For a daily / weekly cron-style sync, schedule `llmwiki sync` directly via your OS's native job runner (`launchd` on macOS, `systemd` on Linux, Task Scheduler on Windows). Paths and adapter selection come from `examples/sessions_config.json`.
  327. ## CLI reference
  328. ```bash
  329. llmwiki init # scaffold raw/ wiki/ site/ + seed nav files
  330. llmwiki sync # convert .jsonl → markdown (auto-build + auto-lint if configured)
  331. llmwiki build # compile static HTML + AI exports
  332. llmwiki serve # local HTTP server on 127.0.0.1:8765
  333. llmwiki adapters # list available adapters + configured state (v1.0)
  334. llmwiki graph # build knowledge graph (v0.2)
  335. llmwiki lint # 16-rule wiki lint (v1.2)
  336. llmwiki export <format> # AI-consumable exports (v0.4)
  337. llmwiki synthesize # auto-ingest synthesis pipeline (v0.5)
  338. llmwiki all # build → graph → export → lint in one shot (v1.2)
  339. llmwiki version
  340. ```
  341. Each subcommand has its own `--help`. All commands are also wrapped in one-click shell/batch scripts: `sync.sh`/`.bat`, `build.sh`/`.bat`, `serve.sh`/`.bat`, `upgrade.sh`/`.bat`.
  342. ## Works with
  343. | Agent | Adapter | Status | Added in |
  344. |---|---|---|---|
  345. | [Claude Code](https://claude.com/claude-code) | `llmwiki.adapters.claude_code` | ✅ Production | v0.1 |
  346. | [Obsidian](https://obsidian.md) (input) | `llmwiki.adapters.obsidian` | ✅ Production | v0.1 |
  347. | [Obsidian](https://obsidian.md) (output) | `llmwiki.obsidian_output` | ✅ Production | v0.2 |
  348. | [Codex CLI](https://github.com/openai/codex) | `llmwiki.adapters.codex_cli` | ✅ Production | v0.3 |
  349. | [Cursor](https://cursor.com) | `llmwiki.adapters.cursor` | 🧪 Beta — needs verification against current Cursor session format | v0.5 |
  350. | [Gemini CLI](https://ai.google.dev/gemini-api) | `llmwiki.adapters.gemini_cli` | 🧪 Beta — layout TBC | v0.5 |
  351. | [Copilot Chat](https://github.com/features/copilot) | `llmwiki.adapters.copilot_chat` | 🧪 Beta | v0.9 |
  352. | [Copilot CLI](https://github.com/features/copilot) | `llmwiki.adapters.copilot_cli` | 🧪 Beta | v0.9 |
  353. | OpenCode / OpenClaw | — | ⏸ Deferred | — |
  354. Adding a new agent is [one small file](docs/framework.md) — subclass `BaseAdapter`, declare `SUPPORTED_SCHEMA_VERSIONS`, ship a fixture + snapshot test.
  355. ## MCP server
  356. llmwiki ships its own MCP server (stdio transport, no SDK dependency) so any MCP client can query your wiki directly.
  357. ```bash
  358. python3 -m llmwiki.mcp # runs on stdin/stdout
  359. ```
  360. Twelve production tools (7 core + 5 added in v1.0 `#159`):
  361. | Tool | What |
  362. |---|---|
  363. | `wiki_query(question, max_pages)` | Keyword search + page content (no LLM synthesis) |
  364. | `wiki_search(term, include_raw)` | Raw grep over wiki/ (+ optional raw/) |
  365. | `wiki_list_sources(project)` | List raw source files with metadata |
  366. | `wiki_read_page(path)` | Read one page (path-traversal guarded) |
  367. | `wiki_lint()` | Orphans + broken-wikilinks report |
  368. | `wiki_sync(dry_run)` | Trigger the converter |
  369. | `wiki_export(format)` | Return any AI-consumable export (llms.txt, jsonld, sitemap, rss, manifest) |
  370. | `wiki_confidence(min, max)` | Pages by confidence range (v1.0) |
  371. | `wiki_lifecycle(state)` | Pages by draft/reviewed/verified/stale/archived (v1.0) |
  372. | `wiki_dashboard()` | Health summary: counts by type, lifecycle, confidence (v1.0) |
  373. | `wiki_entity_search(name, entity_type)` | Search entities by name substring or type (v1.0) |
  374. | `wiki_category_browse(tag)` | Browse tags with counts, drill into specific tag (v1.0) |
  375. Register in your MCP client's config — e.g. for Claude Desktop, add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
  376. ```json
  377. {
  378. "mcpServers": {
  379. "llmwiki": {
  380. "command": "python3",
  381. "args": ["-m", "llmwiki.mcp"]
  382. }
  383. }
  384. }
  385. ```
  386. ## Configuration
  387. Single JSON config at `examples/sessions_config.json`. Copy to `config.json` and edit:
  388. ```json
  389. {
  390. "filters": {
  391. "live_session_minutes": 60,
  392. "exclude_projects": []
  393. },
  394. "redaction": {
  395. "real_username": "YOUR_USERNAME",
  396. "replacement_username": "USER",
  397. "extra_patterns": [
  398. "(?i)(api[_-]?key|secret|token|bearer|password)...",
  399. "sk-[A-Za-z0-9]{20,}"
  400. ]
  401. },
  402. "truncation": {
  403. "tool_result_chars": 500,
  404. "bash_stdout_lines": 5
  405. },
  406. "adapters": {
  407. "obsidian": {
  408. "vault_paths": ["~/Documents/Obsidian Vault"]
  409. }
  410. }
  411. }
  412. ```
  413. All paths, regexes, truncation limits, and per-adapter settings are tunable. See [docs/configuration.md](docs/configuration.md).
  414. ## `.llmwikiignore`
  415. Gitignore-style pattern file at the repo root. Skip entire projects, dates, or specific sessions without touching config:
  416. ```
  417. # Skip a whole project
  418. confidential-client/
  419. # Skip anything before a date
  420. *2025-*
  421. # Keep exception
  422. !confidential-client/public-*
  423. ```
  424. ## Karpathy's LLM Wiki pattern
  425. This project follows the three-layer structure described in [Karpathy's gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f):
  426. 1. **Raw sources** (`raw/`) — immutable. Session transcripts converted from `.jsonl`.
  427. 2. **The wiki** (`wiki/`) — LLM-generated. One page per entity, concept, source. Interlinked via `[[wikilinks]]`.
  428. 3. **The schema** (`CLAUDE.md`, `AGENTS.md`) — tells your agent how to ingest and query.
  429. See [docs/architecture.md](docs/architecture.md) for the full breakdown and how it maps to the file tree.
  430. ## Design principles
  431. - **Stdlib first** — only mandatory runtime dep is `markdown`. Optional extras (`graph`, `dev`, `e2e`) layer in graph engines, lint/test tooling, and Playwright on top.
  432. - **Works offline** — no Google fonts, no external CSS. Syntax highlighting loads from a highlight.js CDN but degrades gracefully without it.
  433. - **Redact by default** — username, API keys, tokens, emails all get redacted before entering the wiki.
  434. - **Idempotent everything** — re-running any command is safe and cheap.
  435. - **Agent-agnostic core** — the converter doesn't know which agent produced the `.jsonl`; adapters translate.
  436. - **Privacy by default** — localhost-only binding, no telemetry, no cloud calls.
  437. - **Dual-format output (v0.4)** — every page ships both for humans (HTML) and AI agents (TXT + JSON + JSON-LD + sitemap + llms.txt).
  438. ## Docs
  439. - [Getting started](docs/getting-started.md) — 5-minute quickstart
  440. - **[Setup guide](docs/tutorials/setup-guide.md)** — 15-minute end-to-end tutorial: local setup → deploy to GitHub Pages → customization (v1.0)
  441. - [Obsidian integration](docs/obsidian-integration.md) — 5-minute setup, 6 recommended plugins, config recipes (v1.0)
  442. - [Architecture](docs/architecture.md) — Karpathy 3-layer + 8-layer build breakdown
  443. - [Configuration](docs/configuration.md) — every tuning knob
  444. - [Privacy](docs/privacy.md) — redaction rules + `.llmwikiignore` + localhost binding
  445. - [Windows setup](docs/windows-setup.md) — Windows-specific gotchas
  446. - [Framework](docs/framework.md) — Open Source Framework v4.1 adapted for agent-native dev tools
  447. - [Research](docs/research.md) — Phase 1.25 analysis of 15 prior LLM Wiki implementations
  448. - [Feature matrix](docs/feature-matrix.md) — all 161 features across 16 categories
  449. - [Roadmap](docs/roadmap.md) — Phase × Layer × Item MoSCoW table
  450. - **Translations**: [i18n/zh-CN](docs/i18n/zh-CN/), [i18n/ja](docs/i18n/ja/), [i18n/es](docs/i18n/es/)
  451. Per-adapter docs:
  452. - [Claude Code adapter](docs/adapters/claude-code.md)
  453. - [Codex CLI adapter](docs/adapters/codex-cli.md)
  454. - [Cursor adapter](docs/adapters/cursor.md)
  455. - [Gemini CLI adapter](docs/adapters/gemini-cli.md)
  456. - [Obsidian adapter](docs/adapters/obsidian.md)
  457. - [Copilot adapter (Chat + CLI)](docs/adapters/copilot.md)
  458. ## Releases
  459. | Version | Focus | Tag |
  460. |---|---|---|
  461. | [v0.1.0](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.1.0) | Core release — Claude Code adapter, god-level HTML UI, schema, CI, plugin scaffolding | `v0.1.0` |
  462. | [v0.2.0](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.2.0) | Extensions — 3 new slash commands, 3 new adapters, Obsidian bidirectional, full MCP server | `v0.2.0` |
  463. | [v0.3.0](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.3.0) | PyPI packaging, eval framework, i18n scaffold | `v0.3.0` |
  464. | [v0.4.0](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.4.0) | AI + human dual format — per-page .txt/.json siblings, llms.txt, JSON-LD graph, sitemap, RSS, schema.org microdata, reading time, related pages, activity heatmap, deep-link anchors, build manifest, link checker, `wiki_export` MCP tool | `v0.4.0` |
  465. | v0.5.0 – v0.9.0 | Internal sprint milestones — features (`_context.md`, auto-ingest, qmd export, model-profile schema, activity heatmap, Copilot adapters, etc.) shipped consolidated under the v0.9.x line. No standalone tags were published. | — |
  466. | [v0.9.1](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.9.1) | Sprint 1 & 2 foundation — link-obsidian CLI, 4-factor confidence scoring, 5-state lifecycle machine, llmbook-reference skill, 7 entity types, flat raw/ naming, pending ingest queue, `_context.md` stubs, meeting + Jira adapters, configurable Web Clipper intake, rich log format | `v0.9.1` |
  467. | [v0.9.2](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.9.2) | Sprint 3 quality — 11 lint rules (8 basic + 3 LLM-powered), Auto Dream MEMORY.md consolidation, Dataview dashboard template, category pages (Dataview + static), auto-build on sync + configurable lint schedule | `v0.9.2` |
  468. | [v0.9.3](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.9.3) | Sprint 3 polish — Obsidian Templater templates, integration guide, two-way editing tests, MCP server 7→12 tools, adapter config validation, pipeline fix (sigstore, PyPI gate) | `v0.9.3` |
  469. | [v0.9.4](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.9.4) | Session C1 (Sprint 4) — multi-agent skill installer, enhanced search with facets, configurable scheduled sync (launchd/systemd/Task Scheduler), CI wiki-checks workflow | `v0.9.4` |
  470. | [v0.9.5](https://github.com/Pratiyush/llm-wiki/releases/tag/v0.9.5) | Docs polish + consistency audit before v1.0.0 | `v0.9.5` |
  471. | [v1.0.0](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.0.0) | Production-ready Obsidian integration — full v1.0 scope | `v1.0.0` |
  472. | [v1.1.0-rc1](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.1.0-rc1) | Solo quick-win sprint — candidates workflow, Ollama scaffold, prompt-cache scaffold | `v1.1.0-rc1` |
  473. | [v1.1.0-rc2](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.1.0-rc2) | Session E — interactive graph viewer + remaining code-only v1.1 work | `v1.1.0-rc2` |
  474. | [v1.1.0-rc3](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.1.0-rc3) | Gap-sweep bundle — state portability, quarantine, sync --status, log CLI, synthesize --estimate breakdown, tag family, stale references, graph context menu, raw immutability, AI-sessions default | `v1.1.0-rc3` |
  475. | [v1.1.0-rc4](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.1.0-rc4) | Navigation + quality — graph `site_url` resolver (99.7% → 0% dead clicks), `llmwiki backlinks` CLI (95% → 0% orphan pages), source-code → GitHub link rewriter (471 → 100 broken), verify-before-fixing contribution rule | `v1.1.0-rc4` |
  476. | [v1.1.0-rc5](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.1.0-rc5) | Site audit + 5 closed batches — session-local ref stripping (351 → 247 broken), cheatsheet, README/CONTRIBUTING compile, expanded E2E, slash-CLI parity test, 4 adapter docs, Ollama tutorial, dual-mode docs skeleton, `/wiki-synthesize` slash | `v1.1.0-rc5` |
  477. | [v1.1.0-rc6](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.1.0-rc6) | rc6 batch — fixed adapter tag hardcoded to `claude-code` for every adapter (#346), tutorial UX polish with in-page TOC + prev/next + edit-on-GitHub (#282), command palette now indexes 107 doc pages + 17 slash commands (#277), content-hash cache for `md_to_html` (#283) | `v1.1.0-rc6` |
  478. | [v1.1.0-rc7](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.1.0-rc7) | rc7 batch — automatic AI-suggested tags during synthesis (#351), link-checker config fix (#348, #350, #353) | `v1.1.0-rc7` |
  479. | [v1.1.0-rc8](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.1.0-rc8) | rc8 batch — complete Mode B agent-delegate backend (#316): new `llmwiki synthesize --list-pending` + `--complete <uuid>` CLI subcommands, `/wiki-sync` step 6 auto-detects pending prompts, Mode B ships end-to-end without an API key | `v1.1.0-rc8` |
  480. | [**v1.2.0**](https://github.com/Pratiyush/llm-wiki/releases/tag/v1.2.0) | **First stable on the 1.x line** — `llmwiki all` one-shot pipeline runner, Playwright + axe-core E2E suite (#384), project-stub auto-seeding, 2 new lint rules, critical export-fidelity + sync-collision fixes, 10 UX-critique items (#387). PyPI distribution name: `llm-notebook`. | `v1.2.0` |
  481. ## Roadmap
  482. Shipped milestones:
  483. - **v0.5.0** — Folder-level `_context.md`, auto-ingest, adapter graduations, lazy search index, scheduled sync, WCAG, E2E tests ([milestone](https://github.com/Pratiyush/llm-wiki/milestone/4))
  484. - **v0.6.0** — qmd export, GitLab Pages CI, PyPI release automation, maintainer governance scaffold ([milestone](https://github.com/Pratiyush/llm-wiki/milestone/5))
  485. - **v0.7.0** — Structured model-profile schema, vs-comparison pages, append-only changelog timeline ([milestone](https://github.com/Pratiyush/llm-wiki/milestone/7))
  486. - **v0.8.0** — 365-day activity heatmap, tool-calling bar chart, token usage card, session metrics frontmatter ([milestone](https://github.com/Pratiyush/llm-wiki/milestone/8))
  487. - **v0.9.0** — Project topics, agent labels, Copilot adapters, image pipeline, highlight.js, public demo deployment
  488. - **v0.9.x** — Sprint 1-4 foundation for v1.0.0 Obsidian integration: confidence scoring, lifecycle state machine, 9 navigation files, 11 lint rules, Auto Dream, Dataview dashboard, multi-agent skills, 12-tool MCP server, meeting + Jira adapters
  489. Active milestones:
  490. | Milestone | Focus | Tracking |
  491. |---|---|---|
  492. | **v1.0.0** | Final docs polish + PyPI trusted publisher + release | [Milestone](https://github.com/Pratiyush/llm-wiki/milestone/9) |
  493. | **v1.1.0** | Ollama backend, prompt caching, interactive graph viewer, Homebrew tap | [Milestone](https://github.com/Pratiyush/llm-wiki/milestone/10) |
  494. | **v1.2.0** | ChatGPT + OpenCode adapters, vault-overlay mode, tree-aware search, cache tiers | [Milestone](https://github.com/Pratiyush/llm-wiki/milestone/11) |
  495. ### Deployment targets
  496. - **GitHub Pages** — shipped in v0.1 via `.github/workflows/pages.yml` (triggers on push to master). See [`docs/deploy/github-pages.md`](docs/deploy/github-pages.md).
  497. - **Docker / GHCR** — pull and run: `docker compose pull && docker compose up -d`. Image published to `ghcr.io/pratiyush/llm-wiki` on every tag push. See [`docs/deploy/docker.md`](docs/deploy/docker.md).
  498. - **GitLab Pages** — copy [`.gitlab-ci.yml.example`](.gitlab-ci.yml.example) → `.gitlab-ci.yml`. See [`docs/deploy/gitlab-pages.md`](docs/deploy/gitlab-pages.md).
  499. - **Vercel / Netlify** — static deploy after `llmwiki build`. See [`docs/deploy/vercel-netlify.md`](docs/deploy/vercel-netlify.md).
  500. - **Any static host** — `llmwiki build` writes to `site/`, which you can `rsync`/`scp` anywhere.
  501. ## Acknowledgements
  502. - [Andrej Karpathy](https://twitter.com/karpathy) for [the LLM Wiki idea](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
  503. - [SamurAIGPT/llm-wiki-agent](https://github.com/SamurAIGPT/llm-wiki-agent), [lucasastorian/llmwiki](https://github.com/lucasastorian/llmwiki), [xoai/sage-wiki](https://github.com/xoai/sage-wiki), and [bashiraziz/llm-wiki-template](https://github.com/bashiraziz/llm-wiki-template) — prior art that shaped this.
  504. - [Python Markdown](https://python-markdown.github.io/) for the rendering pipeline, and [highlight.js](https://highlightjs.org/) for client-side syntax highlighting.
  505. - [llmstxt.org](https://llmstxt.org) for the llms.txt spec used in v0.4.
  506. ## License
  507. [MIT](LICENSE) © Pratiyush
llmwiki/build.py · Bundled script · 2,710 lines · 1 finding marked
  1. """llmwiki — static HTML site builder.
  2. Reads converted markdown sources under `raw/` and produces a complete static
  3. site under `site/` with:
  4. - Home + projects index + sessions index + per-project + per-session pages
  5. - Inter + JetBrains Mono typography, purple accent (#7C3AED)
  6. - Light / dark theme toggle (data-theme + system preference + localStorage)
  7. - Global search index (site/search-index.json) — client-side fuzzy matcher
  8. - Cmd+K command palette (vanilla JS, no framework)
  9. - Keyboard shortcuts: /, g h, g p, g s, j/k, ?
  10. - highlight.js client-side syntax highlighting (CDN, light + dark themes)
  11. - Collapsible tool-result sections (<details>) for long outputs
  12. - Copy-as-markdown + copy-code buttons (Clipboard API + execCommand fallback)
  13. - Breadcrumbs + reading progress bar
  14. - Filter bar on the sessions table
  15. - Mobile-responsive, print-friendly
  16. - ARIA focus rings and prefers-reduced-motion support
  17. Stdlib + `markdown` (required). No optional deps — highlight.js loads from CDN.
  18. Usage:
  19. python3 -m llmwiki build [--synthesize] [--out <dir>]
  20. """
  21. from __future__ import annotations
  22. import argparse
  23. import html
  24. import json
  25. import os
  26. import re
  27. import shutil
  28. import subprocess
  29. import sys
  30. from collections import defaultdict
  31. from datetime import datetime, timezone
  32. from pathlib import Path
  33. from typing import Any, Optional
  34. import markdown
  35. from markdown.preprocessors import Preprocessor
  36. from llmwiki import REPO_ROOT
  37. from llmwiki.changelog_timeline import (
  38. extract_price_points,
  39. find_recently_updated,
  40. parse_changelog,
  41. render_changelog_timeline,
  42. render_price_sparkline,
  43. render_recent_activity,
  44. render_recently_updated,
  45. )
  46. from llmwiki.log_reader import recent_events as _recent_log_events
  47. from llmwiki.context_md import is_context_file
  48. from llmwiki.freshness import freshness_badge, load_freshness_config
  49. from llmwiki.project_topics import (
  50. extract_session_topics,
  51. get_project_topics,
  52. load_project_profile,
  53. render_topic_chips,
  54. )
  55. from llmwiki.viz_heatmap import collect_session_counts, render_heatmap
  56. from llmwiki.viz_tokens import (
  57. render_project_token_card,
  58. render_session_token_card,
  59. render_site_token_stats,
  60. )
  61. from llmwiki.viz_tools import (
  62. render_project_tool_chart,
  63. render_session_tool_chart,
  64. )
  65. # ─── paths ─────────────────────────────────────────────────────────────────
  66. RAW_DIR = REPO_ROOT / "raw"
  67. RAW_SESSIONS = RAW_DIR / "sessions"
  68. DEFAULT_OUT_DIR = REPO_ROOT / "site"
  69. # v0.7+: optional per-project metadata (topics, description, homepage).
  70. # Users drop a `wiki/projects/<slug>.md` file with frontmatter.
  71. PROJECTS_META_DIR = REPO_ROOT / "wiki" / "projects"
  72. # ─── frontmatter ───────────────────────────────────────────────────────────
  73. # #409 / #423: build.py used to ship a divergent regex (LF-only, no BOM
  74. # handling, simpler list parser) which silently dropped frontmatter on
  75. # Windows-authored files. Unified to the canonical parser in
  76. # `_frontmatter.py`. Re-exported under the historical name so external
  77. # consumers (and `tests/test_render_split.py`) keep working.
  78. from llmwiki._frontmatter import ( # noqa: E402
  79. _FRONTMATTER_RE,
  80. parse_frontmatter,
  81. )
  82. # ─── discovery ─────────────────────────────────────────────────────────────
  83. # #405 path-traversal guard. Site paths are composed by joining `out_dir`
  84. # with `project_slug` and other slug values from frontmatter. A poisoned
  85. # `project: ../../etc/passwd` would otherwise write outside `out_dir`.
  86. _SAFE_SLUG_RE = re.compile(r"^[A-Za-z0-9._-]+$")
  87. def _safe_slug(value: str | None, *, fallback: str = "_unknown") -> str:
  88. """Return a path-safe single-segment slug.
  89. Rejects empty values, traversal segments (``..``), absolute paths
  90. (leading ``/`` or backslash), null bytes, and anything containing
  91. characters outside ``[A-Za-z0-9._-]``. Falls back to ``fallback`` so
  92. the build keeps going on poisoned frontmatter — the offending
  93. session lands under a clearly abnormal slug rather than escaping
  94. ``out_dir``.
  95. """
  96. if not value:
  97. return fallback
  98. s = str(value).strip()
  99. # Strip surrounding quotes leaked from naive YAML parsers.
  100. if len(s) >= 2 and s[0] == s[-1] and s[0] in ("'", '"'):
  101. s = s[1:-1]
  102. if not s or s in (".", ".."):
  103. return fallback
  104. if "/" in s or "\\" in s or "\x00" in s:
  105. return fallback
  106. if not _SAFE_SLUG_RE.match(s):
  107. return fallback
  108. return s
  109. def _is_subagent(meta: dict[str, Any], path: Path) -> bool:
  110. """Return True iff a session is a sub-agent run (#492 / #406).
  111. Prefers the adapter-written ``is_subagent`` frontmatter field
  112. (canonical contract since #406). Falls back to the legacy
  113. ``"subagent" in path.name`` substring check ONLY if the field is
  114. missing — needed to keep pre-#406 raw files classified correctly
  115. until they're re-synced.
  116. Accepts the field as either a real bool or one of the
  117. case-insensitive strings ``"true"`` / ``"false"`` since
  118. frontmatter parsers historically coerced inconsistently.
  119. """
  120. raw = meta.get("is_subagent")
  121. if isinstance(raw, bool):
  122. return raw
  123. if isinstance(raw, str):
  124. s = raw.strip().lower()
  125. if s in ("true", "yes", "1"):
  126. return True
  127. if s in ("false", "no", "0"):
  128. return False
  129. # Field absent or unrecognised — fall back to the legacy heuristic
  130. # so pre-#406 raw files (no is_subagent field) still get the right
  131. # answer. The renderer renames sub-agent slugs to
  132. # `<slug>-subagent-<id>`, so the substring match is correct on
  133. # canonically-renamed files even when meta is missing.
  134. return "subagent" in path.name
  135. def discover_sources(root: Path) -> list[tuple[Path, dict[str, Any], str]]:
  136. out: list[tuple[Path, dict[str, Any], str]] = []
  137. if not root.exists():
  138. return out
  139. for p in sorted(root.rglob("*.md")):
  140. # v0.5 (#60): `_context.md` files are folder metadata for LLM
  141. # navigation, not pages. Skip them so they never appear in the
  142. # session index, search index, or AI-consumable exports.
  143. if is_context_file(p):
  144. continue
  145. try:
  146. text = p.read_text(encoding="utf-8")
  147. except OSError:
  148. continue
  149. meta, body = parse_frontmatter(text)
  150. # #405: sanitize the frontmatter values that compose output paths.
  151. # Original values stay available via meta.get(); the *_safe_*
  152. # versions are what every path-composition site downstream uses.
  153. meta["project"] = _safe_slug(
  154. meta.get("project") or p.parent.name,
  155. fallback=_safe_slug(p.parent.name, fallback="_unknown"),
  156. )
  157. if "slug" in meta:
  158. meta["slug"] = _safe_slug(meta["slug"], fallback=p.stem)
  159. out.append((p, meta, body))
  160. return out
  161. def group_by_project(
  162. sources: list[tuple[Path, dict[str, Any], str]],
  163. ) -> dict[str, list[tuple[Path, dict[str, Any], str]]]:
  164. g: dict[str, list[tuple[Path, dict[str, Any], str]]] = defaultdict(list)
  165. for p, meta, body in sources:
  166. project = str(meta.get("project") or p.parent.name)
  167. g[project].append((p, meta, body))
  168. for k in g:
  169. g[k].sort(key=lambda t: str(t[1].get("started", t[0].name)))
  170. return g
  171. _SLUG_SPLIT_RE = re.compile(r"[-_]+")
  172. def _humanize_slug(slug: str) -> str:
  173. """Turn a kebab/snake-case slug into a human-readable title.
  174. `my-cool-project` → `My Cool Project`. Single-letter parts are
  175. upper-cased the same way as multi-letter parts. Empty / whitespace
  176. input returns the original (callers handle the empty case).
  177. """
  178. parts = [p for p in _SLUG_SPLIT_RE.split(slug.strip()) if p]
  179. if not parts:
  180. return slug.strip()
  181. return " ".join(p[:1].upper() + p[1:] for p in parts)
  182. def _derive_stub_description(
  183. sessions: list[tuple[Path, dict[str, Any], str]],
  184. ) -> str:
  185. """Pick a sensible description from the most-recent session.
  186. Prefers an explicit `summary` (truncated to ~140 chars), then the
  187. humanised session slug, then empty. `sessions` arrives sorted oldest
  188. → newest by `discover_sources`/grouping, so we walk the tail.
  189. """
  190. for _path, meta, _body in reversed(sessions):
  191. summary = meta.get("summary")
  192. if isinstance(summary, str) and summary.strip():
  193. text = summary.strip().splitlines()[0].strip()
  194. if len(text) > 140:
  195. text = text[:137].rstrip() + "..."
  196. return text
  197. raw_slug = meta.get("slug")
  198. if isinstance(raw_slug, str) and raw_slug.strip():
  199. humanised = _humanize_slug(raw_slug)
  200. if humanised:
  201. return humanised
  202. return ""
  203. def _derive_stub_topics(
  204. sessions: list[tuple[Path, dict[str, Any], str]],
  205. max_topics: int = 6,
  206. ) -> list[str]:
  207. """Aggregate topics from session frontmatter, then `tools_used` as a
  208. secondary source. Uses `extract_session_topics` for the tags path so
  209. the noise filter stays in sync with `project_topics.py`. Falls back
  210. to `min_count=1` because most projects have only a few sessions and
  211. the `min_count=2` default in `extract_session_topics` would suppress
  212. nearly everything at seed time. Returns at most `max_topics` topics.
  213. """
  214. metas = [meta for _path, meta, _body in sessions]
  215. topics = extract_session_topics(metas, max_topics=max_topics, min_count=1)
  216. if topics:
  217. return topics
  218. # Fallback: tools_used aggregation (filtered by the same noise set).
  219. from llmwiki.project_topics import _NOISE_TAGS
  220. counts: dict[str, int] = {}
  221. for meta in metas:
  222. raw = meta.get("tools_used")
  223. if isinstance(raw, list):
  224. items = raw
  225. elif isinstance(raw, dict):
  226. items = list(raw.keys())
  227. else:
  228. items = []
  229. for item in items:
  230. tag = str(item).strip().lower()
  231. if tag and tag not in _NOISE_TAGS:
  232. counts[tag] = counts.get(tag, 0) + 1
  233. ordered = sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))
  234. return [tag for tag, _ in ordered[:max_topics]]
  235. def _format_topics_yaml(topics: list[str]) -> str:
  236. """Inline-list YAML serialisation that matches the parser in
  237. `project_topics._parse_topics_frontmatter`."""
  238. if not topics:
  239. return "[]"
  240. return "[" + ", ".join(topics) + "]"
  241. def ensure_project_stubs(
  242. groups: dict[str, list[tuple[Path, dict[str, Any], str]]],
  243. meta_dir: Path,
  244. ) -> list[Path]:
  245. """Auto-seed ``wiki/projects/<slug>.md`` for any discovered project
  246. that doesn't already have one (`issues-commands.md` I-12).
  247. Without this, real projects render a bare hero — no description, no
  248. topic chips, no homepage — because those fields come from a hand-
  249. authored file that never gets created on sync. Seeding pre-populates
  250. `topics:` from session tags/tools and `description:` from the most-
  251. recent session's summary or slug, so every real project lights up the
  252. moment its first session lands (closes #425). Existing hand-authored
  253. files are never overwritten — only the absence of a file triggers a
  254. write, so the user's edits always win.
  255. Returns the list of stub paths actually written (empty if all project
  256. metadata files already existed).
  257. """
  258. meta_dir.mkdir(parents=True, exist_ok=True)
  259. written: list[Path] = []
  260. for slug in sorted(groups):
  261. target = meta_dir / f"{slug}.md"
  262. if target.exists():
  263. continue
  264. sessions = groups[slug]
  265. topics = _derive_stub_topics(sessions)
  266. description = _derive_stub_description(sessions)
  267. # Escape embedded double-quotes in description so the YAML stays
  268. # valid — slugs/summaries from real sessions occasionally contain
  269. # quotes (`"why didn't this work"`).
  270. description_safe = description.replace("\\", "\\\\").replace('"', '\\"')
  271. # #py-l8 (#606): drop f-prefix on lines that have no
  272. # placeholders so ruff's F541 doesn't flag the file. Mixed
  273. # f-/plain in a concatenation chain is fine.
  274. stub = (
  275. "---\n"
  276. f'title: "{slug}"\n'
  277. "type: entity\n"
  278. "entity_type: project\n"
  279. f"project: {slug}\n"
  280. f"topics: {_format_topics_yaml(topics)}\n"
  281. f'description: "{description_safe}"\n'
  282. 'homepage: ""\n'
  283. "---\n\n"
  284. f"# {slug}\n\n"
  285. "*Auto-generated project stub. `topics` and `description` are "
  286. "pre-filled from session metadata — edit any field above and "
  287. "the build will pick it up. Fill in `homepage` to add a link "
  288. "chip to the project hero.*\n"
  289. )
  290. target.write_text(stub, encoding="utf-8")
  291. written.append(target)
  292. return written
  293. # ─── markdown normaliser + renderer ───────────────────────────────────────
  294. _H1_LINE_RE = re.compile(r"^#\s+.*\n", re.MULTILINE)
  295. def strip_leading_h1(body: str) -> str:
  296. m = _H1_LINE_RE.search(body)
  297. if m and m.start() < 200:
  298. body = body[: m.start()] + body[m.end() :]
  299. body = body.lstrip("\n")
  300. return body
  301. def normalize_markdown(body: str) -> str:
  302. """Fix common markdown glitches from the converter:
  303. 1. Insert blank line before lists that follow a bold header.
  304. 2. Outdent fenced code blocks that are 2-space-indented under a list item.
  305. """
  306. body = re.sub(
  307. r"(\*\*(?:Tools used|Tool results):\*\*)\n(?!\n)", r"\1\n\n", body,
  308. )
  309. lines = body.split("\n")
  310. out: list[str] = []
  311. i = 0
  312. while i < len(lines):
  313. line = lines[i]
  314. if line.strip() == "```" and line.startswith(" "):
  315. if out and out[-1].strip():
  316. out.append("")
  317. out.append("```")
  318. i += 1
  319. while i < len(lines) and lines[i].strip() != "```":
  320. bl = lines[i]
  321. if bl.startswith(" "):
  322. bl = bl[2:]
  323. out.append(bl)
  324. i += 1
  325. if i < len(lines):
  326. out.append("```")
  327. i += 1
  328. if i < len(lines) and lines[i].strip():
  329. out.append("")
  330. else:
  331. out.append(line)
  332. i += 1
  333. return "\n".join(out)
  334. # v0.5 (#74): Session content frequently mentions HTML-ish strings in prose —
  335. # e.g. an assistant describing how a hidden `<textarea class="md-source">` works.
  336. # The default Python markdown library passes raw HTML tags through unchanged,
  337. # which means a session that mentions `<textarea>` outside of backticks leaks
  338. # an unclosed textarea into the DOM, swallowing every following element
  339. # (including the <script> tag that boots highlight.js). The v0.5 hljs swap
  340. # made this pre-existing bug catastrophic — before, Pygments rendered code
  341. # server-side so the broken tail didn't visibly matter; now, a single
  342. # unescaped tag breaks the whole page's syntax highlighting.
  343. #
  344. # The preprocessor below escapes anything that *looks* like an HTML tag start
  345. # (`<tagname` or `</tagname`) outside of inline backticks. Fenced code blocks
  346. # are already extracted into placeholders by `fenced_code` (priority 25) before
  347. # this runs (priority 22). Priority 22 also ensures we run *before*
  348. # `html_block` (priority 20) so raw HTML blocks never get a chance to be
  349. # preserved as-is. Bare `<` / `<space>` (e.g. `x < 10`) are left alone —
  350. # markdown's own escaper handles those. HTML comments (`<!-- ... -->`) are
  351. # preserved because the regex only matches `<[letter]`, not `<!`, and
  352. # build.py emits an `<!-- llmwiki:metadata -->` comment that AI agents parse.
  353. _TAG_START_RE = re.compile(r"<(/?[A-Za-z][A-Za-z0-9:_-]*)")
  354. # #sec-13 (#557): also neutralise raw `<![CDATA[` blocks in prose. CDATA
  355. # isn't allowed in HTML but some browsers / parsers treat it as a
  356. # start-of-foreign-content marker; surfaces in MathML / SVG islands or
  357. # in legacy XHTML rendering paths. Escape the leading `<` so the
  358. # surrounding markdown processor doesn't pass it through as-is.
  359. _CDATA_START_RE = re.compile(r"<!\[CDATA\[")
  360. _INLINE_CODE_RE = re.compile(r"`[^`]*`")
  361. class _EscapeRawHtmlPreprocessor(Preprocessor):
  362. """Escape HTML tag-start patterns outside code spans so raw `<textarea>`
  363. etc. in session prose can never leak into the DOM as live elements.
  364. See the comment above `md_to_html` for the full rationale."""
  365. def run(self, lines: list[str]) -> list[str]:
  366. out: list[str] = []
  367. for line in lines:
  368. parts: list[tuple[str, str]] = []
  369. last = 0
  370. for m in _INLINE_CODE_RE.finditer(line):
  371. parts.append(("text", line[last : m.start()]))
  372. parts.append(("code", m.group(0)))
  373. last = m.end()
  374. parts.append(("text", line[last:]))
  375. rebuilt: list[str] = []
  376. for kind, part in parts:
  377. if kind == "text":
  378. # #sec-13: neutralise CDATA markers BEFORE the tag-
  379. # start sub so we don't accidentally double-escape.
  380. part = _CDATA_START_RE.sub(r"&lt;![CDATA[", part)
  381. rebuilt.append(_TAG_START_RE.sub(r"&lt;\1", part))
  382. else:
  383. rebuilt.append(part)
  384. out.append("".join(rebuilt))
  385. return out
  386. # #283: in-memory content-hash cache for md_to_html. Same markdown body
  387. # always produces the same HTML, and build steps call md_to_html on the
  388. # same boilerplate (e.g. `## Connections`) across hundreds of pages.
  389. # blake2b(digest_size=8) keyed + bounded by a size cap so repeated
  390. # builds in the same Python process (tests, watch mode, bulk exports)
  391. # don't re-parse.
  392. #
  393. # #417: switched from SHA-256 hex (allocates 64-byte string per call)
  394. # to blake2b(digest_size=8) returning bytes. ~3× faster + 8× less
  395. # allocation per cache key on a 5000-page corpus. The 8-byte (64-bit)
  396. # digest gives a birthday-collision threshold around 4×10^9 entries —
  397. # the 4096-entry cap stays many orders of magnitude below that.
  398. _MD_CACHE: dict[bytes, str] = {}
  399. _PLAIN_CACHE: dict[bytes, str] = {}
  400. _MD_CACHE_MAX = 4096 # entries; ~20 MB ceiling at ~5 KB avg
  401. _md_cache_hits = 0
  402. _md_cache_misses = 0
  403. _plain_cache_hits = 0
  404. _plain_cache_misses = 0
  405. def _content_key(body: str) -> bytes:
  406. """Compute the cache key for a markdown body (#417).
  407. blake2b is significantly faster than SHA-256 for short strings,
  408. and the 8-byte digest is enough headroom for the 4096-entry cap.
  409. Bytes (not hex) avoids the encode-back-to-string allocation.
  410. """
  411. import hashlib as _hl
  412. return _hl.blake2b(body.encode("utf-8"), digest_size=8).digest()
  413. def md_to_html_cache_stats() -> dict[str, int]:
  414. """Return ``{hits, misses, size}`` for observability (#283)."""
  415. return {
  416. "hits": _md_cache_hits,
  417. "misses": _md_cache_misses,
  418. "size": len(_MD_CACHE),
  419. "plain_hits": _plain_cache_hits,
  420. "plain_misses": _plain_cache_misses,
  421. "plain_size": len(_PLAIN_CACHE),
  422. }
  423. def md_to_html_cache_clear() -> None:
  424. """Clear the md_to_html + md_to_plain caches (used in tests)."""
  425. global _md_cache_hits, _md_cache_misses
  426. global _plain_cache_hits, _plain_cache_misses
  427. _MD_CACHE.clear()
  428. _PLAIN_CACHE.clear()
  429. _md_cache_hits = 0
  430. _md_cache_misses = 0
  431. _plain_cache_hits = 0
  432. _plain_cache_misses = 0
  433. def _evict_first(cache: dict) -> None:
  434. """FIFO-evict the oldest cache entry."""
  435. try:
  436. first_key = next(iter(cache))
  437. del cache[first_key]
  438. except StopIteration:
  439. pass
  440. def md_to_html(body: str) -> str:
  441. global _md_cache_hits, _md_cache_misses
  442. key = _content_key(body)
  443. cached = _MD_CACHE.get(key)
  444. if cached is not None:
  445. _md_cache_hits += 1
  446. return cached
  447. _md_cache_misses += 1
  448. result = _md_to_html_uncached(body)
  449. if len(_MD_CACHE) >= _MD_CACHE_MAX:
  450. _evict_first(_MD_CACHE)
  451. _MD_CACHE[key] = result
  452. return result
  453. def _md_to_html_uncached(body: str) -> str:
  454. body = normalize_markdown(body)
  455. # v0.5: highlight.js replaces server-side Pygments/codehilite. The
  456. # fenced_code extension emits `<pre><code class="language-xxx">` and
  457. # highlight.js (loaded via CDN in page_head) picks it up client-side.
  458. # Benefits: lighter builds, no optional dep, consistent look across pages,
  459. # and auto-detection for untagged blocks.
  460. extensions = ["fenced_code", "tables", "toc", "sane_lists"]
  461. ext_configs: dict[str, dict[str, Any]] = {
  462. # #646: drop `permalink: True`. The Python-Markdown TOC
  463. # extension's permalink emits a `<a class="headerlink">¶</a>`
  464. # next to every heading; the site CSS doesn't style
  465. # `.headerlink` (only `.deep-link` is styled), so axe-core
  466. # flags every one as a `link-in-text-block` violation
  467. # (links that aren't visually distinguishable). The JS-
  468. # driven `.deep-link` icon next to each heading (rendered by
  469. # render/js.py) is the canonical deep-link affordance — it
  470. # has CSS + hover state + aria-hidden treatment. Two emitters
  471. # for the same job; the markdown one is the older one. Anchor
  472. # targets (`<h2 id="...">`) still ship via toc — links to
  473. # `#section-name` keep working.
  474. "toc": {"toc_depth": "2-3"},
  475. }
  476. md = markdown.Markdown(extensions=extensions, extension_configs=ext_configs)
  477. # v0.5 (#74): escape raw HTML tags in prose so session content mentioning
  478. # `<textarea>` etc. can't break the page. Runs after fenced_code (25) and
  479. # before html_block (20), so fenced code is preserved verbatim (through
  480. # placeholders), inline code via backticks is preserved by this
  481. # preprocessor's own backtick-skipping, and everything else is safe.
  482. md.preprocessors.register(
  483. _EscapeRawHtmlPreprocessor(md), "escape_raw_html_tags", 22
  484. )
  485. return md.convert(body)
  486. def md_to_plain_text(body: str) -> str:
  487. """Strip markdown to plain text for the search index.
  488. #417: memoized on the same content key as md_to_html. The build
  489. pipeline calls md_to_html and md_to_plain_text on the same body
  490. repeatedly (per-page render + search-index extract + RSS summary
  491. + .txt sibling). Sharing the key makes the second + third + …
  492. calls free.
  493. """
  494. global _plain_cache_hits, _plain_cache_misses
  495. key = _content_key(body)
  496. cached = _PLAIN_CACHE.get(key)
  497. if cached is not None:
  498. _plain_cache_hits += 1
  499. return cached
  500. _plain_cache_misses += 1
  501. result = _md_to_plain_text_uncached(body)
  502. if len(_PLAIN_CACHE) >= _MD_CACHE_MAX:
  503. _evict_first(_PLAIN_CACHE)
  504. _PLAIN_CACHE[key] = result
  505. return result
  506. def _md_to_plain_text_uncached(body: str) -> str:
  507. body = normalize_markdown(strip_leading_h1(body))
  508. # Remove code blocks (they're noisy in search)
  509. body = re.sub(r"```.*?```", " ", body, flags=re.DOTALL)
  510. # Inline code
  511. body = re.sub(r"`([^`]*)`", r"\1", body)
  512. # Links: [text](url) → text
  513. body = re.sub(r"\[([^\]]*)\]\([^\)]*\)", r"\1", body)
  514. # Wikilinks: [[name]] → name
  515. body = re.sub(r"\[\[([^\]]*)\]\]", r"\1", body)
  516. # Headings: strip leading #
  517. body = re.sub(r"^#+\s*", "", body, flags=re.MULTILINE)
  518. # Bold/italic marks
  519. body = re.sub(r"\*\*([^*]*)\*\*", r"\1", body)
  520. body = re.sub(r"\*([^*]*)\*", r"\1", body)
  521. # HTML comments
  522. body = re.sub(r"<!--.*?-->", "", body, flags=re.DOTALL)
  523. # Collapse whitespace
  524. body = re.sub(r"\s+", " ", body).strip()
  525. return body
  526. # ─── helpers for frontmatter consumers ─────────────────────────────────────
  527. def get_tools_list(meta: dict[str, Any]) -> list[str]:
  528. tools = meta.get("tools_used", [])
  529. if isinstance(tools, str):
  530. return [t.strip() for t in tools.strip("[]").split(",") if t.strip()]
  531. return list(tools) if tools else []
  532. def short_started(meta: dict[str, Any]) -> str:
  533. s = str(meta.get("started", ""))
  534. return s[:16].replace("T", " ")
  535. # ─── freshness (content staleness) ────────────────────────────────────────
  536. # Cached once per build so every page sees the same "now" and the same
  537. # thresholds. Populated lazily by render_freshness().
  538. _FRESHNESS_CONFIG: Optional[tuple[int, int]] = None
  539. _BUILD_NOW: Optional[datetime] = None
  540. def render_freshness(meta: dict[str, Any]) -> str:
  541. """Render a freshness badge for a page's frontmatter using cached config.
  542. Thresholds come from ``config.json`` (freshness.green_days /
  543. yellow_days) or the module defaults. Build-time "now" is cached the
  544. first call so the whole site renders with one consistent clock.
  545. """
  546. global _FRESHNESS_CONFIG, _BUILD_NOW
  547. if _FRESHNESS_CONFIG is None:
  548. _FRESHNESS_CONFIG = load_freshness_config()
  549. if _BUILD_NOW is None:
  550. _BUILD_NOW = datetime.now(timezone.utc).replace(tzinfo=None)
  551. green, yellow = _FRESHNESS_CONFIG
  552. return freshness_badge(meta, now=_BUILD_NOW, green_days=green, yellow_days=yellow)
  553. # ─── html template helpers ─────────────────────────────────────────────────
  554. # v0.5: highlight.js for client-side syntax highlighting. Two themes so the
  555. # switcher can swap between light/dark without a network round-trip. Pinned
  556. # to a major version for stability, served from jsdelivr.
  557. HLJS_VERSION = "11.9.0"
  558. HLJS_LIGHT_CSS = (
  559. f"https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@{HLJS_VERSION}"
  560. "/build/styles/github.min.css"
  561. )
  562. HLJS_DARK_CSS = (
  563. f"https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@{HLJS_VERSION}"
  564. "/build/styles/github-dark.min.css"
  565. )
  566. HLJS_SCRIPT = (
  567. f"https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@{HLJS_VERSION}"
  568. "/build/highlight.min.js"
  569. )
  570. def _hljs_head_tags() -> str:
  571. """Return the `<link>` tags for highlight.js themes. The dark theme is
  572. loaded with ``disabled`` and the light theme is the default — the runtime
  573. swaps the ``disabled`` flag when the theme toggles, so code blocks stay in
  574. sync with the rest of the page."""
  575. return (
  576. f' <link id="hljs-light" rel="stylesheet" href="{HLJS_LIGHT_CSS}">\n'
  577. f' <link id="hljs-dark" rel="stylesheet" href="{HLJS_DARK_CSS}" disabled>\n'
  578. )
  579. _PRE_PAINT_THEME_SCRIPT = """ <script>
  580. /* #458: read localStorage.llmwiki-theme BEFORE first paint so users
  581. never see a flash of the wrong theme when navigating between pages.
  582. Falls back to prefers-color-scheme, then dark. Mirrors the same
  583. pre-paint pattern graph.html already uses (#477). */
  584. (function () {
  585. try {
  586. var t = localStorage.getItem('llmwiki-theme');
  587. if (t !== 'dark' && t !== 'light') {
  588. t = (window.matchMedia && window.matchMedia('(prefers-color-scheme: light)').matches) ? 'light' : 'dark';
  589. }
  590. document.documentElement.setAttribute('data-theme', t);
  591. } catch (e) {
  592. document.documentElement.setAttribute('data-theme', 'dark');
  593. }
  594. })();
  595. </script>
  596. """
  597. def page_head(title: str, description: str, css_prefix: str = "", lang: str = "en") -> str:
  598. # #ui-m13 (#576): lang argument lets callers override the default
  599. # `<html lang="en">` for translated docs (`docs/i18n/<locale>/`).
  600. # See also page_head_article() which has the same parameter.
  601. return f"""<!DOCTYPE html>
  602. <html lang="{html.escape(lang)}" dir="auto">
  603. <head>
  604. <meta charset="UTF-8">
  605. <meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
  606. <title>{html.escape(title)}</title>
  607. <meta name="description" content="{html.escape(description)}">
  608. {_PRE_PAINT_THEME_SCRIPT} <link rel="preconnect" href="https://fonts.googleapis.com">
  609. <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
  610. <!-- #ui-m14 (#577): async-load Google Fonts via media="print" + onload swap so it doesn't render-block first paint. <noscript> fallback for JS-disabled users. -->
  611. <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" media="print" onload="this.media='all'">
  612. <noscript><link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet"></noscript>
  613. {_hljs_head_tags()} <link rel="stylesheet" href="{css_prefix}style.css">
  614. </head>
  615. <body>
  616. <a href="#main-content" class="skip-link">Skip to content</a>
  617. <div class="progress-bar" id="progress-bar"></div>
  618. """
  619. def page_head_article(
  620. title: str,
  621. description: str,
  622. css_prefix: str = "",
  623. canonical: str = "",
  624. date: str = "",
  625. metadata_comment: str = "",
  626. lang: str = "en",
  627. ) -> str:
  628. """v0.4: Extended page head for session (Article) pages with schema.org
  629. microdata, canonical link, and an AI-readable metadata HTML comment.
  630. #ui-m13 (#576): `lang` arg lets translated docs override the
  631. default `en`. `dir="auto"` lets the browser infer RTL/LTR per
  632. paragraph for sessions whose body contains Arabic / Hebrew
  633. transliterations or quotes."""
  634. canonical_tag = ""
  635. if canonical:
  636. canonical_tag = f' <link rel="canonical" href="{html.escape(canonical)}">\n'
  637. og_tags = f""" <meta property="og:type" content="article">
  638. <meta property="og:title" content="{html.escape(title)}">
  639. <meta property="og:description" content="{html.escape(description)}">
  640. """
  641. if date:
  642. og_tags += f' <meta property="article:published_time" content="{html.escape(date)}">\n'
  643. return f"""<!DOCTYPE html>
  644. <html lang="{html.escape(lang)}" dir="auto">
  645. <head>
  646. <meta charset="UTF-8">
  647. <meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
  648. <title>{html.escape(title)}</title>
  649. <meta name="description" content="{html.escape(description)}">
  650. {_PRE_PAINT_THEME_SCRIPT}{canonical_tag}{og_tags} <link rel="preconnect" href="https://fonts.googleapis.com">
  651. <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
  652. <!-- #ui-m14 (#577): async-load Google Fonts via media="print" + onload swap so it doesn't render-block first paint. <noscript> fallback for JS-disabled users. -->
  653. <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" media="print" onload="this.media='all'">
  654. <noscript><link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet"></noscript>
  655. {_hljs_head_tags()} <link rel="stylesheet" href="{css_prefix}style.css">
  656. </head>
  657. <body>
  658. {metadata_comment}<a href="#main-content" class="skip-link">Skip to content</a>
  659. <div class="progress-bar" id="progress-bar"></div>
  660. """
  661. def _build_metadata_comment(
  662. meta: dict[str, Any],
  663. slug: str,
  664. project_slug: str,
  665. reading_min: int,
  666. html_stem: str = "",
  667. ) -> str:
  668. """An HTML comment at the top of every session page that AI agents can
  669. parse without needing to fetch the separate .json sibling."""
  670. fields = [
  671. f"slug: {slug}",
  672. f"project: {project_slug}",
  673. ]
  674. for key in ("date", "started", "ended", "model", "gitBranch", "permissionMode", "user_messages", "tool_calls"):
  675. val = meta.get(key)
  676. if val is not None:
  677. fields.append(f"{key}: {val}")
  678. tools = get_tools_list(meta)
  679. if tools:
  680. fields.append(f"tools_used: [{', '.join(tools)}]")
  681. fields.append(f"reading_min: {reading_min}")
  682. sibling_stem = html_stem or slug
  683. fields.append(f"txt_sibling: {sibling_stem}.txt")
  684. fields.append(f"json_sibling: {sibling_stem}.json")
  685. body = "\n".join(fields)
  686. return f"<!-- llmwiki:metadata\n{body}\n-->\n"
  687. def nav_bar(active: str, link_prefix: str = "") -> str:
  688. def link(href: str, label: str, key: str) -> str:
  689. cls = ' class="active"' if key == active else ""
  690. return f'<a href="{link_prefix}{href}"{cls}>{label}</a>'
  691. # #460: hamburger pattern for tablet/mobile (≤1023px). The desktop
  692. # nav-links row is hidden below 1024 (CSS rule), so without this
  693. # button the Graph / Docs / Changelog entries would be unreachable
  694. # on mobile (the bottom nav only carries Home / Projects / Sessions).
  695. # The drawer below mirrors the same 6 links vertically. JS in
  696. # render/js.py wires aria-expanded, ESC-to-close, and focus return.
  697. drawer_link = lambda href, label, key: (
  698. f' <a href="{link_prefix}{href}" class="nav-drawer-link'
  699. + (' active' if key == active else '') + '">'
  700. + label + '</a>'
  701. )
  702. # Post-review: dropped `role="menu"` + `aria-labelledby` — children
  703. # are plain <a>, not role="menuitem", so screen readers were being
  704. # told "press arrow keys" which did nothing. The drawer is a
  705. # disclosure nav, not an ARIA menu. The hamburger's aria-controls
  706. # already provides the trigger→drawer association; no role needed
  707. # on the container.
  708. nav_drawer_html = f"""<div id="nav-drawer" class="nav-drawer" hidden aria-label="Main navigation">
  709. {drawer_link("index.html", "Home", "home")}
  710. {drawer_link("projects/index.html", "Projects", "projects")}
  711. {drawer_link("sessions/index.html", "Sessions", "sessions")}
  712. {drawer_link("graph.html", "Graph", "graph")}
  713. {drawer_link("docs/index.html", "Docs", "docs")}
  714. {drawer_link("changelog.html", "Changelog", "changelog")}
  715. </div>"""
  716. return f"""<header class="nav">
  717. <div class="nav-inner">
  718. <a href="{link_prefix}index.html" class="nav-brand">
  719. <svg aria-hidden="true" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 2L2 7l10 5 10-5-10-5z"/><path d="M2 17l10 5 10-5"/><path d="M2 12l10 5 10-5"/></svg>
  720. LLM Wiki
  721. </a>
  722. <button type="button" class="nav-hamburger" id="nav-hamburger"
  723. aria-expanded="false" aria-controls="nav-drawer"
  724. aria-label="Open navigation menu">
  725. <svg aria-hidden="true" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><line x1="3" y1="6" x2="21" y2="6"/><line x1="3" y1="12" x2="21" y2="12"/><line x1="3" y1="18" x2="21" y2="18"/></svg>
  726. </button>
  727. <nav class="nav-links">
  728. {link("index.html", "Home", "home")}
  729. {link("projects/index.html", "Projects", "projects")}
  730. {link("sessions/index.html", "Sessions", "sessions")}
  731. {link("graph.html", "Graph", "graph")}
  732. {link("docs/index.html", "Docs", "docs")}
  733. {link("changelog.html", "Changelog", "changelog")}
  734. <button class="nav-search-btn" id="open-palette"
  735. aria-label="Open command palette"
  736. aria-haspopup="dialog" aria-expanded="false" aria-controls="palette">
  737. <svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><circle cx="11" cy="11" r="8"/><line x1="21" y1="21" x2="16.65" y2="16.65"/></svg>
  738. <span>Search</span>
  739. <kbd>⌘K</kbd>
  740. </button>
  741. <button class="theme-toggle" id="theme-toggle"
  742. aria-label="Toggle dark mode" aria-pressed="false">
  743. <svg aria-hidden="true" class="icon-moon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/></svg>
  744. <svg aria-hidden="true" class="icon-sun" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="5"/><line x1="12" y1="1" x2="12" y2="3"/><line x1="12" y1="21" x2="12" y2="23"/><line x1="4.22" y1="4.22" x2="5.64" y2="5.64"/><line x1="18.36" y1="18.36" x2="19.78" y2="19.78"/><line x1="1" y1="12" x2="3" y2="12"/><line x1="21" y1="12" x2="23" y2="12"/><line x1="4.22" y1="19.78" x2="5.64" y2="18.36"/><line x1="18.36" y1="5.64" x2="19.78" y2="4.22"/></svg>
  745. </button>
  746. </nav>
  747. </div>
  748. {nav_drawer_html}
  749. </header>
  750. """
  751. def breadcrumbs_bar(crumbs: list[tuple[str, str]], link_prefix: str = "") -> str:
  752. if not crumbs:
  753. return ""
  754. parts = []
  755. for i, (label, href) in enumerate(crumbs):
  756. if href:
  757. parts.append(f'<a href="{link_prefix}{html.escape(href)}">{html.escape(label)}</a>')
  758. else:
  759. parts.append(f'<span aria-current="page">{html.escape(label)}</span>')
  760. sep = ' <span class="crumb-sep">›</span> '
  761. return f'<nav class="breadcrumbs" aria-label="Breadcrumb">{sep.join(parts)}</nav>'
  762. def hero(title: str, subtitle: str, size: str = "", subtitle_is_html: bool = False) -> str:
  763. cls = f"hero {size}".strip()
  764. sub = subtitle if subtitle_is_html else html.escape(subtitle)
  765. return f"""<main id="main-content">
  766. <section class="{cls}">
  767. <div class="container">
  768. <h1>{html.escape(title)}</h1>
  769. <p class="hero-sub">{sub}</p>
  770. </div>
  771. </section>
  772. """
  773. def page_foot(js_prefix: str = "") -> str:
  774. return f"""<footer class="footer">
  775. <div class="container">
  776. <p class="muted">llmwiki · <a href="{js_prefix}index.html">home</a> · press <kbd>?</kbd> for shortcuts</p>
  777. </div>
  778. </footer>
  779. <nav class="mobile-bottom-nav" aria-label="Mobile navigation">
  780. <a href="{js_prefix}index.html" class="mbn-link" data-page="home">
  781. <svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round"><path d="M3 9l9-7 9 7v11a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2z"/><polyline points="9 22 9 12 15 12 15 22"/></svg>
  782. <span>Home</span>
  783. </a>
  784. <a href="{js_prefix}projects/index.html" class="mbn-link" data-page="projects">
  785. <svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round"><path d="M22 19a2 2 0 0 1-2 2H4a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h5l2 3h9a2 2 0 0 1 2 2z"/></svg>
  786. <span>Projects</span>
  787. </a>
  788. <a href="{js_prefix}sessions/index.html" class="mbn-link" data-page="sessions">
  789. <svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round"><line x1="8" y1="6" x2="21" y2="6"/><line x1="8" y1="12" x2="21" y2="12"/><line x1="8" y1="18" x2="21" y2="18"/><line x1="3" y1="6" x2="3.01" y2="6"/><line x1="3" y1="12" x2="3.01" y2="12"/><line x1="3" y1="18" x2="3.01" y2="18"/></svg>
  790. <span>Sessions</span>
  791. </a>
  792. <button type="button" class="mbn-link" id="mbn-search" aria-label="Search">
  793. <svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round"><circle cx="11" cy="11" r="8"/><line x1="21" y1="21" x2="16.65" y2="16.65"/></svg>
  794. <span>Search</span>
  795. </button>
  796. <button type="button" class="mbn-link" id="mbn-theme" aria-label="Toggle theme" aria-pressed="false">
  797. <svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/></svg>
  798. <span>Theme</span>
  799. </button>
  800. </nav>
  801. <div id="palette" class="palette">
  802. <div class="palette-backdrop" id="palette-backdrop"></div>
  803. <div class="palette-modal" role="dialog" aria-modal="true" aria-label="Command palette">
  804. <div class="palette-header">
  805. <svg aria-hidden="true" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="11" cy="11" r="8"/><line x1="21" y1="21" x2="16.65" y2="16.65"/></svg>
  806. <input type="text" id="palette-input" aria-label="Search pages" placeholder="Search… or type:session project:llm-wiki date:>2026-03 sort:date" autocomplete="off" spellcheck="false">
  807. <kbd>ESC</kbd>
  808. </div>
  809. <ul class="palette-results" id="palette-results"></ul>
  810. <div class="palette-footer muted">
  811. <span><kbd>↑↓</kbd> navigate</span>
  812. <span><kbd>↵</kbd> open</span>
  813. <span><kbd>ESC</kbd> close</span>
  814. </div>
  815. </div>
  816. </div>
  817. <div id="help-dialog" class="help-dialog">
  818. <div class="palette-backdrop" id="help-backdrop"></div>
  819. <div class="help-modal">
  820. <h2>Keyboard shortcuts</h2>
  821. <table>
  822. <tr><td><kbd>⌘K</kbd> / <kbd>Ctrl+K</kbd></td><td>Open command palette</td></tr>
  823. <tr><td><kbd>/</kbd></td><td>Focus search</td></tr>
  824. <tr><td><kbd>g h</kbd></td><td>Go to home</td></tr>
  825. <tr><td><kbd>g p</kbd></td><td>Go to projects</td></tr>
  826. <tr><td><kbd>g s</kbd></td><td>Go to sessions</td></tr>
  827. <tr><td><kbd>j</kbd> / <kbd>k</kbd></td><td>Next / prev row (tables)</td></tr>
  828. <tr><td><kbd>?</kbd></td><td>Show this help</td></tr>
  829. <tr><td><kbd>Esc</kbd></td><td>Close dialogs</td></tr>
  830. </table>
  831. <h3>Structured queries</h3>
  832. <p class="muted help-dialog-hint">Mix key:value filters with free text in the palette:</p>
  833. <table>
  834. <tr><td><code>type:session</code></td><td>Only session pages</td></tr>
  835. <tr><td><code>project:llm-wiki</code></td><td>Filter by project name (substring)</td></tr>
  836. <tr><td><code>model:claude</code></td><td>Filter by model name (substring)</td></tr>
  837. <tr><td><code>date:&gt;2026-03-01</code></td><td>Sessions after a date</td></tr>
  838. <tr><td><code>date:&lt;2026-04-01</code></td><td>Sessions before a date</td></tr>
  839. <tr><td><code>tags:rust</code></td><td>Pages mentioning a tag/topic</td></tr>
  840. <tr><td><code>sort:date</code></td><td>Sort results by date (newest first)</td></tr>
  841. </table>
  842. <p class="muted help-dialog-example">Example: <code>type:session project:llm-wiki date:&gt;2026-04 sort:date</code></p>
  843. <button class="btn" id="help-close">Close</button>
  844. </div>
  845. </div>
  846. <script src="{js_prefix}search-index.json" type="application/json" id="search-index-hint"></script>
  847. <script>window.LLMWIKI_INDEX_URL = "{js_prefix}search-index.json";</script>
  848. <script src="{HLJS_SCRIPT}" defer></script>
  849. <script>
  850. // v0.5: Run highlight.js once the CDN script lands. Defer keeps it out of
  851. // the critical path; the DOMContentLoaded fallback covers the case where
  852. // hljs arrives before/after DOM ready depending on cache state.
  853. function __llmwikiHljsInit() {{
  854. if (window.hljs) {{ window.hljs.highlightAll(); }}
  855. else {{ window.addEventListener('load', function() {{ if (window.hljs) window.hljs.highlightAll(); }}); }}
  856. }}
  857. if (document.readyState === 'loading') {{
  858. document.addEventListener('DOMContentLoaded', __llmwikiHljsInit);
  859. }} else {{
  860. __llmwikiHljsInit();
  861. }}
  862. </script>
  863. <script src="{js_prefix}script.js"></script>
  864. </body>
  865. </html>
  866. """
  867. # ─── page renderers ────────────────────────────────────────────────────────
  868. def _pluralize(n: int, singular: str, plural: str | None = None) -> str:
  869. """Return ``"1 session"`` for n=1, ``"3 sessions"`` for n=3.
  870. Closes #387 U7. The hero subtitle and any other count-bearing
  871. user-facing string should never read as ``"1 sessions"``."""
  872. if plural is None:
  873. plural = singular + "s"
  874. return f"{n} {singular if n == 1 else plural}"
  875. def calc_reading_time(body: str, wpm: int = 225) -> int:
  876. """Estimate reading time in minutes from a markdown body."""
  877. words = len(re.findall(r"\w+", body))
  878. return max(1, round(words / wpm))
  879. def render_session(
  880. path: Path,
  881. meta: dict[str, Any],
  882. body: str,
  883. out_dir: Path,
  884. project_slug: str,
  885. ) -> Path:
  886. slug = meta.get("slug", path.stem)
  887. date = meta.get("date", "")
  888. title_raw = meta.get("title", f"Session: {slug}")
  889. body = strip_leading_h1(body)
  890. body_html = md_to_html(body)
  891. # #270: session transcripts often reference files the user had open
  892. # during the session (tasks.md, CLAUDE.md, convert.py, etc). Route
  893. # the ones that look like repo source code or root files to GitHub
  894. # so the links don't dead-end after ingest.
  895. from llmwiki.docs_pages import (
  896. rewrite_md_links_to_html,
  897. rewrite_source_code_links_to_github,
  898. strip_dead_session_refs,
  899. )
  900. body_html = rewrite_source_code_links_to_github(body_html)
  901. # #284: now that README.md and CONTRIBUTING.md compile to
  902. # site/README.html / site/CONTRIBUTING.html, session bodies that
  903. # reference those files should route to the compiled pages.
  904. # Generic .md → .html pass runs AFTER the GitHub rewrite so
  905. # source-code and repo-root-only refs (CLAUDE.md, AGENTS.md) still
  906. # go to GitHub.
  907. body_html = rewrite_md_links_to_html(body_html)
  908. # #336: for remaining session-local refs (tasks.md, user_profile.md,
  909. # wiki/sources/<proj>/... wikilinks), drop the anchor but keep the
  910. # text — they point at files unique to the user's project and don't
  911. # compile to anywhere on the site.
  912. body_html = strip_dead_session_refs(body_html)
  913. raw_md_for_copy = html.escape(body)
  914. reading_min = calc_reading_time(body)
  915. bits: list[str] = []
  916. if meta.get("project"):
  917. bits.append(
  918. f'<a href="../../projects/{html.escape(str(meta["project"]))}.html">{html.escape(str(meta["project"]))}</a>'
  919. )
  920. # Agent badge — shows Claude / Codex / Copilot / Cursor / Gemini
  921. bits.append(render_agent_badge(meta))
  922. if meta.get("gitBranch"):
  923. bits.append(f'branch <code>{html.escape(str(meta["gitBranch"]))}</code>')
  924. if meta.get("model"):
  925. bits.append(f'<code>{html.escape(str(meta["model"]))}</code>')
  926. if meta.get("started"):
  927. bits.append(f'<span class="muted">{html.escape(short_started(meta))}</span>')
  928. if meta.get("user_messages"):
  929. bits.append(f'{html.escape(str(meta["user_messages"]))} msgs')
  930. if meta.get("tool_calls"):
  931. bits.append(f'{html.escape(str(meta["tool_calls"]))} tools')
  932. bits.append(f'<span class="muted">{reading_min} min read</span>')
  933. bits.append(render_freshness(meta))
  934. meta_strip = " · ".join(bits) if bits else ""
  935. tools_list = get_tools_list(meta)
  936. tools_preview = ""
  937. if tools_list:
  938. preview = ", ".join(tools_list[:6])
  939. if len(tools_list) > 6:
  940. preview += f", +{len(tools_list) - 6} more"
  941. tools_preview = f'<div class="meta-tools muted">tools: {html.escape(preview)}</div>'
  942. # v0.8 (#65): horizontal bar chart of tool calls in this session.
  943. # Uses the `tool_counts` JSON dict from frontmatter (#63). Empty
  944. # sessions (no recorded calls) render nothing.
  945. tool_chart_svg = render_session_tool_chart(meta)
  946. tool_chart_block = ""
  947. if tool_chart_svg:
  948. tool_chart_block = (
  949. '<div class="tool-chart-card">'
  950. '<div class="tool-chart-label muted">Tool calls</div>'
  951. f'{tool_chart_svg}'
  952. '</div>'
  953. )
  954. # v0.8 (#66): session token-usage card. Input / cache_creation /
  955. # cache_read / output plus a cache-hit-ratio tier badge. Sessions
  956. # missing token_totals (older converter output) render nothing.
  957. token_card_block = render_session_token_card(meta)
  958. # IMPORTANT: The HTML file is named `<path.stem>.html` (e.g. date-slug),
  959. # NOT `<slug>.html`. The siblings + canonical must use path.stem.
  960. html_stem = path.stem
  961. raw_md_path = f"../../sources/{project_slug}/{path.name}"
  962. actions_html = f"""<div class="session-actions">
  963. <button class="btn btn-primary" type="button" aria-label="Copy session content as markdown" title="Copy as markdown" onclick="copyMarkdown(this)">Copy as markdown</button>
  964. <a class="btn" href="../../projects/{html.escape(project_slug)}.html">← {html.escape(project_slug)}</a>
  965. <a class="btn" href="{html.escape(raw_md_path)}" download>Download .md</a>
  966. <a class="btn" href="{html.escape(html_stem + '.txt')}" title="plain-text sibling for AI agents">.txt</a>
  967. <a class="btn" href="{html.escape(html_stem + '.json')}" title="structured JSON sibling for AI agents">.json</a>
  968. <textarea class="md-source" hidden>{raw_md_for_copy}</textarea>
  969. </div>"""
  970. crumbs = [
  971. ("Home", "index.html"),
  972. ("Projects", "projects/index.html"),
  973. (project_slug, f"projects/{project_slug}.html"),
  974. (str(slug), ""),
  975. ]
  976. breadcrumbs = breadcrumbs_bar(crumbs, link_prefix="../../")
  977. # v0.4: machine-readable metadata appendix (HTML comment that AI agents
  978. # scraping HTML can parse without fetching the .json sibling).
  979. metadata_comment = _build_metadata_comment(meta, slug, project_slug, reading_min, html_stem=html_stem)
  980. # v0.4: page_head with schema.org article microdata + canonical link.
  981. # Canonical is relative to the current page (same dir) so the link checker
  982. # resolves it correctly whether served from a subdomain root or any path.
  983. page = (
  984. page_head_article(
  985. title=f"{title_raw} — LLM Wiki",
  986. description=f"Session transcript from {meta.get('project', '')} on {date}",
  987. css_prefix="../../",
  988. canonical=f"{html_stem}.html",
  989. date=str(meta.get("started") or date),
  990. metadata_comment=metadata_comment,
  991. )
  992. + nav_bar("sessions", link_prefix="../../")
  993. + hero(str(title_raw), meta_strip, size="hero-sm", subtitle_is_html=True)
  994. # #471: human-readable description rendered as a subtitle below
  995. # the hero, before the meta-strip. Only emit if frontmatter
  996. # carries the field; older sessions skip this block cleanly.
  997. + (
  998. f'<div class="container session-description"><p>{html.escape(str(meta["description"]))}</p></div>'
  999. if meta.get("description") else ""
  1000. )
  1001. + f'<section class="section">\n <div class="container">\n{breadcrumbs}\n{tools_preview}\n{actions_html}\n{tool_chart_block}\n{token_card_block}\n <article class="content" itemscope itemtype="https://schema.org/Article">\n'
  1002. + f'<meta itemprop="headline" content="{html.escape(str(title_raw))}">\n'
  1003. + f'<meta itemprop="datePublished" content="{html.escape(str(meta.get("started") or date))}">\n'
  1004. + f'<meta itemprop="inLanguage" content="en">\n'
  1005. + body_html
  1006. + '\n </article>\n </div>\n</section>\n</main>\n'
  1007. + page_foot(js_prefix="../../")
  1008. )
  1009. out_path = out_dir / "sessions" / project_slug / f"{path.stem}.html"
  1010. out_path.parent.mkdir(parents=True, exist_ok=True)
  1011. out_path.write_text(page, encoding="utf-8")
  1012. return out_path
  1013. def render_project_page(
  1014. project_slug: str,
  1015. sessions: list[tuple[Path, dict[str, Any], str]],
  1016. out_dir: Path,
  1017. ) -> Path:
  1018. main_sessions = [s for s in sessions if not _is_subagent(s[1], s[0])]
  1019. subagent_sessions = [s for s in sessions if s not in main_sessions]
  1020. def card(p: Path, meta: dict[str, Any]) -> str:
  1021. slug = meta.get("slug", p.stem)
  1022. title = meta.get("title", slug)
  1023. # Strip "Session: project/" prefix for cleaner display
  1024. if title.startswith("Session: "):
  1025. title = title[9:]
  1026. date = meta.get("date", "")
  1027. model = meta.get("model", "")
  1028. umsgs = meta.get("user_messages", "")
  1029. tcalls = meta.get("tool_calls", "")
  1030. href = f"../sessions/{project_slug}/{p.stem}.html"
  1031. badge = render_freshness(meta)
  1032. return f""" <a class="card" href="{href}">
  1033. <div class="card-title">{html.escape(str(title))}</div>
  1034. <div class="card-meta">{html.escape(str(date))} · {html.escape(str(model))}</div>
  1035. <div class="card-stats muted">{html.escape(str(umsgs))} messages · {html.escape(str(tcalls))} tool calls</div>
  1036. <div class="card-badge">{badge}</div>
  1037. </a>"""
  1038. cards_main = "\n".join(card(p, m) for p, m, _ in main_sessions)
  1039. cards_sub = "\n".join(card(p, m) for p, m, _ in subagent_sessions)
  1040. sub_section = ""
  1041. if subagent_sessions:
  1042. sub_section = (
  1043. '<details class="sub-section"><summary>Sub-agent runs ('
  1044. + str(len(subagent_sessions))
  1045. + ")</summary>\n<div class=\"card-grid\">\n"
  1046. + cards_sub
  1047. + "\n</div>\n</details>\n"
  1048. )
  1049. crumbs = breadcrumbs_bar(
  1050. [("Home", "index.html"), ("Projects", "projects/index.html"), (project_slug, "")],
  1051. link_prefix="../",
  1052. )
  1053. # v0.8 (#64, #72): per-project 365-day heatmap — same window as the
  1054. # aggregate on the home page, but filtered to just this project's
  1055. # sessions. Sparse projects (only a handful of sessions) still render
  1056. # the full grid with the rest as level-0 cells, so the shape matches.
  1057. proj_entries = [m for _, m, _ in sessions]
  1058. proj_counts = collect_session_counts(proj_entries, project_slug=project_slug)
  1059. proj_heatmap = render_heatmap(proj_counts, title_prefix=f"{project_slug} activity")
  1060. heatmap_block = f"""<section class="section heatmap-section">
  1061. <div class="container">
  1062. <div class="activity-heatmap">
  1063. <div class="heatmap-label muted">Activity · last 365 days · {html.escape(project_slug)}</div>
  1064. {proj_heatmap}
  1065. </div>
  1066. </div>
  1067. </section>"""
  1068. # v0.8 (#65): aggregate tool-call bar chart across all sessions in
  1069. # this project. Projects with no recorded tool calls render nothing.
  1070. proj_tool_chart = render_project_tool_chart(proj_entries, project_slug)
  1071. tool_chart_block = ""
  1072. if proj_tool_chart:
  1073. tool_chart_block = f"""<section class="section tool-chart-section">
  1074. <div class="container">
  1075. <div class="tool-chart-card">
  1076. <div class="tool-chart-label muted">Tool calls · {html.escape(project_slug)} aggregate</div>
  1077. {proj_tool_chart}
  1078. </div>
  1079. </div>
  1080. </section>"""
  1081. # v0.8 (#66): project token timeline card (log-scale area chart of
  1082. # total tokens per session date + aggregate cache hit ratio in the
  1083. # header). Empty for projects without any token data.
  1084. proj_token_card_html = render_project_token_card(proj_entries, project_slug)
  1085. token_timeline_block = ""
  1086. if proj_token_card_html:
  1087. token_timeline_block = f"""<section class="section token-timeline-section">
  1088. <div class="container">
  1089. {proj_token_card_html}
  1090. </div>
  1091. </section>"""
  1092. # Project topics strip — renders below the hero, above the heatmap.
  1093. # Explicit profile via wiki/projects/<slug>.md wins over the
  1094. # session-tag fallback. Projects with no topics render an empty
  1095. # strip (no chip row at all).
  1096. proj_profile = load_project_profile(PROJECTS_META_DIR, project_slug)
  1097. proj_topics = get_project_topics(PROJECTS_META_DIR, project_slug, proj_entries)
  1098. topics_html = render_topic_chips(
  1099. proj_topics, max_visible=12, classname="project-topics project-hero-topics"
  1100. )
  1101. description_html = ""
  1102. if proj_profile and proj_profile.get("description"):
  1103. description_html = (
  1104. f'<p class="project-description muted">'
  1105. f'{html.escape(proj_profile["description"])}</p>'
  1106. )
  1107. homepage_html = ""
  1108. if proj_profile and proj_profile.get("homepage"):
  1109. hp = proj_profile["homepage"]
  1110. homepage_html = (
  1111. f'<a class="project-homepage" href="{html.escape(hp)}" '
  1112. f'rel="noopener">{html.escape(hp)} ↗</a>'
  1113. )
  1114. topics_strip = ""
  1115. if topics_html or description_html or homepage_html:
  1116. topics_strip = (
  1117. '<section class="section project-topics-section">\n'
  1118. ' <div class="container">\n'
  1119. f' {description_html}\n'
  1120. f' {topics_html}\n'
  1121. f' {homepage_html}\n'
  1122. ' </div>\n'
  1123. '</section>\n'
  1124. )
  1125. body = f"""{topics_strip}
  1126. {heatmap_block}
  1127. {tool_chart_block}
  1128. {token_timeline_block}
  1129. <section class="section">
  1130. <div class="container">
  1131. {crumbs}
  1132. <h2>Main sessions ({len(main_sessions)})</h2>
  1133. <div class="card-grid">
  1134. {cards_main}
  1135. </div>
  1136. {sub_section}
  1137. </div>
  1138. </section>
  1139. </main>
  1140. """
  1141. page = (
  1142. page_head(
  1143. f"{project_slug} — LLM Wiki",
  1144. f"{len(sessions)} Claude Code sessions from {project_slug}",
  1145. css_prefix="../",
  1146. )
  1147. + nav_bar("projects", link_prefix="../")
  1148. + hero(
  1149. project_slug,
  1150. f"{len(main_sessions)} main sessions · {len(subagent_sessions)} sub-agent runs",
  1151. )
  1152. + body
  1153. + page_foot(js_prefix="../")
  1154. )
  1155. out_path = out_dir / "projects" / f"{project_slug}.html"
  1156. out_path.parent.mkdir(parents=True, exist_ok=True)
  1157. out_path.write_text(page, encoding="utf-8")
  1158. return out_path
  1159. def render_projects_index(
  1160. groups: dict[str, list[tuple[Path, dict[str, Any], str]]],
  1161. out_dir: Path,
  1162. ) -> Path:
  1163. cards = []
  1164. for project, sessions in sorted(groups.items(), key=lambda x: -len(x[1])):
  1165. main_count = sum(1 for p, m, _ in sessions if not _is_subagent(m, p))
  1166. sub_count = len(sessions) - main_count
  1167. # Freshness reflects the newest session in the project.
  1168. newest_meta = max(
  1169. (m for _, m, _ in sessions),
  1170. key=lambda m: str(m.get("ended") or m.get("started") or m.get("date") or ""),
  1171. default={},
  1172. )
  1173. badge = render_freshness(newest_meta)
  1174. cards.append(
  1175. f""" <a class="card" href="{html.escape(project)}.html">
  1176. <div class="card-title">{html.escape(project)}</div>
  1177. <div class="card-meta">{main_count} main · {sub_count} sub-agent</div>
  1178. <div class="card-badge">{badge}</div>
  1179. </a>"""
  1180. )
  1181. crumbs = breadcrumbs_bar(
  1182. [("Home", "index.html"), ("Projects", "")], link_prefix="../"
  1183. )
  1184. body = f"""<section class="section">
  1185. <div class="container">
  1186. {crumbs}
  1187. <div class="card-grid">
  1188. {chr(10).join(cards)}
  1189. </div>
  1190. </div>
  1191. </section>
  1192. </main>
  1193. """
  1194. page = (
  1195. page_head("Projects — LLM Wiki", "All projects with Claude Code session history", css_prefix="../")
  1196. + nav_bar("projects", link_prefix="../")
  1197. + hero("Projects", _pluralize(len(groups), "project"))
  1198. + body
  1199. + page_foot(js_prefix="../")
  1200. )
  1201. out_path = out_dir / "projects" / "index.html"
  1202. out_path.parent.mkdir(parents=True, exist_ok=True)
  1203. out_path.write_text(page, encoding="utf-8")
  1204. return out_path
  1205. def render_sessions_index(
  1206. sources: list[tuple[Path, dict[str, Any], str]],
  1207. groups: dict[str, list[tuple[Path, dict[str, Any], str]]],
  1208. out_dir: Path,
  1209. ) -> Path:
  1210. rows = []
  1211. def key(t: tuple[Path, dict[str, Any], str]) -> str:
  1212. return str(t[1].get("started", "")) or str(t[0].name)
  1213. for p, meta, _ in sorted(sources, key=key, reverse=True):
  1214. project = meta.get("project", p.parent.name)
  1215. slug = meta.get("slug", p.stem)
  1216. title = meta.get("title", slug)
  1217. # Strip "Session: " prefix for cleaner display
  1218. if title.startswith("Session: "):
  1219. title = title[9:]
  1220. # #452: titles auto-generated from session frontmatter follow the
  1221. # pattern "<slug> — <date>" but the table already has a dedicated
  1222. # Date column, so the trailing date is redundant. Strip it so the
  1223. # Session cell shows just the slug (or whatever custom title the
  1224. # user gave) and the Date column carries the date alone.
  1225. date = meta.get("date", "")
  1226. if date and title.endswith(f" — {date}"):
  1227. title = title[: -(len(date) + 3)]
  1228. # Truncate long titles for table display
  1229. display_title = title[:70] + "..." if len(title) > 70 else title
  1230. # #471: human-readable description from the first user turn —
  1231. # if frontmatter carries one, render it as a small muted line
  1232. # below the slug. Falls back to no second line for older
  1233. # sessions without the field.
  1234. description = str(meta.get("description") or "").strip()
  1235. desc_line = (
  1236. f'<div class="session-cell-desc muted">{html.escape(description)}</div>'
  1237. if description else ""
  1238. )
  1239. model = meta.get("model", "")
  1240. umsgs = meta.get("user_messages", "")
  1241. tcalls = meta.get("tool_calls", "")
  1242. href = f"{project}/{p.stem}.html"
  1243. rows.append(
  1244. f""" <tr data-project="{html.escape(str(project))}" data-model="{html.escape(str(model))}" data-date="{html.escape(str(date))}" data-slug="{html.escape(str(slug))}">
  1245. <td><a href="{html.escape(str(href))}">{html.escape(str(display_title))}</a>{desc_line}</td>
  1246. <td>{render_agent_badge(meta)}</td>
  1247. <td><a href="../projects/{html.escape(str(project))}.html">{html.escape(str(project))}</a></td>
  1248. <td>{html.escape(str(date))}</td>
  1249. <td><code>{html.escape(str(model))}</code></td>
  1250. <td class="num">{html.escape(str(umsgs))}</td>
  1251. <td class="num">{html.escape(str(tcalls))}</td>
  1252. </tr>"""
  1253. )
  1254. project_options = "\n".join(
  1255. f' <option value="{html.escape(p)}">{html.escape(p)}</option>'
  1256. for p in sorted(groups.keys())
  1257. )
  1258. models = sorted(
  1259. {str(m.get("model", "")) for _, m, _ in sources if m.get("model")}
  1260. )
  1261. model_options = "\n".join(
  1262. f' <option value="{html.escape(m)}">{html.escape(m)}</option>'
  1263. for m in models
  1264. )
  1265. crumbs = breadcrumbs_bar(
  1266. [("Home", "index.html"), ("Sessions", "")], link_prefix="../"
  1267. )
  1268. body = f"""<section class="section">
  1269. <div class="container">
  1270. {crumbs}
  1271. <div class="filter-bar">
  1272. <label>Project
  1273. <select id="filter-project">
  1274. <option value="">All projects</option>
  1275. {project_options}
  1276. </select>
  1277. </label>
  1278. <label>Model
  1279. <select id="filter-model">
  1280. <option value="">All models</option>
  1281. {model_options}
  1282. </select>
  1283. </label>
  1284. <label>From
  1285. <input type="date" id="filter-date-from">
  1286. </label>
  1287. <label>To
  1288. <input type="date" id="filter-date-to">
  1289. </label>
  1290. <label>Slug
  1291. <input type="text" id="filter-text" placeholder="part of slug…">
  1292. </label>
  1293. <button class="btn" id="filter-clear">Clear</button>
  1294. <span class="filter-count muted" id="filter-count"></span>
  1295. </div>
  1296. <div class="table-wrap">
  1297. <table class="sessions-table">
  1298. <colgroup>
  1299. <col style="width: 30%">
  1300. <col style="width: 8%">
  1301. <col style="width: 16%">
  1302. <col style="width: 10%">
  1303. <col style="width: 22%">
  1304. <col style="width: 7%">
  1305. <col style="width: 7%">
  1306. </colgroup>
  1307. <thead>
  1308. <tr><th>Session</th><th>Agent</th><th>Project</th><th>Date</th><th>Model</th><th>Msgs</th><th>Tools</th></tr>
  1309. </thead>
  1310. <tbody id="sessions-tbody">
  1311. {chr(10).join(rows)}
  1312. </tbody>
  1313. </table>
  1314. </div>
  1315. </div>
  1316. </section>
  1317. </main>
  1318. """
  1319. page = (
  1320. page_head("Sessions — LLM Wiki", "All Claude Code sessions, newest first", css_prefix="../")
  1321. + nav_bar("sessions", link_prefix="../")
  1322. + hero("All sessions", _pluralize(len(sources), "session") + " total")
  1323. + body
  1324. + page_foot(js_prefix="../")
  1325. )
  1326. out_path = out_dir / "sessions" / "index.html"
  1327. out_path.parent.mkdir(parents=True, exist_ok=True)
  1328. out_path.write_text(page, encoding="utf-8")
  1329. return out_path
  1330. def render_index(
  1331. groups: dict[str, list[tuple[Path, dict[str, Any], str]]],
  1332. all_sources: list[tuple[Path, dict[str, Any], str]],
  1333. out_dir: Path,
  1334. synthesis: Optional[str] = None,
  1335. ) -> Path:
  1336. total = len(all_sources)
  1337. mains = sum(1 for p, m, _ in all_sources if not _is_subagent(m, p))
  1338. subs = total - mains
  1339. synth_block = ""
  1340. if synthesis:
  1341. synth_block = f"""<section class="section">
  1342. <div class="container">
  1343. <div class="synthesis">
  1344. <h2>Overview</h2>
  1345. {md_to_html(synthesis)}
  1346. </div>
  1347. </div>
  1348. </section>"""
  1349. # v0.8 (#64, #72): aggregate 365-day GitHub-style heatmap. Counts all
  1350. # main sessions across all projects. Rendered as a pure-SVG block just
  1351. # above the projects grid so the landing page gives a glanceable
  1352. # "last year of activity" view.
  1353. heatmap_entries = [m for _, m, _ in all_sources]
  1354. heatmap_counts = collect_session_counts(heatmap_entries)
  1355. heatmap_svg = render_heatmap(heatmap_counts, title_prefix="Activity")
  1356. heatmap_block = f"""<section class="section heatmap-section">
  1357. <div class="container">
  1358. <div class="activity-heatmap">
  1359. <div class="heatmap-label muted">Activity · last 365 days</div>
  1360. {heatmap_svg}
  1361. </div>
  1362. </div>
  1363. </section>"""
  1364. # v0.8 (#66): site-wide token summary stats — four cards showing
  1365. # total tokens processed, average per session, best cache hit project,
  1366. # heaviest project. Empty if no session has token_totals data.
  1367. metas_by_project: dict[str, list[dict[str, Any]]] = {}
  1368. for project, sessions in groups.items():
  1369. metas_by_project[project] = [m for _, m, _ in sessions]
  1370. token_stats_block = render_site_token_stats(metas_by_project, link_prefix="")
  1371. # Recently updated — show last 10 entries from wiki/log.md.
  1372. log_events = _recent_log_events(
  1373. REPO_ROOT / "wiki" / "log.md", limit=10
  1374. )
  1375. recent_block_inner = render_recent_activity(log_events)
  1376. recent_block = (
  1377. f'<section class="section recently-updated-section">\n'
  1378. f' <div class="container">\n'
  1379. f' {recent_block_inner}\n'
  1380. f' </div>\n'
  1381. f'</section>\n'
  1382. ) if recent_block_inner else ""
  1383. cards = []
  1384. for project, sessions in sorted(groups.items(), key=lambda x: -len(x[1])):
  1385. main_count = sum(1 for p, m, _ in sessions if not _is_subagent(m, p))
  1386. # Project topics — explicit profile in wiki/projects/<slug>.md
  1387. # takes precedence, falls back to aggregated session tags with
  1388. # noise filtered out. Rendered as chips below the card meta.
  1389. proj_metas = [m for _, m, _ in sessions]
  1390. topics = get_project_topics(PROJECTS_META_DIR, project, proj_metas)
  1391. topics_html = render_topic_chips(topics, max_visible=4,
  1392. classname="project-topics card-topics")
  1393. # #455: render the activity date range under the meta line so
  1394. # users can spot fresh vs stale projects without clicking. Pull
  1395. # `date:` from frontmatter (already YYYY-MM-DD strings); ignore
  1396. # missing/blank values; format as `2026-03-12 → 2026-04-01` for
  1397. # multi-day, just `2026-04-01` if first == last.
  1398. dates = sorted(
  1399. {str(m.get("date", "")) for _, m, _ in sessions if m.get("date")}
  1400. )
  1401. if dates:
  1402. if dates[0] == dates[-1]:
  1403. date_range_html = (
  1404. f'<div class="card-date-range">{html.escape(dates[0])}</div>'
  1405. )
  1406. else:
  1407. date_range_html = (
  1408. f'<div class="card-date-range">'
  1409. f'{html.escape(dates[0])} → {html.escape(dates[-1])}'
  1410. f'</div>'
  1411. )
  1412. else:
  1413. date_range_html = ""
  1414. cards.append(
  1415. f""" <a class="card card-project" href="projects/{html.escape(project)}.html">
  1416. <div class="card-title">{html.escape(project)}</div>
  1417. <div class="card-meta">{main_count} main · {len(sessions) - main_count} sub-agent</div>
  1418. {date_range_html}
  1419. {topics_html}
  1420. </a>"""
  1421. )
  1422. body = f"""{heatmap_block}
  1423. {token_stats_block}
  1424. {recent_block}
  1425. <section class="section">
  1426. <div class="container">
  1427. <h2>Projects</h2>
  1428. <div class="card-grid">
  1429. {chr(10).join(cards)}
  1430. </div>
  1431. </div>
  1432. </section>
  1433. </main>
  1434. """
  1435. page = (
  1436. page_head("LLM Wiki", "Karpathy-style knowledge base from Claude Code sessions", css_prefix="")
  1437. + nav_bar("home", link_prefix="")
  1438. + hero(
  1439. "LLM Wiki",
  1440. f"{_pluralize(mains, 'main session')} · {_pluralize(subs, 'sub-agent run')} · {_pluralize(len(groups), 'project')}",
  1441. )
  1442. + synth_block
  1443. + body
  1444. + page_foot(js_prefix="")
  1445. )
  1446. out_path = out_dir / "index.html"
  1447. out_path.write_text(page, encoding="utf-8")
  1448. return out_path
  1449. def _render_root_md_page(
  1450. src_name: str,
  1451. out_name: str,
  1452. title: str,
  1453. subtitle: str,
  1454. meta_description: str,
  1455. out_dir: Path,
  1456. *,
  1457. active_nav: str = "docs",
  1458. ) -> Optional[Path]:
  1459. """Compile a repo-root ``.md`` file to a standalone site page (#284).
  1460. Used for ``README.md`` and ``CONTRIBUTING.md`` so visitors don't get
  1461. bounced out to GitHub for content we're already shipping as HTML.
  1462. """
  1463. src = REPO_ROOT / src_name
  1464. if not src.is_file():
  1465. return None
  1466. raw = src.read_text(encoding="utf-8")
  1467. body_md = raw
  1468. lines = raw.splitlines()
  1469. if lines and lines[0].lstrip().startswith("# "):
  1470. body_md = "\n".join(lines[1:]).lstrip("\n")
  1471. content_html = md_to_html(body_md)
  1472. # #270: route embedded source-code + repo-root links to GitHub, then
  1473. # the generic .md→.html pass for anything remaining. README has
  1474. # plenty of such links.
  1475. from llmwiki.docs_pages import (
  1476. rewrite_md_links_to_html,
  1477. rewrite_source_code_links_to_github,
  1478. )
  1479. content_html = rewrite_source_code_links_to_github(content_html)
  1480. content_html = rewrite_md_links_to_html(content_html)
  1481. body = f"""<section class="section docs-body">
  1482. <div class="container narrow">
  1483. <article class="article docs-article">
  1484. {content_html}
  1485. </article>
  1486. </div>
  1487. </section>
  1488. </main>
  1489. """
  1490. page = (
  1491. page_head(
  1492. f"{title} — LLM Wiki",
  1493. meta_description,
  1494. css_prefix="",
  1495. )
  1496. + nav_bar(active_nav, link_prefix="")
  1497. + hero(title, subtitle)
  1498. + body
  1499. + page_foot(js_prefix="")
  1500. )
  1501. out_path = out_dir / out_name
  1502. out_path.write_text(page, encoding="utf-8")
  1503. return out_path
  1504. def render_404(out_dir: Path) -> Path:
  1505. """Emit ``site/404.html`` with the standard site chrome and a "Page not
  1506. found" panel. Closes #387 U8 — without this, ``llmwiki serve`` falls
  1507. back to the stdlib ``http.server`` default 404 (an unstyled error string
  1508. with no nav). The page itself is not linked from the index, but
  1509. ``serve.py`` injects it as the body of every 404 response.
  1510. """
  1511. head = page_head(
  1512. title="Page not found · llmwiki",
  1513. description="The page you tried to open doesn't exist on this site.",
  1514. )
  1515. nav = nav_bar(active="")
  1516. foot = page_foot()
  1517. body = """<main id="main-content">
  1518. <section class="hero">
  1519. <div class="container">
  1520. <h1>Page not found</h1>
  1521. <p class="hero-sub">The page you tried to open doesn't exist on this site. The link may be stale, the page may have been removed, or the URL may have a typo.</p>
  1522. </div>
  1523. </section>
  1524. <section class="section">
  1525. <div class="container">
  1526. <p>Try one of these:</p>
  1527. <ul class="not-found-links">
  1528. <li><a href="index.html">Home</a> — overview and recent activity</li>
  1529. <li><a href="projects/index.html">Projects</a> — every project with sessions</li>
  1530. <li><a href="sessions/index.html">Sessions</a> — every session, sortable + filterable</li>
  1531. <li><a href="changelog.html">Changelog</a> — what's shipped recently</li>
  1532. </ul>
  1533. <p class="muted">Or press <kbd>⌘K</kbd> / <kbd>Ctrl+K</kbd> to open the command palette and search.</p>
  1534. </div>
  1535. </section>
  1536. </main>
  1537. """
  1538. page = head + nav + body + foot
  1539. out_path = out_dir / "404.html"
  1540. out_path.write_text(page, encoding="utf-8")
  1541. return out_path
  1542. def render_readme_page(out_dir: Path) -> Optional[Path]:
  1543. """Compile ``README.md`` to ``site/README.html`` (#284)."""
  1544. return _render_root_md_page(
  1545. "README.md", "README.html",
  1546. title="README",
  1547. subtitle="The public README of llmwiki, rendered from `README.md`.",
  1548. meta_description="llmwiki — Karpathy-style LLM wiki from your Claude Code, Codex CLI, Cursor, and Obsidian sessions.",
  1549. out_dir=out_dir,
  1550. )
  1551. def render_contributing_page(out_dir: Path) -> Optional[Path]:
  1552. """Compile ``CONTRIBUTING.md`` to ``site/CONTRIBUTING.html`` (#284)."""
  1553. return _render_root_md_page(
  1554. "CONTRIBUTING.md", "CONTRIBUTING.html",
  1555. title="Contributing",
  1556. subtitle="The 8 rules + review bar for contributing to llmwiki.",
  1557. meta_description="Contribution rules, PR checklist, and review bar for llmwiki.",
  1558. out_dir=out_dir,
  1559. )
  1560. # ─── changelog page ────────────────────────────────────────────────────────
  1561. def render_changelog(out_dir: Path) -> Optional[Path]:
  1562. """Render ``CHANGELOG.md`` (repo root) to ``site/changelog.html``.
  1563. Returns None if CHANGELOG.md is missing. Shown as its own top-level page
  1564. so visitors can see what's new / what shipped without clicking through to
  1565. GitHub. Keep-a-changelog headings become an in-page TOC via the existing
  1566. `toc` markdown extension.
  1567. """
  1568. src = REPO_ROOT / "CHANGELOG.md"
  1569. if not src.exists():
  1570. return None
  1571. raw = src.read_text(encoding="utf-8")
  1572. # Pull the top H1 ("Changelog") and use it as the hero title; render
  1573. # everything else as the body. Strip the leading H1 line to avoid a
  1574. # duplicate title.
  1575. body_md = raw
  1576. lines = raw.splitlines()
  1577. if lines and lines[0].lstrip().startswith("# "):
  1578. body_md = "\n".join(lines[1:]).lstrip("\n")
  1579. content_html = md_to_html(body_md)
  1580. body = f"""<section class="section changelog-body">
  1581. <div class="container narrow">
  1582. <article class="article">
  1583. {content_html}
  1584. </article>
  1585. </div>
  1586. </section>
  1587. </main>
  1588. """
  1589. page = (
  1590. page_head(
  1591. "Changelog — LLM Wiki",
  1592. "Release notes for llmwiki — features, fixes, and version history.",
  1593. css_prefix="",
  1594. )
  1595. + nav_bar("changelog", link_prefix="")
  1596. + hero(
  1597. "Changelog",
  1598. "Every release, every fix. Keep-a-changelog format, semver.",
  1599. )
  1600. + body
  1601. + page_foot(js_prefix="")
  1602. )
  1603. out_path = out_dir / "changelog.html"
  1604. out_path.write_text(page, encoding="utf-8")
  1605. return out_path
  1606. # ─── v0.7 (#55) models section ─────────────────────────────────────────────
  1607. def render_models_section(out_dir: Path) -> tuple[Optional[Path], int]:
  1608. """Discover `wiki/entities/*.md` pages with `entity_kind: ai-model`,
  1609. render one detail page per model + a sortable `/models/index.html`.
  1610. Returns `(index_path_or_None, model_count)`. If there's no
  1611. `wiki/entities/` directory OR no model pages there, we still write
  1612. an empty-state index so the nav link doesn't 404.
  1613. """
  1614. # Post-review: imports lazily so this function actually works the
  1615. # next time someone wires it from the CLI. Previously these names
  1616. # were referenced but never imported — function body was reachable
  1617. # but would crash with NameError on first call.
  1618. from llmwiki.models_page import ( # noqa: F401
  1619. discover_model_entities_with_meta,
  1620. render_models_index,
  1621. render_model_info_card,
  1622. )
  1623. entities_dir = REPO_ROOT / "wiki" / "entities"
  1624. entries_with_meta = discover_model_entities_with_meta(entities_dir)
  1625. # Backwards-compatible list without meta for render_models_index.
  1626. entries = [
  1627. (path, profile, warnings, body)
  1628. for path, _meta, profile, warnings, body in entries_with_meta
  1629. ]
  1630. models_out = out_dir / "models"
  1631. models_out.mkdir(parents=True, exist_ok=True)
  1632. # Index page — always write it so the nav link resolves.
  1633. index_body = render_models_index(entries)
  1634. index_page = (
  1635. page_head(
  1636. "Models — LLM Wiki",
  1637. "Directory of AI-model entities tracked by the wiki with pricing, "
  1638. "context windows, and benchmark scores.",
  1639. css_prefix="../",
  1640. )
  1641. + nav_bar("models", link_prefix="../")
  1642. + hero("Models", f"{len(entries)} model entities tracked")
  1643. + index_body
  1644. + "</main>\n"
  1645. + page_foot(js_prefix="../")
  1646. )
  1647. index_path = models_out / "index.html"
  1648. index_path.write_text(index_page, encoding="utf-8")
  1649. # Per-model detail page — info card + body markdown rendered normally.
  1650. for path, meta, profile, warnings, body in entries_with_meta:
  1651. slug = path.stem
  1652. title = profile.get("title", slug)
  1653. info_card = render_model_info_card(profile)
  1654. # v0.7 (#56): changelog timeline + pricing sparkline below the
  1655. # info card. The sparkline only shows if there are ≥2 dated
  1656. # input-price changes in the changelog.
  1657. changelog_entries, changelog_warnings = parse_changelog(meta)
  1658. warnings = list(warnings) + changelog_warnings
  1659. timeline_html = render_changelog_timeline(changelog_entries)
  1660. timeline_block = ""
  1661. if timeline_html:
  1662. price_pts = extract_price_points(
  1663. changelog_entries, field_suffix="pricing.input_per_1m"
  1664. )
  1665. sparkline = render_price_sparkline(price_pts)
  1666. sparkline_block = (
  1667. f'<div class="timeline-sparkline">'
  1668. f'<span class="muted">Input pricing trend</span> {sparkline}'
  1669. f'</div>'
  1670. if sparkline else ""
  1671. )
  1672. timeline_block = (
  1673. '<div class="timeline-card">'
  1674. '<div class="timeline-card-title">Changelog</div>'
  1675. + sparkline_block
  1676. + timeline_html
  1677. + '</div>'
  1678. )
  1679. body_html = md_to_html(body)
  1680. warnings_html = ""
  1681. if warnings:
  1682. items = "".join(f"<li>{html.escape(w)}</li>" for w in warnings)
  1683. warnings_html = (
  1684. '<details class="model-warnings"><summary>Schema warnings '
  1685. f'({len(warnings)})</summary><ul>{items}</ul></details>'
  1686. )
  1687. page = (
  1688. page_head(
  1689. f"{title} — LLM Wiki",
  1690. f"AI-model entity: {title}",
  1691. css_prefix="../",
  1692. )
  1693. + nav_bar("models", link_prefix="../")
  1694. + hero(title, profile.get("provider", ""))
  1695. + f'<section class="section">\n <div class="container narrow">\n'
  1696. + info_card
  1697. + timeline_block
  1698. + warnings_html
  1699. + f' <article class="article content">\n {body_html}\n </article>\n'
  1700. + ' </div>\n</section>\n</main>\n'
  1701. + page_foot(js_prefix="../")
  1702. )
  1703. (models_out / f"{slug}.html").write_text(page, encoding="utf-8")
  1704. return index_path, len(entries)
  1705. # ─── v0.7 (#58) auto-generated vs-comparison pages ────────────────────────
  1706. def render_vs_section(
  1707. out_dir: Path,
  1708. max_pairs: int = 500,
  1709. min_shared_fields: int = 3,
  1710. ) -> tuple[Optional[Path], int]:
  1711. """Generate `/vs/<slug_a>-vs-<slug_b>.html` for every pair of
  1712. comparable model entities + an index at `/vs/index.html`.
  1713. Honors user overrides under `wiki/vs/<slug>.md` — a hand-written
  1714. comparison replaces the auto-gen for that URL. Returns
  1715. `(index_path, pair_count)`. Always writes the index so the nav
  1716. link resolves even when no entities exist.
  1717. """
  1718. # Post-review: lazy imports so this function actually works the
  1719. # next time someone wires it. Previously these names were referenced
  1720. # but never imported — first call would have crashed with NameError.
  1721. from llmwiki.models_page import discover_model_entities # noqa: F401
  1722. from llmwiki.compare import ( # noqa: F401
  1723. discover_user_overrides,
  1724. generate_pairs,
  1725. pair_slug,
  1726. render_comparison_body,
  1727. render_comparisons_index,
  1728. )
  1729. entities_dir = REPO_ROOT / "wiki" / "entities"
  1730. overrides_dir = REPO_ROOT / "wiki" / "vs"
  1731. entries = discover_model_entities(entities_dir)
  1732. # Strip down to (path, profile) for compare.generate_pairs
  1733. pair_entries = [(p, profile) for p, profile, _w, _b in entries]
  1734. pairs = generate_pairs(
  1735. pair_entries,
  1736. min_shared_fields=min_shared_fields,
  1737. max_pairs=max_pairs,
  1738. )
  1739. vs_out = out_dir / "vs"
  1740. vs_out.mkdir(parents=True, exist_ok=True)
  1741. # Index
  1742. index_body = render_comparisons_index(pairs)
  1743. index_page = (
  1744. page_head(
  1745. "Model comparisons — LLM Wiki",
  1746. "Auto-generated side-by-side comparisons of AI-model entities.",
  1747. css_prefix="../",
  1748. )
  1749. + nav_bar("vs", link_prefix="../")
  1750. + hero("Model comparisons", f"{len(pairs)} auto-generated pairs")
  1751. + index_body
  1752. + "</main>\n"
  1753. + page_foot(js_prefix="../")
  1754. )
  1755. index_path = vs_out / "index.html"
  1756. index_path.write_text(index_page, encoding="utf-8")
  1757. # User overrides replace the auto-gen for matching slugs
  1758. overrides = discover_user_overrides(overrides_dir)
  1759. for pair in pairs:
  1760. slug = pair_slug(pair)
  1761. if slug in overrides:
  1762. # User override — render the raw body through md_to_html
  1763. body_html = md_to_html(overrides[slug])
  1764. article_body = (
  1765. '<section class="section"><div class="container narrow">'
  1766. f'<article class="article content">{body_html}</article>'
  1767. '</div></section>'
  1768. )
  1769. else:
  1770. # Auto-gen — three structured sections
  1771. comparison_body = render_comparison_body(pair)
  1772. article_body = (
  1773. '<section class="section"><div class="container narrow">'
  1774. f'{comparison_body}'
  1775. '</div></section>'
  1776. )
  1777. title = f"{pair['title_a']} vs {pair['title_b']}"
  1778. page = (
  1779. page_head(
  1780. f"{title} — LLM Wiki",
  1781. f"Side-by-side comparison of {title}.",
  1782. css_prefix="../",
  1783. )
  1784. + nav_bar("vs", link_prefix="../")
  1785. + hero(title, f"{pair['score']} shared structured fields")
  1786. + article_body
  1787. + "</main>\n"
  1788. + page_foot(js_prefix="../")
  1789. )
  1790. (vs_out / f"{slug}.html").write_text(page, encoding="utf-8")
  1791. return index_path, len(pairs)
  1792. # ─── search index ──────────────────────────────────────────────────────────
  1793. def build_search_index(
  1794. sources: list[tuple[Path, dict[str, Any], str]],
  1795. groups: dict[str, list[tuple[Path, dict[str, Any], str]]],
  1796. out_dir: Path,
  1797. *,
  1798. search_mode: str = "auto",
  1799. ) -> Path:
  1800. """Build a chunked search index for lazy loading (#47).
  1801. Writes:
  1802. search-index.json — meta entries (projects + pages) + _chunks manifest
  1803. + _mode + _tree_eligible_ratio (#53)
  1804. search-chunks/<project>.json — session entries per project, each
  1805. carrying heading_max_depth + count_by_depth
  1806. `search_mode` accepts ``auto`` (default, heuristic), ``tree``, or
  1807. ``flat`` — matches the `llmwiki build --search-mode` flag.
  1808. """
  1809. from llmwiki.search_tree import (
  1810. annotate_entry_headings,
  1811. decide_search_mode,
  1812. search_index_footer_badge,
  1813. )
  1814. # ── session entries grouped by project ──
  1815. from llmwiki.search_facets import enrich_entry
  1816. chunks: dict[str, list[dict[str, Any]]] = {}
  1817. for p, meta, body in sources:
  1818. project = str(meta.get("project") or p.parent.name)
  1819. slug = str(meta.get("slug", p.stem))
  1820. plain = md_to_plain_text(body)[:1200]
  1821. entry = {
  1822. "id": f"session:{project}/{p.stem}",
  1823. "url": f"sessions/{project}/{p.stem}.html",
  1824. "title": slug,
  1825. "type": "session",
  1826. "project": project,
  1827. "date": str(meta.get("date", "")),
  1828. "model": str(meta.get("model", "")),
  1829. "body": plain,
  1830. }
  1831. # v1.0 (#161): enrich with facet fields.
  1832. enrich_entry(entry, meta)
  1833. # v1.2 (#53): inject heading depth so the client can tree-walk.
  1834. annotate_entry_headings(entry, body)
  1835. chunks.setdefault(project, []).append(entry)
  1836. # ── write per-project chunks ──
  1837. chunks_dir = out_dir / "search-chunks"
  1838. chunks_dir.mkdir(exist_ok=True)
  1839. chunk_manifest: list[str] = []
  1840. total_chunk_bytes = 0
  1841. for project_slug, entries in sorted(chunks.items()):
  1842. chunk_path = chunks_dir / f"{project_slug}.json"
  1843. data = json.dumps(entries, ensure_ascii=False)
  1844. chunk_path.write_text(data, encoding="utf-8")
  1845. chunk_manifest.append(f"search-chunks/{project_slug}.json")
  1846. total_chunk_bytes += len(data.encode("utf-8"))
  1847. # ── meta index: projects + static pages + chunk manifest ──
  1848. meta_entries: list[dict[str, Any]] = []
  1849. for project, sessions in groups.items():
  1850. meta_entries.append(
  1851. {
  1852. "id": f"project:{project}",
  1853. "url": f"projects/{project}.html",
  1854. "title": project,
  1855. "type": "project",
  1856. "project": project,
  1857. "date": "",
  1858. "model": "",
  1859. "body": f"{len(sessions)} sessions",
  1860. }
  1861. )
  1862. meta_entries.append(
  1863. {"id": "home", "url": "index.html", "title": "Home", "type": "page",
  1864. "project": "", "date": "", "model": "", "body": "overview index"}
  1865. )
  1866. meta_entries.append(
  1867. {"id": "projects-index", "url": "projects/index.html", "title": "Projects",
  1868. "type": "page", "project": "", "date": "", "model": "", "body": "all projects"}
  1869. )
  1870. meta_entries.append(
  1871. {"id": "sessions-index", "url": "sessions/index.html", "title": "All sessions",
  1872. "type": "page", "project": "", "date": "", "model": "", "body": "sortable sessions table"}
  1873. )
  1874. # #277: index every docs/ page + every slash command so the palette
  1875. # becomes a universal quick-find (not just sessions + projects).
  1876. from llmwiki.docs_pages import iter_docs_pages, _first_paragraph
  1877. docs_dir = REPO_ROOT / "docs"
  1878. if docs_dir.is_dir():
  1879. for page in iter_docs_pages(docs_dir):
  1880. meta_entries.append({
  1881. "id": f"docs:{page.rel}",
  1882. "url": f"docs/{page.rel.replace('.md', '.html')}",
  1883. "title": page.title,
  1884. "type": "docs",
  1885. "project": "",
  1886. "date": "",
  1887. "model": "",
  1888. "body": _first_paragraph(page.body)[:300],
  1889. })
  1890. # Slash commands — read the first non-empty line of each .md as
  1891. # the description so the palette shows what each /wiki-* does.
  1892. slash_dir = REPO_ROOT / ".claude" / "commands"
  1893. if slash_dir.is_dir():
  1894. for p in sorted(slash_dir.glob("*.md")):
  1895. try:
  1896. text = p.read_text(encoding="utf-8")
  1897. except OSError:
  1898. continue
  1899. first_para = next(
  1900. (ln.strip() for ln in text.splitlines() if ln.strip()),
  1901. "",
  1902. )
  1903. meta_entries.append({
  1904. "id": f"slash:{p.stem}",
  1905. # Slashes aren't URLs — the palette shows a non-clickable
  1906. # entry with the command to type inside Claude Code.
  1907. "url": "",
  1908. "title": f"/{p.stem}",
  1909. "type": "slash",
  1910. "project": "",
  1911. "date": "",
  1912. "model": "",
  1913. "body": first_para[:300],
  1914. })
  1915. # v1.0 (#161): aggregate facet counts across all session chunks so the
  1916. # client can render filter checkboxes without scanning the full index.
  1917. from llmwiki.search_facets import aggregate_facets
  1918. all_entries: list[dict[str, Any]] = []
  1919. for chunk_entries in chunks.values():
  1920. all_entries.extend(chunk_entries)
  1921. facets = aggregate_facets(all_entries)
  1922. # v1.2 (#53): decide tree vs flat across *all* entries, surface
  1923. # the ratio so the client can show it in the palette footer.
  1924. mode, tree_ratio = decide_search_mode(all_entries, override=search_mode)
  1925. mode_badge = search_index_footer_badge(mode, tree_ratio)
  1926. index_obj = {
  1927. "entries": meta_entries,
  1928. "_chunks": chunk_manifest,
  1929. "_facets": facets,
  1930. "_mode": mode,
  1931. "_tree_eligible_ratio": round(tree_ratio, 4),
  1932. "_mode_badge": mode_badge,
  1933. }
  1934. out_path = out_dir / "search-index.json"
  1935. out_path.write_text(json.dumps(index_obj, ensure_ascii=False), encoding="utf-8")
  1936. meta_kb = len(json.dumps(index_obj).encode("utf-8")) // 1024
  1937. chunks_kb = total_chunk_bytes // 1024
  1938. print(
  1939. f" wrote search-index.json ({meta_kb} KB meta) + "
  1940. f"{len(chunk_manifest)} chunks ({chunks_kb} KB total) · {mode_badge}"
  1941. )
  1942. return out_path
  1943. # ─── css + js constants ────────────────────────────────────────────────────
  1944. # ─── css + js ────────────────────────────────────────────────────────────
  1945. # CSS + JS constants live in llmwiki/render/ since v1.1 (#217). Re-export
  1946. # here for backwards compatibility — any external caller still doing
  1947. # `from llmwiki.build import CSS` keeps working.
  1948. from llmwiki.render.css import CSS # noqa: F401 (re-exported)
  1949. from llmwiki.render.js import JS # noqa: F401 (re-exported)
  1950. # ─── claude synthesis (optional) ───────────────────────────────────────────
  1951. # #421: shell metacharacters that have no business in a path-to-an-
  1952. # executable. We refuse paths containing any of these rather than
  1953. # trying to escape them — the CLI argv is never shell-interpreted
  1954. # (we use list-form subprocess.run), but the same path may end up in
  1955. # user-facing logs, scripts, or future code paths that *do* interpolate.
  1956. # Reject loudly to keep hygiene tight.
  1957. # #sec-6 (#550): extended to reject NUL + control chars + unprintable
  1958. # bytes. The original list caught the obvious shell-special characters;
  1959. # control chars (0x00–0x1F minus tab) get rejected too because they
  1960. # survive the rejection of `\n` and `\r` only by accident, and can
  1961. # break log parsers / shell prompts in subtle ways.
  1962. _PATH_SHELL_METACHARS = re.compile(r"[;&|`$<>\n\r\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
  1963. def _resolve_claude_path(claude_path: Optional[str]) -> Optional[Path]:
  1964. """Resolve and validate the ``--claude`` path (#421).
  1965. Returns ``None`` when:
  1966. - The path is empty or contains shell metacharacters (rejected loudly).
  1967. - ``shutil.which`` can't find the binary on PATH (when no path passed).
  1968. - The resolved path doesn't exist on disk.
  1969. Returns a ``Path`` when the binary is found and looks safe. Callers
  1970. should treat ``None`` as "skip synthesis" — the synth step is best-
  1971. effort and never fatal.
  1972. """
  1973. if claude_path:
  1974. # Reject explicit paths containing shell metacharacters even
  1975. # though argv is list-form — this keeps the path safe to log.
  1976. if _PATH_SHELL_METACHARS.search(claude_path):
  1977. print(
  1978. f" warning: refusing claude path with shell metacharacters: "
  1979. f"{claude_path!r}",
  1980. file=sys.stderr,
  1981. )
  1982. return None
  1983. candidate = Path(claude_path)
  1984. else:
  1985. # No explicit path: use shutil.which so PATH-based lookups
  1986. # (homebrew, asdf, npm-global, Windows %PATH%) all just work.
  1987. import shutil as _shutil
  1988. found = _shutil.which("claude")
  1989. if not found:
  1990. return None
  1991. candidate = Path(found)
  1992. if not candidate.exists():
  1993. print(f" warning: claude CLI not found at {candidate}", file=sys.stderr)
  1994. return None
  1995. return candidate
  1996. # #486: validate slug shape before it lands in the synthesize_overview
  1997. # prompt. Anything that doesn't match this is replaced by `_invalid_`.
  1998. # - Length cap 80 chars (real slugs are far shorter)
  1999. # - Charset: alphanumerics + `.`, `_`, `-` (no whitespace, no shell
  2000. # metachars, no NUL bytes, no unicode-confusable categories)
  2001. _SAFE_SLUG_RE = re.compile(r"^[A-Za-z0-9._-]{1,80}$")
  2002. # #486: cap the total prompt size sent to the claude CLI. 32 KB is well
  2003. # inside any LLM's context, well inside macOS's ~256 KB argv limit, and
  2004. # small enough that a malicious large slug list can't push the prompt
  2005. # past the OS argv limit. (We also pass via stdin below so this is
  2006. # defence-in-depth.)
  2007. _MAX_OVERVIEW_PROMPT_BYTES = 32_000
  2008. def _validate_overview_slug(s: Any) -> str:
  2009. """Return ``s`` if it's a safe slug, ``"_invalid_"`` otherwise.
  2010. #486: a malicious .jsonl could land arbitrary content in the
  2011. `slug` field of a session's frontmatter. That string then ends
  2012. up inside the prompt sent to the claude CLI for overview
  2013. synthesis. Without validation, an attacker-controlled slug could:
  2014. - inject prompt text ("ignore previous instructions, …")
    • Dismissed after review: Quotes instruction-override phrasing (in quotes, code, a table or an example): "ignore previous instructions"
    • Line 2,250 on GitHub
  2015. - contain `\\x00` and crash subprocess.run with a ValueError
  2016. - balloon argv past the OS limit (~256 KB on macOS)
  2017. Replace anything sketchy with the literal `_invalid_` so the
  2018. prompt stays well-formed and the synthesis output stays
  2019. trustworthy.
  2020. """
  2021. if not isinstance(s, str):
  2022. return "_invalid_"
  2023. if not _SAFE_SLUG_RE.match(s):
  2024. return "_invalid_"
  2025. return s
  2026. def synthesize_overview(
  2027. groups: dict[str, list[tuple[Path, dict[str, Any], str]]],
  2028. claude_path: str,
  2029. ) -> Optional[str]:
  2030. resolved = _resolve_claude_path(claude_path)
  2031. if resolved is None:
  2032. return None
  2033. claude_path = str(resolved)
  2034. lines: list[str] = [
  2035. "You are writing a short (200-300 word) overview for a personal knowledge-base",
  2036. "landing page. Below is a JSON summary of the user's Claude Code session history",
  2037. "across multiple projects. Write 2-3 paragraphs of prose in markdown that:",
  2038. " 1. Names the main projects and what each is about (infer from session slugs and tool usage)",
  2039. " 2. Highlights the busiest project(s) and the overall scale of the work",
  2040. " 3. Is written in third person, referring to the user as 'the developer'",
  2041. "Do NOT use bullet points. Just 2-3 short paragraphs of prose.",
  2042. "",
  2043. "Data:",
  2044. "",
  2045. ]
  2046. brief: dict[str, Any] = {}
  2047. for project, sessions in sorted(groups.items()):
  2048. brief[project] = {
  2049. "session_count": len(sessions),
  2050. "main_sessions": sum(1 for p, m, _ in sessions if not _is_subagent(m, p)),
  2051. "dates": sorted({str(m.get("date", "")) for _, m, _ in sessions if m.get("date")}),
  2052. "models": sorted({str(m.get("model", "")) for _, m, _ in sessions if m.get("model")}),
  2053. # #486: every slug filtered through the safe-slug regex so
  2054. # malicious .jsonl content can't prompt-inject or crash
  2055. # subprocess.run via embedded NUL bytes.
  2056. "slugs": [_validate_overview_slug(m.get("slug", p.stem)) for p, m, _ in sessions[:8]],
  2057. }
  2058. prompt = "\n".join(lines) + json.dumps(brief, indent=2)
  2059. # #486: cap total prompt size so a malicious large slug list can't
  2060. # push the prompt past the OS argv limit. Truncation is fine here —
  2061. # the LLM gets the head of the JSON and produces a partial overview;
  2062. # better than a build that silently fails.
  2063. if len(prompt.encode("utf-8")) > _MAX_OVERVIEW_PROMPT_BYTES:
  2064. prompt = prompt.encode("utf-8")[:_MAX_OVERVIEW_PROMPT_BYTES].decode(
  2065. "utf-8", errors="ignore"
  2066. )
  2067. print(" calling claude CLI for overview synthesis…")
  2068. try:
  2069. # #486: pass the prompt via stdin (`-p -`) instead of argv so we
  2070. # dodge the OS argv-length limit entirely. The byte cap above is
  2071. # defence-in-depth — argv-length DoS path closed regardless.
  2072. result = subprocess.run(
  2073. [claude_path, "-p", "-", "--model", "claude-haiku-4-5-20251001"],
  2074. input=prompt,
  2075. capture_output=True, text=True, timeout=120,
  2076. )
  2077. except subprocess.TimeoutExpired:
  2078. print(" warning: claude CLI timed out after 120s", file=sys.stderr)
  2079. return None
  2080. # #py-m4 (#590): narrow `except Exception` to the families we
  2081. # actually expect from a subprocess call. Catching MemoryError or
  2082. # ImportError silently here would mask real failures.
  2083. except (OSError, subprocess.SubprocessError) as e:
  2084. print(f" warning: claude CLI failed: {e}", file=sys.stderr)
  2085. return None
  2086. if result.returncode != 0:
  2087. print(f" warning: claude CLI exited {result.returncode}", file=sys.stderr)
  2088. return None
  2089. return result.stdout.strip() or None
  2090. # ─── main ──────────────────────────────────────────────────────────────────
  2091. def build_site(
  2092. out_dir: Path = DEFAULT_OUT_DIR,
  2093. synthesize: bool = False,
  2094. claude_path: str = "",
  2095. search_mode: str = "auto",
  2096. seed_project_stubs: bool = False,
  2097. ) -> int:
  2098. if not RAW_SESSIONS.exists():
  2099. print(
  2100. f"error: {RAW_SESSIONS} does not exist. Run `llmwiki init` + `llmwiki sync` first.",
  2101. file=sys.stderr,
  2102. )
  2103. return 2
  2104. print(f"==> scanning {RAW_SESSIONS}")
  2105. sources = discover_sources(RAW_SESSIONS)
  2106. if not sources:
  2107. print(" no sources found.", file=sys.stderr)
  2108. return 2
  2109. print(f" found {len(sources)} source markdowns")
  2110. groups = group_by_project(sources)
  2111. print(f" grouped into {len(groups)} projects")
  2112. # #414: stub-seeding used to be unconditional. `build` is documented
  2113. # as read-only on `wiki/`, but seeding wrote to `wiki/projects/` —
  2114. # CI users running `llmwiki build` on a curated checkout discovered
  2115. # surprise commits in their working tree. Now opt-in: callers that
  2116. # have already accepted mutation (sync, the new `--seed-project-stubs`
  2117. # flag) request seeding explicitly; the default `build` is pure.
  2118. if seed_project_stubs:
  2119. stubs_written = ensure_project_stubs(groups, PROJECTS_META_DIR)
  2120. if stubs_written:
  2121. print(f" seeded {len(stubs_written)} new wiki/projects/ stubs")
  2122. # Reset output dir (clear contents only — the HTTP server may be cwd'd here)
  2123. # #py-m12 (#598): drop ignore_errors=True. A failure to remove a
  2124. # site/ subtree means we'll write a corrupted partial site on top
  2125. # of stale files; users have hit this when one CI runner left a
  2126. # read-only directory behind. Surface OSError instead so the build
  2127. # halts with a clear message.
  2128. if out_dir.exists():
  2129. rmtree_errors: list[str] = []
  2130. def _on_rmtree_error(func, path, exc_info):
  2131. err = exc_info[1] if isinstance(exc_info, tuple) else exc_info
  2132. rmtree_errors.append(f"{path}: {err}")
  2133. for child in out_dir.iterdir():
  2134. if child.is_dir():
  2135. # Python 3.12+ uses onexc; pre-3.12 uses onerror. Use
  2136. # onerror for back-compat with the 3.9 floor.
  2137. shutil.rmtree(child, onerror=_on_rmtree_error)
  2138. else:
  2139. try:
  2140. child.unlink()
  2141. except OSError as e:
  2142. rmtree_errors.append(f"{child}: {e}")
  2143. if rmtree_errors:
  2144. raise OSError(
  2145. "could not reset site dir " + str(out_dir) + ":\n "
  2146. + "\n ".join(rmtree_errors)
  2147. )
  2148. else:
  2149. out_dir.mkdir(parents=True)
  2150. # CSS + JS
  2151. (out_dir / "style.css").write_text(CSS, encoding="utf-8")
  2152. (out_dir / "script.js").write_text(JS, encoding="utf-8")
  2153. print(" wrote style.css, script.js")
  2154. # Copy raw markdown files for "Download .md" links
  2155. sources_out = out_dir / "sources"
  2156. if sources_out.exists():
  2157. shutil.rmtree(sources_out)
  2158. shutil.copytree(RAW_SESSIONS, sources_out)
  2159. print(f" copied raw .md sources to sources/")
  2160. # v0.7 (#96): copy downloaded image assets into site/assets/
  2161. raw_assets = RAW_DIR / "assets"
  2162. if raw_assets.exists() and any(raw_assets.iterdir()):
  2163. site_assets = out_dir / "assets"
  2164. if site_assets.exists():
  2165. shutil.rmtree(site_assets)
  2166. shutil.copytree(raw_assets, site_assets)
  2167. asset_count = sum(1 for _ in site_assets.iterdir() if _.is_file())
  2168. print(f" copied {asset_count} image assets to assets/")
  2169. # Synthesis
  2170. synthesis = None
  2171. if synthesize:
  2172. synthesis = synthesize_overview(groups, claude_path)
  2173. if synthesis:
  2174. print(f" synthesis: {len(synthesis)} chars")
  2175. # Render pages — single pass over `sources` (#py-m8 / #594).
  2176. # Sibling writes (.txt + .json) happen inside the same iteration so
  2177. # we don't re-walk `sources` later. The exporter import is hoisted
  2178. # ABOVE the loop with try/except: a missing module degrades to "no
  2179. # siblings on any session" (the import-fail case is truly fatal —
  2180. # no sense retrying it per-source).
  2181. #
  2182. # #v1378-review (architect + python-reviewer): per-source write
  2183. # failures are now isolated. The previous version set a single
  2184. # `siblings_failed` flag on the first OSError/ValueError/
  2185. # RuntimeError and silently dropped sibling writes for EVERY
  2186. # subsequent session in the loop — a single bad body on session 3
  2187. # of 500 produced 497 silently missing siblings. Now each source's
  2188. # sibling write is wrapped individually and collects errors into
  2189. # a list; the build proceeds and we print a summary at the end.
  2190. sibling_writers_loaded = True
  2191. sibling_import_error: Optional[BaseException] = None
  2192. try:
  2193. from llmwiki.exporters import write_page_json, write_page_txt
  2194. except (ImportError, OSError, ValueError, RuntimeError) as e:
  2195. sibling_writers_loaded = False
  2196. sibling_import_error = e
  2197. n_sessions = 0
  2198. n_siblings = 0
  2199. sibling_failures: list[tuple[str, BaseException]] = []
  2200. _wikilink_re = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]*)?\]\]")
  2201. for path, meta, body in sources:
  2202. project = str(meta.get("project") or path.parent.name)
  2203. render_session(path, meta, body, out_dir, project)
  2204. n_sessions += 1
  2205. if not sibling_writers_loaded:
  2206. continue
  2207. # render_session writes site/sessions/<project>/<stem>.html;
  2208. # mirror its path here so the siblings land alongside it.
  2209. html_path = out_dir / "sessions" / project / f"{path.stem}.html"
  2210. if not html_path.exists():
  2211. continue
  2212. try:
  2213. body_stripped = strip_leading_h1(body)
  2214. wikilinks_out = _wikilink_re.findall(body_stripped)
  2215. write_page_txt(html_path, body_stripped)
  2216. n_siblings += 1
  2217. meta_copy = dict(meta)
  2218. meta_copy.setdefault("slug", str(meta.get("slug", path.stem)))
  2219. meta_copy.setdefault("project", project)
  2220. write_page_json(html_path, meta_copy, body_stripped, wikilinks_out)
  2221. n_siblings += 1
  2222. except (OSError, ValueError, RuntimeError) as e:
  2223. # Isolate to this source — the next iteration still attempts
  2224. # both writes. n_siblings increments only after each
  2225. # successful write so the count never over-reports.
  2226. sibling_failures.append((str(html_path), e))
  2227. # Print warnings BEFORE the success line so a CI log scanner sees
  2228. # them in the right order (architect-review feedback).
  2229. if not sibling_writers_loaded:
  2230. print(
  2231. f" warning: per-page siblings disabled (exporter import failed): "
  2232. f"{sibling_import_error}",
  2233. file=sys.stderr,
  2234. )
  2235. if sibling_failures:
  2236. first_path, first_err = sibling_failures[0]
  2237. print(
  2238. f" warning: per-page sibling write failed on "
  2239. f"{len(sibling_failures)} of {n_sessions} sessions; "
  2240. f"first failure: {first_path}: {first_err}",
  2241. file=sys.stderr,
  2242. )
  2243. print(f" wrote {n_sessions} session pages")
  2244. if sibling_writers_loaded:
  2245. print(f" wrote {n_siblings} per-page siblings (.txt + .json)")
  2246. for project, sessions in groups.items():
  2247. render_project_page(project, sessions, out_dir)
  2248. print(f" wrote {len(groups)} project pages")
  2249. render_projects_index(groups, out_dir)
  2250. render_sessions_index(sources, groups, out_dir)
  2251. render_index(groups, sources, out_dir, synthesis=synthesis)
  2252. cl_path = render_changelog(out_dir)
  2253. # #387 U8: branded 404 page that serve.py returns as the body of any
  2254. # 404 response, instead of the stdlib http.server default.
  2255. not_found_path = render_404(out_dir)
  2256. # #284: compile README + CONTRIBUTING as standalone site pages so
  2257. # they don't bounce visitors out to GitHub for content we're already
  2258. # shipping as HTML.
  2259. readme_path = render_readme_page(out_dir)
  2260. contributing_path = render_contributing_page(out_dir)
  2261. print(
  2262. " wrote index.html, projects/index.html, sessions/index.html, 404.html"
  2263. + (", changelog.html" if cl_path else "")
  2264. )
  2265. # Search index (chunked — #47) + tree/flat auto-routing (#53)
  2266. idx_path = build_search_index(sources, groups, out_dir, search_mode=search_mode)
  2267. # v0.4: AI-consumable exports (llms.txt, llms-full.txt, graph.jsonld,
  2268. # sitemap.xml, rss.xml, robots.txt, ai-readme.md)
  2269. # #py-m4 (#590): narrow the catch so MemoryError, ImportError,
  2270. # and KeyboardInterrupt aren't silently swallowed into a warning
  2271. # line. ImportError in particular hides a broken module — the
  2272. # build should crash loud, not log "warning: AI exports failed:
  2273. # No module named ..." and ship a half-built site.
  2274. try:
  2275. from llmwiki.exporters import export_all
  2276. ai_paths = export_all(out_dir, groups, sources)
  2277. print(f" wrote {len(ai_paths)} AI-consumable exports: {', '.join(sorted(ai_paths.keys()))}")
  2278. except (OSError, ValueError, RuntimeError) as e:
  2279. print(f" warning: AI exports failed: {e}", file=sys.stderr)
  2280. # v1.1 (#118): copy the interactive knowledge graph into the site
  2281. # so the "Graph" nav link works without a separate `llmwiki graph` step.
  2282. try:
  2283. from llmwiki.graph import copy_to_site as copy_graph_to_site
  2284. graph_path = copy_graph_to_site(out_dir)
  2285. if graph_path:
  2286. print(f" wrote {graph_path.relative_to(out_dir.parent)} (interactive graph viewer)")
  2287. except (OSError, ValueError, RuntimeError) as e:
  2288. print(f" warning: graph viewer copy failed: {e}", file=sys.stderr)
  2289. # v1.2 (#265): compile the editorial docs (tutorials + hub) under
  2290. # site/docs/. Only pages with `docs_shell: true` in frontmatter
  2291. # are included — reference docs that stay GitHub-rendered aren't
  2292. # touched.
  2293. try:
  2294. from llmwiki.docs_pages import compile_docs_site
  2295. docs_dir = REPO_ROOT / "docs"
  2296. # nav_builder gets called per-page with the right link_prefix so
  2297. # the nav bar's hrefs resolve from whatever depth the page sits at.
  2298. def _docs_nav(link_prefix: str) -> str:
  2299. return nav_bar(active="docs", link_prefix=link_prefix)
  2300. docs_written = compile_docs_site(
  2301. docs_dir,
  2302. out_dir,
  2303. md_to_html=md_to_html,
  2304. page_head=page_head,
  2305. nav_builder=_docs_nav,
  2306. )
  2307. if docs_written:
  2308. print(
  2309. f" wrote site/docs/ ({len(docs_written)} editorial pages: "
  2310. "hub + tutorials + style guide)"
  2311. )
  2312. # #py-m4: same narrow-catch pattern as above.
  2313. except (OSError, ValueError, RuntimeError) as e:
  2314. print(f" warning: docs compile failed: {e}", file=sys.stderr)
  2315. # v0.4 per-page sibling .txt and .json — collapsed into the
  2316. # render_session loop above (#py-m8 / #594). The duplicate walk and
  2317. # html_path.exists() guard are gone; siblings now write inside the
  2318. # same iteration that produced the HTML.
  2319. # v0.4: Build manifest with SHA-256 hashes
  2320. try:
  2321. from llmwiki.manifest import write_manifest
  2322. manifest_path = write_manifest(out_dir)
  2323. print(f" wrote {manifest_path.relative_to(out_dir.parent) if manifest_path.is_relative_to(out_dir.parent) else manifest_path.name}")
  2324. except (OSError, ValueError, RuntimeError) as e:
  2325. print(f" warning: manifest failed: {e}", file=sys.stderr)
  2326. total_files = sum(1 for _ in out_dir.rglob("*.html"))
  2327. total_bytes = sum(p.stat().st_size for p in out_dir.rglob("*") if p.is_file())
  2328. print(
  2329. f"==> build complete: {total_files} HTML files, {total_bytes / 1024:.0f} KB total"
  2330. )
  2331. print(f" output: {out_dir}")
  2332. return 0
  2333. def parse_args() -> argparse.Namespace:
  2334. p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
  2335. p.add_argument("--out", type=Path, default=DEFAULT_OUT_DIR)
  2336. p.add_argument("--synthesize", action="store_true")
  2337. p.add_argument("--claude", type=str, default="")
  2338. return p.parse_args()
  2339. def main() -> int:
  2340. args = parse_args()
  2341. return build_site(
  2342. out_dir=args.out,
  2343. synthesize=args.synthesize,
  2344. claude_path=args.claude,
  2345. )
  2346. if __name__ == "__main__":
  2347. sys.exit(main())
  2348. # ─── agent label detection ────────────────────────────────────────────────
  2349. # v0.9: Detect which AI agent produced a session from the model name,
  2350. # source_file path, or explicit frontmatter field. Returns a short label
  2351. # + CSS class for badge rendering.
  2352. def detect_agent_label(meta: dict) -> tuple[str, str]:
  2353. """Return (label, css_class) for the agent that produced this session.
  2354. Detection order:
  2355. 1. Explicit `agent:` frontmatter field (set by adapters)
  2356. 2. Model name patterns (claude-* → Claude, gpt-* → Codex/Copilot, etc.)
  2357. 3. Source file path patterns (codex → Codex, copilot → Copilot)
  2358. 4. Default: "Unknown"
  2359. """
  2360. # 1. Explicit field
  2361. agent = str(meta.get("agent", "")).strip().lower()
  2362. if agent:
  2363. return _agent_map(agent)
  2364. # 2. Model name
  2365. model = str(meta.get("model", "")).lower()
  2366. if "claude" in model:
  2367. return ("Claude", "agent-claude")
  2368. if "gpt" in model or "o1" in model or "o3" in model or "o4" in model:
  2369. return ("Codex", "agent-codex")
  2370. if "gemini" in model:
  2371. return ("Gemini", "agent-gemini")
  2372. if "copilot" in model:
  2373. return ("Copilot", "agent-copilot")
  2374. # 3. Source file path
  2375. source = str(meta.get("source_file", "")).lower()
  2376. if "codex" in source or ".codex" in source:
  2377. return ("Codex", "agent-codex")
  2378. if "copilot" in source:
  2379. return ("Copilot", "agent-copilot")
  2380. if "cursor" in source:
  2381. return ("Cursor", "agent-cursor")
  2382. if "gemini" in source:
  2383. return ("Gemini", "agent-gemini")
  2384. if "claude" in source or ".claude" in source:
  2385. return ("Claude", "agent-claude")
  2386. # 4. Tags fallback
  2387. tags = meta.get("tags", [])
  2388. if isinstance(tags, list):
  2389. tag_str = " ".join(str(t).lower() for t in tags)
  2390. else:
  2391. tag_str = str(tags).lower()
  2392. if "codex" in tag_str:
  2393. return ("Codex", "agent-codex")
  2394. if "copilot" in tag_str:
  2395. return ("Copilot", "agent-copilot")
  2396. if "claude" in tag_str:
  2397. return ("Claude", "agent-claude")
  2398. return ("Agent", "agent-unknown")
  2399. def _agent_map(agent: str) -> tuple[str, str]:
  2400. """Map an explicit agent name to (label, css_class)."""
  2401. m = {
  2402. "claude": ("Claude", "agent-claude"),
  2403. "claude-code": ("Claude", "agent-claude"),
  2404. "codex": ("Codex", "agent-codex"),
  2405. "codex-cli": ("Codex", "agent-codex"),
  2406. "copilot": ("Copilot", "agent-copilot"),
  2407. "copilot-chat": ("Copilot", "agent-copilot"),
  2408. "copilot-cli": ("Copilot", "agent-copilot"),
  2409. "cursor": ("Cursor", "agent-cursor"),
  2410. "gemini": ("Gemini", "agent-gemini"),
  2411. "gemini-cli": ("Gemini", "agent-gemini"),
  2412. "obsidian": ("Obsidian", "agent-obsidian"),
  2413. # Simplification sweep removed the PDF adapter. The "pdf" entry
  2414. # used to live here; left as a comment so a future grep sees
  2415. # the rationale instead of guessing why the agent-pdf badge is
  2416. # gone. CSS class .agent-pdf is also removed (see render/css.py).
  2417. }
  2418. return m.get(agent, (agent.title(), "agent-unknown"))
  2419. def render_agent_badge(meta: dict) -> str:
  2420. """Render an inline agent badge chip."""
  2421. label, css_class = detect_agent_label(meta)
  2422. return f'<span class="agent-badge {html.escape(css_class)}">{html.escape(label)}</span>'
  2423. # Inject agent badge CSS into the CSS constant
  2424. # (This is appended at the module level — it'll be picked up on next build)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.