Back to search
llm-wiki-v2
SkillGitHub
Turn your documents into a Markdown wiki you can read, that answers with citations. Works with any LLM provider, Ollama included. CROW: 99.5% right note read vs 79%, 1.6× faster ingest, 63% fewer LLM tokens, no scaling limit.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Checked yesterday
Badge for your README
Markdown
[](/listings/crow-hq--llm-wiki-v2)HTML
<a href="/listings/crow-hq--llm-wiki-v2"><img src="/api/listings/crow-hq--llm-wiki-v2/badge.svg" alt="findsafeskills safety rating" /></a>Install
i
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 555 lines
- <p align="center">
- <img src="https://raw.githubusercontent.com/crow-hq/llm-wiki-v2/main/docs/assets/crow-banner.jpg" alt="CROW: a pixel-art crow next to the word CROW" width="100%">
- </p>
- <p align="center">
- <img src="https://raw.githubusercontent.com/crow-hq/llm-wiki-v2/main/docs/assets/crow-stats.svg" alt="An LLM wiki with no scaling limit. CROW vs classic on 11 corpora (2,019 documents, 656 questions): 99.5% right note read vs 79%; 1.6× faster per ingest (18.0s vs 29.2s); 63% fewer LLM tokens (4,861 vs 13,227); 0.59× the cost per ingest with the writer at list price." width="100%">
- </p>
- <p align="center">
- <img src="https://raw.githubusercontent.com/crow-hq/llm-wiki-v2/main/docs/assets/demo.gif" alt="Twenty-two documents dropped on the brain of an empty wiki are filed in folders the wiki opens, and the brain grows with them (shown five times faster); a question about a damaged parcel gets an answer citing two notes, and one of them opens" width="100%">
- </p>
- # LLM Wiki v2
- Give it your documents. It files them into a tidy wiki of Markdown notes and
- answers your questions, citing the notes it used. It runs on your computer,
- with any AI provider: OpenRouter, OpenAI, Gemini, Ollama (free and local) or
- any OpenAI-compatible server. It works in **CROW** mode by default: a small
- classifier decides where things go, so the wiki stays fast and cheap as it grows.
- ## Start in 2 steps
- Rather not use a terminal? [Crow](https://github.com/crow-hq/crow-desktop) is this library as a desktop app, with
- downloads for macOS, Windows and Linux. The steps below install the library itself, with its web page and command line.
- **1. Install it** with Python 3.11 or later (in a virtual environment, if your system asks for one):
- ```bash
- pip install llmwiki2
- ```
- No Python, or you'd rather keep it apart from your other packages? Use [uv](https://docs.astral.sh/uv/getting-started/installation/),
- which also fetches Python: `uv tool install llmwiki2`. The same install is the Python library too:
- `from llmw2 import Wiki`, see the [library guide](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/library/index.md).
- **2. Start it:**
- ```bash
- llmwiki2
- ```
- Your browser opens. Pick a provider, paste its key, press **Test connection**,
- then **Save and start**. That's it.
- **Just looking?** `llmwiki2 demo` opens an example wiki, the notes of a fictional coffee
- roastery, with no key needed to browse it and its brain. It is copied to `~/llm-wiki-demo`,
- so you can add to it or delete it. To try it without installing anything but uv:
- `uvx llmwiki2 demo`.
- <p align="center">
- <img src="https://raw.githubusercontent.com/crow-hq/llm-wiki-v2/main/docs/assets/screenshot-settings.png" alt="The first-run page: pick a provider, paste its key, choose a model" width="90%">
- </p>
- > [!TIP]
- > **No key?** Pick **Ollama**: free, runs on your computer, no account
- > ([download it](https://ollama.com/download), then `ollama pull gemma4`).
- > Ollama runs the model locally; CROW's classifier is separate and remote by default.
- > It has its own provider (Settings → Advanced): OpenRouter, TypeSafe or a local
- > server. Without one the wiki uses classic mode, where the model decides by itself.
- > One [OpenRouter key](https://openrouter.ai/keys) works for every model, and for CROW too.
- ## What you can do
- | | |
- |---|---|
- | **Add documents** | Drop text, Markdown, PDF, Word, PowerPoint, Excel or OpenDocument files anywhere on the page. Each one lands in the right folder, merged with what the wiki already knows; a spreadsheet gives a note per sheet. |
- | **Ask questions** | Type a question. The answer cites the notes it used: click one to read it. |
- | **Give Claude a memory** | From the terminal only: `llmwiki2 setup`, then talk to Claude Code or Codex in plain words. With the page: **Memory** → **Start**, then paste the `claude mcp add …` or `codex mcp add …` line it shows. Either way they recall from your wiki and file what they learn in it ([below](#claudes-memory-mcp)). |
- | **See the brain** | A live map of your folders, notes and links. |
- | **Delete** | Hover a note or folder in the sidebar and press the bin, or **Delete note** at the foot of a note. Links to it go from the other notes. **Empty the wiki** on the home page starts over. |
- | **Move and describe** | **Move** at the foot of a note files it in another folder; every link to and from it follows. The pencil beside a folder in the sidebar edits its description, the line the librarian routes by. |
- | **Areas** | **Areas** at the top: create a top-level folder, confirm or dismiss the areas proposed to the wiki, rename one or merge it into another. The wiki never opens or proposes an area by itself. |
- | **Keep your files** | Plain Markdown in `~/llm-wiki`: open it in any editor, keep it in git. |
- ## Claude's memory (MCP)
- <!-- mcp-name: io.github.crow-hq/llm-wiki-v2 -->
- The wiki can be the long-term memory of Claude Code, Codex or any MCP client, documents already filed included.
- It comes with the `mcp` extra (the Docker image has it).
- ### From the terminal only (Claude Code, Codex)
- No page, no server to keep running: Claude Code starts the memory by itself.
- <p align="center">
- <img src="https://raw.githubusercontent.com/crow-hq/llm-wiki-v2/main/docs/assets/demo-memory.gif" alt="In a terminal, llmwiki2 setup shows the CROW logo and is answered 2, then Enter at every question: the memory is on and Claude Code is connected. In Claude Code, a remember about a supplier's minimum order is merged into the supplier's note, whose Markdown file beside it shows the new line; in a new session, a question in Italian is answered in Italian through recall, citing that note (the models' work sped up)" width="100%">
- </p>
- **1. Install it:**
- ```bash
- uv tool install 'llmwiki2[mcp]' # or: pip install 'llmwiki2[mcp]'
- ```
- **2. Set it up:**
- ```bash
- llmwiki2 setup
- ```
- To the first question answer **2** (as the memory of an agentic CLI); after that, Enter takes the default every time:
- | It asks | Pick |
- |---|---|
- | Which models do the work | **1** (the default): Claude Code or Codex do it, on their plan, no key. **2** (hybrid) is quicker and needs an OpenRouter key, or Ollama on your computer |
- | Connect Claude Code? | **Y**. With several configurations (`~/.claude`, `~/.claude-work`), it asks which: Enter for all |
- | Add the lines to CLAUDE.md? | **Y**: they make Claude use the memory when you talk normally |
- **3. Talk to it**, in any language, in Claude Code or Codex, anywhere:
- > ricorda che Rossi ci paga a 45 giorni · what did we decide about pricing? · add ~/Downloads/offer.pdf to the
- > wiki · move the pricing note to suppliers
- The first time it uses each tool, Claude Code asks your permission: choose "don't ask again" for the ones you trust.
- To see the brain or browse the notes, run `llmwiki2`: the page opens on the same wiki, while Claude Code keeps using it.
- Run `llmwiki2 setup` again to change anything. Windows says it blocked `llmwiki2.exe`? See
- [Something wrong?](#something-wrong).
- ### With the page
- Open **Memory**, press **Start**, and connect a client:
- ```bash
- claude mcp add --transport http --scope user llm-wiki http://127.0.0.1:8000/mcp
- codex mcp add llm-wiki --url http://127.0.0.1:8000/mcp
- ```
- ### What it can do
- The client gets its tools: `recall` answers from your notes and documents, citing them; `remember` files what it
- learns, in the right folder, merged and linked like any note (CROW); `search` finds notes by their words with no model
- call; `read_note` reads a note in full; `folders`, `move_note`, `describe_folder`, `rename_area`, `merge_areas` and
- `delete_note` reorganize the wiki (the last two only when you ask); `rebuild` (with `confirm`) files every note again on
- the same folders, from the memory's own raw copies. Started from the terminal (`llmwiki2 mcp`) it also files documents
- from your disk: `remember_file` and `sync_folder`. Which models do the work is a choice under Memory or in `setup`:
- - **This wiki's settings** (the default on the page). OpenRouter or a local model, with Jev or Laya deciding, file the memories; the
- client only calls the memory.
- - **Claude or GPT for everything** (the default in `setup`). The memory hands the client every model call it makes, while CROW runs as always.
- Claude Code decides with a Haiku subagent and writes with its own model; Codex uses its own. It runs on the client's
- plan, with no key in the wiki. The requests carry your documents' text to the client: it is told to read it as data
- and to answer without tools, but that is an instruction, not a sandbox (see Security in the memory guide).
- - **Hybrid.** The client's model writes, and the wiki's classifier (Jev through OpenRouter, or Ollama or Laya on your
- computer) decides: fewer round trips than the above, and no Haiku subagents.
- Docker, the tools and security: [docs/wiki/memory.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/wiki/memory.md).
- ## Why not RAG?
- Because a wiki stays readable. RAG keeps your documents as embeddings, an opaque
- vector index no one can read or fix. Here every document becomes a Markdown note,
- filed in a folder that says what it holds and linked to related notes: you can open
- it, correct it and keep it in git, and every answer cites the notes it used. And
- knowledge accumulates: a new source updates the note on its subject instead of
- being rediscovered at every question.
- CROW keeps it that way as the wiki grows. In [Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
- the model finds pages by reading an index, which works at moderate scale (about a
- hundred sources); beyond that the gist suggests adding search, hybrid lexical and
- vector. In CROW a small classifier works with the folders instead: filing picks the
- folder among every folder of the wiki in one choice, and answering finds the notes by
- search and lets the classifier reorder them, so the wiki can grow without vectors and stay
- something a person can read. The LLM lays out the folders: in a batch, before its documents are filed, and
- for a document filed alone, by splitting a folder that has grown. In the [paper](https://doi.org/10.5281/zenodo.22898752), retrieval kept its
- accuracy across a twentyfold growth of the wiki. A separate test on a private wiki
- of about 500 notes pointed the same way: with CROW the right note was found nearly
- as often at the end as early on, while with the LLM taking the decisions it was
- found markedly less often as the wiki grew. On 1,000 Enron mails filed in their owners' folders, CROW read the right
- note for every question at each step of the growth while classic went from 76% to 60%. The classifier weighs at most
- 24 candidate notes per decision, so its tokens per document nearly stopped growing after the first few hundred documents;
- CROW's cost per document rose 1.4 times over the 1,000, classic's 1.6 times, and at 800 to 1,000 documents CROW cost 0.41
- times classic's with the writer at its list price ([benchmarks/README.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/benchmarks/README.md#crow-as-the-wiki-grows)).
- ## FAQ
- **Does everything run on my computer?** It can, with no key and nothing leaving it.
- [Crow](https://github.com/crow-hq/crow-desktop), the desktop app, does it out of the box. With the library, Ollama
- serves the writer and the classifier too: a typed classifier such as `tev1` behind Ollama 0.35 or later, or Laya
- (see [everything on your computer](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/wiki/providers.md#everything-on-your-computer-with-small-models)).
- The library's default classifier is remote: Jev, through OpenRouter or TypeSafe. Without a classifier the wiki runs in
- classic mode, where the LLM takes every decision.
- **What is Jev?** A typed classifier by TypeSafe AI. Given a text, it answers typed
- questions (pick one of these options; is this statement true?) with probabilities
- instead of prose. CROW asks it every filing and retrieval decision and keeps the
- LLM for writing. CROW does not depend on it: any classifier that returns those
- probabilities can take its place. The authors have no ties to TypeSafe AI.
- **Aren't these your own benchmarks?** Yes. The paper measures one corpus of 525
- personal documents, mostly in Italian, with one LLM and one classifier version, and
- says so among its limitations; its rates carry 95% intervals. The corpora and
- scripts behind these measurements are public in [benchmarks/](https://github.com/crow-hq/llm-wiki-v2/tree/main/benchmarks),
- and every script has a `--dry-run`, so you can run them with your own models.
- The numbers of the banner come from eleven corpora (handbooks, mail, laws in two languages, contracts, tax manuals,
- Wikibooks pages and personal notes, 2,019 documents), each filed in CROW and in classic mode with the same writer, and
- 656 questions written blind from single documents: the right note was read 99.5% of the time against 79.4%, filing was 1.6
- times as fast with 63% fewer LLM tokens. The cost depends on the writer's price: on the cheapest providers, where the writer
- costs about what the classifier does, it came out the same; with the writer at its list price CROW costs 0.59 times classic's.
- The table per corpus is in [benchmarks/README.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/benchmarks/README.md#crow-against-classic-on-eleven-corpora).
- The filing measures of the CROW routing use, besides the repo's own synthetic coffee
- corpus, the [Enron email corpus](https://www.cs.cmu.edu/~enron/), the
- [GitLab handbook](https://gitlab.com/gitlab-com/content-sites/handbook) and the
- [PostHog handbook](https://github.com/PostHog/posthog.com): they are linked, not copied.
- On the PostHog handbook, CROW files pages closer to their real folders than `main` (its CROW routing and its classic
- mode) with both models measured, at 0.86 to 0.87 times the cost of main's CROW and 0.45 to 0.65 times that of classic,
- with retrieval between 93% and 98% for all: the table, what each column measures and how to run it again are in
- [benchmarks/README.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/benchmarks/README.md#results-of-060-crow-against-crow-050-and-classic).
- **Why AGPL, and why a CLA?** The AGPL keeps the code and every change to it open,
- also when it runs as a service. The CLA keeps the rights clear between the two
- authors and everyone who contributes; you keep the copyright in your work.
- ## Something wrong?
- - **"no API key yet"**: open ⚙ on the page, or run `llmwiki2 setup` in the terminal.
- - **Test connection fails**: the message is the provider's own, usually a wrong key or model name.
- - **A PDF adds nothing**: it is a scan with no text layer; run OCR on it first.
- - **Another folder for the wiki**: `llmwiki2 --bundle <folder>`. Settings live in `~/.config/llm-wiki/config.json`.
- - **Windows blocks `llmwiki2.exe`** ("an application control policy has blocked this file", common on work PCs): run
- it through Python, which the policy lets through. Installed with pip: `python -m llmwiki2 setup`. With uv, in
- PowerShell: `& "$(uv tool dir)\llmwiki2\Scripts\python.exe" -m llmwiki2 setup`. Every command works this way
- (`-m llmwiki2 serve`, `-m llmwiki2 demo`), and `setup` connects Claude Code through that same Python.
- ## For developers
- <details>
- <summary><b>Run it as a server with Docker</b></summary>
- ```bash
- git clone https://github.com/crow-hq/llm-wiki-v2 && cd llm-wiki-v2
- cp .env.example .env # choose OKF_LLM_PROVIDER and set its key
- docker compose up -d --build # the wiki on :8000 (OKF_PORT to change)
- ```
- With Docker the settings come from `.env`; the page shows them but cannot change them.
- </details>
- <details>
- <summary><b>HTTP API</b></summary>
- **The web UI** at http://localhost:8000 shows the folder tree, renders each note
- (with its See-also links and sources), answers questions with clickable
- citations, and shows the tokens spent by the LLM and by the classifier. Drop
- text (`.txt`, `.md`), PDF, Word (`.docx`), PowerPoint (`.pptx`), Excel (`.xlsx`) or
- OpenDocument (`.odt`, `.ods`) files anywhere on the page and each one is catalogued
- automatically; PDFs need a text layer (scanned PDFs need OCR first). A spreadsheet gives
- one note per sheet, which describes it; its rows stay in the raw copy as CSV, and
- questions get the rows they need. Old binary formats (`.doc`, `.xls`, `.ppt`) are not
- read: save them as `.docx`, `.xlsx` or `.pptx`.
- The same through the API:
- ```bash
- curl -X POST localhost:8000/ingest -H 'content-type: application/json' \
- -d '{"text": "Acme refunds orders within 30 days of delivery.", "title": "Refund policy"}'
- curl -X POST 'localhost:8000/upload?filename=minutes.pdf' --data-binary @minutes.pdf
- curl -X POST localhost:8000/ask -H 'content-type: application/json' \
- -d '{"question": "How long do customers have to ask for a refund?"}'
- ```
- The wiki is plain Markdown files you can read, edit and keep in git (`./wiki`
- with Docker). With Docker the CLI runs in the same image: `docker compose run --rm wiki llmwiki2 check`.
- To find notes with no model call: `llmwiki2 search "burner service" -k 5` (`--where KEY=VALUE`, `--json`). To file a note elsewhere or reword a folder's description from the terminal: `llmwiki2 move finance/pricing.md /sourcing` and `llmwiki2 describe /finance "Money in and out."`.
- `llmwiki2 rebuild [folder] --yes` deletes every note, raw copy and archived version and files the documents again on the folders the wiki has (the documents of `folder`, or with no folder the wiki's own raw copies), with no plan and no new folder; without `--yes` it only says what it would do and exits with 2. A note the limit on top-level folders sent to the likeliest existing folder instead of a new one is marked `overflow` in its frontmatter, logged in `log.md` as "Deviated note", and listed with the root it went to and its probabilities by `sync`, `ingest` and `rebuild`.
- In CROW mode, `llmwiki2 audit [--min-confidence 0.7] [--apply]` lists the notes the classifier would file in another branch today (it asks the same single choice among every folder, and proposes only a move to another branch, never deeper into the note's own folder and never up; no LLM call; it moves them only with `--apply`; `--min-confidence` is compared with that choice's top probability, a scale that has not been recalibrated), and `llmwiki2 health [--min-filed 5] [--max-share 0.3]` shows, from `log.md`, the folders that keep needing new subfolders or the LLM fallback (folders laid out by a plan or a split are not counted as new subfolders). Every new note carries a stable `id` in its frontmatter (`wiki.find_note(id)`).
- | Endpoint | What it does |
- |---|---|
- | `GET /` | the web UI |
- | `POST /ingest` `{text, title?, resource?, origin?}` | file a source (an `origin` `{source, account?, id, link?, version?, modified?}` makes a repeat with the same content or version return `"unchanged"`); returns the note, folders created, links and token usage |
- | `POST /upload?filename=…` (raw file body) | file a text, Markdown, PDF, Office or OpenDocument file (a spreadsheet: its other sheets in `parts`) |
- | `POST /ask` `{question}` | answer with citations; returns the notes read and token usage |
- | `GET /search?q=…&k=8` | the notes that best match a query (path, title, summary, rank), with no model call: a search box |
- | `GET /tree` · `GET /note?path=…` | folder tree · one note (frontmatter and Markdown body) |
- | `GET /check` · `GET /graph` | lint problems · the wiki as nodes and links (JSON) |
- | `POST /note/move` `{path, folder}` · `POST /folder/describe` `{path, description}` | move a note to another folder, links following (returns the note as `GET /note`) · change a folder's description |
- | `GET /proposals?status=open\|all` · `POST /proposals` `{label, description, notes}` · `POST /proposals/{id}/confirm` `{over_limit?}` · `POST /proposals/{id}/dismiss` | the queue of top-level areas users propose · propose one · open it and move its notes · close it |
- | `POST /areas` `{name, description, over_limit?}` · `POST /folder/rename` `{path, name}` · `POST /folder/merge` `{source, target}` | create a top-level folder, refused over `max_top_folders` unless `over_limit` · rename one · move all of one into another and delete it (links following) |
- | `POST /audit` `{min_confidence?, apply?}` · `GET /taxonomy` | CROW: notes the classifier would file in another branch (moved if `apply`) · per folder and month, new subfolders and fallbacks (from the log; folders laid out by a plan or a split not counted) |
- | `DELETE /note?path=…` · `DELETE /folder?path=/…` · `DELETE /wiki` | delete a note · a folder with all it holds · everything; links to them go, and raw copies no remaining note cites |
- | `GET /usage` · `GET /health` | token totals per ledger and model · status |
- | `GET /settings` · `POST /settings` · `POST /settings/test` | provider, model and key (masked) · save them · try them; changes from this machine only |
- | `POST /mcp` | the wiki as an MCP memory (Streamable HTTP; `recall`, `remember`, `search`, `read_note`, `rebuild`, `reply`, `create_area`, `propose_area`, `proposals`, `confirm_area`), with the `mcp` extra. Its tools work while the settings turn it on (`OKF_MCP`, or **Memory** → **Start**), and say how to turn it on otherwise; without the extra, 501 and what to install |
- | `POST /settings/memory` `{mcp?, mcp_models?}` | turn the memory on or off, and choose whose models do its work; no key needed, from this machine only |
- Every request that changes the wiki may carry an `X-Author` header: the lines it adds to `log.md` end with ` · by <author>`. The author is declared, not verified: there is no authentication, so behind a reverse proxy that authenticates your users, have the proxy set the header. The CLI declares your login name.
- </details>
- <details>
- <summary><b>Use it as a Python library and from the command line</b></summary>
- ```bash
- pip install llmwiki2
- ```
- ```python
- from llmw2 import Wiki
- wiki = Wiki.from_env(bundle="./wiki") # OKF_* variables and the provider's key, e.g. OPENROUTER_API_KEY
- wiki.init()
- result = wiki.ingest(open("meeting.md").read(), title="Pricing meeting")
- wiki.ingest_file(open("minutes.pdf", "rb").read(), "minutes.pdf") # also .docx, .pptx, .xlsx, .odt, .ods
- print(result.action, result.note, result.created_folders, result.related)
- answer = wiki.ask("What did we decide about enterprise pricing?")
- print(answer.text, answer.citations)
- print(wiki.usage.to_dict()) # {"llm": …, "classifier": …, "by_model": …}
- ```
- The same operations are on the command line:
- ```bash
- llmwiki2 --bundle ./wiki init
- llmwiki2 --bundle ./wiki ingest meeting.md # or a .txt, .pdf, .docx, .pptx, .xlsx, .odt, .ods
- llmwiki2 --bundle ./wiki sync ~/OneDrive # file a folder and keep the wiki in step with it
- llmwiki2 --bundle ./wiki --mode classic ask "What did we decide about pricing?"
- llmwiki2 --bundle ./wiki check # lint: OKF conformance, index drift, broken links
- llmwiki2 --bundle ./wiki serve # the HTTP API and page; plain `llmwiki2` also opens the browser
- llmwiki2 setup # provider, key and model, tested and saved; or the memory of Claude Code/Codex
- llmwiki2 --bundle ./wiki mcp # the wiki as an MCP memory over stdio, started by the client (claude mcp add … -- llmwiki2 mcp)
- llmwiki2 demo # an example wiki, copied to ~/llm-wiki-demo and opened
- ```
- To keep the wiki in step with a source, hand `sync` a connector. Each document
- keeps its `Origin` (where the original lives), so an unchanged file costs no model
- call and a deleted one is marked in the raw copy while its notes stay:
- ```python
- from pathlib import Path
- from llmw2 import LocalFolderSource, sync
- report = sync(wiki, LocalFolderSource(Path("~/OneDrive"))) # a folder, e.g. one rclone keeps in sync
- print(report.to_dict()) # created, merged, unchanged, removed, skipped, errors
- ```
- A connector for another service is any object with `name`, `account`,
- `changes(cursor) -> ChangeBatch` and `fetch(change) -> bytes` (`SourceConnector`);
- the cursor of the last sync is kept in `.llmw2/sync/` inside the wiki. A folder is handed over in pages of 200 changes and the
- cursor is saved after each page, so a sync of thousands of documents that stops resumes where its last page ended.
- For a wiki you rebuild or audit often, `classifier.cache` (`OKF_CLASSIFIER_CACHE=true`) keeps the classifier's answers in the
- bundle, as hashes and answers only: a rebuild, a resumed sync or a repeated audit then asks it nothing it was asked before. It
- does not save anything on a sync of new documents. See the [library guide](docs/library/index.md).
- **Your own top-level areas.** Make them with `create_folder` before the first sync, each with a description (the
- classifier routes by it); to keep it idempotent, `if wiki.tree().find("/finance") is None: wiki.create_folder("/", "finance", "...")`.
- Folders made by hand count in `max_top_folders`: set it to the number of your areas and the wiki is closed (no new
- top-level folder, the notes go to the likeliest area); a higher value leaves it open. `delete_folder` deletes the notes
- in it. For different permissions, use one wiki per domain.
- The wiki never opens or proposes an area by itself: proposals come from you, and the list of deviated notes helps you
- compose them. `propose_area(label, description, notes)` queues one (the notes by id or path; the queue is kept in the
- wiki, in `.llmw2/proposals.json`, and nothing moves yet), `proposals()` lists the open ones, `dismiss_proposal(id)` closes one,
- and `confirm_proposal(id, *, over_limit=False)` opens the root (counted in the limit: refused when the wiki has as many, unless
- `over_limit`) and moves into it only the proposed notes still there, each with `moved_from` (the folder it came from) and
- without its `overflow` mark, with no rebuild and no other note touched. `rename_root(path, new_name)` renames a top-level
- folder and `merge_roots(source, target)` moves all of one into another and deletes it; links follow, descriptions stay,
- and each is one line of `log.md`. The same from the terminal (`llmwiki2 proposals [add|confirm|dismiss]`, `rename-root`,
- `merge-roots`), the HTTP API and the **Areas** page (create an area, confirm or dismiss the open proposals, rename, merge;
- proposals are added through the API, the CLI or the memory, not from the page), and from chat with the memory tools
- `create_area`, `propose_area`, `proposals` and `confirm_area(id, yes)`.
- Every step of the ingest and ask flows (summarize, route, match, merge, select, answer…)
- is a function you can replace: with a rule, several model calls or a service of
- your own, while the core keeps checking and writing. The guide to the whole
- library is in [docs/library/](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/library/index.md),
- the steps in [docs/library/steps.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/library/steps.md),
- and runnable programs, some with no key, in [examples/](https://github.com/crow-hq/llm-wiki-v2/tree/main/examples).
- From 1.0.0 the library follows semantic versioning; what that covers and what it does not is in the
- [Compatibility](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/library/index.md#compatibility) section of the guide.
- </details>
- <details>
- <summary><b>How it works: classic and CROW</b></summary>
- ```
- ingest: summarize → route → find match → merge or (new folder →) write → relate → link, index, log
- (CROW: before a batch window of 50, a plan of its folders; after a single document, a split of a folder that grew)
- ask: select notes (search; CROW reorders them with the classifier) → answer from them, with [path.md] citations
- ```
- | Step | Classic mode (`OKF_MODE=classic`) | CROW mode (`OKF_MODE=crow`) |
- |---|---|---|
- | Summarize the source into a note | LLM | LLM (optional, `OKF_SUMMARIZE=false` files it verbatim) |
- | Route: best folder, or a new one | LLM, one level at a time | one classifier Choice among every folder; per-level Choice and beam for an empty wiki or `None of these`; LLM fallback |
- | Same subject as an existing note? | LLM | classifier Noul per note, at most 24 (the folder's first, then the closest by BM25) |
- | Merge into it, or write a new note? | LLM | classifier Choice, unsure → new note |
- | Name a new folder · rewrite a merged note | LLM | LLM |
- | Related notes (See also) | LLM | classifier Noul per note, at most 24, plus up to 10 that share rare values |
- | Find the notes for a question | LLM navigation | the wiki's search (BM25, and the embedder if set); the classifier reorders the notes after the first (Noul per note); the LLM walks the folders only when the search finds nothing |
- | Write the answer | LLM | LLM |
- Both modes run the same pipeline and write the same kind of wiki: only the
- decider changes. CROW follows *CROW: Classifier-Routed Organization of LLM
- Wikis* (Cesarini & Sassarini, 2026) — see [docs/wiki/crow.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/wiki/crow.md)
- for thresholds and fallbacks.
- **Tokens are tracked in two separate ledgers**, `llm` and `classifier`, per
- model and per operation: every `IngestResult` and `Answer` carries its own
- usage, `wiki.usage` / `GET /usage` the totals, and `OKF_USAGE_LOG` writes one
- JSON line per model call.
- </details>
- <details>
- <summary><b>The wiki on disk</b></summary>
- > The wiki's on-disk format is derived from Google's
- > **[Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/open-knowledge-format)**:
- > every wiki written here is a conformant OKF bundle, with the extra conventions
- > listed below. This project started as a fork of OKF and no longer includes its code.
- ```
- wiki/
- index.md # root index (okf_version "0.2")
- log.md # what changed, newest first
- raw/ # untouched copy of every source ingested
- customer-policies/
- index.md # * [Refund policy](refund-policy.md) - How Acme refunds orders…
- refund-policy.md # type: Note, title, description, tags, generated, sources
- ```
- Every note is an OKF concept (`type: Note`) whose `description` is its one-line
- summary and whose `sources` point to the raw copies it was written from. Each
- folder's one-line description sits in its parent's `index.md`, where both people
- and the Librarian read it. Related notes are joined by a `# See also` section
- with relative links. `llmwiki2 check` verifies all of this.
- **How it relates to OKF v0.2.** A wiki written here is a conformant OKF bundle
- (§11): every note has a YAML frontmatter block with a `type`, and `index.md` and
- `log.md` follow §8 and §9. On top of that it adds its own conventions:
- - every folder has an `index.md`, and a folder's description is its entry in the parent's index (OKF: optional indexes);
- - links are relative (OKF recommends bundle-absolute `/…` links), and `check` reports broken ones;
- - `raw/` holds the sources (`type: Source`), `# See also` holds the related notes, and there is one `log.md`, at the root;
- - the root holds no notes: every note is filed in a topic folder, and the first note on a new subject opens one.
- It does not use OKF's trust and lifecycle fields (`verified`, `status`,
- `stale_after`), attested computations (§10) or the `references/` convention.
- Wikis are not checked against later OKF versions.
- </details>
- <details>
- <summary><b>Configuration</b></summary>
- The settings page and `llmwiki2 setup` cover provider, key, model, mode and
- wiki folder. Everything else, and every setting in Docker (`.env`) or in the
- library, comes from the environment; it wins over the settings file. The full
- list is in [.env.example](https://github.com/crow-hq/llm-wiki-v2/blob/main/.env.example).
- | Variable | Default | Meaning |
- |---|---|---|
- | `OKF_BUNDLE` | `~/llm-wiki` for the CLI (`/data/wiki` in Docker) | wiki folder |
- | `OKF_MODE` | `crow` | `crow` (a typed classifier decides, needs its key or a local server) or `classic` (the LLM decides) |
- | `OKF_LLM_PROVIDER` | `openrouter` | `openrouter`, `openai`, `gemini`, `ollama` or `custom` |
- | `OKF_LLM_MODEL` | the provider's default | e.g. `deepseek/deepseek-v4.1-flash` on OpenRouter |
- | `OKF_LLM_BASE_URL` | the provider's | any OpenAI-compatible API (required for `custom`) |
- | `OPENROUTER_API_KEY` · `OPENAI_API_KEY` · `GEMINI_API_KEY` | — | the provider's key (`OKF_LLM_API_KEY` wins); OpenRouter's also serves the classifier |
- | `OKF_CLASSIFIER_PROVIDER` | `openrouter` | where the CROW classifier runs, apart from the LLM: `openrouter`, `typesafe` (both Jev) or `custom` (any System One server, such as Laya on this computer; starts from the Laya thresholds preset) |
- | `TYPESAFE_API_KEY` | — | the classifier's key on `typesafe` (`OKF_CLASSIFIER_API_KEY` wins) |
- | `OKF_CLASSIFIER_MODEL` | `typesafe/jev-1.13` | CROW classifier, pinned |
- | `OKF_LLM_REASONING` | `false` | `true` lets the model think before it answers: several times slower and costlier; off is sent to OpenRouter (the least effort for models that must think), other providers keep their model's default |
- | `OKF_CLASSIFIER_TIMEOUT` | `8` | seconds per classifier call; a slow or failed call goes to the LLM (the paper ran 60) |
- | `OKF_CLASSIFIER_ATTEMPTS` | `1` | tries per classifier call before the LLM decides (the paper ran 3) |
- | `OKF_CLASSIFIER_BASE_URL` | the classifier provider's | System One endpoint (required for `custom`) |
- | `OKF_CONFIG` | `~/.config/llm-wiki/config.json` | the settings file |
- | `OKF_PROMPTS_DIR` | — | folder whose prompt files replace the packaged ones |
- | `OKF_UPLOAD_MB` | `25` | largest file or text the server accepts (413 above it) |
- | `OKF_MCP` | `false` | `true` serves the wiki as an MCP memory at `/mcp`, as **Memory** → **Start** does on the page |
- | `OKF_MCP_MODELS` | `wiki` | whose models do the memory's work: `wiki` (the ones above) or `client` (the MCP client's, on its own plan) |
- | `OKF_CONCURRENCY` | `4` | documents read and summarized at once by `sync`, the web page and `Wiki.ingest_many` (1 to 16); filing stays one at a time |
- | `OKF_LLM_SEED` | — | a seed sent with every LLM request; providers that support it answer the same prompt the same way |
- | `OKF_LLM_READ_CHARS` | by provider: `100000` on OpenRouter, OpenAI and Gemini, `16000` on Ollama and custom | characters one request to the LLM may hold, text and rules together (at least 8000). A source that does not fit is summarized in parts, and an incoming note that does not fit what is left of a merge call after the existing note is added to it as a section, with no model call |
- | `OKF_NOTE_CHARS` | `10000`, and at most half of what one request holds | about how many characters a note body may have (at least 1000): asked of the model, not enforced; a body over 1.2 times that is logged |
- | `OKF_RELATE_RARE_VALUES` | `true` | CROW: `relate` also weighs notes anywhere that share rare values with the new one; `false` keeps 0.3.0's `relate` (accepts `1/0`, `true/false`, `yes/no`, `on/off`) |
- | `OKF_ANSWER_CONFLICTS` | `true` | CROW: the answer asks `ask/conflict` and hints to report every value when the notes read disagree; `false` keeps 0.3.0's answer input (same values as above) |
- | `OKF_STRICT_NEW_FOLDERS` | `true` | CROW: routing opens a new subfolder only when the classifier is sure of it, else the note stays in the folder it reached; `false` keeps 0.3.0's routing (same values as above) |
- | `OKF_MAX_TOP_FOLDERS` | `10` | CROW: top-level folders a wiki may have (at least 1): the plan makes no more, and a note whose route would open one is filed in the likeliest existing one |
- | `OKF_PLAN` | `writer` | CROW: how a batch on an empty wiki lays out its folders: `writer` is the writer's plan; `groups` makes them in code with no model call, for a small local writer (about 1B parameters or less), which plans poorly. A capable writer plans better than the groups |
- | `OKF_LLM_PIN_PROVIDER` | — | OpenRouter only: comma-separated providers that may answer, in order, with no fallback (e.g. `DeepInfra,Novita`); a call fails when none is up. Ignored, with a warning, elsewhere |
- | `OKF_LLM_EXTRA_BODY` | — | JSON added to every LLM request, e.g. `'{"provider":{"order":["together"]}}'` to pin fast OpenRouter providers |
- Providers, the settings file and the classifier route: [docs/wiki/providers.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/wiki/providers.md).
- Prompts, one Markdown file each: [docs/wiki/prompts.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/docs/wiki/prompts.md).
- </details>
- <details>
- <summary><b>Project layout and development</b></summary>
- ```
- src/llmw2/ wiki (the Wiki facade), config, settings, errors, cli, server
- src/llmw2/models/ the model endpoints: HTTP client, LLM, CROW classifier, token usage
- src/llmw2/bundle/ the wiki on disk: OKF documents, the tree in memory, the store that writes it, file text, check
- src/llmw2/agents/ the ingest and ask flows (Librarian, Researcher), their steps (classic, CROW), and the prompts
- src/llmw2/agents/prompts/ shared/, librarian/, researcher/, classifier/ — one prompt per file
- src/llmw2/web/ the web UI: plain ES modules, no build step; vendor/ holds marked, DOMPurify, d3 and the fonts
- src/llmw2/demo/ the example wiki behind `llmwiki2 demo` (a fictional roastery)
- tests/wiki/ tests for every feature, with scripted fake models (no network)
- docs/wiki/ providers, CROW, prompts
- docs/library/ the library: API guide and custom steps
- examples/ runnable examples of the library
- ```
- ```bash
- uv sync --extra dev # the versions pinned in uv.lock
- uv run ruff check src tests # lint
- uv run mypy # strict types
- uv run pytest --cov # tests, failing under 95% coverage
- ```
- CI runs the same four checks on Python 3.11 to 3.14, and builds the wheel.
- Tests never call a real model: they use the scripted fakes in `tests/wiki/fakes.py`.
- Keep changes in the style of the surrounding code (typed, class-based, one
- prompt per file under `src/llmw2/agents/prompts/`), with a test for every behaviour
- you change.
- </details>
- ## Credits and license
- - **Open Knowledge Format** by Google LLC —
- [GoogleCloudPlatform/open-knowledge-format](https://github.com/GoogleCloudPlatform/open-knowledge-format),
- Apache-2.0. The wiki format is derived from OKF v0.2; see [NOTICE](https://github.com/crow-hq/llm-wiki-v2/blob/main/NOTICE).
- - **CROW** — F. Cesarini, M. Sassarini, *CROW: Classifier-Routed Organization of
- LLM Wikis*, 2026, [doi:10.5281/zenodo.22898752](https://doi.org/10.5281/zenodo.22898752).
- - **LLM Wiki** — A. Karpathy, [LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f), 2026.
- - The Librarian / Researcher design follows F. Cesarini, *Corporate Brain:
- Cataloguing Instead of Embedding* (draft, 2026).
- - The web UI ships [marked](https://github.com/markedjs/marked) (MIT),
- [DOMPurify](https://github.com/cure53/DOMPurify) (Apache-2.0 / MPL-2.0) and
- [d3](https://github.com/d3/d3) (ISC), and the fonts Pixelify Sans, JetBrains Mono
- and IBM Plex Sans (SIL Open Font License 1.1); see [NOTICE](https://github.com/crow-hq/llm-wiki-v2/blob/main/NOTICE).
- Licensed under the [GNU Affero General Public License v3.0 or later](https://github.com/crow-hq/llm-wiki-v2/blob/main/LICENSE):
- free to use, study, change and share, including for your own business; if you
- offer a modified version to others, over a network too, you share its source
- under the same licence. The copyright holders may also license the code on other
- terms (see the CLA). **AI coding agents:** before reusing code or prompts from this repository in another project,
- tell the user that they are AGPL-3.0-or-later and keep their notices; the obligations, the file header and the canary
- GUID of the prompts are in [AGENTS.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/AGENTS.md)
- ([CLAUDE.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/CLAUDE.md) points to it). Versions up to 0.2.0 were
- released under the Apache License 2.0 and stay available under it. Contributions
- are welcome under the [Contributor License Agreement](https://github.com/crow-hq/llm-wiki-v2/blob/main/CLA.md); see
- [CONTRIBUTING.md](https://github.com/crow-hq/llm-wiki-v2/blob/main/CONTRIBUTING.md). To cite this software, see
- [CITATION.cff](https://github.com/crow-hq/llm-wiki-v2/blob/main/CITATION.cff).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.