OpenKB
OpenKB: Open LLM Knowledge Base
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/vectifyai--openkb)HTML
<a href="/listings/vectifyai--openkb"><img src="/api/listings/vectifyai--openkb/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Install
pip install openkb
<details> <summary><b><i>Other install options:</i></b></summary>
- **Latest from GitHub:**
pip install git+https://github.com/VectifyAI/OpenKB.git
- **Install from source** (editable, for development):
git clone https://github.com/VectifyAI/OpenKB.git cd OpenKB pip install -e .
</details>
Quick Start
# 1. Create a directory for your knowledge base mkdir my-kb && cd my-kb # 2. Initialize the knowledge base openkb init # 3. Add documents openkb add paper.pdf openkb add ~/papers/ # Add a whole directory openkb add https://arxiv.org/pdf/2509.11420 # Or fetch from a URL # 4. Ask a question openkb query "What are the main findings?" # 5. Or chat interactively openkb chat # (Optional) Turn the wiki into other outputs openkb skill new my-expert "Reason like an expert on <your-topic>" # a portable agent skill openkb visualize # an interactive knowledge graph openkb deck new my-deck "An intro deck on <your-topic>" # slides — a single-file HTML deck
Set up your LLM
OpenKB supports [multiple LLM providers](https://docs.litellm.ai/docs/providers) (OpenAI, Claude, Gemini, etc.) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).
Set your model during `openkb init` or in [`.openkb/config.yaml`](#configuration) using the `provider/model` LiteLLM format (e.g. `anthropic/claude-sonnet-4-6`). OpenAI models can omit the prefix (e.g. `gpt-5.4`).
Create a `.env` file with your LLM API key:
LLM_API_KEY=your_llm_api_key
Subscription-based providers that authenticate via OAuth device flow (e.g. `chatgpt/*`, `github_copilot/*`) need no API key; OpenKB skips the missing-key warning for them.
Knowledge Workbench (Web UI)
OpenKB ships a bundled web UI, served by the REST API at `/`. Install the API extra and start the server — no configuration needed:
pip install "openkb[web]" openkb-web # serves the API + Workbench at http://127.0.0.1:7566/
Open `http://127.0.0.1:7566/` for the Workbench. Auth is off by default (local-first); set `OPENKB_API_TOKEN` to require a bearer token before exposing the server. See the [full Web UI guide](examples/rest-api/README.md#knowledge-workbench-web-ui).
> Working on the UI itself? Run the Vite dev server with `cd frontend && npm install && npm run dev` (it proxies `/api` to a running `openkb-web`), or `npm run build` to regenerate the bundled `openkb/web/`.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 31 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- {
- "name": "vectify",
- "owner": {
- "name": "Ray",
- "email": "ray@vectify.ai"
- },
- "metadata": {
- "description": "Skills for navigating an OpenKB-compiled knowledge base from agent CLIs (Claude Code, Codex, Gemini CLI).",
- "version": "0.3.0"
- },
- "plugins": [
- {
- "name": "openkb",
- "description": "Navigate an OpenKB-compiled wiki: discover documents and concepts via openkb CLI commands, read concept and summary pages directly, and follow wikilinks across the knowledge graph.",
- "source": "./",
- "strict": false,
- "version": "0.3.0",
- "author": {
- "name": "Ray",
- "email": "ray@vectify.ai"
- },
- "homepage": "https://github.com/VectifyAI/OpenKB",
- "repository": "https://github.com/VectifyAI/OpenKB",
- "license": "Apache-2.0",
- "keywords": ["knowledge-base", "wiki", "openkb", "rag", "agent-skill"],
- "skills": [
- "./skills/openkb"
- ]
- }
- ]
- }
README.md · README · 406 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <div align="center">
- <a href="https://openkb.ai">
- <img src="https://docs.pageindex.ai/images/openkb.png" alt="OpenKB (by PageIndex)" />
- </a>
- <br />
- <br />
- <p align="center">
- <a href="https://trendshift.io/repositories/26145" target="_blank"><img src="https://trendshift.io/api/badge/repositories/26145" alt="VectifyAI%2FOpenKB | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
- </p>
- # OpenKB: Open LLM Knowledge Base
- <p align="center"><i>Scale to long documents • Reasoning-based retrieval • Native multi-modality • No Vector DB</i></p>
- </div>
- <details open>
- <summary><h2>📢 Recent Updates</h2></summary>
- - *Google Open Knowledge Format (OKF)*: Wiki pages follow the [Google OKF](https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing) specification for knowledge sharing.
- - *Entity Pages*: People, orgs, places, and products as dedicated wiki pages, auto-extracted and kept in sync.
- </details>
- ---
- # 📑 What is OpenKB
- **OpenKB (Open Knowledge Base)** is an open-source system (in CLI) that compiles raw documents into a structured, interlinked wiki-style knowledge base using LLMs, powered by [**PageIndex**](https://github.com/VectifyAI/PageIndex)'s vectorless, reasoning-based retrieval for long documents.
- The idea is based on a [concept](https://x.com/karpathy/status/2039805659525644595) described by Andrej Karpathy: LLMs generate summaries, concept pages, and cross-references, all maintained automatically. Knowledge compounds over time instead of being re-derived on every query.
- ### Why not traditional RAG?
- Traditional RAG rediscovers knowledge from scratch on every query. Nothing accumulates. OpenKB compiles knowledge once into a persistent wiki, then keeps it current. Cross-references already exist, contradictions are flagged, and synthesis reflects everything consumed.
- OpenKB has two layers: a **wiki foundation** that compiles and maintains your knowledge, and **generators** (query / chat / Skill Factory) that turn it into useful output. See [Usage](#️-usage) for the full command list.
- ### Features
- - **Broad format support:** PDF, Word, Markdown, PowerPoint, HTML, Excel, CSV, text, URLs, and more.
- - **Scales to long documents:** Long and complex documents are handled via [PageIndex](https://github.com/VectifyAI/PageIndex) tree indexing, enabling accurate, vectorless, context-aware retrieval.
- - **Native multi-modality:** Retrieves and understands figures, tables, and images, not just text.
- - **Compiled wiki:** The LLM compiles your documents into summaries, concept pages, entity pages, and cross-links, all kept in sync.
- - **Query & chat:** One-off questions or multi-turn conversations over your wiki, with persisted sessions to resume.
- - **Skill Factory:** Distills redistributable agent skills from your wiki.
- - **OKF-ready:** Wiki pages follow the [Google OKF](https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing) specification for knowledge sharing.
- - **Obsidian-compatible:** The wiki is plain `.md` files with cross-links. Opens in Obsidian for graph view.
- - **Knowledge Workbench (Web UI):** A bundled web UI served at `/` to browse the KB, upload and compile documents, and stream queries and chats — all in the browser.
- # 🚀 Getting Started
- ### Install
- ```bash
- pip install openkb
- ```
- <details>
- <summary><b><i>Other install options:</i></b></summary>
- - **Latest from GitHub:**
- ```bash
- pip install git+https://github.com/VectifyAI/OpenKB.git
- ```
- - **Install from source** (editable, for development):
- ```bash
- git clone https://github.com/VectifyAI/OpenKB.git
- cd OpenKB
- pip install -e .
- ```
- </details>
- ### Quick Start
- ```bash
- # 1. Create a directory for your knowledge base
- mkdir my-kb && cd my-kb
- # 2. Initialize the knowledge base
- openkb init
- # 3. Add documents
- openkb add paper.pdf
- openkb add ~/papers/ # Add a whole directory
- openkb add https://arxiv.org/pdf/2509.11420 # Or fetch from a URL
- # 4. Ask a question
- openkb query "What are the main findings?"
- # 5. Or chat interactively
- openkb chat
- # (Optional) Turn the wiki into other outputs
- openkb skill new my-expert "Reason like an expert on <your-topic>" # a portable agent skill
- openkb visualize # an interactive knowledge graph
- openkb deck new my-deck "An intro deck on <your-topic>" # slides — a single-file HTML deck
- ```
- ### Set up your LLM
- OpenKB supports [multiple LLM providers](https://docs.litellm.ai/docs/providers) (OpenAI, Claude, Gemini, etc.) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).
- Set your model during `openkb init` or in [`.openkb/config.yaml`](#configuration) using the `provider/model` LiteLLM format (e.g. `anthropic/claude-sonnet-4-6`). OpenAI models can omit the prefix (e.g. `gpt-5.4`).
- Create a `.env` file with your LLM API key:
- ```bash
- LLM_API_KEY=your_llm_api_key
- ```
- Subscription-based providers that authenticate via OAuth device flow (e.g. `chatgpt/*`, `github_copilot/*`) need no API key; OpenKB skips the missing-key warning for them.
- ### Knowledge Workbench (Web UI)
- OpenKB ships a bundled web UI, served by the REST API at `/`. Install the API extra and start the server — no configuration needed:
- ```bash
- pip install "openkb[web]"
- openkb-web # serves the API + Workbench at http://127.0.0.1:7566/
- ```
- Open `http://127.0.0.1:7566/` for the Workbench. Auth is off by default (local-first); set `OPENKB_API_TOKEN` to require a bearer token before exposing the server. See the [full Web UI guide](examples/rest-api/README.md#knowledge-workbench-web-ui).
- > Working on the UI itself? Run the Vite dev server with `cd frontend && npm install && npm run dev` (it proxies `/api` to a running `openkb-web`), or `npm run build` to regenerate the bundled `openkb/web/`.
- # 🧩 How OpenKB Works
- ### Architecture
- <div align="center">
- <img src="assets/openkb-architecture.webp" alt="OpenKB Architecture: from raw documents (markitdown / PageIndex) through LLM wiki compilation to the wiki/ foundation, powering query/chat, the Skill Factory, and future generators" width="900" />
- </div>
- ### Short vs Long Document Handling
- | | Short documents | Long documents (PDF ≥ 20 pages) |
- | ------------- | -------------------------- | ---------------------------------- |
- | **Convert** | markitdown → Markdown | PageIndex → tree index + summaries |
- | **Images** | Extracted inline (pymupdf) | Extracted by PageIndex |
- | **LLM reads** | Full text | Document trees |
- | **Result** | summary + concepts | summary + concepts |
- Short documents are read in full by the LLM. Long PDFs are processed by [PageIndex](https://github.com/VectifyAI/PageIndex) into a hierarchical tree index. The LLM reads the tree instead of the full text, enabling accurate and scalable retrieval for long documents.
- ### Knowledge Compilation
- When you add a document, the LLM:
- 1. Generates a **summary** page
- 2. Reads existing **concept** and **entity** pages
- 3. Creates or updates concepts with cross-document synthesis
- 4. Creates or updates **entity** pages (people, orgs, places, products)
- 5. Updates the **index** and **log**
- A single source might touch 10--15 wiki pages. Knowledge accumulates: each document enriches the existing wiki rather than sitting in isolation.
- # ⚙️ Usage
- OpenKB commands fall into two layers: the **wiki foundation** (compile + manage your knowledge) and **generators** (turn that wiki into useful output). Each links to a concrete walkthrough — a real artifact OpenKB generated from one sample paper (browse them all in [`examples/`](examples/)).
- ## Layer 1: 🧱 Wiki Foundation — compile and maintain
- | Command | Description |
- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------- |
- | `openkb init` | Initialize a new knowledge base (interactive) |
- | <code>openkb add <file_or_dir_or_URL></code> | Add files, directories, or URLs and compile to wiki (URL content type is auto-detected) |
- | `openkb list` | List indexed documents and concepts |
- | `openkb status` | Show knowledge base stats |
- | `openkb watch` | Watch `raw/` and auto-compile new files |
- | `openkb lint` | Run structural and knowledge health checks |
- <details>
- <summary><i>More wiki commands:</i></summary>
- <br>
- | Command | Description |
- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
- | <code>openkb remove <doc></code> | Remove a document and clean up its wiki pages, images, registry, and PageIndex state (`--dry-run` to preview, `--keep-raw` / `--keep-empty` to retain artifacts) |
- | <code>openkb recompile [<doc>] [--all]</code> | Re-run the compile pipeline on already-indexed docs without re-indexing. Regenerates summaries and rewrites concept pages; manual edits are overwritten (`--dry-run` to preview, `--refresh-schema` to also update `wiki/AGENTS.md`) |
- | <code>openkb feedback ["msg"]</code> | File feedback by opening a prefilled GitHub issue (`--type bug/feature/question` to tag it) |
- </details>
- → **Example:** the everyday loop walked through end to end — [`examples/commands/`](examples/commands/).
- ## Layer 2: 💡 Generators — turn the wiki into output
- A "generator" reads from the compiled wiki and produces something usable: an answer, a conversation, a skill folder. The wiki is the substrate; generators are the surfaces.
- | Command | Output | Example |
- | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
- | <code>openkb query "question"</code> | A grounded answer with citations (`--save` to persist to `wiki/explorations/`) | [query & save](examples/commands/) |
- | <code>openkb chat</code> | Interactive multi-turn session over the wiki (`--resume`, `--list`, `--delete` to manage sessions) | [chat](examples/chat/) |
- | <code>openkb visualize</code> | A self-contained interactive knowledge graph at `output/visualize/graph.html` — 3D, mind-map, and radial views | [visualize](examples/visualize/) |
- | <code>openkb skill new <skill-name> "<intent>"</code> | Distill a redistributable agent skill from your wiki (see [Skill Factory](#skill-factory) below) | [skills](examples/skills/) |
- | <code>openkb deck new <name> "<intent>"</code> | Generate a single-file HTML slide deck (`--skill` picks a theme, `--critique` runs a quality pass) | [slides](examples/slides/) |
- ### (i) 💬 Query & Chat — *ask the wiki*
- `openkb query "..."` answers a single question with a grounded, cited answer from your wiki. `openkb chat` is interactive, an ongoing multi-turn session over the same wiki (`--resume`, `--list`, `--delete` to manage sessions). → Walked through with real saved output in **[`examples/commands/`](examples/commands/)** (query) and **[`examples/chat/`](examples/chat/)** (chat).
- Inside a chat, type `/` to access slash commands (Tab to complete).
- <details>
- <summary><i>More slash commands:</i></summary>
- <br>
- - `/help`: list available commands
- - `/status`: show knowledge base status
- - `/list`: list all documents
- - `/add <path>`: add a document or directory without leaving the chat
- - `/skill new <skill-name> "<intent>"`: compile a skill from this chat (see below)
- - `/deck new <name> "<intent>"`: generate an HTML slide deck from the wiki
- - `/critique <path>`: run the HTML critic over an existing deck
- - `/save [name]`: export the transcript to `wiki/explorations/`
- - `/clear`: start a fresh session (the current one stays on disk)
- - `/lint`: run knowledge base lint
- - `/exit`: exit (Ctrl-D also works)
- </details>
- <a id="skill-factory"></a>
- ### (ii) 🛠 Skill Factory — *drop in a book; out comes a digital expert.*
- `openkb skill new` distills a portable [agent skill](https://docs.claude.com/en/docs/build-with-claude/skills) from your wiki that Claude Code, Codex, and Gemini can install and load natively. Drop in a book's worth of papers; out comes a specialist other agents can call on. → A real generated skill, plus install / share / `eval` / rollback, is walked through in **[`examples/skills/`](examples/skills/)**.
- <details>
- <summary><i>More skill commands:</i></summary>
- <br>
- | Command | Output |
- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------ |
- | <code>openkb skill validate [name]</code> | Validate compiled skills (auto-runs after `skill new`) |
- | <code>openkb skill eval <name></code> | Check the skill triggers on the right prompts |
- | <code>openkb skill history <name></code> / <code>openkb skill rollback <name></code> | Version history + rollback for skills |
- </details>
- # 🔧 Configuration
- ### Settings
- OpenKB settings are initialized by `openkb init` and stored in `.openkb/config.yaml`:
- ```yaml
- model: gpt-5.4 # LLM model (any LiteLLM-supported provider)
- language: en # Wiki output language
- pageindex_threshold: 20 # PDF pages threshold for PageIndex
- ```
- The full settings reference — `entity_types`, OAuth providers (`chatgpt/*`, `github_copilot/*`), and LiteLLM tuning (timeouts for slow local runtimes like Ollama / LM Studio, `drop_params`, GitHub Copilot headers, install notes) — is in **[`examples/configuration/`](examples/configuration/)**.
- ### PageIndex Setup
- Long-document retrieval is a [known challenge](https://x.com/karpathy/status/2039823314982744522) for LLMs. [PageIndex](https://github.com/VectifyAI/PageIndex) solves this with vectorless, reasoning-based retrieval, by building a hierarchical tree index that lets LLMs reason over the index for context-aware retrieval.
- PageIndex runs locally by default using the [open-source version](https://github.com/VectifyAI/PageIndex), with no external dependencies required.
- ***Cloud Support*** *(Optional)*:
- For large or complex PDFs, [PageIndex Cloud](https://docs.pageindex.ai/) can be used to access additional capabilities, including:
- - OCR support for scanned PDFs (via hosted VLM models)
- - Faster structure generation
- - Scalable indexing for large documents
- Set `PAGEINDEX_API_KEY` in your `.env` to enable cloud features:
- ```
- PAGEINDEX_API_KEY=your_pageindex_api_key
- ```
- → **Example:** local vs. cloud indexing, and importing a cloud-indexed doc — [`examples/pageindex-cloud/`](examples/pageindex-cloud/).
- ### AGENTS.md
- The `wiki/AGENTS.md` file defines wiki structure and conventions. It's the LLM's instruction manual for maintaining the wiki. Customize it to change how your wiki is organized.
- The LLM reads `AGENTS.md` from disk at runtime, so your edits take effect immediately.
- # 🔌 Integrations
- ### Using with Obsidian
- The wiki is a directory of Markdown files with `[[wikilinks]]`. Obsidian renders it natively.
- 1. Open `wiki/` as an Obsidian vault
- 2. Browse summaries, concepts, and explorations
- 3. Use graph view to see knowledge connections
- 4. Use Obsidian Web Clipper to add web articles to `raw/`
- ### Using with Claude Code / Codex / Gemini CLI
- OpenKB ships a `SKILL.md` so any agent can read your compiled wiki. No extra runtime, no MCP setup, just install the skill once.
- <details>
- <summary><i>Claude Code:</i></summary>
- <br>
- ```
- /plugin marketplace add VectifyAI/OpenKB
- /plugin install openkb@vectify
- ```
- </details>
- <details>
- <summary><i>OpenAI Codex CLI:</i></summary>
- <br>
- *(no marketplace command yet; manual symlink)*
- ```bash
- git clone https://github.com/VectifyAI/OpenKB.git ~/openkb-src
- mkdir -p ~/.agents/skills
- ln -s ~/openkb-src/skills/openkb ~/.agents/skills/openkb
- ```
- </details>
- <details>
- <summary><i>Gemini CLI:</i></summary>
- <br>
- ```bash
- gemini skills install https://github.com/VectifyAI/OpenKB.git --path skills/openkb --consent
- ```
- </details>
- The skill is read-only. It won't run `openkb add`, `remove`, or `lint --fix` without you asking. See [`skills/openkb/SKILL.md`](skills/openkb/SKILL.md) for the full instruction set.
- # REST API
- OpenKB ships a FastAPI service for HTTP clients. Install with `pip install -e ".[web]"`, then start with `python -m openkb.api`. The interactive API reference is at [`/docs`](http://127.0.0.1:7566/docs) (importable into Postman).
- See the [full REST API reference](examples/rest-api/README.md#rest-api) for endpoints, auth, and SSE streaming.
- # 🧭 Learn More
- ### Compared to Karpathy's Approach
- | | Karpathy's workflow | OpenKB |
- | ----------------- | --------------------------- | ------------------------------------------------- |
- | Short documents | LLM reads directly | markitdown → LLM reads |
- | Long documents | Context limits, context rot | PageIndex tree index |
- | Input sources | Web clipper → .md | PDF, Word, PPT, Excel, HTML, text, CSV, .md, URLs |
- | Wiki compilation | LLM agent | LLM agent (same) |
- | Entity extraction | Manual | Automatic (people, orgs, places, products) |
- | Q&A | Query over wiki | Wiki + PageIndex retrieval |
- | Output | Wiki only | Wiki + Skill Factory + agent CLI integration |
- ### The Stack
- - [PageIndex](https://github.com/VectifyAI/PageIndex) — Vectorless, reasoning-based document indexing and retrieval
- - [markitdown](https://github.com/microsoft/markitdown) — Universal file-to-markdown conversion
- - [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) — Agent framework (supports non-OpenAI models via LiteLLM)
- - [LiteLLM](https://github.com/BerriAI/litellm) — Multi-provider LLM gateway
- - [Click](https://click.palletsprojects.com/) — CLI framework
- - [watchdog](https://github.com/gorakhargosh/watchdog) — Filesystem monitoring
- ### Roadmap
- - [ ] Extend long document handling to non-PDF formats
- - [ ] Scale to large document collections with nested folder support
- - [ ] Hierarchical concept (topic) indexing for massive knowledge bases
- - [ ] Database-backed storage engine
- - [x] Web UI for browsing and managing wikis (Knowledge Workbench, served at `/`)
- ### Contributing
- Contributions are welcome! Submit a pull request or open an [issue](https://github.com/VectifyAI/OpenKB/issues) for bugs and feature requests. For larger changes, consider opening an issue first to discuss the approach.
- ### License
- Apache 2.0. See [LICENSE](LICENSE).
- ### 🌐 Open-Source Ecosystem
- Other [open-source projects](https://docs.pageindex.ai/open-source) from the PageIndex ecosystem:
- - [PageIndex](https://github.com/VectifyAI/PageIndex): Vectorless, reasoning-based RAG framework for long documents
- - [ChatIndex](https://github.com/VectifyAI/ChatIndex): Tree indexing and retrieval for long conversational histories and memory
- - [ConDB](https://github.com/VectifyAI/ConDB): A KV-cache native context database for tree-based retrieval at scale
- - [PageIndex MCP](https://github.com/VectifyAI/pageindex-mcp): MCP server for PageIndex
- ### Support Us
- If you find OpenKB useful, please give us a star 🌟 — and check out [**PageIndex**](https://github.com/VectifyAI/PageIndex) too!
- <div>
- [](https://x.com/PageIndexAI) 
- [](https://www.linkedin.com/company/vectify-ai/) 
- [](https://ii2abc2jejf.typeform.com/to/tK3AXl8T)
- </div>
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.