Skip to content
Back to search

xerj

XERJ: the search engine for AI agents. Plugins that connect Claude Code to a local XERJ node.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/xerj-org/xerj with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/xerj-org/xerj'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/xerj-org--xerj/badge.svg)](/listings/xerj-org--xerj)

HTML

<a href="/listings/xerj-org--xerj"><img src="/api/listings/xerj-org--xerj/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

curl -fsSL https://xerj.org/get | sh

Windows PowerShell:

irm https://xerj.org/get.ps1 | iex

Nix (builds from source; `nix develop` opens a shell with the Rust toolchain):

nix run github:xerj-org/xerj -- --version

One static binary, no JVM, no dependencies. Prebuilt for Linux, macOS and Windows on x86-64 and arm64. The x86-64 builds target x86-64-v2 (SSE4.2: Intel Nehalem 2008 and later, AMD Bulldozer and later) and use AVX2 and FMA at runtime when the CPU has them, so a pre-Haswell machine runs the same binary. It speaks the Elasticsearch API, so existing clients, dashboards and tooling work against it unchanged.

First commands after install (the installer prints where `xerj` landed; add it to your PATH if needed): `xerj --insecure --data-dir ~/xerj-data &`, wait until `http://localhost:9200` responds, then `xerj autoindex ~/my-project`. See [Index a folder](#index-a-folder).

For a host with no runtime internet access, follow the [air-gapped deployment recipe](./docs/recipes/air-gapped-deployment.md). The default lexical embedder is offline; neural mode needs the three model files staged locally before the first semantic operation.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 25 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "xerj",
  3. "description": "XERJ: the search engine for AI agents. Plugins that connect Claude Code to a local XERJ node.",
  4. "owner": {
  5. "name": "xerj-org",
  6. "email": "git@xerj.org",
  7. "url": "https://xerj.org"
  8. },
  9. "plugins": [
  10. {
  11. "name": "xerj",
  12. "source": "./plugins/xerj",
  13. "description": "XERJ MCP server: cited search over your code, docs, specs and logs, plus Corpus Hub packs.",
  14. "category": "search",
  15. "homepage": "https://xerj.org/for-agents"
  16. },
  17. {
  18. "name": "xerj-knowledge",
  19. "source": "./plugins/xerj-knowledge",
  20. "description": "Local document search for Claude Cowork and Claude Code: your folders, notes, PDFs, Office files and mail exports, answered with the file and page. Files stay on your machine.",
  21. "category": "productivity",
  22. "homepage": "https://xerj.org/for-agents"
  23. }
  24. ]
  25. }
README.md · README · 874 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <h1 align="center">XERJ</h1>
  2. <div align="center">
  3. [![XERJ Query-1](https://img.shields.io/badge/XERJ%20Query--1-new%20in%20rc.101-ffc400?style=flat-square)](https://xerj.org/xerj-query-1)
  4. [![CI](https://img.shields.io/github/actions/workflow/status/xerj-org/xerj/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/xerj-org/xerj/actions/workflows/ci.yml)
  5. [![Release](https://img.shields.io/github/v/release/xerj-org/xerj?style=flat-square&label=release)](https://github.com/xerj-org/xerj/releases)
  6. [![License](https://img.shields.io/badge/license-Apache--2.0-blue?style=flat-square)](./LICENSE)
  7. [![Stars](https://img.shields.io/github/stars/xerj-org/xerj?style=flat-square)](https://github.com/xerj-org/xerj/stargazers)
  8. [![Rust](https://img.shields.io/badge/built%20with-Rust-000000?style=flat-square&logo=rust)](https://www.rust-lang.org)
  9. [![Discussions](https://img.shields.io/github/discussions/xerj-org/xerj?style=flat-square&label=discussions)](https://github.com/xerj-org/xerj/discussions)
  10. [![Discord](https://img.shields.io/badge/Discord-join-5865F2?style=flat-square&logo=discord&logoColor=white)](https://discord.com/invite/yetXFusXfs)
  11. [![X](https://img.shields.io/badge/X-%40xerj__org-000000?style=flat-square&logo=x&logoColor=white)](https://x.com/xerj_org)
  12. [![Reference coding](https://img.shields.io/badge/reference%20coding-2.7%C3%97%20fewer%20output%20tokens-ff8c00?style=flat-square)](https://xerj.org/case-studies/reference-coding.html)
  13. [![Autoindex](https://img.shields.io/badge/autoindex-zero%20config-ff8c00?style=flat-square)](https://xerj.org/docs/cli.html)
  14. [![Search](https://img.shields.io/badge/search-BM25%20%2B%20kNN%20%2B%20hybrid-8957e5?style=flat-square)](https://xerj.org/docs/queries.html)
  15. [![Agent memory](https://img.shields.io/badge/agent%20memory-%2F__memory-00b3a4?style=flat-square)](https://xerj.org/docs/recipes/agentic-memory.html)
  16. [![MCP](https://img.shields.io/badge/MCP-native-1f6feb?style=flat-square)](https://xerj.org/llms.txt)
  17. [![llms.txt](https://img.shields.io/badge/llms.txt-agent%20ready-6f42c1?style=flat-square)](https://xerj.org/llms.txt)
  18. [![ES-YAML conformance](https://img.shields.io/badge/ES--YAML%20conformance-0%20failed%20(1448%2F1458)-brightgreen?style=flat-square)](https://xerj.org/benchmarks)
  19. [![Single binary](https://img.shields.io/badge/single%20binary-no%20JVM-orange?style=flat-square)](https://xerj.org/docs/install.html)
  20. [![Default embedder](https://img.shields.io/badge/default%20embedder-lexical%2C%20offline-teal?style=flat-square)](https://xerj.org/docs/vectors.html)
  21. [![Neural](https://img.shields.io/badge/neural%20embedder-opt--in%2C%20downloads%20~90%20MB-9c6ade?style=flat-square)](https://xerj.org/docs/vectors.html)
  22. <a href="https://trendshift.io/repositories/116361?utm_source=repository-badge&amp;utm_medium=badge&amp;utm_campaign=badge-repository-116361" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/116361" alt="xerj-org%2Fxerj | Trendshift" width="250" height="55"/></a>
  23. <a href="https://trendshift.io/repositories/116361#achievements" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/116361/daily?language=Rust" alt="XERJ | #1 Rust Repository Of The Day | Trendshift" width="250" height="55"/></a>
  24. <a href="https://trendshift.io/repositories/116361#achievements" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/116361/weekly?language=Rust" alt="XERJ | #1 Rust Repository Of The Week | Trendshift" width="250" height="55"/></a>
  25. [English](./README.md) | [简体中文](./README.zh-CN.md) | [日本語](./README.ja-JP.md) | [العربية](./README.ar.md) | [Español](./README.es.md) | [한국어](./README.ko.md) | [فارسی](./README.fa.md)
  26. </div>
  27. <p align="center">An AI-native search engine in one Rust binary.</p>
  28. <p align="center">
  29. <a href="https://xerj.org/xerj-query-1"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/media/cards/card-1-dark.png"><img src="docs/media/cards/card-1-light.png" width="400" alt="XERJ Query-1, AI model and AI search. Answers over facts you wrote, and shows the facts it used. Runs offline, no embeddings, no GPU. Wrong answers on 160 questions: 1, versus 9 to 15 for RAG with Opus 5.5. 220 ms p50 on CPU. $42 of CPU per million questions. AI search: BM25, vector and hybrid in one query, one binary. Link: xerj.org/xerj-query-1"></picture></a>
  30. <a href="#the-engine-underneath"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/media/cards/card-2-dark.png"><img src="docs/media/cards/card-2-light.png" width="400" alt="Why XERJ, built for agents: search, memory and a familiar API in one Rust binary. Reference coding: 2.7 times fewer output tokens than a grep-driven agent, same 16 of 16 solved. Agent memory and MCP: /_memory, works with Claude Code, Cursor and any MCP client. Elasticsearch replacement: 0 failed on ES-YAML conformance (1448 of 1458), same API, no JVM. Link: the engine underneath"></picture></a>
  31. </p>
  32. <p align="center"><b>The search engine for AI, now with XERJ Query-1: a model that turns your question into a small program over facts you have written.</b></p>
  33. <p align="center">
  34. <a href="https://xerj.org/xerj-query-1">XERJ Query-1</a> ·
  35. <a href="https://xerj.org/blog/xerj-query-ai-model">Blog post</a> ·
  36. <a href="https://github.com/xerj-org/xerj/releases/tag/v1.0.0-rc.101">Release rc.101</a> ·
  37. <a href="https://xerj.org/llms.txt">llms.txt</a> ·
  38. <a href="https://xerj.org/docs/">Docs</a> ·
  39. <a href="https://xerj.org/donate">Donate</a>
  40. </p>
  41. ## XERJ Query-1, new in rc.101
  42. Most AI search finds text in milliseconds and then asks a language model to write the answer. The model is what
  43. makes a RAG answer good, and it is also where the seconds and the per-question cost come from.
  44. XERJ Query-1 works the other way. It reads your question and writes a short program (look this up, compare these,
  45. count those), and the engine runs that program over facts you stored. Lookup and comparison steps point at a stored
  46. fact and its quoted span; counts return a program hash, not the records. If the data cannot support an answer you get
  47. `UNKNOWN`; if two facts disagree you get `CONFLICT`. It runs on four CPU cores, offline, with no embeddings to build
  48. and no GPU.
  49. | | XERJ Query-1 | the usual way | measured on |
  50. |---|---|---|---|
  51. | Question to answer | **220 ms** (p50) | 1.5 to 2.3 s for Opus 5.5 to write a RAG answer (API time per call, our command-line client's start-up not counted), after a search of 0.8 to 106 ms | same 160 questions, rc.101 release, 4 model threads, laptop CPU |
  52. | Wrong answers | **1 of 160** (124 right, 35 declined) | 9 to 15 (RAG with Opus 5.5), 6 (Opus 5.5 reading the whole store), 5 (Opus 5.5 given each question's records) | our 160-question test, drawn from the model's development set; XERJ Query-1 was given each question's records |
  53. | Cost per million questions | **$42** of CPU | $13,400 (RAG) to $694,000 (Opus re-reading the whole store per question, uncached) | same test, Opus 5.5 list prices; CPU at AWS c7g list price, 4.1 CPU-s per question on a shared ARM server (0.9 on a laptop) |
  54. | Build a search index of 10 GB of text | **17 min** (BM25, measured) | an estimated 62 to 97 hours to embed it with bge-small on CPU | English Wikipedia (`wikimedia/wikipedia` 20231101.en, first 22 of 41 files, 3,299,578 articles); XERJ Query-1 does not use this index |
  55. | Correct a fact | **2.3 ms** (one write; the next answer uses it) | re-embed the changed chunk (about 30 ms with bge-small), and the whole store (19.6 s for our 5,000-record test store) when the embedding model changes | rc.101 release, laptop |
  56. Retrieval quality (R@k, nDCG@k) on BEIR, on SearchTome and on our 160-question test, the latency split and every data
  57. set: [benchmarks/query-1-2026-11](./benchmarks/query-1-2026-11/README.md).
  58. The model is off by default; without the flag there is no `/_facts` route and the MCP tool list is unchanged. rc.101
  59. upgrades the data directory to format 4 on first write, and rc.100 cannot open it again. What it does not do yet,
  60. measured: sums and totals (a sum question can come back as a member count), and large fact sets (on a 95,000-fact
  61. namespace asks took 0.82 s at the median and all 20 test questions were declined). It declines more often than an LLM
  62. would guess. The numbers and their caveats: [blog post](https://xerj.org/blog/xerj-query-ai-model),
  63. [capabilities](./docs/XERJ_QUERY_1.md).
  64. ### Try it with the model
  65. ```sh
  66. curl -fsSL https://xerj.org/get | sh -s -- --with-model
  67. ```
  68. Verify: the installer ends by printing the start line with the model directory it unpacked.
  69. ```sh
  70. xerj --facts-model-dir ~/.local/share/xerj/model --insecure --data-dir ~/xerj-data &
  71. curl -s localhost:9200/_facts | grep -o '"state":"loaded"'
  72. ```
  73. Verify: prints `"state":"loaded"`. In the first few seconds after start it prints nothing, because the model is still loading (`"not_loaded"`); run it again.
  74. Write a fact and ask about it: [docs/MODEL_QUICKSTART.md](./docs/MODEL_QUICKSTART.md).
  75. From an agent with no shell, the same calls work over HTTP and the MCP tool `xerj_facts_ask`;
  76. see [llms.txt](https://xerj.org/llms.txt).
  77. XERJ is research in the open. The model was trained on donated GPU time; if it is useful to
  78. you, [xerj.org/donate](https://xerj.org/donate) funds the next models. The engine is open
  79. source; the model ships as a binary bundle, and its training and evaluation data are not
  80. public yet.
  81. ---
  82. ## The engine underneath
  83. **Local-first search engine and MCP server for AI agents.** One Rust binary indexes code, docs,
  84. logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search and agent
  85. memory, behind an Elasticsearch-compatible API. Works with Claude Code, Cursor and any MCP client.
  86. XERJ is a local AI search engine that indexes any folder automatically, so your
  87. coding agent stops burning tokens reading files one by one and pulls the exact code it
  88. needs instead. Reference coding is its main use case and the clearest win: point an agent
  89. at a task and it downloads the open-source repos closest to it, indexes them, and reuses
  90. how they solved the problem before writing its own code. In a controlled study that cut a
  91. coding agent's output tokens by 2.7x at the same 16/16 solve rate
  92. ([case study](https://xerj.org/case-studies/reference-coding.html)), and people report
  93. roughly 5x in everyday work
  94. ([field reports](./user-feedback/11-reference-coding-field-reports/2026-08-11-token-savings-reports.md)).
  95. It is enough for a smaller, cheaper model to out-code a pricier one working from memory,
  96. while you spend less time in fix loops. Try it now with a one-prompt install:
  97. ```text
  98. Install XERJ (docs: https://xerj.org/llms.txt), index this project's sources, and set up
  99. reference coding: clone and index the open-source repos closest to what we're building,
  100. and search how they solved a problem before writing code.
  101. ```
  102. Or run it yourself:
  103. ```sh
  104. curl -fsSL https://xerj.org/get | sh
  105. xerj --insecure --data-dir ~/xerj-data &
  106. xerj init # wire XERJ into Claude Code/Cursor (MCP + skill), one command
  107. xerj autoindex ~/my-project
  108. xerj search "how do we rotate the WAL segment" # ranked passages, file:line
  109. xerj def "euler_to_rotationmatrix" # go-to-definition: file:line + signature
  110. xerj gain # what it did for you — counted, not estimated
  111. ```
  112. For several IDs or questions at once, use `xerj batch-search --input queries.json`
  113. or MCP `xerj_batch_search`: [ordered results, citations and per-item errors](docs/recipes/batch-search.md).
  114. Keep that data directory outside any folder you will autoindex. `data/` is not in the
  115. built-in ignore list, so `--data-dir ./data` inside a tree you then index makes the node
  116. index its own storage.
  117. At startup, `Data dir` shows the resolved absolute path, `Data source` identifies
  118. the flag, selected config file or default, and `Data status` says whether this
  119. launch created the directory. A new directory can make an older node's indices
  120. or owner account appear missing: restart with the previous `--data-dir` or
  121. `--config`. XERJ does not discover or migrate other data directories automatically.
  122. Windows and a hand-verified install are under [Install by hand](#install-by-hand).
  123. That `--insecure` flag is a trust boundary, not a formality. It turns off TLS and API-key
  124. auth, so every caller that can reach the port is a superuser who can read and delete every
  125. index, including every `/_memory` namespace. It is meant for one user on a dev machine: a
  126. fresh node binds `127.0.0.1` only, and a cleartext bind on any other interface refuses to
  127. start unless you explicitly allow it. The full model is in
  128. [docs/SECURITY_MODEL.md](./docs/SECURITY_MODEL.md).
  129. ## Join the community
  130. You join XERJ by running it and sending back one short field report on what it was like: a
  131. single commit or PR. That report is the membership card, and it is the only thing the
  132. project asks in return for a free binary (the engine source is Apache-2.0; the XERJ Query-1 crates in release
  133. binaries since rc.101 are proprietary). It is what keeps the engine improving
  134. for whoever runs it next.
  135. Humans and AI agents are both welcome; this is deliberately not an agents-only project.
  136. Whether you indexed a repo over lunch or an agent installed XERJ mid-session in someone
  137. else's codebase, the ask is the same: one short report saying what you pointed it at, what
  138. you used it for, and your honest verdict. A negative verdict is a good report. If something
  139. broke, that is a separate issue or fix on top.
  140. Field-report PRs are CLA-exempt, so they merge without a signature. The path that works on
  141. any build is a plain `gh pr create` adding exactly one markdown file under
  142. [`user-feedback/16-agent-field-reports/`](./user-feedback/16-agent-field-reports), with
  143. nothing else in the diff. On a build that has it, `xerj feedback` drafts the report and
  144. fills in version, OS and what was indexed, and `xerj feedback --open-pr` opens the PR for
  145. you. Check `xerj --help` for whether your binary has the command.
  146. ## The measured reason: reference coding
  147. An agent that greps pulls whole files into its context, up to 1.06M input tokens on one
  148. corpus in our measurements, and still has to read them. An agent that queries XERJ gets the
  149. passage. Measured end to end on code the model had not memorised
  150. ([case study](https://xerj.org/case-studies/reference-coding.html): 8 tasks, 4 languages,
  151. 16 runs per arm, real `claude -p` token counts):
  152. | | output tokens | cost | solved |
  153. |---|---:|---:|---:|
  154. | from memory only | 260,916 | $11.18 | 11/16 |
  155. | grep-driven agent | 26,477 | $3.27 | 16/16 |
  156. | XERJ | 9,982 | $1.58 | 16/16 |
  157. That is 2.7x fewer output tokens than grep and 26x fewer than memory alone at the same solve
  158. rate, and up to 278x fewer on a single Java task. In real development, users report roughly
  159. 5x fewer tokens end to end
  160. ([field reports](./user-feedback/11-reference-coding-field-reports/2026-08-11-token-savings-reports.md)).
  161. The value is gated by memorisation: it wins on private, internal, niche or post-cutoff code,
  162. and is neutral to harmful on popular public libraries the model already knows. The honest
  163. limits are in the [case study](https://xerj.org/case-studies/reference-coding.html).
  164. ## Feed it any folder
  165. Reference coding is one use of the same primitive: `xerj autoindex <folder>` makes an agent
  166. know a corpus instead of grepping it. One command indexes code, docs, logs, PDFs, SQLite and
  167. awkward CSVs into typed, queryable indices for search, RAG, security audits and agent
  168. memory, with no schema to write and no pipeline to configure.
  169. ```sh
  170. xerj --insecure --data-dir ~/xerj-data & # start it
  171. xerj autoindex ~/my-project # point it at anything
  172. ```
  173. `--insecure` again means no TLS and no auth — every caller a superuser — so this is the
  174. same one-user-on-a-dev-machine boundary as above ([security model](./docs/SECURITY_MODEL.md)).
  175. XERJ sniffs every file, works out what it is, and creates one index per dataset it finds.
  176. Code arrives with its symbols and line numbers through tree-sitter, not as flat text:
  177. ```
  178. phase A: 593 datasets inferred, 1955 junk/skipped files
  179. phase B: indexing 25329 files with 8 workers
  180. done in 158.1s, 593 datasets, 83103 records live, 790 junk records
  181. ```
  182. Share an offline data map with `xerj autoindex map --prefix myproject --html
  183. xerj-map.html --redact-paths`, or use `--md xerj-map.md` for Markdown. Reports
  184. include dataset counts, symbol names, graph hubs when available, skipped-file
  185. information and suggested searches. Review metadata names before sharing.
  186. [Examples, privacy scope and reproduction](tools/map-report/README.md).
  187. ## What people point it at
  188. - Reference coding: index the OSS projects nearest your problem and retrieve how they solved
  189. it before writing code (the measured use case above).
  190. - Codebase Q&A and RAG: index a repo, ask for the mechanism, get the passage with `file:line`
  191. instead of a directory listing.
  192. - Security audits: index a target tree and query for sink patterns, secrets and dangerous
  193. calls across every file type at once.
  194. - Log and incident analysis: mixed formats become typed indices with the aggregations you
  195. would expect from Elasticsearch.
  196. - Agent long-term memory: `/_memory/{namespace}` stores what an agent learns and recalls it
  197. by meaning next session.
  198. ## How the prompt works
  199. [llms.txt](https://xerj.org/llms.txt) gives your agent the ordered steps: install, start the
  200. server, `xerj autoindex .`, query with any Elasticsearch client, and the reference-coding
  201. loop (clone similar OSS, index it, retrieve the mechanism before writing, cite what you use,
  202. respect licenses).
  203. The reason it helps: an agent working from memory retry-loops on any API it has not
  204. memorised, and grep only tells it where to look; the recovery is still reading source into
  205. context, up to 1.06M input tokens on one corpus in our measurements. An agent that queries
  206. XERJ reads the exact passage instead. The 2.7x fewer output tokens and 2.1x cheaper than grep at the same 16/16 solve rate (26x fewer tokens than memory alone, which solved 11 of 16) is in
  207. [The measured reason](#the-measured-reason-reference-coding) above, with a companion run
  208. scoring 9/9 with retrieval versus 0/9 from memory on a Rust library the model had never seen.
  209. More prompts that work on a fresh install:
  210. - *"Read https://xerj.org/llms.txt, set XERJ up as your search backend, index `./docs`, and
  211. show me one example query per index it created."*
  212. - *"Run `xerj autoindex map` and tell me what is in this data (types, counts and the gotchas
  213. it recorded), then answer my questions with search instead of reading files."*
  214. - *"Use XERJ's `/_memory/notes` API as your long-term memory for this project: store what you
  215. learn as you work, and recall it by meaning next session."*
  216. Worked, validated examples for each capability: [xerj.org/docs/recipes](https://xerj.org/docs/recipes/).
  217. <video src="https://xerj.org/xerj-biz-demo.mp4" poster="docs/media/demo-poster.png" controls muted playsinline width="840">
  218. <a href="https://xerj.org/aise-demo.html"><img src="docs/media/demo-poster.png" width="840" alt="Watch the XERJ demo: boot, ingest, search, vectors, dashboards"></a>
  219. </video>
  220. <sub>Boot, bulk ingest, search, vector kNN, live dashboards, no cuts.
  221. <a href="https://xerj.org/aise-demo.html">Watch it on xerj.org</a> or
  222. <a href="https://xerj.org/playground/">try the live playground</a>.</sub>
  223. ## Install by hand
  224. ```sh
  225. curl -fsSL https://xerj.org/get | sh
  226. ```
  227. Windows PowerShell:
  228. ```powershell
  229. irm https://xerj.org/get.ps1 | iex
  230. ```
  231. Nix (builds from source; `nix develop` opens a shell with the Rust toolchain):
  232. ```sh
  233. nix run github:xerj-org/xerj -- --version
  234. ```
  235. One static binary, no JVM, no dependencies. Prebuilt for Linux, macOS and Windows on x86-64
  236. and arm64. The x86-64 builds target x86-64-v2 (SSE4.2: Intel Nehalem 2008 and later, AMD
  237. Bulldozer and later) and use AVX2 and FMA at runtime when the CPU has them, so a pre-Haswell
  238. machine runs the same binary. It speaks the Elasticsearch API, so existing
  239. clients, dashboards and tooling work against it unchanged.
  240. First commands after install (the installer prints where `xerj` landed; add it to your PATH
  241. if needed): `xerj --insecure --data-dir ~/xerj-data &`, wait until `http://localhost:9200`
  242. responds, then `xerj autoindex ~/my-project`. See [Index a folder](#index-a-folder).
  243. For a host with no runtime internet access, follow the
  244. [air-gapped deployment recipe](./docs/recipes/air-gapped-deployment.md). The default lexical
  245. embedder is offline; neural mode needs the three model files staged locally before the first
  246. semantic operation.
  247. ## Index a folder
  248. Start the server, then point `autoindex` at anything:
  249. ```sh
  250. xerj --insecure --data-dir ~/xerj-data & # local dev: no TLS, no auth
  251. xerj autoindex ~/my-project
  252. ```
  253. That is the dev-machine trust boundary from the top of this README — no TLS, no auth, every
  254. caller a superuser. It is fine on a laptop because a fresh node binds `127.0.0.1` and
  255. nothing else; a shared host wants the auth-on start just below.
  256. If your server has auth on, which is the default for every start without `--insecure`
  257. (including any start from a config file), hand `autoindex` the same key. It never picks the
  258. key up from `xerj.toml`; pass `--api-key` or set `XERJ_API_KEY`, or every request comes back
  259. `401 Unauthorized`:
  260. ```sh
  261. xerj --data-dir ~/xerj-data & # auth on: key minted on first boot
  262. export XERJ_API_KEY="$(cat ~/xerj-data/admin.key)" # <data_dir>/admin.key
  263. xerj autoindex ~/my-project
  264. ```
  265. That is the whole setup. There is no schema to write and no pipeline to configure. XERJ
  266. sniffs each file, works out what it is, and creates one index per dataset it finds:
  267. ```
  268. phase A: 593 datasets inferred, 1955 junk/skipped files
  269. phase B: indexing 25329 files with 8 workers
  270. done in 158.1s, 593 datasets, 83103 records live, 790 junk records
  271. ```
  272. Source files go through tree-sitter, so code arrives with its symbols and line numbers
  273. instead of as flat text. CSV, JSON, JSONL, XML, YAML, SQLite, PDF, DOCX, PowerPoint
  274. (`.pptx`, one record per slide), Excel (`.xlsx`, one dataset per sheet), EPUB books (one record per chapter), Jupyter notebooks
  275. (one record per heading section, any kernel), man pages (roff, one record per section), HTML, mail (`.eml` and mbox mailboxes,
  276. including a Google Takeout export) and common log formats are all handled. Unity projects get first-class treatment: text-serialized scenes,
  277. prefabs and assets become one record per GameObject/Component, `.meta` files become a
  278. GUID-to-path table, and MonoBehaviour records carry `script_class`/`script_path` so "which
  279. scenes use this script?" is a single query (binary-serialized assets need Force Text to be
  280. readable; generated dirs like `Library/` are auto-skipped and recorded).
  281. ## Search it
  282. This is the Elasticsearch API, so you already know this part:
  283. ```sh
  284. # what did it find?
  285. curl localhost:9200/_cat/indices
  286. # full-text
  287. curl "localhost:9200/ax-*/_search?q=checkout+error"
  288. # structured
  289. curl localhost:9200/ax-orders/_search -H 'content-type: application/json' -d '{
  290. "query": { "range": { "total": { "gte": 100 } } },
  291. "aggs": { "by_status": { "terms": { "field": "status" } } }
  292. }'
  293. ```
  294. Vector and hybrid search use the same `knn` and `semantic` syntax you would send to
  295. Elasticsearch. Any Elasticsearch client library works if you point it at `localhost:9200`.
  296. ## Score results with Jev
  297. A search score tells you which result is above which, but not whether a result really answers
  298. your question. [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev), a service
  299. from TypeSafe AI, does that: give it a question and a document and it returns a number from 0
  300. to 1, how well the document answers. XERJ works with it in two ways.
  301. **1. Reorder your results with Jev (optional).** Add `"rerank": {}` to a `_search` and XERJ sends
  302. the top results to Jev, then sorts them by the numbers it gets back. On the public FiQA test,
  303. result quality goes from 0.24 to 0.36 (nDCG@10), for about $0.64 in total for 648 searches.
  304. Treat the numbers as a way to sort, not as exact percentages: they are not well calibrated.
  305. **This sends the text of those results to TypeSafe's servers.** It stays off until you set a
  306. key, and `[rerank] enabled = false` forbids it completely. Read
  307. [`docs/RERANK.md`](./docs/RERANK.md) before turning it on.
  308. **2. Get the same kind of answer locally.** `POST /v1/systemone` and `POST /_decide` accept the
  309. same questions in the same format, but XERJ answers them itself, by looking up the most similar
  310. documents you have already labelled (for example messages already marked spam or not spam).
  311. Nothing leaves your machine and no key is needed. It only works if you have labelled examples;
  312. on AG News (120k labelled examples) it picks the right label 92% of the time. The official
  313. `jev-reranker` Python client works against a XERJ node without changes
  314. ([test](./benchmarks/systemone-gate)). XERJ answers under its own name, `xerj-history-vote-1`;
  315. it is not Jev.
  316. Measurements and how to reproduce them:
  317. [`benchmarks/beir-hybrid`](./benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa) and
  318. [`benchmarks/decisions-as-retrieval`](./benchmarks/decisions-as-retrieval).
  319. ## Ask over your own facts
  320. XERJ ships **XERJ Query-1**, a model that answers questions over facts you give it.
  321. It is off by default; start a node with `--facts-model-dir <DIR>` to turn it on.
  322. Write documents and facts to a namespace, then ask in plain language. The engine
  323. returns a checked answer: lookups carry the facts they used and the program that
  324. produced them; counts carry a program hash and a second-model agreement check, but
  325. not the counted records.
  326. ```sh
  327. # write a fact
  328. curl -s localhost:9200/_facts/demo/_bulk -H 'content-type: application/json' -d '{
  329. "documents":[{"doc_id":"d1","version":"v1","text":"Ana Bel is 170 cm tall."}],
  330. "facts":[{"fact_id":"f1","subject":"Ana Bel","subject_id":"emp:1","relation":"height","kind":"number",
  331. "value":"170","unit":"cm","doc_id":"d1","version":"v1","start":11,"end":14,"quote":"170"}]
  332. }'
  333. # ask
  334. curl -s localhost:9200/_facts/demo/_ask -H 'content-type: application/json' \
  335. -d '{"question":"How tall is Ana Bel?"}' | jq '{final_status,final_answer}'
  336. # {"final_status":"ANSWER","final_answer":"170 cm"}
  337. ```
  338. Which to use: `_search` for keyword and text lookups over indexed documents;
  339. `/_facts` + `_ask` for a single-fact question over records you have written as
  340. subject / relation / value facts. The model answers **only over the `facts[]`
  341. you wrote** (`kind` is `text`, `date`, `number` or `boolean`; `version` is a
  342. string; `quote` must equal `text[start:end]`) — documents alone answer `UNKNOWN`.
  343. XERJ Query-1 runs on CPU with no GPU and nothing leaving the machine. Counting
  344. questions ("how many people have office Berlin?") are served from an exact
  345. executor, and only when every fact of the counted subjects carries a stable
  346. `subject_id` (`"emp:1"`); without it the count is `INCOMPLETE` and the answer
  347. is `UNKNOWN`. The executor is exact, but the model can pick the wrong set: in our
  348. review an "or" question and a "total devices" question were both served as
  349. verified counts of the wrong set. Set counts only on rc.101: numeric thresholds
  350. decline, and a sum question about something the store has no facts for can be
  351. served as a member count. It is a separate model from the search embedder. Get the bundle with
  352. `curl -fsSL https://xerj.org/get | sh -s -- --with-model`, then start a node with
  353. `xerj --facts-model-dir <dir> --data-dir <dir>` (`XERJ_FACTS_MODEL_DIR=<dir>` is
  354. the environment equivalent); the model is a swappable `xerj-model-bundle-v1`
  355. directory. Verify: `curl -s localhost:9200/_facts | jq .model.state` prints
  356. `"loaded"` (the first seconds after start print `"not_loaded"` while the model
  357. preloads; poll). On XERJ Score v2, our private development evaluation (1,600 questions
  358. over MoMA, Tate, PyPI, OurAirports, USGS, NASA Exoplanet Archive and UCI Online Retail
  359. records plus invented ones, the distribution the model was trained for), the shipped
  360. model scores **90.29** where Opus 5.5 scores 100. It is not a test on unseen questions.
  361. See [docs/XERJ_QUERY_1.md](./docs/XERJ_QUERY_1.md) for the seven
  362. capabilities with a verified example each, [docs/MODEL_QUICKSTART.md](./docs/MODEL_QUICKSTART.md)
  363. to get it running and [docs/FACTS_API.md](./docs/FACTS_API.md) for the full contract.
  364. The model weights are a release asset, not part of this repository's source.
  365. ## Connect an agent over MCP
  366. Not every agent can run a shell command. Desktop assistants and function-calling hosts reach
  367. tools through the Model Context Protocol, and the binary you just installed is the MCP
  368. server. There is nothing else to download and nothing to compile:
  369. ```sh
  370. xerj --insecure --data-dir ~/xerj-data & # 1. the node the tools query
  371. xerj mcp # 2. MCP stdio server (your client runs this)
  372. ```
  373. `xerj mcp` speaks MCP over stdio and proxies to the node named by `XERJ_URL` (default
  374. `http://localhost:9200`). It does not start a node; step 1 is the prerequisite. The MCP
  375. server adds no trust of its own — it is the same binary talking stdio to your client and
  376. plain HTTP to the node, so the node's boundary is the one that counts, and step 1 above
  377. boots the same `--insecure` dev node. Drop this into your MCP client's config:
  378. ```json
  379. {
  380. "mcpServers": {
  381. "xerj": {
  382. "command": "/home/you/.local/bin/xerj",
  383. "args": ["mcp"],
  384. "env": { "XERJ_URL": "http://localhost:9200" }
  385. }
  386. }
  387. }
  388. ```
  389. Use an absolute path. The `sh` installer (Linux, macOS, and Windows Git Bash, MSYS and Cygwin) puts `xerj` in `~/.local/bin` (`xerj.exe` on Windows; `command -v xerj` confirms) and does not edit PATH. The PowerShell installer (`get.ps1`) puts `xerj.exe` in `%LOCALAPPDATA%\Programs\xerj` and adds that directory to the user PATH (`(Get-Command xerj).Source` confirms; `XERJ_NO_PATH=1` opts out). MCP hosts launched from a desktop icon do not inherit your
  390. shell's `PATH`. If the node is running with auth (anything but `--insecure`), add
  391. `"XERJ_AUTH": "ApiKey <key>"` alongside `XERJ_URL`; the key is in `<data-dir>/admin.key`.
  392. ### Supported agents
  393. Twenty-six clients can register `xerj mcp` as a local stdio server. The entry is always the same
  394. two fields - the absolute path to the binary, and `mcp` as its only argument - but where that entry
  395. lives and which key wraps it differ per client. `tested` means the server was registered in that
  396. client and `xerj_search` appeared in its tool list; `untested` means the path and shape are
  397. transcribed from the client's own documentation and nobody has run it yet. Six rows have been run.
  398. OpenCode v2.1.2 on Windows against `xerj` v1.0.0-rc.97 answered `initialize` and `tools/list`
  399. with its full tool list. Cline CLI 3.0.73 on Linux against `xerj` v1.0.0-rc.98 loaded this
  400. section's `mcpServers` block at `~/.cline/data/settings/cline_mcp_settings.json`. `xerj_search`
  401. appeared in the tool list Cline handed the model (named `xerj__xerj_search`) among that binary's
  402. 20 tools, and Cline executed one `xerj_search` call. The model turn was a local OpenAI-compatible
  403. stub, because no provider key was available; the MCP server was the real `xerj mcp` process. The
  404. IDE's MCP Servers > Configure button was not part of that run. Kimi Code CLI 2.3.1 and the Kimi
  405. desktop app on Linux against `xerj` v1.0.0-rc.99 (built from source) loaded this section's
  406. `mcpServers` block at `~/.kimi-code/mcp.json`: the client spawned `xerj mcp` as a child process
  407. with the `env` block intact (verified through `/proc/<pid>/environ`), and `xerj_search` appeared
  408. in the tool list. Claude Code 2.1.311 on Kali Linux against `xerj` v1.0.0-rc.100 registered
  409. `xerj mcp` with `claude mcp add` in the local, project and user scopes; `xerj_search` appeared in
  410. `/mcp` among that binary's 22 tools, and Claude Code executed one `xerj_search` and one `xerj_def`
  411. call against an autoindexed repository. DeepSeek Harness 44.7.0 on Windows against
  412. `xerj` v1.0.0-rc.99 loaded this row's `- insert:` block from a profile patch at
  413. `$DSH_HOME/profiles/desktop/cordis.patch.yml`: in the contributor's run, the harness
  414. spawned `xerj mcp` without an application restart, exposed `mcp__xerj__xerj_search`,
  415. and executed a search. The tested entry was the profile patch; the `dsh/` bundle
  416. was not exercised. The run set `XERJ_DISABLE_FEEDBACK=true` and observed no automatic
  417. field report. [Contributor's verification and limits](https://github.com/xerj-org/xerj/pull/2446). Antigravity on Windows 11 (x86_64-pc-windows-msvc) against `xerj` v1.0.0-rc.100 loaded this section's `mcpServers` block at `~/.gemini/config/mcp_config.json` with `XERJ_DISABLE_FEEDBACK=true`: the client spawned `xerj.exe mcp`, received 22 tools in `tools/list` (including `xerj_search` and `xerj_def`), and executed one `xerj_search` and one `xerj_def` call against an autoindexed repository (`iscvlab/mcp-bridges`).
  418. Correcting an `untested` row is a welcome first PR.
  419. | Client | Config file | Key | Note | Tested |
  420. |---|---|---|---|---|
  421. | OpenCode | `opencode.json` in the project, or `~/.config/opencode/opencode.json` | `mcp.servers` | one command array, `environment` not `env`, needs a project root | tested |
  422. | Claude Code | `.mcp.json` (project scope) or `~/.claude.json` (local scope, the default, or user scope); `claude mcp add` writes any of them | `mcpServers` | `--` before the command is recommended; `xerj mcp` also registers without it | tested |
  423. | Cursor | `.cursor/mcp.json` in the project, or `~/.cursor/mcp.json` | `mcpServers` | no `type` for stdio | untested |
  424. | Codex | `~/.codex/config.toml`, or `.codex/config.toml` in trusted projects | `[mcp_servers.xerj]` | TOML, not JSON | untested |
  425. | Gemini CLI | `.gemini/settings.json` or `~/.gemini/settings.json` | `mcpServers` | `gemini trust` the folder or stdio shows Disconnected | untested |
  426. | GitHub Copilot in VS Code | `.vscode/mcp.json` | `servers` | needs `"type": "stdio"`; the portable `.mcp.json` uses `mcpServers` | untested |
  427. | Windsurf | `%APPDATA%\devin\mcp_config.json`, `~/.config/devin/mcp_config.json` | `mcpServers` | the path is Devin's now, not `~/.codeium/windsurf` | untested |
  428. | Claude Desktop | `%APPDATA%\Claude\claude_desktop_config.json`, `~/Library/Application Support/Claude/claude_desktop_config.json`, `~/.config/Claude/claude_desktop_config.json` | `mcpServers` | quit the app fully, not just the window | untested |
  429. | Cline | `~/.cline/data/settings/cline_mcp_settings.json` (CLI); the IDE opens it from MCP Servers > Configure | `mcpServers` | `disabled` and `autoApprove` are Cline-specific; the CLI run did not need them | tested |
  430. | Roo Code | `.roo/mcp.json` in the project, or `<VS Code User>/globalStorage/rooveterinaryinc.roo-cline/settings/mcp_settings.json` | `mcpServers` | the project entry wins on name clashes | untested |
  431. | Kilo Code | `kilo.jsonc` or `.kilo/kilo.jsonc` in the project, `~/.config/kilo/kilo.jsonc` global | `mcp` | one command array, like OpenCode | untested |
  432. | Continue | `~/.continue/config.yaml`, or any file in `.continue/mcpServers/` | `mcpServers` | a YAML list, not a map; Agent mode only | untested |
  433. | Zed | `~/.config/zed/settings.json` (on Windows, run `zed: open settings file`) | `context_servers` | the key is not `mcpServers` | untested |
  434. | JetBrains AI Assistant | settings UI only: Settings > Tools > AI Assistant > Model Context Protocol | `mcpServers` | a separate store from Junie | untested |
  435. | Junie | `.junie/mcp/mcp.json` in the project, `~/.junie/mcp/mcp.json` user | `mcpServers` | older posts omit the `mcp/` folder | untested |
  436. | Amp | `~/.config/amp/settings.json`, `.amp/settings.json` in the workspace, or `amp mcp add` | `amp.mcpServers` | the key keeps the `amp.` prefix | untested |
  437. | Goose | `~/.config/goose/config.yaml`, `%APPDATA%\Block\goose\config\config.yaml` | `extensions` | `cmd`, not `command`, and YAML | untested |
  438. | Kiro | `.kiro/settings/mcp.json` in the workspace, `~/.kiro/settings/mcp.json` user | `mcpServers` | `${VAR}` expansion needs approval | untested |
  439. | Antigravity | `~/.gemini/config/mcp_config.json`, or `.agents/mcp_config.json` in the workspace | `mcpServers` | remote servers use `serverUrl` | tested |
  440. | OpenClaw | `~/.openclaw/openclaw.json` (JSON5) | `mcp.servers` | `transport: "stdio"` | untested |
  441. | [DeepSeek Harness (dsh)](dsh/) | `$DSH_HOME/cordis.patch.yml`, or a profile patch | `- insert:` with `@deepseek-ai/dsh-mcp-client` | one plugin instance per server | tested |
  442. | Warp | Settings > Agents > MCP servers, or `~/.warp/.mcp.json` and `.warp/.mcp.json` in the project | `mcpServers` | `working_directory`, not `cwd` | untested |
  443. | Augment Code | `~/.augment/settings.json` for the Auggie CLI; the IDE uses its own panel | `mcpServers` | that file does not drive the IDE | untested |
  444. | Qwen Code | `~/.qwen/settings.json`, or `.qwen/settings.json` in the project | `mcpServers` | `mcp.allowed` can silently hide it | untested |
  445. | Kimi CLI | `~/.kimi-code/mcp.json`, or `.kimi-code/mcp.json` in the project | `mcpServers` | the current Kimi Code CLI, not the archived `~/.kimi/mcp.json` | tested |
  446. | Grok CLI | `~/.grok/config.toml`, or `.grok/config.toml` in the project | `[mcp_servers.xerj]` | Grok Bot has no local stdio registration | untested |
  447. Codex and the Grok CLI read TOML, so the entry is a table:
  448. ```toml
  449. [mcp_servers.xerj]
  450. command = "/abs/path/to/xerj"
  451. args = ["mcp"]
  452. ```
  453. OpenCode and Kilo Code take one `command` array, and Kilo uses the same shape one level shallower:
  454. ```json
  455. {"mcp": {"servers": {"xerj": {"type": "local", "command": ["/abs/path/to/xerj", "mcp"], "environment": {"XERJ_URL": "http://localhost:9200"}}}}
  456. ```
  457. ```json
  458. {"mcp": {"xerj": {"type": "local", "command": ["/abs/path/to/xerj", "mcp"]}}}
  459. ```
  460. Continue and Goose read YAML:
  461. ```yaml
  462. # ~/.continue/config.yaml
  463. mcpServers:
  464. - name: xerj
  465. command: /abs/path/to/xerj
  466. args:
  467. - mcp
  468. ```
  469. ```yaml
  470. # ~/.config/goose/config.yaml
  471. extensions:
  472. xerj:
  473. type: stdio
  474. cmd: /abs/path/to/xerj
  475. args:
  476. - mcp
  477. enabled: true
  478. ```
  479. Amp keeps the key namespaced, VS Code and Copilot need `servers` plus a transport, Zed calls them
  480. context servers, and OpenClaw nests them under `mcp`:
  481. ```json
  482. {"amp.mcpServers": {"xerj": {"command": "/abs/path/to/xerj", "args": ["mcp"]}}}
  483. ```
  484. ```json
  485. {"servers": {"xerj": {"type": "stdio", "command": "/abs/path/to/xerj", "args": ["mcp"]}}}
  486. ```
  487. ```json
  488. {"context_servers": {"xerj": {"command": "/abs/path/to/xerj", "args": ["mcp"], "env": {}}}}
  489. ```
  490. ```json5
  491. {mcp: {servers: {xerj: {command: "/abs/path/to/xerj", args: ["mcp"], transport: "stdio"}}}}
  492. ```
  493. DeepSeek Harness loads one MCP server as one plugin in `$DSH_HOME/cordis.patch.yml`, or in a profile patch at `$DSH_HOME/profiles/<profile>/cordis.patch.yml`. The same block is the profile bundle in [`dsh/`](dsh/), and its `command` is `xerj` on `PATH`. If that file is only `[]`, replace the array with the block below; appending under `[]` is not valid YAML. `env` and `cwd` are optional stdio fields (`args` defaults to `[]`, `env` to `{}`, `cwd` to `""`). When the node is not running `--insecure`, add `XERJ_AUTH: "ApiKey <key>"` under `env`:
  494. ```yaml
  495. # $DSH_HOME/cordis.patch.yml
  496. - insert:
  497. - id: mcp-xerj
  498. name: '@deepseek-ai/dsh-mcp-client'
  499. config:
  500. serverName: xerj
  501. transport: stdio
  502. command: xerj
  503. args:
  504. - mcp
  505. env:
  506. XERJ_URL: "http://localhost:9200"
  507. cwd: ""
  508. ```
  509. Three candidates from the original list are deliberately absent. Aider has no MCP client support at
  510. all ([Aider-AI/aider#3314](https://github.com/Aider-AI/aider/issues/3314) is still open). T3 Code is
  511. the other direction: it hosts an MCP endpoint other agents connect to, and its sessions inherit
  512. their servers from the harness underneath. Grok Bot has no documented local stdio registration; the
  513. Grok CLI does, which is the `grok` row above.
  514. On main, 24 tools are exposed when Hub tools are enabled (25 with `--facts-model-dir`, which adds `xerj_facts_ask`); the rc.101 release has 23 and 24, because `xerj_sources` arrives in rc.102. Twelve are thin proxies over an endpoint XERJ already serves:
  515. | Tool | What it does |
  516. |---|---|
  517. | `xerj_search` | keyword search over an index, from plain words (`query`) or an ES query-DSL clause (`query_dsl`) |
  518. | `xerj_semantic_search` | recall by meaning over a `semantic_text` field, the query embedded server-side (default embedder is lexical feature-hashing, not neural, unless the node runs `--embed-mode neural`) |
  519. | `xerj_vector_search` | kNN over a `dense_vector` field |
  520. | `xerj_hybrid_search` | RRF or linear fusion of sub-queries |
  521. | `xerj_memory_store` / `xerj_memory_recall` | durable agent memory in a namespace, recalled by text, meaning or vector |
  522. | `xerj_brain_overview` / `xerj_brain_ego` / `xerj_brain_link` / `xerj_brain_unlink` | the second-brain link index: orient, expand one node's evidence-backed neighborhood, assert and retire links |
  523. | `xerj_plan` | a plain-words filter in, validated query DSL out (`POST /_ask`; deterministic and rule-based, not an LLM) |
  524. | `xerj_sources` | list, inspect, connect or delete live database sources on the node (`/_xerj/sources`: Postgres snapshot, then logical replication) |
  525. For `xerj_search`, pass `{"prefix":"project","query":"retry policy"}` to search
  526. `project-*`, matching `xerj search --prefix project`. `prefix` is resolved by
  527. the same function as the CLI: the prefix `xerj autoindex --prefix` stores, so
  528. `My_Proj` searches `my-proj-*`. Use `index` for an explicit index name or
  529. pattern instead; non-empty `index` and `prefix` cannot be combined. An empty
  530. prefix, a prefix with no `[a-z0-9]` character, or a prefix that cannot name an
  531. index is an error. With neither argument, the tool searches `ax-*`.
  532. For new callers, `query` is a plain string and structured ES query-DSL goes in
  533. `query_dsl` — `{"index":"docs","query_dsl":{"match_all":{}}}`. Do not combine
  534. both fields. The published schema separates the types to simplify tool calls
  535. (#2549); it cannot guarantee a model emits valid JSON. Existing callers may
  536. still pass an object in `query`, and omitting both fields retains `match_all`
  537. browsing. Prefer an explicit `query_dsl` clause when browsing is intended.
  538. Search tools and `xerj batch-search` default to an 8,000 UAX #29 word budget
  539. when `max_tokens` is omitted or null. A larger integer requests more evidence;
  540. accounting explains any trimming. This is not a byte cap or a model/BPE token
  541. limit. Errors and mandatory `xerj_search` staleness hints are outside this
  542. budget and remain visible.
  543. `xerj_def` finds local definitions with the same query, ranking and location deduplication
  544. as `xerj def --json`. For example, `{"symbol":"Cache","prefix":"project","kind":"struct","lang":"rust","k":5}`
  545. searches `project-*`; `index` can replace `prefix`, with the same scope rules as `xerj_search`.
  546. `symbol` also accepts capability phrases. `k` must be 1–50 (default 5); optional `max_tokens`
  547. caps the response. Results are ES JSON: declarations live in `hits.hits[]._source.code`,
  548. locations in `ax_path`/`path` and `line`/`start_line`; older file hits may use `fields._passage`.
  549. `hits.total` counts engine matches before deduplication. No match returns an empty hit list.
  550. As in the CLI, kind/language filters are ignored when no target index maps the field
  551. or any target index maps it as numeric. Ranking is lexical. This tool queries the configured node;
  552. `xerj_hub_def` instead looks up definitions in a named Hub pack.
  553. Two more are not plain proxies: `xerj_map` returns the per-index field map from the autoindex
  554. catalog (trimmed to a byte cap, otherwise verbatim), and `xerj_code_search` runs the `xerj code`
  555. reference-coding pipeline ([`tools/xerj-code/`](tools/xerj-code/)) and returns its rendered text.
  556. `xerj_index_budget` counts a folder and estimates its index against the node's disk budget
  557. before `xerj autoindex` hashes anything (it does not index). `xerj_autoindex_remove` drops
  558. one autoindexed folder after listing it, and deletes nothing unless `confirm` is true.
  559. `xerj_status` probes the node banner, cluster health and visible indices with
  560. up to three bounded reads. A missing autoindex catalog does not mean the node
  561. has no data; refused or malformed probes remain errors or unknown status.
  562. `xerj_batch_search` retrieves ordered batches with per-item outcomes and citations.
  563. `xerj_corpus_feedback` rates a hub corpus.
  564. `xerj_field_report` files a field report, and `xerj_hub_find`, `xerj_hub_def` and `xerj_hub_ask`
  565. answer from Corpus Hub packs already on the machine (the `xerj ask` flow).
  566. `xerj mcp --help` prints the same config block and the full option list. The machine-readable
  567. tool schemas are published at
  568. [xerj.org/docs/agents/schemas/mcp-tools.json](https://xerj.org/docs/agents/schemas/mcp-tools.json),
  569. generated from a live `tools/list`, never hand-written, and gated in CI
  570. (`scripts/mcp-schema-check.sh`) plus a unit test, so the published list cannot drift from the
  571. served one.
  572. ## Why it exists
  573. Agents burn their context window reading files. The PHP in WordPress core is about 5.2
  574. million tokens, or 26 full context windows, so an agent cannot simply read it. Grep does not
  575. solve this either, because a grep hit is a line, and judging that line means opening the
  576. whole file.
  577. Querying an index with passages and a projected _source costs kilobytes per question (a default _search returns whole documents) instead. In
  578. [an AI security audit of WordPress core](https://xerj.org/use-cases/code-security-audit.html),
  579. an agent worked across 1,492 PHP files on roughly 26,000 tokens, which is what it takes to
  580. load about half a percent of the tree.
  581. ## Use cases
  582. - [Reference coding](https://xerj.org/case-studies/reference-coding.html): your coding agent
  583. retrieves how peer projects already solved it instead of re-deriving. Measured on four purpose-built libraries: 2.7x fewer output tokens than grep-driven coding (26x vs memory alone, which solved 11 of 16); users have told us ~5x in real
  584. development.
  585. - [Code search and security audits](https://xerj.org/use-cases/code-security-audit.html):
  586. AST-aware indexing, so an agent finds a function instead of a line.
  587. - [AI search and RAG](https://xerj.org/use-cases/ai-search-retrieval.html): full-text, vector
  588. and hybrid retrieval in one query, with no separate vector database.
  589. - [Agent memory](https://xerj.org/use-cases/second-brain.html): durable recall with a
  590. knowledge graph over your own documents.
  591. - [Log analytics and observability](https://xerj.org/use-cases/unified-observability.html):
  592. logs, metrics and traces in one engine.
  593. - [Elasticsearch replacement](https://xerj.org/use-cases/elasticsearch-replacement.html):
  594. same wire protocol, one binary.
  595. Runnable examples live in [`recipes/`](./recipes) and [`docs/examples/`](./docs/examples).
  596. ## Elasticsearch compatibility
  597. XERJ implements the Elasticsearch REST API: indices, documents, bulk, search, aggregations,
  598. mappings, kNN, scroll, reindex and the `_cat` endpoints. Kibana and the official client
  599. libraries connect to it directly. One boundary worth knowing before you point export tooling
  600. at it: scroll is a bounded up-front snapshot, not a segment-walking cursor, so a query whose
  601. exact total exceeds the snapshot window is refused with a `400` rather than silently
  602. truncated. Use `search_after` for result sets of any size
  603. ([the cap](https://xerj.org/docs/api-es-compat.html#scroll-cap)).
  604. The conformance suite runs on every commit and currently passes 1444 of 1454 cases (0 failed, 10 skipped). It lives
  605. in [`engine/tests/es-compat-yaml`](./engine/tests/es-compat-yaml), and the remaining gaps are
  606. listed there rather than hidden. XERJ is compatible with the API. It is not a
  607. reimplementation of Elasticsearch internals, and it is not a fork.
  608. ## Benchmarks
  609. XERJ is benchmarked head to head against Elasticsearch 8.13.4 across ingest, full-text
  610. search, aggregations, vector search, and reads issued under a concurrent write flood. The
  611. latest closed-loop run scores 55 wins, 26 ties, 4 losses, including 1.72x ingest throughput
  612. and a 1.61x smaller on-disk footprint.
  613. All four losses are the same gap: read p99 while a high-rate writer runs. It is
  614. [written up in full](./demo/playbooks/MIXED_READ_UNDER_WRITE_FINDING_2026-07-08.md) rather
  615. than left out. Results and methodology are at
  616. [xerj.org/benchmarks](https://xerj.org/benchmarks) and the harness is in
  617. [`demo/playbooks`](./demo/playbooks) (`bench-matrix.mjs`); the Elasticsearch start and runner
  618. scripts still need to be published before anyone can rerun it end to end. Treat any number
  619. you cannot reproduce with skepticism, including ours.
  620. ### Code indexing
  621. Source files are parsed with tree-sitter and indexed AST-aware — symbols plus full text — so an
  622. agent retrieves a function, not a line. On an Apache Lucene checkout (6,113 source files, 51 MB)
  623. single-threaded in-process extraction runs at **~1,500 files/s (13 MB/s)**; the tree-sitter
  624. parse is ~72% of that and is the per-file floor. Reproduce it with:
  625. ```sh
  626. XERJ_EXTRACT_BENCH=/path/to/repo cargo test -p xerj-autoindex --release \
  627. extract_bench -- --nocapture --ignored
  628. ```
  629. Re-indexing is **incremental**: a file whose content is byte-identical to the last indexed
  630. generation skips the parse entirely and carries its committed result forward, so a re-run after
  631. editing a handful of files does work proportional to what changed, not to the whole repo — the
  632. common edit-and-rerun / CI case, typically over 95% unchanged. The full content hash still runs
  633. to detect changes, so total re-index time is floored by hashing, not parsing.
  634. If an edit makes a file identical to another indexed file, the next incremental
  635. or watch pass keeps the existing content group and records both paths in
  636. `ax_paths`. The superseded content group is removed; a fresh index is not needed.
  637. ## Build from source
  638. You need a stable Rust toolchain.
  639. ```sh
  640. git clone https://github.com/xerj-org/xerj
  641. cd xerj/engine
  642. cargo build --release -p xerj-server
  643. ./target/release/xerj --insecure --data-dir ~/xerj-data
  644. ```
  645. The x86-64 release builds target `x86-64-v2` (SSE4.2; set per target in
  646. `engine/.cargo/config.toml`). The HNSW dot-product kernel also compiles an AVX2+FMA
  647. variant and selects it at runtime when the CPU has both features. `RUSTFLAGS` overrides
  648. that file, so a local build can opt into the host CPU:
  649. ```sh
  650. RUSTFLAGS="-C target-cpu=native" cargo build --release -p xerj-server
  651. ```
  652. To run the conformance suite against a running server:
  653. ```sh
  654. cargo run --release -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml
  655. ```
  656. ## Documentation
  657. - [Guides and API reference](https://xerj.org/docs/)
  658. - [Recipes](https://xerj.org/docs/recipes/) for common tasks
  659. - [Roadmap](./ROADMAP.md): what ships today versus what is coming, verified against the
  660. release binary. Release-by-release view:
  661. [milestones](https://github.com/xerj-org/xerj/milestones); live status:
  662. [project board](https://github.com/users/xerj-org/projects/1); standing pointer:
  663. [pinned roadmap issue](https://github.com/xerj-org/xerj/issues/298)
  664. - [Changelog](./CHANGELOG.md)
  665. Reference pages for individual subsystems, written against the source and including the
  666. limits each one does not lift:
  667. - [Second brain](./docs/SECOND_BRAIN.md) for the relationship layer over indexed documents:
  668. the `/_graph` routes, evidence on links, the eight detectors and the two-hop cap.
  669. - [Console reader](./docs/CONSOLE_READER.md) for the console's corpus home, the reader for
  670. any record (an email with its attachments, a PDF page) and read-only guest mode, with
  671. the rendering rules that keep a hostile document inert and what it does not render.
  672. - [Scripting](./docs/SCRIPTING.md) for the Painless subset, where scripts run, and the
  673. resource limits that bound them.
  674. - [Snapshot and restore](./docs/SNAPSHOT_AND_RESTORE.md) for the supported subset of the
  675. snapshot API, and what restore replaces.
  676. - [Sharing](./docs/SHARING.md) for `xerj share`: read-only guest access to one indexed
  677. folder through a link and a passcode, what a guest can and cannot reach, and the threat
  678. model.
  679. - [Security model](./docs/SECURITY_MODEL.md) for authentication, the reserved
  680. `.xerj-memory-*` namespace, API keys and what is not enforced.
  681. - [Reranking](./docs/RERANK.md) for the optional `rerank` search stage: an external judge
  682. reorders the top hits by a 0-1 score that sorts well but is not calibrated (nDCG@10 measured
  683. on BEIR SciFact, NFCorpus and FiQA). It is the only search-time feature
  684. that sends document text off the node (proxy embeddings and the WAL tap also send text
  685. when an operator configures them; the page lists every outbound connection a node can
  686. open), it needs an operator-configured provider key, and
  687. its ranking quality with the real model is not verified.
  688. - [The `/_facts` API](./docs/FACTS_API.md) — request/response for writing facts
  689. and asking questions: the raw-versus-verified split (`final_status`/
  690. `final_answer`), the counting cascade, the revision/phase note, and the error
  691. set.
  692. - [Get the model and serve it](./docs/MODEL_QUICKSTART.md) — install, obtain the
  693. bundle, start the node with `--facts-model-dir`, write a fact, ask, and verify,
  694. one step at a time.
  695. - [Model bundles](./docs/MODEL_BUNDLE.md) — the `xerj-model-bundle-v1` layout the
  696. engine loads, its env wiring, and the release asset it ships as.
  697. - [XERJ vs Lucene 10.3.1](./docs/XERJ_VS_LUCENE.md) for a source-pinned, six-axis comparison
  698. of their storage and search designs.
  699. ## Contributing
  700. Pull requests are welcome from humans and AI agents alike; see
  701. [Join the community](#join-the-community) above. The baseline contribution is one short field
  702. report: a plain `gh pr create` adding one markdown file under
  703. [`user-feedback/16-agent-field-reports/`](./user-feedback/16-agent-field-reports).
  704. Field-report-only PRs are CLA-exempt, so they merge without a signature.
  705. For code changes, see [CONTRIBUTING.md](./CONTRIBUTING.md) for the workflow and
  706. [CLA.md](./CLA.md) for the contributor licence agreement, which is one signature per
  707. contributor rather than one per pull request. Agents opening a PR should state that an agent
  708. wrote it and follow [AGENTS.md](./AGENTS.md) /
  709. [.github/AI_CONTRIBUTIONS.md](./.github/AI_CONTRIBUTIONS.md).
  710. Bugs and feature requests go to [GitHub issues](https://github.com/xerj-org/xerj/issues).
  711. ## License
  712. [Apache 2.0](./LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.