Skip to content
Back to search

Matryoshka

MCP server for token-efficient large document analysis via the use of REPL state

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/yogthos--matryoshka/badge.svg)](/listings/yogthos--matryoshka)

HTML

<a href="/listings/yogthos--matryoshka"><img src="/api/listings/yogthos--matryoshka/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install from npm:

pnpm add -g matryoshka-rlm

Or run without installing:

npx matryoshka-rlm "How many ERROR entries are there?" ./server.log

Included Tools

The package provides several CLI tools:

| Command | Description | |---------|-------------| | `rlm` | Main CLI for document analysis with LLM reasoning | | `rlm-mcp` | MCP server with full RLM + LLM orchestration (`analyze_document` tool) | | `lattice-mcp` | MCP server exposing direct Nucleus commands (no LLM required) | | `lattice-repl` | Interactive REPL for Nucleus commands | | `lattice-http` | HTTP server for Nucleus queries | | `lattice-pipe` | Pipe adapter for programmatic access | | `lattice-setup` | Setup script for Claude Code integration |

From Source

git clone https://github.com/yogthos/Matryoshka.git
cd Matryoshka
pnpm install
pnpm run build

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 689 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Matryoshka
  2. [![Tests](https://github.com/yogthos/Matryoshka/actions/workflows/test.yml/badge.svg)](https://github.com/yogthos/Matryoshka/actions/workflows/test.yml)
  3. [![SafeSkill 88/100](https://img.shields.io/badge/SafeSkill-88%2F100_Passes%20with%20Notes-yellow)](https://safeskill.dev/scan/yogthos-matryoshka)
  4. Process documents 100x larger than your LLM's context window—without vector databases or chunking heuristics.
  5. ## The Problem
  6. LLMs have fixed context windows. Traditional solutions (RAG, chunking) lose information or miss connections across chunks. RLM takes a different approach: the model reasons about your query and outputs symbolic commands that a logic engine executes against the document.
  7. Based on the [Recursive Language Models paper](https://arxiv.org/abs/2512.24601).
  8. ## How It Works
  9. Unlike traditional approaches where an LLM writes arbitrary code, RLM uses **[Nucleus](https://github.com/michaelwhitford/nucleus)**—a constrained symbolic language based on S-expressions. The LLM outputs Nucleus commands, which are parsed, type-checked, and executed by **Lattice**, our logic engine.
  10. ```
  11. ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
  12. │ User Query │────▶│ LLM Reasons │────▶│ Nucleus Command │
  13. │ "total sales?" │ │ about intent │ │ (sum RESULTS) │
  14. └─────────────────┘ └─────────────────┘ └────────┬────────┘
  15. │
  16. ┌─────────────────┐ ┌─────────────────┐ ┌────────▼────────┐
  17. │ Final Answer │◀────│ Lattice Engine │◀────│ Parser │
  18. │ 13,000,000 │ │ Executes │ │ Validates │
  19. └─────────────────┘ └─────────────────┘ └─────────────────┘
  20. ```
  21. **Why this works better than code generation:**
  22. 1. **Reduced entropy** - Nucleus has a rigid grammar with fewer valid outputs than JavaScript
  23. 2. **Fail-fast validation** - Parser rejects malformed commands before execution
  24. 3. **Safe execution** - Lattice only executes known operations, no arbitrary code
  25. 4. **Small model friendly** - 7B models handle symbolic grammars better than freeform code
  26. ## Architecture
  27. ### The Nucleus DSL
  28. The LLM outputs commands in the Nucleus DSL—an S-expression language designed for document analysis:
  29. ```scheme
  30. ; Search for patterns
  31. (grep "ERROR")
  32. ; Filter results
  33. (filter RESULTS (lambda x (match x "timeout" 0)))
  34. ; Aggregate
  35. (sum RESULTS) ; Auto-extracts numbers from lines
  36. (count RESULTS) ; Count matching items
  37. ; Final answer
  38. <<<FINAL>>>13000000<<<END>>>
  39. ```
  40. #### Feature availability by execution path
  41. Matryoshka has two execution paths and not every primitive works in both:
  42. | Feature | `runRLM` (CLI / programmatic) | `lattice-mcp` (MCP server) |
  43. |---|---|---|
  44. | `(grep …)`, `(filter …)`, `(map …)`, etc. | ✅ | ✅ |
  45. | `(llm_query …)`, `(llm_batch …)` | ✅ | ✅ via MCP sampling protocol |
  46. | `(rlm_query …)`, `(rlm_batch …)` | ✅ (concurrent rlm_batch) | ✅ — child Nucleus session spawns via the same MCP sampling bridge; M suspensions per `rlm_query` call. `rlm_batch` runs **sequentially** (children one at a time) because the multi-turn suspension protocol only carries one pending request at a time — concurrent children would lose suspensions. Round-trip count is the same; wall-clock is N×slower for non-sampling clients. |
  47. | `(context N)` selector | ✅ (multi-doc via `runRLMFromContent(query, string[])`) | partial — `(context 0)` works; multi-doc loading not exposed via `lattice_load` |
  48. | `(grep "X" haystack)` | ✅ | ✅ |
  49. | `(show_vars)` | ✅ | ✅ (internal `_<name>` bindings filtered out) |
  50. | `FINAL_VAR(name)` resolution | ✅ | N/A — MCP returns query results directly |
  51. | `maxTimeoutMs` / `maxChars` / `maxErrors` | ✅ | ❌ — MCP has its own session timeout |
  52. | `compactionThresholdChars` | ✅ | ❌ — MCP doesn't have a multi-turn FSM history |
  53. The resource-limit features remain `runRLM`-only. The recursive primitives (`rlm_query`/`rlm_batch`) work in both paths — the MCP path spawns a child `runRLMFromContent` whose `llmClient` is the same sampling bridge as the parent, so each child turn flows through the existing MCP suspension/sampling protocol.
  54. #### Recursive Primitives
  55. `rlm_query` spawns a child Nucleus session with its own FSM loop. The child runs to FINAL and returns a string — useful when a sub-task needs multi-turn reasoning over a structured handle:
  56. ```scheme
  57. ; Child sees the resolved handle as its working document, NOT a
  58. ; JSON-stringified prompt blob. Lets the child use grep/lines/
  59. ; chunk_by_lines over arrays without JSON-syntax noise.
  60. (rlm_query "extract dates" (context RESULTS))
  61. ; No (context …) → child's document is the prompt itself.
  62. (rlm_query "summarize each error type")
  63. ```
  64. `rlm_batch` runs the same per-item recursion across a collection. Each item produces one entry in the returned array, in input order. Per-item failures surface as `"Error: rlm_batch item N failed — …"` strings without aborting the rest of the batch:
  65. ```scheme
  66. (rlm_batch (chunk_by_lines 100)
  67. (lambda c (rlm_query "extract metrics" (context c))))
  68. ```
  69. - **`runRLM`**: children fan out concurrently via a worker pool capped at `maxConcurrentSubcalls` (default 4).
  70. - **`lattice-mcp`**: children run **sequentially** because the multi-turn suspension protocol can carry only one pending request at a time. Round-trip count is identical to the concurrent path (N children × M turns each); only wall-clock differs.
  71. #### Multi-Context Loading
  72. Pass `string[]` to `runRLMFromContent` to load multiple documents. Address them via `(context N)`; index 0 is the default for primitives that don't specify a haystack:
  73. ```scheme
  74. (grep "DEPLOY" (context 0)) ; deploy.log
  75. (grep "OUTAGE" (context 2)) ; comms.log
  76. ; (context N) is just a term — pipe it anywhere a string is expected
  77. (rlm_query "scan" (context (context 1))) ; child sees doc 1
  78. ```
  79. Per-doc line numbers come back, so the LLM can cite "doc 0 line 4, doc 2 line 2" with confidence rather than inventing absolute offsets across a concatenation.
  80. #### Introspection
  81. ```scheme
  82. (show_vars) ; Returns a string summary of every binding currently
  83. ; in scope. Useful before a (filter RESULTS …) or a
  84. ; FINAL_VAR(name) reference when the LLM lost track of
  85. ; what's bound. Same surface as the `lattice_bindings`
  86. ; MCP tool but reachable from inside a query.
  87. ```
  88. Unknown FINAL_VAR markers surface a clear error rather than passing the literal text through:
  89. ```
  90. <<<FINAL>>>FINAL_VAR(_99)<<<END>>>
  91. → "[FINAL_VAR error: unknown binding "_99". Available: _1, RESULTS]"
  92. ```
  93. #### Resource Limits
  94. All optional. With none set, behavior is unchanged:
  95. ```typescript
  96. runRLM(query, file, {
  97. maxTimeoutMs: 30_000, // wall-clock cap, propagates to children
  98. maxChars: 100_000, // cumulative chars sent + received
  99. maxErrors: 5, // consecutive parse/execution errors
  100. compactionThresholdChars: 50_000, // summarize history when prompt grows past this
  101. })
  102. ```
  103. When a limit hits, the run terminates cleanly with a string of the form:
  104. ```
  105. [aborted: timeout 32100ms of 30000ms]
  106. Best partial answer:
  107. <the most recent meaningful solver result>
  108. ```
  109. The partial answer is always preserved when present — completed work is never silently lost on abort.
  110. ### The Lattice Engine
  111. The Lattice engine (`src/logic/`) processes Nucleus commands:
  112. 1. **Parser** (`lc-parser.ts`) - Parses S-expressions into an AST
  113. 2. **Type Inference** (`type-inference.ts`) - Validates types before execution
  114. 3. **Constraint Resolver** (`constraint-resolver.ts`) - Handles symbolic constraints like `[Σ⚡μ]`
  115. 4. **Solver** (`lc-solver.ts`) - Executes commands against the document
  116. Lattice uses **miniKanren** (a relational programming engine) for pattern classification and filtering operations.
  117. ### In-Memory Handle Storage
  118. For large result sets, RLM uses a handle-based architecture with in-memory SQLite (`src/persistence/`) that achieves **97%+ token savings**:
  119. ```
  120. Traditional: LLM sees full array [15,000 tokens for 1000 results]
  121. Handle-based: LLM sees stub [50 tokens: "$grep_error: Array(1000) [preview...]"]
  122. ```
  123. **How it works:**
  124. 1. Results are stored in SQLite with FTS5 full-text indexing
  125. 2. LLM receives descriptive handle references derived from the command (e.g., `$grep_error`, `$bm25_timeout`, `$filter_status`)
  126. 3. Operations execute server-side, returning new handles
  127. 4. Full data is only materialized when needed
  128. Handle names are auto-generated from the Nucleus command: `(grep "ERROR")` produces `$grep_error`, `(list_symbols "function")` produces `$list_symbols_function`. Repeated commands get a numeric suffix (`$grep_error_2`, `$grep_error_3`).
  129. ### Memory Pad
  130. The Lattice engine doubles as a **context memory** for LLM agents. Instead of roundtripping large text blobs in every message, agents stash context server-side and carry only compact handle stubs:
  131. ```
  132. Agent reads file, summarizes → lattice_memo "auth architecture"
  133. → $memo_auth_architecture: "auth architecture" (2.1KB, 50 lines)
  134. 20 messages later, needs it → lattice_expand $memo_auth_architecture
  135. → Full 50-line summary
  136. ```
  137. **Token math** (30-message session, 3 source files stashed):
  138. - Traditional roundtripping: **836K tokens**
  139. - Memo-based (stubs + 6 expands): **57K tokens** — **93% savings**
  140. Memos persist across document loads (`lattice_load` clears query handles but keeps memos), support LRU eviction (100 memo cap, 10MB budget), and can be explicitly deleted when stale. No document needs to be loaded to use memos.
  141. ### The Role of the LLM
  142. The LLM does **reasoning**, not code generation:
  143. 1. **Understands intent** - Interprets "total of north sales" as needing grep + filter + sum
  144. 2. **Chooses operations** - Decides which Nucleus commands achieve the goal
  145. 3. **Verifies results** - Checks if the current results answer the query
  146. 4. **Iterates** - Refines search if results are too broad or narrow
  147. The LLM never writes JavaScript. It outputs Nucleus commands that Lattice executes safely.
  148. ## Installation
  149. Install from npm:
  150. ```bash
  151. pnpm add -g matryoshka-rlm
  152. ```
  153. Or run without installing:
  154. ```bash
  155. npx matryoshka-rlm "How many ERROR entries are there?" ./server.log
  156. ```
  157. ### Included Tools
  158. The package provides several CLI tools:
  159. | Command | Description |
  160. |---------|-------------|
  161. | `rlm` | Main CLI for document analysis with LLM reasoning |
  162. | `rlm-mcp` | MCP server with full RLM + LLM orchestration (`analyze_document` tool) |
  163. | `lattice-mcp` | MCP server exposing direct Nucleus commands (no LLM required) |
  164. | `lattice-repl` | Interactive REPL for Nucleus commands |
  165. | `lattice-http` | HTTP server for Nucleus queries |
  166. | `lattice-pipe` | Pipe adapter for programmatic access |
  167. | `lattice-setup` | Setup script for Claude Code integration |
  168. ### From Source
  169. ```bash
  170. git clone https://github.com/yogthos/Matryoshka.git
  171. cd Matryoshka
  172. pnpm install
  173. pnpm run build
  174. ```
  175. ## Configuration
  176. Copy `config.example.json` to `~/.config/matryoshka/config.json`:
  177. ```json
  178. {
  179. "llm": {
  180. "provider": "ollama"
  181. },
  182. "providers": {
  183. "ollama": {
  184. "url": "http://localhost:11434/api/generate",
  185. "model": "qwen3-coder:30b",
  186. "options": { "temperature": 0.2, "num_ctx": 8192 }
  187. },
  188. "deepseek": {
  189. "url": "https://api.deepseek.com/chat/completions",
  190. "apiKey": "${DEEPSEEK_API_KEY}",
  191. "model": "deepseek-chat",
  192. "options": { "temperature": 0.2 }
  193. },
  194. "glm": {
  195. "url": "https://open.bigmodel.cn/api/paas/v4/chat/completions",
  196. "apiKey": "${ZHIPU_API_KEY}",
  197. "model": "glm-4-plus",
  198. "options": { "temperature": 0.2 }
  199. }
  200. },
  201. "rlm": {
  202. "maxTurns": 10
  203. },
  204. "grammars": {
  205. "ocaml": {
  206. "package": "tree-sitter-ocaml",
  207. "extensions": [".ml", ".mli"],
  208. "moduleExport": "ocaml",
  209. "symbols": {
  210. "value_definition": "function",
  211. "type_definition": "type",
  212. "module_definition": "module"
  213. }
  214. }
  215. }
  216. }
  217. ```
  218. - **`llm` / `providers` / `rlm`** — LLM provider selection and RLM tuning (shown above with example providers). Each provider takes a full `url` (the complete API endpoint), an optional `apiKey` (supports `${ENV_VAR}` interpolation), a `model` name, and `options`.
  219. - **`grammars`** — custom tree-sitter language mappings for symbol extraction (see [Adding Language Support](#adding-language-support) for the full list of built-in languages). Use the [tree-sitter playground](https://tree-sitter.github.io/tree-sitter/playground) to explore node types for your language.
  220. ## Usage
  221. ### CLI
  222. ```bash
  223. # Basic usage
  224. rlm "How many ERROR entries are there?" ./server.log
  225. # With options
  226. rlm "Count all ERROR entries" ./server.log --max-turns 15 --verbose
  227. # See all options
  228. rlm --help
  229. ```
  230. ### MCP Integration
  231. RLM includes `lattice-mcp`, an MCP (Model Context Protocol) server for direct access to the Nucleus engine. This allows coding agents to analyze documents with **80%+ token savings** compared to reading files directly.
  232. The key advantage is **handle-based results**: query results are stored server-side in SQLite, and the agent receives compact stubs like `$grep_error: Array(1000) [preview...]` instead of full data. Handle names are derived from the command for easy identification. Operations chain server-side without roundtripping data.
  233. #### Available Tools
  234. | Tool | Description |
  235. |------|-------------|
  236. | `lattice_load` | Load a document for analysis |
  237. | `lattice_query` | Execute Nucleus commands on the loaded document |
  238. | `lattice_expand` | Expand a handle to see full data (with optional limit/offset) |
  239. | `lattice_memo` | Store arbitrary context as a memo handle (no document required) |
  240. | `lattice_memo_delete` | Delete a stale memo to free memory |
  241. | `lattice_close` | Close the session and free memory |
  242. | `lattice_status` | Get session status, document info, and memo usage |
  243. | `lattice_bindings` | Show current variable bindings and memo labels |
  244. | `lattice_reset` | Reset all bindings and memos but keep document loaded |
  245. | `lattice_llm_respond` | Respond to a pending `(llm_query ...)` suspension |
  246. | `lattice_llm_batch_respond` | Respond to a pending `(llm_batch ...)` suspension with all N responses |
  247. | `lattice_help` | Get Nucleus command reference |
  248. #### Example MCP config
  249. ```json
  250. {
  251. "mcp": {
  252. "lattice": {
  253. "type": "stdio",
  254. "command": "lattice-mcp"
  255. }
  256. }
  257. }
  258. ```
  259. #### Efficient Usage Pattern
  260. ```
  261. 1. lattice_load("/path/to/large-file.txt") # Load document (use for >500 lines)
  262. 2. lattice_query('(grep "ERROR")') # Search → $grep_error: Array(500) [preview]
  263. 3. lattice_query('(filter RESULTS ...)') # Narrow → $filter_timeout: Array(50) [preview]
  264. 4. lattice_query('(count RESULTS)') # Count without seeing data → 50
  265. 5. lattice_expand("$filter_timeout", limit=10) # Expand only what you need to see
  266. 6. lattice_close() # Free memory when done
  267. ```
  268. **Token efficiency tips:**
  269. - Query results return descriptive handle stubs, not full data
  270. - Use `lattice_expand` with `limit` to see only what you need
  271. - Chain `grep → filter → count/sum` to refine progressively
  272. - Use `RESULTS` in queries (always points to last result)
  273. - Use descriptive handle names (e.g., `$grep_error`) with `lattice_expand` to inspect specific results
  274. #### Chunking and Sub-LLM Recursion
  275. Two primitive families power the paper's `Ω(|P|²)` semantic-horizon pattern:
  276. **Chunking** — pre-slice a document that's too big to map over directly:
  277. ```scheme
  278. (chunk_by_size 2000) ; 2000-character slices
  279. (chunk_by_lines 100) ; 100-line slices
  280. (chunk_by_regex "\\n\\n") ; Split on blank lines; capture groups ignored
  281. ```
  282. **Sub-LLM calls** — `(llm_query ...)` invokes a sub-LLM with an
  283. interpolated prompt. Works at the top level and nested inside
  284. `map` / `filter` / `reduce` lambdas:
  285. ```scheme
  286. (llm_query "Summarize this") ; bare
  287. (llm_query "Classify: {items}" (items RESULTS)) ; with binding
  288. (map (chunk_by_lines 100)
  289. (lambda c (llm_query "summarize: {chunk}" (chunk c)))) ; OOLONG
  290. (filter RESULTS (lambda x (match (llm_query "keep?: {item}" (item x)) "keep" 0)))
  291. ```
  292. The last two patterns fire **one sub-LLM call per item** — classification
  293. or summarization over an entire document, one chunk at a time, without
  294. pulling any of it into the root model's context.
  295. **Batched sub-LLM** — when per-item calls are independent, `llm_batch`
  296. collapses N serial suspensions into one:
  297. ```scheme
  298. (llm_batch RESULTS (lambda x (llm_query "tag: {item}" (item x))))
  299. ```
  300. Same surface syntax as `map` + `llm_query`, but fires a single
  301. `[LLM_BATCH_REQUEST id=... count=N]` suspension. The client replies once
  302. with a JSON array of N responses via `lattice_llm_batch_respond`.
  303. ~92% round-trip reduction on N=12, ~99% on N=100.
  304. **Constrain responses with `(one_of ...)`** for classification tasks:
  305. ```scheme
  306. (llm_batch RESULTS
  307. (lambda x (llm_query "Rate: {item}" (item x)
  308. (one_of "low" "medium" "high"))))
  309. ```
  310. Validates responses case-insensitively against the allowed values,
  311. making downstream `(filter ...)` / `(count ...)` reliable without
  312. re-normalizing free-text output.
  313. **Add `(calibrate)` for subjective-judgment tasks:**
  314. ```scheme
  315. (llm_batch RESULTS
  316. (lambda x (llm_query "Rate: {item}" (item x)
  317. (one_of "low" "medium" "high")
  318. (calibrate))))
  319. ```
  320. Asks the model to scan all N prompts and establish a consistent relative
  321. scale before answering any. Useful when ratings depend on the distribution
  322. of the corpus rather than being absolute.
  323. **Multi-turn suspension protocol (works with any MCP client):**
  324. When `(llm_query ...)` is evaluated, execution suspends and returns a
  325. `[LLM_QUERY_REQUEST id=...]` message. The MCP client responds via
  326. `lattice_llm_respond` to resume execution. For queries with multiple
  327. `llm_query` calls (e.g., inside `map`), each item triggers one
  328. suspension — respond to each in turn until the final handle stub or
  329. scalar is returned. No special client capabilities (like `sampling`)
  330. are required.
  331. For the native recursive sub-RLM implementation,
  332. use `runRLMFromContent(query, content, { subRLMMaxDepth: 1 })` directly
  333. from the programmatic API — see the Programmatic section below.
  334. #### Memory Pad Usage
  335. ```
  336. 1. lattice_memo(content="<file summary>", label="auth module") → $memo_auth_module stub
  337. 2. lattice_memo(content="<analysis>", label="perf bottlenecks") → $memo_perf_bottlenecks stub
  338. 3. # ... many turns later, need the auth context ...
  339. 4. lattice_expand("$memo_auth_module") → Full summary
  340. 5. lattice_memo_delete("$memo_auth_module") → Drop when stale
  341. ```
  342. Memos don't require a loaded document — they create a session automatically.
  343. Limits: 100 memos, 10MB total. Oldest evicted when exceeded.
  344. ### Programmatic
  345. ```typescript
  346. import { runRLM } from "matryoshka-rlm/rlm";
  347. import { createLLMClient } from "matryoshka-rlm";
  348. const llmClient = createLLMClient("ollama", {
  349. url: "http://localhost:11434/api/generate",
  350. model: "qwen3-coder:30b",
  351. options: { temperature: 0.2 }
  352. });
  353. const result = await runRLM("How many ERROR entries are there?", "./server.log", {
  354. llmClient,
  355. maxTurns: 10,
  356. turnTimeoutMs: 30000,
  357. });
  358. ```
  359. ## Nucleus DSL Reference
  360. ### Search Commands
  361. ```scheme
  362. (grep "pattern") ; Regex search, returns matches with line numbers
  363. (fuzzy_search "query" 10) ; Fuzzy search, returns top N matches with scores
  364. (bm25 "query terms" 10) ; BM25 ranked keyword search (TF-IDF scoring)
  365. (semantic "query terms" 10) ; TF-IDF cosine similarity search
  366. (text_stats) ; Document metadata (length, line count, samples)
  367. (lines 10 20) ; Get specific line range (1-indexed)
  368. ```
  369. ### Multi-Signal Fusion & Ranking
  370. Combine results from multiple search operations for better relevance:
  371. ```scheme
  372. ;; Reciprocal Rank Fusion — merge results from different search signals
  373. (fuse (grep "ERROR") (bm25 "error handling") (semantic "failure"))
  374. ;; Gravity dampening — halve scores for false positives lacking query term overlap
  375. (dampen (bm25 "database error") "database error")
  376. ;; Q-value reranking — learns which lines are useful across turns
  377. (rerank (fuse (grep "ERROR") (bm25 "error")))
  378. ;; Full pipeline: fuse → dampen → rerank
  379. (rerank (dampen (fuse (grep "ERROR") (bm25 "error") (semantic "failure")) "error"))
  380. ```
  381. ### Symbol Operations (Code Files)
  382. For code files, Lattice uses tree-sitter to extract structural symbols. This enables code-aware queries that understand functions, classes, methods, and other language constructs.
  383. **Built-in languages (packages included):**
  384. - TypeScript (.ts, .tsx), JavaScript (.js, .jsx), Python (.py), Go (.go)
  385. - HTML (.html), CSS (.css), JSON (.json)
  386. **Additional languages (install package to enable):**
  387. - Rust, C, C++, Java, Ruby, PHP, C#, Kotlin, Swift, Scala, Lua, Haskell, Bash, SQL, and more
  388. ```scheme
  389. (list_symbols) ; List all symbols (functions, classes, methods, etc.)
  390. (list_symbols "function") ; Filter by kind: "function", "class", "method", "interface", "type", "struct"
  391. (get_symbol_body "myFunc") ; Get source code body for a symbol by name
  392. (get_symbol_body RESULTS) ; Get body for symbol from previous query result
  393. (find_references "myFunc") ; Find all references to an identifier
  394. ```
  395. Symbols include metadata like name, kind, start/end lines, and parent relationships (e.g., methods within classes).
  396. ### Knowledge Graph (Code Structure)
  397. When a code file is loaded, Lattice automatically builds an in-memory knowledge graph that tracks call relationships, inheritance, and interface implementations. This enables structural queries beyond simple text search.
  398. ```scheme
  399. (callers "funcName") ; Who calls this function?
  400. (callees "funcName") ; What does this function call?
  401. (ancestors "ClassName") ; Inheritance chain (extends)
  402. (descendants "ClassName") ; All subclasses (transitive)
  403. (implementations "IFace") ; Classes implementing this interface
  404. (dependents "name") ; All transitive dependents
  405. (dependents "name" 2) ; Dependents within depth limit
  406. (symbol_graph "name" 1) ; Neighborhood subgraph around symbol
  407. ```
  408. The graph is built using line-based heuristics (word-boundary matching for calls, syntax pattern matching for extends/implements), so it produces approximate but useful results without requiring a full language server.
  409. #### Graph Analysis
  410. Community detection and structural insights help you understand codebase architecture:
  411. ```scheme
  412. (communities) ; Detect communities with cohesion scores
  413. (community_of "name") ; Which community does this symbol belong to?
  414. (god_nodes) ; Top 10 most-connected nodes (hubs)
  415. (god_nodes 5) ; Top N most-connected nodes
  416. (surprising_connections) ; Cross-community or low-confidence edges
  417. (bridge_nodes) ; Nodes bridging different communities
  418. (suggest_questions) ; Questions the graph can answer
  419. (graph_report) ; Full analysis (all of the above)
  420. ```
  421. #### Adding Language Support
  422. Matryoshka includes built-in symbol mappings for 20+ languages. To enable a language, install its tree-sitter grammar package:
  423. ```bash
  424. # Enable Rust support
  425. pnpm add tree-sitter-rust
  426. # Enable Java support
  427. pnpm add tree-sitter-java
  428. # Enable Ruby support
  429. pnpm add tree-sitter-ruby
  430. # pnpm 10+ blocks install scripts by default — the grammar packages
  431. # need their native bindings built, so approve them after install:
  432. pnpm approve-builds
  433. ```
  434. **Languages with built-in mappings:**
  435. - TypeScript, JavaScript, Python, Go, Rust, C, C++, Java
  436. - Ruby, PHP, C#, Kotlin, Swift, Scala, Lua, Haskell, Elixir
  437. - HTML, CSS, JSON, YAML, TOML, Markdown, SQL, Bash
  438. Once a package is installed, the language is automatically available for symbol extraction.
  439. #### Custom Language Configuration
  440. For languages without built-in mappings, add a `grammars` entry to your config — see the [Configuration](#configuration) section for the full example and details.
  441. ### Control Flow
  442. ```scheme
  443. (if (count RESULTS) (sum RESULTS) 0) ; Conditional: if/then/else
  444. (add 10 20) ; Arithmetic addition
  445. ```
  446. ### Collection Operations
  447. ```scheme
  448. (filter RESULTS (lambda x (match x "pattern" 0))) ; Filter by regex
  449. (map RESULTS (lambda x (match x "(\\d+)" 1))) ; Extract from each
  450. (sum RESULTS) ; Sum numbers in results
  451. (count RESULTS) ; Count items
  452. ```
  453. ### String Operations
  454. ```scheme
  455. (match str "pattern" 0) ; Regex match, return group N
  456. (replace str "from" "to") ; String replacement
  457. (split str "," 0) ; Split and get index
  458. (parseInt str) ; Parse integer
  459. (parseFloat str) ; Parse float
  460. ```
  461. ### Type Coercion
  462. ```scheme
  463. (parseDate "Jan 15, 2024") ; -> "2024-01-15"
  464. (parseDate "01/15/2024" "US") ; -> "2024-01-15" (MM/DD/YYYY)
  465. (parseCurrency "$1,234.56") ; -> 1234.56
  466. (parseNumber "1,234,567") ; -> 1234567
  467. (coerce value "date") ; General coercion (date/currency/number/boolean/string)
  468. (extract str "\\$[\\d,]+" 0 "currency") ; Extract and coerce in one step
  469. ```
  470. ### Program Synthesis
  471. The model provides constraints (input/output examples), not code — the synthesizer builds programs automatically using Barliman-style relational synthesis with miniKanren.
  472. ```scheme
  473. ; Synthesize from input/output pairs
  474. (synthesize
  475. ("$100" 100)
  476. ("$1,234" 1234)
  477. ("$50,000" 50000))
  478. ; Named functions — synthesize once, apply many times
  479. (define-fn "parse_price" (("$100" 100) ("$1,234" 1234)))
  480. (apply-fn "parse_price" "$50,000") ; -> 50000
  481. ; Boolean classifiers from examples
  482. (predicate "is_error" (("ERROR: timeout" true) ("INFO: ok" false)))
  483. ```
  484. ### Cross-Turn State
  485. Results from previous turns are available:
  486. - `RESULTS` - Latest array result (updated by grep, filter)
  487. - `_1`, `_2`, `_3`, ... - Results from specific turns (1-indexed)
  488. ### Final Answer
  489. ```scheme
  490. <<<FINAL>>>your answer here<<<END>>>
  491. ```
  492. ## Development
  493. ```bash
  494. pnpm test # Run tests
  495. pnpm test -- --coverage # With coverage
  496. RUN_E2E=1 pnpm test -- tests/e2e.test.ts # E2E tests (requires Ollama)
  497. pnpm run build # Build
  498. pnpm run typecheck # Type check
  499. ```
  500. ## Acknowledgements
  501. This project incorporates ideas and code from:
  502. - **[Ori-Mnemos](https://github.com/aayoawoyemi/Ori-Mnemos)** - A persistent memory infrastructure for AI agents implementing the Recursive Memory Harness framework. BM25 search, Reciprocal Rank Fusion, gravity dampening, and Q-value learning reranking were ported from Ori-Mnemos and adapted for line-based document analysis.
  503. - **[Nucleus](https://github.com/michaelwhitford/nucleus)** - A symbolic S-expression language by Michael Whitford. RLM uses Nucleus syntax for the constrained DSL that the LLM outputs, providing a rigid grammar that reduces model errors.
  504. - **[ramo](https://github.com/wjlewis/ramo)** - A miniKanren implementation in TypeScript by Will Lewis. Used for constraint-based program synthesis.
  505. - **[Barliman](https://github.com/webyrd/Barliman)** - A prototype smart editor by William Byrd and Greg Rosenblatt that uses program synthesis to assist programmers. The Barliman-style approach of providing input/output constraints instead of code inspired the synthesis workflow.
  506. - **[tree-sitter](https://tree-sitter.github.io/tree-sitter/)** - A parser generator tool and incremental parsing library. Used for extracting structural symbols (functions, classes, methods) from code files to enable code-aware queries.
  507. ## License
  508. Apache-2.0
  509. ## References
  510. - [RLM Paper](https://arxiv.org/abs/2512.24601)
  511. - [Original Implementation](https://github.com/alexzhang13/rlm)
  512. - [Model Context Protocol](https://modelcontextprotocol.io/)
  513. - [miniKanren](http://minikanren.org/)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.