Skip to content
Back to search

roam-code

Local codebase intelligence CLI + MCP server for AI coding agents: SQLite code graph, 28 languages, 287 commands, 246 MCP tools, change-safety gates, audit evidence, zero API keys.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Cranot/roam-code with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Cranot/roam-code'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/cranot--roam-code/badge.svg)](/listings/cranot--roam-code)

HTML

<a href="/listings/cranot--roam-code"><img src="/api/listings/cranot--roam-code/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/plugin.json · Manifest · 32 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
  3. "name": "roam-code",
  4. "version": "14.1.0",
  5. "description": "Local codebase intelligence + change-safety gates for coding agents. REQUIRES Python 3.10+ and `pip install roam-code` first — this plugin launches the `roam` binary from your PATH, and its MCP server cannot start without it. Pre-indexes your repo's symbols, call graph, deps, and git history into SQLite (28 languages, 100% local, zero API keys), then exposes a lean `core` MCP preset: graph-precise search, callers/impact blast radius, coupling, dead code, taint reachability, and pre-merge verify/critique gates that catch regressions, broken references, AI-slop duplication, and convention drift before they ship.",
  6. "author": {
  7. "name": "Cranot",
  8. "url": "https://github.com/Cranot"
  9. },
  10. "homepage": "https://roam-code.com/",
  11. "repository": "https://github.com/Cranot/roam-code",
  12. "license": "Apache-2.0",
  13. "keywords": [
  14. "codebase-intelligence",
  15. "call-graph",
  16. "blast-radius",
  17. "code-review",
  18. "mergeability",
  19. "taint-analysis",
  20. "mcp",
  21. "local-first"
  22. ],
  23. "mcpServers": {
  24. "roam-code": {
  25. "command": "roam",
  26. "args": ["mcp"],
  27. "env": {
  28. "ROAM_MCP_PRESET": "core"
  29. }
  30. }
  31. }
  32. }
README.md · README · 494 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # roam-code
  2. **Local code analysis and ready-to-run checks for coding agents.**
  3. Roam gives your agent a reusable codebase index, pattern findings, candidate
  4. algorithmic alternatives, and checks it can call as it works. It can investigate
  5. code and evaluate changes without building each analysis from scratch.
  6. [![PyPI version](https://img.shields.io/pypi/v/roam-code?style=flat-square&color=blue)](https://pypi.org/project/roam-code/)
  7. [![GitHub stars](https://img.shields.io/github/stars/Cranot/roam-code?style=flat-square)](https://github.com/Cranot/roam-code/stargazers)
  8. [![CI](https://github.com/Cranot/roam-code/actions/workflows/roam-ci.yml/badge.svg)](https://github.com/Cranot/roam-code/actions/workflows/roam-ci.yml)
  9. [![Python 3.10+](https://img.shields.io/badge/python-3.10+-3776AB?logo=python&logoColor=white)](https://www.python.org/downloads/)
  10. [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
  11. <sub>Runs on your machine · free and open source · no account or API key for local analysis · no automatic source-code or telemetry upload</sub>
  12. <!-- BEGIN auto-count:readme-headline-counts -->
  13. <sub>288 commands · 259 MCP tools (17 in the default `core` preset) · 28 languages</sub>
  14. <!-- END auto-count:readme-headline-counts -->
  15. [Connect your agent](#connect-your-agent) · [Try a task](#try-a-task) ·
  16. [Documentation](https://roam-code.com/docs/) · [Explore the code map](https://roam-code.com/explore)
  17. ## Why Roam is different
  18. Your agent can read code, reason about it, and run tools already. Roam adds
  19. reusable local analyses: query indexed connections between files and functions,
  20. inspect a detected pattern with a proposed alternative, or run a scoped check.
  21. The results give your agent something concrete to inspect, test, and build on.
  22. Static checks use local compute, not model calls or a paid Roam API. Your
  23. agent's model usage, including reading those results, is separate. Refresh the
  24. index with `roam index` as the code changes.
  25. Roam is useful before there is a defect or a patch: trace a dependency, compare
  26. an approach, or establish which checks a change will need. Agents use the tools;
  27. people set direction and decide what ships. It complements your editor, search,
  28. tests, security tools, and code review.
  29. <a id="install--first-four-commands"></a>
  30. ## Connect your agent
  31. You need **Python 3.10+** and a Git repository. On Linux, macOS, or Windows:
  32. ```bash
  33. pip install "roam-code[mcp]"
  34. cd path/to/your/repo
  35. roam init
  36. ```
  37. Replace the path with your project. `init` builds the local index and creates
  38. project configuration; use `roam index` if you want only the index. The first
  39. run can download a parser and takes longer than a refresh.
  40. Then **[choose your agent's setup guide](https://roam-code.com/setup#install-mcp)**.
  41. Connect through MCP (a standard way for agents to use tools), or let your agent
  42. call the CLI. Add the usage instructions to its project configuration; connecting
  43. tools alone does not make the agent use them.
  44. Prefer an isolated installation? Use `pipx install "roam-code[mcp]"` or
  45. `uv tool install "roam-code[mcp]"`. If your agent only needs shell commands,
  46. `pip install roam-code` is enough. [Container setup and limits](docs/containers.md).
  47. <details>
  48. <summary>Inspect a first result yourself</summary>
  49. ```bash
  50. roam health
  51. roam preflight <symbol>
  52. ```
  53. Replace `<symbol>` with a function or class in your project; find it with
  54. `roam search <name>`. Read the returned locations, findings and missing checks,
  55. not just the score. The [historical fresh-install transcript](docs/fresh-install-smoke.md)
  56. shows a recorded example, not current measurements of your checkout.
  57. </details>
  58. ## Try a task
  59. Start with code you know. Ask your agent to use Roam to find a function's
  60. definition and references, then inspect the returned locations together.
  61. Follow the [first-result checklist](https://roam-code.com/docs/integration-tutorials#validate-connection)
  62. to confirm the intended repository, connected tools, and any incomplete results.
  63. | What you want to do | What Roam supplies | What the agent still does |
  64. | --- | --- | --- |
  65. | Find where to start | An overview, definitions and indexed connections | Read the relevant source and check the connections |
  66. | Evaluate an approach | Pattern findings with alternatives from the algorithm catalog | Check semantics, try a change, test and measure it |
  67. | Check a change | Impact analysis, related-test guidance and configured check results | Run the required tests and report missing evidence |
  68. For example, `roam algo` can pair repeated list searches with a set or lookup-table
  69. alternative. That is a candidate, not a proven improvement: value types, updates,
  70. ordering, duplicates and returned positions can change the answer. Check behavior
  71. and measure performance before adopting it. For its MCP tool, choose the `review`
  72. preset and restart the server; it is not in default `core`.
  73. Findings are leads. Static connections can be incomplete; a suggested test list
  74. is not test coverage, and a good health score is not permission to merge.
  75. [Read the evidence limits](docs/concepts/detector-evidence.md).
  76. ## Core commands
  77. <!-- BEGIN auto-count:readme-canonical-mention -->
  78. **Start with these five commands.** Use `understand`, `context`, `retrieve`, `preflight`, and `critique` for everyday exploration and change review. You can discover the rest as you need them: **288 commands (281 canonical + 7 aliases) organised into 7 categories**. An alias is another name for the same command; you do not need to memorize them. The remaining ~283 commands beyond those five are detail surface for specialised workflows — they are called by agents on demand, not memorised.
  79. <!-- END auto-count:readme-canonical-mention -->
  80. | Verb | What it does |
  81. |------|--------------|
  82. | `roam understand` | Get an overview of the project and where to start reading |
  83. | `roam context <symbol>` | Read a definition alongside the code that calls it and the code it calls |
  84. | `roam retrieve "<task>"` | Find useful code for a question such as “trace the login flow” |
  85. | `roam preflight <symbol>` | See what a change could affect, including connected code and tests |
  86. | `roam critique` | Review a patch for related code you may have missed; pipe in `git diff`. High-severity findings exit 5 |
  87. <details>
  88. <summary>Discover the wider command surface</summary>
  89. <!-- BEGIN auto-count:readme-sarif-surface-mention -->
  90. The full surface spans **7 categories** — Getting Started, Daily Workflow, Codebase Health, Architecture, Exploration, Reports & CI, and Refactoring. Run `roam --help` for the 5-verb core, `roam --help-all` for every command name, and `roam surface --json` for the machine-readable inventory. Every command accepts `roam --json <cmd>` for structured output and `roam --sarif <cmd>` for CI integration (SARIF 2.1.0, honoured by 39 commands).
  91. <!-- END auto-count:readme-sarif-surface-mention -->
  92. </details>
  93. <details>
  94. <!-- BEGIN auto-count:readme-cli-command-list-summary -->
  95. <summary><strong>Full command reference — canonical command list (all 281)</strong></summary>
  96. <!-- END auto-count:readme-cli-command-list-summary -->
  97. Use the [complete command index](docs/COMMANDS.md) or the
  98. [command reference with examples](https://roam-code.com/docs/command-reference).
  99. </details>
  100. ## MCP Server
  101. MCP lets an agent call Roam tools directly. Install with the `[mcp]` extra
  102. above and follow the [client-specific setup](https://roam-code.com/docs/integration-tutorials).
  103. The server command is `roam mcp`; the client starts it for the intended project.
  104. <!-- BEGIN auto-count:readme-default-preset -->
  105. **Default preset:** `core` (17 tools: 16 core + `roam_expand_toolset` meta-tool).
  106. <!-- END auto-count:readme-default-preset -->
  107. Select a wider preset for a named task by setting `ROAM_MCP_PRESET` in the
  108. server's environment and restarting it. `roam_expand_toolset` reports what is
  109. available; it does not switch the running server. See [presets and tool schemas](docs/mcp-tools.md).
  110. When consuming results, check errors, `partial_success`, scan scope, freshness,
  111. and any response handle before calling a check complete. Preserve useful partial
  112. findings, but do not treat missing evidence as a clean result.
  113. [CLI evidence handling](docs/agent-cli.md) · [MCP usage](https://roam-code.com/docs/mcp-usage).
  114. <details>
  115. <summary>Default tools and package configuration</summary>
  116. There are 8 selectable presets (`core`, `review`, `refactor`, `debug`, `architecture`, `compliance`, `compile-curated`, `full`).
  117. Choose by the task; the [preset guide](docs/mcp-tools.md) explains their contents.
  118. <!-- BEGIN auto-count:readme-mcp-core-preset-tools -->
  119. Core preset tools: `roam_alerts`, `roam_ask`, `roam_batch_search`, `roam_coupling`, `roam_dead_code`, `roam_deps`, `roam_diagnose_issue`, `roam_fetch_handle`, `roam_file_info`, `roam_grep`, `roam_metrics`, `roam_prepare_change`, `roam_search_symbol`, `roam_taint`, `roam_understand`, `roam_uses`.
  120. <!-- END auto-count:readme-mcp-core-preset-tools -->
  121. `roam_ask` (CLI: `roam ask`) routes a supported question to a recipe from the
  122. 31-recipe registry. It is a deterministic dispatcher, not a model conversation.
  123. <!-- BEGIN auto-count:readme-mcp-tool-list-link -->
  124. The full 259-tool table with descriptions lives in [`docs/mcp-tools.md`](docs/mcp-tools.md).
  125. <!-- END auto-count:readme-mcp-tool-list-link -->
  126. **Which preset ships where.** The Claude Code plugin selects `core` for a
  127. smaller tool list. This repository's own `.mcp.json` selects `full` so agents
  128. working on Roam can test the wider surface. The difference is intentional.
  129. </details>
  130. <!-- MCP Registry ownership token: keep identical to server.json -> name. -->
  131. <!-- mcp-name: io.github.cranot/roam-code -->
  132. ## Integration with AI coding tools
  133. Use the [integration guides](https://roam-code.com/docs/integration-tutorials)
  134. for supported clients and project instructions. Review generated instructions
  135. before adding them to an existing agent configuration.
  136. ```bash
  137. roam describe --agent-prompt
  138. ```
  139. This prints guidance to inspect and incorporate; it does not establish that
  140. your agent has followed the workflow. The [agent CLI guide](docs/agent-cli.md)
  141. covers bounded calls and incomplete output.
  142. <a id="the-compiler--your-agents-first-token-already-knows-the-answer"></a>
  143. ## Context for your agent
  144. Roam's task compiler can attach locally prepared context to a prompt: callers,
  145. recent changes, or source around a reported bug. It makes no model calls.
  146. Optional Claude Code hooks combine that preparation with post-edit checks:
  147. ```bash
  148. roam hooks claude --write
  149. ```
  150. Run from the initialized project. Undo with
  151. `roam hooks claude --uninstall --write`. Compile-time context injection is
  152. fail-open; the edited-turn Stop gate blocks failing, unavailable, malformed or
  153. incomplete verification. Read the [hook compatibility and check workflow](docs/agent-cli.md#automatic-claude-hooks)
  154. before enabling it. Installation does not prove the hooks executed.
  155. [compile-code](https://github.com/Cranot/compile-code) provides a dedicated
  156. driver for the same preparation-and-verification loop.
  157. [Historical comparisons and losses](docs/measurements.md#agent-workflow-comparisons--mayjuly-2026)
  158. are dated experiments, not a promise for your model or repository.
  159. ### Check the work after an edit
  160. `roam verify --auto` chooses checks for changed files. The
  161. [verification workflow](docs/agent-cli.md#check-the-work-after-an-edit) explains
  162. what can run, how missing evidence is handled, and how reviewed exceptions work.
  163. It does not replace your project's required test suite.
  164. ## CI/CD integration
  165. Use the [CI integration guide](docs/ci-integration.md) for GitHub, GitLab,
  166. Jenkins, Azure or Bitbucket. Generate configuration explicitly with
  167. `roam ci-setup --platform github --write` or `roam init --with-ci=github`;
  168. plain `roam init` does not change CI configuration.
  169. <details>
  170. <summary>GitHub Actions example</summary>
  171. ```yaml
  172. # .github/workflows/roam.yml
  173. name: Roam Analysis
  174. on: [pull_request]
  175. jobs:
  176. roam:
  177. runs-on: ubuntu-24.04
  178. steps:
  179. - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
  180. with:
  181. fetch-depth: 0
  182. persist-credentials: false
  183. # For production, replace the tag with the reviewed 40-character SHA it
  184. # points at — a release tag is readable but remains movable.
  185. - uses: Cranot/roam-code@v14.1.0
  186. with:
  187. version: '14.1.0'
  188. commands: health
  189. gate: "score>=70"
  190. sarif: 'true'
  191. comment: 'true'
  192. ```
  193. Review permissions and pin actions to reviewed commit SHAs before adoption.
  194. A quality-score gate is one configured check, not permission to merge.
  195. <!-- BEGIN auto-count:readme-sarif-output-count -->
  196. **SARIF output.** 39 commands honour the global `--sarif` flag (health, complexity, dead, smells, clones, vulns, taint, secrets, n1, …). Minimal upload:
  197. <!-- END auto-count:readme-sarif-output-count -->
  198. ```yaml
  199. - run: roam --sarif health > roam-health.sarif
  200. - uses: github/codeql-action/upload-sarif@03e4368ac7daa2bd82b3e85262f3bf87ee112f57 # v3.36.0
  201. with:
  202. sarif_file: roam-health.sarif
  203. ```
  204. </details>
  205. ## Roam Guard for PRs
  206. `roam guard-pr` gathers a change's review record: what changed, which checks
  207. were required, what ran, and what remains missing. Use `--ci` or `--strict`
  208. when it must gate CI; reporting-only mode can exit 0 with a blocking verdict.
  209. ```bash
  210. roam guard-pr --ci --output guard.md
  211. ```
  212. The [adoption guide](templates/examples/roam-guard-pr.README.md) covers verdicts,
  213. exit codes, rules, and CI setup. [Verification evidence](docs/concepts/verification-evidence.md)
  214. explains what a saved record establishes. Signed records can reveal later
  215. changes to evidence; they do not authenticate every supplied claim or establish
  216. that all relevant checks ran.
  217. ## Best for
  218. Repeated repository investigations, implementation choices, refactor planning,
  219. and change checks in an agent's workflow. Try a real task on your own code and
  220. inspect the evidence, false positives and time involved before adding a gate.
  221. ### When NOT to use Roam
  222. Use an LSP for compiler-backed type resolution, ripgrep for raw text search,
  223. and your test runner for runtime behavior. A small script may be quicker to
  224. read directly. Roam complements these tools; it does not replace them.
  225. ### What's measured vs advisory
  226. The [measurement record](docs/measurements.md) separates historical agent
  227. comparisons, held-out repair-sibling retrieval results, and unmeasured detector
  228. accuracy. It preserves losses and limitations alongside wins. No general
  229. accuracy or savings claim follows from a passing regression suite.
  230. <a id="tier-1--full-extraction-dedicated-parsers"></a>
  231. ## Language Support
  232. Python, JavaScript, TypeScript, Go, Rust, Java, C#, PHP, Ruby, Salesforce and more.
  233. The [language support table](docs/language-support.md) lists extractors and
  234. supported constructs. Extraction depth varies by language and framework;
  235. a recognized file extension is not complete semantic or runtime coverage.
  236. ## Performance
  237. Indexing cost depends on your source, Git history, machine and parser cache.
  238. Refresh with `roam index` after changes; unchanged source can be reused.
  239. [Historical timings and benchmark results](docs/measurements.md#timing-notes--historical-readme-record)
  240. are context, not a latency promise. Measure the commands your workflow uses.
  241. ## How It Works
  242. Roam discovers files, parses source, extracts symbols, resolves references,
  243. and stores the local model in SQLite. Graph analysis, Git history, detector
  244. rules and supported experiments supply different kinds of observations.
  245. [See the architecture](https://roam-code.com/docs/architecture) and
  246. [explore the illustrative code atlas](https://roam-code.com/explore).
  247. Exclude paths with `.roamignore` or `roam config --exclude "*.proto"`.
  248. ## How Roam Compares
  249. Choose Roam for callable local code analysis and checks alongside your agent,
  250. not as another coding agent or a replacement for specialist review.
  251. Its MCP inventory is 259 (17 in default core preset); select tools for the task.
  252. [Compare the roles](https://roam-code.com/compare), then evaluate the outputs on
  253. your own repository: resolution, false positives, latency, and usefulness.
  254. ## Paid layers (free CLI stays Apache 2.0)
  255. The CLI and MCP server are free under Apache 2.0 for individuals and teams.
  256. Static checks use local compute, not model calls; your agent's model charges
  257. are separate. You do not need a paid service to use Roam with your agents.
  258. - **PR Replay — paid report by request.** A written assessment of an agreed
  259. change history, reviewed findings, and a founder walk-through. The existing
  260. [Team and Deep scopes](https://roam-code.com/audit#tiers) define the PR window;
  261. agree how commits map to those PRs before kickoff. Try the free local sample
  262. with `roam pr-replay --tier sample`.
  263. Its default range is `HEAD~5..HEAD`, not necessarily five PRs; your checkout
  264. needs that history. A replay does not prove that an incident would have been prevented.
  265. - **Roam Review — planned, not available to subscribe to.** A proposed hosted
  266. pull-request check using Roam's local analysis. The hosted app, installation
  267. flow, and billing are not built yet.
  268. - **Roam Cloud — planned, not available to subscribe to.** A proposed shared
  269. history of codebase measurements. The local `roam metrics-push` command exists;
  270. that does not establish a hosted dashboard. Inspect its payload with `--dry-run`.
  271. See [pricing and paid help](https://roam-code.com/pricing) and the
  272. [PR Replay sample and scope](https://roam-code.com/audit#sample). Questions go to
  273. [hello@roam-code.com](mailto:hello@roam-code.com); describe the task first, without
  274. source code, credentials, or private reports. Availability and written terms
  275. are agreed before work starts.
  276. ## FAQ
  277. **Does Roam send any data externally?**
  278. Not during ordinary local analysis. Roam does not automatically upload source code, indexes, findings, telemetry, or analytics, and it performs no automatic update check. On first use, `tree-sitter-language-pack` downloads a checksum-verified parser bundle and keeps it in a local cache. Explicit features can contact PyPI, GitHub, user-selected URLs, Roam Cloud, or Sigstore services; [`docs/network-boundary.md`](docs/network-boundary.md) lists every built-in trigger, destination, and payload class. Inspect `roam metrics-push` with `--dry-run` before sending its allow-listed payload.
  279. **Can Roam run in air-gapped environments?**
  280. Yes, after installation and parser prewarming. Run `roam index --force` once while connected on each target platform; the retained bundle lets later grammar loads complete without network access. Avoid the explicit network triggers in the [network-boundary inventory](docs/network-boundary.md), use fixture/file inputs and offline-key signing, and enforce egress policy around project commands launched by `roam verify` or hooks.
  281. **Does Roam modify my source code?**
  282. Read-only by default. Creates `.roam/` with an index database. `roam mutate` (move/rename/extract) defaults to `--dry-run`; pass `--apply` explicitly to write changes.
  283. **Can I control where and when the index is built?**
  284. Set `ROAM_DB_DIR` to redirect the SQLite database and its `index.lock` / `index.state` sidecars together, or save a per-project location with `roam config --set-db-dir`. Set `ROAM_NO_AUTO_INDEX=1` to refuse implicit builds: an analysis command needing a missing or incomplete index exits `3` and emits a structured refusal in JSON mode. `roam index` and `roam init` still build when explicitly invoked. Use `roam index` if you only want to build the index; `init` intentionally also creates project configuration. The storage override does not redirect agent ledgers, response evidence, or other requested artifacts. Stop older indexers before upgrading or changing store locations; do not run mixed-version writers against one store.
  285. **How does Roam handle monorepos and multi-repo projects?**
  286. Monorepos: indexes from the root; batched SQL handles 100k+ symbols. Multi-repo: `roam ws init <repo1> <repo2>` builds a workspace overlay DB for cross-repo API edges, then `roam ws resolve` / `ws context` / `ws trace` work across repos.
  287. **Is Roam compatible with SonarQube / CodeScene?**
  288. Yes — they coexist in the same CI pipeline. SARIF output uploads to GitHub Code Scanning.
  289. **Does Roam satisfy SOC 2 / ISO 42001 / EU AI Act on its own?**
  290. No. Configured Roam workflows can produce supporting records: `ChangeEvidence`
  291. packets, a run ledger, and audit-trail records. Read which checks and supplied
  292. decisions were captured, and which evidence is missing. These records do not
  293. authenticate who acted or prove complete coverage. Roam does not certify;
  294. framework applicability and assessment need appropriate specialist review.
  295. **What's the difference between the free CLI and Roam Review / Cloud / PR Replay?**
  296. The CLI and MCP tools are free under Apache 2.0; their static checks run locally
  297. without model calls. Your agent's model usage is separate. PR Replay offers a
  298. free local sample and paid reports with founder review, by agreed scope.
  299. Roam Review and Roam Cloud are planned hosted products, not available to
  300. subscribe to. See [pricing and availability](https://roam-code.com/pricing).
  301. ## Limitations
  302. - **Static analysis primarily** — can't trace dynamic dispatch, reflection, or eval'd code. Runtime trace ingestion (`roam ingest-trace`) adds production data but requires external trace export.
  303. - **Import resolution is heuristic** — complex re-exports or conditional imports may not resolve.
  304. - **Limited cross-language edges** — Salesforce, Protobuf, REST API, and multi-repo edges are supported, but not arbitrary FFI.
  305. - **Tier 2 languages** get basic symbol extraction only via the generic tree-sitter walker.
  306. - **Large monorepos** (100k+ files) may have slow initial indexing.
  307. ## Troubleshooting
  308. | Problem | Solution |
  309. |---------|----------|
  310. | `roam: command not found` | Ensure install location is on PATH. For `uv`: `uv tool update-shell` |
  311. | `Another indexing process owns the workspace` | Wait for the active writer and run `roam doctor`. A live or unprovable owner requires investigation; preserve `.roam/index.lock` and the lifecycle marker. Proven abandoned generations are recovered by the indexer. |
  312. | `database is locked` | Finish the active indexer or watcher, close other database writers, and check for cloud-sync interference. Then retry `roam index`; rebuilding does not bypass a live SQLite lock. |
  313. | `ROAM_NO_AUTO_INDEX` refusal (exit 3) | Run `roam index` explicitly after selecting the intended project and store, or unset the opt-out to restore automatic cold-start builds. JSON refusals distinguish missing and incomplete indexes without opening the database. |
  314. | Unicode errors on Windows | `chcp 65001` for UTF-8 |
  315. | Symbol resolves to wrong file | Use `file:symbol` syntax: `roam symbol myfile:MyFunction` |
  316. | Health score seems wrong | `roam health --explain` for score contributions; use `roam --json health` for structured findings. The architectural score is separate from test and environment health. |
  317. | Index stale after `git pull` or a commit | `roam index` refreshes source and Git metadata, including commits with unchanged file contents. Use `roam index --force` to rebuild derived index data. |
  318. See the [troubleshooting guide](https://roam-code.com/docs/troubleshooting) and
  319. [repository maintenance guide](docs/repository-maintenance.md) for environment
  320. repair, index recovery, and the meaning of doctor advisories.
  321. ## Go deeper
  322. - [Documentation map](docs/README.md) — maintained guides and their owners.
  323. - [Understanding Roam](docs/understanding-roam.md) — product model and evidence boundaries.
  324. - [Network boundary](docs/network-boundary.md) — downloads, online features and payloads.
  325. - [MCP security posture](dev/MCP-SECURITY-POSTURE.md) — gateway integration and receipt limits.
  326. ## Walkthrough
  327. Follow the [worked example](https://roam-code.com/docs/canonical-demo) from a
  328. first scan to a configured review record, or browse
  329. [illustrative changes](https://roam-code.com/docs/demos).
  330. <details>
  331. <summary>Watch a recorded terminal demo</summary>
  332. ![roam terminal demo](docs/assets/roam-terminal-demo.gif)
  333. This is a historical CLI recording, not a live run or a current-output contract.
  334. Use the linked guides above for the maintained workflow.
  335. </details>
  336. ## What's New
  337. <a id="v133-released-2026-05-19--mcp-runtime-security--ux-polish"></a>
  338. <a id="v132-released-2026-05-16--evidence-freshness--resolution-disclosure"></a>
  339. <a id="v131-released-2026-05-15--pattern-2-propagation--shared-yaml-helper--3-flagship-silent-fallback-seals"></a>
  340. <a id="v130-released-2026-05-13--agent-os-substrate--laravel-idioms--vue-sfc"></a>
  341. Read the [changelog](CHANGELOG.md) for released changes and the `Unreleased`
  342. section for work awaiting a release. A source checkout may be ahead of the
  343. package installed from PyPI.
  344. ## Update / Uninstall
  345. ```bash
  346. # Update
  347. pipx upgrade roam-code # or: uv tool upgrade roam-code / pip install --upgrade roam-code
  348. # Uninstall
  349. pipx uninstall roam-code # or: uv tool uninstall roam-code / pip uninstall roam-code
  350. ```
  351. Before removing `.roam/`, back up any rules, annotations, memory, signed run
  352. ledgers, keys, or proof bundles you need to retain. That directory contains
  353. project state as well as the rebuildable index; uninstalling the package does
  354. not require deleting it.
  355. ## Contributing
  356. ```bash
  357. git clone https://github.com/Cranot/roam-code.git
  358. cd roam-code
  359. uv sync --locked --no-default-groups --extra dev --group ci --python 3.12
  360. uv run --no-sync pytest tests/test_basic.py -n 0
  361. ```
  362. Follow [CONTRIBUTING.md](CONTRIBUTING.md) for the full test/release gates and
  363. [the documentation map](docs/README.md) for maintained guides and references.
  364. Good first contributions: add a [Tier 1 language](src/roam/languages/) (see `go_lang.py` or `php_lang.py` as templates), improve reference resolution, add benchmark repos, extend SARIF converters, add MCP tools. Please open an issue first to discuss larger changes.
  365. ## License
  366. [Apache 2.0](LICENSE)
benchmarks/agent-eval/evaluate.py · Bundled script · 432 lines · 4 findings marked
  1. #!/usr/bin/env python3
  2. # ruff: noqa: T201 -- CLI progress and result output is intentional.
  3. """
  4. Evaluate a single agent workspace using roam-code.
  5. Usage:
  6. python evaluate.py <workspace_path> [--output results/result.json]
  7. Runs roam init + all analysis commands, collects scores into a structured JSON result.
  8. """
  9. from __future__ import annotations
  10. import argparse
  11. import importlib.util
  12. import json
  13. import subprocess
  14. import sys
  15. import time
  16. from collections import defaultdict
  17. from functools import lru_cache
  18. from pathlib import Path
  19. from types import ModuleType
  20. # Agent CLI version + model signatures
  21. AGENT_SIGNATURES = {
  22. "claude-code": {
  23. "cli_cmd": "claude",
  24. "version_cmd": ["claude", "--version"],
  25. "model": "claude-opus-4-6",
  26. "invoke": "claude --model opus --dangerously-skip-permissions -p",
  27. },
  28. "claude-code-sonnet": {
  29. "cli_cmd": "claude",
  30. "version_cmd": ["claude", "--version"],
  31. "model": "claude-sonnet-4-5-20250929",
  32. "invoke": "claude --model sonnet --dangerously-skip-permissions -p",
  33. },
  34. "codex": {
  35. "cli_cmd": "codex",
  36. "version_cmd": ["codex", "--version"],
  37. "model": "gpt-5.3-codex",
  38. "invoke": "codex exec --dangerously-bypass-approvals-and-sandbox",
  39. },
  40. "gemini-cli": {
  41. "cli_cmd": "gemini",
  42. "version_cmd": ["gemini", "--version"],
  43. "model": "gemini-3-pro-preview",
  44. "invoke": "gemini --yolo",
  45. },
  46. }
  47. @lru_cache(maxsize=1)
  48. def _load_scoring_module() -> ModuleType:
  49. """Load sibling scoring helpers without creating a phantom top-level import."""
  50. module_path = Path(__file__).with_name("scoring.py")
  51. module_name = "_agent_eval_scoring"
  52. spec = importlib.util.spec_from_file_location(module_name, module_path)
  53. if spec is None or spec.loader is None:
  54. raise ImportError(f"Could not load scoring helpers from {module_path}")
  55. module = importlib.util.module_from_spec(spec)
  56. sys.modules[module_name] = module
  57. spec.loader.exec_module(module)
  58. return module
  59. def get_agent_signature(agent: str) -> dict:
  60. """Detect CLI version and return full agent signature."""
  61. sig = AGENT_SIGNATURES.get(agent, {})
  62. if not sig:
  63. return {"agent": agent, "cli_version": "unknown", "model": "unknown"}
  64. cli_version = "unknown"
  65. version_cmd = sig.get("version_cmd")
  66. if version_cmd:
  67. try:
  68. result = subprocess.run(
  69. version_cmd,
  70. capture_output=True,
  71. text=True,
  72. timeout=10,
  73. )
  74. cli_version = result.stdout.strip().split("\n")[0]
  75. except Exception:
  76. pass
  77. roam_version = "unknown"
  78. try:
  79. result = subprocess.run(
  80. ["roam", "--version"],
  81. capture_output=True,
  82. text=True,
  83. timeout=10,
  84. )
  85. roam_version = result.stdout.strip().split("\n")[0]
  86. except Exception:
  87. pass
  88. return {
  89. "agent": agent,
  90. "cli_version": cli_version,
  91. "model": sig.get("model", "unknown"),
  92. "invoke_command": sig.get("invoke", "unknown"),
  93. "roam_version": roam_version,
  94. }
  95. def run_roam(workspace: Path, command: str, timeout: int = 120) -> dict | None:
  96. """Run a roam command with --json and return parsed output."""
  97. try:
  98. result = subprocess.run(
  99. ["roam", "--json", command],
  100. cwd=str(workspace),
  101. capture_output=True,
  102. text=True,
  103. timeout=timeout,
  104. )
  105. if result.returncode == 0 and result.stdout.strip():
  106. try:
  107. return json.loads(result.stdout)
  108. except json.JSONDecodeError:
  109. return {"raw_output": result.stdout.strip(), "parse_error": True}
  110. return {
  111. "error": result.stderr.strip() or f"exit code {result.returncode}",
  112. "returncode": result.returncode,
  113. }
  114. except subprocess.TimeoutExpired:
  115. return {"error": "timeout", "timeout_seconds": timeout}
  116. except FileNotFoundError:
  117. return {"error": "roam not found in PATH"}
  118. def run_roam_init(workspace: Path, timeout: int = 300) -> dict:
  119. """Run roam init and return status."""
  120. try:
  121. result = subprocess.run(
  122. ["roam", "init"],
  123. cwd=str(workspace),
  124. capture_output=True,
  125. text=True,
  126. timeout=timeout,
  127. )
  128. return {
  129. "success": result.returncode == 0,
  130. "stdout": result.stdout.strip()[-500:], # last 500 chars
  131. "stderr": result.stderr.strip()[-500:],
  132. }
  133. except subprocess.TimeoutExpired:
  134. return {"success": False, "error": "timeout"}
  135. except FileNotFoundError:
  136. return {"success": False, "error": "roam not found in PATH"}
  137. def check_git_init(workspace: Path) -> bool:
  138. """Ensure workspace is a git repo (roam requires it)."""
  139. git_dir = workspace / ".git"
  140. if git_dir.exists():
  141. return True
  142. # Initialize git if needed
  143. try:
  144. subprocess.run(
  145. ["git", "init"],
  146. cwd=str(workspace),
  147. capture_output=True,
  148. timeout=30,
  149. )
  150. subprocess.run(
  151. ["git", "add", "-A"],
  152. cwd=str(workspace),
  153. capture_output=True,
  154. timeout=30,
  155. )
  156. subprocess.run(
  157. ["git", "commit", "-m", "initial"],
  158. cwd=str(workspace),
  159. capture_output=True,
  160. timeout=30,
  161. )
  162. return True
  163. except Exception:
  164. return False
  165. def count_files(workspace: Path) -> dict:
  166. """Count files by type in workspace."""
  167. counts: defaultdict[str, int] = defaultdict(int)
  168. total_lines = 0
  169. total_files = 0
  170. for f in workspace.rglob("*"):
  171. if f.is_file() and ".git" not in f.parts and "node_modules" not in f.parts:
  172. ext = f.suffix.lower() or "(no ext)"
  173. counts[ext] += 1
  174. total_files += 1
  175. try:
  176. total_lines += len(f.read_text(encoding="utf-8", errors="ignore").splitlines())
  177. except Exception:
  178. pass
  179. return {
  180. "total_files": total_files,
  181. "total_lines": total_lines,
  182. "by_extension": dict(sorted(counts.items(), key=lambda x: -x[1])),
  183. }
  184. def check_tests_exist(workspace: Path) -> dict:
  185. """Check if test files exist."""
  186. test_patterns = [
  187. "**/*test*.*",
  188. "**/*spec*.*",
  189. "**/test_*.*",
  190. "**/tests/**/*.*",
  191. "**/__tests__/**/*.*",
  192. ]
  193. test_files = set()
  194. for pattern in test_patterns:
  195. for f in workspace.glob(pattern):
  196. if f.is_file() and ".git" not in str(f) and "node_modules" not in str(f):
  197. test_files.add(str(f.relative_to(workspace)))
  198. return {
  199. "tests_found": len(test_files) > 0,
  200. "test_file_count": len(test_files),
  201. "test_files": sorted(test_files)[:20], # cap at 20
  202. }
  203. def check_readme_exists(workspace: Path) -> bool:
  204. """Check if README exists."""
  205. for name in ["README.md", "README.rst", "README.txt", "README"]:
  206. if (workspace / name).exists():
  207. return True
  208. return False
  209. def check_build_config(workspace: Path) -> dict:
  210. """Check for build/project config files."""
  211. configs = {
  212. "package.json": "node",
  213. "pyproject.toml": "python",
  214. "setup.py": "python",
  215. "CMakeLists.txt": "cmake",
  216. "Makefile": "make",
  217. "go.mod": "go",
  218. "Cargo.toml": "rust",
  219. "vite.config.js": "vite",
  220. "vite.config.ts": "vite",
  221. "astro.config.mjs": "astro",
  222. }
  223. found = {}
  224. for filename, build_type in configs.items():
  225. if (workspace / filename).exists():
  226. found[filename] = build_type
  227. return {
  228. "has_build_config": len(found) > 0,
  229. "configs_found": found,
  230. }
  231. def evaluate_workspace(workspace: Path) -> dict:
  232. """Run full evaluation on a workspace. Returns structured results."""
  233. results = {
  234. "workspace": str(workspace),
  235. "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
  236. "file_stats": {},
  237. "structure": {},
  238. "roam": {},
  239. "scores": {},
  240. }
  241. # --- File stats ---
  242. results["file_stats"] = count_files(workspace)
  243. results["structure"]["tests"] = check_tests_exist(workspace)
  244. results["structure"]["readme"] = check_readme_exists(workspace)
  245. results["structure"]["build"] = check_build_config(workspace)
  246. # --- Git init (roam needs it) ---
  247. if not check_git_init(workspace):
  248. results["roam"]["error"] = "Could not initialize git"
  249. results["scores"] = _empty_scores()
  250. return results
  251. # --- Roam init ---
  252. print(" Running roam init...")
  253. init_result = run_roam_init(workspace)
  254. results["roam"]["init"] = init_result
  255. if not init_result.get("success"):
  256. results["roam"]["error"] = "roam init failed"
  257. results["scores"] = _empty_scores()
  258. return results
  259. # --- Roam analysis commands ---
  260. commands = ["health", "dead", "complexity", "coupling"]
  261. for cmd in commands:
  262. print(f" Running roam {cmd}...")
  263. results["roam"][cmd] = run_roam(workspace, cmd)
  264. # --- Extract scores ---
  265. results["scores"] = extract_scores(results["roam"])
  266. # --- Composite AQS ---
  267. scoring = _load_scoring_module()
  268. aqs = scoring.compute_aqs(results)
  269. results["aqs"] = aqs
  270. return results
  271. def extract_scores(roam_results: dict) -> dict:
  272. """Extract numeric scores from roam JSON output."""
  273. scores = {}
  274. # Health score (0-100) — also includes tangle_ratio
  275. health = roam_results.get("health")
  276. if health and isinstance(health, dict):
  277. scores["health"] = health.get("health_score", health.get("summary", {}).get("health_score"))
  278. scores["health_verdict"] = health.get("summary", {}).get("verdict")
  279. scores["tangle_ratio"] = health.get("tangle_ratio", health.get("summary", {}).get("tangle_ratio"))
  280. scores["propagation_cost"] = health.get("propagation_cost")
  281. scores["issue_count"] = health.get("issue_count")
  282. severity = health.get("severity", health.get("summary", {}).get("severity", {}))
  283. scores["critical_issues"] = severity.get("CRITICAL", 0)
  284. scores["warning_issues"] = severity.get("WARNING", 0)
  285. # Dead code count
  286. dead = roam_results.get("dead")
  287. if dead and isinstance(dead, dict):
  288. summary = dead.get("summary", {})
  289. # sum safe + review counts (intentional are OK)
  290. scores["dead_symbols"] = summary.get("safe", 0) + summary.get("review", 0)
  291. # Complexity
  292. complexity = roam_results.get("complexity")
  293. if complexity and isinstance(complexity, dict):
  294. summary = complexity.get("summary", {})
  295. scores["avg_complexity"] = summary.get("average_complexity")
  296. scores["p90_complexity"] = summary.get("p90_complexity")
  297. scores["high_complexity_count"] = summary.get("high_count", 0)
  298. scores["critical_complexity_count"] = summary.get("critical_count", 0)
  299. # Coupling
  300. coupling = roam_results.get("coupling")
  301. if coupling and isinstance(coupling, dict):
  302. summary = coupling.get("summary", {})
  303. scores["coupling_pairs"] = summary.get("pairs", 0)
  304. scores["hidden_coupling"] = summary.get("hidden_coupling", 0)
  305. return scores
  306. def _empty_scores() -> dict:
  307. return {
  308. "health": None,
  309. "dead_symbols": None,
  310. "avg_complexity": None,
  311. "p90_complexity": None,
  312. "high_complexity_count": None,
  313. "critical_complexity_count": None,
  314. "tangle_ratio": None,
  315. "propagation_cost": None,
  316. "coupling_pairs": None,
  317. "hidden_coupling": None,
  318. "critical_issues": None,
  319. "warning_issues": None,
  320. }
  321. def main():
  322. parser = argparse.ArgumentParser(description="Evaluate agent workspace with roam")
  323. parser.add_argument("workspace", type=Path, help="Path to agent workspace")
  324. parser.add_argument("--output", "-o", type=Path, help="Output JSON file")
  325. parser.add_argument("--agent", type=str, help="Agent name (claude-code, codex, gemini-cli)")
  326. parser.add_argument("--mode", type=str, help="Mode (vanilla, roam-cli, roam-mcp)")
  327. parser.add_argument("--task", type=str, help="Task ID (react-todo, etc.)")
  328. args = parser.parse_args()
  329. workspace = args.workspace.resolve()
  330. if not workspace.is_dir():
  331. print(f"Error: {workspace} is not a directory")
  332. sys.exit(1)
  333. print(f"Evaluating: {workspace}")
  334. results = evaluate_workspace(workspace)
  335. # Add metadata
  336. if args.agent:
  337. results["agent"] = args.agent
  338. results["signature"] = get_agent_signature(args.agent)
  339. if args.mode:
  340. results["mode"] = args.mode
  341. if args.task:
  342. results["task"] = args.task
  343. # Output
  344. output_json = json.dumps(results, indent=2, default=str)
  345. if args.output:
  346. args.output.parent.mkdir(parents=True, exist_ok=True)
  347. args.output.write_text(output_json, encoding="utf-8")
  348. print(f"\nResults saved to: {args.output}")
  349. else:
  350. print("\n" + output_json)
  351. # Print summary
  352. scores = results.get("scores", {})
  353. print("\n=== SCORE SUMMARY ===")
  354. print(f" Health: {scores.get('health', 'N/A')}")
  355. print(f" Dead symbols: {scores.get('dead_symbols', 'N/A')}")
  356. print(f" Avg complexity: {scores.get('avg_complexity', 'N/A')}")
  357. print(f" P90 complexity: {scores.get('p90_complexity', 'N/A')}")
  358. print(f" High complexity: {scores.get('high_complexity_count', 'N/A')}")
  359. print(f" Tangle ratio: {scores.get('tangle_ratio', 'N/A')}")
  360. print(f" Propagation cost: {scores.get('propagation_cost', 'N/A')}")
  361. print(f" Coupling pairs: {scores.get('coupling_pairs', 'N/A')}")
  362. print(f" Hidden coupling: {scores.get('hidden_coupling', 'N/A')}")
  363. print(f" Critical issues: {scores.get('critical_issues', 'N/A')}")
  364. print(f" Warning issues: {scores.get('warning_issues', 'N/A')}")
  365. print(f" Files: {results['file_stats'].get('total_files', 'N/A')}")
  366. print(f" Lines: {results['file_stats'].get('total_lines', 'N/A')}")
  367. print(f" Tests found: {results['structure']['tests'].get('test_file_count', 0)}")
  368. # Print AQS
  369. aqs = results.get("aqs", {})
  370. if aqs:
  371. print("\n=== AGENT QUALITY SCORE ===")
  372. scoring = _load_scoring_module()
  373. print(scoring.format_aqs_report(aqs))
  374. if __name__ == "__main__":
  375. main()

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.