sem
Semantic version control => entity-level diffs, blame, and impact analysis on top of git. 28 languages via tree-sitter. Built for coding agents.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Ataraxy-Labs/sem with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Ataraxy-Labs/sem'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/ataraxy-labs--sem)HTML
<a href="/listings/ataraxy-labs--sem"><img src="/api/listings/ataraxy-labs--sem/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
These steps run a script from the network that findsafeskills does not scan.
curl -fsSL https://raw.githubusercontent.com/Ataraxy-Labs/sem/main/install.sh | sh
Or via Homebrew:
brew install sem-cli
Or via winget on Windows:
winget install AtaraxyLabs.sem
Or via Scoop on Windows:
scoop install sem
Or install the npm wrapper into `node_modules`:
npm install --save-dev @ataraxy-labs/sem
With Bun, trust the package so its `postinstall` script can download the binary:
bun add -d @ataraxy-labs/sem bun pm trust @ataraxy-labs/sem
Once installed, update to the latest release any time:
sem update
Or via cargo, from [crates.io](https://crates.io/crates/sem-cli):
cargo install sem-cli
Or build the latest `main` from source (requires Rust):
cargo install --git https://github.com/Ataraxy-Labs/sem sem-cli
Or grab a binary from [GitHub Releases](https://github.com/Ataraxy-Labs/sem/releases).
Or run via Docker:
docker build -t sem . docker run --rm -it -u "$(id -u):$(id -g)" -v "$(pwd):/repo" sem diff
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 41 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # sem
- Semantic version control CLI. Entity-level diffs for Git (functions, classes, methods instead of lines).
- ## Structure
- Cargo workspace at `crates/`:
- - `sem-core` — entity extraction engine, tree-sitter parsers, dependency graph
- - `sem-cli` — CLI binary (`sem find`, `sem grep`, `sem impact`, `sem check`, `sem certify`, `sem diff`, `sem graph`, `sem history`)
- ## Build & Test
- ```bash
- cargo build --release -p sem-cli # binary at target/release/sem
- cargo test --workspace # 134 tests
- ```
- All cargo commands run from repo root. CI builds from `crates/` directory.
- ## Key Paths
- - Parsers: `crates/sem-core/src/parser/plugins/` (24 language plugins + shebang detection)
- - Commands: `crates/sem-cli/src/commands/`
- - Entity model: `crates/sem-core/src/model/`
- - Git integration: `crates/sem-core/src/git/`
- ## Adding a Language
- 1. Add tree-sitter grammar to `sem-core/Cargo.toml`
- 2. Create parser plugin in `sem-core/src/parser/plugins/`
- 3. Register in `sem-core/src/parser/mod.rs`
- 4. Add tests
- 5. See `CONTRIBUTING.md` for the full 7-step guide
- ## Conventions
- - Entity granularity: functions, classes, methods, structs, traits, interfaces
- - Methods have a `parentId` linking to their class
- - `structural_hash` for matching entities across versions
- - Release on tag push (`v*`)
- - License: MIT OR Apache-2.0
README.md · README · 750 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- > **Part of the [Ataraxy Labs](https://ataraxy-labs.com) stack**: agent-native infrastructure for software development. See also: [weave](https://ataraxy-labs.com/weave) (entity-level git merge driver) · [inspect](https://github.com/Ataraxy-Labs/inspect) (semantic code review) · [opensessions](https://github.com/Ataraxy-Labs/opensessions) (tmux sidebar for coding agents).
- >
- > Read the manifesto: https://ataraxy-labs.com/#thesis · Essays: https://ataraxy-labs.com/blogs · LLMs: https://ataraxy-labs.com/llms.txt
- <p align="center">
- <img src="assets/banner.svg" alt="sem" width="600" />
- </p>
- <p align="center">
- <a href="https://trendshift.io/repositories/25348" target="_blank"><img src="https://trendshift.io/api/badge/repositories/25348" alt="Ataraxy-Labs%2Fsem | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
- </p>
- <p align="center">
- <strong>Semantic version control built on Git.</strong><br>
- Instead of lines changed, sem tells you what entities changed: functions, methods, classes.
- </p>
- <p align="center">
- <a href="https://ataraxy-labs.com/blogs/code-is-not-text">Why sem?</a> ·
- <a href="#using-sem-with-agents">Using sem with agents</a> ·
- <a href="#install">Install</a> ·
- <a href="#commands">Commands</a> ·
- <a href="#use-with-ai-agents-mcp">Agents (MCP)</a> ·
- <a href="docs/cloud-consent.html">Cloud consent</a> ·
- <a href="https://github.com/Ataraxy-Labs/sem/releases/latest">Releases</a>
- </p>
- <p align="center">
- <a href="https://github.com/Ataraxy-Labs/sem/releases/latest"><img src="https://img.shields.io/github/v/release/Ataraxy-Labs/sem?color=blue&label=release" alt="Release"></a>
- <img src="https://img.shields.io/badge/rust-stable-orange" alt="Rust">
- <img src="https://img.shields.io/badge/tests-900%2B_passing-brightgreen" alt="Tests">
- <a href="LICENSE-MIT"><img src="https://img.shields.io/badge/license-MIT-yellow" alt="License"></a>
- <img src="https://img.shields.io/badge/languages-32-blue" alt="Languages">
- </p>
- sem is a semantic version control tool that works on top of Git. It parses your code with tree-sitter, extracts every function, class, and method as an entity, and diffs at the entity level instead of lines. This means you see "function `blahh` was modified" instead of "lines x-y changed."
- It works in any Git repo with no setup.
- Cloud-backed queries are opt-in per repo: logging in does not upload a repo or send a query. See the [cloud consent flow](docs/cloud-consent.html) for the public/private repo states, preview screen, local audit log, and forget controls.
- <p align="center">
- <img src="assets/terminal.svg" alt="sem diff" width="800" />
- </p>
- ## Using sem with agents
- Four questions, four verbs. Every verb takes `--json`.
- | Question | Verb |
- |---|---|
- | Where is it? | `sem find`, `sem grep` |
- | What does my change touch? | `sem impact` |
- | Is it correct? | `sem check` |
- | What should a human review? | `sem certify` |
- **Where is it?** Find a function, class or method by name:
- ```bash
- sem find parse_config
- ```
- ```
- function parse_config src/config.py:1
- ```
- Who calls it (`--callers`), what it uses (`--refs`), or its code with its callers and callees (`--context`):
- ```bash
- sem find parse_config --callers
- ```
- ```
- function parse_config src/config.py:1
- function load src/config.py:6
- function test_parse_config tests/test_config.py:4
- ```
- Search text, like `rg`:
- ```bash
- sem grep splitlines
- ```
- ```
- src/config.py:2: pairs = [line.split("=", 1) for line in text.splitlines() if line]
- ```
- **What does my change touch?** Everything that depends on an entity, and the tests to run:
- ```bash
- sem impact parse_config --tests
- ```
- ```
- ⚡ 1 tests affected:
- tests/test_config.py
- function test_parse_config (L4–5)
- ```
- `sem impact --diff HEAD --tests` answers the same for your whole uncommitted change.
- **Is it correct?** Run the project's own compiler, type checker, linter and tests, rechecking only what the change can affect when that gives the same answer. Exit 0 means pass, 1 fail, 2 could not decide:
- ```bash
- sem check
- ```
- ```
- cmd FAIL full sh (0 rechecked, 279 ms)
- `python3 -m pytest -q tests` exited 1:
- E AssertionError: assert {'a': '1'} == {'a': '2'}
- ```
- **What should a human review?** A review certificate for a commit range: what changed, which callers the change leaves behind, which tests reach it:
- ```bash
- sem certify main..HEAD
- ```
- ```
- # sem certificate 94f4bd74e26e..d73d4d092b43
- 1 files, 2 entities changed (1 added, 1 modified, 0 deleted, 0 renamed, 0 moved).
- - modified `parse_config` (src/config.py:1): 2 static caller(s) in 2 file(s) that this change does not modify
- ```
- For agents that speak MCP, `sem mcp` serves the same verbs as tools (see [Use with AI agents](#use-with-ai-agents-mcp)).
- ## Install
- ```bash
- curl -fsSL https://raw.githubusercontent.com/Ataraxy-Labs/sem/main/install.sh | sh
- ```
- Or via Homebrew:
- ```bash
- brew install sem-cli
- ```
- Or via winget on Windows:
- ```powershell
- winget install AtaraxyLabs.sem
- ```
- Or via Scoop on Windows:
- ```powershell
- scoop install sem
- ```
- Or install the npm wrapper into `node_modules`:
- ```bash
- npm install --save-dev @ataraxy-labs/sem
- ```
- With Bun, trust the package so its `postinstall` script can download the binary:
- ```bash
- bun add -d @ataraxy-labs/sem
- bun pm trust @ataraxy-labs/sem
- ```
- Once installed, update to the latest release any time:
- ```bash
- sem config update
- ```
- Or via cargo, from [crates.io](https://crates.io/crates/sem-cli):
- ```bash
- cargo install --locked sem-cli
- ```
- Or build the latest `main` from source (requires Rust):
- ```bash
- cargo install --locked --git https://github.com/Ataraxy-Labs/sem sem-cli
- ```
- Or grab a binary from [GitHub Releases](https://github.com/Ataraxy-Labs/sem/releases).
- Or run via Docker:
- ```bash
- docker build -t sem .
- docker run --rm -it -u "$(id -u):$(id -g)" -v "$(pwd):/repo" sem diff
- ```
- ## Name conflict with GNU Parallel
- GNU Parallel ships a `sem` binary (`/usr/bin/sem`) as a symlink to `parallel`. If you have both installed, they'll collide. Run `sem --version` to check which one you're using. ([#77](https://github.com/Ataraxy-Labs/sem/issues/77))
- **Quick fixes:**
- ```bash
- # Option 1: alias in your shell profile (~/.bashrc, ~/.zshrc)
- alias sem="$HOME/.cargo/bin/sem"
- # Option 2: make sure cargo bin comes first in PATH
- export PATH="$HOME/.cargo/bin:$PATH"
- # Option 3: if installed via Homebrew
- export PATH="$(brew --prefix)/bin:$PATH"
- ```
- If you installed via npm/bun, the binary lives in `node_modules/.bin/sem` and is invoked through `npx sem` or `bunx sem`, which avoids the conflict entirely.
- ## Commands
- Works in any Git repo. No setup required. Also works outside Git for arbitrary file comparison.
- | Verb | The question it answers |
- |---|---|
- | `sem find` | Where is it defined? Who calls it, what does it use, what is around it? |
- | `sem grep` | Where does this text appear? |
- | `sem impact` | What does changing this entity, or this diff, touch? |
- | `sem check` | Is my change correct? |
- | `sem certify` | What should a human review in this commit range? |
- | `sem diff` | Which functions and classes changed? |
- | `sem graph` | How is the code connected? |
- | `sem history` | How did this entity change over time? Who last changed it? |
- | `sem cloud` | Account, cloud reviews, cross-repo queries |
- | `sem config` | git diff integration, telemetry, completions, updates, stats |
- | `sem mcp` | The same verbs as MCP tools for agents |
- `sem <verb> --help` shows every flag with an example.
- sem stores its SQLite entity cache outside the repository, under the OS cache directory by default. Set `SEM_CACHE_DIR=/path/to/cache` to override the cache root; repo-local overrides are ignored so cache files do not dirty the working tree.
- ### sem find
- Find entity definitions by name, then ask about one: who calls it, what it uses, or its code with the code around it. Answers come from an on-disk query index (`index.sem`, next to the entity cache); the first call in a repo builds it, later calls read it directly, with no daemon.
- ```bash
- sem find "function diff_command" # where it is defined ("type name" narrows by kind)
- sem find diff_command load_config # several names in one call
- sem find diff_command --callers # who calls it; says when the caller set may be incomplete
- sem find diff_command --refs # what it calls and references
- sem find diff_command --context # its code plus callers and callees, within a token budget
- sem find diff_command --context --budget 4000 --hops 1
- sem find --in src/auth.ts # every entity in a file or directory
- sem find diff_command --in src/ # only definitions under src/
- sem find --text "retry budget" # entity bodies containing a string, named by entity
- sem find diff_command --json # JSON on any of the above
- ```
- With `--context`, when the target signature itself does not fit, JSON output reports `target_omitted: true`.
- ### sem grep
- Text search across source files, rg-compatible `file:line:text` output, served from the index's trigram postings when one exists.
- ```bash
- sem grep "TODO"
- sem grep -i todo src/ # case-insensitive, under src/
- sem grep -e foo -e bar --json # several patterns, hits kept apart
- ```
- ### sem impact
- What breaks if an entity changes: its dependencies, its dependents (transitively), and the tests that reach it.
- ```bash
- sem impact authenticateUser # full impact
- sem impact authenticateUser --deps # direct dependencies only
- sem impact authenticateUser --dependents
- sem impact authenticateUser --tests # the tests to run
- sem impact authenticateUser --file src/auth.ts # disambiguate by file
- sem impact --diff HEAD --tests # tests for the uncommitted change
- sem impact --diff main..HEAD --json # one impact report per changed entity
- ```
- With `--diff` and `--tests` in a JS/TS workspace, the answer is the module graph's affected-test selection over the changed files (`sem graph --modules affected-tests`); elsewhere it is the entity graph's. `--no-default-excludes` includes generated, fixture, vendor, benchmark and build trees.
- ### sem check
- Runs the project's compiler, type checker, linter and tests on the working tree and prints one verdict: exit 0 pass, 1 fail, 2 could not decide. Each checker gives the verdict the real tool would give on the whole project; it rechecks only what the change can affect when that is provably the same answer, and otherwise runs the tool in full and says why. Nothing to check is 2, never a pass.
- ```bash
- sem check # every checker the project has
- sem check --checkers ts,lint,tests # only these
- sem check --base origin/main # against origin/main instead of HEAD
- sem check --promises # also prove every promise in .sem/promises can fail
- sem check --json # one JSON object with a verification certificate
- ```
- Checkers are detected from the project. TypeScript, lint and JS tests (vitest, jest) follow the module graph, Go rechecks affected packages, Python type checking (pyright scoped by imports, or mypy) and pytest rerun only the files a change can reach, and C and C++ recompile only the translation units in `compile_commands.json` that include a changed file. Cargo, Gradle or Maven, dotnet and SwiftPM run their own build, which already rebuilds only what changed, and are skipped outright when no input changed since the base passed (a TypeScript edit never reruns the Rust build).
- An edit inside the body of a Python function that declares its return type cannot change what other files see, so only that file is rechecked.
- Commands of your own go in `.sem/check.json`, e.g. `{"commands": [{"run": "make lint", "inputs": ["src/**"]}]}`. A command that lists its `inputs` is skipped when none of them changed since it passed; one without runs every time.
- ### sem certify
- A review certificate for a commit range: entities touched, signature changes and the callers they leave behind, callee deltas, promises kept or broken (with witnesses), module reachability deltas (JS/TS), affected tests, and the static reference cone. `--arch` gives the architecture view instead: new or removed data paths, side effects, dependencies, cycles, and what did not change, ranked.
- ```bash
- sem certify main..HEAD # markdown certificate
- sem certify main..HEAD --json # the full certificate
- sem certify main..HEAD --arch # architecture delta
- sem certify main..HEAD --arch --view # at most 10 ranked items
- sem certify main..HEAD --html > view.html # one self-contained page
- ```
- `<base>...<head>` uses the merge base; one ref means `<ref>..HEAD`.
- ### sem diff
- Entity-level diff with rename detection, structural hashing, and word-level inline highlights.
- ```bash
- # Semantic diff of working changes
- sem diff
- # Staged changes only
- sem diff --staged
- # Specific commit
- sem diff --commit abc1234
- # Commit range
- sem diff --from HEAD~5 --to HEAD
- # Verbose mode (word-level inline diffs for each entity)
- sem diff -v
- # Plain text output (git status style)
- sem diff --format plain
- # JSON output (for AI agents, CI pipelines)
- sem diff --format json
- # Markdown output (for PRs, reports)
- sem diff --format markdown
- # Compare any two files (no git repo needed)
- sem diff file1.ts file2.ts
- # Read file changes from stdin (no git repo needed)
- echo '[{"filePath":"src/main.rs","status":"modified","beforeContent":"...","afterContent":"..."}]' \
- | sem diff --stdin --format json
- # Only specific file types
- sem diff --file-exts .py .rs
- ```
- ### sem graph
- How the code is connected. With no flag, the entity dependency graph (the graph `sem impact` and `sem find --context` are built on); the flags select other layers.
- ```bash
- sem graph --json # entities and the calls/references between them
- sem graph --modules metrics # JS/TS module graph: density, cycles, depth, centrality
- sem graph --modules blast-radius pkg-a # who breaks if pkg-a changes
- sem graph --modules path src/a.ts src/b.ts
- sem graph --dataflow # reads, writes and source -> sink paths
- sem graph --dataflow --witness # experimental: witness tasks per static flow
- sem graph --system # exact dependency versions from every lockfile
- sem graph --system build --out sys/ # the layered whole-system graph
- ```
- ### sem history
- How an entity changed through git history, logic changes told apart from cosmetic ones.
- ```bash
- sem history authenticateUser
- sem history authenticateUser -v # with the content diff of each version
- sem history authenticateUser --limit 20
- sem history # repo hotspots and co-change pairs
- sem history --blame src/auth.ts # who last changed each entity in the file
- ```
- With no entity, `sem history` analyzes recent repo history at the entity level: **hotspots** (most-changed functions/classes, with author counts) and **co-change pairs** (entities that repeatedly change in the same commits: "if you touch one, don't forget the other").
- ### sem cloud, sem config
- ```bash
- sem cloud login # API key, or GitHub when omitted
- sem cloud enable # cloud queries for this public repo (shows what is sent, asks first)
- sem cloud disable # stop cloud for this repo
- sem cloud review listen <diff-id> # attach an agent to a cloud code review
- sem cloud xref --json # cross-repo dependencies
- sem cloud repos # where your code is stored
- sem config setup # make `git diff` show sem's entity diff
- sem config telemetry off
- sem config completions zsh # shell completions
- sem config update
- sem config stats # local diff counters; nothing leaves your machine
- ```
- ### Old command names
- Every earlier command name and flag still works, with the same output. At a terminal, sem prints a one-line note on stderr with the new spelling; in `--json` mode or when output is piped, it prints nothing extra.
- | Old | New |
- |---|---|
- | `sem callers X` | `sem find X --callers` |
- | `sem refs X` | `sem find X --refs` |
- | `sem context X` | `sem find X --context` |
- | `sem entities PATH` | `sem find --in PATH` |
- | `sem log X` | `sem history X` |
- | `sem blame FILE` | `sem history --blame FILE` |
- | `sem arch-diff RANGE` | `sem certify RANGE --arch` |
- | `sem topology OP` | `sem graph --modules OP` |
- | `sem dataflow` | `sem graph --dataflow` |
- | `sem system OP` | `sem graph --system OP` |
- | `sem promises verify` | `sem check --promises` |
- | `sem login`, `logout`, `whoami`, `review`, `xref`, `repos` | `sem cloud ...` |
- | `sem cloud never` | `sem cloud disable` |
- | `sem setup`, `unsetup`, `telemetry`, `completions`, `update`, `stats` | `sem config ...` |
- ### Promises
- A promise is something an agent (or a person) claims about the codebase, written down as a check that either passes or fails. "Done" means the check passes. Promises live in `.sem/promises/*.json`; each law may carry a human `"promise"`. Code-shape laws are tree-sitter queries that work in any language sem parses; every capture whose name does not start with `_` counts as a violation, and nested captures are reported once, at the outermost node. For example, `.sem/promises/jsx-no-logic.json`:
- ```json
- { "laws": [
- { "id": "jsx-no-logic/conditionals",
- "promise": "JSX contains no conditional rendering",
- "forbidPattern": { "from": ["**/*.tsx", "**/*.jsx"], "within": ["jsx_expression"],
- "query": "(ternary_expression) @conditional" } },
- { "id": "jsx-no-logic/inline-handler-bodies",
- "promise": "JSX event handlers are references, not block bodies",
- "forbidPattern": { "from": ["**/*.tsx", "**/*.jsx"], "within": ["jsx_attribute"],
- "query": "(arrow_function body: (statement_block (_))) @handler_body" } }
- ] }
- ```
- ```bash
- sem promises check # KEPT / BROKEN n per promise, then file:line:col capture snippet; exit 1 if any broken
- sem promises check --changed src/App.tsx # only violations in these files (what an edit hook runs)
- sem promises check --since main # only files changed since a ref, uncommitted and untracked included
- sem promises status --json # id, kept, violation count per promise
- sem check --promises # prove every promise can fail (each needs a mutation)
- ```
- `sem promises` is not listed in `sem --help`; `sem certify` reports promises kept or broken for a range. The same file also works with `sem graph --modules check --laws <file>`, which adds graph and import laws (`forbid`, `only`, `acyclic`, `layers`, `forbidImport`, `allowImports`) for JS/TS workspaces. With `--changed` or `--since`, those laws still run over the whole graph, but only violations that involve a changed file (or the package containing it) are reported.
- To have an agent see a broken promise right after it makes an edit, run [`scripts/promises-hook.sh`](scripts/promises-hook.sh) after each edit. The script exits 2 and prints the violations to stderr when a promise breaks, and stays silent otherwise. In Claude Code, add it to your settings as a `PostToolUse` hook:
- ```json
- { "hooks": { "PostToolUse": [ { "matcher": "Edit|Write|MultiEdit",
- "hooks": [ { "type": "command", "command": "sh /path/to/sem/scripts/promises-hook.sh" } ] } ] } }
- ```
- The script reads `tool_input.file_path` from the hook's stdin. Other harnesses can pass the edited paths as arguments instead. In pi, for example, an extension can do this from its `tool_result` handler for the edit and write tools: run `sh promises-hook.sh <path>` and append the stderr to the tool result when the script exits 2.
- ## Use as default Git diff
- Replace `git diff` output with entity-level diffs. Agents and humans get sem output automatically without changing any commands.
- ```bash
- sem config setup
- ```
- Now `git diff` shows entity-level changes instead of line-level. No prompts, no agent configuration needed. Everything that calls `git diff` gets sem output automatically. Also installs a pre-commit hook that shows entity-level blast radius of staged changes.
- On macOS and Linux, `sem config setup` also registers a Claude Code `UserPromptSubmit` hook (`sem hook prompt-submit`) for prompt-time context injection, and `sem hook tool` around the Edit, Write and MultiEdit tools. After each edit that hook tells the agent which sibling implementations of a changed method it did not touch, which callers still use a signature it just changed, and which tests reach the change, with a command to run them; it stays silent when there is nothing to say. It edits `~/.claude/settings.json` idempotently, backs it up first, and leaves any hooks you already have untouched.
- To disable and go back to normal git diff (also removes the session hooks):
- ```bash
- sem config unsetup
- ```
- ## Entity-level diffs on every pull request
- Add the GitHub Action and every PR gets one sticky comment showing which
- functions, classes, and methods changed. It updates in place on each push and
- calls out cosmetic-only PRs (formatting/comments) explicitly:
- ```yaml
- # .github/workflows/entity-diff.yml
- name: Entity diff
- on: pull_request
- permissions:
- contents: read
- pull-requests: write
- jobs:
- entity-diff:
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v4
- - uses: Ataraxy-Labs/sem/action@v0.23.1
- ```
- No config, no API keys, never fails your build. See [action/](action/) for details.
- ## Cloud acceleration (for scale and teams)
- Local is always free and always fast: the on-disk index answers day-to-day queries in single-digit milliseconds even from a cold process, so there's nothing to keep warm and no login required. You do not pay to make your laptop fast.
- Cloud is for what a laptop can't do. On a very large monorepo the first local graph build can take a few seconds; a shared team graph shouldn't be rebuilt per developer; and CI wants the graph without checking anything out. `sem cloud login` connects those cases to sem cloud, which keeps a warm, pre-built graph for your registered repos and serves the heavy queries from it instead of rebuilding locally.
- ```bash
- sem cloud login # GitHub device flow, one time
- sem impact myFunc --file src/foo.rs # served from the cloud's warm graph
- ```
- It is fully optional and transparent:
- - Not logged in, or the cloud is unreachable? sem computes locally and prints the exact same output. No failures, no difference in results.
- - `SEM_LOCAL=1` forces local computation even when logged in.
- - Small repos see no change, local is already fast. The win is for large codebases where rebuilding the graph each time is the bottleneck.
- Related commands, all cloud-account scoped:
- ```bash
- sem cloud logout # log out
- sem cloud whoami # show current cloud identity
- sem cloud enable # turn on cloud queries for a public repo (shows what's sent first)
- sem cloud disable # stop cloud for this repo
- sem cloud xref --json # cross-repo dependencies across your indexed repos
- sem cloud repos # where your code is stored: cloud-indexed repos + local caches
- sem cloud status # cloud + telemetry state for this repo (offline; sends nothing)
- sem cloud share # cloud queries for a private repo, with extra confirmation
- sem cloud forget # delete this repo's cloud index and unregister it
- ```
- `status`, `share`, `forget`, `list`, `preview` and `log` are not shown in `sem cloud --help` but work as listed; each one is read-only or requires explicit confirmation before it sends anything.
- If your team runs code review through sem cloud, `sem cloud review listen <diff-id-or-url>` execs a coding agent pre-configured to join that review as a live listener that answers reviewer questions anchored to specific lines of the diff.
- ## What it parses
- 32 programming languages with full entity extraction via tree-sitter:
- | Language | Extensions | Entities |
- |----------|-----------|----------|
- | TypeScript | `.ts` `.tsx` `.mts` `.cts` | functions, classes, interfaces, types, enums, exports |
- | JavaScript | `.js` `.jsx` `.mjs` `.cjs` `.es6` | functions, classes, variables, exports |
- | Python | `.py` `.pyi` | functions, classes, decorated definitions |
- | Starlark (Bazel, Buck2) | `BUILD` `BUCK` `WORKSPACE` `PACKAGE` `.bzl` `.bazel` `.bxl` `.star` | build targets, functions |
- | Go | `.go` | functions, methods, types, vars, consts |
- | Rust | `.rs` | functions, structs, enums, impls, traits, mods, consts |
- | Java | `.java` | classes, methods, interfaces, enums, fields, constructors |
- | C | `.c` `.h` | functions, structs, enums, unions, typedefs |
- | C++ | `.cpp` `.cc` `.cxx` `.hpp` `.hh` `.hxx` | functions, classes, structs, enums, namespaces, templates |
- | C# | `.cs` | classes, methods, interfaces, enums, structs, properties |
- | Ruby | `.rb` | methods, classes, modules |
- | PHP | `.php` `.inc` `.phtml` `.module` | functions, classes, methods, interfaces, traits, enums |
- | Swift | `.swift` | functions, classes, protocols, structs, enums, properties |
- | Elixir | `.ex` `.exs` | modules, functions, macros, guards, protocols |
- | Bash | `.sh` | functions |
- | Fish | `.fish` | functions |
- | jq (opt-in, `--features lang-jq`) | `.jq` | functions (`def`) |
- | Lua | `.lua` | functions (global, local, table, and method forms) |
- | HCL/Terraform | `.hcl` `.tf` `.tfvars` | blocks, attributes (qualified names for nested blocks) |
- | Kotlin | `.kt` `.kts` | classes, interfaces, objects, functions, properties, companion objects |
- | Fortran | `.f90` `.f95` `.f03` `.f08` `.f` `.for` | functions, subroutines, modules, programs |
- | Vue | `.vue` | template/script/style blocks + inner TS/JS entities |
- | XML | `.xml` `.plist` `.svg` `.csproj` + 9 more MSBuild/resource extensions | elements (nested, tag-name identity) |
- | ERB | `.erb` `.html.erb` | blocks, expressions, code tags |
- | Svelte | `.svelte` `.svelte.js` `.svelte.ts` (+ `.test`/`.spec` variants) | component blocks + rune JS/TS modules |
- | Perl | `.pl` `.pm` `.t` | subroutines, packages |
- | Dart | `.dart` | classes, mixins, extensions, enums, type aliases, functions |
- | OCaml | `.ml` `.mli` | values, modules, types, classes, externals |
- | Scala | `.scala` `.sc` `.sbt` `.kojo` `.mill` | classes, objects, traits, enums, functions, vals, extensions |
- | Nix | `.nix` | bindings, inherit declarations |
- | Haskell | `.hs` | functions, signatures, data types, newtypes, classes, instances, type synonyms |
- | Elm | `.elm` | value declarations, type aliases, type declarations, port annotations, infix declarations |
- | Clojure | `.clj` `.cljs` `.cljc` | vars, functions, macros, multimethods, protocols, records, types |
- | D | `.d` `.di` | modules, functions, classes, structs, interfaces, unions, enums, templates, aliases, unittests |
- | Zig | `.zig` | functions, tests, variables |
- | SQL | `.sql` `.psql` `.pgsql` `.ddl` | tables, views, functions, indexes, types, schemas, triggers, sequences |
- Plus structured data formats:
- | Format | Extensions | Entities |
- |--------|-----------|----------|
- | JSON | `.json` | properties, objects (RFC 6901 paths) |
- | YAML | `.yml` `.yaml` | sections, properties (dot paths) |
- | TOML | `.toml` | sections, properties |
- | EDN | `.edn` | top-level map entries (keyword keys) |
- | CSV | `.csv` `.tsv` | rows (first column as identity) |
- | Markdown | `.md` `.mdx` `.qmd` | heading-based sections |
- | LaTeX | `.tex` `.latex` `.cls` `.sty` | sections (part/chapter/section/…), plus theorem/lemma/proof/figure/table/algorithm and other tracked environments |
- Everything else falls back to chunk-based diffing.
- ### Custom extensions and extensionless files
- For files with non-standard extensions, create a `.semrc` in your project root:
- ```
- .xyz = cpp
- .j = json
- .mypy = python
- ```
- sem also reads `.gitattributes` patterns (`diff=` and `linguist-language=`) if you already have those set up. `.semrc` takes priority when both define the same extension.
- For files with no extension at all, sem detects the language automatically from content (shebang lines, vim modelines, and structural heuristics like `package`/`import`/`use` statements). This covers 30+ languages with no config needed.
- ## How matching works
- Three-phase entity matching:
- 1. **Exact ID match**: same entity in before/after = modified or unchanged
- 2. **Structural hash match**: same AST structure, different name = renamed or moved (ignores whitespace/comments)
- 3. **Fuzzy similarity**: >80% token overlap = probable rename
- This means sem detects renames and moves, not just additions and deletions. Structural hashing also distinguishes cosmetic changes (whitespace, formatting) from real logic changes.
- ## Use with AI agents (MCP)
- Need a URL-based client? The [Streamable HTTP preview](docs/mcp-http.md) adds authenticated, local read-only access; stdio remains the default.
- On macOS and Linux, clients in the same checkout share a warm repository daemon.
- Run `sem mcp --status` to check it. See the [shared runtime contract and reproducible benchmark](docs/shared-mcp.md)
- for session isolation, fallback behavior, and current platform limits.
- `sem mcp` starts a [Model Context Protocol](https://modelcontextprotocol.io) server over stdin/stdout. It's not a command you run and read yourself: it's a server your coding agent launches in the background so it can ask sem questions while it works. That's the reason `mcp` lives alongside the normal commands. The agent gets the core verbs as tools:
- | Tool | Answers |
- |---|---|
- | `sem_find` | Where is it defined? `mode`: `callers`, `refs` or `context`; `in` lists a file or directory |
- | `sem_grep` | Where does this text appear? |
- | `sem_impact` | What does changing this entity touch? Dependents, deps, tests |
- | `sem_check` | Is my change correct? A verdict: pass, fail, could not decide |
- | `sem_certify` | What should a human review in this commit range? |
- | `sem_diff` | Which entities changed between two refs? |
- | `sem_graph` | The entity, module, data-flow or system graph |
- | `sem_history` | How did an entity change? `blame` for a file |
- Clients that call the earlier tool names (`sem_entities`, `sem_context`, `sem_callers`, `sem_log`, `sem_blame`) still get answers; those names are just not listed. A review-listener session (`sem mcp --review`, which `sem cloud review listen` sets up) also lists `join_review`, `wait_for_branch`, `reply_to_branch` and `list_open_branches`.
- Why an agent wants these: instead of reading whole files and burning tokens, it can ask "what breaks if I change `submitOrder`" (`sem_impact`) or "give me just the context to refactor this function" (`sem_find` with mode `context`, which returns the function's source plus its callers and callees) and get a precise, deterministic answer from the dependency graph instead of a grep result that might miss a caller.
- Add it once, then talk to your agent normally. It calls the tools on its own.
- **Claude Code:**
- ```bash
- claude mcp add sem -- sem mcp
- ```
- Or one command that also installs the skill, so the agent knows *when* to reach for sem:
- ```bash
- npx @ataraxy-labs/sem-skill
- ```
- **Cursor, Claude Desktop, or any client with an `mcpServers` config:**
- ```json
- {
- "mcpServers": {
- "sem": {
- "command": "sem",
- "args": ["mcp"]
- }
- }
- }
- ```
- If `sem` isn't on the agent's PATH, use the absolute path to the binary. No separate install is needed: `sem mcp` ships in the same binary as every other command.
- ## JSON output
- ```bash
- sem diff --format json
- ```
- Real output, from a one-line logic change to a Python function:
- ```json
- {
- "summary": {
- "fileCount": 1,
- "added": 0,
- "modified": 1,
- "deleted": 0,
- "moved": 0,
- "renamed": 0,
- "reordered": 0,
- "binary": 0,
- "orphan": 0,
- "total": 1
- },
- "changes": [
- {
- "entityId": "auth.py::function::authenticate_user",
- "changeType": "modified",
- "entityType": "function",
- "entityName": "authenticate_user",
- "startLine": 1,
- "endLine": 6,
- "oldStartLine": 1,
- "oldEndLine": 4,
- "oldEntityName": null,
- "filePath": "auth.py",
- "oldFilePath": null,
- "oldParentId": null,
- "beforeContent": "def authenticate_user(username, password):\n if not username or not password:\n return False\n return check_credentials(username, password)",
- "afterContent": "def authenticate_user(username, password):\n if not username or not password:\n return False\n if not check_credentials(username, password):\n return False\n return True",
- "commitSha": null,
- "author": null,
- "structuralChange": true
- }
- ],
- "binaryChanges": []
- }
- ```
- The named change-type buckets (`added`, `modified`, `deleted`, `moved`, `renamed`, `reordered`) always sum to `total`. `orphan` is a cross-cutting metadata count for module-level changes, and those changes are already included in the named change-type buckets. `beforeContent`/`afterContent` carry the entity's full source on either side of the change; `structuralChange` is `false` when the diff is cosmetic only (whitespace, comments).
- ## As a library
- sem-core can be used as a Rust library dependency, from [crates.io](https://crates.io/crates/sem-core):
- ```toml
- [dependencies]
- sem-core = "0.23"
- ```
- Used by [weave](https://github.com/Ataraxy-Labs/weave) (semantic merge driver) and [inspect](https://github.com/Ataraxy-Labs/inspect) (entity-level code review).
- ## Architecture
- - **tree-sitter** for code parsing (native Rust, not WASM)
- - **git2** for Git operations
- - **rayon** for parallel file processing
- - **xxhash** for structural hashing
- - A per-repo cache directory (SQLite entity cache + an mmap-able query index) backs `find`/`callers`/`refs`/`grep` with cold-process lookups and no background daemon
- - Plugin system for adding new languages and formats (see [CONTRIBUTING.md](CONTRIBUTING.md))
- ## Telemetry
- Local by default: sem counts command names (e.g. `diff`, `impact`) on your own machine only, and in that mode nothing is ever uploaded. No code, file paths, repo names, or user identity is recorded, and no network call is made.
- ```bash
- sem config telemetry preview # see current mode and exactly what would be sent
- sem config telemetry on # opt in: also upload counts to help improve sem
- sem config telemetry off # record nothing at all
- ```
- `SEM_NO_TELEMETRY=1` or `DO_NOT_TRACK=1` force the record-nothing behavior regardless of mode. Development builds (anything run out of a `cargo build` `target/` directory) never record, so working on sem itself doesn't pollute the numbers.
- ## Contributing
- Want to add a new language? See [CONTRIBUTING.md](CONTRIBUTING.md) for a step-by-step guide.
- ## Star History
- [](https://star-history.com/#Ataraxy-Labs/sem&Date)
- ## License
- MIT OR Apache-2.0
install.sh · Bundled script · 304 lines · 2 findings marked
- #!/bin/sh
- # sem installer — https://github.com/Ataraxy-Labs/sem
- #
- # Usage:
- # curl -fsSL https://raw.githubusercontent.com/Ataraxy-Labs/sem/main/install.sh | sh
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 5 on GitHub
- #
- # # pin a version
- # curl -fsSL https://raw.githubusercontent.com/Ataraxy-Labs/sem/main/install.sh | SEM_VERSION=v0.23.1 sh
- #
- # # or with flags (download first if you want flags)
- # curl -fsSL -o install.sh https://raw.githubusercontent.com/Ataraxy-Labs/sem/main/install.sh
- # sh install.sh --version v0.23.1 --install-dir /opt/sem/bin
- #
- # Env vars (all optional):
- # SEM_VERSION version to install, e.g. "v0.23.1" or "0.23.1" (default: latest)
- # SEM_INSTALL_DIR install directory (default: $HOME/.local/bin)
- # NO_COLOR disable colored output (any non-empty value)
- #
- # Supported platforms: macOS (arm64, x86_64), Linux (x86_64, arm64).
- # Windows: see https://github.com/Ataraxy-Labs/sem#installation (Scoop) or
- # download sem-windows-x86_64.zip from the releases page directly.
- #
- # This script never uses sudo and never writes outside INSTALL_DIR. It
- # verifies every downloaded archive against the release's checksums.txt and
- # refuses to install (fail closed) if that file can't be fetched or doesn't
- # cover the archive.
- set -eu
- REPO="Ataraxy-Labs/sem"
- BINARY="sem"
- VERSION="${SEM_VERSION:-latest}"
- INSTALL_DIR="${SEM_INSTALL_DIR:-$HOME/.local/bin}"
- FORCE=0
- SKIP_CHECKSUM=0
- # --- output -----------------------------------------------------------
- # Plain-output degradation: no tty, NO_COLOR set, or CI set -> no ANSI.
- if [ -n "${NO_COLOR:-}" ] || [ -n "${CI:-}" ] || [ ! -t 1 ]; then
- USE_COLOR=0
- else
- USE_COLOR=1
- fi
- info() {
- if [ "$USE_COLOR" -eq 1 ]; then
- printf ' \033[1;32m%s\033[0m %s\n' "$1" "$2"
- else
- printf ' %s %s\n' "$1" "$2"
- fi
- }
- warn() {
- if [ "$USE_COLOR" -eq 1 ]; then
- printf ' \033[1;33mwarning:\033[0m %s\n' "$1"
- else
- printf ' warning: %s\n' "$1"
- fi
- }
- error() {
- if [ "$USE_COLOR" -eq 1 ]; then
- printf ' \033[1;31merror:\033[0m %s\n' "$1" 1>&2
- else
- printf ' error: %s\n' "$1" 1>&2
- fi
- exit 1
- }
- # --- flags --------------------------------------------------------------
- usage() {
- cat <<EOF
- sem installer
- Usage: install.sh [options]
- Options:
- -v, --version VERSION install a specific version, e.g. v0.23.1 (default: latest)
- -d, --install-dir DIR install directory (default: \$HOME/.local/bin)
- -f, --force reinstall even if the target version is already present
- --skip-checksum install without SHA256 verification (NOT recommended)
- --no-color disable colored output
- -h, --help show this help
- Environment variables SEM_VERSION and SEM_INSTALL_DIR set the same defaults
- as --version and --install-dir.
- EOF
- }
- while [ $# -gt 0 ]; do
- case "$1" in
- -v|--version)
- [ $# -ge 2 ] || error "--version requires an argument"
- VERSION="$2"; shift 2 ;;
- -d|--install-dir)
- [ $# -ge 2 ] || error "--install-dir requires an argument"
- INSTALL_DIR="$2"; shift 2 ;;
- -f|--force)
- FORCE=1; shift ;;
- --skip-checksum)
- SKIP_CHECKSUM=1; shift ;;
- --no-color)
- USE_COLOR=0; shift ;;
- -h|--help)
- usage; exit 0 ;;
- *)
- error "unknown option: $1 (see --help)" ;;
- esac
- done
- require_cmd() {
- command -v "$1" >/dev/null 2>&1 || error "required command not found: $1"
- }
- # --- platform detection ---------------------------------------------------
- detect_platform() {
- OS=$(uname -s | tr '[:upper:]' '[:lower:]')
- ARCH=$(uname -m)
- case "$OS" in
- linux) OS_NAME="linux" ;;
- darwin) OS_NAME="darwin" ;;
- *)
- error "Unsupported OS: $OS. sem's install.sh supports macOS and Linux. \
- For Windows, see https://github.com/${REPO}#installation (Scoop) or download \
- sem-windows-x86_64.zip from https://github.com/${REPO}/releases." ;;
- esac
- case "$ARCH" in
- x86_64|amd64) ARCH_NAME="x86_64" ;;
- aarch64|arm64) ARCH_NAME="arm64" ;;
- *) error "Unsupported architecture: $ARCH (supported: x86_64, arm64/aarch64)" ;;
- esac
- ARTIFACT="sem-${OS_NAME}-${ARCH_NAME}"
- }
- # --- version resolution ---------------------------------------------------
- resolve_version() {
- if [ "$VERSION" = "latest" ]; then
- info "Resolving" "latest release"
- VERSION=$(curl -fsSL "https://api.github.com/repos/${REPO}/releases/latest" \
- | grep '"tag_name"' | head -1 | sed 's/.*"tag_name": *"//;s/".*//')
- [ -n "$VERSION" ] || error "Could not determine the latest version from the GitHub API"
- else
- # Accept "0.23.1" or "v0.23.1"; release tags are "vX.Y.Z".
- case "$VERSION" in
- v*) ;;
- *) VERSION="v${VERSION}" ;;
- esac
- fi
- }
- # --- idempotency: skip if this exact version is already installed --------
- already_installed() {
- [ "$FORCE" -eq 0 ] || return 1
- [ -x "${INSTALL_DIR}/${BINARY}" ] || return 1
- current=$("${INSTALL_DIR}/${BINARY}" --version 2>/dev/null || true)
- # `sem --version` output is expected to contain the version number.
- case "$current" in
- *"${VERSION#v}"*) return 0 ;;
- *) return 1 ;;
- esac
- }
- # --- checksum verification (fail closed) ----------------------------------
- sha256_of() {
- if command -v sha256sum >/dev/null 2>&1; then
- sha256sum "$1" | awk '{print $1}'
- elif command -v shasum >/dev/null 2>&1; then
- shasum -a 256 "$1" | awk '{print $1}'
- else
- error "no SHA256 tool found (need sha256sum or shasum) — cannot verify the download. Refusing to install."
- fi
- }
- verify_checksum() {
- archive="$1"
- if [ "$SKIP_CHECKSUM" -eq 1 ]; then
- warn "checksum verification skipped (--skip-checksum) — installing an UNVERIFIED binary"
- return 0
- fi
- sums_url="https://github.com/${REPO}/releases/download/${VERSION}/checksums.txt"
- sums=$(curl -fsSL "$sums_url" 2>/dev/null) || {
- error "could not fetch checksums.txt for ${VERSION} (${sums_url}). \
- Refusing to install an unverified binary. Re-run with --skip-checksum to \
- override at your own risk, or install a different --version."
- }
- expected=$(printf '%s\n' "$sums" | grep -F "${ARTIFACT}.tar.gz" | awk '{print $1}' | head -1)
- if [ -z "$expected" ]; then
- error "checksums.txt for ${VERSION} has no entry for ${ARTIFACT}.tar.gz. \
- Refusing to install an unverified binary. Re-run with --skip-checksum to \
- override at your own risk."
- fi
- actual=$(sha256_of "$archive")
- if [ "$actual" != "$expected" ]; then
- error "checksum mismatch for ${ARTIFACT}.tar.gz
- expected: ${expected}
- actual: ${actual}
- This could mean a corrupted download or a tampered release. Not installing."
- fi
- info "Verified" "SHA256 checksum"
- }
- # --- download + install ----------------------------------------------------
- download_and_install() {
- URL="https://github.com/${REPO}/releases/download/${VERSION}/${ARTIFACT}.tar.gz"
- TMPDIR=$(mktemp -d "${TMPDIR:-/tmp}/sem-install.XXXXXX")
- trap 'rm -rf "$TMPDIR"' EXIT INT TERM
- info "Downloading" "${ARTIFACT} ${VERSION}"
- curl -fsSL "$URL" -o "${TMPDIR}/${ARTIFACT}.tar.gz" \
- || error "download failed: ${URL}
- Check https://github.com/${REPO}/releases for available builds and versions."
- verify_checksum "${TMPDIR}/${ARTIFACT}.tar.gz"
- tar xzf "${TMPDIR}/${ARTIFACT}.tar.gz" -C "$TMPDIR"
- [ -f "${TMPDIR}/${BINARY}" ] || error "binary '${BINARY}' not found inside ${ARTIFACT}.tar.gz"
- mkdir -p "$INSTALL_DIR" || error "could not create install directory: ${INSTALL_DIR}"
- [ -w "$INSTALL_DIR" ] || error "install directory is not writable: ${INSTALL_DIR} \
- (this installer never uses sudo — pass --install-dir to pick a writable location)"
- # Atomic-ish install: write alongside, then rename into place.
- mv "${TMPDIR}/${BINARY}" "${INSTALL_DIR}/${BINARY}.new"
- chmod +x "${INSTALL_DIR}/${BINARY}.new"
- mv -f "${INSTALL_DIR}/${BINARY}.new" "${INSTALL_DIR}/${BINARY}"
- }
- # --- PATH guidance ----------------------------------------------------------
- path_guidance() {
- case ":$PATH:" in
- *":${INSTALL_DIR}:"*) return 0 ;;
- esac
- warn "${INSTALL_DIR} is not on your PATH"
- printf '\n'
- printf ' Add it in your shell rc file, then restart your shell:\n\n'
- printf ' bash/zsh: echo '\''export PATH="%s:$PATH"'\'' >> ~/.bashrc # or ~/.zshrc\n' "$INSTALL_DIR"
- printf ' fish: fish_add_path %s\n' "$INSTALL_DIR"
- printf '\n'
- }
- # --- main -------------------------------------------------------------------
- main() {
- require_cmd curl
- require_cmd tar
- require_cmd mktemp
- if [ "$USE_COLOR" -eq 1 ]; then
- printf '\n \033[1msem\033[0m installer\n\n'
- else
- printf '\n sem installer\n\n'
- fi
- detect_platform
- resolve_version
- if already_installed; then
- info "Already installed" "${VERSION} -> ${INSTALL_DIR}/${BINARY} (use --force to reinstall)"
- path_guidance
- exit 0
- fi
- download_and_install
- if [ -x "${INSTALL_DIR}/${BINARY}" ]; then
- if ! installed_version=$("${INSTALL_DIR}/${BINARY}" --version 2>&1); then
- # the binary is there but won't start (for example a glibc too old for it): say so, don't report success
- error "sem was installed to ${INSTALL_DIR}/${BINARY} but does not start on this system:
- ${installed_version}
- Please report it at https://github.com/${REPO}/issues with your OS version (cat /etc/os-release)."
- fi
- info "Installed" "${installed_version} -> ${INSTALL_DIR}/${BINARY}"
- else
- error "install appears to have failed: ${INSTALL_DIR}/${BINARY} is not executable"
- fi
- path_guidance
- if [ "$USE_COLOR" -eq 1 ]; then
- printf ' Run \033[1msem config setup\033[0m to replace git diff globally.\n'
- printf ' Run \033[1msem cloud login\033[0m to connect to sem cloud.\n'
- else
- printf ' Run "sem config setup" to replace git diff globally.\n'
- printf ' Run "sem cloud login" to connect to sem cloud.\n'
- fi
- printf '\n To uninstall: curl -fsSL https://raw.githubusercontent.com/%s/main/uninstall.sh | sh\n\n' "$REPO"
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 301 on GitHub
- }
- main
pi/src/bridge/bash-audit.ts · Bundled script · 415 lines · 1 finding marked
- import { isCodeFilePath, auditWriteCommand } from "./write-audit.ts";
- /**
- * Pure classifier for pi-sem's bash audit wrapper (no process spawning).
- *
- * Deliberately conservative: rules only ever match a statement's LEADING
- * command word(s) (after stripping an env-var prefix and normalizing an
- * absolute-path binary to its basename), a single level of $(...)/backtick
- * substitution, or (for "write") a redirect/verb pattern anywhere in the
- * statement, never scan mid-command generally beyond that, so a false
- * positive costs at most one audit-log entry by default (and one refusal
- * when PI_SEM_STRICT=1).
- *
- * NOT caught (deliberately, to keep this a "few regexes" classifier rather
- * than a shell parser -- a real adversary can defeat every rule below):
- * - nested command substitution, e.g. `$(echo $(cat f))` -- only one level
- * of $(...)/`...` is inspected
- * - a variable holding a command name, e.g. `CMD=cat; $CMD f.txt`
- * - `xargs cat`, `find . -exec cat {} \;`, or any command that itself
- * invokes a read verb as a sub-process argument rather than via -c/-e
- * - a read/write verb inside a heredoc BODY's own text content (the body
- * itself is stripped before classification -- see stripHeredocsAndHereStrings
- * -- but a verb appearing only as literal text inside that body, e.g. the
- * word "cat" mentioned in a comment being written, was never a command
- * anyway and correctly isn't classified)
- * - `python3 script.py` / `node script.js` running arbitrary file reads or
- * writes internally -- only the inline `-c`/`-e` forms are classified
- * - a `python -c '...'`/`node -e '...'` script's own file-write calls
- * (e.g. `open(path, "w")`) are not parsed -- the whole invocation is
- * already flagged "indirect" regardless of what the embedded script
- * does, which is as far as this classifier goes without a real
- * interpreter for that language
- * - shell aliases, functions, or `env` used as an indirection wrapper
- * (`env cat f.txt` is not normalized the way `/bin/cat` is)
- * - a `sed` in-place flag written via a variable (e.g. `FLAG=-i; sed $FLAG
- * ...`) -- both spellings recognized literally in the statement text
- * (`-i`, `--in-place`) are caught, but an indirect flag is not
- * - a flag that consumes a separate-token VALUE ahead of the trigger flag,
- * e.g. `node -r foo -e "..."` -- the leading-flag scan stops at the
- * first token that doesn't itself start with `-` (here, `foo`), so a
- * trigger flag appearing after such a value-bearing flag is not seen.
- * A flag that is merely ANOTHER leading `-`/`--` token (with no
- * separate-token value of its own), e.g. `bash --posix -c "..."`, does
- * NOT break the scan and so is correctly still caught.
- * - a backslash-escaped command (`\cat f.txt`, defeats shell aliasing),
- * the `command`/`builtin` prefix (`command cat f.txt`), bare `&`
- * backgrounding as a statement separator (`sleep 1 & cat f.txt`),
- * subshell parens (`( cat f.txt )`), or piping into a second-stage
- * shell (`echo 'cat f.txt' | sh`) -- none of these are normalized or
- * split on; each defeats every rule below (a later review's findings,
- * disclosed, not yet fixed)
- * - `base64 -d | sh`-style decode-then-execute chains -- no verb rule for
- Note: Quoted as data in code: Mentions decoding base64 into a shell (no payload shown)
- Line 52 on GitHub
- * decode utilities, and only one level of substitution is inspected
- * (disclosed, not yet fixed)
- * - a write's TARGET PATH extraction (below) is best-effort: quoted paths
- * with embedded spaces, `2>&1`-style fd duplication (deliberately
- * excluded, see REDIRECT_PATTERN), and multi-redirect statements only
- * ever resolve to the LAST redirect target found
- */
- export interface BashAuditMatch {
- ruleId: "read" | "search" | "find" | "list" | "indirect" | "write";
- statement: string;
- suggestedTool: string;
- /** Only set for ruleId "write" when a target path could be extracted (best-effort; see header). */
- path?: string;
- /** Only set alongside `path` -- whether that path's extension looks like a code file (see write-audit.ts's isCodeFilePath). */
- isCodeFile?: boolean;
- }
- /** Statements are split on these operators, then trimmed. `||` must come before `|`. */
- const STATEMENT_SPLIT = /\|\||&&|;|\||\n/;
- /** Leading `KEY=value ` assignments (`FOO=1 BAR=2 cmd`), stripped before matching. */
- const ENV_PREFIX = /^(?:[A-Za-z_][A-Za-z0-9_]*=(?:'[^']*'|"[^"]*"|\S*)\s+)+/;
- /** A single level of `$(...)` or `` `...` `` substitution content. */
- const SUBSTITUTION = /\$\(([^()]*)\)|`([^`]*)`/g;
- /**
- * A `>`/`>>` redirect NOT preceded by `-` (excludes `->`), NOT followed by
- * `=` (excludes `>=`), and NOT followed by `&` (excludes `2>&1`/`1>&2` fd
- * duplication, which never names a real file target). `&> file`/`&>> file`
- * (redirect both streams) IS matched -- the `&` sits before the `>`, which
- * this pattern doesn't look at, so the `>` itself still matches normally.
- */
- const REDIRECT_PATTERN = /(?<!-)>{1,2}(?!=)(?!&)\s*(['"]?)([^\s'"]+)\1/g;
- /**
- * Strips heredoc bodies and here-string tokens from the WHOLE command text,
- * before statement splitting -- both constructs can contain characters
- * STATEMENT_SPLIT treats as separators (`;`, `&&`, `|`, newlines), so this
- * must run first or a heredoc body gets shredded into bogus "statements".
- *
- * Here-strings (`<<< word` / `<<< 'quoted'` / `<<< "quoted"`) are a single
- * token, removed outright. Heredocs (`<<WORD`, `<<-WORD`, `<<~WORD`, with an
- * optionally quoted delimiter) keep whatever follows the marker on the SAME
- * line (e.g. `> out.txt` in `cat <<EOF > out.txt`) and strip everything from
- * the next line through the terminator line (a line consisting of just the
- * delimiter, optionally indented for `<<-`).
- *
- * This closes a real gap: the previous approach (skip
- * the WHOLE statement if it merely contained the substring "<<") meant
- * appending `<<< x` to ANY statement -- `head -100 file <<< x`, `cat foo.ts
- * <<< x` -- evaded classification entirely, a universal evasion suffix
- * disguised as a heredoc-write guard.
- */
- function stripHeredocsAndHereStrings(command: string): string {
- let result = command.replace(/<<<\s*(?:'[^']*'|"[^"]*"|\S+)/g, "");
- result = result.replace(/<<[-~]?(['"]?)(\w+)\1([^\n]*)\n[\s\S]*?^[ \t]*\2[ \t]*$\n?/gm, "$3");
- // Cosmetic only: stripping a token can leave a double space behind
- // (`cat <<EOF > out.txt` -> `cat > out.txt`) -- collapse horizontal
- // whitespace runs, never touching newlines (STATEMENT_SPLIT still needs
- // those as statement separators).
- return result.replace(/[ \t]{2,}/g, " ");
- }
- function stripEnvPrefix(statement: string): string {
- return statement.replace(ENV_PREFIX, "");
- }
- /** `/bin/cat foo.txt` -> `cat foo.txt`; `env`-based indirection is NOT unwrapped (see header). */
- function normalizeLeadingBinary(statement: string): string {
- return statement.replace(/^\/(?:[^\s/]+\/)*([^\s/]+)/, "$1");
- }
- /** Strip env-prefix then normalize an absolute-path binary, for matching only (reported statement stays raw). */
- function normalizeForMatch(statement: string): string {
- return normalizeLeadingBinary(stripEnvPrefix(statement));
- }
- /**
- * Recognizes sed's in-place flag in EITHER spelling: the short `-i`
- * (optionally with an attached suffix, `-i.bak`) or the GNU long form
- * `--in-place` (optionally `--in-place=.bak`). Previously only `-i` was
- * recognized, which meant `sed --in-place ...` fell through to the plain-
- * sed READ rule -- a MISCLASSIFICATION (reported as a read) rather than
- * merely uncaught, since neither the write rule's guard nor the read
- * rule's negated guard recognized the long form.
- */
- function hasSedInPlaceFlag(statement: string): boolean {
- return /(^|\s)-i\b/.test(statement) || /(^|\s)--in-place(\b|=)/.test(statement);
- }
- /**
- * True when `rest` (the tokens after the interpreter name) carries the
- * trigger letter (`c` for sh/bash/zsh/dash/python, `e` for node) in a
- * leading run of flag tokens -- either combined into one cluster
- * (`-lc`) or split across separate tokens (`-l -c`). Scans tokens from the
- * start and stops at the first non-flag token (the interpreter's actual
- * argument, e.g. the `-c` STRING itself, must not be scanned for stray
- * letters).
- */
- function hasIndirectTriggerFlag(rest: string, triggerLetter: string): boolean {
- const tokens = rest.trim().split(/\s+/);
- for (const token of tokens) {
- if (!token.startsWith("-")) break;
- if (token.slice(1).includes(triggerLetter)) return true;
- }
- return false;
- }
- /**
- * `sh|bash|zsh|dash|python(3)|node` followed by a leading run of flag
- * tokens carrying the trigger letter -- combined cluster (`bash -lc "..."`)
- * or split across separate tokens (`bash -l -c "..."`, `sh -e -c "..."`).
- * The split form was previously a documented gap:
- * `bash -lc` was caught but `bash -l -c` was not, even though both invoke
- * an opaque interpreted string identically.
- */
- function classifyIndirectShell(normalized: string): boolean {
- const match = /^(sh|bash|zsh|dash|python3?|node)\s*(.*)$/.exec(normalized);
- if (!match) return false;
- const interpreter = match[1] ?? "";
- const rest = match[2] ?? "";
- const trigger = interpreter === "node" ? "e" : "c";
- return hasIndirectTriggerFlag(rest, trigger);
- }
- /**
- * Best-effort write TARGET extraction for a statement already matched by
- * one of the "write" rules below. Not attempted for anything else -- this
- * is deliberately narrow, not a general shell-argument parser.
- */
- function extractWriteTargetPath(statement: string): string | undefined {
- REDIRECT_PATTERN.lastIndex = 0;
- const redirectMatches = [...statement.matchAll(REDIRECT_PATTERN)];
- if (redirectMatches.length > 0) {
- // Last redirect in the statement wins -- the common single-redirect
- // case has exactly one match; a multi-redirect statement's earlier
- // targets are a documented gap (see module header).
- return redirectMatches[redirectMatches.length - 1]?.[2];
- }
- const normalized = normalizeForMatch(statement);
- if (/^sed\b/.test(normalized) && hasSedInPlaceFlag(statement)) {
- const tokens = statement.trim().split(/\s+/);
- return tokens[tokens.length - 1];
- }
- if (/^tee\b/.test(normalized)) {
- const tokens = statement.trim().split(/\s+/).slice(1).filter((t) => !t.startsWith("-"));
- return tokens[0];
- }
- if (/^(cp|mv|truncate)\b/.test(normalized)) {
- // .slice(1) drops the leading command word itself -- without it, a
- // flags-only invocation with no real path argument (`truncate --help`)
- // would fall through to returning "truncate" itself as the "path"
- // once every flag token got filtered out.
- const tokens = statement.trim().split(/\s+/).slice(1).filter((t) => !t.startsWith("-"));
- return tokens[tokens.length - 1];
- }
- return undefined;
- }
- interface ClassificationRule {
- ruleId: BashAuditMatch["ruleId"];
- pattern: RegExp;
- suggestedTool: string;
- /** Extra predicate; when it returns false the rule does not apply. Receives the RAW (unnormalized) statement. */
- guard?: (statement: string) => boolean;
- }
- // Order matters: first match wins, one match max per statement. WRITE rules
- // come first so a statement that's both read-shaped and write-shaped
- // (`cat <<EOF > out.txt`, which after heredoc-stripping becomes
- // `cat > out.txt`) is classified by its write side -- the security-relevant
- // half -- not misreported as a plain read.
- const RULES: ClassificationRule[] = [
- {
- ruleId: "write",
- // `cmd > file` / `cmd >> file`, any leading command --
- // bash-side writes (redirection, sed -i, tee, cp, mv,
- // truncate) were never classified at all, so PI_SEM_STRICT's write
- // protection was fully bypassable through bash.
- pattern: REDIRECT_PATTERN,
- suggestedTool: "weave_edit for an existing entity, or the write tool for a new file, instead of a bash redirect",
- },
- {
- ruleId: "write",
- // The positive counterpart of the read rule's "sed without -i" guard
- // below -- WITH -i (short OR --in-place long form), sed genuinely
- // edits the file in place.
- guard: hasSedInPlaceFlag,
- pattern: /^sed\b/,
- suggestedTool: "weave_edit for an existing entity instead of bash sed -i/--in-place",
- },
- {
- ruleId: "write",
- pattern: /^(tee|cp|mv|truncate)\b/,
- suggestedTool: "weave_edit for an existing entity, or the write tool for a new file, instead of bash tee/cp/mv/truncate",
- },
- {
- ruleId: "read",
- // No heredoc/redirect guard needed here anymore: heredoc bodies are
- // stripped from the WHOLE command before statement splitting (see
- // stripHeredocsAndHereStrings), and a real `>`/`>>` redirect is caught
- // by the "write" rules above, which run first (first-match-wins) --
- // `cat file > out.txt` is classified "write", never reaching this rule.
- pattern: /^(cat|head|tail|less|more)\b/,
- suggestedTool: "sem_read (or sem_context) instead of bash cat/head/tail/less",
- },
- {
- ruleId: "read",
- // Plain `sed` (an edit) must NOT match; only sed WITHOUT an in-place
- // flag (short -i OR long --in-place) is a read (covers `sed -n ...` and
- // any other read-only use).
- guard: (statement) => !hasSedInPlaceFlag(statement),
- pattern: /^sed\b/,
- suggestedTool: "sem_read (or sem_context) instead of bash sed (no -i)",
- },
- {
- ruleId: "read",
- pattern: /^awk\b/,
- suggestedTool: "sem_read (or sem_context) instead of bash awk",
- },
- {
- ruleId: "search",
- pattern: /^(grep|rg|ag)\b/,
- suggestedTool: "sem_outline's text= (or sem_grep once available) instead of bash grep/rg/ag",
- },
- {
- ruleId: "find",
- pattern: /^(find|fd)\b/,
- suggestedTool: "sem_find (once available) instead of bash find/fd",
- },
- {
- ruleId: "list",
- pattern: /^(ls|tree)\b/,
- suggestedTool: "sem_outline (or sem_find once available) instead of bash ls/tree",
- },
- {
- ruleId: "indirect",
- // `sh -c "..."` / `bash -c "..."` / `python(3) -c "..."` / `node -e "..."`
- // run an opaque interpreted string -- classified as bypass-shaped
- // regardless of what the string contains (auditing its contents would
- // require actually parsing that language). The trigger letter (`c` for
- // sh/bash/zsh/dash/python, `e` for node) is matched whether it appears
- // in a combined CLUSTER (`bash -lc "..."`, `sh -ic "..."`, `bash -xc
- // "..."`) or split across separate leading flag tokens (`bash -l -c
- // "..."`, `sh -e -c "..."` -- previously a documented gap), so
- // `bash -lc` and `bash -l -c` are treated identically.
- // A false positive here only ever costs one audit-log entry, so
- // matching any leading flag run CONTAINING the trigger letter is the
- // conservative direction, even though a cluster like `-norc` (not a
- // real bash short-flag combination, but syntactically matches) would
- // also trip it.
- pattern: /^(sh|bash|zsh|dash|python3?|node)\b/,
- guard: (statement) => classifyIndirectShell(normalizeForMatch(statement)),
- suggestedTool: "an inline interpreter -c/-e invocation can run arbitrary reads/writes opaquely -- prefer sem_read/sem_outline/weave_edit directly",
- },
- ];
- function attachWriteFields(match: BashAuditMatch, statement: string): BashAuditMatch {
- if (match.ruleId !== "write") return match;
- const path = extractWriteTargetPath(statement);
- if (path === undefined) return match;
- return { ...match, path, isCodeFile: isCodeFilePath(path) };
- }
- function classifyLeadingCommand(rawStatement: string): BashAuditMatch | undefined {
- const normalized = normalizeForMatch(rawStatement);
- const rule = RULES.find((r) => r.pattern.test(normalized) && (r.guard?.(rawStatement) ?? true));
- if (!rule) return undefined;
- return attachWriteFields({ ruleId: rule.ruleId, statement: rawStatement, suggestedTool: rule.suggestedTool }, rawStatement);
- }
- /** Only a rule's own bare leading-command patterns are tried here -- not the "indirect" -c/-e rule itself (a substitution nested inside another -c/-e is not chased). */
- const SUBSTITUTION_INNER_RULES = RULES.filter((r) => r.ruleId !== "indirect");
- function classifySubstitutions(rawStatement: string): BashAuditMatch | undefined {
- SUBSTITUTION.lastIndex = 0;
- let match: RegExpExecArray | null;
- while ((match = SUBSTITUTION.exec(rawStatement))) {
- const inner = (match[1] ?? match[2] ?? "").trim();
- if (!inner) continue;
- const normalizedInner = normalizeForMatch(inner);
- const rule = SUBSTITUTION_INNER_RULES.find((r) => r.pattern.test(normalizedInner) && (r.guard?.(inner) ?? true));
- if (!rule) continue;
- return {
- ruleId: "indirect",
- statement: rawStatement,
- suggestedTool: `command substitution runs a ${rule.ruleId} (${inner.split(/\s+/)[0]}) whose output is used elsewhere -- ${rule.suggestedTool}`,
- };
- }
- return undefined;
- }
- export function classifyBashCommand(command: string): BashAuditMatch[] {
- const stripped = stripHeredocsAndHereStrings(command);
- const matches: BashAuditMatch[] = [];
- for (const rawStatement of stripped.split(STATEMENT_SPLIT)) {
- const statement = rawStatement.trim();
- if (statement.length === 0) continue;
- const match = classifyLeadingCommand(statement) ?? classifySubstitutions(statement);
- if (match) matches.push(match);
- }
- return matches;
- }
- export interface BashAuditDecision {
- matches: BashAuditMatch[];
- refuse: boolean;
- refusalMessage?: string;
- }
- /**
- * `resolveTargetExists` lets the caller (extensions/pi-sem.ts, which has
- * real fs access) answer "does this write's target already exist" for a
- * "write"-classified match with an extracted `path` -- mirroring
- * write-audit.ts's own pure-classifier-plus-caller-resolved-existence
- * split. Omitted (the default for every existing call site and test), a
- * "write" match never auto-refuses under strict mode -- audit-only, same
- * as a match with no extractable path at all. When provided and the write
- * targets an existing code file, the refusal is delegated to
- * write-audit.ts's own `auditWriteCommand` so the message is the SAME text
- * the builtin `write` tool wrapper already gives, not a separately-maintained
- * copy that can drift.
- */
- export function auditBashCommand(
- command: string,
- strict: boolean,
- resolveTargetExists?: (path: string) => boolean,
- ): BashAuditDecision {
- const matches = classifyBashCommand(command);
- if (matches.length === 0 || !strict) return { matches, refuse: false };
- const first = matches[0];
- if (first === undefined) return { matches, refuse: false };
- if (first.ruleId === "write") {
- if (first.path === undefined) return { matches, refuse: false };
- const targetExists = resolveTargetExists?.(first.path) ?? false;
- const writeDecision = auditWriteCommand(first.path, 0, targetExists, strict);
- if (!writeDecision.refuse) return { matches, refuse: false };
- // Prefix with the offending bash statement for context, then reuse the
- // builtin write wrapper's own refusalMessage's core guidance text
- // verbatim (its own leading "pi-sem: " is stripped so the two prefixes
- // don't double up) -- this is the "same message as the write wrapper",
- // guaranteed by sharing the string, not by separately maintaining
- // matching prose.
- return {
- matches,
- refuse: true,
- refusalMessage: `pi-sem: "${first.statement}" -- ${writeDecision.refusalMessage?.replace(/^pi-sem: /, "")}`,
- };
- }
- return {
- matches,
- refuse: true,
- refusalMessage:
- `pi-sem: "${first.statement}" looks like a file ${first.ruleId}` +
- ` -- use ${first.suggestedTool}.` +
- ` (PI_SEM_STRICT is set; unset it or set PI_SEM_STRICT=0 to audit instead of refuse.)`,
- };
- }
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.