Skip to content
Back to search

token-saver-marketplace

SkillGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ppgranger--token-saver/badge.svg)](/listings/ppgranger--token-saver)

HTML

<a href="/listings/ppgranger--token-saver"><img src="/api/listings/ppgranger--token-saver/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

/plugin marketplace add ppgranger/token-saver
/plugin install token-saver

(Also available on Anthropic's official community marketplace — see [Installation](#installation) for both options.)

That's the whole setup. Ask Claude to run anything — *"run the tests"*, *"show me the diff"*, *"what changed in the last 10 commits"* — and the output is compressed before it reaches the model. Then:

token-saver stats

Want to see it work before installing it?

git clone https://github.com/ppgranger/token-saver.git && cd token-saver
python3 bin/token-saver benchmark 'git log --oneline -50'
python3 bin/token-saver benchmark 'git diff' --show-removed
python3 bin/token-saver explain 'docker compose logs | grep error'

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 22 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "token-saver-marketplace",
  3. "owner": {
  4. "name": "ppgranger",
  5. "url": "https://github.com/ppgranger"
  6. },
  7. "metadata": {
  8. "description": "Token-Saver: automatic output compression for AI coding assistants"
  9. },
  10. "plugins": [
  11. {
  12. "name": "token-saver",
  13. "source": "./",
  14. "description": "Automatically compresses verbose CLI output to save tokens. 36 specialized processors for git, docker, npm, terraform, kubectl, helm, ansible, cargo, go, and more.",
  15. "version": "3.0.1",
  16. "author": {
  17. "name": "ppgranger"
  18. },
  19. "keywords": ["token-optimization", "compression", "cost-saving"]
  20. }
  21. ]
  22. }
README.md · README · 1,355 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Token-Saver
  2. [![CI](https://github.com/ppgranger/token-saver/actions/workflows/ci.yml/badge.svg)](https://github.com/ppgranger/token-saver/actions/workflows/ci.yml)
  3. [![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://python.org)
  4. [![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
  5. [![Avg Savings](docs/assets/badge-savings.svg)](docs/processors/)
  6. **Keep the signal. Stop rereading the same failures.**
  7. Token-Saver compresses the terminal output your AI coding assistant reads —
  8. `git diff`, `pytest`, `npm install`, `terraform plan`, `kubectl`, `docker` — to
  9. leave more room for the code, instructions, and next decision. Compression runs
  10. locally, without another model call or an output upload.
  11. **New in v3: [Delta](#delta-changes-between-runs).** Run your tests,
  12. edit, run again: see which failures are new, changed, or unchanged. Fresh
  13. diagnostics keep their full details; repeated ones can become a short inventory,
  14. with details available on demand. Delta is an experimental, opt-in feature in
  15. 3.0.0 for supported pytest and Ruff commands in Claude Code.
  16. **36 specialized processors** cover git, pytest, jest, cargo, go, docker,
  17. kubernetes, terraform, pulumi, helm, ansible, aws, gcloud, and more. Their rules
  18. reduce progress logs and boilerplate, with failure fixtures and optional quality
  19. contracts checking important diagnostics.
  20. Ordinary compression runs automatically in **Claude Code** and **Antigravity
  21. CLI** after installation. **Codex** can use an explicit wrapper with
  22. [project instructions](docs/codex.md). Results depend on the command and output;
  23. already concise or unsupported output may stay unchanged.
  24. **Why developers use Token-Saver:**
  25. - 💸 **Less output to process** — many benchmark scenarios save 60-99% of estimated output tokens; billing impact depends on your usage and provider.
  26. - 🔁 **See what changed** — opt-in Delta compares repeated pytest/Ruff runs and keeps retained details one command away.
  27. - 🪟 **Bigger effective context** — fit more real work into the same context window.
  28. - 🎯 **Tested preservation** — failure fixtures check important diagnostics; add quality contracts for your own workflows.
  29. - 🔌 **Install once, forget it** — works automatically in the background, no prompts to change.
  30. - 🛡️ **Local compression** — pure regex/parsing with no network calls in the compression path.
  31. ---
  32. ## Table of Contents
  33. - [Before & After](#before--after)
  34. - [Quick Start](#quick-start)
  35. - [Why Token-Saver Exists](#why-token-saver-exists)
  36. - [Who It's For](#who-its-for)
  37. - [What Gets Compressed — Real Examples](#what-gets-compressed--real-examples)
  38. - [How It Compares](#how-it-compares)
  39. - [How It Works](#how-it-works)
  40. - [Precision Guarantees](#precision-guarantees)
  41. - [Installation](#installation)
  42. - [Codex setup](docs/codex.md)
  43. - [Optional Markdown workflow](docs/markdown-workflow.md)
  44. - [CLI Reference](#cli-reference)
  45. - [Delta: Changes Between Runs](#delta-changes-between-runs)
  46. - [Compression Quality Gates](#compression-quality-gates)
  47. - [Processors](#processors)
  48. - [Configuration](#configuration)
  49. - [Tuning Recipes](#tuning-recipes)
  50. - [Custom Processors](#custom-processors)
  51. - [Savings Tracking](#savings-tracking)
  52. - [Security & Privacy](#security--privacy)
  53. - [Performance](#performance)
  54. - [FAQ](#faq)
  55. - [Troubleshooting](#troubleshooting)
  56. - [Known Limitations](#known-limitations)
  57. - [Project Structure](#project-structure)
  58. - [Tests](#tests)
  59. - [Contributing](#contributing)
  60. ---
  61. ## Before & After
  62. | Command | Raw Output | Compressed | Savings |
  63. |---------|-----------|------------|---------|
  64. | `git diff` (5 files, 20 context lines each) | 2,270 tokens | 546 tokens | **76%** |
  65. | `git diff --stat` (25 files) | 526 tokens | 21 tokens | **96%** |
  66. | `git log --oneline` (50 entries) | 568 tokens | 117 tokens | **79%** |
  67. | `pytest` (500 tests, 2 failures) | 6,744 tokens | 307 tokens | **95%** |
  68. | `pytest` (200 passed, 30 warnings) | 3,543 tokens | 80 tokens | **98%** |
  69. | `jest` (50 suites, all passing) | 570 tokens | 32 tokens | **94%** |
  70. | `npm install` (220 packages) | 3,843 tokens | 4 tokens | **99.9%** |
  71. | `pip install -r requirements.txt` (30 packages) | 1,797 tokens | 88 tokens | **95%** |
  72. | `cargo build` (120 crates) | 934 tokens | 21 tokens | **98%** |
  73. | `docker build` (20 steps) | 1,682 tokens | 207 tokens | **88%** |
  74. | `curl` download (100 progress lines) | 2,122 tokens | 0 tokens | **100%** |
  75. | `ruff check` (110 violations, 4 rules) | 1,475 tokens | 186 tokens | **87%** |
  76. | `eslint` (55 violations, 2 rules) | 1,292 tokens | 144 tokens | **89%** |
  77. | `tree` (350+ lines) | 1,840 tokens | 136 tokens | **93%** |
  78. | `find . -name '*.py'` (205 results) | 1,294 tokens | 217 tokens | **75%** |
  79. These are locked to [`tests/compression_baselines.json`](tests/compression_baselines.json) and gated by CI ([`tests/test_compression_ratchet.py`](tests/test_compression_ratchet.py)) — if a change makes any of them compress worse, the build fails. Run `python3 scripts/audit_compression.py` to see the full scenario set, or `token-saver benchmark '<command>'` to measure savings on your own workloads.
  80. Not every command compresses, and that's deliberate. `cat large_file.py`, `git status -s`, and `tsc` with 7 type errors all sit in the same baseline file at **0%** — they're already dense, or the model needs them verbatim. Token-Saver returns the original bytes rather than shaving a few tokens at the cost of correctness.
  81. ## Quick Start
  82. ```
  83. /plugin marketplace add ppgranger/token-saver
  84. /plugin install token-saver
  85. ```
  86. (Also available on Anthropic's official community marketplace — see [Installation](#installation) for both options.)
  87. That's the whole setup. Ask Claude to run anything — *"run the tests"*, *"show me the diff"*, *"what changed in the last 10 commits"* — and the output is compressed before it reaches the model. Then:
  88. ```bash
  89. token-saver stats
  90. ```
  91. Want to see it work before installing it?
  92. ```bash
  93. git clone https://github.com/ppgranger/token-saver.git && cd token-saver
  94. python3 bin/token-saver benchmark 'git log --oneline -50'
  95. python3 bin/token-saver benchmark 'git diff' --show-removed
  96. python3 bin/token-saver explain 'docker compose logs | grep error'
  97. ```
  98. ## Why Token-Saver Exists
  99. Verbose tool output competes with your code and instructions for context. A
  100. `pytest` run with 200 passing tests or an install with hundreds of progress
  101. lines often repeats information that can be represented much more compactly.
  102. After an edit, another run can repeat the same failures too.
  103. Token-Saver reduces that repetition using local parsing. Delta extends this to
  104. consecutive runs, while keeping a current diagnostic inventory and retrievable
  105. details. Whether this improves task completion, response time, or billing needs
  106. measurement in the agent workflow that uses it.
  107. There are three common ways to attack this:
  108. 1. **Summarize with another LLM.** Accurate-ish, but it costs a second inference per command, adds seconds of latency, and is non-deterministic — the same `pytest` run can be summarized two different ways.
  109. 2. **Truncate blindly.** Free and instant, but it's exactly how you lose the one stack-trace line that mattered.
  110. 3. **Parse the format you already know.** `git diff` has a grammar. `pytest` has a summary line. `npm install` has a progress phase and a result phase. Format-aware parsing can remove repetitive output while retaining tested diagnostics, deterministically and without another model call.
  111. Token-Saver is the third approach, applied to 36 command families. It sits
  112. between the CLI and your AI assistant, applying format-specific rules whose
  113. preservation behavior you can test against your own captured output.
  114. ## Who It's For
  115. - **Developers paying per token** on Claude Code, whether via API credits or a plan with usage limits. Tool output is often the largest single category of input tokens in an agentic session.
  116. - **Teams running agents in CI**, where a long pipeline command (`terraform plan`, `pytest`, `gradle build`) is piped straight into a model.
  117. - **Anyone hitting context limits** mid-task — the "conversation is too long" wall usually arrives right after a big `npm install` or a wide `git diff`.
  118. - **Monorepo and infrastructure work**, where a single `terraform plan` or `kubectl describe` is thousands of tokens of mostly-unchanged resource attributes.
  119. - **Privacy-sensitive environments** that can't route output through a second model or a hosted service. Token-Saver is pure local Python with no network calls in the compression path.
  120. It's probably **not** for you if your agent mostly reads and writes source files and rarely runs shell commands — file reads go through Claude Code's own Read tool, which Token-Saver doesn't intercept.
  121. ## What Gets Compressed — Real Examples
  122. Everything below is actual Token-Saver output, produced by the same scenario corpus the CI ratchet uses.
  123. ### `pytest` — 533 lines → 26 lines (95%)
  124. Passing tests collapse to a count. Every failure keeps its full traceback, assertion diff, file, and line number.
  125. ```
  126. [500 tests passed]
  127. =================================== FAILURES ===================================
  128. __________________________ test_compression_ratio ______________________________
  129. def test_compression_ratio():
  130. engine = CompressionEngine()
  131. result = engine.compress('git status', large_output)
  132. > assert len(result[0]) < len(large_output) * 0.5
  133. E AssertionError: assert 1500 < 1000
  134. ...
  135. FAILED tests/test_engine.py::test_compression_ratio - AssertionError: assert 1500 < 1000
  136. FAILED tests/test_processors.py::test_diff_context_trim - AssertionError: assert 42 < 10
  137. ========================= 2 failed, 510 passed ==
  138. ```
  139. ### `ruff check` — 111 lines → 18 lines (87%)
  140. Violations are grouped by rule with a count and a couple of concrete examples, so the model learns the *pattern* to fix instead of reading 45 near-identical lines.
  141. ```
  142. 110 issues across 4 rules:
  143. E501: 45 occurrences in 45 files
  144. src/module/e501_00.py:10:5: E501 Line too long
  145. src/module/e501_01.py:11:6: E501 Line too long
  146. ... (43 more)
  147. F401: 30 occurrences in 30 files
  148. src/module/f401_00.py:10:5: F401 imported but unused
  149. ... (28 more)
  150. Found 110 errors.
  151. ```
  152. ### `npm install` — 526 lines → 1 line (99.9%)
  153. ```
  154. Build succeeded.
  155. ```
  156. Nothing actionable was lost: the raw output was 220 `added <pkg>@<ver>` lines and a progress bar. Had the install failed, the error and the failing package would survive verbatim — see [Precision Guarantees](#precision-guarantees).
  157. ### `cargo build` — 122 lines → 2 lines (98%)
  158. ```
  159. [121 crates compiled]
  160. Finished dev [unoptimized + debuginfo] target(s) in 45.23s
  161. ```
  162. ### `git diff` — context trimmed, changes untouched (76%)
  163. Unified diffs keep every `+`/`-` line and every `@@` hunk header, with surrounding unchanged context trimmed to `max_diff_context_lines` (default 3):
  164. ```
  165. diff --git a/src/module_0.py b/src/module_0.py
  166. @@ -10,30 +10,32 @@ def some_function_0():
  167. # This is context line 9 that hasn't changed and takes up tokens
  168. - old_value = compute_something(0)
  169. - return old_value
  170. + new_value = compute_something_better(0)
  171. + cached = cache.get(new_value)
  172. + if cached:
  173. + return cached
  174. + return new_value
  175. # This is trailing context line 0 that is unchanged and wastes tokens
  176. ```
  177. ### `tree` — 312 lines → 28 lines (93%)
  178. The directory shape is preserved down to the truncation point, with an explicit marker and the original totals, so the model knows what it *isn't* seeing:
  179. ```
  180. .
  181. ├── src/
  182. │ ├── engine.py
  183. │ ├── config.py
  184. │ ├── processors/
  185. │ │ ├── git.py
  186. ...
  187. ... (287 lines truncated)
  188. 20 directories, 310 files
  189. ```
  190. ### `env` / `printenv` — secrets redacted before the model sees them
  191. Variables whose names look sensitive (`SECRET`, `PASSWORD`, `CREDENTIAL`, `API_KEY`, `GITHUB_TOKEN`, and bare `KEY`/`TOKEN`/`AUTH`/`PWD` at letter boundaries — so `PATH`, `AUTHOR`, and `MONKEY` are left alone) have their values replaced. Token-Saver returns this redacted output **even when it isn't smaller** than the input: a redacted result is never traded back for the raw one to satisfy a compression threshold.
  192. ## How It Compares
  193. | | Token-Saver | LLM summarizer | Blind truncation | Response caching |
  194. |---|---|---|---|---|
  195. | Extra inference cost | None | One call per command | None | None |
  196. | Additional work | Python startup + parsing | Another model call | Truncation | Cache lookup |
  197. | Deterministic | Yes | No | Yes | Yes |
  198. | Diagnostic preservation | Tested fixtures + configurable quality contracts | Depends on prompt/model | No semantic checks | Depends on cached content |
  199. | Works offline | Yes | Needs a model | Yes | Usually not |
  200. | Understands `git diff` | Yes | Sort of | No | N/A |
  201. For a sourced comparison with RTK and Context Mode, including integration caveats, see the [full comparison](docs/comparison.md).
  202. ## How It Works
  203. ### Architecture
  204. **Built around explicit SOLID boundaries.** Processors parse output, a shared
  205. policy decides command eligibility, adapters execute commands, and dedicated
  206. modules own storage and presentation. Delta comparison and statistics formatting
  207. are pure functions; engine dependencies and telemetry writers can be supplied
  208. independently. The [architecture guide](docs/architecture.md) documents the
  209. contracts, compatibility guarantees, tests, and remaining couplings.
  210. ```
  211. CLI command --> Specialized processor --> Compressed output
  212. |
  213. 36 processors
  214. (git, test, cargo, go, build,
  215. lint, package_list, python_install,
  216. maven_gradle, bun, network, docker,
  217. kubectl, terraform, pulumi, cdktf,
  218. nix, mise, env, search, system_info,
  219. gh, db_query, cloud_cli, ansible,
  220. helm, syslog, ssh, jq_yq, just, act,
  221. structured_log, file_listing,
  222. file_content, generic)
  223. ```
  224. The engine (`CompressionEngine`) maintains a priority-ordered chain of processors.
  225. The first processor that can handle the command (`can_handle()`) produces the
  226. compressed output. `GenericProcessor` serves as a fallback and always matches last.
  227. When a specialized processor doesn't achieve the minimum compression ratio,
  228. the engine tries the generic processor as a fallback before returning uncompressed output.
  229. After the specialized processor runs, a lightweight cleanup pass (`clean()`)
  230. strips residual ANSI codes and collapses consecutive blank lines.
  231. ### The Compression Pipeline, Step by Step
  232. 1. **Eligibility.** The hook first decides whether the command is safe to wrap at all — see [Security & Privacy](#security--privacy). Redirections, `sudo`, editors, unquoted newlines, background `&`, and shell-construct fragments are all left alone.
  233. 2. **Size gates.** Output shorter than `min_input_length` is returned untouched. Output longer than `max_output_bytes` (default 10 MB) is hard-capped with an explicit note before compression, so a pathological command can't be held in memory in full.
  234. 3. **Routing.** Processors are tried in priority order; the first `can_handle()` match wins.
  235. 4. **Failure routing.** If the command exited non-zero and the matched processor hasn't opted into `handles_failure`, the engine deliberately routes to the generic processor instead. A specialized processor's heuristics are tuned for the *success* shape of a tool's output; failure text often isn't recognized, and an unrecognized line is exactly what a loop drops silently.
  236. 5. **Chaining.** A processor may declare `chain_to` to hand its result to a second one, bounded by `max_chain_depth`.
  237. 6. **Cleanup.** ANSI escape stripping and blank-line collapsing.
  238. 7. **Critical-line recovery.** The engine diffs the compressed output against the original for error-shaped lines that vanished, and re-appends up to `recover_critical_lines` of them under an explicit `[token-saver] N error line(s) recovered` marker. This is a backstop covering all 36 processors — and every future one — rather than an audit that has to be repeated per processor.
  239. 8. **Ratio gate.** If the result isn't meaningfully smaller, the original is returned. **Exception:** if the processor redacted secrets, the redacted output always wins, regardless of size.
  240. 9. **Tracking.** Sizes — never content — go to a local SQLite database.
  241. ### Platform Integration
  242. The two platforms use different mechanisms:
  243. **Claude Code** (PreToolUse hook):
  244. ```
  245. 1. Claude wants to run `git status`
  246. 2. PreToolUse hook intercepts the command
  247. 3. Rewrites to: /path/to/hook-python wrap.py 'git status'
  248. 4. wrap.py executes the original command
  249. 5. Compresses the output
  250. 6. Claude receives the compressed version
  251. ```
  252. Claude Code's PreToolUse hook cannot modify output after execution.
  253. The only way to reduce tokens is to rewrite the command to go through a wrapper
  254. that executes, compresses, and returns the result.
  255. Because that rewrite is a source-to-source shell transformation, the rewritten
  256. command is syntax-checked (`sh -n -c`) before it runs. If the rewrite wouldn't
  257. parse, the **original** command runs uncompressed — a missed saving, never a
  258. broken command.
  259. **Antigravity CLI** (AfterTool hook):
  260. ```
  261. 1. Antigravity executes the command
  262. 2. AfterTool hook receives the raw output
  263. 3. Compresses the output
  264. 4. Replaces it via {"decision": "deny", "reason": "<compressed output>"}
  265. ```
  266. Antigravity CLI allows direct output replacement through the deny/reason mechanism.
  267. ### Chained Commands
  268. `git add -A && git commit -m wip && git push` is a single Bash tool call, but three
  269. commands with three different output shapes. Token-Saver splits the chain, validates
  270. each segment independently, injects unique boundary markers, and compresses each
  271. segment with its own processor — so the `git push` output is handled by the git
  272. processor even though it shares a shell with two other commands. If *any* segment is
  273. ineligible, the whole chain runs untouched.
  274. ## Precision Guarantees
  275. Compression is aggressive on noise, conservative on signal:
  276. - Tiny outputs are never touched, and a result that doesn't actually get smaller is discarded in favor of the original (both thresholds are configurable — see [Configuration](#configuration))
  277. - Failure fixtures verify specific errors, tracebacks and changed lines; grouping and bounded recovery can omit other details
  278. - Source code files (`cat *.py`, `cat *.ts`, ...) pass through **unchanged** — the model needs exact content
  279. - Secrets in `.env.production`, `.env.local`, and other `.env.*` variants are automatically **redacted** before reaching the model (`.env`, `.env.example`, and `.env.template` pass through unchanged, by design — see the note below)
  280. - Processors remove or group progress bars, passing tests, installation logs, ANSI codes and other repetitive content according to their documented rules
  281. - Generic truncation includes an omission marker; format-specific processors can also summarize or omit lines
  282. - 1300+ tests including precision-specific tests that verify every critical piece of data survives compression, and a compression-ratchet suite that fails CI if a real-world scenario's compression ratio regresses
  283. Two of these are enforced structurally rather than by convention:
  284. - **`TestFailureHandling`** runs a realistic *failing*-command fixture through **every** processor, at **every** exit code (`0`, `1`, and unknown), and asserts the failure reason is still present. A processor that claims `handles_failure = True` has to earn it; one that doesn't gets routed around.
  285. - **Critical-line recovery** (step 7 above) restores recognized error lines within a configurable cap, except after redaction where restoring raw lines could expose secrets.
  286. These are protections on tested output formats, not a universal lossless guarantee.
  287. Use [quality gates](#compression-quality-gates) to assert the exact diagnostics
  288. your workflow requires.
  289. > **Note on `.env`:** `.env`, `.env.example`, and `.env.template` are intentionally left untouched (not redacted, not compressed) — these are the files you're most likely actively editing, where exact values matter. Other `.env.*` variants (`.env.production`, `.env.local`, ...), which you're more likely to be reading than editing, get their values redacted. If you `cat .env` directly, treat that output as sensitive the same way you would without Token-Saver installed.
  290. ## Installation
  291. ### Prerequisites
  292. - Python 3.10+
  293. - Claude Code, Antigravity CLI, or Codex with the [manual wrapper setup](docs/codex.md)
  294. No third-party Python packages are required — Token-Saver uses only the standard library.
  295. For Codex, follow the [explicit wrapper and AGENTS.md recipe](docs/codex.md).
  296. It uses a local checkout and does not require installing either platform plugin.
  297. For section-scoped Markdown work, see the separate, opt-in
  298. [Texio evaluation recipe](docs/markdown-workflow.md).
  299. ### Method 1: Claude Code Plugin (recommended)
  300. Two marketplaces carry Token-Saver — pick one.
  301. **Anthropic's official community marketplace** (`claude-community`), no setup beyond adding it once:
  302. ```
  303. /plugin marketplace add anthropics/claude-plugins-community
  304. /plugin install token-saver@claude-community --scope project
  305. ```
  306. This mirror is a snapshot pinned to a specific commit, refreshed periodically rather than on every push — if you want the exact latest commit on `main`, use the self-hosted marketplace below instead.
  307. **Self-hosted marketplace** (this repo, always current):
  308. ```
  309. /plugin marketplace add ppgranger/token-saver
  310. /plugin install token-saver
  311. ```
  312. Or test directly from a local clone:
  313. ```bash
  314. git clone https://github.com/ppgranger/token-saver.git
  315. claude --plugin-dir ./token-saver
  316. ```
  317. ### Method 2: Manual installation
  318. ```bash
  319. git clone https://github.com/ppgranger/token-saver.git
  320. cd token-saver
  321. python3 install.py --target claude # Claude Code only
  322. python3 install.py --target antigravity # Antigravity CLI only
  323. python3 install.py --target both # Both platforms
  324. ```
  325. The manual installer registers token-saver as a native Claude Code plugin
  326. (equivalent to `/plugin install`). It appears in `/plugin` list and hooks,
  327. skills, and commands are managed natively by Claude Code.
  328. The repo/zip can be deleted after installation. Token-Saver copies everything
  329. it needs to `~/.token-saver/` and the platform plugin directories.
  330. ### Development Mode
  331. ```bash
  332. python3 install.py --target claude --link # Symlinks instead of copies
  333. ```
  334. Changes in the source directory are immediately applied.
  335. Do **not** delete the repo in this mode.
  336. ### Uninstall
  337. ```bash
  338. python3 install.py --uninstall # Remove from all platforms
  339. python3 install.py --uninstall --keep-data # Keep stats DB
  340. ```
  341. ### Updating
  342. **Plugin install**: Run `/plugin update token-saver` from within Claude Code.
  343. Marketplace installs do not add the `token-saver` CLI to your shell's `PATH`.
  344. **Manual install**: Run `token-saver update` from anywhere, or:
  345. ```bash
  346. cd token-saver && git pull && python3 install.py --target claude
  347. ```
  348. **GitHub releases**: The SessionStart hook checks for new releases via the
  349. GitHub API for both install methods and recommends the appropriate updater.
  350. The remote lookup is cached for 24h and fails open — a network outage or an
  351. API rate limit never blocks a session.
  352. ### Upgrading from v1.x to v2.0
  353. If you previously installed token-saver v1.x:
  354. ```bash
  355. cd token-saver
  356. git pull
  357. python3 install.py --target claude
  358. ```
  359. The installer automatically:
  360. - Removes legacy hooks from `~/.claude/settings.json` (no longer needed)
  361. - Removes the old `~/.claude/plugins/token-saver/` directory
  362. - Installs to the plugin cache as a native Claude Code plugin
  363. - Registers in `enabledPlugins` and `installed_plugins.json`
  364. You can also run `token-saver update` from anywhere to auto-upgrade.
  365. ### Avoid dual installation
  366. Do NOT install token-saver via BOTH `/plugin install` AND `python3 install.py`
  367. simultaneously — this could register the plugin twice. Use one method or the
  368. other. The same applies across marketplaces: install from either
  369. `claude-community` or the self-hosted `ppgranger/token-saver` marketplace,
  370. not both.
  371. To switch from manual to marketplace:
  372. ```bash
  373. python3 install.py --uninstall --target claude
  374. /plugin marketplace add ppgranger/token-saver
  375. /plugin install token-saver
  376. ```
  377. ### What the Installer Does
  378. 1. Copies (or symlinks) files to:
  379. - Core: `~/.token-saver/` (CLI, updater, shared source)
  380. - Claude Code: `~/.claude/plugins/cache/token-saver-marketplace/token-saver/`
  381. - Antigravity CLI: `~/.gemini/antigravity-cli/plugins/token-saver/`
  382. 2. Registers as a native Claude Code plugin in `installed_plugins.json` and `enabledPlugins`
  383. 3. Installs `token-saver` CLI to `~/.local/bin/`
  384. 4. Stamps the current version into plugin manifests
  385. 5. Cleans up any legacy `token-saving` or v1.x installation
  386. ## CLI Reference
  387. After installation the `token-saver` command is available. If `~/.local/bin` is not on your PATH, the installer prints instructions.
  388. | Command | What it does |
  389. |---|---|
  390. | `token-saver version` | Print the installed version |
  391. | `token-saver stats` | Show session + lifetime savings |
  392. | `token-saver stats --json` | Same, machine-readable |
  393. | `token-saver update` | Check GitHub for a new release and apply it |
  394. | `token-saver benchmark '<cmd>'` | Run a command and report its compression |
  395. | `token-saver benchmark '<cmd>' --dry-run` | Show which processor would match, without executing |
  396. | `token-saver benchmark '<cmd>' --show-removed` | Line/byte breakdown of what compression removed |
  397. | `token-saver benchmark '<cmd>' --stdin` | Compress output piped on stdin instead of executing |
  398. | `token-saver benchmark '<cmd>' --format json` | Machine-readable benchmark |
  399. | `token-saver explain '<cmd>'` | Explain how a command is routed — or why it's excluded |
  400. | `token-saver explain '<cmd>' --format json` | Same, machine-readable |
  401. | `token-saver compress '<cmd>'` | Read UTF-8 stdin and emit compressed text; never execute the command label |
  402. | `token-saver compress '<cmd>' --exit-code 1 --max-tokens 1500` | Apply failure-aware routing and check an estimated output budget |
  403. | `token-saver compress '<cmd>' --format json` | Compressed output plus measurements and budget verdict |
  404. | `token-saver replay quality.json` | Check captured fixtures against preservation and budget contracts |
  405. | `token-saver replay quality.json --format json` | Content-free quality report for CI |
  406. | `token-saver delta show RUN` | Read a retained, sanitized Delta snapshot without rerunning its command |
  407. | `token-saver delta show RUN --diagnostic EXACT_ID` | Read full details for one diagnostic in that snapshot |
  408. | `token-saver delta clear` | Delete all retained Delta snapshots and reset comparisons |
  409. `explain` is the fastest way to answer "why didn't that get compressed?":
  410. ```console
  411. $ token-saver explain 'git status'
  412. Token-Saver Explain
  413. ========================================
  414. Command: git status
  415. Compressible: yes
  416. Reason: matched a compressible processor pattern
  417. Processor: git
  418. $ token-saver explain 'cat x.py > out.txt'
  419. Command: cat x.py > out.txt
  420. Compressible: no
  421. Reason: contains output redirection (>, >>, 2>, &>)
  422. Excluded by: output redirection
  423. Processor: file_content
  424. ```
  425. And `--stdin` lets you test compression against output you already have, with no re-execution:
  426. ```bash
  427. pytest > /tmp/out.txt 2>&1
  428. token-saver benchmark 'pytest' --stdin --show-removed < /tmp/out.txt
  429. ```
  430. ## Delta: Changes Between Runs
  431. **New in 3.0.0: spend context on the next change.** Delta is experimental and
  432. disabled by default.
  433. Delta compares consecutive `pytest` and `ruff check` results in Claude Code so
  434. the agent can see what changed after an edit. Commands still execute normally.
  435. New or changed diagnostics retain their full details; unchanged diagnostics keep
  436. their identity and summary, with a command to retrieve the details locally.
  437. | In your edit–test loop | What Delta reports |
  438. |---|---|
  439. | First supported run | A baseline with full failure details. |
  440. | Same failures again | `UNCHANGED` with names and summaries when that saves output. |
  441. | An assertion or traceback changes | `CHANGED` with its full current details. |
  442. | A new regression appears | `NEW` with its full current details. |
  443. | A previous failure disappears | `PASSED` only with explicit passing evidence; otherwise `NOT OBSERVED`. |
  444. | You need the original diagnostic | `token-saver delta show RUN --diagnostic EXACT_ID`, without rerunning tests. |
  445. One repeated run from the [real-command fixture benchmark](docs/delta.md#run-the-real-command-benchmark)
  446. returns this instead of 4,079 characters of ordinary compressed output
  447. (`RUN` replaces the generated snapshot identifier):
  448. ```text
  449. [token-saver delta] pytest | exit 1
  450. 3 failed, 21 passed in 0.02s
  451. UNCHANGED test_fixture.py::test_payload — AssertionError: payload diagnostic
  452. UNCHANGED test_fixture.py::test_status — AssertionError: status diagnostic
  453. UNCHANGED test_fixture.py::test_cache — AssertionError: cache diagnostic
  454. Details: token-saver delta show RUN [--diagnostic ID]
  455. ```
  456. The complete five-run fixture sequence saves **32.5% of output characters for
  457. pytest and 8.3% for Ruff**, compared with ordinary compression, **including one
  458. targeted detail retrieval**. These are measured fixture results, not universal
  459. savings or billing figures. A complete snapshot read is more expensive: it
  460. erases the gain in the Ruff scenario. The guide includes every result and the
  461. reproduction command.
  462. Enable it explicitly in your global configuration, or before launching Claude:
  463. ```bash
  464. export TOKEN_SAVER_DELTA_ENABLED=true
  465. ```
  466. The comparison stays within the same session, working directory, and exact
  467. command. Each Delta response includes the current exit status, totals, and diagnostic
  468. inventory. A missing failure is **`NOT OBSERVED`, not confirmed fixed**; `PASSED`
  469. requires an explicit passing test result in the current output. Use `pytest -v`
  470. to expose those per-test results.
  471. Delta is disabled by default. Enabling it retains sanitized output snapshots
  472. locally for 24 hours and at most 100 runs by default. It currently applies only
  473. to supported single commands through the Claude wrapper, with a host session
  474. identifier. Unsupported or ambiguous output uses ordinary compression.
  475. The first run and new or changed failures can produce **more output than v2**
  476. when preserving full details requires it. For unchanged repeats, Token-Saver uses
  477. ordinary sanitized compression whenever Delta would be as large or larger.
  478. Ordinary compression can also win for new failures if it retains every full new
  479. or changed diagnostic. Valid snapshots still establish comparison baselines.
  480. The [reproducible Delta scenario](docs/delta.md#reproduce-the-output-size-benchmark)
  481. compares five synthetic pytest captures, including detail retrieval. It does not
  482. measure agent task success or billing savings; the existing savings table above
  483. still describes ordinary compression.
  484. See the [Delta guide](docs/delta.md) for supported formats, retrieval, storage,
  485. and limits.
  486. ## Compression Quality Gates
  487. Make compression testable on **your** workflow: set per-command and total token
  488. budgets, require a minimum measured reduction, and name literal messages that
  489. must survive. `replay` checks saved fixtures without re-running commands; a typo
  490. in a required message fails against the original fixture too.
  491. ```bash
  492. python3 bin/token-saver replay examples/quality-replay.json
  493. python3 bin/token-saver compress 'pytest -v' --exit-code 1 --max-tokens 1500 < examples/fixtures/pytest_output.txt
  494. ```
  495. An over-budget result returns status `1` and retains the compressed output;
  496. it never truncates more text just to satisfy a limit. Invalid input returns `2`.
  497. Token counts use `ceil(characters / chars_per_token)` and remain estimates,
  498. not exact tokenizer or billing counts. JSON replay reports omit command strings,
  499. raw output and expected text; `compress --format json` includes the compressed output.
  500. Neither command stores captured output or savings history.
  501. See [the manifest format and CI example](docs/quality-gates.md). These additions
  502. are available in this checkout; use `python3 bin/token-saver` to try them before
  503. a release is published.
  504. ## Processors
  505. Each processor handles a family of commands. The first one that matches
  506. (`can_handle()`) processes the output. Detailed documentation for each
  507. processor is in [`docs/processors/`](docs/processors/).
  508. | # | Processor | Priority | Commands | Docs |
  509. |---|---|---|---|---|
  510. | 1 | **Package List** | 15 | pip list/freeze, npm ls, conda list, gem list, brew list | [package_list.md](docs/processors/package_list.md) |
  511. | 2 | **just** | 18 | just --list, just --summary (recipe listing compaction) | [just.md](docs/processors/just.md) |
  512. | 3 | **act** | 19 | act (run GitHub Actions locally via nektos/act) | [act.md](docs/processors/act.md) |
  513. | 4 | **Git** | 20 | status, diff, log, show, push/pull/fetch, branch, stash, reflog, blame, cherry-pick, rebase, merge | [git.md](docs/processors/git.md) |
  514. | 5 | **Test** | 21 | pytest, jest, vitest, mocha, cargo test, go test, rspec, phpunit, bun test, npm/yarn/pnpm test, dotnet test, swift test, mix test | [test_output.md](docs/processors/test_output.md) |
  515. | 6 | **Cargo** | 22 | cargo build, check, doc, update, bench | [cargo.md](docs/processors/cargo.md) |
  516. | 7 | **Go** | 23 | go build, vet, mod, generate, install | [go.md](docs/processors/go.md) |
  517. | 8 | **Python Install** | 24 | pip install, poetry install/update/add, uv pip install, uv sync | [python_install.md](docs/processors/python_install.md) |
  518. | 9 | **Build** | 25 | npm/yarn/pnpm build/install, cargo build, make, cmake, tsc, webpack, vite, next build, turbo, nx, bazel, sbt, mix compile, docker build | [build_output.md](docs/processors/build_output.md) |
  519. | 10 | **Cargo Clippy** | 26 | cargo clippy (multi-line block grouping with span/help preservation) | [cargo_clippy.md](docs/processors/cargo_clippy.md) |
  520. | 11 | **Lint** | 27 | eslint, ruff, flake8, pylint, clippy, mypy, prettier, biome, shellcheck, hadolint, rubocop, golangci-lint | [lint_output.md](docs/processors/lint_output.md) |
  521. | 12 | **Maven/Gradle** | 28 | mvn, ./mvnw, gradle, ./gradlew (download stripping, task noise removal) | [maven_gradle.md](docs/processors/maven_gradle.md) |
  522. | 13 | **Bun** | 29 | bun install, add, remove, update | [bun.md](docs/processors/bun.md) |
  523. | 14 | **Network** | 30 | curl, wget, http/https (httpie) | [network.md](docs/processors/network.md) |
  524. | 15 | **Docker** | 31 | ps, images, logs, pull/push, inspect, stats, compose up/down/build/ps/logs | [docker.md](docs/processors/docker.md) |
  525. | 16 | **Kubernetes** | 32 | kubectl/oc get, describe, logs, top, apply, delete, create | [kubectl.md](docs/processors/kubectl.md) |
  526. | 17 | **Terraform** | 33 | terraform/tofu plan, apply, destroy, init, output, state list/show | [terraform.md](docs/processors/terraform.md) |
  527. | 18 | **Environment** | 34 | env, printenv (with secret redaction) | [env.md](docs/processors/env.md) |
  528. | 19 | **Search** | 35 | grep -r, rg, ag, fd, fdfind | [search.md](docs/processors/search.md) |
  529. | 20 | **System Info** | 36 | du, wc, df | [system_info.md](docs/processors/system_info.md) |
  530. | 21 | **GitHub CLI** | 37 | gh pr/issue/run list/view/diff/checks/status | [gh.md](docs/processors/gh.md) |
  531. | 22 | **Database Query** | 38 | psql, mysql, sqlite3, pgcli, mycli, litecli | [db_query.md](docs/processors/db_query.md) |
  532. | 23 | **Cloud CLI** | 39 | aws, gcloud, az (JSON/table/text output compression) | [cloud_cli.md](docs/processors/cloud_cli.md) |
  533. | 24 | **Ansible** | 40 | ansible-playbook, ansible (ok/skipped counting, error preservation) | [ansible.md](docs/processors/ansible.md) |
  534. | 25 | **Helm** | 41 | helm install/upgrade/list/template/status/history | [helm.md](docs/processors/helm.md) |
  535. | 26 | **Syslog** | 42 | journalctl, dmesg (head/tail with error extraction) | [syslog.md](docs/processors/syslog.md) |
  536. | 27 | **SSH/SCP** | 43 | non-interactive ssh, scp (remote command output compression) | [ssh.md](docs/processors/ssh.md) |
  537. | 28 | **JQ/YQ** | 44 | jq, yq (large JSON/YAML output compaction) | [jq_yq.md](docs/processors/jq_yq.md) |
  538. | 29 | **Structured Log** | 45 | stern, kubetail (JSON Lines grouping by level) | [structured_log.md](docs/processors/structured_log.md) |
  539. | 30 | **Pulumi** | 46 | pulumi up, preview, destroy, refresh | [pulumi.md](docs/processors/pulumi.md) |
  540. | 31 | **CDKTF** | 47 | cdktf deploy, diff, destroy, synth | [cdktf.md](docs/processors/cdktf.md) |
  541. | 32 | **Nix** | 48 | nix build/develop/eval/run, nix-build, nix-shell | [nix.md](docs/processors/nix.md) |
  542. | 33 | **mise** | 49 | mise install, use, upgrade (runtime version manager) | [mise.md](docs/processors/mise.md) |
  543. | 34 | **File Listing** | 50 | ls, find, tree, exa, eza, rsync | [file_listing.md](docs/processors/file_listing.md) |
  544. | 35 | **File Content** | 51 | cat, head, tail, bat, less, more (content-aware: code, config, log, CSV) | [file_content.md](docs/processors/file_content.md) |
  545. | 36 | **Generic** | 999 | Any command (fallback: ANSI strip, dedup, truncation) | [generic.md](docs/processors/generic.md) |
  546. Lower priority numbers run first. The gaps between numbers are intentional — they leave room for [custom processors](#custom-processors) to slot in ahead of, or behind, any built-in.
  547. ## Configuration
  548. Thresholds are configurable via JSON file or environment variables. Precedence, lowest to highest:
  549. ```
  550. built-in defaults < ~/.token-saver/config.json < ./.token-saver.json < TOKEN_SAVER_* env vars
  551. ```
  552. ### Configuration File
  553. `~/.token-saver/config.json`:
  554. ```json
  555. {
  556. "enabled": true,
  557. "min_input_length": 1,
  558. "min_compression_ratio": 0.0,
  559. "max_diff_hunk_lines": 50,
  560. "max_log_entries": 10,
  561. "max_file_lines": 100,
  562. "generic_truncate_threshold": 200,
  563. "debug": false
  564. }
  565. ```
  566. Values are coerced to the type of their default, and a value that can't be coerced is ignored rather than propagated — `{"max_chain_depth": "deep"}` leaves the default in place instead of reaching arithmetic code downstream.
  567. ### Environment Variables
  568. Every key can be overridden with the `TOKEN_SAVER_` prefix:
  569. ```bash
  570. export TOKEN_SAVER_MAX_LOG_ENTRIES=50
  571. export TOKEN_SAVER_DEBUG=true
  572. # Disable compression entirely (bypass mode)
  573. export TOKEN_SAVER_ENABLED=false
  574. ```
  575. List-valued keys take a comma-separated string: `TOKEN_SAVER_DISABLED_PROCESSORS=file_content,network`.
  576. ### Per-Project Configuration
  577. Drop a `.token-saver.json` in your repository root to override global settings:
  578. ```json
  579. {
  580. "max_diff_hunk_lines": 300,
  581. "generic_truncate_threshold": 1000,
  582. "max_log_entries": 50
  583. }
  584. ```
  585. Project settings are merged with global settings. Token-Saver walks up parent directories (like `.gitignore` resolution) to find the nearest `.token-saver.json`, stopping at your home directory or the filesystem root. Useful for monorepos or projects with atypical output patterns (large Terraform plans, verbose test suites, etc.).
  586. **Security note:** project files cannot set `user_processors_dir` (loads Python
  587. code), `disabled_processors` or `redaction_allowlist` (can weaken redaction), or
  588. `delta_enabled`, `delta_retention_hours`, and `delta_max_runs` (control local
  589. output retention). Set these only in global `~/.token-saver/config.json` or via
  590. `TOKEN_SAVER_*` environment variables. A project file that sets them has those
  591. keys dropped with a debug-log line, not coerced. Cloning a repository cannot
  592. opt you into retaining its command output.
  593. ### Complete Parameter List
  594. This table is generated from `src/config.py`'s `_DEFAULTS` dict — that file is the
  595. source of truth if the two ever disagree, and
  596. [`tests/test_readme_config_sync.py`](tests/test_readme_config_sync.py) fails CI if
  597. they drift apart.
  598. | Parameter | Default | Description |
  599. |---|---|---|
  600. | `enabled` | `true` | Master switch -- set to `false` to bypass all compression |
  601. | `min_input_length` | `1` | Minimum threshold (characters) to attempt compression |
  602. | `min_compression_ratio` | `0.0` | Minimum gain to apply compression (0 = apply any gain) |
  603. | `wrap_timeout` | `300` | Wrapper timeout in seconds |
  604. | `max_diff_hunk_lines` | `50` | Max lines per hunk in git diff |
  605. | `max_diff_context_lines` | `3` | Context lines kept before/after each change in diffs |
  606. | `max_log_entries` | `10` | Max entries in git log/reflog |
  607. | `max_file_lines` | `100` | Threshold before file content compression kicks in |
  608. | `file_keep_head` | `80` | Lines kept from the start of file (fallback strategy) |
  609. | `file_keep_tail` | `30` | Lines kept from the end of file (fallback strategy) |
  610. | `generic_truncate_threshold` | `200` | Generic truncation threshold |
  611. | `generic_keep_head` | `100` | Lines kept from the start (generic) |
  612. | `generic_keep_tail` | `50` | Lines kept from the end (generic) |
  613. | `generic_keep_critical` | `20` | Max error/failure lines rescued from the truncated middle (0 disables) |
  614. | `recover_critical_lines` | `20` | Max error lines the engine re-appends when a processor dropped them (0 disables) |
  615. | `ls_compact_threshold` | `15` | Items before ls compaction |
  616. | `find_compact_threshold` | `20` | Results before find compaction |
  617. | `tree_compact_threshold` | `30` | Lines before tree truncation |
  618. | `lint_example_count` | `2` | Examples shown per lint rule |
  619. | `lint_group_threshold` | `3` | Occurrences before grouping by rule |
  620. | `file_code_head_lines` | `15` | Import/header lines to preserve in code files |
  621. | `file_code_body_lines` | `2` | Body lines kept per function/class definition |
  622. | `file_log_context_lines` | `2` | Context lines around errors in log files |
  623. | `file_csv_head_rows` | `3` | Data rows kept from start of CSV files |
  624. | `file_csv_tail_rows` | `2` | Data rows kept from end of CSV files |
  625. | `search_max_per_file` | `3` | Max match lines shown per file |
  626. | `search_max_files` | `15` | Max files shown in search results |
  627. | `kubectl_keep_head` | `5` | Lines kept from start of kubectl logs |
  628. | `kubectl_keep_tail` | `10` | Lines kept from end of kubectl logs |
  629. | `docker_log_keep_head` | `5` | Lines kept from start of docker logs |
  630. | `docker_log_keep_tail` | `10` | Lines kept from end of docker logs |
  631. | `git_branch_threshold` | `15` | Branches before compaction |
  632. | `git_stash_threshold` | `5` | Stash entries before truncation |
  633. | `max_traceback_lines` | `30` | Max traceback lines before truncation |
  634. | `db_max_rows` | `20` | Max rows shown per database query result |
  635. | `db_prune_days` | `90` | Stats retention in days |
  636. | `chars_per_token` | `4` | Chars-per-token ratio used to estimate token counts (display only) |
  637. | `cargo_warning_example_count` | `2` | Example warnings shown per cargo/clippy lint |
  638. | `cargo_warning_group_threshold` | `3` | Occurrences before grouping cargo warnings |
  639. | `jq_passthrough_threshold` | `50` | Below this many lines, jq/yq output passes through unchanged |
  640. | `max_chain_depth` | `3` | Maximum processor chain depth (`chain_to`) |
  641. | `max_output_bytes` | `10000000` | Hard cap on output length (chars) before compression runs (0 disables) |
  642. | `debug` | `false` | Enable debug logging |
  643. | `user_processors_dir` | `""` (falls back to `~/.token-saver/processors/`) | Directory for custom processors — **global config / env var only, not project config** |
  644. | `disabled_processors` | `[]` | Processor names to disable (env: comma-separated) — **global config / env var only, not project config** |
  645. | `redaction_allowlist` | `[]` | Env var name patterns exempt from secret redaction — **global config / env var only, not project config** |
  646. | `delta_enabled` | `false` | Experimental repeated-run diagnostics and local output retention — **global config / env var only, not project config** |
  647. | `delta_retention_hours` | `24` | Delta snapshot lifetime in hours, from 1 to 168 — **global config / env var only, not project config** |
  648. | `delta_max_runs` | `100` | Maximum retained Delta snapshots across all scopes, from 1 to 1000 — **global config / env var only, not project config** |
  649. Parameters marked **global config / env var only** cannot be set from a
  650. project-level `.token-saver.json` — see the security note in
  651. [Per-Project Configuration](#per-project-configuration).
  652. ## Tuning Recipes
  653. **"I want maximum savings and I trust the precision tests."** The defaults already apply
  654. any gain at all (`min_compression_ratio: 0.0`), so turn the truncation thresholds down:
  655. ```json
  656. { "generic_truncate_threshold": 80, "max_log_entries": 5, "max_diff_context_lines": 1 }
  657. ```
  658. **"I'm doing a careful code review and want full diffs."** Loosen the diff limits in a
  659. project-level `.token-saver.json`:
  660. ```json
  661. { "max_diff_hunk_lines": 1000, "max_diff_context_lines": 10 }
  662. ```
  663. **"Terraform plans in this repo are enormous and I need all of them."**
  664. ```json
  665. { "generic_truncate_threshold": 2000, "max_output_bytes": 50000000 }
  666. ```
  667. **"One processor is mangling a tool I care about."** Disable just that one — globally,
  668. not from a project file:
  669. ```bash
  670. export TOKEN_SAVER_DISABLED_PROCESSORS=jq_yq
  671. ```
  672. The generic processor can't be disabled; it's the fallback that provides `clean()`.
  673. **"Turn everything off for this session."**
  674. ```bash
  675. export TOKEN_SAVER_ENABLED=false
  676. ```
  677. **"Show me what's happening."**
  678. ```bash
  679. export TOKEN_SAVER_DEBUG=true
  680. ```
  681. ## Custom Processors
  682. You can extend Token-Saver with your own processors for commands not covered by the built-in 36.
  683. 1. Create a Python file with a class inheriting from `src.processors.base.Processor`
  684. 2. Implement `can_handle()`, `process()`, `name`, and set `priority`
  685. 3. Copy the file to `~/.token-saver/processors/`
  686. ```bash
  687. # Example: install the ansible processor
  688. cp examples/custom_processor/ansible_output.py ~/.token-saver/processors/
  689. ```
  690. A minimal processor looks like this:
  691. ```python
  692. import re
  693. from src.processors.base import Processor
  694. class TfsecProcessor(Processor):
  695. priority = 16 # a free slot: 15 is package_list, 18 is just
  696. hook_patterns = [r"^tfsec\b"]
  697. @property
  698. def name(self) -> str:
  699. return "tfsec"
  700. def can_handle(self, command: str) -> bool:
  701. return bool(re.match(r"tfsec\b", command.strip()))
  702. def process(self, command: str, output: str) -> str:
  703. keep = [ln for ln in output.splitlines() if "Result" in ln or "CRITICAL" in ln]
  704. return "\n".join(keep) if keep else output
  705. ```
  706. Two details worth knowing:
  707. - `hook_patterns` is what the PreToolUse hook matches *before any output exists*; `can_handle()` is what the engine matches *after*. They must agree, and [`tests/test_pattern_consistency.py`](tests/test_pattern_consistency.py) enforces that for every built-in processor — a command that gets wrapped but never routed is pure overhead.
  708. - Returning `output` unchanged is a supported, first-class outcome. It signals "I looked and decided not to compress", which the engine records differently from a failed compression attempt.
  709. User processors are auto-discovered on every invocation. A broken processor (syntax error, missing import) is skipped with a warning — it never crashes the engine.
  710. See [`examples/custom_processor/`](examples/custom_processor/) for a complete example with documentation, and [CONTRIBUTING.md](CONTRIBUTING.md) if you'd like to upstream one.
  711. ## Savings Tracking
  712. Token-Saver records every compression in a local SQLite database:
  713. ```
  714. ~/.token-saver/savings.db
  715. ```
  716. ### Tables
  717. - **savings**: each individual compression (timestamp, command, processor, sizes, platform)
  718. - **sessions**: aggregated totals per session (first/last activity, total original/compressed, command count)
  719. ### Automatic Stats
  720. On every session start, the `SessionStart` hook displays a summary:
  721. ```
  722. [token-saver] Lifetime: 342 cmds, 307.2k tokens saved (67.3%) | Session: 5 cmds, 11.3k tokens saved (72.1%)
  723. ```
  724. If a newer version is available, the notification is appended:
  725. ```
  726. [token-saver] Lifetime: 342 cmds, 307.2k tokens saved (67.3%) | Update available: v1.0.1 -> v1.1.0 -- Run: token-saver update
  727. ```
  728. ### Manual Stats
  729. ```bash
  730. token-saver stats
  731. token-saver stats --json
  732. ```
  733. ```
  734. Token-Saver Statistics
  735. ========================================
  736. Session
  737. ----------------------------------------
  738. Commands compressed: 12
  739. Original tokens: 61.3k tokens
  740. Compressed tokens: 15.5k tokens
  741. Saved: 45.8k tokens (74.7%)
  742. Lifetime
  743. ----------------------------------------
  744. Sessions: 47
  745. Commands compressed: 342
  746. Original tokens: 461.0k tokens
  747. Compressed tokens: 147.3k tokens
  748. Saved: 307.2k tokens (67.3%)
  749. Top Processors
  750. ----------------------------------------
  751. git 142 cmds, 121.8k tokens saved
  752. test 89 cmds, 78.0k tokens saved
  753. build 45 cmds, 49.7k tokens saved
  754. ```
  755. Token counts are estimated from a `chars_per_token` ratio (default 4) — they're for
  756. comparison and trend-watching, not for reconciling an invoice.
  757. ### Maintenance
  758. - Auto-pruning of records older than 90 days (configurable via `db_prune_days`)
  759. - Automatic recovery on database corruption
  760. - Thread-safe (reentrant lock on all operations)
  761. - WAL mode for concurrent write performance
  762. ## Security & Privacy
  763. ### What leaves your machine
  764. Nothing, in the compression path. Compression is pure local Python — regex and string
  765. parsing, no network, no model call, no telemetry. The only outbound request Token-Saver
  766. ever makes is an optional GitHub release check (cached 24h, 1s timeout, fails open),
  767. which sends nothing but a standard HTTP GET.
  768. ### What gets stored
  769. The stats database records command strings, processor names, size measurements,
  770. and timestamps; it does not store captured output. Command strings can themselves
  771. contain sensitive information.
  772. Opting into [Delta](docs/delta.md) additionally stores sanitized diagnostic
  773. snapshots, including captured output context, in a separate private local
  774. database. Defaults are 24-hour retention and 100 snapshots; expiry is enforced
  775. when the store is accessed. Redaction recognizes specific secret formats and
  776. cannot detect every secret. Disable `delta_enabled` to stop new captures and run
  777. `token-saver delta clear` to remove retained snapshots. Deletion does not promise
  778. secure erasure from filesystem snapshots or backups.
  779. ### How the hook is hardened
  780. - **No shell injection**: commands are passed through `shlex.quote()` when rewriting
  781. - **Syntax-checked rewrites**: the rewritten command is validated with `sh -n -c` before it runs; an invalid rewrite falls back to the original command
  782. - **Fail-open**: if the hook fails (Python error, missing file, timeout), the original command executes normally. A broken Token-Saver costs you savings, not a working shell.
  783. - **Secret redaction**: the `env` processor automatically redacts values of variables matching `*KEY*`, `*SECRET*`, `*TOKEN*`, `*PASSWORD*`, `*CREDENTIAL*` patterns, preventing accidental leakage into AI context windows — and a redacted result is never discarded in favor of the raw one by the compression-ratio gate
  784. - **Untrusted project config**: processor-loading, redaction, and Delta retention settings listed as global-only above are ignored in a repo-local `.token-saver.json`
  785. - **Signal forwarding**: the wrapper propagates SIGINT/SIGTERM to the child process
  786. - **Exclusions**: commands with complex pipes, redirections, `sudo`, editors, `ssh`, unquoted newlines, background `&`, or shell-construct fragments (`for`, `while`, `if`, `case`) are never intercepted
  787. - **Safe trailing pipes**: simple trailing pipes (`| head`, `| tail`, `| wc`, `| grep`, `| sort`, `| uniq`, `| cut`) are allowed
  788. - **Chained commands**: `&&` and `;` chains are supported — each segment is validated individually, and one ineligible segment disqualifies the whole chain
  789. - **Self-protection**: commands containing `token-saver` or `wrap.py` are not intercepted (prevents recursion)
  790. Quote-awareness matters here: `echo "a > b"` contains a `>` inside quotes and is *not* a
  791. redirection, while `2>&1` is a file-descriptor duplication rather than a write to a file.
  792. Exclusion scanning uses a quote-aware tokenizer (`src/shell_syntax.py`) rather than a
  793. naive substring search, so quoted metacharacters don't cause false exclusions — or,
  794. worse, false inclusions.
  795. ## Performance
  796. Measured on macOS, Python 3.12, warm filesystem cache:
  797. | | Time |
  798. |---|---|
  799. | Bare `sh -c 'echo hi'` | ~5ms |
  800. | `python3 -c pass` (interpreter startup floor) | ~99ms |
  801. | Full `wrap.py` round-trip | ~158ms |
  802. | Wrapper versus bare shell | ~153ms |
  803. | Portion beyond Python startup | ~60ms |
  804. These previously recorded timings include interpreter startup in the full
  805. wrapper cost. The roughly 60ms remainder is not the total overhead users pay.
  806. Measure the full round-trip on your machine: an extra 153ms is about 15% of a
  807. one-second command, or 0.5% of a thirty-second command. Short commands can make
  808. the overhead more noticeable than the output reduction.
  809. `max_output_bytes` (default 10,000,000, currently counted as characters) limits
  810. the text passed to compression after capture. It does not limit subprocess
  811. buffering or guarantee a bound on every parser's execution time. Delta adds
  812. parsing, comparison, and local SQLite storage; the timings above are for ordinary
  813. compression and do not establish its overhead.
  814. ## FAQ
  815. **Does Token-Saver send my code or output to any server?**
  816. No. Compression is entirely local — regex and string parsing in Python's standard
  817. library. The only network call is an optional GitHub release check, cached for 24 hours
  818. and silently skipped offline.
  819. **Will it hide an error from me or from the model?**
  820. That's the failure mode the preservation tests target. Known failures route
  821. around processors that have not opted into failure handling, and critical-line
  822. recovery covers selected omissions. These protections are tested on a fixture
  823. corpus; unfamiliar formats and configured truncation can still lose useful
  824. context. Use quality contracts for your workload and inspect `--show-removed`
  825. when evaluating a rule. Delta retains full new or changed diagnostics for its
  826. supported formats, with details retrievable while the snapshot is retained.
  827. **How much will I actually save?**
  828. It depends on your command mix, output, and detail retrievals. The tables above
  829. measure individual fixture scenarios, not complete agent sessions. Run
  830. `token-saver stats` for recorded command-output estimates, or
  831. `token-saver benchmark '<command>'` for one command. Delta detail retrievals add
  832. output and are not included in the ordinary savings tracker; account for them
  833. separately when evaluating a complete workflow.
  834. **Does it work with the Claude API or the Claude Agent SDK directly?**
  835. Not as a drop-in — Token-Saver ships as a Claude Code plugin and an Antigravity CLI
  836. plugin. But the engine is importable (`from src.engine import CompressionEngine`) and
  837. has no dependency on either platform, so wiring it into your own agent loop is a few
  838. lines.
  839. **Does it work on Windows?**
  840. Yes. Windows is a supported platform, with a `token-saver.cmd` launcher, an
  841. `%APPDATA%`-based data directory, and UTF-8 stdio forcing at every entry point.
  842. **Does it slow down my shell?**
  843. No — Token-Saver only runs inside your AI assistant's tool calls. Your interactive
  844. terminal is untouched.
  845. **What happens if Token-Saver crashes?**
  846. The hook fails open: the original command runs, uncompressed, exactly as it would
  847. without Token-Saver installed. Same for a Python error, a missing file, or a timeout.
  848. **Can I use it alongside other token-reduction MCP servers?**
  849. Yes. Token-Saver operates at the output level; caching and delegation tools operate at
  850. the task or invocation level. See [docs/comparison.md](docs/comparison.md).
  851. **Why not just tell the model "be brief", or pipe through `| tail -50`?**
  852. Because the model doesn't control tool output, and `tail -50` doesn't know the error was
  853. on line 12. Format-aware compression keeps the 12 lines that matter out of 500 — blind
  854. truncation keeps the last 50 and hopes.
  855. **Does it compress files Claude reads with the Read tool?**
  856. No. Token-Saver intercepts Bash commands; reads via the native file tool don't pass
  857. through it. It *does* handle `cat`, `head`, `tail`, and `bat` run as shell commands —
  858. though source code files pass through unchanged by design.
  859. **Can a repository I clone attack me through `.token-saver.json`?**
  860. Project configuration cannot load processors through `user_processors_dir`,
  861. change `disabled_processors` or `redaction_allowlist`, or enable or adjust Delta
  862. retention through `delta_enabled`, `delta_retention_hours`, or `delta_max_runs`.
  863. Those settings require global configuration or environment variables. Other
  864. project settings still affect compression behavior; review them when needed.
  865. **How do I turn it off temporarily?**
  866. `export TOKEN_SAVER_ENABLED=false`, or set `{"enabled": false}` in
  867. `~/.token-saver/config.json`.
  868. **Is a specific processor's behavior documented anywhere?**
  869. Yes — [`docs/processors/`](docs/processors/) has a page per processor covering what it
  870. matches, what it keeps, what it drops, and its config knobs.
  871. ## Troubleshooting
  872. **The hooks report that Python 3.10 or later is required.** Check the interpreter
  873. used by the host application, which may differ from your interactive terminal's
  874. `python3`. Launch the host with a `PATH` that selects Python 3.10+, or configure
  875. its hook commands to use an absolute supported Python executable. The pre-tool
  876. hook passes that same executable to `wrap.py`, including paths with spaces.
  877. Unsupported interpreters print a clear diagnostic and leave hook commands
  878. unwrapped; directly invoking the wrapper with an old Python exits with an error.
  879. **A command isn't being compressed.** Ask why:
  880. ```bash
  881. token-saver explain 'the exact command'
  882. ```
  883. The most common answers are a redirection (`>`), a `sudo`, a complex pipe, or output
  884. that was simply too small to be worth touching.
  885. **Compression looks wrong for one tool.** Reproduce it in isolation and see what got
  886. removed:
  887. ```bash
  888. token-saver benchmark 'the exact command' --show-removed
  889. ```
  890. If it's genuinely wrong, disable that one processor
  891. (`export TOKEN_SAVER_DISABLED_PROCESSORS=<name>`) and please
  892. [open an issue](https://github.com/ppgranger/token-saver/issues) with the raw output.
  893. **Stats show nothing.** Verify the plugin is actually loaded (`/plugin` in Claude Code)
  894. and that `token-saver version` resolves. If you installed manually, confirm
  895. `~/.local/bin` is on your `PATH`.
  896. **I see `[token-saver] N error line(s) recovered`.** That's the safety net working: a
  897. processor dropped lines that looked like errors and the engine put them back. Worth
  898. reporting — the processor should have kept them itself.
  899. **Everything is broken and I want out.**
  900. ```bash
  901. export TOKEN_SAVER_ENABLED=false # immediate, this session
  902. python3 install.py --uninstall # permanent
  903. ```
  904. **I need to see what the hook is doing.**
  905. ```bash
  906. export TOKEN_SAVER_DEBUG=true
  907. python3 scripts/wrap.py --dry-run 'git status'
  908. ```
  909. ## Known Limitations
  910. - Does not compress commands with complex pipelines, redirections (`> file`), or `||` chains
  911. - Simple trailing pipes are supported (`| head`, `| tail`, `| wc`, `| grep`, `| sort`, `| uniq`, `| cut`)
  912. - Chained commands (`&&`, `;`) are supported — each segment is validated individually
  913. - `sudo`, `ssh`, `vim` commands are never intercepted; remote `rsync` (with host:path) is excluded but local `rsync` is compressible
  914. - Long diff compression truncates per-hunk, not per-file: a diff with many small hunks is not reduced
  915. - The generic processor only deduplicates **consecutive identical lines**, not similar lines
  916. - Token counts in stats are estimated from a chars-per-token ratio, not a real tokenizer
  917. - Experimental Delta covers supported single pytest/Ruff commands in Claude Code; it needs an explicit host session and an unchanged command to compare runs
  918. - Delta retains local output when enabled; fresh diagnostics and detail retrieval can outweigh repeated-run savings
  919. - Antigravity CLI: the deny/reason mechanism may have side effects if other plugins use the same hook
  920. ## Project Structure
  921. ```
  922. token-saver/
  923. ├── .claude-plugin/ # Plugin metadata
  924. │ ├── plugin.json # Plugin manifest
  925. │ └── marketplace.json # Marketplace catalog for distribution
  926. ├── hooks/ # Native hook declarations
  927. │ └── hooks.json # Claude Code reads this automatically
  928. ├── skills/ # Agent skills
  929. │ └── token-saver-config/
  930. │ └── SKILL.md
  931. ├── commands/ # Slash commands
  932. │ └── token-saver-stats.md
  933. ├── scripts/ # Python hook scripts
  934. │ ├── __init__.py # Package init (prevents namespace conflicts)
  935. │ ├── hook_pretool.py # PreToolUse hook (Claude Code)
  936. │ ├── wrap.py # CLI wrapper (Claude Code)
  937. │ ├── hook_session.py # SessionStart hook wrapper
  938. │ └── audit_compression.py # Compression ratio corpus + report
  939. ├── antigravity/ # Antigravity CLI specific files
  940. │ ├── antigravity-plugin.json # Antigravity plugin metadata
  941. │ ├── hooks.json # Antigravity hook definitions
  942. │ └── hook_aftertool.py # AfterTool hook (Antigravity CLI)
  943. ├── bin/ # CLI executables
  944. │ ├── token-saver # Unix CLI wrapper
  945. │ └── token-saver.cmd # Windows CLI wrapper
  946. ├── src/ # Shared source code
  947. │ ├── __init__.py # Version (__version__) + data_dir()
  948. │ ├── chain_utils.py # Chained command splitting (&&, ;)
  949. │ ├── cli.py # CLI entry point (version/stats/benchmark/explain)
  950. │ ├── config.py # Configuration system + project trust boundary
  951. │ ├── console.py # UTF-8 stdio forcing
  952. │ ├── engine.py # Compression engine (orchestrator)
  953. │ ├── diagnostics.py # Immutable processor diagnostic contract
  954. │ ├── delta.py # Compare and render completed runs
  955. │ ├── delta_redaction.py # Recognized secret masking for snapshots
  956. │ ├── delta_store.py # Private, bounded snapshot persistence
  957. │ ├── delta_cli.py # Retrieve details and clear snapshots
  958. │ ├── hook_session.py # SessionStart hook (stats + update notif)
  959. │ ├── platforms.py # Platform detection + I/O abstraction
  960. │ ├── shell_syntax.py # Quote-aware shell scanning (exclusions)
  961. │ ├── stats.py # Stats display
  962. │ ├── tracker.py # SQLite tracking
  963. │ ├── version_check.py # GitHub update check
  964. │ └── processors/ # 36 auto-discovered processors
  965. │ ├── __init__.py
  966. │ ├── base.py # Abstract Processor class
  967. │ ├── critical.py # Shared "is this line an error?" predicate
  968. │ ├── utils.py # Shared utilities (diff compression)
  969. │ ├── package_list.py # pip list/freeze, npm ls, conda list
  970. │ ├── git.py # git status/diff/log/show/blame/push/pull
  971. │ ├── test_output.py # pytest/jest/cargo/go/dotnet/swift/mix test
  972. │ ├── build_output.py # npm/cargo/make/webpack/tsc/turbo/nx/docker build
  973. │ ├── lint_output.py # eslint/ruff/pylint/clippy/mypy/shellcheck/hadolint
  974. │ ├── network.py # curl/wget/httpie
  975. │ ├── docker.py # docker ps/images/logs/inspect/stats/compose
  976. │ ├── kubectl.py # kubectl get/describe/logs/apply/delete/create
  977. │ ├── terraform.py # terraform plan/apply/init/output/state
  978. │ ├── env.py # env/printenv (with secret redaction)
  979. │ ├── search.py # grep/rg/ag/fd/fdfind
  980. │ ├── system_info.py # du/wc/df
  981. │ ├── gh.py # gh pr/issue/run list/view/diff/checks
  982. │ ├── db_query.py # psql/mysql/sqlite3/pgcli/mycli/litecli
  983. │ ├── cloud_cli.py # aws/gcloud/az
  984. │ ├── ansible.py # ansible-playbook/ansible
  985. │ ├── helm.py # helm install/upgrade/list/template/status
  986. │ ├── syslog.py # journalctl/dmesg
  987. │ ├── file_listing.py # ls/find/tree/exa/eza/rsync
  988. │ ├── file_content.py # cat/bat (content-aware compression)
  989. │ └── generic.py # Universal fallback
  990. ├── docs/
  991. │ ├── comparison.md # Token-Saver vs. adjacent tools
  992. │ └── processors/ # Per-processor documentation
  993. ├── examples/
  994. │ └── custom_processor/ # Worked example of a user processor
  995. ├── installers/ # Modular installer package
  996. │ ├── common.py # Shared constants + utilities
  997. │ ├── claude.py # Claude Code installer (native plugin registration)
  998. │ └── antigravity.py # Antigravity CLI installer
  999. ├── install.py # Installer entry point
  1000. ├── CLAUDE.md # Plugin instructions
  1001. ├── tests/
  1002. │ ├── test_engine.py # Engine + registry tests
  1003. │ ├── test_processors.py # Per-processor tests (the largest file by far)
  1004. │ ├── test_hooks.py # Hook pattern + integration tests
  1005. │ ├── test_precision.py # Precision preservation tests
  1006. │ ├── test_pattern_consistency.py # hook_patterns vs. can_handle() agreement, per processor
  1007. │ ├── test_core.py # Shared compression core tests
  1008. │ ├── test_tracker.py # SQLite + concurrency tests
  1009. │ ├── test_config.py # Configuration tests
  1010. │ ├── test_console.py # UTF-8 stdio forcing tests
  1011. │ ├── test_readme_config_sync.py # README parameter table vs. _DEFAULTS drift guard
  1012. │ ├── test_version_check.py # Version check + fail-open tests
  1013. │ ├── test_version_consistency.py # Manifest/marketplace version drift guard
  1014. │ ├── test_cli.py # CLI subcommand tests
  1015. │ ├── test_user_processors.py # Custom processor loading tests
  1016. │ ├── test_installers.py # Installer utility tests
  1017. │ ├── test_install_smoke.py # End-to-end installer smoke tests
  1018. │ ├── failure_fixtures.py # One failing-command fixture per processor
  1019. │ ├── compression_baselines.json # Recorded ratios guarded by the ratchet test
  1020. │ └── test_compression_ratchet.py # Fails if any scenario compresses worse
  1021. ├── pyproject.toml # Python project config + Ruff rules
  1022. ├── CONTRIBUTING.md # Developer guide
  1023. ├── LICENSE # Apache 2.0
  1024. └── README.md
  1025. ```
  1026. ## Tests
  1027. ```bash
  1028. python3 -m pytest tests/ -v
  1029. ```
  1030. 1,300+ tests (run `python3 -m pytest tests/ --collect-only -q | tail -1` for the exact, current count) covering:
  1031. - **test_engine.py**: compression thresholds, processor priority, ANSI cleanup, generic fallback, hook pattern coverage across all supported commands
  1032. - **test_processors.py**: each processor with nominal and edge cases, chained command routing, all subcommands (blame, inspect, stats, compose, apply/delete, init/output/state, fd, exa, httpie, dotnet/swift/mix test, shellcheck/hadolint/biome, traceback truncation, ansible, helm, syslog, parameterized tests, coverage, docker compose logs, tsc typecheck, .env redaction, minified files, search directory grouping, git lockfiles/stat grouping)
  1033. - **test_hooks.py**: matching patterns for all supported commands, exclusions (pipes, sudo, editors, redirections, remote rsync, backgrounded `&`, unquoted newlines, shell-construct fragments), subprocess integration, global options (git, docker, kubectl), chained commands (shared shell state, `&&` short-circuit, `;` continue, unterminated-segment markers), safe trailing pipes
  1034. - **test_precision.py**: verification that every critical piece of data survives compression (filenames, hashes, error messages, stack traces, line numbers, rule IDs, diff changes, warning types, secret redaction under every fallback path, unhealthy pods, terraform changes, unmet dependencies) — includes `TestFailureHandling`, which runs a realistic failing-command fixture through every processor and asserts the failure reason is never lost, for every processor, at every exit code
  1035. - **test_compression_ratchet.py**: locks each real-world scenario's compression ratio, processor routing, and was-compressed status to a recorded baseline — a silent regression fails CI, not just a code review
  1036. - **test_pattern_consistency.py**: every processor's `hook_patterns` (used by the PreToolUse hook, before any output exists) must agree with its `can_handle()` (used by the engine, after output exists) — otherwise a command gets wrapped but never actually routed
  1037. - **test_readme_config_sync.py**: parses this README's parameter table and cross-checks every row against `src.config._DEFAULTS` — documentation drift fails CI instead of shipping
  1038. - **test_core.py**: shared compression core (decision, pass-through-on-error, audit logging) and the platform hook end-to-end
  1039. - **test_tracker.py**: CRUD, concurrency, corruption recovery, session tracking, stats CLI
  1040. - **test_config.py**: defaults, env overrides, invalid values, project-config trust boundaries
  1041. - **test_version_check.py**: version parsing, comparison, fail-open on errors
  1042. - **test_cli.py**: version/stats/help/explain subcommands, bin script execution
  1043. - **test_user_processors.py**: custom processor discovery and loading from `~/.token-saver/processors/`
  1044. - **test_installers.py** / **test_install_smoke.py**: version stamping, legacy migration, CLI install/uninstall, end-to-end install smoke tests
  1045. Lint and type checks match CI:
  1046. ```bash
  1047. ruff check .
  1048. ruff format --check .
  1049. mypy src/ scripts/ --ignore-missing-imports
  1050. ```
  1051. ## Debugging
  1052. To diagnose issues:
  1053. ```bash
  1054. # Test compression on a command without replacing the output
  1055. python3 scripts/wrap.py --dry-run 'git status'
  1056. # Explain routing / exclusion decisions
  1057. token-saver explain 'git status'
  1058. # See exactly what compression removed
  1059. token-saver benchmark 'git diff' --show-removed
  1060. # Enable debug logging
  1061. export TOKEN_SAVER_DEBUG=true
  1062. # Check stats
  1063. token-saver stats
  1064. # Check version
  1065. token-saver version
  1066. ```
  1067. ## Contributing
  1068. Contributions are welcome — especially new processors for tools that aren't covered yet.
  1069. Read [CONTRIBUTING.md](CONTRIBUTING.md) for the development workflow, then:
  1070. 1. Add your processor to `src/processors/`
  1071. 2. Add tests to `tests/test_processors.py` and a failing fixture to `tests/failure_fixtures.py`
  1072. 3. Add a scenario to `scripts/audit_compression.py` so the ratchet guards your ratio
  1073. 4. Document it in `docs/processors/` and add a row to the [Processors](#processors) table
  1074. 5. Run `ruff check . && mypy src/ scripts/ --ignore-missing-imports && python3 -m pytest tests/`
  1075. ## License
  1076. Apache 2.0 — see [LICENSE](LICENSE).
src/command_policy.py · Bundled script · 547 lines · 1 finding marked
  1. # Licensed under the Apache License, Version 2.0 (the "License");
  2. # you may not use this file except in compliance with the License.
  3. # You may obtain a copy of the License at
  4. #
  5. # https://www.apache.org/licenses/LICENSE-2.0
  6. #
  7. # Unless required by applicable law or agreed to in writing, software
  8. # distributed under the License is distributed on an "AS IS" BASIS,
  9. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  10. # See the License for the specific language governing permissions and
  11. # limitations under the License.
  12. """Shared shell eligibility policy independent of host hook protocols.
  13. The evaluator only inspects command text. Processor discovery is deferred until
  14. an application asks for the default policy; callers can inject an explicit
  15. pattern inventory without loading plugins, executing commands or touching the
  16. filesystem. Shell exclusions apply identically to interception and explanation.
  17. """
  18. import functools
  19. import logging
  20. import re
  21. import shlex
  22. from src import chain_utils
  23. from src import shell_syntax
  24. _LOG = logging.getLogger(__name__)
  25. def compile_patterns(
  26. patterns: list[str],
  27. ) -> tuple[list[str], list[re.Pattern]]:
  28. """Compile patterns one by one, dropping (and logging) any invalid one.
  29. Args:
  30. patterns: Regex sources advertised by the processor registry.
  31. Returns:
  32. Parallel lists of valid sources and compiled patterns in input order.
  33. Invalid expressions emit only a content-free warning and are omitted.
  34. """
  35. sources: list[str] = []
  36. compiled: list[re.Pattern] = []
  37. for p in patterns:
  38. try:
  39. compiled.append(re.compile(p))
  40. except re.error: # noqa: PERF203 — isolate each plugin pattern
  41. _LOG.warning("Skipping an invalid hook pattern")
  42. else:
  43. sources.append(p)
  44. return sources, compiled
  45. # Trailing pipe suffixes that are safe to wrap.
  46. # These are stripped before checking exclusions so commands like
  47. # `git log | head -30` or `pip list | grep torch` are still compressed.
  48. # The full original command (with pipe) is passed to wrap.py unchanged.
  49. #
  50. # Allowed trailing pipes (single stage only):
  51. # | head [-N] — truncate output
  52. # | tail [-N] [+N] — truncate output
  53. # | wc [-l] [-w] [-c] — count lines/words/chars
  54. # | grep [-viEc] "pattern" — filter lines (single grep, no chaining)
  55. # | sort [-rnk] [N] — reorder lines
  56. # | uniq [-c] — deduplicate lines
  57. # | cut -fN [-dX] — extract columns
  58. _SAFE_TRAILING_PIPE_RE = re.compile(
  59. r"\s*\|\s*("
  60. r"head(\s+-[n]?\s*\d+|\s+-\d+)*" # | head -30, | head -n 50
  61. r"|tail(\s+[-+]?\d+|\s+-[nf]\s*\d+)*" # | tail -20, | tail -n 50, | tail +5
  62. r"|wc(\s+-[lwc])*" # | wc -l, | wc -w
  63. r"|grep(\s+-[viEcwnHr])*\s+\S+" # | grep -i pattern, | grep -v noise
  64. r"|sort(\s+-[rnktu](\s+\d+)?)*" # | sort -r, | sort -k 2
  65. r"|uniq(\s+-[cd])*" # | uniq -c
  66. r"|cut(\s+-[fd]\s*\S+)+" # | cut -f1 -d,
  67. r")\s*$"
  68. )
  69. # Streaming / follow / watch commands that never terminate on their own.
  70. # Wrapping them would buffer forever (the wrap.py subprocess only flushes
  71. # after the child exits), so they must pass through untouched. Shared by
  72. # both the whole-command and per-segment exclusion lists.
  73. _STREAMING_EXCLUDED_PATTERNS = [
  74. r"^\s*watch\b", # `watch` repeats a command forever
  75. r"\s--follow(=|\s|$)", # --follow on tail/journalctl/kubectl/docker logs
  76. r"^\s*tail\b[^|]*\s-[a-zA-Z]*[fF]\b", # tail -f / tail -F / tail -nf
  77. r"^\s*journalctl\b[^|]*\s-[a-zA-Z]*f\b", # journalctl -f
  78. r"\b(kubectl|oc)\b[^|]*\blogs\b[^|]*\s-[a-zA-Z]*f\b", # kubectl/oc logs -f
  79. r"\bdocker\b[^|]*\blogs\b[^|]*\s-[a-zA-Z]*f\b", # docker logs -f
  80. # Docker stats streams by default.
  81. r"^\s*docker\s+stats\b(?![^|]*--no-stream)",
  82. (
  83. "^\\s*docker(\\s+compose|-compose)\\s+up\\b(?![^|]*\\s-d\\b)"
  84. ), # compose up (attached)
  85. r"\s--watch(=|\s|$)", # generic --watch flag (vitest, jest, tsc, etc.)
  86. r"\s--watchAll\b", # jest --watchAll
  87. r"^\s*(npx\s+)?vitest\b(?![^|]*\brun\b)", # vitest defaults to watch mode
  88. ]
  89. # Commands that should NEVER be wrapped (checked on whole command for
  90. # single-command inputs, or delegated to per-segment checks for chains).
  91. EXCLUDED_PATTERNS = [
  92. r"(?<!['\"])\|(?!['\"])", # unquoted pipe (complex pipelines)
  93. r"^\s*(vi|vim|nano|emacs|code)\b",
  94. r"^\s*ssh\s+(?:-\S+\s+)*\S+\s*$", # interactive ssh only
  95. r"^\s*rsync\b.*\S+:\S+", # only exclude remote rsync (host:path)
  96. r"(?:^|\s)token[-_]saver\s", # avoid wrapping token-saver CLI itself
  97. # Recursion guard: our rewrite is `python3 <…>wrap.py '<cmd>'`. Match
  98. # wrap.py only in script-execution position so `cat wrap.py` stays
  99. # wrappable.
  100. r"(?:^|\s)(?:\S*/)?(?:python\d?(?:\.\d+)*|node|ruby|sh|bash|zsh|perl)\s+"
  101. r"(?:-\S+\s+)*\S*wrap\.py\b",
  102. r"^\s*\.?\S*/?wrap\.py\b",
  103. r"<\(", # process substitution
  104. r"^\s*sudo\b", # never wrap sudo
  105. r"^\s*env\s+\S+=", # env VAR=val prefix — too complex to wrap
  106. # Interactive-flag REPLs even with script args (e.g. `python -i script.py`).
  107. (
  108. "^\\s*(python\\d?(?:\\.\\d+)*|ipython|node|ruby|perl|"
  109. "ghci|deno|php|lua|R|bash|sh|zsh)\\s+(?:-\\S*i\\S*|-"
  110. "-interactive)(\\s|$)"
  111. ),
  112. *_STREAMING_EXCLUDED_PATTERNS,
  113. ]
  114. # re.MULTILINE so `^`/`$`-anchored patterns (sudo, vim, ssh, …) match at the
  115. # start/end of *every* line, not just the start/end of the whole string.
  116. # Defense in depth: has_unquoted_newline() is what actually stops a
  117. # multi-line command from reaching this point at all, but a multi-line
  118. # string that slips past it for any other reason (a future gap in the
  119. # quote-aware scanner, a new caller that skips that check) must still have
  120. # every line's dangerous prefix checked here, not just the first.
  121. COMPILED_EXCLUDED = [re.compile(p, re.MULTILINE) for p in EXCLUDED_PATTERNS]
  122. # Strip leading path prefix so '/usr/bin/git status' → 'git status',
  123. # './node_modules/.bin/jest' → 'jest', '.venv/bin/pip' → 'pip', etc.
  124. # Greedy match: captures everything up to and including the last '/'
  125. # in the first token (before any space).
  126. _PATH_PREFIX_RE = re.compile(r"^(\S*/)(?=\S)")
  127. def _normalize_cmd(cmd: str) -> str:
  128. """Normalize leading paths and Python launchers for matching exclusions.
  129. Decode Python executable quoting so Windows paths with spaces use the same
  130. recursion and interactive-mode guards as bare POSIX Python names. Keep the
  131. argument suffix unchanged: unquoting it would turn literal text into shell
  132. operators. Other executable families retain their existing path handling.
  133. """
  134. stripped = cmd.lstrip()
  135. end = next(
  136. (
  137. index
  138. for index, char in shell_syntax.iter_unquoted(stripped)
  139. if char.isspace()
  140. ),
  141. len(stripped),
  142. )
  143. token = stripped[:end]
  144. if "$" in token or "`" in token:
  145. return cmd
  146. try:
  147. words = shlex.split(token)
  148. except ValueError:
  149. return cmd
  150. if len(words) != 1:
  151. return cmd
  152. executable = words[0].replace("\\", "/").rsplit("/", 1)[-1]
  153. if re.fullmatch(r"python\d?(?:\.\d+)*(?:\.exe)?", executable):
  154. return executable.removesuffix(".exe") + stripped[end:]
  155. return _PATH_PREFIX_RE.sub("", cmd)
  156. # Constructs that break naive chain splitting / per-segment execution.
  157. _DANGEROUS_CONSTRUCTS = ("$(", "`", "<<")
  158. # Both checks now share the single quote-aware scanner in shell_syntax,
  159. # alongside the chain splitters. Re-exported under the historical names so
  160. # the rest of this module (and its tests) read unchanged.
  161. # Preserve the historical callable aliases used by external hook integrations.
  162. # pylint: disable=invalid-name
  163. _has_unquoted_construct = shell_syntax.has_unquoted
  164. _has_output_redirection = shell_syntax.has_output_redirection
  165. # pylint: enable=invalid-name
  166. # Per-segment safety checks applied inside _is_chain_compressible().
  167. # These catch dangerous constructs within individual chain segments.
  168. _SEGMENT_EXCLUDED_PATTERNS = [
  169. r"(?<!['\"])\|(?!['\"])", # pipes inside a segment
  170. r"<\(", # process substitution
  171. r"^\s*sudo\b",
  172. r"^\s*(vi|vim|nano|emacs|code)\b",
  173. r"^\s*ssh\s+(?:-\S+\s+)*\S+\s*$", # interactive ssh only
  174. r"^\s*rsync\b.*\S+:\S+", # only exclude remote rsync (host:path)
  175. r"^\s*env\s+\S+=",
  176. r"(?:^|\s)token[-_]saver\s",
  177. r"(?:^|\s)(?:\S*/)?(?:python\d?(?:\.\d+)*|node|ruby|sh|bash|zsh|perl)\s+"
  178. r"(?:-\S+\s+)*\S*wrap\.py\b",
  179. r"^\s*\.?\S*/?wrap\.py\b",
  180. # Bare interactive REPL launchers: would hang waiting for stdin.
  181. # Only matches when there are no arguments (REPL mode).
  182. r"^\s*(python\d?(?:\.\d+)*|ipython|node|bash|sh|zsh|ruby|irb|pry|gdb|lldb"
  183. r"|mongo|mongosh|redis-cli|psql|mysql|sqlite3|php|perl|lua|R)\s*$",
  184. # Interactive-flag REPLs: -i drops into REPL even with other args.
  185. (
  186. "^\\s*(python\\d?(?:\\.\\d+)*|ipython|node|ruby|perl|"
  187. "ghci|deno|php|lua|R|bash|sh|zsh)\\s+(?:-\\S*i\\S*|-"
  188. "-interactive)(\\s|$)"
  189. ),
  190. *_STREAMING_EXCLUDED_PATTERNS,
  191. ]
  192. # Same defense-in-depth rationale as COMPILED_EXCLUDED above.
  193. _COMPILED_SEGMENT_EXCLUDED = [
  194. re.compile(p, re.MULTILINE) for p in _SEGMENT_EXCLUDED_PATTERNS
  195. ]
  196. # Commands whose *output* is perfectly compressible, but which must never be
  197. # auto-approved on the user's behalf. Wrapping a command means emitting
  198. # `permissionDecision: "allow"`, which per Claude Code's hook semantics runs
  199. # it with no confirmation prompt and without consulting the user's configured
  200. # allow/ask/deny rules. That trade is fine for `git status`; it is not fine
  201. # for a command that force-pushes, tears down infrastructure, or deletes data
  202. # — the human-in-the-loop check is the whole point there, and saving a few
  203. # hundred tokens does not justify removing it. Matching commands are left
  204. # entirely alone (no rewrite, no compression) so Claude Code applies its
  205. # normal permission flow.
  206. #
  207. # Deliberately narrow: each pattern targets the destructive *flag or
  208. # subcommand*, not the tool, so `git push` / `kubectl get` / `terraform plan`
  209. # stay compressible. This list is not exhaustive and is not a security
  210. # boundary — it is a "don't silently bypass the user's own guard rails on the
  211. # obviously irreversible cases" list.
  212. _DESTRUCTIVE_PATTERNS = [
  213. # force-push, but not the safer --force-with-lease
  214. r"\bgit\s+push\b[^\n]*\s(?:--force\b(?!-with-lease)|-f\b)",
  215. r"\bgit\s+reset\b[^\n]*\s--hard\b",
  216. r"\bgit\s+clean\b[^\n]*\s-[a-zA-Z]*f",
  217. r"\b(?:kubectl|oc)\b[^\n]*\bdelete\b",
  218. r"\b(?:terraform|tofu)\s+(?:apply|destroy)\b",
  219. r"\bdocker\b[^\n]*\b(?:system\s+prune|volume\s+rm|rmi)\b",
  220. r"\b(?:docker|podman)\s+rm\b",
  221. (
  222. "\\brm\\s+[^\\n]*-[a-zA-Z]*[rR][a-zA-Z]*f|\\brm\\s+[^\\"
  223. "n]*-[a-zA-Z]*f[a-zA-Z]*[rR]"
  224. ), # rm -rf / -fr
    • Note: Quoted as data in code: Runs `rm -rf` against a root or home-directory path
    • Line 250 on GitHub
  225. ]
  226. _COMPILED_DESTRUCTIVE = [
  227. re.compile(p, re.MULTILINE) for p in _DESTRUCTIVE_PATTERNS
  228. ]
  229. def is_destructive(command: str) -> bool:
  230. """Identify commands whose auto-approval would bypass permission rules.
  231. Checked separately from :func:`is_compressible`: such a command may well
  232. be compressible, but must not be rewritten, because rewriting is what
  233. emits ``permissionDecision: "allow"``.
  234. """
  235. norm = _normalize_cmd(command)
  236. return any(
  237. p.search(command) or p.search(norm) for p in _COMPILED_DESTRUCTIVE
  238. )
  239. def _ends_with_line_continuation(segment: str) -> bool:
  240. """Return True if the segment ends with an unescaped backslash.
  241. ``wrap.py`` closes each chain segment's brace group with a newline; a
  242. trailing backslash would splice that newline away and swallow the closing
  243. brace. An even-length run of backslashes is an escaped backslash, not a
  244. continuation.
  245. """
  246. stripped = segment.rstrip()
  247. trailing = len(stripped) - len(stripped.rstrip("\\"))
  248. return trailing % 2 == 1
  249. def _is_comment_only(segment: str) -> bool:
  250. r"""Return True if the segment is nothing but a shell comment.
  251. Such a segment would produce an empty brace group (``{ # foo\\n}``), which
  252. is a syntax error. Today it also silently comments out the rest of the
  253. rewritten line, so the command never runs — either way, don't wrap it.
  254. """
  255. return segment.lstrip().startswith("#")
  256. def _is_segment_safe(segment: str) -> bool:
  257. """Return True if a single chain segment has no dangerous constructs.
  258. Checks both the raw segment and its path-stripped form, so commands like
  259. ``/usr/bin/vim``, ``./python``, or ``.venv/bin/sudo`` are still caught.
  260. """
  261. if _has_output_redirection(segment):
  262. return False
  263. # A backgrounded segment (`npm run dev &`) detaches from wrap.py's
  264. # subprocess entirely — nothing to compress, and the timeout can never
  265. # reclaim a process it no longer has a handle on.
  266. if shell_syntax.has_unquoted_background_operator(segment):
  267. return False
  268. # An opening fragment of a for/while/if/case/... compound statement is
  269. # not a real, independent segment: the chain splitter only saw one of
  270. # its internal `;`s. Wrapping it in its own brace group is a shell
  271. # syntax error, so the whole command — every segment, not just this
  272. # one — must be declined. See has_shell_construct_open's docstring.
  273. if shell_syntax.has_shell_construct_open(segment):
  274. return False
  275. # Both break wrap.py's brace-group rewrite — see the helpers above.
  276. if _ends_with_line_continuation(segment) or _is_comment_only(segment):
  277. return False
  278. norm = _normalize_cmd(segment)
  279. for pattern in _COMPILED_SEGMENT_EXCLUDED:
  280. if pattern.search(segment) or pattern.search(norm):
  281. return False
  282. return True
  283. def _is_chain_compressible(
  284. command: str, patterns: tuple[tuple[str, re.Pattern], ...]
  285. ) -> bool:
  286. """Check whether a chained command (&&/;) is compressible.
  287. Every segment must pass the segment safety check (no sudo, redirects,
  288. bare REPLs, etc.). At least one segment must be compressible; the
  289. others may be silent or unknown (their output passes through unchanged
  290. in wrap.py's per-segment compression). Safe trailing pipes are only
  291. stripped from the *last* segment.
  292. """
  293. segments = chain_utils.split_chain(command)
  294. if not segments:
  295. return False
  296. has_compressible = False
  297. for i, seg in enumerate(segments):
  298. # Only strip safe trailing pipe from the last segment
  299. check_seg = (
  300. _SAFE_TRAILING_PIPE_RE.sub("", seg)
  301. if i == len(segments) - 1
  302. else seg
  303. )
  304. if not _is_segment_safe(check_seg):
  305. return False
  306. norm_seg = _normalize_cmd(check_seg)
  307. is_comp = any(
  308. pattern.search(check_seg) for _, pattern in patterns
  309. ) or any(pattern.search(norm_seg) for _, pattern in patterns)
  310. if is_comp:
  311. has_compressible = True
  312. return has_compressible
  313. def _matched_exclusion(check_cmd: str, norm_cmd: str) -> str | None:
  314. """Return the source regex of the first exclusion that matches, if any."""
  315. for source, pattern in zip(
  316. EXCLUDED_PATTERNS, COMPILED_EXCLUDED, strict=False
  317. ):
  318. if pattern.search(check_cmd) or pattern.search(norm_cmd):
  319. return source
  320. return None
  321. def _matched_compressible(
  322. check_cmd: str,
  323. norm_cmd: str,
  324. patterns: tuple[tuple[str, re.Pattern], ...],
  325. ) -> list[str]:
  326. """Return source regexes of all compressible patterns that match."""
  327. matched = []
  328. for source, pattern in patterns:
  329. if pattern.search(check_cmd) or pattern.search(norm_cmd):
  330. matched.append(source)
  331. return matched
  332. def _explain_decision(
  333. command: str, patterns: tuple[tuple[str, re.Pattern], ...]
  334. ) -> dict:
  335. """Explain whether ``command`` would be wrapped, and why.
  336. This is the sole evaluator used for interception and explanation.
  337. The supplied pattern pairs belong to one explicit processor inventory.
  338. Args:
  339. command: Original shell text inspected without execution.
  340. patterns: Accepted regex sources paired with compiled expressions.
  341. Returns:
  342. The command, compressible, reason, excluded_by, matched_patterns and
  343. is_chain record used by the public explanation API.
  344. """
  345. result = {
  346. "command": command,
  347. "compressible": False,
  348. "reason": "",
  349. "excluded_by": None,
  350. "matched_patterns": [],
  351. "is_chain": False,
  352. }
  353. cmd = command.strip()
  354. if not cmd:
  355. result["reason"] = "empty command"
  356. return result
  357. if re.search(r"(?<!['\"])\|\|(?!['\"])", cmd):
  358. result["reason"] = (
  359. "contains '||' (error-recovery chains are not wrapped)"
  360. )
  361. result["excluded_by"] = r"||"
  362. return result
  363. if shell_syntax.has_unquoted_newline(cmd):
  364. result["reason"] = "contains an unquoted newline (multiple statements)"
  365. result["excluded_by"] = "unquoted newline"
  366. return result
  367. if _has_unquoted_construct(cmd, _DANGEROUS_CONSTRUCTS):
  368. result["reason"] = "contains unquoted $(), backtick, or heredoc"
  369. result["excluded_by"] = "dangerous shell construct"
  370. return result
  371. if chain_utils.CHAIN_SPLIT_RE.search(cmd):
  372. result["is_chain"] = True
  373. compressible = _is_chain_compressible(cmd, patterns)
  374. result["compressible"] = compressible
  375. seen: list[str] = []
  376. for seg in chain_utils.split_chain(cmd):
  377. check_seg = _SAFE_TRAILING_PIPE_RE.sub("", seg)
  378. norm_seg = _normalize_cmd(check_seg)
  379. for p in _matched_compressible(check_seg, norm_seg, patterns):
  380. if p not in seen:
  381. seen.append(p)
  382. result["matched_patterns"] = seen
  383. result["reason"] = (
  384. "chain with at least one compressible, all-safe segment"
  385. if compressible
  386. else "chain has an unsafe segment or no compressible segment"
  387. )
  388. return result
  389. if _has_output_redirection(cmd):
  390. result["reason"] = "contains output redirection (>, >>, 2>, &>)"
  391. result["excluded_by"] = "output redirection"
  392. return result
  393. if shell_syntax.has_unquoted_background_operator(cmd):
  394. result["reason"] = "contains an unquoted '&' (backgrounded command)"
  395. result["excluded_by"] = "background operator"
  396. return result
  397. check_cmd = _SAFE_TRAILING_PIPE_RE.sub("", cmd)
  398. norm_cmd = _normalize_cmd(check_cmd)
  399. excl = _matched_exclusion(check_cmd, norm_cmd)
  400. if excl is not None:
  401. result["reason"] = "matched an exclusion pattern"
  402. result["excluded_by"] = excl
  403. return result
  404. matched = _matched_compressible(check_cmd, norm_cmd, patterns)
  405. result["matched_patterns"] = matched
  406. result["compressible"] = bool(matched)
  407. result["reason"] = (
  408. "matched a compressible processor pattern"
  409. if matched
  410. else "no processor pattern matched (not wrapped)"
  411. )
  412. return result
  413. class CommandPolicy:
  414. """Evaluate shell safety against one explicit processor-pattern inventory.
  415. Invalid extension regexes are skipped with a content-free warning. The
  416. policy does not discover or import processors itself and does not execute
  417. shell text. Construct a new instance when the processor inventory changes.
  418. Attributes:
  419. pattern_sources: Accepted regex sources in discovery order.
  420. compiled_patterns: Compiled expressions corresponding to those sources.
  421. """
  422. def __init__(self, patterns: list[str]) -> None:
  423. """Compile the supplied patterns without loading processors.
  424. Args:
  425. patterns: Anchored command patterns from an explicit registry.
  426. """
  427. self.pattern_sources, self.compiled_patterns = compile_patterns(
  428. patterns
  429. )
  430. def explain_decision(self, command: str) -> dict:
  431. """Describe the same eligibility decision used by interception.
  432. Args:
  433. command: Original shell text, which is never executed.
  434. Returns:
  435. The historical command, compressible, reason, excluded_by,
  436. matched_patterns and is_chain record used by the explain CLI.
  437. """
  438. patterns = tuple(
  439. zip(self.pattern_sources, self.compiled_patterns, strict=False)
  440. )
  441. return _explain_decision(command, patterns)
  442. def is_compressible(self, command: str) -> bool:
  443. """Return eligibility using the single structured evaluator."""
  444. return bool(self.explain_decision(command)["compressible"])
  445. @functools.lru_cache(maxsize=1)
  446. def default_policy() -> CommandPolicy:
  447. """Lazily discover a process-local policy, failing open on plugin errors.
  448. Returns:
  449. Policy for the discovered processors, or an empty inventory when an
  450. optional registry fails. Host protocol adapters are never imported.
  451. """
  452. try:
  453. # Keep plugin discovery outside import time and inside this boundary.
  454. # pylint: disable-next=import-outside-toplevel
  455. from src import processors # noqa: PLC0415
  456. patterns = processors.collect_hook_patterns()
  457. except Exception: # pylint: disable=broad-exception-caught
  458. # Optional user processors may fail arbitrarily; no captured content
  459. # or exception text is safe to include in this isolation diagnostic.
  460. _LOG.warning("Failed to load command patterns — compression disabled")
  461. patterns = []
  462. return CommandPolicy(patterns)
  463. def is_compressible(command: str) -> bool:
  464. """Return whether the default runtime policy accepts shell command text."""
  465. return default_policy().is_compressible(command)
  466. def explain_decision(command: str) -> dict:
  467. """Explain default routing without loading a host protocol adapter."""
  468. return default_policy().explain_decision(command)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.