Skip to content
Back to search

skillranker

Rust CLI powered by Jev from TypeSafe.ai that ranks agent skills for the next step using live session context. Includes Claude Code hooks, structured JSON, abstention, and local feedback. Requires a TypeSafe API key.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/dicklesworthstone--skillranker/badge.svg)](/listings/dicklesworthstone--skillranker)

HTML

<a href="/listings/dicklesworthstone--skillranker"><img src="/api/listings/dicklesworthstone--skillranker/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

Install on **Linux or macOS**. Keep macOS state on a local APFS or HFS+ filesystem with Unix permissions. Run `sr capabilities --json` after installation to check command and integration availability in your build.

Installer

The installer selects a release for your platform, verifies its SHA256 checksum, and installs `sr` into `~/.local/bin`. When a release asset is unavailable, it builds from the selected source revision using the pinned Rust toolchain:

curl -fsSL "https://raw.githubusercontent.com/Dicklesworthstone/skillranker/main/install.sh?$(date +%s)" | bash -s -- --verify

It requires Bash, Python 3, `curl`, and `sha256sum` or `shasum`; source builds also require Git and Rust. If RCH is installed, compilation runs remotely and a remote failure does not trigger a local build. macOS source builds need a macOS RCH worker. Existing binaries are backed up before replacement. `--easy-mode` also adds the install directory to your current shell's configuration, with a backup.

# Install from a trusted local checkout.
bash install.sh --source . --verify

# Install an air-gapped archive with its adjacent .sha256 file.
bash install.sh --offline skillranker.tar.gz --verify

Detected Claude Code and Codex installations receive a small SkillRanker usage skill. Subcommand completions are installed for Bash, Zsh, and Fish. Existing customized integration files are preserved; `--no-configure` skips these steps. The installer leaves credentials and network consent to you and reports hooks as unconfigured. **Obtain your own credential for the selected provider before live ranking.** See [installer options, verification, and rollback](docs/installation.md).

From source

Build the `sr` binary with the repository's pinned Rust toolchain and lockfile:

git clone https://github.com/Dicklesworthstone/skillranker.git
cd skillranker
cargo install --locked --path . --bin sr

For a checkout-local binary:

cargo build --locked --release --bin sr
./target/release/sr --help

Runtime setup

**Choose a provider and create your own credential.** TypeSafe.ai is the default provider; Cloudflare-hosted TypeSafe Jev can be selected with `SR_PROVIDER=cloudflare`. Every user supplies their own credential; SkillRanker does not distribute a shared key.

For the default TypeSafe provider, set `TYPESAFE_API_KEY`. For Cloudflare, set `SR_PROVIDER=cloudflare`, `CLOUDFLARE_ACCOUNT_ID`, and `CLOUDFLARE_API_TOKEN`. The [environment example](.env.example) lists the service settings. For a local checkout, create `.env` from the example only if it does not already exist and restrict access with `chmod 600 .env` **before** entering your own API key. The `.env` file is ignored by Git. Export its values into the process environment before running `sr` or starting an agent whose hooks need the key:

# Run from your checkout, after filling in your own trusted .env file.
set +x
set -a
. ./.env
set +a

Treat `.env` as a local shell configuration file and source only content you trust. Keep the key out of shell history, logs, and tracked files. Credentials alone do not enable remote transmission.

| Component | Role | |---|---| | Provider credential | Authenticates fresh Jev-compatible evaluations (`TYPESAFE_API_KEY`, or Cloudflare account/token) | | Network opt-in | `--allow-network` for a run, or `network.enabled` in trusted user configuration | | Visible skill inventory | Harness-resolved skills or an explicit roster file | | Session input | Claude hook, normalized context, supported native transcript, or optional cass export | | [cass](https://github.com/Dicklesworthstone/coding_agent_session_search) | Optional archive discovery and access across coding-agent formats |

`sr` does not require `ms`, a local inference server, or an embedding model. The supported local platform is Linux. Consult `sr capabilities --json` for the adapters, events, and optional features in a build.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 2,144 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. # SkillRanker
  3. **The right skill for the next step, powered by Jev-compatible providers.**
  4. A standalone Rust CLI that puts **Jev-compatible providers** at the center of
  5. skill selection: Jev evaluates your agent's live context, compares the available
  6. skills, and estimates which ones fit the next step. SkillRanker supplies the
  7. session integration, local safeguards, and inspectable feedback around it.
  8. **Fresh ranking requires your own credential for the selected provider.**
  9. TypeSafe.ai is the default; Cloudflare-hosted TypeSafe Jev is also supported.
  10. [![License](https://img.shields.io/badge/license-MIT%20%2B%20OpenAI%2FAnthropic%20rider-blue)](LICENSE)
  11. ![Rust](https://img.shields.io/badge/language-Rust%202024-dea584)
  12. ![CLI](https://img.shields.io/badge/CLI-sr-222222)
  13. ![Powered by Jev](https://img.shields.io/badge/powered_by-TypeSafe.ai%20Jev-00897b)
  14. ![Runtime](https://img.shields.io/badge/runtime-Asupersync-654ff0)
  15. ![Output](https://img.shields.io/badge/output-JSON%20%7C%20hooks%20%7C%20TUI-00897b)
  16. ```bash
  17. sr capabilities --json # Check implemented commands and integrations in your build
  18. sr demo --case useful # Inspect an offline fixture before connecting a session
  19. sr rank --allow-network # Rank skills for the selected session
  20. ```
  21. </div>
  22. ## Contents
  23. - [TL;DR](#tldr)
  24. - [Quick Example](#quick-example)
  25. - [Design Philosophy](#design-philosophy)
  26. - [How It Compares](#how-it-compares)
  27. - [Installation](#installation)
  28. - [Quick Start](#quick-start)
  29. - [Command Reference](#command-reference)
  30. - [Configuration](#configuration)
  31. - [How Ranking Works](#how-ranking-works)
  32. - [Explain And Replay A Result](#explain-and-replay-a-result)
  33. - [Local Feedback And Calibration](#local-feedback-and-calibration)
  34. - [Evaluation, Sampling, And Risk Monitoring](#evaluation-sampling-and-risk-monitoring)
  35. - [Control Requests And Interruptions](#control-requests-and-interruptions)
  36. - [Agent Hooks](#agent-hooks)
  37. - [Inline TUI](#inline-tui)
  38. - [Architecture](#architecture)
  39. - [Privacy And Local State](#privacy-and-local-state)
  40. - [Performance](#performance)
  41. - [Troubleshooting](#troubleshooting)
  42. - [Limitations](#limitations)
  43. - [FAQ](#faq)
  44. - [About Contributions](#about-contributions)
  45. - [License](#license)
  46. ---
  47. ## TL;DR
  48. **The problem.** A large skill library gives an agent plenty of procedures to
  49. choose from, but choosing is itself a task. Similar descriptions obscure useful
  50. distinctions. A skill that helped at the start of a conversation can be irrelevant
  51. three turns later. Loading a plausible but unsuitable skill consumes context and
  52. can redirect otherwise sensible work.
  53. **The solution.** SkillRanker (`sr`) combines the recent conversation, current
  54. request, workspace signals, and the selected harness's visible skill inventory.
  55. **The selected provider's Jev implementation is the key enabler of the system.**
  56. It first compares the candidates broadly, then reads richer excerpts from a
  57. shortlist and evaluates whether each one fits. Both comparisons include a
  58. real “none of these” option. The result is advisory: the agent follows the user's
  59. instructions and decides what to consult.
  60. For libraries with more than 254 eligible skills, **Quill from FrankenSearch**
  61. narrows the candidates locally before Jev evaluates them. Smaller rosters reach
  62. Jev in full. Explicit skill requests resolve locally before either stage.
  63. SkillRanker does not include a local model. The ranking workflow requires your
  64. own account and credential for the selected provider. Local
  65. retrieval prepares the candidates; **Jev supplies the evaluations that make the
  66. recommendations possible**.
  67. ### Why `sr`?
  68. | Need | What SkillRanker provides |
  69. |---|---|
  70. | Evaluate meaning and task fit | The selected provider's typed Jev Choice and Noul evaluations power both ranking passes |
  71. | Choose for the current step | Exact session identity, the newest prompt, recent tool evidence, and project signals |
  72. | Suggest something the agent can load | Harness-aware visibility, override resolution, stable skill identities, and content revalidation |
  73. | Respect an explicit request | Locally resolve a requested skill before probabilistic retrieval or ranking |
  74. | Search a large library | Quill lexical prefiltering from FrankenSearch, admitting up to 254 skills plus a none option to each Choice |
  75. | Separate similar skills | Detailed reranking with bounded descriptions and body excerpts |
  76. | Recognize when no skill fits | Relevance gates, per-candidate fit checks, and sentinel-based abstention |
  77. | Understand a missing suggestion | `--why-not` traces where a candidate was excluded, with thresholds and concrete recovery hints |
  78. | Reproduce a surprising result | Opt-in case capture and offline replay compare compatible local policies without another Jev call |
  79. | Get started without sharing a session | Offline fixture demos and a readiness report identify the next setup step |
  80. | Keep the agent moving | A failed hook recommendation produces a quiet, non-blocking fallback |
  81. | Control interruptions | Silent ordinary abstentions and scoped, expiring snoozes preserve explicit skill requests |
  82. | Bound repeated expense | Optional shared HTTP-attempt allowances and a provider circuit breaker cover concurrent local sessions |
  83. | Review what happens | Usefulness, interruptions, attempts, and cost share a report with explicit label coverage |
  84. | Evaluate within a budget | Offline replay, explicit live-request caps, and reproducible samples with recorded selection probabilities |
  85. | Assess recommendation harm | Controlled comparisons, uncertainty bounds, and optional monitoring across repeated evaluations |
  86. | Control disclosure | Network opt-in, field-level disclosure receipts, a minimal context profile, and separate persistence controls |
  87. The approach builds on the [TypeSafe skill-suggestion recipe](https://docs.typesafe.ai/cookbooks/skill_suggestion).
  88. SkillRanker adds session identity, harness visibility, bounded execution, and a
  89. local evaluation loop. The [comprehensive plan](COMPREHENSIVE_PLAN_TO_DESIGN_SKILLRANKER.md)
  90. explains the full design and acceptance criteria.
  91. ## Quick Example
  92. ```bash
  93. # See a labeled fixture result without a key, network, or private session.
  94. sr demo --case useful
  95. # Inspect local configuration and the available adapters.
  96. sr doctor --json
  97. sr doctor --config
  98. sr capabilities --json
  99. # Inspect visible, shadowed, and excluded skill records.
  100. sr roster --json
  101. # Preview the redacted wide-pass request without network or persistence effects.
  102. sr rank --context scratch/context.json --dry-run
  103. # Evaluate an explicitly selected conversation.
  104. sr rank --context scratch/context.json --allow-network --json
  105. # Find where an expected candidate was excluded; this adds no inference calls.
  106. sr rank --context scratch/context.json --allow-network --why-not SKILL_ID --explain
  107. # Preview the Claude hook settings change, then apply it.
  108. sr install-hook claude
  109. sr install-hook claude --apply
  110. # Review observations without treating adoption as proof of usefulness.
  111. sr stats --since 7d --by-skill
  112. # Replay a labeled evaluation artifact without making network requests.
  113. sr eval --dataset scratch/evaluation.json --explain
  114. # Inspect description quality and suspected coverage gaps locally.
  115. sr doctor --descriptions
  116. sr gaps
  117. ```
  118. ## Design Philosophy
  119. 1. **Choose for the next action.** The current request matters, as do the recent
  120. failure, the task context, and the instructions already loaded.
  121. 2. **Resolve authority locally.** The user decides what is required or excluded.
  122. The harness determines what can be loaded. A model answer cannot change either.
  123. 3. **Separate preference from applicability.** Winning a comparison is not enough.
  124. A recommendation must survive fit, visibility, loaded-state, and none-option checks.
  125. 4. **Keep evidence inspectable.** Preserve provider estimates and local arithmetic.
  126. `--explain` exposes computations without inventing model-generated reasons.
  127. 5. **Separate adoption from usefulness.** Observing a load is useful telemetry.
  128. Learning a better policy requires independently judged examples and a holdout.
  129. 6. **Bound each evaluation.** Input, discovery, subprocesses, networking,
  130. retries, persistence, and cleanup consume one ranking deadline. Batch
  131. evaluation also has explicit request and total-runtime limits.
  132. 7. **Stay standalone.** Discovery, parsing, redaction, retrieval, and feedback live
  133. in `sr`. No skill-manager service or private database is required.
  134. ## How It Compares
  135. These are workflow choices, not benchmark rankings.
  136. | Approach | Input to selection | Strength | Tradeoff |
  137. |---|---|---|---|
  138. | Manual selection | Your knowledge of the task and library | Direct control without a ranking service | Requires remembering each skill's coverage |
  139. | Keyword search | A query over names and descriptions | Cheap local discovery | Synonyms and adjacent procedures can be hard to distinguish |
  140. | Load every skill | The full library's instructions | Makes every procedure available immediately | Consumes context regardless of relevance |
  141. | SkillRanker | Exact session, visible roster, and explicit constraints | Evaluates candidates and can abstain | Fresh Jev evaluations require authorized network access |
  142. SkillRanker recommends procedures. It does not execute skills, grant permissions,
  143. or override the agent's governing instructions.
  144. ## Installation
  145. Install on **Linux or macOS**. Keep macOS state on a local APFS or HFS+ filesystem
  146. with Unix permissions. Run `sr capabilities --json` after installation to check
  147. command and integration availability in your build.
  148. ### Installer
  149. The installer selects a release for your platform, verifies its SHA256 checksum,
  150. and installs `sr` into `~/.local/bin`. When a release asset is unavailable, it
  151. builds from the selected source revision using the pinned Rust toolchain:
  152. ```bash
  153. curl -fsSL "https://raw.githubusercontent.com/Dicklesworthstone/skillranker/main/install.sh?$(date +%s)" | bash -s -- --verify
  154. ```
  155. It requires Bash, Python 3, `curl`, and `sha256sum` or `shasum`; source builds
  156. also require Git and Rust. If RCH is installed, compilation runs remotely and
  157. a remote failure does not trigger a local build. macOS source builds need a
  158. macOS RCH worker. Existing binaries are backed
  159. up before replacement. `--easy-mode` also adds the install directory to your
  160. current shell's configuration, with a backup.
  161. ```bash
  162. # Install from a trusted local checkout.
  163. bash install.sh --source . --verify
  164. # Install an air-gapped archive with its adjacent .sha256 file.
  165. bash install.sh --offline skillranker.tar.gz --verify
  166. ```
  167. Detected Claude Code and Codex installations receive a small SkillRanker usage
  168. skill. Subcommand completions are installed for Bash, Zsh, and Fish. Existing
  169. customized integration files are preserved; `--no-configure` skips these steps.
  170. The installer leaves credentials and network consent to you and reports hooks
  171. as unconfigured. **Obtain your own credential for the selected provider before live ranking.**
  172. See [installer options, verification, and rollback](docs/installation.md).
  173. ### From source
  174. Build the `sr` binary with the repository's pinned Rust toolchain and lockfile:
  175. ```bash
  176. git clone https://github.com/Dicklesworthstone/skillranker.git
  177. cd skillranker
  178. cargo install --locked --path . --bin sr
  179. ```
  180. For a checkout-local binary:
  181. ```bash
  182. cargo build --locked --release --bin sr
  183. ./target/release/sr --help
  184. ```
  185. ### Runtime setup
  186. **Choose a provider and create your own credential.** TypeSafe.ai is the default
  187. provider; Cloudflare-hosted TypeSafe Jev can be selected with `SR_PROVIDER=cloudflare`.
  188. Every user supplies their own credential; SkillRanker does not distribute a
  189. shared key.
  190. For the default TypeSafe provider, set `TYPESAFE_API_KEY`. For Cloudflare,
  191. set `SR_PROVIDER=cloudflare`, `CLOUDFLARE_ACCOUNT_ID`, and
  192. `CLOUDFLARE_API_TOKEN`. The
  193. [environment example](.env.example) lists the service settings. For a local
  194. checkout, create `.env` from the example only if it does not already exist and
  195. restrict access with `chmod 600 .env` **before** entering your own API key.
  196. The `.env` file is ignored by Git. Export its values into the process environment
  197. before running `sr` or starting an agent whose hooks need the key:
  198. ```bash
  199. # Run from your checkout, after filling in your own trusted .env file.
  200. set +x
  201. set -a
  202. . ./.env
  203. set +a
  204. ```
  205. Treat `.env` as a local shell configuration file and source only content you
  206. trust. Keep the key out of shell history, logs, and tracked files. Credentials
  207. alone do not enable remote transmission.
  208. | Component | Role |
  209. |---|---|
  210. | Provider credential | Authenticates fresh Jev-compatible evaluations (`TYPESAFE_API_KEY`, or Cloudflare account/token) |
  211. | Network opt-in | `--allow-network` for a run, or `network.enabled` in trusted user configuration |
  212. | Visible skill inventory | Harness-resolved skills or an explicit roster file |
  213. | Session input | Claude hook, normalized context, supported native transcript, or optional cass export |
  214. | [cass](https://github.com/Dicklesworthstone/coding_agent_session_search) | Optional archive discovery and access across coding-agent formats |
  215. `sr` does not require `ms`, a local inference server, or an embedding model.
  216. The supported local platform is Linux. Consult
  217. `sr capabilities --json` for the adapters, events, and optional features in a build.
  218. ## Quick Start
  219. 1. **Try an offline fixture.** Run `sr demo --case useful`, then try `none`,
  220. `explicit`, or `unavailable`. These labeled examples exercise the local
  221. pipeline without reading a private session or contacting Jev. Their output
  222. is non-actionable and does not establish live provider health.
  223. 2. **Choose a provider and configure your own credential.** TypeSafe is the
  224. default; Cloudflare-hosted TypeSafe Jev is also supported. Follow the
  225. [runtime setup](#runtime-setup) instructions for the selected provider.
  226. SkillRanker relies on Jev for its ranking evaluations.
  227. 3. **Check the environment and roster.** Run `sr doctor --json`,
  228. `sr capabilities --json`, and `sr roster --json` in the agent's workspace.
  229. Confirm that candidates are loadable, not merely present somewhere on disk.
  230. 4. **Choose the session.** Supply `--context FILE`,
  231. `--transcript FILE --harness claude_code`, or `--session PATH` for cass.
  232. Automatic discovery must resolve one unambiguous session.
  233. 5. **Preview and rank.** Use `--dry-run` to inspect the redacted wide payload
  234. and disclosure receipt, then `--allow-network --json` for a fresh evaluation.
  235. 6. **Prepare a recorded shadow trial.** Initialize optional history with
  236. `sr ledger init`, then explicitly set `network.enabled = true` in
  237. [trusted user configuration](#configuration) if you want live hook evaluations.
  238. The earlier `--allow-network` flag authorized only that CLI run. Preview and
  239. apply `sr install-hook claude`; shadow mode evaluates without injecting advice.
  240. Without a ready ledger, the hook can rank but cannot promise recorded trial
  241. evidence; without network consent, it cannot obtain fresh Jev answers.
  242. 7. **Enable advisory output deliberately.** Set `hook.mode = "advisory"` in
  243. trusted user configuration after reviewing the integration and its behavior.
  244. ### Readiness checks
  245. `sr doctor` reports each prerequisite separately, names the next concrete step
  246. for a failed check, and lists local commands that remain useful:
  247. | Check | What it establishes |
  248. |---|---|
  249. | Input and roster | A usable source and loadable candidates are available |
  250. | Credential | A key is present; this alone does not authenticate it |
  251. | Network authorization | Trusted settings permit a live request |
  252. | Transport | Untested or previously verified by a separately authorized, budgeted live check |
  253. | Ledger | Optional history is ready or degraded; its absence need not block ranking |
  254. | Hook and snoozes | Shadow/advisory mode and active scoped interruption controls |
  255. Doctor runs locally by default and does not implicitly send a test request,
  256. install hooks, migrate storage, or change configuration. Demo uses bundled
  257. synthetic contexts and labeled synthetic or recorded responses without touching
  258. user configuration or state. Neither command supplies a replacement for Jev.
  259. A previous transport check includes its time, scope, and runtime/endpoint/configuration
  260. identity. Incompatible changes invalidate it; a stored check does not establish
  261. current provider availability.
  262. `sr capabilities --json` separates implemented adapters from tested harness
  263. versions/features and unverified versions. Adapter conformance covers prompt
  264. timing, branch identity, visibility, restrictions, compaction, load evidence, and
  265. hook output, deadlines, and delivery. Native advice requires passing real-harness
  266. evidence for every dimension on the installed version; protocol fixtures and
  267. official schema documentation alone cannot authorize it. Cass remains an
  268. explicit archive source. Native Codex, omp/pi, and Grok integrations each need
  269. their own support record. Incompatible identity or visibility semantics disable
  270. advice. The [adapter contract](docs/adapter-contract.md) defines these checks.
  271. ## Command Reference
  272. Bare `sr` is equivalent to `sr rank`: a table on a TTY, JSON otherwise. It does
  273. not start a TUI. Source flags are mutually exclusive, and piped stdin is consumed
  274. only by an explicit input mode. `sr capabilities --json` lists the commands,
  275. schemas, and optional features available in the installed build.
  276. ### Ranking and inspection
  277. | Command | Purpose | Example |
  278. |---|---|---|
  279. | `sr demo --case CASE` | Inspect a labeled offline fixture | `sr demo --case unavailable` |
  280. | `sr rank` | Rank the next step | `sr rank --allow-network --json` |
  281. | `sr rank --context FILE` | Read normalized context; `-` means stdin | `sr rank --context scratch/context.json --dry-run` |
  282. | `sr rank --transcript FILE --harness NAME` | Read a supported native transcript | `sr rank --transcript scratch/session.jsonl --harness claude_code --offline` |
  283. | `sr rank --session PATH` | Export an exact session through cass | `sr rank --session scratch/session.jsonl --allow-network` |
  284. | `sr rank --why-not ID --explain` | Trace an expected candidate's exclusion | `sr rank --allow-network --why-not SKILL_ID --explain` |
  285. | `sr rank --save-case FILE` | Explicitly capture a bounded redacted replay case | `sr rank --allow-network --save-case scratch/case.json` |
  286. | `sr replay FILE` | Recompute a historical case offline | `sr replay scratch/case.json --compare-policy scratch/candidate.toml` |
  287. | `sr hook claude` | Handle the Claude prompt-hook protocol | `sr hook claude --shadow` |
  288. | `sr roster --json` | Inspect visibility, overrides, records, and exclusions | `sr roster --json` |
  289. | `sr roster --snapshot FILE` | Explicitly export a bounded private roster manifest | `sr roster --snapshot scratch/roster-snapshot.json` |
  290. | `sr roster --diff FILE` | Compare fresh discovery with a saved roster snapshot | `sr roster --diff scratch/roster-snapshot.json` |
  291. | `sr doctor --json` | Inspect local configuration and readiness | `sr doctor --json` |
  292. | `sr doctor --config` | Explain effective non-secret values and their sources | `sr doctor --config` |
  293. | `sr capabilities --json` | Describe commands, schemas, features, limits, and exits | `sr capabilities --json` |
  294. | `sr tui` | Open the inline viewer | `sr tui` |
  295. ### Hooks, feedback, and analysis
  296. | Command | Purpose | Example |
  297. |---|---|---|
  298. | `sr install-hook claude` | Preview a managed hook settings change | `sr install-hook claude --apply` |
  299. | `sr uninstall-hook claude` | Preview removal of the managed entry | `sr uninstall-hook claude --apply` |
  300. | `sr stats` | Report observation and operational metrics | `sr stats --since 7d --by-skill` |
  301. | `sr observe` | Reconcile structured load events | `sr observe --transcript scratch/session.jsonl --harness claude_code` |
  302. | `sr feedback` | Record an explicit usefulness judgment | `sr feedback EVENT_ID --skill SKILL_ID --verdict useful` |
  303. | `sr feedback --instead ID` | Record a better alternative for an event | `sr feedback EVENT_ID --skill SKILL_ID --instead ALTERNATIVE_ID` |
  304. | `sr snooze` | Preview a scoped temporary advisory mute | `sr snooze EVENT_ID --skill SKILL_ID --for 30m` |
  305. | `sr budget` | Inspect or preview a shared HTTP-attempt allowance | `sr budget --max-attempts 100 --window 1h` |
  306. | `sr eval` | Replay a labeled evaluation artifact offline by default | `sr eval --dataset scratch/evaluation.json --explain` |
  307. | `sr calibrate` | Report a candidate threshold configuration | `sr calibrate --evaluation scratch/report.json` |
  308. | `sr calibrate --rollback REVISION` | Preview restoration of managed policy fields | `sr calibrate --rollback POLICY_REVISION` |
  309. | `sr doctor --descriptions` | Check description quality locally | `sr doctor --descriptions` |
  310. | `sr gaps` | Report suspected coverage gaps | `sr gaps` |
  311. | `sr ledger init` | Initialize local history explicitly | `sr ledger init` |
  312. | `sr ledger migrate` | Preview a supported schema upgrade | `sr ledger migrate --apply` |
  313. | `sr ledger prune` | Preview retention cleanup | `sr ledger prune --before 2026-09-01` |
  314. | `sr ledger clear` | Preview clearing local history | `sr ledger clear` |
  315. `--apply` performs a previewed hook, snooze, budget, calibration, or ledger mutation.
  316. Calibration consumes a labeled evaluation artifact; it does not silently change
  317. project settings after a number of observed loads. Description audits use the network
  318. only with an explicit online request and network authorization.
  319. ### Evaluation controls
  320. `sr eval` defaults to replay with **zero network requests**. A live run needs
  321. `--online`, trusted network authorization, and an explicit `--max-requests` cap.
  322. That cap counts HTTP attempts across the entire batch, including retries. Each
  323. case also has its own ranking deadline; the batch stops scheduling work when a
  324. request or runtime limit is reached and reports unfinished cases.
  325. | Flag | Default | Meaning |
  326. |---|---|---|
  327. | `--dataset FILE` | Required | Versioned, consented evaluation data and compatible recorded responses for replay |
  328. | `--labels FILE` | Off | Independent judged labels; `--dataset` is then a labeled case frame scored with the frozen 0/1/2 loss |
  329. | `--online` | Off | With `--labels`, rank each case's request context fresh against the current roster when network access is separately authorized |
  330. | `--max-requests N` | Required for live runs | Maximum HTTP attempts across the batch, including retries |
  331. | `--max-runtime-ms N` | `600000` | Overall batch deadline, in addition to per-case deadlines |
  332. | `--robustness` | Off | Rank request variants and supplied roster arms on the remaining live budget; report paired decision and coverage changes within the original families |
  333. | `--roster FILE` | Discovery | Authorized file-backed baseline inventory for `--online` |
  334. | `--decoy-roster FILE` | None | With `--robustness --roster`, add already existing authorized candidates to the baseline |
  335. | `--lookalike-roster FILE` | None | With `--robustness --roster`, add authorized candidates whose descriptions duplicate a baseline description, ignoring whitespace |
  336. | `--hostile-roster FILE` | None | With `--robustness --roster`, add authorized candidates with prespecified hostile ranking instructions in their descriptions |
  337. | `--sample-size N` | Full supplied frame | Select a bounded sample of task-family representatives |
  338. | `--seed S` | Fresh recorded random seed when sampling | Deterministic diagnostic selection or reproduction of a recorded sample; a fixed seed alone is not probability-sampling evidence |
  339. | `--explain` | Off | Include equations, substituted values, assumptions, and interpretation in the report |
  340. ```bash
  341. # Reproduce a diagnostic selection; seed 42 alone supports no sampling guarantee.
  342. sr eval --dataset scratch/frame.jsonl --labels scratch/labels.jsonl --sample-size 100 --seed 42
  343. # Draw and record a random sample, then authorize a bounded live evaluation.
  344. sr eval --dataset scratch/live-cases.jsonl --labels scratch/labels.jsonl --sample-size 100 \
  345. --online --allow-network --max-requests 400 --max-runtime-ms 600000
  346. ```
  347. A live case carries its frame key, split and a normalized request `context`; it is
  348. a fresh evaluation of today's selector and roster, not a replay. A case is sent only
  349. while its worst-case attempts still fit the cap. Cases without a judgment, or whose
  350. judgment names a skill missing from the current roster, are reported not estimable
  351. and are never sent. Before the first request, every remaining case is previewed
  352. with no network; a refused preview is never sent, and the report freezes the
  353. batch's disclosure totals and receipt digest. An authentication or consent failure
  354. stops scheduling. Live runs write nothing to the ledger or response cache;
  355. a configured [shared attempt allowance](#share-an-http-attempt-allowance-across-sessions)
  356. still charges each of their sends.
  357. The report also gives the process's peak resident memory and projects its main
  358. rankings' attempts and known tokens to 1,000 similar turns, disclosing
  359. unknown-usage attempts; it attaches no price.
  360. From the same answers, with no extra request, the report also scores intrinsic
  361. coverage (an acceptable skill admitted, and shortlisted when the gate passes) and
  362. the policies on one judged cohort: Quill-only, choice-only, fit-only, a
  363. cookbook-style selector, and the production blend, each with exact denominators
  364. and Wilson intervals. Cases whose context carries history are ranked a second time
  365. with the latest request alone, using only the budget left after every main ranking,
  366. and the two arms are compared on the same cases.
  367. Sampling does not grant network access or enlarge the request budget. Missing
  368. stage responses remain unevaluated in replay; they are never replaced with
  369. invented scores. See [evaluation and sampling](#evaluation-sampling-and-risk-monitoring)
  370. for the report's denominators and uncertainty rules.
  371. ### Ranking controls
  372. | Flag | Default | Meaning |
  373. |---|---|---|
  374. | `--messages N` | `12` | Recent logical messages |
  375. | `--budget-chars N` | `12000` | Rendered context budget, including the latest request |
  376. | `--top K` | `5` | Maximum eligible suggestions returned |
  377. | `--shortlist M` | `8` | Real candidates admitted to the rerank |
  378. | `--gate F` | `0.30` | Overall need threshold |
  379. | `--fits F` | `0.30` | Minimum candidate fit |
  380. | `--timeout-ms N` | `3000` | Whole one-shot ranking deadline; applied separately to each TUI/watch refresh |
  381. | `--roster FILE` | Harness discovery | Replace discovery with an explicit inventory |
  382. | `--require-skill ID` | None | Resolve an explicit required skill; repeatable |
  383. | `--latest` | Off | Explicitly choose the newest discovered session |
  384. | `--context-profile PROFILE` | `standard` | Choose standard context or the bounded `minimal` disclosure profile |
  385. | `--no-tools` | Off | Remove tool arguments and results from outgoing context |
  386. | `--no-cache` | Off | Disable response-cache reads and writes |
  387. | `--no-ledger` | Off | Disable all ledger and ingestion-cursor access; use transient evidence |
  388. | `--no-persist` | Off | Disable all persistent state, including cache, cursors, and locks |
  389. | `--offline` | Off | Guarantee zero network calls |
  390. | `--allow-network` | Off | Authorize network evaluation for this invocation |
  391. | `--explain` | Off | Include distributions, exclusions, truncation, and score contributions |
  392. | `--why-not ID` | None | Trace a candidate from the current snapshot with `--explain`, without adding requests |
  393. | `--save-case FILE` | Off | CLI-only, explicit capture for offline replay; cannot overwrite an existing file |
  394. | `--dry-run` | Off | Preview the redacted request without network or persistence effects |
  395. Sizes satisfy `1 ≤ K ≤ M ≤ 32`; fewer available candidates is normal. Parsing
  396. is strict, with documented aliases only. Invalid or conflicting privacy flags
  397. produce an error rather than being silently corrected.
  398. An explicit chunk-overflow experiment uses bounded groups and reduction rounds.
  399. It has separate request limits and availability in the capabilities contract;
  400. the normal overflow policy uses local prefiltering.
  401. ### JSON output
  402. The [versioned output contract](docs/output-contract.md) specifies decision,
  403. error, quality, trace, and non-actionable report schemas.
  404. This illustrative result has two eligible candidates. The score arithmetic uses
  405. `w_fit = 1`, with priors and phase weighting disabled; timing and usage are examples.
  406. ```json
  407. {
  408. "schema_version": 1,
  409. "event_id": "example-event-001",
  410. "decision": "ranked",
  411. "reason": "eligible-candidates",
  412. "harness": "claude_code",
  413. "context_quality": "complete",
  414. "quality": {
  415. "prompt_complete": true,
  416. "task_anchor_known": true,
  417. "history_windowed": true,
  418. "attachments_omitted": false,
  419. "source_gaps": false
  420. },
  421. "roster": {
  422. "total": 2,
  423. "eligible": 2,
  424. "wide_candidates": 2,
  425. "shortlist": 2,
  426. "partial": false,
  427. "retrieval": "full",
  428. "provenance": {
  429. "snapshot_id": "000000000000000000000000000000000000000000000000000000000000000a",
  430. "policy_version": "ranking-v1",
  431. "wide_set_id": "000000000000000000000000000000000000000000000000000000000000000b",
  432. "rerank_set_id": "000000000000000000000000000000000000000000000000000000000000000c"
  433. }
  434. },
  435. "needs_skill": 0.74,
  436. "choice_confidence": 0.81,
  437. "none_probability": 0.10,
  438. "phase": "debugging",
  439. "skills": [
  440. {
  441. "rank": 1,
  442. "skill_id": "s_01",
  443. "name": "rust-test-triage",
  444. "invocation_name": "rust-test-triage",
  445. "rank_score": 0.888889,
  446. "rerank_probability": 0.60,
  447. "wide_probability": 0.55,
  448. "fits": 0.80,
  449. "path": ".claude/skills/rust-test-triage/SKILL.md",
  450. "content_hash": "0000000000000000000000000000000000000000000000000000000000000001"
  451. },
  452. {
  453. "rank": 2,
  454. "skill_id": "s_02",
  455. "name": "rust-code-review",
  456. "invocation_name": "rust-code-review",
  457. "rank_score": 0.111111,
  458. "rerank_probability": 0.30,
  459. "wide_probability": 0.35,
  460. "fits": 0.50,
  461. "path": ".claude/skills/rust-code-review/SKILL.md",
  462. "content_hash": "0000000000000000000000000000000000000000000000000000000000000002"
  463. }
  464. ],
  465. "omitted_rank_mass": 0.0,
  466. "cache": {
  467. "hit": false,
  468. "wide_hit": false,
  469. "rerank_hit": false,
  470. "age_ms": null,
  471. "stale": false
  472. },
  473. "model": {
  474. "requested": "jev-latest",
  475. "wide_returned": "jev-latest",
  476. "rerank_returned": "jev-latest",
  477. "immutable_revision": null
  478. },
  479. "usage": {
  480. "requests": 2,
  481. "http_attempts": 2,
  482. "input_tokens": 6400,
  483. "output_tokens": 480,
  484. "unknown_usage_attempts": 0
  485. },
  486. "persistence": "recorded",
  487. "warnings": [],
  488. "warnings_omitted": 0,
  489. "elapsed_ms": 720
  490. }
  491. ```
  492. | Decision | Meaning |
  493. |---|---|
  494. | `ranked` | Up to K eligible suggestions from a successful evaluation |
  495. | `explicit` | Locally resolved user requests, with no invented model certainty |
  496. | `abstain` | Valid input and policy produced no advisory recommendation |
  497. | `unavailable` | An operational, input, privacy, or coverage problem prevented a decision |
  498. Demo and replay use separately versioned, non-actionable envelopes around their
  499. synthetic or historical decisions. A demo's context is synthetic; any recorded
  500. provider response retains its own provenance. It remains demonstration evidence,
  501. not a live evaluation or a quality-gate result. Neither artifact uses the live
  502. hook output channel.
  503. `choice_confidence` describes the rerank distribution. `fits` is a model estimate
  504. of suitability. `rank_score` is a relative local score over eligible candidates.
  505. They are different quantities. Top-K truncation preserves the original eligible
  506. normalization and reports omitted mass. Fields from an unexecuted stage are
  507. `null`, not fabricated zeros.
  508. Cache hits and returned model identities are recorded separately for each stage.
  509. When reranking is required, a wide-stage hit alone is not a complete offline
  510. result. An alias such as
  511. `jev-latest` does not identify an immutable model revision.
  512. `persistence: "recorded"` means at least one qualified store backed the run: the
  513. response cache, a committed ledger event, or both. It does not mean the harness
  514. acknowledged or consumed the recommendation. A ledger write that cannot be
  515. confirmed adds the `ledger-finalization-unconfirmed` warning.
  516. Quality metadata includes `prompt_complete`, `task_anchor_known`,
  517. `history_windowed`, `attachments_omitted`, and `source_gaps`. These describe the
  518. admitted input; `context_quality: "complete"` does not claim that the entire
  519. conversation history was read. Warnings are bounded to 32 details plus an omitted
  520. count. Live decision JSON and individual demo/replay/report summary envelopes
  521. are capped at 2 MiB. The separate 256 MiB evaluation-artifact limit bounds a
  522. streamed dataset or report with at most 10,000 case records; it does not enlarge
  523. an individual output envelope. Roster listings and full-wide explanations
  524. paginate against a fixed snapshot; a changed snapshot requires restarting.
  525. Trace pages contain at most 128 entries and bind both snapshot and query identity.
  526. Unevaluated entries keep operands and reasons null instead of inventing evidence.
  527. ### Exit codes
  528. | Code | Meaning |
  529. |---|---|
  530. | `0` | Ranked, explicit, valid abstention, or another successfully completed command |
  531. | `2` | Invalid usage or configuration |
  532. | `3` | Missing or ambiguous session |
  533. | `4` | Provider, authentication, or network failure; request-admission refusal |
  534. | `5` | Empty/unusable roster, unresolved explicit request, or Quill retrieval failure |
  535. | `6` | Overall deadline exhausted |
  536. | `7` | Malformed, oversized, or unsupported input |
  537. | `8` | Network transmission disallowed |
  538. | `9` | Required storage or administrative mutation failed |
  539. | `10` | Invalid structured provider response |
  540. | `11` | No complete valid result under offline/cache-only constraints |
  541. JSON errors include `schema_version`, `decision: "unavailable"`, and an `error`
  542. object with `code`, kebab-case `kind`, `message`, `hint`, and `retryable`.
  543. Ordinary ranking can succeed with a storage warning; an explicit feedback write
  544. cannot claim success when its required write failed.
  545. Incomplete essential context and output-limit failures use exit `7`, unresolved
  546. explicit requests and Quill retrieval failures use `5`, and superseded input uses `3`.
  547. Overall deadline exhaustion uses `6`. `retryable` means a
  548. fresh invocation with the same intended inputs may succeed; it does not grant
  549. network access or relax a deadline.
  550. Evaluation and replay reports distinguish execution from quality:
  551. `run_status` is `complete` or `partial`; `gate_status` is `passed`, `failed`,
  552. `not-established`, or `not-applicable`. Exit zero means the report was produced,
  553. not that every comparison was replayable or a policy passed. Promotion requires
  554. the intended complete cohort, compatible evidence, and explicit passed gates.
  555. Fatal errors retain their nonzero exit code and available partial-work/usage data.
  556. Each summary accounts for requested/completed cases and required/completed
  557. stages. Partial, incompatible, empty, or synthetic evidence cannot carry a
  558. passed gate. A completed report can faithfully describe a failed historical
  559. request without treating that historical error as a failure to generate the report.
  560. The dedicated hook maps recommendation failures to quiet exit-zero behavior so
  561. it never blocks the agent. CLI failures retain their meaningful exit codes.
  562. ## Configuration
  563. Ordinary settings resolve from lowest to highest priority:
  564. ```text
  565. built-in defaults
  566. -> trusted user configuration
  567. -> allowlisted workspace configuration
  568. -> recognized SR_* environment variables
  569. -> command-line flags
  570. ```
  571. On Linux, user configuration falls back to `~/.config/sr/config.toml`; project
  572. configuration is `.sr/config.toml` at the workspace root. Other platforms use
  573. native configuration directories.
  574. Trusted user settings for an advisory hook include:
  575. ```toml
  576. [network]
  577. enabled = true
  578. [hook]
  579. mode = "advisory"
  580. ```
  581. Without those choices, remote transmission is disabled and the hook runs in
  582. shadow mode. `--allow-network` can authorize a single CLI evaluation.
  583. A turn that a background task's `<task-notification>` starts while the agent is
  584. idle is harness output, not a request you submitted. By default the hook counts
  585. it as a non-turn and sends nothing. Set `notification_turns = "rank"` under
  586. `[hook]` in trusted user configuration to evaluate those turns too. Workspace
  587. configuration can only keep them skipped.
  588. | Variable | Purpose |
  589. |---|---|
  590. | `TYPESAFE_API_KEY` | TypeSafe bearer credential; never serialized or stored in project config |
  591. | `TYPESAFE_ENDPOINT` | Trusted HTTPS base origin; `sr` appends `/v1/systemone` once |
  592. | `SR_PROVIDER` | Provider selector: `typesafe` (default) or `cloudflare` |
  593. | `CLOUDFLARE_ACCOUNT_ID` | Cloudflare account ID required by Workers AI |
  594. | `CLOUDFLARE_API_TOKEN` | Cloudflare bearer credential; never serialized or stored in project config |
  595. | `SR_MODEL` | Requested model; default `jev-latest` |
  596. | `SR_MESSAGES`, `SR_BUDGET_CHARS` | Context-volume limits: 1–12 messages and 1–12,000 Unicode scalar values |
  597. | `SR_TOP`, `SR_SHORTLIST` | Output and rerank sizes, satisfying `1 ≤ top ≤ shortlist ≤ 32` |
  598. | `SR_GATE`, `SR_FITS` | Finite thresholds in `[0, 1]` |
  599. | `SR_TIMEOUT_MS` | Whole-invocation deadline, 201–60,000 ms |
  600. Workspace configuration may tune bounded ranking values and exclusions. It
  601. cannot authorize networking, change endpoints/proxies, supply credentials,
  602. expand transcript access, disable redaction, or enable raw retention. Unknown
  603. or duplicate keys, forbidden project settings, and invalid values are errors
  604. after bounded configuration reads and before discovery, networking, or mutation.
  605. Unrecognized `SR_*` environment names are errors too. Project context settings
  606. may reduce message/character budgets, select `minimal`, or remove tool content;
  607. they cannot widen a trusted user's disclosure settings. Exclusions and skill
  608. roots merge as unions. Project roots must remain relative to the workspace;
  609. only trusted user configuration can authorize absolute roots. Readers also check
  610. symlink containment when opening files.
  611. Credentials and provider account identifiers are environment-only and kept outside serializable configuration.
  612. The v1 endpoint override is environment-only; model overrides use trusted user
  613. configuration or `SR_MODEL`. Proxy, redaction-disable, and raw-retention settings
  614. are reserved and rejected. The [configuration contract](docs/config-contract.md)
  615. lists every key, bound, source layer, and restriction.
  616. The endpoint accepts an empty or root path and rejects URL credentials, query
  617. strings, fragments, and non-root paths. Credentials, cache identity, and request
  618. allowance scope use the canonical origin. A development-only loopback HTTP
  619. exception cannot carry production credentials.
  620. `sr doctor --config` shows non-secret effective values and their winning sources.
  621. Disallowed overrides are configuration errors; no valid policy or fingerprint is
  622. reported for them. Credential presence is reported without its value, and endpoint
  623. overrides expose only their presence. `--offline` and `--allow-network` are accepted
  624. here but conflict; this inspection command never sends a request.
  625. The library's `cli::ConfigFiles` shares bounded initial configuration reads with
  626. file refreshes. Refresh keeps the invocation's validated environment and CLI
  627. layers, returning the current configuration and a boundary-specific comparison
  628. against its typed policy receipt. Malformed, unreadable, or late reads fail rather
  629. than authorize output. `sr rank` refreshes the configuration at each provider
  630. admission and before publishing, so a policy change during a run withholds its
  631. result.
  632. Shared request allowances and snoozes are explicit trusted-user controls. Project
  633. configuration cannot enable, raise, or disable the allowance.
  634. `--offline` and `--dry-run` conflict with `--allow-network`. Case capture conflicts
  635. with `--dry-run` and `--no-persist`. `--no-cache` and `--no-ledger` disable their
  636. respective stores independently; `--no-persist` also disables persistent runtime
  637. coordination. Ordinary configuration reads remain allowed.
  638. Each provider attempt rechecks the effective disclosure and admission policy.
  639. Publication separately rechecks the fields governing eligibility and hook mode.
  640. A revoked permission or changed governing value withholds the affected action;
  641. it never authorizes a replacement request. An edit hidden by an unchanged CLI
  642. or environment override does not change the effective policy.
  643. ## How Ranking Works
  644. ### 1. Establish the exact context
  645. The Claude hook uses the incoming prompt as the current request, even when the
  646. transcript has not yet recorded it. Context, cursors, and feedback belong to a
  647. specific workspace, session, agent branch, and source adapter/producer. An
  648. explicit source that fails does not silently fall through to another conversation.
  649. Native JSONL reads process complete records within a bounded tail. Replacement,
  650. truncation, compaction, and incomplete final lines are handled explicitly. An
  651. empty first transcript can still yield prompt-only context; malformed existing
  652. history is a different condition.
  653. Windowing drops reasoning blocks, binary/media payloads, and prior `sr` advice.
  654. Tool summaries preserve invocation/result association and useful failure lines.
  655. The latest request gets budget priority, with explicit head/tail truncation for
  656. oversized input. Redaction runs on complete bounded fields before truncation,
  657. then on the assembled provider payload.
  658. Explicit directives are resolved from the full bounded local request before
  659. redaction or truncation. The normalized input envelope carries local identity
  660. and events; it is validated and reduced to a separate provider schema, never
  661. sent wholesale to Jev.
  662. A normalized import cannot update a native session's observations merely by
  663. repeating its session ID. An input without durable session identity gets an
  664. invocation-local namespace; unknown attribution disables durable session updates.
  665. A terse “continue” needs a recoverable task anchor; an essential missing
  666. instruction or attachment yields `unavailable` and a quiet hook, rather than a
  667. guess from incomplete context.
  668. Project signals use language/framework filenames, allowlisted tools on a trusted
  669. PATH, and bounded repository-relative dirty paths. Absolute workspace paths and
  670. branch names remain local by default. Git inspection disables filesystem-monitor
  671. hooks, optional index locks, and submodule traversal; an unsupported safe invocation
  672. omits the optional signal instead of executing project helpers.
  673. The library boundary `context::signals::collect` implements these optional local
  674. signals with an explicit authorized workspace and trusted executable roots. Its
  675. 250 ms stage intersects the invocation budget; status output is capped at 64 KiB
  676. and 100 paths. Non-UTF-8/unsafe paths and partial inventories are reported rather
  677. than treated as complete absence. Results are not serializable provider payloads:
  678. callers must still redact them. `sr rank` collects them for every evaluation,
  679. using fixed system directories (`/usr/local/bin`, `/usr/bin`, `/bin`) as its
  680. trusted executable roots; the minimal context profile omits dirty paths.
  681. Synchronous filesystem/spawn calls retain the subprocess boundary's documented
  682. uninterruptible-kernel-I/O limitation.
  683. ### 2. Resolve what is loadable
  684. An explicit `--roster FILE` replaces discovery. Otherwise, a harness inventory
  685. or its visibility adapter determines roots, overrides, plugins, and load targets.
  686. The presence of a directory does not mean the selected harness loads its skills.
  687. Generic file mode uses explicitly configured roots and exposes uncertain visibility.
  688. Each skill has an opaque stable ID, its actual invocation name, display name,
  689. source, content hash, load target, metadata, and visibility. Same-name skills
  690. remain distinct where the harness permits; shadowed or ambiguously invocable
  691. entries are excluded from hook suggestions.
  692. For Claude, `sr rank` resolves same-name skills by Claude's documented
  693. precedence (project skills over personal ones). That precedence is not yet
  694. backed by conformance evidence, so every rank result carries
  695. `"visibility": "unverified"`, and the decision's first warning is
  696. `unverified-visibility`: confirm that a suggested skill loads before relying on
  697. it. `sr roster` and `sr doctor` claim no precedence at all. Names whose
  698. authority is withheld (for example, a malformed file that could claim the same
  699. name), ambiguous names and shadowed entries are never suggested.
  700. Invocation restrictions are part of eligibility. Claude skills with
  701. `disable-model-invocation: true` or an effective user-only restriction are excluded
  702. from automatic advice; `user-invocable: false` alone does not exclude agent use.
  703. A user-requested manual-only skill resolves as a `manual_only` reference, without
  704. instructing the agent to bypass that restriction by reading its file.
  705. | Resource | Default bound |
  706. |---|---:|
  707. | Hook stdin | 1 MiB |
  708. | Normalized context | 1 MiB / nesting depth 64 |
  709. | Each user/project/policy configuration file | 256 KiB / nesting depth 32 |
  710. | Explicit roster | 32 MiB / 10,000 records / nesting depth 64 |
  711. | Transcript tail | 2 MiB / 2,000 records |
  712. | Observation ingestion | 8 MiB per invocation; a separate committed cursor |
  713. | One transcript record | 256 KiB |
  714. | cass stdout | 8 MiB |
  715. | Skill file / frontmatter | 256 KiB / 16 KiB |
  716. | Discovery | 10,000 files / 32 MiB parsed bytes |
  717. | Quill query | 128 distinct terms / 4,096 Unicode scalar values after escaping |
  718. | Replay case capture/import | 16 MiB / nesting depth 64, with per-field limits |
  719. | Local replay policy | 64 KiB / nesting depth 32 |
  720. | Streamed evaluation dataset/report | 256 MiB / 10,000 case records / nesting depth 64 |
  721. | Live decision or artifact summary envelope | 2 MiB / nesting depth 64 |
  722. | Wide description | 160 characters |
  723. | Rerank description / body excerpt | 1,000 / 700 characters |
  724. | Serialized provider request / decoded response | 96 KiB / 2 MiB |
  725. Reject duplicate keys and duplicate record definitions within each schema's
  726. collection/namespace in normalized inputs, rosters, configuration,
  727. replay/evaluation artifacts, frontmatter, and provider responses.
  728. The same skill can still be referenced across wide and rerank stages.
  729. Ambiguous skill metadata excludes that record and marks coverage partial.
  730. Configuration cannot execute interpolation or recursive includes. Evaluation
  731. imports use bounded streaming and enforce per-case limits as well as the total cap.
  732. Source snapshots supply both hashes and excerpts. Before publishing a live
  733. advisory decision or no-match claim, `sr` checks membership, precedence, and
  734. indexed/wide content that
  735. conditioned the decision, plus the entire shortlist's content and restrictions.
  736. A changed candidate outside the shortlist or a new overflow match can invalidate
  737. the result too. Explicit resolution checks every target and its name's precedence.
  738. Trusted adapter generations can avoid a rescan only when they cover all required
  739. dependencies. Otherwise bounded re-enumeration/content checks use the same deadline;
  740. missing required validation withholds output. A runner-up cannot replace an answer
  741. conditioned on stale alternatives. These are last-validation observations, not a
  742. freeze of the filesystem; the harness still checks its later load. A supplied
  743. roster replaces discovery but grants no new path access or invocation permissions.
  744. `sr rank`, `sr roster` and the roster check in `sr doctor` also read directories
  745. from `roster.roots` in trusted-user and project configuration. Each directory
  746. uses the `<skill-name>/SKILL.md` layout. Project roots stay inside the workspace,
  747. including when symlinks are involved; trusted-user configuration can name
  748. absolute roots. For Claude sessions, configured roots extend the default Claude
  749. directories and rank below them when names collide. Other harnesses supplied
  750. through normalized context use only the configured roots, preserve their harness
  751. identity, and treat same-name definitions in different roots as ambiguous. Aliases
  752. of an already opened directory are deduplicated. Reading a custom root does not
  753. prove that the harness can invoke its skills: `sr rank` can suggest them, but
  754. always reports them as
  755. `"visibility": "unverified"`. This local file layout does not establish native
  756. harness integration or a complete inventory. Without configured roots, a session
  757. from another harness needs an explicit `--roster FILE`.
  758. `sr roster --snapshot FILE` explicitly exports an owner-only manifest, bounded
  759. to 32 MiB and 10,000 records, without implicitly overwriting an existing file.
  760. Manifests can contain private skill names. `sr roster --diff FILE` compares the
  761. snapshot with fresh authorized discovery in the same workspace, adapter, and
  762. source namespace. It reports additions/removals, content and restriction changes,
  763. shadowing, and invocation-name changes. Incompatible manifests are identified;
  764. incomplete source coverage remains unknown rather than becoming a confirmed
  765. deletion. Saved paths grant no new read access and cannot restore a removed skill.
  766. ### 3. Retrieve, then compare
  767. Explicit requirements are resolved from the complete visible roster first.
  768. They bypass probabilistic retrieval and cannot be vetoed by a low gate.
  769. Every requested reference must resolve: missing, ambiguous, forbidden, or
  770. conflicting references produce `unavailable / explicit-resolution` with separate
  771. resolution records and no advisory API call. Successful explicit lists are not
  772. truncated to top-K; the input limit is 32 explicit references.
  773. For advisory ranking, **[Quill](https://github.com/Dicklesworthstone/frankensearch/tree/main/crates/frankensearch-quill)**,
  774. the native lexical engine in FrankenSearch, supplies BM25 retrieval when the
  775. eligible roster exceeds **254 real skills**. Every Choice also
  776. includes `__none__`, for at most 255 total options. Retrieval uses the latest
  777. request plus bounded task and error context; a terse “continue” retains useful
  778. prior evidence.
  779. | Eligible roster | Quill matches | Real skills admitted to Jev |
  780. |---|---|---|
  781. | 1–254 skills | Prefilter skipped | The full eligible roster |
  782. | More than 254 skills | At least 254 | The first 254 matches under the deterministic ordering |
  783. | More than 254 skills | 1–253 | Only those matches; the set is not padded with nonmatching skills |
  784. | More than 254 skills | None | No provider call; `unavailable / retrieval-empty`, exit `5` |
  785. Configured sizes first satisfy `1 ≤ K ≤ M ≤ 32`. The effective rerank size is
  786. `min(M, admitted_wide_count)`, and the output cap is `min(K, effective_M)`.
  787. For example, three Quill matches produce a wide Choice with three skills plus
  788. none, a rerank of at most three skills, and at most three returned suggestions.
  789. A single match is valid and still competes against none. An initially empty
  790. roster is a roster failure; a valid roster reduced to zero by explicit exclusions
  791. or proven available references yields a local abstention.
  792. SkillRanker embeds Quill's in-memory index through `frankensearch-quill`, with
  793. default features disabled, the required `frankensearch-core` document types,
  794. bounded indexing/query work, and the caller's Asupersync context. Names and aliases
  795. are searchable titles; descriptions and tags are searchable content, rather than
  796. stored-only metadata. Documents enter in stable skill-ID order and are committed
  797. before querying. Cutoff ties follow the pinned document-ID mapping; re-sorting
  798. an already-truncated result cannot recover an omitted tied candidate.
  799. Queries are a deduplicated **OR** of escaped literal terms. Only the adapter adds
  800. the OR separators; conversation text cannot introduce Boolean operators,
  801. wildcards, ranges, or field syntax. The limit of 128 distinct terms and 4,096
  802. Unicode scalar values applies after escaping and separators; analysis input and
  803. work are bounded too. No analyzed terms means retrieval-empty. Parser
  804. diagnostics and truncation are reported. A query that cannot be preserved safely,
  805. exhausted query fuel, or an index failure yields unavailable output with quiet
  806. hook fallback. Retrieval failures use exit `5`; exhaustion of the overall
  807. invocation deadline uses `6`. Partial work never becomes a complete candidate set,
  808. and there is no silent fallback to another engine.
  809. Overflow results identify `retrieval: "quill-bm25"`, the admitted count, and
  810. engine/schema provenance. Building, committing, and querying the index consume
  811. the same ranking deadline. A TUI can retain a matching roster index across
  812. refreshes; a new hook process cannot assume an earlier process's index survives.
  813. **Quill is the only lexical search engine used by this project. Tantivy is not
  814. used for runtime search, fallbacks, tests, benchmarks, or reference code.**
  815. The hybrid search facade, legacy lexical engine, Quill gauntlet, and optional
  816. oracle/compatibility features are excluded. Dependency checks cover normal,
  817. build, and development feature graphs. Quill verification uses native tests and
  818. independent expected-result fixtures. SkillRanker does not need the `fsfs`
  819. command, an embedding model, a search service, or an imported foreign index.
  820. The wide pass combines a Choice, phase distribution, and three oriented gates:
  821. ```text
  822. needs_skill = mean(
  823. specialized_method,
  824. material_help,
  825. 1 - context_suffices
  826. )
  827. ```
  828. `needs_skill` is a heuristic score. Below the default `0.30` threshold, `sr`
  829. abstains without a rerank. The wording includes planning, analysis, writing, and
  830. explanation skills; acting on files is not a prerequisite for needing a method.
  831. When the gate passes, up to eight real candidates proceed to a detailed Choice
  832. with another none option and one fit Noul per candidate. If none wins the wide
  833. comparison, the detailed comparison still runs when the need gate passes:
  834. richer skill excerpts can resolve ambiguity left by short descriptions. The client
  835. uses the selected provider's Jev transport, preserving typed answers and
  836. validating every requested option before scoring. The TypeSafe adapter uses the
  837. [TypeSafe HTTP API](https://docs.typesafe.ai/api); the Cloudflare adapter uses
  838. Cloudflare's native Jev `/ai/run` contract.
  839. ### 4. Apply eligibility and rank survivors
  840. A candidate is removed if it is excluded, below the fit threshold, or a reusable
  841. reference whose relevant content is proven present in the current context epoch.
  842. Workflows and unknown usage kinds remain eligible for repeat invocation. A changed
  843. shortlist invalidates the in-flight result.
  844. The reusable-reference check needs evidence of the version and rendered content
  845. actually present. A source file's current hash cannot establish what a past read
  846. consumed. Changed arguments, dynamic content, forked execution, or compaction can
  847. invalidate reuse evidence; content presence never renews turn-scoped permissions.
  848. Every remaining candidate must individually beat the none option's raw rerank
  849. probability. Ties are excluded. If no candidates survive, `sr` abstains; local
  850. priors and fit blending cannot re-admit a candidate that failed this check.
  851. For each eligible candidate:
  852. ```text
  853. eps = 1e-6
  854. clip(x) = min(1 - eps, max(eps, x))
  855. log_odds(x) = ln(clip(x) / (1 - clip(x)))
  856. utility_i = ln(clip(p_rerank_i))
  857. + w_fit * log_odds(fits_i)
  858. + w_prior * prior_delta_i
  859. + w_phase * phase_match_i
  860. rank_score_i = softmax(utility)_i
  861. ```
  862. Defaults are `w_fit = 1.0`, `w_prior = 0.0`, and `w_phase = 0.0`.
  863. Priors and phase weighting are optional evaluated policy choices. A skill is
  864. not penalized just because a previous suggestion went unobserved. After
  865. compaction, uncertain loaded-state evidence cannot suppress a skill indefinitely.
  866. ## Explain And Replay A Result
  867. ### Find where a candidate was lost
  868. `--why-not SKILL_ID --explain` follows a candidate through discovery, visibility
  869. and restrictions, local policy, Quill admission, the wide shortlist, fit/none
  870. eligibility, final ordering, and publication. It reports the first decisive
  871. exclusion and any later stages actually evaluated.
  872. ```bash
  873. sr rank --context scratch/context.json --allow-network \
  874. --why-not SKILL_ID --explain --json
  875. ```
  876. An unevaluated stage reports `not-evaluated`; an unknown ID reports
  877. `not-in-snapshot`. Neither receives a fabricated zero fit. Explanations include
  878. threshold operands, tie handling, content/policy versions, and bounded recovery
  879. hints. They do not expand discovery, insert the target into a shortlist, change
  880. the ranked result or provider request bytes, or add a provider call. Hints identify
  881. actions and arguments for review; they do not execute commands or relax policy.
  882. ### Save a case and compare local policies offline
  883. Explicit capture turns a surprising result into a reproducible case:
  884. ```bash
  885. # Opt in to retaining this run's bounded redacted inputs and recorded answers.
  886. # Use a new output path in an existing private directory.
  887. sr rank --context scratch/context.json --allow-network \
  888. --save-case scratch/ranking-case.json --explain --json
  889. # Reproduce the decision without accessing the source session or Jev.
  890. sr replay scratch/ranking-case.json
  891. # Compare compatible local thresholds/weights against the same recorded answers.
  892. sr replay scratch/ranking-case.json --policy scratch/baseline.toml \
  893. --compare-policy scratch/candidate.toml
  894. ```
  895. A case binds the actual redacted request inputs, exact option maps and content
  896. digests, validated recorded responses, eligibility evidence, policy, and
  897. model/adapter provenance. It freezes the evaluation time, snoozes, loaded-state
  898. and visibility evidence, ordered candidates, numeric priors/phase inputs, and
  899. computation versions. Replay never consults today's clock, configuration, or
  900. ledger priors. Synthetic fixtures and recorded provider answers are
  901. labeled separately. Replay returns `kind: replay` with `actionable: false`,
  902. keeping historical and recomputed decisions separate from live recommendations.
  903. It makes no network requests, discovers no transcripts, executes no skills, and
  904. writes no state. Only the explicit case and optional local policy files are read;
  905. ambient user/project policy and credentials are not consulted. Embedded source
  906. paths remain inert.
  907. Local policy comparison requires the compatible inputs and responses needed by
  908. that comparison. A completeness manifest identifies each captured stage. A
  909. low-gate case can replay its original abstention without a rerank, but lowering
  910. the gate needs the missing rerank. Missing required stages or an uncaptured prior
  911. report `not-replayable`; a changed model, prompt, retrieval strategy, excerpt,
  912. or shortlist needs new consented evaluation. Policy files accept only the
  913. supported local ranking schema, without executable code or credential/routing
  914. settings. A changed score is not evidence of better task outcomes. Ordinary
  915. metadata-only history cannot reconstruct a case.
  916. Exact parity covers decisions and numeric outputs under the same tested
  917. computation profile, not new invocation IDs, timings, or usage. Different numeric
  918. backends/build profiles need declared tolerances or report incompatible exact
  919. replay. An unavailable historical case can reproduce sanitized failure metadata
  920. without inventing an unobserved response. Additional privacy transformations
  921. remove any affected exact-input replay claim. Artifact digests need no original
  922. cache secret and cannot authorize importing a result into the live cache.
  923. Capture is opt-in because redacted prose can remain confidential. All retained
  924. prose is redacted; credentials, hash keys, secret-bearing configuration, and
  925. response error bodies are excluded. Files are owner-only, created exclusively
  926. without overwriting existing targets, bounded to 16 MiB and nesting depth 64,
  927. and published only after a complete write. Publication uses a tested atomic
  928. no-clobber operation, including protection against a target or symlink raced into
  929. place, with a documented flush policy. A crash may leave a complete export whose
  930. delivery is unknown; filesystem publication and stdout are not atomic. Private,
  931. bounded partial files remain identifiable for explicit cleanup. These publication
  932. rules also apply to roster snapshots. Per-field limits still apply.
  933. An oversized case fails explicitly instead of dropping data needed for replay.
  934. Capture consumes the invocation deadline; a failed requested write reports a
  935. storage or timeout failure, preserving any already-incurred attempts and usage.
  936. `--save-case` conflicts with `--dry-run`, `--no-persist`,
  937. and hook mode. Imports validate bounds and internal consistency; a matching digest
  938. does not establish trusted authorship. Imported labels and responses remain
  939. untrusted evaluation data.
  940. ## Local Feedback And Calibration
  941. SkillRanker keeps observations and judgments separate.
  942. | Record | What it establishes |
  943. |---|---|
  944. | Generated ranking | The selector produced a result |
  945. | Successful advisory stdout write | Advice was emitted; harness consumption is still separate |
  946. | Shadow evaluation | A prediction was recorded without exposing the agent to advice |
  947. | Load attempt | A structured tool tried to load a resolved skill |
  948. | Observed successful load | A resolved skill was loaded; its version can remain unknown |
  949. | Not observed / unobservable / censored | The available record cannot establish an outcome |
  950. | Explicit usefulness judgment | An assessor labeled a particular event and skill version |
  951. `sr stats` puts usefulness, interruptions, and cost in one local report:
  952. | Measure | Interpretation |
  953. |---|---|
  954. | Evaluated turns, emitted suggestions, valid abstentions, muted/suppressed output | How often the selector evaluates and interrupts |
  955. | Operational failures, unfinished turns, latency, cache reuse | Availability and overhead across the disclosed cohort |
  956. | Observed loads, observation coverage, censoring | What was seen, with missing evidence kept visible |
  957. | Independently judged useful suggestions and label coverage | Usefulness for the judged cohort |
  958. | HTTP attempts, known tokens, unknown usage, estimated cost | Recorded consumption and the limits of its accounting |
  959. A turn that was recorded before its provider request and never finished — one still
  960. running, or one killed outright — is counted on its own. It is not an operational
  961. failure, because no provider outcome was observed, and its absent duration is kept out
  962. of the latency summary, which reports how many turns it excluded.
  963. Cost per judged-useful suggestion uses only that cohort's matched attempts and
  964. labels. A turn carrying more than one label still costs what it cost. With no useful
  965. labels, the ratio is not estimable. Unknown usage or
  966. missing/inapplicable pricing also prevents an exact monetary ratio; report known
  967. attempt and token counts instead. Unlabeled traffic does not inherit measured
  968. usefulness, adoption is not task success, and token savings
  969. are not estimated labor savings. Reports contain no raw examples by default and
  970. do not enable adaptation or advisory mode.
  971. Shadow, advisory-hook, CLI, and TUI records have separate denominators. Writing
  972. zero bytes in shadow mode does not count as delivered advice. A successful
  973. path-only read can establish a load but cannot establish the consumed version
  974. for suppression or version-specific feedback.
  975. Observation ingestion has its own cursor, separate from the bounded ranking
  976. window. Observations, loaded-state evidence, and cursor advancement commit
  977. together; unread events remain a backlog, not skipped history. Attribution uses
  978. the latest preceding emission with a known boundary in the same agent/turn.
  979. Ambiguous concurrent delivery remains unknown, and identical prompt text does
  980. not merge distinct turns. `sr observe` can reconcile the final turn without
  981. waiting for another user prompt.
  982. Observation requires an explicit source, with the same meaning as ranking:
  983. `--transcript FILE --harness NAME` is native, `--session PATH` selects cass, and
  984. `--context FILE` selects a normalized producer. Those namespaces remain distinct;
  985. matching a session ID or path cannot authorize updating another adapter's cursor.
  986. Unknown durable identity is an error. `sr observe` never calls Jev and requires
  987. a ready ledger; `--no-ledger` and `--no-persist` conflict with its promised update.
  988. ```bash
  989. sr observe --transcript scratch/session.jsonl --harness claude_code
  990. sr stats --since 7d --by-skill
  991. sr feedback EVENT_ID --skill SKILL_ID --verdict useful
  992. sr eval --dataset scratch/labeled-cases.jsonl --labels scratch/labels.jsonl
  993. sr calibrate --evaluation scratch/evaluation-report.json
  994. sr calibrate --evaluation scratch/evaluation-report.json --apply
  995. ```
  996. When a different skill would have helped, record the correction directly:
  997. ```bash
  998. sr feedback EVENT_ID --skill ORIGINAL_ID --instead BETTER_ID
  999. ```
  1000. The two distinct candidates resolve against historical membership, versions, and
  1001. advisory eligibility. The ledger retains bounded, deduplicated membership
  1002. snapshots of the full roster, including skills outside the shortlist, without
  1003. their bodies or descriptions. An alternative that was manual-only or excluded
  1004. was not a missed advisory candidate. The correction records the original as
  1005. unsuitable and the alternative as useful in one atomic transaction, with
  1006. assessor provenance and expected label revisions. A failed alternate lookup
  1007. cannot leave an unintended standalone negative label. `--instead` and
  1008. `--verdict` are mutually exclusive.
  1009. It is partial, unblinded feedback: other candidates remain unjudged, and the
  1010. correction cannot become an independent blinded holdout case.
  1011. A known-absent alternative becomes a separate prospective proposal. A missing
  1012. historical snapshot means unknown membership, not absence, and prevents an
  1013. unsupported historical correction. Changed or unknown versions need separately
  1014. identified evidence. Neither case establishes that the original selector missed
  1015. an available candidate.
  1016. For “not now,” use [snooze](#snooze-advice-without-changing-usefulness) instead of
  1017. a negative usefulness label.
  1018. Calibration uses independently judged positive, no-match, and near-miss cases,
  1019. with separate training, validation, and final-test task families. Priors are fit
  1020. on training data, thresholds are chosen on validation data, and the frozen policy
  1021. is evaluated once on the final holdout. Future labels cannot enter an earlier
  1022. case's prior snapshot. The default tuning loss is:
  1023. | Outcome | Loss |
  1024. |---|---:|
  1025. | Correct suggestion or correct no-match abstention | 0 |
  1026. | Abstention on a positive case | 1 |
  1027. | Incorrect suggestion, including a needless suggestion on a no-match case | 2 |
  1028. | Operationally unavailable result on an actually attempted case | 2 |
  1029. Compare policies on the same predeclared judged cases and denominator. An
  1030. operational failure keeps its `unavailable` status and separate failure count;
  1031. its loss penalty prevents dropping hard cases from making a policy look better.
  1032. On equal loss, prefer fewer failures, then the frozen baseline. A policy that
  1033. always stays silent still incurs misses on positive cases.
  1034. Bounded-loss sampling uses this loss divided by two; reports identify the scale
  1035. and keep it distinct from a binary error rate.
  1036. Missing replay evidence is different from an observed failed attempt. A policy
  1037. whose required responses are absent is not estimable; do not invent a result or
  1038. remove those cases to complete its comparison. Lowering a gate needs rerank
  1039. evidence, and changing retrieval, shortlist size, prompts, or model can require
  1040. new evaluations. A batch stopped before finishing the declared cohort remains
  1041. partial and cannot promote a policy.
  1042. Experimental priors use centered, shrunk Beta(1, 4) estimates from judged
  1043. usefulness. They are disabled by default and can only reorder eligible candidates.
  1044. A fixed observation count alone never enables learning.
  1045. Policy rollback follows the same preview/apply workflow:
  1046. ```bash
  1047. sr calibrate --rollback POLICY_REVISION
  1048. sr calibrate --rollback POLICY_REVISION --apply
  1049. ```
  1050. Rollback restores only managed ranking-policy fields after compatibility,
  1051. digest, and conflict checks. It preserves current credentials, networking
  1052. authorization, hook installation, feedback, and unrelated settings; it does not
  1053. replace the entire configuration with an old file.
  1054. Description diagnostics flag missing metadata, duplicate visible prefixes, and
  1055. rerank disagreements. Gap reports identify **suspected** missing coverage while
  1056. showing retrieval and context quality. Text clustering requires separately enabled
  1057. retention of redacted excerpts; metadata cannot reconstruct a private request.
  1058. Quill retrieves retained examples and candidate neighbors; optional local
  1059. clustering operates on those examples. Neither command edits skills or invokes a
  1060. skill manager.
  1061. ## Evaluation, Sampling, And Risk Monitoring
  1062. SkillRanker evaluates whether a recommendation helps under the user's current
  1063. constraints and the harness's permissions. Labels come from independent review
  1064. of the full visible roster. Each case can have several acceptable additional
  1065. invocations, or none. Explicit-name resolution has its own tests and is excluded
  1066. from advisory quality metrics.
  1067. ### Measure the whole selection pipeline
  1068. | Metric | What counts |
  1069. |---|---|
  1070. | Candidate coverage at 254 or M | Fraction of positive advisory cases where the candidate set contains at least one acceptable skill |
  1071. | Top-one precision | Fraction of emitted advisory suggestions whose first skill is acceptable |
  1072. | Positive-case suggestion rate | Fraction of judged positive cases receiving an acceptable first suggestion; abstention and unavailable count as misses |
  1073. | Needless-suggestion rate | Fraction of judged no-match cases receiving any advisory suggestion |
  1074. | False abstention | Fraction of judged positive cases receiving a valid relevance abstention; operational failures are reported separately |
  1075. | Fit Brier score | Mean squared error of fit estimates on the declared set of independently judged skill/case pairs |
  1076. Candidate coverage asks whether the selector finds **any** acceptable option.
  1077. Optional set recall measures the fraction of all acceptable skills recovered and
  1078. is reported separately. This matters when a case has twenty acceptable skills
  1079. but the shortlist holds eight.
  1080. Benchmark retrieval and rerank stages run independently of the production gate
  1081. so low-gate cases do not disappear from the comparison. Reports also show the
  1082. actual gated decisions, operational failures over all attempted cases, and counts
  1083. of unjudged cases. Baselines include Quill-only lexical selection, the
  1084. cookbook-style selector, Choice-only, fit-only, and the default blend, with both
  1085. recent-context and latest-request-only inputs.
  1086. Reports bind results to dataset and split digests, roster content and visibility,
  1087. prompts, policy, runtime, and returned model identities/time ranges. Related
  1088. sessions and task variants stay in one split. An unversioned Jev alias limits
  1089. reproducibility even when the local sample and arithmetic replay exactly.
  1090. Robustness evaluation tests option-order permutations and opaque-ID renaming as
  1091. exact local parsing/scoring invariants. `sr eval --online --robustness` ranks
  1092. equivalent-whitespace, distracting-text, and hostile-instruction variants on the
  1093. remaining live budget. It reports top-one changes and paired admitted/top-M
  1094. coverage changes, with missing stage evidence counted as unknown. Low-gate wide
  1095. answers retain their intrinsic top-M coverage. Supply `--roster BASE`, then
  1096. `--decoy-roster EXPANDED` or `--lookalike-roster EXPANDED` to generate additional
  1097. paired arms from authorized file-backed manifests. Each expanded roster must
  1098. retain every baseline candidate unchanged and add invocable candidates; every
  1099. lookalike addition must duplicate a nonempty baseline description, ignoring
  1100. whitespace. These flags select existing skills and never create or edit them.
  1101. Choose irrelevant decoys using independent judgments; the command cannot infer
  1102. that an added skill is irrelevant merely because it was absent from the baseline.
  1103. `--robustness` also compares two prespecified wire changes: reversed skill
  1104. Choice option order and a bijection of opaque option handles, including rerank
  1105. fit-question keys. Both retain stable skill identities, content and eligibility,
  1106. freeze the baseline inventory, get separately bound wide previews, and consume
  1107. fresh responses from the same
  1108. attempt/runtime allowance after the request-text arms. Changed Jev decisions
  1109. are measured within the original family; identical probabilities are not required.
  1110. Unsupplied arms remain under `not_computed`. `--hostile-roster EXPANDED` tests
  1111. hostile instructions in added skill descriptions separately from hostile text
  1112. appended to a request. The
  1113. operator prespecifies that inventory; `sr` does not classify hostility or edit
  1114. descriptions. Baseline files remain unchanged, so this arm measures the effect of
  1115. added hostile candidates rather than rewriting a baseline skill's identity.
  1116. Configured roster arms run before request variants, after main rankings and
  1117. ablations. Their wide requests get separate network-free previews. With an
  1118. explicit baseline, all arms retain its
  1119. frozen roster identities, content and eligibility. Changed inventory is refused
  1120. before sending. Jev comparisons measure changes rather than demand identical
  1121. probabilities from a stochastic, candidate-dependent model.
  1122. New provider inputs require new responses and consume the live budget. Related
  1123. variants retain their original task family and split.
  1124. ### Spend the evaluation budget deliberately
  1125. `--sample-size` freezes a sampling manifest before selected cases run. Unless
  1126. replaying a recorded draw, omit `--seed` for a fresh seed from trusted OS randomness.
  1127. The unit is one representative per independent task family, selected by a
  1128. recorded rule before inspecting evaluated outcomes. The manifest records:
  1129. - The consented frame, split digests, and observable strata, such as normal versus
  1130. overflow retrieval and complete versus degraded input.
  1131. - Stratum population and sample sizes, selected IDs, and justified inclusion
  1132. probabilities for random selection without replacement within a stratum;
  1133. diagnostic selections leave these probabilities unknown.
  1134. - The RNG algorithm, version, seed, randomization provenance, design status,
  1135. policy/model identities, and label/request budgets needed to replay selection.
  1136. A manually fixed seed supplies reproducibility, not evidence for the claimed
  1137. inclusion probabilities or design-based uncertainty. Such a run is diagnostic
  1138. unless its input carries a matching prior randomization manifest; replaying that
  1139. manifest is the same draw, not an additional independent sample. Imported
  1140. provenance remains supplied/unverified. Full-frame evaluation needs no sampling
  1141. randomness. Guarantees assume uniform selection within strata; the recorded PRNG
  1142. and bounded-draw checks support the implementation without proving exact
  1143. uniformity over every possible seed. Never redraw to obtain favorable results.
  1144. Every represented stratum receives a positive allocation. Gate-abstained,
  1145. operationally failed, and unknown-metadata cases remain in the sampling frame.
  1146. If the budget cannot cover it, the report must use a declared narrower population
  1147. or a predeclared stratum-merging rule.
  1148. Oversampling rare cases helps inspection, but changes the sample's composition.
  1149. For error indicators or another declared loss in `[0,1]`, the report weights each
  1150. stratum by its share of the original frame:
  1151. ```text
  1152. weight_h = population_h / total_population
  1153. estimated_loss = sum_h(weight_h * sampled_mean_loss_h)
  1154. ```
  1155. For example, take a frame of 900 routine cases and 100 overflow cases, with 50
  1156. sampled from each. If the observed error rates are 2% and 20%, respectively:
  1157. | Calculation | Error estimate |
  1158. |---|---:|
  1159. | Unweighted sample: `(0.02 + 0.20) / 2` | 11% |
  1160. | Frame-weighted: `0.9 × 0.02 + 0.1 × 0.20` | 3.8% |
  1161. The design-weighted mean is a Horvitz–Thompson estimate for that declared frame.
  1162. It does not establish risk on unseen projects. Weighted precision is a ratio
  1163. estimate, with its own uncertainty requirements; weighted rows cannot be treated
  1164. as ordinary binomial counts. [Sampling estimator reference](https://www150.statcan.gc.ca/n1/pub/12-001-x/2019001/article/00007/02-eng.htm)
  1165. Fixed-sample bounded-loss reports use a conservative per-stratum sampling bound
  1166. with a shared error budget; a fully enumerated stratum contributes its exact mean.
  1167. Missing labels receive lower/upper loss assignments rather than silently shrinking
  1168. the denominator. Unknown inclusion probabilities or a changed frame invalidate
  1169. the corresponding estimation claim. These reports supplement the separate
  1170. promotion cohorts below.
  1171. Near-threshold decisions, wide/rerank disagreements, overflow misses, and sparse
  1172. categories also enter a diagnostic review queue. That queue helps choose examples
  1173. to inspect; it is kept out of representative holdout denominators. Optional
  1174. allocation based on pilot variance and labeling cost is compared at equal label,
  1175. token, and request budgets before adoption. Sampling never changes which skill
  1176. the live agent is instructed to execute or adds work to the hook path.
  1177. ### Require evidence before promoting a policy
  1178. The acceptance policy fixes these requirements before tuning. They are evaluation
  1179. thresholds, not claimed benchmark results:
  1180. | Check | Requirement |
  1181. |---|---|
  1182. | Relevance cohort | At least 300 adjudicated primary cases from independent task families: at least 150 positive, 100 no-match, and 50 near-miss cases across those groups |
  1183. | Overflow candidate coverage at 254 | At least 98%, with at least 50 positive overflow cases |
  1184. | Shortlist candidate coverage at M | At least 95% on positive advisory cases |
  1185. | Top-one precision | At least 90%, with the lower endpoint of a 95% interval at least 80% |
  1186. | Positive-case suggestion rate | At least 80%; abstention and unavailable remain misses |
  1187. | Needless suggestions | At most 5%, with the upper endpoint of a 95% interval at most 10% |
  1188. | New-harm risk | One-sided 95% upper bound at most 2%, from a separate controlled paired cohort |
  1189. | Operational fallback | At most 5% over at least 500 representative hook invocations, including provider outages |
  1190. Primary relevance rates use two-sided 95% Wilson intervals, with one preselected
  1191. case per independent task family. Additional variants do not inflate that
  1192. denominator. Deliberately oversampled benchmark categories do not establish
  1193. production prevalence or production precision.
  1194. Independent families alone do not establish a binomial sampling model or remove
  1195. selection bias. Freeze the population, sampling design, endpoint model, and
  1196. interval method before inspecting results. Wilson intervals are nominal
  1197. model-based intervals, not exact coverage guarantees for an arbitrary fixed or
  1198. quota-selected benchmark. Stratified probability samples use design-aware bounds
  1199. and separate ratio methods; weighted rows are not binomial counts. Diagnostic
  1200. rates without a justified inferential model cannot pass promotion gates.
  1201. The harm comparison pairs advice-enabled and baseline runs from equivalent
  1202. isolated snapshots, with identical permissions and budgets, randomized arm order,
  1203. and blinded outcome review. A task family counts as new harm if any planned paired
  1204. run is harmful with advice and non-harmful without it. Missing or unjudgeable
  1205. outcomes are also flagged for this conservative harm-or-unresolved gate. Count
  1206. each family once and report judged harm and unresolved-only outcomes separately.
  1207. Improvements on other tasks do not cancel flagged units; the net harm difference
  1208. is reported separately.
  1209. The gate uses a one-sided 95% Clopper–Pearson upper bound when the prespecified
  1210. binomial model is justified; otherwise it needs a prespecified design-valid
  1211. bound. With zero flagged events in `n` independent identically distributed
  1212. task-family units, the binomial upper bound is
  1213. `1 - 0.05^(1/n)`: about **2.95% for 100 units** and **1.98% for 150 units**.
  1214. Repeating one task 150 times does not provide 150 independent units, and zero
  1215. observed events does not justify a zero uncertainty interval.
  1216. [Exact binomial interval reference](https://itl.nist.gov/div898/software/dataplot/refman2/auxillar/exacbino.htm)
  1217. Insufficient independent cases, no emitted suggestions, or missing subgroup
  1218. coverage means the requirement is not established. The harm cohort is separate
  1219. from the relevance holdout, and its result applies to the declared population
  1220. and experiment.
  1221. ### Monitor repeated evaluations without resetting the evidence
  1222. Optional sequential monitoring tracks a **new-harm-or-unresolved composite**
  1223. from prospectively ordered, controlled task-family pairs. A finalized unit is
  1224. flagged for judged new harm or an outcome still unjudgeable at its frozen deadline.
  1225. Each family counts once; reports separate judged harm, unresolved-only, and
  1226. unflagged units. The null bounds the conditional probability of this composite
  1227. at 2%, using a fixed mixture of alternatives at 5%, 10%, and 20%. Under that
  1228. conditional-risk assumption, its evidence threshold controls the chance of ever
  1229. raising a false alarm across repeated looks.
  1230. [Time-uniform evidence reference](https://arxiv.org/abs/1808.03204)
  1231. This is not a false-alarm guarantee for true harm alone under arbitrary missing
  1232. labels. With zero actual harm but 5% unresolved units, the composite already
  1233. exceeds the 2% null. An alarm can identify inadequate evidence without
  1234. demonstrating harmful advice.
  1235. The mixture starts at one and alarms at `1 / alpha_monitor`; a monitor allocated
  1236. `alpha_monitor = 0.05` has threshold **20**. Each finalized unit contributes once,
  1237. in the predeclared order. Pending labels wait until their adjudication deadline.
  1238. Labels can change before finalization; the analyzed label version is then frozen.
  1239. A later correction invalidates the inferential epoch: preserve its original
  1240. trace, alarms, and spent alpha. Corrected historical recomputation is descriptive
  1241. only. Resume inference with fresh prospective units and a fresh allocation unless
  1242. a separately proven revision-aware method is available.
  1243. The total false-alarm budget is allocated across monitors and restarts in advance.
  1244. A restart does not replenish it. State is bound to the rubric, baseline,
  1245. policy/model cohort, and ordering; missing or corrupt state or an unresolved
  1246. label revision reports `unmonitored`.
  1247. An alarm blocks further policy promotion and recommends the frozen baseline or
  1248. shadow mode. Configuration changes still require the explicit apply path.
  1249. The absence of an alarm does not establish that the risk ceiling is met, and an
  1250. evidence value is not a posterior probability. The monitor cannot be applied to
  1251. an arbitrary fixed sample merely because its sampling estimate is valid, or infer
  1252. harm from skill adoption. It runs on evaluation evidence without launching
  1253. production experiments or making extra network calls.
  1254. `sr eval --explain` includes mathematical explanation cards with equations,
  1255. substituted values, assumptions, and what further evidence would change the
  1256. conclusion. These belong in reports; hook advice stays short.
  1257. ### Opt-in retrieval and description experiments
  1258. The evaluation tools support three bounded experiments. They use held-out cases
  1259. and the existing request, disclosure, and latency budgets; they do not silently
  1260. replace the default selector.
  1261. | Experiment | Scope and comparison |
  1262. |---|---|
  1263. | Relevant skill passages | Compare the fixed opening excerpt with Quill-selected heading-delimited passages inside an already selected skill, retaining its purpose/restriction prefix and the 700-character body budget |
  1264. | Multiple overflow query views | Compare the combined query with up to three Quill views: current request, task anchor, and recent error; deduplicate and fuse their rankings into at most 254 candidates |
  1265. | Description overlays | Compare an original description with a proposed evaluation-only overlay against user-supplied positive and near-miss cases |
  1266. Passage retrieval reads only already authorized, bounded skill bytes, scans them
  1267. for secrets before excerpting, and retains heading, position, and version. A
  1268. successful query with no passage match uses the declared opening excerpt; parser,
  1269. fuel, and index errors retain typed failure behavior. It loads no referenced
  1270. files, scripts, or additional candidates.
  1271. Multiple query views share aggregate term, character, fuel, memory, and deadline
  1272. budgets. Each returns at most 254 matches. A pinned reciprocal-rank rule such as
  1273. `sum_v 1/(60 + rank_v)` uses ranks starting at one, zero contribution for absent
  1274. hits, equal view weights, and stable skill-ID ties. Identical normalized views
  1275. are deduplicated. An attempted view failure invalidates the result; an empty
  1276. union remains `retrieval-empty`. Fusion scores are retrieval heuristics, not
  1277. Jev probabilities, and rosters of 254 or fewer still bypass the prefilter.
  1278. Description overlays bind to source content digests and are unavailable to
  1279. hooks. A changed description requires a new request fingerprint and responses;
  1280. compatible unchanged inputs may use recorded answers. Reports compare coverage,
  1281. false suggestions, truncation, unknown cases, and attempts. Stale or ambiguous
  1282. sources are rejected. The experiment exports a reviewable suggestion without
  1283. rewriting skills or changing loadability. Untouched task families supply final
  1284. validation; a better prose rubric score alone cannot justify promotion.
  1285. ## Control Requests And Interruptions
  1286. ### Share an HTTP-attempt allowance across sessions
  1287. An optional trusted-user allowance bounds HTTP attempts across local `sr`
  1288. processes in a configured user/endpoint scope and named time window. It includes
  1289. retries and live evaluation, and intersects each invocation and batch limit.
  1290. Setup explicitly initializes the allowance and preflight reports the maximum
  1291. attempts. It is neither a cross-machine billing limit nor a hard monetary cap;
  1292. cost estimates use separately versioned prices and retain unknown usage.
  1293. ```bash
  1294. sr budget # Inspect local allowance and health
  1295. sr budget --max-attempts 100 --window 1h # Preview scope and window semantics
  1296. sr budget --max-attempts 100 --window 1h --apply # Explicitly configure this limit
  1297. ```
  1298. The allowance accepts 1–10,000 admissions in fixed one-hour UTC windows. The preview
  1299. names the user/endpoint scope, exact window boundaries, charged attempts, and
  1300. remaining allowance. This is not a rolling-hour cap: adjacent windows can each
  1301. consume their allowance close to the boundary. Configuration changes preserve
  1302. still-applicable charges. Accounting shares the 64 MiB cache/coordinator budget,
  1303. but unexpired charges cannot be evicted; exhausted storage withholds new requests.
  1304. Under the trusted configuration lock, setup first publishes a durable activation
  1305. intent and new guard generation, then prepares accounting while preserving
  1306. charges, then publishes the matching ready generation. Missing or mismatched
  1307. state after the intent blocks admission. Retrying `--apply` resumes that intent;
  1308. before it is visible, the old policy remains in force. Each attempt checks the
  1309. active generation at admission, including processes started earlier. Already
  1310. admitted requests cannot be recalled. Inspection itself makes no network requests.
  1311. The guard lives in `sr/allowance.toml` under your configuration directory, and
  1312. its accounting in `allowance.sqlite3` in the private `sr` cache directory, which
  1313. `--no-cache` does not relocate. Each endpoint origin, port included, has its own
  1314. count.
  1315. Persistent attempts hold the same bounded lock as setup while rereading guard
  1316. configuration, validating the accounting generation, and debiting. They release
  1317. it before HTTP; lock contention cannot turn into an unbounded hook wait. With
  1318. `--no-persist`, the final trusted-config read is the local admission boundary
  1319. only when no guard is active. An intent or enabled guard refuses the request;
  1320. a later activation cannot revoke a stateless request already admitted. Retries
  1321. must recheck, not reuse startup-time settings.
  1322. Each attempt is atomically reserved/debited before transmission. Its single-use
  1323. permit binds an attempt ID, request/endpoint, guard generation, window, and
  1324. remaining monotonic deadline. No permits are preallocated for later windows.
  1325. An expired or wrong-window permit is discarded without refund; a retry needs
  1326. fresh admission. A scheduling pause can still separate the last check from
  1327. network I/O, so the limit counts admissions, not wire-arrival or billing times.
  1328. A timeout or
  1329. crash never refunds a request that may have been sent. Restarting, evicting a
  1330. cache entry, or pruning ordinary history cannot reset accounting. Missing,
  1331. corrupt, or busy enforcement state withholds new requests; local explicit
  1332. resolution and valid cache hits remain available. Clock anomalies retain charges
  1333. until reconciliation. With `--no-persist`, an enabled shared allowance cannot
  1334. admit provider attempts; local explicit resolution remains usable.
  1335. This protected accounting and best-effort cooldown/lease state are separate from
  1336. the optional observation ledger; `--no-ledger` does not disable the allowance.
  1337. The debit must be durably committed before sending. The initial backend verifies
  1338. WAL with `synchronous=FULL`; if that work cannot fit the deadline, it withholds
  1339. the attempt. An asynchronous flush cannot satisfy the accounting guarantee.
  1340. A provider circuit breaker limits repeated outage traffic. Three consecutive
  1341. transient failures open a 30-second cooldown. Failed half-open probes double the
  1342. cooldown to a five-minute cap, with a single fenced probe owner. A longer valid
  1343. provider `Retry-After` prevents early attempts without making a hook wait for it.
  1344. A half-open probe is the next permitted real request, with the same network
  1345. authorization, deadline, and attempt debit; it adds no separate health call or
  1346. background probe. Valid successful responses restore service and
  1347. reset the local failure streak.
  1348. Circuit state lives in `breaker.sqlite3` in the private `sr` cache directory,
  1349. per endpoint origin, and a cooldown refusal is decided before the allowance
  1350. charges anything. With `--no-persist`, each process keeps its own circuit;
  1351. when the shared store is expected but unusable, the result carries a
  1352. `breaker-process-local` warning. This build has no safe shared credential
  1353. identity, so authentication failures never open the circuit: each stays with
  1354. its own invocation and is not retried automatically.
  1355. A valid response clears a stored `Retry-After`. Live `sr eval` batches share
  1356. the same per-origin circuit, so provider failures during an evaluation also
  1357. cool down hook traffic to that endpoint.
  1358. Responses carry their breaker generation: a late obsolete response cannot close
  1359. a newer circuit or release its successor's probe lease. Authentication pauses
  1360. are scoped to a verified credential profile/generation so a bad key cannot block
  1361. another profile's good key. Without a safe shared identity, auth failures remain
  1362. invocation-local with no automatic retry. Never store raw or reversible keys;
  1363. rotating credentials does not reset the user/endpoint allowance. Authentication
  1364. or configuration errors require an explicit retry or relevant configuration
  1365. change. Local configuration/input errors, exclusions, lexical misses, and
  1366. relevance abstentions are not endpoint failures.
  1367. Client-internal automatic retries are disabled. If the wide request fits the
  1368. allowance but rerank does not, return unavailable with partial evidence and usage;
  1369. the wide winner cannot become a completed recommendation.
  1370. Budget refusals, circuit refusals, provider outages, and relevance abstentions
  1371. are reported separately. When circuit persistence is unavailable, protection is
  1372. reported as process-local. Outages never substitute stale advice or another
  1373. inference provider.
  1374. | Admission refusal | CLI result |
  1375. |---|---|
  1376. | Attempt allowance exhausted | `unavailable / request-budget`, exit `4` |
  1377. | Provider circuit prevents a call | `unavailable / provider-cooldown`, exit `4` |
  1378. | Required accounting unusable | `unavailable / budget-state`, exit `4` |
  1379. The hook translates each refusal into quiet fallback.
  1380. ### Snooze advice without changing usefulness
  1381. Ordinary advisory abstentions are silent. When a valid suggestion is unwelcome
  1382. for now, snooze it within the event's verified workspace, session, and agent
  1383. branch:
  1384. ```bash
  1385. # Preview the exact scope, expiry, and effect before applying.
  1386. sr snooze EVENT_ID --skill SKILL_ID --for 30m
  1387. sr snooze EVENT_ID --skill SKILL_ID --for 30m --apply
  1388. # Or mute all advisory candidates, then clear that scope's snoozes explicitly.
  1389. sr snooze EVENT_ID --all --for 30m --apply
  1390. sr snooze EVENT_ID --clear --apply
  1391. ```
  1392. Choose exactly one of `--skill`, `--all`, or `--clear`. The first two require a
  1393. duration; clearing a snooze does not accept one.
  1394. Snoozes are trusted-user configuration entries with backup/conflict checks, a
  1395. maximum of 128 entries, and durations from one minute to 24 hours. Missing or
  1396. ambiguous event attribution cannot create a broadly scoped mute. Doctor shows
  1397. active entries and expiry anomalies; uncertain expiry remains muted until
  1398. resolved or explicitly cleared.
  1399. They live in `sr/snoozes.toml` under your configuration directory, and the
  1400. event's recorded workspace, session and agent branch bound each one. A snooze
  1401. applied while a ranking is in flight withholds that ranking's advice at
  1402. publication; a request already sent may still incur cost, and advice already
  1403. published cannot be recalled.
  1404. Snoozed skills remain visible in explanations but leave advisory eligibility
  1405. before retrieval. Muting every advisory candidate skips Jev. Explicit skill
  1406. requests still resolve normally. Ranking reads these controls with `--no-ledger`
  1407. or `--no-persist`, as ordinary configuration, and never writes expiry cleanup.
  1408. A snooze changes neither usefulness labels nor priors; ignored suggestions do
  1409. not create snoozes.
  1410. Automatic suppression across separate user turns is a distinct opt-in experiment.
  1411. It requires unchanged task evidence, skill version, and policy within a bounded
  1412. interval, and restores advice when renewed instructions or relevant tool changes
  1413. invalidate that match. Publication suppression is recorded separately from
  1414. ranking and exposure; missing history cannot prove advice was previously delivered.
  1415. ## Agent Hooks
  1416. The Claude Code integration uses `UserPromptSubmit` and the dedicated
  1417. `sr hook claude` protocol boundary. Advisory output uses the harness's
  1418. `hookSpecificOutput` envelope:
  1419. ```json
  1420. {
  1421. "hookSpecificOutput": {
  1422. "hookEventName": "UserPromptSubmit",
  1423. "additionalContext": "Suggested skill for the next step: rust-test-triage. Use it only if it fits the user's request and current instructions."
  1424. }
  1425. }
  1426. ```
  1427. The normal hook names at most one locally validated skill. Multiple explicitly
  1428. requested skills remain user requests, not adaptive top-three suggestions.
  1429. A complete explicit list must fit the 1,024-character hook limit; an oversized
  1430. list produces quiet fallback instead of silently dropping requests. Manual-only
  1431. references never become instructions for autonomous invocation.
  1432. Ordinary valid abstentions produce no injected text. A short no-additional-skill
  1433. message is a separate opt-in experiment. Operational failures or unresolved
  1434. target visibility also stay quiet. A partial roster can support a scoped positive
  1435. recommendation when the target and its restrictions are verified; it cannot
  1436. support a global no-match message.
  1437. **The hook never blocks the agent on a recommendation failure.** It emits no
  1438. blocking decision fields and maps errors found before publication to empty
  1439. stdout, a sanitized stderr diagnostic, and exit zero. Shadow mode suppresses both suggestions and
  1440. abstention text while retaining permitted local observations.
  1441. The complete bounded envelope is rendered before publication. If stdout fails
  1442. partway through, already-written bytes cannot be retracted: delivery remains
  1443. unknown, and `sr` does not append replacement JSON or retry the whole message.
  1444. Only a complete successful write can count as an emission.
  1445. ```bash
  1446. sr install-hook claude # Preview the exact settings change
  1447. sr install-hook claude --apply # Merge the managed entry with a backup
  1448. sr hook claude --shadow # Explicitly keep the hook observational
  1449. sr uninstall-hook claude --apply # Remove only the managed entry
  1450. ```
  1451. Installation uses a trusted absolute executable path, escaped arguments, and a
  1452. harness timeout above the ranker's deadline. Repeated installation is idempotent;
  1453. unrelated settings are preserved, and concurrent or malformed edits are reported.
  1454. Backups stay in owner-only private state. Cooperating installers serialize their
  1455. edits, but simultaneous edits by another program are unsupported: atomic rename
  1456. alone cannot prevent every lost update. Raising the internal deadline beyond the
  1457. installed outer timeout requires reinstalling the hook.
  1458. Preview and apply both report what a recorded shadow trial still needs, and
  1459. change none of it: an initialized ledger (`sr ledger init`), trusted network
  1460. consent (`[network] enabled = true` in your user `sr/config.toml`, since a
  1461. one-shot `--allow-network` never reaches a hook), and the selected provider's
  1462. credential in the environment Claude Code gives the hook. The report never
  1463. prints the credential or configuration contents. `--settings-file FILE` targets
  1464. a specific settings file.
  1465. Other harnesses can supply versioned normalized context through
  1466. `sr rank --context FILE`. Native integration support is enumerated by capabilities;
  1467. a post-turn notification is not interchangeable with a pre-turn recommendation hook.
  1468. ## Inline TUI
  1469. ```bash
  1470. sr tui
  1471. ```
  1472. The optional FrankenTUI display uses an inline layout of roughly nine rows in a
  1473. dedicated pane or terminal. It shows relative rank score, fit, source, freshness,
  1474. and decision status. Small terminals use fewer rows and readable text.
  1475. | Key | Action |
  1476. |---|---|
  1477. | `1`–`5` | Select a locally resolved skill target |
  1478. | `r` | Request a refresh |
  1479. | `w` | Toggle bounded transcript watch |
  1480. | `e` | Show distributions and supporting evidence |
  1481. | `q` | Exit and restore terminal state |
  1482. Selection never invokes a loader or shell command. Machine-readable selection
  1483. output stays separate from terminal rendering. Watch mode permits one active
  1484. ranking per session, coalesces changes, and enforces a minimum five-second interval.
  1485. A superseded result cannot replace a newer generation's display.
  1486. The TUI stays open until you exit. Each accepted refresh starts a new bounded
  1487. evaluation; the three-second ranking deadline is not the lifetime of the viewer.
  1488. ## Architecture
  1489. ```mermaid
  1490. flowchart TD
  1491. A[Exact session and trusted configuration] --> B[Capture context and visible roster]
  1492. B --> C[Normalize and resolve directives before redaction and budgeting]
  1493. C -->|explicit request| X[Locally resolved explicit result]
  1494. C -->|exclusions or snoozes remove all| N[Abstain]
  1495. C -->|advisory candidates| Q{More than 254 eligible skills?}
  1496. Q -->|no| K{Exact valid cache?}
  1497. Q -->|yes| QR[Quill BM25: up to 254 actual matches]
  1498. QR -->|matches| K
  1499. QR -->|empty or failed| U[Unavailable or quiet hook fallback]
  1500. K -->|hit| P[Apply current eligibility and policy]
  1501. K -->|miss| BUD[Check network authorization, deadline, attempts, and cooldown]
  1502. BUD -->|admitted| W[TypeSafe Jev: wide Choice with none and gates]
  1503. BUD -->|refused| U
  1504. W -->|low need| N
  1505. W -->|continue| RBUD[Recheck admission for rerank]
  1506. RBUD -->|admitted| R[TypeSafe Jev: detailed rerank with none and fits]
  1507. RBUD -->|refused| U
  1508. R --> P
  1509. P --> O[JSON, table, hook, or TUI]
  1510. W -->|failure| U
  1511. R -->|failure| U
  1512. X --> O
  1513. N --> O
  1514. O --> L[(Bounded local metadata)]
  1515. U --> O
  1516. L --> F[Observe loads and record explicit judgments]
  1517. F --> E[Replay, weighted evaluation, and optional calibration]
  1518. D[Consented cases and independent labels] --> S[Freeze task families and sampling manifest]
  1519. S --> E
  1520. E --> V[Versioned reports and explicit policy apply]
  1521. ```
  1522. | Component | Responsibility |
  1523. |---|---|
  1524. | `context/` | Exact source selection, Claude and normalized adapters, optional cass, context profiles, and project signals |
  1525. | `privacy/` | Local redaction, disclosure receipts, and trusted network, root, field, and persistence policy |
  1526. | `roster/` | Harness visibility, bounded parsing, stable identities, snapshot diffs, and Quill lexical retrieval |
  1527. | `jev/` | Asupersync HTTPS, typed validation, questions, eligibility, scores, request admission, and circuit protection |
  1528. | `ledger/` | SQLite transactions, observations, judgments, sampling manifests, calibration, and optional priors/monitoring |
  1529. | `output/` | Versioned JSON, stage explanations, demo/replay rendering, human table, Claude protocol, and optional TUI |
  1530. | `cache.rs` | Separate request and decision fingerprints, TTL, and revalidation |
  1531. | `hook_install.rs` | Managed configuration preview, apply, backup, and rollback |
  1532. One Rust package contains `sr` and reusable pure pipeline components. Asupersync
  1533. owns task lifetimes, deadlines, HTTP/TLS, and deterministic lab replay. Quill
  1534. provides bounded in-memory lexical retrieval. SQLite persistence uses `rusqlite`
  1535. with bundled SQLite. Concurrent WAL stores require a verified SQLite
  1536. 3.51.3 or later to include the [WAL-reset corruption fix](https://www.sqlite.org/wal.html#walresetbug);
  1537. the actual linked engine is checked separately from the Rust crate version.
  1538. FrankenTUI is optional.
  1539. **The inference engine is the selected provider's Jev implementation.** The
  1540. surrounding Rust code gathers and protects context, constructs typed questions,
  1541. validates Jev's answers, and turns them into useful agent recommendations. Quill
  1542. retrieval, caching, and the ledger support that engine; they do not replace it.
  1543. Selected parsing and redaction code can be adapted from
  1544. [meta_skill](https://github.com/Dicklesworthstone/meta_skill) with source provenance
  1545. and license notices. SkillRanker does not invoke its CLI, link its application,
  1546. read its private database, or write outcomes back to it.
  1547. ### Evaluation numerics and data preparation
  1548. Evaluation tooling draws on three Rust numerical/data projects through narrow,
  1549. versioned interfaces:
  1550. | Project | Role in evaluation |
  1551. |---|---|
  1552. | [FrankenSciPy](https://github.com/Dicklesworthstone/frankenscipy) | Wilson and exact binomial intervals, beta-distribution calculations, and stable log-sum-exp |
  1553. | [FrankenNumPy](https://github.com/Dicklesworthstone/franken_numpy) | Seeded sampling without replacement and rejection-based shuffling |
  1554. | [FrankenPandas](https://github.com/Dicklesworthstone/frankenpandas) | Evaluation-table preparation, duplicate detection, grouping, and checked joins |
  1555. These components stay behind evaluation/tooling boundaries; ordinary hooks do
  1556. not require the numerical stack. Reports identify the actual backend and source
  1557. revision. A compatibility wrapper that falls back to another implementation does
  1558. not count as execution by the named backend.
  1559. Numerical adapters preserve interval sidedness and zero/all-event endpoints.
  1560. The harm gate uses a one-sided 95% upper limit, while relevance reports use
  1561. two-sided 95% intervals. Sampling records the RNG algorithm and version as well
  1562. as the seed. Data preparation checks one-to-one or many-to-one join cardinality,
  1563. resolves revised labels before aggregation, and retains unmatched labels as
  1564. unknown so a duplicated row cannot inflate the sample size.
  1565. ## Privacy And Local State
  1566. **Fresh Jev evaluations send redacted context and skill excerpts to the selected
  1567. provider.**
  1568. Networking requires a trusted setup choice. Project files cannot enable it just
  1569. because an API key is present. `--offline` guarantees zero network requests and
  1570. can use local explicit resolution or an exact valid cache entry.
  1571. Offline and dry-run input uses direct transcripts or normalized context. An
  1572. explicit cass source in those modes returns `unsupported-source-mode` (exit `7`)
  1573. with a hint to choose a supported local source. `--offline` conflicts with
  1574. `--allow-network`.
  1575. | Control | Effect |
  1576. |---|---|
  1577. | `--dry-run` | Preview exact redacted wide-request bytes for a stateless run; no network or persistent state access |
  1578. | `--context-profile minimal` | Keep the request, indispensable task anchor, explicit constraints, and candidate material required by each ranking stage; omit optional history, tool bodies, and dirty paths |
  1579. | `--no-tools` | Remove tool arguments and results from provider context |
  1580. | `--no-cache` | Disable cache reads and writes |
  1581. | `--no-ledger` | Disable all ledger and ingestion-cursor reads/writes; use transient evidence |
  1582. | `--no-persist` | Also disable persistent cache, cursors, locks, and other local state |
  1583. | `--offline` | Disallow all network activity |
  1584. The default context profile is `standard`. Choose a profile through trusted user
  1585. configuration or the CLI. The minimal profile still includes the candidate
  1586. descriptions and bounded skill-body excerpts needed by the respective Jev passes;
  1587. it reduces session disclosure without stripping the material used to compare
  1588. skills. `--no-tools` further restricts either profile, and workspace settings
  1589. cannot widen a trusted minimal profile.
  1590. ```bash
  1591. sr rank --context scratch/context.json --context-profile minimal --dry-run
  1592. ```
  1593. Dry-run and explain include a disclosure receipt showing source categories,
  1594. included/omitted counts, truncation, and redaction counts without matched secret
  1595. fragments. Minimal context returns unavailable if an omitted tool result or
  1596. attachment is essential. Local load observations remain separate; the chosen
  1597. profile and receipt schema enter request provenance. Smaller disclosure alone
  1598. does not establish equivalent recommendation quality.
  1599. A second-stage dry run requires explicit shortlist IDs
  1600. (`--dry-run --shortlist-ids ID,...`) or a validated recorded wide answer. It
  1601. cannot know a model's shortlist without that evidence. The preview
  1602. corresponds to `--no-persist`; a persistent run can include additional historical
  1603. evidence and therefore produce a different payload.
  1604. These persistence controls govern `sr`'s state stores; configured inputs and
  1605. ordinary configuration still require file reads, including explicit snooze
  1606. controls. With `--no-persist`, leases and cooldowns are process-local, so
  1607. cross-process coordination is unavailable. An enabled shared attempt allowance
  1608. requires durable enforcement, so it withholds provider calls under `--no-persist`
  1609. while leaving local explicit resolution usable. Explicit case capture is
  1610. incompatible with `--no-persist`.
  1611. Local data uses platform directories, including `$XDG_DATA_HOME/sr` on Linux
  1612. with `~/.local/share/sr` as the fallback. Database and cache files are owner-only.
  1613. Raw transcripts and request bodies are not retained by default. Event metadata
  1614. has a default 30-day logical retention policy; response-cache entries expire after
  1615. at most ten minutes. Expiry excludes data from ordinary use; physical cleanup is
  1616. an explicit ledger operation outside the hook. It is not a secure-erasure guarantee.
  1617. Hooks do not initialize or migrate the ledger or enabled allowance accounting.
  1618. `sr ledger init` creates an absent observation ledger; repeating it preserves a
  1619. compatible existing store and never resets its history or cursors.
  1620. Migration previews and `--apply` use backups that include committed SQLite WAL
  1621. state. Ordinary ranking remains usable with degraded optional persistence when
  1622. storage is absent, busy, or full. An enabled shared allowance with unavailable
  1623. accounting withholds new HTTP attempts; a requested case capture must either
  1624. complete or report failure. The default quotas are 256 MiB for the ledger,
  1625. sidecars, and backups, and 64 MiB for cache/coordination state; reaching a quota
  1626. stops optional recording rather than growing without a bound. Recording stops
  1627. early enough to preserve a bounded maintenance reserve. Doctor reports usable
  1628. capacity and headroom; administrative operations preflight backup, WAL, temporary
  1629. space, and available disk. If sufficient space is unavailable, they fail before
  1630. mutation with a concrete recovery step instead of exceeding quota or discarding
  1631. the only recoverable copy.
  1632. Full roster membership snapshots share the ledger quota and retention rules;
  1633. retained events reference snapshots deduplicated per workspace and harness, so
  1634. one ledger serves every workspace that sees the same skills. Missing snapshot
  1635. evidence disables unsupported historical corrections while ranking remains
  1636. usable.
  1637. Writes verify the store's incarnation and schema/data generation in their
  1638. transaction. Clearing history advances the generation, preventing older in-flight
  1639. work from repopulating it; new invocations can record afterward. Required stale
  1640. writes fail, while optional recording is skipped visibly. Clearing history does
  1641. not reset request allowances, disable future recording, or unlink a live database.
  1642. Requests use HTTPS with credential-scoped endpoints and redirects disabled.
  1643. Redaction covers outgoing roster excerpts as well as conversation fields, but
  1644. it cannot identify every piece of confidential prose. Review dry-run output
  1645. before sharing it. There is no cross-user telemetry or remote feedback sink.
  1646. Saved replay cases and roster snapshots are explicit private exports; hooks do
  1647. not capture case bodies by default. Demo and replay do not turn fixture or
  1648. historical answers into live service evaluations.
  1649. ## Performance
  1650. Each one-shot rank or hook invocation has a **three-second total budget**,
  1651. beginning at process entry and reserving the final 200 ms for output and cleanup.
  1652. The Claude harness timeout is initially four seconds. TUI/watch refreshes each
  1653. receive a new evaluation budget; live batch evaluation also has its own total
  1654. runtime and HTTP-attempt caps. Local maintenance uses bounded batches and
  1655. transactions rather than the inference timeout.
  1656. | Path | Engineering target |
  1657. |---|---|
  1658. | Exact cache hit | p95 at or below 100 ms |
  1659. | Warm hook requiring the network | p50 at or below 600 ms; p95 at or below 1,500 ms |
  1660. | Cold CLI or cass discovery | The same configurable deadline, with stage timings |
  1661. These targets are not remote-service guarantees or measured benchmark results.
  1662. Process startup, cold TLS, roster size, discovery, and provider load all matter.
  1663. Quill index construction and search are timed separately, including on exact
  1664. cache hits that must re-establish the current candidate set. Include roster
  1665. revalidation before publication in those timings. Logical index allocation
  1666. budgets do not replace measurements of actual process memory.
  1667. Input capture and roster discovery overlap after identity is established.
  1668. The two inference stages remain sequential. Low-need decisions skip the rerank.
  1669. The normal budget is two logical requests and at most four HTTP attempts total;
  1670. retries and `Retry-After` consume the same remaining deadline. The explicit retry
  1671. loop accounts for every attempt; client-internal retries are disabled. A response
  1672. arriving after the deadline cannot be published as a timely result.
  1673. Cache keys cover exact request inputs and current decision policy. New tool
  1674. evidence, compaction, skill-content changes, exclusions, model/endpoint identity,
  1675. and session changes can invalidate reuse. Context profiles, excerpt/query
  1676. strategies, and effective snoozes enter the appropriate request/decision
  1677. fingerprints. Stale results cannot drive hook output.
  1678. Cache hits report zero new requests and tokens; unanswered attempts retain an
  1679. unknown-usage marker rather than being counted as free.
  1680. Concurrent consumers can share an exact validated response while retaining
  1681. separate decisions, eligibility checks, deadlines, and exposure records. Only
  1682. the owner incurs each uniquely identified provider attempt; followers report
  1683. zero new usage, with unavailable owner accounting marked unknown. Cross-process
  1684. result sharing requires the response cache. `--no-cache` permits only in-process
  1685. in-flight sharing; coordination state cannot hide response bodies. Shared
  1686. allowances/cooldowns remain effective. `--no-persist` disables cross-process
  1687. sharing entirely.
  1688. Each stage records cache provenance and requested/returned model identity. With
  1689. an unversioned alias, a cached wide answer is not combined with a fresh rerank;
  1690. the pair is refreshed together or remains unavailable. Different immutable
  1691. returned revisions also invalidate the pair. Offline mode requires a complete,
  1692. valid result and otherwise returns `cache-miss` (exit `11`).
  1693. Performance reports include p50/p95/p99, cold startup and TLS, cache/network
  1694. strata, all-invocation fallback rates, memory, requests, tokens, and unknown usage.
  1695. Successful-call latency alone cannot hide a high timeout rate. Model times and

Lines 2,001–2,144 are not shown. The link to the file on GitHub has every line.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.