Skip to content
Back to search

ypollak2

SkillGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 17 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/ypollak2/llm-router with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/ypollak2/llm-router'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ypollak2--llm-router/badge.svg)](/listings/ypollak2--llm-router)

HTML

<a href="/listings/ypollak2--llm-router"><img src="/api/listings/ypollak2--llm-router/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pip install llm-routing        # installs the `llm-router` command
llm-router install             # wire up Claude Code (default host)
llm-router doctor               # check provider connectivity and setup
llm-router status               # verify it's routing — est. savings vs baseline

Works with **no API keys** on a Claude Pro/Max subscription — routing goes through MCP tools and local models. Adding provider keys widens the pool; nothing requires them. See **[guide/PROVIDERS.md](guide/PROVIDERS.md)**. To install into another host, see [Works With](#works-with). The daily KPI job is installed only with consent (`install --schedule-daily`, or Y when asked; `--no-schedule` skips); `uninstall` removes it.

Python agents route in-process, no server: `from llm_router import Router; r = Router(budget_usd=0.50).run("...")` (`await .arun(...)` in async code; `BudgetExceeded` when the cap is spent).

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 18 lines
  1. {
  2. "name": "ypollak2",
  3. "owner": {
  4. "name": "Yali Pollak"
  5. },
  6. "plugins": [
  7. {
  8. "name": "llm-router",
  9. "description": "Route text, image, video, and audio tasks to 20+ AI providers with smart routing, budget control, and multi-step orchestration",
  10. "version": "16.0.1",
  11. "source": {
  12. "source": "github",
  13. "repo": "ypollak2/llm-router"
  14. }
  15. }
  16. ],
  17. "version": "16.0.1"
  18. }
README.md · README · 271 lines
  1. <h1 align="center">llm-router</h1>
  2. <p align="center">
  3. <strong>Stop spending your Claude Pro/Max quota on questions a free model can answer.</strong>
  4. </p>
  5. <p align="center">
  6. llm-router hooks into your coding tool's own lifecycle, reads each prompt before the
  7. model does, and drafts an answer on a free or local model first. No API keys needed
  8. on a Claude subscription — routing runs through MCP tools and local models.
  9. </p>
  10. <p align="center">
  11. <sub><b>One caveat:</b> routed answers are advisory by default — Claude still takes the
  12. turn unless you opt in to zero-Claude mode. By default some tool calls wait until a
  13. prompt is routed; <code>LLM_ROUTER_ENFORCE=off</code> turns that off (all user settings: <a href="docs/configuration.md">docs/configuration.md</a>). Draft-rate and
  14. acceptance figures live in <a href="docs/MEASUREMENT.md">docs/MEASUREMENT.md</a>, with
  15. their n, window and conditions.</sub>
  16. </p>
  17. <p align="center">
  18. <a href="https://pypi.org/project/llm-routing/"><img src="https://img.shields.io/pypi/v/llm-routing?style=flat-square&color=4F46E5" alt="PyPI"></a>
  19. <a href="https://pepy.tech/projects/llm-routing"><img src="https://static.pepy.tech/personalized-badge/llm-routing?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads" alt="PyPI Downloads"></a>
  20. <a href="https://github.com/ypollak2/llm-router/actions"><img src="https://img.shields.io/github/actions/workflow/status/ypollak2/llm-router/ci.yml?style=flat-square&label=tests" alt="Tests"></a>
  21. <a href="https://github.com/ypollak2/llm-router/stargazers"><img src="https://img.shields.io/github/stars/ypollak2/llm-router?style=flat-square&color=F59E0B&v=2" alt="Stars"></a>
  22. <a href="https://github.com/RouteWorks/RouterArena"><img src="https://img.shields.io/badge/RouterArena-listed-F59E0B?style=flat-square&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZmlsbD0id2hpdGUiIGQ9Ik0xMiAyTDMgN2w5IDUgOS01LTktNXpNMyAxN2w5IDUgOS01TTMgMTJsOSA1IDktNSIvPjwvc3ZnPg==" alt="Listed on RouterArena"></a>
  23. <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-10B981?style=flat-square" alt="License"></a>
  24. </p>
  25. <details>
  26. <summary><b>📑 Table of Contents</b></summary>
  27. - [Why install this](#why-install-this)
  28. - [Quick Start](#quick-start)
  29. - [Works With](#works-with)
  30. - [How It Works](#how-it-works)
  31. - [Features](#features)
  32. - [Reference](#reference)
  33. - [Savings](#savings)
  34. - [Trust and Security](#trust-and-security)
  35. - [Documentation](#documentation)
  36. - [Enterprise, Contributing, License](#enterprise-contributing-license)
  37. </details>
  38. ---
  39. ## Why install this
  40. You are on a Claude Pro or Max plan, have not spent a cent beyond it, and still
  41. hit the five-hour limit — not because you asked too much, but because *every*
  42. prompt went to the premium model: "what does this error mean", "reformat this
  43. JSON", "is the service up" drew the same quota as the question that needed it.
  44. ### Why a proxy cannot do this
  45. Most routers here are a **proxy**: your agent forwards requests using *your
  46. API keys*. That has a hard limit — a proxy cannot intercept a session
  47. authenticated by a subscription, because there is no key to forward.
  48. | | Pays per token | Pays a subscription |
  49. |---|---|---|
  50. | What runs out | your invoice | your five-hour window |
  51. | Needs API keys | yes | **no** |
  52. | A proxy can help | yes | **no — nothing to intercept** |
  53. | llm-router helps | yes | **yes** |
  54. ### Honesty as a feature
  55. `llm-router status` reports **verified** and **unverified** savings
  56. separately, each with its own n — not a blended percentage. This project does
  57. not publish a general savings percentage; any number depends on your own
  58. workload. See [Savings](#savings) below.
  59. ---
  60. ## Quick Start
  61. ```bash
  62. pip install llm-routing # installs the `llm-router` command
  63. llm-router install # wire up Claude Code (default host)
  64. llm-router doctor # check provider connectivity and setup
  65. llm-router status # verify it's routing — est. savings vs baseline
  66. ```
  67. Works with **no API keys** on a Claude Pro/Max subscription — routing goes
  68. through MCP tools and local models. Adding provider keys widens the pool;
  69. nothing requires them. See **[guide/PROVIDERS.md](guide/PROVIDERS.md)**. To
  70. install into another host, see [Works With](#works-with). The daily KPI job is installed only with consent (`install --schedule-daily`, or Y when asked; `--no-schedule` skips); `uninstall` removes it.
  71. Python agents route in-process, no server: `from llm_router import Router; r = Router(budget_usd=0.50).run("...")` (`await .arun(...)` in async code; `BudgetExceeded` when the cap is spent).
  72. ---
  73. ## Works With
  74. | Host | Install |
  75. |------|---------|
  76. | **Claude Code** (default) | `llm-router install` |
  77. | **Codex CLI** | `llm-router install --host codex` |
  78. | **OpenCode** | `llm-router install --host opencode` |
  79. | **Gemini CLI** | `llm-router install --host gemini-cli` |
  80. | **GitHub Copilot CLI** | `llm-router install --host copilot-cli` |
  81. | **OpenClaw** | `llm-router install --host openclaw` |
  82. | **Trae IDE** | `llm-router install --host trae` |
  83. | **Pi (pi.dev)** (MCP server only; end-of-turn line planned in 16.1) | `llm-router install --host pi` |
  84. | **Factory Droid** | `llm-router install --host factory` |
  85. | **Claude Desktop** | `llm-router install --host desktop` |
  86. | **VS Code** (native MCP) | `llm-router install --host vscode` |
  87. | **Cursor** | `llm-router install --host cursor` |
  88. | **GitHub Copilot in VS Code** (capability extension, no cost-routing) | `llm-router install --host copilot` |
  89. | **Windsurf / Cascade** | `llm-router install --host windsurf` |
  90. | **Kimi Code (Moonshot AI)** | planned (16.1); `--host kimi` reports why and exits 2 |
  91. `llm-router install --host all` installs or prints every host config in one
  92. pass. Full per-host detail, including what each host genuinely cannot do:
  93. **[guide/HOST_SUPPORT_MATRIX.md](guide/HOST_SUPPORT_MATRIX.md)**.
  94. ---
  95. ## How It Works
  96. Hooks intercept the prompt before your coding tool's own model sees it. A free
  97. regex heuristic classifies it instantly, scoring about half of real prompts
  98. with confidence — **788 of 1,571 measured** (`scripts/measure_low_signal_rate.py`,
  99. run 2026-09-23); the rest fall back to a default. A routable prompt gets a
  100. draft from a free/local model first, then walks up the chain toward paid
  101. models only if needed.
  102. The draft is **advisory** — handed to Claude as an unverified hint, not a turn
  103. replacement, unless you opt in to zero-Claude mode. Enforcement acts at the
  104. tool-call level: `smart` (default) holds selected tool calls until the prompt
  105. is routed; `off` disables that. Full modes and per-host overrides:
  106. **[guide/GETTING_STARTED.md](guide/GETTING_STARTED.md)**.
  107. ---
  108. ## Features
  109. - **A prompt with a secret goes to a local model or nowhere.** When a routed
  110. call (the `llm_*` text tools, the gateway, streaming) contains an API key,
  111. token or private key, llm-router sends it only to Ollama models on a loopback
  112. daemon — direct, not through an HTTP proxy, with no semantic cache, judge
  113. queue or cloud fallback — or refuses it with the secret's category and how to
  114. proceed (`cloud:` prefix to override one request). Placeholders such as
  115. `AKIA...EXAMPLE` or `postgres:postgres@` are not refused. Not every door
  116. enforces this: the router, gateway, SDK, `llm_stream`, `llm_codex`, `llm_gemini`
  117. and the `agent-route` hook do; the Claude Code proxy (logs only), the opt-in
  118. `auto-route` cloud classifiers, the hook's direct-execution Gemini/OpenAI calls,
  119. agent delegation and other opt-in features do not, and what a host already
  120. sent to its own model is out of reach. **[The per-door table, details and limits](SECURITY.md#data-handling--storage-mapping)**.
  121. - **Automatic fallback with circuit breakers.** A provider that fails or
  122. rate-limits is skipped, not retried into the ground.
  123. - **You can see it working.** A status line, terminal title and OS
  124. notification show the last model routed, savings and health.
  125. - **Session-end summary.** Savings vs baseline, tier mix, per-provider cost,
  126. latency p50/p95/p99 and top routes.
  127. - **Media and pipelines too.** `llm_image` / `llm_video` / `llm_audio`, and
  128. `llm_orchestrate` for multi-step research.
  129. ---
  130. ## Reference
  131. The default MCP surface shows **12 front-door tools** (the `CONSOLIDATED_TOOLS` set in
  132. `src/llm_router/tool_surface.py`; `tests/test_doc_tool_counts_match_registration.py` checks
  133. this count against it); an opt-out shows everything registered (see [docs/configuration.md](docs/configuration.md) for user settings).
  134. | Topic | What | Guide |
  135. |-------|------|-------|
  136. | CLI | `install`, `status`, `gain`, `doctor`, `okf index/status`, `sessions status` and more | [guide/GETTING_STARTED.md](guide/GETTING_STARTED.md) |
  137. | Providers | Free-first — Ollama (local), OpenRouter, Gemini, Groq, your Claude subscription | [guide/PROVIDERS.md](guide/PROVIDERS.md) |
  138. | Routing Policies | `conservative` → `balanced` (default) → `cost_aggressive`; thresholds and YAML schema | [guide/POLICIES.md](guide/POLICIES.md) |
  139. | MCP Tools | Every tool with its signature | [guide/TOOLS.md](guide/TOOLS.md) |
  140. ```bash
  141. llm-router gain today # savings so far today
  142. llm-router okf status # knowledge base freshness
  143. llm-router sessions status # per-session summary
  144. ```
  145. ---
  146. ## Savings
  147. Savings figures are a *counterfactual* — what the same tokens would have cost
  148. at API list price, against what was actually spent — not money saved on a flat
  149. subscription. `llm-router status` and `llm-router savings-report` split
  150. **verified** from **unverified** savings, each with its own n. Methodology,
  151. assumptions and limitations: **[docs/MEASUREMENT.md](docs/MEASUREMENT.md)**.
  152. ---
  153. ## Trust and Security
  154. llm-router runs entirely on your machine — no hosted proxy, nothing sent to an
  155. llm-router service, no account required. A **grounding check**
  156. (`src/llm_router/grounding.py`) discards any draft citing a file or function
  157. absent from its context and the indexed repo, falling through to Claude.
  158. `LLM_ROUTER_DIRECT_EXECUTION` is default on — a local model gets `write_file`,
  159. `edit_file` and `run_command` under a confined, allowlisted path, and none of
  160. that stops targeted deletes, `git push --force`, or exfiltration. Shell
  161. redirects and copies that write outside the project root are refused by a static
  162. tripwire, not a sandbox; an OS sandbox is 16.1. A command the tripwire
  163. cannot judge (an interpreter one-liner, `sudo`, a `$(...)`) returns
  164. `needs_approval`, but approval is not wired anywhere yet, so both command paths
  165. treat it as refused and do not run it. Turn it off
  166. with `LLM_ROUTER_DIRECT_EXECUTION=false`. Full analysis:
  167. **[SECURITY.md](SECURITY.md)**.
  168. Self-audits are published in **[audit/](audit/)** and
  169. **[docs/repo_goals/AUDIT-2026-09-25.md](docs/repo_goals/AUDIT-2026-09-25.md)**
  170. (on 2026-09-25, of 20 goals: 13 met, 6 partial, 1 not met), including negative results.
  171. **Ground Truth accumulation** (opt-in) builds a
  172. corpus of replayable routing decisions instead of unlinked telemetry; off by
  173. default since it's the only part that writes prompt text to disk. Details:
  174. **[guide/GROUND_TRUTH.md](guide/GROUND_TRUTH.md)**.
  175. ---
  176. ## Documentation
  177. Full index: **[guide/README.md](guide/README.md)**
  178. | Document | Purpose |
  179. |----------|---------|
  180. | [Quick Start](guide/QUICKSTART_2MIN.md) | Fastest path to working routing |
  181. | [Getting Started](guide/GETTING_STARTED.md) | Full setup walkthrough |
  182. | [Host Support Matrix](guide/HOST_SUPPORT_MATRIX.md) | Per-host feature comparison |
  183. | [Providers](guide/PROVIDERS.md) | Provider setup and model recommendations |
  184. | [Routing Policies](guide/POLICIES.md) | `routing.yaml` schema and authoring your own policy |
  185. | [Tool Reference](guide/TOOLS.md) | All MCP tools with examples |
  186. | [Architecture](guide/ARCHITECTURE.md) | Internal design and module structure |
  187. | [Troubleshooting](guide/TROUBLESHOOTING.md) | Common issues and fixes |
  188. | [Testing the Router](guide/TESTING.md) | Isolation suite for verifying routing health |
  189. | [Measurement](docs/MEASUREMENT.md) | What "savings" means and how it's computed |
  190. | [RouterArena](docs/ROUTERARENA.md) | Benchmark methodology, results and negative findings |
  191. | [Ground Truth](guide/GROUND_TRUTH.md) | Replayable routing corpus, opt-in |
  192. | [Benchmarks](docs/BENCHMARKS.md) | Model cost/latency/quality table, regenerated by CI |
  193. | [Changelog](CHANGELOG.md) | Release notes ([archive](CHANGELOG-ARCHIVE.md)) |
  194. `llm-router` is scored on the [RouterArena](https://github.com/RouteWorks/RouterArena)
  195. benchmark, graded locally with this repo's harness and **not yet independently
  196. verified**. Results with their n and source files, methodology and negative
  197. results (including a proxy split that misled tuning):
  198. **[docs/ROUTERARENA.md](docs/ROUTERARENA.md)**.
  199. ---
  200. ## Enterprise, Contributing, License
  201. `llm-router` is built for individual developers and small teams: local cost
  202. savings, zero ops overhead, no hosted anything. For team-wide policy
  203. enforcement, audit export, SSO or per-org budgets, see
  204. **[Chuzom](https://github.com/Chuzom/Chuzom)** — a separate, sibling product.
  205. Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
  206. ```bash
  207. git clone https://github.com/ypollak2/llm-router.git
  208. cd llm-router
  209. uv sync --extra dev
  210. uv run pytest tests/ -q # Run tests
  211. uv run ruff check src/ tests/ # Lint
  212. ```
  213. MIT License. [Issues](https://github.com/ypollak2/llm-router/issues) ·
  214. [Discussions](https://github.com/ypollak2/llm-router/discussions) ·
  215. [PyPI](https://pypi.org/project/llm-routing/) · [Changelog](CHANGELOG.md)
scripts/bench_draft_acceptance.py · Bundled script · 280 lines · 1 finding marked
  1. #!/usr/bin/env python3
  2. """Would Claude actually USE these drafts? Measured on real prompts.
  3. The draft use rate has been 0 for every window ever measured, and the obvious
  4. explanations — wrong local model, no repo context — were both real and are both
  5. now fixed. This asks whether fixing them changes the outcome.
  6. The corpus is the user's OWN prompts, recovered from Claude Code session
  7. transcripts. Inventing four questions (the first attempt at this) measured the
  8. questions, not the system.
  9. ACCEPTANCE IS SCORED MECHANICALLY, never by a model judging a model. The rules
  10. are the four reasons drafts were actually discarded, taken from a hand
  11. classification of 101 real injections:
  12. 37.6% asked the user a question instead of answering
  13. 32.7% asserted a status it could not observe ("all tests passed")
  14. 25.7% generic advice with no reference to the repo
  15. 4.0% cited files that do not exist
  16. The first three are string-detectable. The fourth is `grounding.grounding_violations`,
  17. which is already in the repo and checks the one claim settleable without another
  18. model: does the cited file exist.
  19. A draft that passes all four is one Claude could relay without telling the user
    • Dismissed after review: Tells the agent to keep something from the user
    • Line 25 on GitHub
  20. something false. That is not proof it is CORRECT — no mechanical check can give
  21. that — and the report says so.
  22. python3 scripts/bench_draft_acceptance.py --n 30 --model qwen3.8:latest --okf on
  23. """
  24. from __future__ import annotations
  25. import argparse
  26. import json
  27. import os
  28. import random
  29. import re
  30. import hashlib
  31. import subprocess
  32. import sys
  33. import time
  34. from pathlib import Path
  35. # M-02: every row this script causes is benchmark traffic, not usage. Declared
  36. # here rather than inferred later -- `routing_quality.detect_synthetic` prefers
  37. # an explicit statement by the harness, and until now no bench script made one,
  38. # so 1,813 fixture rows were counted as production spend.
  39. import os as _os
  40. _os.environ.setdefault("LLM_ROUTER_SYNTHETIC", "1")
  41. ROOT = Path(__file__).resolve().parent.parent
  42. sys.path.insert(0, str(ROOT / "src"))
  43. HOOK = ROOT / "src/llm_router/hooks/auto-route.py"
  44. PY = str(ROOT / ".venv/bin/python")
  45. # Widened 2026-09-14 after the new system prompt forbade the exact phrases this
  46. # regex looked for. A draft then came back with "Could you share: 1. The workflow
  47. # file content" and scored ACCEPTABLE — the same deferral, reworded around the
  48. # grep. A scorer that a prompt change can teach a model to evade is measuring the
  49. # wording, not the behaviour.
  50. ASKS_BACK = re.compile(
  51. r"would you like me to|shall i |should i |let me know if|please confirm|"
  52. r"do you want me to|would you prefer|"
  53. r"could you (share|provide|clarify|confirm|tell me|send|paste|specify)|"
  54. r"can you (share|provide|clarify|confirm|tell me|send|paste|specify)|"
  55. r"please (share|provide|clarify|send|paste|specify)|"
  56. r"once you (provide|share|send)|"
  57. r"(i|we) (would )?need (you to|more|additional) (know|information|details|context)|"
  58. r"if you (can )?(share|provide|send) ", re.I)
  59. # A draft that says "I'll go look at the codebase" is not an answer, it is a
  60. # promise to produce one. Claude cannot relay it — there is nothing to relay.
  61. # Caught when a rescued continuation scored ACCEPTABLE on the body "I'll continue
  62. # with W3 ... Let me first check the current state of the codebase."
  63. PROMISES_ONLY = re.compile(
  64. r"\b(i'?ll|i will|let me|i'?m going to|first,? (i|let)|i can) "
  65. r"(start|begin|continue|check|look|review|examine|investigate|analyz|read|"
  66. r"proceed|go (through|over)|take a look)", re.I)
  67. # Substance = something only a model that actually engaged could produce: a path,
  68. # a symbol, a command, a number, or a structured list of real steps.
  69. _SUBSTANCE = re.compile(
  70. r"`[^`]+`|\b\w+\.(py|sh|md|json|yaml|toml|js|ts)\b|^\s*[-*\d]+[.)]\s+\S+",
  71. re.M)
  72. def _has_substance(body: str) -> bool:
  73. return len(_SUBSTANCE.findall(body)) >= 2
  74. ASSERTS_STATUS = re.compile(
  75. r"all tests? (pass|passed)|completed successfully|no further action|"
  76. r"has (been )?(completed|finished) successfully|✅|task .* (complete|done)", re.I)
  77. def draft_body(stdout: str) -> str | None:
  78. try:
  79. payload = json.loads(stdout)
  80. except Exception: # noqa: BLE001
  81. return None
  82. blob = json.dumps(payload)
  83. if "UNVERIFIED DRAFT" not in blob:
  84. return None
  85. body = blob.split("UNVERIFIED DRAFT")[1].split("END UNVERIFIED DRAFT")[0]
  86. return body.replace("(no context — verify or discard)", " ")
  87. WANTS_QUESTIONS = re.compile(
  88. r"ask me|american question|multiple choice|give me options|"
  89. r"what do you (need|want) (to know|from me)", re.I)
  90. def verdict(body: str, prompt: str) -> tuple[bool, str]:
  91. """(acceptable, reason-if-not)."""
  92. # The user's own house rule is to be asked in multiple choice. When the
  93. # PROMPT asks for questions, answering with questions is correct, and
  94. # scoring it a failure measured the rule rather than the model: 7 of 20
  95. # ASKS_BACK rejects on 2026-09-14 were exactly this.
  96. if ASKS_BACK.search(body) and not WANTS_QUESTIONS.search(prompt):
  97. return False, "asks the user a question instead of answering"
  98. if ASSERTS_STATUS.search(body):
  99. return False, "asserts a status it cannot observe"
  100. try:
  101. from llm_router.grounding import grounding_violations
  102. bad = grounding_violations(body, "", prompt)
  103. if bad:
  104. return False, f"cites files that do not exist: {bad[:2]}"
  105. except Exception: # noqa: BLE001
  106. pass
  107. if len(body.strip()) < 80:
  108. return False, "too short to be an answer"
  109. if PROMISES_ONLY.search(body) and not _has_substance(body):
  110. return False, "announces intent without answering"
  111. return True, ""
  112. DEBUG_LOG = Path(
  113. os.environ.get("LLM_ROUTER_HOME", "").strip() or (Path.home() / ".llm-router")
  114. ).expanduser() / "auto-route-debug.log"
  115. def _log_size() -> int:
  116. try:
  117. return DEBUG_LOG.stat().st_size
  118. except OSError:
  119. return 0
  120. def _skips_since(offset: int) -> list[str]:
  121. """Why the hook abandoned each model in the chain, for this invocation only.
  122. Without this a missing draft is indistinguishable from a slow one, which is
  123. exactly the confusion that made three runs of one config spread 17%-53%.
  124. """
  125. try:
  126. with DEBUG_LOG.open() as fh:
  127. fh.seek(offset)
  128. tail = fh.read()
  129. except OSError:
  130. return []
  131. return [ln.split("DIRECT MODEL SKIPPED:", 1)[1].strip()
  132. for ln in tail.splitlines() if "DIRECT MODEL SKIPPED:" in ln]
  133. def preflight(model: str) -> None:
  134. """Refuse to benchmark a model this machine does not have.
  135. A run pinned to `qwen3.5:latest` — not installed here — was reported as the
  136. "old configuration" baseline and compared against a new one. Ollama answers a
  137. missing model with a 404 the hook swallows, so the run produced numbers and no
  138. error. An absent model must stop the benchmark, not quietly become its result.
  139. """
  140. if model == "auto":
  141. return
  142. sys.path.insert(0, str(ROOT / "src"))
  143. from llm_router.hooks.direct_executor import available_ollama_models
  144. installed = available_ollama_models(timeout=5.0)
  145. if installed is None:
  146. raise SystemExit("ollama is not reachable — cannot verify the model, refusing to measure")
  147. if model not in installed and model.split(":")[0] not in {m.split(":")[0] for m in installed}:
  148. raise SystemExit(
  149. f"{model} is not installed (have: {', '.join(sorted(installed)) or 'nothing'}).\n"
  150. "Pull it or pass --model auto to measure the real discovered chain."
  151. )
  152. def run(prompt: str, model: str, okf: bool, timeout: int) -> str | None:
  153. env = dict(os.environ)
  154. if model != "auto":
  155. env["LLM_ROUTER_OLLAMA_MODEL"] = model
  156. else:
  157. env.pop("LLM_ROUTER_OLLAMA_MODEL", None)
  158. env["LLM_ROUTER_CONTEXT_INJECTION"] = "on" if okf else "off"
  159. env["LLM_ROUTER_TRACE"] = ""
  160. # One session id per prompt. Sharing one across a shuffled corpus let a draft
  161. # about prompt N become the injected context for unrelated prompt N+1 — 29 of
  162. # 144 drafts in the 2026-09-14 run carried "Gable 5" into prompts that had
  163. # nothing to do with it, and the resulting 72%/55% measured contamination.
  164. sid = hashlib.sha1(f"{prompt}|{os.getpid()}".encode()).hexdigest()[:8]
  165. payload = json.dumps({"session_id": sid, "prompt": prompt,
  166. "cwd": str(ROOT), "transcript_path": ""})
  167. try:
  168. r = subprocess.run([PY, str(HOOK)], input=payload, capture_output=True,
  169. text=True, env=env, cwd=str(ROOT), timeout=timeout)
  170. return draft_body(r.stdout or "")
  171. except subprocess.TimeoutExpired:
  172. return None
  173. def main() -> int:
  174. ap = argparse.ArgumentParser()
  175. ap.add_argument("--n", type=int, default=30)
  176. ap.add_argument("--model", default="qwen3.8:latest")
  177. ap.add_argument("--okf", default="on")
  178. ap.add_argument("--corpus", default="/tmp/corpus.json")
  179. ap.add_argument("--timeout", type=int, default=120)
  180. ap.add_argument("--seed", type=int, default=7)
  181. ap.add_argument("--out", default="/tmp/draft_bodies.json",
  182. help="where draft body is saved for re-scoring")
  183. args = ap.parse_args()
  184. corpus = json.loads(Path(args.corpus).read_text())
  185. random.Random(args.seed).shuffle(corpus)
  186. sample = corpus[:args.n]
  187. okf = args.okf == "on"
  188. preflight(args.model)
  189. print(f"model={args.model} context_injection={args.okf} n={len(sample)} seed={args.seed}")
  190. produced = accepted = 0
  191. reasons: dict[str, int] = {}
  192. skip_causes: dict[str, int] = {}
  193. # Every draft body is written out as it arrives, so a change to the scoring
  194. # rules never costs another hour of model time — and a run that dies halfway
  195. # still leaves everything it managed to collect.
  196. captured: list[dict] = []
  197. for i, prompt in enumerate(sample, 1):
  198. mark = _log_size()
  199. t0 = time.monotonic()
  200. body = run(prompt, args.model, okf, args.timeout)
  201. dt = time.monotonic() - t0
  202. skips = _skips_since(mark)
  203. captured.append({"prompt": prompt, "body": body, "seconds": round(dt, 1),
  204. "skips": skips})
  205. Path(args.out).write_text(json.dumps(captured, indent=1))
  206. if body is None:
  207. reasons["no draft produced"] = reasons.get("no draft produced", 0) + 1
  208. for sk in skips:
  209. key = re.sub(r"\d+ chars", "N chars", sk)
  210. skip_causes[key] = skip_causes.get(key, 0) + 1
  211. why = "; ".join(skips) or "no model abandoned — gated before the call"
  212. print(f" {i:3d}. NO DRAFT [{dt:5.1f}s] {prompt[:40]!r} — {why[:70]}")
  213. continue
  214. produced += 1
  215. ok, why = verdict(body, prompt)
  216. accepted += ok
  217. if not ok:
  218. reasons[why.split(":")[0]] = reasons.get(why.split(":")[0], 0) + 1
  219. print(f" {i:3d}. {'ACCEPT ' if ok else 'reject '} {prompt[:46]!r} "
  220. f"{'' if ok else '— ' + why[:50]}")
  221. n = len(sample)
  222. print(f"\n drafts produced : {produced}/{n} ({100*produced/n:.0f}%)")
  223. print(f" ACCEPTABLE : {accepted}/{n} ({100*accepted/n:.0f}%)")
  224. if skip_causes:
  225. print("\n why models were abandoned (transport level):")
  226. for cause, count in sorted(skip_causes.items(), key=lambda x: -x[1]):
  227. print(f" {count:3d} {cause}")
  228. print("\n why the rest were not:")
  229. for reason, count in sorted(reasons.items(), key=lambda x: -x[1]):
  230. print(f" {count:3d} {reason}")
  231. print("\n 'acceptable' means Claude could relay it without saying something")
  232. print(" false. It is NOT a claim that the draft is correct — no mechanical")
  233. print(" check can establish that.")
  234. return 0
  235. if __name__ == "__main__":
  236. raise SystemExit(main())

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.