ypollak2
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 17 Nov 2026. Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/ypollak2/llm-router with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/ypollak2/llm-router'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/ypollak2--llm-router)HTML
<a href="/listings/ypollak2--llm-router"><img src="/api/listings/ypollak2--llm-router/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
pip install llm-routing # installs the `llm-router` command llm-router install # wire up Claude Code (default host) llm-router doctor # check provider connectivity and setup llm-router status # verify it's routing — est. savings vs baseline
Works with **no API keys** on a Claude Pro/Max subscription — routing goes through MCP tools and local models. Adding provider keys widens the pool; nothing requires them. See **[guide/PROVIDERS.md](guide/PROVIDERS.md)**. To install into another host, see [Works With](#works-with). The daily KPI job is installed only with consent (`install --schedule-daily`, or Y when asked; `--no-schedule` skips); `uninstall` removes it.
Python agents route in-process, no server: `from llm_router import Router; r = Router(budget_usd=0.50).run("...")` (`await .arun(...)` in async code; `BudgetExceeded` when the cap is spent).
---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 18 lines
- {
- "name": "ypollak2",
- "owner": {
- "name": "Yali Pollak"
- },
- "plugins": [
- {
- "name": "llm-router",
- "description": "Route text, image, video, and audio tasks to 20+ AI providers with smart routing, budget control, and multi-step orchestration",
- "version": "16.0.1",
- "source": {
- "source": "github",
- "repo": "ypollak2/llm-router"
- }
- }
- ],
- "version": "16.0.1"
- }
README.md · README · 271 lines
- <h1 align="center">llm-router</h1>
- <p align="center">
- <strong>Stop spending your Claude Pro/Max quota on questions a free model can answer.</strong>
- </p>
- <p align="center">
- llm-router hooks into your coding tool's own lifecycle, reads each prompt before the
- model does, and drafts an answer on a free or local model first. No API keys needed
- on a Claude subscription — routing runs through MCP tools and local models.
- </p>
- <p align="center">
- <sub><b>One caveat:</b> routed answers are advisory by default — Claude still takes the
- turn unless you opt in to zero-Claude mode. By default some tool calls wait until a
- prompt is routed; <code>LLM_ROUTER_ENFORCE=off</code> turns that off (all user settings: <a href="docs/configuration.md">docs/configuration.md</a>). Draft-rate and
- acceptance figures live in <a href="docs/MEASUREMENT.md">docs/MEASUREMENT.md</a>, with
- their n, window and conditions.</sub>
- </p>
- <p align="center">
- <a href="https://pypi.org/project/llm-routing/"><img src="https://img.shields.io/pypi/v/llm-routing?style=flat-square&color=4F46E5" alt="PyPI"></a>
- <a href="https://pepy.tech/projects/llm-routing"><img src="https://static.pepy.tech/personalized-badge/llm-routing?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads" alt="PyPI Downloads"></a>
- <a href="https://github.com/ypollak2/llm-router/actions"><img src="https://img.shields.io/github/actions/workflow/status/ypollak2/llm-router/ci.yml?style=flat-square&label=tests" alt="Tests"></a>
- <a href="https://github.com/ypollak2/llm-router/stargazers"><img src="https://img.shields.io/github/stars/ypollak2/llm-router?style=flat-square&color=F59E0B&v=2" alt="Stars"></a>
- <a href="https://github.com/RouteWorks/RouterArena"><img src="https://img.shields.io/badge/RouterArena-listed-F59E0B?style=flat-square&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZmlsbD0id2hpdGUiIGQ9Ik0xMiAyTDMgN2w5IDUgOS01LTktNXpNMyAxN2w5IDUgOS01TTMgMTJsOSA1IDktNSIvPjwvc3ZnPg==" alt="Listed on RouterArena"></a>
- <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-10B981?style=flat-square" alt="License"></a>
- </p>
- <details>
- <summary><b>📑 Table of Contents</b></summary>
- - [Why install this](#why-install-this)
- - [Quick Start](#quick-start)
- - [Works With](#works-with)
- - [How It Works](#how-it-works)
- - [Features](#features)
- - [Reference](#reference)
- - [Savings](#savings)
- - [Trust and Security](#trust-and-security)
- - [Documentation](#documentation)
- - [Enterprise, Contributing, License](#enterprise-contributing-license)
- </details>
- ---
- ## Why install this
- You are on a Claude Pro or Max plan, have not spent a cent beyond it, and still
- hit the five-hour limit — not because you asked too much, but because *every*
- prompt went to the premium model: "what does this error mean", "reformat this
- JSON", "is the service up" drew the same quota as the question that needed it.
- ### Why a proxy cannot do this
- Most routers here are a **proxy**: your agent forwards requests using *your
- API keys*. That has a hard limit — a proxy cannot intercept a session
- authenticated by a subscription, because there is no key to forward.
- | | Pays per token | Pays a subscription |
- |---|---|---|
- | What runs out | your invoice | your five-hour window |
- | Needs API keys | yes | **no** |
- | A proxy can help | yes | **no — nothing to intercept** |
- | llm-router helps | yes | **yes** |
- ### Honesty as a feature
- `llm-router status` reports **verified** and **unverified** savings
- separately, each with its own n — not a blended percentage. This project does
- not publish a general savings percentage; any number depends on your own
- workload. See [Savings](#savings) below.
- ---
- ## Quick Start
- ```bash
- pip install llm-routing # installs the `llm-router` command
- llm-router install # wire up Claude Code (default host)
- llm-router doctor # check provider connectivity and setup
- llm-router status # verify it's routing — est. savings vs baseline
- ```
- Works with **no API keys** on a Claude Pro/Max subscription — routing goes
- through MCP tools and local models. Adding provider keys widens the pool;
- nothing requires them. See **[guide/PROVIDERS.md](guide/PROVIDERS.md)**. To
- install into another host, see [Works With](#works-with). The daily KPI job is installed only with consent (`install --schedule-daily`, or Y when asked; `--no-schedule` skips); `uninstall` removes it.
- Python agents route in-process, no server: `from llm_router import Router; r = Router(budget_usd=0.50).run("...")` (`await .arun(...)` in async code; `BudgetExceeded` when the cap is spent).
- ---
- ## Works With
- | Host | Install |
- |------|---------|
- | **Claude Code** (default) | `llm-router install` |
- | **Codex CLI** | `llm-router install --host codex` |
- | **OpenCode** | `llm-router install --host opencode` |
- | **Gemini CLI** | `llm-router install --host gemini-cli` |
- | **GitHub Copilot CLI** | `llm-router install --host copilot-cli` |
- | **OpenClaw** | `llm-router install --host openclaw` |
- | **Trae IDE** | `llm-router install --host trae` |
- | **Pi (pi.dev)** (MCP server only; end-of-turn line planned in 16.1) | `llm-router install --host pi` |
- | **Factory Droid** | `llm-router install --host factory` |
- | **Claude Desktop** | `llm-router install --host desktop` |
- | **VS Code** (native MCP) | `llm-router install --host vscode` |
- | **Cursor** | `llm-router install --host cursor` |
- | **GitHub Copilot in VS Code** (capability extension, no cost-routing) | `llm-router install --host copilot` |
- | **Windsurf / Cascade** | `llm-router install --host windsurf` |
- | **Kimi Code (Moonshot AI)** | planned (16.1); `--host kimi` reports why and exits 2 |
- `llm-router install --host all` installs or prints every host config in one
- pass. Full per-host detail, including what each host genuinely cannot do:
- **[guide/HOST_SUPPORT_MATRIX.md](guide/HOST_SUPPORT_MATRIX.md)**.
- ---
- ## How It Works
- Hooks intercept the prompt before your coding tool's own model sees it. A free
- regex heuristic classifies it instantly, scoring about half of real prompts
- with confidence — **788 of 1,571 measured** (`scripts/measure_low_signal_rate.py`,
- run 2026-09-23); the rest fall back to a default. A routable prompt gets a
- draft from a free/local model first, then walks up the chain toward paid
- models only if needed.
- The draft is **advisory** — handed to Claude as an unverified hint, not a turn
- replacement, unless you opt in to zero-Claude mode. Enforcement acts at the
- tool-call level: `smart` (default) holds selected tool calls until the prompt
- is routed; `off` disables that. Full modes and per-host overrides:
- **[guide/GETTING_STARTED.md](guide/GETTING_STARTED.md)**.
- ---
- ## Features
- - **A prompt with a secret goes to a local model or nowhere.** When a routed
- call (the `llm_*` text tools, the gateway, streaming) contains an API key,
- token or private key, llm-router sends it only to Ollama models on a loopback
- daemon — direct, not through an HTTP proxy, with no semantic cache, judge
- queue or cloud fallback — or refuses it with the secret's category and how to
- proceed (`cloud:` prefix to override one request). Placeholders such as
- `AKIA...EXAMPLE` or `postgres:postgres@` are not refused. Not every door
- enforces this: the router, gateway, SDK, `llm_stream`, `llm_codex`, `llm_gemini`
- and the `agent-route` hook do; the Claude Code proxy (logs only), the opt-in
- `auto-route` cloud classifiers, the hook's direct-execution Gemini/OpenAI calls,
- agent delegation and other opt-in features do not, and what a host already
- sent to its own model is out of reach. **[The per-door table, details and limits](SECURITY.md#data-handling--storage-mapping)**.
- - **Automatic fallback with circuit breakers.** A provider that fails or
- rate-limits is skipped, not retried into the ground.
- - **You can see it working.** A status line, terminal title and OS
- notification show the last model routed, savings and health.
- - **Session-end summary.** Savings vs baseline, tier mix, per-provider cost,
- latency p50/p95/p99 and top routes.
- - **Media and pipelines too.** `llm_image` / `llm_video` / `llm_audio`, and
- `llm_orchestrate` for multi-step research.
- ---
- ## Reference
- The default MCP surface shows **12 front-door tools** (the `CONSOLIDATED_TOOLS` set in
- `src/llm_router/tool_surface.py`; `tests/test_doc_tool_counts_match_registration.py` checks
- this count against it); an opt-out shows everything registered (see [docs/configuration.md](docs/configuration.md) for user settings).
- | Topic | What | Guide |
- |-------|------|-------|
- | CLI | `install`, `status`, `gain`, `doctor`, `okf index/status`, `sessions status` and more | [guide/GETTING_STARTED.md](guide/GETTING_STARTED.md) |
- | Providers | Free-first — Ollama (local), OpenRouter, Gemini, Groq, your Claude subscription | [guide/PROVIDERS.md](guide/PROVIDERS.md) |
- | Routing Policies | `conservative` → `balanced` (default) → `cost_aggressive`; thresholds and YAML schema | [guide/POLICIES.md](guide/POLICIES.md) |
- | MCP Tools | Every tool with its signature | [guide/TOOLS.md](guide/TOOLS.md) |
- ```bash
- llm-router gain today # savings so far today
- llm-router okf status # knowledge base freshness
- llm-router sessions status # per-session summary
- ```
- ---
- ## Savings
- Savings figures are a *counterfactual* — what the same tokens would have cost
- at API list price, against what was actually spent — not money saved on a flat
- subscription. `llm-router status` and `llm-router savings-report` split
- **verified** from **unverified** savings, each with its own n. Methodology,
- assumptions and limitations: **[docs/MEASUREMENT.md](docs/MEASUREMENT.md)**.
- ---
- ## Trust and Security
- llm-router runs entirely on your machine — no hosted proxy, nothing sent to an
- llm-router service, no account required. A **grounding check**
- (`src/llm_router/grounding.py`) discards any draft citing a file or function
- absent from its context and the indexed repo, falling through to Claude.
- `LLM_ROUTER_DIRECT_EXECUTION` is default on — a local model gets `write_file`,
- `edit_file` and `run_command` under a confined, allowlisted path, and none of
- that stops targeted deletes, `git push --force`, or exfiltration. Shell
- redirects and copies that write outside the project root are refused by a static
- tripwire, not a sandbox; an OS sandbox is 16.1. A command the tripwire
- cannot judge (an interpreter one-liner, `sudo`, a `$(...)`) returns
- `needs_approval`, but approval is not wired anywhere yet, so both command paths
- treat it as refused and do not run it. Turn it off
- with `LLM_ROUTER_DIRECT_EXECUTION=false`. Full analysis:
- **[SECURITY.md](SECURITY.md)**.
- Self-audits are published in **[audit/](audit/)** and
- **[docs/repo_goals/AUDIT-2026-09-25.md](docs/repo_goals/AUDIT-2026-09-25.md)**
- (on 2026-09-25, of 20 goals: 13 met, 6 partial, 1 not met), including negative results.
- **Ground Truth accumulation** (opt-in) builds a
- corpus of replayable routing decisions instead of unlinked telemetry; off by
- default since it's the only part that writes prompt text to disk. Details:
- **[guide/GROUND_TRUTH.md](guide/GROUND_TRUTH.md)**.
- ---
- ## Documentation
- Full index: **[guide/README.md](guide/README.md)**
- | Document | Purpose |
- |----------|---------|
- | [Quick Start](guide/QUICKSTART_2MIN.md) | Fastest path to working routing |
- | [Getting Started](guide/GETTING_STARTED.md) | Full setup walkthrough |
- | [Host Support Matrix](guide/HOST_SUPPORT_MATRIX.md) | Per-host feature comparison |
- | [Providers](guide/PROVIDERS.md) | Provider setup and model recommendations |
- | [Routing Policies](guide/POLICIES.md) | `routing.yaml` schema and authoring your own policy |
- | [Tool Reference](guide/TOOLS.md) | All MCP tools with examples |
- | [Architecture](guide/ARCHITECTURE.md) | Internal design and module structure |
- | [Troubleshooting](guide/TROUBLESHOOTING.md) | Common issues and fixes |
- | [Testing the Router](guide/TESTING.md) | Isolation suite for verifying routing health |
- | [Measurement](docs/MEASUREMENT.md) | What "savings" means and how it's computed |
- | [RouterArena](docs/ROUTERARENA.md) | Benchmark methodology, results and negative findings |
- | [Ground Truth](guide/GROUND_TRUTH.md) | Replayable routing corpus, opt-in |
- | [Benchmarks](docs/BENCHMARKS.md) | Model cost/latency/quality table, regenerated by CI |
- | [Changelog](CHANGELOG.md) | Release notes ([archive](CHANGELOG-ARCHIVE.md)) |
- `llm-router` is scored on the [RouterArena](https://github.com/RouteWorks/RouterArena)
- benchmark, graded locally with this repo's harness and **not yet independently
- verified**. Results with their n and source files, methodology and negative
- results (including a proxy split that misled tuning):
- **[docs/ROUTERARENA.md](docs/ROUTERARENA.md)**.
- ---
- ## Enterprise, Contributing, License
- `llm-router` is built for individual developers and small teams: local cost
- savings, zero ops overhead, no hosted anything. For team-wide policy
- enforcement, audit export, SSO or per-org budgets, see
- **[Chuzom](https://github.com/Chuzom/Chuzom)** — a separate, sibling product.
- Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
- ```bash
- git clone https://github.com/ypollak2/llm-router.git
- cd llm-router
- uv sync --extra dev
- uv run pytest tests/ -q # Run tests
- uv run ruff check src/ tests/ # Lint
- ```
- MIT License. [Issues](https://github.com/ypollak2/llm-router/issues) ·
- [Discussions](https://github.com/ypollak2/llm-router/discussions) ·
- [PyPI](https://pypi.org/project/llm-routing/) · [Changelog](CHANGELOG.md)
scripts/bench_draft_acceptance.py · Bundled script · 280 lines · 1 finding marked
- #!/usr/bin/env python3
- """Would Claude actually USE these drafts? Measured on real prompts.
- The draft use rate has been 0 for every window ever measured, and the obvious
- explanations — wrong local model, no repo context — were both real and are both
- now fixed. This asks whether fixing them changes the outcome.
- The corpus is the user's OWN prompts, recovered from Claude Code session
- transcripts. Inventing four questions (the first attempt at this) measured the
- questions, not the system.
- ACCEPTANCE IS SCORED MECHANICALLY, never by a model judging a model. The rules
- are the four reasons drafts were actually discarded, taken from a hand
- classification of 101 real injections:
- 37.6% asked the user a question instead of answering
- 32.7% asserted a status it could not observe ("all tests passed")
- 25.7% generic advice with no reference to the repo
- 4.0% cited files that do not exist
- The first three are string-detectable. The fourth is `grounding.grounding_violations`,
- which is already in the repo and checks the one claim settleable without another
- model: does the cited file exist.
- A draft that passes all four is one Claude could relay without telling the user
- Dismissed after review: Tells the agent to keep something from the user
- Line 25 on GitHub
- something false. That is not proof it is CORRECT — no mechanical check can give
- that — and the report says so.
- python3 scripts/bench_draft_acceptance.py --n 30 --model qwen3.8:latest --okf on
- """
- from __future__ import annotations
- import argparse
- import json
- import os
- import random
- import re
- import hashlib
- import subprocess
- import sys
- import time
- from pathlib import Path
- # M-02: every row this script causes is benchmark traffic, not usage. Declared
- # here rather than inferred later -- `routing_quality.detect_synthetic` prefers
- # an explicit statement by the harness, and until now no bench script made one,
- # so 1,813 fixture rows were counted as production spend.
- import os as _os
- _os.environ.setdefault("LLM_ROUTER_SYNTHETIC", "1")
- ROOT = Path(__file__).resolve().parent.parent
- sys.path.insert(0, str(ROOT / "src"))
- HOOK = ROOT / "src/llm_router/hooks/auto-route.py"
- PY = str(ROOT / ".venv/bin/python")
- # Widened 2026-09-14 after the new system prompt forbade the exact phrases this
- # regex looked for. A draft then came back with "Could you share: 1. The workflow
- # file content" and scored ACCEPTABLE — the same deferral, reworded around the
- # grep. A scorer that a prompt change can teach a model to evade is measuring the
- # wording, not the behaviour.
- ASKS_BACK = re.compile(
- r"would you like me to|shall i |should i |let me know if|please confirm|"
- r"do you want me to|would you prefer|"
- r"could you (share|provide|clarify|confirm|tell me|send|paste|specify)|"
- r"can you (share|provide|clarify|confirm|tell me|send|paste|specify)|"
- r"please (share|provide|clarify|send|paste|specify)|"
- r"once you (provide|share|send)|"
- r"(i|we) (would )?need (you to|more|additional) (know|information|details|context)|"
- r"if you (can )?(share|provide|send) ", re.I)
- # A draft that says "I'll go look at the codebase" is not an answer, it is a
- # promise to produce one. Claude cannot relay it — there is nothing to relay.
- # Caught when a rescued continuation scored ACCEPTABLE on the body "I'll continue
- # with W3 ... Let me first check the current state of the codebase."
- PROMISES_ONLY = re.compile(
- r"\b(i'?ll|i will|let me|i'?m going to|first,? (i|let)|i can) "
- r"(start|begin|continue|check|look|review|examine|investigate|analyz|read|"
- r"proceed|go (through|over)|take a look)", re.I)
- # Substance = something only a model that actually engaged could produce: a path,
- # a symbol, a command, a number, or a structured list of real steps.
- _SUBSTANCE = re.compile(
- r"`[^`]+`|\b\w+\.(py|sh|md|json|yaml|toml|js|ts)\b|^\s*[-*\d]+[.)]\s+\S+",
- re.M)
- def _has_substance(body: str) -> bool:
- return len(_SUBSTANCE.findall(body)) >= 2
- ASSERTS_STATUS = re.compile(
- r"all tests? (pass|passed)|completed successfully|no further action|"
- r"has (been )?(completed|finished) successfully|✅|task .* (complete|done)", re.I)
- def draft_body(stdout: str) -> str | None:
- try:
- payload = json.loads(stdout)
- except Exception: # noqa: BLE001
- return None
- blob = json.dumps(payload)
- if "UNVERIFIED DRAFT" not in blob:
- return None
- body = blob.split("UNVERIFIED DRAFT")[1].split("END UNVERIFIED DRAFT")[0]
- return body.replace("(no context — verify or discard)", " ")
- WANTS_QUESTIONS = re.compile(
- r"ask me|american question|multiple choice|give me options|"
- r"what do you (need|want) (to know|from me)", re.I)
- def verdict(body: str, prompt: str) -> tuple[bool, str]:
- """(acceptable, reason-if-not)."""
- # The user's own house rule is to be asked in multiple choice. When the
- # PROMPT asks for questions, answering with questions is correct, and
- # scoring it a failure measured the rule rather than the model: 7 of 20
- # ASKS_BACK rejects on 2026-09-14 were exactly this.
- if ASKS_BACK.search(body) and not WANTS_QUESTIONS.search(prompt):
- return False, "asks the user a question instead of answering"
- if ASSERTS_STATUS.search(body):
- return False, "asserts a status it cannot observe"
- try:
- from llm_router.grounding import grounding_violations
- bad = grounding_violations(body, "", prompt)
- if bad:
- return False, f"cites files that do not exist: {bad[:2]}"
- except Exception: # noqa: BLE001
- pass
- if len(body.strip()) < 80:
- return False, "too short to be an answer"
- if PROMISES_ONLY.search(body) and not _has_substance(body):
- return False, "announces intent without answering"
- return True, ""
- DEBUG_LOG = Path(
- os.environ.get("LLM_ROUTER_HOME", "").strip() or (Path.home() / ".llm-router")
- ).expanduser() / "auto-route-debug.log"
- def _log_size() -> int:
- try:
- return DEBUG_LOG.stat().st_size
- except OSError:
- return 0
- def _skips_since(offset: int) -> list[str]:
- """Why the hook abandoned each model in the chain, for this invocation only.
- Without this a missing draft is indistinguishable from a slow one, which is
- exactly the confusion that made three runs of one config spread 17%-53%.
- """
- try:
- with DEBUG_LOG.open() as fh:
- fh.seek(offset)
- tail = fh.read()
- except OSError:
- return []
- return [ln.split("DIRECT MODEL SKIPPED:", 1)[1].strip()
- for ln in tail.splitlines() if "DIRECT MODEL SKIPPED:" in ln]
- def preflight(model: str) -> None:
- """Refuse to benchmark a model this machine does not have.
- A run pinned to `qwen3.5:latest` — not installed here — was reported as the
- "old configuration" baseline and compared against a new one. Ollama answers a
- missing model with a 404 the hook swallows, so the run produced numbers and no
- error. An absent model must stop the benchmark, not quietly become its result.
- """
- if model == "auto":
- return
- sys.path.insert(0, str(ROOT / "src"))
- from llm_router.hooks.direct_executor import available_ollama_models
- installed = available_ollama_models(timeout=5.0)
- if installed is None:
- raise SystemExit("ollama is not reachable — cannot verify the model, refusing to measure")
- if model not in installed and model.split(":")[0] not in {m.split(":")[0] for m in installed}:
- raise SystemExit(
- f"{model} is not installed (have: {', '.join(sorted(installed)) or 'nothing'}).\n"
- "Pull it or pass --model auto to measure the real discovered chain."
- )
- def run(prompt: str, model: str, okf: bool, timeout: int) -> str | None:
- env = dict(os.environ)
- if model != "auto":
- env["LLM_ROUTER_OLLAMA_MODEL"] = model
- else:
- env.pop("LLM_ROUTER_OLLAMA_MODEL", None)
- env["LLM_ROUTER_CONTEXT_INJECTION"] = "on" if okf else "off"
- env["LLM_ROUTER_TRACE"] = ""
- # One session id per prompt. Sharing one across a shuffled corpus let a draft
- # about prompt N become the injected context for unrelated prompt N+1 — 29 of
- # 144 drafts in the 2026-09-14 run carried "Gable 5" into prompts that had
- # nothing to do with it, and the resulting 72%/55% measured contamination.
- sid = hashlib.sha1(f"{prompt}|{os.getpid()}".encode()).hexdigest()[:8]
- payload = json.dumps({"session_id": sid, "prompt": prompt,
- "cwd": str(ROOT), "transcript_path": ""})
- try:
- r = subprocess.run([PY, str(HOOK)], input=payload, capture_output=True,
- text=True, env=env, cwd=str(ROOT), timeout=timeout)
- return draft_body(r.stdout or "")
- except subprocess.TimeoutExpired:
- return None
- def main() -> int:
- ap = argparse.ArgumentParser()
- ap.add_argument("--n", type=int, default=30)
- ap.add_argument("--model", default="qwen3.8:latest")
- ap.add_argument("--okf", default="on")
- ap.add_argument("--corpus", default="/tmp/corpus.json")
- ap.add_argument("--timeout", type=int, default=120)
- ap.add_argument("--seed", type=int, default=7)
- ap.add_argument("--out", default="/tmp/draft_bodies.json",
- help="where draft body is saved for re-scoring")
- args = ap.parse_args()
- corpus = json.loads(Path(args.corpus).read_text())
- random.Random(args.seed).shuffle(corpus)
- sample = corpus[:args.n]
- okf = args.okf == "on"
- preflight(args.model)
- print(f"model={args.model} context_injection={args.okf} n={len(sample)} seed={args.seed}")
- produced = accepted = 0
- reasons: dict[str, int] = {}
- skip_causes: dict[str, int] = {}
- # Every draft body is written out as it arrives, so a change to the scoring
- # rules never costs another hour of model time — and a run that dies halfway
- # still leaves everything it managed to collect.
- captured: list[dict] = []
- for i, prompt in enumerate(sample, 1):
- mark = _log_size()
- t0 = time.monotonic()
- body = run(prompt, args.model, okf, args.timeout)
- dt = time.monotonic() - t0
- skips = _skips_since(mark)
- captured.append({"prompt": prompt, "body": body, "seconds": round(dt, 1),
- "skips": skips})
- Path(args.out).write_text(json.dumps(captured, indent=1))
- if body is None:
- reasons["no draft produced"] = reasons.get("no draft produced", 0) + 1
- for sk in skips:
- key = re.sub(r"\d+ chars", "N chars", sk)
- skip_causes[key] = skip_causes.get(key, 0) + 1
- why = "; ".join(skips) or "no model abandoned — gated before the call"
- print(f" {i:3d}. NO DRAFT [{dt:5.1f}s] {prompt[:40]!r} — {why[:70]}")
- continue
- produced += 1
- ok, why = verdict(body, prompt)
- accepted += ok
- if not ok:
- reasons[why.split(":")[0]] = reasons.get(why.split(":")[0], 0) + 1
- print(f" {i:3d}. {'ACCEPT ' if ok else 'reject '} {prompt[:46]!r} "
- f"{'' if ok else '— ' + why[:50]}")
- n = len(sample)
- print(f"\n drafts produced : {produced}/{n} ({100*produced/n:.0f}%)")
- print(f" ACCEPTABLE : {accepted}/{n} ({100*accepted/n:.0f}%)")
- if skip_causes:
- print("\n why models were abandoned (transport level):")
- for cause, count in sorted(skip_causes.items(), key=lambda x: -x[1]):
- print(f" {count:3d} {cause}")
- print("\n why the rest were not:")
- for reason, count in sorted(reasons.items(), key=lambda x: -x[1]):
- print(f" {count:3d} {reason}")
- print("\n 'acceptable' means Claude could relay it without saying something")
- print(" false. It is NOT a claim that the draft is correct — no mechanical")
- print(" check can establish that.")
- return 0
- if __name__ == "__main__":
- raise SystemExit(main())
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.