Skip to content
Back to search

semutjahat

MarketplaceWorks with Claude CodeGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/semutjahat--model-judge/badge.svg)](/listings/semutjahat--model-judge)

HTML

<a href="/listings/semutjahat--model-judge"><img src="/api/listings/semutjahat--model-judge/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

The repo is its own marketplace, named `semutjahat`. To install model-judge for your user (all your Claude Code sessions), from a terminal:

claude plugin marketplace add SemutJahat/model-judge
claude plugin install model-judge@semutjahat --scope user
claude plugin list                     # model-judge@semutjahat … Scope: user · Status: ✔ enabled

Or at the Claude Code prompt: `/plugin install model-judge --marketplace SemutJahat/model-judge`.

The same marketplace also lists **[rag-first](https://github.com/SemutJahat/rag-first)**, which makes Claude Code search the project's enowx-rag RAG before it greps or reads around: `claude plugin install rag-first@semutjahat --scope user`. New sessions load the plugin; sessions that are already open don't. It needs Claude Code with hook modules (2.1.328 has them; see [Older Claude Code](#auto-mode-why-the-call-is-not-rewritten)), Node on the PATH, and your Claude Code login, which the judge uses.

Claude Code installs a copy of the plugin in `~/.claude/plugins/cache/semutjahat/model-judge/<version>/` and runs it from there. To get a newer version after the repo changes:

claude plugin marketplace update semutjahat     # fetch the repo's latest marketplace and plugin
claude plugin update model-judge@semutjahat    # then open a new session

To turn it off for a while, or remove it:

claude plugin disable model-judge@semutjahat --scope user    # off; on again: claude plugin enable model-judge@semutjahat --scope user
claude plugin uninstall model-judge@semutjahat --scope user  # remove the plugin
claude plugin marketplace remove semutjahat                   # remove the marketplace

To try it for one session without installing anything, from a clone: `claude --plugin-dir .`

**Developing it.** A clone added as a local marketplace (`claude plugin marketplace add ./model-judge`) is read straight from that folder, not from a copy. Edits apply without reinstalling: `scripts/` and `config.json` from the next subagent call, and `hooks/` and `commands/` from the next new session.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 27 lines
  1. {
  2. "name": "semutjahat",
  3. "owner": {
  4. "name": "SemutJahat",
  5. "url": "https://github.com/SemutJahat"
  6. },
  7. "metadata": {
  8. "description": "SemutJahat's Claude Code plugins: model-judge (a small judge model picks each subagent's model and effort) and rag-first (search the project's RAG before grepping)."
  9. },
  10. "plugins": [
  11. {
  12. "name": "model-judge",
  13. "source": "./",
  14. "description": "Picks each subagent's model and effort with a small judge model, before it starts; opt-in, suggests delegating big multi-part prompts.",
  15. "version": "0.4.0"
  16. },
  17. {
  18. "name": "rag-first",
  19. "source": {
  20. "source": "github",
  21. "repo": "SemutJahat/rag-first"
  22. },
  23. "description": "Makes Claude Code search the project's enowx-rag RAG before it greps or reads around.",
  24. "version": "0.2.1"
  25. }
  26. ]
  27. }
README.md · README · 557 lines
  1. # model-judge
  2. A Claude Code plugin. Just before Claude Code starts a subagent, a small judge model rates the subagent's
  3. prompt **easy / medium / hard**. The plugin then sets the subagent's **model** and **effort** to match,
  4. replacing whatever the caller or the subagent's definition had set.
  5. It needs nothing besides Claude Code and Node: the judge runs on your own Claude Code login
  6. (`claude -p --model haiku`). When `ANTHROPIC_API_KEY` is set, it calls the Messages API directly, which is faster.
  7. ## What it does
  8. Two parts work together; both ship in `hooks/hooks.json`:
  9. 1. **A `PreToolUse` hook** on the subagent tool (matcher `Agent|Task`; in Claude Code 2.1.328 the tool is `Agent`,
  10. `Task` its legacy alias), `scripts/route.mjs`. It sends the call's `prompt`, `description` and `subagent_type`
  11. to the judge, which answers `{"why", "difficulty", "confidence"}`, logs the decision, and shows you one line
  12. under the call. It **does not change the call**: no `updatedInput`, no `permissionDecision`. It leaves its
  13. decision (model, effort) in `~/.claude/model-judge/pending/<tool_use_id>.json`.
  14. 2. **A hooks module**, `hooks/register.ts`, which Claude Code runs inside the session. At **`agent.spawn`**, when
  15. the call has passed every permission check and the subagent is about to start, it takes that decision and
  16. sets the subagent's **model**. On that subagent's **`turn.step`** (each model request it makes) it sets the
  17. **effort**. It runs `scripts/spawn.mjs` for the hand-off and logs what was really applied.
  18. Why two parts: auto mode reviews a tool call as the model wrote it, and **refuses a call that a hook rewrote
  19. after that review**. A `PreToolUse` hook that returned `updatedInput` (what model-judge 0.1 did) got every
  20. changed subagent call refused in auto mode, and the model then sent each call a second time. Setting the model
  21. at spawn time leaves the reviewed call untouched. See [Auto mode](#auto-mode-why-the-call-is-not-rewritten).
  22. It also adds a `/model-judge:log` command and ships an optional status line. See
  23. [Seeing what the judge does](#seeing-what-the-judge-does). Opt-in, it can also suggest that the main agent
  24. delegate a big multi-part prompt to subagents: see [Delegation nudge](#delegation-nudge-opt-in-off-by-default).
  25. Default routes (in `config.json`):
  26. | difficulty | model | effort |
  27. |------------|----------|----------|
  28. | easy | `haiku` | `low` |
  29. | medium | `sonnet` | `medium` |
  30. | hard | `opus` | `high` |
  31. ### The override rule
  32. When the judge answers with confidence ≥ the threshold (default **0.75**), its model **always replaces** the
  33. subagent's model, whether the main agent set one, the agent definition's frontmatter set one, or nobody did.
  34. It does this even when the model is the same. `agent.spawn`'s `model` is the Agent call's `model` parameter,
  35. which outranks the definition's `model:` frontmatter and `CLAUDE_CODE_SUBAGENT_MODEL`. The judge's effort
  36. replaces the subagent's effort on every request it makes.
  37. The subagent starts **as asked** (and the hook exits 0) when:
  38. - the judge is unsure (confidence < threshold);
  39. - the judge fails: `claude` not found, it exits non-zero, or its answer isn't the expected JSON;
  40. - the judge takes longer than `timeoutMs` (default **15 s**); the judge is killed;
  41. - the subagent is a `fork`, since Claude Code ignores model and effort for forks;
  42. - `CLAUDE_CODE_SUBAGENT_MODEL_FORCE` is set, since Claude Code then removes `model` from the tool;
  43. - the hook runs inside the judge itself (`MODEL_JUDGE_INNER=1`, the recursion guard).
  44. It never blocks a subagent: every path exits 0, the module's `agent.spawn` hook starts the subagent as asked
  45. if anything in it fails, and any error only leaves the call as it was.
  46. The same call judged again within `cacheTtlMs` (default 10 minutes) reuses the verdict instead of paying the
  47. judge again: the log says `"judge": "cache"`, and the line shows `(0.0s)`. The cache key covers the judge's
  48. model, its instructions and the task text (prompt, description, subagent type), and failed calls are not
  49. cached. It lives in `~/.claude/model-judge/cache.json`.
  50. ## Auto mode: why the call is not rewritten
  51. What model-judge 0.1 did, and what happened in auto mode (reproduced in a headless `claude -p --permission-mode
  52. auto` run of `examples/prompts/mixed.md`): all three Agent calls came back with
  53. > The server-side auto mode classifier gave no verdict for Agent: a hook changed this call's input after auto
  54. > mode reviewed it, so the review does not cover what would run. This is not a judgment that the action is
  55. > unsafe. The hook produced this input: {"description":"count words",…,"model":"haiku","effort":"low",…}
  56. The same refusal came for `slugify with tests` (sonnet→sonnet), because adding `effort` alone changes the input.
  57. Claude Code's text tells the model to "issue the call once more with exactly that input". So the model sent all
  58. three again, and the hook judged them again: 6 Agent calls and 6 judge calls for 3 subagents. In the owner's
  59. session, the second round carried the hook's own model and effort, so the hook changed nothing and the calls
  60. ran. In the headless run, the second round asked for `sonnet` again and was refused again.
  61. This is by design in Claude Code 2.1.328. Auto mode's classifier reviews each tool call server-side, as the
  62. model wrote it, so any `updatedInput` voids the review. A `permissionDecision` of `"allow"` with
  63. `updatedInput` doesn't help: in auto mode a hook's allow is still sent to the classifier ("auto mode requires
  64. classifier adjudication"), which has only seen the original input. And an `"allow"` would skip the user's
  65. permission prompts in the other modes, which model-judge must not do.
  66. The fix uses the hooks-module events that Claude Code 2.1.328 gives plugins for this:
  67. - **`agent.spawn`:** "A hook sets this to pick the subagent's model". It runs after the call was reviewed
  68. and allowed.
  69. - **`turn.step`:** "A hook rewrites `model` or `effort` going down", for one model request.
  70. Neither event changes the tool call, so no review is voided, and nothing is approved that wouldn't be anyway.
  71. Tested in real headless runs of the mixed prompt in each permission mode, with 3 subagents requesting `sonnet`:
  72. | mode | Agent calls | refused | judge calls | first requests (model / effort) |
  73. |---|---|---|---|---|
  74. | auto (0.1, before) | 6 | 6 | 6 | none ran |
  75. | default | 3 | 0 | 3 | haiku/low · sonnet/medium · opus/high |
  76. | acceptEdits | 3 | 0 | 3 | haiku/low · sonnet/medium · opus/high |
  77. | auto | 3 | 0 | 3 | haiku/low · sonnet/medium · opus/high |
  78. | bypassPermissions | 3 | 0 | 3 | haiku/low · sonnet/medium · opus/high |
  79. An interactive session in auto mode, using the installed plugin, gave the same result (see
  80. [Seeing what the judge does](#seeing-what-the-judge-does)).
  81. **Older Claude Code.** The module needs hook modules (`agent.spawn`, `turn.step`), which Claude Code 2.1.328
  82. has. On a version without them, set `"apply": "input"` in the config. That brings back the 0.1 behaviour
  83. (`updatedInput` on the call), which works in default, acceptEdits and bypassPermissions modes but not in auto.
  84. ## Effort: it is set per request
  85. `agent.spawn` takes no effort, but `turn.step` does: each model request of a subagent passes through it, and a
  86. hook may set its `effort` (`low | medium | high | xhigh | max`). The module remembers the effort for each
  87. subagent it started (by `agentId`) and sets it on every request that subagent makes. After the first request,
  88. it reads what that request really carried (`$.agent.list()`'s `lastRequest`, "the request as it went out, past
  89. caps, fallbacks and a hook's rewrite") and logs it as a `"request"` line. That is the evidence the effort
  90. reached the API. The real runs above show low / medium / high there.
  91. Limits found in Claude Code:
  92. - Since the call isn't rewritten, a subagent's `meta.json` in the transcript shows the model but no effort
  93. (`effort: null`). The `"request"` lines in the log show the effort that was sent.
  94. - **`CLAUDE_CODE_EFFORT_LEVEL`** pins the session's effort. When it is set (to anything but `auto`/`unset`),
  95. model-judge sets only the model and leaves effort alone (the log notes it).
  96. - The module's effort table lives in its memory: if Claude Code reloads the plugin while a subagent runs,
  97. that subagent's later requests go back to their default effort, but its model stays.
  98. - Forks inherit the parent's model and effort, and the hook doesn't touch them.
  99. ## Latency
  100. Each routed subagent waits for one judge call before it starts. Measured on this Mac (macOS, Claude Code
  101. 2.1.328, judge `claude -p --model haiku` → Claude Haiku 5.5, no API key), over 12 real calls:
  102. **2.7 s to 4.2 s, median ≈ 3.2 s**, about $0.0002 per call. A bare `claude -p` judge call timed with `time`:
  103. 3.07 s wall, of which the model took 2.0 s (`duration_ms`). The API path skips the CLI's start-up and should be
  104. faster, but no API key was available here to time it. A cache hit (the same call within 10 minutes) costs
  105. nothing: no judge call. The module's hand-off adds two short `node` runs per subagent (about 50 ms each).
  106. ## Delegation nudge (opt-in, off by default)
  107. The judge can also look at **your prompt**, before the main agent sees it, and suggest that the main agent hand
  108. parts of the work to subagents. It is off unless you turn it on:
  109. ```sh
  110. mkdir -p ~/.claude/model-judge
  111. echo '{"delegationNudge": "on"}' > ~/.claude/model-judge/config.json # merged over the plugin's config.json
  112. ```
  113. A `UserPromptSubmit` hook (`scripts/nudge.mjs`) sends the prompt to the same judge (`claude -p --model haiku`, or
  114. the Messages API with `ANTHROPIC_API_KEY`; same recursion guard, same `timeoutMs`, fails open). The judge answers
  115. with the task's difficulty and the parts of it that could run **independently**: changes to separate files or
  116. modules, several separate questions, research or a long search that can run on its own. Only when the task is
  117. at least `medium`, the judge is sure enough, and there are at least 2 such parts does the hook add context for
  118. the main agent:
  119. ```
  120. model-judge delegation hint (from the model-judge plugin, not from the user): this task looks hard (confidence 0.85) and splits into 3 parts that can run independently:
  121. 1. Trace how a chat message flows from internal/server to the model call in internal/providers (medium → likely sonnet / medium effort)
  122. 2. Review internal/sandbox and list every way a command could reach the host outside the sandbox (hard → likely opus / high effort)
  123. 3. List every configuration key read in internal/config and check whether docs/ documents each (medium → likely sonnet / medium effort)
  124. Recommended: hand these parts to subagents with the Agent tool, one subagent per part, all in one message so they run in parallel. […]
  125. Keep in the main agent any part that is only a quick lookup or a small edit, work that is tightly coupled, and putting the parts' results together. This is a recommendation, not a requirement: […]
  126. ```
  127. and shows you one line (it follows `display`: shown for `all` and `changes`, hidden for `off`):
  128. ```
  129. UserPromptSubmit says: ⚖ model-judge: suggested delegation: 3 parts (hard 0.85, 3.5s)
  130. ```
  131. It never blocks the prompt and never forces anything: the main agent decides. The models in the hint are
  132. guesses from `routes`; the subagent judge still decides each subagent's model and effort for real when it
  133. starts, as always. Easy, single-step, tightly coupled and conversational prompts get nothing. The judge's
  134. answer is logged either way (`"kind": "nudge"`, status `nudge`, `none` or `failed`) and shows in
  135. `/model-judge:log` and `log.mjs` as a `prompt: …` row with `N parts` and result `nudge` / `no-nudge`.
  136. **What is never judged** (no judge call, nothing logged): prompts shorter than `nudgeMinChars` (80; this also
  137. covers "yes", "go on" and other short follow-ups), slash commands, prompts Claude Code submits itself (a
  138. background subagent's hand-back `<agent-message …>`, task and shell notifications, local command output), and
  139. anything inside the judge's own runs. The same prompt within `cacheTtlMs` reuses the cached answer.
  140. **The trade-off.** A subagent starts with a fresh, empty context: it has to rediscover what the main agent
  141. already knows, and its report then has to be read back. That pays off for big or parallel work (several
  142. independent investigations across a large codebase run at once, and keep the file dumps out of the main
  143. context). For small tasks it is pure overhead. Every qualifying prompt also waits for one judge call before the
  144. main agent starts: **~3–5 s** (measured 3.0–5.5 s, median 4.4 s, over 23 real calls here, `claude -p` judge
  145. on Haiku 5.5, no API key; the nudge judge writes a longer answer than the subagent judge), about $0.0002 each. With the nudge off, the hook only starts `node`, reads the config and exits (≈50 ms).
  146. What happened in the real runs (Claude Code 2.1.328, `--plugin-dir` the repo, main model Sonnet, nudge on):
  147. | prompt | where | nudge | what the main agent did |
  148. |---|---|---|---|
  149. | "What is confidenceThreshold in config.json…" (easy) | this repo, headless default + auto, interactive default + auto | none (easy 0.97), no line | answered itself |
  150. | three read-only questions about this ~20-file repo | this repo, headless | nudge (medium, 3 parts) | answered all three itself, with Bash/Read (Sonnet and Opus as main) |
  151. | three independent investigations of a 1,147-file Go repo | headless default + auto, interactive default + auto | nudge (hard 0.85, 3 parts) | started 3 subagents in one message; model-judge set sonnet/medium, opus/high, sonnet/medium at spawn |
  152. The second row is the trade-off at work: the main agent judged a small repo cheaper to read itself and skipped
  153. the hint, which is what "never forces" allows. The settings that tune it:
  154. | key | default | meaning |
  155. |---|---|---|
  156. | `delegationNudge` | `"off"` | `"on"` turns the nudge on |
  157. | `nudgeMinChars` | `80` | shorter prompts are never judged |
  158. | `nudgeMinDifficulty` | `"medium"` | the whole task must be at least this hard |
  159. | `nudgeMinConfidence` | `0.7` | the judge's confidence must reach this |
  160. | `nudgeMinParts` | `2` | at least this many independent parts (never less than 2) |
  161. ## Seeing what the judge does
  162. Three views. All of them run without enowX, and none touches the decision or the permission check.
  163. ### 1. A line under each subagent call (on by default)
  164. The hook adds a `systemMessage` to its answer. Claude Code 2.1.328 shows a PreToolUse hook's `systemMessage`
  165. to the user right under the tool call, as `PreToolUse:Agent says: …`. This was captured from a real interactive
  166. session (`claude --plugin-dir … --permission-mode default`, run in a pseudo-terminal), where one prompt
  167. started four subagents, all requesting `sonnet`:
  168. ```
  169. ⏺ 4 background agents launched (↓ to manage)
  170. ├ Agent (count words)
  171. ├ Agent (slugify with tests)
  172. ├ Agent (rate limiter design)
  173. └ Explore (vague task)
  174. ⎿ PreToolUse:Agent says: ⚖ model-judge: "count words" → easy 0.97 · sonnet→haiku · effort low (3.4s)
  175. ⎿ PreToolUse:Agent says: ⚖ model-judge: "slugify with tests" → medium 0.85 · sonnet→sonnet · effort medium (2.7s)
  176. ⎿ PreToolUse:Agent says: ⚖ model-judge: "rate limiter design" → hard 0.85 · sonnet→opus · effort high (4.3s)
  177. ⎿ PreToolUse:Agent says: ⚖ model-judge: "vague task" unsure 0.40 (easy?) → left as sonnet (3.5s)
  178. ```
  179. The subagents' transcripts confirm it: `count words` ran on `claude-haiku-5-5` (effort low), `slugify with tests`
  180. on `claude-sonnet-5-5` (medium), `rate limiter design` on `claude-opus-5-5` (high), and `vague task` stayed on
  181. `claude-sonnet-5-5` with no effort set.
  182. In **auto mode**, with the plugin installed (`claude --permission-mode auto --model sonnet`, the mixed prompt),
  183. the capture shows one launch, no refusals, and model-judge's status line at the bottom:
  184. ```
  185. ⏺ 3 background agents launched (↓ to manage)
  186. ├ count words
  187. ├ slugify with tests
  188. └ distributed rate limiter
  189. ⎿ PreToolUse:Agent says: ⚖ model-judge: "count words" → easy 0.97 · sonnet→haiku · effort low (3.1s)
  190. ⎿ PreToolUse:Agent says: ⚖ model-judge: "slugify with tests" → medium 0.90 · sonnet→sonnet · effort medium (4.0s)
  191. ⎿ PreToolUse:Agent says: ⚖ model-judge: "distributed rate limiter" → hard 0.90 · sonnet→opus · effort high (3.0s)
  192. …
  193. ⏺ Agent "distributed rate limiter" finished · 24s
  194. ──────────────────────────────────────────────
  195. ❯
  196. ──────────────────────────────────────────────
  197. ⚖ opus/high · hard 0.90 · "distributed rate limiter"
  198. ⏵⏵ auto mode on (shift+tab to cycle) · /tasks to see subagents
  199. ```
  200. The other forms of the line:
  201. ```
  202. ⚖ model-judge: "x" judge failed (cli: claude -p timed out after 15000 ms) → left as sonnet (15.0s)
  203. ⚖ model-judge: "x" skipped (fork) → left as default
  204. ```
  205. `"display"` in `config.json` sets how much it says:
  206. - `"all"` (default): every decision.
  207. - `"changes"`: only when the model or effort changed, or the judge failed.
  208. - `"off"`: nothing; the log still records everything.
  209. When the input is left unchanged, the hook prints only `{"systemMessage": …}`, with no `hookSpecificOutput`.
  210. ### 2. `/model-judge:log [N]`, and the same in a terminal
  211. The plugin adds a slash command, `/model-judge:log`. It runs `scripts/log.mjs` and shows the last N decisions,
  212. 10 by default. It runs `node` only, which its `allowed-tools` frontmatter permits, so it asks for no permission.
  213. Captured from the interactive session:
  214. ```
  215. ❯ /model-judge:log
  216. ⏺ time subagent verdict conf model effort judge result
  217. ──────── ──────────────────────────── ─────── ──── ───────────────── ───────────── ────── ─────────
  218. 22:44:07 count words easy 0.97 sonnet→haiku -→low 3.4s override
  219. 22:44:08 slugify with tests medium 0.85 sonnet→sonnet -→medium 2.7s override
  220. 22:44:09 vague task easy 0.40 sonnet→sonnet - 3.5s unsure
  221. 22:44:09 rate limiter design hard 0.85 sonnet→opus -→high 4.3s override
  222. (4 of 4, from …/decisions.jsonl)
  223. ```
  224. (The test session logged to a temp folder, so its path is shortened here.)
  225. The command's output reaches the model, which then repeats it to you. That costs one short model turn.
  226. In a terminal, with no model involved (from a clone of this repo, or the installed copy's folder):
  227. ```sh
  228. node scripts/log.mjs # last 10 (or: log.mjs 25, -n 25)
  229. node scripts/log.mjs --watch # then follow new decisions live, in another terminal
  230. node scripts/log.mjs --session SESSION_ID --json
  231. ```
  232. ### 3. Status line (opt-in)
  233. `scripts/statusline.mjs` shows the current session's latest decision under the prompt. It prints nothing until
  234. the session's first subagent. Captured at the bottom of the interactive session above:
  235. ```
  236. ❯
  237. ───────────────────────────────────────────
  238. ⚖ opus/high · hard 0.85 · "rate limiter design"
  239. ```
  240. The other forms: `⚖ sonnet (unsure 0.40) · "vague task"`, `⚖ default (judge failed)`. Nothing sets this up
  241. for you. A status line is a command in `~/.claude/settings.json`, and an installed plugin's folder changes with
  242. each version, so point it at the stable launcher instead. `scripts/statusline-launcher.mjs` finds the installed
  243. model-judge (its `installPath` in `~/.claude/plugins/installed_plugins.json`) and runs that version's
  244. `statusline.mjs`. To opt in, copy the launcher out of the plugin once:
  245. ```sh
  246. mkdir -p ~/.claude/model-judge
  247. cp "$(jq -r '.plugins["model-judge@semutjahat"][0].installPath' ~/.claude/plugins/installed_plugins.json)/scripts/statusline-launcher.mjs" \
  248. ~/.claude/model-judge/statusline.mjs
  249. ```
  250. Then add this to `~/.claude/settings.json`, with your home folder in the path:
  251. ```json
  252. "statusLine": { "type": "command", "command": "node /Users/YOU/.claude/model-judge/statusline.mjs" }
  253. ```
  254. To try it for one session without editing any settings file:
  255. `claude --settings '{"statusLine":{"type":"command","command":"node /Users/YOU/.claude/model-judge/statusline.mjs"}}'`.
  256. If you already have a status line, call both from one command, and pass the same stdin to each:
  257. `sh -c 'i=$(cat); printf %s "$i" | your-statusline; printf " "; printf %s "$i" | node …/statusline.mjs'`.
  258. To remove it, delete the `"statusLine"` entry from `~/.claude/settings.json` (and, if you like,
  259. `~/.claude/model-judge/statusline.mjs`). New sessions then show no status line.
  260. ## Try it yourself
  261. None of these install anything. They use the plugin straight from a clone of this repo
  262. (`git clone https://github.com/SemutJahat/model-judge && cd model-judge`).
  263. **1. One-command demo** (about 30–60 s, a few cents):
  264. ```sh
  265. ./examples/demo.sh # examples/prompts/mixed.md: easy + medium + hard, all requesting sonnet
  266. ./examples/demo.sh examples/prompts/ambiguous.md # or any other prompt file
  267. ```
  268. It runs a headless session in a throwaway temp folder and prints one row per subagent: the model requested, the
  269. judge's verdict, what the hook set, and what the subagent started with and ran on. The real run is shown under
  270. [Real run](#real-run-evidence).
  271. **2. Prompts to paste into an interactive session.** Start Claude Code with the plugin loaded for that session:
  272. ```sh
  273. claude --plugin-dir . # from the clone
  274. ```
  275. Then paste a file from `examples/prompts/`. Each one asks for subagents with `model: "sonnet"` and gives each
  276. subagent's prompt in a ```` ```task ```` block:
  277. | file | what to expect |
  278. |---|---|
  279. | `easy.md` | count words: easy → **haiku**, effort low |
  280. | `medium.md` | slugify function with edge cases and tests: medium → **sonnet**, effort medium |
  281. | `hard.md` | distributed rate limiter design: hard → **opus**, effort high |
  282. | `mixed.md` | all three in one go |
  283. | `ambiguous.md` | "Look into it and do what's needed.": the judge is unsure (confidence ≈ 0.3–0.45 < 0.75), so the call is **left unchanged** (sonnet, no effort) |
  284. **3. Just the judge, no subagent** (about 3 s per prompt):
  285. ```sh
  286. node examples/dry-run.mjs "Count the words in: the quick brown fox"
  287. echo "Design a sharded job queue with exactly-once delivery" | node examples/dry-run.mjs --model sonnet
  288. node examples/dry-run.mjs --model sonnet --file examples/prompts/mixed.md # judges each task block
  289. node examples/dry-run.mjs --json --file examples/prompts/hard.md # the log line it would write
  290. ```
  291. It runs the same `decide()` as the hook, with the same config, judge and threshold. It prints the verdict and the
  292. model and effort it would set, and writes nothing to the log unless you pass `--log`.
  293. **Watching while you try.** In an interactive session, each subagent call shows its model-judge line (see
  294. [Seeing what the judge does](#seeing-what-the-judge-does)). Type `/model-judge:log` for the table, or keep
  295. `node scripts/log.mjs --watch` running in a second terminal.
  296. **Reading the log.** Every subagent the hook sees in a normal session adds a line to
  297. `~/.claude/model-judge/decisions.jsonl` (the demo writes to its temp folder instead). Use `node scripts/log.mjs`
  298. for a table, or `jq` for the raw lines:
  299. ```sh
  300. jq -c '{ts, description, difficulty, confidence, model: "\(.model_before) -> \(.model_after)", effort_after, action, reason}' ~/.claude/model-judge/decisions.jsonl | tail
  301. ```
  302. **Seeing the real model in a subagent's transcript.** Claude Code keeps each subagent's transcript next to the
  303. session's, in `~/.claude/projects/<project folder>/<session id>/subagents/`:
  304. ```sh
  305. S=$(echo ~/.claude/projects/*/SESSION_ID/subagents) # SESSION_ID: from the log line's session_id
  306. cat $S/agent-*.meta.json # {"description":…, "model":"haiku", "effort":null, "toolUseId":…}
  307. jq -r 'select(.type=="assistant") | .message.model' $S/agent-*.jsonl | sort | uniq -c # e.g. claude-haiku-5-5
  308. ```
  309. `toolUseId` in `meta.json` is the `tool_use_id` of the log line, so the two can be matched. `meta.json`'s
  310. `effort` stays `null`, because the effort is set per request, not on the call. The log's `"request"` line
  311. for the subagent (matched by `agent_id`) shows the effort its first request carried.
  312. ## Install / uninstall
  313. The repo is its own marketplace, named `semutjahat`. To install model-judge for your user (all your Claude
  314. Code sessions), from a terminal:
  315. ```sh
  316. claude plugin marketplace add SemutJahat/model-judge
  317. claude plugin install model-judge@semutjahat --scope user
  318. claude plugin list # model-judge@semutjahat … Scope: user · Status: ✔ enabled
  319. ```
  320. Or at the Claude Code prompt: `/plugin install model-judge --marketplace SemutJahat/model-judge`.
  321. The same marketplace also lists **[rag-first](https://github.com/SemutJahat/rag-first)**, which makes Claude
  322. Code search the project's enowx-rag RAG before it greps or reads around:
  323. `claude plugin install rag-first@semutjahat --scope user`.
  324. New sessions load the plugin; sessions that are already open don't. It needs Claude Code with hook modules
  325. (2.1.328 has them; see [Older Claude Code](#auto-mode-why-the-call-is-not-rewritten)), Node on the PATH, and
  326. your Claude Code login, which the judge uses.
  327. Claude Code installs a copy of the plugin in `~/.claude/plugins/cache/semutjahat/model-judge/<version>/`
  328. and runs it from there. To get a newer version after the repo changes:
  329. ```sh
  330. claude plugin marketplace update semutjahat # fetch the repo's latest marketplace and plugin
  331. claude plugin update model-judge@semutjahat # then open a new session
  332. ```
  333. To turn it off for a while, or remove it:
  334. ```sh
  335. claude plugin disable model-judge@semutjahat --scope user # off; on again: claude plugin enable model-judge@semutjahat --scope user
  336. claude plugin uninstall model-judge@semutjahat --scope user # remove the plugin
  337. claude plugin marketplace remove semutjahat # remove the marketplace
  338. ```
  339. To try it for one session without installing anything, from a clone: `claude --plugin-dir .`
  340. **Developing it.** A clone added as a local marketplace (`claude plugin marketplace add ./model-judge`) is read
  341. straight from that folder, not from a copy. Edits apply without reinstalling: `scripts/` and `config.json` from
  342. the next subagent call, and `hooks/` and `commands/` from the next new session.
  343. ## Config
  344. `config.json` in the plugin:
  345. ```json
  346. {
  347. "judgeModel": "haiku", // model for `claude -p --model …`
  348. "judgeApiModel": "claude-haiku-5-5", // model id when ANTHROPIC_API_KEY is used
  349. "useApiKey": true, // use ANTHROPIC_API_KEY when it is set, falling back to claude -p
  350. "confidenceThreshold": 0.75,
  351. "timeoutMs": 15000,
  352. "maxPromptChars": 6000, // the prompt is cut to this before it goes to the judge
  353. "setEffort": true,
  354. "display": "all", // the line under each subagent call: "all" | "changes" | "off"
  355. "apply": "spawn", // "spawn": set at agent.spawn/turn.step (works in auto mode); "input": 0.1's updatedInput
  356. "cacheTtlMs": 600000, // reuse a verdict for the same call for 10 min; 0 turns the cache off
  357. "delegationNudge": "off", // "on": suggest delegation for big multi-part prompts (see Delegation nudge)
  358. "nudgeMinChars": 80, "nudgeMinDifficulty": "medium", "nudgeMinConfidence": 0.7, "nudgeMinParts": 2,
  359. "routes": { "easy": {...}, "medium": {...}, "hard": {...} }
  360. }
  361. ```
  362. An installed plugin is a copy that each update replaces, so put your own settings in
  363. `~/.claude/model-judge/config.json`, which only needs the keys you change (e.g. `{"display": "changes"}`).
  364. Settings are read from the plugin's `config.json`, then `~/.claude/model-judge/config.json`, then the file
  365. in `$MODEL_JUDGE_CONFIG`. Later files win, key by key, and `routes` merge by level. Route models can be
  366. aliases (`haiku`, `sonnet`, `opus`, `fable`) or full ids (`claude-opus-5-5` → `opus`). `MODEL_JUDGE_CLAUDE_BIN`
  367. names another `claude` binary.
  368. How the judge is called: `claude -p --model haiku --tools "" --strict-mcp-config --setting-sources ""
  369. --no-session-persistence --disable-slash-commands --output-format json --system-prompt …`. It runs in a fresh
  370. empty temp folder (removed afterwards) with `MODEL_JUDGE_INNER=1` in its environment. Loading no settings
  371. files means it loads no plugins, so this hook can't run again inside it, and the env guard covers the rest.
  372. ## The log
  373. Every decision is one JSON line in **`~/.claude/model-judge/decisions.jsonl`** (or in `$MODEL_JUDGE_DIR`):
  374. ```json
  375. {"ts":"2026-11-19T11:46:17.930Z","session_id":"e370a8ce-…","tool_use_id":"toolu_016V…","tool":"Agent",
  376. "description":"trivial echo","subagent_type":"general-purpose",
  377. "prompt_head":"Reply with exactly the word pong. Do not use any tools.",
  378. "model_before":"sonnet","effort_before":null,"judge":"cli","judge_ms":2819,
  379. "difficulty":"easy","confidence":0.99,"why":"The task is a trivial reply of a single fixed word with no tools, requiring no reasoning.",
  380. "action":"override","model_after":"haiku","effort_after":"low"}
  381. ```
  382. `action` is `override` or `unchanged`. `status` is `override`, `unsure`, `failed` or `skipped`, and an
  383. unchanged line has a `reason`. `prompt_head` is the prompt's first 200 characters. `judge` is `cli`, `api` or
  384. `cache`. `apply` is `spawn` or `input`.
  385. A delegation nudge (when it is on) is a line with `"kind": "nudge"`: `status` `nudge`, `none` or `failed`,
  386. `prompt_head`, `prompt_chars`, `difficulty`, `confidence`, `why`, `parts` (`[{task, difficulty}]`), `judge`,
  387. `judge_ms`, and a `reason` when no nudge was added. The status line ignores these lines.
  388. The hooks module adds two lines per routed subagent, with an `event` field (the views skip them):
  389. ```json
  390. {"event":"applied","tool_use_id":"toolu_…","agent_id":"a168964b54e9438ea","model":"claude-haiku-5-5","effort":"low"}
  391. {"event":"request","agent_id":"a168964b54e9438ea","model":"claude-haiku-5-5","effort":"low","wanted":"low"}
  392. ```
  393. `applied` is written at `agent.spawn`, and `model` is what Claude Code resolved for the subagent. `request` is
  394. what its first model request really carried. A decision with no `applied` line shows as `pending` in
  395. `log.mjs`, meaning the module didn't apply it (for example, because hook modules aren't available).
  396. ```sh
  397. tail -f ~/.claude/model-judge/decisions.jsonl | jq -c '{description, difficulty, confidence, model_before, model_after, effort_after, why}'
  398. ```
  399. ## Tests
  400. ```sh
  401. npm test # node --test test/: 70 tests, the judge stubbed (test/fixtures/fake-claude.mjs, a local fake Messages API)
  402. ./test/real-run.sh # the real check below, costs a few cents (same machinery as examples/demo.sh)
  403. ```
  404. The unit tests cover: easy, medium and hard prompts; a model and effort already set and then overridden;
  405. `Task` as the tool name; the judge's flags, temp folder and recursion guard; the judge missing, failing,
  406. answering garbage, unsure or too slow; forks and other tools; `CLAUDE_CODE_EFFORT_LEVEL` and
  407. `CLAUDE_CODE_SUBAGENT_MODEL_FORCE`; config overrides; the API path and its fallback to `claude -p`; the line
  408. shown to the user in each case and each `display` mode; the status line text; the `log.mjs` table, its
  409. `--session` and `--json` options and an empty log. `test/nudge.test.mjs` covers the delegation nudge: off by
  410. default, on through the personal config, a multi-part prompt (context, line, log), easy / coupled / unsure
  411. prompts, the cheap skips (short, slash command, subagent hand-back, recursion guard), `nudgeMinChars`, the
  412. cache, the judge failing, too slow or missing, `display`, the thresholds, the hint's models and effort, and
  413. the `log.mjs` rows.
  414. ### Real run (evidence)
  415. `examples/demo.sh` and `test/real-run.sh` run a headless `claude -p --plugin-dir <the clone> --setting-sources ""
  416. --strict-mcp-config --model sonnet` session in a temp folder (`test/headless-run.sh`; set `PERMISSION_MODE` to
  417. pick the mode). Nothing is installed and no settings files are read. Afterwards `test/run-report.mjs` matches
  418. each Agent call's tool call, result, log lines and transcript by `tool_use_id`. Here is the auto-mode run of
  419. 2026-11-19 (three subagents in one go, all requesting `sonnet`):
  420. ```
  421. | Subagent | Requested | Judge verdict | Decision | Call | Started on (agent.spawn) | First request (model / effort) | Ran on (transcript) |
  422. |--------------------------|-----------|---------------|-----------------|------|--------------------------|--------------------------------|---------------------|
  423. | count words | sonnet | easy 0.97 | haiku / low | ran | claude-haiku-5-5 | claude-haiku-5-5 / low | claude-haiku-5-5 |
  424. | slugify with tests | sonnet | medium 0.9 | sonnet / medium | ran | claude-sonnet-5-5 | claude-sonnet-5-5 / medium | claude-sonnet-5-5 |
  425. | distributed rate limiter | sonnet | hard 0.9 | opus / high | ran | claude-opus-5-5 | claude-opus-5-5 / high | claude-opus-5-5 |
  426. Agent calls: 3 for 3 subagent task(s); refused: 0; judge calls: 3 (+0 from cache)
  427. ```
  428. How to read the columns:
  429. - **Started on:** the `applied` log line, written at `agent.spawn`.
  430. - **First request:** the `request` log line, the model and effort as sent.
  431. - **Ran on:** the `message.model` of the subagent's responses in its `agent-*.jsonl`.
  432. The main transcript still shows each call as the model wrote it (`model: "sonnet"`), and that is the point: the
  433. call isn't rewritten, so auto mode's review of it holds. The default, acceptEdits and bypassPermissions runs
  434. gave the same table.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.