Skip to content
Back to search

skill-conductor

Architecture-first skill lifecycle for AI agents. BinEval binary scoring with threshold-blind, cross-family-calibrated judges, gated self-update loop, pressure testing, 10 authoring principles grounded in empirical research.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/smixs--skill-conductor/badge.svg)](/listings/smixs--skill-conductor)

HTML

<a href="/listings/smixs--skill-conductor"><img src="/api/listings/smixs--skill-conductor/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# skills.sh — installs into ~/.claude/skills
npx skills add smixs/skill-conductor
# Claude Code plugin
/plugin marketplace add smixs/skill-conductor
/plugin install skill-conductor@smixs

<details> <summary><strong>v3.2.0 — Evidence-based upgrade: form-matching, judge calibration, pressure testing</strong></summary>

- **Principle #10: Match the form to the failure** — classify the baseline failure before writing a rule; prohibitions bulletproof discipline failures but measurably backfire on shaping failures (obra/superpowers wording tests + Guardrails polarity data). Plus: no nuance clauses, exemption clauses don't scope. - **Critique-before-verdict judges** — all three eval agents (grader, comparator, bineval) now write the detailed evidence critique BEFORE committing to the 1/0 verdict, with a borderline few-shot example in each (Hamel Husain's judge methodology). - **Threshold-blind judging** — the BinEval judge no longer computes the overall score or the GATE; the orchestrator aggregates. A judge that knows the bar is biased toward it. - **Automatic cross-family judge calibration** — a second judge from a different model family answers the same bank; stable disagreement flags a badly worded question, not a dispute. Self-preference-bias guard on final acceptance. - **Variance discipline** — improvements on non-critical questions count only when they reproduce in 2 consecutive runs; the 3-iteration cap now carries its STICK rationale. - **`references/pressure-testing.md`** — micro-test protocol (no-guidance control, 5+ reps, variance as a metric) + pressure scenarios for discipline skills (7 pressure types, forced A/B/C choice, rationalization tables). - **Pushy description formula** — `[What] + Use when [4-5 phrasings] + "even if they don't explicitly say '<canonical term>'" + Do NOT use for [...]`, deduped to a single canonical home in Principle #2. - **Question bank v1.1** — 5 new questions: pushy triggers, nuance clauses, directive reference loading, time-rot language, redundant-content (E:A:R). - **Self-hosted proof** — this release was produced by Conductor evaluating and improving itself: 3 gated iterations, dual-family judges (Claude + GPT via codex), all critical questions passing.

</details>

<details> <summary><strong>v3.1.0 — Gated self-update: held-out gate + edit budget (SkillOpt core)</strong></summary>

- **Held-out gate for body edits** — Mode 2 IMPROVE now splits evals into train/held-out (`scripts/split_evals.py`, deterministic, stratified by optional `evals[].category`). Lessons and edits come from TRAIN only; a candidate is accepted iff no held-out assertion regresses (flip-confirmation re-run for noise), train pass-rate strictly improves, and no new critical failure. Methodology borrowed from [microsoft/SkillOpt](https://github.com/microsoft/SkillOpt). - **Edit budget** — at most 3 atomic edits per iteration (one edit = one lesson, labeled); no wholesale rewrites, so gate rejections stay attributable. - **Case transitions** — assertion-level diff parent→candidate (improved / regressed / persistent-fail / stable-success) recorded as an additive `transitions` block in benchmark.json. - **Refactor** — `split_eval_set` generalized into `utils.split_evals(stratify_key=...)`; `run_loop.py` (Mode 5 OPTIMIZE) delegates to it, split unchanged bit-for-bit.

</details>

<details> <summary><strong>v3.0.0 — BinEval scoring, English canon, dual-channel install</strong></summary>

- **BinEval evaluation** — replaces the old 5-axis 1-10 scoring with atomic binary yes/no questions across 5 dimensions (Discovery, Clarity, Structure, Robustness, Completeness). Each answer carries grounding evidence; the pass criterion is a **gate on critical questions**, not an opaque number. Adapted from *"Ask, Don't Judge"* ([arXiv 2606.27226](https://arxiv.org/abs/2606.27226)).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 19 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "smixs",
  3. "version": "1.0.0",
  4. "description": "Agent-skill tooling by smixs.",
  5. "owner": {
  6. "name": "smixs",
  7. "email": "shimaozb@gmail.com"
  8. },
  9. "plugins": [
  10. {
  11. "name": "skill-conductor",
  12. "source": "./",
  13. "description": "Create, edit, evaluate, and package Claude Code agent skills — full draft→test→review→improve lifecycle.",
  14. "version": "3.2.0",
  15. "author": { "name": "smixs" },
  16. "category": "development"
  17. }
  18. ]
  19. }
README.md · README · 229 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src="assets/conductor.png" alt="Skill Conductor" width="100%">
  3. </p>
  4. # Skill Conductor
  5. > A skill that creates, evaluates, and improves other skills. Meta-level.
  6. [![Release](https://img.shields.io/github/v/release/smixs/skill-conductor)](https://github.com/smixs/skill-conductor/releases)
  7. [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
  8. [![Install](https://img.shields.io/badge/install-npx%20skills%20add-black)](https://skills.sh)
  9. [![Claude Code](https://img.shields.io/badge/Claude%20Code-plugin-d97757)](https://code.claude.com/docs/en/plugins)
  10. Architecture-first skill lifecycle: **design → build → test → evaluate → package**.
  11. Most skill tools jump straight to "write SKILL.md." Conductor makes you choose the architecture first — because rewriting a wrong pattern costs more than writing it right.
  12. ## Install
  13. ```bash
  14. # skills.sh — installs into ~/.claude/skills
  15. npx skills add smixs/skill-conductor
  16. ```
  17. ```bash
  18. # Claude Code plugin
  19. /plugin marketplace add smixs/skill-conductor
  20. /plugin install skill-conductor@smixs
  21. ```
  22. <details>
  23. <summary><strong>v3.2.0 — Evidence-based upgrade: form-matching, judge calibration, pressure testing</strong></summary>
  24. - **Principle #10: Match the form to the failure** — classify the baseline failure before writing a rule; prohibitions bulletproof discipline failures but measurably backfire on shaping failures (obra/superpowers wording tests + Guardrails polarity data). Plus: no nuance clauses, exemption clauses don't scope.
  25. - **Critique-before-verdict judges** — all three eval agents (grader, comparator, bineval) now write the detailed evidence critique BEFORE committing to the 1/0 verdict, with a borderline few-shot example in each (Hamel Husain's judge methodology).
  26. - **Threshold-blind judging** — the BinEval judge no longer computes the overall score or the GATE; the orchestrator aggregates. A judge that knows the bar is biased toward it.
  27. - **Automatic cross-family judge calibration** — a second judge from a different model family answers the same bank; stable disagreement flags a badly worded question, not a dispute. Self-preference-bias guard on final acceptance.
  28. - **Variance discipline** — improvements on non-critical questions count only when they reproduce in 2 consecutive runs; the 3-iteration cap now carries its STICK rationale.
  29. - **`references/pressure-testing.md`** — micro-test protocol (no-guidance control, 5+ reps, variance as a metric) + pressure scenarios for discipline skills (7 pressure types, forced A/B/C choice, rationalization tables).
  30. - **Pushy description formula** — `[What] + Use when [4-5 phrasings] + "even if they don't explicitly say '<canonical term>'" + Do NOT use for [...]`, deduped to a single canonical home in Principle #2.
  31. - **Question bank v1.1** — 5 new questions: pushy triggers, nuance clauses, directive reference loading, time-rot language, redundant-content (E:A:R).
  32. - **Self-hosted proof** — this release was produced by Conductor evaluating and improving itself: 3 gated iterations, dual-family judges (Claude + GPT via codex), all critical questions passing.
  33. </details>
  34. <details>
  35. <summary><strong>v3.1.0 — Gated self-update: held-out gate + edit budget (SkillOpt core)</strong></summary>
  36. - **Held-out gate for body edits** — Mode 2 IMPROVE now splits evals into train/held-out (`scripts/split_evals.py`, deterministic, stratified by optional `evals[].category`). Lessons and edits come from TRAIN only; a candidate is accepted iff no held-out assertion regresses (flip-confirmation re-run for noise), train pass-rate strictly improves, and no new critical failure. Methodology borrowed from [microsoft/SkillOpt](https://github.com/microsoft/SkillOpt).
  37. - **Edit budget** — at most 3 atomic edits per iteration (one edit = one lesson, labeled); no wholesale rewrites, so gate rejections stay attributable.
  38. - **Case transitions** — assertion-level diff parent→candidate (improved / regressed / persistent-fail / stable-success) recorded as an additive `transitions` block in benchmark.json.
  39. - **Refactor** — `split_eval_set` generalized into `utils.split_evals(stratify_key=...)`; `run_loop.py` (Mode 5 OPTIMIZE) delegates to it, split unchanged bit-for-bit.
  40. </details>
  41. <details>
  42. <summary><strong>v3.0.0 — BinEval scoring, English canon, dual-channel install</strong></summary>
  43. - **BinEval evaluation** — replaces the old 5-axis 1-10 scoring with atomic binary yes/no questions across 5 dimensions (Discovery, Clarity, Structure, Robustness, Completeness). Each answer carries grounding evidence; the pass criterion is a **gate on critical questions**, not an opaque number. Adapted from *"Ask, Don't Judge"* ([arXiv 2606.27226](https://arxiv.org/abs/2606.27226)).
  44. - **Deterministic + LLM split** — `eval_skill.py --json` emits structural checks as binary question records; an evaluator agent answers the judgment questions with evidence and a self-update loop feeds failing questions back into edits.
  45. - **9 authoring principles** — a universal canon (pre-flight, no-process-in-description, MOC, fresh-practitioner author, TWI "why", blind-agent test, inline checklists, one-term-per-concept, cut-the-fat) in `references/sop-practices.md`, applied to every skill.
  46. - **Dual-channel install** — one repo, one source of truth, installable via skills.sh and the Claude Code plugin marketplace.
  47. </details>
  48. <details>
  49. <summary><strong>v3: SOP practices + smoke tests</strong></summary>
  50. - **`references/sop-practices.md`** — 80 years of Standard Operating Procedure wisdom applied to skill authoring. Inline checklists at risk-points, pre-flight checks, programmatic validation, exception handling patterns. Use for procedural skills (client intake, onboarding, reporting, escalation)
  51. - **`scripts/test_smoke.py`** — fast safety net for skill-conductor scripts themselves. Verifies critical scripts execute on known-good skills, fail on known-bad, produce expected output shapes. Run: `uv run scripts/test_smoke.py`
  52. - Updated eval agents (grader, comparator, analyzer) with refined rubrics
  53. - Improved `package_skill.py`, `eval_skill.py`, and schema validation
  54. - Updated `patterns.md` and `schemas.md` with tighter definitions
  55. </details>
  56. <details>
  57. <summary><strong>v2: Anthropic's eval engine meets architecture-first design</strong></summary>
  58. Anthropic [updated their skill-creator](https://claude.com/blog/improving-skill-creator-test-measure-and-refine-agent-skills) with serious eval infrastructure. We took the best of it:
  59. **From Anthropic's skill-creator:**
  60. - 3 specialized agents: **grader** (assertion checking + claim extraction), **comparator** (blind A/B testing), **analyzer** (post-hoc root cause analysis)
  61. - Parallel eval execution with isolated contexts (no cross-contamination)
  62. - Automated description optimization with train/test split (60/40)
  63. - Benchmark tracking: pass rate, tokens, time with variance analysis
  64. - HTML eval viewer with qualitative + quantitative tabs
  65. **What Conductor adds on top:**
  66. - **Architecture before code.** 5 patterns (Sequential, Iterative, Context-Aware, Domain Intelligence, Multi-MCP) with selection criteria. Pick wrong = rewrite everything later
  67. - **Degrees of freedom.** Low (deterministic scripts) → Medium (pseudocode) → High (free text). Match freedom to risk tolerance
  68. - **TDD RED before writing.** Verify the agent fails WITHOUT the skill first. If it already handles the task — you don't need a skill
  69. - **Quality scoring with a gate** (now BinEval — see v1.0.0). Numbers and evidence, not a "vibe check"
  70. - **Skill categorization.** Capability uplift (teaching something new) vs Encoded preference (sequencing known abilities). Different skills need different testing strategies
  71. </details>
  72. ## Synthesized from
  73. 1. **[Anthropic Skill Creator](https://github.com/anthropics/claude-plugins-official/tree/main/plugins/skill-creator)** — eval infrastructure, grader/comparator/analyzer agents, benchmark pipeline
  74. 2. **[The Complete Guide to Building Skills for Claude](https://claude.com/blog/complete-guide-to-building-skills-for-claude)** — architecture patterns, success metrics
  75. 3. **[Superpowers / writing-skills](https://github.com/obra/superpowers/blob/main/skills/writing-skills/SKILL.md)** by Jesse Vincent — TDD approach, the "description trap" discovery, match-the-form-to-the-failure, the micro-test protocol, pressure scenarios and rationalization tables
  76. 4. **[Skills Best Practices](https://github.com/mgechev/skills-best-practices)** by Minko Gechev — three-stage LLM validation, eval methodology
  77. 5. **[hamelsmu/evals-skills](https://github.com/hamelsmu/evals-skills)** by Hamel Husain — critique-before-verdict judge outputs, borderline few-shot examples, judge calibration discipline
  78. 6. **[grafana/skills — skill-authoring](https://github.com/grafana/skills)** — the pushy description pattern in production, judge score-variance discipline ("three consecutive local passes before shipping")
  79. 7. **[softaworks/agent-toolkit — skill-judge](https://github.com/softaworks/agent-toolkit)** — the Expert/Activation/Redundant knowledge-delta taxonomy, directive loading triggers, the freedom-consequence test
  80. 8. **[neolabhq/context-engineering-kit](https://github.com/neolabhq/context-engineering-kit)** — threshold-blind judges (never tell the judge the bar)
  81. 9. **[trailofbits/skills — skill-improver](https://github.com/trailofbits/skills)** — the stop-hook pattern for unattended improvement loops (referenced, not implemented)
  82. ### Methodology foundations
  83. - **[Standard Operating Procedures: A Writing Guide](https://extension.psu.edu/standard-operating-procedures-a-writing-guide)** — Richard Stup, Penn State Extension. Format selection, hierarchical vs. flowchart procedures.
  84. - **[Procedure Writing: Principles and Practices](https://books.google.com/books/about/Procedure_Writing.html?id=Tm5RAAAAMAAJ)** — Wieringa, Moore & Barnes (Battelle Press, 1998). Imperative steps, removing modal weasel-words.
  85. - **[Toyota TWI (Training Within Industry)](https://www.allaboutlean.com/wp-content/uploads/2019/01/TWI_Job_Instruction_Manual.pdf)** — the "Job Instruction" method: step → key point → why; the 5 Whys root-cause practice ([Job Methods manual](https://www.allaboutlean.com/wp-content/uploads/2019/01/TWI_Job_Methods_Manual.pdf), [The Roots of Lean](https://www.lean.org/downloads/105.pdf)).
  86. - **McDonald's Operations Manual** — the canonical 600+ page SOP system; checklists at the point of use. The manual itself is proprietary; it is documented in John F. Love's [*McDonald's: Behind the Arches*](https://archive.org/details/mcdonaldsbehinda0000love).
  87. - **[Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement](https://arxiv.org/abs/2606.27226)** — the BinEval method behind Conductor's evaluation.
  88. - **[Guardrails Beat Guidance](https://arxiv.org/abs/2604.11088)** (5000+ Claude Code runs on SWE-bench) — rule polarity: helpful rules are negative constraints, harmful ones positive directives → Principle #10.
  89. - **[TICK: Generated Checklists Improve LLM Evaluation and Generation](https://arxiv.org/abs/2410.03608)** — checklist as spec + eval + feedback; refinement plateaus and degrades past 3–4 iterations → the 3-iteration cap.
  90. - **[CheckEval](https://arxiv.org/abs/2403.18771)** and **[Prosa](https://arxiv.org/abs/2605.01630)** — binary decomposition makes judges reproducible across model families → cross-family judge calibration.
  91. - **[Self-Preference Bias in Rubric-Based Evaluation](https://arxiv.org/abs/2604.06996)** — judges favor their own family even on binary rubrics → the out-of-family acceptance rule.
  92. - **[LLMs Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798)** and **[BIG-Bench Mistake](https://arxiv.org/abs/2311.08516)** — self-correction needs an external gate; models fix errors well only when an external checker locates them → the gated self-update loop.
  93. - **[SkillJuror](https://arxiv.org/abs/2606.11543)** — progressive disclosure with explicit loading triggers beats both flat files and passive reference lists → directive loading rules.
  94. - **[SkillReducer](https://arxiv.org/abs/2603.29919)** — >60% of public skill-body text changes no agent behavior → the actionability test in Principle #9.
  95. - **[IFScale](https://arxiv.org/abs/2507.11538)** and **[Prompt Design at Scale](https://arxiv.org/abs/2607.19257)** — compliance collapses near 80 simultaneous rules; format matters less than rule count → the rule budget and MOC structure.
  96. ## 6 Modes
  97. | Mode | What it does |
  98. |------|-------------|
  99. | **CREATE** | Architecture selection → TDD baseline → scaffold → write → verify → refactor |
  100. | **IMPROVE** | Diagnose → eval loop → self-update loop (failing questions → targeted edits) → iterate |
  101. | **VALIDATE** | Structural checks + trigger testing + BinEval scoring |
  102. | **REVIEW** | Pass/fail quality gate for third-party skills before you install them |
  103. | **OPTIMIZE** | Auto-tune the description for accurate triggering with a train/test split |
  104. | **PACKAGE** | Validate structure + package as `.skill` for distribution |
  105. ## Architecture patterns
  106. Choose before writing a single line:
  107. | Pattern | Use when |
  108. |---|---|
  109. | Sequential workflow | Clear step-by-step process |
  110. | Iterative refinement | Output improves with cycles |
  111. | Context-aware selection | Same goal, different tools by context |
  112. | Domain intelligence | Specialized knowledge beyond tool access |
  113. | Multi-MCP coordination | Workflow spans multiple services |
  114. ## Eval infrastructure
  115. ```
  116. ┌─────────┐
  117. │ SKILL │
  118. └────┬────┘
  119. │
  120. ┌──────────┼──────────┐
  121. │ │ │
  122. ┌────▼────┐ ┌──▼───┐ ┌───▼────┐
  123. │ Grader │ │ A/B │ │Analyzer│
  124. │ │ │Blind │ │ │
  125. │assertions│ │compare│ │root │
  126. │+ claims │ │ │ │cause │
  127. └─────────┘ └──────┘ └────────┘
  128. │ │ │
  129. └──────────┼──────────┘
  130. │
  131. ┌─────▼─────┐
  132. │ Benchmark │
  133. │ mean±std │
  134. └───────────┘
  135. ```
  136. Quality is scored with **BinEval**: binary yes/no questions per dimension, each answered with evidence; the skill passes when every *critical* question answers yes — not when a scalar clears a threshold.
  137. ## Installation layout
  138. ```
  139. skills/
  140. └── skill-conductor/
  141. ├── SKILL.md
  142. ├── agents/
  143. │ ├── grader.md
  144. │ ├── comparator.md
  145. │ ├── analyzer.md
  146. │ └── bineval.md
  147. ├── eval-viewer/
  148. │ ├── generate_review.py
  149. │ └── viewer.html
  150. ├── references/
  151. │ ├── patterns.md
  152. │ ├── schemas.md
  153. │ ├── sop-practices.md
  154. │ ├── bineval-method.md
  155. │ ├── quality-questions.md
  156. │ ├── pressure-testing.md
  157. │ └── runtime-setup.md
  158. ├── assets/
  159. │ └── eval_review.html
  160. └── scripts/
  161. ├── init_skill.py
  162. ├── eval_skill.py
  163. ├── run_eval.py
  164. ├── run_loop.py
  165. ├── improve_description.py
  166. ├── aggregate_benchmark.py
  167. ├── generate_report.py
  168. ├── package_skill.py
  169. ├── quick_validate.py
  170. ├── test_smoke.py
  171. └── utils.py
  172. ```
  173. **Claude Code:** the install commands above drop it into `.claude/skills/`. Auto-activates when the agent detects a skill-building task.
  174. ## Key discovery
  175. Never put process steps in the skill description. If your description says "exports assets, generates specs, creates tasks" — the model follows the description and skips the body. Tested experimentally.
  176. ```yaml
  177. # ✅ Good
  178. description: Analyze design files for developer handoff. Use when user uploads .fig files.
  179. # ❌ Bad - model follows this and ignores SKILL.md body
  180. description: Exports Figma assets, generates specs, creates Linear tasks, posts to Slack.
  181. ```
  182. ## License
  183. MIT — see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.