Skip to content
Back to search

autoresearch-skill

SkillGitHubIncomplete

autoresearch-skill (no description available)

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/olelehmann100kmrr--autoresearch-skill/badge.svg)](/listings/olelehmann100kmrr--autoresearch-skill)

HTML

<a href="/listings/olelehmann100kmrr--autoresearch-skill"><img src="/api/listings/olelehmann100kmrr--autoresearch-skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 329 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: autoresearch
  3. description: "Autonomously optimize any Claude Code skill by running it repeatedly, scoring outputs against binary evals, mutating the prompt, and keeping improvements. Based on Karpathy's autoresearch methodology. Use when: optimize this skill, improve this skill, run autoresearch on, make this skill better, self-improve skill, benchmark skill, eval my skill, run evals on. Outputs: an improved SKILL.md, a results log, and a changelog of every mutation tried."
  4. ---
  5. # Autoresearch for Skills
  6. Most skills work about 70% of the time. The other 30% you get garbage. The fix isn't to rewrite the skill from scratch. It's to let an agent run it dozens of times, score every output, and tighten the prompt until that 30% disappears.
  7. This skill adapts Andrej Karpathy's autoresearch methodology (autonomous experimentation loops) to Claude Code skills. Instead of optimizing ML training code, we optimize skill prompts.
  8. ---
  9. ## the core job
  10. Take any existing skill, define what "good output" looks like as binary yes/no checks, then run an autonomous loop that:
  11. 1. Generates outputs from the skill using test inputs
  12. 2. Scores every output against the eval criteria
  13. 3. Mutates the skill prompt to fix failures
  14. 4. Keeps mutations that improve the score, discards the rest
  15. 5. Repeats until the score ceiling is hit or the user stops it
  16. **Output:** An improved SKILL.md + `results.tsv` log + `changelog.md` of every mutation attempted + a live HTML dashboard you can watch in your browser.
  17. ---
  18. ## before starting: gather context
  19. **STOP. Do not run any experiments until all fields below are confirmed with the user. Ask for any missing fields before proceeding.**
  20. 1. **Target skill** — Which skill do you want to optimize? (need the exact path to SKILL.md)
  21. 2. **Test inputs** — What 3-5 different prompts/scenarios should we test the skill with? (variety matters — pick inputs that cover different use cases so we don't overfit to one scenario)
  22. 3. **Eval criteria** — What 3-6 binary yes/no checks define a good output? (these are your "test questions" — see [references/eval-guide.md](references/eval-guide.md) for how to write good evals)
  23. 4. **Runs per experiment** — How many times should we run the skill per mutation? Default: 5. (more runs = more reliable scores, but slower and more expensive. 5 is the sweet spot for most skills.)
  24. 5. **Run interval** — How often should experiments cycle? Default: every 2 minutes. (shorter = faster iteration, but costs more)
  25. 6. **Budget cap** — Optional. Max number of experiment cycles before stopping. Default: no cap (runs until you stop it).
  26. ---
  27. ## step 1: read the skill
  28. Before changing anything, read and understand the target skill completely.
  29. 1. Read the full SKILL.md file
  30. 2. Read any files in `references/` that the skill links to
  31. 3. Identify the skill's core job, process steps, and output format
  32. 4. Note any existing quality checks or anti-patterns already in the skill
  33. Do NOT skip this. You need to understand what the skill does before you can improve it.
  34. ---
  35. ## step 2: build the eval suite
  36. Convert the user's eval criteria into a structured test. Every check must be binary — pass or fail, no scales.
  37. **Format each eval as:**
  38. ```
  39. EVAL [number]: [Short name]
  40. Question: [Yes/no question about the output]
  41. Pass condition: [What "yes" looks like — be specific]
  42. Fail condition: [What triggers a "no"]
  43. ```
  44. **Rules for good evals:**
  45. - Binary only. Yes or no. No "rate 1-7" scales. Scales compound variability and give unreliable results.
  46. - Specific enough to be consistent. "Is the text readable?" is too vague. "Are all words spelled correctly with no truncated sentences?" is testable.
  47. - Not so narrow that the skill games the eval. "Contains fewer than 200 words" will make the skill optimize for brevity at the expense of everything else.
  48. - 3-6 evals is the sweet spot. More than that and the skill starts parroting eval criteria back instead of actually improving.
  49. See [references/eval-guide.md](references/eval-guide.md) for detailed examples of good vs bad evals.
  50. **Max score calculation:**
  51. ```
  52. max_score = [number of evals] × [runs per experiment]
  53. ```
  54. Example: 4 evals × 5 runs = max score of 20.
  55. ---
  56. ## step 3: generate the live dashboard
  57. Before running any experiments, create a live HTML dashboard at `autoresearch-[skill-name]/dashboard.html` and open it in the browser.
  58. The dashboard must:
  59. - Auto-refresh every 10 seconds (reads from results.tsv)
  60. - Show a score progression line chart (experiment number on X axis, pass rate % on Y axis)
  61. - Show a colored bar for each experiment: green = keep, red = discard, blue = baseline
  62. - Show a table of all experiments with: experiment #, score, pass rate, status, description
  63. - Show per-eval breakdown: which evals pass most/least across all runs
  64. - Show current status: "Running experiment [N]..." or "Idle"
  65. - Use clean styling with soft colors (white background, pastel accents, clean sans-serif font)
  66. Generate the dashboard as a single self-contained HTML file with inline CSS and JavaScript. Use Chart.js loaded from CDN for the line chart. The JS should fetch `results.json` (which you update after each experiment alongside results.tsv) and re-render.
  67. **Open it immediately** after creating it: `open dashboard.html` (macOS) so the user can see it in their browser.
  68. **Update `results.json`** after every experiment so the dashboard stays current. The JSON format:
  69. ```json
  70. {
  71. "skill_name": "[name]",
  72. "status": "running",
  73. "current_experiment": 3,
  74. "baseline_score": 70.0,
  75. "best_score": 90.0,
  76. "experiments": [
  77. {
  78. "id": 0,
  79. "score": 14,
  80. "max_score": 20,
  81. "pass_rate": 70.0,
  82. "status": "baseline",
  83. "description": "original skill — no changes"
  84. }
  85. ],
  86. "eval_breakdown": [
  87. {"name": "Text legibility", "pass_count": 8, "total": 10},
  88. {"name": "Pastel colors", "pass_count": 9, "total": 10}
  89. ]
  90. }
  91. ```
  92. When the run finishes (user stops it or ceiling hit), update `status` to `"complete"` so the dashboard shows a "Done" state with final summary.
  93. ---
  94. ## step 4: establish baseline
  95. Run the skill AS-IS before changing anything. This is experiment #0.
  96. 1. Create a working directory: `autoresearch-[skill-name]/` inside the skill's folder
  97. 2. Create `results.tsv` with the header row
  98. 3. Create `results.json` and `dashboard.html`, then open the dashboard in the browser
  99. 4. Back up the original SKILL.md as `SKILL.md.baseline`
  100. 5. Run the skill [N] times using the test inputs
  101. 6. Score every output against every eval
  102. 7. Record the baseline score and update both results.tsv and results.json
  103. **results.tsv format (tab-separated):**
  104. ```
  105. experiment score max_score pass_rate status description
  106. 0 14 20 70.0% baseline original skill — no changes
  107. ```
  108. **IMPORTANT:** After establishing baseline, confirm the score with the user before proceeding. If baseline is already 90%+, the skill may not need optimization — ask the user if they want to continue.
  109. ---
  110. ## step 5: run the experiment loop
  111. This is the core autoresearch loop. Once started, run autonomously until stopped.
  112. **LOOP:**
  113. 1. **Analyze failures.** Look at which evals are failing most. Read the actual outputs that failed. Identify the pattern — is it a formatting issue? A missing instruction? An ambiguous directive?
  114. 2. **Form a hypothesis.** Pick ONE thing to change. Don't change 5 things at once — you won't know what helped.
  115. Good mutations:
  116. - Add a specific instruction that addresses the most common failure
  117. - Reword an ambiguous instruction to be more explicit
  118. - Add an anti-pattern ("Do NOT do X") for a recurring mistake
  119. - Move a buried instruction higher in the skill (priority = position)
  120. - Add or improve an example that shows the correct behavior
  121. - Remove an instruction that's causing the skill to over-optimize for one thing at the expense of others
  122. Bad mutations:
  123. - Rewriting the entire skill from scratch
  124. - Adding 10 new rules at once
  125. - Making the skill longer without a specific reason
  126. - Adding vague instructions like "make it better" or "be more creative"
  127. 3. **Make the change.** Edit SKILL.md with ONE targeted mutation.
  128. 4. **Run the experiment.** Execute the skill [N] times with the same test inputs.
  129. 5. **Score it.** Run every output through every eval. Calculate total score.
  130. 6. **Decide: keep or discard.**
  131. - Score improved → **KEEP.** Log it. This is the new baseline.
  132. - Score stayed the same → **DISCARD.** Revert SKILL.md to previous version. The change added complexity without improvement.
  133. - Score got worse → **DISCARD.** Revert SKILL.md to previous version.
  134. 7. **Log the result** in results.tsv.
  135. 8. **Repeat.** Go back to step 1 of the loop.
  136. **NEVER STOP.** Once the loop starts, do not pause to ask the user if you should continue. They may be away from the computer. Run autonomously until:
  137. - The user manually stops you
  138. - You hit the budget cap (if one was set)
  139. - You hit 95%+ pass rate for 3 consecutive experiments (diminishing returns)
  140. **If you run out of ideas:** Re-read the failing outputs. Try combining two previous near-miss mutations. Try a completely different approach to the same problem. Try removing things instead of adding them. Simplification that maintains the score is a win.
  141. ---
  142. ## step 6: write the changelog
  143. After each experiment (whether kept or discarded), append to `changelog.md`:
  144. ```markdown
  145. ## Experiment [N] — [keep/discard]
  146. **Score:** [X]/[max] ([percent]%)
  147. **Change:** [One sentence describing what was changed]
  148. **Reasoning:** [Why this change was expected to help]
  149. **Result:** [What actually happened — which evals improved/declined]
  150. **Failing outputs:** [Brief description of what still fails, if anything]
  151. ```
  152. This changelog is the most valuable artifact. It's a research log that any future agent (or smarter future model) can pick up and continue from.
  153. ---
  154. ## step 7: deliver results
  155. When the user returns or the loop stops, present:
  156. 1. **Score summary:** Baseline score → Final score (percent improvement)
  157. 2. **Total experiments run:** How many mutations were tried
  158. 3. **Keep rate:** How many mutations were kept vs discarded
  159. 4. **Top 3 changes that helped most** (from the changelog)
  160. 5. **Remaining failure patterns** (what the skill still gets wrong, if anything)
  161. 6. **The improved SKILL.md** (already saved in place)
  162. 7. **Location of results.tsv and changelog.md** for reference
  163. ---
  164. ## output format
  165. The skill produces four files in `autoresearch-[skill-name]/`:
  166. ```
  167. autoresearch-[skill-name]/
  168. ├── dashboard.html # live browser dashboard (auto-refreshes)
  169. ├── results.json # data file powering the dashboard
  170. ├── results.tsv # score log for every experiment
  171. ├── changelog.md # detailed mutation log
  172. └── SKILL.md.baseline # original skill before optimization
  173. ```
  174. Plus the improved SKILL.md saved back to its original location.
  175. **results.tsv example:**
  176. ```
  177. experiment score max_score pass_rate status description
  178. 0 14 20 70.0% baseline original skill — no changes
  179. 1 16 20 80.0% keep added explicit instruction to avoid numbering in diagrams
  180. 2 16 20 80.0% discard tried enforcing left-to-right layout — no improvement
  181. 3 18 20 90.0% keep added color palette hex codes instead of vague "pastel" description
  182. 4 18 20 90.0% discard added anti-pattern for neon colors — no improvement
  183. 5 19 20 95.0% keep added worked example showing correct label formatting
  184. ```
  185. ---
  186. ## example: optimizing a diagram-generator skill
  187. **Context gathered:**
  188. - Target skill: `~/.claude/skills/diagram-generator/SKILL.md`
  189. - Test inputs: "OAuth flow diagram", "CI/CD pipeline", "microservices architecture", "user onboarding funnel", "database schema relationships"
  190. - Evals: (1) All text legible and spelled correctly? (2) Uses only pastel/soft colors? (3) Linear layout — left-to-right or top-to-bottom? (4) Free of numbers, ordinals, and ordering?
  191. - Runs per experiment: 10
  192. - Max score: 40
  193. **Baseline run (experiment 0):**
  194. Generated 10 diagrams. Scored each against 4 evals. Result: 32/40 (80%).
  195. Common failures: 3 diagrams had numbered steps, 2 had bright red elements, 3 had illegible small text.
  196. **Experiment 1 — KEEP (35/40, 87.5%):**
  197. Change: Added "NEVER include step numbers, ordinal numbers (1st, 2nd), or any numerical ordering in diagrams" to the anti-patterns section.
  198. Result: Numbering failures dropped from 3 to 1. Other evals held steady.
  199. **Experiment 2 — DISCARD (34/40, 85%):**
  200. Change: Added "All text must be minimum 14px font size."
  201. Result: Legibility improved by 1, but color compliance dropped by 2. Reverted.
  202. **Experiment 3 — KEEP (37/40, 92.5%):**
  203. Change: Replaced vague "pastel colors" instruction with specific hex codes: `#A8D8EA, #AA96DA, #FCBAD3, #FFFFD2, #B5EAD7`.
  204. Result: Color eval went from 8/10 to 10/10. Other evals held.
  205. **Experiment 4 — DISCARD (37/40, 92.5%):**
  206. Change: Added anti-pattern "Do NOT use red (#FF0000), orange (#FF8C00), or neon green (#39FF14)."
  207. Result: No change. The hex codes from experiment 3 already solved the color problem. Reverted to keep skill simpler.
  208. **Experiment 5 — KEEP (39/40, 97.5%):**
  209. Change: Added a worked example showing a correct diagram with properly formatted labels (no numbers, pastel fills, left-to-right flow, legible text).
  210. Result: Hit 39/40. One remaining failure: a complex diagram with overlapping labels. Diminishing returns — stopped.
  211. **Final delivery:**
  212. - Baseline: 32/40 (80%) → Final: 39/40 (97.5%)
  213. - 5 experiments, 3 kept, 2 discarded
  214. - Top changes: specific hex codes for colors, explicit anti-numbering rule, worked example
  215. - Remaining issue: very complex diagrams occasionally get overlapping labels (1/40 failure rate)
  216. ---
  217. ## how this connects to other skills
  218. **What feeds into autoresearch:**
  219. - Any existing skill that needs optimization
  220. - User-defined eval criteria (or help them define evals using the eval guide)
  221. **What autoresearch feeds into:**
  222. - The improved skill replaces the original
  223. - The changelog can be passed to future models for continued optimization
  224. - The eval suite can be reused whenever the skill is updated
  225. ---
  226. ## the test
  227. A good autoresearch run:
  228. 1. **Started with a baseline** — never changed anything before measuring the starting point
  229. 2. **Used binary evals only** — no scales, no vibes, no "rate this 1-10"
  230. 3. **Changed one thing at a time** — so you know exactly what helped
  231. 4. **Kept a complete log** — every experiment recorded, kept or discarded
  232. 5. **Improved the score** — measurable improvement from baseline to final
  233. 6. **Didn't overfit** — the skill got better at the actual job, not just at passing the specific test inputs
  234. 7. **Ran autonomously** — didn't stop to ask permission between experiments
  235. If the skill "passes" all evals but the actual output quality hasn't improved — the evals are bad, not the skill. Go back to step 2 and write better evals.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.