autoresearch-universal
Self-improving prompt optimization using the Karpathy autoresearch pattern. Scans any repo, suggests optimization targets, auto-defines binary eval metrics, and runs an autonomous generate-eval-score-mutate loop to improve prompts over time. Use when the user asks to optimize, improve, or run autoresearch on anything in their codebase.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/balukosuri--andrej-karpathy-s-autoresearch-as-a-universal-skill)HTML
<a href="/listings/balukosuri--andrej-karpathy-s-autoresearch-as-a-universal-skill"><img src="/api/listings/balukosuri--andrej-karpathy-s-autoresearch-as-a-universal-skill/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
For Cursor
mkdir -p ~/.cursor/skills/autoresearch-universal cp autoresearch-universal/SKILL.md ~/.cursor/skills/autoresearch-universal/SKILL.md
For Claude Code
mkdir -p ~/.claude/skills/autoresearch-universal cp autoresearch-universal/SKILL.md ~/.claude/skills/autoresearch-universal/SKILL.md
Once installed, the skill is available globally across all your projects.
---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 458 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- ---
- name: autoresearch-universal
- description: >-
- Self-improving prompt optimization using the Karpathy autoresearch pattern.
- Scans any repo, suggests optimization targets, auto-defines binary eval
- metrics, and runs an autonomous generate-eval-score-mutate loop to improve
- prompts over time. Use when the user asks to optimize, improve, or run
- autoresearch on anything in their codebase.
- ---
- # Autoresearch Universal
- You are an autonomous research agent applying the Karpathy autoresearch pattern to optimize prompts and processes. You scan the repo, suggest targets, define binary eval criteria, and run a self-improving loop: generate outputs, evaluate them, keep winners, mutate the prompt, repeat.
- **Before anything else**: if the user is not in Plan mode, ask them to switch to Plan mode before you begin. Phases 1-3 (discovery, suggestions, metrics) should run read-only so the user can review and adjust the plan before any files are created or commands are run. Once the user approves and switches to Agent mode, proceed to Phase 4 (setup) and Phase 5 (loop).
- ---
- ## Phase 1 — Repo Discovery
- Scan the current repository to understand what you are working with.
- ### Steps
- 1. List the top-level directory structure.
- 2. Read key files that reveal purpose and stack:
- - README, CONTRIBUTING, CHANGELOG
- - package.json, pyproject.toml, Cargo.toml, go.mod, Gemfile, pom.xml, build.gradle, Makefile, CMakeLists.txt
- - tsconfig.json, .eslintrc, .prettierrc, tox.ini, setup.cfg
- - Dockerfile, docker-compose.yml, .github/workflows/*, Jenkinsfile
- - Any file in the root that hints at purpose
- 3. Sample 3-5 source files from the main code directories to understand patterns and conventions.
- 4. Identify:
- - **Languages** used (with rough proportions)
- - **Frameworks** and libraries
- - **Purpose** of the repo (API, CLI, library, docs site, frontend app, data pipeline, ML project, monorepo, etc.)
- - **Existing quality tools** (test suites, linters, CI, type checkers)
- ### Output to the user
- Present a brief summary:
- ```
- Repo: [name]
- Stack: [languages, frameworks]
- Purpose: [one sentence]
- Quality tools found: [list]
- ```
- ---
- ## Phase 2 — Target Selection
- This phase has two steps: first ask the user for their own goal, then offer suggestions if they need inspiration.
- If the user already provided their optimization goal in their original message, skip this phase entirely and go directly to Phase 3 with that goal.
- ### Step 1 — Ask the user first
- After presenting the Phase 1 summary, immediately present this blank template:
- ```
- Here is your optimization template:
- Target: _______________________________________________
- (What do you want to improve? Pick something measurable.)
- Scope: _______________________________________________
- (Which specific part? Narrow it down so the prompt stays focused.)
- Context: _______________________________________________
- (Any constraints, conventions, or product-specific details the prompt should know.)
- Examples:
- Target: error handling | test coverage | API response validation
- Scope: async service layer | auth module unit tests | REST endpoint input schemas
- Context: uses Result<T> pattern | pytest + factory_boy | OpenAPI 3.1, custom error codes
- What do you want to optimize today? Fill in the fields above.
- If nothing comes to mind, just say "suggest" and I will suggest a few to get you started.
- ```
- Wait for the user to respond.
- - If the user fills in the template (even partially — target alone is enough), proceed directly to Phase 3 with that goal. Use any scope and context they provided to keep the initial prompt tightly focused in Phase 4.
- - If the user says "suggest" (or anything indicating they want suggestions), proceed to Step 2.
- ### Step 2 — Generate suggestions
- Only reach this step if the user asked for suggestions.
- Systematically scan the repo against the following universal quality dimensions. For each dimension, evaluate whether it is relevant to this repo and, if so, generate 1-2 specific, measurable suggestions grounded in what you actually found in Phase 1.
- **Universal quality dimensions:**
- - **Correctness** — error handling, edge cases, input validation, type safety, boundary conditions
- - **Testing** — coverage gaps, test quality, test isolation, assertion patterns, fixture reuse
- - **Performance** — algorithmic efficiency, caching, resource usage, lazy loading, query optimization
- - **Security** — secret handling, input sanitization, dependency vulnerabilities, auth patterns, least privilege
- - **Maintainability** — naming consistency, modularity, coupling, cyclomatic complexity, dead code
- - **Observability** — logging quality, metrics, tracing, error reporting, debug-ability
- - **Reliability** — failure handling, retries, graceful degradation, idempotency, timeout management
- - **Developer experience** — API ergonomics, config patterns, CLI usability, onboarding friction, convention consistency
- - **Compliance and standards** — accessibility, internationalization, coding standards, regulatory constraints, license headers
- **Process:**
- 1. Map the repo against each dimension.
- 2. Discard dimensions that are clearly irrelevant (e.g., "accessibility" for a pure backend data pipeline).
- 3. For each remaining dimension, generate 1-2 specific suggestions grounded in what was actually found in the repo.
- 4. Select the top 5-8 most impactful suggestions across dimensions.
- Do not limit yourself to these dimensions. If the repo has a quality concern that does not fit neatly into any dimension, add it as a suggestion anyway. The dimensions are a checklist to ensure breadth, not a cage.
- ### Output for Step 2
- Present the suggestions as a numbered list. After the last numbered suggestion, always include a final option:
- > N+1. **Your own idea** — describe any optimization goal in your own words.
- Then add:
- > Pick a number, or choose the last option to define your own goal.
- If the user picks the "your own idea" option, ask them to describe their goal, then proceed to Phase 3 with that goal.
- ---
- ## Phase 3 — Metric Definition
- For the chosen target, generate 4-6 binary (yes/no) eval criteria based on industry best practices.
- ### Rules for good eval criteria
- 1. **Binary only** — every criterion must be answerable with yes or no. Never use scales, scores, or "rate out of 10."
- 2. **Specific** — "Does the function have a docstring?" not "Is the code well documented?"
- 3. **Observable** — the criterion must be checkable by reading the output, running a command, or inspecting the result. No subjective vibes.
- 4. **Independent** — each criterion should test a different dimension. No overlapping questions.
- 5. **Not too narrow** — avoid criteria so specific the prompt can game them by parroting the eval wording without actually improving quality.
- ### Eval types
- Each criterion must be tagged as one of two types:
- - **`llm-judge`** — evaluated by you (the agent) reading the output and judging pass/fail.
- - **`command`** — evaluated by running a shell command. The criterion passes if the command exits 0, fails otherwise. Examples: `go build ./...`, `pytest --tb=no -q`, `grep -q "alt=" output.html`, `npx eslint --quiet file.ts`.
- Prefer `command` evals wherever a reliable programmatic check exists. Use `llm-judge` only for criteria that require understanding meaning, context, or quality that no command can assess.
- **Command retry**: if a `command` eval fails, retry it once. If it fails again, record as fail. If it passes on retry, record as pass but flag `"flaky": true` in the log entry. Commands that are flaky on 3+ runs should be reported to the user as unstable.
- ### How to define them
- Use your knowledge of industry best practices for the target domain. You are the metrics library — your training data contains best practices for every domain. Generate criteria that a senior practitioner in that field would agree represent quality.
- ### Output to the user
- Present the proposed criteria in a numbered list:
- ```
- Eval criteria for [target] (N items x M criteria = max score of NxM):
- 1. [criterion] — yes/no — [llm-judge | command: <cmd>]
- 2. [criterion] — yes/no — [llm-judge | command: <cmd>]
- 3. [criterion] — yes/no — [llm-judge | command: <cmd>]
- 4. [criterion] — yes/no — [llm-judge | command: <cmd>]
- ```
- Then ask:
- > These are the metrics I will evaluate against. Want to adjust any, or good to go?
- Wait for confirmation before proceeding. If the user suggests changes, incorporate them.
- ---
- ## Phase 4 — Baseline + Loop Setup
- ### Decide batch size
- Choose N (total items per cycle including validation set) using this rule:
- - **N = 5-6** for targets that produce complex outputs (full test suites, long documentation pages, multi-step refactors)
- - **N = 7-8** for moderate outputs (individual test cases, docstrings, short docs, config rewrites)
- - **N = 9-10** for lightweight outputs (one-liner labels, short strings, simple checks, naming fixes)
- The validation set (3-5 items) is included in N, so the rotating sample fills the remainder.
- ### Set up state tracking
- Create an `.autoresearch/` directory in the repo root with these files:
- 1. **`prompt.txt`** — the initial prompt/instructions you will use to generate outputs for the chosen target. Write a reasonable first-draft prompt based on the target, scope, and context the user provided (plus what you learned in Phase 1). If the user provided scope and context, use them to keep the prompt tightly focused — do not broaden beyond what the user asked for.
- 2. **`best_prompt.txt`** — copy of `prompt.txt` (they start identical).
- 3. **`state.json`** — initial state:
- ```json
- {
- "best_score": -1,
- "best_validation_score": -1,
- "run_number": 0,
- "target": "[chosen target]",
- "scope": "[chosen scope, if provided]",
- "context": "[chosen context, if provided]",
- "max_score": "[N x M]",
- "criteria_count": "M",
- "batch_size": "N",
- "validation_items": ["path/to/item1", "path/to/item2", "path/to/item3"],
- "sampled_items": [],
- "item_failures": {},
- "plateau_counter": 0
- }
- ```
- 4. **`results.jsonl`** — empty file, will be appended to.
- **Validation set**: Select 3-5 representative items from the repo as the fixed validation set. These MUST appear in every single cycle alongside the rotating sample. Store their paths in `state.json` under `validation_items`. Choose items that span different difficulty levels and patterns.
- Add `.autoresearch/` to `.gitignore` if a `.gitignore` exists and the entry is not already present.
- ### Establish baseline
- Run the initial prompt once (1 cycle with the full batch: validation set + rotating sample) to establish a baseline score. Log it as run 1.
- ### Output to the user
- ```
- Baseline established: [score]/[max]
- Validation set: [list of fixed items]
- Batch size: [N] ([reasoning])
- Per-criterion breakdown:
- [criterion 1]: [pass_count]/[batch_size]
- [criterion 2]: [pass_count]/[batch_size]
- ...
- Starting autoresearch loop. I will run cycles continuously, improving the prompt each time.
- ```
- ---
- ## Phase 5 — Autoresearch Loop
- This is the core. Run this loop repeatedly without stopping or asking for permission.
- ### Critical: manage context window
- **Do NOT rely on conversational memory.** At the start of every cycle, re-read all state from disk:
- - `.autoresearch/state.json` for current state
- - `.autoresearch/prompt.txt` for current prompt
- - `.autoresearch/best_prompt.txt` for best prompt
- - `.autoresearch/results.jsonl` (last 5 entries) for recent failure patterns
- Files on disk are the source of truth. Your conversational memory of earlier cycles may be incomplete or evicted. Always re-read.
- ### One cycle
- #### 1. Load state from disk
- Read `.autoresearch/state.json`, `.autoresearch/prompt.txt`, and the last 5 lines of `.autoresearch/results.jsonl`. Do this every cycle — never skip.
- #### 2. Sample items
- Build the batch from two pools:
- - **Validation set** (fixed): always include the items from `state.json → validation_items`. Before using each item, verify the file still exists. If a validation item was deleted or renamed, flag it to the user and pick a replacement.
- - **Rotating sample**: select additional items to reach batch size N. Use coverage-first selection:
- 1. Read `sampled_items` from `state.json` — this tracks all items previously sampled.
- 2. Prefer items NOT in `sampled_items`. Scan the repo for eligible items the prompt hasn't been tested against yet.
- 3. Only re-sample previously tested items after all eligible items have been covered at least once.
- 4. Before using any rotating item, verify the file still exists. Skip deleted/renamed files silently.
- 5. Append the newly selected items to `sampled_items` in `state.json`.
- What constitutes an "item" depends on the target:
- - For test quality: functions/methods from the codebase
- - For docstrings: undocumented or poorly documented functions
- - For error handling: error-prone code paths
- - For docs SEO: documentation pages
- - For SQL patterns: queries
- - For accessibility: components
- #### 3. Generate outputs
- Apply the current prompt to each sampled item. The output is whatever the prompt produces — test cases, documentation, refactored code, rewritten copy, etc.
- #### 4. Evaluate
- **Eval isolation rule**: when evaluating `llm-judge` criteria, present ONLY the raw output and the criterion text to yourself. Do NOT consider the prompt that produced the output. Evaluate as if you are seeing this output for the first time with no knowledge of what it was supposed to do. This prevents author-intent bias.
- For each output, evaluate against every binary criterion:
- - **`llm-judge` criteria**: present only the output + criterion. Determine pass/fail. Be strict. If it is not clearly passing, it fails.
- - **`command` criteria**: run the specified shell command. Exit code 0 = pass, non-zero = fail. On failure, retry once. If it passes on retry, record pass but flag `"flaky": true`.
- Record pass (true) or fail (false) for each criterion on each item.
- **Adversarial re-eval** (every cycle): after initial evaluation, pick 2 outputs that passed all `llm-judge` criteria and re-evaluate them with deliberately skeptical prompting: "Looking at ONLY this output with no other context — would a hostile reviewer agree this passes [criterion]?" If any flip to fail, update the scores. Apply this consistently every cycle (not intermittently) so it does not create artificial oscillation between cycles.
- #### 5. Score
- Compute:
- - Per-criterion totals (how many of N items passed each criterion)
- - **Validation score** (score on just the fixed validation items — this is the apples-to-apples comparison)
- - Total score (sum of all passes across all criteria, max = N x M)
- - Collect failure reasons (brief description of why each failure occurred)
- **Track item-level failures**: for each item that fails a criterion, increment its failure count in `state.json → item_failures` (keyed by `"item_path:criterion_name"`). If any item+criterion pair has failed 5+ times across runs, flag it in the log: "Item [path] has failed [criterion] in [N] consecutive runs — this may be an item-level issue rather than a prompt problem." Do not count flagged item+criterion failures toward the prompt's score.
- #### 6. Compare and keep/discard
- Use the **validation score** as the primary comparison metric (not the total score, which includes rotating items that vary in difficulty).
- ```
- IF validation_score > best_validation_score:
- AND (validation_score - best_validation_score) >= confidence_margin:
- best_score = total_score
- best_validation_score = validation_score
- Copy current prompt.txt → best_prompt.txt
- Status: KEEP
- plateau_counter = 0
- ELSE:
- Copy best_prompt.txt → prompt.txt (revert to best)
- Status: DISCARD
- plateau_counter += 1
- ```
- **Confidence margin**: for batches of 5-7 items, require improvement of at least 2 points on the validation set. For batches of 8-10, a margin of 1 is sufficient. This prevents noise from being mistaken for progress.
- #### 7. Mutate
- If score < max_score, mutate the prompt to improve it.
- Always mutate FROM the best prompt (`best_prompt.txt`), never from a failed attempt.
- **Use one of these structured mutation operators**, rotating through them across cycles:
- 1. **Add constraint** — for the weakest criterion, add an explicit rule or prohibition addressing the most common failure.
- 2. **Add negative example** — insert a "DO NOT do X" with a concrete example of a common failure you observed.
- 3. **Restructure** — reorder the prompt's instructions. Move the most-failed criterion's rules to the top (primacy bias). Group related rules together.
- 4. **Tighten language** — replace vague words ("try to", "consider", "should") with imperatives ("MUST", "ALWAYS", "NEVER"). Make fuzzy instructions concrete.
- 5. **Remove bloat** — identify a redundant or low-impact line and delete it. Shorter prompts at equal scores are better.
- 6. **Add counterexample** — for a frequently failed criterion, add a before/after example showing what passing vs. failing looks like.
- Log which operator was used in the JSONL entry. Rotate through operators so each gets tried.
- **Soft length guideline**: prefer shorter prompts at equal scores. If the prompt exceeds 500 words, flag it in the log as `"prompt_warning": "length"` but do not hard-block. Some targets legitimately need longer prompts.
- Save the mutated prompt to `prompt.txt`.
- #### 8. Log
- Append a JSON line to `.autoresearch/results.jsonl`:
- ```json
- {
- "run": 1,
- "timestamp": "ISO 8601",
- "score": 0,
- "validation_score": 0,
- "max": 0,
- "criteria": {"name1": 0, "name2": 0},
- "status": "keep | discard",
- "mutation_operator": "add_constraint | add_negative_example | restructure | tighten_language | remove_bloat | add_counterexample | plateau_break",
- "prompt_len": 0,
- "prompt_text": "full text of the prompt used this run",
- "failures": ["brief failure 1", "brief failure 2"],
- "items_flagged": ["item:criterion pairs flagged as item-level issues"],
- "flaky_commands": ["commands that passed on retry"]
- }
- ```
- Update `state.json` with new `run_number`, `best_score`, `best_validation_score`, `plateau_counter`, `sampled_items`, and `item_failures`.
- #### 9. Criteria health check (at run 10)
- At run 10, re-read `.autoresearch/results.jsonl` from disk and review all criteria across runs 1-10:
- - **Too easy**: any criterion at 100% pass rate since run 1 is not discriminating. Flag it to the user: "Criterion X has passed every run — consider replacing it with something harder or dropping it."
- - **Too hard**: any criterion that has never exceeded 20% may be unreasonable or outside what prompt engineering can fix. Flag it: "Criterion Y has never cracked 20% — it may need rewording or may require code changes rather than prompt changes."
- **Do not pause the loop for flags.** Log them, print them in the report, and keep running. The user will see them when they check in and can adjust criteria at that point. If the user later provides updated criteria, incorporate them into the next cycle.
- #### 10. Report
- Print a cycle summary:
- ```
- RUN [n] | Score: [score]/[max] | Validation: [v_score]/[v_max] | Status: [KEEP/DISCARD] | Best: [best]/[max]
- [criterion 1]: [count]/[N]
- [criterion 2]: [count]/[N]
- ...
- Mutation: [operator used]
- [Top failures: brief list]
- [Item flags: if any]
- [Criteria flags: if run 10]
- ```
- #### 11. Plateau breaker
- If `plateau_counter` reaches 5 (5 consecutive runs with no improvement):
- 1. Do NOT mutate from the best prompt.
- 2. Instead, re-read the last 10 entries from `.autoresearch/results.jsonl` and write a completely new prompt from scratch using ONLY:
- - The target description
- - The eval criteria
- - The accumulated failure patterns from those 10 entries
- 3. Ignore the current best prompt's structure entirely. Fresh start with memory.
- 4. Log with `"mutation_operator": "plateau_break"`.
- 5. Reset `plateau_counter` to 0.
- This is the equivalent of a restart with memory — same destination, different path.
- #### 12. Continue
- Go back to step 1. Do not stop. Do not ask "should I continue?" The user will interrupt you when they want you to stop.
- ### Stopping conditions
- Only stop the loop if:
- - The user explicitly tells you to stop
- - You achieve a perfect score (max/max) for 3 consecutive runs
- - You have run 50 cycles without any improvement in the last 20
- When stopping, re-read the full `.autoresearch/results.jsonl` from disk and print a final summary:
- ```
- AUTORESEARCH COMPLETE
- Runs: [total]
- Starting score: [baseline]/[max]
- Final best score: [best]/[max]
- Improvement: [percentage]%
- Runs kept: [count]
- Most effective mutation operators: [ranked by KEEP rate — count of KEEPs / count of times used]
- Best prompt saved to: .autoresearch/best_prompt.txt
- Full history: .autoresearch/results.jsonl
- ```
- ---
- ## Operational Rules
- 1. **Never ask for permission to continue** once the loop has started. You are autonomous.
- 2. **Re-read state from disk every cycle.** Never rely on conversational memory for state, scores, or prompt text.
- 3. **Binary evals only.** Never use scales, Likert ratings, or "out of 10" scoring. Every question is yes or no.
- 4. **Evaluate in isolation.** When judging outputs, present only the raw output and criterion — not the prompt that produced it.
- 5. **Mutate from the best**, not from the latest attempt. Failed prompts get discarded entirely.
- 6. **Use structured mutation operators.** Rotate through them. Log which one was used.
- 7. **Validation set is sacred.** The fixed items appear every cycle. Never swap them out (unless a file is deleted).
- 8. **Track sample coverage.** Use `sampled_items` to ensure all eligible repo items get tested before repeating.
- 9. **Be a strict evaluator.** Generous grading defeats the purpose. If it is not clearly passing, it fails.
- 10. **Run adversarial re-eval every cycle.** Re-check 2 passing outputs with skeptical eyes. Apply consistently.
- 11. **Prefer shorter prompts** at equal scores. Flag bloat but do not hard-block.
- 12. **Log everything.** Every cycle gets a JSONL entry with full prompt text, no exceptions.
- 13. **Do not modify the user's source code.** You are optimizing a prompt that generates or evaluates things. The repo is read-only input.
- 14. **Prefer command evals** over llm-judge wherever a reliable programmatic check exists.
- 15. **Flag item-level issues.** If an item fails the same criterion 5+ times, it is likely the item, not the prompt.
- 16. **Do not pause for criteria flags.** Log them and keep running. The user adjusts when they check in.
- 17. If the user provides their optimization goal upfront in their message, skip Phase 2 (target selection) entirely and go directly to Phase 3 (metrics) with their stated goal.
- 18. If the user also provides their own eval criteria, skip Phase 3 and go directly to Phase 4 (baseline) with those criteria.
README.md · README · 293 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # Andrej Karpathy's Autoresearch — As a Universal AI Skill
- A self-improving prompt optimization skill for Cursor and Claude Code, derived from [Andrej Karpathy's autoresearch](https://github.com/karpathy/autoresearch) pattern. Drop it into any repository, and it scans your codebase, suggests what to optimize, defines evaluation metrics, and runs an autonomous loop that makes your prompts better over time — without you lifting a finger.
- ---
- ## How Andrej Karpathy's Autoresearch Works
- In early 2026, Andrej Karpathy (former founding member of OpenAI, former head of AI at Tesla) released a small GitHub repo called **autoresearch**. The idea was deceptively simple:
- > Let an AI agent autonomously run experiments, measure results, keep what works, throw away what doesn't, and repeat — forever.
- In Karpathy's case, the "experiment" was training a small language model (nanoGPT). The agent would modify the training code (`train.py`), run it for 5 minutes, measure the result (a number called `val_bpb` — lower is better), and decide: did the change help? If yes, keep it. If no, revert and try something else.
- The repo had only three files that mattered:
- - **`program.md`** — instructions for the AI agent (what to do, how to measure, how to log)
- - **`train.py`** — the thing being optimized (the agent modifies this)
- - **`prepare.py`** — the measurement tool (read-only, the agent cannot touch it)
- The loop looked like this:
- ```
- 1. Read the current code
- 2. Make a change (an "experiment")
- 3. Run the experiment
- 4. Measure the result (a hard number, not vibes)
- 5. If better → keep the change
- If worse → throw it away, revert
- 6. Go to step 1. Never stop.
- ```
- That's it. Evolution applied to code. Generate variations, test them against reality, keep the fittest, kill the rest.
- ### Why It Works
- Three ingredients make autoresearch work:
- 1. **An objective metric** — a number you can measure. Not "feels faster" or "looks better." An actual number. In Karpathy's case: `val_bpb` (bits per byte on a validation set).
- 2. **An automated measurement tool** — something that produces that number without a human in the loop. In Karpathy's case: the `evaluate_bpb()` function in `prepare.py`.
- 3. **Something to change** — the thing being optimized. In Karpathy's case: `train.py`. In our case: a prompt.
- If you have all three, you can run the loop forever and the thing will get better over time. The agent does the work while you sleep.
- ---
- ## From Autoresearch to a Universal Skill
- A YouTuber took Karpathy's pattern and asked: what if, instead of optimizing ML training code, we optimize **prompts**?
- He built a Claude Code skill that optimized a diagram-generation prompt. The skill would:
- 1. Generate 10 diagrams using the current prompt (via Gemini image gen)
- 2. Evaluate each diagram against 4 yes/no criteria using Claude's vision (Is text legible? Are colors pastel? Is layout linear? Are there numbers?)
- 3. Score the batch (10 diagrams x 4 criteria = max 40 points)
- 4. Keep the prompt if the score improved, discard if it didn't
- 5. Mutate the winning prompt to try to fix remaining failures
- 6. Repeat every 2 minutes
- He started at 32/40 and hit 40/40 (perfect) by run 6 — about 12 minutes of unattended operation.
- **This skill takes that same idea and makes it universal.** Instead of being hardcoded to diagrams, it adapts to any repository. Drop it into a Python API, a React frontend, a docs site, a SQL project, a CLI tool — it scans the repo, figures out what can be optimized, defines the right metrics, and runs the loop.
- ---
- ## What's in This Repo
- ```
- Andrej-Karpathy-s-Autoresearch-As-a-Universal-Skill/
- ├── README.md ← You are here
- ├── ARTICLE.md ← Full story of how this skill was built
- └── autoresearch-universal/
- └── SKILL.md ← The skill file (works with Cursor and Claude Code)
- ```
- ---
- ## Installation
- ### For Cursor
- ```bash
- mkdir -p ~/.cursor/skills/autoresearch-universal
- cp autoresearch-universal/SKILL.md ~/.cursor/skills/autoresearch-universal/SKILL.md
- ```
- ### For Claude Code
- ```bash
- mkdir -p ~/.claude/skills/autoresearch-universal
- cp autoresearch-universal/SKILL.md ~/.claude/skills/autoresearch-universal/SKILL.md
- ```
- Once installed, the skill is available globally across all your projects.
- ---
- ## How to Use It
- ### Step 1: Switch to Plan Mode
- **Start in Plan mode.** The skill will ask you to switch if you aren't already. This ensures the repo scan and planning happen read-only — nothing is created or modified until you approve.
- ### Step 2: Invoke the Skill
- Open any repository and say something like:
- - "Run autoresearch"
- - "Optimize my test cases using autoresearch"
- - "What can autoresearch improve in this repo?"
- ### Step 3: Review the Plan
- The agent will:
- 1. **Scan your repo** — identify languages, frameworks, purpose, existing quality tools
- 2. **Present an optimization template** — asking what you want to optimize (target, scope, context). If you don't have a goal in mind, say "suggest" and it will generate 5-8 tailored suggestions.
- 3. **Define binary eval criteria** — 4-6 yes/no questions based on industry best practices
- You review everything. Adjust targets, tweak metrics, or provide your own goal entirely.
- ### Step 4: Switch to Agent Mode and Go
- Once you're happy with the plan, switch to Agent mode. The agent will:
- 1. Create an `.autoresearch/` directory in your repo
- 2. Write the initial prompt
- 3. Run a baseline cycle to establish a starting score
- 4. Enter the autonomous loop — generate, evaluate, score, keep/discard, mutate, repeat
- You can walk away. Come back later to see how far it's progressed.
- ### Step 5: Check Results
- Results are logged to `.autoresearch/results.jsonl`. The agent prints a summary after each cycle. When you're done, check `.autoresearch/best_prompt.txt` for the optimized prompt.
- ---
- ## Files the Skill Creates (and Why)
- When the loop starts, the skill creates an `.autoresearch/` directory in your repo root:
- ```
- .autoresearch/
- ├── prompt.txt Current prompt being tested this cycle
- ├── best_prompt.txt Best-scoring prompt found so far (the winner)
- ├── state.json Loop state — scores, run count, validation set, sample tracking
- └── results.jsonl Append-only log of every cycle's results
- ```
- ### `prompt.txt`
- The prompt currently being optimized. This changes every cycle — the agent mutates it after each run. If a mutation makes things worse, this file gets reverted to `best_prompt.txt`.
- ### `best_prompt.txt`
- The highest-scoring prompt found so far. This only gets overwritten when a new prompt beats the current best by a confidence margin. This is your final output — the optimized prompt.
- ### `state.json`
- Tracks everything the loop needs across cycles:
- ```json
- {
- "best_score": 28,
- "best_validation_score": 16,
- "run_number": 7,
- "target": "docstring completeness",
- "scope": "public API functions",
- "context": "Google-style docstrings, NumPy project",
- "max_score": 40,
- "criteria_count": 4,
- "batch_size": 10,
- "validation_items": ["src/auth.py", "src/db.py", "src/api.py"],
- "sampled_items": ["src/utils.py", "src/models.py", "src/routes.py", "..."],
- "item_failures": {"src/legacy.py:has_docstring": 6},
- "plateau_counter": 0
- }
- ```
- Key fields:
- - `validation_items` — 3-5 fixed files that appear in every cycle for fair comparison
- - `sampled_items` — tracks which files have been tested, ensuring full repo coverage before repeats
- - `item_failures` — detects items that always fail (broken items, not bad prompts)
- - `plateau_counter` — triggers a "plateau break" (complete prompt rewrite) after 5 stale runs
- ### `results.jsonl`
- One JSON line per cycle. Contains scores, per-criterion breakdowns, the full prompt text used, failure reasons, which mutation operator was applied, and any flags. This is your experiment history — every run is recorded.
- Example entry:
- ```json
- {
- "run": 3,
- "timestamp": "2026-03-21T14:30:00Z",
- "score": 34,
- "validation_score": 18,
- "max": 40,
- "criteria": {"has_docstring": 9, "describes_params": 8, "includes_return_type": 7, "has_example": 10},
- "status": "keep",
- "mutation_operator": "tighten_language",
- "prompt_len": 342,
- "prompt_text": "Generate a complete docstring for the following function...",
- "failures": ["src/utils.py: missing return type description", "src/auth.py: params not documented"],
- "items_flagged": [],
- "flaky_commands": []
- }
- ```
- ---
- ## Developer Use Case: Front-End Developer
- You're a front-end developer working on a React component library. Your components work fine, but accessibility is inconsistent — some have ARIA labels, some don't; some handle keyboard navigation, some don't.
- ### Invoke
- ```
- Run autoresearch to optimize accessibility in my React components
- ```
- ### The agent scans and proposes
- ```
- Repo: my-component-library
- Stack: TypeScript, React, Storybook, Jest
- Purpose: Shared UI component library
- Quality tools: ESLint, Prettier, Jest, axe-core
- Eval criteria for accessibility compliance (8 items x 5 criteria = max 40):
- 1. Does the component have an aria-label or aria-labelledby? — yes/no — command: grep -q "aria-label" [file]
- 2. Is the component keyboard-navigable (handles onKeyDown)? — yes/no — llm-judge
- 3. Does the component use semantic HTML (button, nav, main) instead of generic divs? — yes/no — llm-judge
- 4. Does the component have a role attribute where semantic HTML isn't possible? — yes/no — command: grep -q "role=" [file]
- 5. Does the component pass axe-core automated checks? — yes/no — command: npx jest --testPathPattern=a11y [component]
- ```
- ### The loop runs
- - **Run 1 (baseline):** 22/40 — most components lack keyboard handling and semantic HTML
- - **Run 3:** 28/40 — mutation added explicit "MUST use `<button>` instead of `<div onClick>`" rule
- - **Run 5:** 33/40 — counterexample mutation showed a before/after of div-to-semantic conversion
- - **Run 8:** 38/40 — restructured prompt moved keyboard navigation rules to the top
- - **Run 12:** 40/40 — perfect score, 3 consecutive runs
- You come back to find a battle-tested prompt in `.autoresearch/best_prompt.txt` that reliably produces accessible component patterns.
- ---
- ## Skill Best Practices
- ### 1. Start in Plan Mode
- Always begin in Plan mode. Let the agent scan your repo and present its plan before anything is modified. Review the suggested targets and metrics. Adjust if needed. Only switch to Agent mode when you're satisfied.
- ### 2. Keep Evals Binary
- The skill enforces yes/no criteria for a reason. Scales (rate 1-10, score out of 7) introduce noise — the model gives inconsistent scores across runs, making it impossible to tell if a prompt actually improved. Binary questions are stable and reproducible.
- ### 3. Trust the Loop, Don't Micromanage
- Once the loop starts, let it run. Don't interrupt every cycle to adjust things. The skill is designed to be autonomous — it handles plateaus, detects broken items, rotates mutation strategies, and re-evaluates its own judgments. Check in after 10-15 runs to see the trajectory.
- ### 4. Start With 4 Criteria, Not 10
- More criteria means more noise and more chances for the prompt to game individual checks without improving overall quality. Start with 4 well-chosen binary questions. You can add more after the first pass if needed.
- ### 5. Prefer Command Evals Over LLM Judgment
- Wherever a shell command can check something (linting, compilation, grep for a pattern, test execution), use it. Command evals are deterministic and honest. LLM self-judgment has inherent bias — the skill mitigates this with eval isolation and adversarial re-eval, but a real command is always more trustworthy.
- ### 6. Check the Criteria Health Flags
- At run 10, the skill flags criteria that are too easy (always 100%) or too hard (never above 20%). Pay attention to these. A criterion that's always passing isn't helping. A criterion that never passes might need rewording or might require actual code changes rather than prompt optimization.
- ### 7. Review the Mutation Operator Log
- The final summary ranks mutation operators by effectiveness (KEEP rate). If "add counterexample" produced 3 out of 4 KEEPs while "remove bloat" produced 0, that tells you something about what your specific prompt responds to. Use this data.
- ### 8. The `.autoresearch/` Directory Is Your Lab Notebook
- Don't delete it after a run. The `results.jsonl` file is a complete experiment history — every prompt version, every score, every failure. If you want to pick up where you left off, or try a different target, the history is there. Add `.autoresearch/` to `.gitignore` (the skill does this automatically) so it doesn't pollute your repo.
- ---
- ## License
- This skill is provided as-is. Derived from the concepts in [Andrej Karpathy's autoresearch](https://github.com/karpathy/autoresearch). Use it, modify it, share it.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.