Skip to content
Back to search

autoresearch-claude-code

Autonomous experiment loop skill for Claude Code — port of pi-autoresearch

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/drivelineresearch--autoresearch-claude-code/badge.svg)](/listings/drivelineresearch--autoresearch-claude-code)

HTML

<a href="/listings/drivelineresearch--autoresearch-claude-code"><img src="/api/listings/drivelineresearch--autoresearch-claude-code/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires **Python 3.10+, Bash, Git**, and your chosen agent client. Linux and macOS are supported; use WSL on Windows. The optional ML example has its own dependencies.

gh repo clone drivelineresearch/autoresearch-claude-code
cd autoresearch-claude-code
./install.sh --codex      # Codex skill in ~/.agents/skills/autoresearch
./install.sh --claude     # Claude skill, command, and four hooks
# Or: ./install.sh --all

No argument defaults to Claude for compatibility. Installations use symlinks, so keep this checkout in place. Existing foreign files/directories are preserved and conflicts are reported. Claude settings are validated before changes, updated atomically, and backed up when modified. Codex installation does not edit its configuration or register Claude hooks.

Claude also supports a session-local plugin load:

claude --plugin-dir /absolute/path/to/autoresearch-claude-code

Choose plugin loading or manual Claude installation; installing both can duplicate hooks. For removal, use `./uninstall.sh --codex`, `--claude`, or `--all`. Removal only deletes symlinks owned by this checkout and corresponding exact hook entries; manually copied files and other installations remain untouched. See [installation/review notes](docs/review.md) for migration limits.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 158 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # autoresearch-claude-code
  2. ![autoresearch](imgs/autoresearch-banner.png)
  3. [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
  4. A measured experiment loop for **Claude Code and Codex**. Define a goal, a fixed
  5. benchmark, and files to modify. The agent tries changes, measures results, commits
  6. supported improvements, and records failed ideas within a budget.
  7. Inspired by [pi-autoresearch](https://github.com/davebcn87/pi-autoresearch).
  8. The experiment protocol is a shared skill with standard-library Python helpers;
  9. no MCP server is required.
  10. ## Install
  11. Requires **Python 3.10+, Bash, Git**, and your chosen agent client. Linux and macOS
  12. are supported; use WSL on Windows. The optional ML example has its own dependencies.
  13. ```bash
  14. gh repo clone drivelineresearch/autoresearch-claude-code
  15. cd autoresearch-claude-code
  16. ./install.sh --codex # Codex skill in ~/.agents/skills/autoresearch
  17. ./install.sh --claude # Claude skill, command, and four hooks
  18. # Or: ./install.sh --all
  19. ```
  20. No argument defaults to Claude for compatibility. Installations use symlinks, so
  21. keep this checkout in place. Existing foreign files/directories are preserved and
  22. conflicts are reported. Claude settings are validated before changes, updated
  23. atomically, and backed up when modified. Codex installation does not edit its
  24. configuration or register Claude hooks.
  25. Claude also supports a session-local plugin load:
  26. ```bash
  27. claude --plugin-dir /absolute/path/to/autoresearch-claude-code
  28. ```
  29. Choose plugin loading or manual Claude installation; installing both can duplicate
  30. hooks. For removal, use `./uninstall.sh --codex`, `--claude`, or `--all`. Removal
  31. only deletes symlinks owned by this checkout and corresponding exact hook entries;
  32. manually copied files and other installations remain untouched. See
  33. [installation/review notes](docs/review.md) for migration limits.
  34. ## Quick start
  35. In **Codex CLI/IDE**, invoke the installed skill:
  36. ```text
  37. $autoresearch optimize test suite runtime with at most 20 runs
  38. $autoresearch status
  39. $autoresearch report
  40. $autoresearch off
  41. ```
  42. In the Codex app, select the skill or ask to use `autoresearch` by name. Codex
  43. [discovers symlinked skills in ~/.agents/skills](https://learn.chatgpt.com/docs/build-skills).
  44. For **Claude Code**, manual installation uses `/autoresearch`; plugin loading uses
  45. the namespaced `/autoresearch:autoresearch`, with the same goal or subcommands.
  46. The agent establishes scope, creates an experiment branch, prepares a fixed
  47. benchmark, calibrates noise, and logs a baseline. It preserves existing work;
  48. use a dedicated worktree when another task is in progress. Artifacts must be
  49. ignored in the target project too; this plugin's ignores do not transfer.
  50. For unattended Codex execution, first initialize and pause the session with the
  51. skill, review the baseline, then explicitly resume it and run:
  52. ```bash
  53. # Run from the initialized experiment worktree after reviewing/removing its pause sentinel.
  54. python3 ~/.agents/skills/autoresearch/scripts/codex_loop.py --dry-run
  55. python3 ~/.agents/skills/autoresearch/scripts/codex_loop.py \
  56. --max-turns 10 --max-seconds 3600 --turn-timeout 300 \
  57. --protect path/to/scorer.py
  58. ```
  59. Replace `path/to/scorer.py` with your real scorer file; repeat `--protect` for other
  60. locked inputs. See the [Codex guide](skills/autoresearch/references/codex.md) for
  61. initialization, flags, logs, pause/cancellation, and recovery.
  62. ## What is enforced
  63. | Capability | Mechanism and limit |
  64. |---|---|
  65. | Shared state | Config/result JSONL validated for finite numbers, sequential runs, segments, status, and metadata; writes locked and atomic |
  66. | Claude continuation | Stop hook blocks turn completion while an active valid session has budget; malformed state allows stopping with a diagnostic |
  67. | Codex continuation | Bounded `codex exec` supervisor requires one result per invocation; stops on no progress, failure, altered config, branch, or protected files |
  68. | Budgets | Current-segment run/time/target checks; Codex also caps invocations and actively times out process groups |
  69. | Pause | `.autoresearch-off`; Codex checks during active turns as well as between them |
  70. | Recovery | JSONL, worklog, dashboard; Claude PreCompact snapshots and SessionStart context |
  71. | Noise/correctness | Agent follows measured-noise and checks protocol; helpers do not independently attest a score or statistical significance |
  72. | Locked scorer | Off-limits scope instructions, plus Codex hashes for specified files between turns; this is not a hostile-agent sandbox |
  73. Run caps count logged experiments (including baseline and failures), not tokens or
  74. dollars. Setup/calibration and seed confirmations have additional cost. Claude's
  75. Stop hook checks at turn boundaries; a benchmark needs its own timeout. A budget
  76. extension or scorer change must be deliberate; removing a pause sentinel alone
  77. never resets the budget.
  78. Keep/discard uses **strict improvement beyond the noise floor** and passing
  79. correctness checks. Fix dataset splits, training-only preprocessing, and scoring
  80. before optimizing; changing the evaluation definition starts a new segment.
  81. The exact workflow is in [SKILL.md](skills/autoresearch/SKILL.md), with
  82. [state commands/schema](skills/autoresearch/references/state.md). The
  83. [full review and roadmap](docs/review.md) records findings, changes, validation,
  84. and remaining limitations.
  85. ## Example: fastball velocity prediction
  86. The [OpenBiomechanics example](examples/obp-autoresearch.md) demonstrates the model
  87. interface, athlete-grouped evaluation, and `METRIC name=number` output.
  88. ```bash
  89. cd examples
  90. uv sync
  91. mkdir -p third_party
  92. gh repo clone drivelineresearch/openbiomechanics third_party/openbiomechanics -- --depth 1
  93. ./autoresearch.sh 42
  94. ```
  95. Run in `examples/`; do not copy training files out of their uv project. Data paths
  96. are relative to the script. Optional backends: `uv sync --extra torch`,
  97. `--extra boost`, `--extra tabpfn`, `--extra tabnet`, or `--extra all`. Some optional
  98. backends need compatible GPU builds, credentials/model downloads, or extra setup.
  99. The 19 registered models cover boosting, neural/tabular models, linear/Bayesian
  100. regression, SVR/KNN, and stacking. They load optional dependencies only when selected;
  101. GPU support depends on the backend and installed build. Start with a CPU backend
  102. to verify your data and evaluation contract.
  103. **Historical result caveat:** the archived worklog reports R² 0.440 → 0.783, but it
  104. used feature selection informed by held-out data and changed aggregation/CV during
  105. the search. Those scores are not a validated like-for-like improvement or an
  106. independent new-player accuracy estimate. The [original narrative](experiments/worklog.md)
  107. is preserved with that qualification. Current code uses training-fold feature
  108. selection and keeps held-out labels out of fitting; rerun a fixed protocol to
  109. establish a new baseline. No replacement accuracy claim is supplied by this review.
  110. ## Development
  111. ```bash
  112. python3 -m unittest discover -s tests -v
  113. shellcheck install.sh uninstall.sh hooks/*.sh skills/autoresearch/scripts/*.sh examples/autoresearch.sh
  114. ```
  115. Core tests need only the standard library; example tests skip if their scientific
  116. packages are absent. To exercise them: `uv sync --project examples`, then
  117. `uv run --project examples python -m unittest discover -s tests -p test_examples.py -v`.
  118. CI runs core tests on Linux/macOS and the example suite with core dependencies.
  119. Fake-CLI tests validate supervisor behavior; authenticated Codex runs and optional
  120. GPU/model backends require separate integration checks.
  121. The real Codex keep/commit/log cycle is now verified after repairing this host's
  122. Ubuntu AppArmor profile and granting scoped Git metadata writes. Sandbox boundary
  123. checks also passed. See the [sandbox repair and verification report](docs/codex-sandbox.md).
  124. Contributor guidance: [AGENTS.md](AGENTS.md). License: [MIT](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.