Skip to content
Back to search

auto-deep-researcher-24x7

๐Ÿ”ฅ An autonomous AI agent that runs your deep learning experiments 24/7 while you sleep. Zero-cost monitoring, Leader-Worker architecture, constant-size memory.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/xiangyue-zhang--auto-deep-researcher-24x7/badge.svg)](/listings/xiangyue-zhang--auto-deep-researcher-24x7)

HTML

<a href="/listings/xiangyue-zhang--auto-deep-researcher-24x7"><img src="/api/listings/xiangyue-zhang--auto-deep-researcher-24x7/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

> **Complete beginner?** Follow every step below. You'll go from zero to a running experiment agent in ~10 minutes. > > **Prefer AI-guided setup?** Open [`AI_GUIDE.md`](AI_GUIDE.md) in Claude Code, ChatGPT, or Codex โ€” the AI will walk you through everything interactively.

Step 0: What You Need

| Requirement | Why | How to Check | |-------------|-----|-------------| | Python 3.10+ | Runtime | `python3 --version` | | [Claude Code](https://claude.ai/claude-code) | The AI backbone | `claude --version` | | 1+ NVIDIA GPU | For training | `nvidia-smi` | | Anthropic API key | LLM calls | `echo $ANTHROPIC_API_KEY` |

Don't have an API key? Get one at [console.anthropic.com](https://console.anthropic.com/) and set it:

export ANTHROPIC_API_KEY="sk-ant-xxxxx"
# Add to ~/.bashrc or ~/.zshrc to make it permanent

Step 1: Install

# Clone the repo
git clone https://github.com/Xiangyue-Zhang/auto-deep-researcher-24x7.git
cd auto-deep-researcher-24x7

# Install Python dependencies
pip install -r requirements.txt

# Install 8 Claude slash commands and 8 Codex local skills
python install.py

# Verify everything works
python -m core.loop --check

You should see:

  Deep Researcher Agent โ€” Installer
  ========================================

    โœ“ Claude /auto-experiment
    โœ“ Claude /experiment-status
    โœ“ Claude /gpu-monitor
    โœ“ Claude /daily-papers
    โœ“ Claude /paper-analyze
    โœ“ Claude /conf-search
    โœ“ Claude /progress-report
    โœ“ Claude /obsidian-sync
    โœ“ Codex $auto-experiment
    ...

  Done! 8 Claude commands and 8 Codex skills installed.

Step 2: Create Your First Project

Let's say you want to train a ResNet on CIFAR-100. Create a project folder with a `PROJECT_BRIEF.md`:

mkdir ~/my-first-experiment
cd ~/my-first-experiment

Now write the brief โ€” **this is the most important file**. It tells the agent what you want:

cat > PROJECT_BRIEF.md << 'EOF'
# Goal
Train a ResNet-50 on CIFAR-100 to reach 80%+ test accuracy.

# Codebase
The agent should create the training code from scratch using PyTorch.
- Use torchvision for the dataset (auto-download)
- Save checkpoints to ./checkpoints/
- Log metrics to ./logs/

# What to Try
- Start with a basic ResNet-50, lr=0.1, SGD, 100 epochs
- If accuracy < 75%, try cosine annealing + warmup
- If accuracy 75-80%, try adding mixup or cutout augmentation
- If accuracy > 80%, the goal is reached

# Constraints
- Use GPU 0 only
- Max 100 epochs per run
- Batch size 128

# Current Status
No experiments run yet. Starting from scratch.
EOF

**Tips for writing a good brief:** - Be specific about the goal (metric + target value) - Tell it where the code/data is (or say "create from scratch") - List constraints (which GPU, max epochs, etc.) - Give it a decision tree ("if X, try Y") โ€” this guides the agent like you would guide a junior student

Step 3: Launch the Agent

**Option A: Through Claude Code (recommended)**

Open Claude Code and type:

/auto-experiment --project ~/my-first-experiment --gpu 0

**Option B: Through Python directly**

python -m core.loop \
  --project ~/my-first-experiment \
  --gpu 0 \
  --max-cycles 5    # Stop after 5 cycles (remove for unlimited)

Step 4: Watch What Happens

The agent will now do everything automatically. Here's what each cycle looks like:

=== Cycle 1 ===

[THINK] Reading PROJECT_BRIEF.md...
        Goal: ResNet-50 on CIFAR-100, target 80%+
        No previous experiments. Starting with baseline.
        Plan: Basic ResNet-50, lr=0.1, SGD with momentum, 100 epochs.

[EXECUTE] Creating train.py...
          Creating config.yaml...
          Running dry-run (2 steps)... โœ“ No errors
          Launching training: nohup python train.py --config config.yaml
          PID: 12345, Log: logs/exp001.log

[MONITOR] Training in progress... (zero LLM cost)
          15:00 โ€” PID alive, GPU 98%, Epoch 12/100, loss=2.34

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md ยท README ยท 1,204 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src="assets/banner.png" alt="Deep Researcher Agent" width="700"/>
  3. </p>
  4. <h1 align="center">Deep Researcher Agent</h1>
  5. <h3 align="center">24/7 Autonomous Deep Learning Experiment Agent</h3>
  6. <p align="center">
  7. <strong>An AI agent that autonomously runs your deep learning experiments 24/7 while you sleep.</strong>
  8. </p>
  9. <p align="center">
  10. <a href="README.md">English</a> |
  11. <a href="docs/README_CN.md">ไธญๆ–‡</a> |
  12. <a href="docs/README_JP.md">ๆ—ฅๆœฌ่ชž</a> |
  13. <a href="docs/README_KR.md">ํ•œ๊ตญ์–ด</a>
  14. </p>
  15. <p align="center">
  16. <a href="#quickstart"><img src="https://img.shields.io/badge/-Quick_Start-blue?style=for-the-badge" alt="Quick Start"/></a>
  17. <a href="docs/architecture.md"><img src="https://img.shields.io/badge/-Architecture-orange?style=for-the-badge" alt="Architecture"/></a>
  18. </p>
  19. <p align="center">
  20. <img src="https://img.shields.io/badge/python-3.10+-blue.svg" alt="Python"/>
  21. <img src="https://img.shields.io/badge/Claude_Code-compatible-blueviolet.svg" alt="Claude Code"/>
  22. <img src="https://img.shields.io/badge/Codex_CLI-compatible-green.svg" alt="Codex CLI"/>
  23. <img src="https://img.shields.io/badge/license-Apache_2.0-green.svg" alt="License"/>
  24. <a href="https://github.com/Xiangyue-Zhang/auto-deep-researcher-24x7/stargazers"><img src="https://img.shields.io/github/stars/Xiangyue-Zhang/auto-deep-researcher-24x7?color=yellow&logo=github&label=Stars" alt="Stars"/></a>
  25. </p>
  26. <p align="center">
  27. <a href="https://arxiv.org/abs/2604.05854"><img src="https://img.shields.io/badge/Technical%20Report-2604.05854-b31b1b.svg" alt="Technical Report"/></a>
  28. </p>
  29. ---
  30. ## Recent Updates
  31. **2026-06-03 โ€” Domestic LLM API presets**
  32. - Run the agent on a **Chinese LLM API instead of a Claude/Codex subscription** by
  33. setting `agent.provider` to a one-word preset โ€” `deepseek`, `qwen` (`dashscope`),
  34. `kimi` (`moonshot`), or `glm` (`zhipu`). The preset auto-fills the OpenAI-compatible
  35. `base_url` and the default key env (`DEEPSEEK_API_KEY` / `DASHSCOPE_API_KEY` /
  36. `MOONSHOT_API_KEY` / `ZHIPUAI_API_KEY`); you just set `model` to that vendor's model
  37. id. `base_url` / `api_key_env` stay overridable for self-hosted or proxied endpoints.
  38. This is a thin alias over the existing OpenAI-compatible path โ€” no new dependency.
  39. (`core/agents.py`)
  40. ```yaml
  41. agent:
  42. provider: "deepseek" # or qwen / kimi / glm
  43. model: "deepseek-chat" # vendor's model id
  44. ```
  45. **2026-06-02 โ€” Slurm execution backend + truthful experiment outcomes**
  46. - **Slurm execution backend** โ€” added `execution.mode: "slurm"` so the agent can
  47. drive experiments on a Slurm cluster. The controller stays local; training is
  48. submitted to the login node with `sbatch --parsable` over a single transient
  49. SSH call that exits immediately โ€” **no process is ever left running on the
  50. login node**. `sacct` is the sole liveness authority (Slurm enforces `--time`),
  51. GPU status is read from the partition's `squeue` occupancy, and two bounds
  52. inside the liveness check (consecutive-unknown grace + a `--time`-derived
  53. wall-clock backstop) guarantee the monitor loop terminates even if the cluster
  54. goes unreachable โ€” without ever reaping a job `sacct` still reports as queued
  55. or running. File and repo-reading ops reuse the SSH path (the login node shares
  56. the NFS workspace). (`core/execution.py`)
  57. - **Truthful experiment outcomes** โ€” the monitor now asks the backend for a
  58. finished job's real terminal state via `final_status()`, so a `FAILED` /
  59. `TIMEOUT` / `CANCELLED` run is no longer silently recorded as `completed`. The
  60. outcome flows into `state.json`, the experiment ledger, and the REFLECT
  61. context, so the agent reasons over what actually happened. On Slurm the state
  62. comes from `sacct`; pid-only backends (`local`/`ssh`) report it as
  63. indeterminate and keep prior behavior. (`core/monitor.py`, `core/loop.py`)
  64. - Additive and opt-in; `local`/`ssh` behavior is unchanged. (+21 unit tests, no
  65. cluster required.)
  66. **2026-06-01 โ€” v2.0 (major update)**
  67. This release gives the agent (a) a persistent, queryable memory of its own
  68. experiments, (b) explicit progress/quality/safety signals derived from that
  69. memory, and (c) much stronger code- and literature-reading tools. Every change
  70. is additive and backward-compatible โ€” existing projects keep working unchanged,
  71. the new gate and rate limit are opt-in, and the whole suite is unit-tested
  72. without a GPU or network (60 โ†’ 99 tests).
  73. *New: autonomy layer*
  74. - **Experiment ledger** โ€” every cycle's hypothesis, metrics, and outcome are appended to `workspace/experiments.jsonl`. Crash-safe, zero token cost, and fed back into planning so the agent remembers what it already tried. (`core/ledger.py`)
  75. - **Data-driven stagnation signal** โ€” the planner is told, from the ledger's metric trajectory, whether results are still improving or have stalled (set `ledger.metric_key`), instead of only a binary repeat-counter.
  76. - **Append-only research journals** โ€” `DEAD_ENDS.md` (failed approaches โ€” do not retry) and `INSIGHTS.md` (durable observations). Never compacted; rotated to dated backups when large, so history is never silently dropped. (`core/journal.py`)
  77. - **Zero-cost violation scanner + advisory phase gate** โ€” surface stuck/stale states and whether a baseline metric bar is met, as pure functions over state + ledger. (`core/safety.py`, `core/ledger.py`)
  78. - **Proactive anti-burn rate limiting** โ€” optional `agent.max_cycles_per_hour` cap protects budget when the agent is stuck in a loop.
  79. *New: agent tools*
  80. - **Code comprehension** โ€” `search_code` (regex grep across the workspace), `list_tree` (recursive, depth-limited repo map), and `read_file` line ranges so large files are no longer blindly truncated. Symlink-safe (never escapes the workspace).
  81. - **Literature** โ€” `get_paper` (paper details + reference/citation snowballing) and `search_arxiv` (freshest preprints), alongside the existing Semantic Scholar search.
  82. - All new tools work identically in **local and SSH** execution modes.
  83. *Config:* new optional sections `ledger:`, `stagnation:`, `journal:`, `safety:`, `gates:`, and `agent.max_cycles_per_hour` โ€” all default to current behavior. See `config.yaml`.
  84. **2026-04-22**
  85. - Added explicit compatible-API configuration, dual Claude/Codex skill installation, and safer skill-installer ownership checks.
  86. **2026-04-21**
  87. - Added an optional SSH execution backend so the controller can stay local while code edits, training, logs, PID checks, and GPU queries run on one remote host.
  88. **2026-04-19**
  89. - Added a real multi-turn worker tool-use loop with authoritative tool-result handoff, stricter CLI behavior, and safer tool-call parsing.
  90. **2026-04-18**
  91. - Added subscription-backed `claude_cli` and `codex_cli` provider modes with fail-fast provider validation and more defensive CLI subprocess handling.
  92. **2026-04-09**
  93. - Reduced token growth and tightened loop/tool safeguards with leader-history resets, no-progress fallback, and stronger path and shell protections.
  94. **2026-04-08**
  95. - Added progress tracking exports with optional Obsidian sync and local text fallback when no vault is configured.
  96. ## Start In 3 Steps
  97. If you only want the shortest path to a working experiment loop, do this:
  98. 1. Create a project folder with one file: `PROJECT_BRIEF.md`
  99. 2. Run `/auto-experiment --project /path/to/project --gpu 0`
  100. 3. Check progress with `/experiment-status` or optional Obsidian/local text notes
  101. Prefer AI-guided setup? Open [`AI_GUIDE.md`](AI_GUIDE.md) in Claude / ChatGPT / Codex and let the assistant walk you through it.
  102. ## What You Actually Need
  103. | Requirement | Required | Notes |
  104. |-------------|----------|-------|
  105. | Python 3.10+ | Yes | Runtime |
  106. | 1+ NVIDIA GPU | Yes | For training |
  107. | API key | Yes | Anthropic-compatible or OpenAI-compatible endpoint |
  108. | `PROJECT_BRIEF.md` | Yes | Main control file |
  109. | Project `config.yaml` | Optional | Only if you want to override defaults |
  110. | Obsidian vault | Optional | If absent, notes fall back to local text files |
  111. ## Minimum Working Example
  112. The smallest project you can launch looks like this:
  113. ```text
  114. my-first-experiment/
  115. โ”œโ”€โ”€ PROJECT_BRIEF.md
  116. โ””โ”€โ”€ workspace/ # auto-created
  117. ```
  118. Minimal `PROJECT_BRIEF.md`:
  119. ```md
  120. # Goal
  121. Train a ResNet-50 on CIFAR-100 to reach 80%+ accuracy.
  122. # Codebase
  123. Create the training code from scratch in PyTorch.
  124. # What to Try
  125. - Start with a basic ResNet-50 baseline.
  126. - If accuracy < 75%, improve optimization and schedule.
  127. - If accuracy is 75-80%, try augmentation.
  128. - If accuracy > 80%, stop and report.
  129. # Constraints
  130. - Use GPU 0 only
  131. - Max 100 epochs per run
  132. ```
  133. That is enough to start. Everything else is optional refinement.
  134. ## What This Project Is Good At
  135. This project is for people who already know what experiment they want to run, but do not want to babysit the loop:
  136. - edit code
  137. - launch training
  138. - monitor runs
  139. - parse logs
  140. - decide the next variation
  141. - keep going while you sleep
  142. It is not trying to replace the researcher. It is trying to take over the repetitive experiment-ops layer.
  143. ## Why It Feels Different From A Simple Script
  144. - It does not just launch one run. It keeps iterating.
  145. - It does not just monitor. It reflects and decides the next step.
  146. - It stays cheap because training-time monitoring makes zero LLM calls.
  147. - It stays controllable because the human can override direction at any cycle.
  148. - It now supports persistent progress notes in Obsidian or local text files.
  149. ## How You Stay In Control
  150. You control the research direction through three files:
  151. - `PROJECT_BRIEF.md`: stable goal, constraints, allowed search space
  152. - `HUMAN_DIRECTIVE.md`: temporary redirect for the next cycle
  153. - `workspace/MEMORY_LOG.md`: rolling memory of results and decisions
  154. Common control patterns:
  155. ```md
  156. # Keep the search narrow
  157. - Only tune augmentation.
  158. - Do not change the backbone.
  159. - Keep training budget fixed.
  160. ```
  161. ```md
  162. # Make the agent stop exploring a weak direction
  163. - If gain stays below 0.3 points for 3 runs, stop this branch.
  164. - Return to the last trusted baseline and try a different idea.
  165. ```
  166. ```md
  167. # Force result verification
  168. - If a result looks unusually strong, rerun with the same seed and one new seed.
  169. - Do not claim improvement until both reproduce.
  170. ```
  171. ## How You See Progress
  172. You should never have to guess what the agent is doing.
  173. - `/experiment-status` shows current goal, best result, cycle count, running status, and recent decisions
  174. - `/progress-report` generates a structured summary
  175. - `/obsidian-sync` refreshes persistent notes manually
  176. - `workspace/progress_tracking/` stores local text notes when no Obsidian vault is configured
  177. If you want a dashboard outside the terminal:
  178. ```yaml
  179. obsidian:
  180. enabled: true
  181. vault_path: "~/Documents/MyObsidianVault" # Optional
  182. auto_append_daily: true
  183. ```
  184. If `vault_path` is empty, the same information is saved locally:
  185. ```text
  186. workspace/progress_tracking/Dashboard.txt
  187. workspace/progress_tracking/Daily/YYYY-MM-DD.txt
  188. ```
  189. ---
  190. ## ๐Ÿ’› A Note on Why We Built This โ€” and How We Hope You'll Use It
  191. > **Our hope is simple: science stays pure, and the human stays in the loop.**
  192. We built this framework for one reason โ€” to take the *repetitive, mechanical* parts of running deep learning experiments off the researcher's plate (launching jobs, watching GPUs, parsing logs, sweeping hyperparameters) so that more of your time can go into **the part that actually matters: thinking**.
  193. If you're here because you want to spend less time babysitting training runs and more time reading, reasoning, and chasing your own ideas โ€” welcome. That's exactly who we built this for.
  194. **A gentle thought we'd love every user to share with us:**
  195. The agent is happy to run the experiments. But please let the *ideas*, the *interpretation*, and the *scientific judgment* remain yours. We don't see automation and academic integrity as being in tension โ€” quite the opposite. The hours this tool gives back are meant to be reinvested in **deeper thinking**, not in skipping it.
  196. So we'd kindly ask that this project not be used to fabricate results, to generate "research" with no human in the loop, or to shortcut the parts of science that depend on a human actually understanding what they're doing. That isn't the future we want to help build โ€” and we don't think it's the one most of you want either.
  197. > **Science should stay pure. The agent can run the experiments โ€” but the ideas, the interpretation, and the responsibility belong to the human.**
  198. >
  199. > **ๅญฆๆœฏๅบ”ๅฝ“ไฟๆŒ็บฏ็ฒนใ€‚** Agent ๅฏไปฅๆ›ฟไฝ ่ท‘ๅฎž้ชŒ๏ผŒไฝ† ideaใ€ๅˆคๆ–ญไธŽ่ดฃไปป๏ผŒ่ฏท็•™็ป™ไบบๆฅๆ‰ฟๆ‹…ใ€‚ๆˆ‘ไปฌ็œŸๅฟƒๅธŒๆœ›ๆฏไธ€ไฝไฝฟ็”จ่€…้ƒฝ่ƒฝ **human in the loop ๅœฐๅŽปๆ€่€ƒ**๏ผŒๆŠŠ่ฟ™ไธชๅทฅๅ…ท็œไธ‹ๆฅ็š„ๆ—ถ้—ด๏ผŒๆŠ•ๅ…ฅๅˆฐ็œŸๆญฃๅฑžไบŽไฝ ่‡ชๅทฑ็š„็ ”็ฉถๆ–นๅ‘้‡Œใ€‚
  200. >
  201. > **็ง‘ๅญฆใฏ็ด”็ฒ‹ใงใ‚ใ‚‹ในใใงใ™ใ€‚** Agent ใฏๅฎŸ้จ“ใ‚’่ตฐใ‚‰ใ›ใ‚‹ใ“ใจใŒใงใใพใ™ใŒใ€ใ‚ขใ‚คใƒ‡ใ‚ขใƒป่งฃ้‡ˆใƒป่ฒฌไปปใฏใ€ใฉใ†ใ‹ไบบ้–“ใฎๆ‰‹ใซๆฎ‹ใ—ใฆใใ ใ•ใ„ใ€‚
  202. >
  203. > **๊ณผํ•™์€ ์ˆœ์ˆ˜ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.** Agent๋Š” ์‹คํ—˜์„ ๋Œ€์‹  ์‹คํ–‰ํ•ด ์ค„ ์ˆ˜ ์žˆ์ง€๋งŒ, ์•„์ด๋””์–ด์™€ ํ•ด์„, ๊ทธ๋ฆฌ๊ณ  ์ฑ…์ž„์€ ๋ถ€๋”” ์‚ฌ๋žŒ์˜ ๋ชซ์œผ๋กœ ๋‚จ๊ฒจ์ฃผ์„ธ์š”.
  204. We trust the people who pick up this tool to take that seriously โ€” and we built it because we believe most of you already do. Thank you for being one of them. ๐Ÿ’›
  205. ---
  206. ## The Core Idea
  207. You design the experiment. The agent handles the repetitive loop.
  208. **Deep Researcher Agent**:
  209. 1. **Thinks** โ€” Reads your project brief, analyzes previous results, plans the next experiment
  210. 2. **Executes** โ€” Modifies code/configs, runs a dry-run, launches training on GPU
  211. 3. **Monitors** โ€” Watches training at **zero LLM cost** (just process checks + log reads)
  212. 4. **Reflects** โ€” Parses results, compares with baselines, decides what to try next
  213. 5. **Repeats** โ€” 24/7, without human intervention
  214. ```
  215. You sleep 8 hours โ†’ Agent runs 3 experiment cycles
  216. You go on vacation โ†’ Agent explores 50+ hyperparameter configs
  217. You write your paper โ†’ Agent already has the results table ready
  218. ```
  219. ---
  220. ## Battle-Tested Results
  221. > Not benchmarks. Real results from months of 24/7 autonomous operation across research projects.
  222. | Metric | Result |
  223. |--------|--------|
  224. | Autonomous experiment cycles completed | 500+ |
  225. | Best single-project improvement | 52% over baseline (across 200+ auto-run experiments) |
  226. | Concurrent projects managed | 4 projects across 4 GPU servers |
  227. | Longest continuous autonomous operation | 30+ days without human intervention |
  228. | Average LLM cost per 24h cycle | ~$0.08 |
  229. ---
  230. ## Key Innovation: Zero-Cost Monitoring
  231. The #1 concern with running LLM agents 24/7: **cost**.
  232. Most agent frameworks call the LLM every few minutes to "check progress". That's $50+/day.
  233. Experiment Agent **sleeps** during training โ€” zero API calls. It only wakes the LLM when training finishes.
  234. ```
  235. LLM Active Zero Cost LLM Active
  236. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  237. โ”‚ THINK โ”‚ โ”‚ TRAIN & MONITOR โ”‚ โ”‚ REFLECT โ”‚
  238. โ”‚ (5-10 min) โ”‚ โ”‚ (hours/days) โ”‚ โ”‚ (5-10 min) โ”‚
  239. โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
  240. โ”‚ โ€ข Analyze โ”‚ โ”‚ โ€ข kill -0 $PID โ”‚ โ”‚ โ€ข Parse โ”‚
  241. โ”‚ โ€ข Plan โ”‚ โ”‚ โ€ข nvidia-smi โ”‚ โ”‚ logs โ”‚
  242. โ”‚ โ€ข Code โ”‚ โ”‚ โ€ข tail log โ”‚ โ”‚ โ€ข Compare โ”‚
  243. โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ€ข Decide โ”‚
  244. โ”‚ ~$0.05 โ”‚ โ”‚ $0.00 โ”‚ โ”‚ ~$0.03 โ”‚
  245. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  246. ```
  247. **24-hour cycle with 8 hours of training: ~$0.08 in LLM calls.**
  248. ---
  249. ## Architecture
  250. ### The THINK โ†’ EXECUTE โ†’ REFLECT Loop
  251. ```
  252. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  253. โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
  254. โ”‚ โ”‚ THINK โ”‚โ”€โ”€โ”€โ†’โ”‚ EXECUTE โ”‚โ”€โ”€โ”€โ†’โ”‚ REFLECT โ”‚โ”€โ”€โ” โ”‚
  255. โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
  256. โ”‚ โ”‚ Analyze โ”‚ โ”‚ Dry-run โ”‚ โ”‚ Evaluate โ”‚ โ”‚ โ”‚
  257. โ”‚ โ”‚ Plan โ”‚ โ”‚ Launch โ”‚ โ”‚ Compare โ”‚ โ”‚ โ”‚
  258. โ”‚ โ”‚ Decide โ”‚ โ”‚ Monitor โ”‚ โ”‚ Update โ”‚ โ”‚ โ”‚
  259. โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚
  260. โ”‚ โ†‘ โ”‚ โ”‚
  261. โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
  262. โ”‚ โ†ป 24/7 Loop โ”‚
  263. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  264. ```
  265. ### Leader-Worker Agent System
  266. Only ONE worker runs at a time. Others idle at zero cost.
  267. ```
  268. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  269. โ”‚ Leader โ”‚ Persistent conversation
  270. โ”‚ (Planner) โ”‚ within each cycle
  271. โ””โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”˜
  272. โ”‚ โ”‚ โ”‚
  273. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  274. โ†“ โ†“ โ†“
  275. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  276. โ”‚ Idea โ”‚ โ”‚ Code โ”‚ โ”‚ Writing โ”‚
  277. โ”‚ Agent โ”‚ โ”‚ Agent โ”‚ โ”‚ Agent โ”‚
  278. โ”‚ (4 tools)โ”‚ โ”‚ (5 tools)โ”‚ โ”‚ (3 tools)โ”‚
  279. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  280. ```
  281. ### Two-Tier Memory (Constant Size Forever)
  282. ```
  283. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  284. โ”‚ Tier 1: PROJECT_BRIEF.md โ”‚
  285. โ”‚ โ€ข Frozen project reference โ”‚
  286. โ”‚ โ€ข Max 3,000 chars โ”‚
  287. โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
  288. โ”‚ Tier 2: MEMORY_LOG.md โ”‚
  289. โ”‚ โ€ข Key Results (auto-compact at 1,200ch) โ”‚
  290. โ”‚ โ€ข Recent Decisions (rolling last 15) โ”‚
  291. โ”‚ โ€ข Max 2,000 chars โ”‚
  292. โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
  293. โ”‚ Total: ~5K chars / ~1,500 tokens โ”‚
  294. โ”‚ SAME whether running 1 day or 6 months โ”‚
  295. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  296. ```
  297. ### Cost Control Strategies (8 Total)
  298. | # | Strategy | Savings |
  299. |---|----------|---------|
  300. | 1 | Zero-LLM monitoring during training | 90%+ of runtime is free |
  301. | 2 | Two-Tier memory with auto-compaction | Fixed context window |
  302. | 3 | Leader conversation persists within cycle | Brief sent once per cycle |
  303. | 4 | Anthropic prompt caching | System/tools cached |
  304. | 5 | Per-agent minimal tool sets (3-5 tools) | Less schema overhead |
  305. | 6 | Slim system prompts | Fewer input tokens |
  306. | 7 | State trimmed before sending | No bloat |
  307. | 8 | Single worker at a time | No parallel LLM costs |
  308. ---
  309. <a name="quickstart"></a>
  310. ## Getting Started (Step by Step)
  311. > **Complete beginner?** Follow every step below. You'll go from zero to a running experiment agent in ~10 minutes.
  312. >
  313. > **Prefer AI-guided setup?** Open [`AI_GUIDE.md`](AI_GUIDE.md) in Claude Code, ChatGPT, or Codex โ€” the AI will walk you through everything interactively.
  314. ### Step 0: What You Need
  315. | Requirement | Why | How to Check |
  316. |-------------|-----|-------------|
  317. | Python 3.10+ | Runtime | `python3 --version` |
  318. | [Claude Code](https://claude.ai/claude-code) | The AI backbone | `claude --version` |
  319. | 1+ NVIDIA GPU | For training | `nvidia-smi` |
  320. | Anthropic API key | LLM calls | `echo $ANTHROPIC_API_KEY` |
  321. Don't have an API key? Get one at [console.anthropic.com](https://console.anthropic.com/) and set it:
  322. ```bash
  323. export ANTHROPIC_API_KEY="sk-ant-xxxxx"
  324. # Add to ~/.bashrc or ~/.zshrc to make it permanent
  325. ```
  326. ### Step 1: Install
  327. ```bash
  328. # Clone the repo
  329. git clone https://github.com/Xiangyue-Zhang/auto-deep-researcher-24x7.git
  330. cd auto-deep-researcher-24x7
  331. # Install Python dependencies
  332. pip install -r requirements.txt
  333. # Install 8 Claude slash commands and 8 Codex local skills
  334. python install.py
  335. # Verify everything works
  336. python -m core.loop --check
  337. ```
  338. You should see:
  339. ```
  340. Deep Researcher Agent โ€” Installer
  341. ========================================
  342. โœ“ Claude /auto-experiment
  343. โœ“ Claude /experiment-status
  344. โœ“ Claude /gpu-monitor
  345. โœ“ Claude /daily-papers
  346. โœ“ Claude /paper-analyze
  347. โœ“ Claude /conf-search
  348. โœ“ Claude /progress-report
  349. โœ“ Claude /obsidian-sync
  350. โœ“ Codex $auto-experiment
  351. ...
  352. Done! 8 Claude commands and 8 Codex skills installed.
  353. ```
  354. ### Step 2: Create Your First Project
  355. Let's say you want to train a ResNet on CIFAR-100. Create a project folder with a `PROJECT_BRIEF.md`:
  356. ```bash
  357. mkdir ~/my-first-experiment
  358. cd ~/my-first-experiment
  359. ```
  360. Now write the brief โ€” **this is the most important file**. It tells the agent what you want:
  361. ```bash
  362. cat > PROJECT_BRIEF.md << 'EOF'
  363. # Goal
  364. Train a ResNet-50 on CIFAR-100 to reach 80%+ test accuracy.
  365. # Codebase
  366. The agent should create the training code from scratch using PyTorch.
  367. - Use torchvision for the dataset (auto-download)
  368. - Save checkpoints to ./checkpoints/
  369. - Log metrics to ./logs/
  370. # What to Try
  371. - Start with a basic ResNet-50, lr=0.1, SGD, 100 epochs
  372. - If accuracy < 75%, try cosine annealing + warmup
  373. - If accuracy 75-80%, try adding mixup or cutout augmentation
  374. - If accuracy > 80%, the goal is reached
  375. # Constraints
  376. - Use GPU 0 only
  377. - Max 100 epochs per run
  378. - Batch size 128
  379. # Current Status
  380. No experiments run yet. Starting from scratch.
  381. EOF
  382. ```
  383. **Tips for writing a good brief:**
  384. - Be specific about the goal (metric + target value)
  385. - Tell it where the code/data is (or say "create from scratch")
  386. - List constraints (which GPU, max epochs, etc.)
  387. - Give it a decision tree ("if X, try Y") โ€” this guides the agent like you would guide a junior student
  388. ### Step 3: Launch the Agent
  389. **Option A: Through Claude Code (recommended)**
  390. Open Claude Code and type:
  391. ```
  392. /auto-experiment --project ~/my-first-experiment --gpu 0
  393. ```
  394. **Option B: Through Python directly**
  395. ```bash
  396. python -m core.loop \
  397. --project ~/my-first-experiment \
  398. --gpu 0 \
  399. --max-cycles 5 # Stop after 5 cycles (remove for unlimited)
  400. ```
  401. ### Step 4: Watch What Happens
  402. The agent will now do everything automatically. Here's what each cycle looks like:
  403. ```
  404. === Cycle 1 ===
  405. [THINK] Reading PROJECT_BRIEF.md...
  406. Goal: ResNet-50 on CIFAR-100, target 80%+
  407. No previous experiments. Starting with baseline.
  408. Plan: Basic ResNet-50, lr=0.1, SGD with momentum, 100 epochs.
  409. [EXECUTE] Creating train.py...
  410. Creating config.yaml...
  411. Running dry-run (2 steps)... โœ“ No errors
  412. Launching training: nohup python train.py --config config.yaml
  413. PID: 12345, Log: logs/exp001.log
  414. [MONITOR] Training in progress... (zero LLM cost)
  415. 15:00 โ€” PID alive, GPU 98%, Epoch 12/100, loss=2.34
  416. 15:15 โ€” PID alive, GPU 97%, Epoch 25/100, loss=1.87
  417. 15:30 โ€” PID alive, GPU 98%, Epoch 38/100, loss=1.54
  418. ...
  419. 17:45 โ€” PID alive, GPU 97%, Epoch 100/100, loss=0.82
  420. 18:00 โ€” PID terminated. Training complete.
  421. [REFLECT] Parsing logs... test accuracy = 76.3%
  422. Result: 76.3% โ€” below 80% target
  423. Brief says: "If < 75%, try cosine annealing"
  424. 76.3% > 75%, so try augmentation instead.
  425. Decision: Add mixup augmentation, keep lr=0.1 + cosine
  426. Milestone logged: "Exp001: ResNet-50 baseline, 76.3%"
  427. === Cycle 2 ===
  428. [THINK] Best so far: 76.3% (Exp001)
  429. Plan: Add mixup (alpha=0.2) + cosine annealing schedule
  430. ...
  431. ```
  432. ### Step 5: Check Progress Anytime
  433. While the agent is running, you can check on it:
  434. ```bash
  435. # In Claude Code:
  436. /experiment-status --project ~/my-first-experiment
  437. # Or check GPU usage:
  438. /gpu-monitor
  439. ```
  440. You'll see something like:
  441. ```
  442. # Experiment Status โ€” my-first-experiment
  443. ## Goal
  444. ResNet-50 on CIFAR-100 โ†’ 80%+ accuracy
  445. ## Progress
  446. - Cycles completed: 3
  447. - Current best: 79.1% (Exp003: ResNet-50 + mixup + cosine)
  448. - Status: TRAINING (PID 12389, GPU 0, running 1.5h)
  449. ## Key Results
  450. [04-07 15:00] Exp001: ResNet-50 baseline, 76.3%
  451. [04-07 18:30] Exp002: + cosine annealing, 77.8%
  452. [04-07 22:00] Exp003: + mixup ฮฑ=0.2, 79.1% โ† best
  453. ## Current Training
  454. Epoch 67/100 | loss: 0.71 | acc: 79.4%
  455. ```
  456. ### Step 5.5: Save Progress to Obsidian or Local Text
  457. Enable progress export in your project `config.yaml`:
  458. ```yaml
  459. obsidian:
  460. enabled: true
  461. vault_path: "~/Documents/MyObsidianVault" # Optional
  462. project_subdir: "DeepResearcher/{project_name}"
  463. auto_append_daily: true
  464. ```
  465. If `vault_path` is set, the agent writes:
  466. ```text
  467. DeepResearcher/my-first-experiment/Dashboard.md
  468. DeepResearcher/my-first-experiment/Daily/YYYY-MM-DD.md
  469. ```
  470. If `vault_path` is empty, it falls back to project-local files:
  471. ```text
  472. workspace/progress_tracking/Dashboard.txt
  473. workspace/progress_tracking/Daily/YYYY-MM-DD.txt
  474. ```
  475. Manual refresh:
  476. ```bash
  477. /obsidian-sync --project ~/my-first-experiment
  478. # or
  479. python -m core.obsidian --project ~/my-first-experiment
  480. ```
  481. ### Step 6: Intervene If Needed
  482. Want to change direction? Three ways, from anywhere:
  483. ```bash
  484. # Way 1: Drop a directive file (agent reads it next cycle)
  485. echo "Stop trying ResNet. Switch to ViT-B/16, start with lr=1e-3" \
  486. > ~/my-first-experiment/workspace/HUMAN_DIRECTIVE.md
  487. # Way 2: Command-line flag
  488. python -m core.loop --project ~/my-first-experiment \
  489. --directive "Try label smoothing 0.1"
  490. # Way 3: Edit memory directly (for permanent changes)
  491. vim ~/my-first-experiment/workspace/MEMORY_LOG.md
  492. ```
  493. ## Human-in-the-Loop Playbook
  494. Use the agent as an operator, not a replacement researcher.
  495. ```text
  496. Human decides:
  497. - goal
  498. - constraints
  499. - forbidden directions
  500. - when to pivot
  501. Agent executes:
  502. - code edits
  503. - runs
  504. - monitoring
  505. - summaries
  506. ```
  507. Write stable rules in `PROJECT_BRIEF.md`, and temporary steering in `HUMAN_DIRECTIVE.md`.
  508. ```md
  509. # HUMAN_DIRECTIVE.md
  510. - Do not change the dataset.
  511. - Try label smoothing 0.1 before changing the backbone.
  512. - Stop this direction if gain stays below 0.3 for 3 runs.
  513. - Compare against the last trusted baseline, not just the latest run.
  514. ```
  515. Case 1: Safer ablation
  516. ```md
  517. - Only change augmentation.
  518. - Keep model, optimizer, and training budget fixed.
  519. - Report a clean comparison table after each run.
  520. ```
  521. Case 2: Deliberate pivot
  522. ```md
  523. - Current ResNet line is saturated.
  524. - Switch to ViT-B/16 only if the last 3 runs plateau.
  525. - Before switching, write a short rationale.
  526. ```
  527. Case 3: Suspicious result
  528. ```md
  529. - Accuracy jumped unexpectedly.
  530. - Re-run with the same seed and one new seed.
  531. - Do not claim improvement until both runs reproduce.
  532. ```
  533. Rule of thumb: let the agent handle repetition, but keep direction, interpretation, and responsibility human.
  534. ### Step 7: Mobile Monitoring with [Happy Coder](https://github.com/slopus/happy) (Optional)
  535. Want to check experiments from your phone? Install [Happy Coder](https://happy.engineering/) ([iOS](https://apps.apple.com/us/app/happy-codex-claude-code-app/id6748571505) / [Android](https://play.google.com/store/apps/details?id=com.ex3ndr.happy)):
  536. ```bash
  537. # Install CLI (one time)
  538. npm install -g happy-coder
  539. # Start session through Happy instead of claude
  540. happy
  541. # Inside the session, launch your experiment:
  542. /auto-experiment --project ~/my-first-experiment --gpu 0
  543. ```
  544. Now on your phone you can:
  545. - Get **push notifications** when experiments finish or the agent needs input
  546. - **Check results** while commuting
  547. - **Send directives** ("try learning rate 1e-5") from anywhere
  548. - **Switch between phone and desktop** seamlessly
  549. - All communication is **end-to-end encrypted**
  550. ```
  551. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” encrypted โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  552. โ”‚ Desktop โ”‚ โ—„โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บ โ”‚ Phone โ”‚
  553. โ”‚ Claude โ”‚ relay โ”‚ Happy โ”‚
  554. โ”‚ Code โ”‚ โ”‚ Coder โ”‚
  555. โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
  556. โ”‚ Agent โ”‚ โ† push notify โ”€โ”€ โ”‚ "Try โ”‚
  557. โ”‚ running โ”‚ โ”‚ lr=1e-5"โ”‚
  558. โ”‚ 24/7 โ”‚ โ”€โ”€ status โ”€โ”€โ”€โ”€โ–บ โ”‚ โœ“ Got it โ”‚
  559. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  560. ```
  561. ### What a Good PROJECT_BRIEF.md Looks Like
  562. The brief is your main lever. Here are examples for different scenarios:
  563. <details>
  564. <summary><b>Example: Fine-tuning a pretrained model</b></summary>
  565. ```markdown
  566. # Goal
  567. Fine-tune ViT-B/16 (pretrained on ImageNet-21K) on Oxford Flowers-102.
  568. Target: 95%+ test accuracy.
  569. # Codebase
  570. - Training script: finetune.py (already exists)
  571. - Config: configs/vit_flowers.yaml
  572. - Data: /data/flowers102/ (already downloaded)
  573. - Pretrained weights: /models/vit-b16-21k.pth
  574. # What to Try
  575. 1. First: freeze backbone, train classifier head only (10 epochs, lr=1e-2)
  576. 2. Then: unfreeze all, fine-tune end-to-end (30 epochs, lr=1e-4)
  577. 3. If stuck below 93%: try layer-wise lr decay (0.65)
  578. 4. If above 94%: try test-time augmentation
  579. # Constraints
  580. - GPU 0, batch size 64
  581. - Save best checkpoint based on val accuracy
  582. ```
  583. </details>
  584. <details>
  585. <summary><b>Example: Hyperparameter search</b></summary>
  586. ```markdown
  587. # Goal
  588. Find the best hyperparameters for our GAN on CelebA-HQ 256x256.
  589. Target: FID < 15.
  590. # Codebase
  591. - train_gan.py, configs/celeba_gan.yaml
  592. - Data: /data/celeba_hq_256/
  593. - Evaluation: eval_fid.py --real_dir /data/celeba_hq_256/val
  594. # Search Space
  595. - Learning rate: [1e-4, 2e-4, 5e-4]
  596. - Beta1: [0.0, 0.5]
  597. - Discriminator steps per generator step: [1, 2, 5]
  598. - Spectral norm: [yes, no]
  599. # Strategy
  600. Start with lr=2e-4, beta1=0.0, d_steps=1, spectral_norm=yes (baseline).
  601. Change ONE variable at a time. Run each for 50K steps.
  602. Always evaluate FID after training.
  603. # Constraints
  604. - GPU 0-1 (can use both)
  605. - Max 50K steps per run (~4 hours)
  606. ```
  607. </details>
  608. <details>
  609. <summary><b>Example: Debugging a training issue</b></summary>
  610. ```markdown
  611. # Goal
  612. Figure out why our transformer model diverges after epoch 20.
  613. Currently: loss explodes from 0.5 to NaN around epoch 20-25.
  614. # Codebase
  615. - train_transformer.py, model/transformer.py
  616. - Config: configs/base.yaml
  617. - Logs from failed runs: logs/failed_run_001.log, logs/failed_run_002.log
  618. # What to Investigate
  619. 1. Check gradient norms โ€” add gradient clipping (max_norm=1.0)
  620. 2. Try lower learning rate (current: 1e-3, try: 1e-4, 5e-5)
  621. 3. Check if it's a specific layer โ€” add per-layer gradient logging
  622. 4. Try warmup (1000 steps) if not already present
  623. 5. Check data โ€” are there any NaN/Inf in the dataset?
  624. # Constraints
  625. - GPU 0, run each test for 30 epochs (enough to see if it diverges)
  626. - Log gradient norms every 100 steps
  627. ```
  628. </details>
  629. ### FAQ
  630. <details>
  631. <summary><b>Q: How much does it cost to run?</b></summary>
  632. About $0.08 per 24-hour cycle (if training takes 8 hours). The secret: zero LLM calls during training. You only pay for the THINK and REFLECT phases (~10 min each).
  633. </details>
  634. <details>
  635. <summary><b>Q: Can it modify my existing code?</b></summary>
  636. Yes. The Code Agent can read, write, and modify any file in your project. It will make changes, dry-run to verify, then launch training. It won't touch protected files (PROJECT_BRIEF.md, MEMORY_LOG.md).
  637. </details>
  638. <details>
  639. <summary><b>Q: What if the agent goes in a wrong direction?</b></summary>
  640. Drop a directive: `echo "Stop. Go back to the ResNet approach" > workspace/HUMAN_DIRECTIVE.md`. The agent reads it next cycle with highest priority.
  641. </details>
  642. <details>
  643. <summary><b>Q: Can I run multiple projects at the same time?</b></summary>
  644. Yes. Launch separate agent instances in different terminals/tmux sessions, each pointing to a different project and GPU.
  645. </details>
  646. <details>
  647. <summary><b>Q: What happens if training crashes?</b></summary>
  648. The monitor detects the process died, captures the error log, and passes it to REFLECT. The agent will analyze the crash, fix the code, and retry.
  649. </details>
  650. <details>
  651. <summary><b>Q: Can I use it with PyTorch / TensorFlow / JAX?</b></summary>
  652. Yes. The agent works with any training framework. It just launches shell commands and reads log files โ€” it doesn't care what framework produces them.
  653. </details>
  654. ---
  655. ## One-Click Install (Claude + Codex)
  656. All features are packaged as Claude Code slash commands and Codex local skills.
  657. **One command to install:**
  658. ```bash
  659. python install.py
  660. ```
  661. After installation, you get:
  662. - **8 slash commands** in Claude Code
  663. - **8 local skills** in Codex (restart Codex after install)
  664. ### Core Skills
  665. | Command | What It Does |
  666. |---------|-------------|
  667. | `/auto-experiment` | Launch the 24/7 autonomous THINKโ†’EXECUTEโ†’REFLECT experiment loop |
  668. | `/experiment-status` | Check running experiments: progress, metrics, cycle count, GPU usage |
  669. | `/gpu-monitor` | Quick GPU status: free/busy, memory, utilization, running processes |
  670. ### Research Skills
  671. | Command | What It Does |
  672. |---------|-------------|
  673. | `/daily-papers` | Daily arXiv recommendations with automatic dedup |
  674. | `/paper-analyze 2312.12345` | Deep paper analysis + extract real figures from arXiv source |
  675. | `/conf-search --venue CVPR2025 --query "motion"` | Search CVPR/NeurIPS/ICML/ICLR/AAAI/ECCV... |
  676. | `/progress-report` | Generate structured progress report with metrics |
  677. | `/obsidian-sync` | Refresh Obsidian or local progress notes |
  678. ### Usage Example
  679. ```bash
  680. # Step 1: Install skills (one time)
  681. python install.py
  682. # Step 2a: In Claude Code, launch an experiment loop
  683. /auto-experiment --project /path/to/my_project --gpu 0
  684. # Step 2b: In Codex, use the matching local skill
  685. $auto-experiment
  686. # Step 3: Check how it's going
  687. /experiment-status --project /path/to/my_project
  688. # Step 4: Check GPU resources
  689. /gpu-monitor
  690. # Step 5: Read papers while the agent trains for you
  691. /daily-papers --topics "vision transformer, image classification"
  692. ```
  693. ### Uninstall
  694. ```bash
  695. python install.py --uninstall
  696. ```
  697. ---
  698. ## Supported LLM Providers
  699. Works with **Anthropic-compatible and OpenAI-compatible APIs** out of the box,
  700. and can also run on a **flat-rate subscription** instead of per-token billing
  701. via the local CLIs.
  702. | Tier | Anthropic (Claude) | OpenAI (Codex/GPT) | Best For |
  703. |------|-------------------|-------------------|----------|
  704. | **Fast** | `claude-sonnet-4-6` | `codex-5.3` | Daily experiments, iteration |
  705. | **Strongest** | `claude-opus-4-6` | `gpt-5.4` | Complex reasoning, architecture decisions |
  706. ### Authentication mode: API key vs. subscription
  707. | Mode | `provider` value | Billing | Requires | Tool-use support |
  708. |------|------------------|---------|----------|------------------|
  709. | API โ€” Anthropic-compatible | `anthropic` | Per-token, via `ANTHROPIC_API_KEY` or custom env | `pip install anthropic` | โœ… Full |
  710. | API โ€” OpenAI-compatible | `openai` | Per-token, via `OPENAI_API_KEY` or custom env | `pip install openai` | โœ… Full |
  711. | **Subscription โ€” Claude** | `claude_cli` | Flat-rate, uses your Claude Code / Pro / Max plan | `claude` CLI installed and logged in | โœ… Full |
  712. | **Subscription โ€” ChatGPT** | `codex_cli` | Flat-rate, uses your ChatGPT Plus / Pro plan | `codex` CLI installed and logged in | โš ๏ธ Leader only |
  713. Tool execution is driven by a text-based `<tool_call>` protocol injected
  714. into the worker's system prompt. All three "Full" providers can be forced
  715. into pure text-oracle mode so they honor the protocol (for `claude_cli`
  716. the framework passes `--tools ""` to disable built-in CLI tools). The
  717. `codex` CLI currently offers no equivalent flag โ€” its internal agentic
  718. loop will bypass the protocol and the framework cannot recover PIDs from
  719. experiments it launches. Use `codex_cli` only for the leader/think path
  720. where no tools are needed.
  721. Switch provider in `config.yaml`:
  722. ```yaml
  723. agent:
  724. # Pay-per-token (needs API key):
  725. provider: "anthropic" # or "openai"
  726. model: "claude-sonnet-4-6" # or "codex-5.3"
  727. base_url: "" # optional compatible endpoint override
  728. api_key_env: "" # optional custom key env var name
  729. auth_token_env: "" # optional custom bearer token env var
  730. # Flat-rate subscription (needs CLI login instead of API key):
  731. # provider: "claude_cli" # or "codex_cli"
  732. ```
  733. Compatible API examples
  734. (illustrative only in this repo โ€” these endpoint/model combinations have not
  735. been live-smoke-tested here):
  736. ```yaml
  737. # Qwen / DashScope
  738. agent:
  739. provider: "openai"
  740. model: "qwen-plus"
  741. base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"
  742. api_key_env: "DASHSCOPE_API_KEY"
  743. # GLM / BigModel
  744. agent:
  745. provider: "openai"
  746. model: "glm-4.5"
  747. base_url: "https://open.bigmodel.cn/api/paas/v4"
  748. api_key_env: "ZHIPUAI_API_KEY"
  749. # MiniMax via OpenAI-compatible endpoint
  750. agent:
  751. provider: "openai"
  752. model: "MiniMax-M1"
  753. base_url: "https://api.minimaxi.com/v1"
  754. api_key_env: "MINIMAX_API_KEY"
  755. ```
  756. Or set via environment (API-key modes only):
  757. ```bash
  758. # For Anthropic-compatible provider:
  759. export ANTHROPIC_API_KEY="sk-ant-xxxxx"
  760. export ANTHROPIC_BASE_URL="https://your-anthropic-compatible-endpoint"
  761. # For OpenAI-compatible provider:
  762. export OPENAI_API_KEY="sk-xxxxx"
  763. export OPENAI_BASE_URL="https://your-openai-compatible-endpoint/v1"
  764. # For subscription providers (claude_cli / codex_cli): no env var โ€” just
  765. # install the CLI once and run `claude` or `codex login` to sign in.
  766. ```
  767. ---
  768. ## Configuration
  769. By default, everything runs locally inside `project.workspace`. If you want
  770. to keep the controller on your laptop but run code, training, logs, and GPU
  771. checks on one remote server, enable the optional SSH execution mode:
  772. ```yaml
  773. execution:
  774. mode: "ssh"
  775. ssh_host: "user@your-server"
  776. remote_workspace: "/home/user/my_project/workspace"
  777. remote_python: "python3"
  778. ssh_args: [] # optional, e.g. ["-p", "2222"]
  779. ```
  780. In SSH mode, controller state still stays local:
  781. - `PROJECT_BRIEF.md`
  782. - `workspace/MEMORY_LOG.md`
  783. - `workspace/state.json`
  784. - `workspace/HUMAN_DIRECTIVE.md`
  785. - local progress / Obsidian exports
  786. The remote host only handles the tool-visible workspace, training process,
  787. training logs, PID checks, and `nvidia-smi`.
  788. On a **Slurm cluster**, set `mode: "slurm"`. The controller still stays on your
  789. laptop; training is submitted to the login node with `sbatch --parsable` over a
  790. single transient SSH call that exits immediately (no process is left running on
  791. the login node), and `sacct` is the sole liveness authority โ€” Slurm enforces
  792. `--time`, so a job is always reaped by its time limit plus a safety buffer:
  793. ```yaml
  794. execution:
  795. mode: "slurm"
  796. ssh_host: "user@login-node"
  797. remote_workspace: "/nfs/home/user/my_project/workspace"
  798. slurm_partition: "gpu-h200" # required
  799. slurm_time: "24:00:00" # required (--time wall limit)
  800. slurm_gpus_per_job: 1 # -> --gres=gpu:N
  801. slurm_setup: "module load cuda/12.4" # optional shell line prepended to the job
  802. ```
  803. In Slurm mode the `gpu` argument to `launch_experiment` is ignored โ€” Slurm
  804. assigns GPUs via `--gres`, so `CUDA_VISIBLE_DEVICES` is not pinned by the agent.
  805. See `config.yaml` for the full set of `slurm_*` options.
  806. ```yaml
  807. # config.yaml
  808. project:
  809. name: "my-research"
  810. brief: "PROJECT_BRIEF.md"
  811. execution:
  812. mode: "local" # or "ssh" / "slurm"
  813. ssh_host: "" # required in ssh/slurm mode
  814. remote_workspace: "" # required in ssh/slurm mode
  815. remote_python: "python3"
  816. ssh_args: []
  817. agent:
  818. provider: "anthropic" # "anthropic" or "openai"
  819. model: "claude-sonnet-4-6" # See model table above
  820. base_url: "" # Optional compatible API endpoint override
  821. api_key_env: "" # Optional custom API key env var
  822. auth_token_env: "" # Optional custom bearer token env var
  823. max_cycles: -1 # -1 = run forever
  824. max_steps_per_cycle: 3 # Max worker dispatches per cycle
  825. cooldown_interval: 300 # Smart cooldown polling (seconds)
  826. memory:
  827. brief_max_chars: 3000 # Tier 1 cap
  828. log_max_chars: 2000 # Tier 2 cap
  829. milestone_max_chars: 1200 # Key results cap
  830. max_recent_entries: 15 # Rolling decision count
  831. gpu:
  832. auto_detect: true
  833. reserve_last: true # Reserve last GPU for keep-alive
  834. monitor:
  835. poll_interval: 900 # Check every 15 min during training
  836. zero_llm: true # No LLM during monitoring
  837. experiment:
  838. mandatory_dry_run: true # Always dry-run before real training
  839. max_parallel: 1 # Concurrent experiments
  840. ```
  841. ---
  842. ## How It Compares
  843. | | Deep Researcher Agent | [Claude Scholar](https://github.com/Galaxy-Dawn/claude-scholar) | [AI Scientist](https://github.com/SakanaAI/AI-Scientist) | [OpenHands](https://github.com/All-Hands-AI/OpenHands) | [SWE-Agent](https://github.com/princeton-nlp/SWE-agent) |
  844. |--|:--:|:--:|:--:|:--:|:--:|
  845. | **Runs experiments autonomously** | โœ… | โŒ | โŒ | โŒ | โŒ |
  846. | **Zero-cost training monitoring** | โœ… | โŒ | โŒ | โŒ | โŒ |
  847. | **GPU management** | โœ… | โŒ | โŒ | โŒ | โŒ |
  848. | **24/7 continuous operation** | โœ… | โŒ | โŒ | โŒ | โŒ |
  849. | **Constant-size memory** | โœ… | โŒ | โŒ | โŒ | โŒ |
  850. | Paper writing | Basic | โœ… | โœ… | โŒ | โŒ |
  851. | Knowledge management | Basic | โœ… | โŒ | โŒ | โŒ |
  852. | General coding | โŒ | โŒ | โŒ | โœ… | โœ… |
  853. **Deep Researcher Agent is the only framework built for _running_ deep learning research, not just writing about it.**
  854. ---
  855. ## Project Structure
  856. ```
  857. auto-deep-researcher-24x7/
  858. โ”œโ”€โ”€ core/ # Autonomous experiment loop engine
  859. โ”‚ โ”œโ”€โ”€ loop.py # THINK โ†’ EXECUTE โ†’ REFLECT cycle
  860. โ”‚ โ”œโ”€โ”€ execution.py # Local / SSH execution backends
  861. โ”‚ โ”œโ”€โ”€ memory.py # Two-Tier constant-size memory
  862. โ”‚ โ”œโ”€โ”€ monitor.py # Zero-LLM experiment monitoring
  863. โ”‚ โ”œโ”€โ”€ agents.py # Leader-Worker agent dispatch
  864. โ”‚ โ””โ”€โ”€ tools.py # Minimal per-agent tool registry
  865. โ”œโ”€โ”€ skills/ # Source skills for Claude slash commands + Codex local skills
  866. โ”‚ โ”œโ”€โ”€ auto-experiment/ # 24/7 autonomous experiment loop
  867. โ”‚ โ”œโ”€โ”€ experiment-status/ # Check experiment progress
  868. โ”‚ โ”œโ”€โ”€ gpu-monitor/ # GPU status & availability
  869. โ”‚ โ”œโ”€โ”€ daily-papers/ # Daily arXiv recommendations
  870. โ”‚ โ”œโ”€โ”€ paper-analyze/ # Deep paper analysis + figure extraction
  871. โ”‚ โ”œโ”€โ”€ conf-search/ # Conference paper search
  872. โ”‚ โ””โ”€โ”€ progress-report/ # Progress report generation
  873. โ”œโ”€โ”€ agents/ # Agent prompt definitions
  874. โ”‚ โ”œโ”€โ”€ leader.md # Central decision-maker
  875. โ”‚ โ”œโ”€โ”€ idea_agent.md # Literature & hypothesis
  876. โ”‚ โ”œโ”€โ”€ code_agent.md # Experiment execution
  877. โ”‚ โ””โ”€โ”€ writing_agent.md # Reporting & writing
  878. โ”œโ”€โ”€ gpu/ # GPU utilities
  879. โ”‚ โ”œโ”€โ”€ detect.py # Detection & monitoring
  880. โ”‚ โ””โ”€โ”€ keeper.py # Cloud instance keep-alive
  881. โ”œโ”€โ”€ examples/ # Ready-to-run demos
  882. โ”œโ”€โ”€ docs/ # Docs + translations (CN/JP)
  883. โ”œโ”€โ”€ install.py # Claude + Codex skill installer
  884. โ”œโ”€โ”€ config.yaml # Default configuration
  885. โ””โ”€โ”€ requirements.txt # Dependencies
  886. ```
  887. ---
  888. ## Contributing
  889. Areas where we'd love help:
  890. - More cloud GPU platforms (AWS, GCP, Lambda Labs, RunPod)
  891. - Experiment tracker integration (W&B, MLflow, TensorBoard)
  892. - New research skills (visualization, result comparison)
  893. - Metric extraction for more training frameworks
  894. See [CONTRIBUTING.md](CONTRIBUTING.md).
  895. ---
  896. ## Citation
  897. If you find this work useful, please cite our paper:
  898. ```bibtex
  899. @article{zhang2026autodeepresearcher,
  900. title={Deep Researcher Agent: Autonomous Deep Learning Experiment Framework},
  901. author={Zhang, Xiangyue},
  902. journal={arXiv preprint arXiv:2604.05854},
  903. year={2026},
  904. url={https://arxiv.org/abs/2604.05854}
  905. }
  906. ```
  907. Or cite the software release:
  908. ```bibtex
  909. @software{auto_deep_researcher_24x7,
  910. title={Deep Researcher Agent: Autonomous Deep Learning Experiment Framework},
  911. author={Xiangyue Zhang},
  912. year={2026},
  913. url={https://github.com/Xiangyue-Zhang/auto-deep-researcher-24x7}
  914. }
  915. ```
  916. ## Star History
  917. [![Star History Chart](https://api.star-history.com/svg?repos=Xiangyue-Zhang/auto-deep-researcher-24x7&type=Date&v=20260408-1)](https://www.star-history.com/#Xiangyue-Zhang/auto-deep-researcher-24x7&Date)
  918. ## License
  919. Apache 2.0 โ€” see [LICENSE](LICENSE).
  920. ---
  921. <p align="center">
  922. <strong><i>"Experiments run through the night. Results arrive at dawn."</i></strong>
  923. </p>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If itโ€™s helped, pass it on.