Skip to content
Back to search

Bilevel-Autoresearch

Use Autoresearch to research Autoresearch.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/edwardoptimization--bilevel-autoresearch/badge.svg)](/listings/edwardoptimization--bilevel-autoresearch)

HTML

<a href="/listings/edwardoptimization--bilevel-autoresearch"><img src="/api/listings/edwardoptimization--bilevel-autoresearch/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**Prerequisites:** Python 3.10+, an LLM API key (DeepSeek, OpenAI, or Anthropic). Training demo also needs a GPU and [Karpathy's autoresearch](https://github.com/karpathy/autoresearch) cloned.

pip install -e .
cp .env.example .env  # fill in your API keys

# Article optimization — lightweight demo (no GPU needed)
python -m domains.article_opt.cli --provider openai once --article article1  # smoke test
python -m domains.article_opt.cli run --articles article1 --max-inner 5 --max-outer 4   # full bilevel

# Training optimization — reproduce the paper result (needs GPU)
git clone https://github.com/karpathy/autoresearch.git ~/karpathy_autoresearch
python -m domains.train_opt.cli --provider deepseek bilevel --inner-budget 5 --outer-cycles 2

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 176 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Bilevel Autoresearch
  2. **Autoresearch that researches itself.** An outer loop autonomously discovers new mechanisms for the inner loop — not by tuning prompts, but by inventing and code-generating structural changes to the search process.
  3. **Paper:** [Bilevel Autoresearch: Meta-Autoresearching Itself](https://arxiv.org/abs/2603.23420) (arXiv:2603.23420 | [AISC 2026](https://aixiv.science/abs/aixiv.260323.000006))
  4. [![arXiv](https://img.shields.io/badge/arXiv-2603.23420-b31b1b.svg)](https://arxiv.org/abs/2603.23420)
  5. [![AISC2026](https://img.shields.io/badge/AISC2026-aixiv.260323.000006-blue.svg)](https://aixiv.science/abs/aixiv.260323.000006)
  6. [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://python.org)
  7. [![CI](https://github.com/EdwardOptimization/Bilevel-Autoresearch/actions/workflows/ci.yml/badge.svg)](https://github.com/EdwardOptimization/Bilevel-Autoresearch/actions)
  8. [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
  9. ---
  10. ## Core Idea
  11. ```
  12. Inner loop: Optimizes the task output (propose → execute → evaluate → keep/discard)
  13. Outer loop: Optimizes how the inner loop works (analyze trace → modify mechanisms → re-run)
  14. ```
  15. Both levels use the same pattern: **propose × evaluate × iterate**. The inner loop improves the task. The outer loop improves the inner loop — not by tuning prompts, but by structurally changing how it searches.
  16. **Autoresearch anything with a measurable objective.** The same bilevel principle applies to project management, multi-agent coordination, experiment scheduling, and the research process itself.
  17. **Why 3 layers in the experiment?** The design is bilevel (inner + outer), but in practice we split the outer loop into two responsibilities for cleaner separation of concerns:
  18. | Layer | Role | What it changes |
  19. |-------|------|-----------------|
  20. | **Level 1** | Inner loop | Task output (hyperparameters) |
  21. | **Level 1.5** | Outer loop — config | Runtime search parameters (freeze/unfreeze, strategy shift) |
  22. | **Level 2** | Outer loop — mechanism | Inner loop code structure (generate new Python mechanisms) |
  23. Level 1.5 handles tactical adjustments ("stop searching WEIGHT_DECAY, focus on LR"). Level 2 handles strategic innovation ("invent a Tabu Search mechanism to prevent repetitive proposals"). Separating them lets Level 2 focus purely on mechanism discovery without being distracted by parameter tuning.
  24. ### The Ultimate Goal: Recursive Bootstrapping
  25. Not only does Level 2 generate mechanisms to accelerate Level 1, but the architecture inherently supports **recursive mechanism feedback**.
  26. If Level 2 discovers that a mechanism (e.g., parallel multi-agent debate, persistent memory, or Tabu search) significantly improves Level 1's search efficiency, **this same mechanism can be abstracted and applied inversely to Level 2 itself.** The system learns how to learn, and then applies those lessons to its own meta-learning process. This moves the framework beyond automated optimization and towards a truly self-improving digital ecosystem.
  27. ## Key Result: Controlled Ablation Experiment
  28. On Karpathy's GPT pretraining benchmark (val_bpb, 300s budget, RTX 5090), we ran a controlled ablation with **4 groups × 3 independent repeats × 30 iterations**, using the **same LLM (DeepSeek)** for all levels:
  29. | Group | What it does | Mean Δval_bpb | vs Group A |
  30. |-------|-------------|--------------|------------|
  31. | **A** — Level 1 | Standard autoresearch (propose → train → keep/discard) | -0.009 ± 0.002 | 1× |
  32. | **B** — Level 1 + 1.5 | + Outer loop adjusts search config | -0.006 ± 0.006 | 0.7× |
  33. | **C** — Level 1 + 1.5 + 2 | + Outer loop generates new mechanisms as code | **-0.045 ± 0.030** | **5×** |
  34. | **D** — Level 1 + 2 | + Mechanisms without config adjustment | -0.034 ± 0.031 | 3.8× |
  35. *Baseline val_bpb ≈ 1.10. More negative = better. 3 independent repeats × 30 iterations each.* The outer loop autonomously generated Python code for new search mechanisms, dynamically loaded them via `importlib`, and injected them into the running inner loop.
  36. ### Mechanisms discovered autonomously by Level 2
  37. Each repeat independently discovered different mechanisms from different domains — no human specified which domains to explore:
  38. | Mechanism | Domain | What It Does |
  39. |-----------|--------|-------------|
  40. | **Tabu Search Manager** | Combinatorial optimization | Prevents revisiting recently explored parameter regions |
  41. | **Multi-Scale Bandit** | Online learning / MAB | Balances exploration and exploitation across parameters |
  42. | **Orthogonal Exploration** | Design of experiments | Forces search across orthogonal parameter dimensions |
  43. | **GP Regressor** (reverted) | Bayesian optimization | Surrogate model for val_bpb prediction (sklearn not installed) |
  44. ### Why Level 2 wins
  45. Group A (no outer loop) follows a **deterministic search path** — it tries WEIGHT_DECAY, then WINDOW_PATTERN, then gets stuck repeating the same proposals for 20+ iterations. Level 2's mechanisms (Tabu Search, Orthogonal Exploration) break this loop and guide the LLM to discover that **reducing TOTAL_BATCH_SIZE** dramatically improves val_bpb — a direction A and B never explored.
  46. ```python
  47. # Agent-generated mechanism: Tabu Search prevents the LLM from repeating failed proposals
  48. class TabuSearchManager:
  49. def __init__(self, max_tabu_size=10):
  50. self._tabu_list = [] # recently visited parameter regions
  51. def is_tabu(self, changes: dict) -> bool:
  52. """Check if proposed changes are too similar to recent attempts."""
  53. for tabu_entry in self._tabu_list:
  54. if self._similarity(changes, tabu_entry) > 0.8:
  55. return True # block this proposal, force the LLM to try something new
  56. return False
  57. ```
  58. Full ablation report: [`experiments/ablations/paper_ablation/run2_results/REPORT.md`](experiments/ablations/paper_ablation/run2_results/REPORT.md)
  59. ---
  60. ## Quick Start
  61. **Prerequisites:** Python 3.10+, an LLM API key (DeepSeek, OpenAI, or Anthropic). Training demo also needs a GPU and [Karpathy's autoresearch](https://github.com/karpathy/autoresearch) cloned.
  62. ```bash
  63. pip install -e .
  64. cp .env.example .env # fill in your API keys
  65. # Article optimization — lightweight demo (no GPU needed)
  66. python -m domains.article_opt.cli --provider openai once --article article1 # smoke test
  67. python -m domains.article_opt.cli run --articles article1 --max-inner 5 --max-outer 4 # full bilevel
  68. # Training optimization — reproduce the paper result (needs GPU)
  69. git clone https://github.com/karpathy/autoresearch.git ~/karpathy_autoresearch
  70. python -m domains.train_opt.cli --provider deepseek bilevel --inner-budget 5 --outer-cycles 2
  71. ```
  72. ---
  73. ## Architecture
  74. ```
  75. core/ # Bilevel framework (shared, domain-agnostic)
  76. ├── inner_loop.py # InnerLoopController
  77. ├── state.py # State management with isolation boundaries
  78. └── llm_client.py # Multi-provider LLM client
  79. domains/
  80. ├── article_opt/ # Article optimization demo
  81. │ ├── cli.py # Entry point: python -m domains.article_opt.cli
  82. │ ├── runner.py # InnerRunner + inject_stage()
  83. │ ├── outer.py # OuterAnalyzer + OuterLoopController
  84. │ ├── mechanism_research.py # Level 2: generate new pipeline stages as code
  85. │ ├── pipeline/ # Article stages (A→E)
  86. │ ├── evaluator/ # Article rubric evaluator
  87. │ └── reference_frameworks.md # Optimization strategy reference doc
  88. └── train_opt/ # Training demo: GPT pretraining optimization
  89. ├── runner.py # Inner loop with 12 agent-invented mechanisms
  90. ├── outer.py # Outer loop: trace analysis → config updates
  91. └── config.py # SearchConfig (outer loop's control surface)
  92. articles/ # Article demo input data (root-level, referenced by path)
  93. experiments/ # Ablation results and experiment records
  94. paper/ # LaTeX paper (submitted to AISC2026)
  95. tests/ # 110 unit tests
  96. ```
  97. ---
  98. ## How It Works
  99. The **inner loop** runs a task repeatedly, learning from each run. The **outer loop** analyzes the inner loop's trace and modifies its configuration. **Level 2** goes further — an autonomous agent reads the inner loop's code, identifies bottlenecks, and writes new Python code to fix them.
  100. Two demo domains are included:
  101. **Article optimization** — 5-stage pipeline (Analysis → Hypotheses → Planning → Assessment → Revision) evaluated against a rubric. Outer loop injects prompt overrides. Level 2 generates new pipeline stages via `importlib`.
  102. **Training optimization** — LLM proposes hyperparameter changes to Karpathy's `train.py`, trains for 5 minutes, measures `val_bpb`. Outer loop freezes ineffective params and shifts strategy. Level 2 generates new Python mechanisms (Tabu Search, Multi-Scale Bandit, Orthogonal Exploration) and injects them via `importlib`. Validated with 3×3 ablation on RTX 5090 — Level 2 achieves 5× improvement over baseline autoresearch.
  103. ---
  104. ## Related Work
  105. | Project | Contribution | Link |
  106. |---------|-------------|------|
  107. | **AutoResearch** (Karpathy) | The single-track autoresearch loop | [GitHub](https://github.com/karpathy/autoresearch) |
  108. | **AutoResearchClaw** (AIMing Lab) | Multi-batch parallel search | [GitHub](https://github.com/aiming-lab/AutoResearchClaw) |
  109. | **EvoScientist** | Persistent experience memory | [GitHub](https://github.com/EvoScientist/EvoScientist) |
  110. Each of the above is a **human-designed** mechanism change. This project asks: can an outer loop discover such improvements **autonomously**?
  111. ---
  112. ## Contributing
  113. ```bash
  114. git clone https://github.com/EdwardOptimization/Bilevel-Autoresearch.git
  115. cd Bilevel-Autoresearch && pip install -e ".[dev]"
  116. python -m pytest tests/ -v # run tests (offline, no API key)
  117. ruff check core/ domains/ tests/ # lint
  118. ```
  119. **Add a new domain:** create `domains/your_domain/` with `runner.py`, `outer.py`, `config.py`, `cli.py`. See [`domains/README.md`](domains/README.md) and `train_opt/` as a template. Domains are self-contained — import `core.llm_client` for LLM access, but implement your own runner and outer loop.
  120. **Add a new LLM provider:** add an entry to `PROVIDERS` in `core/llm_client.py`. All OpenAI-compatible providers work with `native_sdk: False`.
  121. **Key constraint:** the evaluator must NEVER receive lesson memory. Lessons influence proposals, never judgments.
  122. ## License
  123. MIT — see [LICENSE](LICENSE)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.