Skip to content
Back to search

ASI-Evolve

ASI-Evolve — Let AI Do the Research, You Keep the Insight

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/gair-nlp--asi-evolve/badge.svg)](/listings/gair-nlp--asi-evolve)

HTML

<a href="/listings/gair-nlp--asi-evolve"><img src="/api/listings/gair-nlp--asi-evolve/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pip install -r requirements.txt

Requirements: - Python 3.10+ - `bash` and `python3` on your system path - Any OpenAI-compatible API endpoint (GPT-4o, Claude, Gemini, local models via LiteLLM, etc.) - Optional: Weights & Biases for experiment tracking

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 314 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # ASI-Evolve — Let AI Do the Research, You Keep the Insight
  2. ![ASI-Evolve overview](assets/Overview.png)
  3. > **"What if you could run a tireless AI researcher on your hardest problem — one that reads the literature, designs experiments, runs them, and learns from every failure?"**
  4. That's ASI-Evolve. It is a general agentic framework that closes the loop between **knowledge → hypothesis → experiment → analysis** — and repeats it autonomously, round after round, until it finds something that works.
  5. We built it for AI research. But the loop doesn't care about domain.
  6. A financial analyst, a biomedical engineer, a climate scientist, or a game developer can all plug their own problem into ASI-Evolve and let it search for better solutions than any human has time to manually explore.
  7. **Quick try:** Install the Evolve Agent Skill under `skills/evolve` to have a lightweight first pass.
  8. <div align="center">
  9. [![Paper](https://img.shields.io/badge/Paper-ASI--Evolve-blue?style=flat&logo=googledocs&logoColor=white)](https://github.com/GAIR-NLP/ASI-Evolve/blob/main/assets/paper.pdf)
  10. [![arXiv](https://img.shields.io/badge/arXiv-2603.29640-b31b1b?style=flat&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2603.29640)
  11. </div>
  12. ---
  13. ## What ASI-Evolve Proved
  14. The paper validated ASI-Evolve on three hard AI problems — each requiring expensive compute, open-ended search, and multi-dimensional feedback:
  15. | Domain | What It Discovered | Gain |
  16. |---|---|---|
  17. | **Neural Architecture Design** | 105 SOTA linear attention architectures | **+0.97 pts** over DeltaNet (≈3× recent human gains) |
  18. | **Pretraining Data Curation** | Evolved pipeline that selects cleaner training data | **+3.96 pts** avg, **+18 pts** on MMLU |
  19. | **RL Algorithm Design** | Novel optimization mechanisms with mathematical innovations | **+12.5 pts** on AMC32 vs GRPO |
  20. | **Biomedical (Drug-Target Interaction)** | Stronger architecture for cold-start generalization | **+6.94 AUROC** |
  21. These are not toy benchmarks. These are real, frontier-level results — produced autonomously.
  22. ---
  23. ## The Core Loop (Works in Any Domain)
  24. | Step | Action |
  25. |---|---|
  26. | 1 | **LEARN** — retrieve relevant prior knowledge |
  27. | 2 | **DESIGN** — propose the next candidate program/idea |
  28. | 3 | **EXPERIMENT** — run it and collect metrics |
  29. | 4 | **ANALYZE** — turn results into reusable lessons |
  30. > Repeat the loop and improve each round.
  31. Three agents drive this loop:
  32. - **Researcher** — reads the database and cognition store, proposes the next candidate
  33. - **Engineer** — executes the candidate and collects structured metrics
  34. - **Analyzer** — distills outcomes into transferable lessons for future rounds
  35. Two memory systems keep the loop from going in circles:
  36. - **Cognition Store** — inject your domain knowledge, papers, or heuristics upfront so the AI doesn't start from zero
  37. - **Experiment Database** — every trial is stored with its motivation, code, result, and analysis; parent selection uses UCB1, greedy, random, or MAP-Elites island sampling
  38. ---
  39. ## Who Should Use This
  40. You don't need to be an AI researcher. You need:
  41. 1. **A problem where better code = better outcome** — optimization, algorithm design, pipeline tuning, simulation strategy
  42. 2. **An evaluation script** — something that takes a program and returns a score
  43. 3. **Some domain knowledge** — papers, rules of thumb, known good approaches
  44. If you have those three things, ASI-Evolve can search the space for you.
  45. **Examples across domains:**
  46. | Industry | Problem You Can Throw At It |
  47. |---|---|
  48. | 🧬 Biomedicine | Drug-target interaction models, protein folding strategies, clinical trial algorithms |
  49. | ⚡ ML Infra | Inference schedulers, KV-cache eviction policies, batching strategies, kernel tiling |
  50. | 🌍 Climate / Energy | Grid load-balancing heuristics, carbon footprint optimization pipelines |
  51. | 🎮 Game AI | Bot strategies, procedural level generation algorithms, reward shaping functions |
  52. | 🏭 Manufacturing | Quality control classifiers, scheduling heuristics, defect detection pipelines |
  53. | 🔬 Materials Science | Crystal structure search algorithms, synthesis route optimization |
  54. | 📦 Logistics | Routing algorithms, warehouse assignment policies, demand forecasting models |
  55. ---
  56. ## Walkthrough: Inference Throughput Optimization
  57. > **Scenario:** You're an ML infra engineer. Your LLM serving system uses a fixed continuous-batching scheduler. You want to automatically discover a better request-scheduling policy that maximizes throughput while keeping P99 latency under budget — without spending weeks hand-tuning heuristics.
  58. ### 1. Clone and install
  59. ```bash
  60. git clone https://github.com/GAIR-NLP/ASI-Evolve.git
  61. cd ASI-Evolve
  62. pip install -r requirements.txt
  63. ```
  64. ### 2. Create your experiment
  65. ```bash
  66. mkdir -p experiments/infer_scheduler/prompts
  67. ```
  68. ```
  69. experiments/infer_scheduler/
  70. ├── input.md ← problem description
  71. ├── config.yaml ← API key + overrides
  72. ├── initial_program ← baseline scheduler to evolve from
  73. ├── init_cognition.py ← inject your domain knowledge
  74. ├── evaluator.py ← benchmark the candidate scheduler
  75. └── eval.sh ← shell wrapper called each round
  76. ```
  77. ### 3. Describe the problem (`input.md`)
  78. ```markdown
  79. # LLM Serving Scheduler
  80. Write a Python function `schedule(queue, gpu_mem_free_gb)` that selects
  81. a batch of requests from `queue` (list of dicts with keys: seq_len, priority,
  82. wait_ms) given available GPU memory, and returns a list of selected request ids.
  83. Optimize for: tokens/sec throughput (primary), P99 latency ms (must stay < 500ms).
  84. ```
  85. ### 4. Baseline program (`initial_program`)
  86. ```python
  87. def schedule(queue, gpu_mem_free_gb):
  88. """FCFS baseline — just take requests in arrival order."""
  89. budget = int(gpu_mem_free_gb * 1024) # rough token budget
  90. selected, used = [], 0
  91. for req in sorted(queue, key=lambda r: r["wait_ms"], reverse=True):
  92. if used + req["seq_len"] <= budget:
  93. selected.append(req["id"])
  94. used += req["seq_len"]
  95. return selected
  96. ```
  97. ### 5. Seed the cognition store (`init_cognition.py`)
  98. ```python
  99. from cognition.store import CognitionStore
  100. store = CognitionStore(storage_dir="experiments/infer_scheduler/cognition_data")
  101. store.add([
  102. {
  103. "title": "Continuous Batching",
  104. "content": "Orca-style iteration-level scheduling avoids head-of-line blocking "
  105. "by preempting long requests and inserting shorter ones mid-flight.",
  106. },
  107. {
  108. "title": "Sequence Length Bucketing",
  109. "content": "Grouping requests by similar sequence length reduces padding waste "
  110. "and improves GPU utilization significantly.",
  111. },
  112. {
  113. "title": "Priority + Starvation",
  114. "content": "Pure priority scheduling causes starvation. Age-weighted priority "
  115. "(priority * log(1 + wait_ms)) balances latency and throughput.",
  116. },
  117. ])
  118. ```
  119. ```bash
  120. python experiments/infer_scheduler/init_cognition.py
  121. ```
  122. ### 6. Evaluator (`evaluator.py` + `eval.sh`)
  123. ```python
  124. import importlib.util, sys, json
  125. from benchmark import run_serving_simulation # your internal benchmark harness
  126. def load(path):
  127. spec = importlib.util.spec_from_file_location("candidate", path)
  128. mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)
  129. return mod.schedule
  130. if __name__ == "__main__":
  131. fn = load(sys.argv[1])
  132. result = run_serving_simulation(scheduler_fn=fn, trace="traces/sharegpt_1k.jsonl")
  133. # score = throughput; constraint violation penalizes automatically
  134. score = result["tokens_per_sec"] if result["p99_latency_ms"] < 500 else 0
  135. print(json.dumps({"score": score, "metrics": result}))
  136. ```
  137. ```bash
  138. # eval.sh
  139. #!/bin/bash
  140. python /path/to/experiments/infer_scheduler/evaluator.py "$1"
  141. ```
  142. ### 7. Run
  143. ```bash
  144. python main.py \
  145. --experiment infer_scheduler \
  146. --steps 40 \
  147. --sample-n 3 \
  148. --eval-script /path/to/experiments/infer_scheduler/eval.sh
  149. ```
  150. ASI-Evolve will iterate through scheduling policies — from simple heuristics to multi-factor priority functions with dynamic chunking — and write a structured lesson after every trial. After 40 rounds you have a ranked database of every policy tried, why each worked or failed, and the best-performing code ready to deploy.
  151. ---
  152. ## What ASI-Evolve Does That You Can't Do Manually
  153. | Manual Research | ASI-Evolve |
  154. |---|---|
  155. | Try 5–10 ideas per week | 50–200 candidates per run |
  156. | Knowledge stays in one person's head | Every insight written to the database |
  157. | Cold start on each new hypothesis | Cognition store primes every round |
  158. | Hard to know why something worked | Analyzer explains every outcome |
  159. | Results hard to reproduce | Full experiment tree stored on disk |
  160. ---
  161. ## Repository Layout
  162. ```
  163. ASI-Evolve/
  164. ├── main.py ← entry point
  165. ├── config.yaml ← global defaults
  166. ├── pipeline/ ← Researcher, Engineer, Analyzer agents
  167. ├── cognition/ ← cognition store (embedding + FAISS)
  168. ├── database/ ← experiment database (nodes + sampling)
  169. ├── utils/
  170. └── experiments/
  171. ├── circle_packing_demo/ ← included runnable demo
  172. └── best/circle_packing/ ← top programs from our ablation runs
  173. ```
  174. > **Skill vs. repository:** When using the Skill, prior experiment traces and instructions accumulate in the model context, and the assistant is not constrained to the same fixed control flow as the official pipeline. End-to-end search quality and throughput are usually lower than when you run this repository directly. Use the Skill for a quick try-out; for very complex or tightly controlled work, prefer cloning and running from here.
  175. ---
  176. ## Installation
  177. ```bash
  178. pip install -r requirements.txt
  179. ```
  180. Requirements:
  181. - Python 3.10+
  182. - `bash` and `python3` on your system path
  183. - Any OpenAI-compatible API endpoint (GPT-4o, Claude, Gemini, local models via LiteLLM, etc.)
  184. - Optional: Weights & Biases for experiment tracking
  185. ---
  186. ## Quick Start (Circle-Packing Demo)
  187. The included demo evolves a program to pack 26 circles in a unit square — a clean benchmark used in our ablation studies.
  188. ```bash
  189. # Initialize the cognition store
  190. python experiments/circle_packing_demo/init_cognition.py
  191. # Run 10 evolution steps
  192. python main.py \
  193. --experiment circle_packing_demo \
  194. --steps 10 \
  195. --sample-n 3 \
  196. --eval-script /path/to/experiments/circle_packing_demo/eval.sh
  197. ```
  198. ASI-Evolve reaches SOTA-level circle-packing results in as few as **17 rounds**.
  199. ---
  200. ## Configuration Reference
  201. Configuration merges in this order (later overrides earlier):
  202. 1. `config.yaml` at repository root
  203. 2. `experiments/<name>/config.yaml`
  204. 3. An explicit file passed with `--config`
  205. Key settings:
  206. | Key | What It Controls |
  207. |---|---|
  208. | `api.model` | The LLM driving all agents |
  209. | `pipeline.sample_n` | How many historical nodes to show the Researcher each round |
  210. | `pipeline.parallel.num_workers` | Parallel evolution workers (2–4 for production) |
  211. | `database.sampling.algorithm` | `ucb1` / `greedy` / `random` / `island` |
  212. | `cognition.retrieval.top_k` | How many cognition items to retrieve per round |
  213. ---
  214. ## Citation
  215. If you use ASI-Evolve in your work, please cite:
  216. ```bibtex
  217. @misc{asi_evolve_2026,
  218. title = {ASI-Evolve: AI Accelerates AI},
  219. author = {Xu, Weixian and Mi, Tiantian and Liu, Yixiu and Nan, Yang and Zhou, Zhimeng and Ye, Lyumanshan and Zhang, Lin and Qiao, Yu and Liu, Pengfei},
  220. year = {2026},
  221. note = {SJTU / SII / GAIR. https://github.com/GAIR-NLP/ASI-Evolve}
  222. }
  223. ```
  224. ---
  225. <div align="center">
  226. **ASI-Evolve is open-source and ready for your domain.**
  227. Fork it. Point it at your problem. Let it run.
  228. [📄 Paper](https://github.com/GAIR-NLP/ASI-Evolve/blob/main/assets/paper.pdf) · [🐛 Issues](https://github.com/GAIR-NLP/ASI-Evolve/issues) · [💬 Discussions](https://github.com/GAIR-NLP/ASI-Evolve/discussions)
  229. </div>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.