Skip to content
Back to search

Recuris

Recursive Experiential–Working Memory Evolution for Long-Horizon Agent Harnesses

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/gen-verse--recuris/badge.svg)](/listings/gen-verse--recuris)

HTML

<a href="/listings/gen-verse--recuris"><img src="/api/listings/gen-verse--recuris/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

This repository provides the code for running Recuris on τ²-Bench, SkillFlow and Terminal-Bench 2.1, the Skill Memory packages produced by the evolution loop, and the frozen evaluation splits.

📦 Install Packages

Python 3.12 and `git`. SkillFlow and Terminal-Bench 2.1 also need Docker with the Compose V2 plugin (`docker compose version` must work; harbor shells out to it for every task).

git clone https://github.com/Gen-Verse/Recuris.git recuris
cd recuris

uv sync --extra all          # or: pip install -e ".[all]"

⚙️ Setup Environment Variables

Put your endpoint in a `.env` file at the repository root, or export it:

OPENAI_API_KEY=...
OPENAI_BASE_URL=...

Any OpenAI-compatible endpoint works. This is needed even when the agent itself is an open-source model, because τ²-Bench scores every episode with an LLM user simulator and an LLM assertion judge, and both stay pinned to a reference model.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 446 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <a name="readme-top"></a>
  2. <p align="center">
  3. <picture>
  4. <source media="(prefers-color-scheme: dark)" srcset="assets/recuris-logo-dark.png">
  5. <img src="assets/recuris-logo.png" alt="Recuris" width="460">
  6. </picture>
  7. </p>
  8. <h3 align="center">
  9. Recursive Experiential–Working Memory Evolution for Long-Horizon Agent Harnesses
  10. </h3>
  11. <p align="center">
  12. <a href="https://arxiv.org/abs/2608.24876"><img src="https://img.shields.io/badge/arxiv-2608.24876-B31B1B.svg?logo=arxiv" alt="Paper"></a>
  13. <a href="https://huggingface.co/papers/2608.24876"><img src="https://img.shields.io/badge/Huggingface-DailyPaper-FFD21E.svg?logo=huggingface" alt="Hugging Face Daily Paper"></a>
  14. <a href="https://x.com/lingyang_pu/status/2092432103954841925"><img src="https://img.shields.io/badge/Coverage-Recuris-2176BC.svg?logo=x" alt="X"></a>
  15. </p>
  16. <p align="center">
  17. <a href="https://paperswithcode.co/benchmark/tau2-bench-retail?task=agents"><img src="https://paperswithcode.co/api/v1/papers/2608.24876/badge.svg?eval=28331" alt="SOTA on τ²-Bench (Retail)"></a>
  18. <a href="https://paperswithcode.co/benchmark/tau2-bench-airline?task=agents"><img src="https://paperswithcode.co/api/v1/papers/2608.24876/badge.svg?eval=28814" alt="SOTA on τ²-Bench (Airline)"></a>
  19. </p>
  20. <p align="center">
  21. <b>National University of Singapore</b> &nbsp;·&nbsp; <b>Stanford University</b> &nbsp;·&nbsp; <b>University of Oxford</b> &nbsp;·&nbsp; <b>Princeton University</b>
  22. </p>
  23. ---
  24. <p align="center">
  25. <img src="assets/results.png" width="1000">
  26. </p>
  27. ## 💡 Introduction
  28. **Recuris** is a recursive self-improvement framework that **improves a
  29. long-horizon agent by evolving its memory instead of its weights or its
  30. prompt**. A frozen agent is paired with a **Skill Memory** `M = (E, W, ρ, C)`.
  31. A meta-agent reads structured execution traces, locates each failure in one
  32. component of that memory, and patches only that component. A deterministic
  33. validation gate then decides, on paired held-out evidence, whether the patch
  34. survives. Recuris has the following key features:
  35. - **State-grounded memory use.** Working memory drives skill invocation, so
  36. retrieval is conditioned on verified task state rather than on a chat history
  37. that grows until the state is buried.
  38. - **Targeted memory evolution.** Structured trajectories `(w_t, E_t, a_t, o_t)`
  39. locate a failure in a specific component, instead of nudging a monolithic
  40. prompt from outcomes alone.
  41. - **Bounded by a validation gate.** Candidates are admitted by paired held-out
  42. arithmetic and nothing else. No model votes on its own patch.
  43. - **Training-free and model-agnostic.** The downstream agent stays frozen, and a
  44. memory evolved on one model transfers to others unchanged.
  45. Overall, Recuris achieves **higher task success**, **larger gains on longer
  46. horizons**, and **substantially fewer long-horizon failures**, on both frontier
  47. and open-source agents.
  48. <p align="center">
  49. <img src="assets/motivation.png" width="1000">
  50. </p>
  51. ## 🔔 News
  52. - **[2026-08]** 🎉 Initial release: evaluation and evolution code, the evolved
  53. Skill Memory packages, and the frozen evaluation splits.
  54. ## 📊 Results
  55. Task success (`avg@4`, %), each model run with the benchmark's own reference
  56. agent alone and with that same agent plus Recuris. **Bold** marks the better of
  57. each pair, the subscript is Δ, † marks a paired task-clustered bootstrap 95% CI
  58. excluding zero, and `n/a` means the model was not run on that benchmark.
  59. <table>
  60. <thead>
  61. <tr>
  62. <th align="left" rowspan="2">Model</th>
  63. <th align="center" colspan="3"><i>Cross-task evolution</i></th>
  64. <th align="center"><i>Within-task adaptation</i></th>
  65. </tr>
  66. <tr>
  67. <th align="right">τ²-Retail</th>
  68. <th align="right">τ²-Airline</th>
  69. <th align="right">SkillFlow</th>
  70. <th align="right">Terminal-Bench 2.1</th>
  71. </tr>
  72. </thead>
  73. <tbody>
  74. <tr><td colspan="5"><i>Open-source models</i></td></tr>
  75. <tr><td align="left">Granite-4.1-3B</td>
  76. <td align="right">9.7</td><td align="right">34.3</td>
  77. <td align="right"><b>0.3</b></td><td align="right">0.6</td></tr>
  78. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  79. <td align="right"><b>23.0</b> <sub>+13.4†</sub></td>
  80. <td align="right"><b>39.8</b> <sub>+5.5</sub></td>
  81. <td align="right">0.0 <sub>−0.3</sub></td>
  82. <td align="right"><b>3.1</b> <sub>+2.5</sub></td></tr>
  83. <tr><td align="left">Qwen3.5-4B</td>
  84. <td align="right">68.0</td><td align="right">75.3</td>
  85. <td align="right">6.0</td><td align="right">10.1</td></tr>
  86. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  87. <td align="right"><b>68.3</b> <sub>+0.3</sub></td>
  88. <td align="right"><b>79.0</b> <sub>+3.8</sub></td>
  89. <td align="right"><b>7.1</b> <sub>+1.1</sub></td>
  90. <td align="right"><b>13.0</b> <sub>+2.9</sub></td></tr>
  91. <tr><td align="left">Qwen3.5-9B</td>
  92. <td align="right">77.6</td><td align="right">75.5</td>
  93. <td align="right">15.1</td><td align="right">17.4</td></tr>
  94. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  95. <td align="right"><b>79.6</b> <sub>+2.0</sub></td>
  96. <td align="right"><b>78.4</b> <sub>+2.9</sub></td>
  97. <td align="right"><b>18.4</b> <sub>+3.4</sub></td>
  98. <td align="right"><b>20.5</b> <sub>+3.1</sub></td></tr>
  99. <tr><td align="left">GPT-OSS-20B</td>
  100. <td align="right">50.6</td><td align="right">54.8</td>
  101. <td align="right">7.8</td><td align="right">3.9</td></tr>
  102. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  103. <td align="right"><b>60.8</b> <sub>+10.2†</sub></td>
  104. <td align="right"><b>59.3</b> <sub>+4.5†</sub></td>
  105. <td align="right"><b>10.4</b> <sub>+2.6†</sub></td>
  106. <td align="right"><b>6.7</b> <sub>+2.8</sub></td></tr>
  107. <tr><td align="left">Qwen3.6-27B</td>
  108. <td align="right">62.8</td><td align="right">79.0</td>
  109. <td align="right">42.2</td><td align="right">38.8</td></tr>
  110. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  111. <td align="right"><b>71.2</b> <sub>+8.3†</sub></td>
  112. <td align="right"><b>80.0</b> <sub>+1.0</sub></td>
  113. <td align="right"><b>58.7</b> <sub>+16.6†</sub></td>
  114. <td align="right"><b>42.1</b> <sub>+3.3</sub></td></tr>
  115. <tr><td align="left">Qwen3.6-35B</td>
  116. <td align="right">78.2</td><td align="right">80.3</td>
  117. <td align="right">35.3</td><td align="right">33.1</td></tr>
  118. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  119. <td align="right"><b>78.5</b> <sub>+0.3</sub></td>
  120. <td align="right"><b>81.5</b> <sub>+1.3</sub></td>
  121. <td align="right"><b>48.8</b> <sub>+13.5†</sub></td>
  122. <td align="right"><b>36.4</b> <sub>+3.3</sub></td></tr>
  123. <tr><td colspan="5"><i>Frontier models</i></td></tr>
  124. <tr><td align="left">Gemini 3.7 Flash</td>
  125. <td align="right">73.5</td><td align="right"><b>86.5</b></td>
  126. <td align="right">n/a</td><td align="right">79.8</td></tr>
  127. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  128. <td align="right"><b>78.3</b> <sub>+4.8</sub></td>
  129. <td align="right">85.0 <sub>−1.5</sub></td>
  130. <td align="right">n/a</td>
  131. <td align="right"><b>82.4</b> <sub>+2.6</sub></td></tr>
  132. <tr><td align="left">GPT-5.6 Sol</td>
  133. <td align="right">58.3</td><td align="right">79.0</td>
  134. <td align="right">n/a</td><td align="right">83.2</td></tr>
  135. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  136. <td align="right"><b>76.1</b> <sub>+17.8†</sub></td>
  137. <td align="right"><b>86.0</b> <sub>+7.0†</sub></td>
  138. <td align="right">n/a</td>
  139. <td align="right"><b>86.4</b> <sub>+3.2</sub></td></tr>
  140. <tr><td align="left">Claude Opus 5</td>
  141. <td align="right">72.4</td><td align="right">89.5</td>
  142. <td align="right">n/a</td><td align="right">84.6</td></tr>
  143. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  144. <td align="right"><b>87.9</b> <sub>+15.6†</sub></td>
  145. <td align="right"><b>90.5</b> <sub>+1.0</sub></td>
  146. <td align="right">n/a</td>
  147. <td align="right"><b>88.4</b> <sub>+3.8</sub></td></tr>
  148. <tr><td align="left">Doubao-2.0-Pro <i>(deployment)</i></td>
  149. <td align="right">58.1</td><td align="right">75.5</td>
  150. <td align="right">34.6</td><td align="right">46.1</td></tr>
  151. <tr><td align="left">&nbsp;&nbsp;<b>+ Recuris</b></td>
  152. <td align="right"><b>81.4</b> <sub>+23.3†</sub></td>
  153. <td align="right"><b>80.5</b> <sub>+5.0</sub></td>
  154. <td align="right"><b>51.4</b> <sub>+16.8†</sub></td>
  155. <td align="right"><b>48.9</b> <sub>+2.9</sub></td></tr>
  156. </tbody>
  157. </table>
  158. Recuris improves task success in **35 of the 37** completed model and benchmark
  159. pairs, from a 3B open-source agent up to the strongest frontier models. The
  160. largest gains reach **+23.3** on τ²-Retail and **+16.8** on SkillFlow. Gains
  161. grow with the interaction horizon, reaching **+32.2** on the longest tasks, and
  162. common long-horizon failure modes drop by up to **80%**.
  163. ## 🛠️ Getting Started
  164. This repository provides the code for running Recuris on τ²-Bench, SkillFlow and
  165. Terminal-Bench 2.1, the Skill Memory packages produced by the evolution loop,
  166. and the frozen evaluation splits.
  167. ### 📦 Install Packages
  168. Python 3.12 and `git`. SkillFlow and Terminal-Bench 2.1 also need Docker
  169. with the Compose V2 plugin (`docker compose version` must work; harbor
  170. shells out to it for every task).
  171. ```bash
  172. git clone https://github.com/Gen-Verse/Recuris.git recuris
  173. cd recuris
  174. uv sync --extra all # or: pip install -e ".[all]"
  175. ```
  176. ### ⚙️ Setup Environment Variables
  177. Put your endpoint in a `.env` file at the repository root, or export it:
  178. ```bash
  179. OPENAI_API_KEY=...
  180. OPENAI_BASE_URL=...
  181. ```
  182. Any OpenAI-compatible endpoint works. This is needed even when the agent itself
  183. is an open-source model, because τ²-Bench scores every episode with an LLM user
  184. simulator and an LLM assertion judge, and both stay pinned to a reference model.
  185. ## 🚀 Quick Start
  186. Each benchmark is run twice, once with a Skill Memory loaded and once without.
  187. The two runs differ only in the flags shown below. Both are needed, because the
  188. number that matters is the difference between them.
  189. ### 🔹 **τ²-Bench (retail and airline)**
  190. Set up the benchmark:
  191. ```bash
  192. bash third_party/tau2/setup.sh
  193. uv pip install -e external/tau2-bench
  194. recuris check-data --benchmark tau2
  195. ```
  196. Install tau2 after `uv sync`, not before: `uv sync` resolves the environment to
  197. exactly what `pyproject.toml` declares, so running it again removes anything
  198. added with `uv pip install`.
  199. Serve an open-source model locally. τ²-Bench drives the agent through tool
  200. calls, so the two tool-calling flags are required, not optional: without them
  201. vLLM rejects every request and every episode ends ungraded.
  202. ```bash
  203. vllm serve <model-id> --port 8000 --served-model-name qwen3.6-27b \
  204. --enable-auto-tool-choice --tool-call-parser hermes
  205. ```
  206. `hermes` is the parser for Qwen; other families need their own (see vLLM's
  207. tool-calling docs). A frontier model served by a provider needs none of this.
  208. Now point the agent at it:
  209. ```bash
  210. export TAU2_GATE_TERM=1 TAU2_GATE_TERM_WM=1 TAU2_STATUS_BOARD=1
  211. # open-source example
  212. export MODEL=openai/qwen3.6-27b
  213. export ARGS='{"api_base":"http://127.0.0.1:8000/v1","api_key":"dummy","temperature":0.0,"timeout":360,"num_retries":2}'
  214. # frontier example
  215. # export MODEL=openai/<provider-model>
  216. # export ARGS='{"api_base":"'"$OPENAI_BASE_URL"'","api_key":"'"$OPENAI_API_KEY"'","temperature":0.0,"timeout":360,"num_retries":2,"reasoning_effort":"high","allowed_openai_params":["reasoning_effort"]}'
  217. ```
  218. Run both configurations and compare them:
  219. ```bash
  220. # with Skill Memory
  221. recuris tau2 --domain retail --agent recuris_agent --skill-memory tau2_retail \
  222. --open-downstream --agent-llm "$MODEL" --agent-llm-args "$ARGS" \
  223. --num-trials 4 --max-concurrency 4 --save-to retail_skill
  224. # without
  225. recuris tau2 --domain retail --agent llm_agent \
  226. --open-downstream --agent-llm "$MODEL" --agent-llm-args "$ARGS" \
  227. --num-trials 4 --max-concurrency 4 --save-to retail_bare
  228. recuris compare --a retail_skill --b retail_bare
  229. ```
  230. #### Notes:
  231. * **`--domain`** is `retail` or `airline`. For airline, use
  232. `--skill-memory tau2_airline`.
  233. * **`--agent-llm-args`** must be identical in both runs. It is validated rather
  234. than merged, so an unknown key raises an error instead of being dropped
  235. silently.
  236. * Switching models means changing `$MODEL` and `$ARGS`. Nothing else changes.
  237. Some servers need extras, for example
  238. `"extra_body":{"chat_template_kwargs":{"enable_thinking":false}}` for Qwen.
  239. ### 🔹 **SkillFlow**
  240. Set up the benchmark and build the task images once:
  241. ```bash
  242. pip install huggingface_hub
  243. bash third_party/skillflow/setup.sh
  244. ./external/SkillFlow/docker/harbor-cli-base/build.sh
  245. python external/SkillFlow/utils/prebuild_task_images.py \
  246. --tasks-root external/SkillFlow/test_tasks
  247. ```
  248. Generate the configs for both runs, then execute them:
  249. ```bash
  250. export MODEL=openai/qwen3.6-27b
  251. export BASE=http://127.0.0.1:8000/v1
  252. recuris skillflow render-configs --arm bare \
  253. --model "$MODEL" --base-url "$BASE" --out configs/skillflow/generated
  254. recuris skillflow render-configs --arm skill --routing default \
  255. --model "$MODEL" --base-url "$BASE" \
  256. --skill-memory skillflow --out configs/skillflow/generated
  257. for cfg in configs/skillflow/generated/bare_*.yaml; do harbor run -c "$cfg" --yes; done
  258. for cfg in configs/skillflow/generated/skill_*.yaml; do harbor run -c "$cfg" --yes; done
  259. recuris skillflow score --bare jobs/bare --skill jobs/skill
  260. ```
  261. #### Notes:
  262. * Run the jobs **one at a time**. Concurrent harbor jobs exhaust the Docker IPv4
  263. address pool, and the resulting failure looks like something else entirely.
  264. * Configs are generated rather than committed, so the two runs cannot drift
  265. apart and no credential is ever written to disk.
  266. * **`--routing default`** is model-agnostic and is the right choice for new
  267. work. `--routing frozen_insample` reproduces our reported numbers and applies
  268. six per-family overrides that were chosen in-sample.
  269. ### 🔹 **Terminal-Bench 2.1 (test-time adaptation)**
  270. ```bash
  271. bash third_party/tb21/setup.sh
  272. recuris check-data --benchmark tb21
  273. ```
  274. On this benchmark a task may be attempted several times in a row, and it stops
  275. as soon as one attempt succeeds. `--rounds` sets how many attempts each task
  276. gets. There are three configurations:
  277. | Configuration | What the agent carries | After a failed attempt |
  278. |---|---|---|
  279. | `bare` | nothing, the stock agent | nothing changes, the next attempt starts over |
  280. | `m0` | a fixed Skill Memory, the seed package | nothing changes, the next attempt gets the same package |
  281. | `tta` | the same package, as a per-task copy | the meta-agent reads the failed trajectory and writes a new card into that copy, which the next attempt carries |
  282. ```bash
  283. # smoke test: one task, one attempt
  284. recuris tta run --taskset splits/tb21/tta_taskset_v3.json \
  285. --run-id smoke --arm m0 --limit 1 --rounds 1
  286. # all three configurations, four attempts each
  287. for cfg in bare m0 tta; do
  288. recuris tta run --taskset splits/tb21/tta_taskset_v3.json \
  289. --run-id demo --arm "$cfg" --rounds 4 --concurrency 3
  290. done
  291. ```
  292. #### Notes:
  293. * Give all three configurations the same `--rounds`. Comparing `tta` at four
  294. attempts against `bare` at one mostly measures the extra attempts rather than
  295. adaptation.
  296. * `m0` against `bare` isolates the value of having a Skill Memory at all. `tta`
  297. against `m0` isolates the value of updating it between attempts, since both
  298. carry a package and both get the same number of attempts.
  299. * At four attempts, that second comparison is worth **+2.3** points, 60.9%
  300. against 58.6%, which is not significant at this sample size. We report it
  301. that way rather than as a headline number.
  302. ### 🔹 **Evolving a Skill Memory**
  303. This is the recursive loop. A meta-agent, the **upstream** model, reads failed
  304. trajectories from the agent being improved, the **downstream** model. It patches
  305. one component of the memory, and a gate admits the patch only on paired held-out
  306. evidence.
  307. ```bash
  308. npm install -g @anthropic-ai/claude-code
  309. ```
  310. ```bash
  311. RECURIS_META_MODEL=... # the upstream meta-agent's model
  312. RECURIS_META_BASE_URL=...
  313. RECURIS_META_API_KEY=...
  314. ```
  315. ```bash
  316. # one scoped session, zero simulations: checks the plumbing first
  317. recuris metaagent qualify --run-id qsmoke --proxy-port 4047
  318. recuris metaagent run --domain retail --run-id retail_v1 \
  319. --splits splits/tau2/retail_from0_v1_k4.json \
  320. --rounds 4 --k 4 --arm autonomous --base neutral \
  321. --round-gate progressive --power-gate warn --reg-cap 1 \
  322. --meta-workflow hierarchical --diagnosis-workers 3 \
  323. --max-concurrency 6 --max-sims 1400 --proxy-port 4047
  324. ```
  325. To evolve a memory **for an open-source downstream model**, unfreeze the worker
  326. only. The user simulator stays pinned, so rounds remain comparable:
  327. ```bash
  328. recuris metaagent run --domain retail --run-id retail_gptoss_v1 \
  329. --splits splits/tau2/retail_from0_v1_k4.json \
  330. --rounds 4 --k 4 --arm autonomous --base neutral \
  331. --open-worker --worker-model openai/gpt-oss-20b \
  332. --worker-llm-args '{"api_base":"http://127.0.0.1:8000/v1","api_key":"dummy","temperature":0.0,"timeout":360,"num_retries":2,"stop_token_ids":[200002,200012]}' \
  333. --round-gate progressive --power-gate warn --reg-cap 1 \
  334. --meta-workflow hierarchical --max-concurrency 6 --proxy-port 4047
  335. ```
  336. #### Notes:
  337. * **`--meta-model`** is the upstream meta-agent and **`--worker-model`** is the
  338. downstream agent being improved. Both default to Doubao.
  339. * **`--base neutral`** starts from a deterministic seed package, so no
  340. hand-written domain profile enters the loop.
  341. * Start with `qualify` and then a single round. Each round writes a full record:
  342. the evidence the session was given, the plan it produced, the gate arithmetic,
  343. and the ledger entry. A round that admits nothing is a valid outcome.
  344. * Evolving a package for a specific model beats reusing one evolved elsewhere.
  345. On GPT-OSS-20B a rebuilt package gained **+10.2**, while the general-purpose
  346. package transferred negatively.
  347. ## 📖 Citation
  348. ```bibtex
  349. @article{yu2026recuris,
  350. title = {Recursive Experiential--Working Memory Evolution for Long-Horizon Agent Harnesses},
  351. author = {Yu, Zhaochen and Wu, Yingcheng and Yin, Zhenfei and Chen, Kaiyuan and Zhao, Zhe and Wang, Mengdi and Yan, Shuicheng and Yang, Ling},
  352. journal = {arXiv preprint arXiv:2608.24876},
  353. year = {2026}
  354. }
  355. ```
  356. <p align="right">(<a href="#readme-top">back to top</a>)</p>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.