Skip to content
Back to search

SDAR

Official code for "Self-Distilled Agentic Reinforcement Learning"

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/zju-real--sdar/badge.svg)](/listings/zju-real--sdar)

HTML

<a href="/listings/zju-real--sdar"><img src="/api/listings/zju-real--sdar/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Python environment

conda create -n sdar python==3.12 -y
conda activate sdar

pip3 install vllm==0.11.0

pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .

Log in to Weights & Biases if you use WandB logging (scripts pass `trainer.logger=['console','wandb']` in many cases):

export WANDB_API_KEY=your_key_here

Install Supported Environments

1. ALFWorld

Install with pip:

pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip3 install alfworld

Download PDDL & Game files and pre-trained MaskRCNN detector (will be stored in `~/.cache/alfworld/`):

alfworld-download -f

2. WebShop

WebShop requires Python <=3.10, so begin by creating a new environment:

conda create -n verl-webshop python==3.10 -y
conda activate verl-webshop

Install WebShop:

cd ./agent_system/environments/env_package/webshop/webshop
./setup.sh -d all

After WebShop is installed, return to the root directory and install the verl package:

cd repo_root/
pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn==2.7.4.post1 --no-build-isolation
pip3 install -e .
pip3 install vllm==0.8.2
# spacy 3.7.2 requires typer<0.10.0,>=0.3.0, but you have typer 0.15.2 which is incompatible.
# weasel 0.3.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.15.2 which is incompatible.

The warnings can be safely ignored.

3. Search

cd ./agent_system/environments/env_package/search/third_party
pip install -e .
pip install gym==0.26.2

Prepare dataset (data will be saved at `~/data/searchR1_processed_direct`):

cd repo_root/
python examples/data_preprocess/preprocess_search_r1_dataset.py

Since faiss-gpu is not available via pip, we setup a separate conda environment for the local retrieval server. Running this server will use around 6GB of GPU memory per GPU, so make sure to account for this in your training run configuration. Build Retriever environments:

# Create and activate the retriever environment with Python 3.10
conda create -n retriever python=3.10 -y
conda activate retriever

# Install PyTorch (with GPU support) and related libraries
conda install numpy==1.26.4 # needed to stop incompatible version of numpy from being installed via pip
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124

# Install other Python packages
pip install transformers datasets pyserini huggingface_hub

# Install the GPU version of faiss
conda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y

# Install the API service framework
pip install uvicorn fastapi

Download the index:

conda activate retriever

local_dir=~/data/searchR1
python examples/search/searchr1_download.py --local_dir $local_dir
cat $local_dir/part_* > $local_dir/e5_Flat.index
gzip -d $local_dir/wiki-18.jsonl.gz

Start the local flat e5 retrieval server:

conda activate retriever

# redirect the output to a file to avoid cluttering the terminal
# we have observed outputting to the terminal causing spikes in server response times
bash examples/search/retriever/retrieval_launch.sh > retrieval_server.log 

Training

1. SDAR

All scripts live under `examples/` and assume the repo root as working directory. You can run e.g.:

bash examples/sdar_trainer/run_alfworld_3b.sh
bash examples/sdar_trainer/run_search_3b.sh
bash examples/sdar_trainer/run_webshop_3b.sh

2. RetireOPD

RetireOPD uses two-stage training (ALFWorld for example):

# Phase 1: Train the teacher

bash examples/skill_grpo_trainer/run_alfworld_3b.sh

# If Phase 1 produces a sharded FSDP checkpoint, merge the actor checkpoint into Hugging Face format before starting Phase 2.
# Record the teacher's validation score for the selected environment, which is the same metric as Phase 2 (`val/success_rate` by default).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 307 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <h1 align="center">
  2. Self-Distilled Agentic Reinforcement Learning
  3. </h1>
  4. <div align='center' style="font-size:18px;">
  5. <p>
  6. <a href="https://arxiv.org/abs/2605.15155">
  7. <img src="https://img.shields.io/badge/Paper-arxiv%3A2605.15155-blue" alt="Paper"/>
  8. </a>
  9. <a href="https://huggingface.co/papers/2605.15155">
  10. <img src="https://img.shields.io/badge/Daily%20Paper-SDAR-yellow" alt="HF Paper"/>
  11. </a>
  12. <a href="https://arxiv.org/abs/2609.20784">
  13. <img src="https://img.shields.io/badge/Paper-arxiv%3A2609.20784-blue" alt="Paper"/>
  14. </a>
  15. <a href="https://huggingface.co/papers/2609.20784">
  16. <img src="https://img.shields.io/badge/Daily%20Paper-RetireOPD-yellow" alt="HF Paper"/>
  17. </a>
  18. </p>
  19. </div>
  20. ## 🔥 Overview
  21. We introduce **SDAR**, a Self-Distilled Agentic Reinforcement learning method with gating mechanism.
  22. <div align="center" style="display:flex; justify-content:center; gap:20px; align-items:flex-start;">
  23. <img src="docs/sdar/sdar_teaser.png" alt="motivation" style="width:40%;">
  24. <img src="docs/sdar/sdart_method.png" alt="method" style="width:58%;">
  25. </div>
  26. ## 🗞️ Awesome Work Powered by SDAR
  27. SDAR is known as the **first** open-sourced framework that unifies Agentic RL with OP(S)D, providing a codebase that has supported the following works (listed in reverse time order and most recent first):
  28. - **AlignOPSD**: Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents [[Paper]](https://arxiv.org/abs/2609.33391) [[Code]](https://github.com/mingju-c/Align-OPSD)
  29. - **PR-OPD**: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2609.36642) [[Code]](https://github.com/balibata/PR-OPD)
  30. - **UniOPSD**: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2609.34810) [[Code]](https://github.com/Zenghuang-Fu/Uniopsd)
  31. - **RetireOPD**: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2609.20784) [[Code]](https://github.com/ZJU-REAL/SDAR)
  32. - **TASPO**: Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization [[Paper]](https://arxiv.org/abs/2608.31077)
  33. - **AHEAD**: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL [[Paper]](https://arxiv.org/abs/2608.24114) [[Code]](https://jinxiaolong1129.github.io/AHEAD/)
  34. - **ICSD**: Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL [[Paper]](https://arxiv.org/abs/2608.14945) [[Code]](https://github.com/lanqz7766/Influence-Calibration-for-On-Policy-Self-Distillation-in-Agentic-RL)
  35. - **BCSD**: Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents [[Paper]](https://arxiv.org/abs/2608.09555)
  36. - **AgentOPSD**: Recursive Self-Distillation for Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2608.05987) [[Code]](https://github.com/ZethWang/AgentOPSD)
  37. - **OCSD**: Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation [[Paper]](https://arxiv.org/pdf/2608.04788) [[Code]](https://github.com/yiy1x/OCSD)
  38. - **ADRS**: Agentic Reinforcement Learning with Self-Distilled Reward Shaping [[Paper]](https://arxiv.org/abs/2608.03223) [[Code]](https://github.com/gitrxh/ADRS-arxiv)
  39. - **PCSD**: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2608.01837)
  40. - **GRSD**: Group-Reflective Self-Distillation for Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2607.28076) [[Code]](https://github.com/BinbZheng1/GRSD)
  41. - **OVCSD**: From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents [[Paper]](https://arxiv.org/abs/2607.27937) [[Code]](https://github.com/shane990928-xia/OVCSD)
  42. - **MAPD**: From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search [[Paper]](https://arxiv.org/abs/2607.24280) [[Code]](https://github.com/AaronLiu0702/MAPD)
  43. - **SEED**: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2607.14777) [[Code]](https://github.com/jinyangwu/SEED)
  44. - **UCOB**: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation [[Paper]](https://arxiv.org/abs/2606.29502) [[Code]](https://github.com/TU2021/UCOB)
  45. - **CRAFT**: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2606.29476)
  46. - **ATOD**: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks [[Paper]](https://arxiv.org/abs/2606.27814) [[Code]](https://github.com/TanQitai/ATOD)
  47. - **OPID**: On-Policy Skill Distillation for Agentic Reinforcement Learning [[Paper]](https://arxiv.org/abs/2606.26790) [[Code]](https://github.com/jinyangwu/OPID)
  48. - **StepOPSD**: Step-Aware Online Preference Distillation for Agent Reinforcement Learning [[Paper]](https://arxiv.org/abs/2605.27140)
  49. ## 📢 News
  50. - **`2026-9-18`**: We released [RetireOPD](https://github.com/ZJU-REAL/SDAR), introducing Self-Retiring OPD for Agentic RL.
  51. - **`2026-8-24`**: 🔥🔥 We released [Agent-G2](https://github.com/ZJU-REAL/Agent-G2), introducing Gaussian hint guidance for Agentic RL.
  52. - **`2026-8-6`**: 🔥 We released [AgentOPSD](https://github.com/ZethWang/AgentOPSD), introducing **recursive credit update** for SDAR. Featured as 🤗 HF Daily Paper #1!
  53. - **`2026-7-29`**: We released [SkillRise](https://github.com/Within-yao/SkillRise), introducing **cross-task skill evolution** via agentic RL.
  54. - **`2026-7-17`**: We released [SEED](https://github.com/jinyangwu/SEED), introducing **self-evolving** opd based on SDAR. Featured as 🤗 HF Daily Paper #3!
  55. - **`2026-6-25`**: We released [OPID](https://github.com/jinyangwu/OPID), introducing **skill evolving** based on SDAR.
  56. - **`2026-6-22`**: We fixed a bug ([issue #35](https://github.com/ZJU-REAL/SDAR/issues/35)) about AlfWorld teacher skill retrieval problem. Please clone the repo again and have a try.
  57. - **`2026-5-15`**: We released our paper and code for SDAR. Featured as 🤗 HF Daily Paper #2!
  58. - **`2026-4`**: Previously, we released [Skill0](https://github.com/ZJU-REAL/SkillZero) and [Skill1](https://github.com/AlphaLab-USTC/Skill1), about lifecycle of agent skills. They are both featured as 🤗 HF Daily Paper #2!
  59. ## 📖 Quick Feature Summary
  60. | Feature Category | Supported Capabilities |
  61. | ----------------- | ------------------------------------------------------------ |
  62. | **Method** | ✅ OP(S)D<br> ✅ GRPO<br> ✅ GRPO+OP(S)D<br> ✅ RLSD<br> ✅ Skill-SD<br> ✅ **SDAR (Ours)**<br> ✅ **RetireOPD (Ours)**<br>|
  63. | **Environment** | ✅ ALFWorld<br> ✅ WebShop<br> ✅ Search-QA |
  64. | **Model Support** | ✅ Qwen3<br> ✅ Qwen2.5 |
  65. ## 📖 Results
  66. SDAR achieves substantial improvements over the standard RL baseline on ALFWorld, WebShop, and Search-QA.
  67. <div align="center">
  68. <img src="docs/sdar/metric.png" alt="Logo" style="width:80%;">
  69. </div>
  70. | Algorithm | Task | Model | Success Rate |
  71. |-----------|----------|------------------------|----------------------|
  72. | RetireOPD | ALFWorld | Qwen2.5-1.5B-Instruct | 89.8% |
  73. | RetireOPD | ALFWorld | Qwen2.5-3B-Instruct | 93.8% |
  74. | RetireOPD | ALFWorld | Qwen2.5-7B-Instruct | 95.3% |
  75. | RetireOPD | WebShop | Qwen2.5-1.5B-Instruct | 75.8% |
  76. | RetireOPD | WebShop | Qwen2.5-3B-Instruct | 77.3% |
  77. | RetireOPD | WebShop | Qwen2.5-7B-Instruct | 84.4% |
  78. ## 🛠️ Installation
  79. ### Python environment
  80. ```bash
  81. conda create -n sdar python==3.12 -y
  82. conda activate sdar
  83. pip3 install vllm==0.11.0
  84. pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
  85. pip install -e .
  86. ```
  87. Log in to Weights & Biases if you use WandB logging (scripts pass `trainer.logger=['console','wandb']` in many cases):
  88. ```bash
  89. export WANDB_API_KEY=your_key_here
  90. ```
  91. ### Install Supported Environments
  92. #### 1. ALFWorld
  93. Install with pip:
  94. ```bash
  95. pip3 install gymnasium==0.29.1
  96. pip3 install stable-baselines3==2.6.0
  97. pip3 install alfworld
  98. ```
  99. Download PDDL & Game files and pre-trained MaskRCNN detector (will be stored in `~/.cache/alfworld/`):
  100. ```bash
  101. alfworld-download -f
  102. ```
  103. #### 2. WebShop
  104. WebShop requires Python <=3.10, so begin by creating a new environment:
  105. ```bash
  106. conda create -n verl-webshop python==3.10 -y
  107. conda activate verl-webshop
  108. ```
  109. Install WebShop:
  110. ```bash
  111. cd ./agent_system/environments/env_package/webshop/webshop
  112. ./setup.sh -d all
  113. ```
  114. After WebShop is installed, return to the root directory and install the verl package:
  115. ```bash
  116. cd repo_root/
  117. pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
  118. pip3 install flash-attn==2.7.4.post1 --no-build-isolation
  119. pip3 install -e .
  120. pip3 install vllm==0.8.2
  121. # spacy 3.7.2 requires typer<0.10.0,>=0.3.0, but you have typer 0.15.2 which is incompatible.
  122. # weasel 0.3.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.15.2 which is incompatible.
  123. ```
  124. The warnings can be safely ignored.
  125. #### 3. Search
  126. ```bash
  127. cd ./agent_system/environments/env_package/search/third_party
  128. pip install -e .
  129. pip install gym==0.26.2
  130. ```
  131. Prepare dataset (data will be saved at `~/data/searchR1_processed_direct`):
  132. ```bash
  133. cd repo_root/
  134. python examples/data_preprocess/preprocess_search_r1_dataset.py
  135. ```
  136. Since faiss-gpu is not available via pip, we setup a separate conda environment for the local retrieval server. Running this server will use around 6GB of GPU memory per GPU, so make sure to account for this in your training run configuration. Build Retriever environments:
  137. ```bash
  138. # Create and activate the retriever environment with Python 3.10
  139. conda create -n retriever python=3.10 -y
  140. conda activate retriever
  141. # Install PyTorch (with GPU support) and related libraries
  142. conda install numpy==1.26.4 # needed to stop incompatible version of numpy from being installed via pip
  143. pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
  144. # Install other Python packages
  145. pip install transformers datasets pyserini huggingface_hub
  146. # Install the GPU version of faiss
  147. conda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y
  148. # Install the API service framework
  149. pip install uvicorn fastapi
  150. ```
  151. Download the index:
  152. ```bash
  153. conda activate retriever
  154. local_dir=~/data/searchR1
  155. python examples/search/searchr1_download.py --local_dir $local_dir
  156. cat $local_dir/part_* > $local_dir/e5_Flat.index
  157. gzip -d $local_dir/wiki-18.jsonl.gz
  158. ```
  159. Start the local flat e5 retrieval server:
  160. ```bash
  161. conda activate retriever
  162. # redirect the output to a file to avoid cluttering the terminal
  163. # we have observed outputting to the terminal causing spikes in server response times
  164. bash examples/search/retriever/retrieval_launch.sh > retrieval_server.log
  165. ```
  166. ### Training
  167. #### 1. SDAR
  168. All scripts live under `examples/` and assume the repo root as working directory. You can run e.g.:
  169. ```bash
  170. bash examples/sdar_trainer/run_alfworld_3b.sh
  171. bash examples/sdar_trainer/run_search_3b.sh
  172. bash examples/sdar_trainer/run_webshop_3b.sh
  173. ```
  174. #### 2. RetireOPD
  175. RetireOPD uses two-stage training (ALFWorld for example):
  176. ```bash
  177. # Phase 1: Train the teacher
  178. bash examples/skill_grpo_trainer/run_alfworld_3b.sh
  179. # If Phase 1 produces a sharded FSDP checkpoint, merge the actor checkpoint into Hugging Face format before starting Phase 2.
  180. # Record the teacher's validation score for the selected environment, which is the same metric as Phase 2 (`val/success_rate` by default).
  181. ```
  182. ```bash
  183. # Phase 2: Train the RetireOPD
  184. TEACHER_MODEL_PATH=/path/to/phase1_teacher_hf \
  185. TEACHER_PERFORMANCE=0.50 \
  186. bash examples/retireopd_trainer/run_alfworld_3b.sh
  187. # Replace 0.50 with the teacher's validation success rate.
  188. ```
  189. #### 3. Other Baselines
  190. Our reproduced codes for GRPO, Skill-GRPO, OP(S)D, GRPO+OP(S)D, Skill-SD, and RLSD are also provided in `examples/`.
  191. ### Merge checkpoints
  192. See `scripts/model_merger.py` for FSDP/Megatron merge examples using paths under `./checkpoints/...`.
  193. ## ⭐️ Citation
  194. If you find this project useful, welcome to cite us.
  195. ```bibtex
  196. @article{lu2026sdar,
  197. title={Self-distilled agentic reinforcement learning},
  198. author={Lu, Zhengxi and Yao, Zhiyuan and Han, Zhuowen and Wang, Zi-Han and Wu, Jinyang and Gu, Qi and Cai, Xunliang and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and others},
  199. journal={arXiv preprint arXiv:2605.15155},
  200. year={2026}
  201. }
  202. @article{lu2026skill0,
  203. title={Skill0: In-context agentic reinforcement learning for skill internalization},
  204. author={Lu, Zhengxi and Yao, Zhiyuan and Wu, Jinyang and Han, Chengcheng and Gu, Qi and Cai, Xunliang and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
  205. journal={arXiv preprint arXiv:2604.02268},
  206. year={2026}
  207. }
  208. @article{wang2026agentopsd,
  209. title={AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning},
  210. author={Wang, Zi-Han and Lu, Zhengxi and Yao, Zhiyuan and Wu, Jinyang and Wu, Jie and Cai, Zhengzhou and Sun, Yueqing and Ye, Ziang and Hao, Linji and Gu, Qi and others},
  211. journal={arXiv preprint arXiv:2608.05987},
  212. year={2026}
  213. }
  214. @article{wu2026seed,
  215. title={SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning},
  216. author={Wu, Jinyang and Yang, Shuo and Lu, Zhengxi and Zhang, Fan and Shen, Yuhao and Feng, Lang and Luo, Haoran and Lian, Zheng and Zhang, Shuai and Wen, Zhengqi and others},
  217. journal={arXiv preprint arXiv:2607.14777},
  218. year={2026}
  219. }
  220. @article{yang2026opid,
  221. title={Opid: On-policy skill distillation for agentic reinforcement learning},
  222. author={Yang, Shuo and Wu, Jinyang and Lu, Zhengxi and Shen, Yuhao and Zhang, Fan and Feng, Lang and Zhang, Shuai and Luo, Haoran and Lian, Zheng and Wen, Zhengqi and others},
  223. journal={arXiv preprint arXiv:2606.26790},
  224. year={2026}
  225. }
  226. @article{shi2026skill1,
  227. title={Skill1: Unified evolution of skill-augmented agents via reinforcement learning},
  228. author={Shi, Yaorui and Chen, Yuxin and Lu, Zhengxi and Miao, Yuchun and Liu, Shugui and Gu, Qi and Cai, Xunliang and Wang, Xiang and Zhang, An},
  229. journal={arXiv preprint arXiv:2605.06130},
  230. year={2026}
  231. }
  232. @article{yao2026skillrise,
  233. title={SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution},
  234. author={Yao, Zhiyuan and Chen, Yuxin and Lu, Zhengxi and Xu, Zishan and Sun, Yueqing and Guo, Yifu and Lu, Yuquan and Cai, Zhengzhou and Zhang, Kangning and Han, Zhuowen and others},
  235. journal={arXiv preprint arXiv:2607.26784},
  236. year={2026}
  237. }
  238. @article{wang2026agentg2,
  239. title={Agent-G $\^{} 2$: Gaussian Guidance for Agentic Reinforcement Learning},
  240. author={Wang, Zixuan and Miao, Yanrui and Lu, Zhengxi and Pan, Teng and Qiu, Yiwen and Li, Hongxing and Qiu, Peng and Zhang, Ruiqing and Shen, Yongliang},
  241. journal={arXiv preprint arXiv:2608.23318},
  242. year={2026}
  243. }
  244. @article{yu2026retireopd,
  245. title={RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning},
  246. author={Yu, Yan and Lu, Zhengxi and Liu, Yizhou and Pan, Yichen and Wang, Aozhe and Chen, Qipeng and Yang, Hua and Zhang, Wenqi and Lu, Weiming and Chen, Qianglong and others},
  247. journal={arXiv preprint arXiv:2609.20784},
  248. year={2026}
  249. }
  250. ```
  251. ## 🤝 Acknowledgement
  252. This project builds on [verl-agent](https://github.com/langfengQ/verl-agent), [veRL](https://github.com/volcengine/verl), [ALFWorld](https://github.com/alfworld/alfworld), [SkillRL](https://github.com/aiming-lab/SkillRL), and [Search-R1](https://github.com/PeterGriffinJin/Search-R1). We thank the authors of those projects.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.