Skip to content
Back to search

AgentOPSD

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/zethwang--agentopsd/badge.svg)](/listings/zethwang--agentopsd)

HTML

<a href="/listings/zethwang--agentopsd"><img src="/api/listings/zethwang--agentopsd/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Python environment

conda create -n agentopsd python==3.12 -y
conda activate agentopsd

pip3 install vllm==0.11.0
pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .

Log in to Weights & Biases if you use WandB logging (scripts pass `trainer.logger=['console','wandb']`):

export WANDB_API_KEY=your_key_here

Install Supported Environments

1. ALFWorld

pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip3 install alfworld
alfworld-download -f   # PDDL & game files + MaskRCNN detector, stored in ~/.cache/alfworld/

2. WebShop

WebShop requires Python <=3.10, so create a separate environment:

conda create -n verl-webshop python==3.10 -y
conda activate verl-webshop

cd ./agent_system/environments/env_package/webshop/webshop
./setup.sh -d all

cd repo_root/
pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn==2.7.4.post1 --no-build-isolation
pip3 install -e .
pip3 install vllm==0.8.2

The `typer` version warnings can be safely ignored.

3. Search

cd ./agent_system/environments/env_package/search/third_party
pip install -e .
pip install gym==0.26.2

cd repo_root/
python examples/data_preprocess/preprocess_search_r1_dataset.py   # -> ~/data/searchR1_processed_direct

Set up a separate retriever environment (faiss-gpu is not available via pip). The retrieval server uses ~6GB GPU memory per GPU:

conda create -n retriever python=3.10 -y
conda activate retriever
conda install numpy==1.26.4
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip install transformers datasets pyserini huggingface_hub
conda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y
pip install uvicorn fastapi

Download the index and start the retrieval server:

conda activate retriever
local_dir=~/data/searchR1
python examples/search/searchr1_download.py --local_dir $local_dir
cat $local_dir/part_* > $local_dir/e5_Flat.index
gzip -d $local_dir/wiki-18.jsonl.gz

bash examples/search/retriever/retrieval_launch.sh > retrieval_server.log

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 168 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <h1 align="center">
  2. AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
  3. </h1>
  4. <div align='center' style="font-size:18px;">
  5. <p>
  6. <a href="https://arxiv.org/abs/2608.05987">
  7. <img src="https://img.shields.io/badge/Paper-arXiv%3A2608.05987-blue" alt="Paper"/>
  8. </a>
  9. <a href="https://huggingface.co/papers/2608.05987">
  10. <img src="https://img.shields.io/badge/Daily%20Paper-huggingface-yellow" alt="HF Paper"/>
  11. </a>
  12. <a href="https://github.com/ZethWang/AgentOPSD">
  13. <img src="https://img.shields.io/badge/Code-GitHub-black" alt="Code"/>
  14. </a>
  15. </p>
  16. </div>
  17. ## 🔥 Overview
  18. **AgentOPSD** is a **critic-free, recursive turn-level credit assignment** method for agentic
  19. reinforcement learning. In long-horizon multi-turn tasks, standard RL with verifiable rewards
  20. only constructs a trajectory-level advantage and struggles to credit the few *pivotal* decisions
  21. that actually drive the outcome.
  22. AgentOPSD turns sparse outcome supervision into dense, turn-level credit:
  23. 1. It computes a **token-level teacher–student log-probability gap** using privileged
  24. self-distillation (the same self-oracle signal used by SDAR/RLSD).
  25. 2. It **aggregates these token-level gaps into a turn-level gap** for each interaction turn.
  26. 3. It **recursively updates a Bayesian belief state in log-odds space** across the episode's
  27. turns, and identifies pivotal turns by the marginal revision between consecutive belief states.
  28. The resulting reweighting is fully compatible with standard policy optimization (e.g. GRPO) and
  29. requires **no value network and no extra rollouts**.
  30. <div align="center">
  31. <img src="docs/agentopsd/method.png" alt="method" style="width:100%;">
  32. </div>
  33. ## 📖 Results
  34. AgentOPSD improves over GRPO and strong self-distillation baselines on **ALFWorld**, **WebShop**,
  35. and **Search-QA** with Qwen2.5 (3B / 7B).
  36. <div align="center">
  37. <img src="docs/agentopsd/dynamics.png" alt="training dynamics" style="width:100%;">
  38. </div>
  39. ## 🛠️ Installation
  40. ### Python environment
  41. ```bash
  42. conda create -n agentopsd python==3.12 -y
  43. conda activate agentopsd
  44. pip3 install vllm==0.11.0
  45. pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
  46. pip install -e .
  47. ```
  48. Log in to Weights & Biases if you use WandB logging (scripts pass `trainer.logger=['console','wandb']`):
  49. ```bash
  50. export WANDB_API_KEY=your_key_here
  51. ```
  52. ### Install Supported Environments
  53. #### 1. ALFWorld
  54. ```bash
  55. pip3 install gymnasium==0.29.1
  56. pip3 install stable-baselines3==2.6.0
  57. pip3 install alfworld
  58. alfworld-download -f # PDDL & game files + MaskRCNN detector, stored in ~/.cache/alfworld/
  59. ```
  60. #### 2. WebShop
  61. WebShop requires Python <=3.10, so create a separate environment:
  62. ```bash
  63. conda create -n verl-webshop python==3.10 -y
  64. conda activate verl-webshop
  65. cd ./agent_system/environments/env_package/webshop/webshop
  66. ./setup.sh -d all
  67. cd repo_root/
  68. pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
  69. pip3 install flash-attn==2.7.4.post1 --no-build-isolation
  70. pip3 install -e .
  71. pip3 install vllm==0.8.2
  72. ```
  73. The `typer` version warnings can be safely ignored.
  74. #### 3. Search
  75. ```bash
  76. cd ./agent_system/environments/env_package/search/third_party
  77. pip install -e .
  78. pip install gym==0.26.2
  79. cd repo_root/
  80. python examples/data_preprocess/preprocess_search_r1_dataset.py # -> ~/data/searchR1_processed_direct
  81. ```
  82. Set up a separate retriever environment (faiss-gpu is not available via pip). The retrieval server
  83. uses ~6GB GPU memory per GPU:
  84. ```bash
  85. conda create -n retriever python=3.10 -y
  86. conda activate retriever
  87. conda install numpy==1.26.4
  88. pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
  89. pip install transformers datasets pyserini huggingface_hub
  90. conda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y
  91. pip install uvicorn fastapi
  92. ```
  93. Download the index and start the retrieval server:
  94. ```bash
  95. conda activate retriever
  96. local_dir=~/data/searchR1
  97. python examples/search/searchr1_download.py --local_dir $local_dir
  98. cat $local_dir/part_* > $local_dir/e5_Flat.index
  99. gzip -d $local_dir/wiki-18.jsonl.gz
  100. bash examples/search/retriever/retrieval_launch.sh > retrieval_server.log
  101. ```
  102. ## 🚀 Training
  103. All scripts live under `examples/` and assume the repo root as the working directory.
  104. AgentOPSD scripts are in `examples/agentopsd_trainer/`:
  105. ```bash
  106. bash examples/agentopsd_trainer/run_alfworld_3b.sh
  107. bash examples/agentopsd_trainer/run_alfworld_7b.sh
  108. bash examples/agentopsd_trainer/run_search_3b.sh
  109. bash examples/agentopsd_trainer/run_webshop_3b.sh
  110. ```
  111. Hyperparameters are exposed at the top of every script. In the code the method is named `opsd`
  112. (`verl.trainer.main_opsd`, `algorithm.opsd.*`).
  113. Baselines used in the paper (GRPO, Skill-GRPO, OPSD, GRPO+OPSD, Skill-SD, RLSD, and the SDAR
  114. method) are also provided under `examples/` for reproduction.
  115. ### Merge checkpoints
  116. See `scripts/model_merger.py` for FSDP/Megatron merge examples using paths under `./checkpoints/...`.
  117. ## ⭐️ Citation
  118. If you find this project useful, please consider citing us:
  119. ```bibtex
  120. @article{wang2026agentopsd,
  121. title={AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning},
  122. author={Wang, Zi-Han and Lu, Zhengxi and Yao, Zhiyuan and Wu, Jinyang and Wu, Jie and Cai, Zhengzhou and Sun, Yueqing and Ye, Ziang and Hao, Linji and Gu, Qi and others},
  123. journal={arXiv preprint arXiv:2608.05987},
  124. year={2026}
  125. }
  126. ```
  127. ## 🤝 Acknowledgement
  128. AgentOPSD is built on top of [SDAR](https://github.com/ZJU-REAL/SDAR),
  129. [verl-agent](https://github.com/langfengQ/verl-agent), and [veRL](https://github.com/volcengine/verl),
  130. and uses the [ALFWorld](https://github.com/alfworld/alfworld), WebShop, and
  131. [Search-R1](https://github.com/PeterGriffinJin/Search-R1) environments. We thank the authors of
  132. those projects.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.