Skip to content
Back to search

ToolOPD

ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/magicagent-search--toolopd/badge.svg)](/listings/magicagent-search--toolopd)

HTML

<a href="/listings/magicagent-search--toolopd"><img src="/api/listings/magicagent-search--toolopd/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Python environment

conda create -n toolopd python==3.12 -y
conda activate toolopd

pip3 install vllm==0.11.0
pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .

Log in to Weights & Biases if you use WandB logging (scripts pass `trainer.logger=['console','wandb']`):

export WANDB_API_KEY=your_key_here

Install Supported Environments

1. ALFWorld

pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip3 install alfworld
alfworld-download -f   # PDDL & game files + MaskRCNN detector, stored in ~/.cache/alfworld/

2. WebShop

WebShop requires Python <=3.10, so create a separate environment:

conda create -n verl-webshop python==3.10 -y
conda activate verl-webshop

cd ./agent_system/environments/env_package/webshop/webshop
./setup.sh -d all

cd repo_root/
pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn==2.7.4.post1 --no-build-isolation
pip3 install -e .
pip3 install vllm==0.8.2

The `typer` version warnings can be safely ignored.

3. Search

cd ./agent_system/environments/env_package/search/third_party
pip install -e .
pip install gym==0.26.2

cd repo_root/
python examples/data_preprocess/preprocess_search_r1_dataset.py   # -> ~/data/searchR1_processed_direct

Set up a separate retriever environment (faiss-gpu is not available via pip). The retrieval server uses ~6GB GPU memory per GPU:

conda create -n retriever python=3.10 -y
conda activate retriever
conda install numpy==1.26.4
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip install transformers datasets pyserini huggingface_hub
conda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y
pip install uvicorn fastapi

Download the index and start the retrieval server:

conda activate retriever
local_dir=~/data/searchR1
python examples/search/searchr1_download.py --local_dir $local_dir
cat $local_dir/part_* > $local_dir/e5_Flat.index
gzip -d $local_dir/wiki-18.jsonl.gz

bash examples/search/retriever/retrieval_launch.sh > retrieval_server.log

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 167 lines
  1. <h1 align="center" style="text-align: center;">
  2. ToolOPD
  3. </h1>
  4. <p align="center" style="text-align: center;">
  5. Credit-Aware On-Policy Distillation for Tool-Use Agent
  6. </p>
  7. ## 🔥 Overview
  8. **ToolOPD** is a **credit-aware on-policy distillation** framework for tool-use agents in long-horizon tasks. Standard on-policy distillation (OPD) provides dense, token-level teacher supervision on student-generated trajectories, but assigns the same supervision strength to different actions. Our credit-distribution analysis and outcome-reversal counterfactual experiments show that credit is concentrated on a small number of critical actions and depends on the final outcome, even when the trajectory is held fixed.
  9. ToolOPD uses completed trajectories and their outcomes to identify which teacher signals support useful decisions:
  10. 1. **Hindsight skill construction.** Summarize each completed trajectory and its outcome into a reusable hindsight skill.
  11. 2. **Action credit estimation.** Measure how this skill changes the teacher's probability of each student action token and convert the change into action credit informed by the trajectory outcome.
  12. 3. **Selective supervision.** Preserve supervision that supports successful decisions or discourages failed ones, while suppressing irrelevant and conflicting signals.
  13. 4. **Joint optimization.** Reweight the action-level distillation loss with these credits and jointly optimize it with reinforcement learning.
  14. <p align="center" style="text-align: center;">
  15. <img src="docs/tool_opd/framework.jpg" alt="ToolOPD framework">
  16. </p>
  17. ## 📄 Abstract
  18. On-policy distillation (OPD) enhances the capabilities of small language agents by providing dense, token-level supervision from a teacher model on student-generated trajectories. However, standard OPD implicitly assumes that credit is uniformly distributed across long-horizon trajectories, assigning the same strength of teacher supervision to different actions. Through credit-distribution analysis and outcome-reversal counterfactual experiments, we find that trajectory credit is concentrated on a small number of critical actions. Moreover, even when the trajectory is held fixed, reversing only the final outcome substantially changes the credit assigned to multiple interaction turns. These findings show that the uniform-credit assumption fails to identify critical decisions and ultimately degrades the model's reasoning accuracy. To address these challenges, we propose ToolOPD, a credit-aware on-policy distillation framework designed for tool-use agents in long-horizon tasks. ToolOPD first summarizes each completed trajectory and its outcome into a reusable hindsight skill. It then measures how this skill changes the teacher's probability of each student action token and converts the change into action credit informed by the trajectory outcome. The resulting credit preserves supervision that supports successful decisions or discourages failed ones, while suppressing irrelevant and conflicting signals. ToolOPD uses these credits to reweight the action-level distillation loss and jointly optimizes it with reinforcement learning. Extensive experiments across multiple long-horizon tool interaction benchmarks demonstrate that ToolOPD significantly outperforms existing reinforcement learning and self-distillation baselines.
  19. ## 🔍 Credit Analysis
  20. Our analysis motivates credit-aware supervision in two ways:
  21. - **Concentrated credit.** A small number of critical actions account for much of the trajectory credit.
  22. - **Outcome-dependent credit.** Reversing only the final outcome substantially changes the credit assigned to multiple interaction turns, even when the trajectory remains fixed.
  23. <p align="center" style="text-align: center;">
  24. <img src="docs/tool_opd/analysis.jpg" alt="Credit distribution, outcome sensitivity, conflicting evidence, and training dynamics" width="80%">
  25. </p>
  26. ## 📖 Results
  27. Extensive experiments across multiple long-horizon tool interaction benchmarks demonstrate that ToolOPD significantly outperforms existing reinforcement learning and self-distillation baselines.
  28. <p align="center" style="text-align: center;">
  29. <img src="docs/tool_opd/results.jpg" alt="ToolOPD main experimental results">
  30. </p>
  31. ## 🛠️ Installation
  32. ### Python environment
  33. ```bash
  34. conda create -n toolopd python==3.12 -y
  35. conda activate toolopd
  36. pip3 install vllm==0.11.0
  37. pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
  38. pip install -e .
  39. ```
  40. Log in to Weights & Biases if you use WandB logging (scripts pass `trainer.logger=['console','wandb']`):
  41. ```bash
  42. export WANDB_API_KEY=your_key_here
  43. ```
  44. ### Install Supported Environments
  45. #### 1. ALFWorld
  46. ```bash
  47. pip3 install gymnasium==0.29.1
  48. pip3 install stable-baselines3==2.6.0
  49. pip3 install alfworld
  50. alfworld-download -f # PDDL & game files + MaskRCNN detector, stored in ~/.cache/alfworld/
  51. ```
  52. #### 2. WebShop
  53. WebShop requires Python <=3.10, so create a separate environment:
  54. ```bash
  55. conda create -n verl-webshop python==3.10 -y
  56. conda activate verl-webshop
  57. cd ./agent_system/environments/env_package/webshop/webshop
  58. ./setup.sh -d all
  59. cd repo_root/
  60. pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
  61. pip3 install flash-attn==2.7.4.post1 --no-build-isolation
  62. pip3 install -e .
  63. pip3 install vllm==0.8.2
  64. ```
  65. The `typer` version warnings can be safely ignored.
  66. #### 3. Search
  67. ```bash
  68. cd ./agent_system/environments/env_package/search/third_party
  69. pip install -e .
  70. pip install gym==0.26.2
  71. cd repo_root/
  72. python examples/data_preprocess/preprocess_search_r1_dataset.py # -> ~/data/searchR1_processed_direct
  73. ```
  74. Set up a separate retriever environment (faiss-gpu is not available via pip). The retrieval server
  75. uses ~6GB GPU memory per GPU:
  76. ```bash
  77. conda create -n retriever python=3.10 -y
  78. conda activate retriever
  79. conda install numpy==1.26.4
  80. pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
  81. pip install transformers datasets pyserini huggingface_hub
  82. conda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y
  83. pip install uvicorn fastapi
  84. ```
  85. Download the index and start the retrieval server:
  86. ```bash
  87. conda activate retriever
  88. local_dir=~/data/searchR1
  89. python examples/search/searchr1_download.py --local_dir $local_dir
  90. cat $local_dir/part_* > $local_dir/e5_Flat.index
  91. gzip -d $local_dir/wiki-18.jsonl.gz
  92. bash examples/search/retriever/retrieval_launch.sh > retrieval_server.log
  93. ```
  94. ## 🚀 Training
  95. Run commands from the repository root. ToolOPD training scripts are available under `examples/tool_trainer/`, with the Python entry point `verl.trainer.main_tool_opd`.
  96. Configure the student and teacher model paths before launching ALFWorld training:
  97. ```bash
  98. export STUDENT_MODEL_PATH=/path/to/student
  99. export TEACHER_MODEL_PATH=/path/to/teacher
  100. bash examples/tool_trainer/run_alfworld_qwen3_0_6B.sh vllm
  101. ```
  102. | Environment | Qwen3-0.6B | Qwen3-1.7B |
  103. | ----------- | ------------------------------ | ------------------------------ |
  104. | ALFWorld | `run_alfworld_qwen3_0_6B.sh` | `run_alfworld_qwen3_1_7B.sh` |
  105. | SearchQA | `run_searchqa_qwen3_0_6B.sh` | `run_searchqa_qwen3_1_7B.sh` |
  106. | WebShop | `run_webshop_qwen3_0_6B.sh` | `run_webshop_qwen3_1_7B.sh` |
  107. The student and teacher currently need to share the same tokenizer so their action-token probabilities align. See `examples/tool_trainer/README.md` and the selected launcher for implementation details, dataset paths, GPU settings, and hyperparameters.
  108. Additional launchers for ALFWorld, WebShop, and Search-QA are available in `examples/tool_trainer/`. Configure their model, dataset, and output paths for your environment before use.
  109. Baseline training scripts are also provided under `examples/`.
  110. ### Merge checkpoints
  111. See `scripts/model_merger.py` for FSDP/Megatron merge examples using paths under `./checkpoints/...`.
  112. ## ⭐️ Citation
  113. **TODO**
  114. ## 🤝 Acknowledgement
  115. ToolOPD builds on open-source reinforcement learning and agent frameworks and uses the ALFWorld, WebShop, and Search-R1 environments. We thank the authors and contributors of these projects.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.