Skip to content
Back to search

Self-Distillation-AgentRL

More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked yesterday
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/aim-uofa--self-distillation-agentrl/badge.svg)](/listings/aim-uofa--self-distillation-agentrl)

HTML

<a href="/listings/aim-uofa--self-distillation-agentrl"><img src="/api/listings/aim-uofa--self-distillation-agentrl/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

All entry points are in [launchers/](launchers/). The workflow below covers environment setup, data preparation, training, checkpoint conversion, and evaluation.

1. Environment Setup

Use Linux x86_64 with Conda, NVIDIA GPUs, and a compatible CUDA toolchain/C++ compiler. Run all commands in Bash from the repository root.

chmod +x launchers/*.sh
export PROJECT_ROOT="$(dirname "$PWD")"
export CONDA_ENV=agentrl
export DATA_ROOT="$PROJECT_ROOT/data"
export HF_HOME="$PROJECT_ROOT/hf_cache"
export VERL_AUTO_PADDING=1

bash launchers/setup_env.sh
source launchers/conda.sh
agentrl_activate_conda

The installer creates the environment, installs the local packages and pinned dependencies, and applies the bundled decord/TextWorld compatibility wheels automatically. Dependency specifications are in [requirements.txt](requirements.txt), [requirements-kernels.txt](requirements-kernels.txt), and [constraints.txt](constraints.txt).

Choose an unused `CONDA_ENV`; use `REUSE_ENV=1` only to resume an interrupted installation. If Conda is not detected, set `CONDA_BASE` to its installation directory. In new terminals, repeat the exports and activation commands, not the installation. Models, data, logs, and checkpoints default to `$PROJECT_ROOT`.

2. Models and Data

Choose `2B`, `4B`, or `9B` for `MODEL_SIZE` and download the corresponding model:

export MODEL_SIZE=2B
export MODEL_PATH="$PROJECT_ROOT/models/Qwen3.5-${MODEL_SIZE}"
MODEL_ID="Qwen/Qwen3.5-${MODEL_SIZE}" bash launchers/prepare_model.sh

For an existing local model, set `MODEL_SIZE` and `MODEL_PATH` and skip the download. Interrupted downloads resume with the same command. When switching models, update these variables and regenerate `EXPERIMENT_NAME` below.

Prepare only the benchmarks you need:

bash launchers/prepare_alfworld.sh
bash launchers/prepare_webshop.sh small
bash launchers/prepare_search.sh qa
bash launchers/prepare_search.sh retrieval
bash launchers/install_scienceworld.sh
bash launchers/prepare_scienceworld.sh

For Search-QA, start the retrieval service in a second terminal with the same environment and path exports:

GPU_ID=0,1,2,3,4,5,6,7 PORT=19080 FAISS_GPU=1 \
  bash launchers/start_search_retriever.sh

This shards the FAISS index across GPUs 0–7 in FP16 and runs the E5 encoder on GPU 0. Keep the service running and reserve memory for it when sharing GPUs with training or evaluation.

In each training/evaluation terminal, set the retrieval endpoint:

export SEARCH_URL=http://127.0.0.1:19080/retrieve

Update `SEARCH_URL` if you change the service host or port.

3. Training

Choose `METHOD` and `TASK` from the identifiers in [Benchmark Coverage](#benchmark-coverage). The same command works across the supported models, methods, and benchmarks; `PAPER_CONFIG=1` selects the study-oriented launcher defaults.

export METHOD=grpo TASK=alfworld
export EXPERIMENT_NAME="${METHOD}_qwen35_${MODEL_SIZE,,}_${TASK}_8gpu"

GPU_ID=0,1,2,3,4,5,6,7 PAPER_CONFIG=1 \
TOTAL_TRAINING_STEPS=100 TEST_FREQ=5 \
SAVE_FREQ=20 MAX_ACTOR_CKPTS=5 RESUME_MODE=disable \
bash "launchers/train_${METHOD}_${TASK}.sh" \
  trainer.n_gpus_per_node=8 ray_init.num_cpus=64 \
  actor_rollout_ref.rollout.enable_chunked_prefill=True \
  actor_rollout_ref.rollout.enforce_eager=False \
  +actor_rollout_ref.rollout.engine_kwargs.vllm.gdn_prefill_backend=triton

Keep `GPU_ID` and `trainer.n_gpus_per_node` consistent; for one GPU, use `GPU_ID=0` and `trainer.n_gpus_per_node=1`. To resume training, keep the original experiment name and set `RESUME_MODE=auto`.

With `PAPER_CONFIG=1`, rollout GPU memory utilization defaults to 0.30 for 2B/4B training, 0.65 for 9B training (including validation), and 0.30 for standalone evaluation. Model size is selected by `MODEL_SIZE`, or the standard model directory name when unset. Override with `GPU_MEMORY_UTILIZATION` if needed.

4. Checkpoint Conversion

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 211 lines
  1. <div align="center">
  2. # More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
  3. [![arXiv](https://img.shields.io/badge/arXiv-2610.19789-b31b1b)](https://arxiv.org/abs/2610.19789) [![Project](https://img.shields.io/badge/Project-Page-blue)](https://aim-uofa.github.io/Self-Distillation-AgentRL) [![Code](https://img.shields.io/badge/Code-GitHub-orange)](https://github.com/aim-uofa/Self-Distillation-AgentRL)
  4. </div>
  5. **Self-Distillation-AgentRL** is the official codebase for our study of self-distillation in agentic reinforcement learning.
  6. ## Overview
  7. Do more self-distillation methods lead to better agents? We compare ten representative approaches against GRPO across four benchmarks and three model sizes, using shared environments and evaluation procedures while retaining method-specific training settings. Beyond final task performance, we examine training progress, stability, and measured computational cost.
  8. ![Study overview: limitations of prior comparisons and a unified benchmark across methods, tasks, and model sizes.](assets/overview.png)
  9. **Study overview.** From fragmented evidence to a common benchmark for assessing the breadth of gains, training stability, and computational cost.
  10. Our evaluation yields three main findings:
  11. - **Gains remain setting-dependent.** No method consistently outperforms GRPO across settings.
  12. - **Strong final performance does not imply stable training.** Final scores can conceal instability along the training trajectory.
  13. - **More computation does not consistently yield better agents.** Additional training cost does not reliably translate into improved performance.
  14. This repository provides a common framework for training, checkpoint conversion, and evaluation, supporting direct comparisons and further research on reliable, cost-effective agent training.
  15. ## Benchmark Coverage
  16. The study spans **12 model–task settings** and **132 training runs**, with repeated evaluations over **eight rollout seeds** for pretrained and trained models.
  17. ### Supported Models
  18. - ✅ Qwen3.5-2B (`MODEL_SIZE=2B`)
  19. - ✅ Qwen3.5-4B (`MODEL_SIZE=4B`)
  20. - ✅ Qwen3.5-9B (`MODEL_SIZE=9B`)
  21. ### Supported Benchmarks
  22. - ✅ ALFWorld (`TASK=alfworld`)
  23. - ✅ ScienceWorld (`TASK=scienceworld`)
  24. - ✅ Search-QA (`TASK=search`)
  25. - ✅ WebShop (`TASK=webshop`)
  26. ### Supported Methods
  27. - ✅ GRPO — baseline (`METHOD=grpo`)
  28. - ✅ GRPO+OPSD (`METHOD=grpo_opsd`)
  29. - ✅ SDAR (`METHOD=sdar`)
  30. - ✅ Skill-SD (`METHOD=skillsd`)
  31. - ✅ RLSD (`METHOD=rlsd`)
  32. - ✅ OCSD (`METHOD=ocsd`)
  33. - ✅ OVCSD (`METHOD=ovcsd`)
  34. - ✅ AgentOPSD (`METHOD=agentopsd`)
  35. - ✅ ADRS (`METHOD=adrs`)
  36. - ✅ ICSD (`METHOD=icsd`)
  37. - ✅ PCSD (`METHOD=pcsd`)
  38. The launchers support training-time validation, standalone evaluation, rollout recording, benchmark mean/standard deviation, and pass@k.
  39. ## Getting Started
  40. All entry points are in [launchers/](launchers/). The workflow below covers environment setup, data preparation, training, checkpoint conversion, and evaluation.
  41. ### 1. Environment Setup
  42. Use Linux x86_64 with Conda, NVIDIA GPUs, and a compatible CUDA toolchain/C++ compiler. Run all commands in Bash from the repository root.
  43. ```bash
  44. chmod +x launchers/*.sh
  45. export PROJECT_ROOT="$(dirname "$PWD")"
  46. export CONDA_ENV=agentrl
  47. export DATA_ROOT="$PROJECT_ROOT/data"
  48. export HF_HOME="$PROJECT_ROOT/hf_cache"
  49. export VERL_AUTO_PADDING=1
  50. bash launchers/setup_env.sh
  51. source launchers/conda.sh
  52. agentrl_activate_conda
  53. ```
  54. The installer creates the environment, installs the local packages and pinned dependencies, and applies the bundled decord/TextWorld compatibility wheels automatically. Dependency specifications are in [requirements.txt](requirements.txt), [requirements-kernels.txt](requirements-kernels.txt), and [constraints.txt](constraints.txt).
  55. Choose an unused `CONDA_ENV`; use `REUSE_ENV=1` only to resume an interrupted installation. If Conda is not detected, set `CONDA_BASE` to its installation directory. In new terminals, repeat the exports and activation commands, not the installation. Models, data, logs, and checkpoints default to `$PROJECT_ROOT`.
  56. ### 2. Models and Data
  57. Choose `2B`, `4B`, or `9B` for `MODEL_SIZE` and download the corresponding model:
  58. ```bash
  59. export MODEL_SIZE=2B
  60. export MODEL_PATH="$PROJECT_ROOT/models/Qwen3.5-${MODEL_SIZE}"
  61. MODEL_ID="Qwen/Qwen3.5-${MODEL_SIZE}" bash launchers/prepare_model.sh
  62. ```
  63. For an existing local model, set `MODEL_SIZE` and `MODEL_PATH` and skip the download. Interrupted downloads resume with the same command. When switching models, update these variables and regenerate `EXPERIMENT_NAME` below.
  64. Prepare only the benchmarks you need:
  65. ```bash
  66. bash launchers/prepare_alfworld.sh
  67. bash launchers/prepare_webshop.sh small
  68. bash launchers/prepare_search.sh qa
  69. bash launchers/prepare_search.sh retrieval
  70. bash launchers/install_scienceworld.sh
  71. bash launchers/prepare_scienceworld.sh
  72. ```
  73. For Search-QA, start the retrieval service in a second terminal with the same environment and path exports:
  74. ```bash
  75. GPU_ID=0,1,2,3,4,5,6,7 PORT=19080 FAISS_GPU=1 \
  76. bash launchers/start_search_retriever.sh
  77. ```
  78. This shards the FAISS index across GPUs 0–7 in FP16 and runs the E5 encoder on GPU 0. Keep the service running and reserve memory for it when sharing GPUs with training or evaluation.
  79. In each training/evaluation terminal, set the retrieval endpoint:
  80. ```bash
  81. export SEARCH_URL=http://127.0.0.1:19080/retrieve
  82. ```
  83. Update `SEARCH_URL` if you change the service host or port.
  84. ### 3. Training
  85. Choose `METHOD` and `TASK` from the identifiers in [Benchmark Coverage](#benchmark-coverage). The same command works across the supported models, methods, and benchmarks; `PAPER_CONFIG=1` selects the study-oriented launcher defaults.
  86. ```bash
  87. export METHOD=grpo TASK=alfworld
  88. export EXPERIMENT_NAME="${METHOD}_qwen35_${MODEL_SIZE,,}_${TASK}_8gpu"
  89. GPU_ID=0,1,2,3,4,5,6,7 PAPER_CONFIG=1 \
  90. TOTAL_TRAINING_STEPS=100 TEST_FREQ=5 \
  91. SAVE_FREQ=20 MAX_ACTOR_CKPTS=5 RESUME_MODE=disable \
  92. bash "launchers/train_${METHOD}_${TASK}.sh" \
  93. trainer.n_gpus_per_node=8 ray_init.num_cpus=64 \
  94. actor_rollout_ref.rollout.enable_chunked_prefill=True \
  95. actor_rollout_ref.rollout.enforce_eager=False \
  96. +actor_rollout_ref.rollout.engine_kwargs.vllm.gdn_prefill_backend=triton
  97. ```
  98. Keep `GPU_ID` and `trainer.n_gpus_per_node` consistent; for one GPU, use `GPU_ID=0` and `trainer.n_gpus_per_node=1`. To resume training, keep the original experiment name and set `RESUME_MODE=auto`.
  99. With `PAPER_CONFIG=1`, rollout GPU memory utilization defaults to 0.30 for 2B/4B training, 0.65 for 9B training (including validation), and 0.30 for standalone evaluation. Model size is selected by `MODEL_SIZE`, or the standard model directory name when unset. Override with `GPU_MEMORY_UTILIZATION` if needed.
  100. ### 4. Checkpoint Conversion
  101. Convert a saved checkpoint to Hugging Face format before evaluation. Change `global_step_100` to select a different training step.
  102. ```bash
  103. CHECKPOINT_TASK="$TASK"
  104. if [[ "$TASK" == search ]]; then CHECKPOINT_TASK=searchQA; fi
  105. STEP_DIR="$PROJECT_ROOT/checkpoints/$METHOD/$CHECKPOINT_TASK/$EXPERIMENT_NAME/global_step_100"
  106. bash "launchers/merge_${METHOD}_checkpoint.sh" "$STEP_DIR"
  107. export MERGED_MODEL="$STEP_DIR/merged_hf"
  108. ```
  109. Allow enough CPU RAM and disk space for merging; evaluation uses `merged_hf/`, not the raw `actor/` shards.
  110. ### 5. Evaluation
  111. To evaluate pretrained weights, complete steps 1–2 and select a benchmark, for example `export TASK=alfworld`. To evaluate a trained checkpoint, first convert it as described above.
  112. #### Single Evaluation
  113. ```bash
  114. PAPER_CONFIG=1 GPU_ID=0 EVAL_DO_SAMPLE=true EVAL_TEMPERATURE=0.4 \
  115. bash "launchers/test_pretrained_${TASK}.sh" \
  116. ray_init.num_cpus=64 \
  117. +actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16 \
  118. actor_rollout_ref.rollout.enable_chunked_prefill=True \
  119. +actor_rollout_ref.rollout.engine_kwargs.vllm.gdn_prefill_backend=triton
  120. ```
  121. To evaluate a trained model, use `bash "launchers/test_${METHOD}_${TASK}.sh" "$MERGED_MODEL"` in place of the pretrained script, keeping the same environment variables and overrides.
  122. #### Multiple Seeds and Pass@k
  123. ```bash
  124. PAPER_CONFIG=1 MODEL_KIND=pretrained GPU_IDS=0,1,2,3,4,5,6,7 \
  125. MAX_PASS_K=8 TEMPERATURE=0.4 \
  126. RUN_ID=eval_run_01 \
  127. bash launchers/eval_pass_at_k.sh
  128. ```
  129. For a merged checkpoint from any method, replace `MODEL_KIND=pretrained` with `MODEL_KIND=grpo GRPO_MODEL_PATH="$MERGED_MODEL" MODEL_TAG="${EXPERIMENT_NAME}_step100"`. Here `grpo` names the shared checkpoint-loading route, not the training method; adjust the tag to match the checkpoint step.
  130. With `MAX_PASS_K=8`, seeds default to 0–7, with one seed per GPU.
  131. Logs are saved under `$PROJECT_ROOT/logs/`. Repeated evaluation also saves rollouts, benchmark mean/standard deviation, and pass@k under `logs/pass_at_k/`.
  132. ## Acknowledgments
  133. Our implementation builds on [SDAR](https://github.com/ZJU-REAL/SDAR), [veRL](https://github.com/volcengine/verl), and [verl-agent](https://github.com/langfengQ/verl-agent). We thank the authors and contributors of these projects and the benchmark environments for their open-source work.
  134. ## 🖊️ Citation
  135. If you find this work useful, please consider citing:
  136. ```bibtex
  137. @misc{zhong2026methodsbetteragentsreality,
  138. title={More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning},
  139. author={Linhao Zhong and Bozhen Fang and Xuehao Wang and Duochao Shi and Zekai Luo and Guangming Yao and Yi Yuan and Hao Chen and Jingdong Chen and Chunhua Shen},
  140. year={2026},
  141. eprint={2610.19789},
  142. archivePrefix={arXiv},
  143. primaryClass={cs.CL},
  144. url={https://arxiv.org/abs/2610.19789},
  145. }
  146. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.