Skip to content
Back to search

CORD-release

Learning from rewards and skills with conflict-aware updates.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/lvaoao--cord-release/badge.svg)](/listings/lvaoao--cord-release)

HTML

<a href="/listings/lvaoao--cord-release"><img src="/api/listings/lvaoao--cord-release/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

The reference setup is **Linux, Python 3.12, CUDA 12.8, and 8 × NVIDIA A100 40 GB**. FlashAttention also requires a CUDA toolkit with `nvcc` and a C++ compiler. Run the commands below from the repository root.

git clone https://github.com/LvAoAo/CORD-release.git
cd CORD-release

conda create -n cord python=3.12 -y
conda activate cord
pip install -r requirements.txt
pip install flash-attn==2.8.3 --no-build-isolation
pip install -e . --no-deps
pip check

The default training launcher requires **96 GiB of free disk space** at startup, including a 64 GiB reserve. Model downloads, caches, logs, and checkpoints are stored locally. A separate logging-service account is not required.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 120 lines
  1. # CORD
  2. **Learning from rewards and skills with conflict-aware updates.**
  3. CORD handles gradient conflicts between two training objectives. This repository provides **Skill-SD + CORD** for multi-turn agents on **ALFWorld** and **WebShop**, using Qwen2.5-3B-Instruct and bundled skill banks.
  4. In this recipe, the reinforcement-learning update is protected: when the two gradients conflict, CORD projects the distillation gradient before combining them. Gradients are accumulated over the full PPO mini-batch before projection. The experiment settings are fixed, so the same commands cover training and independent checkpoint evaluation.
  5. ## Installation
  6. The reference setup is **Linux, Python 3.12, CUDA 12.8, and 8 × NVIDIA A100 40 GB**. FlashAttention also requires a CUDA toolkit with `nvcc` and a C++ compiler. Run the commands below from the repository root.
  7. ```bash
  8. git clone https://github.com/LvAoAo/CORD-release.git
  9. cd CORD-release
  10. conda create -n cord python=3.12 -y
  11. conda activate cord
  12. pip install -r requirements.txt
  13. pip install flash-attn==2.8.3 --no-build-isolation
  14. pip install -e . --no-deps
  15. pip check
  16. ```
  17. The default training launcher requires **96 GiB of free disk space** at startup, including a 64 GiB reserve. Model downloads, caches, logs, and checkpoints are stored locally. A separate logging-service account is not required.
  18. ## Prepare the environments
  19. Choose either task, or prepare both. The skill banks and evaluation manifests are already included; benchmark assets are downloaded separately.
  20. ### ALFWorld
  21. ```bash
  22. export ALFWORLD_DATA="$PWD/data/alfworld"
  23. bash scripts/setup_alfworld.sh
  24. ```
  25. The setup downloads the game assets and checks that the games in the fixed evaluation manifest are available. To reuse an existing installation, set `ALFWORLD_DATA` to its asset directory.
  26. ### WebShop
  27. Install Java 11 and the WebShop dependencies in the same environment:
  28. ```bash
  29. conda install -c conda-forge openjdk=11 -y
  30. export JAVA_HOME="$CONDA_PREFIX/lib/jvm"
  31. pip install -r requirements-webshop.txt
  32. pip install --no-deps https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
  33. bash scripts/setup_webshop.sh
  34. export WEBSHOP_DATA="$PWD/data/webshop"
  35. export WEBSHOP_INDEX="$PWD/data/webshop/indexes"
  36. ```
  37. This downloads the fixed **1,000-product catalog** and builds its Lucene index. To reuse existing assets, point `WEBSHOP_DATA` at the directory containing `items_shuffle_1000.json`, `items_ins_v2_1000.json`, and `items_human_ins.json`, and `WEBSHOP_INDEX` at the matching index. Keep these environment variables set when training or evaluating.
  38. ## Train
  39. Run Skill-SD + CORD on the task you prepared:
  40. ```bash
  41. # ALFWorld
  42. bash scripts/train.sh --task alfworld --method skillsd
  43. # WebShop
  44. bash scripts/train.sh --task webshop --method skillsd
  45. ```
  46. Both recipes use **150 training steps**, learning rate **1e-6**, and distillation coefficient **0.001** with the importance-weighted k3 loss. CORD applies projection only when the gradients conflict. Training uses seed 0 and the included task-specific skills; validation runs every five steps on a fixed 128-assignment manifest.
  47. Outputs go to `outputs/alfworld-skillsd/` or `outputs/webshop-skillsd/`. The final checkpoint is saved under `checkpoints/global_step_150/`. For another run, choose a new directory with `--output-dir`; existing checkpoints are never overwritten.
  48. Use `--model /path/to/model` to load a local copy of Qwen2.5-3B-Instruct. GPU count, tensor parallelism, and micro-batch sizes can be adjusted through `--gpus`, `--tensor-parallel`, `--micro-batch-size`, and `--logprob-micro-batch-size`; see `bash scripts/train.sh --help`. The reference configuration uses eight GPUs and tensor parallelism of two.
  49. ## Evaluate
  50. After training, reload the final checkpoint in a separate evaluation run:
  51. ```bash
  52. # ALFWorld
  53. bash scripts/evaluate.sh --task alfworld --method skillsd
  54. # WebShop
  55. bash scripts/evaluate.sh --task webshop --method skillsd
  56. ```
  57. If training used a custom `--output-dir` or `--model`, pass the same values here. Use `--checkpoint /path/to/global_step_150` to select a different checkpoint. **Evaluation must use the same GPU count as training**, because model weights are saved as FSDP shards. Checkpoints support evaluation; optimizer state is not saved for exact training continuation.
  58. Final evaluation uses four sampled passes at temperature 0.4:
  59. | Task | Fixed evaluation set |
  60. | --- | --- |
  61. | ALFWorld | 128 assignments covering 91 distinct games |
  62. | WebShop | 500 goals from the bundled test manifest |
  63. Each evaluation writes `per_pass.json` and `summary.json`. The summary reports the mean and population standard deviation across the four passes, rather than across independently trained seeds.
  64. ```text
  65. outputs/alfworld-skillsd/
  66. ├── checkpoints/global_step_150/actor/
  67. └── runs/
  68. ├── train-<run-id>/run.log
  69. └── evaluate-<run-id>/evaluation/
  70. ├── per_pass.json
  71. └── summary.json
  72. ```
  73. ## Code layout
  74. | Path | Contents |
  75. | --- | --- |
  76. | `cord/` | Task settings, data setup, and launch commands |
  77. | `verl/trainer/ppo/cord_utils.py` | Distillation loss and CORD gradient combination |
  78. | `verl/workers/actor/dp_actor.py` | Gradient accumulation and actor updates |
  79. | `agent_system/` | Environment interfaces and multi-turn rollouts |
  80. | `skills/` · `manifests/` | Fixed skill banks and evaluation assignments |
  81. ## License
  82. Apache-2.0. See [LICENSE](LICENSE) for the license and third-party attribution notices.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.