Skip to content
Back to search

trl

Train transformer language models with reinforcement learning.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/huggingface/trl with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/huggingface/trl'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/huggingface--trl/badge.svg)](/listings/huggingface--trl)

HTML

<a href="/listings/huggingface--trl"><img src="/api/listings/huggingface--trl/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Python Package

Install the library using `pip`:

pip install trl

From source

If you want to use the latest features before an official release, you can install TRL from source:

pip install git+https://github.com/huggingface/trl.git

Repository

If you want to use the examples you can clone the repository with the following command:

git clone https://github.com/huggingface/trl.git

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 262 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # TRL - Transformers Reinforcement Learning
  2. <div style="text-align: center">
  3. <picture>
  4. <source media="(prefers-color-scheme: light)" srcset="https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_light.png">
  5. <img src="https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png" alt="TRL Banner">
  6. </picture>
  7. </div>
  8. <hr> <br>
  9. <h3 align="center">
  10. <p>A comprehensive library to post-train foundation models</p>
  11. </h3>
  12. <p align="center">
  13. <a href="https://github.com/huggingface/trl/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/huggingface/trl.svg?color=blue"></a>
  14. <a href="https://huggingface.co/docs/trl/index"><img alt="Documentation" src="https://img.shields.io/website?label=documentation&url=https%3A%2F%2Fhuggingface.co%2Fdocs%2Ftrl%2Findex&down_color=red&down_message=offline&up_color=blue&up_message=online"></a>
  15. <a href="https://github.com/huggingface/trl/releases"><img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/trl.svg"></a>
  16. <a href="https://huggingface.co/trl-lib"><img alt="Hugging Face Hub" src="https://img.shields.io/badge/🤗%20Hub-trl--lib-yellow"></a>
  17. </p>
  18. ## 🎉 What's New
  19. **⚡ AsyncGRPOTrainer is now stable:** [`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer) graduates to the stable API. It runs GRPO much faster by decoupling generation from training: a vLLM server streams rollouts while the trainer updates the model.
  20. ## Overview
  21. TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the [🤗 Transformers](https://github.com/huggingface/transformers) ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.
  22. ## Highlights
  23. - **Trainers**: Various fine-tuning methods are easily accessible via trainers like [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer), [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), [`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer), [`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer), [`KTOTrainer`](https://huggingface.co/docs/trl/kto_trainer) and more.
  24. - **Efficient and scalable**:
  25. - Leverages [🤗 Accelerate](https://github.com/huggingface/accelerate) to scale from single GPU to multi-node clusters using methods like [DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) and [DeepSpeed](https://github.com/deepspeedai/DeepSpeed).
  26. - Full integration with [🤗 PEFT](https://github.com/huggingface/peft) enables training on large models with modest hardware via quantization and LoRA/QLoRA.
  27. - Integrates [🦥 Unsloth](https://github.com/unslothai/unsloth) for accelerating training using optimized kernels.
  28. - **Command Line Interface (CLI)**: A simple interface lets you fine-tune with models without needing to write code.
  29. ## Installation
  30. ### Python Package
  31. Install the library using `pip`:
  32. ```bash
  33. pip install trl
  34. ```
  35. ### From source
  36. If you want to use the latest features before an official release, you can install TRL from source:
  37. ```bash
  38. pip install git+https://github.com/huggingface/trl.git
  39. ```
  40. ### Repository
  41. If you want to use the examples you can clone the repository with the following command:
  42. ```bash
  43. git clone https://github.com/huggingface/trl.git
  44. ```
  45. ## Quick Start
  46. For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the 🤗 Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.
  47. ### `SFTTrainer`
  48. Here is a basic example of how to use the [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer):
  49. ```python
  50. from trl import SFTTrainer
  51. from datasets import load_dataset
  52. dataset = load_dataset("trl-lib/Capybara", split="train")
  53. trainer = SFTTrainer(
  54. model="Qwen/Qwen2.5-0.5B",
  55. train_dataset=dataset,
  56. )
  57. trainer.train()
  58. ```
  59. ### `GRPOTrainer`
  60. [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer) implements the [Group Relative Policy Optimization (GRPO) algorithm](https://huggingface.co/papers/2402.03300) that is more memory-efficient than PPO and was used to train [Deepseek AI's R1](https://huggingface.co/deepseek-ai/DeepSeek-R1). `accuracy_reward` needs math-verify, which you can install with `pip install trl[math_verify]`.
  61. ```python
  62. from datasets import load_dataset
  63. from trl import GRPOTrainer
  64. from trl.rewards import accuracy_reward
  65. dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
  66. trainer = GRPOTrainer(
  67. model="Qwen/Qwen2.5-0.5B-Instruct",
  68. reward_funcs=accuracy_reward,
  69. train_dataset=dataset,
  70. )
  71. trainer.train()
  72. ```
  73. > [!NOTE]
  74. > For reasoning models, use the `reasoning_accuracy_reward()` function for better results.
  75. ### `AsyncGRPOTrainer`
  76. [`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer) runs the same algorithm as [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), much faster: a vLLM server streams rollouts while the trainer updates the model, so generation and training overlap instead of alternating. The trainer needs vLLM and kernels, and `accuracy_reward` needs math-verify, which you can install with `pip install trl[vllm,kernels,math_verify]`.
  77. ```python
  78. # train_async_grpo.py
  79. from datasets import load_dataset
  80. from trl import AsyncGRPOTrainer
  81. from trl.rewards import accuracy_reward
  82. if __name__ == "__main__":
  83. dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
  84. trainer = AsyncGRPOTrainer(
  85. model="Qwen/Qwen3.5-2B",
  86. reward_funcs=accuracy_reward,
  87. train_dataset=dataset,
  88. )
  89. trainer.train()
  90. ```
  91. Generation runs on a vLLM server, on a different GPU than the trainer. Start it on GPU 0:
  92. ```bash
  93. CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \
  94. --weight-transfer-config '{"backend": "nccl"}' --logprobs-mode processed_logprobs --max-model-len 4096
  95. ```
  96. Then run the training on GPU 1:
  97. ```bash
  98. CUDA_VISIBLE_DEVICES=1 python train_async_grpo.py
  99. ```
  100. ### `DPOTrainer`
  101. [`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer) implements the popular [Direct Preference Optimization (DPO) algorithm](https://huggingface.co/papers/2305.18290) that was used to post-train [Llama 3](https://huggingface.co/papers/2407.21783) and many other models. Here is a basic example of how to use the `DPOTrainer`:
  102. ```python
  103. from datasets import load_dataset
  104. from trl import DPOTrainer
  105. dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
  106. trainer = DPOTrainer(
  107. model="Qwen/Qwen3-0.6B",
  108. train_dataset=dataset,
  109. )
  110. trainer.train()
  111. ```
  112. ### `KTOTrainer`
  113. [`KTOTrainer`](https://huggingface.co/docs/trl/kto_trainer) implements the [Kahneman-Tversky Optimization (KTO) algorithm](https://huggingface.co/papers/2402.01306), which aligns models from simple binary (desirable / undesirable) feedback rather than paired preferences. Here is a basic example of how to use the `KTOTrainer`:
  114. ```python
  115. from datasets import load_dataset
  116. from trl import KTOTrainer
  117. dataset = load_dataset("trl-lib/kto-mix-14k", split="train")
  118. trainer = KTOTrainer(
  119. model="Qwen/Qwen3-0.6B",
  120. train_dataset=dataset,
  121. )
  122. trainer.train()
  123. ```
  124. ### `RewardTrainer`
  125. Here is a basic example of how to use the [`RewardTrainer`](https://huggingface.co/docs/trl/reward_trainer):
  126. ```python
  127. from trl import RewardTrainer
  128. from datasets import load_dataset
  129. dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
  130. trainer = RewardTrainer(
  131. model="Qwen/Qwen2.5-0.5B-Instruct",
  132. train_dataset=dataset,
  133. )
  134. trainer.train()
  135. ```
  136. ## Command Line Interface (CLI)
  137. You can use the TRL Command Line Interface (CLI) to quickly get started with post-training methods like Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO):
  138. **SFT:**
  139. ```bash
  140. trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
  141. --dataset_name trl-lib/Capybara \
  142. --output_dir Qwen2.5-0.5B-SFT
  143. ```
  144. **DPO:**
  145. ```bash
  146. trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
  147. --dataset_name argilla/Capybara-Preferences \
  148. --output_dir Qwen2.5-0.5B-DPO
  149. ```
  150. **KTO:**
  151. ```bash
  152. trl kto --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
  153. --dataset_name trl-lib/kto-mix-14k \
  154. --output_dir Qwen2.5-0.5B-KTO
  155. ```
  156. Read more about CLI in the [relevant documentation section](https://huggingface.co/docs/trl/clis) or use `--help` for more details.
  157. ## Development
  158. If you want to contribute to `trl` or customize it to your needs make sure to read the [contribution guide](https://github.com/huggingface/trl/blob/main/CONTRIBUTING.md) and make sure you make a dev install:
  159. ```bash
  160. git clone https://github.com/huggingface/trl.git
  161. cd trl/
  162. pip install -e .[dev]
  163. ```
  164. ## Experimental
  165. A minimal incubation area is available under `trl.experimental` for unstable / fast-evolving features. Anything there may change or be removed in any release without notice.
  166. Example:
  167. ```python
  168. from trl.experimental.new_trainer import NewTrainer
  169. ```
  170. Read more in the [Experimental docs](https://huggingface.co/docs/trl/experimental_overview).
  171. ## Citation
  172. ```bibtex
  173. @software{vonwerra2020trl,
  174. title = {{TRL: Transformers Reinforcement Learning}},
  175. author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin and Dirhoussi, Amine and Villanova del Moral, Albert and Paniego Blanco, Sergio},
  176. license = {Apache-2.0},
  177. url = {https://github.com/huggingface/trl},
  178. year = {2020}
  179. }
  180. ```
  181. ## License
  182. This repository's source code is available under the [Apache-2.0 License](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.