trl
Train transformer language models with reinforcement learning.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/huggingface/trl with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/huggingface/trl'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/huggingface--trl)HTML
<a href="/listings/huggingface--trl"><img src="/api/listings/huggingface--trl/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Python Package
Install the library using `pip`:
pip install trl
From source
If you want to use the latest features before an official release, you can install TRL from source:
pip install git+https://github.com/huggingface/trl.git
Repository
If you want to use the examples you can clone the repository with the following command:
git clone https://github.com/huggingface/trl.git
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 262 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # TRL - Transformers Reinforcement Learning
- <div style="text-align: center">
- <picture>
- <source media="(prefers-color-scheme: light)" srcset="https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_light.png">
- <img src="https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png" alt="TRL Banner">
- </picture>
- </div>
- <hr> <br>
- <h3 align="center">
- <p>A comprehensive library to post-train foundation models</p>
- </h3>
- <p align="center">
- <a href="https://github.com/huggingface/trl/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/huggingface/trl.svg?color=blue"></a>
- <a href="https://huggingface.co/docs/trl/index"><img alt="Documentation" src="https://img.shields.io/website?label=documentation&url=https%3A%2F%2Fhuggingface.co%2Fdocs%2Ftrl%2Findex&down_color=red&down_message=offline&up_color=blue&up_message=online"></a>
- <a href="https://github.com/huggingface/trl/releases"><img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/trl.svg"></a>
- <a href="https://huggingface.co/trl-lib"><img alt="Hugging Face Hub" src="https://img.shields.io/badge/🤗%20Hub-trl--lib-yellow"></a>
- </p>
- ## 🎉 What's New
- **⚡ AsyncGRPOTrainer is now stable:** [`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer) graduates to the stable API. It runs GRPO much faster by decoupling generation from training: a vLLM server streams rollouts while the trainer updates the model.
- ## Overview
- TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the [🤗 Transformers](https://github.com/huggingface/transformers) ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.
- ## Highlights
- - **Trainers**: Various fine-tuning methods are easily accessible via trainers like [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer), [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), [`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer), [`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer), [`KTOTrainer`](https://huggingface.co/docs/trl/kto_trainer) and more.
- - **Efficient and scalable**:
- - Leverages [🤗 Accelerate](https://github.com/huggingface/accelerate) to scale from single GPU to multi-node clusters using methods like [DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) and [DeepSpeed](https://github.com/deepspeedai/DeepSpeed).
- - Full integration with [🤗 PEFT](https://github.com/huggingface/peft) enables training on large models with modest hardware via quantization and LoRA/QLoRA.
- - Integrates [🦥 Unsloth](https://github.com/unslothai/unsloth) for accelerating training using optimized kernels.
- - **Command Line Interface (CLI)**: A simple interface lets you fine-tune with models without needing to write code.
- ## Installation
- ### Python Package
- Install the library using `pip`:
- ```bash
- pip install trl
- ```
- ### From source
- If you want to use the latest features before an official release, you can install TRL from source:
- ```bash
- pip install git+https://github.com/huggingface/trl.git
- ```
- ### Repository
- If you want to use the examples you can clone the repository with the following command:
- ```bash
- git clone https://github.com/huggingface/trl.git
- ```
- ## Quick Start
- For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the 🤗 Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.
- ### `SFTTrainer`
- Here is a basic example of how to use the [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer):
- ```python
- from trl import SFTTrainer
- from datasets import load_dataset
- dataset = load_dataset("trl-lib/Capybara", split="train")
- trainer = SFTTrainer(
- model="Qwen/Qwen2.5-0.5B",
- train_dataset=dataset,
- )
- trainer.train()
- ```
- ### `GRPOTrainer`
- [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer) implements the [Group Relative Policy Optimization (GRPO) algorithm](https://huggingface.co/papers/2402.03300) that is more memory-efficient than PPO and was used to train [Deepseek AI's R1](https://huggingface.co/deepseek-ai/DeepSeek-R1). `accuracy_reward` needs math-verify, which you can install with `pip install trl[math_verify]`.
- ```python
- from datasets import load_dataset
- from trl import GRPOTrainer
- from trl.rewards import accuracy_reward
- dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
- trainer = GRPOTrainer(
- model="Qwen/Qwen2.5-0.5B-Instruct",
- reward_funcs=accuracy_reward,
- train_dataset=dataset,
- )
- trainer.train()
- ```
- > [!NOTE]
- > For reasoning models, use the `reasoning_accuracy_reward()` function for better results.
- ### `AsyncGRPOTrainer`
- [`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer) runs the same algorithm as [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), much faster: a vLLM server streams rollouts while the trainer updates the model, so generation and training overlap instead of alternating. The trainer needs vLLM and kernels, and `accuracy_reward` needs math-verify, which you can install with `pip install trl[vllm,kernels,math_verify]`.
- ```python
- # train_async_grpo.py
- from datasets import load_dataset
- from trl import AsyncGRPOTrainer
- from trl.rewards import accuracy_reward
- if __name__ == "__main__":
- dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
- trainer = AsyncGRPOTrainer(
- model="Qwen/Qwen3.5-2B",
- reward_funcs=accuracy_reward,
- train_dataset=dataset,
- )
- trainer.train()
- ```
- Generation runs on a vLLM server, on a different GPU than the trainer. Start it on GPU 0:
- ```bash
- CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \
- --weight-transfer-config '{"backend": "nccl"}' --logprobs-mode processed_logprobs --max-model-len 4096
- ```
- Then run the training on GPU 1:
- ```bash
- CUDA_VISIBLE_DEVICES=1 python train_async_grpo.py
- ```
- ### `DPOTrainer`
- [`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer) implements the popular [Direct Preference Optimization (DPO) algorithm](https://huggingface.co/papers/2305.18290) that was used to post-train [Llama 3](https://huggingface.co/papers/2407.21783) and many other models. Here is a basic example of how to use the `DPOTrainer`:
- ```python
- from datasets import load_dataset
- from trl import DPOTrainer
- dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
- trainer = DPOTrainer(
- model="Qwen/Qwen3-0.6B",
- train_dataset=dataset,
- )
- trainer.train()
- ```
- ### `KTOTrainer`
- [`KTOTrainer`](https://huggingface.co/docs/trl/kto_trainer) implements the [Kahneman-Tversky Optimization (KTO) algorithm](https://huggingface.co/papers/2402.01306), which aligns models from simple binary (desirable / undesirable) feedback rather than paired preferences. Here is a basic example of how to use the `KTOTrainer`:
- ```python
- from datasets import load_dataset
- from trl import KTOTrainer
- dataset = load_dataset("trl-lib/kto-mix-14k", split="train")
- trainer = KTOTrainer(
- model="Qwen/Qwen3-0.6B",
- train_dataset=dataset,
- )
- trainer.train()
- ```
- ### `RewardTrainer`
- Here is a basic example of how to use the [`RewardTrainer`](https://huggingface.co/docs/trl/reward_trainer):
- ```python
- from trl import RewardTrainer
- from datasets import load_dataset
- dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
- trainer = RewardTrainer(
- model="Qwen/Qwen2.5-0.5B-Instruct",
- train_dataset=dataset,
- )
- trainer.train()
- ```
- ## Command Line Interface (CLI)
- You can use the TRL Command Line Interface (CLI) to quickly get started with post-training methods like Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO):
- **SFT:**
- ```bash
- trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
- --dataset_name trl-lib/Capybara \
- --output_dir Qwen2.5-0.5B-SFT
- ```
- **DPO:**
- ```bash
- trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
- --dataset_name argilla/Capybara-Preferences \
- --output_dir Qwen2.5-0.5B-DPO
- ```
- **KTO:**
- ```bash
- trl kto --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
- --dataset_name trl-lib/kto-mix-14k \
- --output_dir Qwen2.5-0.5B-KTO
- ```
- Read more about CLI in the [relevant documentation section](https://huggingface.co/docs/trl/clis) or use `--help` for more details.
- ## Development
- If you want to contribute to `trl` or customize it to your needs make sure to read the [contribution guide](https://github.com/huggingface/trl/blob/main/CONTRIBUTING.md) and make sure you make a dev install:
- ```bash
- git clone https://github.com/huggingface/trl.git
- cd trl/
- pip install -e .[dev]
- ```
- ## Experimental
- A minimal incubation area is available under `trl.experimental` for unstable / fast-evolving features. Anything there may change or be removed in any release without notice.
- Example:
- ```python
- from trl.experimental.new_trainer import NewTrainer
- ```
- Read more in the [Experimental docs](https://huggingface.co/docs/trl/experimental_overview).
- ## Citation
- ```bibtex
- @software{vonwerra2020trl,
- title = {{TRL: Transformers Reinforcement Learning}},
- author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin and Dirhoussi, Amine and Villanova del Moral, Albert and Paniego Blanco, Sergio},
- license = {Apache-2.0},
- url = {https://github.com/huggingface/trl},
- year = {2020}
- }
- ```
- ## License
- This repository's source code is available under the [Apache-2.0 License](LICENSE).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.