Skip to content
Back to search

halo

Halo is an open-source framework built by White Circle for training large language and multimodal models

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/whitecircle/halo with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/whitecircle/halo'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/whitecircle--halo/badge.svg)](/listings/whitecircle--halo)

HTML

<a href="/listings/whitecircle--halo"><img src="/api/listings/whitecircle--halo/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Installation

Halo ships as separate Docker images for Blackwell and Hopper, with PyTorch, Flash Attention, and DeepEP V2 already built in — pull anonymously, no account required:

# B200/B300 — FA4 + FA2 + DeepEP
docker pull public.ecr.aws/whitecircle/halo:blackwell

# H100/H200 — FA2 + FA3 + DeepEP
docker pull public.ecr.aws/whitecircle/halo:hopper

Versioned tags (`:blackwell-1.1.0`, `:hopper-1.1.0`) pin the release. There is no `latest` tag, since the images are architecture-specific. The RL inference images are published alongside them (`:vllm-0.26.0`, `:sglang-0.5.17`), pinned as `:vllm-0.26.0-1.1.0` and `:sglang-0.5.17-1.1.0`.

To build locally instead:

# B200/B300 — FA4 + FA2 + DeepEP
make build-blackwell

# H100/H200 — FA2 + FA3 + DeepEP
make build-hopper

No host Python environment is required. See [Installation](human-docs/installation.md) for image details and build options.

Launch a training run

`halo launch` takes a training method and a YAML config:

# SFT + LoRA (fits into a single GPU)
halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml

# SFT + Expert Parallelism (MoE, 8 GPUs → torchrun)
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8

# FSDP-2 + Accelerate
halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
    -a launcher-configs/accelerate/fsdp2_gradop_config.yaml -n 8

# all training methods (sft, smpo, dpo, rlvr, environmental-grpo, ...)
halo launch --list

`halo` selects `python`, `torchrun`, or `accelerate launch` based on the arguments. Options after the config that are not the launcher's own are passed to the trainer and override the YAML config:

halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml \
    -n 8 \
    --expert_parallel_size=8 \
    --learning_rate=1e-5 \
    --max_length=32000

Use `--dry-run` to print the command without running it, and `--list` to see the available methods:

halo launch --list
halo launch sft config.yaml --dry-run

Other tools

Checkpoint, inference, data, and environment utilities use halo run:

halo run merge-ep-shards --input_dir <ep-checkpoint-dir> --output_dir <merged-dir>
halo run quantize-to-lowp --input_dir <bf16-model-dir> --output_dir <nvfp4-out-dir> --format nvfp4

# all tools (merge-*, quantize-to-lowp, rm-scoring, run-env, …)
halo run --list             

The underlying scripts can also be run directly:

python scripts/training/sft.py \
    examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml

torchrun --nproc_per_node=8 scripts/training/sft.py \
    examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml

Coding agents

The training images ship repo-specific agent skills (discoverable by Claude Code and Codex; add `--build-arg INSTALL_CLAUDE_CODE=1` to bake the CLI into the image) for [launching jobs](skills/launch), [choosing parallelism settings](skills/parallelism), [debugging distributed runs](skills/debug), [adding models](skills/add-model), [managing checkpoints](skills/checkpoints), [preparing data](skills/data), [optimizing throughput](skills/optimize), and [setting up RL](skills/rl-setup).

See [`skills/`](skills/) for the available skills.

</br>

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 431 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <p align="center">
  2. <img src="/logo.png" height="100" alt="Halo logo: a white ring on black">
  3. <h3 align="center">Halo</h3>
  4. </p>
  5. <p align="center">
  6. Frontier-Lab Training for Everyone
  7. </p>
  8. <p align="center">
  9. <a href="https://whitecircle.com/research/halo"><strong>Blog post</strong></a> ·
  10. <a href="https://whitecircle.com/halo/docs"><strong>Docs</strong></a> ·
  11. <a href="examples/"><strong>Examples</strong></a> ·
  12. <a href="https://whitecircle.com/halo/docs/performance"><strong>Benchmarks</strong></a>
  13. </p>
  14. <br/>
  15. ## What is Halo
  16. Halo is an open-source framework built by [White Circle](https://whitecircle.com) for training large language and multimodal models. It supports pre-training, SFT, preference optimization, fully asynchronous multi-turn RL, and much more.
  17. The same codebase runs on one GPU or across multiple nodes, with EP, CP, TP, ETP, PP, FSDP2, fused kernels, BF16 training, and distributed rollouts.
  18. Halo trains Hugging Face models directly. Checkpoints still load with `from_pretrained`, and the EP layer a MoE family adds is under 90 lines for every family but GPT-OSS, at 168 ([Model Integration Cost](human-docs/model-integration-cost.md)).
  19. On 8× B300, Halo 1.2.0 trains Qwen3.6-35B-A3B on packed 64k and 128k rows at 1.10–1.12× Megatron Bridge, 1.47× TRL 1.17.1 and 1.9–2.0× NeMo AutoModel 0.7.0, and Qwen3.5-122B-A10B at 1.50× TRL and 1.7–1.9× NeMo; it trails a tuned Megatron by 11.5% on dense Qwen3.5-9B at 64k ([Benchmarks](#benchmarks)). It runs 1.11–2.07× as fast as Halo 1.1.0 on the twelve shapes benchmarked against it.
  20. `Pre- & Post-Training` · `EP / CP / TP / ETP / PP` ·
  21. `Multi-Node` · `Verifiable & Multi-Turn RL` · `FA4 + Liger + Grouped GEMM` · `Full BF16`
  22. </br>
  23. ## Why Halo
  24. - **Halo trains Hugging Face models directly.** Trainers subclass Hugging Face or TRL trainers, checkpoints remain standard SafeTensors, and `from_pretrained` continues to work.
  25. - **Parallelism is added to the existing model.** EP wraps MoE blocks, CP wraps attention, and TP/ETP shard weights in place. There is no separate distributed implementation of the model.
  26. - **The distributed stack is built on PyTorch.** FSDP2, DTensor, and DeviceMesh provide the primitives, and DeepEP handles the all-to-all communication for Expert Parallelism.
  27. - **Each parallelism axis has its own setting.** EP, CP, TP, ETP, and PP are configured separately. EP combines with CP, TP, or ETP, PP combines with EP or ETP, and other pairings are rejected at config time.
  28. - **Training methods share the same infrastructure.** Pre-training, SFT, preference optimization, distillation, and RL use the same parallelism and checkpointing code, so a new method doesn't need its own distributed implementation.
  29. - **SFT can train a release's MTP head.** The multi-token-prediction head that ships for speculative decoding trains beside a fine-tune or over a frozen trunk ([MTP heads](agent-docs/optimization/mtp-heads.md)).
  30. - **Performance optimizations are integrated.** Halo includes DeepEP V2, FlashAttention 4, Liger, Grouped GEMM, padding-free and tree packing, AdamWBF16, and kernels for GLM-5 Next, with fallbacks for Blackwell, Hopper, and older GPUs.
  31. - **RL is asynchronous, with a clean Transformers ↔ vLLM/SGLang split.** Multi-turn environment rollouts run as Ray actors against vLLM or SGLang servers, and the trainer pushes updated weights to them over native NCCL. The training side stays plain Transformers, with no Megatron backend or veRL dependency. See [Async GRPO with Environments](human-docs/training-methods/async-grpo-environments.md).
  32. </br>
  33. ## Latest updates
  34. <!-- Newest first. This feed and GitHub Releases are Halo's changelog. -->
  35. - **2026-11-07 — Halo 1.3.0rc1.** A release candidate for faster RL at scale. On synchronous GRPO with
  36. Qwen3.5-35B-A3B on two 8× B300 nodes, a step takes 0.137 s per 1k response tokens against Miles v0.1.3's 0.176
  37. ([GRPO against Miles](agent-docs/optimization/throughput-benchmarks.md#grpo-against-miles)).
  38. - **Faster GRPO:** rows pack into token-budget micro-batches, importance weights can come from the loss forward,
  39. and weight sync can run one group across servers.
  40. - **New in RL:** score centering on SGLang, FP8-weight rollouts for Qwen3.5/3.6 MoE, adapter-only sync to
  41. SGLang, and GLM-5.3-Flash under GRPO.
  42. - **Scale:** gathered EP saves striped over several writers, adapter merges with no model load, and startup
  43. checks for setups that used to fail hours into a run.
  44. - **2026-11-04 — Halo 1.2.0.** Training steps are faster than in 1.1.0 on all twelve shapes benchmarked
  45. against it, from 1.11× to 2.07×, and GLM-5.3-Flash more than doubles its LoRA throughput with two new kernels.
  46. Halo also gains pipeline parallelism and three more model families ([benchmarks](human-docs/performance.md)).
  47. - **GLM-5.3-Flash:** new attention and hyper-connection kernels. LoRA at EP8 runs 2.2–2.7× faster than
  48. before them with the same peak memory, and a full fine-tune on two 8-GPU machines up to 2.3× faster.
  49. - **Pipeline parallelism** for SFT, preference, offline GRPO, reward, and classification training, with EP
  50. or ETP inside each stage ([docs](agent-docs/parallelism/pipeline-parallelism.md)).
  51. - **New models:** NVIDIA Nemotron 3.5 (Super VL and Lightning), Qwen3-Next, and experimental
  52. Qwen3.8-Flash-Next, for eighteen MoE families in all.
  53. - **SFT features:** tree packing, which stores a shared prefix once, and training of MTP heads.
  54. - **Upgrade notes:** the import package is now `halo`, on transformers 5.20, TRL 1.14, vLLM 0.32, and
  55. SGLang 0.5.22. PaliGemma SFT and distillation now use a causal mask, which 1.1.0 left off.
  56. - **2026-10-11 — Halo 1.1.0.** Faster MoE training, exact resume, and more reliable async RL. Fused MoE
  57. kernels and FlexAttention on sliding-window layers made Gemma 4 26B-A4B 1.67× faster than 1.0.0 on two
  58. GPUs, and an atomic-free expert permute made gpt-oss at EP8 up to 27% faster. Checkpoint and resume problems now stop the run instead of
  59. passing silently, and offline GRPO gained context parallelism.
  60. - **2026-08-20 — Halo 1.0.0.** First public release: EP / CP / TP / ETP on native HuggingFace models
  61. (LLM and VLM), 15 MoE families, pre-training through multi-turn RL with vLLM or SGLang rollouts,
  62. FlashAttention-4, DeepEP V2, DeepGEMM, and the AdamWBF16 optimizer.
  63. </br>
  64. ## Quick start
  65. ### Installation
  66. Halo ships as separate Docker images for Blackwell and Hopper, with PyTorch, Flash Attention, and
  67. DeepEP V2 built in. Pull them anonymously, with no account:
  68. ```bash
  69. # B200/B300: FA4 + FA2 + DeepEP
  70. docker pull public.ecr.aws/whitecircle/halo:blackwell
  71. # H100/H200: FA2 + FA3 + DeepEP
  72. docker pull public.ecr.aws/whitecircle/halo:hopper
  73. ```
  74. Versioned tags (`:blackwell-1.3.0rc1`, `:hopper-1.3.0rc1`) pin the release. There is no `latest` tag, since
  75. the images are architecture-specific. The RL inference images are published alongside them
  76. (`:vllm-0.32.0`, `:sglang-0.5.22`), pinned as `:vllm-0.32.0-1.3.0rc1` and `:sglang-0.5.22-1.3.0rc1`.
  77. To build locally instead:
  78. ```bash
  79. # B200/B300: FA4 + FA2 + DeepEP
  80. make build-blackwell
  81. # H100/H200: FA2 + FA3 + DeepEP
  82. make build-hopper
  83. ```
  84. No host Python environment is required. See [Installation](human-docs/installation.md) for image details and build options.
  85. ### Launch a training run
  86. `halo launch` takes a training method and a YAML config:
  87. ```bash
  88. # SFT + LoRA (fits into a single GPU)
  89. halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml
  90. # SFT + Expert Parallelism (MoE, 8 GPUs → torchrun)
  91. halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8
  92. # FSDP-2 + Accelerate
  93. halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
  94. -a launcher-configs/accelerate/fsdp2_gradop_config.yaml -n 8
  95. # all training methods (sft, preference/smpo, grpo/online, grpo/environmental, ...)
  96. halo launch --list
  97. ```
  98. `halo` selects `python`, `torchrun`, or `accelerate launch` based on the arguments, and `--dry_run`
  99. prints the command instead of running it. Options after the config that are not the launcher's own are
  100. passed to the trainer and override the YAML config:
  101. ```bash
  102. halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml \
  103. -n 8 \
  104. --expert_parallel_size=8 \
  105. --learning_rate=1e-5 \
  106. --max_length=32000
  107. ```
  108. ### Other tools
  109. Checkpoint, inference, data, and environment utilities use `halo run`:
  110. ```bash
  111. halo run merge-ep-shards --input_dir <ep-checkpoint-dir> --output_dir <merged-dir>
  112. halo run quantize-to-lowp --input_dir <bf16-model-dir> --output_dir <nvfp4-out-dir> --format nvfp4
  113. # all tools (merge-*, quantize-to-lowp, inference/reward-model/score, eval-env, …)
  114. halo run --list
  115. ```
  116. The underlying scripts can also be run directly:
  117. ```bash
  118. python scripts/training/sft.py \
  119. examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml
  120. torchrun --nproc_per_node=8 scripts/training/sft.py \
  121. examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml
  122. ```
  123. ### Coding agents
  124. The training images ship repo-specific agent skills (discoverable by Claude Code and Codex; add
  125. `--build-arg INSTALL_CLAUDE_CODE=1` to bake the CLI into the image) for [launching jobs](skills/launch),
  126. [choosing parallelism settings](skills/parallelism), [debugging distributed runs](skills/debug), [adding models](skills/add-model), [managing checkpoints](skills/checkpoints),
  127. [preparing data](skills/data), [optimizing throughput](skills/optimize), [setting up RL](skills/rl-setup),
  128. [writing tests](skills/write-tests), and [keeping docs current](skills/docs).
  129. </br>
  130. ## Documentation
  131. We keep two versions of the docs, depending on whether or not you're AI:
  132. - **[`human-docs/`](human-docs/README.md)**: shorter, task-oriented guides for installing, configuring,
  133. and running Halo.
  134. - **[`agent-docs/`](agent-docs/README.md)**: the full technical reference, written for coding agents,
  135. covering model integrations, parallelism, optimization internals, infrastructure, and troubleshooting.
  136. The full reference is also perfectly readable if you're human and want the details.
  137. **The human guide:**
  138. [Installation](human-docs/installation.md) ·
  139. [Quickstart](human-docs/quickstart.md) ·
  140. [Choosing a Method](human-docs/choosing-a-method.md) ·
  141. [The halo CLI](human-docs/cli.md) ·
  142. [Writing a Config](human-docs/configuration.md) ·
  143. [Datasets](human-docs/data.md) ·
  144. [Training Methods](human-docs/training-methods/README.md) ·
  145. [Rollout Servers](human-docs/rollout-servers.md) ·
  146. [Parallelism](human-docs/parallelism.md) ·
  147. [Clusters & Multi-Node](human-docs/clusters.md) ·
  148. [Performance](human-docs/performance.md) ·
  149. [Supported Matrix](human-docs/supported-matrix.md) ·
  150. [Supported Models](human-docs/models.md) ·
  151. [Model Cookbooks](human-docs/cookbooks/README.md) ·
  152. [Model Integration Cost](human-docs/model-integration-cost.md) ·
  153. [Checkpoints & Export](human-docs/checkpoints.md) ·
  154. [Monitoring](human-docs/monitoring.md) ·
  155. [Troubleshooting](human-docs/troubleshooting.md) ·
  156. [Environment Variables](human-docs/environment-variables.md) ·
  157. [AI Tooling](human-docs/ai-tooling.md) ·
  158. [Contributing](human-docs/contributing.md)
  159. </br>
  160. ## Repository layout
  161. A map of the repository:
  162. | Path | What's inside |
  163. |---|---|
  164. | **`halo/`** | Trainers, parallelism, optimizers, kernels, collators, and callbacks |
  165. | **`scripts/`** | Training entry points, inference, data-prep, and checkpoint tools |
  166. | **`examples/`** | Ready-to-run YAML configs, grouped by method and model |
  167. | **`launcher-configs/`** | Accelerate, SkyPilot, and Nomad launch configs |
  168. | **`human-docs/`** | Short, task-oriented documentation |
  169. | **`agent-docs/`** | Full technical reference |
  170. | **`skills/`** | Agent skills (linked from `.claude/skills` and `.agents/skills`) |
  171. | **`tests/`** | CPU and distributed GPU tests |
  172. | **`Dockerfile` · `Makefile`** | Image builds, tests, and training commands |
  173. | **`AGENTS.md`** | Contributor and agent instructions |
  174. A typical run is: **pick an `examples/` config → `halo launch <method> <config>`** (or `python scripts/training/<method>.py <config>`), inside the Docker image.
  175. </br>
  176. ## Benchmarks
  177. All rows ran on 8× B300 with the Blackwell image unless a row says otherwise. Rates are tokens/s per GPU on a warm
  178. kernel cache; a fresh Triton cache costs the first steps (Qwen3.5-35B-A3B's first ~20 run at about 40% of the steady
  179. rate).
  180. ### Against other frameworks
  181. Full-parameter SFT on UltraChat 200k packed into rows of exactly 64k or 128k tokens with documents isolated, the same rows
  182. for every framework. Each cell is tok/s/GPU (peak GiB); † is an nvidia-smi peak, which reads higher than the others.
  183. | Model, tokens per row | Halo 1.2.0 | Megatron Bridge | TRL 1.17.1 | NeMo AutoModel 0.7.0 |
  184. |---|---:|---:|---:|---:|
  185. | Qwen3.5-9B, 64k | 17,206 (54.3) | **19,431** (245.5†) | 14,735 (56.0) | 10,360 (99.3) |
  186. | Qwen3.5-9B, 128k | 17,093 (82.1) | **17,348** (241.1†) | 14,601 (86.8) | 9,451 (166.4) |
  187. | Qwen3.6-35B-A3B, 64k | **23,637** (139.1) | 21,559 (224.6†) | 16,126 (138.8) | 12,340 (99.1) |
  188. | Qwen3.6-35B-A3B, 128k | **23,647** (139.1) | 21,109 (244.2†) | 16,112 (138.8) | 12,026 (139.3) |
  189. | Qwen3.5-122B-A10B, 64k | **8,933** (188.7) | 4,781 (254.0†) | 5,936 (194.5) | 5,397 (219.3) |
  190. | Qwen3.5-122B-A10B, 128k | **8,943** (188.7) | does not fit | 5,951 (194.5) | 4,683 (227.1) |
  191. Megatron Bridge is a 0.7.0 release-candidate commit with Megatron Core 0.20.0. How to read the table:
  192. - Optimizer regimes differ: Halo trains bf16 parameters with stochastic rounding, TRL and NeMo bf16 parameters with bf16 Adam
  193. state, Megatron fp32 main parameters (fp16 master on the 122B).
  194. - Megatron's peaks are nvidia-smi samples; the others are torch allocated peaks. Do not compare them directly.
  195. - Each Halo row is the median of 3 runs of a chosen setup; the others are single runs of their best layouts.
  196. Rates per parallelism layout, the 16k and 4k tables, and the other conditions: [Throughput Benchmarks](agent-docs/optimization/throughput-benchmarks.md#full-parameter-sft-framework-comparison);
  197. the TRL side: [Halo vs stock TRL](agent-docs/optimization/halo-vs-stock-trl.md).
  198. ### GRPO against Miles
  199. GRPO on Qwen3.5-35B-A3B (GSM8K, 32 prompts × 8 samples, one 8× B300 node training and one serving SGLang
  200. rollouts, same prompts on both sides): on-policy, Halo takes 0.137 s per 1k response tokens against Miles v0.1.3's 0.176, 22% less,
  201. with a 14.0 s step against 18.6 s and 63 against 109 GiB on the trainer node. With rollouts overlapped it takes 0.102
  202. against 0.111, 8% less. Layouts and conditions:
  203. [GRPO against Miles](agent-docs/optimization/throughput-benchmarks.md#grpo-against-miles).
  204. ### Halo 1.2.0 against 1.1.0
  205. | Result | Configuration |
  206. |---|---|
  207. | **1.11–2.07× Halo 1.1.0** | Twelve SFT shapes, none slower: TP2 1.82×, EP4+ETP2 2.07×, EP 1.12–1.29×, CP2 1.15×, FSDP2 1.11×. Qwen3-8B, gpt-oss-20b and Qwen3.5-35B-A3B at batch 1, GC on. |
  208. | **2.2–2.7× on GLM-5.3-Flash LoRA** | EP8, real text: 2,840 vs 1,282 at 8k, 2,390 vs 893 at 16,384. Same peak memory. The baseline is 1.2.0 without the two new kernels. |
  209. | **1.65–2.3× on GLM-5.3-Flash full fine-tune** | Two 8-GPU machines: EP16 1,504 vs 913, PP2 + EP8 2,423 vs about 1,060. EP8 with DP runs out of memory. |
  210. | **PP across two machines** | gpt-oss-20b PP2 + EP8 at 8k: 15,715, 15.9% below EP8 with DP. Qwen3-8B PP2 at 4k: 17,608, 4.0% below DP16. Qwen3.5-35B-A3B PP2 + EP8 reaches 0.97× EP8 with DP, at 126.7 GiB on the first stage and 62.7 on the last, against 148.7 (GC off, packed rows of 8k). |
  211. | **13.8× DP16 on plain TCP sockets** | Machines forced onto plain TCP sockets: Qwen3-8B PP2 holds 15,414 while DP16 falls to 1,117. |
  212. ### Kernels and memory (Halo 1.1.0, not rerun on 1.2.0)
  213. These rows have no 1.2.0 measurement; read each as a 1.1.0 number.
  214. | Result | Configuration |
  215. |---|---|
  216. | **28,700 tok/s/GPU** | gpt-oss-20b, dense EP1, 4k, batch 4, GC off: ~230k tok/s across 8 GPUs. |
  217. | **16,447 tok/s/GPU at 1,908 TFLOPS** | Qwen3.5-35B-A3B, 4k, batch 4, EP2, SDPA: the highest achieved TFLOPS of any EP>1 run benchmarked on 1.1.0. |
  218. | **25–76 GiB/GPU on the same 16k workload** | EP8+CP8: 24.6 GiB at 7,149 tok/s/GPU. Dense EP1: 75.8 GiB at 20,690 tok/s/GPU. |
  219. | **256k context** | gpt-oss-20b, dense EP1 ZeRO-3 with fused linear cross-entropy: 5,728 tok/s/GPU at 112 GiB (measured on Halo 1.0.0). |
  220. | **2.45× Grouped GEMM** | Qwen3-30B-A3B, 2× B300, EP2, 8k, batch 4, GC on; 4.10× at batch 1. |
  221. | **3.6–3.9× FA4 kernel throughput** | FA4 vs FA2 on the isolated kernel (B2×S8192, head_dim 128); up to 2.3× end-to-end on dense Qwen3-4B at 32k (1× B300). |
  222. Full results and methodology: [Performance](human-docs/performance.md)
  223. </br>
  224. ## Parallelism
  225. Each axis has its own flag. EP combines with one of CP, TP, or ETP, and PP with EP or ETP; any other pairing is rejected at config time:
  226. ```bash
  227. --expert_parallel_size=N # MoE experts across GPUs (DeepEP)
  228. --context_parallel_size=N # long sequences (Ulysses attention)
  229. --tensor_parallel_size=N # model weights (DTensor; attention only on MoE)
  230. --expert_tensor_parallel_size=N # expert-FFN sharding (MoE, experimental)
  231. --pipeline_parallel_size=N # decoder layers in stages across nodes (registered families)
  232. ```
  233. | Mode | Best for |
  234. |---|-------------------------------------------|
  235. | EP | Distribute MoE experts across GPUs |
  236. | TP | Shard model weights across GPUs (a dense model past one GPU) |
  237. | CP | Split long sequences across GPUs |
  238. | ETP | Shard individual expert FFNs |
  239. | EP+CP | MoE + long-context training |
  240. | EP+TP | MoE + weight sharding |
  241. | EP+ETP | MoE + sharded experts |
  242. | PP | Split a model's layers across nodes |
  243. | PP+EP, PP+ETP | MoE across nodes, experts inside each stage |
  244. Over a fast link, PP is slower than data parallelism when the model fits. It pays on slower links and for
  245. models that do not fit at EP8 on one machine. PP does not support LoRA yet.
  246. Pass them directly to `torchrun`, or after the config with `halo launch`:
  247. ```bash
  248. halo launch sft config.yaml -n 8 --expert_parallel_size=8
  249. ```
  250. Multi-node runs support shared (NFS/Lustre) and node-local filesystems.
  251. See [Parallelism](human-docs/parallelism.md) to pick a mode, [Supported Matrix](human-docs/supported-matrix.md)
  252. for the per-model matrix, and [Clusters & Multi-Node](human-docs/clusters.md) for the multi-node recipes.
  253. </br>
  254. ## Optimizations
  255. Enabled by default where supported:
  256. - **Attention and kernels.** Flash Attention 4 on Blackwell, FA3 on Hopper, FA2 elsewhere, Liger fused
  257. CE/SwiGLU/RMSNorm (Qwen3 MoE, GLM-4.7-Flash, GPT-OSS, Gemma 4 and Nemotron-H take torch's fused RMSNorm instead),
  258. fused GLU kernels, and Grouped GEMM for MoE expert compute.
  259. - **GLM-5 Next and DeepSeek-V4 kernels.** DSA attention on Blackwell (`HALO_DSA_MATMUL`, GLM-5 Next) and
  260. hyper-connections (`HALO_HC_KERNELS`) run on dedicated kernels
  261. ([GLM-5 Next](agent-docs/models/glm5-next.md), [hyper-connection kernels](agent-docs/optimization/hyper-connection-kernels.md)).
  262. - **Parallel steps.** TP linears on their local shards, one Ulysses all-to-all per exchange, a sync-free
  263. MoE routing sort, and two decoder layers per FSDP2 shard group under ZeRO-2.
  264. - **Precision and optimizers.** `AdamWBF16` with stochastic rounding keeps weights and optimizer
  265. state in 6 bytes/parameter instead of 12 with FP32 master weights. Muon and FlashAdamW are also
  266. supported. FP8/FP4 QAT runs on MoE experts and dense MLPs through a fake-quant path, with an opt-in
  267. DeepGEMM forward; `quantize-to-lowp` writes block-scaled mxfp8/mxfp4/nvfp4 weights.
  268. - **Memory and PEFT.** Padding-free packing with `cu_seq_lens`, SFT tree packing, LoRA, and QLoRA. The
  269. [Qwen3-4B QLoRA example](examples/sft/qwen3/qwen3-4b-ultrachat-qlora.yaml) peaks at 7.9 GiB (batch 1, 4k)
  270. and Qwen3-8B at 32k at 20.9 GiB with the fused loss ([PEFT](agent-docs/optimization/peft.md)). Single-GPU
  271. LoRA/QLoRA also runs on Ampere and Ada cards from `halo:blackwell`, but that path is not validated.
  272. - **Checkpoints.** Large gathered checkpoints are automatically sharded, with tools for merging per-rank
  273. EP shards, per-node PP saves and PEFT adapters back into a standard HuggingFace checkpoint
  274. ([Checkpoints](human-docs/checkpoints.md)).
  275. - **Data.** Offline tokenization, packing, and sharding, plus native `s3://` dataset loading with a local cache.
  276. [Performance](human-docs/performance.md) covers what each lever buys and the bottlenecks they attack.
  277. </br>
  278. ## Built with AI
  279. We use GPT and Claude extensively while developing Halo, through
  280. [PI-Agent](https://github.com/earendil-works/pi),
  281. [Claude Code](https://github.com/anthropics/claude-code), and
  282. [Codex](https://github.com/openai/codex).
  283. [`AGENTS.md`](AGENTS.md) contains the instructions and conventions used by both coding agents and
  284. human contributors.
  285. AI-assisted contributions are welcome. We ask that contributors understand and stand behind
  286. the code they submit. See [`CONTRIBUTING.md`](CONTRIBUTING.md).
  287. </br>
  288. ## Acknowledgments
  289. Halo builds on a lot of great open-source work, in particular:
  290. - **[Hugging Face](https://huggingface.co)**: Transformers, TRL, Accelerate, Datasets, and PEFT.
  291. - **[Liger Kernel](https://github.com/linkedin/Liger-Kernel)**: fused cross-entropy, SwiGLU, and RMSNorm.
  292. - **[FlashAdamW](https://github.com/databricks/flashoptim)** by Databricks AI Research: memory-efficient
  293. AdamW with quantized optimizer states and compact master weights.
  294. - **[DeepEP](https://github.com/deepseek-ai/DeepEP)** and **[DeepGEMM](https://github.com/deepseek-ai/DeepGEMM)**
  295. by DeepSeek-AI: expert communication and grouped GEMM. Halo's aux-loss-free router balancing also
  296. follows the approach introduced with DeepSeek-V3.
  297. - **[Flash Attention](https://github.com/Dao-AILab/flash-attention)** by Tri Dao and contributors:
  298. FA2, FA3, and FA4.
  299. - **[vLLM](https://github.com/vllm-project/vllm)**: the rollout/serving engine behind online and
  300. environmental RL.
  301. - **[PyTorch](https://pytorch.org)**: DTensor, FSDP2, and `torch.distributed`.
  302. - **[SGLang](https://github.com/sgl-project/sglang)**: the alternative rollout engine, with NCCL weight
  303. synchronization, for multi-turn environment RL.
  304. - **[SDPG](https://arxiv.org/abs/2606.04036)**, led by researchers at UCLA: self-distilled policy
  305. gradient for online reinforcement learning.
  306. DeepEP is built from upstream at a pinned commit in the training image and keeps its upstream license;
  307. the build pins, Flash Attention's included, live in the [`Dockerfile`](Dockerfile).
  308. See [`pyproject.toml`](pyproject.toml) for the full dependency set.
  309. Halo is built and maintained by **[White Circle](https://whitecircle.com)**.
  310. </br>
  311. ## License
  312. Halo is released under the **Halo License**: the Apache License 2.0
  313. ([`APACHE-2.0.txt`](APACHE-2.0.txt)) as modified by the Supplemental Terms in [`LICENSE`](LICENSE).
  314. Copyright © 2026 White Circle, PBC.
  315. In plain terms: free to use, modify, self-host, and to train your own models and ship them in your
  316. own products, at any scale. The one commercial condition is offering Halo's training functionality
  317. to third parties as a service: that's free up to US \$20M in such revenue over any 12 months, and
  318. above that requires a commercial agreement with White Circle. Models trained with Halo by anyone
  319. over \$20M in annual revenue must carry a "Trained with Halo" credit.
  320. Questions or commercial licensing: [hello@whitecircle.com](mailto:hello@whitecircle.com).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.