Skip to content
Back to search

TorchSpec

A PyTorch native library for training speculative decoding models

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/lightseekorg/TorchSpec with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/lightseekorg/TorchSpec'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/lightseekorg--torchspec/badge.svg)](/listings/lightseekorg--torchspec)

HTML

<a href="/listings/lightseekorg--torchspec"><img src="/api/listings/lightseekorg--torchspec/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Train an Eagle3 draft model for Qwen3-8B on a single node with 4 GPUs (2 for training and 2 for inference):

./examples/qwen3-8b-single-node/run.sh

Override config values directly from the CLI:

./examples/qwen3-8b-single-node/run.sh training.learning_rate=5e-5 training.num_train_steps=500

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 328 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # TorchSpec
  2. TorchSpec is a torch-native speculative decoding training framework. We introduce a disaggregated way of training speculative decoding draft models where inference and training are fully decoupled and stream hidden states directly from inference engine groups to distributed training workers via [Mooncake](https://github.com/kvcache-ai/Mooncake) store, allowing each side to scale independently.
  3. ## Adoption
  4. TorchSpec has been adopted by production inference platforms and the vLLM ecosystem:
  5. - [DigitalOcean](https://www.digitalocean.com/blog/how-we-built-fastest-deepseek-minimax-qwen-on-blackwell-ultra) used TorchSpec to train an EAGLE3 draft model for MiniMax-M2.5 on DigitalOcean Serverless Inference.
  6. - [vLLM](https://vllm.ai/blog/2026-05-11-vllm-tops-artificial-analysis) used TorchSpec and vLLM to train the custom EAGLE3 draft model featured in its Artificial Analysis leaderboard work.
  7. - [CoreWeave](https://www.coreweave.com/blog/kimi-k2-7-code-now-available-on-serverless-inference-with-leading-benchmark-price-performance) used TorchSpec to train a DFlash speculative decoding model for Kimi K2.7 Code and contributed D-PACE support upstream.
  8. - [fal](https://blog.fal.ai/how-we-achieved-1000-tok-s-and-16x-throughput-with-dspark-for-ideogram-v4-prompt-expander/) used TorchSpec to train a DSpark speculative decoding model for its Ideogram V4 prompt expander, reporting 16x throughput gains with DSpark.
  9. - [Tencent Hunyuan](https://x.com/TencentHunyuan/status/2082447023626944936) open-sourced [**AngelSpec**](https://arxiv.org/abs/2607.25852), adopting TorchSpec for training and featuring it in the paper.
  10. ## 🤗 Released Models
  11. Draft models trained with TorchSpec, available on the [LightSeek Foundation](https://huggingface.co/lightseekorg) Hugging Face organization:
  12. - [lightseekorg/kimi-k2.5-eagle3](https://huggingface.co/lightseekorg/kimi-k2.5-eagle3)
  13. - [lightseekorg/kimi-k2.5-eagle3-mla](https://huggingface.co/lightseekorg/kimi-k2.5-eagle3-mla)
  14. - [lightseekorg/kimi-k2.6-eagle3](https://huggingface.co/lightseekorg/kimi-k2.6-eagle3)
  15. - [lightseekorg/kimi-k2.6-eagle3-mla](https://huggingface.co/lightseekorg/kimi-k2.6-eagle3-mla)
  16. - [lightseekorg/kimi-k2.6-eagle3.1-mla](https://huggingface.co/lightseekorg/kimi-k2.6-eagle3.1-mla)
  17. - [lightseekorg/kimi-k3-eagle3-mla](https://huggingface.co/lightseekorg/kimi-k3-eagle3-mla)
  18. - [lightseekorg/kimi-k3-dflash2](https://huggingface.co/lightseekorg/kimi-k3-dflash2)
  19. - [lightseekorg/kimi-k3-dspark](https://huggingface.co/lightseekorg/kimi-k3-dspark)
  20. Draft models trained with TorchSpec, available from other organizations:
  21. - [Inferact/MiniMax-M3-EAGLE3](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3)
  22. - [Inferact/Kimi-K3-DSpark](https://huggingface.co/Inferact/Kimi-K3-DSpark)
  23. ## 🚀 Blogs and Announcements
  24. - TorchSpec now integrates [SMG](https://github.com/smg-project/smg) for chat templating and tokenization in supported text training paths. (Oct 2026)
  25. - 🔥 [TorchSpec: Speculative Decoding Training at Scale](https://www.youtube.com/watch?v=M7fbxwBSIeI) (Youtube Video, Ray Summit 2026)
  26. - [Releasing Kimi K3 Draft Collection: Efficient Draft Training in TorchSpec](https://lightseek.org/blog/kimi-k3-draft-collection.html) (Sep 2026)
  27. - [EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec](https://lightseek.org/blog/eagle-3-1.html) (May 2026)
  28. - [TorchSpec: Speculative Decoding Training at Scale](https://pytorch.org/blog/torchspec-speculative-decoding-training-at-scale/) (PyTorch Blog, March 2026)
  29. - [TorchSpec: Speculative Decoding Training at Scale](https://lightseek.org/blog/torchspec-speculative-decoding-training-at-scale.html) (March 2026)
  30. ## Table of Contents
  31. - [Architecture Overview](#architecture-overview)
  32. - [Inference Backend Support](#inference-backend-support)
  33. - [Quick Start](#quick-start)
  34. - [Setup](#setup)
  35. - [Examples](#examples)
  36. - [Training Modes](#training-modes)
  37. - [Checkpoint Conversion](#checkpoint-conversion)
  38. - [Metrics Reporting](#metrics-reporting)
  39. - [Troubleshooting](#troubleshooting)
  40. ## Architecture Overview
  41. <p align="center">
  42. <img src="docs/torchspec_architecture.png" alt="TorchSpec Architecture" width="100%">
  43. </p>
  44. TorchSpec is built around a disaggregated training pipeline:
  45. - **Inference engines** generate target-model hidden states with inference engines.
  46. - **Mooncake store** transfers tensors between inference and training without materializing them on disk.
  47. - **Training workers** consume streamed hidden states to train speculative decoding draft models.
  48. This separation keeps the training side focused on optimization while letting the inference side scale for hidden-state generation throughput.
  49. ## Inference Backend Support
  50. TorchSpec streams hidden states from inference engines into training workers.
  51. | Backend | Support Tier | Status |
  52. |---------|--------------|--------|
  53. | [vLLM](https://github.com/vllm-project/vllm) | First-class | Available |
  54. | [TokenSpeed](https://github.com/lightseekorg/tokenspeed) | First-class | In progress |
  55. | [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | First-class | Available |
  56. | [SGLang](https://github.com/sgl-project/sglang) | Best community effort | Available |
  57. | [HuggingFace Transformers](https://github.com/huggingface/transformers) | Best community effort | Available |
  58. ## Quick Start
  59. Train an Eagle3 draft model for Qwen3-8B on a single node with 4 GPUs (2 for training and 2 for inference):
  60. ```bash
  61. ./examples/qwen3-8b-single-node/run.sh
  62. ```
  63. Override config values directly from the CLI:
  64. ```bash
  65. ./examples/qwen3-8b-single-node/run.sh training.learning_rate=5e-5 training.num_train_steps=500
  66. ```
  67. ## Setup
  68. ### Quick Setup
  69. Source installs build the bundled Rust tokenizer. Install [Rust/rustup](https://rustup.rs/)
  70. and select the pinned toolchain first (prebuilt wheels and Docker runtime images
  71. do not need a compiler):
  72. ```bash
  73. rustup toolchain install 1.98.0 --profile minimal
  74. ```
  75. ```bash
  76. # Install with vLLM
  77. ./tools/build_conda.sh 1 vllm
  78. micromamba activate torchspec
  79. # Or install with SGLang
  80. ./tools/build_conda.sh
  81. micromamba activate torchspec
  82. # Or install TokenSpeed from an editable source checkout
  83. ./tools/build_conda.sh 1 tokenspeed
  84. micromamba activate torchspec
  85. ```
  86. To install into your current environment instead:
  87. ```bash
  88. ./tools/build_conda.sh current tokenspeed # or 'sglang', 'vllm', or 'both'
  89. ```
  90. The TokenSpeed backend currently requires a Python 3.12 environment because
  91. its native kernel dependency wheels do not support Python 3.14.
  92. Optional: install Flash Attention support:
  93. ```bash
  94. pip install -e ".[fa]"
  95. ```
  96. ### Training tokenizer
  97. Supported text profiles (`llama3`, `qwen`, `qwen3-thinking`) and
  98. Kimi-K3 use SMG's Rust tokenizer in-process by default. Its native extension ships
  99. in the TorchSpec wheel, and `pip install -e .` builds it automatically. Source
  100. builds need Rust 1.98.0; prebuilt wheels and our Docker runtime images do not.
  101. See [installation details](docs/smg-tokenization.md#installation).
  102. To explicitly use the old tokenizer, set `dataset.rendering_backend: huggingface`
  103. or pass `dataset.rendering_backend=huggingface` on the CLI. Non-K3 deferred/decode training,
  104. other templates, and custom renderers retain their existing auto routing. Both built-in
  105. K3 backends require complete, unformatted conversations; the HF fallback does not
  106. enable deferred/decode or preformatted/generation-prefix modes. Missing
  107. SMG bindings and rendering errors never silently change the selected backend.
  108. See [SMG setup, supported paths, and correctness gates](docs/smg-tokenization.md).
  109. ### Backend-Specific Usage
  110. **vLLM**
  111. ```bash
  112. ./examples/qwen3-8b-single-node/run.sh configs/vllm_qwen3_8b.yaml
  113. ```
  114. **SGLang**
  115. ```bash
  116. ./examples/qwen3-8b-single-node/run.sh
  117. ```
  118. **TensorRT-LLM**
  119. Run inside the TensorRT-LLM image (`docker/trtllm/v1.3.0rc18/Dockerfile`), which ships `tensorrt_llm` pre-patched for Mooncake hidden-state capture:
  120. ```bash
  121. ./examples/qwen3-8b-single-node/run.sh configs/trtllm_qwen3_8b.yaml
  122. ```
  123. Single-node tensor parallelism only for now (multi-node TP is not yet wired up).
  124. TorchSpec uses vLLM's **Worker Extension** mechanism to hook into the model forward pass and capture hidden states directly inside worker processes, which avoids RPC serialization overhead during extraction. For SGLang, TorchSpec applies a patch to the existing codebase to enable hidden-state extraction. For TensorRT-LLM, TorchSpec builds on its native **SaveHiddenStates** speculative mode and applies a small patch that redirects the captured aux + final hidden states to Mooncake instead of writing them to disk.
  125. ### Updating the SGLang patch
  126. Commit your changes in the `_sglang` checkout, then regenerate `patches/sglang/<version>/sglang.patch` from the pinned base commit with `./tools/update_sglang_patch.sh`.
  127. ## Examples
  128. | Example | Backend | Model |
  129. |---------|---------|-------|
  130. | [hf-quickstart](examples/hf-quickstart/) | HuggingFace | Qwen3-8B |
  131. | [qwen3-8b-single-node](examples/qwen3-8b-single-node/) | Inference engine | Qwen3-8B |
  132. | [kimi-k25-2node-h200](examples/kimi-k25-2node-h200/) | Inference engine | Kimi-K2.5 |
  133. | [kimi-k25-3node-h100](examples/kimi-k25-3node-h100/) | Inference engine | Kimi-K2.5 |
  134. See [examples/README.md](examples/README.md) for more details about each example.
  135. ## Training Modes
  136. ### Offline Replay Training
  137. Offline replay ([Docs](docs/offline_training.md)) is a training mode that separates target-output generation from draft-model training by reading hidden states from disk. Recommended for testing and development on 1 GPU.
  138. ### Resume vs. Continual Training
  139. Both modes use `training.load_path`, but they restore different states:
  140. | Goal | `training.load_path` | `training.continual_training` | What gets restored |
  141. |------|----------------------|-------------------------------|--------------------|
  142. | Resume an interrupted run | Required | `false` (default) | Model, optimizer, LR scheduler, RNG, and step metadata |
  143. | Start a new run from existing weights | Required | `true` | Model weights only |
  144. Resume the same run:
  145. ```yaml
  146. training:
  147. load_path: /path/to/old_run/checkpoints
  148. output_dir: /path/to/old_run
  149. ```
  150. Start a new run from existing weights:
  151. ```yaml
  152. training:
  153. load_path: /path/to/old_run/checkpoints
  154. continual_training: true
  155. learning_rate: 1e-5
  156. warmup_ratio: 0.01
  157. num_epochs: 1
  158. output_dir: /path/to/new_run
  159. ```
  160. Start a new run from an existing Hugging Face checkpoint. `--input` accepts a Hugging Face Hub repo id (downloaded automatically), a local HF/safetensors directory or `.safetensors` file, or a TorchSpec DCP checkpoint dir:
  161. ```bash
  162. python tools/convert_to_torchspec.py \
  163. --input org/dflash-checkpoint \
  164. --config torchspec/config/dspark_draft_config_qwen36_35b.json \
  165. --output ./outputs/dspark_init
  166. ```
  167. Then warm-start from the generated init:
  168. ```yaml
  169. training:
  170. load_path: ./outputs/dspark_init
  171. continual_training: true
  172. ```
  173. Using this technique, you can also warm-start DSpark training runs from a pre-trained DFlash model for faster convergence.
  174. ## Checkpoint Conversion
  175. Convert an FSDP checkpoint to HuggingFace format:
  176. ```bash
  177. python tools/convert_to_hf.py --input-dir ./outputs/my_experiment/iter_0010000/
  178. ```
  179. Vocabulary pruning, which reduces the draft model `lm_head` to a smaller token set and emits `d2t` and `t2d` mappings, can be applied either during training or at conversion time.
  180. DFlash2 requires the full target vocabulary. Do not set `draft_vocab_size` or pass `--prune-vocab` for a DFlash2 model.
  181. - **Pre-pruning**: set `draft_vocab_size` in your training config. The checkpoint already contains the pruned `lm_head` and `d2t`/`t2d` buffers, so the basic conversion command is enough.
  182. - **Post-pruning**: train with the full vocabulary, then pass `--prune-vocab` at conversion time together with a representative dataset to compute token frequencies.
  183. ```bash
  184. python tools/convert_to_hf.py \
  185. --input-dir ./outputs/my_experiment/iter_0010000/ \
  186. --prune-vocab \
  187. --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
  188. --draft-vocab-size 32000 \
  189. --tokenizer Qwen/Qwen3-8B \
  190. --chat-template qwen \
  191. --prompt-key conversations
  192. ```
  193. Pass `--cache-dir ./cache` to reuse the tokenized dataset cache from training.
  194. ## Metrics Reporting
  195. W&B logging is disabled by default with `report_to: none`. To enable it, set `report_to: wandb` in your config and provide your API key.
  196. ## Troubleshooting
  197. Set `TORCHSPEC_LOG_LEVEL=DEBUG` for more verbose logging when diagnosing issues:
  198. ```bash
  199. TORCHSPEC_LOG_LEVEL=DEBUG ./examples/qwen3-8b-single-node/run.sh
  200. ```
  201. ### Mooncake SEGFAULT
  202. Current Mooncake version has a bug with TCP-only hosts causing a SEGFAULT error. Set `MC_STORE_MEMCPY=0` until the [upstream issue](https://github.com/kvcache-ai/Mooncake/issues/1986) is fixed.
  203. ### RDMA failure
  204. If you get the error `... libcudart symbols not found globally. Make sure PyTorch with CUDA is installed before using TileLang`, reinstall mooncake to match your system CUDA version, i.e. if you run on CUDA 13, mooncake is probably installed for cu12 wheels, re-install it using `uv pip uninstall mooncake-transfer-engine && uv pip install mooncake-transfer-engine-cuda13`.
  205. ### Per-Rank File Logging
  206. Set `TORCHSPEC_LOG_DIR` to an absolute path on a shared filesystem (NFS) to enable per-rank log files for every Ray actor on both training and inference:
  207. ```bash
  208. export TORCHSPEC_LOG_DIR=/my_project/running_logs
  209. ```
  210. This creates a structured directory with one file per actor, organized by role and node:
  211. ```text
  212. running_logs/
  213. training/
  214. 10.0.0.1/
  215. training_g0_rank0_20260301_080012.log
  216. training_g0_rank1_20260301_080012.log
  217. 10.0.0.2/
  218. training_g0_rank2_20260301_080013.log
  219. inference/
  220. 10.0.0.1/
  221. inference_g0_rank0_20260301_080014.log
  222. 10.0.0.2/
  223. inference_g0_rank1_20260301_080015.log
  224. ```
  225. The path must be absolute and writable from all nodes. If `TORCHSPEC_LOG_DIR` is unset or not writable, per-rank file logging stays disabled and Ray falls back to stdout/stderr capture.
  226. | Issue | Reference |
  227. |-------|-----------|
  228. | Stuck or failing distributed runs, Ray actor errors | [docs/debugging_ray_jobs.md](docs/debugging_ray_jobs.md) |
  229. | Ray cluster setup, actor hierarchy, placement groups | [docs/ray.md](docs/ray.md) |
  230. | Pipeline bottlenecks, slow steps, throughput analysis | [docs/performance_metrics.md](docs/performance_metrics.md) |

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.