Skip to content
Back to search

LightLLM

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/ModelTC/LightLLM with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/ModelTC/LightLLM'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/modeltc--lightllm/badge.svg)](/listings/modeltc--lightllm)

HTML

<a href="/listings/modeltc--lightllm"><img src="/api/listings/modeltc--lightllm/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 126 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. <picture>
  3. <img alt="LightLLM" src="assets/logo_new.png" width=90%>
  4. </picture>
  5. </div>
  6. ---
  7. <div align="center">
  8. [![docs](https://img.shields.io/badge/docs-latest-blue)](https://lightllm-en.readthedocs.io/en/latest/)
  9. [![Docker](https://github.com/ModelTC/lightllm/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/ModelTC/lightllm/actions/workflows/docker-publish.yml)
  10. [![stars](https://img.shields.io/github/stars/ModelTC/lightllm?style=social)](https://github.com/ModelTC/lightllm)
  11. ![visitors](https://komarev.com/ghpvc/?username=lightllm&label=visitors)
  12. [![Discord Banner](https://img.shields.io/discord/1139835312592392214?logo=discord&logoColor=white)](https://discord.gg/WzzfwVSguU)
  13. [![license](https://img.shields.io/github/license/ModelTC/lightllm)](https://github.com/ModelTC/lightllm/blob/main/LICENSE)
  14. </div>
  15. LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. LightLLM harnesses the strengths of numerous well-regarded open-source implementations, including but not limited to FasterTransformer, TGI, vLLM, and FlashAttention.
  16. [English Docs](https://lightllm-en.readthedocs.io/en/latest/) | [中文文档](https://lightllm-cn.readthedocs.io/en/latest/) | [Blogs](https://modeltc.github.io/lightllm-blog/)
  17. ## Tech Blogs
  18. - [2025/11] 🚀 Prefix KV Cache Transfer between DP rankers is now supported! Check out the technical deep dive in our [blog post](https://light-ai.top/lightllm-blog/2025/11/18/dp_kv_fetch.html).
  19. ## News
  20. - [2025/09] 🔥 LightLLM [v1.1.0](https://www.light-ai.top/lightllm-blog/2025/09/03/lightllm.html) release!
  21. - [2025/08] Pre $^3$ achieves the outstanding paper award of [ACL2025](https://2025.aclweb.org/program/awards/).
  22. - [2025/05] LightLLM paper on constrained decoding accepted by [ACL2025](https://arxiv.org/pdf/2506.03887) (Pre $^3$: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation). For a more accessible overview of the research with key insights and examples, check out our blog post: [LightLLM Blog](https://www.light-ai.top/lightllm-blog/2025/06/15/pre3.html)
  23. - [2025/04] LightLLM paper on request scheduler published in [ASPLOS’25](https://dl.acm.org/doi/10.1145/3676641.3716011) (Past-Future Scheduler for LLM Serving under SLA Guarantees)
  24. - [2025/02] 🔥 LightLLM v1.0.0 release, achieving the **fastest DeepSeek-R1** serving performance on single H200 machine.
  25. ## Get started
  26. - [Install LightLLM](https://lightllm-en.readthedocs.io/en/latest/getting_started/installation.html)
  27. - [Quick Start](https://lightllm-en.readthedocs.io/en/latest/getting_started/quickstart.html)
  28. - [TuTorial](https://lightllm-en.readthedocs.io/en/latest/tutorial/deepseek_deployment.html)
  29. ## Performance
  30. Learn more in the release blogs: [v1.1.0 blog](https://www.light-ai.top/lightllm-blog/2025/09/03/lightllm.html).
  31. ## FAQ
  32. Please refer to the [FAQ](https://lightllm-en.readthedocs.io/en/latest/faq.html) for more information.
  33. ## Projects using LightLLM
  34. We welcome any coopoeration and contribution. If there is a project requires LightLLM's support, please contact us via email or create a pull request.
  35. Projects based on LightLLM or referenced LightLLM components:
  36. - [LoongServe, Peking University](https://github.com/LoongServe/LoongServe)
  37. - [vLLM](https://github.com/vllm-project/vllm) (some LightLLM's kernel used)
  38. - [SGLang](https://github.com/sgl-project/sglang) (some LightLLM's kernel used)
  39. - [ParrotServe](https://github.com/microsoft/ParrotServe), Microsoft
  40. - [Aphrodite](https://github.com/aphrodite-engine/aphrodite-engine) (some LightLLM's kernel used)
  41. - [S-LoRA](https://github.com/S-LoRA/S-LoRA)
  42. - [OmniKV, Ant Group](https://github.com/antgroup/OmniKV)
  43. - [Lab4AI LightLLM+LlamaIndex](https://www.lab4ai.cn/project/detail?utm_source=LLM1&id=b417085ae8cd4dd0bef7161c3d583b15&type=project), [Lab4AI LightLLM+Qwen3-8B](https://www.lab4ai.cn/project/detail?utm_source=lightllmcapp&id=c98ff5d09528423d8dd06f5a063cb2a6&type=project)
  44. - [LazyLLM](https://github.com/LazyAGI/LazyLLM)
  45. Also, LightLLM's pure-python design and token-level KC Cache management make it easy to use as the basis for research projects.
  46. Academia works based on or use part of LightLLM:
  47. - [ParrotServe (OSDI’24)](https://www.usenix.org/conference/osdi24/presentation/lin-chaofan)
  48. - [SLoRA (MLSys’24)](https://proceedings.mlsys.org/paper_files/paper/2024/hash/906419cd502575b617cc489a1a696a67-Abstract-Conference.html)
  49. - [LoongServe (SOSP’24)](https://dl.acm.org/doi/abs/10.1145/3694715.3695948)
  50. - [ByteDance’s CXL (Eurosys’24)](https://dl.acm.org/doi/10.1145/3627703.3650061)
  51. - [VTC (OSDI’24)](https://www.usenix.org/conference/osdi24/presentation/sheng)
  52. - [OmniKV (ICLR’25)](https://openreview.net/forum?id=ulCAPXYXfa)
  53. - [CaraServe](https://arxiv.org/abs/2401.11240), [LoRATEE](https://ieeexplore.ieee.org/abstract/document/10890445), [FastSwitch](https://arxiv.org/abs/2411.18424) ...
  54. ## Community
  55. For further information and discussion, [join our discord server](https://discord.gg/WzzfwVSguU). Welcome to be a member and look forward to your contribution!
  56. ## License
  57. This repository is released under the [Apache-2.0](LICENSE) license.
  58. ## Acknowledgement
  59. We learned a lot from the following projects when developing LightLLM.
  60. - [Faster Transformer](https://github.com/NVIDIA/FasterTransformer)
  61. - [Text Generation Inference](https://github.com/huggingface/text-generation-inference)
  62. - [vLLM](https://github.com/vllm-project/vllm)
  63. - [SGLang](https://github.com/sgl-project/sglang)
  64. - [flashinfer](https://github.com/flashinfer-ai/flashinfer/tree/main)
  65. - [Flash Attention 1&2](https://github.com/Dao-AILab/flash-attention)
  66. - [OpenAI Triton](https://github.com/openai/triton)
  67. ## Citation
  68. We have published a number of papers around components or features of LightLLM, if you use LightLLM in your work, please consider citing the relevant paper.
  69. **constrained decoding**: accepted by [ACL2025](https://arxiv.org/pdf/2506.03887) and achieved the outstanding paper award.
  70. ```bibtex
  71. @inproceedings{chen-etal-2025-pre3,
  72. title = "Pre$^3$: Enabling Deterministic Pushdown Automata for Faster Structured {LLM} Generation",
  73. author = "Chen, Junyi and Bai, Shihao and Wang, Zaijun and Wu, Siyu and Du, Chuheng and Yang, Hailong and Gong, Ruihao and Liu, Shengzhong and Wu, Fan and Chen, Guihai",
  74. editor = "Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher",
  75. booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
  76. month = jul,
  77. year = "2025",
  78. address = "Vienna, Austria",
  79. publisher = "Association for Computational Linguistics",
  80. url = "https://aclanthology.org/2025.acl-long.551/",
  81. doi = "10.18653/v1/2025.acl-long.551",
  82. pages = "11253--11267",
  83. ISBN = "979-8-89176-251-0",
  84. }
  85. ```
  86. **Request scheduler**: accepted by [ASPLOS’25](https://dl.acm.org/doi/10.1145/3676641.3716011):
  87. ```bibtex
  88. @inproceedings{gong2025past,
  89. title={Past-Future Scheduler for LLM Serving under SLA Guarantees},
  90. author={Gong, Ruihao and Bai, Shihao and Wu, Siyu and Fan, Yunqian and Wang, Zaijun and Li, Xiuhong and Yang, Hailong and Liu, Xianglong},
  91. booktitle={Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2},
  92. pages={798--813},
  93. year={2025}
  94. }
  95. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.