Skip to content
Back to search

glm-flash-lite

GLM-5.3-Flash EXL3 on one 24 GB RTX 3090 + DDR4: elastic GPU expert cache, zero-copy experts, AVX2 CPU tier, OpenAI API

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/sybil-solutions--glm-flash-lite/badge.svg)](/listings/sybil-solutions--glm-flash-lite)

HTML

<a href="/listings/sybil-solutions--glm-flash-lite"><img src="/api/listings/sybil-solutions--glm-flash-lite/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 176 lines
  1. # glm-flash-lite
  2. Run **GLM-5.3-Flash** (125 GB, EXL3 3.05 bpw) on **one 24 GB RTX 3090** with an OpenAI-compatible API, using RAM and
  3. NVMe for the experts that do not fit on the GPU.
  4. | RAM for the server | mode | decode, 1 user | prefill 8k / 32k | output vs stock exllamav3 |
  5. |---|---|---|---|---|
  6. | 218 GiB free | `fast` | 28.2 tok/s | 710 / 951 tok/s | near-exact (decode KL 0.005) |
  7. | **55 GiB cap** + NVMe | **`nvme`** | **19.5 tok/s** | 664 / 969 tok/s | near-exact (decode KL 0.008) |
  8. | 16 GiB cap + NVMe | `nvme` | 14.7 tok/s (lab; 14.38 sweep) | 650 / not run | near-exact (same CPU lane) |
  9. | 55 GiB cap + NVMe | `nvme-exact` | 8.3 tok/s | 564 / 806 tok/s | bit-exact |
  10. | 55 GiB cap + NVMe + 1x Arc Pro B70 | `nvme` + `GLM53_B70=1` | 26.5 tok/s | 658 / 968 tok/s | near-exact (decode KL 0.003) |
  11. Measured on 1x RTX 3090 (PCIe 4.0 x16), AMD EPYC 7443P (24 cores), 8-channel DDR4, and 4x Samsung 9100 PRO in RAID0.
  12. Decode runs each answer to its natural end. Full tables, dates and raw files: [docs/results.md](docs/results.md).
  13. Image: ghcr.io/sybil-solutions/glm53-flash-offload (name kept for existing installs)
  14. ## Run it
  15. **Omarchy Local AI.** Run the line below. In Local AI, pick **GLM-5.3-Flash, EXL3 3.05 bpw (55 GB RAM, experts on
  16. NVMe)** and press **Start**. The plugin downloads the weights, builds the store and starts the server.
  17. ```bash
  18. omarchy plugin add https://github.com/sybil-solutions/omarchy-local-ai --enable
  19. ```
  20. **Docker.** Needs the NVIDIA container toolkit, the `hf` CLI, 125 GB for the weights, and 117 GB for the expert store
  21. on a fast local NVMe filesystem (xfs or ext4).
  22. ```bash
  23. IMG=ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 # v4.6-nvme
  24. hf download turboderp/GLM-5.3-Flash-exl3 --revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 --local-dir /data/glm53
  25. docker run --rm -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx "$IMG" pack-store # once: 117 GB expert store
  26. docker run -d --name glm53 --gpus '"device=0"' --memory 55g --memory-swap 55g --shm-size 1g --ulimit memlock=-1 \
  27. -e GLM53_MODE=nvme -e GLM53_EC_MAX_SLOTS=1376 -p 127.0.0.1:30000:30000 -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx:ro "$IMG"
  28. ```
  29. The server is ready in about 80 s at `http://127.0.0.1:30000/v1`, model `glm-5.3-flash`. For 16 GB of RAM use
  30. `--memory 16g --memory-swap 16g`. For bit-exact output use `GLM53_MODE=nvme-exact`. The all-RAM `fast` mode uses an
  31. older image; see [docs/results.md](docs/results.md).
  32. ## Supported recipes
  33. Every [local-ai-registry](https://github.com/sybil-solutions/local-ai-registry) recipe whose launch image is
  34. `ghcr.io/sybil-solutions/glm53-flash-offload` or `ghcr.io/0xsero/glm53-flash-offload`, single-GPU only. Only one
  35. hardware card has such recipes: the RTX 3090 24 GB. The GLM-5.3-Flash recipes on other cards (DGX Spark GB10,
  36. RTX PRO 6000 Blackwell, CMP 170HX, Apple M5 Ultra) use other engines and images (vLLM, SGLang, oMLX). No recipe runs
  37. this image on an Arc Pro B70 alone.
  38. Status: **accepted** = the registry lab passed all six gates (load, chat, reasoning, tools, context, speed) on that
  39. digest; **reported** = the lab ran it and a gate failed, or only the owner's campaign measured it; **candidate** = not
  40. in the plugin catalog. Lab rows: prefill is the context gate (one 93,437-token prompt, tokens / request time), decode is
  41. the speed gate (one stream, first 30 s, natural completion, gate 15 tok/s). Sweep rows: `bench/sweep.py`, see
  42. [docs/results.md](docs/results.md). "agg" is the aggregate over all streams.
  43. ### RTX 3090 24 GB
  44. | prefill size | prefill speed | decode speed | concurrency | kv cache | gpu count | recipe | RAM / storage | status | image digest | source |
  45. |---|---|---|---|---|---|---|---|---|---|---|
  46. | 93,437 | 1,015 tok/s | 20.0 tok/s | 1 | 131,072 | 1 | [55 GB + NVMe][r55] | 55 GiB cap; 242.6 GB NVMe | accepted | `aa74200f` (v4.6-nvme) | [lab run 2026-10-09][l55] |
  47. | 8,192 | 664 tok/s | 19.47 tok/s | 1 | 131,072 | 1 | [55 GB + NVMe][r55] | 55 GiB cap; 242.6 GB NVMe | accepted | `aa74200f` (v4.6-nvme) | [sweep N129][res] |
  48. | 8,192 | not run | 19.09 tok/s agg | 2 | 131,072 | 1 | [55 GB + NVMe][r55] | 55 GiB cap; 242.6 GB NVMe | accepted | `aa74200f` (v4.6-nvme) | [sweep N129][res] |
  49. | 8,192 | not run | 19.75 tok/s agg | 4 | 131,072 | 1 | [55 GB + NVMe][r55] | 55 GiB cap; 242.6 GB NVMe | accepted | `aa74200f` (v4.6-nvme) | [sweep N129][res] |
  50. | 32,768 | 969 tok/s | 18.30 tok/s | 1 | 131,072 | 1 | [55 GB + NVMe][r55] | 55 GiB cap; 242.6 GB NVMe | accepted | `aa74200f` (v4.6-nvme) | [sweep N129][res] |
  51. | 32,768 | not run | 18.44 tok/s agg | 2 | 131,072 | 1 | [55 GB + NVMe][r55] | 55 GiB cap; 242.6 GB NVMe | accepted | `aa74200f` (v4.6-nvme) | [sweep N129][res] |
  52. | 93,437 | 1,010 tok/s | 14.7 tok/s | 1 | 131,072 | 1 | [16 GB + NVMe][r16] | 16 GiB cap; 242.6 GB NVMe | reported (speed gate 15) | `aa74200f` (v4.6-nvme) | [lab run 2026-10-09][l16] |
  53. | 8,192 | 650 tok/s | 14.38 tok/s | 1 | 131,072 | 1 | [16 GB + NVMe][r16] | 16 GiB cap; 242.6 GB NVMe | reported | `aa74200f` (v4.6-nvme) | [sweep N129][res] |
  54. | 8,192 | not run | not run | 2 | 131,072 | 1 | [16 GB + NVMe][r16] | 16 GiB cap; 242.6 GB NVMe | reported | `aa74200f` (v4.6-nvme) | not run |
  55. | 8,192 | not run | not run | 4 | 131,072 | 1 | [16 GB + NVMe][r16] | 16 GiB cap; 242.6 GB NVMe | reported | `aa74200f` (v4.6-nvme) | not run |
  56. | 32,768 | not run | not run | 1 | 131,072 | 1 | [16 GB + NVMe][r16] | 16 GiB cap; 242.6 GB NVMe | reported | `aa74200f` (v4.6-nvme) | not run |
  57. | 32,768 | not run | not run | 2 | 131,072 | 1 | [16 GB + NVMe][r16] | 16 GiB cap; 242.6 GB NVMe | reported | `aa74200f` (v4.6-nvme) | not run |
  58. | 93,437 | 989 tok/s | 28.6 tok/s | 1 | 131,072 | 1 | [all experts in RAM][rram] | ~238 GB free RAM; 125.3 GB NVMe | accepted | `c5a58a40` (v4.4-nvme, `fast`) | [lab run 2026-10-09][lram] |
  59. | 8,192 | 710 tok/s | 28.15 tok/s | 1 | 131,072 | 1 | [all experts in RAM][rram] | ~238 GB free RAM; 125.3 GB NVMe | accepted | `bb633b0b` (previous image) | [run G067][res] |
  60. | 8,192 | not run | 31.57 tok/s agg | 2 | 131,072 | 1 | [all experts in RAM][rram] | ~238 GB free RAM; 125.3 GB NVMe | accepted | `bb633b0b` (previous image) | [run G067][res] |
  61. | 8,192 | not run | 33.98 tok/s agg | 4 | 131,072 | 1 | [all experts in RAM][rram] | ~238 GB free RAM; 125.3 GB NVMe | accepted | `bb633b0b` (previous image) | [run G067][res] |
  62. | 32,768 | 951 tok/s | 26.89 tok/s | 1 | 131,072 | 1 | [all experts in RAM][rram] | ~238 GB free RAM; 125.3 GB NVMe | accepted | `bb633b0b` (previous image) | [run G067][res] |
  63. | 32,768 | not run | not run | 2 | 131,072 | 1 | [all experts in RAM][rram] | ~238 GB free RAM; 125.3 GB NVMe | accepted | `bb633b0b` (previous image) | not run |
  64. - RAM / storage: the NVMe recipes run in a container capped at 55 or 16 GiB (the registry asks for 59.1 / 17.2 GB
  65. free) and need 242.6 GB on a local NVMe filesystem with O_DIRECT (xfs, ext4): 125.3 GB of weights plus the 117.3 GB
  66. expert store from `pack-store`. All three need an x86-64 CPU with AVX2+FMA+F16C; the NVMe recipes ask for 40 or more
  67. logical CPUs. The all-RAM recipe needs ~238 GB of free RAM and the 125.3 GB of weights on local NVMe.
  68. - The `fast` sweep rows ran on the previous image `bb633b0b`. The full sweep was not re-run on `c5a58a40`, the digest
  69. the recipe pins; only the lab run was.
  70. - Image digests: v4.6-nvme `sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69` (55 GB recipe
  71. accepted in local-ai-registry PR [#199](https://github.com/sybil-solutions/local-ai-registry/pull/199));
  72. v4.4-nvme `sha256:c5a58a40bd93e57227d3986b81e9648d422c465a3ff00e14f17dad65dadf2310`;
  73. previous all-RAM image `ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0bcb85573ad40b6c408af5e1036ae062c593e42479f4e4d2ab521e551d`.
  74. **Hardware cost** (prices dated 2026-10-08, from the earlier campaign): the cheapest build is the 16 GB RTX 3090
  75. build at ~$5.0k, about $340 per tok/s at its 14.7 tok/s lab decode (reported, below the 15 tok/s gate). The 55 GB build
  76. is ~$5.4k, about $270 per tok/s at its accepted 20.0 tok/s.
  77. **Experimental, not supported** (multi-GPU or companion): [glm-5.3-flash.exllamav3-nvme-b70][rb70], the 55 GB NVMe
  78. recipe on an RTX 3090 plus an Intel Arc Pro B70 running the `glm53-b70-expert-server` companion
  79. (`sha256:8e95f5f9f5e2373d5a80c556cdbc24ae5be8e63dd9f1520bdeb0bc0d02bc5fc5`). It is a candidate with a reported proof
  80. from the owner's campaign N137 (26.5 tok/s C1 decode, 968 tok/s 32k prefill), not lab-run, and the plugin cannot start
  81. companions yet. See [docs/b70-tier.md](docs/b70-tier.md).
  82. [r55]: https://github.com/sybil-solutions/local-ai-registry/blob/main/registry/recipes/nvidia/rtx-3090-24gb/glm-5.3-flash.exllamav3-nvme.128k.json
  83. [r16]: https://github.com/sybil-solutions/local-ai-registry/blob/main/registry/recipes/nvidia/rtx-3090-24gb/glm-5.3-flash.exllamav3-nvme16g.128k.json
  84. [rram]: https://github.com/sybil-solutions/local-ai-registry/blob/main/registry/recipes/nvidia/rtx-3090-24gb/glm-5.3-flash.exllamav3.128k.json
  85. [rb70]: https://github.com/sybil-solutions/local-ai-registry/blob/main/registry/recipes/nvidia/rtx-3090-24gb/glm-5.3-flash.exllamav3-nvme-b70.128k.json
  86. [l55]: https://github.com/sybil-solutions/local-ai-registry/blob/main/lab/runs/rtx-3090-24gb.glm-5.3-flash.exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-55g-128k-rtx-3090-24gb.128k.20261009T123730.json
  87. [l16]: https://github.com/sybil-solutions/local-ai-registry/blob/main/lab/runs/rtx-3090-24gb.glm-5.3-flash.exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-16g-128k-rtx-3090-24gb.128k.20261009T124547.json
  88. [lram]: https://github.com/sybil-solutions/local-ai-registry/blob/main/lab/runs/rtx-3090-24gb.glm-5.3-flash.exllamav3-glm-5.3-flash-exl3-3.05bpw-offload-128k-rtx-3090-24gb.128k.20261009T064723.json
  89. [res]: docs/results.md
  90. ## How it works
  91. The model has 12,096 routed experts (42 MoE layers x 288), 9.4 MB each, and each token uses 8 per layer. Each expert
  92. is kept in one of three tiers. Two lanes compute them at the same time:
  93. ```text
  94. GPU lane CPU lane
  95. +-------------------------------+ +---------------------------+
  96. | VRAM expert cache, ~1,300 | | 22 AVX2 threads compute |
  97. | hottest experts (12.4 GB) | | RAM experts in place |
  98. +---------------^---------------+ +-------------^-------------+
  99. | PCIe copy when admitted | reads
  100. +---------------+--------------------------------------------+-------------+
  101. | RAM tier, pinned, sized from --memory: ~4,800 experts at 55 GiB |
  102. | holds only experts that are NOT in VRAM |
  103. +---------------^----------------------------------------------------------+
  104. | O_DIRECT reads, deep queue, layer-ahead prefetch
  105. +---------------+----------------------------------------------------------+
  106. | NVMe store: every expert, 117 GB, one 4K-aligned record each |
  107. +--------------------------------------------------------------------------+
  108. ```
  109. - **Expert cache in VRAM.** The VRAM left after load holds the hottest experts. exllamav3's own kernels read them
  110. through repointed tables. On the all-RAM path, cache plus direct reads took decode from 7.5-8.4 tok/s (stock
  111. exllamav3) to 12.6.
  112. - **RAM tier with no duplicates.** An expert evicted from VRAM moves into RAM, so VRAM and RAM never hold the same
  113. expert. 55 GiB of RAM then covers ~6,100 different experts instead of ~4,800. Per token at C1: ~120 picks hit
  114. VRAM, ~175 hit RAM and ~23 come from NVMe.
  115. - **NVMe record store.** Each expert is one 4K-aligned record, read with `O_DIRECT` (no page cache) by 16 threads
  116. across a 4-drive RAID0. Capping reads at 8 GB/s cost 8 % of C1 decode, 33 % of C4 and 27 % of prefill.
  117. - **CPU tier.** Cold experts already in RAM are computed by the CPU where they sit, instead of being copied over PCIe.
  118. This runs in parallel with the GPU, which adds the CPU's partial result back. Adding it, together with prefetch,
  119. took NVMe decode from 8.3 to 15.3 tok/s, at a decode KL of 0.005.
  120. - **Host engine without stalls.** One controller thread plans each layer, issues the reads and answers the GPU
  121. through mapped memory. The GPU waits on per-expert "landed" flags, with no host sync per layer. Together with the
  122. no-duplicates RAM tier, this took exact decode from 7.3 to 8.3 tok/s.
  123. - **Layer-ahead prefetch.** The next layer's router runs on the current layer's input, and its NVMe experts are read
  124. one layer early. NVMe misses fall from 40 to 22 per token, and it is 13 % faster in the same session. An earlier
  125. "prefetch off is faster" result was drift between sessions.
  126. - **Victim ring.** Evicted experts are parked in a 24-slot VRAM ring. A copy engine drains them to RAM off the
  127. critical path (~7 GB/s of write-backs at C1).
  128. **Where the time goes now** (v4.6, 55 GiB, C1, 55.8 ms per token): copying admitted experts to VRAM 31 %, non-MoE GPU
  129. work 20 % (KDA/DSA attention GEMVs at 50-57 % of VRAM bandwidth), the MoE kernel 18 %, waiting for the CPU lane 18 %,
  130. NVMe waits 10 %, host 2 % (the decode lookahead removed the step-boundary gaps). No resource is saturated: the GPU does
  131. real work 70 % of the time, the CPU lane 60 %, PCIe and NVMe run at under 40 % of their ceilings, because every layer
  132. waits on its slowest lane. Better scheduling alone would reach about 20 tok/s; every lane at its hardware floor about
  133. 41. Profile: moetier `docs/latency-glm53-v46.md`. Details: [docs/how-it-works.md](docs/how-it-works.md).
  134. ## Tune it for your machine
  135. Point your coding agent (Claude Code, Codex, others) at
  136. [`skills/glm53-offload-setup/SKILL.md`](skills/glm53-offload-setup/SKILL.md). It checks your GPU, RAM, CPU and NVMe,
  137. picks the mode, builds the store, measures, and tunes one setting at a time.
  138. ## More
  139. - [docs/results.md](docs/results.md): every measured table, per mode, with dates and raw files
  140. - [docs/how-it-works.md](docs/how-it-works.md): the mechanisms, how each tier is sized, and every setting
  141. - [docs/b70-tier.md](docs/b70-tier.md): optional Intel Arc Pro B70 as a second expert tier (`GLM53_B70=1`)
  142. - [docs/troubleshooting.md](docs/troubleshooting.md): error messages and fixes
  143. - [docs/reference.md](docs/reference.md): the full campaign record (image smokes, quality method, reproduction pins)
  144. Credits: [exllamav3](https://github.com/turboderp-org/exllamav3) and the
  145. [EXL3 checkpoint](https://huggingface.co/turboderp/GLM-5.3-Flash-exl3) by turboderp; FreeToken (FlashML) for the
  146. host-memory expert tier idea; [SGLang](https://github.com/sgl-project/sglang) as the base image; GLM-5.3-Flash by
  147. Z.ai. MIT license (this repository); the weights carry their own license.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.