Skip to content
Back to search

mlx-vlm

MarketplaceWorks with Claude CodeGitHub

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Neywa-Labs/mlx-vlm with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Neywa-Labs/mlx-vlm'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/blaizzy--mlx-vlm/badge.svg)](/listings/blaizzy--mlx-vlm)

HTML

<a href="/listings/blaizzy--mlx-vlm"><img src="/api/listings/blaizzy--mlx-vlm/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

The easiest way to get started is to install the `mlx-vlm` package using pip:

pip install -U mlx-vlm

The [Gradio chat UI](#chat-ui-with-gradio) needs an extra dependency that is not part of the base install:

pip install -U 'mlx-vlm[ui]'

Quote the package name so that shells which expand square brackets, such as `zsh`, do not treat `[ui]` as a glob pattern.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 18 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "mlx-vlm",
  3. "metadata": {
  4. "description": "Agent skills for MLX-VLM: command-line and server inference, convert/quantize, adding new models, benchmarking, contributing, cached-model listing, and reproducible GitHub issues.",
  5. "version": "0.1.0"
  6. },
  7. "owner": {
  8. "name": "MLX-VLM contributors"
  9. },
  10. "plugins": [
  11. {
  12. "name": "mlx-vlm-skills",
  13. "description": "Skills for MLX-VLM: command-line and server inference, convert/quantize, adding new models, benchmarking, contributing, cached-model listing, and reproducible GitHub issues.",
  14. "version": "0.1.0",
  15. "source": "./skills"
  16. }
  17. ]
  18. }
README.md · README · 2,042 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <h1 align="center">MLX-VLM</h1>
  2. [![Any input. Any output. All local. Runs locally on your Mac. Install with pip install -U mlx-vlm or try it in Nativ.](https://raw.githubusercontent.com/Neywa-Labs/mlx-vlm/b7fda105064b609fcc2ef243bd628c3dad317564/docs/assets/mlx-vlm-banner.png)](https://runnativ.ai)
  3. <p align="center">
  4. <strong>Run and fine-tune multimodal models on Apple Silicon.</strong> Text, images, video, and audio in — any out. Built on MLX.
  5. </p>
  6. <p align="center">
  7. <a href="#installation"><strong>Installation</strong></a> &middot;
  8. <a href="https://neywa-labs.github.io/mlx-vlm/">Documentation</a> &middot;
  9. <a href="https://github.com/Neywa-Labs/mlx-vlm/releases">Releases</a>
  10. </p>
  11. <p align="center">
  12. macOS &middot; Apple silicon
  13. </p>
  14. <p align="center">
  15. Built by <a href="https://www.neywalabs.ai">Neywa Labs</a>, the local AI company behind <a href="https://runnativ.ai">Nativ</a>.
  16. </p>
  17. <p align="center">
  18. <a href="https://github.com/Neywa-Labs/mlx-vlm/actions/workflows/python-publish.yml"><img src="https://github.com/Neywa-Labs/mlx-vlm/actions/workflows/python-publish.yml/badge.svg" alt="Upload Python Package"></a>
  19. <a href="https://pypi.org/project/mlx-vlm/"><img src="https://img.shields.io/pypi/v/mlx-vlm" alt="PyPI version"></a>
  20. <a href="https://pypi.org/project/mlx-vlm/"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a>
  21. <a href="https://github.com/Neywa-Labs/mlx-vlm/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Neywa-Labs/mlx-vlm" alt="MIT license"></a>
  22. </p>
  23. ## Table of Contents
  24. - [Installation](#installation)
  25. - [Usage](#usage)
  26. - [Command Line Interface (CLI)](#command-line-interface-cli)
  27. - [Thinking Budget](#thinking-budget)
  28. - [Speculative Decoding](#speculative-decoding)
  29. - [DFlash, DFlash2, and DSpark](#dflash-dflash2-and-dspark)
  30. - [Bundled native heads](#bundled-native-heads)
  31. - [Gemma 4 MTP](#gemma-4-mtp)
  32. - [Gemma 4 EAGLE-3](#gemma-4-eagle-3)
  33. - [MiniMax M3 EAGLE-3](#minimax-m3-eagle-3)
  34. - [Chat UI with Gradio](#chat-ui-with-gradio)
  35. - [Python Script](#python-script)
  36. - [Server (FastAPI)](#server-fastapi)
  37. - [Continuous Batching](#continuous-batching)
  38. - [Automatic Prefix Caching (APC)](#automatic-prefix-caching-apc)
  39. - [KV Cache Quantization](#kv-cache-quantization)
  40. - [1-bit Affine Inference](#1-bit-affine-inference)
  41. - [Activation Quantization (CUDA)](#activation-quantization-cuda)
  42. - [Multi-Image Chat Support](#multi-image-chat-support)
  43. - [Supported Models](#supported-models)
  44. - [Usage Examples](#usage-examples)
  45. - [Model-Specific Documentation](#model-specific-documentation)
  46. - [Vision Feature Caching](#vision-feature-caching)
  47. - [TurboQuant KV Cache](#turboquant-kv-cache)
  48. - [Distributed Inference](#distributed-inference)
  49. - [Fine-tuning](#fine-tuning)
  50. ## Model-Specific Documentation
  51. Some models have detailed documentation with prompt formats, examples, and best practices:
  52. | Model | Documentation |
  53. |-------|---------------|
  54. | Inkling 2 Small | [Docs](mlx_vlm/models/inkling2/README.md) |
  55. | LFM3 / LFM3-VL | [Docs](mlx_vlm/models/lfm3/README.md) |
  56. | SenseNova-Loop-9B | [Docs](mlx_vlm/models/sensenova_loop/README.md) |
  57. | LFM3-Audio | [Docs](mlx_vlm/models/lfm3_audio/README.md) |
  58. | LFM3-Omni | [Docs](mlx_vlm/models/lfm3_omni/README.md) |
  59. | DeepSeek-OCR | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/deepseekocr/README.md) |
  60. | DeepSeek-OCR-2 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/deepseekocr_2/README.md) |
  61. | Unlimited-OCR | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/unlimited_ocr/README.md) |
  62. | DOTS-OCR | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/dots_ocr/README.md) |
  63. | DOTS-MOCR | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/dots_ocr/README.md) |
  64. | ERNIE 5.5 Flash Base (text) | [Docs](mlx_vlm/models/ernie5_5_moe/README.md) |
  65. | ERNIE 4.5 VL | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/ernie4_5_moe_vl/README.md) |
  66. | GLM-OCR | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/glm_ocr/README.md) |
  67. | Phi-4 Reasoning Vision | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/phi4_siglip/README.md) |
  68. | MiniCPM-o | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/minicpmo/README.md) |
  69. | PaddleOCR-VL | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/paddleocr_vl/README.md) |
  70. | OXR | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/oxr/README.md) |
  71. | Phi-4 Multimodal | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/phi4mm/README.md) |
  72. | MolmoPoint | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/molmo_point/README.md) |
  73. | LocateAnything | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/locateanything/README.md) |
  74. | Moondream2 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/moondream2/README.md) |
  75. | Moondream3 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/moondream3/README.md) |
  76. | EmbeddingGemma 2 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/embedding_gemma2/README.md) |
  77. | Gemma 4 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/gemma4/README.md) |
  78. | Muse Spark Open | [Docs](mlx_vlm/models/muse_spark/README.md) |
  79. | MiniMax M3 / M3.1 Flash | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/minimax_m3_vl/README.md) |
  80. | MiniMax-H3-Image | [Docs](mlx_vlm/models/minimax_h3_image/README.md) |
  81. | Ling 3.1 Flash | [Docs](mlx_vlm/models/bailing_hybrid/README.md) |
  82. | Falcon-OCR | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/falcon_ocr/README.md) |
  83. | IndicOCR | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/indic_ocr/README.md) |
  84. | PP-DocLayoutV3 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/pp_doclayout_v3/README.md) |
  85. | Granite Vision 3.2 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/granite_vision/README.md) |
  86. | Granite Vision (4.0, 4.1) | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/granite4_vision/README.md) |
  87. | MiniCPM-OCR 1.0 | [Docs](mlx_vlm/models/minicpm_ocr/README.md) |
  88. | MiniCPM-OCR Jev | [Docs](mlx_vlm/models/minicpm_ocr_jev/README.md) |
  89. | MiniCPM-V 4.7 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/minicpmv4_7/README.md) |
  90. | MiniCPM-V 4.6 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/minicpmv4_6/README.md) |
  91. | GLiNER2.5 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/gliner2_5/README.md) |
  92. | BERT | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/bert/README.md) |
  93. | LLaVA-OneVision | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/llava_onevision/README.md) |
  94. | K2-Horizon | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/k2_horizon/README.md) |
  95. | Z1T-0 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/z1t/README.md) |
  96. | Spark-X2.5 | [Docs](https://github.com/Neywa-Labs/mlx-vlm/blob/main/mlx_vlm/models/spark2_5/README.md) |
  97. | Ternary Bonsai 2 | [Docs](mlx_vlm/models/prism_hadamard_qwen35/README.md) |
  98. | Flint | [Docs](mlx_vlm/models/flint/README.md) |
  99. | Ripple | [Docs](mlx_vlm/models/ricursive/README.md) |
  100. | Nemotron 3.5 Super VL / Bolt / Nano Omni | [Docs](mlx_vlm/models/nemotron_h_nano_omni/README.md) |
  101. | Idol (image to 3D Gaussian splats) | [Docs](mlx_vlm/models/idol/README.md) |
  102. ## Installation
  103. The easiest way to get started is to install the `mlx-vlm` package using pip:
  104. ```sh
  105. pip install -U mlx-vlm
  106. ```
  107. The [Gradio chat UI](#chat-ui-with-gradio) needs an extra dependency that is not
  108. part of the base install:
  109. ```sh
  110. pip install -U 'mlx-vlm[ui]'
  111. ```
  112. Quote the package name so that shells which expand square brackets, such as
  113. `zsh`, do not treat `[ui]` as a glob pattern.
  114. ## Agent Skills
  115. This repo ships an agent-skills bundle under `skills/` for common MLX-VLM workflows — usage, conversion, development, and support. Skills load into a coding agent (Claude Code, Codex, Gemini) so it follows the right project conventions instead of guessing.
  116. | Skill | Description |
  117. |-------|-------------|
  118. | `cli-inference` | Run and debug command-line inference (`mlx_vlm.generate`) — text/image/audio inputs and image-generation flags. |
  119. | `server-inference` | Run and debug the local server across the models, chat, responses, messages, audio, image, cache, and metrics endpoints. |
  120. | `convert-quantize` | Convert and quantize Hugging Face models to MLX (`mlx_vlm.convert`) — bits/group size, quant modes, RTN/AWQ, mixed recipes. |
  121. | `add-new-model` | Port a new architecture into `mlx_vlm/models` — config, weight-name mapping, reuse a similar model, add a test class. |
  122. | `benchmarking` | Produce credible, reproducible perf numbers and fork-vs-main A/B tables for PRs. |
  123. | `contributing` | Shape a change to pass review — code/config/test placement, pre-commit hooks, and PR expectations. |
  124. | `hf-cache-models` | List cached, custom, and loaded models through the server's model discovery endpoint. |
  125. | `reproducible-github-issues` | Turn CLI or server failures into concise, reproducible GitHub issues. |
  126. Validate the bundle at any time:
  127. ```sh
  128. python3 skills/scripts/validate_skills.py
  129. ```
  130. Install from a local checkout:
  131. ```sh
  132. # Claude Code
  133. /plugin marketplace add /path/to/mlx-vlm
  134. /plugin install mlx-vlm-skills@mlx-vlm
  135. # Codex CLI
  136. codex plugin marketplace add /path/to/mlx-vlm
  137. codex plugin add mlx-vlm-skills@mlx-vlm
  138. # Gemini CLI
  139. gemini extensions install /path/to/mlx-vlm/skills
  140. ```
  141. ## Usage
  142. ### Command Line Interface (CLI)
  143. Generate output from a model using the CLI:
  144. ```sh
  145. # Text generation
  146. mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Hello, how are you?"
  147. # Image generation
  148. mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg
  149. # Audio understanding
  150. mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you hear" --audio /path/to/audio.wav
  151. # Multi-modal understanding (Image + Audio)
  152. mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you see and hear" --image /path/to/image.jpg --audio /path/to/audio.wav
  153. # Speech generation
  154. mlx_vlm.generate --model openbmb/MiniCPM-o-4_5 --output-modality audio --prompt "Say hello." --ref-audio /path/to/voice.wav --output speech.wav --max-tokens 256
  155. # Image to 3D Gaussian splats
  156. mlx_vlm.generate --model speridlabs/idol --output-modality 3d --image object.png --output object.ply
  157. ```
  158. #### Thinking Budget
  159. For thinking models (e.g., Qwen3.5), you can limit the number of tokens spent in the thinking block:
  160. ```sh
  161. mlx_vlm.generate --model mlx-community/Qwen3.5-2B-4bit \
  162. --thinking-budget 50 \
  163. --thinking-start-token "<think>" \
  164. --thinking-end-token "</think>" \
  165. --enable-thinking \
  166. --prompt "Solve 2+2"
  167. ```
  168. | Flag | Description |
  169. |------|-------------|
  170. | `--enable-thinking` | Activate thinking mode in the chat template |
  171. | `--thinking-budget` | Max tokens allowed inside the thinking block |
  172. | `--thinking-start-token` | Token that opens a thinking block (default: `<think>`) |
  173. | `--thinking-end-token` | Token that closes a thinking block (default: `</think>`) |
  174. When the budget is exceeded, the model is forced to emit `\n</think>` and transition to the answer. If `--enable-thinking` is passed but the model's chat template does not support it, the budget is applied only if the model generates the start token on its own.
  175. On the server, thinking mode is disabled by default. Start the server with `--enable-thinking` to make thinking mode the default for requests that do not specify it:
  176. ```sh
  177. mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking
  178. ```
  179. You can also set server defaults for the thinking budget and delimiter tokens:
  180. ```sh
  181. mlx_vlm.server --model Qwen/Qwen3.5-4B \
  182. --enable-thinking \
  183. --thinking-budget 512 \
  184. --thinking-start-token "<think>" \
  185. --thinking-end-token "</think>"
  186. ```
  187. Requests can override the server defaults with `enable_thinking`, `thinking_budget`, `thinking_start_token`, or `thinking_end_token`. `chat_template_kwargs.enable_thinking` is also accepted; an explicit top-level `enable_thinking` takes precedence.
  188. ### Speculative Decoding
  189. Speed up generation by drafting several candidate tokens with a small "drafter" model and verifying them in a single target forward pass. Three drafter families are supported.
  190. | Flag | Description |
  191. |------|-------------|
  192. | `--draft-model` | HuggingFace repo or local path for the drafter |
  193. | `--draft-kind` | Drafter family — `dflash` (default), `eagle3`, or `mtp` (native/assistant MTP) |
  194. | `--draft-block-size` | Override the drafter's configured block size |
  195. See [docs/usage.md](docs/usage.md) for Python API examples including batch generation.
  196. #### DFlash, DFlash2, and DSpark
  197. A lightweight block-diffusion drafter that predicts multiple tokens per round, typically 2–3× faster.
  198. ```sh
  199. # Text generation with speculative decoding
  200. mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  201. --draft-model z-lab/Qwen3.5-4B-DFlash \
  202. --prompt "Write a quicksort in Python." \
  203. --max-tokens 512 --temperature 0 --enable-thinking
  204. # Also works with images
  205. mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  206. --draft-model z-lab/Qwen3.5-4B-DFlash \
  207. --image examples/images/cats.jpg \
  208. --prompt "Describe this image." \
  209. --max-tokens 256 --temperature 0 --enable-thinking
  210. # Server with speculative decoding
  211. mlx_vlm.server --model Qwen/Qwen3.5-4B \
  212. --draft-model z-lab/Qwen3.5-4B-DFlash
  213. ```
  214. DFlash2 adds dynamic convolutions and a candidate-path selector. The published
  215. Qwen3.8-27B checkpoint is auto-detected and uses the shared exact DFlash target
  216. verification path. For the fastest quantized setup, convert the drafter to
  217. 4-bit; the verifier adapts between three and five rows from recent acceptance:
  218. ```sh
  219. mlx_vlm.convert --hf-path z-lab/Qwen3.8-27B-DFlash2 \
  220. --mlx-path Qwen3.8-27B-DFlash2-4bit \
  221. --quantize --q-bits 4 --q-group-size 64
  222. mlx_vlm.generate --model mlx-community/Qwen3.8-27B-4bit \
  223. --draft-model Qwen3.8-27B-DFlash2-4bit \
  224. --prompt "Write a quicksort in Python." \
  225. --max-tokens 512 --temperature 0
  226. mlx_vlm.server --model mlx-community/Qwen3.8-27B-4bit \
  227. --draft-model Qwen3.8-27B-DFlash2-4bit
  228. ```
  229. Liquid AI's DSpark checkpoint uses a Qwen3-style block drafter plus a learned
  230. Markov correction head. It is auto-detected and runs through the exact target
  231. verification path:
  232. ```sh
  233. mlx_vlm.generate --model LiquidAI/LFM2.5-2.6B \
  234. --draft-model LiquidAI/LFM2.5-2.6B-DSpark \
  235. --prompt "Explain speculative decoding in three sentences." \
  236. --max-tokens 256 --temperature 0
  237. mlx_vlm.server --model LiquidAI/LFM2.5-2.6B \
  238. --draft-model LiquidAI/LFM2.5-2.6B-DSpark
  239. ```
  240. The published DSpark `block_size: 9` means nine proposals, or ten target rows
  241. after adding the anchor token. On MLX, DSpark verifies seven proposals plus the
  242. anchor by default: eight rows exactly fill the verifier threadgroup, while nine
  243. or ten rows pad to sixteen and run slower. The trained width remains available
  244. with `--draft-block-size 10`. The checkpoint's confidence head is loaded for
  245. parity, and DSpark decoding currently requires greedy sampling
  246. (`temperature=0`).
  247. Official OpenBMB and Red Hat Speculators checkpoints can also be loaded directly:
  248. ```sh
  249. mlx_vlm.generate --model openbmb/MiniCPM5-2B \
  250. --draft-model openbmb/MiniCPM5-2B-DSpark \
  251. --prompt "Write a Python function that adds two integers." --temperature 0
  252. mlx_vlm.generate --model Qwen/Qwen3-4B \
  253. --draft-model RedHatAI/Qwen3-4B-speculator.dspark \
  254. --prompt "Explain speculative decoding." --temperature 0
  255. ```
  256. The loader handles nested Speculators configs, reduced draft vocabularies,
  257. checkpoint-owned output heads, and each checkpoint's anchor convention.
  258. See [official checkpoint support](docs/speculative-decoding.md#official-checkpoint-support).
  259. Muse Glimmer's published assistant checkpoint is auto-detected as DFlash:
  260. ```sh
  261. mlx_vlm.generate --model meta-models/Muse-Glimmer-30B \
  262. --draft-model meta-models/Muse-Glimmer-30B-assistant \
  263. --prompt "Write a quicksort in Python." \
  264. --max-tokens 512 --temperature 0
  265. mlx_vlm.server --model meta-models/Muse-Glimmer-30B \
  266. --draft-model meta-models/Muse-Glimmer-30B-assistant
  267. ```
  268. DFlash draft-cache windowing is available from the Python API. During
  269. speculative decoding the target model still verifies every proposed token with
  270. its full KV cache; this knob only changes the DFlash drafter cache. When
  271. `draft_window_size` is set, the drafter keeps at most that many recent committed
  272. tokens in its own KV cache instead of attending over the full generated prefix.
  273. That reduces draft-side cache length and memory, but it can lower acceptance
  274. because the drafter has less context than the target verifier. On MLX, the full
  275. draft cache is usually faster for Qwen3.5 DFlash, so windowing defaults to
  276. `None`; set it only when you want to experiment with this compact recent-token
  277. cache tradeoff:
  278. ```python
  279. from mlx_vlm import load
  280. from mlx_vlm.generate import generate
  281. from mlx_vlm.speculative.drafters import load_drafter
  282. model, processor = load("Qwen/Qwen3.5-4B")
  283. draft_model, draft_kind = load_drafter("z-lab/Qwen3.5-4B-DFlash")
  284. draft_model.config.draft_window_size = 256 # None disables windowing
  285. result = generate(
  286. model,
  287. processor,
  288. "Write a quicksort in Python.",
  289. max_tokens=512,
  290. temperature=0,
  291. draft_model=draft_model,
  292. draft_kind=draft_kind,
  293. )
  294. ```
  295. #### Bundled native heads
  296. Bundled native heads are detected when the same original checkpoint is passed
  297. as the target and drafter. For example:
  298. ```sh
  299. mlx_vlm.generate --model JetBrains/Mellum2.1-12B-A2.5B-Thinking-MTP \
  300. --draft-model JetBrains/Mellum2.1-12B-A2.5B-Thinking-MTP \
  301. --prompt "Write a Python function that adds two integers." --temperature 0
  302. ```
  303. This also applies to DeepSeek V4.1's bundled DSpark stages. Use
  304. `mlx_vlm.convert --mtp` to extract a native head alongside a converted target.
  305. #### Gemma 4 MTP
  306. [Multi-Token Prediction](https://ai.google.dev/gemma/docs/mtp/mtp): Google's 4-layer "assistant" drafter that shares K/V with the target and drafts multiple tokens autoregressively from a constant position. Pass `--draft-kind mtp` to dispatch the MTP round-loop.
  307. ```sh
  308. mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  309. --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  310. --draft-kind mtp --draft-block-size 4 \
  311. --prompt "Explain speculative decoding in 3 sentences." \
  312. --max-tokens 256 --temperature 0
  313. # Server
  314. mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  315. --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  316. --draft-kind mtp --draft-block-size 4
  317. ```
  318. Supported pairings (target ↔ drafter):
  319. | Target | Drafter |
  320. |---------------------------------|------------------------------------------|
  321. | `mlx-community/gemma-4-E2B-it-bf16` | `mlx-community/gemma-4-E2B-it-assistant-bf16` |
  322. | `mlx-community/gemma-4-E4B-it-bf16` | `mlx-community/gemma-4-E4B-it-assistant-bf16` |
  323. | `mlx-community/gemma-4-26B-A4B-it-bf16` | `mlx-community/gemma-4-26B-A4B-it-assistant-bf16` |
  324. | `mlx-community/gemma-4-31B-it-bf16` | `mlx-community/gemma-4-31B-it-assistant-bf16` |
  325. Measured speedups (greedy, byte-identical output): up to **3.94×** on 26B-A4B and **2.29×** on 31B at B=4. See [`mlx_vlm/speculative/drafters/gemma4_assistant/README.md`](mlx_vlm/speculative/drafters/gemma4_assistant/README.md) for full sweeps and architecture notes.
  326. #### Gemma 4 EAGLE-3
  327. [EAGLE-3](https://sgl-project.github.io/SpecForge/concepts/EAGLE3.html) drafts from three target hidden-state captures with a lightweight one-layer speculator. The Red Hat Speculators checkpoint auto-detects as `--draft-kind eagle3`.
  328. ```sh
  329. mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  330. --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3 \
  331. --prompt "Explain speculative decoding in 3 sentences." \
  332. --max-tokens 256 --temperature 0
  333. # Server
  334. mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  335. --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3
  336. ```
  337. #### MiniMax M3 EAGLE-3
  338. MiniMax M3 supports the released `Inferact/MiniMax-M3-EAGLE3` drafter. Convert
  339. the target with `mlx_vlm.convert` because `mlx_lm.convert` does not know the
  340. `minimax_m3_vl` model type.
  341. ```sh
  342. mlx_vlm.convert \
  343. --hf-path MiniMaxAI/MiniMax-M3 \
  344. --mlx-path ~/MiniMax-M3-4bit \
  345. --quantize --q-bits 4 \
  346. --trust-remote-code
  347. mlx_vlm.convert \
  348. --hf-path Inferact/MiniMax-M3-EAGLE3 \
  349. --mlx-path ~/MiniMax-M3-EAGLE3
  350. mlx_vlm.generate \
  351. --model ~/MiniMax-M3-4bit \
  352. --draft-model ~/MiniMax-M3-EAGLE3 \
  353. --draft-kind eagle3 \
  354. --draft-block-size 3 \
  355. --prompt "Explain MiniMax Sparse Attention in one paragraph." \
  356. --max-tokens 256 --temperature 0
  357. ```
  358. The public MiniMax M3 BF16 checkpoint advertises MTP metadata but does not
  359. publish `mtp` or `nextn` tensors, so use the released EAGLE-3 drafter for that
  360. checkpoint.
  361. MiniMax M3 also supports image/video prompts, MiniMax thinking tags, MiniMax
  362. tool-call parsing, MSA index caches, and MXFP8 config loading. See
  363. [`mlx_vlm/models/minimax_m3_vl/README.md`](mlx_vlm/models/minimax_m3_vl/README.md)
  364. for model-specific conversion and runtime notes.
  365. #### Reading the server's draft counters
  366. With a drafter loaded, `mlx_vlm.server` adds four fields to a response's
  367. `timings`, named after llama.cpp's:
  368. | Field | Meaning |
  369. |---|---|
  370. | `draft_kind` | The drafter family that ran |
  371. | `draft_rounds` | Verification rounds: target forward passes after the prefill |
  372. | `draft_n` | Tokens proposed, `--draft-block-size` minus one per round (the block includes the anchor token) |
  373. | `draft_n_accepted` | Proposed tokens the target's greedy choice matched |
  374. `draft_n_accepted` is a measure of the drafter, not of the output. A round
  375. is recorded when the target verifies it, before the stop token or `max_tokens`
  376. cuts the reply, so matches past the end of the reply are counted and the field
  377. can exceed `predicted_n`. `predicted_n - draft_n_accepted` is therefore not
  378. the number of target forward passes. Use `draft_rounds`:
  379. ```text
  380. accept_length = (predicted_n - 1) / draft_rounds
  381. ```
  382. The first token comes from the prefill, before any draft exists, so it is
  383. left out of the numerator. The three counters are differences of a tally kept
  384. on the drafter, so they belong to one request only when one request is in
  385. flight; with concurrent requests they are shared across the batch.
  386. ### Chat UI with Gradio
  387. The Gradio chat UI requires the optional `ui` extra, which the base `mlx-vlm`
  388. install does not include:
  389. ```sh
  390. pip install -U 'mlx-vlm[ui]'
  391. ```
  392. Then launch the chat interface:
  393. ```sh
  394. mlx_vlm.chat_ui --model mlx-community/Qwen2-VL-2B-Instruct-4bit
  395. ```
  396. ### Python Script
  397. Here's an example of how to use MLX-VLM in a Python script:
  398. ```python
  399. import mlx.core as mx
  400. from mlx_vlm import load, generate
  401. from mlx_vlm.prompt_utils import apply_chat_template
  402. from mlx_vlm.utils import load_config
  403. # Load the model
  404. model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
  405. model, processor = load(model_path)
  406. config = load_config(model_path)
  407. # Prepare input
  408. image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
  409. # image = [Image.open("...")] can also be used with PIL.Image.Image objects
  410. prompt = "Describe this image."
  411. # Apply chat template
  412. formatted_prompt = apply_chat_template(
  413. processor, config, prompt, num_images=len(image)
  414. )
  415. # Generate output
  416. output = generate(model, processor, formatted_prompt, image, verbose=False)
  417. print(output)
  418. ```
  419. #### Audio Example
  420. ```python
  421. from mlx_vlm import load, generate
  422. from mlx_vlm.prompt_utils import apply_chat_template
  423. from mlx_vlm.utils import load_config
  424. # Load model with audio support
  425. model_path = "mlx-community/gemma-3n-E2B-it-4bit"
  426. model, processor = load(model_path)
  427. config = model.config
  428. # Prepare audio input
  429. audio = ["/path/to/audio1.wav", "/path/to/audio2.mp3"]
  430. prompt = "Describe what you hear in these audio files."
  431. # Apply chat template with audio
  432. formatted_prompt = apply_chat_template(
  433. processor, config, prompt, num_audios=len(audio)
  434. )
  435. # Generate output with audio
  436. output = generate(model, processor, formatted_prompt, audio=audio, verbose=False)
  437. print(output)
  438. ```
  439. #### Multi-Modal Example (Image + Audio)
  440. ```python
  441. from mlx_vlm import load, generate
  442. from mlx_vlm.prompt_utils import apply_chat_template
  443. from mlx_vlm.utils import load_config
  444. # Load multi-modal model
  445. model_path = "mlx-community/gemma-3n-E2B-it-4bit"
  446. model, processor = load(model_path)
  447. config = model.config
  448. # Prepare inputs
  449. image = ["/path/to/image.jpg"]
  450. audio = ["/path/to/audio.wav"]
  451. prompt = ""
  452. # Apply chat template
  453. formatted_prompt = apply_chat_template(
  454. processor, config, prompt,
  455. num_images=len(image),
  456. num_audios=len(audio)
  457. )
  458. # Generate output
  459. output = generate(model, processor, formatted_prompt, image, audio=audio, verbose=False)
  460. print(output)
  461. ```
  462. ### Decision models
  463. Run shared typed decision prediction from the CLI:
  464. ```sh
  465. python -m mlx_vlm decide --model nativ-community/decider-2b \
  466. --state "Please refund my duplicate charge" \
  467. --questions '{"department":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":["billing","technical","sales"]}}'
  468. ```
  469. Use `--state-file request.txt` for UTF-8 text and `--questions-file questions.json`
  470. for named questions. `mlx_vlm.decide` runs the same command.
  471. Models that read images take a repeatable `--image`, and models that read
  472. speech take one `--audio` (each a path or URL). The state is optional when
  473. media are given:
  474. ```sh
  475. python -m mlx_vlm decide --model LiquidAI/d1-omni-600M \
  476. --image http://images.cocodataset.org/val2017/000000039769.jpg \
  477. --questions '{"cats":{"type":"choice","instructions":"How many cats are there?","criteria":["one","two","more"]}}'
  478. python -m mlx_vlm decide --model LiquidAI/d1-omni-600M \
  479. --audio https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac \
  480. --questions '{"speaking":{"type":"bool","instructions":"Is anyone speaking in this clip?"}}'
  481. ```
  482. Preload a decision model in the existing server:
  483. ```sh
  484. python -m mlx_vlm server --decision-model nativ-community/decider-2b
  485. curl http://localhost:8080/v1/decisions \
  486. -H 'Content-Type: application/json' \
  487. -d '{"state":"Please refund my duplicate charge","questions":{"department":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":["billing","technical","sales"]}}}'
  488. ```
  489. An explicit `model` in the request selects a checkpoint; otherwise the currently
  490. loaded decision model is used. Without either, the endpoint returns HTTP 400.
  491. Requests may add `"images"`, a list of URLs, paths, or data URIs, or `"audio"`,
  492. one URL, path, or base64 data URI; `state` may then be `null`. Decision requests
  493. are non-streaming and use the server's API-key authentication.
  494. Both interfaces return the shared `predict()` result and preserve model-specific
  495. scoring. Decider, Laya, [d1-3B](mlx_vlm/models/d1/README.md), and
  496. [d1-omni-600M](mlx_vlm/models/d1_omni/README.md) support `choice`, `bool`, and
  497. `score`; the two LiquidAI d1 models also read images, and d1-omni reads speech.
  498. Consult each model's README for criteria and calibration details.
  499. ### Server (FastAPI)
  500. Start the server:
  501. ```sh
  502. mlx_vlm.server --port 8080
  503. # Preload a model at startup (Hugging Face repo or local path)
  504. mlx_vlm.server --model <hf_repo_or_local_path>
  505. # Preload separate model kinds at startup
  506. mlx_vlm.server --model <language_model> \
  507. --image-model <image_generation_model> \
  508. --tts-model <text_to_speech_model> \
  509. --stt-model <speech_to_text_model>
  510. # Preload a model with adapter
  511. mlx_vlm.server --model <hf_repo_or_local_path> --adapter-path <adapter_path>
  512. # With trust remote code enabled (required for some models)
  513. mlx_vlm.server --trust-remote-code
  514. # Enable thinking mode by default for requests that do not override it
  515. mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking
  516. # Configure thinking defaults at startup
  517. mlx_vlm.server --model Qwen/Qwen3.5-4B \
  518. --enable-thinking \
  519. --thinking-budget 512 \
  520. --thinking-start-token "<think>" \
  521. --thinking-end-token "</think>"
  522. # Require bearer authentication for API endpoints
  523. mlx_vlm.server --api-key <secret-token>
  524. # Include local models outside the Hugging Face cache
  525. mlx_vlm.server --model-dir /Volumes/Models --model-dir ~/my-custom-model
  526. ```
  527. Use `distributed: true` per request to run a model across configured Macs while
  528. keeping ordinary requests on the local loader. See [distributed serving](docs/distributed-serving.md)
  529. for worker setup, automatic checkpoint transfer, and current limitations.
  530. #### Server Options
  531. - `--model`: Preload a language model at server startup, accepts a Hugging Face repo ID or local path (optional, loads lazily on first request if omitted)
  532. - `--image-model`: Preload an image generation model at server startup
  533. - `--mesh-model`: Preload an image-to-3D model for asynchronous PLY generation
  534. - `--tts-model`: Preload a text-to-speech model at server startup
  535. - `--stt-model`: Preload a speech-to-text model at server startup
  536. - `--embedding-model`: Preload an embedding model at server startup
  537. - `--decision-model`: Preload a decision model at server startup
  538. - `--reranker-model`: Preload a supported reranker model at server startup
  539. - `--model-dir`: Additional model folder or parent containing model folders to include in discovery; repeat for multiple paths. Overrides `MLX_VLM_MODEL_PATHS` (paths separated by `os.pathsep`, `:` on macOS/Linux)
  540. - `--adapter-path`: Path for adapter weights to use with the preloaded model
  541. - `--draft-model`: Speculative drafter path or HF id (e.g. `z-lab/Qwen3.8-27B-DFlash2`, `z-lab/Qwen3.5-4B-DFlash`, `RedHatAI/gemma-4-31B-it-speculator.eagle3`, `google/gemma-4-31B-it-assistant`, `Inferact/MiniMax-M3-EAGLE3`) — enables speculative decoding for ~2× or higher throughput
  542. - `--draft-kind`: Drafter family — `dflash` (default), `eagle3`, or `mtp` (native/assistant MTP)
  543. - `--draft-block-size`: Override the drafter's configured block size
  544. - `--host`: Host address (default: `0.0.0.0`)
  545. - `--port`: Port number (default: `8080`)
  546. - `--trust-remote-code`: Trust remote code when loading models from Hugging Face Hub
  547. - `--enable-thinking`: Enable thinking mode by default for requests that do not set `enable_thinking`
  548. - `--thinking-budget`: Default maximum number of tokens allowed inside a thinking block
  549. - `--thinking-start-token`: Default token that opens a thinking block
  550. - `--thinking-end-token`: Default token that closes a thinking block (`--thinking-eos-token` is also accepted)
  551. - `--kv-bits`: Number of bits for KV cache quantization (e.g. `8` for uniform, `3.5` for TurboQuant)
  552. - `--kv-quant-scheme`: KV cache quantization backend (`uniform` or `turboquant`)
  553. - `--kv-key-bits` / `--kv-value-bits`: Override the bit-width for keys or values individually (see [Per-tensor KV quantization](#per-tensor-kv-quantization))
  554. - `--kv-key-scheme` / `--kv-value-scheme`: Override the quantization backend for keys or values individually
  555. - `--kv-group-size`: Group size for uniform KV cache quantization (default: `64`)
  556. - `--max-kv-size`: Maximum KV cache size in tokens
  557. - `--vision-cache-size`: Max number of cached vision features (default: `20`)
  558. - `--min-pixels` / `--max-pixels`: Default minimum/maximum input image pixel area for processors that support these options (also `MLX_VLM_MIN_PIXELS` / `MLX_VLM_MAX_PIXELS`). Unset values use the model's preprocessing defaults.
  559. - `--log-progress-interval`: Decoded tokens between progress log messages; `0` disables periodic decode progress (default: `10`)
  560. - `--api-key`: Bearer token required for inference, model discovery, and management endpoints
  561. - `--log-level`: Logging level — `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` (default: `INFO`)
  562. Streaming `/v1/chat/completions` and `/v1/responses` send invisible SSE comments
  563. every 15 seconds while idle, keeping connections alive during queueing and long
  564. prefills. Set `MLX_VLM_STREAM_KEEPALIVE_INTERVAL` to a different interval in seconds
  565. or `0` to disable it. This does not change model output or generation timeouts.
  566. For higher-resolution image input, including OCR with Bonsai 2, set pixel-area
  567. defaults at startup:
  568. ```sh
  569. mlx_vlm.server --model prism-ml/Ternary-Bonsai-2-2B-mlx-2bit \
  570. --min-pixels 1048576 --max-pixels 4194304
  571. ```
  572. Chat Completions, Responses, and Messages requests can override these defaults
  573. with top-level `min_pixels` and `max_pixels` fields (use `extra_body` with an
  574. OpenAI client). For example, `"min_pixels": 1048576` requests at least 1024 × 1024
  575. pixels of area, preserving aspect ratio and rounding to the processor's patch
  576. size. On Bonsai 2 this corresponds to approximately 1024 image tokens. Higher
  577. resolution uses more context, memory, and processing time. Both limits must be
  578. positive and the effective minimum must not exceed the maximum. Omitted or null
  579. request values inherit the server defaults.
  580. Change these defaults on a running server without restarting or reloading the
  581. model:
  582. ```sh
  583. curl -X PATCH http://localhost:8080/v1/settings \
  584. -H 'Content-Type: application/json' \
  585. -d '{"min_pixels": 1048576, "max_pixels": 4194304}'
  586. ```
  587. Use `GET /v1/settings` to inspect the current values. New requests inherit the
  588. updated limits; requests already queued for generation keep their resolved
  589. options, and per-request overrides still take precedence. The pair is updated
  590. atomically; invalid limits appear in `rejected` and leave both limits unchanged.
  591. Set either setting to `null` to use that limit from the model's preprocessing
  592. configuration. Updates last until the server restarts; the
  593. `{"op": "replace", "values": {}}` form resets **all** runtime settings to their
  594. startup defaults, including CLI pixel limits. Include
  595. `Authorization: Bearer <key>` when the server has an API key configured.
  596. The existing `resize_shape` request field also applies before image processing
  597. in the batching server. Processor pixel limits still apply afterward. Token
  598. counting and streaming context checks use the same options as generation.
  599. At `INFO`, the server logs request start/completion, chunked-prefill progress,
  600. time to first token, periodic decode throughput, and the final token counts. Set
  601. `--log-level DEBUG` to emit decode progress for every token and add its token
  602. number, token ID, and decoded text to the same log entry. Decode progress uses
  603. `rate` for the instantaneous inter-token rate; decode completion uses the same
  604. field name for aggregate decode throughput measured across completed token
  605. intervals.
  606. OpenAI-compatible streaming responses expose throughput under
  607. `timings.predicted_per_second`. Token-bearing SSE chunks report the instantaneous
  608. inter-token rate, while terminal and usage chunks report the aggregate rate as
  609. `(tokens - 1) / (last_token_time - first_token_time)`. The first token reports
  610. `null` because it has no preceding token interval.
  611. You can also set trust remote code via environment variable:
  612. ```sh
  613. MLX_TRUST_REMOTE_CODE=true mlx_vlm.server
  614. ```
  615. The server provides multiple endpoints for different use cases and supports dynamic model loading/unloading with caching (one model at a time).
  616. ### Continuous Batching
  617. The server supports continuous batching for higher throughput when handling multiple concurrent requests. New requests join the active batch immediately without waiting for existing requests to finish, and mixed batches of image and text-only requests are supported.
  618. Prompt processing and decoding are interleaved, with decoding taking priority. The
  619. server's `--prefill-step-size` (default: 2048) is a shared prompt-token budget per
  620. step, including padding, so increasing the batch size does not multiply the prompt
  621. work between decode steps. While peers decode, the scheduler also caps prompt
  622. work at 256 tokens per step. Lower the configured budget to reduce pauses further.
  623. Models that require unchunked prefill do not use this budget.
  624. Ready decode tokens are published before the next prefill step. Pending prompts
  625. are grouped by comparable length within the oldest candidate window to limit
  626. padding without starving long prompts. APC retains its cached-suffix grouping and
  627. memory admission checks. Python streaming integrations can
  628. use `BatchGenerator.decode_step()` and `prefill_step()` separately; `next()`
  629. continues to advance both phases for existing callers.
  630. Continuous batching is enabled automatically when the server loads a model. You can pre-load a model at startup so it's ready to serve immediately:
  631. ```sh
  632. mlx_vlm.server --port 8080 --model mlx-community/Qwen2.5-VL-3B-Instruct-4bit
  633. ```
  634. Verify via the health endpoint:
  635. ```sh
  636. curl http://localhost:8080/health
  637. # {"status":"healthy","loaded_model":"...","apc_enabled":false}
  638. ```
  639. If `--model` is omitted, the model is loaded on the first request.
  640. ### Automatic Prefix Caching (APC)
  641. Automatic Prefix Caching reuses model cache state across requests that share the same prefix. It is useful for repeated long documents, long chat histories, or retrieval contexts where each request appends a short new suffix.
  642. APC builds a cache plan from `model.make_cache()` in the same style as vLLM's hybrid cache manager: each layer gets a cache spec, compatible specs form cache groups, and one coordinator selects a common reusable prefix across the groups. Dense attention models use pageable K/V blocks. Hybrid full/sliding-attention, recurrent/SSM, MLA/composite, VLM, and Omni layouts use restorable state checkpoints for the components that cannot be concatenated safely. Generation code uses the same coordinator API for both paths.
  643. APC has two tiers:
  644. - **Warm memory**: keeps reusable `APCBlock` tensors in process memory. This is the fastest path, but it keeps both the reusable block pool and the runtime `KVCache`.
  645. - **Warm disk**: persists cached prefixes as safetensors shards so they survive process restarts. Warm-disk reads build the layer-major prompt cache directly without promoting restored blocks into the `APCBlock` pool; writes can still populate both memory and disk tiers.
  646. #### Python Script
  647. Use `APCManager` directly when calling `stream_generate`:
  648. ```python
  649. from pathlib import Path
  650. from mlx_vlm import load, stream_generate
  651. from mlx_vlm.apc import APCManager, DiskBlockStore
  652. from mlx_vlm.prompt_utils import apply_chat_template
  653. model_id = "Qwen/Qwen3-VL-4B-Instruct"
  654. model, processor = load(model_id)
  655. disk = DiskBlockStore(
  656. Path("~/.cache/mlx-vlm/caching").expanduser(),
  657. namespace=model_id,
  658. max_bytes=3 * (1 << 30), # 3 GB disk cap; use None for uncapped
  659. )
  660. apc = APCManager(num_blocks=4096, block_size=16, disk=disk)
  661. # Optional diagnostics: inspect the automatically inferred cache groups.
  662. print(apc.coordinator(model).plan.describe())
  663. document = Path("long_document.txt").read_text()
  664. try:
  665. # First request computes the full prefix and stores reusable K/V blocks.
  666. prompt1 = apply_chat_template(
  667. processor,
  668. model.config,
  669. prompt=f"{document}\n\nSummarize the key decisions.",
  670. num_images=0,
  671. )
  672. for _ in stream_generate(
  673. model, processor, prompt1, max_tokens=128, temperature=0.0, apc_manager=apc
  674. ):
  675. pass
  676. # Second request shares the same document prefix and only prefills the suffix.
  677. prompt2 = apply_chat_template(
  678. processor,
  679. model.config,
  680. prompt=f"{document}\n\nList the open engineering risks.",
  681. num_images=0,
  682. )
  683. for chunk in stream_generate(
  684. model, processor, prompt2, max_tokens=128, temperature=0.0, apc_manager=apc
  685. ):
  686. print(chunk.text, end="", flush=True)
  687. print(apc.stats_snapshot())
  688. finally:
  689. apc.close()
  690. ```
  691. To compare cold, warm-memory, warm-disk, and disk-eviction behavior with a
  692. model, use the same direct API path:
  693. ```python
  694. import os
  695. import tempfile
  696. import time
  697. from pathlib import Path
  698. from mlx_vlm import load, stream_generate
  699. from mlx_vlm.apc import APCManager, DiskBlockStore
  700. from mlx_vlm.prompt_utils import apply_chat_template
  701. model_id = "Qwen/Qwen3-VL-4B-Instruct"
  702. contexts = [8000, 20000, 50000, 100000]
  703. disk_cap_gb = 0 # 0 means uncapped
  704. shard_max_blocks = 256
  705. context_sweep_max_tokens = 1 # one token is enough to measure prefill reuse
  706. test_prompt_tokens = 8000
  707. fill_prompts = 80
  708. eviction_disk_cap_gb = 3.0
  709. os.environ["APC_DISK_SHARD_MAX_BLOCKS"] = str(shard_max_blocks)
  710. model, processor = load(model_id)
  711. tokenizer = processor.tokenizer if hasattr(processor, "tokenizer") else processor
  712. def disk_cap_bytes(gb: float):
  713. return None if gb <= 0 else int(gb * (1 << 30))
  714. def make_context(target_tokens: int, seed: int = 0) -> str:
  715. line = (
  716. f"Document {seed}: APC benchmark content with deterministic facts, "
  717. "dates, identifiers, and repeated technical notes.\n"
  718. )
  719. line_tokens = max(1, len(tokenizer.encode(line, add_special_tokens=False)))
  720. text = line * max(1, target_tokens // line_tokens)
  721. while len(tokenizer.encode(text, add_special_tokens=False)) < target_tokens:
  722. text += line
  723. return text
  724. def make_prompt(context: str, question: str) -> str:
  725. return apply_chat_template(
  726. processor,
  727. model.config,
  728. prompt=f"{context}\n\n{question}",
  729. num_images=0,
  730. )
  731. def run_once(apc: APCManager, context: str, question: str, max_tokens: int = 32):
  732. prompt = make_prompt(context, question)
  733. apc.reset_stats()
  734. last = None
  735. output = []
  736. start = time.perf_counter()
  737. for chunk in stream_generate(
  738. model,
  739. processor,
  740. prompt,
  741. max_tokens=max_tokens,
  742. temperature=0.0,
  743. apc_manager=apc,
  744. ):
  745. output.append(chunk.text)
  746. last = chunk
  747. if last is None:
  748. raise RuntimeError("generation returned no chunks")
  749. return {
  750. "wall_s": time.perf_counter() - start,
  751. "prompt_tokens": last.prompt_tokens,
  752. "prompt_tps": last.prompt_tps,
  753. "generation_tps": last.generation_tps,
  754. "apc": apc.stats_snapshot(),
  755. "text": "".join(output).strip(),
  756. }
  757. def print_result(label: str, result: dict) -> None:
  758. stats = result["apc"]
  759. print(
  760. f"{label:<12} "
  761. f"prompt_tokens={result['prompt_tokens']:>7} "
  762. f"prompt_tps={result['prompt_tps']:>8.1f} "
  763. f"gen_tps={result['generation_tps']:>7.1f} "
  764. f"matched={stats.get('matched_tokens', 0):>7} "
  765. f"disk_hits={stats.get('disk_hits', 0):>5} "
  766. f"disk_evictions={stats.get('disk_evictions', 0):>5}"
  767. )
  768. def open_apc(cache_root: Path, namespace: str, disk_gb: float) -> APCManager:
  769. disk = DiskBlockStore(
  770. cache_root,
  771. namespace=namespace,
  772. max_bytes=disk_cap_bytes(disk_gb),
  773. )
  774. return APCManager(num_blocks=4096, block_size=16, disk=disk)
  775. def run_context_sweep() -> None:
  776. print("cold / warm-memory / warm-disk")
  777. with tempfile.TemporaryDirectory() as tmp:
  778. cache_root = Path(tmp)
  779. for target_tokens in contexts:
  780. context = make_context(target_tokens)
  781. namespace = f"{model_id}-context-{target_tokens}"
  782. apc = open_apc(cache_root, namespace, disk_cap_gb)
  783. try:
  784. print(f"\ncontext ~= {target_tokens} text tokens")
  785. print_result(
  786. "cold",
  787. run_once(
  788. apc,
  789. context,
  790. "Summarize the key decisions.",
  791. max_tokens=context_sweep_max_tokens,
  792. ),
  793. )
  794. print_result(
  795. "warm-memory",
  796. run_once(
  797. apc,
  798. context,
  799. "List the open engineering risks.",
  800. max_tokens=context_sweep_max_tokens,
  801. ),
  802. )
  803. finally:
  804. # Closing waits for queued disk writes before reopening the disk tier.
  805. apc.close()
  806. apc = open_apc(cache_root, namespace, disk_cap_gb)
  807. try:
  808. print_result(
  809. "warm-disk",
  810. run_once(
  811. apc,
  812. context,
  813. "Extract the implementation timeline.",
  814. max_tokens=context_sweep_max_tokens,
  815. ),
  816. )
  817. finally:
  818. apc.close()
  819. def run_disk_eviction_workload() -> None:
  820. print("\ndisk eviction workload")
  821. with tempfile.TemporaryDirectory() as tmp:
  822. cache_root = Path(tmp)
  823. namespace = f"{model_id}-eviction"
  824. test_context = make_context(test_prompt_tokens, seed=0)
  825. apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
  826. try:
  827. print_result(
  828. "seed",
  829. run_once(apc, test_context, "Summarize the retained test prefix."),
  830. )
  831. finally:
  832. apc.close()
  833. apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
  834. try:
  835. for i in range(fill_prompts):
  836. fill_context = make_context(test_prompt_tokens, seed=i + 1)
  837. run_once(
  838. apc,
  839. fill_context,
  840. f"Summarize filler document {i + 1}.",
  841. max_tokens=1,
  842. )
  843. if (i + 1) % 10 == 0:
  844. stats = apc.stats_snapshot()
  845. print(
  846. f"filled={i + 1:>3} "
  847. f"disk_gb={stats.get('disk_bytes', 0) / (1 << 30):.2f} "
  848. f"disk_evictions={stats.get('disk_evictions', 0)}"
  849. )
  850. finally:
  851. apc.close()
  852. apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
  853. try:
  854. print_result(
  855. "post-fill",
  856. run_once(
  857. apc,
  858. test_context,
  859. "Check whether the retained test prefix still restores.",
  860. ),
  861. )
  862. finally:
  863. apc.close()
  864. run_context_sweep()
  865. run_disk_eviction_workload()
  866. ```
  867. #### Server
  868. Enable in-memory APC for the server with environment variables:
  869. ```sh
  870. APC_ENABLED=1 \
  871. APC_NUM_BLOCKS=4096 \
  872. mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080
  873. ```
  874. APC works with KV-cache quantization (`--kv-bits`):
  875. ```sh
  876. APC_ENABLED=1 \
  877. APC_NUM_BLOCKS=4096 \
  878. mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --kv-bits 8 --port 8080
  879. ```
  880. APC persists caches to disk by default when enabled, under
  881. `$MLX_VLM_CACHE_HOME/apc` (or `~/.cache/mlx-vlm/apc`), with a 20 GiB cap per
  882. model namespace. Customize the location and cap:
  883. ```sh
  884. APC_ENABLED=1 \
  885. APC_NUM_BLOCKS=4096 \
  886. APC_DISK_PATH=~/.cache/mlx-vlm/caching \
  887. APC_DISK_MAX_GB=3 \
  888. APC_DISK_SHARD_MAX_BLOCKS=256 \
  889. mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080
  890. ```
  891. Repeated requests with the same long prefix will hit APC automatically:
  892. ```sh
  893. curl -X POST "http://localhost:8080/v1/chat/completions" \
  894. -H "Content-Type: application/json" \
  895. -H "X-APC-Tenant: demo" \
  896. -d '{
  897. "model": "Qwen/Qwen3-VL-4B-Instruct",
  898. "messages": [{
  899. "role": "user",
  900. "content": "Paste a long shared document here.\n\nNow answer question A."
  901. }],
  902. "max_tokens": 128
  903. }'
  904. ```
  905. Use the same `X-APC-Tenant` value for requests that may share cached prefixes. Use different tenant values to isolate cache entries between users or workspaces.
  906. Inspect and reset APC state:
  907. ```sh
  908. curl http://localhost:8080/v1/cache/stats
  909. curl -X POST http://localhost:8080/v1/cache/reset
  910. ```
  911. `restored_tokens` counts prefix tokens successfully materialized for generation
  912. across block and exact caches, in memory or on disk. A lookup alone does not
  913. increase it, nor does a failed batch restore. A successful restore remains counted
  914. if the request later fails or is cancelled.
  915. The previous `served_tokens` block-storage count is now `stored_tokens`;
  916. `token_hit_rate` retains its existing calculation,
  917. `matched_tokens / (matched_tokens + stored_tokens)`.
  918. Configure APC on a running server with `PATCH /v1/settings`. Use
  919. `GET /v1/settings` to discover supported settings and read their current values:
  920. ```sh
  921. curl http://localhost:8080/v1/settings
  922. curl -X PATCH http://localhost:8080/v1/settings \
  923. -H 'Content-Type: application/json' \
  924. -d '{
  925. "apc_enabled": true,
  926. "apc_disk_enabled": true,
  927. "apc_memory_max_gb": 2,
  928. "apc_disk_max_gb": 20,
  929. "apc_checkpoint_interval_tokens": 1024
  930. }'
  931. ```
  932. The endpoint also accepts `apc_memory_reserve_gb`, `apc_disk_queue_max_gb`,
  933. `apc_disk_path`, `apc_disk_shard_max_blocks`, `apc_block_size`, `apc_num_blocks`,
  934. `apc_checkpoint_entries`, and `apc_checkpoint_guard_tokens`. When
  935. `MLX_VLM_SERVER_API_KEY` is configured, include `Authorization: Bearer <key>`.
  936. Settings apply on the next text-generation request through the existing model
  937. reload path; the server process stays running. Reloading clears resident caches
  938. and retires the previous disk writer after its generation worker finishes.
  939. Existing disk files remain available. Updating APC options while APC is disabled
  940. stages them for the next enable. The response reports `applied`, `rejected`,
  941. `reload_kinds`, and the resulting settings; invalid sizes are rejected without
  942. changing those values. Unchanged settings do not trigger a reload.
  943. Use `null` for automatic memory budgets or the default disk path/cap. Zero has
  944. specific meanings: `apc_memory_max_gb: 0` retains caches only on disk,
  945. `apc_disk_max_gb: 0` removes the disk cap, and `apc_disk_queue_max_gb: 0` writes
  946. synchronously. An empty `apc_disk_path` or `apc_disk_enabled: false` disables
  947. persistence. PATCH merges with current settings; the optional
  948. `{"op": "replace", "values": {...}}` form first restores startup environment
  949. defaults. `/v1/cache/stats` shows the active manager after the next request.
  950. Common APC environment variables:
  951. | Variable | Default | Description |
  952. |----------|---------|-------------|
  953. | `APC_ENABLED` | `0` | Set to `1` to enable APC |
  954. | `APC_NUM_BLOCKS` | `2048` | Number of in-memory APC blocks |
  955. | `APC_BLOCK_SIZE` | `16` | Tokens per APC block |
  956. | `APC_CHECKPOINT_ENTRIES` | `2` | In-memory checkpoint entries; also bounds snapshots captured per hybrid prompt (disk-only mode captures two) |
  957. | `APC_CHECKPOINT_GUARD_TOKENS` | `1` | Tokens retained after a reusable hybrid checkpoint boundary; the default preserves the normal final-token prefill boundary |
  958. | `APC_CHECKPOINT_INTERVAL_TOKENS` | `2048` | Spacing of intermediate hybrid checkpoints, rounded up to a multiple of `APC_BLOCK_SIZE`; observed prompt divergence takes priority within the entry budget; `0` keeps only the final checkpoint |
  959. | `APC_MEMORY_MAX_GB` | auto | Resident block and checkpoint budget in GiB: 10% of Metal's recommended working set, capped at 8 GiB; `0` retains caches only on disk |
  960. | `APC_MEMORY_RESERVE_GB` | auto | Additional memory headroom in GiB: 10% of Metal's recommended working set, at least 1 GiB |
  961. | `APC_DISK_ENABLED` | `1` | Set to `0` to disable disk persistence |
  962. | `MLX_VLM_CACHE_HOME` | `~/.cache/mlx-vlm` | Base cache directory; APC uses its `apc` subdirectory unless `APC_DISK_PATH` is set |
  963. | `APC_DISK_PATH` | cache directory above | Directory for persistent disk shards; an empty value disables persistence |
  964. | `APC_DISK_MAX_GB` | `20` | Disk cap per model namespace in GiB; `0` means uncapped |
  965. | `APC_DISK_QUEUE_MAX_GB` | `1` | Maximum tensor bytes held by queued disk writes in GiB; larger writes run synchronously; `0` makes all writes synchronous |
  966. | `APC_DISK_SHARD_MAX_BLOCKS` | `256` | Max blocks per disk segment shard |
  967. | `APC_MAX_POOL_TENSORS` | `450000` | Stops adding memory blocks before the Metal resource limit; disk writes continue |
  968. | `APC_LAYER_MAJOR_MEMORY_MIN_TOKENS` | `50000` | Store long warm-memory prefixes as compact layer-major snapshots instead of per-block tensors |
  969. | `APC_HASH` | `fast` | Set to `sha256` for a stable cryptographic hash |
  970. | `APC_TRACE` | unset | Set to `1` for greppable store/reject/self-check log lines |
  971. Custom cache layouts can opt in without APC model-name checks by implementing `prefix_cache_snapshot()` and `prefix_cache_restore(snapshot)`. In-tree dense, sliding-window, recurrent, composite, VLM, and Omni cache layouts are detected automatically. APC works with `--kv-bits` (including TurboQuant): the live KV cache stays quantized; pageable APC K/V blocks are stored as dequantized float K/V, so block-pool size does not shrink with quant.
  972. When APC is enabled on the server, a non-fatal layout self-check runs at model load.
  973. Requests with a shared document and different questions can reuse their common
  974. prefix. Dense K/V caches, including compact layer-major memory snapshots, match
  975. complete blocks before the first differing token. Hybrid, recurrent and sliding
  976. window caches must restore a state captured before that divergence: their final
  977. state cannot be rolled back by slicing K/V tensors.
  978. Hybrid prefill now captures intermediate checkpoints as well as the final guard
  979. checkpoint, in streaming, continuous batching and DiffusionGemma generation.
  980. For text API requests with `transient_context`, the server verifies and prioritizes
  981. the token boundary before that mutable suffix. Otherwise, observed divergence
  982. from resident prompts takes priority. With neither hint, each prompt stores its
  983. latest 2,048-token boundary before the final checkpoint and the final checkpoint
  984. itself. For example, a 30,000-token
  985. document followed by 20 instruction tokens can reuse 28,672 tokens when the
  986. instructions change. The same checkpoints persist across server restarts when
  987. the disk tier is enabled.
  988. Captures are bounded by `APC_CHECKPOINT_ENTRIES` to avoid copying the growing
  989. hybrid cache at every prefill chunk. More entries retain more earlier boundaries;
  990. a smaller interval gives finer reuse near the end. Reuse still requires a
  991. retained boundary before the divergence. Short prompts below the interval,
  992. divergence before the earliest retained checkpoint, and evicted checkpoints can
  993. miss. Media checkpoints include all media tokens so the remaining suffix is text.
  994. Changing checkpoint boundaries can change floating-point execution shapes, as
  995. with other chunked or cached prefill paths.
  996. The resident byte budget includes exact snapshots as well as pageable blocks.
  997. Before embeddings and prefill, APC drains pending disk writes and evicts idle
  998. checkpoints and blocks in LRU order within each tier. Admission uses current
  999. Metal allocations, available system RAM, and the incoming prompt's estimated
  1000. cache growth, based on observed bytes per token. Leased blocks remain valid for
  1001. active requests. Snapshots that exceed the budget are written directly to disk
  1002. without making another resident copy, and disk restores stay out of the memory
  1003. LRU when promotion would exceed the budget. Disk pressure may therefore trade
  1004. latency for lower memory use.
  1005. These controls limit APC's memory overhead; model weights and an individual
  1006. request must still fit in memory. Tune the reserve for models with larger
  1007. prefill or vision temporaries. `/v1/cache/stats` reports resident bytes,
  1008. the memory budget, prefill reserve, memory evictions, and pending disk bytes.
  1009. #### KV Cache Quantization
  1010. Reduce KV cache memory during continuous batching with `--kv-bits`. Both uniform quantization and TurboQuant are supported. Compatible with Automatic Prefix Caching (`APC_ENABLED=1`).
  1011. ```sh
  1012. # Uniform 8-bit KV cache quantization
  1013. mlx_vlm.server --model google/gemma-4-26b-a4b-it --kv-bits 8
  1014. # TurboQuant 3.5-bit (3-bit keys + 4-bit values)
  1015. mlx_vlm.server --model google/gemma-4-26b-a4b-it --kv-bits 3.5 --kv-quant-scheme turboquant
  1016. ```
  1017. Full-attention layers use quantized batch caches while sliding-window layers keep their fixed-size rotating caches. The last full-attention layer stays unquantized (sensitive in deep models).
  1018. ##### Per-tensor KV quantization
  1019. Keys and values do not have to share a bit-width or a backend. A fractional `--kv-bits` already splits the budget — `3.5` gives 3-bit keys and 4-bit values — and `--kv-key-bits` / `--kv-value-bits` override either side:
  1020. ```sh
  1021. # 8-bit keys, 3-bit values, both TurboQuant
  1022. mlx_vlm.generate --model mlx-community/Qwen3.5-9B-MLX-4bit \
  1023. --kv-bits 3.5 --kv-quant-scheme turboquant \
  1024. --kv-key-bits 8 --kv-value-bits 3
  1025. ```
  1026. `--kv-key-scheme` / `--kv-value-scheme` go further and select a different backend per tensor, which builds a hybrid cache:
  1027. ```sh
  1028. # uniform 8-bit keys beside TurboQuant 3-bit values
  1029. mlx_vlm.generate --model mlx-community/Qwen3.5-9B-MLX-4bit \
  1030. --kv-bits 8 --kv-quant-scheme uniform \
  1031. --kv-value-bits 3 --kv-value-scheme turboquant
  1032. ```
  1033. Two limitations apply to mixed *schemes* specifically:
  1034. - The hybrid cache dequantizes on every step instead of using a fused kernel, so it is slower than either homogeneous path.
  1035. - Mixed schemes are not supported during continuous batching or for batch prefix caches, and raise `NotImplementedError` there. Mixed bit-widths under a single scheme work everywhere.
  1036. Note that values are given the extra bit by default for a reason: value error passes straight through the attention output, whereas key error is partly reabsorbed by the softmax. Measured on Qwen3.5, spending an equal budget key-heavy was consistently worse than value-heavy, so prefer measuring before overriding.
  1037. Tested with gemma-4-26b-a4b-it at 20K context:
  1038. | Config | Gen tok/s | KV Cache | KV Reduction |
  1039. |--------|-----------|----------|--------------|
  1040. | No quant | 50.3 | 0.624 GB | 1x |
  1041. | Uniform 8-bit | 52.6 | 0.469 GB | **1.33x** |
  1042. | TurboQuant 3.5-bit | 25.6 | 0.365 GB | **1.71x** |
  1043. > Models with all full-attention layers (e.g. Qwen, LLaMA) see larger reductions — up to 3.6x at 8-bit and 6.4x at 4-bit.
  1044. #### Log Probabilities
  1045. The `/chat/completions` endpoint supports OpenAI-compatible per-token log probabilities. Pass `logprobs: true` (and optionally `top_logprobs: N`, up to 20) in the request:
  1046. ```sh
  1047. curl -X POST "http://localhost:8080/v1/chat/completions" \
  1048. -H "Content-Type: application/json" \
  1049. -d '{
  1050. "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
  1051. "messages": [{"role":"user","content":"Say hi in 3 words."}],
  1052. "max_tokens": 8,
  1053. "logprobs": true,
  1054. "top_logprobs": 3
  1055. }'
  1056. ```
  1057. Each choice gets a `logprobs.content[]` list with one entry per generated token: `{token, logprob, bytes, top_logprobs: [{token, logprob, bytes}, ...]}`. Works for both streaming and non-streaming.
  1058. `top_logprobs` requires the server to be started with a non-zero cap on how many alternatives it will compute per token (default `0` = disabled, max `20`). Set it via the `--top-logprobs-k` flag or the `TOP_LOGPROBS_K` env var:
  1059. ```sh
  1060. mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit --top-logprobs-k 5
  1061. # or
  1062. TOP_LOGPROBS_K=5 mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit
  1063. ```
  1064. Per-request `top_logprobs` is clamped to `TOP_LOGPROBS_K`. When `TOP_LOGPROBS_K=0`, requests with `logprobs: true` still return chosen-token logprobs; only the `top_logprobs` list stays empty. Leaving the cap at `0` keeps the vocab-wide sort out of the decode graph, so deployments that don't need logprobs pay zero overhead.
  1065. #### Structured Outputs
  1066. The `/v1/chat/completions` and `/v1/responses` endpoints support OpenAI-compatible `json_schema` structured outputs. The server constrains generation to the supplied JSON schema and supports both streaming and non-streaming responses.
  1067. You can define the schema with Pydantic:
  1068. ```python
  1069. from typing import Literal
  1070. from pydantic import BaseModel, ConfigDict, Field
  1071. class AnimalResult(BaseModel):
  1072. model_config = ConfigDict(extra="forbid")
  1073. animal: Literal["dog", "cat", "bird", "unknown"]
  1074. species: str = Field(max_length=60)
  1075. description: str = Field(max_length=200)
  1076. schema = AnimalResult.model_json_schema()
  1077. ```
  1078. Call the local server with the OpenAI Python client:
  1079. ```python
  1080. from openai import OpenAI
  1081. client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
  1082. response = client.chat.completions.create(
  1083. model="mlx-community/Qwen3.5-4B-MLX-4bit",
  1084. messages=[
  1085. {"role": "user", "content": "Return a dog object."},
  1086. ],
  1087. response_format={
  1088. "type": "json_schema",
  1089. "json_schema": {
  1090. "name": "AnimalResult",
  1091. "strict": True,
  1092. "schema": schema,
  1093. },
  1094. },
  1095. )
  1096. result = AnimalResult.model_validate_json(response.choices[0].message.content)
  1097. print(result)
  1098. ```
  1099. Example output:
  1100. ```text
  1101. animal='dog' species='Canis lupus familiaris' description='A domesticated canine known for companionship and loyalty.'
  1102. ```
  1103. Chat completions use top-level `response_format`. The same format works for text-only and multimodal requests:
  1104. ```sh
  1105. curl -X POST "http://localhost:8080/v1/chat/completions" \
  1106. -H "Content-Type: application/json" \
  1107. -d '{
  1108. "model": "mlx-community/Qwen3.5-4B-MLX-4bit",
  1109. "messages": [{
  1110. "role": "user",
  1111. "content": [
  1112. {"type": "text", "text": "Identify the main animal in this image."},
  1113. {"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}}
  1114. ]
  1115. }],
  1116. "response_format": {
  1117. "type": "json_schema",
  1118. "json_schema": {
  1119. "name": "AnimalResult",
  1120. "strict": true,
  1121. "schema": {
  1122. "type": "object",
  1123. "properties": {
  1124. "animal": {"type": "string", "enum": ["dog", "cat", "bird", "unknown"]},
  1125. "species": {"type": "string", "maxLength": 60},
  1126. "description": {"type": "string", "maxLength": 200}
  1127. },
  1128. "required": ["animal", "species", "description"],
  1129. "additionalProperties": false
  1130. }
  1131. }
  1132. },
  1133. "max_tokens": 256
  1134. }'
  1135. ```
  1136. Structured outputs are also supported with:
  1137. - Streaming chat completions by setting `"stream": true`
  1138. - The responses API via `text.format` on `/v1/responses`
  1139. - Text-only requests using the same `response_format` shape
  1140. Structured outputs are not currently supported with speculative decoding.
  1141. #### How It Works
  1142. - A dedicated generation thread runs a `BatchGenerator` that processes multiple requests in parallel
  1143. - Image requests are prefilled individually with their own vision embeddings, then join the shared decoding batch
  1144. - Text-only requests are batched together for efficient prefill
  1145. - After prefill, all requests decode together in a single batch, sharing GPU compute
  1146. #### Available Endpoints
  1147. - `/models` and `/v1/models` - Discover cached and local models, including their loaded status; accepts repeated `model_dir` query parameters
  1148. - `/chat/completions` and `/v1/chat/completions` - OpenAI-compatible chat-style interaction endpoint with support for images, audio, and text
  1149. - `/responses` and `/v1/responses` - OpenAI-compatible responses endpoint
  1150. - `/responses/compact` and `/v1/responses/compact` - Compact conversation history into replayable state; see [compaction and APC](docs/usage.md#conversation-compaction-and-apc)
  1151. - `/embeddings` and `/v1/embeddings` - OpenAI-compatible embeddings endpoint backed by native MLX embedding models
  1152. - `/v1/decisions` - Predict named typed decisions using the shared decision API
  1153. - `/v1/meshes` - Asynchronous image-to-3D jobs, with polling and downloads at `/v1/meshes/{id}` and `/v1/meshes/{id}/content`; see [3D serving](mlx_vlm/models/idol/README.md#server)
  1154. - `/v1/rerank` - Rank text or multimodal documents by relevance to a query
  1155. - `/audio/speech` and `/v1/audio/speech` - OpenAI-compatible text-to-speech endpoint backed by `mlx-audio` TTS models
  1156. - `/audio/transcriptions` and `/v1/audio/transcriptions` - OpenAI-compatible speech-to-text endpoint backed by `mlx-audio` STT models
  1157. - `/audio/translations` and `/v1/audio/translations` - OpenAI-compatible audio translation endpoint for STT models that expose a translation task
  1158. - `/audio/diarizations` and `/v1/audio/diarizations` - Recording-wide speaker turns using MLX-Audio Nemotron diarization
  1159. - `/audio/speaker_embeddings` and `/v1/audio/speaker_embeddings` - Normalized ECAPA speaker vectors from separate speech windows
  1160. - `/v1/realtime` - WebSocket-based realtime full-duplex speech. See the [Nemotron VoiceChat guide](mlx_vlm/models/nemotron_voicechat/README.md#realtime-websocket-api) for supported models and usage.
  1161. - `/health` - Check server status
  1162. - `/metrics` and `/v1/metrics` - Inspect rolling request metrics, throughput, and runtime counters
  1163. - `/unload` - Unload all loaded model caches from memory
  1164. #### Usage Examples
  1165. ##### List available models
  1166. ```sh
  1167. curl "http://localhost:8080/models"
  1168. ```
  1169. `/models` and `/v1/models` list loaded models and local model candidates from the
  1170. Hugging Face cache and any configured or request-specific model directories. Each entry includes
  1171. `loaded: true` when the model is currently loaded in this server process, or
  1172. `loaded: false` when it is discovered but not loaded. Discovery does not load
  1173. models or guarantee that every discovered architecture is supported.
  1174. Discovery reads model metadata (`config.json`, or `model_index.json` for image
  1175. pipelines) and checks for nonempty safetensors files. Sharded checkpoints must
  1176. have every shard referenced by their weight index. Tokenizer metadata is not
  1177. required, and checkpoint Python code is never executed by discovery.
  1178. All cached revisions are considered, preferring `main`. If only another revision
  1179. is usable, the entry's ID is its absolute snapshot path so selecting it uses that
  1180. local revision. Custom models also use absolute paths. Symlink aliases and
  1181. loaded local models are deduplicated by their resolved directory.
  1182. Each `--model-dir` can be a model folder or a parent whose immediate children
  1183. are model folders; it does not recursively scan arbitrary directories. Missing
  1184. paths and invalid model folders are skipped. Hugging Face's `HF_HUB_CACHE` and
  1185. `HF_HOME` settings select a custom Hugging Face cache directory as usual.
  1186. You can also pass directories directly to either models endpoint:
  1187. ```sh
  1188. curl --get "http://localhost:8080/v1/models" \
  1189. --data-urlencode "model_dir=/Volumes/Models" \
  1190. --data-urlencode "model_dir=/Users/me/my custom model"
  1191. ```
  1192. These paths refer to the server's filesystem. They supplement the cache and
  1193. configured directories for this request only; they do not change server settings.
  1194. Use repeated `model_dir` parameters for multiple paths. Empty values are ignored.
  1195. ##### Embeddings
  1196. ```sh
  1197. curl -X POST "http://localhost:8080/v1/embeddings" \
  1198. -H "Content-Type: application/json" \
  1199. -d '{
  1200. "model": "sentence-transformers/all-MiniLM-L6-v2",
  1201. "input": ["The quick brown fox.", "A fast auburn fox."]
  1202. }'
  1203. ```
  1204. Preload a default with `--embedding-model <repo-or-path>`. Supported architectures: BERT, XLM-RoBERTa, ModernBERT, Qwen3-Embedding, EmbeddingGemma (gemma3), LFM2, SigLIP (text), Qwen3-VL-Embedding, and Llama-Nemotron-VL, plus LLM2Vec bidirectional Llama. ColBERT-style multi-vector models (ColIdefics3, ColQwen2.5, LFM2-ColBERT) are also available for late-interaction use.
  1205. ##### Reranking
  1206. ```sh
  1207. curl -X POST "http://localhost:8080/v1/rerank" \
  1208. -H "Content-Type: application/json" \
  1209. -d '{
  1210. "model": "mlx-community/Qwen3-Reranker-0.6B-4bit",
  1211. "query": "What is the capital of France?",
  1212. "documents": ["Berlin is in Germany.", "Paris is the capital of France."],
  1213. "top_n": 1,
  1214. "return_documents": true
  1215. }'
  1216. ```
  1217. Preload a default with `--reranker-model <repo-or-path>`. Supported text rerankers include one-label BERT, XLM-RoBERTa, and ModernBERT sequence-classification checkpoints, plus Qwen3 generative rerankers. Qwen3-VL rerankers also accept objects containing `text`, `image`, `image_url`, `video`, or `video_url`. Sequence-classification rerankers accept text pairs and do not support custom instructions.
  1218. ##### Text Input
  1219. ```sh
  1220. curl -X POST "http://localhost:8080/chat/completions" \
  1221. -H "Content-Type: application/json" \
  1222. -d '{
  1223. "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
  1224. "messages": [
  1225. {
  1226. "role": "user",
  1227. "content": "Hello, how are you"
  1228. }
  1229. ],
  1230. "stream": true,
  1231. "max_tokens": 100
  1232. }'
  1233. ```
  1234. ##### Image Input
  1235. ```sh
  1236. curl -X POST "http://localhost:8080/chat/completions" \
  1237. -H "Content-Type: application/json" \
  1238. -d '{
  1239. "model": "mlx-community/Qwen2.5-VL-32B-Instruct-8bit",
  1240. "messages":
  1241. [
  1242. {
  1243. "role": "system",
  1244. "content": "You are a helpful assistant."
  1245. },
  1246. {
  1247. "role": "user",
  1248. "content": [
  1249. {
  1250. "type": "text",
  1251. "text": "This is today's chart for energy demand in California. Can you provide an analysis of the chart and comment on the implications for renewable energy in California?"
  1252. },
  1253. {
  1254. "type": "input_image",
  1255. "image_url": "/path/to/repo/examples/images/renewables_california.png"
  1256. }
  1257. ]
  1258. }
  1259. ],
  1260. "stream": true,
  1261. "max_tokens": 1000
  1262. }'
  1263. ```
  1264. ##### Audio Input
  1265. ```sh
  1266. curl -X POST "http://localhost:8080/chat/completions" \
  1267. -H "Content-Type: application/json" \
  1268. -d '{
  1269. "model": "mlx-community/gemma-3n-E2B-it-4bit",
  1270. "messages": [
  1271. {
  1272. "role": "user",
  1273. "content": [
  1274. { "type": "text", "text": "Describe what you hear in these audio files" },
  1275. { "type": "input_audio", "input_audio": "/path/to/audio1.wav" },
  1276. { "type": "input_audio", "input_audio": "https://example.com/audio2.mp3" }
  1277. ]
  1278. }
  1279. ],
  1280. "stream": true,
  1281. "max_tokens": 500
  1282. }'
  1283. ```
  1284. ##### Text-to-Speech
  1285. ```sh
  1286. curl -X POST "http://localhost:8080/v1/audio/speech" \
  1287. -H "Content-Type: application/json" \
  1288. -d '{
  1289. "model": "mlx-community/pocket-tts",
  1290. "input": "Hello from MLX VLM.",
  1291. "voice": "fantine",
  1292. "response_format": "mp3"
  1293. }' \
  1294. --output speech.mp3
  1295. ```
  1296. ##### Speech-to-Text
  1297. ```sh
  1298. curl -X POST "http://localhost:8080/v1/audio/transcriptions" \
  1299. -F model=mlx-community/parakeet-tdt-0.6b-v3 \
  1300. -F file=@/path/to/audio.mp3 \
  1301. -F response_format=json
  1302. ```
  1303. Models with instruction-based audio tasks can also receive `task`, `prompt`, and
  1304. `system_prompt` as multipart fields. For example, with Nemotron Transcribe support
  1305. installed in `mlx-audio`:
  1306. ```sh
  1307. curl -X POST "http://localhost:8080/v1/audio/transcriptions" \
  1308. -F model=/path/to/nemotron-transcribe \
  1309. -F file=@/path/to/meeting.wav \
  1310. -F task=qa \
  1311. -F 'prompt=What time is the meeting?'
  1312. ```
  1313. Nemotron supports `asr`, `qa`, `saa` (speaker attribution), `rich`, and `translate`.
  1314. For a spoken request, use `task=qa` with `system_prompt=Answer the spoken question.`
  1315. and omit `prompt`. Translation requires a target `language`, such as `German`;
  1316. `/v1/audio/translations` selects `task=translate` regardless of the supplied task.
  1317. Task values and system instructions depend on the selected model. The server
  1318. forwards `prompt` to the model's declared `prompt`, `context`, or `text` parameter,
  1319. preserving an explicitly supplied `context` or `text` value. `stream=true` returns
  1320. NDJSON text chunks; Nemotron encodes the complete audio before emitting text.
  1321. ##### Multi-Modal (Image + Audio)
  1322. ```sh
  1323. curl -X POST "http://localhost:8080/chat/completions" \
  1324. -H "Content-Type: application/json" \
  1325. -d '{
  1326. "model": "mlx-community/gemma-3n-E2B-it-4bit",
  1327. "messages": [
  1328. {
  1329. "role": "user",
  1330. "content": [
  1331. {"type": "input_image", "image_url": "/path/to/image.jpg"},
  1332. {"type": "input_audio", "input_audio": "/path/to/audio.wav"}
  1333. ]
  1334. }
  1335. ],
  1336. "max_tokens": 100
  1337. }'
  1338. ```
  1339. ##### Responses Endpoint
  1340. ```sh
  1341. curl -X POST "http://localhost:8080/responses" \
  1342. -H "Content-Type: application/json" \
  1343. -d '{
  1344. "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
  1345. "messages": [
  1346. {
  1347. "role": "user",
  1348. "content": [
  1349. {"type": "input_text", "text": "What is in this image?"},
  1350. {"type": "input_image", "image_url": "/path/to/image.jpg"}
  1351. ]
  1352. }
  1353. ],
  1354. "max_tokens": 100
  1355. }'
  1356. ```
  1357. #### Request Parameters
  1358. - `model`: Model identifier (required)
  1359. - `messages`: Chat messages for chat/OpenAI endpoints
  1360. - `max_tokens`: Maximum tokens to generate
  1361. - `temperature`: Sampling temperature
  1362. - `top_p`: Top-p sampling parameter
  1363. - `top_k`: Top-k sampling cutoff
  1364. - `min_p`: Min-p sampling threshold
  1365. - `repetition_penalty`: Penalty applied to repeated tokens
  1366. - `enable_thinking`: Override the server thinking-mode default for a request (`true` or `false`)
  1367. - `thinking_budget`: Maximum tokens allowed inside the thinking block
  1368. - `thinking_start_token`: Token that opens a thinking block
  1369. - `thinking_end_token`: Token that closes a thinking block
  1370. - `stream`: Enable streaming responses
  1371. ## 1-bit Affine Inference
  1372. MLX-VLM can load existing affine 1-bit MLX checkpoints without a custom MLX
  1373. build. When a checkpoint declares `"bits": 1`, compatible `Linear` and
  1374. `Embedding` layers are replaced automatically with an inference-only module
  1375. that JIT-compiles its Metal kernel from Python.
  1376. The checkpoint must use MLX's packed `uint32` weight layout, include `scales`
  1377. and `biases`, and declare a group size of `32`, `64`, or `128`:
  1378. ```json
  1379. {
  1380. "quantization": {
  1381. "group_size": 64,
  1382. "bits": 1,
  1383. "mode": "affine"
  1384. }
  1385. }
  1386. ```
  1387. Load and generate normally; no extra inference flag is needed:
  1388. ```python
  1389. from mlx_vlm import generate, load
  1390. model, processor = load("path/to/1bit-model")
  1391. result = generate(model, processor, "Describe this image", image=["image.jpg"])
  1392. ```
  1393. This path is for inference from an already quantized checkpoint. Converting a
  1394. floating-point model to 1-bit still requires a quantizer that can produce the
  1395. packed weights and affine parameters.
  1396. ## Activation Quantization (CUDA)
  1397. When running on NVIDIA GPUs with MLX CUDA, models quantized with `mxfp8` or `nvfp4` modes require activation quantization to work properly. This converts `QuantizedLinear` layers to `QQLinear` layers which quantize both weights and activations.
  1398. ### Command Line
  1399. Use the `-qa` or `--quantize-activations` flag:
  1400. ```sh
  1401. mlx_vlm.generate --model /path/to/mxfp8-model --prompt "Describe this image" --image /path/to/image.jpg -qa
  1402. ```
  1403. ### Python API
  1404. Pass `quantize_activations=True` to the `load` function:
  1405. ```python
  1406. from mlx_vlm import load, generate
  1407. # Load with activation quantization enabled
  1408. model, processor = load(
  1409. "path/to/mxfp8-quantized-model",
  1410. quantize_activations=True
  1411. )
  1412. # Generate as usual
  1413. output = generate(model, processor, "Describe this image", image=["image.jpg"])
  1414. ```
  1415. ### Supported Quantization Modes
  1416. - `mxfp8` - 8-bit MX floating point
  1417. - `nvfp4` - 4-bit NVIDIA floating point
  1418. > **Note**: This feature is required for mxfp/nvfp quantized models on CUDA. On Apple Silicon (Metal), these models work without the flag.
  1419. ## Multi-Image Chat Support
  1420. MLX-VLM supports analyzing multiple images simultaneously with select models. This feature enables more complex visual reasoning tasks and comprehensive analysis across multiple images in a single conversation.
  1421. ### Usage Examples
  1422. #### Python Script
  1423. ```python
  1424. from mlx_vlm import load, generate
  1425. from mlx_vlm.prompt_utils import apply_chat_template
  1426. from mlx_vlm.utils import load_config
  1427. model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
  1428. model, processor = load(model_path)
  1429. config = model.config
  1430. images = ["path/to/image1.jpg", "path/to/image2.jpg"]
  1431. prompt = "Compare these two images."
  1432. formatted_prompt = apply_chat_template(
  1433. processor, config, prompt, num_images=len(images)
  1434. )
  1435. output = generate(model, processor, formatted_prompt, images, verbose=False)
  1436. print(output)
  1437. ```
  1438. #### Command Line
  1439. ```sh
  1440. mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Compare these images" --image path/to/image1.jpg path/to/image2.jpg
  1441. ```
  1442. ## Video Understanding
  1443. MLX-VLM also supports video analysis such as captioning, summarization, and more, with select models.
  1444. ### Supported Models
  1445. The following models support video chat:
  1446. 1. Qwen2-VL
  1447. 2. Qwen2.5-VL
  1448. 3. Idefics3
  1449. 4. LLaVA
  1450. 5. MiniMax M3 / M3.1 Flash
  1451. 6. LLaVA-OneVision
  1452. 7. Mage-VL
  1453. 8. Muse Spark Open
  1454. With more coming soon.
  1455. ### Usage Examples
  1456. #### Command Line
  1457. ```sh
  1458. mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Describe this video" --video path/to/video.mp4 --fps 1.0
  1459. ```
  1460. #### Python
  1461. Pass the video to `apply_chat_template` with `video=` (and optionally `fps=`) so the prompt gets a video placeholder, then pass the same path to `generate`. There is no `num_videos` argument.
  1462. ```python
  1463. from mlx_vlm import load, generate
  1464. from mlx_vlm.prompt_utils import apply_chat_template
  1465. from mlx_vlm.utils import load_config
  1466. model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
  1467. model, processor = load(model_path)
  1468. config = load_config(model_path)
  1469. video = "path/to/video.mp4"
  1470. formatted_prompt = apply_chat_template(
  1471. processor, config, "Describe this video.", video=video, fps=1.0
  1472. )
  1473. output = generate(
  1474. model, processor, formatted_prompt, video=video, fps=1.0, max_tokens=100
  1475. )
  1476. print(output)
  1477. ```
  1478. ## Vision Feature Caching
  1479. In multi-turn conversations about an image, the vision encoder runs on every turn even though the image hasn't changed. `VisionFeatureCache` stores projected vision features in an LRU cache keyed by image path, so the expensive vision encoder is only called once per unique image.
  1480. ### How It Works
  1481. 1. **First turn (cache miss)** -- `encode_image()` runs the full vision pipeline (vision tower + projector), stores the result in the cache, and passes it to the language model.
  1482. 2. **Subsequent turns (cache hit)** -- the cached features are passed directly via `cached_image_features`, skipping the vision encoder entirely.
  1483. 3. **Image switch** -- when the image changes, it's a new cache key so features are computed and cached. Switching back to a previous image is a cache hit.
  1484. The cache holds up to 8 entries (configurable) and uses LRU eviction.
  1485. ### CLI
  1486. All chat interfaces use `VisionFeatureCache` automatically:
  1487. ```sh
  1488. # Gradio chat UI
  1489. python -m mlx_vlm.chat_ui --model google/gemma-4-26b-a4b-it
  1490. # Interactive chat with Rich UI (load images with /image command)
  1491. python -m mlx_vlm.chat --model google/gemma-4-26b-a4b-it
  1492. # Inline chat mode
  1493. python -m mlx_vlm.generate \
  1494. --model google/gemma-4-26b-a4b-it \
  1495. --image path/to/image.jpg \
  1496. --chat \
  1497. --max-tokens 200
  1498. ```
  1499. ### Python
  1500. ```python
  1501. from mlx_vlm import load, stream_generate, VisionFeatureCache
  1502. from mlx_vlm.prompt_utils import apply_chat_template
  1503. model, processor = load("google/gemma-4-26b-a4b-it")
  1504. cache = VisionFeatureCache()
  1505. image = "path/to/image.jpg"
  1506. # Turn 1 -- cache miss, encodes image
  1507. prompt1 = apply_chat_template(processor, model.config, "Describe this image.", num_images=1)
  1508. for chunk in stream_generate(model, processor, prompt1, image=[image],
  1509. max_tokens=200, vision_cache=cache):
  1510. print(chunk.text, end="")
  1511. # Turn 2 -- cache hit, skips vision encoder
  1512. prompt2 = apply_chat_template(processor, model.config, "What colors do you see?", num_images=1)
  1513. for chunk in stream_generate(model, processor, prompt2, image=[image],
  1514. max_tokens=200, vision_cache=cache):
  1515. print(chunk.text, end="")
  1516. ```
  1517. ### Server
  1518. The server caches vision features automatically across requests for the same image. No configuration needed -- the cache is created when a model loads and cleared on unload.
  1519. ```sh
  1520. mlx_vlm.server --model google/gemma-4-26b-a4b-it
  1521. ```
  1522. Multi-turn conversations via `/v1/chat/completions` (streaming and non-streaming) and `/responses` all benefit. The same image sent across multiple requests will only be encoded once.
  1523. ### Performance
  1524. Tested on `google/gemma-4-26b-a4b-it` over 10 multi-turn conversation turns:
  1525. | Metric | Without Cache | With Cache |
  1526. |--------|--------------|------------|
  1527. | Prompt TPS | ~48 | ~550-825 |
  1528. | Speedup | -- | **11x+** |
  1529. | Peak Memory | 52.66 GB | 52.66 GB (flat) |
  1530. Generation speed (~31 tok/s) and memory are unaffected -- only prompt processing gets faster.
  1531. ## TurboQuant KV Cache
  1532. TurboQuant compresses the KV cache during generation, enabling longer context lengths with less memory while maintaining quality.
  1533. ### Quick Start
  1534. ```sh
  1535. # 3.5-bit KV cache quantization (3-bit keys + 4-bit values)
  1536. mlx_vlm generate \
  1537. --model mlx-community/Qwen3.5-4B-4bit \
  1538. --kv-bits 3.5 \
  1539. --kv-quant-scheme turboquant \
  1540. --prompt "Your long prompt here..."
  1541. ```
  1542. ```python
  1543. from mlx_vlm import generate
  1544. result = generate(
  1545. model, processor, prompt,
  1546. kv_bits=3.5,
  1547. kv_quant_scheme="turboquant",
  1548. max_tokens=256,
  1549. )
  1550. ```
  1551. ```sh
  1552. # Server with TurboQuant
  1553. mlx_vlm server \
  1554. --model google/gemma-4-26b-a4b-it \
  1555. --kv-bits 3.5 \
  1556. --kv-quant-scheme turboquant
  1557. ```
  1558. ### How It Works
  1559. TurboQuant uses random rotation + codebook quantization ([arXiv:2504.19874](https://arxiv.org/abs/2504.19874)) to compress KV cache entries from 16-bit to 2-4 bits per dimension:
  1560. - **Keys & Values**: MSE codebook quantization with Hadamard rotation
  1561. - **Fractional bits** (e.g. 3.5): uses lower bits for keys, higher for values (3-bit K + 4-bit V)
  1562. Custom Metal kernels fuse score computation and value aggregation directly on packed quantized data, avoiding full dequantization during decode.
  1563. ### Performance
  1564. Tested on Qwen3.5-4B-4bit at 128k context:
  1565. | Metric | Baseline | TurboQuant 3.5-bit |
  1566. |--------|----------|-------------------|
  1567. | KV Memory | 4.1 GB | 0.97 GB (**76% reduction**) |
  1568. | Peak Memory | 18.3 GB | 17.3 GB (**-1.0 GB**) |
  1569. At 512k+ contexts, TurboQuant's per-layer attention is **faster than FP16 SDPA** due to reduced memory bandwidth requirements.
  1570. Tested on gemma-4-31b-it at 128k context:
  1571. | Metric | Baseline | TurboQuant 3.5-bit |
  1572. |--------|----------|-------------------|
  1573. | KV Memory | 13.3 GB | 4.9 GB (**63% reduction**) |
  1574. | Peak Memory | 75.2 GB | 65.8 GB (**-9.4 GB**) |
  1575. ### Supported Bit Widths
  1576. | Bits | Compression | Best For |
  1577. |------|------------|----------|
  1578. | 2 | ~8x | Maximum compression, some quality loss |
  1579. | 3 | ~5x | Good balance of quality and compression |
  1580. | 3.5 | ~4.5x | Recommended default (3-bit keys + 4-bit values) |
  1581. | 4 | ~4x | Best quality, moderate compression |

Lines 2,001–2,042 are not shown. The link to the file on GitHub has every line.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.