Skip to content
Back to search

vllm-mlx

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/waybarrios--vllm-mlx/badge.svg)](/listings/waybarrios--vllm-mlx)

HTML

<a href="/listings/waybarrios--vllm-mlx"><img src="/api/listings/waybarrios--vllm-mlx/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

New users: follow the [isolated release install](docs/getting-started/installation.md#install-a-release) and [first-response check](docs/getting-started/quickstart.md#first-response). The example below assumes that environment is activated. Initial model download/loading time depends on the machine and connection.

The version below is this walkthrough's pinned reference release, not an automatically updated latest version.

python -m pip install 'vllm-mlx==0.4.1'
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --host 127.0.0.1 --port 8000

After the first response, restart with `--continuous-batching` to try [continuous batching and its cache options](docs/guides/continuous-batching.md). The minimal command above uses the default engine without that flag.

**OpenAI SDK:**

Install `openai` in your Python client environment first; it is not installed by the server package. See the [client setup](docs/getting-started/quickstart.md#option-1-openai-compatible-server).

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.create(model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hi!"}])
print(r.choices[0].message.content)

**Anthropic SDK / Claude Code:**

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 340 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # vllm-mlx
  2. **Continuous batching + OpenAI + Anthropic APIs in one server. Native Apple Silicon inference.**
  3. **Read this in other languages:** [English](README.md) · [Español](README.es.md) · [Français](README.fr.md) · [中文](README.zh.md)
  4. [![PyPI version](https://img.shields.io/pypi/v/vllm-mlx.svg)](https://pypi.org/project/vllm-mlx/)
  5. [![PyPI Downloads](https://static.pepy.tech/badge/vllm-mlx)](https://pepy.tech/projects/vllm-mlx)
  6. [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
  7. [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
  8. [![Apple Silicon](https://img.shields.io/badge/Apple-Silicon-black.svg)](https://support.apple.com/en-us/HT211814)
  9. [![GitHub stars](https://img.shields.io/github/stars/waybarrios/vllm-mlx.svg?style=social)](https://github.com/waybarrios/vllm-mlx)
  10. ---
  11. ## What is vllm-mlx?
  12. A vLLM-style inference server for Apple Silicon Macs. Unlike `Ollama` or `mlx-lm` used directly, it ships **continuous batching, paged KV cache, prefix caching, and SSD-tiered cache**, and exposes **both OpenAI `/v1/*` and Anthropic `/v1/messages`** from a single process. Run LLMs, vision models, audio, and embeddings on Metal with unified memory, no conversion step.
  13. ## Quick start
  14. New users: follow the [isolated release install](docs/getting-started/installation.md#install-a-release)
  15. and [first-response check](docs/getting-started/quickstart.md#first-response).
  16. The example below assumes that environment is activated. Initial model
  17. download/loading time depends on the machine and connection.
  18. The version below is this walkthrough's pinned reference release, not an
  19. automatically updated latest version.
  20. ```bash
  21. python -m pip install 'vllm-mlx==0.4.1'
  22. vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --host 127.0.0.1 --port 8000
  23. ```
  24. After the first response, restart with `--continuous-batching` to try
  25. [continuous batching and its cache options](docs/guides/continuous-batching.md).
  26. The minimal command above uses the default engine without that flag.
  27. **OpenAI SDK:**
  28. Install `openai` in your Python client environment first; it is not installed
  29. by the server package. See the [client setup](docs/getting-started/quickstart.md#option-1-openai-compatible-server).
  30. ```python
  31. from openai import OpenAI
  32. client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
  33. r = client.chat.completions.create(model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hi!"}])
  34. print(r.choices[0].message.content)
  35. ```
  36. **Anthropic SDK / Claude Code:**
  37. ```bash
  38. export ANTHROPIC_BASE_URL=http://localhost:8000
  39. export ANTHROPIC_API_KEY=not-needed
  40. claude
  41. ```
  42. ## Coding CLI compatibility
  43. **Validated with OpenCode, pi, Codex, Claude Code, GitHub Copilot CLI, Cline CLI,
  44. and OpenClaw's embedded agent.** All seven completed a streamed tool interaction
  45. and an exact file edit in one local run with Qwen3.8-27B-4bit on September 19,
  46. 2026. Results apply to the tested client versions and settings.
  47. See the [validated CLI matrix and setup guide](docs/guides/client-acceptance.md#validated-cli-versions)
  48. for versions, API transports, reproduction commands, and coverage limits.
  49. ## Features
  50. ### APIs
  51. - **OpenAI-compatible**: `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/rerank`, `/v1/responses`
  52. - **Anthropic-compatible**: `/v1/messages` (streaming, tool use, system prompts)
  53. - **MCP Tool Calling**: 19 parsers (OpenAI, Anthropic, Gemini, Qwen, DeepSeek, Gemma, and more)
  54. - **Structured output**: JSON Schema via `response_format` (lm-format-enforcer)
  55. ### Throughput & memory
  56. - **Continuous batching**: high throughput for concurrent requests
  57. - **Paged KV cache**: memory-efficient with prefix sharing
  58. - **SSD-tiered KV cache**: spill prefix cache to disk for long-context agents (`--ssd-cache-dir`)
  59. - **Warm prompts**: preload popular prefixes at startup (`--warm-prompts`) for 1.3-2.25x TTFT
  60. - **Prefix cache**: trie-based, shared across requests
  61. ### Multimodal
  62. - **Text + image + video + audio** from one server
  63. - Vision models: Gemma 3, Gemma 4, Qwen3-VL, Pixtral, Llama vision
  64. - **Audio input** in chat (`audio_url` content blocks)
  65. - **Native TTS**: 11 voices, 15+ languages (Kokoro, Chatterbox, VibeVoice, VoxCPM)
  66. - **STT**: Whisper family with RTF up to 197x on M4 Max
  67. ### Reasoning & advanced
  68. - **Reasoning extraction**: Qwen3 and DeepSeek-R1 parsers (`--reasoning-parser`)
  69. - **MoE expert reduction**: `--moe-top-k` for +7-16% on Qwen3-30B-A3B
  70. - **Speculative decoding**: `--enable-mtp` for supported models
  71. - **Sparse prefill**: attention-based `--specprefill` for supported model/draft combinations
  72. ### Observability
  73. - **Prometheus metrics**: `/metrics` endpoint with `--enable-metrics`
  74. - **Built-in benchmarker**: `vllm-mlx bench-serve` for prompt sweeps with CSV/JSON output
  75. ### Native GPU acceleration
  76. - Apple Silicon only (M1, M2, M3, M4, M5) with Metal kernels via MLX
  77. - Unified memory, no model conversion
  78. ## Performance
  79. **LLM decode (M4 Max, 128 GB, greedy, single stream):**
  80. | Model | Tok/s | Memory |
  81. |-------|------:|-------:|
  82. | Qwen3-0.6B-8bit | 417.9 | 0.7 GB |
  83. | Llama-3.2-3B-Instruct-4bit | 205.6 | 1.8 GB |
  84. | Qwen3-30B-A3B-4bit | 127.7 | ~18 GB |
  85. **Audio speech-to-text (M4 Max, RTF = real-time factor):**
  86. | Model | RTF | Use case |
  87. |-------|----:|----------|
  88. | whisper-tiny | 197x | Real-time / low latency |
  89. | whisper-large-v3-turbo | 55x | Quality + speed |
  90. | whisper-large-v3 | 24x | Highest accuracy |
  91. See [docs/benchmarks/](docs/benchmarks/) for continuous-batching results, KV-cache quantization (4-bit / 8-bit / fp16), and MoE top-k sweeps.
  92. ## Examples
  93. ### Anthropic API (Claude Code, OpenCode)
  94. ```bash
  95. vllm-mlx serve mlx-community/Qwen3-8B-4bit --port 8000
  96. export ANTHROPIC_BASE_URL=http://localhost:8000
  97. export ANTHROPIC_API_KEY=not-needed
  98. claude
  99. ```
  100. ### Reasoning models (Qwen3, DeepSeek-R1)
  101. ```bash
  102. vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3
  103. ```
  104. ```python
  105. r = client.chat.completions.create(
  106. model="mlx-community/Qwen3-8B-4bit",
  107. messages=[{"role": "user", "content": "What is 17 * 23?"}],
  108. )
  109. print("Thinking:", r.choices[0].message.reasoning)
  110. print("Answer:", r.choices[0].message.content)
  111. ```
  112. ### Multimodal (image + text)
  113. ```bash
  114. vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000
  115. ```
  116. ```python
  117. r = client.chat.completions.create(
  118. model="mlx-community/Qwen3-VL-4B-Instruct-3bit",
  119. messages=[{"role": "user", "content": [
  120. {"type": "text", "text": "What is in this image?"},
  121. {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}},
  122. ]}],
  123. )
  124. ```
  125. ### Structured output (JSON Schema)
  126. With the text server from the quick start:
  127. ```python
  128. r = client.chat.completions.create(
  129. model="mlx-community/Llama-3.2-3B-Instruct-4bit",
  130. messages=[{"role": "user", "content": "List 3 colors."}],
  131. response_format={
  132. "type": "json_schema",
  133. "json_schema": {
  134. "schema": {"type": "object", "properties": {"colors": {"type": "array", "items": {"type": "string"}}}}
  135. },
  136. },
  137. )
  138. ```
  139. ### Reranking (`/v1/rerank`)
  140. ```bash
  141. curl http://localhost:8000/v1/rerank -H 'Content-Type: application/json' -d '{
  142. "model": "default",
  143. "query": "apple silicon inference",
  144. "documents": ["MLX is Apples framework", "Metal kernels on M-series", "CUDA on NVIDIA"]
  145. }'
  146. ```
  147. The built-in MLX reranker forward path supports standard BERT/XLM-RoBERTa
  148. sequence-classification weights with `gelu`, `gelu_new`/`gelu_fast`, `relu`, or
  149. `silu`/`swish` `hidden_act` values. Other activations fail explicitly so custom
  150. reranker architectures can add a dedicated adapter instead of silently using the
  151. wrong activation.
  152. ### Embeddings
  153. ```bash
  154. vllm-mlx serve <llm-model> --embedding-model mlx-community/all-MiniLM-L6-v2-4bit
  155. ```
  156. ```python
  157. emb = client.embeddings.create(model="mlx-community/all-MiniLM-L6-v2-4bit", input=["Hello", "World"])
  158. ```
  159. ### Audio (TTS / STT)
  160. ```bash
  161. pip install vllm-mlx[audio]
  162. brew install espeak-ng # macOS, needed for non-English TTS
  163. python examples/tts_example.py "Hello, how are you?" --play
  164. python examples/tts_multilingual.py "Hola mundo" --lang es --play
  165. ```
  166. ### Built-in benchmarking
  167. ```bash
  168. vllm-mlx bench-serve --url http://localhost:8000 --concurrency 5 --prompts prompts.txt --output results.csv
  169. # Product-style workload with quality checks and metrics deltas
  170. vllm-mlx bench-serve --url http://localhost:8000 --workload workload.json --repetitions 5 --output results.json
  171. # Append workload rows into SQLite for longitudinal comparisons
  172. vllm-mlx bench-serve --url http://localhost:8000 --workload workload.json --repetitions 5 --format sqlite --output bench.db
  173. ```
  174. ### Model acquisition and conversion
  175. ```bash
  176. # Inspect repo metadata, file sizes, config, and rough fit before downloading weights
  177. vllm-mlx model inspect mlx-community/Llama-3.2-3B-Instruct-4bit
  178. # Acquire with resumable Hugging Face transfer and write a local artifact manifest
  179. vllm-mlx model acquire mlx-community/Llama-3.2-3B-Instruct-4bit --target-dir ./models/llama-3b-4bit
  180. # Wrap mlx-lm conversion and record the exact recipe in the converted artifact
  181. vllm-mlx model convert meta-llama/Llama-3.2-3B-Instruct --output ./models/llama-3b-mlx-q4 --quantize --q-bits 4 --q-group-size 64 --q-mode affine
  182. ```
  183. ### Prometheus metrics
  184. ```bash
  185. vllm-mlx serve <model> --enable-metrics
  186. curl http://localhost:8000/metrics
  187. ```
  188. ## Installation
  189. Use the [release walkthrough](docs/getting-started/installation.md#install-a-release)
  190. for an isolated environment and a pinned server version. If you already manage
  191. CLI tools with uv, its isolated equivalent is:
  192. ```bash
  193. uv tool install 'vllm-mlx==0.4.1'
  194. ```
  195. Keep [development checkouts](docs/getting-started/installation.md#development-checkout)
  196. separate. See the [Installation Guide](docs/getting-started/installation.md) for
  197. optional extras and [Audio Guide](docs/guides/audio.md) for audio setup.
  198. ## Documentation
  199. Browse the complete documentation at [vllm-mlx.is-a.dev](https://vllm-mlx.is-a.dev/).
  200. - **Getting started**: [Installation](docs/getting-started/installation.md) · [Quick Start](docs/getting-started/quickstart.md)
  201. - **Servers & APIs**: [OpenAI server](docs/guides/server.md) · [Anthropic Messages API](docs/guides/server.md#anthropic-messages-api) · [Python API](docs/guides/python-api.md)
  202. - **Client compatibility**: [Run coding-client acceptance checks](docs/guides/client-acceptance.md)
  203. - **Features**: [Multimodal](docs/guides/multimodal.md) · [Audio](docs/guides/audio.md) · [Embeddings](docs/guides/embeddings.md) · [Reasoning](docs/guides/reasoning.md) · [MCP & Tool Calling](docs/guides/mcp-tools.md) · [Tool Parsers](docs/guides/tool-calling.md)
  204. - **Performance**: [Continuous Batching](docs/guides/continuous-batching.md) · [Multi-Model Serving](docs/guides/model-registry.md) · [Warm Prompts](docs/guides/warm-prompts.md) · [MoE Top-K](docs/guides/moe-top-k.md)
  205. - **Reference**: [CLI](docs/reference/cli.md) · [Models](docs/reference/models.md) · [Configuration](docs/reference/configuration.md)
  206. - **Benchmarks**: [LLM](docs/benchmarks/llm.md) · [Image](docs/benchmarks/image.md) · [Video](docs/benchmarks/video.md) · [Audio](docs/benchmarks/audio.md)
  207. ## Architecture
  208. ```
  209. ┌─────────────────────────────────────────────────────────────────────────┐
  210. │ vllm-mlx Server │
  211. │ OpenAI /v1/* · Anthropic /v1/messages · /v1/rerank · /metrics │
  212. └─────────────────────────────────────────────────────────────────────────┘
  213. │
  214. ▼
  215. ┌─────────────────────────────────────────────────────────────────────────┐
  216. │ Continuous batching · Paged KV cache · Prefix cache · SSD tiering │
  217. └─────────────────────────────────────────────────────────────────────────┘
  218. │
  219. ┌─────────────┬────────────┴────────────┬─────────────┐
  220. ▼ ▼ ▼ ▼
  221. ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐
  222. │ mlx-lm │ │ mlx-vlm │ │ mlx-audio │ │mlx-embeddings │
  223. │ (LLMs) │ │ (Vision) │ │ (TTS + STT) │ │ (Embeddings) │
  224. └───────────────┘ └───────────────┘ └───────────────┘ └───────────────┘
  225. │
  226. ▼
  227. ┌─────────────────────────────────────────────────────────────────────────┐
  228. │ MLX · Metal kernels · Unified memory │
  229. └─────────────────────────────────────────────────────────────────────────┘
  230. ```
  231. ## Contributing
  232. Bug fixes, perf work, docs, and benchmarks on different Apple Silicon chips all welcome. See the [Contributing Guide](docs/development/contributing.md).
  233. ## License
  234. Apache 2.0. See [LICENSE](LICENSE).
  235. ## Citation
  236. ```bibtex
  237. @software{vllm_mlx2025,
  238. author = {Barrios, Wayner},
  239. title = {vllm-mlx: Apple Silicon MLX Backend for vLLM},
  240. year = {2025},
  241. url = {https://github.com/waybarrios/vllm-mlx},
  242. note = {Native GPU-accelerated LLM and vision-language model inference on Apple Silicon}
  243. }
  244. ```
  245. ## Acknowledgments
  246. - [MLX](https://github.com/ml-explore/mlx). Apple's ML framework.
  247. - [mlx-lm](https://github.com/ml-explore/mlx-lm). LLM inference library.
  248. - [mlx-vlm](https://github.com/Blaizzy/mlx-vlm). Vision-language models.
  249. - [mlx-audio](https://github.com/Blaizzy/mlx-audio). Text-to-Speech and Speech-to-Text.
  250. - [mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings). Text embeddings.
  251. - [Rapid-MLX](https://github.com/raullenchai/Rapid-MLX). Community fork of vllm-mlx.
  252. - [vLLM](https://github.com/vllm-project/vllm). High-throughput LLM serving. vllm-mlx is inspired by vLLM and adopts its continuous-batching and paged KV-cache design for Apple Silicon via MLX.
  253. ## Star history
  254. [![Star History Chart](https://star-history.dera.page/svg?repos=waybarrios/vllm-mlx&type=Date)](https://star-history.dera.page/#waybarrios/vllm-mlx&Date)
  255. ---
  256. **If vllm-mlx helped you, please star the repo. It helps more Apple Silicon devs find it.**

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.