Skip to content
Back to search

vmlx

vMLX - Use MLX models easily - JANGQ (GGUF for MLX) - Not dependant on mlx_vlm

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/jjang-ai--vmlx/badge.svg)](/listings/jjang-ai--vmlx)

HTML

<a href="/listings/jjang-ai--vmlx"><img src="/api/listings/jjang-ai--vmlx/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install from PyPI

Published on [PyPI as `vmlx`](https://pypi.org/project/vmlx/) -- install and run in one command:

# Recommended: uv (fast, no venv hassle)
brew install uv
uv tool install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

# Or: pipx (isolates from system Python)
brew install pipx
pipx install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

# Or: pip in a virtual environment
python3 -m venv ~/.vmlx-env && source ~/.vmlx-env/bin/activate
pip install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

> **Note:** On macOS 14+, bare `pip install` fails with "externally-managed-environment". Use `uv`, `pipx`, or a venv.

The vMLX inference server is now running at `http://0.0.0.0:8000` with an OpenAI + Anthropic compatible API. Works with any model from [mlx-community](https://huggingface.co/mlx-community) -- thousands of models ready to go.

Or download the desktop app

Get [MLX Studio](https://github.com/jjang-ai/mlxstudio/releases/latest) -- a native macOS app with chat UI, model management, image generation, and developer tools. No terminal required. Just download the DMG and drag to Applications.

Use with OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Use with Anthropic SDK

import anthropic

client = anthropic.Anthropic(base_url="http://localhost:8000/v1", api_key="not-needed")
message = client.messages.create(
    model="local",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}],
)
print(message.content[0].text)

Use with curl

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 914 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <picture>
  3. <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-dark.png">
  4. <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-light.png">
  5. <img alt="vMLX" src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-light.png" width="400">
  6. </picture>
  7. </p>
  8. <h3 align="center">MLX Inference Server for Apple Silicon</h3>
  9. <p align="center">
  10. Self-hosted inference server for LLMs, VLMs, and image generation on Apple Silicon.<br>
  11. OpenAI + Anthropic + Ollama compatible HTTP API. Self-hosted; no third-party API keys required.<br>
  12. Native MTP artifact detection and family-specific cache policy gates keep speculative/cache settings explicit and model-safe.
  13. </p>
  14. <p align="center">
  15. <em>Looking for a native Swift macOS app or Swift inference engine? See <a href="https://osaurus.ai">osaurus.ai</a>.</em>
  16. </p>
  17. <p align="center">
  18. <a href="https://pypi.org/project/vmlx/"><img src="https://img.shields.io/pypi/v/vmlx?color=%234B8BBE&label=PyPI&logo=python&logoColor=white" alt="PyPI" /></a>
  19. <a href="https://github.com/jjang-ai/vmlx/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-green?logo=apache" alt="License" /></a>
  20. <a href="https://github.com/jjang-ai/vmlx"><img src="https://img.shields.io/github/stars/jjang-ai/vmlx?style=social" alt="Stars" /></a>
  21. <img src="https://img.shields.io/badge/Apple_Silicon-M1%2FM2%2FM3%2FM4-black?logo=apple" alt="Apple Silicon" />
  22. <img src="https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white" alt="Python" />
  23. <img src="https://img.shields.io/badge/Electron-28-47848F?logo=electron&logoColor=white" alt="Electron" />
  24. <a href="https://ko-fi.com/jangml"><img src="https://img.shields.io/badge/Support-Ko--fi-FF5E5B?logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
  25. </p>
  26. <p align="center">
  27. <a href="#quickstart">Quickstart</a> &bull;
  28. <a href="#model-support">Models</a> &bull;
  29. <a href="#features">Features</a> &bull;
  30. <a href="#image-generation--editing">Image Gen</a> &bull;
  31. <a href="#api-reference">API</a> &bull;
  32. <a href="#desktop-app">Desktop App</a> &bull;
  33. <a href="#advanced-quantization">JANG</a> &bull;
  34. <a href="#cli-commands">CLI</a> &bull;
  35. <a href="#configuration">Config</a> &bull;
  36. <a href="#contributing">Contributing</a> &bull;
  37. <a href="#한국어-korean">한국어</a>
  38. </p>
  39. ---
  40. > **JANG 2-bit destroys MLX 4-bit on [MiniMax M2.5](https://huggingface.co/JANGQ-AI/MiniMax-M2.5-JANG_2L):**
  41. >
  42. > | Quantization | MMLU (200q) | Size |
  43. > |---|---|---|
  44. > | **JANG\_2L (2-bit)** | **74%** | **89 GB** |
  45. > | MLX 4-bit | 26.5% | 120 GB |
  46. > | MLX 3-bit | 24.5% | 93 GB |
  47. > | MLX 2-bit | 25% | 68 GB |
  48. >
  49. > Adaptive mixed-precision keeps critical layers at higher precision. Scores at [jangq.ai](https://jangq.ai). Models at [JANGQ-AI](https://huggingface.co/JANGQ-AI).
  50. <table align="center">
  51. <tr>
  52. <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/chat-tab.png" width="500" alt="Chat interface" /></td>
  53. <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/agentic-chat.png" width="500" alt="Agentic coding chat" /></td>
  54. </tr>
  55. <tr>
  56. <td align="center"><em>Chat with any MLX model -- thinking mode, streaming, and syntax highlighting</em></td>
  57. <td align="center"><em>Agentic chat with full coding capabilities -- tool use and structured output</em></td>
  58. </tr>
  59. </table>
  60. ---
  61. ## Quickstart
  62. ### Install from PyPI
  63. Published on [PyPI as `vmlx`](https://pypi.org/project/vmlx/) -- install and run in one command:
  64. ```bash
  65. # Recommended: uv (fast, no venv hassle)
  66. brew install uv
  67. uv tool install vmlx
  68. vmlx serve mlx-community/Qwen3-8B-4bit
  69. # Or: pipx (isolates from system Python)
  70. brew install pipx
  71. pipx install vmlx
  72. vmlx serve mlx-community/Qwen3-8B-4bit
  73. # Or: pip in a virtual environment
  74. python3 -m venv ~/.vmlx-env && source ~/.vmlx-env/bin/activate
  75. pip install vmlx
  76. vmlx serve mlx-community/Qwen3-8B-4bit
  77. ```
  78. > **Note:** On macOS 14+, bare `pip install` fails with "externally-managed-environment". Use `uv`, `pipx`, or a venv.
  79. The vMLX inference server is now running at `http://0.0.0.0:8000` with an OpenAI + Anthropic compatible API. Works with any model from [mlx-community](https://huggingface.co/mlx-community) -- thousands of models ready to go.
  80. ### Or download the desktop app
  81. Get [MLX Studio](https://github.com/jjang-ai/mlxstudio/releases/latest) -- a native macOS app with chat UI, model management, image generation, and developer tools. No terminal required. Just download the DMG and drag to Applications.
  82. ### Use with OpenAI SDK
  83. ```python
  84. from openai import OpenAI
  85. client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
  86. response = client.chat.completions.create(
  87. model="local",
  88. messages=[{"role": "user", "content": "Hello!"}],
  89. stream=True,
  90. )
  91. for chunk in response:
  92. print(chunk.choices[0].delta.content or "", end="", flush=True)
  93. ```
  94. ### Use with Anthropic SDK
  95. ```python
  96. import anthropic
  97. client = anthropic.Anthropic(base_url="http://localhost:8000/v1", api_key="not-needed")
  98. message = client.messages.create(
  99. model="local",
  100. max_tokens=1024,
  101. messages=[{"role": "user", "content": "Hello!"}],
  102. )
  103. print(message.content[0].text)
  104. ```
  105. ### Use with curl
  106. ```bash
  107. curl http://localhost:8000/v1/chat/completions \
  108. -H "Content-Type: application/json" \
  109. -d '{
  110. "model": "local",
  111. "messages": [{"role": "user", "content": "Hello!"}],
  112. "stream": true
  113. }'
  114. ```
  115. ---
  116. ## Model Support
  117. vMLX runs any MLX model. Point it at a HuggingFace repo or local path and go.
  118. | Type | Models |
  119. |------|--------|
  120. | **Text LLMs** | Qwen 2/2.5/3/3.5/3.6, Llama 3/3.1/3.2/3.3/4, Mistral/Mixtral, **Mistral-Medium-3.5** (ministral3), Mistral-Small-4, Gemma 3/4, Phi-4, DeepSeek V2/V3/V4, GLM-4/5, MiniMax M2.5/M2.7, Nemotron, **Laguna** (poolside), **ZAYA** (CCA + MoE), Kimi K2.5/K2.6, StepFun, and any mlx-lm model |
  121. | **Vision LLMs** | Qwen-VL, Qwen3.5-VL / Qwen3.6-VL, Pixtral, InternVL, LLaVA, Gemma 3n / 4-VL, Mistral-Medium-3.5 (PIXTRAL) |
  122. | **Multimodal Omni** | **Nemotron-3-Nano-Omni** (text + image + audio + video) — Parakeet audio encoder + RADIO ViT vision tower; routed via OmniMultimodalDispatcher across `/v1/chat/completions`, `/v1/messages`, `/v1/responses`, `/api/chat` |
  123. | **MoE Models** | Qwen 3.5/3.6 MoE (A3B/A10B), Mixtral, DeepSeek V2/V3/V4, MiniMax M2.5/M2.7, Llama 4, Laguna (256 routed experts top-8) |
  124. | **Hybrid SSM** | Nemotron-H, Jamba, GatedDeltaNet (Mamba + Attention), Qwen3.5-A3B hybrid, Granite MoE Hybrid, LFM2 |
  125. | **Image Gen** | Flux Schnell/Dev, Z-Image Turbo (via mflux) |
  126. | **Image Edit** | Qwen Image Edit (via mflux) |
  127. | **Embeddings** | Any mlx-lm compatible embedding model |
  128. | **Reranking** | Cross-encoder reranking models |
  129. | **Audio** | Kokoro TTS, Whisper STT (via mlx-audio) |
  130. ---
  131. ## Features
  132. ### Inference Engine
  133. | Feature | Description |
  134. |---------|-------------|
  135. | **Continuous Batching** | Handle multiple concurrent requests efficiently |
  136. | **Prefix Cache** | Reuse KV states for repeated prompts -- makes follow-up messages instant |
  137. | **Paged KV Cache** | Block-based caching with content-addressable deduplication |
  138. | **KV Cache Quantization** | Compress cached states to q4/q8 for 2-4x memory savings |
  139. | **Disk Cache (L2)** | Persist prompt caches to SSD -- survives server restarts |
  140. | **Block Disk Cache** | Per-block persistent cache paired with paged KV cache |
  141. | **Speculative Decoding** | Small draft model proposes tokens for 20-90% speedup |
  142. | **Prompt Lookup Decoding** | No draft model needed — reuses n-gram matches from the prompt/context. Best for structured or repetitive output (code, JSON, schemas). Enable with `--enable-pld`. |
  143. | **JIT Compilation** | `mx.compile` Metal kernel fusion (experimental) |
  144. | **Hybrid SSM Support** | Mamba/GatedDeltaNet layers handled correctly alongside attention |
  145. | **Distributed Compute** | Pipeline parallelism across multiple Macs via Thunderbolt 5 / Ethernet / WiFi |
  146. ### Distributed Inference (Multi-Mac)
  147. Run models too large for a single Mac across 2+ machines. Each Mac loads a subset of transformer layers and they communicate hidden states over the network.
  148. ```bash
  149. # On worker Macs:
  150. pip install vmlx
  151. vmlx-worker --secret mysecret
  152. # On coordinator Mac (runs the server):
  153. vmlx serve JANGQ-AI/Qwen3.5-Coder-Rerank-397B-A27B-JANG_2L --distributed --cluster-secret mysecret
  154. ```
  155. | Feature | Description |
  156. |---------|-------------|
  157. | **Pipeline Parallelism** | Split layers across nodes -- hidden state (~8KB/step) flows sequentially |
  158. | **Auto-Discovery** | Bonjour mDNS, UDP broadcast, HTTP probes, Tailscale, cached peers, manual IP |
  159. | **Capability-Scored Election** | Most powerful Mac becomes coordinator automatically |
  160. | **Any Network Works** | TB5 (120 Gbps), 10GbE, 1GbE, WiFi, Tailscale -- PP is not bandwidth-bound |
  161. | **JANG Support** | Each worker loads its layer range from JANG safetensors (mmap) |
  162. | **Live Node List** | Desktop app shows discovered nodes, link type, latency, layer assignments |
  163. | **Cluster API** | `/v1/cluster/status`, `/v1/cluster/nodes`, `/v1/cluster/scan` REST endpoints |
  164. ### 5-Layer Cache Architecture
  165. ```
  166. Request -> Tokens
  167. |
  168. L1: Memory-Aware Prefix Cache (or Paged Cache)
  169. | miss
  170. L2: Disk Cache (or Block Disk Store)
  171. | miss
  172. Inference -> float16 KV states
  173. |
  174. KV Quantization -> q4/q8 for storage
  175. |
  176. Store back into L1 + L2
  177. ```
  178. ### Tool Calling
  179. Auto-detected parsers for every major model family:
  180. `qwen` - `llama` - `mistral` - `hermes` - `deepseek` - `glm47` - `minimax` - `nemotron` - `granite` - `functionary` - `xlam` - `kimi` - `step3p5`
  181. ### Reasoning / Thinking Mode
  182. Auto-detected reasoning parsers that extract `<think>` blocks:
  183. `qwen3` (Qwen3, QwQ, StepFun) - `minimax_m2` (MiniMax M2/M2.5/M2.7) - `deepseek_r1` (DeepSeek R1, Gemma 3, GLM, Phi-4) - `openai_gptoss` (GLM Flash, GPT-OSS)
  184. ### Audio
  185. | Feature | Description |
  186. |---------|-------------|
  187. | **Text-to-Speech** | Kokoro TTS via mlx-audio -- multiple voices, streaming output |
  188. | **Speech-to-Text** | Whisper STT via mlx-audio -- transcription and translation |
  189. ---
  190. ## Image Generation & Editing
  191. Generate and edit images locally with Flux models via [mflux](https://github.com/filipstrand/mflux).
  192. ```bash
  193. pip install vmlx[image]
  194. # Image generation
  195. vmlx serve schnell # or dev, z-image-turbo
  196. vmlx serve ~/.mlxstudio/models/image/flux1-schnell-4bit
  197. # Image editing
  198. vmlx serve qwen-image-edit # instruction-based editing
  199. ```
  200. ### Generation API
  201. ```bash
  202. curl http://localhost:8000/v1/images/generations \
  203. -H "Content-Type: application/json" \
  204. -d '{
  205. "model": "schnell",
  206. "prompt": "A cat astronaut floating in space with Earth in the background",
  207. "size": "1024x1024",
  208. "n": 1
  209. }'
  210. ```
  211. ```python
  212. # Python (OpenAI SDK)
  213. response = client.images.generate(
  214. model="schnell",
  215. prompt="A cat astronaut floating in space",
  216. size="1024x1024",
  217. n=1,
  218. )
  219. ```
  220. ### Editing API
  221. ```bash
  222. # Edit an image with a text prompt (Flux Kontext / Qwen Image Edit)
  223. curl http://localhost:8000/v1/images/edits \
  224. -H "Content-Type: application/json" \
  225. -d '{
  226. "model": "flux-kontext",
  227. "prompt": "Change the background to a sunset",
  228. "image": "<base64-encoded-image>",
  229. "size": "1024x1024",
  230. "strength": 0.8
  231. }'
  232. ```
  233. ```python
  234. # Python
  235. import base64
  236. with open("source.png", "rb") as f:
  237. image_b64 = base64.b64encode(f.read()).decode()
  238. response = requests.post("http://localhost:8000/v1/images/edits", json={
  239. "model": "flux-kontext",
  240. "prompt": "Make the sky purple",
  241. "image": image_b64,
  242. "size": "1024x1024",
  243. "strength": 0.8,
  244. })
  245. ```
  246. ### Supported Image Models
  247. **Generation Models:**
  248. | Model | Steps | Speed | Memory |
  249. |-------|-------|-------|--------|
  250. | **Flux Schnell** | 4 | Fastest | ~6-24 GB |
  251. | **Z-Image Turbo** | 4 | Fast | ~6-24 GB |
  252. | **Flux Dev** | 20 | Slow | ~6-24 GB |
  253. **Editing Models:**
  254. | Model | Steps | Type | Memory |
  255. |-------|-------|------|--------|
  256. | **Qwen Image Edit** | 28 | Instruction-based editing | ~54 GB |
  257. ---
  258. ## API Reference
  259. ### API Gateway
  260. The desktop app runs an **API Gateway** on a single port (default `8080`) that routes requests to all loaded models by name. Run multiple models simultaneously and access them all through one URL.
  261. ```bash
  262. # All models accessible through the gateway
  263. curl http://localhost:8080/v1/chat/completions \
  264. -d '{"model": "Qwen3.5-122B", "messages": [{"role": "user", "content": "Hi"}]}'
  265. # Works with Ollama CLI too
  266. OLLAMA_HOST=http://localhost:8080 ollama run Qwen3.5-122B
  267. ```
  268. The gateway supports **OpenAI**, **Anthropic**, and **Ollama** wire formats. Configure the port in the API tab.
  269. ### Endpoints
  270. **OpenAI / Anthropic**
  271. | Method | Path | Description |
  272. |--------|------|-------------|
  273. | `POST` | `/v1/chat/completions` | OpenAI Chat Completions API (streaming + non-streaming) |
  274. | `POST` | `/v1/messages` | Anthropic Messages API |
  275. | `POST` | `/v1/responses` | OpenAI Responses API |
  276. | `POST` | `/v1/completions` | Text completions |
  277. | `POST` | `/v1/images/generations` | Image generation |
  278. | `POST` | `/v1/images/edits` | Image editing (Qwen Image Edit) |
  279. | `POST` | `/v1/embeddings` | Text embeddings |
  280. | `POST` | `/v1/rerank` | Document reranking |
  281. | `POST` | `/v1/audio/transcriptions` | Speech-to-text (Whisper) |
  282. | `POST` | `/v1/audio/speech` | Text-to-speech (Kokoro) |
  283. | `GET` | `/v1/models` | List loaded models |
  284. | `GET` | `/v1/cache/stats` | Cache statistics |
  285. | `GET` | `/health` | Server health check |
  286. **Ollama**
  287. | Method | Path | Description |
  288. |--------|------|-------------|
  289. | `POST` | `/api/chat` | Chat completion (NDJSON streaming) |
  290. | `POST` | `/api/generate` | Text generation (NDJSON streaming) |
  291. | `GET` | `/api/tags` | List loaded models |
  292. | `POST` | `/api/show` | Model details |
  293. | `POST` | `/api/embeddings` | Generate embeddings |
  294. Ollama terminal responses preserve available backend cache and phase measurements
  295. under `vmlx_usage`: `prompt_tokens_details.cached_tokens`, `vmlx_prefill`, and
  296. `vmlx_decode`. Phase records retain their own token count, seconds, and scope;
  297. `prompt_eval_count` includes cached tokens and must not be divided by cold-prefill
  298. time. Missing backend timings stay absent rather than being estimated from
  299. request wall time.
  300. ### curl Examples
  301. **Chat completion (streaming)**
  302. ```bash
  303. curl http://localhost:8000/v1/chat/completions \
  304. -H "Content-Type: application/json" \
  305. -d '{
  306. "model": "local",
  307. "messages": [{"role": "user", "content": "Explain quantum computing in 3 sentences."}],
  308. "stream": true,
  309. "temperature": 0.7
  310. }'
  311. ```
  312. **Chat completion with thinking mode**
  313. ```bash
  314. curl http://localhost:8000/v1/chat/completions \
  315. -H "Content-Type: application/json" \
  316. -d '{
  317. "model": "local",
  318. "messages": [{"role": "user", "content": "Solve: what is 23 * 47?"}],
  319. "enable_thinking": true,
  320. "stream": true
  321. }'
  322. ```
  323. **Tool calling**
  324. ```bash
  325. curl http://localhost:8000/v1/chat/completions \
  326. -H "Content-Type: application/json" \
  327. -d '{
  328. "model": "local",
  329. "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
  330. "tools": [{
  331. "type": "function",
  332. "function": {
  333. "name": "get_weather",
  334. "description": "Get current weather for a location",
  335. "parameters": {
  336. "type": "object",
  337. "properties": {
  338. "location": {"type": "string", "description": "City name"}
  339. },
  340. "required": ["location"]
  341. }
  342. }
  343. }]
  344. }'
  345. ```
  346. **Anthropic Messages API**
  347. ```bash
  348. curl http://localhost:8000/v1/messages \
  349. -H "Content-Type: application/json" \
  350. -H "x-api-key: not-needed" \
  351. -H "anthropic-version: 2023-06-01" \
  352. -d '{
  353. "model": "local",
  354. "max_tokens": 1024,
  355. "messages": [{"role": "user", "content": "Hello!"}]
  356. }'
  357. ```
  358. **Embeddings**
  359. ```bash
  360. curl http://localhost:8000/v1/embeddings \
  361. -H "Content-Type: application/json" \
  362. -d '{
  363. "model": "local",
  364. "input": "The quick brown fox jumps over the lazy dog"
  365. }'
  366. ```
  367. **Text-to-speech**
  368. ```bash
  369. curl http://localhost:8000/v1/audio/speech \
  370. -H "Content-Type: application/json" \
  371. -d '{
  372. "model": "kokoro",
  373. "input": "Hello, welcome to vMLX!",
  374. "voice": "af_heart"
  375. }' --output speech.wav
  376. ```
  377. **Speech-to-text**
  378. ```bash
  379. curl http://localhost:8000/v1/audio/transcriptions \
  380. -F file=@audio.wav \
  381. -F model=whisper
  382. ```
  383. **Image generation**
  384. ```bash
  385. curl http://localhost:8000/v1/images/generations \
  386. -H "Content-Type: application/json" \
  387. -d '{
  388. "model": "schnell",
  389. "prompt": "A mountain landscape at sunset",
  390. "size": "1024x1024"
  391. }'
  392. ```
  393. **Reranking**
  394. ```bash
  395. curl http://localhost:8000/v1/rerank \
  396. -H "Content-Type: application/json" \
  397. -d '{
  398. "model": "local",
  399. "query": "What is machine learning?",
  400. "documents": [
  401. "ML is a subset of AI",
  402. "The weather is sunny today",
  403. "Neural networks learn from data"
  404. ]
  405. }'
  406. ```
  407. **Cache stats**
  408. ```bash
  409. curl http://localhost:8000/v1/cache/stats
  410. ```
  411. **Health check**
  412. ```bash
  413. curl http://localhost:8000/health
  414. ```
  415. ---
  416. ## Desktop App
  417. vMLX includes a native macOS desktop app (MLX Studio) with 5 modes:
  418. | Mode | Description |
  419. |------|-------------|
  420. | **Chat** | Conversation interface with chat history, thinking mode, tool calling, agentic coding |
  421. | **Server** | Manage model sessions -- start, stop, configure, monitor |
  422. | **Image** | Text-to-image generation and image editing with Flux, Kontext, Qwen, and Fill models |
  423. | **Tools** | Model converter, GGUF-to-MLX, inspector, diagnostics |
  424. | **API** | Live endpoint reference with copy-pasteable code snippets |
  425. <table align="center">
  426. <tr>
  427. <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/image-edit-tab.png" width="450" alt="Image generation and editing" /></td>
  428. <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/gguf-to-mlx.png" width="450" alt="Developer tools" /></td>
  429. </tr>
  430. <tr>
  431. <td align="center"><em>Image generation and editing with Flux models</em></td>
  432. <td align="center"><em>Developer tools -- model conversion and diagnostics</em></td>
  433. </tr>
  434. <tr>
  435. <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/anthropic-api.png" width="450" alt="Anthropic API endpoint" /></td>
  436. <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/gguf-to-mlx.png" width="450" alt="GGUF to MLX conversion" /></td>
  437. </tr>
  438. <tr>
  439. <td align="center"><em>Anthropic Messages API endpoint -- full compatibility</em></td>
  440. <td align="center"><em>GGUF to MLX conversion -- bring your own models</em></td>
  441. </tr>
  442. </table>
  443. ### Download
  444. Get the latest DMG from [MLX Studio Releases](https://github.com/jjang-ai/mlxstudio/releases/latest), or build from source:
  445. ```bash
  446. git clone https://github.com/jjang-ai/vmlx.git
  447. cd vmlx/panel
  448. npm install && npm run build
  449. npx electron-builder --mac dmg
  450. ```
  451. ### Menu Bar
  452. vMLX lives in your menu bar showing all running models, GPU memory usage, and quick controls.
  453. <p align="center">
  454. <img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/menu-bar.png" width="300" alt="Menu Bar" />
  455. </p>
  456. ---
  457. ## Advanced Quantization
  458. vMLX supports standard MLX quantization (4-bit, 8-bit uniform) out of the box. For users who want to push further, **JANG adaptive mixed-precision** assigns different bit widths to different layer types -- attention gets more bits, MLP layers get fewer -- achieving better quality at the same model size.
  459. ### JANG Profiles
  460. | Profile | Attention | Embeddings | MLP | Avg Bits | Use Case |
  461. |---------|-----------|------------|-----|----------|----------|
  462. | `JANG_2M` | 8-bit | 4-bit | 2-bit | ~2.5 | Balanced compression |
  463. | `JANG_2L` | 8-bit | 6-bit | 2-bit | ~2.7 | Quality 2-bit |
  464. | `JANG_3M` | 8-bit | 3-bit | 3-bit | ~3.2 | **Recommended** |
  465. | `JANG_4M` | 8-bit | 4-bit | 4-bit | ~4.2 | Standard quality |
  466. | `JANG_6M` | 8-bit | 6-bit | 6-bit | ~6.2 | Near lossless |
  467. ### Convert
  468. ```bash
  469. pip install vmlx[jang]
  470. # Standard MLX quantization
  471. vmlx convert my-model --bits 4
  472. # JANG adaptive quantization
  473. vmlx convert my-model --jang-profile JANG_3M
  474. # Activation-aware calibration (better at 2-3 bit)
  475. vmlx convert my-model --jang-profile JANG_2L --calibration-method activations
  476. # Serve the converted model
  477. vmlx serve ./my-model-JANG_3M --continuous-batching --use-paged-cache
  478. ```
  479. Pre-quantized JANG models are available at [JANGQ-AI on HuggingFace](https://huggingface.co/JANGQ-AI).
  480. ### Smelt Mode (Partial Expert Loading)
  481. For MoE models that don't fit in RAM, **Smelt** loads only a subset of experts per layer from SSD and keeps the backbone resident. Routing is biased toward the resident experts, so response quality stays coherent while RAM usage drops. Trade-off: throughput scales inversely with expert % loaded, because expert swaps hit SSD on the hot path.
  482. ```bash
  483. # Load 50% of experts per layer (default when --smelt alone)
  484. vmlx serve ./MyMoE-JANG_4M --smelt --smelt-experts 50
  485. # Aggressive: load 25% — smallest RAM, slowest
  486. vmlx serve ./MyMoE-JANG_4M --smelt --smelt-experts 25
  487. ```
  488. **Benchmarks on `Nemotron-Cascade-2-30B-A3B-JANG_4M`** (23 MoE layers × 128 experts, Apple M3 Ultra / 128 GB, dedicated machine, no parallel models):
  489. | `--smelt-experts` | Active RAM | Decode tok/s | RAM saving | Coherent |
  490. |---|---:|---:|---|---|
  491. | _off (baseline)_ | **17,408 MB** | **89.9** | — | ✓ |
  492. | `50` | 9,529 MB | **66.5** | **−45%** | ✓ |
  493. | `25` | 5,590 MB | * | **−68%** | ✓ |
  494. \* Responses too short (2-5 tokens) for reliable steady-state tok/s measurement at 25 %. Subjectively responsive.
  495. All three configurations produced coherent, non-looping output. No quality degradation observed — routing bias keeps the model on-topic.
  496. > **Credit**: Smelt mode is inspired by [Anemll's **flash-moe**](https://github.com/Anemll/flash-moe) — a pure C / Objective‑C / Metal inference engine that showed huge MoE models (Qwen3.5-397B) can run on modest Apple Silicon hardware by streaming expert weights from SSD with `pread()` on demand. vMLX Smelt takes a different implementation path: Python/MLX, tied to the JANG quantization format, and loading a fixed subset of experts per layer at startup (backbone resident, routing biased toward the loaded subset) rather than on-demand per-token. It plugs into the full vMLX server with continuous batching, paged cache, and OpenAI-compatible API. Different engine, same core insight — thanks to the flash-moe team for validating the approach.
  497. **Smelt is mutually exclusive with VLM mode.** vMLX detects smelt and automatically disables `--is-mllm` (with a warning) because the vision tower is not wired through the partial-expert loader — image input on a smelt-loaded VLM would produce garbage logits. Use a text-only model when running smelt, or disable smelt when running a VLM.
  498. Smelt requires an MoE model in JANG format. Not compatible with dense models (no experts to partial-load) or with non-JANG formats.
  499. ---
  500. ## CLI Commands
  501. ```bash
  502. vmlx serve <model> # Start inference server
  503. vmlx convert <model> --bits 4 # MLX uniform quantization
  504. vmlx convert <model> -j JANG_3M # JANG adaptive quantization
  505. vmlx info <model> # Model metadata and config
  506. vmlx doctor <model> # Run diagnostics
  507. vmlx bench <model> # Performance benchmarks
  508. vmlx-worker --secret <secret> # Start distributed worker node
  509. ```
  510. ---
  511. ## Configuration
  512. ### Server Options
  513. ```bash
  514. vmlx serve <model> \
  515. --host 0.0.0.0 \ # Bind address (default: 0.0.0.0)
  516. --port 8000 \ # Port (default: 8000)
  517. --api-key sk-your-key \ # Optional API key authentication
  518. --continuous-batching \ # Enable concurrent request handling
  519. --enable-prefix-cache \ # Reuse KV states for repeated prompts
  520. --use-paged-cache \ # Block-based KV cache with dedup
  521. --kv-cache-quantization q8 \ # Quantize cache: q4 or q8
  522. --enable-disk-cache \ # Persist cache to SSD
  523. --enable-jit \ # JIT Metal kernel compilation
  524. --tool-call-parser auto \ # Auto-detect tool call format
  525. --reasoning-parser auto \ # Auto-detect thinking format
  526. --log-level INFO \ # Logging: DEBUG, INFO, WARNING, ERROR
  527. --max-model-len 8192 \ # Max context length
  528. --speculative-model <model> \ # Draft model for speculative decoding
  529. --enable-pld \ # Prompt Lookup Decoding — no draft model, best for code/JSON/schemas
  530. --distributed \ # Enable multi-Mac pipeline parallelism
  531. --cluster-secret <secret> \ # Shared auth secret for workers
  532. --distributed-mode pipeline \ # pipeline (default) or tensor (coming soon)
  533. --worker-nodes ip:port,... \ # Manual worker IPs (overrides auto-discovery)
  534. --cors-origins "*" # CORS allowed origins
  535. ```
  536. ### Quantization Options
  537. ```bash
  538. vmlx convert <model> \
  539. --bits 4 \ # Uniform quantization bits: 2, 3, 4, 6, 8
  540. --group-size 64 \ # Quantization group size (default: 64)
  541. --output ./output-dir \ # Output directory
  542. --jang-profile JANG_3M \ # JANG mixed-precision profile
  543. --calibration-method activations # Activation-aware calibration
  544. ```
  545. ### Image Generation & Editing Options
  546. ```bash
  547. pip install vmlx[image]
  548. # Generation models
  549. vmlx serve schnell \ # or dev, z-image-turbo
  550. --image-quantize 4 \ # Quantization: 4, 8 (omit for full precision)
  551. --port 8001
  552. # Editing models
  553. vmlx serve qwen-image-edit \ # Instruction-based editing (full precision only)
  554. --port 8001
  555. # Local model directory
  556. vmlx serve ~/.mlxstudio/models/image/FLUX.1-schnell-mflux-4bit
  557. ```
  558. ### Audio Options
  559. TTS and STT require the `mlx-audio` package:
  560. ```bash
  561. pip install mlx-audio
  562. # TTS: serve Kokoro model
  563. vmlx serve kokoro --port 8002
  564. # STT: serve Whisper model
  565. vmlx serve whisper --port 8003
  566. ```
  567. ### Optional Dependencies
  568. ```bash
  569. pip install vmlx # Core: text LLMs, VLMs, embeddings, reranking
  570. pip install vmlx[image] # + Image generation (mflux)
  571. pip install vmlx[jang] # + JANG quantization tools
  572. pip install vmlx[dev] # + Development/testing tools
  573. pip install vmlx[image,jang] # Multiple extras
  574. ```
  575. ---
  576. ## Architecture
  577. ```
  578. +--------------------------------------------+
  579. | Desktop App (Electron) |
  580. | Chat | Server | Image | Tools | API |
  581. +--------------------------------------------+
  582. | Session Manager (TypeScript) |
  583. | Process spawn | Health monitor | Tray |
  584. +--------------------------------------------+
  585. | vMLX Engine (Python / FastAPI) |
  586. | +--------+ +---------+ +-----------+ |
  587. | |Simple | | Batched | | ImageGen | |
  588. | |Engine | | Engine | | Engine | |
  589. | +---+----+ +----+----+ +-----+-----+ |
  590. | | | | |
  591. | +---+------------+--+ +-----+-----+ |
  592. | | mlx-lm / mlx-vlm | | mflux | |
  593. | +--------+-----------+ +-----------+ |
  594. | | |
  595. | +--------+----------------------------+ |
  596. | | MLX Metal GPU Backend | |
  597. | | quantized_matmul | KV cache | SDPA | |
  598. | +--------------------------------------+ |
  599. +--------------------------------------------+
  600. | L1: Prefix Cache (Memory-Aware / Paged) |
  601. | L2: Disk Cache (Persistent / Block Store) |
  602. | KV Quant: q4/q8 at storage boundary |
  603. +--------------------------------------------+
  604. ```
  605. ---
  606. ## Contributing
  607. Contributions are welcome. Here is how to set up a development environment:
  608. ```bash
  609. git clone https://github.com/jjang-ai/vmlx.git
  610. cd vmlx
  611. # Python engine
  612. python -m venv .venv && source .venv/bin/activate
  613. pip install -e ".[dev,jang,image]"
  614. pytest tests/ -k "not Async" # 2000+ tests
  615. # Electron desktop app
  616. cd panel && npm install
  617. npm run dev # Development mode with hot reload
  618. npx vitest run # 1545+ tests
  619. ```
  620. ### Project Structure
  621. ```
  622. vmlx/
  623. vmlx_engine/ # Python inference engine (FastAPI server)
  624. panel/ # Electron desktop app (React + TypeScript)
  625. src/main/ # Electron main process
  626. src/renderer/ # React frontend
  627. src/preload/ # IPC bridge
  628. tests/ # Python test suite
  629. assets/ # Screenshots and logos
  630. ```
  631. ### Guidelines
  632. - Run the full test suite before submitting PRs
  633. - Follow existing code style and patterns
  634. - Include tests for new features
  635. - Update documentation for user-facing changes
  636. ---
  637. ## License
  638. Apache License 2.0 -- see [LICENSE](LICENSE).
  639. ---
  640. <p align="center">
  641. Built by <a href="https://github.com/jjang-ai">Jinho Jang</a> (eric@jangq.ai)<br>
  642. <a href="https://jangq.ai">JANGQ AI</a> &bull; <a href="https://pypi.org/project/vmlx/">PyPI</a> &bull; <a href="https://github.com/jjang-ai/vmlx">GitHub</a> &bull; <a href="https://github.com/jjang-ai/mlxstudio/releases">Downloads</a>
  643. </p>
  644. ---
  645. ## 한국어 (Korean)
  646. ### vMLX — Apple Silicon을 위한 로컬 AI 엔진
  647. Mac에서 LLM, VLM, 이미지 생성 및 편집 모델을 완전히 로컬로 실행하세요.
  648. OpenAI + Anthropic 호환 API. 클라우드 없음. API 키 불필요. 데이터가 기기를 떠나지 않습니다.
  649. ### 빠른 시작
  650. ```bash
  651. pip install vmlx
  652. vmlx serve mlx-community/Llama-3.2-3B-Instruct-4bit
  653. ```
  654. ### 주요 기능
  655. | 기능 | 설명 |
  656. |------|------|
  657. | **텍스트 생성** | MLX 및 JANG 형식의 LLM 추론 |
  658. | **비전-언어 모델** | 이미지 + 텍스트 멀티모달 추론 |
  659. | **이미지 생성** | Flux Schnell/Dev, Z-Image Turbo (mflux 기반) |
  660. | **이미지 편집** | Qwen Image Edit (텍스트 지시 기반 이미지 편집) |
  661. | **5단계 캐싱** | 프리픽스, 페이지드, KV 양자화, 디스크, 메모리 인식 캐시 |
  662. | **연속 배칭** | 다중 동시 요청 처리 |
  663. | **에이전트 도구** | 30개 내장 도구 (파일, 웹 검색, Git, 터미널) |
  664. | **OpenAI API** | /v1/chat/completions, /v1/images/generations, /v1/images/edits |
  665. | **Anthropic API** | /v1/messages (스트리밍, 도구 호출, 시스템 프롬프트) |
  666. ### 이미지 생성
  667. ```bash
  668. pip install vmlx[image]
  669. vmlx serve schnell # 빠른 생성 (4 단계)
  670. vmlx serve dev # 고품질 생성 (20 단계)
  671. ```
  672. ### 이미지 편집
  673. ```bash
  674. vmlx serve qwen-image-edit # 텍스트 지시 기반 이미지 편집
  675. ```
  676. ```bash
  677. # 이미지 편집 API
  678. curl http://localhost:8000/v1/images/edits \
  679. -H "Content-Type: application/json" \
  680. -d '{
  681. "model": "qwen-image-edit",
  682. "prompt": "배경을 해질녘으로 변경",
  683. "image": "<base64 인코딩된 이미지>",
  684. "size": "1024x1024",
  685. "strength": 0.8
  686. }'
  687. ```
  688. ### 데스크톱 앱 (MLX Studio)
  689. macOS 네이티브 데스크톱 앱으로 5가지 모드를 제공합니다:
  690. | 모드 | 설명 |
  691. |------|------|
  692. | **채팅** | 대화 인터페이스, 채팅 기록, 도구 호출, 에이전트 코딩 |
  693. | **서버** | 모델 세션 관리 — 시작, 정지, 설정, 모니터링 |
  694. | **이미지** | 텍스트-이미지 생성 및 이미지 편집 (Flux, Qwen 모델) |
  695. | **도구** | 모델 변환기, GGUF-MLX 변환, 진단 |
  696. | **API** | 실시간 엔드포인트 참조 및 코드 스니펫 |
  697. <p align="center">
  698. <img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/image-edit-tab.png" width="450" alt="이미지 생성 및 편집" />
  699. </p>
  700. ### 설치
  701. ```bash
  702. pip install vmlx # 기본: 텍스트 LLM, VLM, 임베딩
  703. pip install vmlx[image] # + 이미지 생성/편집 (mflux)
  704. pip install vmlx[jang] # + JANG 양자화 도구
  705. pip install vmlx[audio] # + TTS/STT (mlx-audio)
  706. ```
  707. ### 라이선스
  708. Apache License 2.0 — [LICENSE](LICENSE) 참조.
  709. ---
  710. <p align="center">
  711. 개발자: <a href="https://github.com/jjang-ai">장진호</a> (eric@jangq.ai)<br>
  712. <a href="https://jangq.ai">JANGQ AI</a> &bull;
  713. <a href="https://ko-fi.com/jangml">Ko-fi로 후원하기</a>
  714. </p>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.