vmlx
vMLX - Use MLX models easily - JANGQ (GGUF for MLX) - Not dependant on mlx_vlm
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/jjang-ai--vmlx)HTML
<a href="/listings/jjang-ai--vmlx"><img src="/api/listings/jjang-ai--vmlx/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Install from PyPI
Published on [PyPI as `vmlx`](https://pypi.org/project/vmlx/) -- install and run in one command:
# Recommended: uv (fast, no venv hassle) brew install uv uv tool install vmlx vmlx serve mlx-community/Qwen3-8B-4bit # Or: pipx (isolates from system Python) brew install pipx pipx install vmlx vmlx serve mlx-community/Qwen3-8B-4bit # Or: pip in a virtual environment python3 -m venv ~/.vmlx-env && source ~/.vmlx-env/bin/activate pip install vmlx vmlx serve mlx-community/Qwen3-8B-4bit
> **Note:** On macOS 14+, bare `pip install` fails with "externally-managed-environment". Use `uv`, `pipx`, or a venv.
The vMLX inference server is now running at `http://0.0.0.0:8000` with an OpenAI + Anthropic compatible API. Works with any model from [mlx-community](https://huggingface.co/mlx-community) -- thousands of models ready to go.
Or download the desktop app
Get [MLX Studio](https://github.com/jjang-ai/mlxstudio/releases/latest) -- a native macOS app with chat UI, model management, image generation, and developer tools. No terminal required. Just download the DMG and drag to Applications.
Use with OpenAI SDK
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
response = client.chat.completions.create(
model="local",
messages=[{"role": "user", "content": "Hello!"}],
stream=True,
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="", flush=True)Use with Anthropic SDK
import anthropic
client = anthropic.Anthropic(base_url="http://localhost:8000/v1", api_key="not-needed")
message = client.messages.create(
model="local",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello!"}],
)
print(message.content[0].text)Use with curl
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "local",
"messages": [{"role": "user", "content": "Hello!"}],
"stream": true
}'---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 914 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <p align="center">
- <picture>
- <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-dark.png">
- <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-light.png">
- <img alt="vMLX" src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-light.png" width="400">
- </picture>
- </p>
- <h3 align="center">MLX Inference Server for Apple Silicon</h3>
- <p align="center">
- Self-hosted inference server for LLMs, VLMs, and image generation on Apple Silicon.<br>
- OpenAI + Anthropic + Ollama compatible HTTP API. Self-hosted; no third-party API keys required.<br>
- Native MTP artifact detection and family-specific cache policy gates keep speculative/cache settings explicit and model-safe.
- </p>
- <p align="center">
- <em>Looking for a native Swift macOS app or Swift inference engine? See <a href="https://osaurus.ai">osaurus.ai</a>.</em>
- </p>
- <p align="center">
- <a href="https://pypi.org/project/vmlx/"><img src="https://img.shields.io/pypi/v/vmlx?color=%234B8BBE&label=PyPI&logo=python&logoColor=white" alt="PyPI" /></a>
- <a href="https://github.com/jjang-ai/vmlx/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-green?logo=apache" alt="License" /></a>
- <a href="https://github.com/jjang-ai/vmlx"><img src="https://img.shields.io/github/stars/jjang-ai/vmlx?style=social" alt="Stars" /></a>
- <img src="https://img.shields.io/badge/Apple_Silicon-M1%2FM2%2FM3%2FM4-black?logo=apple" alt="Apple Silicon" />
- <img src="https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white" alt="Python" />
- <img src="https://img.shields.io/badge/Electron-28-47848F?logo=electron&logoColor=white" alt="Electron" />
- <a href="https://ko-fi.com/jangml"><img src="https://img.shields.io/badge/Support-Ko--fi-FF5E5B?logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
- </p>
- <p align="center">
- <a href="#quickstart">Quickstart</a> •
- <a href="#model-support">Models</a> •
- <a href="#features">Features</a> •
- <a href="#image-generation--editing">Image Gen</a> •
- <a href="#api-reference">API</a> •
- <a href="#desktop-app">Desktop App</a> •
- <a href="#advanced-quantization">JANG</a> •
- <a href="#cli-commands">CLI</a> •
- <a href="#configuration">Config</a> •
- <a href="#contributing">Contributing</a> •
- <a href="#한국어-korean">한국어</a>
- </p>
- ---
- > **JANG 2-bit destroys MLX 4-bit on [MiniMax M2.5](https://huggingface.co/JANGQ-AI/MiniMax-M2.5-JANG_2L):**
- >
- > | Quantization | MMLU (200q) | Size |
- > |---|---|---|
- > | **JANG\_2L (2-bit)** | **74%** | **89 GB** |
- > | MLX 4-bit | 26.5% | 120 GB |
- > | MLX 3-bit | 24.5% | 93 GB |
- > | MLX 2-bit | 25% | 68 GB |
- >
- > Adaptive mixed-precision keeps critical layers at higher precision. Scores at [jangq.ai](https://jangq.ai). Models at [JANGQ-AI](https://huggingface.co/JANGQ-AI).
- <table align="center">
- <tr>
- <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/chat-tab.png" width="500" alt="Chat interface" /></td>
- <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/agentic-chat.png" width="500" alt="Agentic coding chat" /></td>
- </tr>
- <tr>
- <td align="center"><em>Chat with any MLX model -- thinking mode, streaming, and syntax highlighting</em></td>
- <td align="center"><em>Agentic chat with full coding capabilities -- tool use and structured output</em></td>
- </tr>
- </table>
- ---
- ## Quickstart
- ### Install from PyPI
- Published on [PyPI as `vmlx`](https://pypi.org/project/vmlx/) -- install and run in one command:
- ```bash
- # Recommended: uv (fast, no venv hassle)
- brew install uv
- uv tool install vmlx
- vmlx serve mlx-community/Qwen3-8B-4bit
- # Or: pipx (isolates from system Python)
- brew install pipx
- pipx install vmlx
- vmlx serve mlx-community/Qwen3-8B-4bit
- # Or: pip in a virtual environment
- python3 -m venv ~/.vmlx-env && source ~/.vmlx-env/bin/activate
- pip install vmlx
- vmlx serve mlx-community/Qwen3-8B-4bit
- ```
- > **Note:** On macOS 14+, bare `pip install` fails with "externally-managed-environment". Use `uv`, `pipx`, or a venv.
- The vMLX inference server is now running at `http://0.0.0.0:8000` with an OpenAI + Anthropic compatible API. Works with any model from [mlx-community](https://huggingface.co/mlx-community) -- thousands of models ready to go.
- ### Or download the desktop app
- Get [MLX Studio](https://github.com/jjang-ai/mlxstudio/releases/latest) -- a native macOS app with chat UI, model management, image generation, and developer tools. No terminal required. Just download the DMG and drag to Applications.
- ### Use with OpenAI SDK
- ```python
- from openai import OpenAI
- client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
- response = client.chat.completions.create(
- model="local",
- messages=[{"role": "user", "content": "Hello!"}],
- stream=True,
- )
- for chunk in response:
- print(chunk.choices[0].delta.content or "", end="", flush=True)
- ```
- ### Use with Anthropic SDK
- ```python
- import anthropic
- client = anthropic.Anthropic(base_url="http://localhost:8000/v1", api_key="not-needed")
- message = client.messages.create(
- model="local",
- max_tokens=1024,
- messages=[{"role": "user", "content": "Hello!"}],
- )
- print(message.content[0].text)
- ```
- ### Use with curl
- ```bash
- curl http://localhost:8000/v1/chat/completions \
- -H "Content-Type: application/json" \
- -d '{
- "model": "local",
- "messages": [{"role": "user", "content": "Hello!"}],
- "stream": true
- }'
- ```
- ---
- ## Model Support
- vMLX runs any MLX model. Point it at a HuggingFace repo or local path and go.
- | Type | Models |
- |------|--------|
- | **Text LLMs** | Qwen 2/2.5/3/3.5/3.6, Llama 3/3.1/3.2/3.3/4, Mistral/Mixtral, **Mistral-Medium-3.5** (ministral3), Mistral-Small-4, Gemma 3/4, Phi-4, DeepSeek V2/V3/V4, GLM-4/5, MiniMax M2.5/M2.7, Nemotron, **Laguna** (poolside), **ZAYA** (CCA + MoE), Kimi K2.5/K2.6, StepFun, and any mlx-lm model |
- | **Vision LLMs** | Qwen-VL, Qwen3.5-VL / Qwen3.6-VL, Pixtral, InternVL, LLaVA, Gemma 3n / 4-VL, Mistral-Medium-3.5 (PIXTRAL) |
- | **Multimodal Omni** | **Nemotron-3-Nano-Omni** (text + image + audio + video) — Parakeet audio encoder + RADIO ViT vision tower; routed via OmniMultimodalDispatcher across `/v1/chat/completions`, `/v1/messages`, `/v1/responses`, `/api/chat` |
- | **MoE Models** | Qwen 3.5/3.6 MoE (A3B/A10B), Mixtral, DeepSeek V2/V3/V4, MiniMax M2.5/M2.7, Llama 4, Laguna (256 routed experts top-8) |
- | **Hybrid SSM** | Nemotron-H, Jamba, GatedDeltaNet (Mamba + Attention), Qwen3.5-A3B hybrid, Granite MoE Hybrid, LFM2 |
- | **Image Gen** | Flux Schnell/Dev, Z-Image Turbo (via mflux) |
- | **Image Edit** | Qwen Image Edit (via mflux) |
- | **Embeddings** | Any mlx-lm compatible embedding model |
- | **Reranking** | Cross-encoder reranking models |
- | **Audio** | Kokoro TTS, Whisper STT (via mlx-audio) |
- ---
- ## Features
- ### Inference Engine
- | Feature | Description |
- |---------|-------------|
- | **Continuous Batching** | Handle multiple concurrent requests efficiently |
- | **Prefix Cache** | Reuse KV states for repeated prompts -- makes follow-up messages instant |
- | **Paged KV Cache** | Block-based caching with content-addressable deduplication |
- | **KV Cache Quantization** | Compress cached states to q4/q8 for 2-4x memory savings |
- | **Disk Cache (L2)** | Persist prompt caches to SSD -- survives server restarts |
- | **Block Disk Cache** | Per-block persistent cache paired with paged KV cache |
- | **Speculative Decoding** | Small draft model proposes tokens for 20-90% speedup |
- | **Prompt Lookup Decoding** | No draft model needed — reuses n-gram matches from the prompt/context. Best for structured or repetitive output (code, JSON, schemas). Enable with `--enable-pld`. |
- | **JIT Compilation** | `mx.compile` Metal kernel fusion (experimental) |
- | **Hybrid SSM Support** | Mamba/GatedDeltaNet layers handled correctly alongside attention |
- | **Distributed Compute** | Pipeline parallelism across multiple Macs via Thunderbolt 5 / Ethernet / WiFi |
- ### Distributed Inference (Multi-Mac)
- Run models too large for a single Mac across 2+ machines. Each Mac loads a subset of transformer layers and they communicate hidden states over the network.
- ```bash
- # On worker Macs:
- pip install vmlx
- vmlx-worker --secret mysecret
- # On coordinator Mac (runs the server):
- vmlx serve JANGQ-AI/Qwen3.5-Coder-Rerank-397B-A27B-JANG_2L --distributed --cluster-secret mysecret
- ```
- | Feature | Description |
- |---------|-------------|
- | **Pipeline Parallelism** | Split layers across nodes -- hidden state (~8KB/step) flows sequentially |
- | **Auto-Discovery** | Bonjour mDNS, UDP broadcast, HTTP probes, Tailscale, cached peers, manual IP |
- | **Capability-Scored Election** | Most powerful Mac becomes coordinator automatically |
- | **Any Network Works** | TB5 (120 Gbps), 10GbE, 1GbE, WiFi, Tailscale -- PP is not bandwidth-bound |
- | **JANG Support** | Each worker loads its layer range from JANG safetensors (mmap) |
- | **Live Node List** | Desktop app shows discovered nodes, link type, latency, layer assignments |
- | **Cluster API** | `/v1/cluster/status`, `/v1/cluster/nodes`, `/v1/cluster/scan` REST endpoints |
- ### 5-Layer Cache Architecture
- ```
- Request -> Tokens
- |
- L1: Memory-Aware Prefix Cache (or Paged Cache)
- | miss
- L2: Disk Cache (or Block Disk Store)
- | miss
- Inference -> float16 KV states
- |
- KV Quantization -> q4/q8 for storage
- |
- Store back into L1 + L2
- ```
- ### Tool Calling
- Auto-detected parsers for every major model family:
- `qwen` - `llama` - `mistral` - `hermes` - `deepseek` - `glm47` - `minimax` - `nemotron` - `granite` - `functionary` - `xlam` - `kimi` - `step3p5`
- ### Reasoning / Thinking Mode
- Auto-detected reasoning parsers that extract `<think>` blocks:
- `qwen3` (Qwen3, QwQ, StepFun) - `minimax_m2` (MiniMax M2/M2.5/M2.7) - `deepseek_r1` (DeepSeek R1, Gemma 3, GLM, Phi-4) - `openai_gptoss` (GLM Flash, GPT-OSS)
- ### Audio
- | Feature | Description |
- |---------|-------------|
- | **Text-to-Speech** | Kokoro TTS via mlx-audio -- multiple voices, streaming output |
- | **Speech-to-Text** | Whisper STT via mlx-audio -- transcription and translation |
- ---
- ## Image Generation & Editing
- Generate and edit images locally with Flux models via [mflux](https://github.com/filipstrand/mflux).
- ```bash
- pip install vmlx[image]
- # Image generation
- vmlx serve schnell # or dev, z-image-turbo
- vmlx serve ~/.mlxstudio/models/image/flux1-schnell-4bit
- # Image editing
- vmlx serve qwen-image-edit # instruction-based editing
- ```
- ### Generation API
- ```bash
- curl http://localhost:8000/v1/images/generations \
- -H "Content-Type: application/json" \
- -d '{
- "model": "schnell",
- "prompt": "A cat astronaut floating in space with Earth in the background",
- "size": "1024x1024",
- "n": 1
- }'
- ```
- ```python
- # Python (OpenAI SDK)
- response = client.images.generate(
- model="schnell",
- prompt="A cat astronaut floating in space",
- size="1024x1024",
- n=1,
- )
- ```
- ### Editing API
- ```bash
- # Edit an image with a text prompt (Flux Kontext / Qwen Image Edit)
- curl http://localhost:8000/v1/images/edits \
- -H "Content-Type: application/json" \
- -d '{
- "model": "flux-kontext",
- "prompt": "Change the background to a sunset",
- "image": "<base64-encoded-image>",
- "size": "1024x1024",
- "strength": 0.8
- }'
- ```
- ```python
- # Python
- import base64
- with open("source.png", "rb") as f:
- image_b64 = base64.b64encode(f.read()).decode()
- response = requests.post("http://localhost:8000/v1/images/edits", json={
- "model": "flux-kontext",
- "prompt": "Make the sky purple",
- "image": image_b64,
- "size": "1024x1024",
- "strength": 0.8,
- })
- ```
- ### Supported Image Models
- **Generation Models:**
- | Model | Steps | Speed | Memory |
- |-------|-------|-------|--------|
- | **Flux Schnell** | 4 | Fastest | ~6-24 GB |
- | **Z-Image Turbo** | 4 | Fast | ~6-24 GB |
- | **Flux Dev** | 20 | Slow | ~6-24 GB |
- **Editing Models:**
- | Model | Steps | Type | Memory |
- |-------|-------|------|--------|
- | **Qwen Image Edit** | 28 | Instruction-based editing | ~54 GB |
- ---
- ## API Reference
- ### API Gateway
- The desktop app runs an **API Gateway** on a single port (default `8080`) that routes requests to all loaded models by name. Run multiple models simultaneously and access them all through one URL.
- ```bash
- # All models accessible through the gateway
- curl http://localhost:8080/v1/chat/completions \
- -d '{"model": "Qwen3.5-122B", "messages": [{"role": "user", "content": "Hi"}]}'
- # Works with Ollama CLI too
- OLLAMA_HOST=http://localhost:8080 ollama run Qwen3.5-122B
- ```
- The gateway supports **OpenAI**, **Anthropic**, and **Ollama** wire formats. Configure the port in the API tab.
- ### Endpoints
- **OpenAI / Anthropic**
- | Method | Path | Description |
- |--------|------|-------------|
- | `POST` | `/v1/chat/completions` | OpenAI Chat Completions API (streaming + non-streaming) |
- | `POST` | `/v1/messages` | Anthropic Messages API |
- | `POST` | `/v1/responses` | OpenAI Responses API |
- | `POST` | `/v1/completions` | Text completions |
- | `POST` | `/v1/images/generations` | Image generation |
- | `POST` | `/v1/images/edits` | Image editing (Qwen Image Edit) |
- | `POST` | `/v1/embeddings` | Text embeddings |
- | `POST` | `/v1/rerank` | Document reranking |
- | `POST` | `/v1/audio/transcriptions` | Speech-to-text (Whisper) |
- | `POST` | `/v1/audio/speech` | Text-to-speech (Kokoro) |
- | `GET` | `/v1/models` | List loaded models |
- | `GET` | `/v1/cache/stats` | Cache statistics |
- | `GET` | `/health` | Server health check |
- **Ollama**
- | Method | Path | Description |
- |--------|------|-------------|
- | `POST` | `/api/chat` | Chat completion (NDJSON streaming) |
- | `POST` | `/api/generate` | Text generation (NDJSON streaming) |
- | `GET` | `/api/tags` | List loaded models |
- | `POST` | `/api/show` | Model details |
- | `POST` | `/api/embeddings` | Generate embeddings |
- Ollama terminal responses preserve available backend cache and phase measurements
- under `vmlx_usage`: `prompt_tokens_details.cached_tokens`, `vmlx_prefill`, and
- `vmlx_decode`. Phase records retain their own token count, seconds, and scope;
- `prompt_eval_count` includes cached tokens and must not be divided by cold-prefill
- time. Missing backend timings stay absent rather than being estimated from
- request wall time.
- ### curl Examples
- **Chat completion (streaming)**
- ```bash
- curl http://localhost:8000/v1/chat/completions \
- -H "Content-Type: application/json" \
- -d '{
- "model": "local",
- "messages": [{"role": "user", "content": "Explain quantum computing in 3 sentences."}],
- "stream": true,
- "temperature": 0.7
- }'
- ```
- **Chat completion with thinking mode**
- ```bash
- curl http://localhost:8000/v1/chat/completions \
- -H "Content-Type: application/json" \
- -d '{
- "model": "local",
- "messages": [{"role": "user", "content": "Solve: what is 23 * 47?"}],
- "enable_thinking": true,
- "stream": true
- }'
- ```
- **Tool calling**
- ```bash
- curl http://localhost:8000/v1/chat/completions \
- -H "Content-Type: application/json" \
- -d '{
- "model": "local",
- "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
- "tools": [{
- "type": "function",
- "function": {
- "name": "get_weather",
- "description": "Get current weather for a location",
- "parameters": {
- "type": "object",
- "properties": {
- "location": {"type": "string", "description": "City name"}
- },
- "required": ["location"]
- }
- }
- }]
- }'
- ```
- **Anthropic Messages API**
- ```bash
- curl http://localhost:8000/v1/messages \
- -H "Content-Type: application/json" \
- -H "x-api-key: not-needed" \
- -H "anthropic-version: 2023-06-01" \
- -d '{
- "model": "local",
- "max_tokens": 1024,
- "messages": [{"role": "user", "content": "Hello!"}]
- }'
- ```
- **Embeddings**
- ```bash
- curl http://localhost:8000/v1/embeddings \
- -H "Content-Type: application/json" \
- -d '{
- "model": "local",
- "input": "The quick brown fox jumps over the lazy dog"
- }'
- ```
- **Text-to-speech**
- ```bash
- curl http://localhost:8000/v1/audio/speech \
- -H "Content-Type: application/json" \
- -d '{
- "model": "kokoro",
- "input": "Hello, welcome to vMLX!",
- "voice": "af_heart"
- }' --output speech.wav
- ```
- **Speech-to-text**
- ```bash
- curl http://localhost:8000/v1/audio/transcriptions \
- -F file=@audio.wav \
- -F model=whisper
- ```
- **Image generation**
- ```bash
- curl http://localhost:8000/v1/images/generations \
- -H "Content-Type: application/json" \
- -d '{
- "model": "schnell",
- "prompt": "A mountain landscape at sunset",
- "size": "1024x1024"
- }'
- ```
- **Reranking**
- ```bash
- curl http://localhost:8000/v1/rerank \
- -H "Content-Type: application/json" \
- -d '{
- "model": "local",
- "query": "What is machine learning?",
- "documents": [
- "ML is a subset of AI",
- "The weather is sunny today",
- "Neural networks learn from data"
- ]
- }'
- ```
- **Cache stats**
- ```bash
- curl http://localhost:8000/v1/cache/stats
- ```
- **Health check**
- ```bash
- curl http://localhost:8000/health
- ```
- ---
- ## Desktop App
- vMLX includes a native macOS desktop app (MLX Studio) with 5 modes:
- | Mode | Description |
- |------|-------------|
- | **Chat** | Conversation interface with chat history, thinking mode, tool calling, agentic coding |
- | **Server** | Manage model sessions -- start, stop, configure, monitor |
- | **Image** | Text-to-image generation and image editing with Flux, Kontext, Qwen, and Fill models |
- | **Tools** | Model converter, GGUF-to-MLX, inspector, diagnostics |
- | **API** | Live endpoint reference with copy-pasteable code snippets |
- <table align="center">
- <tr>
- <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/image-edit-tab.png" width="450" alt="Image generation and editing" /></td>
- <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/gguf-to-mlx.png" width="450" alt="Developer tools" /></td>
- </tr>
- <tr>
- <td align="center"><em>Image generation and editing with Flux models</em></td>
- <td align="center"><em>Developer tools -- model conversion and diagnostics</em></td>
- </tr>
- <tr>
- <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/anthropic-api.png" width="450" alt="Anthropic API endpoint" /></td>
- <td align="center"><img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/gguf-to-mlx.png" width="450" alt="GGUF to MLX conversion" /></td>
- </tr>
- <tr>
- <td align="center"><em>Anthropic Messages API endpoint -- full compatibility</em></td>
- <td align="center"><em>GGUF to MLX conversion -- bring your own models</em></td>
- </tr>
- </table>
- ### Download
- Get the latest DMG from [MLX Studio Releases](https://github.com/jjang-ai/mlxstudio/releases/latest), or build from source:
- ```bash
- git clone https://github.com/jjang-ai/vmlx.git
- cd vmlx/panel
- npm install && npm run build
- npx electron-builder --mac dmg
- ```
- ### Menu Bar
- vMLX lives in your menu bar showing all running models, GPU memory usage, and quick controls.
- <p align="center">
- <img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/menu-bar.png" width="300" alt="Menu Bar" />
- </p>
- ---
- ## Advanced Quantization
- vMLX supports standard MLX quantization (4-bit, 8-bit uniform) out of the box. For users who want to push further, **JANG adaptive mixed-precision** assigns different bit widths to different layer types -- attention gets more bits, MLP layers get fewer -- achieving better quality at the same model size.
- ### JANG Profiles
- | Profile | Attention | Embeddings | MLP | Avg Bits | Use Case |
- |---------|-----------|------------|-----|----------|----------|
- | `JANG_2M` | 8-bit | 4-bit | 2-bit | ~2.5 | Balanced compression |
- | `JANG_2L` | 8-bit | 6-bit | 2-bit | ~2.7 | Quality 2-bit |
- | `JANG_3M` | 8-bit | 3-bit | 3-bit | ~3.2 | **Recommended** |
- | `JANG_4M` | 8-bit | 4-bit | 4-bit | ~4.2 | Standard quality |
- | `JANG_6M` | 8-bit | 6-bit | 6-bit | ~6.2 | Near lossless |
- ### Convert
- ```bash
- pip install vmlx[jang]
- # Standard MLX quantization
- vmlx convert my-model --bits 4
- # JANG adaptive quantization
- vmlx convert my-model --jang-profile JANG_3M
- # Activation-aware calibration (better at 2-3 bit)
- vmlx convert my-model --jang-profile JANG_2L --calibration-method activations
- # Serve the converted model
- vmlx serve ./my-model-JANG_3M --continuous-batching --use-paged-cache
- ```
- Pre-quantized JANG models are available at [JANGQ-AI on HuggingFace](https://huggingface.co/JANGQ-AI).
- ### Smelt Mode (Partial Expert Loading)
- For MoE models that don't fit in RAM, **Smelt** loads only a subset of experts per layer from SSD and keeps the backbone resident. Routing is biased toward the resident experts, so response quality stays coherent while RAM usage drops. Trade-off: throughput scales inversely with expert % loaded, because expert swaps hit SSD on the hot path.
- ```bash
- # Load 50% of experts per layer (default when --smelt alone)
- vmlx serve ./MyMoE-JANG_4M --smelt --smelt-experts 50
- # Aggressive: load 25% — smallest RAM, slowest
- vmlx serve ./MyMoE-JANG_4M --smelt --smelt-experts 25
- ```
- **Benchmarks on `Nemotron-Cascade-2-30B-A3B-JANG_4M`** (23 MoE layers × 128 experts, Apple M3 Ultra / 128 GB, dedicated machine, no parallel models):
- | `--smelt-experts` | Active RAM | Decode tok/s | RAM saving | Coherent |
- |---|---:|---:|---|---|
- | _off (baseline)_ | **17,408 MB** | **89.9** | — | ✓ |
- | `50` | 9,529 MB | **66.5** | **−45%** | ✓ |
- | `25` | 5,590 MB | * | **−68%** | ✓ |
- \* Responses too short (2-5 tokens) for reliable steady-state tok/s measurement at 25 %. Subjectively responsive.
- All three configurations produced coherent, non-looping output. No quality degradation observed — routing bias keeps the model on-topic.
- > **Credit**: Smelt mode is inspired by [Anemll's **flash-moe**](https://github.com/Anemll/flash-moe) — a pure C / Objective‑C / Metal inference engine that showed huge MoE models (Qwen3.5-397B) can run on modest Apple Silicon hardware by streaming expert weights from SSD with `pread()` on demand. vMLX Smelt takes a different implementation path: Python/MLX, tied to the JANG quantization format, and loading a fixed subset of experts per layer at startup (backbone resident, routing biased toward the loaded subset) rather than on-demand per-token. It plugs into the full vMLX server with continuous batching, paged cache, and OpenAI-compatible API. Different engine, same core insight — thanks to the flash-moe team for validating the approach.
- **Smelt is mutually exclusive with VLM mode.** vMLX detects smelt and automatically disables `--is-mllm` (with a warning) because the vision tower is not wired through the partial-expert loader — image input on a smelt-loaded VLM would produce garbage logits. Use a text-only model when running smelt, or disable smelt when running a VLM.
- Smelt requires an MoE model in JANG format. Not compatible with dense models (no experts to partial-load) or with non-JANG formats.
- ---
- ## CLI Commands
- ```bash
- vmlx serve <model> # Start inference server
- vmlx convert <model> --bits 4 # MLX uniform quantization
- vmlx convert <model> -j JANG_3M # JANG adaptive quantization
- vmlx info <model> # Model metadata and config
- vmlx doctor <model> # Run diagnostics
- vmlx bench <model> # Performance benchmarks
- vmlx-worker --secret <secret> # Start distributed worker node
- ```
- ---
- ## Configuration
- ### Server Options
- ```bash
- vmlx serve <model> \
- --host 0.0.0.0 \ # Bind address (default: 0.0.0.0)
- --port 8000 \ # Port (default: 8000)
- --api-key sk-your-key \ # Optional API key authentication
- --continuous-batching \ # Enable concurrent request handling
- --enable-prefix-cache \ # Reuse KV states for repeated prompts
- --use-paged-cache \ # Block-based KV cache with dedup
- --kv-cache-quantization q8 \ # Quantize cache: q4 or q8
- --enable-disk-cache \ # Persist cache to SSD
- --enable-jit \ # JIT Metal kernel compilation
- --tool-call-parser auto \ # Auto-detect tool call format
- --reasoning-parser auto \ # Auto-detect thinking format
- --log-level INFO \ # Logging: DEBUG, INFO, WARNING, ERROR
- --max-model-len 8192 \ # Max context length
- --speculative-model <model> \ # Draft model for speculative decoding
- --enable-pld \ # Prompt Lookup Decoding — no draft model, best for code/JSON/schemas
- --distributed \ # Enable multi-Mac pipeline parallelism
- --cluster-secret <secret> \ # Shared auth secret for workers
- --distributed-mode pipeline \ # pipeline (default) or tensor (coming soon)
- --worker-nodes ip:port,... \ # Manual worker IPs (overrides auto-discovery)
- --cors-origins "*" # CORS allowed origins
- ```
- ### Quantization Options
- ```bash
- vmlx convert <model> \
- --bits 4 \ # Uniform quantization bits: 2, 3, 4, 6, 8
- --group-size 64 \ # Quantization group size (default: 64)
- --output ./output-dir \ # Output directory
- --jang-profile JANG_3M \ # JANG mixed-precision profile
- --calibration-method activations # Activation-aware calibration
- ```
- ### Image Generation & Editing Options
- ```bash
- pip install vmlx[image]
- # Generation models
- vmlx serve schnell \ # or dev, z-image-turbo
- --image-quantize 4 \ # Quantization: 4, 8 (omit for full precision)
- --port 8001
- # Editing models
- vmlx serve qwen-image-edit \ # Instruction-based editing (full precision only)
- --port 8001
- # Local model directory
- vmlx serve ~/.mlxstudio/models/image/FLUX.1-schnell-mflux-4bit
- ```
- ### Audio Options
- TTS and STT require the `mlx-audio` package:
- ```bash
- pip install mlx-audio
- # TTS: serve Kokoro model
- vmlx serve kokoro --port 8002
- # STT: serve Whisper model
- vmlx serve whisper --port 8003
- ```
- ### Optional Dependencies
- ```bash
- pip install vmlx # Core: text LLMs, VLMs, embeddings, reranking
- pip install vmlx[image] # + Image generation (mflux)
- pip install vmlx[jang] # + JANG quantization tools
- pip install vmlx[dev] # + Development/testing tools
- pip install vmlx[image,jang] # Multiple extras
- ```
- ---
- ## Architecture
- ```
- +--------------------------------------------+
- | Desktop App (Electron) |
- | Chat | Server | Image | Tools | API |
- +--------------------------------------------+
- | Session Manager (TypeScript) |
- | Process spawn | Health monitor | Tray |
- +--------------------------------------------+
- | vMLX Engine (Python / FastAPI) |
- | +--------+ +---------+ +-----------+ |
- | |Simple | | Batched | | ImageGen | |
- | |Engine | | Engine | | Engine | |
- | +---+----+ +----+----+ +-----+-----+ |
- | | | | |
- | +---+------------+--+ +-----+-----+ |
- | | mlx-lm / mlx-vlm | | mflux | |
- | +--------+-----------+ +-----------+ |
- | | |
- | +--------+----------------------------+ |
- | | MLX Metal GPU Backend | |
- | | quantized_matmul | KV cache | SDPA | |
- | +--------------------------------------+ |
- +--------------------------------------------+
- | L1: Prefix Cache (Memory-Aware / Paged) |
- | L2: Disk Cache (Persistent / Block Store) |
- | KV Quant: q4/q8 at storage boundary |
- +--------------------------------------------+
- ```
- ---
- ## Contributing
- Contributions are welcome. Here is how to set up a development environment:
- ```bash
- git clone https://github.com/jjang-ai/vmlx.git
- cd vmlx
- # Python engine
- python -m venv .venv && source .venv/bin/activate
- pip install -e ".[dev,jang,image]"
- pytest tests/ -k "not Async" # 2000+ tests
- # Electron desktop app
- cd panel && npm install
- npm run dev # Development mode with hot reload
- npx vitest run # 1545+ tests
- ```
- ### Project Structure
- ```
- vmlx/
- vmlx_engine/ # Python inference engine (FastAPI server)
- panel/ # Electron desktop app (React + TypeScript)
- src/main/ # Electron main process
- src/renderer/ # React frontend
- src/preload/ # IPC bridge
- tests/ # Python test suite
- assets/ # Screenshots and logos
- ```
- ### Guidelines
- - Run the full test suite before submitting PRs
- - Follow existing code style and patterns
- - Include tests for new features
- - Update documentation for user-facing changes
- ---
- ## License
- Apache License 2.0 -- see [LICENSE](LICENSE).
- ---
- <p align="center">
- Built by <a href="https://github.com/jjang-ai">Jinho Jang</a> (eric@jangq.ai)<br>
- <a href="https://jangq.ai">JANGQ AI</a> • <a href="https://pypi.org/project/vmlx/">PyPI</a> • <a href="https://github.com/jjang-ai/vmlx">GitHub</a> • <a href="https://github.com/jjang-ai/mlxstudio/releases">Downloads</a>
- </p>
- ---
- ## 한국어 (Korean)
- ### vMLX — Apple Silicon을 위한 로컬 AI 엔진
- Mac에서 LLM, VLM, 이미지 생성 및 편집 모델을 완전히 로컬로 실행하세요.
- OpenAI + Anthropic 호환 API. 클라우드 없음. API 키 불필요. 데이터가 기기를 떠나지 않습니다.
- ### 빠른 시작
- ```bash
- pip install vmlx
- vmlx serve mlx-community/Llama-3.2-3B-Instruct-4bit
- ```
- ### 주요 기능
- | 기능 | 설명 |
- |------|------|
- | **텍스트 생성** | MLX 및 JANG 형식의 LLM 추론 |
- | **비전-언어 모델** | 이미지 + 텍스트 멀티모달 추론 |
- | **이미지 생성** | Flux Schnell/Dev, Z-Image Turbo (mflux 기반) |
- | **이미지 편집** | Qwen Image Edit (텍스트 지시 기반 이미지 편집) |
- | **5단계 캐싱** | 프리픽스, 페이지드, KV 양자화, 디스크, 메모리 인식 캐시 |
- | **연속 배칭** | 다중 동시 요청 처리 |
- | **에이전트 도구** | 30개 내장 도구 (파일, 웹 검색, Git, 터미널) |
- | **OpenAI API** | /v1/chat/completions, /v1/images/generations, /v1/images/edits |
- | **Anthropic API** | /v1/messages (스트리밍, 도구 호출, 시스템 프롬프트) |
- ### 이미지 생성
- ```bash
- pip install vmlx[image]
- vmlx serve schnell # 빠른 생성 (4 단계)
- vmlx serve dev # 고품질 생성 (20 단계)
- ```
- ### 이미지 편집
- ```bash
- vmlx serve qwen-image-edit # 텍스트 지시 기반 이미지 편집
- ```
- ```bash
- # 이미지 편집 API
- curl http://localhost:8000/v1/images/edits \
- -H "Content-Type: application/json" \
- -d '{
- "model": "qwen-image-edit",
- "prompt": "배경을 해질녘으로 변경",
- "image": "<base64 인코딩된 이미지>",
- "size": "1024x1024",
- "strength": 0.8
- }'
- ```
- ### 데스크톱 앱 (MLX Studio)
- macOS 네이티브 데스크톱 앱으로 5가지 모드를 제공합니다:
- | 모드 | 설명 |
- |------|------|
- | **채팅** | 대화 인터페이스, 채팅 기록, 도구 호출, 에이전트 코딩 |
- | **서버** | 모델 세션 관리 — 시작, 정지, 설정, 모니터링 |
- | **이미지** | 텍스트-이미지 생성 및 이미지 편집 (Flux, Qwen 모델) |
- | **도구** | 모델 변환기, GGUF-MLX 변환, 진단 |
- | **API** | 실시간 엔드포인트 참조 및 코드 스니펫 |
- <p align="center">
- <img src="https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/image-edit-tab.png" width="450" alt="이미지 생성 및 편집" />
- </p>
- ### 설치
- ```bash
- pip install vmlx # 기본: 텍스트 LLM, VLM, 임베딩
- pip install vmlx[image] # + 이미지 생성/편집 (mflux)
- pip install vmlx[jang] # + JANG 양자화 도구
- pip install vmlx[audio] # + TTS/STT (mlx-audio)
- ```
- ### 라이선스
- Apache License 2.0 — [LICENSE](LICENSE) 참조.
- ---
- <p align="center">
- 개발자: <a href="https://github.com/jjang-ai">장진호</a> (eric@jangq.ai)<br>
- <a href="https://jangq.ai">JANGQ AI</a> •
- <a href="https://ko-fi.com/jangml">Ko-fi로 후원하기</a>
- </p>
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.