localrouter
Fast & opinionated local AI routing: one MCP server that loads local image and video models on demand on your own GPU box (DGX Spark first)
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/jayleaton/localrouter with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/jayleaton/localrouter'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/jayleaton--localrouter)HTML
<a href="/listings/jayleaton--localrouter"><img src="/api/listings/jayleaton--localrouter/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
git clone <this repo> localrouter && bash localrouter/tools/spark/install.sh
The installer checks the machine (GPU, Docker with the NVIDIA runtime, disk), downloads the official checkpoints and converts them inside NVIDIA's PyTorch container (the first run takes a while and needs about 200 GB free; the converted image DiT weights are verified against committed reference digests), builds and starts the service (it restarts with the machine), makes one image and a one-second clip through MCP as a check (printing the clip's wall time), and prints how to connect agents. `MODELS=image` or `MODELS=video` installs just one (about 110 GB free for images only; the default, both, needs about 200 GB). The image model is FP8 (`qwen-image-2.1`, kept warm); `PRECISIONS="fp8s nvfp4"` also builds the faster NVFP4 pack and serves it as `qwen-image-2.1-nvfp4`. Qwen-Image 2.1 Turbo is installed beside it as `qwen-image-2.1-turbo` (about 25 GB more: only its DiT differs, the text encoder and VAE are shared) and becomes the image model a request gets when it names none, kept warm; `IMAGE_MODELS=qwen-image-2.1` installs the base model only, `IMAGE_DEFAULT=qwen-image-2.1` keeps it the default. LocalRouter listens on `127.0.0.1` only unless you say otherwise: on a terminal the installer asks "Make LocalRouter reachable from your tailnet? [y/N]" once, and remembers the answer in `.env` under `LOCALROUTER_HOME`. Re-running it updates the service using the existing packs. The complete two-model installer still needs an end-to-end check on a DGX Spark; the video installation path has only been exercised with stubs.
Prebuilt Qwen-Image 2.1 packs are available as an optional download from [Hugging Face](https://huggingface.co/jayleaton/localrouter-qwen-image-2.1), for non-commercial research or evaluation only under the Qwen Research License. The installer still converts the official checkpoint locally by default.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 257 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <p align="center"><img src="docs/assets/localrouter-hero.png" alt="LocalRouter" width="100%"></p>
- <p align="center"><a href="https://x.com/jayleaton"><img src="https://img.shields.io/badge/Follow-%40jayleaton-000000?logo=x&logoColor=white" alt="Follow @jayleaton"></a> <a href="https://buymeacoffee.com/jayleaton"><img src="https://img.shields.io/badge/Buy_Me_a_Coffee-Support-FFDD00?logo=buymeacoffee&logoColor=black" alt="Buy Me a Coffee: Support"></a></p>
- > [!WARNING]
- > **Early preview.** LocalRouter is new and under active development: expect bugs and breaking changes. Please [open an issue](https://github.com/jayleaton/localrouter/issues) if something doesn't work.
- **LocalRouter turns one NVIDIA DGX Spark into a full-duplex "Jarvis" box.** You talk to it and it talks back. You can
- interrupt it mid-sentence, and it can make a picture or a video while you talk. Everything runs on the box:
- - **Chat:** Qwen3.8 27B on TensorFold, with the DFlash2 drafter.
- - **Voice:** Fish Audio Drama 3, streamed as it speaks, with six fixed voices.
- - **Hearing:** Whisper large-v3-turbo with the Silero VAD.
- - **Images:** Qwen-Image 2.1 Turbo: text to image, and edits of one or two input images (an instruction, optionally a mask).
- - **Video with audio:** MiniMax H3: text to video, and image to video (the input image as the first frame), up to 1280x704 and 8 s.
- One small Zig daemon owns the machine. It keeps the voice loop and chat warm, loads the other models when asked,
- makes room for them, and hands every tool to your agents over MCP and an OpenAI-compatible API.
- ## Quick start
- Tell your coding agent (Claude Code, Codex, T3, ...):
- > *Read INSTALL.md and set up LocalRouter Jarvis on this box.*
- [INSTALL.md](INSTALL.md) is written for agents. It is a deterministic sequence of commands, with what to check at each
- step and what to ask you: which licences you accept, and whether the GPU may be used. To run it yourself:
- ```bash
- git clone https://github.com/jayleaton/localrouter && cd localrouter
- tools/install/bootstrap.sh --profile jarvis-gb10 --dry-run # the plan: steps, downloads, disk, memory, licences
- tools/install/bootstrap.sh --profile jarvis-gb10 --accept # after reading the licences the plan lists
- ```
- **What it does.** It installs every model of [deploy/profiles/jarvis-gb10.json](deploy/profiles/jarvis-gb10.json), checks
- every download against its pin and qualifies chat and Drama 3 on the GPU. It leaves one systemd user service on
- `127.0.0.1:8190`, which starts at boot and is reachable on your tailnet through `tailscale serve`. It is resumable: after
- a failure, do what it says and run it again. Models already on the box are reused, not downloaded again.
- `tools/install/doctor.sh --profile jarvis-gb10` lists anything missing, with its fix.
- **On an Apple Silicon Mac (preview).** `tools/install/bootstrap.sh --profile mac-apple-silicon --dry-run`, then
- `--accept`, installs and qualifies chat, Drama 3, Whisper, KnowLine and Qwen-Image on Metal. It leaves one launchd
- user agent on `127.0.0.1:8190`, which starts at login, and `deploy/mac/` scripts to start, stop, check and pair it.
- [docs/dev/MAC-PORT.md](docs/dev/MAC-PORT.md#installing-on-a-mac) has the steps and the measured memory and speed.
- **A fresh install on DGX-01** (measured 2026-10-22):
- | Part | Measured |
- | --- | --- |
- | Downloads and builds | 31.5 min, 175.7 GB downloaded |
- | Chat qualification on the GPU | 113 s |
- | Drama 3 qualification on the GPU | 20.5 min |
- | Image and video pack builds | not yet timed end to end |
- ## What you get
- | Where | What |
- | --- | --- |
- | `/voice` | The web app. A full-duplex voice conversation: barge-in, spoken picture requests, and a voice picker with previews |
- | `/v1` | OpenAI-compatible API: chat completions (whole or streamed, with tools), audio speech and transcriptions, images, videos, and the Realtime API at `/v1/realtime` |
- | `/mcp` | Every tool over MCP: `chat`, `synthesize_speech`, `transcribe_audio`, `generate_image`, `generate_video`, `list_models`, `list_voices`, jobs. Setup for Claude Code, Codex, Cursor and any client: [docs/MCP.md](docs/MCP.md) |
- ```bash
- claude mcp add --transport http localrouter http://127.0.0.1:8190/mcp
- ```
- ## Desktop app (beta)
- **LocalRouter for desktop** brings Jarvis to your Mac, Windows or Linux computer: chat and calls with your box's
- models, and, after your approval, control of that computer. **[Download it from the Releases
- page](https://github.com/jayleaton/localrouter/releases)**: a `.dmg` for Apple silicon or Intel Macs, an installer for
- Windows, an AppImage or `.deb` for Linux. The builds are not notarized (macOS) or code-signed (Windows); each release's
- notes say how to open them. More in [apps/companion](apps/companion/README.md).
- ## Measured on a DGX Spark
- DGX-01, NVIDIA GB10, 2026-10-22. Only figures we measured are listed.
- | What | Measured |
- | --- | --- |
- | Voice turn: end of speech to first audio | about 0.9 s median (916 to 959 ms across today's smoke runs); 0.72 to 0.95 s for the first turn after an image or a video |
- | Drama 3: first audio | 0.14 s p50, 0.18 s p95. Real-time factor 0.68. From its GPU qualification |
- | Chat: first content | 0.15 s for a short prompt, streamed |
- | "Generate a picture of a dog", spoken | picture done 8.1 to 10.2 s after the end of speech; spoken confirmation 8.8 to 11.4 s |
- | Image: Qwen-Image 2.1 Turbo, 1024x1024 | 6.3 s warm |
- | Image edit: Qwen-Image 2.1 Turbo, one input at 1024x1024 | 11 to 12.5 s (a 1024x576 edit 7.2 s; two inputs 15.9 s) |
- | Video: MiniMax H3 with audio, the engine alone | 864x480, 5.17 s: 83 s warm. 1280x704, 5.17 s: 244 s. 1280x704, 8 s: 397 s |
- | Video through the API (load included) | a 2.3 s clip at 768x448, text or image to video: 33 to 39 s |
- | Memory with chat, Drama 3, Whisper and the image model warm | 24 + 42 + 3.6 + 35.2 GiB resident; 13.7 GiB still available after a start, 11 to 12 GiB in use, of 121.7 GiB |
- A video job unloads only the image model while it runs, and the image model comes back after it. Chat, Drama 3 and
- Whisper are never unloaded for an image or a video.
- ## Hardware
- | Need | jarvis-gb10 (the tested profile) |
- | --- | --- |
- | Box | NVIDIA DGX Spark (GB10, `aarch64`): 128 GB unified memory, at least 110 GiB visible to the OS |
- | Software | DGX OS / Ubuntu 24.04, NVIDIA driver 580 or newer, CUDA 13 (`nvcc`), Docker with the NVIDIA runtime (for the image and video packs), systemd user services |
- | Disk | about 250 GiB free for a fresh install (216 GiB plus a 30 GiB margin). `--dry-run` prints the exact figure |
- | Network | HTTPS to huggingface.co, github.com, ziglang.org, pypi.org, download.pytorch.org and nvcr.io during the install |
- No other profile is tested yet. `mac-apple-silicon` is a preview: an Apple Silicon Mac with at least 64 GiB,
- [docs/dev/MAC-PORT.md](docs/dev/MAC-PORT.md#installing-on-a-mac). The GB10 profile's chat and Drama 3 engines are GB10
- builds, and their qualification refuses other GPUs.
- ## How it works
- - **One door for every agent.** Add one MCP server, or point any OpenAI client at `/v1`.
- - **Memory is managed for you.** Each model says what it needs before it loads. If it does not fit, idle models of
- lower priority give way. Kept-warm models (the voice set and chat) come back by themselves, and only with room to
- spare.
- - **Correct.** The image and video engines are LocalRouter's own, written in Zig with their own GPU kernels. They are
- verified operation by operation against a Python reference of each model, then from prompt to pixels and audio
- samples. Chat and Drama 3 are qualified on the box's own GPU before the service serves them.
- - **Small.** One Zig binary. Every loaded model is its own process, so unloading returns all of its memory and a crash
- costs one job.
- ## Image and video only (Docker)
- `bash tools/spark/install.sh` installs just the image and video models as a Docker service. The script's header
- documents it: `MODELS`, `PRECISIONS` (FP8 by default, NVFP4 optional), `IMAGE_MODELS`, and binding with
- `LOCALROUTER_BIND=localhost|tailscale|all`. Prebuilt Qwen-Image 2.1 packs are optional, from
- [Hugging Face](https://huggingface.co/jayleaton/localrouter-qwen-image-2.1), for non-commercial research or evaluation
- only under the Qwen Research License.
- ## Connect an agent
- Any MCP client: `{"mcpServers": {"localrouter": {"type": "http", "url": "http://<host>:8190/mcp"}}}`. For clients that
- only speak stdio: `localrouter mcp-stdio --url http://<host>:8190`.
- | MCP tool | Does |
- | --- | --- |
- | `list_models`, `list_voices` | the models (what each can do, `default_for`, state, memory) and the speech voices |
- | `reserve_for`, `unload_model`, `load_model` | make room for a heavy job (what it unloads comes back after it), unload or load one model; never the chat model. The procedure is a skill: [skills/jarvis-model-manager](skills/jarvis-model-manager/SKILL.md) |
- | `generate_image` | text to image. Waits up to `wait_s`, returns the PNGs inline and as links, or a job id |
- | `edit_image` | an edit of one or more input images by an instruction, optionally with a mask; the same reply as `generate_image` |
- | `generate_video` | text to video, or image to video with `image`. Waits up to `wait_s`, then returns the MP4's link or a job id |
- | `chat` | a chat completion from the chat model: the reply's text (or its tool calls), then the whole completion. `reasoning: false` answers without thinking |
- | `synthesize_speech`, `transcribe_audio` | speech as WAV (inline and a link) in any listed voice; the transcript of a URL, base64 audio or a file on the box |
- | `get_job`, `cancel_job` | follow or stop a long job |
- | `release_models` | unload everything now |
- Config snippets for every client, an example call for every tool and the security note are in [docs/MCP.md](docs/MCP.md).
- The HTTP API, field by field, is in [docs/AGENTS.md](docs/AGENTS.md), and an agent skill is in
- [skill/localrouter](skill/localrouter/SKILL.md). What a harness on another device may rely on (versioned, with
- `GET /v1/capabilities` to discover it and what is not offered) is [docs/HARNESS.md](docs/HARNESS.md).
- ## Configuration
- `localrouter serve --config tools.json` (the installer writes one):
- ```json
- {
- "reserve_bytes": 8589934592,
- "defaults": {"text_to_image": "qwen-image-2.1-turbo"},
- "tools": [
- {"id": "qwen-image-2.1-turbo", "kind": "image", "engine": "qwen_image_turbo", "weights": "/models/qwen-image-2.1",
- "priority": 10, "keep_loaded": true, "options": {"precision": "fp8s"}},
- {"id": "qwen-image-2.1", "kind": "image", "engine": "qwen_image", "weights": "/models/qwen-image-2.1",
- "priority": 5, "options": {"precision": "fp8s"}},
- {"id": "qwen-image-2.1-nvfp4", "kind": "image", "engine": "qwen_image", "weights": "/models/qwen-image-2.1",
- "priority": 10, "options": {"precision": "nvfp4"}},
- {"id": "minimax-h3", "kind": "video", "engine": "minimax_h3", "weights": "/models/minimax-h3",
- "idle_ttl_s": 30, "options": {"max_width": 768, "max_height": 448, "max_seconds": 5}}
- ]
- }
- ```
- | Key | Default | Meaning |
- | --- | --- | --- |
- | `host`, `port` | `127.0.0.1`, `8190` | where the API listens. `host` is an IP, `localhost`, `all` (`0.0.0.0`) or `tailscale` (this machine's tailnet IPv4); `--host` takes the same |
- | `data_dir` | `/data` | job inputs and outputs (`jobs/`) and worker logs (`logs/<tool>.log`) |
- | `reserve_bytes` | 8 GiB | memory always left free for the OS and other services |
- | `budget_bytes` | 0 (none) | a cap on what loaded models and running jobs may use |
- | `keep_outputs_s` | 86400 | how long outputs stay (in `data_dir`/jobs; on start the daemon also removes older ones left from before a restart) |
- | `defaults` | none | the model a request that names none gets, per capability: `{"text_to_image": "qwen-image-2.1-turbo"}`. Unset: the first model (in config order) that can do it. Each must name a configured model with that capability, or `serve` refuses to start. `localrouter models` and `list_models` show the result |
- | `gpu_lock` | none | the host's GPU lock file, the one GPU jobs outside LocalRouter take exclusively. LocalRouter holds it once, shared and without waiting, while any of its models is loaded, and releases it when none is; while someone else holds it nothing loads (503). LocalRouter's own models never take it: they are admitted by memory alone, so chat, speech and transcription stay loaded side by side |
- | `allow_private_urls` | false | let input image URLs (edits, image to video) name loopback, private and link-local hosts; by default they are refused, redirects are never followed, and only the daemon's own output URLs are fetched from the machine itself |
- | `tools[].engine` / `cmd` | | a compiled-in engine, or any program speaking the [worker protocol](docs/ARCHITECTURE.md#any-program-the-cmd-adapter) |
- | `tools[].priority` | 0 (chat: 100) | higher is unloaded later when memory is needed |
- | `tools[].keep_loaded` | false (chat: true) | load at start, never unload for idleness, reload as soon as the scheduler is idle (unloading idle tools of lower priority) |
- | `tools[].qualification` | none | a file the model's qualify script writes on this machine. Until it exists and says `"qualified": true` the model is listed (`"qualified": false` in `/v1/models`) and refused with a 503 saying it is not GPU-qualified on this host |
- | `tools[].idle_ttl_s` | 120 | unload after this long unused (0: right after each job) |
- | `tools[].capabilities` | from the engine | `text_to_image`, `image_edit`, `text_to_video`, `image_to_video`, `text_to_speech`, `speech_to_text`, `chat_completion` |
- | `tools[].engine` | | `qwen_image` (Qwen-Image 2.1, 25 steps), `qwen_image_turbo` (Qwen-Image 2.1 Turbo, its checkpoint's 8 steps; other step counts are refused), `minimax_h3`, `kandinsky6` (refuses to load until qualified; its options `fp32_linear`, `clip_linear`, `qwen_rope_freq`, `qwen_inv_freq` and `audio` are required; the exact `audio` values are in `kernels/cuda/kandinsky6/audio/QUALIFICATION.md`), `openai_server` (a model's own OpenAI-compatible server run as the worker: chat), `testpattern` |
- | `tools[].options` | | engine settings (precision: `fp8s`, the default, or `nvfp4`; `max_side`; `max_pixels`, the largest width x height, which sizes the image engines' scratch (about 4.6 KiB a pixel on GB10 FP8); default steps; the Qwen engines' `dit`: the DiT pack directory, default `<precision>` or `turbo-<precision>`) |
- There is no authentication: it listens on localhost unless you choose otherwise; use a private network or a tailnet
- (the Jarvis install puts `tailscale serve` HTTPS in front of it; `--host tailscale` binds the tailnet address) and never
- `all` on an untrusted network.
- ## Build and test
- Zig 0.17.0; ffmpeg for video output. The GPU engines need nvcc and an NVIDIA GPU (sm_120 / sm_121); without `-Dnvcc` the
- binary builds without embedded CUDA kernels, and the GPU-free test engine runs the tests. The model engines
- remain registered but cannot generate without their kernels. To check a running box from any machine as a client:
- `tools/harness/nemo_check.sh --url http://<box>:8190` ([HARNESS.md](docs/HARNESS.md#harnesses)).
- ```bash
- zig build # zig-out/bin/localrouter
- zig build test # unit + integration tests (a real daemon and real workers)
- zig build integration-exe && zig-out/bin/integration-test # the integration tests one at a time, each named as it runs
- zig build -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121,120 # with the GPU engines
- localrouter check selftest # a private daemon, one image and one video, no GPU
- ```
- ## How it was built
- The engines are ports, not wrappers. Each model was first reproduced in Python on the project's own kernels (the
- "twin", in `tools/twin`), measured against the reference implementation, then rewritten in Zig and gated against the
- twin: every operation alone and chained, then the whole generation, on the target GPU. The measured results and the
- milestones are in [docs/dev](docs/dev/).
- ## Credits
- LocalRouter stands on the work of these projects and teams:
- - [**TensorFold**](https://github.com/ashhart/TensorFold) by **Ash Hart**. Its CUDA runtime, NVFP4 kernels and Python
- model implementations were the baselines our image and video engines were ported from and verified against. Its
- native server runs the chat model, and Drama 3 runs on it. Thank you, Ash; we expect to contribute parts of the Zig
- work back upstream.
- - **Qwen** (Alibaba): Qwen3.8 27B for chat, and Qwen-Image 2.1 and 2.1 Turbo for images.
- - **z-lab**: the [DFlash2](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2) speculative drafter for Qwen3.8 27B.
- - **Fish Audio**: [Drama 3](https://huggingface.co/fishaudio/drama-3), the voice. Built with Fish Audio.
- - [**drama-tensorfold**](https://github.com/plotarmordev/drama-tensorfold) by **plotarmordev** (Apache-2.0), which ports
- Drama 3 to TensorFold.
- - **OpenAI**: [Whisper](https://github.com/openai/whisper) large-v3-turbo, for speech to text.
- - [**Silero VAD**](https://github.com/snakers4/silero-vad): voice activity detection.
- - **MiniMax**: MiniMax H3, video with audio.
- - **Kandinsky Lab**: Kandinsky-6 Pro distilled, video with coupled audio (built in; it refuses to load until its GPU
- qualification passes).
- - **NVIDIA**: the DGX Spark (GB10) and the NGC PyTorch container the packs are built in. The INT8 attention kernels
- are SageAttention's, as shipped in NVIDIA's comfy-kitchen.
- The weights are downloaded from their publishers under their own licences. Check these terms before installing or
- using a model (a summary, not legal advice; the licence text is what binds):
- | Model | Licence | Use | Restrictions / notice |
- | --- | --- | --- | --- |
- | Qwen3.8 27B (chat; TensorFold's MLX 4-bit checkpoint and z-lab's DFlash2 drafter) | [Apache 2.0](https://huggingface.co/TensorFold/Qwen3.8-27B-MLX-4bit) | any, commercial included | |
- | KnowLine-4B Gen5.5 (decisions) | [Apache 2.0](https://huggingface.co/PelaAI/KnowLine-4B-Gen5.5) | any, commercial included | |
- | Whisper large-v3-turbo (speech to text) | [MIT](https://huggingface.co/openai/whisper-large-v3-turbo) | any, commercial included | |
- | Fish Audio Drama 3 (speech) | [Fish Audio Community License v1.0](https://huggingface.co/fishaudio/drama-3/blob/e0a9218aa48888270f06ef1ed9ea67622c0e17d2/LICENSE) | research and non-commercial; commercial only under USD 500,000 annual revenue | no Hosted Service (third parties generating speech, e.g. through an API) without a separate licence from Fish Audio, so keep LocalRouter on loopback or your tailnet for it ([details](docs/dev/DRAMA3.md#licences)). "Built with Fish Audio" |
- | Kitten TTS 2 (optional speech tool) | [Stellon Labs Community License](https://huggingface.co/KittenML/kitten-tts-2/blob/f2eee74f4467855ad58fe16f1fb7b7df473263fd/LICENSE.md) | research, non-commercial; commercial after registering with Stellon Labs, until USD 1,000,000 annual revenue or total funding | outputs may not train another model. "Powered by Stellon Labs" ([details](docs/dev/KITTEN-TTS2.md#licences)) |
- | Qwen-Image 2.1 and 2.1 Turbo (images) | [Qwen Research License](https://huggingface.co/Qwen/Qwen-Image-2.1/blob/main/LICENSE) | non-commercial research or evaluation only | commercial use needs a separate licence from Qwen |
- | MiniMax H3 (video) | [MiniMax H3 Community License](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE) | outside the excluded territories, under its Acceptable Use Policy; commercial included, over USD 20 million yearly revenue with MiniMax's written authorization | the model and its outputs are not licensed in the US, EU, UK or South Korea. Content published to the public must be disclosed as machine-generated; a commercial product shows "MiniMax H3" in its interface |
- | Kandinsky-6 Pro distilled (video; not yet available) | [MIT](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers) | any, commercial included | |
- `GET /v1/models` lists each model's licence as `license` (from the tool's config, else its engine's), and
- `GET /v1/capabilities` lists them all. LocalRouter informs; it does not enforce any of these terms, and installing
- locally does not waive them.
- The Apache-2.0 licence below covers the software. See [NOTICE](NOTICE).
- ## License
- Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.