self-hosted-ai-stack
Deploy a self-hosted AI stack with Docker Compose: local LLMs, chat, document parsing, embeddings, transcription, text-to-speech and MCP tools. Powered by Ollama, LiteLLM, AnythingLLM and more. Local-first with auto-configured authentication, lightweight stacks, optional HTTPS and NVIDIA GPU support. MIT licensed. Multi-arch: amd64, arm64.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/hwdsl2/self-hosted-ai-stack with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/hwdsl2/self-hosted-ai-stack'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/hwdsl2--self-hosted-ai-stack)HTML
<a href="/listings/hwdsl2--self-hosted-ai-stack"><img src="/api/listings/hwdsl2--self-hosted-ai-stack/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
**Requirements:**
- A Linux server (local or cloud) with Docker installed - At least 8 GB of RAM (with small models). For larger LLM models (8B+), 16 GB or more is recommended. - You can comment out services you don't need to reduce memory usage.
**Start the full stack:**
# Clone the repository to get the compose files git clone https://github.com/hwdsl2/self-hosted-ai-stack cd self-hosted-ai-stack docker compose up -d
> **PostgreSQL credentials:** Fresh installs and existing default installs are handled automatically. If you previously set a custom database password, see [PostgreSQL credentials](#postgresql-credentials) before starting.
**Pull a model** (required before making LLM requests):
docker exec ollama ollama_manage --pull llama3.2:3b
Run the health check to verify all services are working:
./stack-check.sh
> **Tip:** On first start, services may take a few minutes to initialize. If any checks fail, wait and run `./stack-check.sh` again. Use `docker compose logs` to check progress.
For detailed troubleshooting, see the [Troubleshooting](docs/troubleshooting.md) guide.
**Get the GatewayCrate master key** (used to log into the Admin UI and for LLM requests):
docker exec litellm litellm_manage --showkey
<details> <summary>Show core API keys (InferCrate, GatewayCrate, UplinkCrate)</summary>
docker exec ollama ollama_manage --showkey docker exec litellm litellm_manage --showkey docker exec mcp mcp_manage --showkey
</details>
**Access AnythingLLM (Chat UI):**
AnythingLLM is pre-configured to connect to your local LLM via GatewayCrate. On first start, it may take a few minutes to become available (check progress with `docker logs anythingllm`).
**Password-protected by default.** A random admin password is auto-generated on first start, printed once to `docker logs anythingllm`, and saved to `/app/server/storage/.initial_admin_password` inside the `anythingllm-data` volume. The seeded password persists across container upgrades. Change it any time from **Settings → Security**; after you do, `.initial_admin_password` may no longer match the current login password.
Retrieve the auto-generated password:
# At any time from the data volume: docker exec anythingllm cat /app/server/storage/.initial_admin_password # Or from the live logs (only shown on first start): docker compose logs anythingllm | grep -A4 "FIRST RUN"
Open `http://<server-ip>:3001` in your browser and log in with the password above.
> [!NOTE] > When exposing AnythingLLM beyond `localhost` or a trusted LAN, use the included Caddy HTTPS overlay so the password is encrypted in transit and direct HTTP ports are bound to localhost. See [Internet-facing deployments](#internet-facing-deployments) below.
**Access GatewayCrate's LiteLLM Admin UI:**
Open `http://<server-ip>:4000/ui` in your browser. Log in with username `admin` and your GatewayCrate master key as the password. The UI provides virtual key management, spend tracking, and model configuration.
> **Tip:** In the Admin UI, click **Playground** in the left menu. Select a local model (e.g., `ollama-chat/llama3.2:3b`) from the dropdown and start chatting — a quick way to verify your local LLM is working end-to-end.
**Use the stack with goose:**
Install goose on your workstation and connect it to the stack's GatewayCrate endpoint for supervised agent and coding tasks. The [goose setup guide](https://selfhostedaistack.com/goose) covers native installation, a restricted GatewayCrate key, local-model expectations, permissions, testing, and optional UplinkCrate access.
**Stop the stack:**
# Stop and remove all containers (data is preserved in Docker volumes) docker compose down
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 702 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- [English](README.md) | [简体中文](docs/README-zh.md) | [繁體中文](docs/README-zh-Hant.md) | [Русский](docs/README-ru.md)
- # Self-Hosted AI Stack
- [](https://docs.docker.com/compose/) [](https://hub.docker.com/u/hwdsl2) [](https://opensource.org/licenses/MIT)
- <p align="center">
- <img src="docs/images/self-hosted-ai-stack-overview.png"
- alt="Self-Hosted AI Stack: deploy a complete self-hosted AI stack with one command"
- width="100%">
- </p>
- Run local AI models, chat with your documents, transcribe audio, and connect MCP tools with Docker Compose. Optional services add document parsing, text-to-speech, and real-time transcription.
- **Features:**
- - **Zero-config:** all services auto-configure on first start.
- - **Secure by default:** AnythingLLM password protection is enabled, and bundled API services auto-generate keys.
- - **HTTPS-ready:** optional Caddy overlay provides automatic TLS and binds direct HTTP ports to localhost.
- - **Local by default:** runs locally, with optional external provider support via GatewayCrate.
- - **AI agent ready:** connect [goose](https://github.com/aaif-goose/goose) on your workstation to GatewayCrate with the [setup guide](https://selfhostedaistack.com/goose).
- - **Flexible configuration:** customize models, ports, providers, and API keys with simple env files.
- - **[Lightweight stacks](#lightweight-stacks):** for lower memory requirements (as low as ~4.5 GB).
- - **GPU acceleration:** via NVIDIA CUDA.
- - **Multi-architecture support:** `linux/amd64`, `linux/arm64`.
- > 📘 **The Self-Hosted AI Builder’s Guide** is the hands-on companion to this stack, covering deployment, security, backups, and upgrades. [Kindle](https://books2read.com/aiguide?store=amazon) · [Google Play](https://books2read.com/aiguide?store=google) · [paperback](https://books2read.com/aiguide?store=amazon-paperback&format=PAPERBACK)
- ## Included services
- | Service | Role | Default port |
- |---|---|---|
- | **[InferCrate (Ollama LLM)](https://github.com/hwdsl2/infercrate)** | Runs local LLM models (llama3, qwen, mistral, etc.) | `11434` |
- | **[AnythingLLM](https://github.com/mintplex-labs/anything-llm)** | Web-based chat UI — password-protected by default | `3001` |
- | **[GatewayCrate (LiteLLM)](https://github.com/hwdsl2/gatewaycrate)** | AI gateway with Admin UI — routes requests to InferCrate and 100+ providers | `4000` |
- | **[EmbedCrate (Hugging Face TEI)](https://github.com/hwdsl2/embedcrate)** | Converts text to vectors for semantic search and RAG | `8000` |
- | **[ScribeCrate (Whisper STT)](https://github.com/hwdsl2/scribecrate)** | Transcribes spoken audio to text | `9000` |
- | **[ScribeCrate Live (real-time STT)](https://github.com/hwdsl2/scribecrate-live)** | Real-time speech-to-text transcription over WebSocket | `9090` |
- | **[SpeakCrate (Kokoro TTS)](https://github.com/hwdsl2/speakcrate)** | Converts text to natural-sounding speech | `8880` |
- | **[UplinkCrate (MCPHub)](https://github.com/hwdsl2/uplinkcrate)** | Provides MCP tools (filesystem, fetch, GitHub, search, databases) to AI clients | `3000` |
- | **[ParseCrate (Docling)](https://github.com/hwdsl2/parsecrate)** | Converts documents (PDF, DOCX, etc.) to structured text/Markdown | `5001` |
- ## Quick start
- **Requirements:**
- - A Linux server (local or cloud) with Docker installed
- - At least 8 GB of RAM (with small models). For larger LLM models (8B+), 16 GB or more is recommended.
- - You can comment out services you don't need to reduce memory usage.
- **Start the full stack:**
- ```bash
- # Clone the repository to get the compose files
- git clone https://github.com/hwdsl2/self-hosted-ai-stack
- cd self-hosted-ai-stack
- docker compose up -d
- ```
- > **PostgreSQL credentials:** Fresh installs and existing default installs are handled automatically. If you previously set a custom database password, see [PostgreSQL credentials](#postgresql-credentials) before starting.
- **Pull a model** (required before making LLM requests):
- ```bash
- docker exec ollama ollama_manage --pull llama3.2:3b
- ```
- Run the health check to verify all services are working:
- ```bash
- ./stack-check.sh
- ```
- > **Tip:** On first start, services may take a few minutes to initialize. If any checks fail, wait and run `./stack-check.sh` again. Use `docker compose logs` to check progress.
- For detailed troubleshooting, see the [Troubleshooting](docs/troubleshooting.md) guide.
- **Get the GatewayCrate master key** (used to log into the Admin UI and for LLM requests):
- ```bash
- docker exec litellm litellm_manage --showkey
- ```
- <details>
- <summary>Show core API keys (InferCrate, GatewayCrate, UplinkCrate)</summary>
- ```bash
- docker exec ollama ollama_manage --showkey
- docker exec litellm litellm_manage --showkey
- docker exec mcp mcp_manage --showkey
- ```
- </details>
- **Access AnythingLLM (Chat UI):**
- AnythingLLM is pre-configured to connect to your local LLM via GatewayCrate. On first start, it may take a few minutes to become available (check progress with `docker logs anythingllm`).
- **Password-protected by default.** A random admin password is auto-generated on first start, printed once to `docker logs anythingllm`, and saved to `/app/server/storage/.initial_admin_password` inside the `anythingllm-data` volume. The seeded password persists across container upgrades. Change it any time from **Settings → Security**; after you do, `.initial_admin_password` may no longer match the current login password.
- Retrieve the auto-generated password:
- ```bash
- # At any time from the data volume:
- docker exec anythingllm cat /app/server/storage/.initial_admin_password
- # Or from the live logs (only shown on first start):
- docker compose logs anythingllm | grep -A4 "FIRST RUN"
- ```
- Open `http://<server-ip>:3001` in your browser and log in with the password above.
- > [!NOTE]
- > When exposing AnythingLLM beyond `localhost` or a trusted LAN, use the included Caddy HTTPS overlay so the password is encrypted in transit and direct HTTP ports are bound to localhost. See [Internet-facing deployments](#internet-facing-deployments) below.
- **Access GatewayCrate's LiteLLM Admin UI:**
- Open `http://<server-ip>:4000/ui` in your browser. Log in with username `admin` and your GatewayCrate master key as the password. The UI provides virtual key management, spend tracking, and model configuration.
- > **Tip:** In the Admin UI, click **Playground** in the left menu. Select a local model (e.g., `ollama-chat/llama3.2:3b`) from the dropdown and start chatting — a quick way to verify your local LLM is working end-to-end.
- **Use the stack with goose:**
- Install goose on your workstation and connect it to the stack's GatewayCrate endpoint for supervised agent and coding tasks. The [goose setup guide](https://selfhostedaistack.com/goose) covers native installation, a restricted GatewayCrate key, local-model expectations, permissions, testing, and optional UplinkCrate access.
- **Stop the stack:**
- ```bash
- # Stop and remove all containers (data is preserved in Docker volumes)
- docker compose down
- ```
- ## Community
- - 📬 [Get project updates and free deployment guides](https://selfhostedstack.beehiiv.com/subscribe?utm_campaign=ai) (1–2 emails/month)
- - 💬 Join the [r/selfhostedstack](https://www.reddit.com/r/selfhostedstack/) community for discussions and showcases
- - ⭐ Star the repository if you find it useful — it helps others discover it
- Self-Hosted AI Stack is maintained by the author of [Setup IPsec VPN](https://github.com/hwdsl2/setup-ipsec-vpn) (28k+ stars).
- ## GPU acceleration (NVIDIA CUDA)
- For NVIDIA GPU acceleration, use the CUDA compose file:
- ```bash
- docker compose -f docker-compose.cuda.yml up -d
- ```
- > **Tip:** To avoid adding `-f docker-compose.cuda.yml` to every subsequent `docker compose` command (`down`, `pull`, `logs`, etc.), set it once for your shell session:
- >
- > ```bash
- > export COMPOSE_FILE=docker-compose.cuda.yml
- > ```
- >
- > Then run plain `docker compose` commands as usual. To make it persistent, add `COMPOSE_FILE=docker-compose.cuda.yml` to a `.env` file in this directory. Run `unset COMPOSE_FILE` to switch back to the CPU configuration.
- **Requirements:** NVIDIA GPU, [NVIDIA driver](https://www.nvidia.com/en-us/drivers/) 575.57.08+ (Linux) or 576.57+ (Windows), and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed on the host. CUDA images are `linux/amd64` only.
- > [!IMPORTANT]
- > The Compose `deploy:` GPU block is ignored by Podman. Use CDI instead — see [Using Podman](#using-podman).
- ## Lightweight stacks
- Don't need the full stack? Use a pre-configured subset from the `stacks/` folder:
- > [!IMPORTANT]
- > The lightweight stacks share default container names, ports, and Docker volume names. Run one stack variant at a time with the default compose files; stop the current variant before switching to another. To combine capabilities, use the full stack or customize Compose project names, container names, ports, and volumes.
- | Stack | Services | Memory | Use case |
- |---|---|---|---|
- | **[chat-ui](stacks/chat-ui/)** | InferCrate + GatewayCrate + AnythingLLM | ~5 GB | Web-based ChatGPT-like chat interface |
- | **[voice-pipeline](stacks/voice-pipeline/)** | ScribeCrate + InferCrate + GatewayCrate + SpeakCrate | ~6 GB | Speech-to-text → LLM → text-to-speech |
- | **[voice-chat](stacks/voice-chat/)** | ScribeCrate + InferCrate + GatewayCrate + SpeakCrate + AnythingLLM | ~6.5 GB | Chat UI with voice input/output |
- | **[rag-pipeline](stacks/rag-pipeline/)** | InferCrate + GatewayCrate + EmbedCrate | ~5 GB | Semantic search + LLM Q&A |
- | **[rag-pipeline-full](stacks/rag-pipeline-full/)** | InferCrate + GatewayCrate + EmbedCrate + ParseCrate | ~6 GB | Document parsing + semantic search + LLM Q&A |
- | **[code-assistant](stacks/code-assistant/)** | InferCrate + GatewayCrate + UplinkCrate + EmbedCrate | ~5 GB | AI coding with tools + semantic code search |
- | **[ai-tools](stacks/ai-tools/)** | InferCrate + GatewayCrate + UplinkCrate | ~5 GB | AI coding assistant with tool access |
- | **[chat-only](stacks/chat-only/)** | InferCrate + GatewayCrate | ~4.5 GB | Minimal local ChatGPT replacement |
- ```bash
- git clone https://github.com/hwdsl2/self-hosted-ai-stack
- cd self-hosted-ai-stack/stacks/chat-ui # or voice-pipeline, voice-chat, rag-pipeline, rag-pipeline-full, code-assistant, ai-tools, chat-only
- docker compose up -d
- ```
- ## Architecture
- ```mermaid
- graph LR
- A["🎤 Audio input"] -->|transcribe| W["ScribeCrate<br/>(speech-to-text)"]
- D["📄 Documents"] -->|parse| DC["ParseCrate<br/>(document → text)"]
- DC -->|embed| E["EmbedCrate<br/>(text → vectors)"]
- E -->|store| VDB["pgvector<br/>(in shared Postgres)"]
- W -->|query| E
- VDB -->|context| L["GatewayCrate<br/>(AI gateway)"]
- W -->|text| L
- L -->|routes to| O["InferCrate<br/>(local LLM)"]
- L -->|response| T["SpeakCrate TTS<br/>(text-to-speech)"]
- T --> B["🔊 Audio output"]
- C["🤖 AI client<br/>(goose, Cline, Claude, etc.)"] -->|MCP tools| M["UplinkCrate<br/>(MCP endpoint)"]
- C -->|chat| L
- L -->|MCP protocol| M
- U["👤 User"] -->|chat| AN["AnythingLLM<br/>(chat UI)"]
- AN -->|LLM requests| L
- AN -->|MCP tools| M
- U -->|use| C
- U -->|speak| A
- U -->|upload| D
- ```
- **Notes:**
- - InferCrate's port (`11434`) and UplinkCrate's port (`3000`) are internal to the Docker network and not exposed to the host by default. Access your LLM through GatewayCrate on port `4000`.
- - SpeakCrate (TTS), ParseCrate (document parsing), and ScribeCrate Live (real-time STT) are disabled by default to reduce memory usage. Uncomment these services in `docker-compose.yml` to enable them.
- ## Running without Docker Compose
- If you prefer using `docker run` commands directly, first create a shared network so services can communicate:
- ```bash
- docker network create ai-stack
- ```
- Then generate a PostgreSQL password and start each service on the shared network:
- > **Note:** With manual `docker run`, start PostgreSQL first, then the model and tool services included in your stack. Wait for dependencies to be ready before starting GatewayCrate, then start AnythingLLM if included. The examples below generate one PostgreSQL password variable and reuse it for Postgres and GatewayCrate.
- ```bash
- LITELLM_POSTGRES_PASSWORD=$(LC_ALL=C tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
- # PostgreSQL with pgvector (required by GatewayCrate; pgvector enables vector storage for RAG)
- docker run -d --name litellm-db --restart always \
- --network ai-stack \
- -e POSTGRES_USER=litellm \
- -e POSTGRES_PASSWORD="$LITELLM_POSTGRES_PASSWORD" \
- -e POSTGRES_DB=litellm \
- -v litellm-db:/var/lib/postgresql \
- pgvector/pgvector:pg18-trixie
- # InferCrate (LLM)
- docker run -d --name ollama --restart always \
- --network ai-stack \
- -v ollama-data:/var/lib/ollama \
- -v ollama-shared:/var/lib/ollama-shared \
- hwdsl2/ollama-server
- # UplinkCrate
- docker run -d --name mcp --restart always \
- --network ai-stack \
- -v mcp-data:/var/lib/mcp \
- -v mcp-shared:/var/lib/mcp-shared \
- hwdsl2/mcp-gateway
- # GatewayCrate (AI gateway)
- docker run -d --name litellm --restart always \
- --network ai-stack \
- -p 4000:4000 \
- -e LITELLM_OLLAMA_BASE_URL=http://ollama:11434 \
- -e LITELLM_MCP_URL=http://mcp:3000/mcp \
- -e LITELLM_DATABASE_URL="postgresql://litellm:${LITELLM_POSTGRES_PASSWORD}@litellm-db:5432/litellm" \
- -v litellm-data:/etc/litellm \
- -v ollama-shared:/var/lib/ollama-shared:ro \
- -v mcp-shared:/var/lib/mcp-shared:ro \
- -v litellm-shared:/var/lib/litellm-shared \
- hwdsl2/litellm-server
- # EmbedCrate
- docker run -d --name embeddings --restart always \
- --network ai-stack \
- -p 127.0.0.1:8000:8000 \
- -v embeddings-data:/var/lib/embeddings \
- hwdsl2/embeddings-server
- # ScribeCrate (Whisper STT)
- docker run -d --name whisper --restart always \
- --network ai-stack \
- -p 127.0.0.1:9000:9000 \
- -v whisper-data:/var/lib/whisper \
- hwdsl2/whisper-server
- # ScribeCrate Live (real-time STT)
- docker run -d --name whisper-live --restart always \
- --network ai-stack \
- -p 127.0.0.1:9090:9090 \
- -v whisper-live-data:/var/lib/whisper-live \
- hwdsl2/whisper-live-server
- # AnythingLLM (chat UI)
- docker run -d --name anythingllm --restart always \
- --network ai-stack \
- -p 3001:3001 \
- -e STORAGE_DIR=/app/server/storage \
- -e LLM_PROVIDER=generic-openai \
- -e GENERIC_OPEN_AI_BASE_PATH=http://litellm:4000/v1 \
- -e GENERIC_OPEN_AI_MODEL_PREF=ollama/llama3.2:3b \
- -e GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=131072 \
- -e ANYTHINGLLM_DEFAULT_CHAT_MODE=chat \
- -e EMBEDDING_ENGINE=native \
- -e DISABLE_TELEMETRY=true \
- -v anythingllm-data:/app/server/storage \
- -v litellm-shared:/var/lib/litellm-shared:ro \
- -v "$(pwd)/chat-ui-bootstrap.sh:/usr/local/bin/chat-ui-bootstrap.sh:ro" \
- --entrypoint /bin/bash \
- mintplexlabs/anythingllm:1.18.0 \
- /usr/local/bin/chat-ui-bootstrap.sh
- # SpeakCrate (TTS)
- docker run -d --name kokoro --restart always \
- --network ai-stack \
- -p 127.0.0.1:8880:8880 \
- -v kokoro-data:/var/lib/kokoro \
- hwdsl2/kokoro-server
- # ParseCrate (document parsing)
- docker run -d --name docling --restart always \
- --network ai-stack \
- -p 127.0.0.1:5001:5001 \
- -v docling-data:/var/lib/docling \
- hwdsl2/docling-server
- ```
- **Note:** The shared network allows services to reach each other by container name (e.g., GatewayCrate connects to InferCrate via `http://ollama:11434`). You can start only the services you need — they don't all have to run together.
- **Pull a model** (required before making LLM requests):
- ```bash
- docker exec ollama ollama_manage --pull llama3.2:3b
- ```
- ## Using Podman
- The stack runs under [Podman](https://podman.io/) on a best-effort basis. The CPU compose files work as-is; GPU acceleration and SELinux-enabled hosts need a couple of extra steps described below. Podman **4.1+** is recommended.
- **1. Install the Docker CLI shim.** So that the `docker` commands in this README and the `stack-check.sh` health check work unchanged, install the `podman-docker` package (provides a `docker` → `podman` wrapper):
- ```bash
- # Fedora / RHEL / CentOS Stream
- sudo dnf install -y podman-docker
- # Debian / Ubuntu
- sudo apt-get install -y podman-docker
- ```
- > [!NOTE]
- > A shell `alias docker=podman` is **not** sufficient — aliases are not seen by scripts such as `stack-check.sh`. Use the `podman-docker` package (or a `docker` → `podman` symlink in your `PATH`) instead. Alternatively, `stack-check.sh` auto-detects Podman; you can also force it with `CONTAINER_ENGINE=podman ./stack-check.sh`.
- **2. Install a Compose provider.** `podman compose` delegates to an external provider. Install either `podman-compose` or `docker-compose`:
- ```bash
- # Fedora / RHEL / CentOS Stream
- sudo dnf install -y podman-compose
- # Debian / Ubuntu
- sudo apt-get install -y podman-compose
- ```
- **3. Start the stack.** With the shim installed, every command in this README works unchanged. Without it, substitute `podman` for `docker`:
- ```bash
- git clone https://github.com/hwdsl2/self-hosted-ai-stack
- cd self-hosted-ai-stack
- podman compose up -d
- ```
- Run the health check (auto-detects the engine):
- ```bash
- ./stack-check.sh
- ```
- **GPU acceleration (CDI).** Podman does not read the Compose `deploy.resources` GPU block. Instead, use the [Container Device Interface (CDI)](https://github.com/cncf-tags/container-device-interface). After installing the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html), generate a CDI spec:
- ```bash
- sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
- ```
- Then expose the GPU to the relevant services. For `podman compose`, replace the `deploy:` block in `docker-compose.cuda.yml` with a `devices:` entry for the `ollama` (and `whisper`) services:
- ```yaml
- devices:
- - nvidia.com/gpu=all
- ```
- For a plain `podman run` command, add `--device nvidia.com/gpu=all`.
- **SELinux.** On SELinux-enabled hosts (Fedora, RHEL, CentOS Stream), bind-mounted files need a relabel suffix, or the container will be denied access. Add `:z` (shared) to the `chat-ui-bootstrap.sh` bind mount:
- - In `docker-compose.yml`: change `./chat-ui-bootstrap.sh:/usr/local/bin/chat-ui-bootstrap.sh:ro` to `./chat-ui-bootstrap.sh:/usr/local/bin/chat-ui-bootstrap.sh:ro,z`
- - In the `podman run` command above: change `"$(pwd)/chat-ui-bootstrap.sh:/usr/local/bin/chat-ui-bootstrap.sh:ro"` to `"$(pwd)/chat-ui-bootstrap.sh:/usr/local/bin/chat-ui-bootstrap.sh:ro,z"`
- Named volumes do not need relabeling.
- **Next steps:** Pull a model and access the services — follow the instructions in [Quick start](#quick-start) starting from "Pull a model." With the `podman-docker` shim installed, all commands work unchanged.
- ## Connect UplinkCrate to GatewayCrate
- GatewayCrate and UplinkCrate are **automatically wired** when using the compose files in this repository — no manual key setup is needed.
- API keys are shared automatically between services via Docker shared volumes:
- - InferCrate generates an API key on first start and copies it to a shared volume
- - UplinkCrate does the same
- - GatewayCrate reads both keys from the shared volumes on startup
- The `LITELLM_MCP_URL=http://mcp:3000/mcp` and `LITELLM_OLLAMA_BASE_URL=http://ollama:11434` environment variables are pre-configured in the compose files, so all services are connected automatically with a single `docker compose up -d`.
- Once connected, AI clients that call GatewayCrate can use MCP tools (filesystem, fetch, GitHub, etc.) directly through the GatewayCrate proxy.
- ## Voice pipeline example
- Transcribe a spoken question, get a local LLM response via InferCrate, and convert it to speech:
- **Note:** SpeakCrate (TTS) is disabled by default. To use this example, first uncomment the `kokoro` service in `docker-compose.yml`, then run `docker compose up -d`.
- **Tip:** Need a sample audio file? Download this English speech sample (WAV, MIT License) from the [Azure Samples](https://github.com/Azure-Samples/cognitive-services-speech-sdk) repository:
- ```bash
- curl -L -o sample_speech.wav \
- "https://github.com/Azure-Samples/cognitive-services-speech-sdk/raw/master/sampledata/audiofiles/katiesteve.wav"
- ```
- ```bash
- gateway_master_key="$(docker exec litellm litellm_manage --getkey)"
- scribe_api_key="$(docker exec whisper whisper_manage --getkey)"
- speak_api_key="$(docker exec kokoro kokoro_manage --getkey)"
- # Step 1: Transcribe audio to text (ScribeCrate)
- TEXT=$(curl -s http://localhost:9000/v1/audio/transcriptions \
- -H "Authorization: Bearer $scribe_api_key" \
- -F file=@sample_speech.wav -F model=whisper-1 | jq -r .text)
- # Step 2: Send text to InferCrate via GatewayCrate and get a response
- RESPONSE=$(curl -s http://localhost:4000/v1/chat/completions \
- -H "Authorization: Bearer $gateway_master_key" \
- -H "Content-Type: application/json" \
- -d "{\"model\":\"ollama/llama3.2:3b\",\"messages\":[{\"role\":\"user\",\"content\":\"$TEXT\"}]}" \
- | jq -r '.choices[0].message.content')
- # Step 3: Convert the response to speech (SpeakCrate TTS)
- curl -s http://localhost:8880/v1/audio/speech \
- -H "Content-Type: application/json" \
- -H "Authorization: Bearer $speak_api_key" \
- -d "{\"model\":\"tts-1\",\"input\":\"$RESPONSE\",\"voice\":\"af_heart\"}" \
- --output response.mp3
- ```
- ## Vector database
- The stack's PostgreSQL ships with the [pgvector](https://github.com/pgvector/pgvector) extension, so you can store and query embeddings in the same database that GatewayCrate uses — no separate vector database required.
- Enable the extension once (the database persists, so this only needs to be done a single time):
- ```bash
- docker exec litellm-db psql -U litellm -d litellm -c 'CREATE EXTENSION IF NOT EXISTS vector;'
- ```
- Verify it is enabled:
- ```bash
- docker exec litellm-db psql -U litellm -d litellm -c "SELECT extname, extversion FROM pg_extension WHERE extname='vector';"
- ```
- You can then create a table with a `vector` column (use the dimension of your embedding model — e.g. `384` for the default `BAAI/bge-small-en-v1.5`) and run similarity search with the `<=>` operator. For larger-scale or hybrid search, you can run a dedicated vector database such as Qdrant or Chroma instead.
- ## RAG pipeline example
- Embed documents for semantic search, retrieve context, then answer questions with a local InferCrate model:
- ```bash
- gateway_master_key="$(docker exec litellm litellm_manage --getkey)"
- embed_api_key="$(docker exec embeddings embed_manage --getkey)"
- # Step 1: Embed a document chunk and store the vector in your vector DB
- curl -s http://localhost:8000/v1/embeddings \
- -H "Content-Type: application/json" \
- -H "Authorization: Bearer $embed_api_key" \
- -d '{"input": "Docker simplifies deployment by packaging apps in containers.", "model": "text-embedding-ada-002"}' \
- | jq '.data[0].embedding'
- # → Store the returned vector alongside the source text in pgvector (included in the stack's Postgres), or another vector DB such as Qdrant or Chroma.
- # Step 2: At query time, embed the question, retrieve the top matching chunks from
- # the vector DB, then send the question and retrieved context to InferCrate via GatewayCrate.
- curl -s http://localhost:4000/v1/chat/completions \
- -H "Authorization: Bearer $gateway_master_key" \
- -H "Content-Type: application/json" \
- -d '{
- "model": "ollama/llama3.2:3b",
- "messages": [
- {"role": "system", "content": "Answer using only the provided context."},
- {"role": "user", "content": "What does Docker do?\n\nContext: Docker simplifies deployment by packaging apps in containers."}
- ]
- }' \
- | jq -r '.choices[0].message.content'
- ```
- ## MCP tools example
- Use UplinkCrate to give your AI assistant access to files, web, and GitHub:
- UplinkCrate is internal to the Docker network by default. Before using `http://localhost:3000/mcp` from a host-side AI client or host-side `curl`, uncomment the `3000:3000/tcp` port mapping for the `mcp` service in `docker-compose.yml` and restart it.
- ```bash
- uplink_api_key="$(docker exec mcp mcp_manage --getkey)"
- # Use MCP endpoint with an AI client (e.g., Cline in VS Code)
- # Set the MCP server URL: http://localhost:3000/mcp
- # Set Authorization header: Bearer <api_key>
- # Or test the MCP endpoint directly with an initialize request
- curl -s http://localhost:3000/mcp \
- -X POST \
- -H "Authorization: Bearer $uplink_api_key" \
- -H "Content-Type: application/json" \
- -H "Accept: application/json, text/event-stream" \
- -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}'
- ```
- ## Usage counts
- Self-Hosted AI Stack uses anonymous, aggregate GitHub release asset download counts to help understand usage and prioritize future improvements. It does not send a telemetry payload or use a private collector.
- To disable usage counts when starting the stack:
- ```bash
- AI_STACK_DISABLE_USAGE_COUNTS=1 docker compose up -d
- ```
- ## Customization
- Each service can be configured with an optional env file. Copy the example env file from the respective repository, edit it, and uncomment the volume mount in `docker-compose.yml`:
- | Service | Env file | Repository |
- |---|---|---|
- | InferCrate | `ollama.env` | [infercrate](https://github.com/hwdsl2/infercrate) |
- | GatewayCrate | `litellm.env` | [gatewaycrate](https://github.com/hwdsl2/gatewaycrate) |
- | EmbedCrate | `embed.env` | [embedcrate](https://github.com/hwdsl2/embedcrate) |
- | ScribeCrate | `whisper.env` | [ScribeCrate](https://github.com/hwdsl2/scribecrate) |
- | ScribeCrate Live | `whisper-live.env` | [ScribeCrate Live](https://github.com/hwdsl2/scribecrate-live) |
- | SpeakCrate | `kokoro.env` | [SpeakCrate](https://github.com/hwdsl2/speakcrate) |
- | UplinkCrate | `mcp.env` | [uplinkcrate](https://github.com/hwdsl2/uplinkcrate) |
- | ParseCrate | `docling.env` | [ParseCrate](https://github.com/hwdsl2/parsecrate) |
- AnythingLLM is configured through its web UI at `http://<server-ip>:3001`. You can change the LLM provider, model, embedding engine, and other settings in **Settings**. See [AnythingLLM docs](https://docs.useanything.com/) for more details.
- **Use the stack's EmbedCrate service (optional).** By default AnythingLLM embeds documents in-process with its bundled MiniLM model and stores the vectors in its own LanceDB. To use the stack's [EmbedCrate (Hugging Face TEI)](https://github.com/hwdsl2/embedcrate) service (BAAI/bge-small-en-v1.5) and/or the stack's pgvector-enabled Postgres instead, edit the `anythingllm` service in `docker-compose.yml`: comment out `EMBEDDING_ENGINE=native` and uncomment the opt-in block beneath it. Also uncomment the `depends_on` note so the embeddings/db services start first.
- When `VECTOR_DB=pgvector` is enabled and no `PGVECTOR_CONNECTION_STRING` is set, AnythingLLM uses the generated Postgres password from `ai-stack-shared` automatically. AnythingLLM auto-creates the `vector` extension and `anythingllm_vectors` table on first use.
- > [!CAUTION]
- > Switching the embedder or vector store on an existing deployment makes previously embedded documents incompatible — re-embed your workspaces after the change.
- For detailed configuration options, API reference, and model management, see the documentation in each service's repository.
- ## Optional secret files
- Services can read credentials from mounted files. This is optional; the default generated and shared credentials remain unchanged.
- Use a compatible image and the [override for that service](docs/examples/secrets). Apply it only to a service already in your stack, remove matching direct credential settings (including empty values), and keep your existing volumes.
- Review the setup, rotation and rollback instructions below before changing credential sources.
- <details>
- <summary>Setup, rotation and rollback</summary>
- **Setup**
- Use a compatible published image, or optionally build one from the matching service revision. Older images ignore the new file inputs. The example below uses a local ScribeCrate image; see its [build instructions](https://github.com/hwdsl2/scribecrate#secret-files). Set `<SERVICE>_SECRET_IMAGE` and `<SERVICE>_SECRET_PATH` for the selected service (for example, `WHISPER_SECRET_IMAGE` and `WHISPER_SECRET_PATH`).
- Choose only an override for a service already present in your stack. Keep your existing project name, base CPU/CUDA file, proxy overlays, env-file arguments and volumes. Remove the corresponding direct variable everywhere, including an empty value in a mounted service env file. For example, from the repository root:
- ```bash
- export WHISPER_SECRET_IMAGE=scribecrate:secret-files-candidate
- export WHISPER_SECRET_PATH=/absolute/path/to/api-key
- docker compose -f docker-compose.yml -f docs/examples/secrets/whisper.yml config
- ```
- After reviewing `config`, start only the affected service with the same file set. For a service that uses a CUDA image in your stack, select a matching CUDA candidate. For a prebuilt stack, use its existing base file and an absolute path to the selected override. With HTTPS, append the secret override after the existing proxy overlay. Do not apply overlays for absent services: Compose would introduce those services. Live remains disabled/commented unless deliberately configured separately.
- **Credentials and access**
- **Note:** If a direct credential comes from Compose `environment` or `env_file`, remove it there and recreate only the affected service, keeping its volumes. Service documentation describes the `unset` workaround for shell-sourced bind-mounted env files. Docker `--env-file` and Compose `env_file` do not execute shell commands.
- The overrides work with the existing root/prebuilt CPU/CUDA/proxy configurations. The existing PostgreSQL-password file, `ai-stack-init`, shared-key mounts, ports and optional-service choices are retained.
- Files must be readable by the container user (ParseCrate uses UID 1001). Service docs describe all supported secondary inputs, source validation and persistence: [ScribeCrate](https://github.com/hwdsl2/scribecrate#secret-files), [ParseCrate](https://github.com/hwdsl2/parsecrate#secret-files), [SpeakCrate](https://github.com/hwdsl2/speakcrate#secret-files), [Live](https://github.com/hwdsl2/scribecrate-live#secret-files), [EmbedCrate](https://github.com/hwdsl2/embedcrate#secret-files), [InferCrate](https://github.com/hwdsl2/infercrate#secret-files), [UplinkCrate](https://github.com/hwdsl2/uplinkcrate#secret-files), [GatewayCrate](https://github.com/hwdsl2/gatewaycrate#secret-files). Add secondary file inputs only to their owning service, with distinct secret definitions and mounts.
- **Rotation, backups and rollback**
- Rotation takes effect at startup. Make new bytes visible, restart/recreate only affected producers as needed, then refresh consumers that cache shared credentials. Existing AnythingLLM credentials and saved provider/admin/database settings are not automatically rotated. Atomic replacement of host bind-mounted files may leave an old inode visible until recreation. Preserve volumes and back up related database/credential state together. Before rollback, restore an old-image-supported direct/saved credential source and verify protected requests; file-only configuration is not a safe old-image rollback target.
- **Compatibility**
- Stack static and Compose-configuration checks passed. Related service images passed their own build, authentication and standalone Docker Compose secret-mount checks. Full-stack runtime, Podman/rootless operation, Podman Compose, Swarm, Kubernetes and GPU acceleration were not tested.
- </details>
- ## Internet-facing deployments
- By default, all services listen over plain HTTP. For internet-facing deployments, use the included Caddy overlay to add automatic HTTPS. In proxy mode, Caddy is the only public listener on ports `80` and `443`; the direct AnythingLLM and GatewayCrate ports are rebound to `127.0.0.1`.
- Prerequisites:
- - Docker Compose `2.24.4+` (required for the proxy overlay's port override)
- - A DNS `A`/`AAAA` record for your domain pointing to this server
- - Inbound `80/tcp`, `443/tcp`, and ideally `443/udp` open in your firewall/security group
- - No other service already using ports `80` or `443` on the host
- **CPU stack:**
- ```bash
- DOMAIN=chat.example.com ACME_EMAIL=you@example.com \
- docker compose -f docker-compose.yml -f docker-compose.proxy.yml up -d
- ```
- **CUDA stack:**
- ```bash
- DOMAIN=chat.example.com ACME_EMAIL=you@example.com \
- docker compose -f docker-compose.cuda.yml -f docker-compose.proxy.yml up -d
- ```
- Open `https://chat.example.com` (replace with your `DOMAIN`) to access AnythingLLM. In proxy mode, `http://127.0.0.1:3001` and `http://127.0.0.1:4000/ui` remain available on the host, but the direct `3001` and `4000` ports are not reachable from outside the server.
- The standard compose files publish GatewayCrate on port `4000`. The proxy overlay changes that direct port to localhost-only, and the included Caddyfile routes only AnythingLLM by default. Uncommenting the optional GatewayCrate hostname block exposes GatewayCrate through Caddy, so keep the GatewayCrate master key secret.
- Troubleshooting:
- ```bash
- docker logs ai-stack-caddy
- # Use the same -f files you used to start the stack
- docker compose -f docker-compose.yml -f docker-compose.proxy.yml ps
- ```
- If Caddy reports an unknown `request_body` directive, pull the current `caddy:2` image and restart the overlay.
- For older Docker Compose versions or Podman, use a host-based reverse proxy instead: bind direct HTTP ports to localhost in the compose file (for example, `"127.0.0.1:3001:3001/tcp"` and `"127.0.0.1:4000:4000/tcp"`) and proxy to those localhost ports. For stack-specific Caddy and nginx examples, see the [Chat UI manual reverse proxy section](stacks/chat-ui/#manual-reverse-proxy).
- When exposing services to the internet, use the generated API keys where present. For existing no-key deployments, set API keys via the relevant env files before publishing those services.
- ## Backup and restore
- Your API keys, models, and configuration are stored in Docker volumes. Back up before upgrading or making changes:
- ```bash
- # Export API keys (while containers are running)
- docker exec ollama ollama_manage --getkey
- docker exec litellm litellm_manage --getkey
- docker exec mcp mcp_manage --getkey
- # Optional services; ignored if the container is not enabled/running
- docker exec whisper whisper_manage --getkey 2>/dev/null || true
- docker exec whisper-live whisper_live_manage --getkey 2>/dev/null || true
- docker exec kokoro kokoro_manage --getkey 2>/dev/null || true
- docker exec embeddings embed_manage --getkey 2>/dev/null || true
- docker exec docling docling_manage --getkey 2>/dev/null || true
- # Back up all volumes (stop services first)
- # Stop and remove all containers (data is preserved in Docker volumes)
- docker compose down
- mkdir -p backups
- for vol in ollama-data litellm-data litellm-db ai-stack-shared embeddings-data whisper-data whisper-live-data kokoro-data mcp-data docling-data anythingllm-data caddy-data caddy-config; do
- docker volume inspect "$vol" >/dev/null 2>&1 && \
- docker run --rm -v "${vol}:/source:ro" -v "$(pwd)/backups:/backup" \
- alpine tar czf "/backup/${vol}.tar.gz" -C /source .
- done
- ```
- > [!IMPORTANT]
- > Back up `ai-stack-shared` with `litellm-db`; fresh installs store the generated PostgreSQL password there. The `ollama-shared`, `mcp-shared`, and `litellm-shared` volumes are ephemeral key-sharing volumes and do not need to be backed up.
- For restore instructions, server migration, and the full pre-upgrade checklist, see the [Backup and Restore](docs/backup-restore.md) guide.
- ## PostgreSQL credentials
- Fresh Docker Compose installs generate a random PostgreSQL password automatically and store it in the `ai-stack-shared` volume. Existing default installs continue to use the legacy `litellm` database password for compatibility.
- If you previously customized the database password, set `LITELLM_POSTGRES_PASSWORD` in your shell environment to that current password before running `docker compose up -d`, or keep an explicit `LITELLM_DATABASE_URL` override in `litellm.env`.
- ## Update images
- To update all services to the latest versions:
- ```bash
- git pull
- docker compose pull
- docker compose up -d
- ./stack-check.sh
- ```
- After the stack restarts, run `./stack-check.sh` to confirm the services and generated credential wiring are healthy.
- `git pull` updates all project files (including any changes to compose files); `docker compose pull` updates the service images. If you've customized `docker-compose.yml`, `git pull` will merge changes automatically, or prompt you to resolve conflicts on the same lines.
- AnythingLLM is pinned to a stable release tag instead of `latest` because the upstream `latest` image tracks the master branch. When a newer AnythingLLM release is available, back up first, update the tag in the compose files, then run the commands above.
- Your data is preserved in the Docker volumes. **Always [back up](#backup-and-restore) before upgrading.**
- ## License
- Copyright (C) 2026 Lin Song
- This work is licensed under the [MIT License](https://opensource.org/licenses/MIT).
- This project is an independent Docker configuration and is not affiliated with, endorsed by, or sponsored by Docker, Inc., Ollama, Berri AI (LiteLLM), Hugging Face, hexgrad (Kokoro), OpenAI, SYSTRAN, or MCPHub. Docker is a trademark or registered trademark of Docker, Inc.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.