Skip to content
Back to search

llm-server-docs

End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/varunvasudeva1--llm-server-docs/badge.svg)](/listings/varunvasudeva1--llm-server-docs)

HTML

<a href="/listings/varunvasudeva1--llm-server-docs"><img src="/api/listings/varunvasudeva1--llm-server-docs/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 1,772 lines · 1 finding marked

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Local LLaMA Server Setup Documentation
  2. _TL;DR_: End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS, along with steps for configuring SSH, firewall, and secure remote access via Tailscale.
  3. Software Stack:
  4. - Inference Engine ([Ollama](https://github.com/ollama/ollama), [llama.cpp](https://github.com/ggml-org/llama.cpp), [vLLM](https://github.com/vllm-project/vllm))
  5. - Search Engine ([SearXNG](https://github.com/searxng/searxng))
  6. - Model Server ([llama-swap](https://github.com/mostlygeek/llama-swap), `systemd` service)
  7. - Chat Platform ([Open WebUI](https://github.com/open-webui/open-webui))
  8. - MCP Proxy Server ([mcp-proxy](https://github.com/sparfenyuk/mcp-proxy), [MCPJungle](https://github.com/mcpjungle/MCPJungle))
  9. - Text-to-Speech Server ([Kokoro FastAPI](https://github.com/remsky/Kokoro-FastAPI))
  10. - Image Generation Server ([ComfyUI](https://github.com/comfyanonymous/ComfyUI))
  11. - VPN/Remote Access ([Tailscale](https://tailscale.com))
  12. - Reverse Proxy ([Caddy](https://github.com/caddyserver/caddy))
  13. - DNS Provider ([Cloudflare](https://www.cloudflare.com))
  14. ![Software Stack Architectural Diagram](media/llm-server-architecture.png)
  15. ## Table of Contents
  16. - [Local LLaMA Server Setup Documentation](#local-llama-server-setup-documentation)
  17. - [Table of Contents](#table-of-contents)
  18. - [About](#about)
  19. - [Priorities](#priorities)
  20. - [Prerequisites](#prerequisites)
  21. - [General](#general)
  22. - [Allow `sudo` Permissions](#allow-sudo-permissions)
  23. - [Update System Packages](#update-system-packages)
  24. - [Schedule Startup Script](#schedule-startup-script)
  25. - [Configure Script Permissions](#configure-script-permissions)
  26. - [Configure Auto-Login (optional)](#configure-auto-login-optional)
  27. - [Docker](#docker)
  28. - [Add User to Docker Group](#add-user-to-docker-group)
  29. - [Nvidia Container Toolkit](#nvidia-container-toolkit)
  30. - [Create a Network](#create-a-network)
  31. - [Harden Docker Containers](#harden-docker-containers)
  32. - [Helpful Commands](#helpful-commands)
  33. - [HuggingFace CLI](#huggingface-cli)
  34. - [Manage Models](#manage-models)
  35. - [Download Models](#download-models)
  36. - [Delete Models](#delete-models)
  37. - [Search Engine](#search-engine)
  38. - [SearXNG](#searxng)
  39. - [Open WebUI Integration](#open-webui-integration)
  40. - [Inference Engine](#inference-engine)
  41. - [Ollama](#ollama)
  42. - [llama.cpp](#llamacpp)
  43. - [vLLM](#vllm)
  44. - [Model Server](#model-server)
  45. - [llama-swap](#llama-swap)
  46. - [`systemd` Service](#systemd-service)
  47. - [Open WebUI Integration](#open-webui-integration-1)
  48. - [llama-swap](#llama-swap-1)
  49. - [`systemd` Service](#systemd-service-1)
  50. - [Chat Platform](#chat-platform)
  51. - [Open WebUI](#open-webui)
  52. - [MCP Proxy Server](#mcp-proxy-server)
  53. - [mcp-proxy](#mcp-proxy)
  54. - [MCPJungle](#mcpjungle)
  55. - [Comparison](#comparison)
  56. - [Open WebUI Integration](#open-webui-integration-2)
  57. - [mcp-proxy](#mcp-proxy-1)
  58. - [MCPJungle](#mcpjungle-1)
  59. - [VS Code/Claude Desktop Integration](#vs-codeclaude-desktop-integration)
  60. - [Text-to-Speech Server](#text-to-speech-server)
  61. - [Kokoro FastAPI](#kokoro-fastapi)
  62. - [Open WebUI Integration](#open-webui-integration-3)
  63. - [Image Generation Server](#image-generation-server)
  64. - [ComfyUI](#comfyui)
  65. - [Open WebUI Integration](#open-webui-integration-4)
  66. - [SSH](#ssh)
  67. - [Firewall](#firewall)
  68. - [Remote Access](#remote-access)
  69. - [Updating](#updating)
  70. - [General](#general-1)
  71. - [Nvidia Drivers \& CUDA](#nvidia-drivers--cuda)
  72. - [Docker Services](#docker-services)
  73. - [Ollama](#ollama-1)
  74. - [llama.cpp](#llamacpp-1)
  75. - [vLLM](#vllm-1)
  76. - [ComfyUI](#comfyui-1)
  77. - [Troubleshooting](#troubleshooting)
  78. - [SearXNG](#searxng-1)
  79. - [Docker](#docker-1)
  80. - [`ssh`](#ssh-1)
  81. - [Nvidia Drivers](#nvidia-drivers)
  82. - [Ollama](#ollama-2)
  83. - [vLLM](#vllm-2)
  84. - [Open WebUI](#open-webui-1)
  85. - [Monitoring](#monitoring)
  86. - [Notes](#notes)
  87. - [Software](#software)
  88. - [Hardware](#hardware)
  89. - [References](#references)
  90. - [Acknowledgements](#acknowledgements)
  91. ## About
  92. This repository outlines the steps to run a server for running local language models. It uses Debian specifically, but most Linux distros should follow a very similar process. It aims to be a guide for Linux beginners like me who are setting up a server for the first time.
  93. The process involves installing the requisite drivers, setting the GPU power limit, setting up auto-login, and scheduling the `init.bash` script to run at boot. All these settings are based on my ideal setup for a language model server that runs most of the day but a lot can be customized to suit your needs.
  94. > [!IMPORTANT]
  95. > No part of this guide was written using AI - any hallucinations are the good old human kind. AI is used to format Markdown elements and vet for any technical inaccuracies. While I've done my absolute best to ensure correctness in every step/command, check **everything** you execute in a terminal. Enjoy!
  96. ## Priorities
  97. - **Simplicity**: It should be relatively straightforward to set up the components of the solution.
  98. - **Stability**: The components should be stable and capable of running for weeks at a time without any intervention necessary.
  99. - **Security**: The components should be able to be tightly secured and limited in their capability to damage the system in case of a known vulnerability affecting any of the components.
  100. - **Maintainability**: The components and their interactions should be uncomplicated enough that you know enough to maintain them as they evolve (because they *will* evolve).
  101. - **Aesthetics**: The result should be as close to a cloud provider's chat platform as possible. A homelab solution doesn't necessarily need to feel like it was cobbled together haphazardly.
  102. - **Modularity**: Components in the setup should be able to be swapped out for newer/more performant/better maintained alternatives easily. Standard protocols (OpenAI-compatibility, MCPs, etc.) help with this a lot and, in this guide, they are always preferred over bundled solutions.
  103. - **Open source**: The code should be able to be verified by a community of engineers. Chat platforms and LLMs involve large amounts of personal data conveyed in natural language and it's important to know that data isn't going outside your machine.
  104. ## Prerequisites
  105. Any modern CPU and GPU combination should work for this guide. Previously, compatibility with AMD GPUs was an issue but the latest releases of Ollama have worked through this and [AMD GPUs are now supported natively](https://ollama.com/blog/amd-preview).
  106. For reference, this guide was built around the following system:
  107. - **CPU**: Intel Core i5-12600KF
  108. - **Memory**: 96GB 3200MHz DDR4 RAM
  109. - **Storage**: 1TB M.2 NVMe SSD
  110. - **GPU**: 2x Nvidia RTX 3090 (24GB)
  111. > [!NOTE]
  112. > **AMD GPUs**: Power limiting is skipped for AMD GPUs as [AMD has recently made it difficult to set power limits on their GPUs](https://www.reddit.com/r/linux_gaming/comments/1b6l1tz/no_more_power_limiting_for_amd_gpus_because_it_is/). Naturally, skip any steps involving `nvidia-smi` or `nvidia-persistenced` and the power limit in the `init.bash` script.
  113. >
  114. > **CPU-only**: You can skip the GPU driver installation and power limiting steps. The rest of the guide should work as expected.
  115. > [!NOTE]
  116. > This guide uses `~/` (or `/home/<username>`) as the base directory. If you're working in different directory, please modify all your commands accordingly.
  117. To begin the process of setting up your server, you will need the following:
  118. - Fresh install of Debian
  119. - Internet connection
  120. - Basic understanding of the Linux terminal
  121. - Peripherals like a monitor, keyboard, and mouse
  122. To install Debian on your newly built server hardware:
  123. - Download the [Debian ISO](https://www.debian.org/distrib/) from the official website.
  124. - Create a bootable USB using a tool like [Rufus](https://rufus.ie/en/) for Windows or [Balena Etcher](https://etcher.balena.io) for MacOS.
  125. - Boot into the USB and install Debian.
  126. For a more detailed guide on installing Debian, refer to the [official documentation](https://www.debian.org/releases/buster/amd64/). For those who aren't yet experienced with Linux, I recommend using the graphical installer - you will be given an option between the text-based installer and graphical installer.
  127. I also recommend installing a lightweight desktop environment like XFCE for ease of use. Other options like GNOME or KDE are also available - GNOME may be a better option for those using their server as a primary workstation as it is more feature-rich (and, as such, heavier) than XFCE.
  128. ## General
  129. ### Allow `sudo` Permissions
  130. To do a bunch of things in this guide, we need to be root. But we don't have the ability to act as root until root allows our user to. First, we'll switch to root and grant our user permission to run commands with `sudo`.
  131. > [!TIP]
  132. > `sudo` stands for "superuser doer" - in Linux, it signals to the OS that you want the command you're running to be running on behalf of the root user. This should be used sparingly on a highly secure system (we want user-specific permissions for most processes) and carefully since it can affect the system with the same power as the root user. Don't overthink its use or worry when using it - just know that it **can** be dangerous if used incorrectly.
  133. - Switch to root:
  134. ```bash
  135. su root
  136. ```
  137. - Run the following command to add your user to the `sudo` group (which has the permissions we're looking for):
  138. ```bash
  139. sudo usermod -a -G sudo <username>
  140. ```
  141. > Replace `<username>` with your username.
  142. Save and exit (`Ctrl+X`).
  143. - Close your existing terminal and open a new session. This is required to see the changes.
  144. - (Optional) Test your new permissions by running `ls` (command to view the contents of a directory) with `sudo`:
  145. ```bash
  146. sudo ls
  147. ```
  148. ### Update System Packages
  149. - Update the system by running the following commands:
  150. ```
  151. sudo apt update
  152. sudo apt upgrade
  153. ```
  154. Now, we'll install the required GPU drivers that allow programs to utilize their compute capabilities.
  155. **Nvidia GPUs**
  156. - Follow Nvidia's [guide on downloading CUDA Toolkit](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Debian). The instructions are specific to your machine and the website will lead you to them interactively.
  157. - Run the following commands:
  158. ```bash
  159. sudo apt install linux-headers-amd64
  160. sudo apt install nvidia-driver firmware-misc-nonfree
  161. ```
  162. - Reboot the server.
  163. - Run the following command to verify the installation:
  164. ```bash
  165. nvidia-smi
  166. ```
  167. **AMD GPUs**
  168. - Run the following commands:
  169. ```bash
  170. deb http://deb.debian.org/debian bookworm main contrib non-free-firmware
  171. apt install firmware-amd-graphics libgl1-mesa-dri libglx-mesa0 mesa-vulkan-drivers xserver-xorg-video-all
  172. ```
  173. - Reboot the server.
  174. We'll also install some packages that are not installed on Debian by default but may be required later:
  175. ```
  176. sudo apt install libcurl cmake build-essential
  177. ```
  178. ### Schedule Startup Script
  179. In this step, we'll create a script called `init.bash`. This script will be run at boot to set the GPU power limit and start the server using Ollama. We set the GPU power limit lower because it has been seen in testing and inference that there is only a 5-15% performance decrease for a 30% reduction in power consumption. This is especially important for servers that are running 24/7.
  180. - Run the following commands:
  181. ```bash
  182. touch init.bash
  183. nano init.bash
  184. ```
  185. - Add the following lines to the script:
  186. ```bash
  187. #!/bin/bash
  188. sudo nvidia-smi -pm 1
  189. sudo nvidia-smi -pl <power_limit>
  190. ```
  191. > Replace `<power_limit>` with the desired power limit in watts. For example, `sudo nvidia-smi -pl 250`.
  192. For multiple GPUs, modify the script to set the power limit for each GPU:
  193. ```bash
  194. sudo nvidia-smi -i 0 -pl <power_limit>
  195. sudo nvidia-smi -i 1 -pl <power_limit>
  196. ```
  197. - Save and exit the script.
  198. - Make the script executable:
  199. ```bash
  200. chmod +x init.bash
  201. ```
  202. Adding the `init.bash` script to the crontab will schedule it to run at boot.
  203. - Run the following command:
  204. ```bash
  205. crontab -e
  206. ```
  207. - Add the following line to the file:
  208. ```bash
  209. @reboot /path/to/init.bash
  210. ```
  211. > Replace `/path/to/init.bash` with the path to the `init.bash` script.
  212. - (Optional) Add the following line to shutdown the server at 12am:
  213. ```bash
  214. 0 0 * * * /sbin/shutdown -h now
  215. ```
  216. - Save and exit the file.
  217. ### Configure Script Permissions
  218. We want `init.bash` to run the `nvidia-smi` commands without having to enter a password. This is done by giving `nvidia-persistenced` and `nvidia-smi` passwordless `sudo` permissions, and can be achieved by editing the `sudoers` file.
  219. AMD users can skip this step as power limiting is not supported on AMD GPUs.
  220. - Run the following command to edit the sudoers file:
  221. ```bash
  222. sudo visudo
  223. ```
  224. - Add the following lines to the file:
  225. ```
  226. <username> ALL=(ALL) NOPASSWD: /usr/bin/nvidia-persistenced
  227. <username> ALL=(ALL) NOPASSWD: /usr/bin/nvidia-smi
  228. ```
  229. > Replace `<username>` with your username.
  230. - Save and exit the file.
  231. > [!IMPORTANT]
  232. > Ensure that you add these lines AFTER `%sudo ALL=(ALL:ALL) ALL`. The order of the lines in the file matters - the last matching line will be used so if you add these lines before `%sudo ALL=(ALL:ALL) ALL`, they will be ignored.
  233. ### Configure Auto-Login (optional)
  234. When the server boots up, we may want it to automatically log in to a user account and run the `init.bash` script. This is done by configuring the `lightdm` display manager.
  235. - Run the following command:
  236. ```bash
  237. sudo nano /etc/lightdm/lightdm.conf
  238. ```
  239. - Find the following commented line. It should be in the `[Seat:*]` section.
  240. ```
  241. # autologin-user=
  242. ```
  243. - Uncomment the line and add your username:
  244. ```
  245. autologin-user=<username>
  246. ```
  247. > Replace `<username>` with your username.
  248. - Save and exit the file.
  249. ## Docker
  250. 📖 [**Documentation**](https://docs.docker.com/engine/)
  251. Docker is a containerization platform that allows you to run applications in isolated environments. This subsection follows [Docker's guide](https://docs.docker.com/engine/install/debian/) to install Docker Engine on Debian. The commands are listed below, but visiting the guide is recommended in case instructions have changed.
  252. - If you already have a Docker installation on your system, it's a good idea to re-install so there are no broken/out-of-date dependencies. The command below will iterate through your system's installed packages and remove the ones associated with Docker.
  253. ```bash
  254. for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt purge $pkg; done
  255. ```
  256. - Run the following commands:
  257. ```bash
  258. # Add Docker's official GPG key:
  259. sudo apt update
  260. sudo apt install ca-certificates curl
  261. sudo install -m 0755 -d /etc/apt/keyrings
  262. sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
  263. sudo chmod a+r /etc/apt/keyrings/docker.asc
  264. # Add the repository to Apt sources:
  265. sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
  266. Types: deb
  267. URIs: https://download.docker.com/linux/debian
  268. Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
  269. Components: stable
  270. Signed-By: /etc/apt/keyrings/docker.asc
  271. EOF
  272. sudo apt update
  273. ```
  274. - Install the Docker packages:
  275. ```bash
  276. sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  277. ```
  278. - Verify the installation (optional):
  279. ```bash
  280. sudo systemctl status docker
  281. ```
  282. If it hasn't started, try manually starting the daemon:
  283. ```bash
  284. sudo systemctl start docker
  285. ```
  286. ### Add User to Docker Group
  287. So that we can forgo the use of `sudo` for Docker commands and run them directly under our user, we will add our user to the `docker` group. This isn't strictly necessary but convenient. This sub-section uses [Docker's own post-install documentation](https://docs.docker.com/engine/install/linux-postinstall/) as a reference - have a look just in case things have changed since the time of writing.
  288. > [!CAUTION]
  289. > Being in the `docker` group grants significant system access - effectively root-equivalent privileges to the Docker daemon. Only add trusted users to this group.
  290. 1. Add your user to the Docker group:
  291. ```bash
  292. sudo usermod -aG docker $USER
  293. ```
  294. 2. Apply the changes:
  295. ```bash
  296. newgrp docker
  297. ```
  298. 3. Verify your new permissions by checking running containers:
  299. ```bash
  300. docker ps -a
  301. ```
  302. If you encounter "permission denied" and Docker commands still require sudo, log out and log back in.
  303. > [!NOTE]
  304. > If you decide not to do this, you will need to append `sudo` in front of any `docker` commands.
  305. ### Nvidia Container Toolkit
  306. You will most likely want to use GPUs via Docker - this will require Nvidia Container Toolkit, which allows Docker to allocate/de-allocate memory on Nvidia GPUs. The steps for installing this are listed below, but it is recommended to reference [Nvidia's documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) for the most up-to-date commands.
  307. 1. Configure the repository
  308. ```bash
  309. curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  310. && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  311. sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  312. sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
  313. ```
  314. 2. Update packages:
  315. ```bash
  316. sudo apt update
  317. ```
  318. 3. Install Nvidia Container Toolkit packages:
  319. ```bash
  320. sudo apt install -y nvidia-container-toolkit
  321. ```
  322. 4. Restart the Docker daemon:
  323. ```bash
  324. sudo systemctl restart docker
  325. ```
  326. ### Create a Network
  327. We'll be running most services via Docker containers. To allow multiple containers to communicate with each other, we can open up ports via UFW (which we'll [configure later](#firewall)) but this is less optimal than creating a Docker network. This way, all containers on the network can securely talk to each other without needing to open ports for the many services we could run via UFW, inherently creating a more secure setup.
  328. We'll call this network `app-net`: you can call it anything you like, just be sure to update the commands that use it later.
  329. Run the following:
  330. ```bash
  331. docker network create app-net
  332. ```
  333. That's it! Now, when we create containers, we can reference it as follows:
  334. **Docker Run**
  335. ```bash
  336. docker run <container> --network app-net
  337. ```
  338. **Docker Compose**
  339. ```yaml
  340. services:
  341. <container>:
  342. # add this
  343. networks:
  344. - app-net
  345. # add this
  346. networks:
  347. app-net:
  348. external: true
  349. ```
  350. > Replace `<container>` with the actual service - don't forget to add other parameters too.
  351. With this configured, we can now call containers by name and port. Let's pretend we're calling the `/health` endpoint in `llama-swap` from `open-webui` (two actual containers we'll be creating later on) to ensure that the containers can see and speak to each other. Run (`CTRL+C` to quit):
  352. ```bash
  353. docker exec -i open-webui curl http://llama-swap:8080/health
  354. ```
  355. You could also run it the other way to be extra sure:
  356. ```bash
  357. docker exec -it llama-swap curl http://open-webui:8080
  358. ```
  359. > [!IMPORTANT]
  360. > The port is always the **internal port** the container is running on. If a container runs on 1111:8080, for example, 1111 is the port on the host (where you might access it, like `http://localhost:1111` or `http://<server_ip>:1111`) and 8080 is the internal port the container is running on. Thus, trying to access the container on 1111 via `app-net` will not work. Remembering this when specifying URLs in services will save you a lot of unnecessary "why isn't this working?" pains.
  361. ### Harden Docker Containers
  362. Docker security is is **critical** for public-facing services. It involves reducing attack surfaces, eliminating terminal abilities for containers, and limiting resource usage. Even if you only intend to access your services via a private network (preferred if possible), shoring up security is never a bad idea because it limits how much damage can be done to your server and its services if it ever was to become compromised somehow.
  363. 📖 See the full guide at [Docker Security](docs/docker-security.md).
  364. > [!TIP]
  365. > The combined example at the end provides a ready-to-paste YAML block with all hardening directives. You can copy it directly into your existing Compose files and customize as needed.
  366. ### Helpful Commands
  367. In the process of setting up this server (or anywhere down the rabbit hole of setting up services), you're likely to use Docker often. For the uninitiated, here are helpful commands that will make navigating and troubleshooting containers easier:
  368. - See available/running containers: `docker ps -a`
  369. - Restart a container: `docker restart <container_name>`
  370. - View a container's logs (live): `docker logs -f <container_name>` (`CTRL+C` to quit)
  371. - Rename a container: `docker rename <container_name> <new_container_name>`
  372. - Sometimes, a single service will spin up multiple containers, e.g. `xyz-server` and `xyz-db`. To restart both simultaneously, run the following command **from inside the directory containing the Compose file**: `docker compose restart`
  373. - Recreate a service: `docker compose down && docker compose up -d`
  374. - Test a container's configuration: `docker compose config --quiet`
  375. - View a container's resolved configuration: `docker compose config`
  376. - List Docker networks: `docker network ls`
  377. > [!TIP]
  378. > There are no set rules when it comes to how you set up your Docker containers/services. However, here are my two cents:
  379. > It's cleanest to use Docker Compose (`docker compose up -d` with a `docker-compose.yaml` file as opposed to `docker run -d <image_name>`). Unless you take copious notes on your homelab and its setup, this method is almost self-documenting and keeps a neat trail of the services you run via their compose files. One compose file per directory is standard.
  380. > [!TIP]
  381. > Occasionally, restarting a container is not enough to persist changes to configurations. If ever you find that changes you made to a service are not showing up, recreate the service altogether first before continuing to other troubleshooting.
  382. ## HuggingFace CLI
  383. 📖 [**Documentation**](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli)
  384. HuggingFace is the leading open-source ML/AI platform - it hosts models (including LLMs), datasets, and demo apps that can be used to test models. For the purpose of this guide, we'll be using HuggingFace to download popular open-source LLMs.
  385. > [!NOTE]
  386. > Only needed for llama.cpp/vLLM.
  387. - Create a new virtual environment:
  388. ```bash
  389. python3 -m venv hf-env
  390. source hf-env/bin/activate
  391. ```
  392. - Download the `huggingface_hub` package using `pip`:
  393. ```bash
  394. pip install -U "huggingface_hub[cli]"
  395. ```
  396. - Create an authentication token on https://huggingface.com
  397. - Log in to HF Hub:
  398. ```bash
  399. hf auth login
  400. ```
  401. - Enter your token when prompted.
  402. - Run the following to verify your login:
  403. ```bash
  404. hf auth whoami
  405. ```
  406. The result should be your username.
  407. ### Manage Models
  408. Models can be downloaded either to the default location (`.cache/huggingface/hub`) or to any local directory you specify. Where the model is stored can be defined using the `--local-dir` command line flag. Not specifying this will result in the model being stored in the default location. Storing the model in the folder where the packages for the inference engine are stored is good practice - this way, everything you need to run inference on a model is stored in the same place. However, if you use the same models with multiple backends frequently (e.g. using Qwen_QwQ-32B-Q4_K_M.gguf with both llama.cpp and vLLM), either set a common model directory or use the default HF option without specifying this flag.
  409. First, activate the virtual environment that contains `huggingface_hub`:
  410. ```
  411. source hf-env/bin/activate
  412. ```
  413. ### Download Models
  414. Models are downloaded using their HuggingFace tag. Here, we'll use bartowski/Qwen_QwQ-32B-GGUF as an example. To download a model, run:
  415. ```
  416. hf download bartowski/Qwen_QwQ-32B-GGUF Qwen_QwQ-32B-Q4_K_M.gguf --local-dir models
  417. ```
  418. Ensure that you are in the correct directory when you run this.
  419. ### Delete Models
  420. To delete a model in the specified location, run:
  421. ```
  422. rm <model_name>
  423. ```
  424. To delete a model in the default location, run:
  425. ```
  426. hf delete-cache
  427. ```
  428. This will start an interactive session where you can remove models from the HuggingFace directory. In case you've been saving models in a different location than `.cache/huggingface`, deleting models from there will free up space but the metadata will remain in the HF cache until it is deleted properly. This can be done via the above command but you can also simply delete the model directory from `.cache/huggingface/hub`.
  429. ## Search Engine
  430. > [!NOTE]
  431. > This step is optional but highly recommended for grounding LLMs with relevant search results from reputable sources. Targeted web searches via MCP tool calls make reports generated by LLMs much less prone to random hallucinations.
  432. ### SearXNG
  433. 🌟 [**GitHub**](https://github.com/searxng/searxng)
  434. 📖 [**Documentation**](https://docs.searxng.org)
  435. 📖 [**Docker Installation Reference**](https://docs.searxng.org/admin/installation-docker.html)
  436. To power our search-based workflows, we don't want to rely on a search provider that can monitor searches. While using any search engine has this problem, metasearch engines like SearXNG mitigate it to a decent degree. SearXNG aggregates results from over 245 [search services](https://docs.searxng.org/user/configured_engines.html#configured-engines) and does not track/profile users. You can use a hosted instance on the Internet but, considering the priorities of this guide and how trivial it is to set one up, we'll be spinning up our own instance on its default port (8080).
  437. 1. Create the project structure and fetch the compose file:
  438. ```bash
  439. mkdir -p searxng/core-config && cd searxng
  440. curl -fsSL -O https://raw.githubusercontent.com/searxng/searxng/master/container/docker-compose.yml
  441. ```
  442. 2. Append `app-net` network to the `core` service in `docker-compose.yml`:
  443. ```yaml
  444. services:
  445. core:
  446. # ...existing config...
  447. networks:
  448. default:
  449. app-net:
  450. aliases:
  451. # this allows reference to the "searxng-core" container via "searxng"
  452. - searxng
  453. networks:
  454. default:
  455. app-net:
  456. external: true
  457. ```
  458. 3. Create `core-config/settings.yml`:
  459. ```yaml
  460. use_default_settings: true
  461. search:
  462. formats:
  463. - html
  464. - json
  465. server:
  466. secret_key: "CHANGE-ME-TO-RANDOM-STRING"
  467. image_proxy: true
  468. limiter: false
  469. ```
  470. 4. Start the services:
  471. ```bash
  472. docker compose up -d
  473. ```
  474. > [!TIP]
  475. > Run `openssl rand -base64 32` to generate a secure random key.
  476. SearXNG is now accessible at `http://localhost:8080` externally and `http://searxng:8080` from containers on `app-net`.
  477. ### Open WebUI Integration
  478. In case you want a simple web search workflow and want to skip MCP servers/agentic setups, Open WebUI supports web search functionality natively. Navigate to `Admin Panel > Settings > Web Search` and set the following values:
  479. - Enable `Web Search`
  480. - Web Search Engine: `searxng`
  481. - Searxng Query URL: `http://searxng:8080/search?q=<query>`
  482. - API Key: `anything-you-like`
  483. ## Inference Engine
  484. The inference engine is the software layer that loads model weights and manages text generation. This guide supports three popular engines: Ollama, llama.cpp, and vLLM. Each offers different trade-offs between ease of use, control, and performance.
  485. If you're unsure where to begin and want some more context, see a detailed comparison in [Comparing Inference Engines](docs/inference-engine-comparison.md). It goes into what inference is, how the engines tackle it differently, and what the best fit for you might be.
  486. ### Ollama
  487. 🌟 [**GitHub**](https://github.com/ollama/ollama)
  488. 📖 [**Documentation**](https://github.com/ollama/ollama/tree/main/docs)
  489. 🔧 [**Engine Arguments**](https://github.com/ollama/ollama/blob/main/docs/modelfile.md)
  490. Ollama will be installed as a service, so it runs automatically at boot.
  491. - Download Ollama from the official repository:
  492. ```
  493. curl -fsSL https://ollama.com/install.sh | sh
  494. ```
  495. We want our API endpoint to be reachable by the rest of the LAN. For Ollama, this means setting `OLLAMA_HOST=0.0.0.0` in the `ollama.service`.
  496. - Run the following command to edit the service:
  497. ```
  498. systemctl edit ollama.service
  499. ```
  500. - Find the `[Service]` section and add `Environment="OLLAMA_HOST=0.0.0.0"` under it. It should look like this:
  501. ```
  502. [Service]
  503. Environment="OLLAMA_HOST=0.0.0.0"
  504. ```
  505. - Save and exit.
  506. - Reload the environment.
  507. ```
  508. systemctl daemon-reload
  509. systemctl restart ollama
  510. ```
  511. > [!TIP]
  512. > If you installed Ollama manually or don't use it as a service, remember to run `ollama serve` to properly start the server. Refer to [Ollama's troubleshooting steps](#ollama-3) if you encounter an error.
  513. ### llama.cpp
  514. 🌟 [**GitHub**](https://github.com/ggml-org/llama.cpp)
  515. 📖 [**Documentation**](https://github.com/ggml-org/llama.cpp/tree/master/docs)
  516. 🔧 [**Engine Arguments**](https://github.com/ggml-org/llama.cpp/tree/master/examples/server)
  517. > [!TIP]
  518. > This is my recommended inference engine. My biggest reasons are
  519. > 1. the `--fit` flag and its ability to automatically load any model by calculating offload requirements based on available system resources, and
  520. > 2. insanely quick model load times, especially compared to vLLM.
  521. - Clone the llama.cpp GitHub repository:
  522. ```
  523. git clone https://github.com/ggml-org/llama.cpp.git
  524. cd llama.cpp
  525. ```
  526. - Build the binary:
  527. **CPU**
  528. ```
  529. cmake -B build
  530. cmake --build build --config Release -j <threads>
  531. ```
  532. **CUDA**
  533. ```
  534. cmake -B build -DGGML_CUDA=ON
  535. cmake --build build --config Release -j <threads>
  536. ```
  537. > Replace `<threads>` with `[number of threads your CPU has] - 2`. This offers a massive boost to the build process as it leverages your system resources better.
  538. For other systems looking to use Metal, Vulkan and other low-level graphics APIs, view the complete [llama.cpp build documentation](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) to leverage accelerated inference.
  539. ### vLLM
  540. 🌟 [**GitHub**](https://github.com/vllm-project/vllm)
  541. 📖 [**Documentation**](https://docs.vllm.ai/en/stable/index.html)
  542. 🔧 [**Engine Arguments**](https://docs.vllm.ai/en/stable/serving/engine_args.html)
  543. vLLM comes with its own OpenAI-compatible API that we can use just like Ollama. Where Ollama runs GGUF model files, vLLM can run AWQ, GPTQ, GGUF, BitsAndBytes, and safetensors (the default release type) natively.
  544. **Manual Installation (Recommended)**
  545. - Create a directory and virtual environment for vLLM:
  546. ```
  547. mkdir vllm
  548. cd vllm
  549. python3 -m venv .venv
  550. source .venv/bin/activate
  551. ```
  552. - Install vLLM using `pip`:
  553. ```
  554. pip install vllm
  555. ```
  556. - Serve with your desired flags. It uses port 8000 by default, but I'm using port 8556 here so it doesn't conflict with any other services:
  557. ```
  558. vllm serve <model> --port 8556
  559. ```
  560. - To use as a service, add the following block to `init.bash` to serve vLLM on startup:
  561. ```
  562. source .venv/bin/activate
  563. vllm serve <model> --port 8556
  564. ```
  565. > Replace `<model>` with your desired model tag, copied from HuggingFace.
  566. **Docker Installation**
  567. - Run:
  568. ```
  569. docker run --gpus all \
  570. -v ~/.cache/huggingface:/root/.cache/huggingface \
  571. --env "HUGGING_FACE_HUB_TOKEN=<your_hf_hub_token>" \
  572. -p 8556:8000 \
  573. --ipc=host \
  574. vllm/vllm-openai:latest \
  575. --model <model>
  576. ```
  577. > Replace `<your_hf_hub_token>` with your HuggingFace Hub token and `<model>` with your desired model tag, copied from HuggingFace.
  578. To serve a different model:
  579. - First stop the existing container:
  580. ```
  581. docker ps -a
  582. docker stop <vllm_container_ID>
  583. ```
  584. - If you want to run the exact same setup again in the future, skip this step. Otherwise, run the following to delete the container and not clutter your Docker container environment:
  585. ```
  586. docker rm <vllm_container_ID>
  587. ```
  588. - Rerun the Docker command from the installation with the desired model.
  589. ```
  590. docker run --gpus all \
  591. -v ~/.cache/huggingface:/root/.cache/huggingface \
  592. --env "HUGGING_FACE_HUB_TOKEN=<your_hf_hub_token>" \
  593. -p 8556:8000 \
  594. --ipc=host \
  595. vllm/vllm-openai:latest \
  596. --model <model>
  597. ```
  598. ## Model Server
  599. > [!NOTE]
  600. > Only needed for manual installations of llama.cpp/vLLM. Ollama manages model management via its CLI.
  601. While the above steps will help you get up and running with an OpenAI-compatible LLM server, they will not help with this server persisting after you close your terminal window or restart your physical server. They also won't allow a chat platform to reliably reference and swap between the various models you have available - a likely use-case in a landscape where different models specialize in different tasks. Running the inference engine via Docker can achieve this persistence with the `-d` (for "detach") flag but (i) services like llama.cpp and vLLM are usually configured without Docker and (ii) it can't swap models on-demand. This necessitates a server that can manage loading/unloading, swapping, and listing available models.
  602. ### llama-swap
  603. 🌟 [**GitHub**](https://github.com/mostlygeek/llama-swap)
  604. 📖 [**Documentation**](https://github.com/mostlygeek/llama-swap/wiki)
  605. 📖 [**Configuration**](https://github.com/mostlygeek/llama-swap/wiki/Configuration)
  606. > [!TIP]
  607. > This is my recommended way to run llama.cpp/vLLM models.
  608. llama-swap is a lightweight proxy server for LLMs that solves our pain points from above. It's an extremely configurable tool that allows a single point of entry for models from various backends. Models can be set up in groups, listed/unlisted easily, configured with customized hyperparameters, and monitored using streamed logs in the llama-swap web UI.
  609. In the installation below, we'll use `Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf` for llama.cpp and `Qwen/Qwen3-4B-Instruct-2507` for vLLM. We'll also use port 7000 to serve the models on.
  610. 1. Create a new directory with the `config.yaml` file:
  611. ```bash
  612. sudo mkdir llama-swap
  613. cd llama-swap
  614. sudo nano config.yaml
  615. ```
  616. 2. Enter the following and save:
  617. **llama.cpp**
  618. ```yaml
  619. models:
  620. "qwen3-4b":
  621. proxy: "http://127.0.0.1:7000"
  622. cmd: |
  623. /app/llama-server
  624. -m /models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
  625. --port 7000 --host 0.0.0.0
  626. ```
  627. **vLLM (Docker)**
  628. ```yaml
  629. models:
  630. "qwen3-4b":
  631. proxy: "http://127.0.0.1:7000"
  632. cmd: |
  633. docker run --name qwen-vllm
  634. --init --rm -p 7000:8080 --ipc=host
  635. vllm/vllm-openai:latest
  636. -m /models/Qwen/Qwen3-4B-Instruct-2507
  637. cmdStop: docker stop qwen-vllm
  638. ```
  639. **vLLM (local)**:
  640. ```yaml
  641. models:
  642. "qwen3-4b":
  643. proxy: "http://127.0.0.1:7000"
  644. cmd: |
  645. source /app/vllm/.venv/bin/activate &&
  646. /app/vllm/.venv/bin/vllm serve
  647. -m /models/Qwen/Qwen3-4B-Instruct-2507
  648. --port 7000 --host 0.0.0.0
  649. cmdStop: pkill -f "vllm serve"
  650. ```
  651. 3. Install the container:
  652. We use the `cuda` tag here, but llama-swap offers `cpu`, `intel`, `vulkan`, and `musa` tags as well. Releases can be found [here](https://github.com/mostlygeek/llama-swap/pkgs/container/llama-swap).
  653. **llama.cpp**
  654. ```bash
  655. docker run -d --gpus all --restart unless-stopped --network app-net --pull=always --name llama-swap -p 9292:8080 \
  656. -v /path/to/models:/models \
  657. -v /home/<username>/llama-swap/config.yaml:/app/config.yaml \
  658. -v /home/<username>/llama.cpp/build/bin/llama-server:/app/llama-server \
  659. ghcr.io/mostlygeek/llama-swap:cuda
  660. ```
  661. **vLLM (Docker/local)**
  662. ```bash
  663. docker run -d --gpus all --restart unless-stopped --network app-net --pull=always --name llama-swap -p 9292:8080 \
  664. -v /path/to/models:/models \
  665. -v /home/<username>/vllm:/app/vllm \
  666. -v /home/<username>/llama-swap/config.yaml:/app/config.yaml \
  667. ghcr.io/mostlygeek/llama-swap:cuda
  668. ```
  669. > Replace `<username>` with your actual username and `/path/to/models` with the path to your model files.
  670. > [!NOTE]
  671. > llama-swap prefers Docker-based vLLM due to cleanliness of environments and adherence to SIGTERM signals sent by the server. I've written out both options here.
  672. This should result in a functioning llama-swap instance running at `http://localhost:9292`, which can be confirmed by running `curl http://localhost:9292/health`. It is **highly recommended** that you read the configuration documentation. llama-swap is thoroughly documented and highly configurable - utilizing its capabilities will result in a tailored setup ready to deploy as you need it.
  673. ### `systemd` Service
  674. 📖 [**llama.cpp Server Documentation**](https://github.com/ggml-org/llama.cpp/tree/master/tools/server)
  675. 📖 [**llama.cpp Preset Documentation**](https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md)
  676. 📖 [**vLLM Online Serving Documentation**](https://docs.vllm.ai/en/stable/serving/online_serving.html)
  677. The other way to persist a model across system reboots is to start the inference engine in a `.service` file that will run alongside the Linux operating system when booting, ensuring that it is available whenever the server is on.
  678. llama.cpp can now natively allow for model-switching via its `--models-preset` flag. Compared to llama-swap, llama.cpp's in-built switcher naturally has less overhead because it's bundled with the binary but it also doesn't come with a UI where one can switch models, view activity & logs, monitor system performance during inference, etc. llama.cpp and llama-swap are functionally equivalent with respect to model-switching. For vLLM, if you're willing to live with the relative compromise of not being able to swap models/backends and are satisfied with running one model, `systemd` is the lowest overhead solution and works great.
  679. Let's call the service we're about to build `llm-server.service`. We'll assume all models are in the `models` child directory - you can change this as you need to.
  680. Create the `systemd` service file:
  681. ```bash
  682. sudo nano /etc/systemd/system/llm-server.service
  683. ```
  684. **llama.cpp**
  685. 1. Create the `models.ini` file (in your llama.cpp directory):
  686. ```ini
  687. version = 1
  688. [*]
  689. jinja = true
  690. no-mmap = true
  691. no-warmup = true
  692. flash-attn = true
  693. [qwen3.6-27b]
  694. model = models/Qwen3.6-27B/Qwen_Qwen3.6-27B-Q8_0.gguf
  695. mmproj = models/Qwen3.6-27B/mmproj-BF16.gguf
  696. ctx-size = 180000
  697. parallel = 1
  698. context-shift = true
  699. fit = off
  700. tensor-split = 12,11
  701. spec-type = draft-mtp
  702. spec-draft-n-max = 2
  703. temp = 0.6
  704. min-p = 0.0
  705. top-p = 0.95
  706. top-k = 20
  707. presence-penalty = 0.0
  708. repeat-penalty = 1.0
  709. chat-template-kwargs = {"preserve_thinking": true}
  710. [gemma-4-31b]
  711. model = models/Gemma4-31B/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
  712. mmproj = models/Gemma4-31B/mmproj-BF16.gguf
  713. model-draft = models/Gemma4-31B/gemma-4-31b-it-qat-q4_0-assistant.gguf
  714. ctx-size = 130000
  715. main-gpu = 0
  716. n-gpu-layers = 99
  717. parallel = 1
  718. fit = on
  719. spec-type = draft-mtp
  720. spec-draft-n-max = 2
  721. chat-template-kwargs = {"preserve_thinking": true}
  722. temp = 1.0
  723. min-p = 0.0
  724. top-p = 0.95
  725. top-k = 64
  726. cache-ram = 2048
  727. ctx-checkpoints = 2
  728. image-min-tokens = 300
  729. image-max-tokens = 512
  730. ```
  731. 2. Configure the service file:
  732. ```ini
  733. [Unit]
  734. Description=llama.cpp Server
  735. After=network.target
  736. [Service]
  737. Type=simple
  738. User=<username>
  739. Group=<username>
  740. WorkingDirectory=/home/<username>/llama.cpp
  741. ExecStart=/home/<username>/llama.cpp/build/bin/llama-server \
  742. --models-preset models.ini \
  743. --port 9292 \
  744. --host 0.0.0.0
  745. # Restart policy
  746. Restart=on-failure
  747. RestartSec=5
  748. [Install]
  749. WantedBy=multi-user.target
  750. ```
  751. **vLLM**
  752. 1. Configure the service file:
  753. ```ini
  754. [Unit]
  755. Description=LLM Server Service
  756. After=network.target
  757. [Service]
  758. User=<user>
  759. Group=<user>
  760. WorkingDirectory=/home/<user>/vllm/
  761. ExecStart=/bin/bash -c 'source .venv/bin/activate && vllm serve --port <port> --host 0.0.0.0 -m /home/<username>/vllm/models/<model>'
  762. Restart=always
  763. RestartSec=10s
  764. [Install]
  765. WantedBy=multi-user.target
  766. ```
  767. > Replace `<username>`, `<port>`, and `<model>` with your Linux username, desired port for serving, and desired model respectively.
  768. Reload the `systemd` daemon:
  769. ```bash
  770. sudo systemctl daemon-reload
  771. ```
  772. Run the service:
  773. - If `llm-server.service` doesn't exist:
  774. ```bash
  775. sudo systemctl enable llm-server.service
  776. sudo systemctl start llm-server
  777. ```
  778. - If `llm-server.service` does exist:
  779. ```bash
  780. sudo systemctl restart llm-server
  781. ```
  782. (Optional) Check the service's status:
  783. ```bash
  784. sudo systemctl status llm-server
  785. ```
  786. ### Open WebUI Integration
  787. #### llama-swap
  788. Navigate to `Admin Panel > Settings > Connections` and set the following values:
  789. - Enable OpenAI API
  790. - API Base URL: `http://llama-swap:8080/v1`
  791. - API Key: `anything-you-like`
  792. #### `systemd` Service
  793. Follow the same steps as above.
  794. - Enable OpenAI API
  795. - API Base URL: `http://localhost:<port>/v1`
  796. - API Key: `anything-you-like`
  797. > Replace `<port>` with your desired port.
  798. > [!WARNING]
  799. > If you use the `systemd` service, ensure the port is open to the local network via `ufw`. Refer to [Firewall](#firewall).
  800. ## Chat Platform
  801. ### Open WebUI
  802. 🌟 [**GitHub**](https://github.com/open-webui/open-webui)
  803. 📖 [**Documentation**](https://docs.openwebui.com)
  804. Open WebUI is a web-based interface for managing models and chats, and provides a beautiful, performant UI for communicating with your models. You will want to do this if you want to access your models from a web interface. If you're fine with using the command line or want to consume models through a plugin/extension, you can skip this step.
  805. To install without Nvidia GPU support, run the following command:
  806. ```bash
  807. docker run -d -p 3000:8080 --network app-net --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
  808. ```
  809. For Nvidia GPUs, run the following command:
  810. ```bash
  811. docker run -d -p 3000:8080 --network app-net --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda
  812. ```
  813. You can access it by navigating to `http://localhost:3000` in your browser or `http://<server_ip>:3000` from another device on the same network. There's no need to add this to the `init.bash` script as Open WebUI will start automatically at boot via Docker Engine.
  814. Read more about Open WebUI [here](https://github.com/open-webui/open-webui).
  815. > [!NOTE]
  816. > `host.docker.internal` is a magic hostname that resolves to the internal IP address assigned to the host by Docker. This allows containers to communicate with services running on the host, such as databases or web servers, without needing to know the host's IP address. It simplifies communication between containers and host-based services, making it easier to develop and deploy applications.
  817. ## MCP Proxy Server
  818. Model Context Protocol (MCP) is a protocol that tools (functions/scripts written in code) to LLMs in a standardized way. Generally, models are being trained more and more with the ability to natively call tools in order to power agentic tasks - think along the lines of having a model use sequential thinking to formulate multiple thoughts, execute multiple targeted web searches, and provide a response leveraging real-time information. MCP also, probably more importantly for most people, enables models to call third-party tools like those for GitHub, Azure, etc. A complete list, curated and maintained by Anthropic, can be found [here](https://github.com/modelcontextprotocol/servers).
  819. Most guides on the Internet concerning MCP will have you spin up an MCP server via a client like VS Code, Cline, etc. since most agentic uses are for coding or Claude Desktop, which is a proprietary app by Anthropic and not at all what we're aiming to achieve in this guide with respect to privacy. There *are* other chat clients that support MCP server management from the UI itself (LobeChat, Cherry Studio, etc.), but we want to be able to manage MCP servers in a central and modular way. That way, (i) they aren't tied to a specific client and are available to every client you may use them with and (ii) if you switch chat platforms in the future, your MCP servers require zero changes because they run as a decoupled service - little bit more maintenance for a lot more flexibility down the line. We can do this by setting up an MCP proxy server.
  820. This proxy server will take the MCP servers running via stdio (standard IO) protocol (that can only be accessed by an application running on that device) and make them compatible with streamable HTTP. Any MCP-enabled client can use streamable HTTP so they'll also be able to use all the servers we install on our physical server. This centralizes the management of your MCP servers: create/edit/delete servers in one place, use any of them from your various clients (Open WebUI, VS Code, etc.).
  821. We'll use the [fetch](https://github.com/zcaceres/fetch-mcp), [sequential-thinking](https://github.com/arben-adm/mcp-sequential-thinking), and [searxng](https://github.com/ihor-sokoliuk/MCP-searxng) MCP servers to get started. The process for adding more servers will be identical.
  822. ### mcp-proxy
  823. 🌟 [**GitHub**](https://github.com/sparfenyuk/mcp-proxy)
  824. mcp-proxy is a server proxy that allows switching between transports (stdio to streamable HTTP and vice versa). I'll be using port 3131 to avoid conflicts - feel free to change this as you need to. I will also be extending mcp-proxy to include `uv`: most MCP servers use either `npx` or `uv` and setting mcp-proxy up without `uv` is going to hamper your ability to run the MCP servers you'd like. If you don't require `uv`, (i) don't add the `build` section in the compose file and (ii) skip step 4.
  825. 1. Create a compose file
  826. ```bash
  827. mkdir mcp-proxy
  828. cd mcp-proxy
  829. sudo nano docker-compose.yaml
  830. ```
  831. 2. Enter the following:
  832. ```yaml
  833. services:
  834. mcp-proxy:
  835. container_name: mcp-proxy
  836. build:
  837. context: .
  838. dockerfile: Dockerfile
  839. networks:
  840. - app-net
  841. volumes:
  842. - .:/config
  843. - /:/<server_hostname>:ro
  844. restart: unless-stopped
  845. ports:
  846. - 3131:3131
  847. command: "--pass-environment --port=3131 --host 0.0.0.0 --transport streamablehttp --named-server-config /config/servers.json"
  848. networks:
  849. app-net:
  850. external: true
  851. ```
  852. > Replace `<server_hostname>` with your actual server's hostname (or whatever else). This is primarily useful when adding `filesystem` or similar MCP servers that read from and write files to the file system. Feel free to skip if that isn't your goal.
  853. 3. Create a `servers.json` file:
  854. ```json
  855. {
  856. "mcpServers": {
  857. "fetch": {
  858. "disabled": false,
  859. "timeout": 60,
  860. "command": "uvx",
  861. "args": [
  862. "mcp-server-fetch"
  863. ],
  864. "transportType": "stdio"
  865. },
  866. "sequential-thinking": {
  867. "command": "npx",
  868. "args": [
  869. "-y",
  870. "@modelcontextprotocol/server-sequential-thinking"
  871. ]
  872. },
  873. "searxng": {
  874. "command": "npx",
  875. "args": ["-y", "mcp-searxng"],
  876. "env": {
  877. "SEARXNG_URL": "http://searxng:8080/search?q=<query>"
  878. }
  879. }
  880. }
  881. }
  882. ```
  883. 4. Create a `Dockerfile`:
  884. ```bash
  885. sudo nano Dockerfile
  886. ```
  887. Enter the following:
  888. ```Dockerfile
  889. FROM ghcr.io/sparfenyuk/mcp-proxy:latest
  890. # Install dependencies for nvm and Node.js
  891. RUN apk add --update npm
  892. # Install the 'uv' package
  893. RUN python3 -m ensurepip && pip install --no-cache-dir uv
  894. ENV PATH="/usr/local/bin:/usr/bin:$PATH" \
  895. UV_PYTHON_PREFERENCE=only-system
  896. ENTRYPOINT ["catatonit", "--", "mcp-proxy"]
  897. ```
  898. 5. Start the container with `docker compose up -d`
  899. Your mcp-proxy container should be up and running! Adding servers is simple: add the relevant server to `servers.json` (you can use the same configuration that the MCP server's developer provides for VS Code, it's identical) and then restart the container with `docker restart mcp-proxy`.
  900. ### MCPJungle
  901. 🌟 [**GitHub**](https://github.com/mcpjungle/MCPJungle?tab=readme-ov-file)
  902. MCPJungle is another MCP proxy server with a different focus. It focuses on providing more of a "production-grade" experience, a lot of which is disabled by default in the development mode of the application. We'll use the standard development version of the container here on port 4141.
  903. 1. Create a compose file:
  904. ```bash
  905. mkdir mcpjungle
  906. cd mcpjungle
  907. sudo nano docker-compose.yaml
  908. ```
  909. Enter the following and save:
  910. ```yaml
  911. # MCPJungle Docker Compose configuration for individual users.
  912. # Use this compose file if you want to run MCPJungle locally for your personal MCP management & Gateway.
  913. # The mcpjungle server runs in development mode.
  914. services:
  915. db:
  916. image: postgres:latest
  917. container_name: mcpjungle-db
  918. environment:
  919. POSTGRES_USER: mcpjungle
  920. POSTGRES_PASSWORD: mcpjungle
  921. POSTGRES_DB: mcpjungle
  922. ports:
  923. - "5432:5432"
  924. networks:
  925. - app-net
  926. volumes:
  927. - db_data:/var/lib/postgresql/data
  928. healthcheck:
  929. test: ["CMD-SHELL", "PGPASSWORD=mcpjungle pg_isready -U mcpjungle"]
  930. interval: 10s
  931. timeout: 5s
  932. retries: 5
  933. restart: unless-stopped
  934. mcpjungle:
  935. image: mcpjungle/mcpjungle:${MCPJUNGLE_IMAGE_TAG:-latest-stdio}
  936. container_name: mcpjungle-server
  937. environment:
  938. DATABASE_URL: postgres://mcpjungle:mcpjungle@db:5432/mcpjungle
  939. SERVER_MODE: ${SERVER_MODE:-development}
  940. OTEL_ENABLED: ${OTEL_ENABLED:-false}
  941. ports:
  942. - "4141:8080"
  943. networks:
  944. - app-net
  945. volumes:
  946. # Mount host filesystem current directory to enable filesystem MCP server access
  947. - .:/host/project:ro
  948. - /home/<username>:/host:ro
  949. # Other options:
  950. # - ${HOME}:/host/home:ro
  951. # - /tmp:/host/tmp:rw
  952. depends_on:
  953. db:
  954. condition: service_healthy
  955. restart: always
  956. volumes:
  957. db_data:
  958. networks:
  959. app-net:
  960. external: true
  961. ```
  962. 2. Start the container with `docker compose up -d`
  963. 3. Create a tool file:
  964. ```bash
  965. sudo nano fetch.json
  966. ```
  967. Enter the following and save:
  968. ```json
  969. {
  970. "name": "fetch",
  971. "transport": "stdio",
  972. "command": "npx",
  973. "args": ["mcp-server-fetch"]
  974. }
  975. ```
  976. 4. Register the tool:
  977. ```bash
  978. docker exec -i mcpjungle-server /mcpjungle register -c /host/project/fetch.json
  979. ```
  980. Repeat steps 3 and 4 for every tool mentioned. Commands for `sequential-thinking` and `searxng` can be found below.
  981. **sequential-thinking**
  982. ```json
  983. {
  984. "name": "sequential-thinking",
  985. "transport": "stdio",
  986. "command": "npx",
  987. "args": ["-y", "@modelcontextprotocol/server-sequential-thinking"]
  988. }
  989. ```
  990. **searxng**
  991. ```json
  992. {
  993. "name": "searxng",
  994. "transport": "stdio",
  995. "command": "npx",
  996. "args": ["-y", "mcp-searxng"],
  997. "env": {
  998. "SEARXNG_URL": "http://searxng:8080/search?q=<query>"
  999. }
  1000. }
  1001. ```
  1002. ### Comparison
  1003. The choice between the two services is yours entirely: I use mcp-proxy because I find the workflow slightly less cumbersome than MCPJungle. Here's a comparison with the stengths of each service.
  1004. **mcp-proxy > MCPJungle**
  1005. - Servers can just be added to `servers.json` and will be registered automatically on container restart - MCPJungle requires manual registration of tools via the CLI
  1006. - Uses the standard MCP syntax that most clients accept for configuration
  1007. - Lighter footprint in that it doesn't need to spin up a separate database container
  1008. - Uses stateful connections - MCPJungle spins up a new connection per tool call, which can lead to some performance overhead
  1009. **MCPJungle > mcp-proxy**
  1010. - Combines all tools under one endpoint, making it very easy to integrate into a chat frontend
  1011. - Capable of creating a very configurable setup with tool groups, access control, selective tool enabling/disabling
  1012. - Supports enterprise features like telemetry
  1013. ### Open WebUI Integration
  1014. Open WebUI recently added support for streamable HTTP - where once you may have had to use [mcpo](https://github.com/open-webui/mcpo), Open WebUI's way of automatically generating an OpenAPI-compatible HTTP server, you can use the MCP servers you've set up as-is with no changes.
  1015. #### mcp-proxy
  1016. Navigate to `Admin Panel > Settings > External Tools`. Click the `+` button to add a new tool and enter the following information:
  1017. - URL: `http://mcp-proxy:<port>/servers/<tool_name>/mcp`
  1018. - API Key: `anything-you-like`
  1019. - ID: `<tool_name>`
  1020. - Name: `<tool_name>`
  1021. > Replace `<port>` with the port of the MCP service and `<tool_name>` with the specific tool you're adding.
  1022. #### MCPJungle
  1023. Follow the same steps as above. By design, MCPJungle exposes all tools via one endpoint, so you should only have to add once:
  1024. - URL: `http://mcpjungle-server:8080/mcp`
  1025. - API Key: `anything-you-like`
  1026. - ID: `<tool_name>`
  1027. - Name: `<tool_name>`
  1028. > [!IMPORTANT]
  1029. > When configuring models in Open WebUI (via `Admin Panel > Settings > Models > my-cool-model > Advanced Params`), change the `Function Calling` parameter from `Default` to `Native`. This step will allow the model to use multiple tool calls to formulate a single response instead of just one.
  1030. ### VS Code/Claude Desktop Integration
  1031. The steps for integrating your MCP proxy server in another client such as VS Code (Claude Desktop, Zed, etc.) will be similar, if not exactly the same.
  1032. Add the following key and value to your `mcp.json` file:
  1033. ```json
  1034. "your-mcp-proxy-name": {
  1035. "timeout": 60,
  1036. "type": "stdio",
  1037. "command": "npx",
  1038. "args": [
  1039. "mcp-remote",
  1040. "http://<your-server-url>/mcp",
  1041. "--allow-http"
  1042. ]
  1043. }
  1044. ```
  1045. ## Text-to-Speech Server
  1046. ### Kokoro FastAPI
  1047. 🌟 [**GitHub**](https://github.com/remsky/Kokoro-FastAPI)
  1048. Kokoro FastAPI is a text-to-speech server that wraps around and provides OpenAI-compatible API inference for [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M), a state-of-the-art TTS model. The documentation for this project is fantastic and covers most, if not all, of the use cases for the project itself.
  1049. To install Kokoro-FastAPI, run
  1050. ```bash
  1051. git clone https://github.com/remsky/Kokoro-FastAPI.git
  1052. cd Kokoro-FastAPI
  1053. docker compose up --build
  1054. ```
  1055. The server can be used in two ways: an API and a UI. By default, the API is served on port 8880 and the UI is served on port 7860.
  1056. ### Open WebUI Integration
  1057. Navigate to `Admin Panel > Settings > Audio` and set the following values:
  1058. - Text-to-Speech Engine: `OpenAI`
  1059. - API Base URL: `http://host.docker.internal:8880/v1`
  1060. - API Key: `anything-you-like`
  1061. - Set Model: `kokoro`
  1062. - Response Splitting: None (this is crucial - Kokoro uses a novel audio splitting system)
  1063. The server can be used in two ways: an API and a UI. By default, the API is served on port 8880 and the UI is served on port 7860.
  1064. ## Image Generation Server
  1065. ### ComfyUI
  1066. 🌟 [**GitHub**](https://github.com/comfyanonymous/ComfyUI)
  1067. 📖 [**Documentation**](https://docs.comfy.org)
  1068. ComfyUI is a popular open-source graph-based tool for generating images using image generation models such as Stable Diffusion XL, Stable Diffusion 3, and the Flux family of models.
  1069. - Clone and navigate to the repository:
  1070. ```
  1071. git clone https://github.com/comfyanonymous/ComfyUI
  1072. cd ComfyUI
  1073. ```
  1074. - Set up a new virtual environment:
  1075. ```
  1076. python3 -m venv comfyui-env
  1077. source comfyui-env/bin/activate
  1078. ```
  1079. - Download the platform-specific dependencies:
  1080. - Nvidia GPUs
  1081. ```
  1082. pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
  1083. ```
  1084. - AMD GPUs
  1085. ```
  1086. pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
  1087. ```
  1088. - Intel GPUs
  1089. Read the installation instructions from [ComfyUI's GitHub](https://github.com/comfyanonymous/ComfyUI?tab=readme-ov-file#intel-gpus).
  1090. - Download the general dependencies:
  1091. ```
  1092. pip install -r requirements.txt
  1093. ```
  1094. Now, we have to download and load a model. Here, we'll use FLUX.1 [dev], a new, state-of-the-art medium-tier model by Black Forest Labs that fits well on an RTX 3090 24GB. Since we want this to be set up as easily as possible, we'll use a complete checkpoint that can be loaded directly into ComfyUI. For a completely customized workflow, CLIPs, VAEs, and models can be downloaded separately. Follow [this guide](https://comfyanonymous.github.io/ComfyUI_examples/flux/#simple-to-use-fp8-checkpoint-version) by ComfyUI's creator to install the FLUX.1 models in a fully customizable way.
  1095. > [!NOTE]
  1096. > [FLUX.1 [schnell] HuggingFace](https://huggingface.co/Comfy-Org/flux1-schnell/blob/main/flux1-schnell-fp8.safetensors) (smaller, ideal for <24GB VRAM)
  1097. >
  1098. > [FLUX.1 [dev] HuggingFace](https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors) (larger, ideal for 24GB VRAM)
  1099. - Download your desired model into `/models/checkpoints`.
  1100. - If you want ComfyUI to be served at boot and effectively run as a service, add the following lines to `init.bash`:
  1101. ```
  1102. cd /path/to/comfyui
  1103. source comfyui/bin/activate
  1104. python main.py --listen
  1105. ```
  1106. > Replace `/path/to/comfyui` with the correct relative path to `init.bash`.
  1107. Otherwise, to run it just once, simply execute the above lines in a terminal window.
  1108. ### Open WebUI Integration
  1109. Navigate to `Admin Panel > Settings > Images` and set the following values:
  1110. - Image Generation Engine: `ComfyUI`
  1111. - API Base URL: `http://localhost:8188`
  1112. > [!TIP]
  1113. > You'll either need more than 24GB of VRAM or to use a small language model mostly on CPU to use Open WebUI with FLUX.1 [dev]. FLUX.1 [schnell] and a small language model, however, should fit cleanly in 24GB of VRAM, making for a faster experience if you intend to regularly use both text and image generation together.
  1114. ## SSH
  1115. Enabling SSH allows you to connect to the server remotely. After configuring SSH, you can connect to the server from another device on the same network using an SSH client like PuTTY or the terminal. This lets you run your server headlessly without needing a monitor, keyboard, or mouse after the initial setup.
  1116. On the server:
  1117. - Run the following command:
  1118. ```
  1119. sudo apt install openssh-server
  1120. ```
  1121. - Start the SSH service:
  1122. ```
  1123. sudo systemctl start ssh
  1124. ```
  1125. - Enable the SSH service to start at boot:
  1126. ```
  1127. sudo systemctl enable ssh
  1128. ```
  1129. - Find the server's IP address:
  1130. ```
  1131. ip a
  1132. ```
  1133. On the client:
  1134. - Connect to the server using SSH:
  1135. ```
  1136. ssh <username>@<ip_address>
  1137. ```
  1138. > Replace `<username>` with your username and `<ip_address>` with the server's IP address.
  1139. > [!NOTE]
  1140. > If you expect to tunnel into your server often, I highly recommend following [this guide](https://www.raspberrypi.com/documentation/computers/remote-access.html#configure-ssh-without-a-password) to enable passwordless SSH using `ssh-keygen` and `ssh-copy-id`. It worked perfectly on my Debian system despite having been written for Raspberry Pi OS.
  1141. ## Firewall
  1142. Setting up a firewall is essential for securing your server. The Uncomplicated Firewall (UFW) is a simple and easy-to-use firewall for Linux. You can use UFW to allow or deny incoming and outgoing traffic to and from your server.
  1143. - Install UFW:
  1144. ```bash
  1145. sudo apt install ufw
  1146. ```
  1147. - Allow SSH, HTTPS, and HTTP to your local network:
  1148. ```bash
  1149. # Allow all <ip_range> hosts access to port <port>
  1150. sudo ufw allow from <ip_range> to any port <port> proto tcp
  1151. ```
  1152. Start with running the above command to open ports 22 (SSH), 80 (HTTP), and 443 (HTTPS) to our local network. Since we use the `app-net` Docker network for our containers, there's no need to open anything else up. Open ports up carefully, ideally only to specific IPs or to your local network. To allow a port for a specific IP, you can replace the IP range with a single IP and it'll work exactly the same way.
  1153. > [!TIP]
  1154. > You can find your local network's IP address range by running `ip route show`. The result will be something like this:
  1155. > ```
  1156. > me@my-cool-server:~$ ip route show
  1157. > default via <router_ip> dev enp3s0 proto dhcp src <server_ip> metric 100
  1158. > <network_ip_range> dev enp3s0 proto kernel scope link src <server_ip> metric 100
  1159. > # more routes
  1160. > ```
  1161. - Enable UFW:
  1162. ```bash
  1163. sudo ufw enable
  1164. ```
  1165. - Check the status of UFW:
  1166. ```bash
  1167. sudo ufw status verbose
  1168. ```
  1169. > [!WARNING]
  1170. > Enabling UFW without allowing access to port 22 will disrupt your existing SSH connections. If you run a headless setup, this means connecting a monitor to your server and then allowing SSH access through UFW. Be careful to ensure that this port is allowed when making changes to UFW's configuration.
  1171. Refer to [this guide](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-debian-10) for more information on setting up UFW.
  1172. ## Remote Access
  1173. This section covers the software required to allow secure and efficient access to the homelab environment. It involves leveraging Caddy as a reverse proxy engine, Cloudflare as a DNS provider, and Tailscale for the mesh VPN that provides built-in security due to firewall hole punching. It is highly recommended if you're looking at building a system that will be used professionally or even by family and friends.
  1174. 📖 See the full guide at [Remote Access](docs/remote-access.md).
  1175. ## Updating
  1176. Updating your system is a good idea to keep software running optimally and with the latest security patches. Updates to Ollama allow for inference from new model architectures and updates to Open WebUI enable new features like voice calling, function calling, pipelines, and more.
  1177. I've compiled steps to update these "primary function" installations in a standalone section because I think it'd be easier to come back to one section instead of hunting for update instructions in multiple subsections.
  1178. ### General
  1179. Upgrade Debian packages by running the following commands:
  1180. ```
  1181. sudo apt update
  1182. sudo apt upgrade
  1183. ```
  1184. ### Nvidia Drivers & CUDA
  1185. Follow Nvidia's guide [here](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Debian) to install the latest CUDA drivers.
  1186. > [!WARNING]
  1187. > Don't skip this step. Not installing the latest drivers after upgrading Debian packages will throw your installations out of sync, leading to broken functionality. When updating, target everything important at once. Also, rebooting after this step is a good idea to ensure that your system is operating as expected after upgrading these crucial drivers.
  1188. ### Docker Services
  1189. Most services in this guide run via Docker and can be updated using the same general pattern. Navigate to the directory containing the service's compose file or container and use the appropriate commands below.
  1190. **Docker Compose services**:
  1191. ```bash
  1192. cd <service-directory>
  1193. docker compose down
  1194. docker compose pull
  1195. docker compose up -d
  1196. ```
  1197. > Replace `<service-directory>` with the directory containing the `docker-compose.yaml` file (e.g., `mcp-proxy`, `mcpjungle`, `Kokoro-FastAPI`, `searxng`).
  1198. **Standalone containers** (llama-swap, Open WebUI):
  1199. For llama-swap, stop and remove the existing container, then re-run the `docker run` command from the [llama-swap section](#llama-swap):
  1200. ```bash
  1201. docker stop llama-swap
  1202. docker rm llama-swap
  1203. # Re-run the docker run command from the installation section
  1204. ```
  1205. For Open WebUI, use Watchtower to update:
  1206. ```bash
  1207. # One-time update
  1208. docker run --rm --volume /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once open-webui
  1209. # Or set up automatic updates
  1210. docker run -d --name watchtower --volume /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower open-webui
  1211. ```
  1212. **vLLM (Docker installation)**:
  1213. Simply re-run your `docker run` command and the latest Docker image will be pulled automatically.
  1214. ### Ollama
  1215. Rerun the command that installs Ollama - it acts as an updater too:
  1216. ```
  1217. curl -fsSL https://ollama.com/install.sh | sh
  1218. ```
  1219. ### llama.cpp
  1220. Enter your llama.cpp folder and run the following commands:
  1221. ```
  1222. cd llama.cpp
  1223. git pull
  1224. # Rebuild according to your setup - uncomment `-DGGML_CUDA=ON` for CUDA support
  1225. cmake -B build # -DGGML_CUDA=ON
  1226. cmake --build build --config Release
  1227. ```
  1228. ### vLLM
  1229. For a manual installation, enter your virtual environment and update via `pip`:
  1230. ```
  1231. source vllm/.venv/bin/activate
  1232. pip install vllm --upgrade
  1233. ```
  1234. ### ComfyUI
  1235. Navigate to the directory, pull the latest changes, and update dependencies:
  1236. ```
  1237. cd ComfyUI
  1238. git pull
  1239. source comfyui-env/bin/activate
  1240. pip install -r requirements.txt
  1241. ```
  1242. ## Troubleshooting
  1243. ### SearXNG
  1244. - **403 Forbidden on search requests**: If you receive a `403 Forbidden` response on search requests, there are two probable causes:
  1245. - **JSON format not enabled**: By default, SearXNG only allows `html` output. Thus, requests with `?format=json` return 403. Ensure that `json` is in the formats list in `core-config/settings.yml`:
  1246. ```yaml
  1247. search:
  1248. formats:
  1249. - html
  1250. - json
  1251. ```
  1252. - **Bot detection blocking**: The bot detection middleware blocks requests when it can't determine the client IP (no reverse proxy/`X-Forwarded-For` headers). Check logs with `docker compose logs core` and look for `X-Forwarded-For nor X-Real-IP header is set!`. For private instances, set `limiter: false` under `server:` in `settings.yml`.
  1253. - **Container not reachable on `app-net`**: If `http://searxng:8080` is unreachable from other containers, verify the alias:
  1254. ```bash
  1255. docker inspect searxng-core --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool
  1256. ```
  1257. The `app-net` entry should list `searxng` under `Aliases`. If missing, ensure the network block is added to `docker-compose.yml` and run `docker compose up -d`. Or, you could refer to it as `searxng-core` and leave aliases altogether - the choice is yours.
  1258. - **Port mapping changes both host and internal port**: Setting `SEARXNG_PORT=5050` in `.env` changes *both* the host port and the internal container port. If you change it, update any internal references from `searxng:8080` to `searxng:5050` accordingly.
  1259. - **Permission denied writing to `core-config/`**: The container runs as user `searxng` and takes ownership of the mounted volume. If you cannot write files to `core-config/`, fix ownership:
  1260. ```bash
  1261. sudo chown -R $USER:$USER searxng/core-config/
  1262. ```
  1263. Note that this will revert the next time you restart the container unless you populate `user: <user_id>:><group_id>` in the compose file (which I don't recommend doing because searxng is particular about directory ownership and throws warnings if you change it).
  1264. ### Docker
  1265. - **Volume mount permission denied**: Since we aim to use our user instead of root to run services, you may encounter permissions issues when trying to mount a volume that is not technically owned by our user. To fix this, you can do one of two things:
  1266. - Change the ownership of the directory:
  1267. ```bash
  1268. sudo chown -R $(id -u):$(id -g) /path/to/volume
  1269. ```
  1270. - Grant access to our user with access control lists (ACLs, in short):
  1271. ```bash
  1272. # Grant read/write to specific user
  1273. sudo setfacl -R -m u:$(id -u):rw /path/to/volume
  1274. # Grant to group
  1275. sudo setfacl -R -m g:$(id -g):rw /path/to/volume
  1276. ```
  1277. In case you don't have the `acl` package, install with `sudo apt update && sudo apt install acl`. The official package page can be found [here](https://packages.debian.org/bookworm/acl).
  1278. > Replace `/path/to/volume` with the desired path.
  1279. I like ACLs better - it solves the access issue cleanly without changing ownership unnecessarily and, in my experience, tends to break things less. However, if the resource is clearly meant to be owned by a specific user, changing ownership is preferred purely for separation of concerns. Gauge for yourself, both are completely acceptable options. Avoid changing permissions if your container is already up and running, though.
  1280. ### `ssh`
  1281. - **`ssh-copy-id` fails to set up passwordless SSH**: If you encounter an issue using `ssh-copy-id` to set up passwordless SSH, try running `ssh-keygen -t rsa` on the client before running `ssh-copy-id`. This generates the RSA key pair that `ssh-copy-id` needs to copy to the server.
  1282. ### Nvidia Drivers
  1283. - **`nvidia-smi` can't communicate with the driver**: Disable Secure Boot in the BIOS if you're having trouble with the Nvidia drivers not working. For me, all packages were at the latest versions and `nvidia-detect` was able to find my GPU correctly, but `nvidia-smi` kept returning the `NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver` error. [Disabling Secure Boot](https://askubuntu.com/a/927470) fixed this for me. Better practice than disabling Secure Boot is to sign the Nvidia drivers yourself but I didn't want to go through that process for a private server that can afford to have Secure Boot disabled.
  1284. - **`unknown or invalid runtime name: nvidia` error in Docker**: If you run into `docker: Error response from daemon: unknown or invalid runtime name: nvidia.`, you probably have `--runtime nvidia` in your Docker statement. This is meant for `nvidia-docker`, [which is deprecated now](https://stackoverflow.com/questions/52865988/nvidia-docker-unknown-runtime-specified-nvidia). Removing this flag from your command should get rid of this error.
  1285. ### Ollama
  1286. - **`could not connect to ollama app, is it running?`**: If you receive this error, your Ollama instance wasn't served properly. This could be because of a manual installation or the desire to use it at-will and not as a service. To run the Ollama server once, run:
  1287. ```
  1288. ollama serve
  1289. ```
  1290. Then, **in a new terminal**, you should be able to access your models regularly by running:
  1291. ```
  1292. ollama run <model>
  1293. ```
  1294. For detailed instructions on _manually_ configuring Ollama to run as a service (to run automatically at boot), read the official documentation [here](https://github.com/ollama/ollama/blob/main/docs/linux.md). You shouldn't need to do this unless your system faces restrictions using Ollama's automated installer.
  1295. - **Permission denied when editing `ollama.service`**: If you receive the `Failed to open "/etc/systemd/system/ollama.service.d/.#override.confb927ee3c846beff8": Permission denied` error from Ollama after running `systemctl edit ollama.service`, simply creating the file works to eliminate it. Use the following steps to edit the file.
    • Dismissed after review: Sets up something that runs again on its own (cron, launchd, systemd, startup entry or shell profile) and fetches from the network
    • Line 1,668 on GitHub
  1296. - Run:
  1297. ```
  1298. sudo mkdir -p /etc/systemd/system/ollama.service.d
  1299. sudo nano /etc/systemd/system/ollama.service.d/override.conf
  1300. ```
  1301. - Retry the remaining steps.
  1302. - **Can't connect to API endpoint**: If you still can't connect to your API endpoint, check your firewall settings. [This guide to UFW (Uncomplicated Firewall) on Debian](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-debian-10) is a good resource.
  1303. ### vLLM
  1304. - **`RuntimeError` downloading with `hf_transfer`**: If you encounter ```RuntimeError: An error occurred while downloading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.```, add `HF_HUB_ENABLE_HF_TRANSFER=0` to the `--env` flag after your HuggingFace Hub token. If this still doesn't fix the issue:
  1305. - Ensure your user has all the requisite permissions for HuggingFace to be able to write to the cache. To give read+write access over the HF cache to your user (and, thus, `huggingface-cli`), run:
  1306. ```
  1307. sudo chmod 777 ~/.cache/huggingface
  1308. sudo chmod 777 ~/.cache/huggingface/hub
  1309. ```
  1310. - Manually download a model via the HuggingFace CLI and specify `--download-dir=~/.cache/huggingface/hub` in the engine arguments. If your `.cache/huggingface` directory is being troublesome, specify another directory to the `--download-dir` in the engine arguments and remember to do the same with the `--local-dir` flag in any `huggingface-cli` commands.
  1311. ### Open WebUI
  1312. - **`llama runner process has terminated: signal: killed`**: If you encounter `Ollama: llama runner process has terminated: signal: killed`, check your `Advanced Parameters`, under `Settings > General > Advanced Parameters`. For me, bumping the context length past what certain models could handle was breaking the Ollama server. Leave it to the default (or higher, but make sure it's still under the limit for the model you're using) to fix this issue.
  1313. ## Monitoring
  1314. To monitor GPU usage, power draw, and temperature, you can use the `nvidia-smi` command. To monitor GPU usage, run:
  1315. ```
  1316. watch -n 1 nvidia-smi
  1317. ```
  1318. This will update the GPU usage every second without cluttering the terminal environment. Press `Ctrl+C` to exit.
  1319. ## Notes
  1320. This is my first foray into setting up a server and ever working with Linux so there may be better ways to do some of the steps. I will update this repository as I learn more.
  1321. ### Software
  1322. - I chose Debian because it is, apparently, one of the most stable Linux distros. I also went with an XFCE desktop environment because it is lightweight and I wasn't yet comfortable going full command line.
  1323. - Use a user for auto-login, don't log in as root unless for a specific reason.
  1324. - To switch to root in the command line without switching users, run `sudo -i`.
  1325. - If something using a Docker container doesn't work, try running `docker ps -a` to see if the container is running. If it isn't, try running `docker compose up -d` again. If it is and isn't working, try running `docker restart <container_id>` to restart the container.
  1326. - If something isn't working no matter what you do, try rebooting the server. It's a common solution to many problems. Try this before spending hours troubleshooting. Sigh.
  1327. - While it takes some time to get comfortable with, using an inference engine like llama.cpp and vLLM (as compared to Ollama) is really the way to go to squeeze the maximum performance out of your hardware. If you're reading this guide in the first place and haven't already thrown up your hands and used a cloud provider, it's a safe assumption that you care about the ethos of hosting all this stuff locally. Thus, get your experience as close to a cloud provider as it can be by optimizing your server.
  1328. ### Hardware
  1329. - The power draw of my EVGA FTW3 Ultra RTX 3090 was 350W at stock settings. I set the power limit to 250W and the performance decrease was negligible for my use case, which is primarily code completion in VS Code and the Q&A via chat.
  1330. - Using a power monitor, I measured the power draw of my server for multiple days - the running average is ~60W. The power can spike to 350W during prompt processing and token generation, but this only lasts for a few seconds. For the remainder of the generation time, it tended to stay at the 250W power limit and dropped back to the average power draw after the model wasn't in use for about 20 seconds.
  1331. - Ensure your power supply has enough headroom for transient spikes (particularly in multi GPU setups) or you may face random shutdowns. Your GPU can blow past its rated power draw and also any software limit you set for it based on the chip's actual draw. I usually aim for +50% of my setup's estimated total power draw.
  1332. ## References
  1333. Adding user to `sudo` group:
  1334. - https://askubuntu.com/questions/168280/how-do-i-grant-sudo-privileges-to-an-existing-user
  1335. Downloading Nvidia drivers:
  1336. - https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Debian
  1337. - https://wiki.debian.org/NvidiaGraphicsDrivers
  1338. Downloading AMD drivers:
  1339. - https://wiki.debian.org/AtiHowTo
  1340. Secure Boot:
  1341. - https://askubuntu.com/a/927470
  1342. Monitoring GPU usage, power draw:
  1343. - https://unix.stackexchange.com/questions/38560/gpu-usage-monitoring-cuda/78203#78203
  1344. Passwordless `sudo`:
  1345. - https://stackoverflow.com/questions/25215604/use-sudo-without-password-inside-a-script
  1346. - https://www.reddit.com/r/Fedora/comments/11lh9nn/set_nvidia_gpu_power_and_temp_limit_on_boot/
  1347. - https://askubuntu.com/questions/100051/why-is-sudoers-nopasswd-option-not-working
  1348. Auto-login:
  1349. - https://forums.debian.net/viewtopic.php?t=149849
  1350. - https://wiki.archlinux.org/title/LightDM#Enabling_autologin
  1351. Expose Ollama to LAN:
  1352. - https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux
  1353. - https://github.com/ollama/ollama/issues/703
  1354. Firewall:
  1355. - https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-debian-10
  1356. Passwordless `ssh`:
  1357. - https://www.raspberrypi.com/documentation/computers/remote-access.html#configure-ssh-without-a-password
  1358. Adding CUDA to PATH:
  1359. - https://askubuntu.com/questions/885610/nvcc-version-command-says-nvcc-is-not-installed
  1360. Docs:
  1361. - [Debian](https://www.debian.org/releases/buster/amd64/)
  1362. - [Docker](https://docs.docker.com/engine/install/debian/)
  1363. - [Ollama](https://github.com/ollama/ollama/blob/main/docs/api.md)
  1364. - [vLLM](https://docs.vllm.ai/en/stable/index.html)
  1365. - [Open WebUI](https://github.com/open-webui/open-webui)
  1366. - [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
  1367. ## Acknowledgements
  1368. Cheers to all the fantastic work done by the open-source community. This guide wouldn't exist without the effort of the many contributors to the projects and guides referenced here. To stay up-to-date on the latest developments in the field of machine learning, LLMs, and other vision/speech models, check out [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/). To stay in touch with (or fall down the rabbit hole of) the world of self-hosted apps out there for your new server, check out [r/selfhosted](https://www.reddit.com/r/selfhosted).
  1369. > [!NOTE]
  1370. > Please star any projects you find useful and consider contributing to them if you can. Stars on this guide would also be appreciated if you found it helpful, as it helps others find it too.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.