Skip to content
Back to search

freellmpool

Free LLM gateway: 22 LLM providers, 178 enabled chat routes, 431 cataloged chat models; keyless start when available.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/0xzr--freellmpool/badge.svg)](/listings/0xzr--freellmpool)

HTML

<a href="/listings/0xzr--freellmpool"><img src="/api/listings/0xzr--freellmpool/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Fresh install to first free-model reply is measured at about 19 seconds under the 30-second target on a clean Linux/Python 3.12 environment, with no API keys when a keyless provider is up:

# One command when uv is installed; no checkout or manually managed virtualenv:
uvx freellmpool ask --max-tokens 32 "Reply with one short sentence: freellmpool is ready."

Portable virtual-environment path:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install freellmpool
freellmpool ask --max-tokens 32 "Reply with one short sentence: freellmpool is ready."

CI runs the same path from this checkout with `FREELLMPOOL_QUICKSTART_PACKAGE=. scripts/quickstart-test.sh`.

The catalog covers provider-operated free tiers, keyless routes, and a few explicit finite-trial or disabled candidates. Eligibility and limits differ by provider. freellmpool automatically uses only enabled routes you can access, fails over when one is rate limited or down, and tracks local per-day usage.

Several providers (Pollinations, OVHcloud, and Kilo Gateway) need no API key, and LLM7 works without one, so the quickstart can answer without signup when a keyless provider is available.

To inspect your local provider keys, agent CLIs, proxy config, and Tailscale state before wiring tools, run the print-only init wizard:

freellmpool init --yes
freellmpool init --yes --agent opencode
freellmpool init --yes --agent metaswarm --tailnet

Add keys for the other providers to unlock more models and higher limits.

Local runtimes (explicit, pin-only)

LM Studio, Ollama, and llama.cpp can be previewed from a fixed list of literal loopback endpoints without scanning your LAN or processes:

freellmpool local discover
freellmpool local discover --name lm_studio
freellmpool local import --name lm_studio --yes
freellmpool ask --providers local_lm_studio --model "<model-from-discover>" "hello"

Discovery is read-only. Import is a separate affirmative step, writes no credential, and keeps every imported model out of automatic routing (`auto = false`). `freellmpool local remove local_lm_studio --yes` reverses only the block managed by the importer. Custom endpoints must be canonical literal loopback URLs such as `http://127.0.0.1:1234/v1`; hostnames, LAN addresses, redirects, and broad network scans are rejected.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 757 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # freellmpool
  2. <!-- mcp-name: io.github.0xzr/freellmpool -->
  3. ![freellmpool tokenmax terminal demo](assets/demo.svg)
  4. ![178 enabled chat routes, 23 LLM providers cataloged, keyless start when available](assets/tokenmax-results.svg)
  5. freellmpool catalogs 23 LLM providers as distinct groups spanning recurring
  6. free tiers, keyless endpoints, finite trials, pin-only routes, and disabled
  7. candidates. It exposes 178 enabled chat routes and 443 cataloged chat models,
  8. and automatically pools only
  9. enabled routes you can access behind one OpenAI-compatible endpoint — as a CLI,
  10. a Python library, or a local proxy. It can start without credentials when an
  11. enabled keyless route is available.
  12. [![PyPI](https://img.shields.io/pypi/v/freellmpool.svg)](https://pypi.org/project/freellmpool/)
  13. [![CI](https://github.com/0xzr/freellmpool/actions/workflows/ci.yml/badge.svg)](https://github.com/0xzr/freellmpool/actions/workflows/ci.yml)
  14. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
  15. [![Website](https://img.shields.io/badge/docs-0xzr.github.io%2Ffreellmpool-6ea8ff)](https://0xzr.github.io/freellmpool/)
  16. [FAQ](FAQ.md): where prompts go, ToS posture, failover, bans, and comparisons.
  17. ## Release and distribution status
  18. - **Latest release: 0.13.0.** The GitHub release and PyPI package are both
  19. 0.13.0; `pip install freellmpool` and `uvx freellmpool` install the audited
  20. provider catalog, bounded streaming and sentinel hardening, Hermes profile,
  21. proxy readiness/provider APIs, `spread` routing, and OpenCode
  22. registry-readiness hardening.
  23. - **Registry publication status: pending.** `opencode-freellmpool` and
  24. `opencode-freellmpool-tui` are tested but not published on npm as of
  25. 2026-08-29. Use their repository-local installation instructions for now.
  26. ## 30-second quickstart
  27. Fresh install to first free-model reply is measured at about 19 seconds under
  28. the 30-second target on a clean Linux/Python 3.12 environment, with no API keys
  29. when a keyless provider is up:
  30. ```bash
  31. # One command when uv is installed; no checkout or manually managed virtualenv:
  32. uvx freellmpool ask --max-tokens 32 "Reply with one short sentence: freellmpool is ready."
  33. ```
  34. Portable virtual-environment path:
  35. ```bash
  36. python3 -m venv .venv
  37. . .venv/bin/activate
  38. python -m pip install --upgrade pip
  39. python -m pip install freellmpool
  40. freellmpool ask --max-tokens 32 "Reply with one short sentence: freellmpool is ready."
  41. ```
  42. CI runs the same path from this checkout with
  43. `FREELLMPOOL_QUICKSTART_PACKAGE=. scripts/quickstart-test.sh`.
  44. The catalog covers provider-operated free tiers, keyless routes, and a few
  45. explicit finite-trial or disabled candidates. Eligibility and limits differ by
  46. provider. freellmpool automatically uses only enabled routes you can access,
  47. fails over when one is rate limited or down, and tracks local per-day usage.
  48. Several providers (Pollinations, OVHcloud, and Kilo Gateway) need no API key,
  49. and LLM7 works without one, so the quickstart can answer without signup when a
  50. keyless provider is available.
  51. To inspect your local provider keys, agent CLIs, proxy config, and Tailscale
  52. state before wiring tools, run the print-only init wizard:
  53. ```bash
  54. freellmpool init --yes
  55. freellmpool init --yes --agent opencode
  56. freellmpool init --yes --agent metaswarm --tailnet
  57. ```
  58. Add keys for the other providers to unlock more models and higher limits.
  59. ### Local runtimes (explicit, pin-only)
  60. LM Studio, Ollama, and llama.cpp can be previewed from a fixed list of literal
  61. loopback endpoints without scanning your LAN or processes:
  62. ```bash
  63. freellmpool local discover
  64. freellmpool local discover --name lm_studio
  65. freellmpool local import --name lm_studio --yes
  66. freellmpool ask --providers local_lm_studio --model "<model-from-discover>" "hello"
  67. ```
  68. Discovery is read-only. Import is a separate affirmative step, writes no
  69. credential, and keeps every imported model out of automatic routing
  70. (`auto = false`). `freellmpool local remove local_lm_studio --yes` reverses only
  71. the block managed by the importer. Custom endpoints must be canonical literal
  72. loopback URLs such as `http://127.0.0.1:1234/v1`; hostnames, LAN addresses,
  73. redirects, and broad network scans are rejected.
  74. ## First-run setup with `freellmpool init`
  75. `freellmpool init` inspects provider keys, installed agent CLIs, Tailscale
  76. state, and proxy config, then prints one copy-pastable next step without editing
  77. files. Run it detect-only first:
  78. ```bash
  79. freellmpool init --yes
  80. ```
  81. `--json` emits the same detection as versioned JSON for scripts and agents.
  82. ### Tailnet / remote agent gateway
  83. Serve the proxy on your Tailscale 100.x address with a generated API key:
  84. ```bash
  85. freellmpool tailnet serve --port 8080
  86. ```
  87. From a remote machine:
  88. ```bash
  89. freellmpool tailnet connect <tailnet-ip> --port 8080
  90. ```
  91. Both sides support `--api-key <shared-secret>` if you want to pin a key instead
  92. of using a generated token. Tailnet serving requires auth by default; do not
  93. run unauthenticated over non-loopback interfaces.
  94. ### Metaswarm agent lanes
  95. This project uses one Umans/Kimi K2.7 worker lane, one MiniMax M3 lane, Codex as
  96. escalation, and Claude Opus only for final pre-ship review. The installable
  97. Metaswarm profile mirrors that posture: one free/cheap worker lane through the
  98. local proxy, one larger freellmpool reviewer lane, and Codex/Opus as explicit
  99. user-owned paid escalation/final-review lanes only (never silent).
  100. ```bash
  101. freellmpool init --yes --agent metaswarm --tailnet
  102. freellmpool profile install metaswarm
  103. freellmpool tailnet serve --port 8080
  104. freellmpool profile doctor metaswarm --dry-run
  105. ```
  106. ## Run a coding agent on free models
  107. freellmpool's proxy speaks the OpenAI API and includes an experimental
  108. Anthropic-compatible path, so coding agents can run against pooled free tiers —
  109. just point them at the proxy:
  110. ```bash
  111. freellmpool proxy # starts http://localhost:8080
  112. freellmpool code claude # prints the one-line setup for Claude Code
  113. freellmpool profile list # richer installable profiles
  114. freellmpool profile show metaswarm # Tailnet-aware Metaswarm profile
  115. freellmpool profile install hermes # Hermes custom-endpoint config
  116. # (also: codex, aider, cline, continue, cursor, hermes, opencode, metaswarm)
  117. ```
  118. The Hermes profile prints (and never writes) this supported custom-endpoint
  119. block; `hermes model` provides the interactive equivalent:
  120. ```yaml
  121. model:
  122. provider: custom
  123. default: quality
  124. base_url: http://localhost:8080/v1
  125. api_key: anything
  126. ```
  127. Claude Code gateway mode can also be launched directly:
  128. ```bash
  129. ANTHROPIC_BASE_URL=http://localhost:8080 \
  130. ANTHROPIC_AUTH_TOKEN=dummy \
  131. ANTHROPIC_API_KEY=dummy \
  132. ANTHROPIC_MODEL=auto \
  133. ANTHROPIC_SMALL_FAST_MODEL=auto \
  134. CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 \
  135. claude
  136. ```
  137. Existing OpenAI-compatible apps work the same way: set
  138. `OPENAI_BASE_URL=http://localhost:8080/v1` and keep your code unchanged.
  139. Anthropic-compatible tools can use the experimental bridge with
  140. `ANTHROPIC_BASE_URL=http://localhost:8080`.
  141. Text-only `/v1/responses` and `/v1/messages` requests stream incrementally from
  142. the selected provider. Tool calls and richer content intentionally stay on the
  143. buffered compatibility path. The browser dashboard and playground share one
  144. public, data-free shell; when proxy auth is enabled it prompts for the bearer
  145. token and keeps it only in page memory while protected status, inventory,
  146. model, and battle calls continue to require the `Authorization` header.
  147. **OpenCode** gets a deeper integration in 0.12.0: a live in-editor **dashboard** (routing mode,
  148. estimated savings, tokens served free, provider race, latency), per-request
  149. **agent routing** via the model picker (`freellmpool/agent|spread|auto|fast|quality|fair`), and `freellmpool_status`
  150. / `freellmpool_models` tools — see [integrations/opencode-tui](integrations/opencode-tui)
  151. and the [guide](https://0xzr.github.io/freellmpool/run-opencode-on-free-models.html).
  152. The plugin registers its routing aliases automatically on supported OpenCode
  153. versions without rewriting user configuration. Restart OpenCode and check
  154. `opencode models freellmpool`.
  155. The package tarballs are validated in CI, but npm publication remains pending;
  156. the linked local-file instructions remain the working install path.
  157. **New in 0.11:** capacity tools — `freellmpool capacity status` shows which free
  158. tiers are usable right now, `freellmpool providers health` live-probes them, and
  159. `freellmpool keys add` walks you through configuring more (see
  160. [Capacity & provider health](#capacity--provider-health) and
  161. [docs/CAPACITY.md](docs/CAPACITY.md)).
  162. **New in 0.10:** an async API (`AsyncPool`), an MCP server (`freellmpool mcp`),
  163. latency-aware routing with `freellmpool benchmark`, observability hooks, and a
  164. plugin system for custom providers. See the [changelog](CHANGELOG.md).
  165. ## Install
  166. ```bash
  167. pip install freellmpool # or: pipx install freellmpool
  168. ```
  169. Only dependency is `httpx`. Python 3.11+.
  170. ### Docker and Open WebUI
  171. The published container runs the local proxy as an unprivileged user. Persist
  172. quota, stats, route-health, cache, and configuration state in a named volume:
  173. ```bash
  174. docker volume create freellmpool-data
  175. docker run --rm -p 127.0.0.1:8080:8080 \
  176. --volume freellmpool-data:/home/freellmpool/.config/freellmpool \
  177. --env GROQ_API_KEY \
  178. ghcr.io/0xzr/freellmpool:0.13.0
  179. ```
  180. Omit `--env GROQ_API_KEY` if you want a credential-free start; the proxy can
  181. answer only while an enabled keyless route is available. Keep the published
  182. port on loopback. If you deliberately expose it, set
  183. `FREELLMPOOL_PROXY_KEY` and require that Bearer token from clients.
  184. For the bundled Open WebUI stack, place any provider credentials in `.env` and
  185. run `docker compose up -d`. Compose waits for the proxy health check and keeps
  186. both freellmpool and Open WebUI state in named volumes; open
  187. `http://localhost:3000` after both services are healthy.
  188. ## Command line
  189. ```bash
  190. freellmpool ask "Write a haiku about sqlite"
  191. git diff | freellmpool ask "Write a commit message for this"
  192. freellmpool tokenmax "Hardest question you've got" # 🌈 blast models, print answers, optional synthesis
  193. freellmpool providers # which providers are configured
  194. freellmpool models # cataloged provider/model ids
  195. freellmpool stats # lifetime tokens served free + estimated cost avoided
  196. freellmpool badge -o badge.svg # a shareable SVG badge of that total
  197. freellmpool badge --summary -o summary.svg # a larger usage summary card
  198. ```
  199. `freellmpool tokenmax` is the tongue-in-cheek maximum-effort mode: it fans your
  200. prompt out to many available models at once and prints each answer. The CLI adds
  201. a synthesized verdict by default unless you pass `--no-synthesize`; the MCP tool
  202. returns the model answers for the calling agent to synthesize. (See
  203. [docs/MCP.md](docs/MCP.md).)
  204. `freellmpool stats` is a running, **persistent** lifetime total (it survives restarts
  205. and upgrades). Embed `freellmpool badge` in a README, or use
  206. `freellmpool badge --summary -o summary.svg` for a larger card with tokens,
  207. requests, estimated savings, and provider mix. The proxy can serve both
  208. `/badge.svg` and `/summary.svg` live when `FREELLMPOOL_PUBLIC_BADGE=1` makes
  209. public badges embeddable.
  210. Pin a provider or model; common OpenAI/Anthropic model names are mapped to a free
  211. equivalent so existing scripts keep working:
  212. ```bash
  213. freellmpool ask -m groq/openai/gpt-oss-20b "hi"
  214. freellmpool ask -p cerebras,groq "hi"
  215. freellmpool ask -m gpt-4o-mini "hi" # routed to a free model
  216. ```
  217. ### Roles
  218. `freellmpool roles` lists ask-role presets (`coder`, `critic`, `summarizer`,
  219. `grounded-reader`, `long-context`, `cheap`, `fast`, `second-opinion`, ...). Each
  220. role sets routing, token budget, temperature, task intent, and system-prompt hints
  221. without inventing a second routing engine. Explicit flags (`--model`, `--providers`,
  222. `--routing`, `--task`, `--max-tokens`) win over role defaults.
  223. ```bash
  224. freellmpool ask --role coder "write a pytest for this function"
  225. freellmpool ask --role grounded-reader "read this Markdown file"
  226. freellmpool ask --routing quality --task general "ignore automatic task classification"
  227. FREELLMPOOL_MODE=wise freellmpool ask --role cheap "summarize this patch"
  228. ```
  229. ## As a proxy
  230. Run a local server that speaks the OpenAI API, then point any OpenAI-compatible
  231. tool at it. On loopback, any placeholder API key works unless you configured
  232. `FREELLMPOOL_PROXY_KEY` or passed `--api-key`; Tailnet/LAN serving requires a
  233. real proxy bearer token by default.
  234. ```bash
  235. freellmpool proxy
  236. export OPENAI_BASE_URL=http://localhost:8080/v1
  237. export OPENAI_API_KEY=unused
  238. ```
  239. ```python
  240. from openai import OpenAI
  241. client = OpenAI()
  242. print(client.chat.completions.create(
  243. model="auto",
  244. messages=[{"role": "user", "content": "hi"}],
  245. ).choices[0].message.content)
  246. # audio → text (Whisper), same client:
  247. print(client.audio.transcriptions.create(
  248. model="auto", file=open("audio.mp3", "rb"),
  249. ).text)
  250. ```
  251. Or with `curl` (multipart upload):
  252. ```bash
  253. curl -s http://localhost:8080/v1/audio/transcriptions \
  254. -F file=@audio.mp3 -F model=auto
  255. ```
  256. The proxy also implements the OpenAI Responses API (for the Codex CLI) and an
  257. experimental Anthropic Messages API path (for Claude Code), so coding agents can
  258. run on free models too. `freellmpool code <agent>` prints the exact setup, while
  259. `freellmpool profile install <agent>` prints the fuller copy-pastable profile
  260. without mutating third-party config:
  261. ```bash
  262. freellmpool code aider # also: claude, codex, cline, continue, cursor, hermes, opencode
  263. freellmpool profile show opencode
  264. freellmpool profile doctor opencode --dry-run
  265. ```
  266. Release 0.12.0 proxy surfaces:
  267. - `/v1/chat/completions` — OpenAI-compatible chat, token streaming, tool calling.
  268. - `/v1/responses` — Responses API shim with incremental text streaming; tools
  269. and rich content use the buffered compatibility path.
  270. - `/v1/messages` — experimental Anthropic-compatible Messages path with the
  271. same incremental-text / buffered-tool split.
  272. - `/v1/embeddings` and `/v1/audio/transcriptions` — OpenAI-compatible embedding
  273. and Whisper-style multipart transcription.
  274. - `/v1/models` — routing aliases plus concrete `provider/model` ids.
  275. - `/v1/models?ready=true` — the same shape, limited to locally ready targets.
  276. - `/v1/providers` — authenticated, secret-free provider/model readiness details.
  277. - `/freellmpool/battle` and `/playground` — bounded browser/JSON model comparisons.
  278. - `/healthz` and `/livez` — public process liveness aliases.
  279. - `/readyz` — public advisory local-capacity readiness (200 when at least one
  280. provider is ready, otherwise 503); it never live-probes an upstream.
  281. - `/dashboard`, `/status`, `/badge.svg`, and `/summary.svg` — local operations surfaces.
  282. `/`, `/dashboard`, and `/playground` serve one public, data-free shell. When a
  283. proxy key is set, the shell prompts for it and keeps it only in page memory;
  284. status, provider/model inventory, battle, and model API calls remain header-auth
  285. protected. Liveness/readiness stay public for orchestrator probes; readiness is
  286. a local quota/cooldown snapshot, not an upstream health guarantee.
  287. Setup snippets for specific tools are in [docs/INTEGRATIONS.md](docs/INTEGRATIONS.md)
  288. and [docs/AGENTS.md](docs/AGENTS.md). The repo also includes an experimental
  289. [metaswarm review adapter](integrations/metaswarm) for using `freellmpool` as an
  290. external-tools reviewer/second opinion. `freellmpool profile show metaswarm`
  291. documents a free/cheap worker lane, a larger reviewer lane, Tailnet client setup,
  292. and paid Codex/Opus lanes as explicit user-owned escalation paths only.
  293. ## As a library
  294. ```python
  295. from freellmpool import Pool
  296. pool = Pool.from_default_config()
  297. reply = pool.ask("Summarize the plot of Hamlet in 20 words.")
  298. print(reply.text, "—", reply.provider_id)
  299. vectors = pool.embed(["first document", "second document"]).vectors
  300. with open("audio.mp3", "rb") as f:
  301. text = pool.transcribe(f.read(), "audio.mp3").text # Whisper, failover across providers
  302. ```
  303. Async is the same API with `await`:
  304. ```python
  305. from freellmpool import AsyncPool
  306. async with AsyncPool.from_default_config() as pool:
  307. reply = await pool.aask("Summarize the plot of Hamlet in 20 words.")
  308. ```
  309. Pass `on_event=...` to either pool to receive structured routing/cache events
  310. (`attempt`/`success`/`error`/`cooldown`/`cache_hit`/`cache_miss`/`exhausted`) for logging or tracing. Add
  311. your own endpoint with `register_provider(...)`, or a new request shape with
  312. `register_adapter(name, fn)`.
  313. ## Benchmark your providers
  314. `freellmpool benchmark` times one call per configured provider and prints
  315. latency and success, so you can see which of your free tiers are fastest right
  316. now. The router learns the same latency/success signal from real traffic as it
  317. runs; set `FREELLMPOOL_ROUTING=fast` to prefer the lowest-latency provider
  318. instead of the default least-used-first.
  319. ```
  320. $ freellmpool benchmark
  321. provider/model status latency note
  322. cerebras/gpt-oss-120b ok 180 ms 6 tok
  323. groq/openai/gpt-oss-20b ok 240 ms 6 tok
  324. ovh/Meta-Llama-3_3-70B-Instruct FAIL - HTTP 429
  325. ```
  326. ## Capacity & provider health
  327. Free tiers drift through the day — keys expire, providers go down, daily caps
  328. fill. These commands tell you what's usable right now and what to set up next:
  329. ```bash
  330. freellmpool capacity status --target 5 # who's healthy / near quota / missing a key
  331. freellmpool capacity status --refresh # explicitly refresh advisory provider metadata
  332. freellmpool quota-wise status # local headroom + recommended mode
  333. freellmpool providers health # send one tiny request to each, time it
  334. freellmpool keys checklist --target 5 # which keys to add to reach N healthy providers
  335. freellmpool keys add groq # configure a key (and record metadata)
  336. ```
  337. Protocol support is verified separately from basic provider health. Run
  338. `freellmpool conformance run` to check chat, streaming, tools, JSON object/schema,
  339. vision, Responses, and Anthropic Messages with bounded synthetic requests, then inspect
  340. the sanitized evidence with `freellmpool conformance status --json`.
  341. Feature-specific auto-routing uses only verified targets; an exact
  342. provider/model pin is the explicit override. Full operator and privacy contract:
  343. [docs/PROTOCOL_CONFORMANCE.md](docs/PROTOCOL_CONFORMANCE.md).
  344. `capacity status` is offline and cache-first: it reads your catalog, environment, and
  345. per-day quota counters and labels each provider `healthy`, `low_quota`,
  346. `exhausted`, `invalid_key`, or `missing`. Cached metadata from the advisory
  347. external catalog ([mnfst/awesome-free-llm-apis](https://github.com/mnfst/awesome-free-llm-apis))
  348. can suggest free providers you could add; only an explicit
  349. `freellmpool capacity status --refresh` or `freellmpool catalog sync` uses the
  350. network for that metadata. It remains advisory only; your `providers.toml`
  351. stays the source of truth for routing. `keys add <name>` can import a
  352. suggested provider from that catalog or create an OpenAI-compatible stub and
  353. autodiscover its models. The proxy `/dashboard` shows the same capacity at a
  354. glance. Full reference: [docs/CAPACITY.md](docs/CAPACITY.md).
  355. `FREELLMPOOL_MODE=wise` is the conservative quota mode: `ask` defaults to a
  356. smaller output budget and spread routing, `tokenmax` narrows its default fan-out,
  357. and broad multi-model calls require confirmation unless you pass `--yes`.
  358. Per-command `--mode normal|wise` overrides the environment, and
  359. `[settings] mode = "wise"` works from `config.toml`. The `conserve` role is a
  360. quota-conscious shorthand for small, spread-routed answers.
  361. For a bounded second opinion instead of a full `tokenmax` blast:
  362. ```bash
  363. freellmpool ask --second-opinion --opinions 3 "is this implementation plan sound?"
  364. freellmpool ask --role second-opinion --synthesize "which release note is clearer?"
  365. ```
  366. The shared panel asks a few diverse providers, keeps individual failures visible,
  367. and can append a non-fatal synthesis when you pass `--synthesize`.
  368. For a side-by-side comparison you can inspect in the terminal or local browser:
  369. ```bash
  370. freellmpool battle "which changelog entry is clearer?" --synthesize
  371. freellmpool proxy --port 8080
  372. freellmpool playground --port 8080
  373. ```
  374. Bundled recipes wrap common workflows in JSON files you can inspect and run:
  375. ```bash
  376. freellmpool recipe list
  377. freellmpool recipe run second-opinion "is this launch plan clear?" --synthesize
  378. freellmpool recipe run pr-review --input patch.diff
  379. freellmpool recipe run repo-summary --path 'src/freellmpool/*.py'
  380. freellmpool recipe run metaswarm-worker-review --input worker.md --validation-output-file validation.txt
  381. ```
  382. Recipes use the same role presets and shared panel helper as `ask` and `battle`;
  383. there is no separate routing engine.
  384. ### Local foreground job queue
  385. For slow, quota-aware work that should not block a live session, queue jobs to
  386. an append-only JSONL log under your config dir (override with
  387. `FREELLMPOOL_JOBS_PATH`). The queue is foreground-only: `jobs run` processes
  388. one job at a time and records started/completed/failed/cancelled events.
  389. Completed ask jobs keep their output in the job log; completed recipe jobs also
  390. write run records and Markdown reports via the same report helpers used by
  391. `freellmpool report`.
  392. ```bash
  393. # queue a recipe job
  394. freellmpool jobs add --recipe pr-review --input patch.diff
  395. # queue an ask job with a role preset
  396. freellmpool jobs add --role summarizer "summarize the latest changelog"
  397. freellmpool jobs list # replayed state (idempotent across restarts)
  398. freellmpool jobs watch # one-shot refresh render, no daemon
  399. freellmpool jobs run --dry-run # print execution order, mutate nothing
  400. freellmpool jobs run --max-failures 2 # halt after N consecutive failures
  401. freellmpool jobs cancel <job-id> # append a cancel tombstone, not a mutation
  402. freellmpool report list
  403. freellmpool report last --markdown
  404. freellmpool report last --html --path
  405. freellmpool cost show <run-id>
  406. ```
  407. Cancellation is a new tombstone event, not a re-write of the earlier queued
  408. record — a crash before `jobs run` finishes still leaves the queue
  409. replayable, and cancelled jobs stay cancelled after restart. Duplicate
  410. submissions create distinct jobs; pass `--dedupe` to reject re-submission of
  411. the same recipe or role while a job is still pending.
  412. ## As an MCP server
  413. `freellmpool mcp` runs a Model Context Protocol server over newline-delimited
  414. JSON-RPC stdio, so Claude Desktop, Claude Code, or Cursor can hand subtasks to
  415. free models. It exposes ask, panel/second-opinion, battle, recipe, roles,
  416. Tailnet-info, quota-wise, route-preview, models, quota, stats, and `tokenmax`
  417. tools. See [docs/MCP.md](docs/MCP.md). A [`server.json`](server.json) is included
  418. for the [MCP registry](https://registry.modelcontextprotocol.io/).
  419. ## In Simon Willison's `llm` CLI
  420. There's a plugin: `llm install llm-freellmpool` → `llm -m freellmpool "..."`.
  421. It can answer without an API key while an enabled keyless route is available.
  422. Source: [plugins/llm-freellmpool](plugins/llm-freellmpool).
  423. ## Provider keys
  424. freellmpool reads keys from the environment and uses whatever is set. No key is
  425. required while an enabled keyless route is available; credentials expand the
  426. set of accessible routes. Step-by-step signup links, including verification and
  427. payment-method caveats, are in
  428. [docs/ACCOUNTS.md](docs/ACCOUNTS.md).
  429. | Provider | Env var | Notes |
  430. |---|---|---|
  431. | Pollinations | — | no key needed |
  432. | OVHcloud | — | no key needed (anonymous tier) |
  433. | Kilo Gateway | — | no key needed |
  434. | LLM7 | `LLM7_API_KEY` | optional |
  435. | Groq | `GROQ_API_KEY` | current free-plan routes; exact limits vary by model |
  436. | Cerebras | `CEREBRAS_API_KEY` | finite $5 trial; current routes are explicit pins, not automatic recurring capacity |
  437. | NVIDIA NIM | `NVIDIA_API_KEY` | |
  438. | OpenRouter | `OPENROUTER_API_KEY` | free models |
  439. | Requesty | `REQUESTY_API_KEY` | zero-priced models only; 200 requests/day per account, no card |
  440. | Google Gemini | `GEMINI_API_KEY` | |
  441. | Cloudflare | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` | |
  442. | Hugging Face router | `HF_TOKEN` | router free tier |
  443. | OpenCode Zen | — | cataloged, disabled by default pending opt-in |
  444. | Aion Labs | `AION_API_KEY` | 20K free tokens/day, no card |
  445. | ModelScope API Inference | `MODELSCOPE_API_KEY` | 2,000 free calls/day |
  446. | Morph | `MORPH_API_KEY` | current priced aliases are retained disabled for explicit future verification |
  447. | Vercel AI Gateway | `AI_GATEWAY_API_KEY` | automatic routing is limited to the publicly price-verified zero-price Poolside route; other routes are pin-only or disabled; set a gateway-side budget |
  448. | SiliconFlow | `SILICONFLOW_API_KEY` | free models; identity verification required |
  449. | Mistral, Cohere, SambaNova, Z.ai, Ollama Cloud | see `.env.example` | |
  450. A `config.toml` (see [config.toml.example](config.toml.example)) can hold keys,
  451. model aliases, and settings instead of env vars.
  452. ## Local diagnostics and operations
  453. Run `freellmpool doctor` for a no-network local check of package version, config
  454. paths, configured provider count, routing mode, quota/cache locations, external
  455. catalog cache age, and bundled catalog validity.
  456. Response caching is off unless `FREELLMPOOL_CACHE_TTL` (seconds) or
  457. `[settings] cache_ttl` is positive. When enabled, cache rows live in SQLite with
  458. WAL mode and TTL pruning; `FREELLMPOOL_CACHE_MAX_ENTRIES` caps retained rows
  459. (default `10000`, set `0` to disable size pruning).
  460. Quota counters are written immediately by default. Long-running proxy/MCP
  461. processes can reduce file churn with `FREELLMPOOL_QUOTA_FLUSH_EVERY=N`, which
  462. batches up to `N` successful requests before flushing. Shutdown paths and
  463. `quota.snapshot()` flush pending counts, so dashboards and process exits still
  464. see current totals.
  465. Recent per-route health is persisted at
  466. `~/.config/freellmpool/route_health.json` (override with
  467. `FREELLMPOOL_HEALTH_FILE`). The bounded, atomic state contains route names,
  468. timings, counters, and normalized failure classes only—never prompts, responses,
  469. headers, or credentials. `/status` reports circuit state, sample age, and reset
  470. time for each enabled route.
  471. ## How routing works
  472. For each request, freellmpool builds the list of `(provider, model)` pairs you
  473. have access to, then orders providers least-used-first and picks a least-used
  474. model inside that provider. This keeps providers with large catalogs, like
  475. NVIDIA, from receiving more traffic only because they expose more models. A
  476. provider that returns a 429 is set aside for its advertised `Retry-After` or
  477. rate-limit reset window. Repeated availability failures open a per-route circuit;
  478. after cooldown, one half-open probe can restore the route. Daily counts are kept
  479. in `~/.config/freellmpool/quota.json` and reset at UTC midnight.
  480. Every call records latency and success per model target. A provider whose targets
  481. are currently failing sinks to the back automatically; with
  482. `FREELLMPOOL_ROUTING=fast` the fastest measured provider goes first instead.
  483. `FREELLMPOOL_ROUTING=fair` spreads requests across providers to preserve daily
  484. quota. `freellmpool benchmark` warms these metrics on demand. To restore the old
  485. per-model balancing behavior, set `FREELLMPOOL_ROUTING=legacy` or
  486. `FREELLMPOOL_ROUTING=model` (or `FREELLMPOOL_ROUTING=model-fast` for the old
  487. per-model fastest-first ordering).
  488. **Quality routing (`FREELLMPOOL_ROUTING=quality`).** Free tiers' strongest models
  489. have the smallest daily caps, so a naive pool gets weaker as the day fills. Quality
  490. routing matches each prompt's *difficulty* to each model's *capability*: hard
  491. prompts (long input, code, reasoning cues) go to the strongest available model, and
  492. easy ones go to lightweight models — which rations scarce strong-model quota so the
  493. pool stays sharp for longer. It also recognizes high-confidence grounded Markdown
  494. reading/extraction requests locally. When repeated evidence from the versioned,
  495. sanitized fixture exists for a candidate's exact model identity, a bounded task-fit
  496. term influences quality ordering. Unmeasured models remain reachable, and if no
  497. candidate has current evidence the ordering is unchanged. Capability is grounded in
  498. real benchmark data, not guessed from names; models that no benchmark lists cover
  499. fall back to a name heuristic.
  500. Use `--task grounded-reading` to declare that intent, `--task general` to suppress
  501. automatic classification, or `--task auto` to classify locally. OpenAI-compatible,
  502. Responses, and Anthropic proxy clients can send the body extension
  503. `"task": "grounded-reading"` or the `X-Freellmpool-Task` header. Explicit intent
  504. wins over automatic classification. Task evidence stores aggregate pass counts,
  505. fixture hashes, and scores only—never prompts, documents, responses, or provider
  506. secrets.
  507. The bundled, offline scores come from [LMArena](https://lmarena.ai/) Elo (an
  508. MIT-licensed snapshot) and the [Aider](https://aider.chat/) code-editing
  509. leaderboard (Apache-2.0), normalized to a common percentile scale. For much
  510. broader coverage, run `freellmpool capability sync` with a free
  511. [Artificial Analysis](https://artificialanalysis.ai/) API key
  512. (`FREELLMPOOL_AA_API_KEY`) — its Intelligence Index covers most current and
  513. open-weight models and takes precedence. The fetched AA data is cached locally
  514. under your own key (never bundled, per AA's terms). `freellmpool capability
  515. status` shows current coverage. Scores via LMArena and Aider; intelligence index
  516. via Artificial Analysis when keyed.
  517. **Context windows.** Free models often have small context windows. freellmpool
  518. never truncates your input; instead, when a model rejects a request as too long,
  519. it learns that model's limit and stops routing oversized requests there, escalating
  520. only to larger-window models. If nothing fits it raises a clear
  521. `ContextWindowExceeded` (with the estimated input size) instead of a generic
  522. failure — over the proxy that's a `413`. You can declare a model's window with
  523. `context = N` in `providers.toml` to skip it proactively.
  524. Architecture notes: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
  525. ## Limitations
  526. - Free-tier models are smaller than frontier models. They're good for drafting,
  527. summarizing, classification, triage, and everyday coding — not a replacement
  528. for GPT-class reasoning on hard problems.
  529. - Quality and capacity vary through the day as high-cap tiers exhaust.
  530. freellmpool's local per-day counters reset at UTC midnight; upstream
  531. providers use their own limit and reset windows.
  532. - Free tiers change without notice. When a model id or limit goes stale, a
  533. one-line PR to `providers.toml` fixes it for everyone.
  534. - The proxy is meant for local/single-user use. It binds to `127.0.0.1` by
  535. default; if you expose it, set a key (`--api-key`).
  536. - The Claude Code / Anthropic path is experimental (text and tool use; no vision).
  537. - These are free tiers shared by everyone — don't abuse them.
  538. - Upstream providers receive and may prompt-title/moderate your requests; use
  539. the FAQ privacy table and each provider's current terms before sending
  540. sensitive prompts.
  541. - Free-tier availability, model IDs, and rate limits drift without notice.
  542. freellmpool does not bypass provider limits, rotate accounts, or evade
  543. quotas.
  544. ## How it compares
  545. Comparison snapshot: 2026-07-19. Competitor rows link to the exact README commit
  546. used for the claims; scopes change quickly, so treat this as a dated map rather
  547. than a permanent ranking.
  548. | Tool | Keyless start | # providers | Failover | MCP server | CLI | Transcription | Local/self-hosted | License |
  549. |---|---|---:|---|---|---|---|---|---|
  550. | **freellmpool** | Yes, when a configured keyless provider is available | 23 chat providers cataloged locally | Yes: retryable provider failures, empty replies, and transport errors | Yes: `freellmpool mcp` | One-shot CLI plus profiles, library, and proxy commands | Yes: `/v1/audio/transcriptions` with failover | Yes: small Python package and local proxy | MIT |
  551. | [OpenRouter free models](https://openrouter.ai/openrouter/free/providers) | No: hosted API use requires an account/key | Hosted router; the linked free router listed 22 models at snapshot | [Yes](https://openrouter.ai/docs/guides/routing/model-fallbacks): hosted provider/model fallbacks | [Yes](https://openrouter.ai/blog/announcements/openrouter-mcp-server/): hosted remote MCP server | Hosted API/SDK and agent SDK, not a local gateway CLI | [Audio input/transcription](https://openrouter.ai/docs/guides/overview/multimodal/audio) through multimodal chat | No: hosted service | Proprietary service |
  552. | [LiteLLM](https://github.com/BerriAI/litellm/blob/5d4c4d0fce45c73c4b56b48e46dfc4e56e8b0aa5/README.md) | No: bring provider or hosted-gateway credentials | README claims 100+ LLMs/providers | Yes: router retries/fallbacks | Yes: AI Gateway includes an MCP Gateway | Python SDK and proxy/gateway CLI surface | Yes: `/audio/transcriptions` | Yes: self-hosted proxy or hosted offering | MIT core; commercial enterprise features |
  553. | [OmniRoute](https://github.com/diegosouzapw/OmniRoute/blob/d8ff51874c8add566d43225988b9bc67e0542d65/README.md) | Yes: its setup documents a no-auth OpenCode option | README claims 268 integrations/providers and 90+ free options | Yes: layered fallback/circuit-breaker routing | Yes: MCP and A2A control planes | Broad management CLI and one-command agent setup | Audio translation is documented; other media capabilities vary by provider | Yes: Node application with dashboard, Docker, desktop/PWA paths | MIT |
  554. | [FreeLLMAPI](https://github.com/tashfeenahmed/freellmapi/blob/759de8e7ed1edc1cd513c9777cd0a807fb5ceee3/README.md) | Server starts with a Docker one-liner; inference capacity is configured afterward | README claims **28 free LLM providers** and 339 endpoints | Yes: provider/key routing and fallback | Yes: **MCP (Streamable HTTP)** | Dashboard/server and desktop apps | Not documented; audio speech/TTS is supported | Yes: Node/Docker server and desktop apps | MIT |
  555. FreeLLMAPI predates this project, and the overlap is independent convergence
  556. around legitimate free tiers. OmniRoute
  557. optimizes for an expansive control plane and many protocols; FreeLLMAPI for a
  558. dashboard-centric self-hosted router with encrypted key management; freellmpool
  559. for the smallest Python/CLI/library path with a keyless-capable first reply.
  560. Those are product-scope choices, not a claim that one tool is best for every
  561. deployment.
  562. ## FAQ
  563. **Is there a free, OpenAI-compatible LLM API gateway?** Yes — freellmpool is a
  564. free, MIT-licensed gateway that exposes one OpenAI-compatible endpoint over the
  565. enabled routes you can access. Its 23 cataloged provider groups span recurring
  566. free tiers, keyless endpoints, finite trials, pin-only routes, and disabled
  567. candidates. `pip install freellmpool` and point any OpenAI client at the local
  568. proxy.
  569. **How do I use multiple free LLM APIs at once?** freellmpool pools them: each request
  570. goes to a provider you have access to, fails over to the next when one is rate-limited
  571. or down, and tracks per-day usage so load spreads across tiers.
  572. **Can I run Claude Code or Codex on free models?** Yes — the proxy speaks the
  573. OpenAI API and has an experimental Anthropic-compatible path. Set
  574. `OPENAI_BASE_URL=http://localhost:8080/v1` for OpenAI-compatible tools or
  575. `ANTHROPIC_BASE_URL=http://localhost:8080` for Anthropic-compatible tools, then
  576. run Codex, Claude Code, aider, Cline, Continue, or Cursor against pooled free tiers. For Claude Code, set
  577. `CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1` so `/v1/models` is discovered
  578. through the Anthropic bridge. See `freellmpool code <agent>`. (Claude Code path is
  579. experimental: text + tools, no vision.)
  580. **Do I need an API key?** Not while an enabled keyless route is available:
  581. Pollinations, OVHcloud, and Kilo Gateway expose keyless routes, and LLM7 is
  582. key-optional. Add applicable free-tier or trial credentials for more routes and
  583. capacity; availability and terms remain provider-specific.
  584. **Is it free and open source?** Yes, MIT-licensed. More at the
  585. [project page](https://0xzr.github.io/freellmpool/).
  586. ## Featured in
  587. - Community videos (Spanish, by lytohlg AI): ["Accede a 18 modelos de IA GRATIS con 1 solo comando"](https://www.youtube.com/watch?v=1UfIlWoedho) and ["Prueba 18 IAs GRATIS sin API key en 30 segundos"](https://www.youtube.com/watch?v=oaM_E92WVGQ) (from an earlier catalog; freellmpool now catalogs 23 providers).
  588. - Directory: [FreeLLM Pool on MCP Market](https://mcpmarket.com/server/freellm-pool).
  589. ## Contributing
  590. New providers and fixes to stale limits are the most useful contributions, and
  591. both are usually a small change to `providers.toml`. See
  592. [CONTRIBUTING.md](CONTRIBUTING.md). Maintainer-ready newcomer tasks are drafted in
  593. [docs/GOOD_FIRST_ISSUES.md](docs/GOOD_FIRST_ISSUES.md). Tests run with no network access:
  594. ```bash
  595. python -m pip install -e ".[dev]" && ruff check . && pytest
  596. ```
  597. Source-first verification in this repo uses `PYTHONPATH=src` so `pytest` exercises
  598. the checkout without requiring an editable install first; CI runs the same
  599. configuration. Release readiness uses `PYTHONPATH=src python3 scripts/check_release_ready.py`.
  600. Security scanners, severity gates, the expiring exception process, local
  601. reproduction, SBOMs, and provenance verification are documented in
  602. [SECURITY.md](SECURITY.md).
  603. ## License
  604. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.