Skip to content
Back to search

ardu

Motivational quotes app with a small local LLM: Expo app, Node backend, eval harness (LLM-as-judge), and an MCP server

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/athospy--ardu/badge.svg)](/listings/athospy--ardu)

HTML

<a href="/listings/athospy--ardu"><img src="/api/listings/athospy--ardu/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 214 lines
  1. # Ardu
  2. **A motivational-quotes app whose AI runs on a 1.5B-parameter model on a 2 GB VPS — and the evals that show what prompt tuning can and can't fix at that size.**
  3. Ardu (iOS and Android, English and Spanish) asks people what they're working toward and writes
  4. short quotes about it. The quotes come from `qwen2.5:1.5b` running under Ollama on the same small
  5. server as the API: no hosted LLM, no per-token bill. This repo has the full app, the backend, the
  6. eval harness used to tune the prompt, and an MCP server so Claude can use the generator directly.
  7. - **App:** in beta on iOS and Android ([Android beta on Google Play](https://play.google.com/apps/testing/dev.santiagomorel.ardu)). More at [santiagomorel.dev](https://santiagomorel.dev).
  8. - **Evals:** [results and method](#evals-what-prompt-tuning-fixed-and-what-it-didnt) · [full report](evals/results/REPORT.md)
  9. - **MCP:** [connect it to Claude](#mcp-server-use-the-generator-from-claude)
  10. ## How it works
  11. ```mermaid
  12. flowchart LR
  13. subgraph Phone
  14. A[Expo / React Native app]
  15. end
  16. subgraph VPS["2 GB VPS"]
  17. B[Express API<br/>JWT auth, rate limits]
  18. C[(Postgres<br/>users, quotes, reflections)]
  19. D[Quote job<br/>node-cron]
  20. E[Ollama<br/>qwen2.5:1.5b]
  21. end
  22. A -- REST --> B
  23. B --> C
  24. D -- "pending AI quotes" --> C
  25. D -- "prompt v1 (streaming)" --> E
  26. E -- draft --> G{Gates}
  27. G -- "echoes the motivation /<br/>repeats an old quote" --> E
  28. G -- accepted --> C
  29. B -- Expo push --> A
  30. ```
  31. **Generation path** ([`backend/src/config/ollama.js`](backend/src/config/ollama.js)):
  32. 1. Build the prompt from a versioned template ([`backend/src/prompts/quote/`](backend/src/prompts/quote/)).
  33. 2. Stream the completion from Ollama. Streaming keeps the HTTP connection alive on a slow CPU box.
  34. 3. Run two cheap gates on the draft. A failed draft is regenerated (up to 2 drafts in total):
  35. - **Overlap gate:** share of the draft's words that also appear in the motivation. Above 30%, the model just echoed the input.
  36. - **Similarity gate:** shared words divided by the larger quote's word count, compared against the 200 most recent AI quotes (all users). Above 75%, it's a repeat.
  37. 4. Strip wrapping quotes and save the result.
  38. Before the gates, a draft with letters outside the Latin script is also regenerated: `qwen2.5` sometimes
  39. switches to Chinese mid-quote. (Until November 2026 the gates' text cleanup dropped accented letters,
  40. `más` → `ms`, and that bug was what happened to catch those drafts. Replaying all 300 eval quotes, fixing it
  41. changed 2 of 150 Spanish gate decisions, one of them the Chinese draft, so the explicit check replaced it.)
  42. ## Evals: what prompt tuning fixed, and what it didn't
  43. Every configuration generated 60 quotes (20 motivations, 10 English and 10 Spanish, × 3 runs) through
  44. the production code path ([`backend/bin/generate.js`](backend/bin/generate.js)). Two LLM judges scored
  45. each quote 1–3 on relevance, originality, inspiration and clarity: a local `llama3` 8B and Claude
  46. (`claude-opus-5-5`). Then a human scored 30 of the quotes blind, to check the judges.
  47. Method and commands: [`evals/`](evals/). Every number: [`evals/results/REPORT.md`](evals/results/REPORT.md).
  48. **Claude judge, mean of 60 quotes per row** (1–3 per dimension, total out of 12). The last column
  49. needs no judge: it counts final quotes that still contain a cliché from the prompt's ban list.
  50. | Generator | relevance | originality | inspiration | clarity | total | banned cliché in final quote |
  51. |---|---|---|---|---|---|---|
  52. | prompt v0 · qwen2.5:1.5b | 2.25 | 1.15 | 1.47 | 1.60 | 6.5 | 3% |
  53. | prompt v1 · qwen2.5:1.5b (production) | 2.65 | 1.17 | 1.48 | 1.93 | 7.2 | 20% |
  54. | prompt v1 + gate 3 · qwen2.5:1.5b | 2.58 | 1.15 | 1.40 | 1.78 | 6.9 | 12% |
  55. | prompt v1 · qwen2.5:3b | 2.75 | 1.28 | 1.52 | 1.82 | 7.4 | 2% |
  56. | prompt v1 · gemma3:4b | **2.93** | **1.47** | **1.97** | **2.27** | **8.6** | 0% |
  57. What this shows:
  58. 1. **The tuned prompt reads clearer. Its relevance gain is unconfirmed.** Paired by motivation, Claude
  59. sees v0 → v1 clarity +0.33 (95% CI +0.10 to +0.57) and relevance +0.40 (+0.20 to +0.60). The blind
  60. human scores back the clarity gain (+0.67 on the sampled quotes) but not the relevance gain (+0.06).
  61. 2. **At 1.5B, naming the clichés to avoid makes the model use them.** v1 added a ban list ("Empower",
  62. "Embrace", "one step at a time"…), and banned clichés in final quotes went from 3% to 20%. The same
  63. prompt on 3B and 4B models: 2% and 0%.
  64. 3. **A regex filter (gate 3) treats the symptom, not the quality.** Rejecting drafts with a banned
  65. cliché cut them to 12% (only 2 drafts are allowed), Claude scored the quotes no better, and every
  66. rejection costs another full generation. It stays off.
  67. 4. **Model size beats prompt work here.** No prompt moved originality or inspiration at 1.5B;
  68. `gemma3:4b` is the only model that did. It needs more than 3 GB of RAM, so it doesn't fit the current
  69. 2 GB server: that is the next decision, not another prompt.
  70. 5. **The prompt fights its own gate, and the gate is right.** v1 asks for a concrete reference to the
  71. motivation, and the overlap gate rejects drafts that reuse too many of its words: 43% of v1 quotes needed
  72. a retry (12% for v0). In 60 extra first drafts, a third were rejected, and almost all of them really
  73. restate the motivation ("Ahorra para comprar tu primera casa."). Ignoring filler words only cut
  74. rejections from 20 to 17, so the gate stays. A retry costs a few seconds in a background job, not user
  75. time, and a prompt fix would need its own eval round.
  76. **How far to trust the judges.** The 8B judge gave every quote the lowest originality score and almost
  77. every quote the lowest inspiration score, so it can't tell configurations apart on those. Claude's
  78. agreement with the human scorer (quadratic-weighted kappa) is higher than llama3's on every dimension,
  79. but still only slight: kappa 0.06–0.20, within one point 83–93% of the time, and Claude is much
  80. stricter on originality and inspiration. Read the table as direction, not as calibrated quality.
  81. The human check is small: 30 quotes, one person, 18 from v0 and 12 from v1 (not paired by
  82. motivation). On that sample v1 also scored a little lower on originality (−0.31) and inspiration (−0.14).
  83. An earlier grader (May 2026) reported a large relevance gain. It set the generator model after the
  84. code had already read it, so it most likely tested the wrong model; [`evals/README.md`](evals/README.md)
  85. has the details. This harness passes the model explicitly and replaces it.
  86. ## MCP server: use the generator from Claude
  87. [`mcp/`](mcp/) exposes four tools:
  88. | Tool | What it does |
  89. |---|---|
  90. | `generate_quote(motivation, language)` | Runs the production generator (same prompt, model and overlap gate) and returns the quote, drafts used and timing. Saves nothing. |
  91. | `search_quotes(query, language?, limit?)` | Searches the curated quote library by text, author or category. |
  92. | `random_quote(language?, category?)` | One random quote from the library. |
  93. | `get_stats()` | Library size, demo generation stats (volume, latency, gate retries) and the latest eval scores. |
  94. It only reads the curated library; real users' quotes and reflections aren't reachable. The
  95. generation log stores numbers only (latency, drafts, gate rejections), never what was typed.
  96. **Local (Claude Code or Claude Desktop, stdio):** needs Postgres with the library loaded and Ollama
  97. running (see [Run it locally](#run-it-locally)).
  98. ```bash
  99. claude mcp add ardu -e DATABASE_URL=postgresql://ardu_user:ardu_password@localhost:5432/quotes_app \
  100. -- node /path/to/ardu/mcp/src/stdio.js
  101. ```
  102. For Claude Desktop, add the same command to `claude_desktop_config.json` under `mcpServers`.
  103. **Remote (Streamable HTTP + bearer token):** [`deploy/`](deploy/) runs the server behind Caddy with
  104. HTTPS. A hosted demo runs at `https://mcp.ardu.santiagomorel.dev/mcp` (token-protected).
  105. ```bash
  106. # Claude Code
  107. claude mcp add --transport http ardu https://<your-domain>/mcp \
  108. --header "Authorization: Bearer <MCP_TOKEN>"
  109. ```
  110. In claude.ai: **Customize → Connectors → Add custom connector**. Set the URL to
  111. `https://<your-domain>/mcp` and Authentication to *No sign-in*. Then add a request header
  112. `authorization` with the value `Bearer <MCP_TOKEN>`.
  113. **A session with the hosted demo** (recorded 2026-11-17; long outputs shortened with …):
  114. ```text
  115. > generate_quote(motivation: "finishing my first half marathon in under two hours", language: "en")
  116. { "quote": "Finish each mile in your own way, crush my first half marathon in under two hours!",
  117. "model": "qwen2.5:1.5b", "prompt_version": "v1", "drafts": 2,
  118. "rejected_drafts": { "overlap": 1, "similarity": 0, "template": 0 }, "seconds": 9.4 }
  119. > generate_quote(motivation: "volver a estudiar inglés después de diez años", language: "es")
  120. { "quote": "Volviendo a estudiar inglés, cada nuevo día es una nueva oportunidad.",
  121. "model": "qwen2.5:1.5b", "prompt_version": "v1", "drafts": 1,
  122. "rejected_drafts": { "overlap": 0, "similarity": 0, "template": 0 }, "seconds": 9.5 }
  123. > search_quotes(query: "fear", limit: 2)
  124. "Everything you've ever wanted is on the other side of fear." — George Addair [courage, en]
  125. > get_stats()
  126. { "library": [ { "language": "en", "category": "action", "count": 5 }, … ], // 50 quotes, en + es
  127. "generation": [ { "model": "qwen2.5:1.5b", "total": 4, "failed": 0, "avg_duration_ms": 9855,
  128. "avg_drafts": 1.25, "overlap_gate_retries": 1, "similarity_gate_retries": 0 } ],
  129. "evals": [ { "run": "v1 · gemma3:4b", "scores": { "claude:claude-opus-5-5": { "total_of_12": 8.63, … } } }, … ] }
  130. ```
  131. The first call shows both sides of the 1.5B model: the overlap gate rejected a draft that copied the
  132. motivation, and the second draft still slipped into first person ("crush **my** first half marathon").
  133. Each generation takes about 10 seconds on the demo's 2-vCPU, CPU-only server.
  134. ## Run it locally
  135. Requirements: Node 20+, Docker, [Ollama](https://ollama.com), and [uv](https://docs.astral.sh/uv/) for the evals.
  136. ```bash
  137. ollama pull qwen2.5:1.5b # the production model
  138. ollama pull llama3 # the eval judge (8B)
  139. # Backend: Postgres in Docker, API on :3000
  140. cd backend
  141. cp .env.example .env
  142. npm install
  143. npm run dev:docker # starts Postgres, runs migrations, starts the API
  144. npm run seed # loads the curated quote library
  145. npx jest # unit tests (no model needed)
  146. # Generate one quote with the production code path
  147. echo '{"id":"1","motivation":"learning to code","language":"en"}' | node bin/generate.js
  148. # Evals
  149. cd ../evals
  150. uv run pytest -q # harness unit tests
  151. ./scripts/run-all.sh # re-run every experiment (~1 hour on an M-series Mac)
  152. # MCP server tests
  153. cd ../mcp && npm install && npm test
  154. # App (needs Xcode or Android Studio)
  155. cd ../app && npm install && cp .env.example .env && npx expo run:ios
  156. ```
  157. ## Repository layout
  158. ```
  159. app/ React Native (Expo) client
  160. backend/ Express API, Postgres migrations, quote generator, cron jobs
  161. evals/ Python eval harness, test cases, results
  162. mcp/ MCP server (stdio + Streamable HTTP)
  163. deploy/ Docker Compose stack for the hosted MCP demo
  164. docs/ Deployment guide (backend and MCP demo)
  165. ```
  166. To run it on a server, see [docs/deploy.md](docs/deploy.md).
  167. ## License
  168. Code: [MIT](LICENSE). The Ardu name, logo, app icon and screenshots are not covered by the
  169. license; all rights reserved.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.