Skip to content
Back to search

player-review-intelligence

AI agent that turns mobile game app-store reviews into weekly, version-aware insights, built with Claude (tool use, prompt caching, batches), RAG, evals and an MCP server.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/lightwonkas--player-review-intelligence/badge.svg)](/listings/lightwonkas--player-review-intelligence)

HTML

<a href="/listings/lightwonkas--player-review-intelligence"><img src="/api/listings/lightwonkas--player-review-intelligence/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires [uv](https://docs.astral.sh/uv/), which installs Python 3.12 for you. Only an `ANTHROPIC_API_KEY` is needed.

git clone https://github.com/LightWonkas/player-review-intelligence && cd player-review-intelligence
uv sync
cp .env.example .env          # then set ANTHROPIC_API_KEY in .env
uv run review-intel fetch --app critical-strike --store all --limit 100
uv run review-intel classify --app critical-strike --limit 100     # ~$0.02
uv run review-intel report --app critical-strike --days 7          # ~$0.03
uv run review-intel dashboard                                      # http://127.0.0.1:8765

This exact sequence was run from a fresh clone. The database is created on first use, and every command is safe to re-run. `uv run review-intel --help` lists the rest: `eval`, `label`, `patches`, `index`, `ask`, `spikes`.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 319 lines
  1. # Player Review Intelligence
  2. [![CI](https://github.com/LightWonkas/player-review-intelligence/actions/workflows/ci.yml/badge.svg)](https://github.com/LightWonkas/player-review-intelligence/actions/workflows/ci.yml)
  3. Player Review Intelligence collects public app-store reviews for mobile games, classifies them with
  4. Claude, links complaints to game updates with RAG, and serves the results as a weekly report, a
  5. dashboard and an MCP server. Claude Code skills and subagents run the workflows on top.
  6. ![Dashboard demo](docs/demo.gif)
  7. ## The problem
  8. A live-service mobile game gets hundreds of store reviews a week across Google Play and the App
  9. Store, in many languages, with a star rating that says little about *why*. The useful signals are
  10. buried: an ad change that annoys players, a crash after an update, a weapon that feels pay-to-win.
  11. Game teams usually find these late, by skimming. This project turns the review stream into numbers
  12. a team can trust, plus a short weekly read that says what changed, what players say and who should
  13. act. Every number comes from the database, never from the model.
  14. Games are configured in [`config/apps.yaml`](config/apps.yaml); adding one needs no code changes.
  15. ## What it does
  16. | Step | How | Measured on real data |
  17. |---|---|---|
  18. | Collect | Google Play scraper plus the App Store RSS feed: polite pacing, retries, idempotent upserts, no usernames stored | 950 reviews across both stores |
  19. | Classify | `claude-haiku-5-5`, one strict tool call per review, Pydantic validation, prompt caching, Message Batches | 0.89 test accuracy against a stronger model's labels; ~$0.08 per 1,000 reviews |
  20. | Evaluate | Blind reference labels, dev/test split, per-class P/R/F1, confusion matrix, Sonnet judge | README table updated by `eval --write-readme` |
  21. | Link to releases | App Store version history, local embeddings in `sqlite-vec`, cited answers, per-store spike test | Answers cite real review ids and the right version |
  22. | Report | Code computes the facts, Sonnet writes the prose, code checks it, a judge scores it | [Example report](docs/example-report.md), ~$0.03 each |
  23. | Serve | FastAPI dashboard, MCP server (tools, resources, prompts), Slack webhook | Works in Claude Code via [`.mcp.json`](.mcp.json) |
  24. | Automate | Claude Code skills and subagents in [`.claude/`](.claude/) | `/weekly-review-report`, `/add-new-game`, `/improve-classifier` |
  25. ## Architecture
  26. ```mermaid
  27. flowchart LR
  28. subgraph stores["Public stores"]
  29. GP["Google Play"]
  30. AS["App Store feed<br/>and version history"]
  31. end
  32. GP --> COL
  33. AS --> COL
  34. COL["collectors<br/>pacing, retries"] --> DB[("SQLite<br/>reviews, labels, patch notes, reports<br/>plus sqlite-vec vectors")]
  35. DB --> CLS["classify<br/>Haiku 5.5, strict tool<br/>prompt cache, batches"]
  36. CLS --> DB
  37. DB --> IDX["index<br/>local MiniLM embeddings"]
  38. IDX --> DB
  39. EV["evals<br/>blind labels, metrics, judge"] -.->|keeps the prompt honest| CLS
  40. DB --> INS["insights<br/>SQL answers"]
  41. DB --> RAG["ask and spikes<br/>RAG, z-test"]
  42. INS --> REP["weekly report<br/>Sonnet prose, checks, judge"]
  43. INS --> DASH["dashboard<br/>FastAPI, SVG"]
  44. INS --> MCP["MCP server<br/>tools, resources, prompts"]
  45. RAG --> MCP
  46. REP --> SL["Slack"]
  47. MCP --> CC["Claude Code / Desktop<br/>skills, subagents"]
  48. ```
  49. Each module is described in two sentences in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
  50. ## Quickstart
  51. Requires [uv](https://docs.astral.sh/uv/), which installs Python 3.12 for you. Only an
  52. `ANTHROPIC_API_KEY` is needed.
  53. ```bash
  54. git clone https://github.com/LightWonkas/player-review-intelligence && cd player-review-intelligence
  55. uv sync
  56. cp .env.example .env # then set ANTHROPIC_API_KEY in .env
  57. uv run review-intel fetch --app critical-strike --store all --limit 100
  58. uv run review-intel classify --app critical-strike --limit 100 # ~$0.02
  59. uv run review-intel report --app critical-strike --days 7 # ~$0.03
  60. uv run review-intel dashboard # http://127.0.0.1:8765
  61. ```
  62. This exact sequence was run from a fresh clone. The database is created on first use, and every
  63. command is safe to re-run. `uv run review-intel --help` lists the rest: `eval`, `label`,
  64. `patches`, `index`, `ask`, `spikes`.
  65. ## Example report
  66. [`docs/example-report.md`](docs/example-report.md) is a real report from the review data,
  67. unedited. It has headline numbers against the previous week, the top issues with player quotes
  68. and review ids, a version watch, a four-week sentiment table, prioritized actions for dev,
  69. product and marketing, and the judge's score.
  70. ## Classification
  71. Each review gets one strict tool call (`record_classification`) to `claude-haiku-5-5`. The result
  72. is a primary category plus up to three more from the 12 in
  73. [`config/taxonomy.yaml`](config/taxonomy.yaml), along with sentiment, severity (1-5), a
  74. bug-report flag, mentioned features and a one-line summary.
  75. - **Validation:** Pydantic checks every response. Invalid output is retried once and then kept in
  76. `classification_failures`, never dropped.
  77. - **Untrusted input:** Review text goes in as escaped data inside `<review>` tags, and a test
  78. checks that a malicious review can't break out.
  79. - **Prompt versions:** `prompt_version` (e.g. `v1-10aea4a5`) is a hash of the prompt and tool
  80. schema, so editing either creates a new version and results never mix.
  81. - **Cost:** The ~1,900-token shared prefix is prompt-cached, so each review pays for about 300
  82. input and 210 output tokens. `classify --dry-run` prints the count and the estimated cost
  83. before you spend anything.
  84. ## Evals
  85. The reference set has 100 reviews, labeled blind: the labeler never sees the model's answer. The
  86. current labels were set by `claude-opus-5-5`, a stronger model than the classifier, and recorded
  87. per item in the `labeler` field. So the scores measure agreement with a stronger reference, not
  88. human accuracy; `review-intel label` lets a person relabel or extend the set.
  89. The sample is 40% random (mostly praise, like the real mix) and 60% low-rated reviews with real
  90. text, where the hard categories are, with at least 25% from the App Store. A hash of each review
  91. fixes it in a 70/30 dev/test split: prompts are tuned on dev, and the headline number is test.
  92. With 35 test reviews, one review moves accuracy by about 3 points, so every score comes with a
  93. 95% bootstrap interval.
  94. `review-intel eval` reports:
  95. - primary-category accuracy, per-category precision/recall/F1 and macro-F1
  96. - sentiment accuracy and a confusion matrix
  97. - a comparison with the previous prompt or model
  98. - an LLM judge's faithfulness score for the per-review summaries (`claude-sonnet-5-5`)
  99. Predictions are cached per prompt version, so re-running an unchanged prompt is free.
  100. <!-- eval:start -->
  101. | Date | Prompt | Model | Test acc [95% CI] | Test macro-F1 [95% CI] | Sentiment acc (all) | Judge |
  102. |---|---|---|---:|---:|---:|---:|
  103. | 2026-11-19 | `v1-10aea4a5` | claude-haiku-5-5 | 0.89 [0.77-0.97] (n=35) | 0.70 [0.52-0.95] | 0.97 | 4.65/5 |
  104. <!-- eval:end -->
  105. Read the intervals before the point estimates. Accuracy is solid (0.77-0.97), but macro-F1
  106. spans 0.52-0.95: categories with 1-3 examples pull the average down, and the set is too small
  107. to pin down rare categories. The weakest spot is the game's own auto-aim, which gets read as cheating
  108. (`controls_ux -> cheating_hackers`). The [`improve-classifier`](.claude/skills/improve-classifier/SKILL.md)
  109. skill is built for exactly this kind of fix.
  110. ## Linking complaints to releases (RAG)
  111. ```bash
  112. uv run review-intel patches fetch --app critical-strike # release notes, one request
  113. uv run review-intel index --app critical-strike # local embeddings, free
  114. uv run review-intel ask "When did matchmaking complaints start?" --app critical-strike
  115. uv run review-intel spikes --app critical-strike
  116. ```
  117. - **Patch notes:** The App Store page lists the last ~25 versions with dates and notes, the only
  118. public release history with dates. Notes for other stores can be imported from markdown
  119. (`patches import`). Version numbers are shared across stores, so one note serves both.
  120. - **Embeddings:** Reviews and single patch-note lines are embedded locally with
  121. `all-MiniLM-L6-v2` and stored in `sqlite-vec` tables in the same SQLite file, so search results
  122. join straight back to SQL. Re-indexing only embeds changed text.
  123. - **`ask`:** Retrieves similar reviews and patch notes, adds the notes for the versions those
  124. reviews were written on, and adds a per-version timeline from SQL. Sonnet answers and cites
  125. `[r<id>]` and `[v<version>]`. Every citation is checked against what was retrieved; invented
  126. ones are removed and reported. Asked when complaints about a new weapon started, it named the
  127. release that added the weapon and the earliest review that mentions it.
  128. - **`spikes`:** No LLM is involved. Per store, a version's share of reviews mentioning a category
  129. is compared with the previous versions using a two-proportion z-test. Stores are kept apart, so
  130. a shift in store mix can't fake a spike. Each spike links to the patch note in force, with its
  131. lines ranked by similarity to the complaints. When nothing is significant, the closest
  132. candidates are shown and labeled as not significant.
  133. ## Weekly report and dashboard
  134. Code lays out the report and fills in every number. `claude-sonnet-5-5` writes only the prose, as
  135. structured output, and code then checks it:
  136. - **Numbers:** every number in the prose must appear in the facts the model was given. If one
  137. doesn't, the model gets one retry with the offending numbers listed.
  138. - **References:** review ids and categories that weren't in the facts are dropped.
  139. - **Judge:** a separate call scores the prose for faithfulness, and the score goes in the footer.
  140. On real data the first version scored 2/5: it said "players say" on the strength of one review,
  141. and it treated an old version as current. Fixing the facts (per-store versions, marked partial
  142. weeks, the severity scale) and asking for claims sized to their evidence brought it to 4/5.
  143. The dashboard is FastAPI and Jinja2 with server-side SVG charts and no JavaScript. It shows
  144. 7-day stat tiles, top issues, reviews by category (7/30/90 days), weekly sentiment and the latest
  145. report. Every chart has a table view, and both color modes were run through a color-vision
  146. deficiency check. JSON routes live under `/api/apps/{app}/...`. The dashboard is read-only and
  147. listens on localhost.
  148. ![Dashboard](docs/dashboard.png)
  149. ## MCP server
  150. `review-intel-mcp` serves the data over the [Model Context Protocol](https://modelcontextprotocol.io)
  151. with the official Python SDK (`MCPServer`, the SDK 2.x name for FastMCP). It runs over stdio, or
  152. over streamable HTTP with `--transport streamable-http`.
  153. | Kind | Name | What it does |
  154. |---|---|---|
  155. | Tool | `top_issues(app, days, limit)` | Top complaint categories vs the previous period, with example reviews |
  156. | Tool | `sentiment_trend(app, days, bucket)` | Sentiment and average rating per day or week |
  157. | Tool | `compare_versions(app, version_a, version_b)` | Category shares with z-scores, rating, sentiment and patch notes |
  158. | Tool | `search_reviews(app, query, category?, version?, limit)` | Semantic search with filters |
  159. | Tool | `ask(app, question)` | Cited answer (calls Claude, ~$0.014) |
  160. | Resource | `reviews://{app}/{review_id}`, `reports://{app}/latest`, `taxonomy://categories` | One review, the latest report, the categories |
  161. | Prompt | `weekly_report(app)`, `triage_bugs(app, days)` | Step-by-step plans that use the tools |
  162. All tools except `ask` are read-only and free, and they share the [`insights`](src/review_intel/insights.py)
  163. query layer with the report and the dashboard, so all three show the same numbers.
  164. **Claude Code:** a session opened in this folder picks up [`.mcp.json`](.mcp.json); approve it
  165. once. To add the server elsewhere:
  166. `claude mcp add --transport stdio review-intel -- uv run --directory /path/to/repo review-intel-mcp`.
  167. **Claude Desktop:** add this to `claude_desktop_config.json`:
  168. ```json
  169. {
  170. "mcpServers": {
  171. "review-intel": {
  172. "command": "uv",
  173. "args": ["run", "--directory", "C:\\path\\to\\player-review-intelligence", "review-intel-mcp"]
  174. }
  175. }
  176. }
  177. ```
  178. **Inspector:** `npx @modelcontextprotocol/inspector uv run review-intel-mcp`
  179. ## Claude Code skills and subagents
  180. The repo uses Claude Code to operate the system, not just to build it.
  181. | | Name | What it does |
  182. |---|---|---|
  183. | Skill | [`weekly-review-report`](.claude/skills/weekly-review-report/SKILL.md) | Fetch, classify and report for a game, then summarize. Prices each paid step before running it. |
  184. | Skill | [`add-new-game`](.claude/skills/add-new-game/SKILL.md) | Adds a game to the config, checks both store ids against the stores, fetches a sample. |
  185. | Skill | [`improve-classifier`](.claude/skills/improve-classifier/SKILL.md) | Eval, confusion matrix, one proposed change shown as a diff, re-eval, before/after table. |
  186. | Subagent | [`bug-analyst`](.claude/agents/bug-analyst.md) | Deduplicated bug list with versions, severity and example review ids. |
  187. | Subagent | [`sentiment-analyst`](.claude/agents/sentiment-analyst.md) | Sentiment by week and version, backed by numbers. |
  188. | Subagent | [`report-reviewer`](.claude/agents/report-reviewer.md) | Checks a report's quotes and claims against the data before it is shared. |
  189. Each subagent only gets the free, read-only MCP tools. [`CLAUDE.md`](CLAUDE.md) gives every
  190. session the commands, the conventions and the rule: run tests and evals before claiming a change
  191. works.
  192. ## Cost
  193. Prices as of November 2026 ([`pricing.py`](src/review_intel/llm/pricing.py)); check Anthropic's
  194. pricing page before budgeting.
  195. | What | Model | Cost |
  196. |---|---|---:|
  197. | Classify 1,000 reviews, `--batch` | Haiku 5.5 | ~$0.08 |
  198. | Classify 1,000 reviews, synchronous | Haiku 5.5 | ~$0.17 |
  199. | Weekly report, with judge | Sonnet 5.5 | ~$0.03 |
  200. | `ask`, one question | Sonnet 5.5 | ~$0.014 |
  201. | Eval run (100 labels + 20 judged summaries) | Haiku + Sonnet | ~$0.08; ~$0.06 when cached |
  202. | Embeddings and index (950 reviews in 19 s on CPU) | local MiniLM | $0 |
  203. A game with 1,000 new reviews a week costs about **$0.11 per week**: a batch classification plus
  204. one report. All the live runs behind this README cost under $1 in total.
  205. ## Design decisions
  206. - **Haiku to classify, Sonnet to write and judge.** Classification is high-volume and narrow, and
  207. Haiku 5.5 reaches 0.89 test accuracy at about $0.08 per 1,000 reviews. Prose a product manager
  208. forwards, and grading other models' output, need the stronger model. The judge is never the
  209. model it grades.
  210. - **Tool use for structured output.** One forced, strict tool call with enum categories gives
  211. parseable JSON every time, and Pydantic covers what JSON Schema can't express. Results are
  212. keyed by prompt version and model, so comparisons stay clean.
  213. - **Code computes, the model writes.** Every number in a report, answer or dashboard comes from
  214. SQL. The model's prose is checked for numbers and ids that aren't in the facts, and a judge
  215. scores what's left. This is what makes the report safe to forward.
  216. - **Local embeddings in the same database.** MiniLM on CPU is free, keeps review text on the
  217. machine, and needs no second vendor. `sqlite-vec` keeps vectors next to the rows they describe,
  218. so retrieval joins straight back to SQL. The trade-off is an English-focused model, which is one
  219. setting to change.
  220. - **Evals keep the prompt honest.** Labels are blind, the dev/test split is fixed by hash, and
  221. predictions are cached per prompt version. The README table updates from real runs, so any
  222. prompt change shows up as a number before it ships.
  223. - **Untrusted input everywhere.** Review text is escaped and delimited in every prompt, flattened
  224. and escaped in reports, and autoescaped in the dashboard. Tests cover injection attempts at each
  225. layer.
  226. - **Statistics before stories.** Spikes need a z-score and a minimum sample, per store. With
  227. today's data that honestly means "no significant spikes" plus the closest candidates, rather
  228. than a confident narrative.
  229. ## Limitations and next steps
  230. - **Reference labels come from a stronger model, not people, and there are only 100.** Scores
  231. measure agreement with Opus, and the macro-F1 interval is wide (0.52-0.95). A larger,
  232. human-labeled set with more rare-category examples would measure real accuracy
  233. (`review-intel label` is ready for it).
  234. - **Shallow history.** The App Store feed serves only ~500 recent reviews per country, and
  235. fetching newest-first covers a few weeks of Google Play. With ~20 reviews per version, spikes
  236. rarely reach significance. A scheduled weekly fetch would build the history.
  237. - **One storefront.** Only the US store is tracked. Other countries are a config entry each, and
  238. multilingual reviews would want a multilingual embedding model (`EMBEDDING_MODEL`).
  239. - **Judge scores vary.** Scores for the same week ranged from 3/5 to 4/5 between runs. The
  240. `report-reviewer` subagent is the second check before sharing.
  241. - **SQLite.** Time windows and counts run as SQL filters and aggregates, but per-version
  242. timelines and spikes still read a game's full classified history, and vector search uses
  243. `sqlite-vec`. For hundreds of thousands of reviews: Postgres through `DATABASE_URL`, pgvector,
  244. and per-version counts in SQL.
  245. - **Next steps:** human labels, a scheduled pipeline with Slack alerts when a spike becomes
  246. significant, and more storefronts.
  247. ## Development
  248. ```bash
  249. uv run pytest # 217 tests, no network, no real LLM
  250. uv run ruff check . && uv run ruff format --check .
  251. uv run mypy # strict
  252. uv run python scripts/record_fixtures.py # re-record store fixtures (by hand)
  253. uv run scripts/make_demo_gif.py # rebuild docs/demo.gif from a running dashboard
  254. ```
  255. CI runs ruff, mypy and pytest on every push. The full build plan is in [`docs/SPEC.md`](docs/SPEC.md).
  256. ## License
  257. [MIT](LICENSE)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.