Skip to content
Back to search

grounded-reports

LLM-written performance summaries that can't be wrong about the numbers — fact-sheet grounding, rule-based claim validation, retry with feedback, deterministic fallback. Includes evals and an MCP server.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mike-larchick--grounded-reports/badge.svg)](/listings/mike-larchick--grounded-reports)

HTML

<a href="/listings/mike-larchick--grounded-reports"><img src="/api/listings/mike-larchick--grounded-reports/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/mike-larchick/grounded-reports
cd grounded-reports
python3 -m grounded_reports.cli --mock

You'll see the fact sheet, attempt 1 failing with a flagged sentence, attempt 2 passing, and the final output. Then:

python3 evals/run_evals.py          # 18 validator cases (2 pinned limitations), exit 1 on any failure
pip install pytest && pytest -q     # unit tests

With a real model

pip install 'anthropic>=0.40'
export ANTHROPIC_API_KEY=...
python3 -m grounded_reports.cli            # default model: claude-sonnet-4-5
python3 -m grounded_reports.cli --json     # machine-readable, includes every attempt

Validate your own text

python3 -m grounded_reports.cli --validate-only my_draft.txt

Bring your own data

Any CSV with `date,sessions,signups,revenue,cost` and at least 2× the comparison window:

python3 -m grounded_reports.cli --data path/to/metrics.csv --period 14 --mock

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 169 lines
  1. # grounded-reports
  2. LLM-written performance summaries with a deterministic gate on the numbers: every direction
  3. word and percentage the model writes about a known metric is checked against pre-computed facts
  4. before anything ships.
  5. A small, runnable distillation of a pattern I run in production: compute the facts in code, let
  6. the model write the prose, validate the claim patterns the gate knows about, and retry or fall
  7. back when it doesn't pass. It catches the failure modes that actually show up — flipped
  8. direction, hallucinated percentages, polarity traps — not every possible falsehood. The
  9. [Known limitations](#known-limitations) section says exactly where it stops.
  10. ```
  11. daily metrics ──▶ fact sheet ──▶ LLM draft ──▶ claim validator ──┬─▶ ship
  12. ▲ │
  13. └──────── retry with violations ◀─────────┘
  14. └─▶ (budget exhausted) deterministic fallback
  15. ```
  16. No API key needed to try it. `--mock` runs the whole loop offline, including a deliberately
  17. wrong first draft so you can watch the validator catch it.
  18. ## Why this exists
  19. A narrative that says "signups grew" when they fell by 10% is worse than no narrative. The
  20. model isn't bad at writing; it's unreliable at arithmetic and at keeping polarity straight
  21. ("cost per signup improved" means it went *down*). The fix is structural, not prompt-tuning:
  22. 1. **Ground.** Compute current vs prior, percent change, direction, and polarity *in code*.
  23. The model receives a fact sheet and is told to only restate it.
  24. 2. **Validate.** Parse the draft for direction words and percentages near metric mentions and
  25. check each one against the fact sheet. Rule-based, deterministic, free.
  26. 3. **Retry with feedback.** Violations go back to the model in the next turn. Static context is
  27. marked cacheable so retries are cheap.
  28. 4. **Fall back.** If the retry budget is exhausted, ship a plain template built from the facts.
  29. Boring, never wrong.
  30. 5. **Eval.** A fixed set of narratives with known defects keeps the validator honest as it
  31. evolves.
  32. ## Quickstart
  33. ```bash
  34. git clone https://github.com/mike-larchick/grounded-reports
  35. cd grounded-reports
  36. python3 -m grounded_reports.cli --mock
  37. ```
  38. You'll see the fact sheet, attempt 1 failing with a flagged sentence, attempt 2 passing, and
  39. the final output. Then:
  40. ```bash
  41. python3 evals/run_evals.py # 18 validator cases (2 pinned limitations), exit 1 on any failure
  42. pip install pytest && pytest -q # unit tests
  43. ```
  44. ### With a real model
  45. ```bash
  46. pip install 'anthropic>=0.40'
  47. export ANTHROPIC_API_KEY=...
  48. python3 -m grounded_reports.cli # default model: claude-sonnet-4-5
  49. python3 -m grounded_reports.cli --json # machine-readable, includes every attempt
  50. ```
  51. ### Validate your own text
  52. ```bash
  53. python3 -m grounded_reports.cli --validate-only my_draft.txt
  54. ```
  55. ### Bring your own data
  56. Any CSV with `date,sessions,signups,revenue,cost` and at least 2× the comparison window:
  57. ```bash
  58. python3 -m grounded_reports.cli --data path/to/metrics.csv --period 14 --mock
  59. ```
  60. ## As an MCP server
  61. The same three steps are exposed as tools so an agent can call them directly.
  62. ```bash
  63. pip install 'mcp>=1.2'
  64. python3 -m grounded_reports.mcp_server
  65. ```
  66. Claude Desktop / any MCP client config:
  67. ```json
  68. {
  69. "mcpServers": {
  70. "grounded-reports": {
  71. "command": "python3",
  72. "args": ["-m", "grounded_reports.mcp_server"],
  73. "cwd": "/path/to/grounded-reports"
  74. }
  75. }
  76. }
  77. ```
  78. Tools: `fact_sheet`, `validate`, `generate_report`. An agent that drafts its own narrative can
  79. call `validate` on it before showing it to a user.
  80. ## What the validator checks
  81. | Claim type | Example | How it's resolved |
  82. |---|---|---|
  83. | Direction | "signups **grew**", "cost **eased**" | Word → up/down/flat; compared to fact sheet |
  84. | Polarity-dependent direction | "cost per signup **improved**" | *improved* = down for lower-is-better metrics |
  85. | Percent change | "sessions were up **16%**" | Compared to actual within ±1.5 points |
  86. | Level, not change | "conversion rate **of 3.2%**", "from 4.1%" | Recognized by preceding word; skipped |
  87. | Multiple metrics per sentence | "revenue climbed while CPA rose" | Each claim attaches to the nearest preceding metric |
  88. | Negation | "signups **did not** grow" | Negator within 3 words inverts up/down; negated "flat" is dropped, not guessed |
  89. | Percent before its metric | "a 16% rise **in sessions**" | Attaches forward when only a change-noun + preposition sits between them |
  90. Run `python3 evals/run_evals.py` to see each of these exercised.
  91. ## Known limitations
  92. The gate is rules, so it has a boundary. These are pinned in `evals/cases.json` as
  93. `known_limitation: true` so they stay visible rather than quietly assumed away:
  94. - **Spelled-out numbers.** "up forty percent" isn't parsed; a wrong value passes.
  95. - **Implicit subjects.** "Sessions set a new high. It fell sharply." — the second sentence has
  96. no metric mention, so the claim can't be attributed and is skipped.
  97. - **Metrics it doesn't know.** Anything outside the fact sheet's alias table is invisible. Safe
  98. for a report that must only restate given data; not a general fact-checker.
  99. - **Causal and comparative language.** "because", "driven by", "outpaced" aren't adjudicated.
  100. - **False positives route to the fallback.** A correct draft the rules misread ships as the
  101. plain template. That's the safe direction, but it lowers quality silently, so the eval set
  102. matters as much for what it *doesn't* flag as for what it does.
  103. The next layer up — a model-based judge for tone, causality, and the cases above — can sit
  104. behind this gate. It should not replace it.
  105. ## What it deliberately doesn't do
  106. - **No second LLM as judge.** The gate is rules so it's deterministic and costs nothing per run.
  107. A model-based judge can layer on top for softer qualities (tone, causality) — this catches the
  108. hard failures first.
  109. - **No causal claims.** The prompt forbids inferring *why* a number moved; the validator doesn't
  110. try to adjudicate causality.
  111. - **Not a general fact-checker.** It knows about the metrics in the fact sheet and nothing else.
  112. Unknown nouns are ignored, which is the safe default for a report that must only restate given data.
  113. ## Layout
  114. ```
  115. grounded_reports/
  116. facts.py load CSV → FactSheet (current, prior, % change, direction, polarity)
  117. validate.py narrative + FactSheet → ValidationResult (violations with sentence context)
  118. generate.py AnthropicGenerator (cacheable system prompt) and offline MockGenerator
  119. fallback.py deterministic narrative from facts
  120. pipeline.py generate → validate → retry → fallback
  121. cli.py command line
  122. mcp_server.py FastMCP server exposing the same steps as tools
  123. data/sample_metrics.csv 28 days of synthetic data
  124. evals/ cases.json + runner
  125. tests/ pytest
  126. ```
  127. ## Background
  128. The production version of this pattern runs weekly against a BigQuery warehouse and ships
  129. client-facing executive summaries with no human review step. This repo is a clean-room rewrite
  130. on synthetic data; none of the production code, prompts, or data are included.
  131. MIT License.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.