Skip to content
Back to search

hallbayes

Berry is a local MCP server that checks whether an LLM's answer is supported by the evidence it cites. For each claim it measures how much the prediction depends on the cited evidence (an…

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/leochlon--hallbayes/badge.svg)](/listings/leochlon--hallbayes)

HTML

<a href="/listings/leochlon--hallbayes"><img src="/api/listings/leochlon--hallbayes/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pipx install -e .   # install
berry init          # in each repo: provisions a key, writes MCP + skill configs

Reload MCP servers in your client afterward. To use your own key or a different backend, run `berry setup`.

`berry install` registers Berry with a specific assistant. It writes a skill file, an always-on instruction file or managed section (`AGENTS.md`, `CLAUDE.md`, `.cursor/rules/berry.mdc`), and MCP config plus hooks where the host exposes them. Generated artifacts embed the resolved `berry` path, so re-run it after reinstalling Berry or moving its environment.

| Assistant | Command | |---|---| | Claude Code | `berry install` | | Codex | `berry install --platform codex` | | Cursor | `berry cursor install` | | Gemini CLI | `berry install --platform gemini` | | GitHub Copilot CLI | `berry install --platform copilot` | | VS Code Copilot Chat | `berry vscode install` | | Aider | `berry install --platform aider` | | Amp | `berry amp install` | | Kiro | `berry kiro install` | | Devin | `berry devin install` | | Anything else | `berry install --list-platforms` |

By default `berry install` writes user-profile config; pass `--project` to install into the current repo (requires a git repo unless `BERRY_ALLOW_NON_GIT_ROOT=1`). The full platform list, flags, and the idempotent-write safety model are in [`docs/INSTALL.md`](docs/INSTALL.md).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 131 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Berry
  2. [![CI](https://github.com/leochlon/hallbayes/actions/workflows/ci.yml/badge.svg)](https://github.com/leochlon/hallbayes/actions/workflows/ci.yml)
  3. Berry is a local MCP server that checks whether an LLM's answer is supported by the evidence it cites. For each claim it measures how much the prediction depends on the cited evidence (an information-budget test), returns a fail-closed answer/flag decision against a target, and records every decision in a tamper-evident ledger.
  4. It flags unsupported claims at 0.82 to 0.93 AUROC across grounding benchmarks, in a single logprob pass. It runs alongside Cursor, Codex, Claude Code, and Gemini CLI via config files committed to your repo.
  5. ## The problem
  6. Agents assert things their context does not support. A claim can be true in the world yet ungrounded in the evidence the model was given, and world-truth fact-checkers miss exactly that case. Berry measures grounding directly: the information sufficiency of the cited evidence for each claim.
  7. ## Install
  8. ```bash
  9. pipx install -e . # install
  10. berry init # in each repo: provisions a key, writes MCP + skill configs
  11. ```
  12. Reload MCP servers in your client afterward. To use your own key or a different backend, run `berry setup`.
  13. `berry install` registers Berry with a specific assistant. It writes a skill file, an always-on instruction file or managed section (`AGENTS.md`, `CLAUDE.md`, `.cursor/rules/berry.mdc`), and MCP config plus hooks where the host exposes them. Generated artifacts embed the resolved `berry` path, so re-run it after reinstalling Berry or moving its environment.
  14. | Assistant | Command |
  15. |---|---|
  16. | Claude Code | `berry install` |
  17. | Codex | `berry install --platform codex` |
  18. | Cursor | `berry cursor install` |
  19. | Gemini CLI | `berry install --platform gemini` |
  20. | GitHub Copilot CLI | `berry install --platform copilot` |
  21. | VS Code Copilot Chat | `berry vscode install` |
  22. | Aider | `berry install --platform aider` |
  23. | Amp | `berry amp install` |
  24. | Kiro | `berry kiro install` |
  25. | Devin | `berry devin install` |
  26. | Anything else | `berry install --list-platforms` |
  27. By default `berry install` writes user-profile config; pass `--project` to install into the current repo (requires a git repo unless `BERRY_ALLOW_NON_GIT_ROOT=1`). The full platform list, flags, and the idempotent-write safety model are in [`docs/INSTALL.md`](docs/INSTALL.md).
  28. ## Usage
  29. Your agent calls `detect_hallucination` with an answer and the evidence spans it cites; Berry scores each claim and flags the ones the evidence does not support. A real run:
  30. > **Evidence (S1):** "The James Webb Space Telescope launched on 25 December 2021 and observes primarily in infrared light."
  31. >
  32. > **Answer:** "JWST launched in December 2021 [S1]. It is the largest optical telescope in space [S1]."
  33. | Claim | Decision | Budget gap |
  34. |---|---|---|
  35. | JWST launched in December 2021 | PASS (grounded) | -18.4 bits |
  36. | It is the largest optical telescope in space | FLAG (not entailed) | +37.6 bits |
  37. The first claim is entailed by S1; the second is not (S1 says infrared, not "largest"), so Berry flags it. A negative budget gap means the evidence carries the claim; a positive gap means it does not.
  38. ## How it works
  39. ```mermaid
  40. flowchart LR
  41. A[Answer + cited spans] --> B[Split into claims]
  42. B --> C[Resolve evidence pack<br/>from server-owned ledger]
  43. C --> D[Per-claim information budget<br/>token logprobs / EDFL]
  44. D --> E{budget gap >= target?}
  45. E -- yes --> F[FLAG: unsupported]
  46. E -- no --> G[PASS: grounded]
  47. F --> H[Hash-chained audit record]
  48. G --> H
  49. ```
  50. - **Verifier**: `detect_hallucination` / `audit_trace_budget` (and run-scoped variants) score each claim's grounding budget in bits and decide answer vs flag against a prespecified target.
  51. - **Span ledger**: typed, immutable, provenance-bearing evidence spans; the verifier scores spans resolved from the server-owned ledger, not caller-supplied text.
  52. - **Claim/evidence graph**: `create_claim`, `link_claim_evidence`, `audit_claims`, `list_audits`; `supports`/`contradicts` edges must point at citable evidence.
  53. - **Tamper-evident SQLite ledger**: incremental, hash-chained writes (O(1) per span); load replays the event chain and fails closed on any row, metadata, or chain mismatch.
  54. Each run is one auditable artifact under `~/.berry/runs/<id>/`:
  55. ```text
  56. run.sqlite source of truth: spans, claims, audits, hash-chained events
  57. ledger_events.jsonl the append-only event log
  58. run.json, *.tsv inspection exports (evidence, attempts, claims, audits)
  59. ```
  60. ## Benchmarks
  61. Information-sufficiency / grounding detection: does Berry flag claims the cited evidence does not support? Single logprob pass, `gpt-4.1-mini`.
  62. ![Berry grounding-detection AUROC by dataset](bench/benchmarks.png)
  63. | Dataset | What it measures | AUROC | Recall |
  64. |---|---|---|---|
  65. | VitaminC (SUPPORTS vs NEI) | evidence sufficiency, contrastive | **0.929** | 0.93 |
  66. | HotpotQA (gold vs distractor) | sufficiency, controlled ablation | 0.841 | 0.975 |
  67. | RAGTruth (atomic claims) | grounding on real RAG outputs | 0.819 | 0.884 |
  68. Recall is the fraction of unsupported claims flagged at the default 0.95 target.
  69. **Contrastive control (VitaminC):** hold the claim fixed and change only the evidence. Berry's budget moves in the correct direction in **95%** of paired claims (369/390). The claim is identical, so this isolates evidence sufficiency, not correctness or prior plausibility.
  70. **Ledger:** incremental writes are **~0.15 ms/span and flat** (O(1)), about 25× faster than a full-snapshot JSON store at 600 spans, and they do not degrade as runs grow; span-row and event-chain edits fail closed on load.
  71. Numbers are pilots on `gpt-4.1-mini`, which runs both the atomic decomposition and the verification; a stronger reasoning model (Claude Opus or an o-series thinking model) is expected to lift the RAGTruth recall, where the misses are plausible additions in long responses. The default target is recall-first, so pick an operating point from the risk-coverage curve. Full methodology, baselines, and reproduce steps in [`bench/BENCHMARK.md`](bench/BENCHMARK.md).
  72. ## Supported verifier backends
  73. The verifier requires token logprobs (Chat Completions-style `logprobs` + `top_logprobs`).
  74. - `openai` (default): OpenAI-compatible Chat Completions with logprobs (OpenAI, OpenRouter, local vLLM, or any compatible `base_url`)
  75. - `gemini`: Gemini Developer API `generateContent` with token logprobs
  76. - `vertex`: Vertex AI `generateContent` (Gemini) with token logprobs
  77. - `dummy`: deterministic offline backend for tests/dev
  78. - Anthropic is not supported yet (the OpenAI-compat layer drops `logprobs`)
  79. ## Workflow playbooks
  80. Each playbook has a before/after worked example (uncited output vs evidence-backed + verifier):
  81. - `docs/workflows/README.md`: Search & Learn, Generate Boilerplate/Content, Inline Completions, Greenfield Prototyping, RCA Fix Agent
  82. ## Docs
  83. - `docs/USAGE.md`: task-oriented guides
  84. - `docs/CLI.md`: command reference
  85. - `docs/INSTALL.md`: full platform list, installer flags, scopes, and safety model
  86. - `docs/CONFIGURATION.md`: config files, defaults, env vars
  87. - `docs/MCP.md`: tools, prompts, transport
  88. - `docs/SPANS.md`: span-ledger model, incremental SQLite/event-log persistence, claim/evidence graph, evidence-pack policy
  89. ## Tests
  90. ```bash
  91. pytest
  92. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.