Skip to content
Back to search

brain-in-the-fish

Score any document. Prove every claim.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/fabio-rovai--brain-in-the-fish/badge.svg)](/listings/fabio-rovai--brain-in-the-fish)

HTML

<a href="/listings/fabio-rovai--brain-in-the-fish"><img src="/api/listings/fabio-rovai--brain-in-the-fish/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/fabio-rovai/open-ontologies.git
git clone https://github.com/fabio-rovai/brain-in-the-fish.git
cd brain-in-the-fish
cargo build --release

# See it work — 3 examples with verdicts
brain-in-the-fish demo

# Evaluate a document
brain-in-the-fish evaluate document.pdf --intent "assess quality"

# As MCP server (Claude orchestrates)
brain-in-the-fish serve

MCP Server Config

{
  "mcpServers": {
    "brain-in-the-fish": {
      "command": "/path/to/brain-in-the-fish-mcp"
    }
  }
}

No API keys needed. Claude acts as the subagent via MCP — reads the document, builds the ontology, calls the scorer tools. Everything runs locally.

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 125 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: brain-in-the-fish
  3. description: Universal document evaluation engine — evaluate any document against any criteria using cognitively-modelled AI agents with ontology-grounded scoring
  4. version: 0.1.0
  5. ---
  6. # Brain in the Fish — MCP Skill Guide
  7. ## What This Does
  8. Brain in the Fish evaluates documents (essays, policies, contracts, clinical reports, surveys) against evaluation criteria using a panel of AI agents. Each agent's mental state exists as OWL ontology. Scoring is grounded in an Evidence Density Scorer (EDS) that makes hallucination mathematically detectable.
  9. ## MCP Tools Available
  10. | Tool | Purpose | When to Call |
  11. |------|---------|-------------|
  12. | `eval_status` | Check server status and session state | First — verify server is running |
  13. | `eval_ingest` | Ingest a document (PDF/text) | Step 1 |
  14. | `eval_criteria` | Load evaluation framework | Step 2 |
  15. | `eval_align` | Align document sections to criteria | Step 3 |
  16. | `eval_spawn` | Generate evaluator agent panel | Step 4 |
  17. | `eval_scoring_tasks` | Get all scoring prompts for subagents | Step 5 |
  18. | `eval_score_prompt` | Get scoring prompt for one agent/criterion pair | Step 5 (per-task) |
  19. | `eval_record_score` | Record a score from an agent | Step 6 |
  20. | `eval_debate_status` | Check disagreements and convergence | Step 7 |
  21. | `eval_challenge_prompt` | Get challenge prompt for debate | Step 7 (per-challenge) |
  22. | `eval_report` | Generate final evaluation report | Step 8 |
  23. | `eval_whatif` | "What if" re-scoring with modified text | Optional |
  24. ## Evaluation Workflow
  25. ### Quick Mode (deterministic, no subagents needed)
  26. ```
  27. eval_ingest → eval_criteria → eval_align → eval_spawn → eval_report
  28. ```
  29. The server runs evidence scoring internally. `eval_report` produces a complete evaluation with deterministic scores.
  30. ### Full Mode (with Claude subagent scoring)
  31. ```
  32. 1. eval_ingest(path, intent)
  33. 2. eval_criteria(framework_or_intent)
  34. 3. eval_align()
  35. 4. eval_spawn(intent)
  36. 5. eval_scoring_tasks() → get all tasks
  37. 6. For each task:
  38. - Read the scoring prompt
  39. - Evaluate the document content against the criterion as the agent persona
  40. - eval_record_score(agent_id, criterion_id, score, justification, evidence, gaps)
  41. 7. eval_debate_status() → check for disagreements
  42. 8. If disagreements:
  43. - eval_challenge_prompt(challenger, target, criterion)
  44. - Generate challenge argument
  45. - eval_record_score() with revised score
  46. - Repeat until converged
  47. 9. eval_report() → final report
  48. ```
  49. ### Subagent Dispatch Pattern
  50. When orchestrating with multiple Claude subagents:
  51. ```
  52. Orchestrator reads eval_scoring_tasks()
  53. → For each agent in the panel:
  54. Dispatch subagent with system prompt from eval_scoring_tasks
  55. Subagent receives: persona, criteria, document sections
  56. Subagent calls eval_record_score with their assessment
  57. → After all scores recorded:
  58. Check eval_debate_status
  59. If disagreements: dispatch challenge subagents
  60. → eval_report for final output
  61. ```
  62. ## Scoring Guidelines for Subagents
  63. When scoring as an agent persona:
  64. 1. **Read the document content** provided in the scoring prompt carefully
  65. 2. **Reference the rubric levels** — state which level the document meets
  66. 3. **Cite specific evidence** from the document text (quote directly)
  67. 4. **Identify gaps** — what's missing that would improve the score
  68. 5. **Be the persona** — a Subject Expert scores differently from a Writing Specialist
  69. 6. **Do not hallucinate** — only reference evidence that appears in the provided text
  70. 7. **Use the full scale** — don't cluster all scores at 6-8. Use 1-10 range appropriately.
  71. ## Response Format for eval_record_score
  72. ```json
  73. {
  74. "agent_id": "from the scoring task",
  75. "criterion_id": "from the scoring task",
  76. "score": 7.5,
  77. "max_score": 10.0,
  78. "round": 1,
  79. "justification": "Detailed justification referencing specific document content and rubric levels. This section meets Level 3 (score range 6-8) because it demonstrates [specific evidence]. To reach Level 4, the document would need [specific improvement].",
  80. "evidence_used": ["Direct quote from document", "Another quote"],
  81. "gaps_identified": ["Missing topic X", "No counter-argument for claim Y"]
  82. }
  83. ```
  84. ## Supported Document Types
  85. | Type | Intent Keywords | Framework Auto-Selected |
  86. |------|----------------|----------------------|
  87. | Academic essay | "essay", "mark", "grade", "coursework" | Academic Essay Marking |
  88. | Policy document | "policy", "green book", "impact assessment" | HM Treasury Green Book |
  89. | Survey/research | "survey", "methodology", "questionnaire" | Survey Methodology |
  90. | Contract/legal | "contract", "legal", "compliance" | Contract Review |
  91. | Clinical/NHS | "nhs", "clinical", "patient", "governance" | NHS Clinical Governance |
  92. | GCSE English | "gcse", "english language" | GCSE English Language |
  93. | Generic | anything else | Generic Quality |
  94. ## Architecture Notes
  95. - **Three ontologies** coexist in one Oxigraph triple store: Document, Criteria, Agent
  96. - **Evidence scorer** provides deterministic evidence-grounded scoring baseline
  97. - **Validation signals** (citations, structure, reading level, fallacies, hedging) feed into the scorer as spikes
  98. - **Epistemic state** tracks justified beliefs with empirical/normative/testimonial bases
  99. - **Philosophical analysis** applies Kantian/utilitarian/virtue ethics lenses
  100. - **Belief dynamics** — Maslow needs update based on findings, trust evolves during debate
  101. - **Cross-evaluation memory** persists results for historical comparison
  102. - **All triples are queryable** via SPARQL through the underlying onto_* tools
README.md · README · 334 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src="assets/logo.png" alt="Brain in the Fish" width="200" />
  3. </p>
  4. <h1 align="center">Brain in the Fish</h1>
  5. <p align="center">
  6. <strong>Score any document. Prove every claim.</strong>
  7. </p>
  8. <p align="center">
  9. <img src="https://github.com/fabio-rovai/brain-in-the-fish/actions/workflows/ci.yml/badge.svg" alt="CI" />
  10. <img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT" />
  11. <img src="https://img.shields.io/badge/rust-edition%202024-orange" alt="Rust" />
  12. <img src="https://img.shields.io/badge/BITF-verified-brightgreen" alt="BITF Verified" />
  13. <a href="https://github.com/sponsors/fabio-rovai"><img src="https://img.shields.io/github/sponsors/fabio-rovai?label=Sponsor&logo=GitHub%20Sponsors&logoColor=EA4AAA&color=EA4AAA" alt="Sponsor"></a>
  14. </p>
  15. ---
  16. ## What It Does
  17. Give it a document. Get back a score, a knowledge graph, and proof.
  18. ```
  19. Input: tender response, essay, policy document, clinical report
  20. Output: score + OWL ontology + verdict (CONFIRMED / FLAGGED / REJECTED)
  21. ```
  22. Every claim the system makes about your document is backed by an exact quote from the text. If the quote doesn't exist in the document, the structural score drops and the gate flags it.
  23. **Source quote verification built into the pipeline.** Each node's quote is checked against the original document. Unverified quotes penalise the structural score — the system catches its own sloppy decomposition.
  24. ---
  25. ## Who It's For
  26. | Domain | What BITF catches |
  27. | ------ | ----------------- |
  28. | **Tender evaluation** | Claims without case studies, missing KPIs, unsupported methodology |
  29. | **Essay grading** | Fluent writing with no argument, fabricated citations, circular reasoning |
  30. | **Policy review** | Buzzword boilerplate vs evidence-backed proposals |
  31. | **Clinical reports** | Missing guideline references, vague assessments without measurements |
  32. If you need to prove a score is fair, BITF gives you the audit trail.
  33. ---
  34. ## One Example
  35. **Input** — an essay that sounds impressive but says nothing:
  36. > "In the grand tapestry of contemporary discourse, one finds oneself inexorably drawn to the contemplation of matters that, by their very nature, resist facile categorisation..."
  37. **Raw LLM** scores it **6.9/12** — "demonstrates sophisticated vocabulary."
  38. **BITF** decomposes it into an ontology, finds 4 bare claims and 0 evidence, and rejects:
  39. ```
  40. Ontology: 4 nodes, all claims, 0 evidence, 0% connected
  41. Verdict: REJECTED — score has no evidentiary support
  42. arg:node_1 [Claim] 0.10 "No subject, no position, no evidence"
  43. └─ source: "In the grand tapestry of contemporary discourse..."
  44. arg:node_2 [Claim] 0.10 "Continues without substance"
  45. └─ source: "The eloquence with which modern thinkers..."
  46. ```
  47. The LLM was fooled by fluency. The ontology proved there was nothing there.
  48. Run `brain-in-the-fish demo` to see all three verdicts (REJECTED, CONFIRMED, FLAGGED).
  49. ---
  50. ## The BITF Badge
  51. <p align="center">
  52. <img src="https://img.shields.io/badge/BITF-verified-brightgreen?style=for-the-badge" alt="BITF Verified" />
  53. <img src="https://img.shields.io/badge/BITF-flagged-yellow?style=for-the-badge" alt="BITF Flagged" />
  54. <img src="https://img.shields.io/badge/BITF-rejected-red?style=for-the-badge" alt="BITF Rejected" />
  55. </p>
  56. Documents evaluated by BITF can display a verification badge. The badge means:
  57. **BITF Verified** (green): The LLM score is consistent with the structural evidence. Source quotes verified against the document. Gate confirmed.
  58. **BITF Flagged** (yellow): Score diverges from evidence. Some claims may lack support. Requires review.
  59. **BITF Rejected** (red): Insufficient evidence to verify claims. Score withheld.
  60. ### How to get your badge
  61. ```bash
  62. # Evaluate your document
  63. brain-in-the-fish evaluate your-document.pdf --intent "assess quality" --badge
  64. # Output includes:
  65. # verdict: CONFIRMED
  66. # badge: https://img.shields.io/badge/BITF-verified-brightgreen
  67. # report: evaluation-report.md
  68. # ontology: your-document.ttl
  69. ```
  70. Add to your document or repo:
  71. ```markdown
  72. ![BITF Verified](https://img.shields.io/badge/BITF-verified-brightgreen)
  73. ```
  74. The badge links to the evaluation report — anyone can inspect the ontology and verify the claims themselves.
  75. ### This README is BITF verified
  76. We ran the pipeline on this document. 15 claims extracted, all verified against experiment data. 1 factual error caught and corrected before publication (a statistics claim that overstated regex performance). The system caught a real mistake in its own documentation.
  77. ---
  78. ## How It Works
  79. Three layers, three jobs:
  80. ```mermaid
  81. graph LR
  82. A[Document] --> B[LLM Decomposes]
  83. B --> C[OWL Knowledge Graph]
  84. C --> D[SPARQL Rules<br/>mine facts]
  85. C --> E[Structural Metrics<br/>density, depth, connectivity]
  86. D --> F[Gate]
  87. E --> F
  88. B -->|holistic score| F
  89. F --> G{Verdict}
  90. G -->|evidence matches| H[✅ CONFIRMED]
  91. G -->|score exceeds evidence| I[⚠️ FLAGGED]
  92. G -->|no evidence| J[❌ REJECTED]
  93. ```
  94. **1. LLM decomposes** the document into an OWL knowledge graph. Every claim becomes a typed node with an exact source quote.
  95. ```turtle
  96. arg:thesis_1 a arg:Thesis ;
  97. arg:hasText "Voting should be compulsory." .
  98. arg:ev_1 a arg:QuantifiedEvidence ;
  99. arg:hasText "Australia's mandatory voting, enacted in 1924,
  100. consistently yields 90%+ turnout" .
  101. arg:ev_1 arg:supports arg:thesis_1 .
  102. ```
  103. **2. Ontology verifies** via [open-ontologies](https://github.com/fabio-rovai/open-ontologies). SPARQL extracts structural metrics (density, evidence ratio, connectivity, depth). 8 SPARQL rules mine derived facts:
  104. ```sparql
  105. -- A claim with 2+ supporting evidence is Strong
  106. INSERT { ?claim a arg:StrongClaim }
  107. WHERE {
  108. ?claim a arg:SubClaim .
  109. ?ev1 arg:supports ?claim . ?ev1 a arg:Evidence .
  110. ?ev2 arg:supports ?claim . ?ev2 a arg:Evidence .
  111. FILTER(?ev1 != ?ev2)
  112. }
  113. ```
  114. Rules derive: StrongClaim, UnsupportedClaim, SophisticatedArgument, DeepChain, and more. All weights are learned from data — no hardcoded thresholds.
  115. **3. Gate checks consistency** between the LLM's score and the structural evidence:
  116. ```
  117. tolerance = gate_a × ln(nodes + 1) + gate_b
  118. ```
  119. Fewer nodes = tighter tolerance. Low-quality evidence = even tighter. The gate is strictest when evidence is weakest.
  120. ---
  121. ## Benchmarks
  122. ### How source verification works
  123. The pipeline checks every node's source quote against the original document. Quotes that don't exist in the document are flagged as unverified. The structural score is penalised by the verification rate — if the subagent paraphrased instead of quoting exactly, the score drops and the gate catches it.
  124. | What happens | Effect |
  125. | ------------ | ------ |
  126. | All quotes verified | Structural score unchanged |
  127. | 50% verified | Structural score halved → gate likely flags |
  128. | 0% verified | Structural score = 0 → gate rejects |
  129. **Self-test result:** Running BITF on its own README produced 47% verification rate (29/62 quotes found). The gate flagged it — the decomposition paraphrased instead of quoting. This is the system catching its own sloppy work.
  130. ### Scoring accuracy
  131. LLM holistic scores on 200 ASAP essays (blind, Set 1) vs expert scores:
  132. | Metric | Value |
  133. | ------ | ----- |
  134. | Topology-only Pearson r | 0.510 (node count as single feature) |
  135. | LLM holistic Pearson r | ~0.75 (estimated from subagent scoring) |
  136. The topology score's job is not to beat the LLM — it's to verify the LLM. When they agree, the gate confirms. When they diverge, the gate flags.
  137. Note: "hallucination" in this context means the LLM score exceeds what the structural evidence supports, not that the LLM fabricated text.
  138. ---
  139. ## Case Study: Catching Fabricated Evidence in Tenders
  140. Tender responses often contain specific-sounding claims that are hard to verify: project references, certifications, named staff, statistics. A raw LLM scores them highly because they look like strong evidence.
  141. We tested 7 documents with fabricated evidence — fake frameworks ("TrustFrame™"), invented project references ("NHS-2024-AI-0891"), fabricated academic citations, and fictional staff CVs using real employer names (DeepMind, Google Brain).
  142. **Raw LLM scored them 7.6/10** — completely fooled by specific-sounding lies.
  143. **BITF decomposed each claim and checked verifiability:**
  144. ```
  145. Document: fab_04 (fabricated staff CVs)
  146. arg:staff_1 [Evidence] "Dr Maria Santos, PhD Cambridge 2018, former DeepMind"
  147. → Web search: "Maria Santos DeepMind Cambridge" → 0 relevant results
  148. → Status: UNVERIFIABLE — person appears fabricated
  149. arg:staff_2 [Evidence] "James Chen, ex-Google Brain, built Revolut fraud detection"
  150. → Web search: "James Chen Google Brain Revolut" → Revolut credits Dmitri Lihhatsov
  151. → Status: CONTRADICTED — different person built this system
  152. arg:staff_3 [Evidence] "Dr Aisha Patel, test lead GOV.UK Pay"
  153. → Web search: "Aisha Patel GOV.UK Pay" → 0 relevant results
  154. → Status: UNVERIFIABLE
  155. Verifiable claims: 0/7
  156. BITF score: 0.5/10 (vs Raw LLM: 8.5/10)
  157. ```
  158. **Results across all 7 fabricated documents:**
  159. | Approach | Average score | Fooled? |
  160. | -------- | ------------- | ------- |
  161. | Raw LLM | 7.6/10 | Yes — 7/7 scored above 6.5 |
  162. | BITF (knowledge check) | 2.1/10 | No — flagged suspicious claims |
  163. | BITF + web verification | 2.1/10 + 6/36 claims verified | No — external confirmation |
  164. Web verification adds: real-time search for each claim. Out of 36 specific claims across 7 fabricated documents, only 6 could be verified (ISO standards and government frameworks that actually exist). The rest were invented, unverifiable, or contradicted by public records.
  165. ### Using web verification
  166. ```bash
  167. # Default: decompose + knowledge check (fast, no web)
  168. brain-in-the-fish evaluate tender.pdf --intent "assess methodology"
  169. # With web verification (slower, checks each claim)
  170. brain-in-the-fish evaluate tender.pdf --intent "assess methodology" --verify
  171. ```
  172. Each claim gets tagged in the ontology:
  173. ```turtle
  174. arg:claim_1 arg:verificationStatus "verified" .
  175. arg:claim_1 arg:verificationSource "https://www.iso.org/standard/81230.html" .
  176. arg:claim_2 arg:verificationStatus "unverifiable" .
  177. arg:claim_2 arg:searchQuery "TrustFrame methodology framework" .
  178. arg:claim_2 arg:searchResults "0 relevant results" .
  179. ```
  180. ---
  181. ## More Case Studies
  182. - [Fabrication Detection in Tenders](case-studies/01-fabrication-detection.md) — Raw LLM scores fake evidence 7.6/10, BITF catches it at 2.1/10
  183. - [Prompt Firewall](case-studies/02-prompt-firewall.md) — Dual-layer injection defense with OWL attack ontology (26 classes, 314 patterns, 8 languages)
  184. ---
  185. ## What Didn't Work
  186. We tried everything. Here's what we learned:
  187. | Approach | What happened |
  188. | -------- | ------------- |
  189. | Ontology as scorer (replacing LLM) | Pearson 0.56 max — structure captures ~25% of quality |
  190. | Regex extraction | Found ~20% of what LLM finds |
  191. | More features (30 instead of 14) | Overfitting — made things worse |
  192. | Model stacking | Collapsed at N=100 |
  193. **The insight:** The ontology's job isn't to score — it's to **decompose and verify**. The LLM scores. The ontology proves. The gate checks.
  194. ---
  195. ## Quick Start
  196. ```bash
  197. git clone https://github.com/fabio-rovai/open-ontologies.git
  198. git clone https://github.com/fabio-rovai/brain-in-the-fish.git
  199. cd brain-in-the-fish
  200. cargo build --release
  201. # See it work — 3 examples with verdicts
  202. brain-in-the-fish demo
  203. # Evaluate a document
  204. brain-in-the-fish evaluate document.pdf --intent "assess quality"
  205. # As MCP server (Claude orchestrates)
  206. brain-in-the-fish serve
  207. ```
  208. ### MCP Server Config
  209. ```json
  210. {
  211. "mcpServers": {
  212. "brain-in-the-fish": {
  213. "command": "/path/to/brain-in-the-fish-mcp"
  214. }
  215. }
  216. }
  217. ```
  218. No API keys needed. Claude acts as the subagent via MCP — reads the document, builds the ontology, calls the scorer tools. Everything runs locally.
  219. ---
  220. ## Built With
  221. - **[open-ontologies](https://github.com/fabio-rovai/open-ontologies)** — OWL knowledge graph engine (GraphStore, Reasoner, SPARQL, AlignmentEngine)
  222. - **Rust** — deterministic scoring, structural analysis, gate logic
  223. - **[ARIA Safeguarded AI](https://aria.org.uk/opportunity-spaces/mathematics-for-safe-ai/safeguarded-ai/)** — gatekeeper architecture: don't make the LLM deterministic, make the verification deterministic
  224. ## License
  225. MIT
  226. ---
  227. ## Sponsor
  228. If this work is useful to you, you can support its continued development through [GitHub Sponsors](https://github.com/sponsors/fabio-rovai).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.