brain-in-the-fish
Score any document. Prove every claim.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/fabio-rovai--brain-in-the-fish)HTML
<a href="/listings/fabio-rovai--brain-in-the-fish"><img src="/api/listings/fabio-rovai--brain-in-the-fish/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
git clone https://github.com/fabio-rovai/open-ontologies.git git clone https://github.com/fabio-rovai/brain-in-the-fish.git cd brain-in-the-fish cargo build --release # See it work — 3 examples with verdicts brain-in-the-fish demo # Evaluate a document brain-in-the-fish evaluate document.pdf --intent "assess quality" # As MCP server (Claude orchestrates) brain-in-the-fish serve
MCP Server Config
{
"mcpServers": {
"brain-in-the-fish": {
"command": "/path/to/brain-in-the-fish-mcp"
}
}
}No API keys needed. Claude acts as the subagent via MCP — reads the document, builds the ontology, calls the scorer tools. Everything runs locally.
---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 125 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- ---
- name: brain-in-the-fish
- description: Universal document evaluation engine — evaluate any document against any criteria using cognitively-modelled AI agents with ontology-grounded scoring
- version: 0.1.0
- ---
- # Brain in the Fish — MCP Skill Guide
- ## What This Does
- Brain in the Fish evaluates documents (essays, policies, contracts, clinical reports, surveys) against evaluation criteria using a panel of AI agents. Each agent's mental state exists as OWL ontology. Scoring is grounded in an Evidence Density Scorer (EDS) that makes hallucination mathematically detectable.
- ## MCP Tools Available
- | Tool | Purpose | When to Call |
- |------|---------|-------------|
- | `eval_status` | Check server status and session state | First — verify server is running |
- | `eval_ingest` | Ingest a document (PDF/text) | Step 1 |
- | `eval_criteria` | Load evaluation framework | Step 2 |
- | `eval_align` | Align document sections to criteria | Step 3 |
- | `eval_spawn` | Generate evaluator agent panel | Step 4 |
- | `eval_scoring_tasks` | Get all scoring prompts for subagents | Step 5 |
- | `eval_score_prompt` | Get scoring prompt for one agent/criterion pair | Step 5 (per-task) |
- | `eval_record_score` | Record a score from an agent | Step 6 |
- | `eval_debate_status` | Check disagreements and convergence | Step 7 |
- | `eval_challenge_prompt` | Get challenge prompt for debate | Step 7 (per-challenge) |
- | `eval_report` | Generate final evaluation report | Step 8 |
- | `eval_whatif` | "What if" re-scoring with modified text | Optional |
- ## Evaluation Workflow
- ### Quick Mode (deterministic, no subagents needed)
- ```
- eval_ingest → eval_criteria → eval_align → eval_spawn → eval_report
- ```
- The server runs evidence scoring internally. `eval_report` produces a complete evaluation with deterministic scores.
- ### Full Mode (with Claude subagent scoring)
- ```
- 1. eval_ingest(path, intent)
- 2. eval_criteria(framework_or_intent)
- 3. eval_align()
- 4. eval_spawn(intent)
- 5. eval_scoring_tasks() → get all tasks
- 6. For each task:
- - Read the scoring prompt
- - Evaluate the document content against the criterion as the agent persona
- - eval_record_score(agent_id, criterion_id, score, justification, evidence, gaps)
- 7. eval_debate_status() → check for disagreements
- 8. If disagreements:
- - eval_challenge_prompt(challenger, target, criterion)
- - Generate challenge argument
- - eval_record_score() with revised score
- - Repeat until converged
- 9. eval_report() → final report
- ```
- ### Subagent Dispatch Pattern
- When orchestrating with multiple Claude subagents:
- ```
- Orchestrator reads eval_scoring_tasks()
- → For each agent in the panel:
- Dispatch subagent with system prompt from eval_scoring_tasks
- Subagent receives: persona, criteria, document sections
- Subagent calls eval_record_score with their assessment
- → After all scores recorded:
- Check eval_debate_status
- If disagreements: dispatch challenge subagents
- → eval_report for final output
- ```
- ## Scoring Guidelines for Subagents
- When scoring as an agent persona:
- 1. **Read the document content** provided in the scoring prompt carefully
- 2. **Reference the rubric levels** — state which level the document meets
- 3. **Cite specific evidence** from the document text (quote directly)
- 4. **Identify gaps** — what's missing that would improve the score
- 5. **Be the persona** — a Subject Expert scores differently from a Writing Specialist
- 6. **Do not hallucinate** — only reference evidence that appears in the provided text
- 7. **Use the full scale** — don't cluster all scores at 6-8. Use 1-10 range appropriately.
- ## Response Format for eval_record_score
- ```json
- {
- "agent_id": "from the scoring task",
- "criterion_id": "from the scoring task",
- "score": 7.5,
- "max_score": 10.0,
- "round": 1,
- "justification": "Detailed justification referencing specific document content and rubric levels. This section meets Level 3 (score range 6-8) because it demonstrates [specific evidence]. To reach Level 4, the document would need [specific improvement].",
- "evidence_used": ["Direct quote from document", "Another quote"],
- "gaps_identified": ["Missing topic X", "No counter-argument for claim Y"]
- }
- ```
- ## Supported Document Types
- | Type | Intent Keywords | Framework Auto-Selected |
- |------|----------------|----------------------|
- | Academic essay | "essay", "mark", "grade", "coursework" | Academic Essay Marking |
- | Policy document | "policy", "green book", "impact assessment" | HM Treasury Green Book |
- | Survey/research | "survey", "methodology", "questionnaire" | Survey Methodology |
- | Contract/legal | "contract", "legal", "compliance" | Contract Review |
- | Clinical/NHS | "nhs", "clinical", "patient", "governance" | NHS Clinical Governance |
- | GCSE English | "gcse", "english language" | GCSE English Language |
- | Generic | anything else | Generic Quality |
- ## Architecture Notes
- - **Three ontologies** coexist in one Oxigraph triple store: Document, Criteria, Agent
- - **Evidence scorer** provides deterministic evidence-grounded scoring baseline
- - **Validation signals** (citations, structure, reading level, fallacies, hedging) feed into the scorer as spikes
- - **Epistemic state** tracks justified beliefs with empirical/normative/testimonial bases
- - **Philosophical analysis** applies Kantian/utilitarian/virtue ethics lenses
- - **Belief dynamics** — Maslow needs update based on findings, trust evolves during debate
- - **Cross-evaluation memory** persists results for historical comparison
- - **All triples are queryable** via SPARQL through the underlying onto_* tools
README.md · README · 334 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <p align="center">
- <img src="assets/logo.png" alt="Brain in the Fish" width="200" />
- </p>
- <h1 align="center">Brain in the Fish</h1>
- <p align="center">
- <strong>Score any document. Prove every claim.</strong>
- </p>
- <p align="center">
- <img src="https://github.com/fabio-rovai/brain-in-the-fish/actions/workflows/ci.yml/badge.svg" alt="CI" />
- <img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT" />
- <img src="https://img.shields.io/badge/rust-edition%202024-orange" alt="Rust" />
- <img src="https://img.shields.io/badge/BITF-verified-brightgreen" alt="BITF Verified" />
- <a href="https://github.com/sponsors/fabio-rovai"><img src="https://img.shields.io/github/sponsors/fabio-rovai?label=Sponsor&logo=GitHub%20Sponsors&logoColor=EA4AAA&color=EA4AAA" alt="Sponsor"></a>
- </p>
- ---
- ## What It Does
- Give it a document. Get back a score, a knowledge graph, and proof.
- ```
- Input: tender response, essay, policy document, clinical report
- Output: score + OWL ontology + verdict (CONFIRMED / FLAGGED / REJECTED)
- ```
- Every claim the system makes about your document is backed by an exact quote from the text. If the quote doesn't exist in the document, the structural score drops and the gate flags it.
- **Source quote verification built into the pipeline.** Each node's quote is checked against the original document. Unverified quotes penalise the structural score — the system catches its own sloppy decomposition.
- ---
- ## Who It's For
- | Domain | What BITF catches |
- | ------ | ----------------- |
- | **Tender evaluation** | Claims without case studies, missing KPIs, unsupported methodology |
- | **Essay grading** | Fluent writing with no argument, fabricated citations, circular reasoning |
- | **Policy review** | Buzzword boilerplate vs evidence-backed proposals |
- | **Clinical reports** | Missing guideline references, vague assessments without measurements |
- If you need to prove a score is fair, BITF gives you the audit trail.
- ---
- ## One Example
- **Input** — an essay that sounds impressive but says nothing:
- > "In the grand tapestry of contemporary discourse, one finds oneself inexorably drawn to the contemplation of matters that, by their very nature, resist facile categorisation..."
- **Raw LLM** scores it **6.9/12** — "demonstrates sophisticated vocabulary."
- **BITF** decomposes it into an ontology, finds 4 bare claims and 0 evidence, and rejects:
- ```
- Ontology: 4 nodes, all claims, 0 evidence, 0% connected
- Verdict: REJECTED — score has no evidentiary support
- arg:node_1 [Claim] 0.10 "No subject, no position, no evidence"
- └─ source: "In the grand tapestry of contemporary discourse..."
- arg:node_2 [Claim] 0.10 "Continues without substance"
- └─ source: "The eloquence with which modern thinkers..."
- ```
- The LLM was fooled by fluency. The ontology proved there was nothing there.
- Run `brain-in-the-fish demo` to see all three verdicts (REJECTED, CONFIRMED, FLAGGED).
- ---
- ## The BITF Badge
- <p align="center">
- <img src="https://img.shields.io/badge/BITF-verified-brightgreen?style=for-the-badge" alt="BITF Verified" />
- <img src="https://img.shields.io/badge/BITF-flagged-yellow?style=for-the-badge" alt="BITF Flagged" />
- <img src="https://img.shields.io/badge/BITF-rejected-red?style=for-the-badge" alt="BITF Rejected" />
- </p>
- Documents evaluated by BITF can display a verification badge. The badge means:
- **BITF Verified** (green): The LLM score is consistent with the structural evidence. Source quotes verified against the document. Gate confirmed.
- **BITF Flagged** (yellow): Score diverges from evidence. Some claims may lack support. Requires review.
- **BITF Rejected** (red): Insufficient evidence to verify claims. Score withheld.
- ### How to get your badge
- ```bash
- # Evaluate your document
- brain-in-the-fish evaluate your-document.pdf --intent "assess quality" --badge
- # Output includes:
- # verdict: CONFIRMED
- # badge: https://img.shields.io/badge/BITF-verified-brightgreen
- # report: evaluation-report.md
- # ontology: your-document.ttl
- ```
- Add to your document or repo:
- ```markdown
- 
- ```
- The badge links to the evaluation report — anyone can inspect the ontology and verify the claims themselves.
- ### This README is BITF verified
- We ran the pipeline on this document. 15 claims extracted, all verified against experiment data. 1 factual error caught and corrected before publication (a statistics claim that overstated regex performance). The system caught a real mistake in its own documentation.
- ---
- ## How It Works
- Three layers, three jobs:
- ```mermaid
- graph LR
- A[Document] --> B[LLM Decomposes]
- B --> C[OWL Knowledge Graph]
- C --> D[SPARQL Rules<br/>mine facts]
- C --> E[Structural Metrics<br/>density, depth, connectivity]
- D --> F[Gate]
- E --> F
- B -->|holistic score| F
- F --> G{Verdict}
- G -->|evidence matches| H[✅ CONFIRMED]
- G -->|score exceeds evidence| I[⚠️ FLAGGED]
- G -->|no evidence| J[❌ REJECTED]
- ```
- **1. LLM decomposes** the document into an OWL knowledge graph. Every claim becomes a typed node with an exact source quote.
- ```turtle
- arg:thesis_1 a arg:Thesis ;
- arg:hasText "Voting should be compulsory." .
- arg:ev_1 a arg:QuantifiedEvidence ;
- arg:hasText "Australia's mandatory voting, enacted in 1924,
- consistently yields 90%+ turnout" .
- arg:ev_1 arg:supports arg:thesis_1 .
- ```
- **2. Ontology verifies** via [open-ontologies](https://github.com/fabio-rovai/open-ontologies). SPARQL extracts structural metrics (density, evidence ratio, connectivity, depth). 8 SPARQL rules mine derived facts:
- ```sparql
- -- A claim with 2+ supporting evidence is Strong
- INSERT { ?claim a arg:StrongClaim }
- WHERE {
- ?claim a arg:SubClaim .
- ?ev1 arg:supports ?claim . ?ev1 a arg:Evidence .
- ?ev2 arg:supports ?claim . ?ev2 a arg:Evidence .
- FILTER(?ev1 != ?ev2)
- }
- ```
- Rules derive: StrongClaim, UnsupportedClaim, SophisticatedArgument, DeepChain, and more. All weights are learned from data — no hardcoded thresholds.
- **3. Gate checks consistency** between the LLM's score and the structural evidence:
- ```
- tolerance = gate_a × ln(nodes + 1) + gate_b
- ```
- Fewer nodes = tighter tolerance. Low-quality evidence = even tighter. The gate is strictest when evidence is weakest.
- ---
- ## Benchmarks
- ### How source verification works
- The pipeline checks every node's source quote against the original document. Quotes that don't exist in the document are flagged as unverified. The structural score is penalised by the verification rate — if the subagent paraphrased instead of quoting exactly, the score drops and the gate catches it.
- | What happens | Effect |
- | ------------ | ------ |
- | All quotes verified | Structural score unchanged |
- | 50% verified | Structural score halved → gate likely flags |
- | 0% verified | Structural score = 0 → gate rejects |
- **Self-test result:** Running BITF on its own README produced 47% verification rate (29/62 quotes found). The gate flagged it — the decomposition paraphrased instead of quoting. This is the system catching its own sloppy work.
- ### Scoring accuracy
- LLM holistic scores on 200 ASAP essays (blind, Set 1) vs expert scores:
- | Metric | Value |
- | ------ | ----- |
- | Topology-only Pearson r | 0.510 (node count as single feature) |
- | LLM holistic Pearson r | ~0.75 (estimated from subagent scoring) |
- The topology score's job is not to beat the LLM — it's to verify the LLM. When they agree, the gate confirms. When they diverge, the gate flags.
- Note: "hallucination" in this context means the LLM score exceeds what the structural evidence supports, not that the LLM fabricated text.
- ---
- ## Case Study: Catching Fabricated Evidence in Tenders
- Tender responses often contain specific-sounding claims that are hard to verify: project references, certifications, named staff, statistics. A raw LLM scores them highly because they look like strong evidence.
- We tested 7 documents with fabricated evidence — fake frameworks ("TrustFrame™"), invented project references ("NHS-2024-AI-0891"), fabricated academic citations, and fictional staff CVs using real employer names (DeepMind, Google Brain).
- **Raw LLM scored them 7.6/10** — completely fooled by specific-sounding lies.
- **BITF decomposed each claim and checked verifiability:**
- ```
- Document: fab_04 (fabricated staff CVs)
- arg:staff_1 [Evidence] "Dr Maria Santos, PhD Cambridge 2018, former DeepMind"
- → Web search: "Maria Santos DeepMind Cambridge" → 0 relevant results
- → Status: UNVERIFIABLE — person appears fabricated
- arg:staff_2 [Evidence] "James Chen, ex-Google Brain, built Revolut fraud detection"
- → Web search: "James Chen Google Brain Revolut" → Revolut credits Dmitri Lihhatsov
- → Status: CONTRADICTED — different person built this system
- arg:staff_3 [Evidence] "Dr Aisha Patel, test lead GOV.UK Pay"
- → Web search: "Aisha Patel GOV.UK Pay" → 0 relevant results
- → Status: UNVERIFIABLE
- Verifiable claims: 0/7
- BITF score: 0.5/10 (vs Raw LLM: 8.5/10)
- ```
- **Results across all 7 fabricated documents:**
- | Approach | Average score | Fooled? |
- | -------- | ------------- | ------- |
- | Raw LLM | 7.6/10 | Yes — 7/7 scored above 6.5 |
- | BITF (knowledge check) | 2.1/10 | No — flagged suspicious claims |
- | BITF + web verification | 2.1/10 + 6/36 claims verified | No — external confirmation |
- Web verification adds: real-time search for each claim. Out of 36 specific claims across 7 fabricated documents, only 6 could be verified (ISO standards and government frameworks that actually exist). The rest were invented, unverifiable, or contradicted by public records.
- ### Using web verification
- ```bash
- # Default: decompose + knowledge check (fast, no web)
- brain-in-the-fish evaluate tender.pdf --intent "assess methodology"
- # With web verification (slower, checks each claim)
- brain-in-the-fish evaluate tender.pdf --intent "assess methodology" --verify
- ```
- Each claim gets tagged in the ontology:
- ```turtle
- arg:claim_1 arg:verificationStatus "verified" .
- arg:claim_1 arg:verificationSource "https://www.iso.org/standard/81230.html" .
- arg:claim_2 arg:verificationStatus "unverifiable" .
- arg:claim_2 arg:searchQuery "TrustFrame methodology framework" .
- arg:claim_2 arg:searchResults "0 relevant results" .
- ```
- ---
- ## More Case Studies
- - [Fabrication Detection in Tenders](case-studies/01-fabrication-detection.md) — Raw LLM scores fake evidence 7.6/10, BITF catches it at 2.1/10
- - [Prompt Firewall](case-studies/02-prompt-firewall.md) — Dual-layer injection defense with OWL attack ontology (26 classes, 314 patterns, 8 languages)
- ---
- ## What Didn't Work
- We tried everything. Here's what we learned:
- | Approach | What happened |
- | -------- | ------------- |
- | Ontology as scorer (replacing LLM) | Pearson 0.56 max — structure captures ~25% of quality |
- | Regex extraction | Found ~20% of what LLM finds |
- | More features (30 instead of 14) | Overfitting — made things worse |
- | Model stacking | Collapsed at N=100 |
- **The insight:** The ontology's job isn't to score — it's to **decompose and verify**. The LLM scores. The ontology proves. The gate checks.
- ---
- ## Quick Start
- ```bash
- git clone https://github.com/fabio-rovai/open-ontologies.git
- git clone https://github.com/fabio-rovai/brain-in-the-fish.git
- cd brain-in-the-fish
- cargo build --release
- # See it work — 3 examples with verdicts
- brain-in-the-fish demo
- # Evaluate a document
- brain-in-the-fish evaluate document.pdf --intent "assess quality"
- # As MCP server (Claude orchestrates)
- brain-in-the-fish serve
- ```
- ### MCP Server Config
- ```json
- {
- "mcpServers": {
- "brain-in-the-fish": {
- "command": "/path/to/brain-in-the-fish-mcp"
- }
- }
- }
- ```
- No API keys needed. Claude acts as the subagent via MCP — reads the document, builds the ontology, calls the scorer tools. Everything runs locally.
- ---
- ## Built With
- - **[open-ontologies](https://github.com/fabio-rovai/open-ontologies)** — OWL knowledge graph engine (GraphStore, Reasoner, SPARQL, AlignmentEngine)
- - **Rust** — deterministic scoring, structural analysis, gate logic
- - **[ARIA Safeguarded AI](https://aria.org.uk/opportunity-spaces/mathematics-for-safe-ai/safeguarded-ai/)** — gatekeeper architecture: don't make the LLM deterministic, make the verification deterministic
- ## License
- MIT
- ---
- ## Sponsor
- If this work is useful to you, you can support its continued development through [GitHub Sponsors](https://github.com/sponsors/fabio-rovai).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.