Skip to content
Back to search

deepdive

Deepdive skill for Claude Code — 12-phase research pipeline: plan-review gate, parallel sub-agent search, claims-ledger triangulation with dissent protection, relevance × authority evidence filter, multi-angle red team, four-layer citation verification. 105 blocks, 29 channels, 460+ stat sources, 47 APIs, 1072 verified endpoints.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Socialpranker/deepdive with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Socialpranker/deepdive'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/socialpranker--deepdive/badge.svg)](/listings/socialpranker--deepdive)

HTML

<a href="/listings/socialpranker--deepdive"><img src="/api/listings/socialpranker--deepdive/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

<details open> <summary><b>For Claude Code (CLI)</b></summary>

git clone https://github.com/Socialpranker/deepdive.git \
  ~/.claude/skills/deepdive

That's it. Now type any of these in a Claude Code session: - "Investigate X" - "Изучи тему" - "Validate this hypothesis"

</details>

<details> <summary><b>For Claude Desktop (Skills enabled)</b></summary>

# Clone
git clone https://github.com/Socialpranker/deepdive.git
cd deepdive

# Package as .skill bundle
zip -r ../deepdive.skill . -x ".*" -x "*.zip"

# Upload via Claude.app → Settings → Skills → Add Skill

</details>

<details> <summary><b>For other LLMs (Codex, Gemini, local)</b></summary>

The 13-phase methodology is portable. Load `SKILL.md` + `references/*.md` into the LLM's context manually. Skip the sub-agent parts and use separate chat sessions per subtopic.

[Full instructions →](#use-with-other-llms-codex-gemini-etc)

</details>

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 16 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "deepdive",
  3. "owner": {
  4. "name": "Socialpranker"
  5. },
  6. "plugins": [
  7. {
  8. "name": "deepdive",
  9. "source": "./",
  10. "description": "Meta-research под вопрос или решение: веб-поиск, академические источники, Q&A отчёт с атомарными пере-используемыми тезисами.",
  11. "skills": [
  12. "./"
  13. ]
  14. }
  15. ]
  16. }
README.md · README · 580 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. <img src="assets/logo.svg" width="96" height="96" alt="deepdive logo">
  3. # Deepdive
  4. ### A structured meta-research skill for Claude Code
  5. **Stop ad-hoc Googling. Start documented investigation.**
  6. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=flat-square)](LICENSE)
  7. [![Claude Code](https://img.shields.io/badge/Claude%20Code-Compatible-d97757?style=flat-square)](https://claude.com/claude-code)
  8. [![Skills](https://img.shields.io/badge/Anthropic-Agent%20Skills-d97757?style=flat-square)](https://docs.anthropic.com/claude/docs/skills)
  9. [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen?style=flat-square)](CONTRIBUTING.md)
  10. [![Auto-Validated](https://img.shields.io/badge/APIs-Auto--Validated%20Weekly-blueviolet?style=flat-square)](.github/workflows/catalog-sync.yml)
  11. [![Stars](https://img.shields.io/github/stars/Socialpranker/deepdive?style=flat-square&logo=github)](https://github.com/Socialpranker/deepdive/stargazers)
  12. <br>
  13. **[Docs](https://socialpranker.github.io/deepdive/)** · **[Install](#install-in-30-seconds)** · **[How it works](#how-it-works)** · **[Contribute](CONTRIBUTING.md)**
  14. <br>
  15. ```
  16. You: investigate the trade-offs between Postgres logical replication and CDC tooling
  17. Claude: ✓ Reframed your question (3 hypotheses) + decision spec: what you'll do with the answer
  18. ✓ Picked genre: decision (comparison + validation)
  19. ✓ Wrote plan.md (17 sections)
  20. ✓ Checked your env: 4 APIs available, 2 fallback to HTML
  21. ✓ Launched 4 sub-agents across 12 channels
  22. ✓ Saved 23 sources to sources/ with quotes, provenance-checked
  23. ✓ Ran adversarial pass (3 counter-arguments + an "execute this" role)
  24. ✓ Report ready: research/postgres-replication-vs-cdc/2026-05-21_decision.md
  25. ✓ Walked through your decision forks — you picked CDC tooling, logged to application.md
  26. ```
  27. </div>
  28. ---
  29. ## What this is
  30. A [Claude Code skill](https://docs.anthropic.com/claude/docs/skills) that turns **"research this topic"** into a **<!--gen:count:phases-->13<!--/gen-->-phase pipeline** with hypothesis testing, parallel sub-agent search, source triangulation, and adversarial review.
  31. The output is a folder you can return to in a month. Every claim traces to a specific source file. The plan documents *why* you made every choice. No re-research needed.
  32. > **New here?** Start with the [Quickstart](QUICKSTART.md) — install → invoke → first result in ~5 min.
  33. <table>
  34. <tr>
  35. <th width="50%">Without this</th>
  36. <th width="50%">With this</th>
  37. </tr>
  38. <tr>
  39. <td>
  40. One-shot prompt → wall of text
  41. Sources lost in chat history
  42. No way to detect bias
  43. No reuse next time
  44. Generic Google results
  45. Sources include... *(vague)*
  46. </td>
  47. <td>
  48. 17-section `plan.md` documents every choice
  49. Each source = file with verbatim quotes
  50. Mandatory adversarial pass + opposition queries
  51. Atomic theses in `findings/FN.md` reusable
  52. <!--gen:count:channels-->29<!--/gen--> named channels + <!--gen:count:stat_sources-->460<!--/gen-->+ stat sources
  53. Every claim → `[s12]` link → specific quote
  54. </td>
  55. </tr>
  56. </table>
  57. ---
  58. ## Install in 30 seconds
  59. <details open>
  60. <summary><b>For Claude Code (CLI)</b></summary>
  61. ```bash
  62. git clone https://github.com/Socialpranker/deepdive.git \
  63. ~/.claude/skills/deepdive
  64. ```
  65. That's it. Now type any of these in a Claude Code session:
  66. - "Investigate X"
  67. - "Изучи тему"
  68. - "Validate this hypothesis"
  69. </details>
  70. <details>
  71. <summary><b>For Claude Desktop (Skills enabled)</b></summary>
  72. ```bash
  73. # Clone
  74. git clone https://github.com/Socialpranker/deepdive.git
  75. cd deepdive
  76. # Package as .skill bundle
  77. zip -r ../deepdive.skill . -x ".*" -x "*.zip"
  78. # Upload via Claude.app → Settings → Skills → Add Skill
  79. ```
  80. </details>
  81. <details>
  82. <summary><b>For other LLMs (Codex, Gemini, local)</b></summary>
  83. The <!--gen:count:phases-->13<!--/gen-->-phase methodology is portable. Load `SKILL.md` + `references/*.md` into the LLM's context manually. Skip the sub-agent parts and use separate chat sessions per subtopic.
  84. [Full instructions →](#use-with-other-llms-codex-gemini-etc)
  85. </details>
  86. ---
  87. ## How it works
  88. The skill runs **<!--gen:count:phases-->13<!--/gen--> phases** in order:
  89. | Phase | Name | What happens |
  90. |:---:|:---|:---|
  91. <!--gen:phases:table:en-->
  92. | **1** | **Reframing** | opus / high |
  93. | **2** | **Genre & block selection** | sonnet / medium |
  94. | **3** | **Plan** | opus / medium |
  95. | **3.5** | **Capability Discovery** | sonnet / low |
  96. | **3.7** | **Plan-review gate** | sonnet / low |
  97. | **4** | **Search** | sonnet / medium |
  98. | **5** | **Claims-ledger + triangulation** | haiku / low |
  99. | **5.5** | **Evidence filter** | sonnet / low |
  100. | **5.7** | **Wiki reconcile** | sonnet / low |
  101. | **6** | **Synthesis + multi-angle red team** | sonnet → opus on deep / high |
  102. | **6.5** | **Verify** | haiku / low |
  103. | **7** | **Refresh targets** | sonnet / medium |
  104. | **8** | **Decision walkthrough** | opus / high |
  105. <!--/gen-->
  106. Each phase runs on a model matched to its task — Opus where reasoning multiplies (1/3/6), Haiku for the parallel fan-out (4). The skill announces the routing and an estimated cost up front, once.
  107. Every phase is **transparent**: you see what's happening, you confirm key decisions, and you get a folder you can return to. Before any search fires, the **plan-review gate** (3.7) shows you the reframing, hypotheses, genre, and channels and lets you approve or edit them — strictness scales with mode (deep waits for an explicit go-ahead, medium is a soft check, shallow skips it). Editing the plan before execution is the single highest-leverage step in the whole pipeline — Gemini Deep Research calls plan review its "biggest lever over output quality," and a wrong plan executed perfectly still produces a wrong report.
  108. Reframing (1) doesn't just restate the question — a **router** classifies its profile (factual / multi-step / relational / comparative / landscape) and that classification picks the decomposition method: factual questions get flat independent subquestions, multi-step ones ("X given Y") get least-to-most leveling, comparative ones get a shared axis matrix with mandatory opposition queries per candidate. Picking the wrong decomposition for a question's shape is a silent failure mode — the router makes the choice explicit instead of defaulting to "flat parallel" for everything.
  109. Phase 4 (Search) isn't a single pass — it's a bounded loop with three cheap safeguards so it doesn't quietly waste budget or silently give up:
  110. - **Cheap goal-check** — after each round, a Haiku pass tags every subquestion `met` / `partial` / `unmet` with a one-line reason. This is what the expensive Opus evaluation reads instead of re-deriving the gap from scratch, and it's what targets the next round's dispatch.
  111. - **No-progress circuit breaker** — two consecutive rounds that add nothing new to the source pool stop the loop immediately, regardless of remaining budget. The unresolved thread goes to Open Questions instead of burning tokens chasing a dead end.
  112. - **Least-to-most decomposition** — for layered questions ("X given Y"), subquestions are leveled `L1 → L2` instead of dispatched flat in parallel: L1 rounds run first, concrete facts they surface get carried forward, and L2 queries are launched already sharpened by that context. Independent subquestions still run flat.
  113. Scoring (5) doesn't stop at the usual Credibility/Recency/Bias rating — it also flags **input-level skepticism**: a source that measures its own product, self-reports a benchmark, or is directly disputed by another collected source gets a strict `caveat:` marker (`vendor` / `self-reported` / `disputed:sNN`) *before* the claim reaches `claims.csv`, not after synthesis has already built on it. A claim whose key number carries that marker is capped at `confidence: medium` (or `low` for an unresolved dispute) — the same rule shape as primary-first sourcing. Vendor benchmarks are the numbers that most often get quietly repeated as fact; catching them on the way in, not in the red team pass at the end, is the point.
  114. For medium/deep depth, the pipeline runs two more machine-checked passes most one-shot research skips entirely:
  115. - **Evidence filter (5.5)** — a CRAG-style relevance classifier runs on every (claim, source) pair *before* synthesis and keeps only the quotes that actually support that specific claim. Dumping every found source into synthesis measurably hurts quality (Search-o1 dropped 33%→24% doing exactly that); this is the fix, not a nice-to-have.
  116. - **Faithfulness verification (6.5)** — beyond checking that a cited link is alive, the skill checks that the source *entails* the claim it's attached to (RAGAS/ALCE-style claim⊨quote), and writes `SUPPORTED` / `PARTIAL` / `UNSUPPORTED` verdicts to `.verify/faithfulness.json`. Citation fabrication is common enough industry-wide — the Tow Center found a >60% error rate in AI-generated citations — that checking for it, not just for dead links, is a real differentiator.
  117. None of this is enforced by discipline alone: `scripts/validate_phases.py` reads a finished run's `mode:` and checks that every phase mandatory for that mode actually left its file artifact (`plan.md`, `claims.csv`, `evidence/`, `.verify/*.json`, the dated report, ...). A skipped phase fails the check instead of silently passing — the model can't just claim "done." As of finish-up, this check is a **blocker, not a suggestion**: the skill won't report a research as done on a red gate, symmetrically to how a report isn't "done" without its verification header. `sources.csv` itself is now built the same deterministic way — `scripts/build_sources_csv.py` generates it from `sources/NN.md` frontmatter (with a `--check` mode for CI) instead of being assembled by hand each run.
  118. A well-cited report that changes nothing is still a failure — the skill's answer to that is a **decision spine** running through the whole pipeline, not a bolt-on question at the end:
  119. - **Reframing (1) captures a decision spec**, not just a topic: the action you'll take, who reads the report and what they do next, and at least one falsifiable if-then fork ("if the research shows X, I do A"). No fork means the research changes nothing — the skill downgrades to shallow "curiosity mode" with an explicit label instead of quietly running a full pipeline for a dead-end doc.
  120. - **Sourcing (4-5) tracks provenance, not just source count.** A `root:` field on every source flags what it's actually retelling — ten articles quoting one press release are one voice, not ten, so triangulation now requires ≥2 distinct roots in addition to ≥3 sources and ≥2 types. A **snowball pass** also chains citations backward (to the primary study everyone's retelling) and forward (who's citing it since), catching sources no keyword query would surface.
  121. - **Synthesis (6) produces `memo.md`** — a one-page decision memo (recommendation, resolved forks, 3 key numbers with sources, the main risk, next actions) designed to be the thing that actually enters your decision process, with the full report as its evidence base. A conditional recommendation mapped to your forks is required; an unconditional "it depends" is not an acceptable ending.
  122. - **Red team (6) gets a fourth role — the Executor** — who plays your own decision-spec consumer and tries to actually act on the report using only its content, flagging every place a hedge phrase or a missing number blocks a real decision.
  123. - **Decision walkthrough (8) is mandatory at every depth, including shallow.** The report isn't discussed, it's executed: the skill walks you through each fork one at a time ("research showed X [s03][s11] — does fork A fire? your call?") and logs the outcome — decided, blocked on missing data (one bounded gap-search, then honest `blocked`), or deferred — to `application.md`. A global ledger tracks whether past research actually led anywhere, and Phase 1 reads it back on your next research to flag a pattern of dead-end runs.
  124. Want to compare models head-to-head? The [eval harness](eval/README.md) scores any run on 6 axes.
  125. ---
  126. ## What's inside
  127. <table>
  128. <tr>
  129. <td width="33%" valign="top">
  130. ### <!--gen:count:blocks-->106<!--/gen--> Report Blocks
  131. 10 categories: **FRAME** · **EXPLAIN** · **COMPARE** · **MAP** · **VALIDATE** · **ANALYZE** · **CLOSE** · **PEOPLE** · **NUMBERS** · **CONTEXT**
  132. Each block has its own template, anti-patterns, and composition rules.
  133. [Block library →](references/blocks/INDEX.md)
  134. </td>
  135. <td width="33%" valign="top">
  136. ### <!--gen:count:channels-->29<!--/gen--> Search Channels
  137. Named strategies with query patterns + paywall fallbacks:
  138. `web-general` · `academic` · `preprint-servers` · `code-github` · `forum-discussion` · `news-current` · `industry-reports` · `regulatory-legal` · `competitive-signals` · `data-statistical-gov` · `product-analytics` · `crypto-analytics` · `api-direct` · *and more*
  139. [Channels catalog →](references/channels.md)
  140. </td>
  141. <td width="33%" valign="top">
  142. ### <!--gen:count:stat_sources-->460<!--/gen-->+ Stat Sources
  143. 14 cross-industry + 19 industry categories. Each entry: URL · Type · Access · Quality · Limitations · Combine-with · Fallback.
  144. Categories: `gov_macro` · `companies_public` · `crypto` · `health` · `education` · `climate_env` · `science` · 19 industries
  145. [Sources catalog →](references/stat_sources/INDEX.md)
  146. **Plus a signal registry:** 1072 endpoints verified live and filed by what question they answer — parliamentary APIs, advisory feeds, climate and macro endpoints. Loaded one file at a time, outside the base context budget.
  147. [Registry →](references/registry/INDEX.md)
  148. </td>
  149. </tr>
  150. <tr>
  151. <td valign="top">
  152. ### 6 Report Genres
  153. | Genre | When |
  154. |:---|:---|
  155. | `qa` | Open meta-research |
  156. | `explainer` | "How does X work" |
  157. | `decision` | "X or Y" |
  158. | `landscape` | "Who's in this space" |
  159. | `validation` | "Is X true" |
  160. | `custom` | Hybrid, assembled per question |
  161. [Genres →](references/genres.md)
  162. </td>
  163. <td valign="top">
  164. ### <!--gen:count:api-->47<!--/gen-->+ API Endpoints
  165. Free no-auth APIs prioritized:
  166. `Semantic Scholar` · `OpenAlex` · `CrossRef` · `arXiv` · `DefiLlama` · `CoinGecko` · `HN Algolia` · `World Bank` · `SEC EDGAR` · `ClinicalTrials.gov` · `PubMed` · `GDELT` · `NSF Awards` · `NIH RePORTER` · `Grants.gov` · `EPO Linked Open Data`
  167. Patents and grants are first-class categories: `patents/` (USPTO ODP, EPO OPS, EPO LOD, WIPO) and `grants/` (NSF, NIH RePORTER, CORDIS, Grants.gov). Every entry carries a `Verified:` date — access terms rot, and a stale "free, no key" sends a run into a paywall.
  168. Auth via env vars only — skill never asks for keys inline.
  169. [API catalog →](references/api_sources/INDEX.md)
  170. </td>
  171. <td valign="top">
  172. ### Weekly Auto-Sync
  173. GitHub Actions cron validates all endpoints + discovers upstream additions:
  174. - HEAD-check <!--gen:count:api-->47<!--/gen-->+ APIs weekly
  175. - Scan public-apis & awesome-public-datasets
  176. - Auto-PR for dead endpoints
  177. - Reports committed to `reports/` branch
  178. [Workflow →](.github/workflows/catalog-sync.yml)
  179. </td>
  180. </tr>
  181. <tr>
  182. <td valign="top">
  183. ### Model Routing
  184. Per-phase model selection — quality where it multiplies, cheap where it parallelizes:
  185. - Reframing / plan / adversarial → **Opus**
  186. - Sub-agent fan-out (search) → **Haiku** (cheap × N)
  187. - Synthesis → **Sonnet/high**
  188. ~$2 instead of ~$8 on a deep run, *and* higher quality on critical phases. Override with `with all on opus` / `with cheap mode`.
  189. [Routing →](references/model_routing.md)
  190. </td>
  191. <td valign="top">
  192. ### Eval Harness
  193. Compare research quality across models. Same question, different configs, scored on 6 axes:
  194. - Deterministic (script): citation integrity, source diversity, cost
  195. - Semantic (LLM-judge): accuracy, coverage, adversarial honesty
  196. Weighted sum with a **citation floor** — hallucinated sources can't win on depth. Verdict = quality per dollar.
  197. [Eval →](eval/README.md)
  198. </td>
  199. <td valign="top">
  200. ### Citation Check
  201. `check_citations.py` resolves every source URL — dead `OPEN` links flagged as likely hallucinations; transport flaps marked `UNKNOWN`, not penalized.
  202. Ignores env proxies (`trust_env=False`). `--strict` for CI.
  203. Verification runs four layers: **liveness** (does the source exist), **faithfulness** (does it actually entail the claim it's cited for), **qualifier preservation** (does the report still say what the ledger said), and **construct provenance** (do the frameworks, taxonomies and named "laws" the report uses exist outside it). Verdicts land in `.verify/*.json`, one producer per file.
  204. The fourth layer exists because the first three all join on `claim_id` — and a fabricated *name* has none. That is the largest measured class of generation defect in research agents ([FINDER/DEFT](https://arxiv.org/abs/2512.01948): strategic content fabrication, 18.95% of errors), and it passes a citation check with every URL alive.
  205. Numbers get two independent passes: `check_number_provenance.py` on **origin** (who produced the figure; does one value circulate across supposedly independent roots) and `check_number_arithmetic.py` on **computation** — every `derived` figure in `numbers.csv` is recomputed from its own declared `formula` + `inputs`, and `share` groups must sum to 100. A percentage computed in prose is otherwise never re-checked by anything.
  206. [Check →](eval/check_citations.py)
  207. </td>
  208. <td valign="top">
  209. ### Phase-gate Validator
  210. `scripts/validate_phases.py` reads a finished run's `mode:` frontmatter and checks that every phase mandatory for that depth left its file artifact — `plan.md`, `state.md`, `claims.csv`, `numbers.csv`, `outline.md`, `evidence/`, `.verify/*.json`, the dated report.
  211. A skipped phase fails the check (`--strict` for CI) instead of the model just asserting "done" — it's a **finish-up blocker**, not advice. Machine insurance against the one failure mode a markdown methodology can't fix by discipline alone.
  212. It also checks three things a missing-file test cannot: that `state.md` is a **rebuilt** round window (Known/Gaps/Next, size-capped) and not an appended second transcript; that every `triangulated`/`contested` claim is mapped to a section in `outline.md` — the **synthesis gap**, where retrieval succeeded and the finding never reached the report; and that no `unsourced` construct sits in the memo or TL;DR.
  213. Its own inputs are machine-built too: `scripts/build_sources_csv.py` generates `sources.csv` deterministically from `sources/NN.md` frontmatter (`--check` for CI drift), and `eval/validate_structure.py` enforces the `caveat:` field as a strict enum (`-` / `vendor` / `self-reported` / `disputed:sNN`) instead of free text, so it stays greppable.
  214. [Validator →](scripts/README.md)
  215. </td>
  216. </tr>
  217. </table>
  218. ---
  219. ## Example folder
  220. Sample output for a typical `decision`-genre research:
  221. ```
  222. research/<topic-slug>/
  223. ├── plan.md # 17-section plan
  224. ├── state.md # Round window, rewritten each round
  225. ├── sources.csv # Index with C/R/B scoring
  226. ├── claims.csv # Claim ledger + triangulation status
  227. ├── numbers.csv # Every figure the report stands on
  228. ├── outline.md # section → block → claim_id map
  229. ├── sources/ # One file per source
  230. │ ├── 01_vendor-docs.md # Primary, total=14
  231. │ ├── 02_benchmark-paper.md # Academic, total=12
  232. │ ├── 03_industry-report.md # Industry, total=13
  233. │ ├── 04_forum-thread.md # Forum, total=9 (opposition)
  234. │ └── ... (19 more)
  235. ├── findings/
  236. │ ├── F1_<atomic-thesis>.md # confidence: high
  237. │ └── F2_<atomic-thesis>.md # confidence: medium
  238. ├── .verify/ # One producer per file, many consumers
  239. │ ├── authority.json # Phase 5.5 — who may assert this
  240. │ ├── citations.json # Layer 1 — liveness
  241. │ ├── faithfulness.json # Layer 2 — entailment
  242. │ ├── qualifiers.json # Layer 3 — scope drift
  243. │ └── constructs.json # Layer 4 — named-construct provenance
  244. ├── memo.md # One-page decision memo (always)
  245. ├── application.md # Decision walkthrough verdict (always)
  246. └── 2026-05-21_decision.md # Final report
  247. ```
  248. Final report structure (assembled from the blocks chosen in plan.md):
  249. ```markdown
  250. ## TL;DR
  251. - Claim A holds under condition X [confidence: high]
  252. - Claim B holds conditionally on threshold Y [confidence: medium]
  253. - Claim C is disputed by opposition sources [confidence: low]
  254. ## Mental model
  255. [How the underlying mechanism works...]
  256. ## Falsification criteria
  257. What would disprove H1, H2, H3...
  258. ## Verdict conditional
  259. Recommendation IF: <conditions met>
  260. Different recommendation OTHERWISE: <conditions broken>
  261. ## Counter-arguments (steel-man)
  262. CA1: "<the strongest opposing claim>" [source: s09]
  263. → Our answer: <conditions under which CA1 fails>
  264. CA2: ...
  265. ```
  266. Every claim is clickable to its source. A month later, you don't re-research — you read.
  267. ---
  268. ## Contribute
  269. The catalog is most valuable when **it grows**. Easy contributions:
  270. | Time | Type | Example |
  271. |:---:|:---|:---|
  272. | 15 min | Add a stat source | `Add SimilarWeb Pro to consumer_digital` |
  273. | 15 min | Improve a query pattern | `Better arxiv channel queries for biology` |
  274. | 30 min | New search channel | `Add patent-search with USPTO+EPO fallback` |
  275. | 1-2h | New industry category | `Add industries/aerospace.md` |
  276. | 2-4h | New report block | `Add decision-tree to compare.md` |
  277. | Half-day | LLM adapter | `Add codex/ folder with adapted protocols` |
  278. [Full contributing guide →](CONTRIBUTING.md)
  279. [![Contributors](https://contrib.rocks/image?repo=Socialpranker/deepdive)](https://github.com/Socialpranker/deepdive/graphs/contributors)
  280. ---
  281. ## FAQ
  282. <details>
  283. <summary><b>How is this different from ChatGPT Deep Research / Perplexity?</b></summary>
  284. Those are **products** — closed UI, fixed flow, opaque source selection. This is **open methodology** — you control every step, the protocol is markdown you can fork, the source catalog is yours to extend.
  285. They also don't separate sources into files, don't do explicit triangulation, don't run adversarial passes, and don't produce reusable atomic theses. Nor do they filter evidence for relevance before synthesis (feeding a model everything you found measurably hurts quality — Search-o1 dropped from 33% to 24% accuracy doing that) or verify that a cited source actually *supports* the claim it's attached to, rather than just existing (faithfulness, not just liveness). Citation fabrication is common enough industry-wide — the Tow Center found a >60% error rate in AI-generated citations — that checking for it is a real differentiator, not a nice-to-have.
  286. Honesty about sources goes further than checking they exist: scoring flags a source that's measuring its own product, self-reporting a benchmark, or directly disputed by another collected source, and caps the confidence of any claim resting on that number — *before* it ever reaches the report. Vendor benchmarks getting quietly repeated as fact is a market-wide problem; catching it on input, not as an afterthought, is the same honesty principle as faithfulness applied one step earlier.
  287. </details>
  288. <details>
  289. <summary><b>Does it work without Claude Code CLI?</b></summary>
  290. Yes — on Claude Desktop with Skills enabled. Also works manually with any LLM by loading the markdown files into context (see ["Use with other LLMs"](#use-with-other-llms-codex-gemini-etc) below).
  291. </details>
  292. <details>
  293. <summary><b>What's a research output look like?</b></summary>
  294. See the [example folder](#example-folder) above. TL;DR: a folder with `plan.md` + `sources/NN.md` per source + `findings/FN.md` atomic theses + final `<date>_<genre>.md` report.
  295. Every claim in the final report links to a specific `sources/NN.md` file.
  296. </details>
  297. <details>
  298. <summary><b>Why so many files? Isn't this overkill?</b></summary>
  299. For a 5-minute "what's the latest X" question — yes. That's why `shallow` mode exists (5-7 sources, no sub-agents, ~15 min). The full machinery is for `medium` (1 hour) and `deep` (3 hours) when you need to actually use the output for a decision.
  300. The file-per-source structure is the key **reuse** mechanism. A single research often informs 3-5 future researches because you can cite individual `sources/NN.md` directly.
  301. </details>
  302. <details>
  303. <summary><b>Is this just prompt engineering?</b></summary>
  304. It's **structured methodology + curated catalog + reusable templates + automation**.
  305. - The <!--gen:count:phases-->13<!--/gen-->-phase workflow forces discipline
  306. - <!--gen:count:stat_sources-->460<!--/gen-->+ stat sources catalog is curated knowledge
  307. - <!--gen:count:blocks-->106<!--/gen--> reusable blocks compose any report shape
  308. - `scripts/validate_phases.py` machine-checks phase completeness, not just style
  309. - A decision spec + mandatory walkthrough (phase 8) ties every research to an actual action, not just a document
  310. - Weekly auto-validation keeps the catalog alive
  311. - 25+ upstream awesome-lists give infinite discovery layer
  312. Prompts are an implementation detail, not the value.
  313. </details>
  314. <details>
  315. <summary><b>What stops the report from being unactionable — "well-researched but useless"?</b></summary>
  316. The most common failure mode of open-ended research isn't bad sourcing, it's a report nobody knows what to do with. Three things guard against that specifically:
  317. 1. **Reframing won't accept a topic without a decision spec** — an action, a consumer, and at least one falsifiable if-then fork. No fork, no medium/deep run; it downgrades to a labeled shallow "curiosity" pass instead.
  318. 2. **Synthesis is banned from ending on "it depends"** without mapping the condition to one of your forks, and a fourth red-team role tries to actually *execute* the recommendation using only the report, flagging every hedge and missing number that blocks a real decision.
  319. 3. **Phase 8 (Decision walkthrough) is mandatory at every depth** — the skill walks you through your forks one at a time and logs whether each was decided, blocked on data, or deferred to `application.md`. A global ledger tracks unapplied research across sessions so the pattern doesn't repeat silently.
  320. </details>
  321. <details>
  322. <summary><b>Can I use this commercially?</b></summary>
  323. Yes — MIT licensed. Use it, modify it, integrate it into products. Attribution appreciated but not required.
  324. </details>
  325. ---
  326. ### Use with other LLMs (Codex, Gemini, etc.)
  327. The methodology is portable. ~70% of content is LLM-agnostic markdown templates.
  328. | Component | Claude-specific | Universal |
  329. |:---|:---:|:---:|
  330. | `SKILL.md` frontmatter | ✓ | — |
  331. | Sub-agent `Explore` type | ✓ | — |
  332. | <!--gen:count:phases-->13<!--/gen-->-phase workflow | — | ✓ |
  333. | <!--gen:count:blocks-->106<!--/gen--> report blocks | — | ✓ |
  334. | <!--gen:count:channels-->29<!--/gen--> search channels | — | ✓ |
  335. | <!--gen:count:stat_sources-->460<!--/gen-->+ stat sources | — | ✓ |
  336. **To adapt:**
  337. 1. Load `SKILL.md` + relevant `references/*.md` into the LLM's context
  338. 2. Replace sub-agent parallelism with separate chat sessions per subtopic
  339. 3. Manage source files (`sources/NN.md`) externally — LLM produces content
  340. 4. PRs welcome for `codex/`, `gemini/`, `local/` adapters
  341. ---
  342. <div align="center">
  343. ## Star History
  344. [![Star History Chart](https://api.star-history.com/svg?repos=Socialpranker/deepdive&type=Date)](https://star-history.com/#Socialpranker/deepdive&Date)
  345. </div>
  346. ---
  347. <details>
  348. <summary><h2>На русском</h2></summary>
  349. **Deepdive** — скилл для [Claude Code](https://claude.com/claude-code), превращающий «загугли это» в дисциплинированный <!--gen:count:phases-->13<!--/gen-->-фазный процесс.
  350. ### Что внутри
  351. - **<!--gen:count:phases-->13<!--/gen--> фаз workflow**: <!--gen:phases:list:ru-->Reframing → Genre & block selection → Plan → Capability Discovery → Plan-review gate → Поиск → Claims-ledger + триангуляция → Evidence-фильтр → Сверка с вики → Синтез + multi-angle red team → Verify → Refresh targets → Decision walkthrough<!--/gen-->
  352. - **<!--gen:count:genres-->6<!--/gen--> жанров отчёта**: qa / explainer / decision / landscape / validation / custom
  353. - **<!--gen:count:blocks-->106<!--/gen--> блоков** в 10 категориях — переиспользуемые секции с шаблонами и анти-паттернами
  354. - **<!--gen:count:channels-->29<!--/gen--> каналов поиска** с paywall fallback протоколом (включая api-direct)
  355. - **<!--gen:count:stat_sources-->460<!--/gen-->+ статистических источников** в 14 cross-industry + 19 отраслевых категориях
  356. - **<!--gen:count:api-->47<!--/gen-->+ API endpoints** для programmatic доступа (free no-auth приоритетны)
  357. - **plan.md** с 17 секциями для прозрачности
  358. - **Plan-review gate** (фаза 3.7) — единственная human-in-the-loop точка перед дорогой Фазой 4: план (гипотезы, жанр, каналы) показывается и утверждается ДО поиска; жёсткость по режиму (deep — ждать «Ок», medium — soft, shallow — skip)
  359. - **Multi-angle red team** из враждебных ролей (Skeptic/Contrarian/Gap-hunter) с триажем severity (обязателен для medium/deep)
  360. - **Evidence-фильтр** (фаза 5.5) — CRAG-классификатор keep/drop по паре (тезис, источник) перед синтезом: наивная подача всего найденного снижает качество (Search-o1 33%→24%), в синтез идут только relevant-цитаты из `evidence/`
  361. - **Faithfulness-верификация** (фаза 6.5, второй слой) — помимо liveness (жива ли ссылка) проверяется entailment «источник ⊨ тезис» (RAGAS-декомпозиция + ALCE), вердикты SUPPORTED/PARTIAL/UNSUPPORTED в `.verify/faithfulness.json`
  362. - **No-progress circuit breaker** (фаза 4) — 2 раунда подряд без новой информации → стоп, нерешённое уходит в Open Questions вместо сжигания бюджета
  363. - **Дешёвый goal-check** (фаза 4) — Haiku между раундами помечает каждый подвопрос met/partial/unmet, направляя следующий раунд и удешевляя дорогую Opus-оценку
  364. - **Least-to-most декомпозиция** (фаза 4) — многошаговые подвопросы («X учитывая Y») раскладываются по уровням L1→L2 с накоплением контекста между ними вместо плоского параллельного запуска
  365. - **Phase-gate валидатор** (`scripts/validate_phases.py`) — машинная проверка, что каждая обязательная для режима фаза оставила артефакт; пропущенная фаза не проходит проверку
  366. - **Decision Spec** (фаза 1) — решение+срок / потребитель→шаг / ≥1 опровергаемая if-then вилка; без вилки ресёрч ничего не меняет — честный даунгрейд в shallow вместо мёртвого deep-отчёта
  367. - **Provenance-триангуляция** (фазы 4-5) — поле `root:` у каждого источника + snowball-пасс по цепочкам цитирований; триангуляция требует ≥2 разных первоисточников, не только ≥3 источников разного типа (десять пересказов одного пресс-релиза — один голос)
  368. - **`memo.md`** (фаза 6, всегда) — одностраничный decision-меморандум: рекомендация без «it depends», разрешённые вилки, ключевые числа со ссылками, next actions — то, что реально входит в твой процесс принятия решений
  369. - **Роль «Исполнитель»** в red team (фаза 6, medium/deep) — симулирует потребителя из Decision Spec и пытается исполнить решение по одному отчёту, ловя обтекаемый язык и пропущенные числа
  370. - **Decision walkthrough** (фаза 8, обязательна всегда) — отчёт не обсуждается, а исполняется: скилл проводит по вилкам решения одну за одной и записывает вердикт (принято / заблокировано данными / отложено) в `application.md`; глобальный ledger отслеживает, приводят ли твои ресёрчи к решениям вообще
  371. - **Weekly auto-validation** через GitHub Actions
  372. ### Установка
  373. ```bash
  374. git clone https://github.com/Socialpranker/deepdive.git ~/.claude/skills/deepdive
  375. ```
  376. Триггеры: «проведи ресёрч», «изучи тему», «копни глубоко», «deep dive»
  377. ### Вклад
  378. Каталог растёт через PRs. Самые ценные — новые источники в `stat_sources/` и `api_sources/`. См. [CONTRIBUTING.md](CONTRIBUTING.md).
  379. </details>
  380. ---
  381. <div align="center">
  382. ### Built by [Socialpranker](https://github.com/Socialpranker) · [MIT License](LICENSE) · [Roadmap](https://github.com/Socialpranker/deepdive/discussions)
  383. **If this skill saves you time, [give it a star](https://github.com/Socialpranker/deepdive)** — it's the only metric I check.
  384. </div>
runner/wiki.py · Bundled script · 665 lines
  1. """Cross-run knowledge layer: the wiki that sits between raw traces and the skill.
  2. The skill already had two of the three layers. Raw: sources/NN.md per run plus
  3. deepdive-state/observations.jsonl. Skills: references/channels.md and api_sources/,
  4. rewritten by promote_candidates.py behind a real gate (3 wins across 3 distinct runs,
  5. liveness, demotion). What was missing is the layer in between — so the only thing that
  6. compounded across runs was WHERE to search, never WHAT was found. A source fetched in
  7. two researches produced two unrelated files scored from zero; a claim triangulated in
  8. March was invisible to a run in August; two runs could conclude opposite things and
  9. nobody would notice.
  10. This module owns that layer. Three registries with real join keys:
  11. sources — key is the canonical URL (or DOI when present). Purely mechanical.
  12. entities — key is a folded name; aliases map variants onto one canonical page.
  13. claims — NO auto-key. Claims get an opaque id on first ingest and are linked to
  14. each other only through adjudicated pairs (see below).
  15. The claims asymmetry is deliberate. Deciding that two sentences from different runs are
  16. "the same claim" is a semantic judgement, and a wrong one manufactures a contradiction
  17. out of nothing — a broken measurement producing a false diagnosis, which is the most
  18. expensive failure this skill can have. So claim identity is never inferred here. This
  19. module only proposes CANDIDATE PAIRS through a mechanical prefilter and resolves the
  20. cases that need no judgement at all (a stale figure, a unit mismatch, a different
  21. scope). Whatever survives that goes to an adjudicator in phase 5.7, and its verdict is
  22. appended to pairs.jsonl as an audit trail. `unknown` is quarantine, never a conflict.
  23. Storage lives under ~/.claude/research/wiki/ — global on purpose. Research folders land
  24. per-project (research/, 06_Деск-ресёрч/, ~/deepdive/), and a per-project wiki would
  25. fragment exactly the thing that is supposed to accumulate. The precedent is already
  26. there: applications_ledger.csv and deepdive-state/ are both global.
  27. """
  28. from __future__ import annotations
  29. import csv
  30. import hashlib
  31. import io
  32. import json
  33. import logging
  34. import re
  35. import unicodedata
  36. from dataclasses import asdict, dataclass, field
  37. from datetime import date, datetime
  38. from pathlib import Path
  39. from urllib.parse import parse_qsl, urlencode, urlsplit
  40. log = logging.getLogger(__name__)
  41. DEFAULT_ROOT = Path.home() / ".claude"
  42. WIKI_DIRNAME = "research/wiki"
  43. REGISTRY_DIRNAME = ".wiki"
  44. # A figure that is simply newer is not a disagreement. Two claims that carry numbers and
  45. # whose as_of dates are further apart than this are treated as a version history, not a
  46. # conflict — the older one is marked superseded without ever reaching an adjudicator.
  47. # 180 days is the point past which most quantities in this skill's typical subject
  48. # matter (pricing, market size, model benchmarks) are expected to have moved anyway.
  49. SUPERSEDE_DAYS = 180
  50. # Query parameters that never change what a page says.
  51. TRACKING_PARAMS = frozenset(
  52. {
  53. "utm_source",
  54. "utm_medium",
  55. "utm_campaign",
  56. "utm_term",
  57. "utm_content",
  58. "utm_id",
  59. "ref",
  60. "referrer",
  61. "source",
  62. "fbclid",
  63. "gclid",
  64. "msclkid",
  65. "yclid",
  66. "_hsenc",
  67. "_hsmi",
  68. "mc_cid",
  69. "mc_eid",
  70. "igshid",
  71. "spm",
  72. "share",
  73. "s",
  74. "__twitter_impression",
  75. }
  76. )
  77. STOPWORDS = frozenset(
  78. """
  79. a an the and or but if then than that this these those of in on at to for from with
  80. without by as is are was were be been being it its their his her our your not no more
  81. most less least very can could may might will would should must do does did has have
  82. had about over under between across per vs versus
  83. и в во не что он на я с со как а то все она так его но да ты к у же вы за бы по только
  84. ее мне было вот от меня еще нет о из ему теперь когда даже ну вдруг ли если уже или ни
  85. быть был него до вас нибудь опять уж вам ведь там потом себя ничего ей может они тут
  86. где есть надо ней для мы тебя их чем была сам чтоб без будто чего раз тоже себе под
  87. будет ж тогда кто этот того потому этого какой совсем ним здесь этом один почти мой
  88. тем чтобы нее сейчас были куда зачем всех никогда можно при наконец два об другой хоть
  89. после над больше тот через эти нас про них какая много разве три эту моя впрочем хорошо
  90. свою этой перед иногда лучше чуть том нельзя такой им более всегда конечно всю между
  91. """.split()
  92. )
  93. # Scope qualifiers. Two claims that carry different ones are talking about different
  94. # things, which is the second-biggest source of manufactured contradictions after stale
  95. # numbers. Matched as whole words against the folded claim text.
  96. QUALIFIER_PATTERNS = (
  97. r"\b(?:в|for|in)\s+(?:сша|us|usa|eu|ес|ru|рф|россии|china|китае|uk|eea)\b",
  98. r"\b(?:enterprise|smb|self-hosted|cloud|on-prem|on-premise|free|paid|trial)\b",
  99. r"\b(?:median|медиан\w*|average|средн\w*|p50|p90|p95|p99)\b",
  100. r"\b(?:per\s+\w+|на\s+\w+|годов\w*|annual|monthly|месячн\w*)\b",
  101. r"\b(?:20\d{2})\b",
  102. )
  103. NUM_RE = re.compile(
  104. r"(?P<value>-?\d[\d\s .,]*\d|-?\d)\s*(?P<unit>%|процент\w*|млрд|млн|тыс|"
  105. r"[a-zA-Zа-яА-Я$€₽]{0,12})",
  106. )
  107. UNIT_ALIASES = {
  108. "процент": "%",
  109. "процента": "%",
  110. "процентов": "%",
  111. "проц": "%",
  112. "pct": "%",
  113. "usd": "$",
  114. "dollars": "$",
  115. "dollar": "$",
  116. "долл": "$",
  117. "доллар": "$",
  118. "eur": "€",
  119. "rub": "₽",
  120. "руб": "₽",
  121. "рублей": "₽",
  122. "bn": "b",
  123. "млрд": "b",
  124. "billion": "b",
  125. "billions": "b",
  126. "m": "m",
  127. "млн": "m",
  128. "million": "m",
  129. "millions": "m",
  130. "mn": "m",
  131. "k": "k",
  132. "тыс": "k",
  133. "thousand": "k",
  134. }
  135. WORD_RE = re.compile(r"[0-9a-zA-Zа-яёА-ЯЁ][0-9a-zA-Zа-яёА-ЯЁ_-]*", re.UNICODE)
  136. # Verdicts an adjudicator may return for a candidate pair. `unknown` is a quarantine
  137. # state: it is neither agreement nor conflict, it never reaches a report, and wiki_lint
  138. # reports it as debt once it goes stale.
  139. VERDICTS = ("same-claim-agree", "same-claim-conflict", "different-claim", "unknown")
  140. # Verdicts this module assigns mechanically, before any adjudicator sees the pair.
  141. AUTO_VERDICTS = ("superseded", "different-claim")
  142. # --------------------------------------------------------------------------- paths
  143. def wiki_dir(root: Path | None = None) -> Path:
  144. return (Path(root) if root is not None else DEFAULT_ROOT) / WIKI_DIRNAME
  145. def registry_dir(root: Path | None = None) -> Path:
  146. return wiki_dir(root) / REGISTRY_DIRNAME
  147. def ensure_wiki(root: Path | None = None) -> Path:
  148. d = wiki_dir(root)
  149. for sub in ("sources", "entities", "claims", REGISTRY_DIRNAME):
  150. (d / sub).mkdir(parents=True, exist_ok=True)
  151. return d
  152. # ---------------------------------------------------------------------- normalizing
  153. def canonical_url(raw: str) -> str:
  154. """Fold a URL to the identity of the document it points at.
  155. Scheme, `www.`, tracking parameters, fragments and a trailing slash never change
  156. what a page says, so none of them may split one source into two wiki pages. arXiv
  157. gets an explicit rule because its three renderings of one paper (/abs/, /pdf/,
  158. /html/) are the single most common way the same source enters two runs looking
  159. like two sources.
  160. """
  161. s = (raw or "").strip()
  162. if not s:
  163. return ""
  164. if "//" not in s:
  165. s = "https://" + s
  166. parts = urlsplit(s)
  167. host = parts.netloc.lower().removeprefix("www.").rstrip(".")
  168. path = parts.path.rstrip("/") or "/"
  169. if host.endswith("arxiv.org"):
  170. host = "arxiv.org"
  171. m = re.search(r"/(?:abs|pdf|html|format)/([^/]+?)(?:v\d+)?(?:\.pdf)?$", path)
  172. if m:
  173. path = f"/abs/{m.group(1)}"
  174. kept = [
  175. (k, v)
  176. for k, v in parse_qsl(parts.query, keep_blank_values=True)
  177. if k.lower() not in TRACKING_PARAMS
  178. ]
  179. query = urlencode(sorted(kept))
  180. return f"{host}{path}" + (f"?{query}" if query else "")
  181. def canonical_doi(raw: str) -> str:
  182. s = (raw or "").strip().lower()
  183. if not s:
  184. return ""
  185. s = re.sub(r"^(?:https?://)?(?:dx\.)?doi\.org/", "", s)
  186. s = s.removeprefix("doi:").strip()
  187. return s if s.startswith("10.") else ""
  188. def source_key(url: str = "", doi: str = "") -> str:
  189. """A DOI outranks a URL: it survives a publisher moving the article."""
  190. d = canonical_doi(doi)
  191. if d:
  192. return f"doi:{d}"
  193. u = canonical_url(url)
  194. return f"url:{u}" if u else ""
  195. def fold(text: str) -> str:
  196. """Casefold + strip diacritics + collapse whitespace. `ё` folds to `е`."""
  197. s = unicodedata.normalize("NFKD", (text or "").replace(" ", " "))
  198. s = "".join(c for c in s if not unicodedata.combining(c))
  199. return re.sub(r"\s+", " ", s.replace("ё", "е").replace("Ё", "Е")).strip().casefold()
  200. def entity_key(name: str) -> str:
  201. """Fold a name to its page key, dropping corporate-form noise and punctuation."""
  202. s = fold(name)
  203. s = re.sub(
  204. r"\b(?:inc|llc|ltd|corp|corporation|gmbh|s\.a\.|ооо|оао|пао|зао)\b", " ", s
  205. )
  206. s = re.sub(r"[^0-9a-zа-я]+", "-", s).strip("-")
  207. return re.sub(r"-{2,}", "-", s)
  208. def tokens(text: str) -> set[str]:
  209. return {
  210. t
  211. for t in (w.casefold() for w in WORD_RE.findall(fold(text)))
  212. if t not in STOPWORDS and len(t) > 2
  213. }
  214. def numbers(text: str) -> set[tuple[float, str]]:
  215. """Extract (value, folded unit) pairs. Unit-less numbers get the unit `""`."""
  216. out: set[tuple[float, str]] = set()
  217. for m in NUM_RE.finditer(text or ""):
  218. raw = m.group("value").replace(" ", "").replace(" ", "")
  219. # 1.234,56 (EU) vs 1,234.56 (US): whichever separator comes last is the decimal
  220. if "," in raw and "." in raw:
  221. raw = (
  222. raw.replace(",", "")
  223. if raw.rfind(".") > raw.rfind(",")
  224. else raw.replace(".", "").replace(",", ".")
  225. )
  226. else:
  227. raw = (
  228. raw.replace(",", ".")
  229. if raw.count(",") == 1 and len(raw.split(",")[-1]) <= 2
  230. else raw.replace(",", "")
  231. )
  232. try:
  233. value = float(raw)
  234. except ValueError:
  235. continue
  236. unit = m.group("unit").strip().casefold()
  237. out.add((value, UNIT_ALIASES.get(unit, unit)))
  238. return out
  239. def qualifiers(text: str) -> set[str]:
  240. folded = fold(text)
  241. return {
  242. m.group(0).strip()
  243. for pat in QUALIFIER_PATTERNS
  244. for m in re.finditer(pat, folded)
  245. }
  246. def parse_date(raw: str) -> date | None:
  247. s = (raw or "").strip()[:10]
  248. for fmt in ("%Y-%m-%d", "%Y-%m", "%Y"):
  249. try:
  250. return datetime.strptime(s[: len(fmt.replace("%Y", "2000"))], fmt).date()
  251. except ValueError:
  252. continue
  253. return None
  254. # ------------------------------------------------------------------------- records
  255. @dataclass
  256. class SourcePage:
  257. key: str
  258. url: str = ""
  259. title: str = ""
  260. type: str = ""
  261. channel: str = ""
  262. doi: str = ""
  263. credibility: str = ""
  264. runs: list[str] = field(default_factory=list)
  265. roots: list[str] = field(default_factory=list)
  266. caveats: list[str] = field(default_factory=list)
  267. first_seen: str = ""
  268. last_seen: str = ""
  269. @property
  270. def page(self) -> str:
  271. return f"sources/{slug_for(self.key)}.md"
  272. @dataclass
  273. class EntityPage:
  274. key: str
  275. name: str = ""
  276. aliases: list[str] = field(default_factory=list)
  277. runs: list[str] = field(default_factory=list)
  278. first_seen: str = ""
  279. last_seen: str = ""
  280. @property
  281. def page(self) -> str:
  282. return f"entities/{self.key}.md"
  283. @dataclass
  284. class ClaimPage:
  285. """One observation of a claim, owned by the run that produced it.
  286. Never merged with another run's claim by this module. Cross-run links exist only as
  287. adjudicated pairs, so a wrong adjudication is a bad edge in pairs.jsonl — reversible
  288. and auditable — instead of two records silently fused into one.
  289. """
  290. id: str
  291. run: str
  292. claim_id: str = ""
  293. text: str = ""
  294. status: str = ""
  295. confidence: str = ""
  296. as_of: str = ""
  297. entities: list[str] = field(default_factory=list)
  298. sources: list[str] = field(default_factory=list)
  299. roots: list[str] = field(default_factory=list)
  300. ingested: str = ""
  301. @property
  302. def page(self) -> str:
  303. return f"claims/{self.id}.md"
  304. def slug_for(key: str) -> str:
  305. """Filesystem-safe page name: readable prefix + hash, so two hosts never collide."""
  306. body = key.split(":", 1)[-1]
  307. head = re.sub(r"[^0-9a-zA-Z.-]+", "-", body)[:48].strip("-").lower() or "src"
  308. return f"{head}-{hashlib.sha1(key.encode('utf-8')).hexdigest()[:8]}"
  309. def claim_uid(run: str, claim_id: str) -> str:
  310. """Identity of a claim OBSERVATION: (run, claim_id). Deliberately excludes the text.
  311. Phase 5.7 pairs claims before synthesis; ingest writes them after. Between the two
  312. the wording gets edited — remediation, a restored qualifier, a softened overclaim.
  313. Hashing the text would move the id under those edits and leave every pair written in
  314. 5.7 pointing at nothing. (run, claim_id) survives them, so a pair adjudicated before
  315. synthesis still refers to the same row afterwards. Not semantic: two runs asserting
  316. the same thing still get two ids, which is the whole point.
  317. """
  318. h = hashlib.sha1(f"{run}\x00{claim_id}".encode("utf-8")).hexdigest()
  319. return f"c-{h[:10]}"
  320. # ------------------------------------------------------------------------ registries
  321. SOURCE_COLUMNS = [
  322. "key",
  323. "url",
  324. "title",
  325. "type",
  326. "channel",
  327. "doi",
  328. "credibility",
  329. "runs",
  330. "roots",
  331. "caveats",
  332. "first_seen",
  333. "last_seen",
  334. "page",
  335. ]
  336. ENTITY_COLUMNS = ["key", "name", "aliases", "runs", "first_seen", "last_seen", "page"]
  337. CLAIM_COLUMNS = [
  338. "id",
  339. "run",
  340. "claim_id",
  341. "text",
  342. "status",
  343. "confidence",
  344. "as_of",
  345. "entities",
  346. "sources",
  347. "roots",
  348. "ingested",
  349. "page",
  350. ]
  351. _LIST_FIELDS = {"runs", "roots", "caveats", "aliases", "entities", "sources"}
  352. def _to_row(rec, columns: list[str]) -> dict[str, str]:
  353. d = asdict(rec)
  354. d["page"] = rec.page
  355. return {
  356. c: ("; ".join(d.get(c) or []) if c in _LIST_FIELDS else str(d.get(c, "")))
  357. for c in columns
  358. }
  359. def _from_row(row: dict[str, str], cls, columns: list[str]):
  360. kw = {}
  361. for c in columns:
  362. if c == "page":
  363. continue
  364. raw = (row.get(c) or "").strip()
  365. kw[c] = (
  366. [p.strip() for p in raw.split(";") if p.strip()]
  367. if c in _LIST_FIELDS
  368. else raw
  369. )
  370. return cls(**kw)
  371. def _read_csv(path: Path, cls, columns: list[str]) -> dict[str, object]:
  372. if not path.is_file():
  373. return {}
  374. out = {}
  375. with path.open(encoding="utf-8", newline="") as fh:
  376. for row in csv.DictReader(fh):
  377. try:
  378. rec = _from_row(row, cls, columns)
  379. except TypeError as exc: # schema drift — skip the row, never the file
  380. log.warning("%s: unreadable row (%s) — skipped", path.name, exc)
  381. continue
  382. out[rec.id if hasattr(rec, "id") else rec.key] = rec
  383. return out
  384. def _write_csv(path: Path, recs, columns: list[str]) -> None:
  385. buf = io.StringIO()
  386. w = csv.DictWriter(buf, fieldnames=columns, lineterminator="\n")
  387. w.writeheader()
  388. for rec in recs:
  389. w.writerow(_to_row(rec, columns))
  390. path.parent.mkdir(parents=True, exist_ok=True)
  391. tmp = path.with_suffix(path.suffix + ".tmp")
  392. tmp.write_text(buf.getvalue(), encoding="utf-8")
  393. tmp.replace(path)
  394. def load_sources(root: Path | None = None) -> dict[str, SourcePage]:
  395. return _read_csv(registry_dir(root) / "sources.csv", SourcePage, SOURCE_COLUMNS)
  396. def load_entities(root: Path | None = None) -> dict[str, EntityPage]:
  397. return _read_csv(registry_dir(root) / "entities.csv", EntityPage, ENTITY_COLUMNS)
  398. def load_claims(root: Path | None = None) -> dict[str, ClaimPage]:
  399. return _read_csv(registry_dir(root) / "claims.csv", ClaimPage, CLAIM_COLUMNS)
  400. def save_sources(recs: dict[str, SourcePage], root: Path | None = None) -> None:
  401. _write_csv(
  402. registry_dir(root) / "sources.csv",
  403. sorted(recs.values(), key=lambda r: r.key),
  404. SOURCE_COLUMNS,
  405. )
  406. def save_entities(recs: dict[str, EntityPage], root: Path | None = None) -> None:
  407. _write_csv(
  408. registry_dir(root) / "entities.csv",
  409. sorted(recs.values(), key=lambda r: r.key),
  410. ENTITY_COLUMNS,
  411. )
  412. def save_claims(recs: dict[str, ClaimPage], root: Path | None = None) -> None:
  413. _write_csv(
  414. registry_dir(root) / "claims.csv",
  415. sorted(recs.values(), key=lambda r: r.id),
  416. CLAIM_COLUMNS,
  417. )
  418. def append_pairs(pairs: list[dict], root: Path | None = None) -> None:
  419. """Append-only: a re-adjudication is a new line, never an overwrite of the old one."""
  420. if not pairs:
  421. return
  422. d = registry_dir(root)
  423. d.mkdir(parents=True, exist_ok=True)
  424. with (d / "pairs.jsonl").open("a", encoding="utf-8") as fh:
  425. for p in pairs:
  426. fh.write(json.dumps(p, ensure_ascii=False, sort_keys=True) + "\n")
  427. def read_pairs(root: Path | None = None) -> list[dict]:
  428. p = registry_dir(root) / "pairs.jsonl"
  429. if not p.is_file():
  430. return []
  431. out = []
  432. for lineno, line in enumerate(p.read_text(encoding="utf-8").splitlines(), 1):
  433. line = line.strip()
  434. if not line:
  435. continue
  436. try:
  437. out.append(json.loads(line))
  438. except json.JSONDecodeError as exc:
  439. log.warning("pairs.jsonl:%d unreadable (%s) — skipped", lineno, exc)
  440. return out
  441. def latest_verdicts(root: Path | None = None) -> dict[tuple[str, str], dict]:
  442. """Last write wins per unordered pair — the audit trail keeps every earlier one."""
  443. out: dict[tuple[str, str], dict] = {}
  444. for p in read_pairs(root):
  445. a, b = p.get("left", ""), p.get("right", "")
  446. if a and b:
  447. out[tuple(sorted((a, b)))] = p
  448. return out
  449. # --------------------------------------------------------------------------- pairing
  450. def prefilter(new: ClaimPage, old: ClaimPage) -> str | None:
  451. """Mechanical gate. Returns why the pair is comparable, or None to never emit it.
  452. Two claims are only worth comparing when they are about the same thing. Shared
  453. entity is necessary but not sufficient — "Postgres is popular" and "Postgres 16
  454. ships logical replication" share an entity and compare to nothing. So a pair also
  455. needs a comparable slot: the same unit on both sides (they measure the same
  456. quantity), or enough shared content beyond the entity name.
  457. """
  458. if new.id == old.id or new.run == old.run:
  459. return None
  460. shared_entities = {e for e in new.entities if e} & {e for e in old.entities if e}
  461. if not shared_entities:
  462. return None
  463. units_new = {u for _, u in numbers(new.text) if u}
  464. units_old = {u for _, u in numbers(old.text) if u}
  465. if units_new & units_old:
  466. return f"entity+unit:{sorted(shared_entities)[0]}"
  467. t_new, t_old = tokens(new.text), tokens(old.text)
  468. entity_tokens = {
  469. tok for e in shared_entities for tok in tokens(e.replace("-", " "))
  470. }
  471. t_new, t_old = t_new - entity_tokens, t_old - entity_tokens
  472. if not t_new or not t_old:
  473. return None
  474. jaccard = len(t_new & t_old) / len(t_new | t_old)
  475. if jaccard >= 0.34:
  476. return f"entity+overlap:{jaccard:.2f}"
  477. return None
  478. def auto_verdict(new: ClaimPage, old: ClaimPage) -> tuple[str, str] | None:
  479. """Resolve the pairs that need no judgement. Returns (verdict, reason) or None.
  480. Every branch here exists because it is a way to manufacture a contradiction that
  481. isn't one. These are decided before an adjudicator is ever asked, so no amount of
  482. model confidence can turn a stale price or a different scope into a disagreement.
  483. """
  484. d_new, d_old = parse_date(new.as_of), parse_date(old.as_of)
  485. nums_new, nums_old = numbers(new.text), numbers(old.text)
  486. # A newer figure for the same quantity is a version, not a disagreement.
  487. if d_new and d_old and nums_new and nums_old:
  488. if abs((d_new - d_old).days) > SUPERSEDE_DAYS:
  489. older = old.id if d_new > d_old else new.id
  490. return (
  491. "superseded",
  492. f"as_of {d_old}/{d_new} > {SUPERSEDE_DAYS}d; older={older}",
  493. )
  494. # Different units measure different quantities. 40% and 40 млн do not disagree.
  495. units_new = {u for _, u in nums_new if u}
  496. units_old = {u for _, u in nums_old if u}
  497. if units_new and units_old and not (units_new & units_old):
  498. return (
  499. "different-claim",
  500. f"unit mismatch {sorted(units_old)}/{sorted(units_new)}",
  501. )
  502. # Different scope: US vs EU, median vs average, enterprise vs free, 2024 vs 2026.
  503. q_new, q_old = qualifiers(new.text), qualifiers(old.text)
  504. if (q_new or q_old) and not (q_new & q_old) and (q_new ^ q_old):
  505. return "different-claim", f"scope {sorted(q_old)}/{sorted(q_new)}"
  506. return None
  507. def candidate_pairs(
  508. new_claims: list[ClaimPage], wiki_claims: dict[str, ClaimPage]
  509. ) -> tuple[list[dict], list[dict]]:
  510. """Split comparable pairs into (auto-resolved, needs-adjudication).
  511. Only the second list costs a model call, and only it can ever produce a conflict.
  512. """
  513. auto: list[dict] = []
  514. todo: list[dict] = []
  515. for new in new_claims:
  516. for old in wiki_claims.values():
  517. reason = prefilter(new, old)
  518. if reason is None:
  519. continue
  520. entry = {
  521. "left": old.id,
  522. "right": new.id,
  523. "left_run": old.run,
  524. "right_run": new.run,
  525. "left_as_of": old.as_of,
  526. "right_as_of": new.as_of,
  527. "left_text": old.text,
  528. "right_text": new.text,
  529. "left_roots": old.roots,
  530. "right_roots": new.roots,
  531. "prefilter": reason,
  532. }
  533. resolved = auto_verdict(new, old)
  534. if resolved:
  535. entry["verdict"], entry["reason"], entry["by"] = *resolved, "auto"
  536. auto.append(entry)
  537. else:
  538. todo.append(entry)
  539. return auto, todo
  540. def open_conflicts(root: Path | None = None) -> list[dict]:
  541. """Adjudicated conflicts that nobody has resolved yet.
  542. A conflict is only real with both sides fully attributed: two claim ids, two runs,
  543. two as_of dates, two root sets. A half-populated conflict is a bug in whatever wrote
  544. it, not a finding — wiki_lint fails on those rather than letting them reach a report.
  545. """
  546. out = []
  547. for p in latest_verdicts(root).values():
  548. if p.get("verdict") != "same-claim-conflict" or p.get("resolved"):
  549. continue
  550. out.append(p)
  551. return sorted(out, key=lambda p: p.get("right_as_of", ""), reverse=True)
scripts/validate_phases.py · Bundled script · 965 lines
  1. #!/usr/bin/env python3
  2. """
  3. Phase-gate validator for a completed deepdive run directory.
  4. The methodology in references/ is executed only by the model's discipline inside a
  5. Claude Code session — nothing forces a phase to actually run. This script closes
  6. that gap: it checks that a finished run's output folder contains the file artifact
  7. of every phase that is MANDATORY for the run's depth mode. It catches "the model
  8. skipped a phase" — the failure mode the whole skill is most exposed to (H3).
  9. Depth mode drives which phases are mandatory. The source of truth for that is the
  10. `depth_gate` field in phases.yaml (loaded via phases_manifest — no second YAML
  11. parser). This script owns only the phase->artifact mapping, because not every phase
  12. emits a file and some emit an either/or or a set of files, which does not fit a
  13. scalar YAML field.
  14. It validates PHASE COMPLETENESS, not artifact format (that's eval/validate_structure.py)
  15. and not research quality (that's eval/score_run.py). The two structural validators
  16. are complementary: run both.
  17. Checks (errors fail --strict; warnings never do):
  18. - every phase mandatory for the mode emitted its artifact(s)
  19. - medium/deep: read-set size and sphere quotas, candidates.csv breadth
  20. - mode is known explicitly (--mode) or read from the report/plan frontmatter
  21. - the phase->artifact table still covers every file-emitting phase in phases.yaml
  22. (a self-check: adding such a phase without updating this table warns loudly)
  23. Usage:
  24. python scripts/validate_phases.py --research-dir research/<slug>
  25. python scripts/validate_phases.py --research-dir research/<slug> --mode deep
  26. python scripts/validate_phases.py --research-dir research/<slug> --strict # exit 1 on errors
  27. python scripts/validate_phases.py --research-dir research/<slug> --json
  28. """
  29. from __future__ import annotations
  30. import argparse
  31. import csv
  32. import json
  33. import re
  34. import sys
  35. from pathlib import Path
  36. sys.path.insert(0, str(Path(__file__).resolve().parent))
  37. import phases_manifest # noqa: E402
  38. from quiet import CAP_ERRORS, CAP_WARNINGS, capped # noqa: E402
  39. MODES = ("shallow", "medium", "deep")
  40. # depth_gate = the MINIMUM mode at which a phase is mandatory. A phase applies to a
  41. # run when the run's mode is at least its gate. Ranked so we can compare.
  42. GATE_RANK = {"shallow": 0, "medium": 1, "deep": 2}
  43. # Phase id -> what a completed run must contain when that phase is mandatory.
  44. # `any_of`: at least one of these paths must exist (relative to the run dir).
  45. # `all_of`: every one of these paths must exist.
  46. # `medium_of`: required from medium up — for artifacts of a phase that itself runs at
  47. # every depth but whose bookkeeping is not worth it on a shallow run.
  48. # A directory entry (trailing kind="dir") must be a non-empty directory.
  49. # Phases with no file output (reframing/genre/capability/plan-gate) are absent here
  50. # on purpose; the self-check below knows they are intentionally artifact-less.
  51. PHASE_ARTIFACTS: dict[str, dict[str, list[str]]] = {
  52. "3": {"all_of": ["plan.md"]},
  53. # state.md is the round workspace, rewritten (not appended) each round — the
  54. # orchestrator plans the next round from it instead of from the transcript.
  55. # candidates.csv is the wide, unread pool (title+snippet) that the read set is
  56. # picked from by sphere quotas — without it breadth is whatever search returned.
  57. "4": {"any_of": ["sources", "sources.csv"], "medium_of": ["state.md", "candidates.csv"]},
  58. "5": {"all_of": ["claims.csv"]},
  59. # 5.5 filters the INPUT to synthesis on two axes: relevance (evidence/) and
  60. # authority (.verify/authority.json). The authority verdicts are fail-closed —
  61. # an absent file means the axis never ran, not that everything qualified.
  62. "5.5": {"all_of": ["evidence", ".verify/authority.json"]},
  63. # 5.7 leaves the candidate pairs it built against the cross-run wiki. Fail-closed
  64. # for the same reason as authority.json: an absent file means the run never
  65. # confronted what earlier researches established, not that nothing conflicted.
  66. "5.7": {"all_of": [".verify/wiki_pairs.json"]},
  67. # Phase 6 emits the dated <YYYY-MM-DD>_<genre>.md report (matched by pattern)
  68. # AND the one-page decision memo the consumer's process actually ingests.
  69. # outline.md (section -> block -> claim_id) and numbers.csv (every figure the
  70. # report stands on) are the machine-checkable half of synthesis; both are
  71. # bookkeeping a shallow run is not asked to keep.
  72. # readout.md is the human version (page + full PDF): written after verify,
  73. # passed through humanizer-ru. Without it the run hands a reader the ledger.
  74. "6": {
  75. "report": [],
  76. "all_of": ["memo.md"],
  77. "medium_of": ["outline.md", "numbers.csv", "readout.md"],
  78. },
  79. # Layer 1 (liveness) / Layer 2 (faithfulness) / Layer 3 (qualifier preservation) /
  80. # Layer 4 (construct provenance). All four are required from medium up — 6.5's own
  81. # depth_gate is medium, so a shallow run never reaches this entry at all.
  82. "6.5": {
  83. "all_of": [
  84. ".verify/citations.json",
  85. ".verify/faithfulness.json",
  86. ".verify/qualifiers.json",
  87. ".verify/constructs.json",
  88. ]
  89. },
  90. "7": {"all_of": ["refresh_targets.md"]},
  91. # Phase 8 (decision walkthrough) must leave application.md with ANY status —
  92. # incl. `deferred`. The gate requires the file, not a made decision, so
  93. # unattended runs are never blocked on a human.
  94. "8": {"all_of": ["application.md"]},
  95. }
  96. # Phases that legitimately produce no file artifact — used only by the self-check so
  97. # it does not flag them as an un-mapped file-emitting phase.
  98. NO_FILE_PHASES = {"1", "2", "3.5", "3.7"}
  99. REPORT_RE = re.compile(r"^\d{4}-\d{2}-\d{2}_[a-z]+\.md$")
  100. MODE_RE = re.compile(r"^mode:\s*(\w+)", re.MULTILINE)
  101. DIR_ARTIFACTS = {"sources", "evidence"}
  102. class Report:
  103. def __init__(self) -> None:
  104. self.errors: list[str] = []
  105. self.warnings: list[str] = []
  106. def err(self, m: str) -> None:
  107. self.errors.append(m)
  108. def warn(self, m: str) -> None:
  109. self.warnings.append(m)
  110. def detect_mode(d: Path) -> str | None:
  111. """Read `mode:` from the report frontmatter, falling back to plan.md."""
  112. dated = sorted(p for p in d.glob("*.md") if REPORT_RE.match(p.name))
  113. for candidate in (*dated, d / "plan.md"):
  114. if candidate.is_file():
  115. m = MODE_RE.search(candidate.read_text(encoding="utf-8"))
  116. if m and m.group(1).lower() in MODES:
  117. return m.group(1).lower()
  118. return None
  119. def artifact_present(d: Path, rel: str) -> bool:
  120. p = d / rel
  121. if rel in DIR_ARTIFACTS:
  122. return p.is_dir() and any(p.iterdir())
  123. return p.is_file()
  124. def has_report(d: Path) -> bool:
  125. return any(REPORT_RE.match(p.name) for p in d.glob("*.md"))
  126. def check_phase(
  127. d: Path, phase_id: str, spec: dict[str, list[str]], mode: str, r: Report
  128. ) -> None:
  129. if "report" in spec:
  130. if not has_report(d):
  131. r.err(f"phase {phase_id}: no final report <YYYY-MM-DD>_<genre>.md")
  132. # no return: a phase may require the report AND fixed-name artifacts (6: memo.md)
  133. if "all_of" in spec:
  134. for rel in spec["all_of"]:
  135. if not artifact_present(d, rel):
  136. r.err(f"phase {phase_id}: missing required artifact '{rel}'")
  137. if "medium_of" in spec and GATE_RANK[mode] >= GATE_RANK["medium"]:
  138. for rel in spec["medium_of"]:
  139. if not artifact_present(d, rel):
  140. r.err(
  141. f"phase {phase_id}: missing required artifact '{rel}' "
  142. f"(required from medium up)"
  143. )
  144. if "any_of" in spec:
  145. if not any(artifact_present(d, rel) for rel in spec["any_of"]):
  146. joined = "' or '".join(spec["any_of"])
  147. r.err(f"phase {phase_id}: missing artifact (need '{joined}')")
  148. SOURCE_FILE_RE = re.compile(r"^(\d+)_.+\.md$")
  149. # Schema contract (references/source_scoring.md). Audit of 70 real runs found 26
  150. # header variants of numbers.csv alone: every reader guessed columns by substring and
  151. # the rules reading a missed column silently never fired. The header is now fixed at
  152. # write time: core columns in this exact order, then only the listed optional tail.
  153. CLAIMS_CORE = (
  154. "claim_id",
  155. "claim",
  156. "hypothesis",
  157. "sources",
  158. "source_types",
  159. "roots",
  160. "paths",
  161. "status",
  162. "confidence",
  163. "primary_source",
  164. "source_caveat",
  165. "dissent",
  166. "as_of",
  167. )
  168. CLAIMS_OPTIONAL = ("claim_class", "evidence_kind", "plain", "detail")
  169. NUMBERS_CORE = (
  170. "num_id",
  171. "value",
  172. "unit",
  173. "kind",
  174. "formula",
  175. "inputs",
  176. "group",
  177. "claim_id",
  178. "sources",
  179. "as_of",
  180. )
  181. NUMBERS_OPTIONAL = ("label",)
  182. # LLM-judge verdict files: one shape, so faithfulness is comparable across runs
  183. # (audit: 0.26–1.0 across runs was not comparable — each run invented its own form).
  184. VERDICT_FILES = {
  185. ".verify/faithfulness.json": {"SUPPORTED", "PARTIAL", "UNSUPPORTED"},
  186. ".verify/qualifiers.json": {
  187. "PRESERVED",
  188. "BROADENED",
  189. "SCOPE-DROPPED",
  190. "UNTRACEABLE",
  191. },
  192. }
  193. VERDICT_KEYS = ("claim_id", "verdict", "model")
  194. # Artifacts that only a medium/deep run produces — used to infer the mode when the
  195. # run never wrote a `mode:` frontmatter (2 of 12 real runs had one).
  196. MEDIUM_MARKERS = (
  197. "state.md",
  198. "outline.md",
  199. "numbers.csv",
  200. "evidence",
  201. "refresh_targets.md",
  202. ".verify/authority.json",
  203. ".verify/wiki_pairs.json",
  204. )
  205. # SKILL.md depth table: shallow is 5–7 sources, medium starts at 12.
  206. MEDIUM_SOURCE_COUNT = 10
  207. def check_source_perimeter(d: Path, r: Report) -> None:
  208. """Each fetch sub-agent owns a disjoint id range, enforced only by prompt text.
  209. One number claimed by two files means an agent wrote outside its range (or two
  210. collided): a source file was silently overwritten, so sources.csv no longer
  211. describes what is on disk."""
  212. src = d / "sources"
  213. if not src.is_dir():
  214. return
  215. by_id: dict[str, list[str]] = {}
  216. for p in sorted(src.glob("*.md")):
  217. m = SOURCE_FILE_RE.match(p.name)
  218. if not m:
  219. r.warn(f"sources/{p.name}: name is not NN_slug.md — not indexable by id")
  220. continue
  221. by_id.setdefault(m.group(1).lstrip("0") or "0", []).append(p.name)
  222. for num, names in sorted(by_id.items()):
  223. if len(names) > 1:
  224. r.err(
  225. f"source id {num} claimed by {len(names)} files ({', '.join(names)}) — "
  226. f"a sub-agent wrote outside its assigned range; that agent's results "
  227. f"are not trustworthy, re-run it with a narrowed task"
  228. )
  229. def infer_mode(d: Path) -> tuple[str, str]:
  230. """Best-effort depth from what the run left on disk. Never returns deep: on disk
  231. deep and medium are the same set of files, and guessing the stricter one would
  232. fail a run for a phase (3.5) that leaves no artifact."""
  233. present = [m for m in MEDIUM_MARKERS if artifact_present(d, m)]
  234. if present:
  235. return "medium", f"medium-only artifacts present: {', '.join(present)}"
  236. src = d / "sources"
  237. n = len(list(src.glob("*.md"))) if src.is_dir() else 0
  238. if n >= MEDIUM_SOURCE_COUNT:
  239. return "medium", f"{n} sources (shallow is 5–7)"
  240. return "shallow", f"no medium artifacts, {n} sources"
  241. def resolve_mode(d: Path, explicit: str | None, r: Report) -> str:
  242. """Explicit flag > frontmatter > inference. A run with no `mode:` is validated
  243. anyway — refusing would let the least disciplined runs skip the gate entirely."""
  244. if explicit:
  245. return explicit
  246. found = detect_mode(d)
  247. if found:
  248. return found
  249. mode, why = infer_mode(d)
  250. r.warn(
  251. f"no 'mode:' frontmatter in report or plan.md — inferred '{mode}' ({why}); "
  252. f"add `mode: {mode}` to plan.md or pass --mode"
  253. )
  254. return mode
  255. def header_violation(
  256. header: list[str], core: tuple[str, ...], optional: tuple[str, ...]
  257. ) -> str | None:
  258. """-> why `header` is not canonical, or None."""
  259. head = [c.strip() for c in header]
  260. if tuple(head[: len(core)]) != core:
  261. missing = [c for c in core if c not in head]
  262. if missing:
  263. return f"missing column(s) {', '.join(missing)}"
  264. return f"core columns out of order (expected {','.join(core)})"
  265. unknown = [c for c in head[len(core) :] if c not in optional]
  266. if unknown:
  267. return (
  268. f"unknown column(s) {', '.join(unknown)} after the core "
  269. f"(allowed: {', '.join(optional)})"
  270. )
  271. return None
  272. def check_schema_contract(d: Path, r: Report) -> None:
  273. """A non-canonical ledger is not a formatting nit: the rule that reads a column
  274. it cannot find silently never fires — the 'green check, no behavior' failure."""
  275. for name, core, optional in (
  276. ("claims.csv", CLAIMS_CORE, CLAIMS_OPTIONAL),
  277. ("numbers.csv", NUMBERS_CORE, NUMBERS_OPTIONAL),
  278. ):
  279. path = d / name
  280. if not path.is_file():
  281. continue
  282. header = next(csv.reader(path.read_text(encoding="utf-8").splitlines()), None)
  283. if not header:
  284. r.err(f"{name} is empty")
  285. continue
  286. why = header_violation(header, core, optional)
  287. if why:
  288. r.err(
  289. f"{name}: {why} — the header is fixed by references/source_scoring.md; "
  290. f"rename/reorder the columns, do not invent new ones"
  291. )
  292. for rel, verdicts in VERDICT_FILES.items():
  293. path = d / rel
  294. if not path.is_file():
  295. continue # presence is the phase-gate's business (6.5)
  296. try:
  297. data = json.loads(path.read_text(encoding="utf-8"))
  298. except json.JSONDecodeError as exc:
  299. r.err(f"{rel} is not valid JSON ({exc.msg})")
  300. continue
  301. results = data.get("results") if isinstance(data, dict) else None
  302. if not isinstance(results, list) or not results:
  303. r.err(f"{rel}: no non-empty `results` list — the layer left no verdicts")
  304. continue
  305. bad = []
  306. for i, item in enumerate(results):
  307. if not isinstance(item, dict):
  308. bad.append(f"#{i} not an object")
  309. continue
  310. empty = [k for k in VERDICT_KEYS if not str(item.get(k) or "").strip()]
  311. if empty:
  312. bad.append(f"#{i} lacks {'/'.join(empty)}")
  313. elif str(item["verdict"]).strip() not in verdicts:
  314. bad.append(f"#{i} verdict '{item['verdict']}'")
  315. if bad:
  316. r.err(
  317. f"{rel}: {len(bad)} result(s) off-contract ({'; '.join(bad[:5])}) — "
  318. f"each needs {', '.join(VERDICT_KEYS)} with verdict in {sorted(verdicts)}"
  319. )
  320. # --- source quality ------------------------------------------------------------
  321. # A source that carries fewer than 2 verbatim quotes, or that was written off a
  322. # paywall/consent/abstract stub, is not evidence: it is a URL with a title. It
  323. # happens when WebFetch returned a thin page and the fetch ladder
  324. # (scripts/fetch_source.py) was never run. Such a source must not be a
  325. # triangulation leg — otherwise `triangulated` is backed by nothing.
  326. MIN_QUOTES = 2
  327. THIN_ACCESS_PREFIXES = ("partial", "closed", "paywalled", "abstract-only")
  328. THIN_SHARE_WARN = 0.25
  329. ACCESS_RE = re.compile(r"^access:\s*(.+?)\s*$", re.MULTILINE)
  330. SOURCE_ID_COLUMNS = ("sources", "source_ids", "source", "source_id")
  331. SOURCE_ID_SPLIT_RE = re.compile(r"[;,|\s\[\]()]+")
  332. # Real runs use both `NN_slug.md` and `sNN_slug.md`; the id is the same either way.
  333. SOURCE_ID_FILE_RE = re.compile(r"^s?(\d+)_.+\.md$")
  334. def _norm_source_id(token: str) -> str | None:
  335. """`s01`, `[s01]`, `01`, `1` -> canonical `1`. Anything else -> None."""
  336. tok = token.strip().strip("[]()").lower()
  337. if tok.startswith("s"):
  338. tok = tok[1:]
  339. if not tok.isdigit():
  340. return None
  341. return tok.lstrip("0") or "0"
  342. def _split_frontmatter(text: str) -> tuple[str, str]:
  343. if text.startswith("---"):
  344. parts = text.split("---", 2)
  345. if len(parts) == 3:
  346. return parts[1], parts[2]
  347. return "", text
  348. def source_is_thin(text: str) -> bool:
  349. front, body = _split_frontmatter(text)
  350. m = ACCESS_RE.search(front)
  351. if m:
  352. access = m.group(1).strip().strip("\"'").lower()
  353. if access.startswith(THIN_ACCESS_PREFIXES):
  354. return True
  355. quotes = sum(1 for line in body.splitlines() if line.lstrip().startswith(">"))
  356. return quotes < MIN_QUOTES
  357. def collect_thin_sources(d: Path) -> tuple[dict[str, str], set[str]]:
  358. """-> ({normalized id: label}, {normalized id of every source file})."""
  359. src = d / "sources"
  360. thin: dict[str, str] = {}
  361. seen: set[str] = set()
  362. if not src.is_dir():
  363. return thin, seen
  364. for p in sorted(src.glob("*.md")):
  365. m = SOURCE_ID_FILE_RE.match(p.name)
  366. if not m:
  367. continue
  368. num = m.group(1)
  369. key = num.lstrip("0") or "0"
  370. seen.add(key)
  371. if source_is_thin(p.read_text(encoding="utf-8")):
  372. thin[key] = f"s{num}"
  373. return thin, seen
  374. # Breadth of the read set (references/phases/p4_search.md «Квоты»). Benchmark
  375. # 2026-10-28: 6 of 6 runs had 16–19 sources, 0 Academic, 3–4 types — the search
  376. # stopped at whatever the first queries returned.
  377. MIN_READ = {"medium": 25, "deep": 35}
  378. MIN_CANDIDATES = {"medium": 60, "deep": 100}
  379. TYPE_QUOTAS = {
  380. "academic": ({"academic"}, 2),
  381. "primary": ({"primary"}, 3),
  382. "media": ({"industry-media", "general-media"}, 2),
  383. "expert": ({"expert-blog"}, 2),
  384. "practice": ({"forum"}, 2),
  385. }
  386. MIN_CONTRADICTING = 3
  387. CANDIDATE_COLUMNS = ("url", "title", "snippet", "channel", "sphere", "stance", "vendor",
  388. "round", "target", "selected")
  389. ACADEMIC_CHANNELS = {"openalex", "semantic_scholar", "arxiv", "crossref"}
  390. MIN_TARGETED_ROUND = 5
  391. WAIVER_RE = re.compile(r"^quota_waiver:\s*(\w+)\s*[—-]\s*(\S.*)$", re.MULTILINE)
  392. # Depth and pool size are what the skill is for; N2 (2026-10-28) waived them in
  393. # plan.md before searching, citing the run budget.
  394. NON_WAIVABLE = {"count", "candidates", "round2"}
  395. BUDGET_REASON_RE = re.compile(r"бюджет|budget|\$\s?\d|деньг|дорог", re.IGNORECASE)
  396. FIELD_RE = re.compile(r"^(type|caveat):\s*(.+?)\s*$", re.MULTILINE)
  397. VERDICT_TABLE_RE = re.compile(r"^\|[^\n]*гипотез[^\n]*\|[^\n]*вердикт", re.IGNORECASE | re.MULTILINE)
  398. def check_verdict_table(d: Path, r: Report) -> None:
  399. """Benchmark 2026-10-28: explicit hypothesis verdicts in 2 of 6 reports."""
  400. dated = sorted(p for p in d.glob("*.md") if REPORT_RE.match(p.name))
  401. if dated and not VERDICT_TABLE_RE.search(dated[-1].read_text(encoding="utf-8")):
  402. r.err(f"{dated[-1].name}: no `| Гипотеза | Вердикт | Опора |` table — every "
  403. f"plan hypothesis needs an explicit verdict (phases/p6_synthesis.md)")
  404. # An unconditional pick: "по умолчанию", or an imperative with no "если" in the sentence.
  405. DEFAULT_RE = re.compile(r"(?<![«\w])(по умолчанию|берите|выбирайте|начинать стоит|начните с|рекомендуем)(?![»\w])",
  406. re.IGNORECASE)
  407. SENTENCE_SPLIT_RE = re.compile(r"(?<=[.!?])\s+|\n")
  408. def _default_sentences(text: str) -> list[str]:
  409. """Unconditional picks in «Ответ» / «Рекомендация» with no [sNN] behind them."""
  410. sections = re.split(r"^## ", text, flags=re.MULTILINE)
  411. scope = [sec for sec in sections if sec.lower().startswith("ответ")]
  412. scope += re.findall(r"^\**Рекомендаци[яи][^\n]*", text, re.MULTILINE)
  413. hits = []
  414. for chunk in scope:
  415. for sent in SENTENCE_SPLIT_RE.split(chunk):
  416. if DEFAULT_RE.search(sent) and not re.search(r"\bне\b[^.]*по умолчанию", sent, re.IGNORECASE) \
  417. and not re.search(r"\bесли\b", sent, re.IGNORECASE) \
  418. and not re.search(r"\[s\d{2,3}\]", sent):
  419. hits.append(sent.strip())
  420. return hits
  421. def check_unmeasured_default(d: Path, r: Report) -> None:
  422. """Benchmark 2026-10-28: 7 of 8 runs said 'Fivetran по умолчанию' with no price found."""
  423. dated = sorted(p for p in d.glob("*.md") if REPORT_RE.match(p.name))
  424. targets = [p for p in [d / "readout.md", *dated[-1:]] if p.is_file()]
  425. for p in targets:
  426. for sent in _default_sentences(p.read_text(encoding="utf-8")):
  427. r.err(f"{p.name}: default recommendation without a measured source — «{sent[:90]}». "
  428. f"Give conditional branches + the cheapest way to measure (phases/p6_synthesis.md, Антибиас)")
  429. PRE_REMEDIATION = "findings/draft_pre_remediation.md"
  430. HIGH_ID_RE = re.compile(r"^\W*(?:HIGH|H)-(\d+)\b", re.MULTILINE)
  431. REVIEW_HEADING_RE = re.compile(r"adversarial review|ремедиац|red team", re.IGNORECASE)
  432. # N4/N6 (2026-10-28): fixes were appended after the body, TL;DR kept the refuted claims.
  433. APPENDIX_RE = re.compile(r"приоритет над текстом|заменяется на|поправки ниже|см\. ремедиац", re.IGNORECASE)
  434. # "- HIGH-1 → §Ответ (TL;DR), §Ветки: что изменено" — a target ends at ":", "§", "|" or ", §".
  435. FIX_TARGET_RE = re.compile(r"§\s*([^§|:\n]+?)\s*(?=,\s*§|[§|:\n]|$)")
  436. def _sections(text: str) -> dict[str, str]:
  437. parts = re.split(r"^(#{2,3} .+)$", text, flags=re.MULTILINE)
  438. return {h.lstrip("# ").strip().casefold(): body for h, body in zip(parts[1::2], parts[2::2])}
  439. def _same(a: str | None, b: str | None) -> bool:
  440. return a is not None and b is not None and a.split() == b.split()
  441. def _find_section(sections: dict[str, str], target: str) -> str | None:
  442. t = target.strip(" *`").casefold()
  443. return next((h for h in sections if h == t or h.startswith(t)), None)
  444. def check_remediation_in_body(d: Path, mode: str, r: Report) -> None:
  445. """Each HIGH from the red-team log is fixed in the body section it names, not in an appendix."""
  446. logs = sorted((d / "findings").glob("redteam_*.md")) if (d / "findings").is_dir() else []
  447. dated = sorted(p for p in d.glob("*.md") if REPORT_RE.match(p.name))
  448. if GATE_RANK[mode] < GATE_RANK["medium"] or not dated:
  449. return
  450. if not logs:
  451. # N6 (2026-10-28) kept no log, so nothing below could check its remediation.
  452. r.err("findings/redteam_<date>.md missing — red team triage log (adversarial_pass.md)")
  453. return
  454. highs = sorted(set(HIGH_ID_RE.findall(logs[-1].read_text(encoding="utf-8"))), key=int)
  455. if not highs:
  456. return
  457. report = dated[-1]
  458. text = report.read_text(encoding="utf-8")
  459. for m in APPENDIX_RE.finditer(text):
  460. r.err(f"{report.name}: «{m.group(0)}» — remediation overrides the body from an appendix; "
  461. f"edit the section itself (phases/p6_synthesis.md, шаг 6)")
  462. snap = d / PRE_REMEDIATION
  463. if not snap.is_file():
  464. r.err(f"{PRE_REMEDIATION} missing — copy the draft there before `remediate`, "
  465. f"otherwise body fixes cannot be told from an appended postscript")
  466. return
  467. now, before = _sections(text), _sections(snap.read_text(encoding="utf-8"))
  468. review = next((body for h, body in now.items() if REVIEW_HEADING_RE.search(h)), "")
  469. changed_any = False
  470. for hid in highs:
  471. line = next((ln for ln in review.splitlines() if re.search(rf"\b(?:HIGH|H)-{hid}\b", ln)), None)
  472. if line is None:
  473. r.err(f"HIGH-{hid} from {logs[-1].name} not listed in `## Adversarial review` "
  474. f"with `→ §<section>` or `→ Open Questions`")
  475. continue
  476. if "open questions" in line.casefold():
  477. continue
  478. targets = FIX_TARGET_RE.findall(line)
  479. if not targets:
  480. r.err(f"HIGH-{hid}: no `§<section>` it was fixed in — «{line.strip()[:90]}»")
  481. for t in targets:
  482. h = _find_section(now, t)
  483. if h is None:
  484. r.err(f"HIGH-{hid}: section «§{t}» not found in {report.name}")
  485. elif _same(now[h], before.get(h)):
  486. r.err(f"HIGH-{hid}: «§{t}» unchanged vs {PRE_REMEDIATION} — the fix is not in the body")
  487. else:
  488. changed_any = True
  489. head = next((h for h in now if h.startswith(("ответ", "tl;dr"))), None)
  490. if changed_any and head and _same(now[head], before.get(head)):
  491. r.warn(f"{report.name}: body fixed after red team but «{head}» unchanged — "
  492. f"check it no longer states the refuted claims")
  493. def check_source_breadth(d: Path, mode: str, r: Report) -> None:
  494. if GATE_RANK[mode] < GATE_RANK["medium"]:
  495. return
  496. plan = d / "plan.md"
  497. waived: set[str] = set()
  498. for key, reason in WAIVER_RE.findall(plan.read_text(encoding="utf-8")) if plan.is_file() else []:
  499. if key in NON_WAIVABLE:
  500. r.err(f"quota_waiver: {key} — not waivable: depth and candidate pool are the point of medium/deep")
  501. elif BUDGET_REASON_RE.search(reason):
  502. r.err(f"quota_waiver: {key} — budget is not a reason; waive only what the search "
  503. f"did not find (cite candidates.csv)")
  504. else:
  505. waived.add(key)
  506. def gate(key: str, msg: str) -> None:
  507. hint = "" if key in waived or key in NON_WAIVABLE else f" — or add `quota_waiver: {key} — <why>` to plan.md"
  508. (r.warn if key in waived else r.err)(msg + hint)
  509. files = sorted((d / "sources").glob("*.md")) if (d / "sources").is_dir() else []
  510. if len(files) < MIN_READ[mode]:
  511. gate("count", f"{len(files)} sources read, {mode} needs ≥{MIN_READ[mode]}")
  512. types: list[str] = []
  513. vendor = contradicting = 0
  514. for p in files:
  515. front, _ = _split_frontmatter(p.read_text(encoding="utf-8"))
  516. fields = dict(FIELD_RE.findall(front))
  517. types.append(fields.get("type", "").strip("\"'").lower())
  518. vendor += fields.get("caveat", "").strip("\"'").lower() == "vendor"
  519. contradicting += bool(re.search(r"^\s+H\d+:\s*contradicts\b", front, re.MULTILINE))
  520. for key, (kinds, need) in TYPE_QUOTAS.items():
  521. have = sum(t in kinds for t in types)
  522. if have < need:
  523. gate(key, f"{have} {key} sources (type {'/'.join(sorted(kinds))}), quota ≥{need}")
  524. if contradicting < MIN_CONTRADICTING:
  525. gate("against", f"{contradicting} sources contradict a hypothesis "
  526. f"(hypothesis_evidence: contradicts), quota ≥{MIN_CONTRADICTING}")
  527. if files and vendor * 2 > len(files):
  528. gate("vendor", f"{vendor} of {len(files)} sources are vendor material — "
  529. f"at most half; the rest must be vendor-independent")
  530. no_card = [p.name for p in files if "\n## Разбор" not in p.read_text(encoding="utf-8")]
  531. if no_card:
  532. r.warn(f"{len(no_card)} of {len(files)} sources have no `## Разбор` card "
  533. f"(claim type, method, scope, modality) — e.g. {', '.join(no_card[:3])}")
  534. check_verdict_table(d, r)
  535. check_unmeasured_default(d, r)
  536. cand = d / "candidates.csv"
  537. if not cand.is_file():
  538. return # absence is reported by the phase-4 artifact check
  539. with cand.open(encoding="utf-8") as fh:
  540. reader = csv.DictReader(fh)
  541. missing = [c for c in CANDIDATE_COLUMNS if c not in (reader.fieldnames or [])]
  542. rows = list(reader)
  543. if missing:
  544. r.err(f"candidates.csv: missing columns {', '.join(missing)}")
  545. return
  546. if len(rows) < MIN_CANDIDATES[mode]:
  547. gate("candidates", f"candidates.csv has {len(rows)} rows, {mode} needs ≥{MIN_CANDIDATES[mode]}")
  548. if not any(row["channel"].strip().lower() in ACADEMIC_CHANNELS for row in rows):
  549. gate("academic", "no candidate came from an academic API "
  550. f"({', '.join(sorted(ACADEMIC_CHANNELS))})")
  551. targeted = [row for row in rows if row["round"].strip() == "2" and row["target"].strip()]
  552. if len(targeted) < MIN_TARGETED_ROUND:
  553. gate("round2", f"{len(targeted)} round-2 candidates with a `target` — the second "
  554. f"round must aim at the decision variable, ≥{MIN_TARGETED_ROUND}")
  555. selected = sum(row["selected"].strip().upper() == "Y" for row in rows)
  556. if files and selected and selected != len(files):
  557. r.warn(f"candidates.csv marks {selected} selected, sources/ has {len(files)}")
  558. def check_source_quality(d: Path, mode: str, r: Report) -> None:
  559. """Thin sources silently backing `triangulated` claims.
  560. Measured on 12 real runs: 27% of source files carry <2 quotes and every one
  561. of them reports `fetch_tier: webfetch` — the ladder was never used. A claim
  562. standing only on such sources is not triangulated, it is one bad fetch.
  563. """
  564. thin, seen = collect_thin_sources(d)
  565. if not seen:
  566. return
  567. for row in load_claim_rows(d):
  568. if (row.get("status") or "").strip().lower() != "triangulated":
  569. continue
  570. cid = (row.get("claim_id") or "").strip() or "?"
  571. cell = next(
  572. (row[c] for c in SOURCE_ID_COLUMNS if (row.get(c) or "").strip()), ""
  573. )
  574. cited = []
  575. for tok in SOURCE_ID_SPLIT_RE.split(cell):
  576. key = _norm_source_id(tok)
  577. if key is not None and key in seen and key not in cited:
  578. cited.append(key)
  579. if not cited:
  580. continue
  581. bad = [thin[k] for k in cited if k in thin]
  582. if not bad:
  583. continue
  584. listed = ", ".join(bad)
  585. if len(bad) == len(cited):
  586. msg = (
  587. f"claim {cid} is triangulated on thin sources only ({listed}) — "
  588. f"thin = <{MIN_QUOTES} quotes or access PARTIAL/closed; refetch via "
  589. f"scripts/fetch_source.py or downgrade"
  590. )
  591. (r.err if GATE_RANK[mode] >= GATE_RANK["medium"] else r.warn)(msg)
  592. else:
  593. r.warn(
  594. f"claim {cid} is triangulated partly on thin sources ({listed}) — "
  595. f"refetch them via scripts/fetch_source.py or drop them as legs"
  596. )
  597. share = len(thin) / len(seen)
  598. if share > THIN_SHARE_WARN:
  599. r.warn(
  600. f"{len(thin)} of {len(seen)} sources are thin ({share * 100:.0f}%) — "
  601. f"the fetch ladder was not used; see subagents_v2.md §fetch"
  602. )
  603. OUTLINE_ROW_RE = re.compile(r"^\|(?P<cells>.+)\|\s*$")
  604. CLAIM_CELL_SPLIT_RE = re.compile(r"[;,\s]+")
  605. # Claims this strong are the run's product: reaching one and leaving it out of the
  606. # report is work thrown away silently, not an editorial choice.
  607. CARRYING_STATUSES = {"triangulated", "contested"}
  608. STATE_SECTIONS = ("## Known", "## Gaps", "## Next")
  609. STATE_SOFT_LIMIT = 6 * 1024
  610. STATE_HARD_LIMIT = 12 * 1024
  611. def parse_outline(text: str) -> dict[str, list[str]]:
  612. """Read the `| section | block | claims |` table -> {section: [claim_id, ...]}.
  613. The claims cell is tokenized rather than pattern-matched: claim ids are whatever
  614. claims.csv calls them, and a validator that hard-codes `CL\\d+` silently reads
  615. zero claims from a run that numbered them differently.
  616. """
  617. out: dict[str, list[str]] = {}
  618. for line in text.splitlines():
  619. m = OUTLINE_ROW_RE.match(line.strip())
  620. if not m:
  621. continue
  622. cells = [c.strip() for c in m.group("cells").split("|")]
  623. if len(cells) < 3:
  624. continue
  625. section = cells[0]
  626. if not section or section.lower() == "section" or set(section) <= set("-: "):
  627. continue # header or separator row
  628. tokens = [
  629. t for t in CLAIM_CELL_SPLIT_RE.split(cells[2]) if t and t not in {"-", "—"}
  630. ]
  631. out[section] = tokens
  632. return out
  633. def load_claim_rows(d: Path) -> list[dict[str, str]]:
  634. ledger = d / "claims.csv"
  635. if not ledger.is_file():
  636. return []
  637. return list(csv.DictReader(ledger.read_text(encoding="utf-8").splitlines()))
  638. def check_outline_coverage(d: Path, mode: str, r: Report) -> None:
  639. """The outline is the join between what was found and what got written.
  640. Three failures live here: a section with no claims (prose with no ledger behind
  641. it), a claim_id that resolves to nothing, and — the expensive one — a carrying
  642. claim that never reached any section. The last is the synthesis gap: retrieval
  643. worked, organization dropped it.
  644. """
  645. outline = d / "outline.md"
  646. if not outline.is_file():
  647. return # presence is the phase-gate's business, above
  648. sections = parse_outline(outline.read_text(encoding="utf-8"))
  649. if not sections:
  650. r.err(
  651. "outline.md has no `| section | block | claims |` table rows — "
  652. "the section->claim map is what makes per-section synthesis checkable"
  653. )
  654. return
  655. rows = load_claim_rows(d)
  656. known = {(row.get("claim_id") or "").strip() for row in rows} - {""}
  657. mapped: set[str] = set()
  658. for section, claims in sections.items():
  659. if not claims:
  660. r.err(f"outline.md: section '{section}' maps to no claim_id")
  661. continue
  662. for cid in claims:
  663. if known and cid not in known:
  664. r.err(
  665. f"outline.md: section '{section}' cites {cid}, absent from claims.csv"
  666. )
  667. mapped.add(cid)
  668. if not rows:
  669. return
  670. orphans = []
  671. for row in rows:
  672. status = (row.get("status") or "").strip().lower()
  673. cid = (row.get("claim_id") or "").strip()
  674. if status in CARRYING_STATUSES and cid and cid not in mapped:
  675. orphans.append(f"{cid} ({status})")
  676. if orphans:
  677. message = (
  678. f"claims reached but never placed in the report: {', '.join(orphans)} — "
  679. f"triangulated/contested work dropped at synthesis, not at search "
  680. f"(map it to a section or say in Open Questions why it is out)"
  681. )
  682. # On deep this is a blocker; on medium a loud warning — a medium run may
  683. # legitimately carry more than its outline holds.
  684. (r.err if mode == "deep" else r.warn)(message)
  685. def check_constructs(d: Path, r: Report) -> None:
  686. """Layer 4: an unsourced named construct in the memo/TL;DR blocks finish."""
  687. path = d / ".verify/constructs.json"
  688. if not path.is_file():
  689. return # presence is enforced by PHASE_ARTIFACTS for 6.5
  690. try:
  691. data = json.loads(path.read_text(encoding="utf-8"))
  692. except json.JSONDecodeError as exc:
  693. r.err(
  694. f".verify/constructs.json is not valid JSON ({exc.msg}) — Layer 4 cannot be read"
  695. )
  696. return
  697. results = data.get("results")
  698. if not isinstance(results, list):
  699. r.err(
  700. ".verify/constructs.json has no `results` list — Layer 4 produced nothing"
  701. )
  702. return
  703. for item in results:
  704. if not isinstance(item, dict):
  705. continue
  706. status = str(item.get("status", "")).strip().lower()
  707. name = item.get("name", "?")
  708. locations = [str(loc) for loc in item.get("locations", [])]
  709. if status == "unsourced":
  710. carried = [loc for loc in locations if loc in {"memo.md", "F1", "F9"}]
  711. if carried:
  712. r.err(
  713. f"construct '{name}' is unsourced and appears in {', '.join(carried)} — "
  714. f"source it, mark it as ours, or remove it (Layer 4 fail-closed)"
  715. )
  716. else:
  717. r.warn(
  718. f"construct '{name}' is unsourced (in {', '.join(locations) or '?'})"
  719. )
  720. elif status not in {"sourced", "author-construct"}:
  721. r.warn(
  722. f"construct '{name}': status '{status or 'empty'}' is not one of "
  723. f"sourced/author-construct/unsourced"
  724. )
  725. def check_state_window(d: Path, r: Report) -> None:
  726. """state.md is a rebuilt window, not a second transcript."""
  727. state = d / "state.md"
  728. if not state.is_file():
  729. return # presence is enforced by PHASE_ARTIFACTS for phase 4
  730. text = state.read_text(encoding="utf-8")
  731. missing = [s for s in STATE_SECTIONS if s not in text]
  732. if missing:
  733. r.err(
  734. f"state.md is missing section(s) {', '.join(missing)} — the next round is "
  735. f"planned from Known/Gaps/Next, so a missing one is a silently empty input"
  736. )
  737. size = len(text.encode("utf-8"))
  738. if size > STATE_HARD_LIMIT:
  739. r.err(
  740. f"state.md is {size // 1024} KB (limit {STATE_HARD_LIMIT // 1024} KB) — "
  741. f"it is being appended to, not rebuilt; the archive belongs in "
  742. f"sources/, deviations.md and plan.md §15"
  743. )
  744. elif size > STATE_SOFT_LIMIT:
  745. r.warn(
  746. f"state.md is {size // 1024} KB (soft limit {STATE_SOFT_LIMIT // 1024} KB) — "
  747. f"compress Known to subquestion statuses with claim_id/[sNN] references"
  748. )
  749. def self_check(phases: list[dict], r: Report) -> None:
  750. """Warn if phases.yaml gained a file-emitting phase this table does not cover."""
  751. known = set(PHASE_ARTIFACTS) | NO_FILE_PHASES
  752. for p in phases:
  753. if p["id"] not in known:
  754. r.warn(
  755. f"phase {p['id']} ({p['name_en']}) is in phases.yaml but not in this "
  756. f"validator's artifact table — add it to PHASE_ARTIFACTS or NO_FILE_PHASES"
  757. )
  758. def check_wiki_ingest(d: Path, r: Report) -> None:
  759. """The run must have been compiled into the cross-run wiki (finish-up step 0).
  760. Not in PHASE_ARTIFACTS because it belongs to no phase: ingest runs at finish-up, at
  761. every depth, and its real output lives outside the run directory. The receipt is
  762. what makes it checkable here at all — without a gate this is precisely the kind of
  763. step that keeps being declared and quietly stops running.
  764. """
  765. receipt = d / ".verify" / "wiki_ingest.json"
  766. if not receipt.is_file():
  767. r.err(
  768. "finish-up: нет .verify/wiki_ingest.json — прогон не попал в кросс-прогонную "
  769. "вики (python scripts/wiki_ingest.py --research-dir <dir>)"
  770. )
  771. return
  772. try:
  773. data = json.loads(receipt.read_text(encoding="utf-8"))
  774. except json.JSONDecodeError as exc:
  775. r.err(f"wiki_ingest.json нечитаем ({exc}) — перезапусти wiki_ingest.py")
  776. return
  777. if data.get("run") != d.name:
  778. r.err(
  779. f"wiki_ingest.json от прогона '{data.get('run')}', а папка '{d.name}' — "
  780. "квитанция скопирована из чужого ресёрча"
  781. )
  782. def validate(d: Path, mode: str, phases: list[dict], r: Report) -> None:
  783. self_check(phases, r)
  784. check_source_perimeter(d, r)
  785. check_schema_contract(d, r)
  786. check_source_quality(d, mode, r)
  787. check_source_breadth(d, mode, r)
  788. check_remediation_in_body(d, mode, r)
  789. check_state_window(d, r)
  790. check_outline_coverage(d, mode, r)
  791. check_constructs(d, r)
  792. check_wiki_ingest(d, r)
  793. gate_of = {p["id"]: p["depth_gate"] for p in phases}
  794. run_rank = GATE_RANK[mode]
  795. for phase_id, spec in PHASE_ARTIFACTS.items():
  796. gate = gate_of.get(phase_id)
  797. if gate is None:
  798. r.warn(f"phase {phase_id} is mapped here but not present in phases.yaml")
  799. continue
  800. if GATE_RANK[gate] <= run_rank: # mandatory for this mode
  801. check_phase(d, phase_id, spec, mode, r)
  802. def main() -> int:
  803. ap = argparse.ArgumentParser(
  804. description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
  805. )
  806. ap.add_argument("--research-dir", required=True, type=Path)
  807. ap.add_argument(
  808. "--mode",
  809. choices=MODES,
  810. help="run depth; frontmatter, else inferred from artifacts, if omitted",
  811. )
  812. ap.add_argument("--strict", action="store_true", help="Exit 1 if any error")
  813. ap.add_argument("--json", action="store_true")
  814. ap.add_argument("--verbose", action="store_true", help="print every warning, not the first few")
  815. args = ap.parse_args()
  816. d = args.research_dir
  817. if not d.is_dir():
  818. print(f"ERROR: not a directory: {d}")
  819. return 2
  820. r = Report()
  821. mode = resolve_mode(d, args.mode, r)
  822. phases = phases_manifest.load_phases(
  823. Path(__file__).resolve().parents[1] / "phases.yaml"
  824. )
  825. validate(d, mode, phases, r)
  826. if args.json:
  827. print(
  828. json.dumps(
  829. {
  830. "research_dir": str(d),
  831. "mode": mode,
  832. "errors": r.errors,
  833. "warnings": r.warnings,
  834. },
  835. indent=2,
  836. )
  837. )
  838. else:
  839. print(f"Validating phases: {d} (mode: {mode})")
  840. for line in capped((f" ERROR {m}" for m in r.errors), CAP_ERRORS, args.verbose):
  841. print(line)
  842. for line in capped((f" warn {m}" for m in r.warnings), CAP_WARNINGS, args.verbose):
  843. print(line)
  844. if not r.errors and not r.warnings:
  845. print(" OK — all mandatory phases produced their artifacts")
  846. elif not r.errors:
  847. print(f"\n{len(r.warnings)} warning(s), 0 errors.")
  848. else:
  849. print(f"\n{len(r.errors)} error(s), {len(r.warnings)} warning(s).")
  850. if args.strict and r.errors:
  851. return 1
  852. return 0
  853. if __name__ == "__main__":
  854. sys.exit(main())

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.