deepdive
Deepdive skill for Claude Code — 12-phase research pipeline: plan-review gate, parallel sub-agent search, claims-ledger triangulation with dissent protection, relevance × authority evidence filter, multi-angle red team, four-layer citation verification. 105 blocks, 29 channels, 460+ stat sources, 47 APIs, 1072 verified endpoints.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Socialpranker/deepdive with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Socialpranker/deepdive'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/socialpranker--deepdive)HTML
<a href="/listings/socialpranker--deepdive"><img src="/api/listings/socialpranker--deepdive/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
<details open> <summary><b>For Claude Code (CLI)</b></summary>
git clone https://github.com/Socialpranker/deepdive.git \ ~/.claude/skills/deepdive
That's it. Now type any of these in a Claude Code session: - "Investigate X" - "Изучи тему" - "Validate this hypothesis"
</details>
<details> <summary><b>For Claude Desktop (Skills enabled)</b></summary>
# Clone git clone https://github.com/Socialpranker/deepdive.git cd deepdive # Package as .skill bundle zip -r ../deepdive.skill . -x ".*" -x "*.zip" # Upload via Claude.app → Settings → Skills → Add Skill
</details>
<details> <summary><b>For other LLMs (Codex, Gemini, local)</b></summary>
The 13-phase methodology is portable. Load `SKILL.md` + `references/*.md` into the LLM's context manually. Skip the sub-agent parts and use separate chat sessions per subtopic.
[Full instructions →](#use-with-other-llms-codex-gemini-etc)
</details>
---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 16 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- {
- "name": "deepdive",
- "owner": {
- "name": "Socialpranker"
- },
- "plugins": [
- {
- "name": "deepdive",
- "source": "./",
- "description": "Meta-research под вопрос или решение: веб-поиск, академические источники, Q&A отчёт с атомарными пере-используемыми тезисами.",
- "skills": [
- "./"
- ]
- }
- ]
- }
README.md · README · 580 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <div align="center">
- <img src="assets/logo.svg" width="96" height="96" alt="deepdive logo">
- # Deepdive
- ### A structured meta-research skill for Claude Code
- **Stop ad-hoc Googling. Start documented investigation.**
- [](LICENSE)
- [](https://claude.com/claude-code)
- [](https://docs.anthropic.com/claude/docs/skills)
- [](CONTRIBUTING.md)
- [](.github/workflows/catalog-sync.yml)
- [](https://github.com/Socialpranker/deepdive/stargazers)
- <br>
- **[Docs](https://socialpranker.github.io/deepdive/)** · **[Install](#install-in-30-seconds)** · **[How it works](#how-it-works)** · **[Contribute](CONTRIBUTING.md)**
- <br>
- ```
- You: investigate the trade-offs between Postgres logical replication and CDC tooling
- Claude: ✓ Reframed your question (3 hypotheses) + decision spec: what you'll do with the answer
- ✓ Picked genre: decision (comparison + validation)
- ✓ Wrote plan.md (17 sections)
- ✓ Checked your env: 4 APIs available, 2 fallback to HTML
- ✓ Launched 4 sub-agents across 12 channels
- ✓ Saved 23 sources to sources/ with quotes, provenance-checked
- ✓ Ran adversarial pass (3 counter-arguments + an "execute this" role)
- ✓ Report ready: research/postgres-replication-vs-cdc/2026-05-21_decision.md
- ✓ Walked through your decision forks — you picked CDC tooling, logged to application.md
- ```
- </div>
- ---
- ## What this is
- A [Claude Code skill](https://docs.anthropic.com/claude/docs/skills) that turns **"research this topic"** into a **<!--gen:count:phases-->13<!--/gen-->-phase pipeline** with hypothesis testing, parallel sub-agent search, source triangulation, and adversarial review.
- The output is a folder you can return to in a month. Every claim traces to a specific source file. The plan documents *why* you made every choice. No re-research needed.
- > **New here?** Start with the [Quickstart](QUICKSTART.md) — install → invoke → first result in ~5 min.
- <table>
- <tr>
- <th width="50%">Without this</th>
- <th width="50%">With this</th>
- </tr>
- <tr>
- <td>
- One-shot prompt → wall of text
- Sources lost in chat history
- No way to detect bias
- No reuse next time
- Generic Google results
- Sources include... *(vague)*
- </td>
- <td>
- 17-section `plan.md` documents every choice
- Each source = file with verbatim quotes
- Mandatory adversarial pass + opposition queries
- Atomic theses in `findings/FN.md` reusable
- <!--gen:count:channels-->29<!--/gen--> named channels + <!--gen:count:stat_sources-->460<!--/gen-->+ stat sources
- Every claim → `[s12]` link → specific quote
- </td>
- </tr>
- </table>
- ---
- ## Install in 30 seconds
- <details open>
- <summary><b>For Claude Code (CLI)</b></summary>
- ```bash
- git clone https://github.com/Socialpranker/deepdive.git \
- ~/.claude/skills/deepdive
- ```
- That's it. Now type any of these in a Claude Code session:
- - "Investigate X"
- - "Изучи тему"
- - "Validate this hypothesis"
- </details>
- <details>
- <summary><b>For Claude Desktop (Skills enabled)</b></summary>
- ```bash
- # Clone
- git clone https://github.com/Socialpranker/deepdive.git
- cd deepdive
- # Package as .skill bundle
- zip -r ../deepdive.skill . -x ".*" -x "*.zip"
- # Upload via Claude.app → Settings → Skills → Add Skill
- ```
- </details>
- <details>
- <summary><b>For other LLMs (Codex, Gemini, local)</b></summary>
- The <!--gen:count:phases-->13<!--/gen-->-phase methodology is portable. Load `SKILL.md` + `references/*.md` into the LLM's context manually. Skip the sub-agent parts and use separate chat sessions per subtopic.
- [Full instructions →](#use-with-other-llms-codex-gemini-etc)
- </details>
- ---
- ## How it works
- The skill runs **<!--gen:count:phases-->13<!--/gen--> phases** in order:
- | Phase | Name | What happens |
- |:---:|:---|:---|
- <!--gen:phases:table:en-->
- | **1** | **Reframing** | opus / high |
- | **2** | **Genre & block selection** | sonnet / medium |
- | **3** | **Plan** | opus / medium |
- | **3.5** | **Capability Discovery** | sonnet / low |
- | **3.7** | **Plan-review gate** | sonnet / low |
- | **4** | **Search** | sonnet / medium |
- | **5** | **Claims-ledger + triangulation** | haiku / low |
- | **5.5** | **Evidence filter** | sonnet / low |
- | **5.7** | **Wiki reconcile** | sonnet / low |
- | **6** | **Synthesis + multi-angle red team** | sonnet → opus on deep / high |
- | **6.5** | **Verify** | haiku / low |
- | **7** | **Refresh targets** | sonnet / medium |
- | **8** | **Decision walkthrough** | opus / high |
- <!--/gen-->
- Each phase runs on a model matched to its task — Opus where reasoning multiplies (1/3/6), Haiku for the parallel fan-out (4). The skill announces the routing and an estimated cost up front, once.
- Every phase is **transparent**: you see what's happening, you confirm key decisions, and you get a folder you can return to. Before any search fires, the **plan-review gate** (3.7) shows you the reframing, hypotheses, genre, and channels and lets you approve or edit them — strictness scales with mode (deep waits for an explicit go-ahead, medium is a soft check, shallow skips it). Editing the plan before execution is the single highest-leverage step in the whole pipeline — Gemini Deep Research calls plan review its "biggest lever over output quality," and a wrong plan executed perfectly still produces a wrong report.
- Reframing (1) doesn't just restate the question — a **router** classifies its profile (factual / multi-step / relational / comparative / landscape) and that classification picks the decomposition method: factual questions get flat independent subquestions, multi-step ones ("X given Y") get least-to-most leveling, comparative ones get a shared axis matrix with mandatory opposition queries per candidate. Picking the wrong decomposition for a question's shape is a silent failure mode — the router makes the choice explicit instead of defaulting to "flat parallel" for everything.
- Phase 4 (Search) isn't a single pass — it's a bounded loop with three cheap safeguards so it doesn't quietly waste budget or silently give up:
- - **Cheap goal-check** — after each round, a Haiku pass tags every subquestion `met` / `partial` / `unmet` with a one-line reason. This is what the expensive Opus evaluation reads instead of re-deriving the gap from scratch, and it's what targets the next round's dispatch.
- - **No-progress circuit breaker** — two consecutive rounds that add nothing new to the source pool stop the loop immediately, regardless of remaining budget. The unresolved thread goes to Open Questions instead of burning tokens chasing a dead end.
- - **Least-to-most decomposition** — for layered questions ("X given Y"), subquestions are leveled `L1 → L2` instead of dispatched flat in parallel: L1 rounds run first, concrete facts they surface get carried forward, and L2 queries are launched already sharpened by that context. Independent subquestions still run flat.
- Scoring (5) doesn't stop at the usual Credibility/Recency/Bias rating — it also flags **input-level skepticism**: a source that measures its own product, self-reports a benchmark, or is directly disputed by another collected source gets a strict `caveat:` marker (`vendor` / `self-reported` / `disputed:sNN`) *before* the claim reaches `claims.csv`, not after synthesis has already built on it. A claim whose key number carries that marker is capped at `confidence: medium` (or `low` for an unresolved dispute) — the same rule shape as primary-first sourcing. Vendor benchmarks are the numbers that most often get quietly repeated as fact; catching them on the way in, not in the red team pass at the end, is the point.
- For medium/deep depth, the pipeline runs two more machine-checked passes most one-shot research skips entirely:
- - **Evidence filter (5.5)** — a CRAG-style relevance classifier runs on every (claim, source) pair *before* synthesis and keeps only the quotes that actually support that specific claim. Dumping every found source into synthesis measurably hurts quality (Search-o1 dropped 33%→24% doing exactly that); this is the fix, not a nice-to-have.
- - **Faithfulness verification (6.5)** — beyond checking that a cited link is alive, the skill checks that the source *entails* the claim it's attached to (RAGAS/ALCE-style claim⊨quote), and writes `SUPPORTED` / `PARTIAL` / `UNSUPPORTED` verdicts to `.verify/faithfulness.json`. Citation fabrication is common enough industry-wide — the Tow Center found a >60% error rate in AI-generated citations — that checking for it, not just for dead links, is a real differentiator.
- None of this is enforced by discipline alone: `scripts/validate_phases.py` reads a finished run's `mode:` and checks that every phase mandatory for that mode actually left its file artifact (`plan.md`, `claims.csv`, `evidence/`, `.verify/*.json`, the dated report, ...). A skipped phase fails the check instead of silently passing — the model can't just claim "done." As of finish-up, this check is a **blocker, not a suggestion**: the skill won't report a research as done on a red gate, symmetrically to how a report isn't "done" without its verification header. `sources.csv` itself is now built the same deterministic way — `scripts/build_sources_csv.py` generates it from `sources/NN.md` frontmatter (with a `--check` mode for CI) instead of being assembled by hand each run.
- A well-cited report that changes nothing is still a failure — the skill's answer to that is a **decision spine** running through the whole pipeline, not a bolt-on question at the end:
- - **Reframing (1) captures a decision spec**, not just a topic: the action you'll take, who reads the report and what they do next, and at least one falsifiable if-then fork ("if the research shows X, I do A"). No fork means the research changes nothing — the skill downgrades to shallow "curiosity mode" with an explicit label instead of quietly running a full pipeline for a dead-end doc.
- - **Sourcing (4-5) tracks provenance, not just source count.** A `root:` field on every source flags what it's actually retelling — ten articles quoting one press release are one voice, not ten, so triangulation now requires ≥2 distinct roots in addition to ≥3 sources and ≥2 types. A **snowball pass** also chains citations backward (to the primary study everyone's retelling) and forward (who's citing it since), catching sources no keyword query would surface.
- - **Synthesis (6) produces `memo.md`** — a one-page decision memo (recommendation, resolved forks, 3 key numbers with sources, the main risk, next actions) designed to be the thing that actually enters your decision process, with the full report as its evidence base. A conditional recommendation mapped to your forks is required; an unconditional "it depends" is not an acceptable ending.
- - **Red team (6) gets a fourth role — the Executor** — who plays your own decision-spec consumer and tries to actually act on the report using only its content, flagging every place a hedge phrase or a missing number blocks a real decision.
- - **Decision walkthrough (8) is mandatory at every depth, including shallow.** The report isn't discussed, it's executed: the skill walks you through each fork one at a time ("research showed X [s03][s11] — does fork A fire? your call?") and logs the outcome — decided, blocked on missing data (one bounded gap-search, then honest `blocked`), or deferred — to `application.md`. A global ledger tracks whether past research actually led anywhere, and Phase 1 reads it back on your next research to flag a pattern of dead-end runs.
- Want to compare models head-to-head? The [eval harness](eval/README.md) scores any run on 6 axes.
- ---
- ## What's inside
- <table>
- <tr>
- <td width="33%" valign="top">
- ### <!--gen:count:blocks-->106<!--/gen--> Report Blocks
- 10 categories: **FRAME** · **EXPLAIN** · **COMPARE** · **MAP** · **VALIDATE** · **ANALYZE** · **CLOSE** · **PEOPLE** · **NUMBERS** · **CONTEXT**
- Each block has its own template, anti-patterns, and composition rules.
- [Block library →](references/blocks/INDEX.md)
- </td>
- <td width="33%" valign="top">
- ### <!--gen:count:channels-->29<!--/gen--> Search Channels
- Named strategies with query patterns + paywall fallbacks:
- `web-general` · `academic` · `preprint-servers` · `code-github` · `forum-discussion` · `news-current` · `industry-reports` · `regulatory-legal` · `competitive-signals` · `data-statistical-gov` · `product-analytics` · `crypto-analytics` · `api-direct` · *and more*
- [Channels catalog →](references/channels.md)
- </td>
- <td width="33%" valign="top">
- ### <!--gen:count:stat_sources-->460<!--/gen-->+ Stat Sources
- 14 cross-industry + 19 industry categories. Each entry: URL · Type · Access · Quality · Limitations · Combine-with · Fallback.
- Categories: `gov_macro` · `companies_public` · `crypto` · `health` · `education` · `climate_env` · `science` · 19 industries
- [Sources catalog →](references/stat_sources/INDEX.md)
- **Plus a signal registry:** 1072 endpoints verified live and filed by what question they answer — parliamentary APIs, advisory feeds, climate and macro endpoints. Loaded one file at a time, outside the base context budget.
- [Registry →](references/registry/INDEX.md)
- </td>
- </tr>
- <tr>
- <td valign="top">
- ### 6 Report Genres
- | Genre | When |
- |:---|:---|
- | `qa` | Open meta-research |
- | `explainer` | "How does X work" |
- | `decision` | "X or Y" |
- | `landscape` | "Who's in this space" |
- | `validation` | "Is X true" |
- | `custom` | Hybrid, assembled per question |
- [Genres →](references/genres.md)
- </td>
- <td valign="top">
- ### <!--gen:count:api-->47<!--/gen-->+ API Endpoints
- Free no-auth APIs prioritized:
- `Semantic Scholar` · `OpenAlex` · `CrossRef` · `arXiv` · `DefiLlama` · `CoinGecko` · `HN Algolia` · `World Bank` · `SEC EDGAR` · `ClinicalTrials.gov` · `PubMed` · `GDELT` · `NSF Awards` · `NIH RePORTER` · `Grants.gov` · `EPO Linked Open Data`
- Patents and grants are first-class categories: `patents/` (USPTO ODP, EPO OPS, EPO LOD, WIPO) and `grants/` (NSF, NIH RePORTER, CORDIS, Grants.gov). Every entry carries a `Verified:` date — access terms rot, and a stale "free, no key" sends a run into a paywall.
- Auth via env vars only — skill never asks for keys inline.
- [API catalog →](references/api_sources/INDEX.md)
- </td>
- <td valign="top">
- ### Weekly Auto-Sync
- GitHub Actions cron validates all endpoints + discovers upstream additions:
- - HEAD-check <!--gen:count:api-->47<!--/gen-->+ APIs weekly
- - Scan public-apis & awesome-public-datasets
- - Auto-PR for dead endpoints
- - Reports committed to `reports/` branch
- [Workflow →](.github/workflows/catalog-sync.yml)
- </td>
- </tr>
- <tr>
- <td valign="top">
- ### Model Routing
- Per-phase model selection — quality where it multiplies, cheap where it parallelizes:
- - Reframing / plan / adversarial → **Opus**
- - Sub-agent fan-out (search) → **Haiku** (cheap × N)
- - Synthesis → **Sonnet/high**
- ~$2 instead of ~$8 on a deep run, *and* higher quality on critical phases. Override with `with all on opus` / `with cheap mode`.
- [Routing →](references/model_routing.md)
- </td>
- <td valign="top">
- ### Eval Harness
- Compare research quality across models. Same question, different configs, scored on 6 axes:
- - Deterministic (script): citation integrity, source diversity, cost
- - Semantic (LLM-judge): accuracy, coverage, adversarial honesty
- Weighted sum with a **citation floor** — hallucinated sources can't win on depth. Verdict = quality per dollar.
- [Eval →](eval/README.md)
- </td>
- <td valign="top">
- ### Citation Check
- `check_citations.py` resolves every source URL — dead `OPEN` links flagged as likely hallucinations; transport flaps marked `UNKNOWN`, not penalized.
- Ignores env proxies (`trust_env=False`). `--strict` for CI.
- Verification runs four layers: **liveness** (does the source exist), **faithfulness** (does it actually entail the claim it's cited for), **qualifier preservation** (does the report still say what the ledger said), and **construct provenance** (do the frameworks, taxonomies and named "laws" the report uses exist outside it). Verdicts land in `.verify/*.json`, one producer per file.
- The fourth layer exists because the first three all join on `claim_id` — and a fabricated *name* has none. That is the largest measured class of generation defect in research agents ([FINDER/DEFT](https://arxiv.org/abs/2512.01948): strategic content fabrication, 18.95% of errors), and it passes a citation check with every URL alive.
- Numbers get two independent passes: `check_number_provenance.py` on **origin** (who produced the figure; does one value circulate across supposedly independent roots) and `check_number_arithmetic.py` on **computation** — every `derived` figure in `numbers.csv` is recomputed from its own declared `formula` + `inputs`, and `share` groups must sum to 100. A percentage computed in prose is otherwise never re-checked by anything.
- [Check →](eval/check_citations.py)
- </td>
- <td valign="top">
- ### Phase-gate Validator
- `scripts/validate_phases.py` reads a finished run's `mode:` frontmatter and checks that every phase mandatory for that depth left its file artifact — `plan.md`, `state.md`, `claims.csv`, `numbers.csv`, `outline.md`, `evidence/`, `.verify/*.json`, the dated report.
- A skipped phase fails the check (`--strict` for CI) instead of the model just asserting "done" — it's a **finish-up blocker**, not advice. Machine insurance against the one failure mode a markdown methodology can't fix by discipline alone.
- It also checks three things a missing-file test cannot: that `state.md` is a **rebuilt** round window (Known/Gaps/Next, size-capped) and not an appended second transcript; that every `triangulated`/`contested` claim is mapped to a section in `outline.md` — the **synthesis gap**, where retrieval succeeded and the finding never reached the report; and that no `unsourced` construct sits in the memo or TL;DR.
- Its own inputs are machine-built too: `scripts/build_sources_csv.py` generates `sources.csv` deterministically from `sources/NN.md` frontmatter (`--check` for CI drift), and `eval/validate_structure.py` enforces the `caveat:` field as a strict enum (`-` / `vendor` / `self-reported` / `disputed:sNN`) instead of free text, so it stays greppable.
- [Validator →](scripts/README.md)
- </td>
- </tr>
- </table>
- ---
- ## Example folder
- Sample output for a typical `decision`-genre research:
- ```
- research/<topic-slug>/
- ├── plan.md # 17-section plan
- ├── state.md # Round window, rewritten each round
- ├── sources.csv # Index with C/R/B scoring
- ├── claims.csv # Claim ledger + triangulation status
- ├── numbers.csv # Every figure the report stands on
- ├── outline.md # section → block → claim_id map
- ├── sources/ # One file per source
- │ ├── 01_vendor-docs.md # Primary, total=14
- │ ├── 02_benchmark-paper.md # Academic, total=12
- │ ├── 03_industry-report.md # Industry, total=13
- │ ├── 04_forum-thread.md # Forum, total=9 (opposition)
- │ └── ... (19 more)
- ├── findings/
- │ ├── F1_<atomic-thesis>.md # confidence: high
- │ └── F2_<atomic-thesis>.md # confidence: medium
- ├── .verify/ # One producer per file, many consumers
- │ ├── authority.json # Phase 5.5 — who may assert this
- │ ├── citations.json # Layer 1 — liveness
- │ ├── faithfulness.json # Layer 2 — entailment
- │ ├── qualifiers.json # Layer 3 — scope drift
- │ └── constructs.json # Layer 4 — named-construct provenance
- ├── memo.md # One-page decision memo (always)
- ├── application.md # Decision walkthrough verdict (always)
- └── 2026-05-21_decision.md # Final report
- ```
- Final report structure (assembled from the blocks chosen in plan.md):
- ```markdown
- ## TL;DR
- - Claim A holds under condition X [confidence: high]
- - Claim B holds conditionally on threshold Y [confidence: medium]
- - Claim C is disputed by opposition sources [confidence: low]
- ## Mental model
- [How the underlying mechanism works...]
- ## Falsification criteria
- What would disprove H1, H2, H3...
- ## Verdict conditional
- Recommendation IF: <conditions met>
- Different recommendation OTHERWISE: <conditions broken>
- ## Counter-arguments (steel-man)
- CA1: "<the strongest opposing claim>" [source: s09]
- → Our answer: <conditions under which CA1 fails>
- CA2: ...
- ```
- Every claim is clickable to its source. A month later, you don't re-research — you read.
- ---
- ## Contribute
- The catalog is most valuable when **it grows**. Easy contributions:
- | Time | Type | Example |
- |:---:|:---|:---|
- | 15 min | Add a stat source | `Add SimilarWeb Pro to consumer_digital` |
- | 15 min | Improve a query pattern | `Better arxiv channel queries for biology` |
- | 30 min | New search channel | `Add patent-search with USPTO+EPO fallback` |
- | 1-2h | New industry category | `Add industries/aerospace.md` |
- | 2-4h | New report block | `Add decision-tree to compare.md` |
- | Half-day | LLM adapter | `Add codex/ folder with adapted protocols` |
- [Full contributing guide →](CONTRIBUTING.md)
- [](https://github.com/Socialpranker/deepdive/graphs/contributors)
- ---
- ## FAQ
- <details>
- <summary><b>How is this different from ChatGPT Deep Research / Perplexity?</b></summary>
- Those are **products** — closed UI, fixed flow, opaque source selection. This is **open methodology** — you control every step, the protocol is markdown you can fork, the source catalog is yours to extend.
- They also don't separate sources into files, don't do explicit triangulation, don't run adversarial passes, and don't produce reusable atomic theses. Nor do they filter evidence for relevance before synthesis (feeding a model everything you found measurably hurts quality — Search-o1 dropped from 33% to 24% accuracy doing that) or verify that a cited source actually *supports* the claim it's attached to, rather than just existing (faithfulness, not just liveness). Citation fabrication is common enough industry-wide — the Tow Center found a >60% error rate in AI-generated citations — that checking for it is a real differentiator, not a nice-to-have.
- Honesty about sources goes further than checking they exist: scoring flags a source that's measuring its own product, self-reporting a benchmark, or directly disputed by another collected source, and caps the confidence of any claim resting on that number — *before* it ever reaches the report. Vendor benchmarks getting quietly repeated as fact is a market-wide problem; catching it on input, not as an afterthought, is the same honesty principle as faithfulness applied one step earlier.
- </details>
- <details>
- <summary><b>Does it work without Claude Code CLI?</b></summary>
- Yes — on Claude Desktop with Skills enabled. Also works manually with any LLM by loading the markdown files into context (see ["Use with other LLMs"](#use-with-other-llms-codex-gemini-etc) below).
- </details>
- <details>
- <summary><b>What's a research output look like?</b></summary>
- See the [example folder](#example-folder) above. TL;DR: a folder with `plan.md` + `sources/NN.md` per source + `findings/FN.md` atomic theses + final `<date>_<genre>.md` report.
- Every claim in the final report links to a specific `sources/NN.md` file.
- </details>
- <details>
- <summary><b>Why so many files? Isn't this overkill?</b></summary>
- For a 5-minute "what's the latest X" question — yes. That's why `shallow` mode exists (5-7 sources, no sub-agents, ~15 min). The full machinery is for `medium` (1 hour) and `deep` (3 hours) when you need to actually use the output for a decision.
- The file-per-source structure is the key **reuse** mechanism. A single research often informs 3-5 future researches because you can cite individual `sources/NN.md` directly.
- </details>
- <details>
- <summary><b>Is this just prompt engineering?</b></summary>
- It's **structured methodology + curated catalog + reusable templates + automation**.
- - The <!--gen:count:phases-->13<!--/gen-->-phase workflow forces discipline
- - <!--gen:count:stat_sources-->460<!--/gen-->+ stat sources catalog is curated knowledge
- - <!--gen:count:blocks-->106<!--/gen--> reusable blocks compose any report shape
- - `scripts/validate_phases.py` machine-checks phase completeness, not just style
- - A decision spec + mandatory walkthrough (phase 8) ties every research to an actual action, not just a document
- - Weekly auto-validation keeps the catalog alive
- - 25+ upstream awesome-lists give infinite discovery layer
- Prompts are an implementation detail, not the value.
- </details>
- <details>
- <summary><b>What stops the report from being unactionable — "well-researched but useless"?</b></summary>
- The most common failure mode of open-ended research isn't bad sourcing, it's a report nobody knows what to do with. Three things guard against that specifically:
- 1. **Reframing won't accept a topic without a decision spec** — an action, a consumer, and at least one falsifiable if-then fork. No fork, no medium/deep run; it downgrades to a labeled shallow "curiosity" pass instead.
- 2. **Synthesis is banned from ending on "it depends"** without mapping the condition to one of your forks, and a fourth red-team role tries to actually *execute* the recommendation using only the report, flagging every hedge and missing number that blocks a real decision.
- 3. **Phase 8 (Decision walkthrough) is mandatory at every depth** — the skill walks you through your forks one at a time and logs whether each was decided, blocked on data, or deferred to `application.md`. A global ledger tracks unapplied research across sessions so the pattern doesn't repeat silently.
- </details>
- <details>
- <summary><b>Can I use this commercially?</b></summary>
- Yes — MIT licensed. Use it, modify it, integrate it into products. Attribution appreciated but not required.
- </details>
- ---
- ### Use with other LLMs (Codex, Gemini, etc.)
- The methodology is portable. ~70% of content is LLM-agnostic markdown templates.
- | Component | Claude-specific | Universal |
- |:---|:---:|:---:|
- | `SKILL.md` frontmatter | ✓ | — |
- | Sub-agent `Explore` type | ✓ | — |
- | <!--gen:count:phases-->13<!--/gen-->-phase workflow | — | ✓ |
- | <!--gen:count:blocks-->106<!--/gen--> report blocks | — | ✓ |
- | <!--gen:count:channels-->29<!--/gen--> search channels | — | ✓ |
- | <!--gen:count:stat_sources-->460<!--/gen-->+ stat sources | — | ✓ |
- **To adapt:**
- 1. Load `SKILL.md` + relevant `references/*.md` into the LLM's context
- 2. Replace sub-agent parallelism with separate chat sessions per subtopic
- 3. Manage source files (`sources/NN.md`) externally — LLM produces content
- 4. PRs welcome for `codex/`, `gemini/`, `local/` adapters
- ---
- <div align="center">
- ## Star History
- [](https://star-history.com/#Socialpranker/deepdive&Date)
- </div>
- ---
- <details>
- <summary><h2>На русском</h2></summary>
- **Deepdive** — скилл для [Claude Code](https://claude.com/claude-code), превращающий «загугли это» в дисциплинированный <!--gen:count:phases-->13<!--/gen-->-фазный процесс.
- ### Что внутри
- - **<!--gen:count:phases-->13<!--/gen--> фаз workflow**: <!--gen:phases:list:ru-->Reframing → Genre & block selection → Plan → Capability Discovery → Plan-review gate → Поиск → Claims-ledger + триангуляция → Evidence-фильтр → Сверка с вики → Синтез + multi-angle red team → Verify → Refresh targets → Decision walkthrough<!--/gen-->
- - **<!--gen:count:genres-->6<!--/gen--> жанров отчёта**: qa / explainer / decision / landscape / validation / custom
- - **<!--gen:count:blocks-->106<!--/gen--> блоков** в 10 категориях — переиспользуемые секции с шаблонами и анти-паттернами
- - **<!--gen:count:channels-->29<!--/gen--> каналов поиска** с paywall fallback протоколом (включая api-direct)
- - **<!--gen:count:stat_sources-->460<!--/gen-->+ статистических источников** в 14 cross-industry + 19 отраслевых категориях
- - **<!--gen:count:api-->47<!--/gen-->+ API endpoints** для programmatic доступа (free no-auth приоритетны)
- - **plan.md** с 17 секциями для прозрачности
- - **Plan-review gate** (фаза 3.7) — единственная human-in-the-loop точка перед дорогой Фазой 4: план (гипотезы, жанр, каналы) показывается и утверждается ДО поиска; жёсткость по режиму (deep — ждать «Ок», medium — soft, shallow — skip)
- - **Multi-angle red team** из враждебных ролей (Skeptic/Contrarian/Gap-hunter) с триажем severity (обязателен для medium/deep)
- - **Evidence-фильтр** (фаза 5.5) — CRAG-классификатор keep/drop по паре (тезис, источник) перед синтезом: наивная подача всего найденного снижает качество (Search-o1 33%→24%), в синтез идут только relevant-цитаты из `evidence/`
- - **Faithfulness-верификация** (фаза 6.5, второй слой) — помимо liveness (жива ли ссылка) проверяется entailment «источник ⊨ тезис» (RAGAS-декомпозиция + ALCE), вердикты SUPPORTED/PARTIAL/UNSUPPORTED в `.verify/faithfulness.json`
- - **No-progress circuit breaker** (фаза 4) — 2 раунда подряд без новой информации → стоп, нерешённое уходит в Open Questions вместо сжигания бюджета
- - **Дешёвый goal-check** (фаза 4) — Haiku между раундами помечает каждый подвопрос met/partial/unmet, направляя следующий раунд и удешевляя дорогую Opus-оценку
- - **Least-to-most декомпозиция** (фаза 4) — многошаговые подвопросы («X учитывая Y») раскладываются по уровням L1→L2 с накоплением контекста между ними вместо плоского параллельного запуска
- - **Phase-gate валидатор** (`scripts/validate_phases.py`) — машинная проверка, что каждая обязательная для режима фаза оставила артефакт; пропущенная фаза не проходит проверку
- - **Decision Spec** (фаза 1) — решение+срок / потребитель→шаг / ≥1 опровергаемая if-then вилка; без вилки ресёрч ничего не меняет — честный даунгрейд в shallow вместо мёртвого deep-отчёта
- - **Provenance-триангуляция** (фазы 4-5) — поле `root:` у каждого источника + snowball-пасс по цепочкам цитирований; триангуляция требует ≥2 разных первоисточников, не только ≥3 источников разного типа (десять пересказов одного пресс-релиза — один голос)
- - **`memo.md`** (фаза 6, всегда) — одностраничный decision-меморандум: рекомендация без «it depends», разрешённые вилки, ключевые числа со ссылками, next actions — то, что реально входит в твой процесс принятия решений
- - **Роль «Исполнитель»** в red team (фаза 6, medium/deep) — симулирует потребителя из Decision Spec и пытается исполнить решение по одному отчёту, ловя обтекаемый язык и пропущенные числа
- - **Decision walkthrough** (фаза 8, обязательна всегда) — отчёт не обсуждается, а исполняется: скилл проводит по вилкам решения одну за одной и записывает вердикт (принято / заблокировано данными / отложено) в `application.md`; глобальный ledger отслеживает, приводят ли твои ресёрчи к решениям вообще
- - **Weekly auto-validation** через GitHub Actions
- ### Установка
- ```bash
- git clone https://github.com/Socialpranker/deepdive.git ~/.claude/skills/deepdive
- ```
- Триггеры: «проведи ресёрч», «изучи тему», «копни глубоко», «deep dive»
- ### Вклад
- Каталог растёт через PRs. Самые ценные — новые источники в `stat_sources/` и `api_sources/`. См. [CONTRIBUTING.md](CONTRIBUTING.md).
- </details>
- ---
- <div align="center">
- ### Built by [Socialpranker](https://github.com/Socialpranker) · [MIT License](LICENSE) · [Roadmap](https://github.com/Socialpranker/deepdive/discussions)
- **If this skill saves you time, [give it a star](https://github.com/Socialpranker/deepdive)** — it's the only metric I check.
- </div>
runner/wiki.py · Bundled script · 665 lines
- """Cross-run knowledge layer: the wiki that sits between raw traces and the skill.
- The skill already had two of the three layers. Raw: sources/NN.md per run plus
- deepdive-state/observations.jsonl. Skills: references/channels.md and api_sources/,
- rewritten by promote_candidates.py behind a real gate (3 wins across 3 distinct runs,
- liveness, demotion). What was missing is the layer in between — so the only thing that
- compounded across runs was WHERE to search, never WHAT was found. A source fetched in
- two researches produced two unrelated files scored from zero; a claim triangulated in
- March was invisible to a run in August; two runs could conclude opposite things and
- nobody would notice.
- This module owns that layer. Three registries with real join keys:
- sources — key is the canonical URL (or DOI when present). Purely mechanical.
- entities — key is a folded name; aliases map variants onto one canonical page.
- claims — NO auto-key. Claims get an opaque id on first ingest and are linked to
- each other only through adjudicated pairs (see below).
- The claims asymmetry is deliberate. Deciding that two sentences from different runs are
- "the same claim" is a semantic judgement, and a wrong one manufactures a contradiction
- out of nothing — a broken measurement producing a false diagnosis, which is the most
- expensive failure this skill can have. So claim identity is never inferred here. This
- module only proposes CANDIDATE PAIRS through a mechanical prefilter and resolves the
- cases that need no judgement at all (a stale figure, a unit mismatch, a different
- scope). Whatever survives that goes to an adjudicator in phase 5.7, and its verdict is
- appended to pairs.jsonl as an audit trail. `unknown` is quarantine, never a conflict.
- Storage lives under ~/.claude/research/wiki/ — global on purpose. Research folders land
- per-project (research/, 06_Деск-ресёрч/, ~/deepdive/), and a per-project wiki would
- fragment exactly the thing that is supposed to accumulate. The precedent is already
- there: applications_ledger.csv and deepdive-state/ are both global.
- """
- from __future__ import annotations
- import csv
- import hashlib
- import io
- import json
- import logging
- import re
- import unicodedata
- from dataclasses import asdict, dataclass, field
- from datetime import date, datetime
- from pathlib import Path
- from urllib.parse import parse_qsl, urlencode, urlsplit
- log = logging.getLogger(__name__)
- DEFAULT_ROOT = Path.home() / ".claude"
- WIKI_DIRNAME = "research/wiki"
- REGISTRY_DIRNAME = ".wiki"
- # A figure that is simply newer is not a disagreement. Two claims that carry numbers and
- # whose as_of dates are further apart than this are treated as a version history, not a
- # conflict — the older one is marked superseded without ever reaching an adjudicator.
- # 180 days is the point past which most quantities in this skill's typical subject
- # matter (pricing, market size, model benchmarks) are expected to have moved anyway.
- SUPERSEDE_DAYS = 180
- # Query parameters that never change what a page says.
- TRACKING_PARAMS = frozenset(
- {
- "utm_source",
- "utm_medium",
- "utm_campaign",
- "utm_term",
- "utm_content",
- "utm_id",
- "ref",
- "referrer",
- "source",
- "fbclid",
- "gclid",
- "msclkid",
- "yclid",
- "_hsenc",
- "_hsmi",
- "mc_cid",
- "mc_eid",
- "igshid",
- "spm",
- "share",
- "s",
- "__twitter_impression",
- }
- )
- STOPWORDS = frozenset(
- """
- a an the and or but if then than that this these those of in on at to for from with
- without by as is are was were be been being it its their his her our your not no more
- most less least very can could may might will would should must do does did has have
- had about over under between across per vs versus
- и в во не что он на я с со как а то все она так его но да ты к у же вы за бы по только
- ее мне было вот от меня еще нет о из ему теперь когда даже ну вдруг ли если уже или ни
- быть был него до вас нибудь опять уж вам ведь там потом себя ничего ей может они тут
- где есть надо ней для мы тебя их чем была сам чтоб без будто чего раз тоже себе под
- будет ж тогда кто этот того потому этого какой совсем ним здесь этом один почти мой
- тем чтобы нее сейчас были куда зачем всех никогда можно при наконец два об другой хоть
- после над больше тот через эти нас про них какая много разве три эту моя впрочем хорошо
- свою этой перед иногда лучше чуть том нельзя такой им более всегда конечно всю между
- """.split()
- )
- # Scope qualifiers. Two claims that carry different ones are talking about different
- # things, which is the second-biggest source of manufactured contradictions after stale
- # numbers. Matched as whole words against the folded claim text.
- QUALIFIER_PATTERNS = (
- r"\b(?:в|for|in)\s+(?:сша|us|usa|eu|ес|ru|рф|россии|china|китае|uk|eea)\b",
- r"\b(?:enterprise|smb|self-hosted|cloud|on-prem|on-premise|free|paid|trial)\b",
- r"\b(?:median|медиан\w*|average|средн\w*|p50|p90|p95|p99)\b",
- r"\b(?:per\s+\w+|на\s+\w+|годов\w*|annual|monthly|месячн\w*)\b",
- r"\b(?:20\d{2})\b",
- )
- NUM_RE = re.compile(
- r"(?P<value>-?\d[\d\s .,]*\d|-?\d)\s*(?P<unit>%|процент\w*|млрд|млн|тыс|"
- r"[a-zA-Zа-яА-Я$€₽]{0,12})",
- )
- UNIT_ALIASES = {
- "процент": "%",
- "процента": "%",
- "процентов": "%",
- "проц": "%",
- "pct": "%",
- "usd": "$",
- "dollars": "$",
- "dollar": "$",
- "долл": "$",
- "доллар": "$",
- "eur": "€",
- "rub": "₽",
- "руб": "₽",
- "рублей": "₽",
- "bn": "b",
- "млрд": "b",
- "billion": "b",
- "billions": "b",
- "m": "m",
- "млн": "m",
- "million": "m",
- "millions": "m",
- "mn": "m",
- "k": "k",
- "тыс": "k",
- "thousand": "k",
- }
- WORD_RE = re.compile(r"[0-9a-zA-Zа-яёА-ЯЁ][0-9a-zA-Zа-яёА-ЯЁ_-]*", re.UNICODE)
- # Verdicts an adjudicator may return for a candidate pair. `unknown` is a quarantine
- # state: it is neither agreement nor conflict, it never reaches a report, and wiki_lint
- # reports it as debt once it goes stale.
- VERDICTS = ("same-claim-agree", "same-claim-conflict", "different-claim", "unknown")
- # Verdicts this module assigns mechanically, before any adjudicator sees the pair.
- AUTO_VERDICTS = ("superseded", "different-claim")
- # --------------------------------------------------------------------------- paths
- def wiki_dir(root: Path | None = None) -> Path:
- return (Path(root) if root is not None else DEFAULT_ROOT) / WIKI_DIRNAME
- def registry_dir(root: Path | None = None) -> Path:
- return wiki_dir(root) / REGISTRY_DIRNAME
- def ensure_wiki(root: Path | None = None) -> Path:
- d = wiki_dir(root)
- for sub in ("sources", "entities", "claims", REGISTRY_DIRNAME):
- (d / sub).mkdir(parents=True, exist_ok=True)
- return d
- # ---------------------------------------------------------------------- normalizing
- def canonical_url(raw: str) -> str:
- """Fold a URL to the identity of the document it points at.
- Scheme, `www.`, tracking parameters, fragments and a trailing slash never change
- what a page says, so none of them may split one source into two wiki pages. arXiv
- gets an explicit rule because its three renderings of one paper (/abs/, /pdf/,
- /html/) are the single most common way the same source enters two runs looking
- like two sources.
- """
- s = (raw or "").strip()
- if not s:
- return ""
- if "//" not in s:
- s = "https://" + s
- parts = urlsplit(s)
- host = parts.netloc.lower().removeprefix("www.").rstrip(".")
- path = parts.path.rstrip("/") or "/"
- if host.endswith("arxiv.org"):
- host = "arxiv.org"
- m = re.search(r"/(?:abs|pdf|html|format)/([^/]+?)(?:v\d+)?(?:\.pdf)?$", path)
- if m:
- path = f"/abs/{m.group(1)}"
- kept = [
- (k, v)
- for k, v in parse_qsl(parts.query, keep_blank_values=True)
- if k.lower() not in TRACKING_PARAMS
- ]
- query = urlencode(sorted(kept))
- return f"{host}{path}" + (f"?{query}" if query else "")
- def canonical_doi(raw: str) -> str:
- s = (raw or "").strip().lower()
- if not s:
- return ""
- s = re.sub(r"^(?:https?://)?(?:dx\.)?doi\.org/", "", s)
- s = s.removeprefix("doi:").strip()
- return s if s.startswith("10.") else ""
- def source_key(url: str = "", doi: str = "") -> str:
- """A DOI outranks a URL: it survives a publisher moving the article."""
- d = canonical_doi(doi)
- if d:
- return f"doi:{d}"
- u = canonical_url(url)
- return f"url:{u}" if u else ""
- def fold(text: str) -> str:
- """Casefold + strip diacritics + collapse whitespace. `ё` folds to `е`."""
- s = unicodedata.normalize("NFKD", (text or "").replace(" ", " "))
- s = "".join(c for c in s if not unicodedata.combining(c))
- return re.sub(r"\s+", " ", s.replace("ё", "е").replace("Ё", "Е")).strip().casefold()
- def entity_key(name: str) -> str:
- """Fold a name to its page key, dropping corporate-form noise and punctuation."""
- s = fold(name)
- s = re.sub(
- r"\b(?:inc|llc|ltd|corp|corporation|gmbh|s\.a\.|ооо|оао|пао|зао)\b", " ", s
- )
- s = re.sub(r"[^0-9a-zа-я]+", "-", s).strip("-")
- return re.sub(r"-{2,}", "-", s)
- def tokens(text: str) -> set[str]:
- return {
- t
- for t in (w.casefold() for w in WORD_RE.findall(fold(text)))
- if t not in STOPWORDS and len(t) > 2
- }
- def numbers(text: str) -> set[tuple[float, str]]:
- """Extract (value, folded unit) pairs. Unit-less numbers get the unit `""`."""
- out: set[tuple[float, str]] = set()
- for m in NUM_RE.finditer(text or ""):
- raw = m.group("value").replace(" ", "").replace(" ", "")
- # 1.234,56 (EU) vs 1,234.56 (US): whichever separator comes last is the decimal
- if "," in raw and "." in raw:
- raw = (
- raw.replace(",", "")
- if raw.rfind(".") > raw.rfind(",")
- else raw.replace(".", "").replace(",", ".")
- )
- else:
- raw = (
- raw.replace(",", ".")
- if raw.count(",") == 1 and len(raw.split(",")[-1]) <= 2
- else raw.replace(",", "")
- )
- try:
- value = float(raw)
- except ValueError:
- continue
- unit = m.group("unit").strip().casefold()
- out.add((value, UNIT_ALIASES.get(unit, unit)))
- return out
- def qualifiers(text: str) -> set[str]:
- folded = fold(text)
- return {
- m.group(0).strip()
- for pat in QUALIFIER_PATTERNS
- for m in re.finditer(pat, folded)
- }
- def parse_date(raw: str) -> date | None:
- s = (raw or "").strip()[:10]
- for fmt in ("%Y-%m-%d", "%Y-%m", "%Y"):
- try:
- return datetime.strptime(s[: len(fmt.replace("%Y", "2000"))], fmt).date()
- except ValueError:
- continue
- return None
- # ------------------------------------------------------------------------- records
- @dataclass
- class SourcePage:
- key: str
- url: str = ""
- title: str = ""
- type: str = ""
- channel: str = ""
- doi: str = ""
- credibility: str = ""
- runs: list[str] = field(default_factory=list)
- roots: list[str] = field(default_factory=list)
- caveats: list[str] = field(default_factory=list)
- first_seen: str = ""
- last_seen: str = ""
- @property
- def page(self) -> str:
- return f"sources/{slug_for(self.key)}.md"
- @dataclass
- class EntityPage:
- key: str
- name: str = ""
- aliases: list[str] = field(default_factory=list)
- runs: list[str] = field(default_factory=list)
- first_seen: str = ""
- last_seen: str = ""
- @property
- def page(self) -> str:
- return f"entities/{self.key}.md"
- @dataclass
- class ClaimPage:
- """One observation of a claim, owned by the run that produced it.
- Never merged with another run's claim by this module. Cross-run links exist only as
- adjudicated pairs, so a wrong adjudication is a bad edge in pairs.jsonl — reversible
- and auditable — instead of two records silently fused into one.
- """
- id: str
- run: str
- claim_id: str = ""
- text: str = ""
- status: str = ""
- confidence: str = ""
- as_of: str = ""
- entities: list[str] = field(default_factory=list)
- sources: list[str] = field(default_factory=list)
- roots: list[str] = field(default_factory=list)
- ingested: str = ""
- @property
- def page(self) -> str:
- return f"claims/{self.id}.md"
- def slug_for(key: str) -> str:
- """Filesystem-safe page name: readable prefix + hash, so two hosts never collide."""
- body = key.split(":", 1)[-1]
- head = re.sub(r"[^0-9a-zA-Z.-]+", "-", body)[:48].strip("-").lower() or "src"
- return f"{head}-{hashlib.sha1(key.encode('utf-8')).hexdigest()[:8]}"
- def claim_uid(run: str, claim_id: str) -> str:
- """Identity of a claim OBSERVATION: (run, claim_id). Deliberately excludes the text.
- Phase 5.7 pairs claims before synthesis; ingest writes them after. Between the two
- the wording gets edited — remediation, a restored qualifier, a softened overclaim.
- Hashing the text would move the id under those edits and leave every pair written in
- 5.7 pointing at nothing. (run, claim_id) survives them, so a pair adjudicated before
- synthesis still refers to the same row afterwards. Not semantic: two runs asserting
- the same thing still get two ids, which is the whole point.
- """
- h = hashlib.sha1(f"{run}\x00{claim_id}".encode("utf-8")).hexdigest()
- return f"c-{h[:10]}"
- # ------------------------------------------------------------------------ registries
- SOURCE_COLUMNS = [
- "key",
- "url",
- "title",
- "type",
- "channel",
- "doi",
- "credibility",
- "runs",
- "roots",
- "caveats",
- "first_seen",
- "last_seen",
- "page",
- ]
- ENTITY_COLUMNS = ["key", "name", "aliases", "runs", "first_seen", "last_seen", "page"]
- CLAIM_COLUMNS = [
- "id",
- "run",
- "claim_id",
- "text",
- "status",
- "confidence",
- "as_of",
- "entities",
- "sources",
- "roots",
- "ingested",
- "page",
- ]
- _LIST_FIELDS = {"runs", "roots", "caveats", "aliases", "entities", "sources"}
- def _to_row(rec, columns: list[str]) -> dict[str, str]:
- d = asdict(rec)
- d["page"] = rec.page
- return {
- c: ("; ".join(d.get(c) or []) if c in _LIST_FIELDS else str(d.get(c, "")))
- for c in columns
- }
- def _from_row(row: dict[str, str], cls, columns: list[str]):
- kw = {}
- for c in columns:
- if c == "page":
- continue
- raw = (row.get(c) or "").strip()
- kw[c] = (
- [p.strip() for p in raw.split(";") if p.strip()]
- if c in _LIST_FIELDS
- else raw
- )
- return cls(**kw)
- def _read_csv(path: Path, cls, columns: list[str]) -> dict[str, object]:
- if not path.is_file():
- return {}
- out = {}
- with path.open(encoding="utf-8", newline="") as fh:
- for row in csv.DictReader(fh):
- try:
- rec = _from_row(row, cls, columns)
- except TypeError as exc: # schema drift — skip the row, never the file
- log.warning("%s: unreadable row (%s) — skipped", path.name, exc)
- continue
- out[rec.id if hasattr(rec, "id") else rec.key] = rec
- return out
- def _write_csv(path: Path, recs, columns: list[str]) -> None:
- buf = io.StringIO()
- w = csv.DictWriter(buf, fieldnames=columns, lineterminator="\n")
- w.writeheader()
- for rec in recs:
- w.writerow(_to_row(rec, columns))
- path.parent.mkdir(parents=True, exist_ok=True)
- tmp = path.with_suffix(path.suffix + ".tmp")
- tmp.write_text(buf.getvalue(), encoding="utf-8")
- tmp.replace(path)
- def load_sources(root: Path | None = None) -> dict[str, SourcePage]:
- return _read_csv(registry_dir(root) / "sources.csv", SourcePage, SOURCE_COLUMNS)
- def load_entities(root: Path | None = None) -> dict[str, EntityPage]:
- return _read_csv(registry_dir(root) / "entities.csv", EntityPage, ENTITY_COLUMNS)
- def load_claims(root: Path | None = None) -> dict[str, ClaimPage]:
- return _read_csv(registry_dir(root) / "claims.csv", ClaimPage, CLAIM_COLUMNS)
- def save_sources(recs: dict[str, SourcePage], root: Path | None = None) -> None:
- _write_csv(
- registry_dir(root) / "sources.csv",
- sorted(recs.values(), key=lambda r: r.key),
- SOURCE_COLUMNS,
- )
- def save_entities(recs: dict[str, EntityPage], root: Path | None = None) -> None:
- _write_csv(
- registry_dir(root) / "entities.csv",
- sorted(recs.values(), key=lambda r: r.key),
- ENTITY_COLUMNS,
- )
- def save_claims(recs: dict[str, ClaimPage], root: Path | None = None) -> None:
- _write_csv(
- registry_dir(root) / "claims.csv",
- sorted(recs.values(), key=lambda r: r.id),
- CLAIM_COLUMNS,
- )
- def append_pairs(pairs: list[dict], root: Path | None = None) -> None:
- """Append-only: a re-adjudication is a new line, never an overwrite of the old one."""
- if not pairs:
- return
- d = registry_dir(root)
- d.mkdir(parents=True, exist_ok=True)
- with (d / "pairs.jsonl").open("a", encoding="utf-8") as fh:
- for p in pairs:
- fh.write(json.dumps(p, ensure_ascii=False, sort_keys=True) + "\n")
- def read_pairs(root: Path | None = None) -> list[dict]:
- p = registry_dir(root) / "pairs.jsonl"
- if not p.is_file():
- return []
- out = []
- for lineno, line in enumerate(p.read_text(encoding="utf-8").splitlines(), 1):
- line = line.strip()
- if not line:
- continue
- try:
- out.append(json.loads(line))
- except json.JSONDecodeError as exc:
- log.warning("pairs.jsonl:%d unreadable (%s) — skipped", lineno, exc)
- return out
- def latest_verdicts(root: Path | None = None) -> dict[tuple[str, str], dict]:
- """Last write wins per unordered pair — the audit trail keeps every earlier one."""
- out: dict[tuple[str, str], dict] = {}
- for p in read_pairs(root):
- a, b = p.get("left", ""), p.get("right", "")
- if a and b:
- out[tuple(sorted((a, b)))] = p
- return out
- # --------------------------------------------------------------------------- pairing
- def prefilter(new: ClaimPage, old: ClaimPage) -> str | None:
- """Mechanical gate. Returns why the pair is comparable, or None to never emit it.
- Two claims are only worth comparing when they are about the same thing. Shared
- entity is necessary but not sufficient — "Postgres is popular" and "Postgres 16
- ships logical replication" share an entity and compare to nothing. So a pair also
- needs a comparable slot: the same unit on both sides (they measure the same
- quantity), or enough shared content beyond the entity name.
- """
- if new.id == old.id or new.run == old.run:
- return None
- shared_entities = {e for e in new.entities if e} & {e for e in old.entities if e}
- if not shared_entities:
- return None
- units_new = {u for _, u in numbers(new.text) if u}
- units_old = {u for _, u in numbers(old.text) if u}
- if units_new & units_old:
- return f"entity+unit:{sorted(shared_entities)[0]}"
- t_new, t_old = tokens(new.text), tokens(old.text)
- entity_tokens = {
- tok for e in shared_entities for tok in tokens(e.replace("-", " "))
- }
- t_new, t_old = t_new - entity_tokens, t_old - entity_tokens
- if not t_new or not t_old:
- return None
- jaccard = len(t_new & t_old) / len(t_new | t_old)
- if jaccard >= 0.34:
- return f"entity+overlap:{jaccard:.2f}"
- return None
- def auto_verdict(new: ClaimPage, old: ClaimPage) -> tuple[str, str] | None:
- """Resolve the pairs that need no judgement. Returns (verdict, reason) or None.
- Every branch here exists because it is a way to manufacture a contradiction that
- isn't one. These are decided before an adjudicator is ever asked, so no amount of
- model confidence can turn a stale price or a different scope into a disagreement.
- """
- d_new, d_old = parse_date(new.as_of), parse_date(old.as_of)
- nums_new, nums_old = numbers(new.text), numbers(old.text)
- # A newer figure for the same quantity is a version, not a disagreement.
- if d_new and d_old and nums_new and nums_old:
- if abs((d_new - d_old).days) > SUPERSEDE_DAYS:
- older = old.id if d_new > d_old else new.id
- return (
- "superseded",
- f"as_of {d_old}/{d_new} > {SUPERSEDE_DAYS}d; older={older}",
- )
- # Different units measure different quantities. 40% and 40 млн do not disagree.
- units_new = {u for _, u in nums_new if u}
- units_old = {u for _, u in nums_old if u}
- if units_new and units_old and not (units_new & units_old):
- return (
- "different-claim",
- f"unit mismatch {sorted(units_old)}/{sorted(units_new)}",
- )
- # Different scope: US vs EU, median vs average, enterprise vs free, 2024 vs 2026.
- q_new, q_old = qualifiers(new.text), qualifiers(old.text)
- if (q_new or q_old) and not (q_new & q_old) and (q_new ^ q_old):
- return "different-claim", f"scope {sorted(q_old)}/{sorted(q_new)}"
- return None
- def candidate_pairs(
- new_claims: list[ClaimPage], wiki_claims: dict[str, ClaimPage]
- ) -> tuple[list[dict], list[dict]]:
- """Split comparable pairs into (auto-resolved, needs-adjudication).
- Only the second list costs a model call, and only it can ever produce a conflict.
- """
- auto: list[dict] = []
- todo: list[dict] = []
- for new in new_claims:
- for old in wiki_claims.values():
- reason = prefilter(new, old)
- if reason is None:
- continue
- entry = {
- "left": old.id,
- "right": new.id,
- "left_run": old.run,
- "right_run": new.run,
- "left_as_of": old.as_of,
- "right_as_of": new.as_of,
- "left_text": old.text,
- "right_text": new.text,
- "left_roots": old.roots,
- "right_roots": new.roots,
- "prefilter": reason,
- }
- resolved = auto_verdict(new, old)
- if resolved:
- entry["verdict"], entry["reason"], entry["by"] = *resolved, "auto"
- auto.append(entry)
- else:
- todo.append(entry)
- return auto, todo
- def open_conflicts(root: Path | None = None) -> list[dict]:
- """Adjudicated conflicts that nobody has resolved yet.
- A conflict is only real with both sides fully attributed: two claim ids, two runs,
- two as_of dates, two root sets. A half-populated conflict is a bug in whatever wrote
- it, not a finding — wiki_lint fails on those rather than letting them reach a report.
- """
- out = []
- for p in latest_verdicts(root).values():
- if p.get("verdict") != "same-claim-conflict" or p.get("resolved"):
- continue
- out.append(p)
- return sorted(out, key=lambda p: p.get("right_as_of", ""), reverse=True)
scripts/validate_phases.py · Bundled script · 965 lines
- #!/usr/bin/env python3
- """
- Phase-gate validator for a completed deepdive run directory.
- The methodology in references/ is executed only by the model's discipline inside a
- Claude Code session — nothing forces a phase to actually run. This script closes
- that gap: it checks that a finished run's output folder contains the file artifact
- of every phase that is MANDATORY for the run's depth mode. It catches "the model
- skipped a phase" — the failure mode the whole skill is most exposed to (H3).
- Depth mode drives which phases are mandatory. The source of truth for that is the
- `depth_gate` field in phases.yaml (loaded via phases_manifest — no second YAML
- parser). This script owns only the phase->artifact mapping, because not every phase
- emits a file and some emit an either/or or a set of files, which does not fit a
- scalar YAML field.
- It validates PHASE COMPLETENESS, not artifact format (that's eval/validate_structure.py)
- and not research quality (that's eval/score_run.py). The two structural validators
- are complementary: run both.
- Checks (errors fail --strict; warnings never do):
- - every phase mandatory for the mode emitted its artifact(s)
- - medium/deep: read-set size and sphere quotas, candidates.csv breadth
- - mode is known explicitly (--mode) or read from the report/plan frontmatter
- - the phase->artifact table still covers every file-emitting phase in phases.yaml
- (a self-check: adding such a phase without updating this table warns loudly)
- Usage:
- python scripts/validate_phases.py --research-dir research/<slug>
- python scripts/validate_phases.py --research-dir research/<slug> --mode deep
- python scripts/validate_phases.py --research-dir research/<slug> --strict # exit 1 on errors
- python scripts/validate_phases.py --research-dir research/<slug> --json
- """
- from __future__ import annotations
- import argparse
- import csv
- import json
- import re
- import sys
- from pathlib import Path
- sys.path.insert(0, str(Path(__file__).resolve().parent))
- import phases_manifest # noqa: E402
- from quiet import CAP_ERRORS, CAP_WARNINGS, capped # noqa: E402
- MODES = ("shallow", "medium", "deep")
- # depth_gate = the MINIMUM mode at which a phase is mandatory. A phase applies to a
- # run when the run's mode is at least its gate. Ranked so we can compare.
- GATE_RANK = {"shallow": 0, "medium": 1, "deep": 2}
- # Phase id -> what a completed run must contain when that phase is mandatory.
- # `any_of`: at least one of these paths must exist (relative to the run dir).
- # `all_of`: every one of these paths must exist.
- # `medium_of`: required from medium up — for artifacts of a phase that itself runs at
- # every depth but whose bookkeeping is not worth it on a shallow run.
- # A directory entry (trailing kind="dir") must be a non-empty directory.
- # Phases with no file output (reframing/genre/capability/plan-gate) are absent here
- # on purpose; the self-check below knows they are intentionally artifact-less.
- PHASE_ARTIFACTS: dict[str, dict[str, list[str]]] = {
- "3": {"all_of": ["plan.md"]},
- # state.md is the round workspace, rewritten (not appended) each round — the
- # orchestrator plans the next round from it instead of from the transcript.
- # candidates.csv is the wide, unread pool (title+snippet) that the read set is
- # picked from by sphere quotas — without it breadth is whatever search returned.
- "4": {"any_of": ["sources", "sources.csv"], "medium_of": ["state.md", "candidates.csv"]},
- "5": {"all_of": ["claims.csv"]},
- # 5.5 filters the INPUT to synthesis on two axes: relevance (evidence/) and
- # authority (.verify/authority.json). The authority verdicts are fail-closed —
- # an absent file means the axis never ran, not that everything qualified.
- "5.5": {"all_of": ["evidence", ".verify/authority.json"]},
- # 5.7 leaves the candidate pairs it built against the cross-run wiki. Fail-closed
- # for the same reason as authority.json: an absent file means the run never
- # confronted what earlier researches established, not that nothing conflicted.
- "5.7": {"all_of": [".verify/wiki_pairs.json"]},
- # Phase 6 emits the dated <YYYY-MM-DD>_<genre>.md report (matched by pattern)
- # AND the one-page decision memo the consumer's process actually ingests.
- # outline.md (section -> block -> claim_id) and numbers.csv (every figure the
- # report stands on) are the machine-checkable half of synthesis; both are
- # bookkeeping a shallow run is not asked to keep.
- # readout.md is the human version (page + full PDF): written after verify,
- # passed through humanizer-ru. Without it the run hands a reader the ledger.
- "6": {
- "report": [],
- "all_of": ["memo.md"],
- "medium_of": ["outline.md", "numbers.csv", "readout.md"],
- },
- # Layer 1 (liveness) / Layer 2 (faithfulness) / Layer 3 (qualifier preservation) /
- # Layer 4 (construct provenance). All four are required from medium up — 6.5's own
- # depth_gate is medium, so a shallow run never reaches this entry at all.
- "6.5": {
- "all_of": [
- ".verify/citations.json",
- ".verify/faithfulness.json",
- ".verify/qualifiers.json",
- ".verify/constructs.json",
- ]
- },
- "7": {"all_of": ["refresh_targets.md"]},
- # Phase 8 (decision walkthrough) must leave application.md with ANY status —
- # incl. `deferred`. The gate requires the file, not a made decision, so
- # unattended runs are never blocked on a human.
- "8": {"all_of": ["application.md"]},
- }
- # Phases that legitimately produce no file artifact — used only by the self-check so
- # it does not flag them as an un-mapped file-emitting phase.
- NO_FILE_PHASES = {"1", "2", "3.5", "3.7"}
- REPORT_RE = re.compile(r"^\d{4}-\d{2}-\d{2}_[a-z]+\.md$")
- MODE_RE = re.compile(r"^mode:\s*(\w+)", re.MULTILINE)
- DIR_ARTIFACTS = {"sources", "evidence"}
- class Report:
- def __init__(self) -> None:
- self.errors: list[str] = []
- self.warnings: list[str] = []
- def err(self, m: str) -> None:
- self.errors.append(m)
- def warn(self, m: str) -> None:
- self.warnings.append(m)
- def detect_mode(d: Path) -> str | None:
- """Read `mode:` from the report frontmatter, falling back to plan.md."""
- dated = sorted(p for p in d.glob("*.md") if REPORT_RE.match(p.name))
- for candidate in (*dated, d / "plan.md"):
- if candidate.is_file():
- m = MODE_RE.search(candidate.read_text(encoding="utf-8"))
- if m and m.group(1).lower() in MODES:
- return m.group(1).lower()
- return None
- def artifact_present(d: Path, rel: str) -> bool:
- p = d / rel
- if rel in DIR_ARTIFACTS:
- return p.is_dir() and any(p.iterdir())
- return p.is_file()
- def has_report(d: Path) -> bool:
- return any(REPORT_RE.match(p.name) for p in d.glob("*.md"))
- def check_phase(
- d: Path, phase_id: str, spec: dict[str, list[str]], mode: str, r: Report
- ) -> None:
- if "report" in spec:
- if not has_report(d):
- r.err(f"phase {phase_id}: no final report <YYYY-MM-DD>_<genre>.md")
- # no return: a phase may require the report AND fixed-name artifacts (6: memo.md)
- if "all_of" in spec:
- for rel in spec["all_of"]:
- if not artifact_present(d, rel):
- r.err(f"phase {phase_id}: missing required artifact '{rel}'")
- if "medium_of" in spec and GATE_RANK[mode] >= GATE_RANK["medium"]:
- for rel in spec["medium_of"]:
- if not artifact_present(d, rel):
- r.err(
- f"phase {phase_id}: missing required artifact '{rel}' "
- f"(required from medium up)"
- )
- if "any_of" in spec:
- if not any(artifact_present(d, rel) for rel in spec["any_of"]):
- joined = "' or '".join(spec["any_of"])
- r.err(f"phase {phase_id}: missing artifact (need '{joined}')")
- SOURCE_FILE_RE = re.compile(r"^(\d+)_.+\.md$")
- # Schema contract (references/source_scoring.md). Audit of 70 real runs found 26
- # header variants of numbers.csv alone: every reader guessed columns by substring and
- # the rules reading a missed column silently never fired. The header is now fixed at
- # write time: core columns in this exact order, then only the listed optional tail.
- CLAIMS_CORE = (
- "claim_id",
- "claim",
- "hypothesis",
- "sources",
- "source_types",
- "roots",
- "paths",
- "status",
- "confidence",
- "primary_source",
- "source_caveat",
- "dissent",
- "as_of",
- )
- CLAIMS_OPTIONAL = ("claim_class", "evidence_kind", "plain", "detail")
- NUMBERS_CORE = (
- "num_id",
- "value",
- "unit",
- "kind",
- "formula",
- "inputs",
- "group",
- "claim_id",
- "sources",
- "as_of",
- )
- NUMBERS_OPTIONAL = ("label",)
- # LLM-judge verdict files: one shape, so faithfulness is comparable across runs
- # (audit: 0.26–1.0 across runs was not comparable — each run invented its own form).
- VERDICT_FILES = {
- ".verify/faithfulness.json": {"SUPPORTED", "PARTIAL", "UNSUPPORTED"},
- ".verify/qualifiers.json": {
- "PRESERVED",
- "BROADENED",
- "SCOPE-DROPPED",
- "UNTRACEABLE",
- },
- }
- VERDICT_KEYS = ("claim_id", "verdict", "model")
- # Artifacts that only a medium/deep run produces — used to infer the mode when the
- # run never wrote a `mode:` frontmatter (2 of 12 real runs had one).
- MEDIUM_MARKERS = (
- "state.md",
- "outline.md",
- "numbers.csv",
- "evidence",
- "refresh_targets.md",
- ".verify/authority.json",
- ".verify/wiki_pairs.json",
- )
- # SKILL.md depth table: shallow is 5–7 sources, medium starts at 12.
- MEDIUM_SOURCE_COUNT = 10
- def check_source_perimeter(d: Path, r: Report) -> None:
- """Each fetch sub-agent owns a disjoint id range, enforced only by prompt text.
- One number claimed by two files means an agent wrote outside its range (or two
- collided): a source file was silently overwritten, so sources.csv no longer
- describes what is on disk."""
- src = d / "sources"
- if not src.is_dir():
- return
- by_id: dict[str, list[str]] = {}
- for p in sorted(src.glob("*.md")):
- m = SOURCE_FILE_RE.match(p.name)
- if not m:
- r.warn(f"sources/{p.name}: name is not NN_slug.md — not indexable by id")
- continue
- by_id.setdefault(m.group(1).lstrip("0") or "0", []).append(p.name)
- for num, names in sorted(by_id.items()):
- if len(names) > 1:
- r.err(
- f"source id {num} claimed by {len(names)} files ({', '.join(names)}) — "
- f"a sub-agent wrote outside its assigned range; that agent's results "
- f"are not trustworthy, re-run it with a narrowed task"
- )
- def infer_mode(d: Path) -> tuple[str, str]:
- """Best-effort depth from what the run left on disk. Never returns deep: on disk
- deep and medium are the same set of files, and guessing the stricter one would
- fail a run for a phase (3.5) that leaves no artifact."""
- present = [m for m in MEDIUM_MARKERS if artifact_present(d, m)]
- if present:
- return "medium", f"medium-only artifacts present: {', '.join(present)}"
- src = d / "sources"
- n = len(list(src.glob("*.md"))) if src.is_dir() else 0
- if n >= MEDIUM_SOURCE_COUNT:
- return "medium", f"{n} sources (shallow is 5–7)"
- return "shallow", f"no medium artifacts, {n} sources"
- def resolve_mode(d: Path, explicit: str | None, r: Report) -> str:
- """Explicit flag > frontmatter > inference. A run with no `mode:` is validated
- anyway — refusing would let the least disciplined runs skip the gate entirely."""
- if explicit:
- return explicit
- found = detect_mode(d)
- if found:
- return found
- mode, why = infer_mode(d)
- r.warn(
- f"no 'mode:' frontmatter in report or plan.md — inferred '{mode}' ({why}); "
- f"add `mode: {mode}` to plan.md or pass --mode"
- )
- return mode
- def header_violation(
- header: list[str], core: tuple[str, ...], optional: tuple[str, ...]
- ) -> str | None:
- """-> why `header` is not canonical, or None."""
- head = [c.strip() for c in header]
- if tuple(head[: len(core)]) != core:
- missing = [c for c in core if c not in head]
- if missing:
- return f"missing column(s) {', '.join(missing)}"
- return f"core columns out of order (expected {','.join(core)})"
- unknown = [c for c in head[len(core) :] if c not in optional]
- if unknown:
- return (
- f"unknown column(s) {', '.join(unknown)} after the core "
- f"(allowed: {', '.join(optional)})"
- )
- return None
- def check_schema_contract(d: Path, r: Report) -> None:
- """A non-canonical ledger is not a formatting nit: the rule that reads a column
- it cannot find silently never fires — the 'green check, no behavior' failure."""
- for name, core, optional in (
- ("claims.csv", CLAIMS_CORE, CLAIMS_OPTIONAL),
- ("numbers.csv", NUMBERS_CORE, NUMBERS_OPTIONAL),
- ):
- path = d / name
- if not path.is_file():
- continue
- header = next(csv.reader(path.read_text(encoding="utf-8").splitlines()), None)
- if not header:
- r.err(f"{name} is empty")
- continue
- why = header_violation(header, core, optional)
- if why:
- r.err(
- f"{name}: {why} — the header is fixed by references/source_scoring.md; "
- f"rename/reorder the columns, do not invent new ones"
- )
- for rel, verdicts in VERDICT_FILES.items():
- path = d / rel
- if not path.is_file():
- continue # presence is the phase-gate's business (6.5)
- try:
- data = json.loads(path.read_text(encoding="utf-8"))
- except json.JSONDecodeError as exc:
- r.err(f"{rel} is not valid JSON ({exc.msg})")
- continue
- results = data.get("results") if isinstance(data, dict) else None
- if not isinstance(results, list) or not results:
- r.err(f"{rel}: no non-empty `results` list — the layer left no verdicts")
- continue
- bad = []
- for i, item in enumerate(results):
- if not isinstance(item, dict):
- bad.append(f"#{i} not an object")
- continue
- empty = [k for k in VERDICT_KEYS if not str(item.get(k) or "").strip()]
- if empty:
- bad.append(f"#{i} lacks {'/'.join(empty)}")
- elif str(item["verdict"]).strip() not in verdicts:
- bad.append(f"#{i} verdict '{item['verdict']}'")
- if bad:
- r.err(
- f"{rel}: {len(bad)} result(s) off-contract ({'; '.join(bad[:5])}) — "
- f"each needs {', '.join(VERDICT_KEYS)} with verdict in {sorted(verdicts)}"
- )
- # --- source quality ------------------------------------------------------------
- # A source that carries fewer than 2 verbatim quotes, or that was written off a
- # paywall/consent/abstract stub, is not evidence: it is a URL with a title. It
- # happens when WebFetch returned a thin page and the fetch ladder
- # (scripts/fetch_source.py) was never run. Such a source must not be a
- # triangulation leg — otherwise `triangulated` is backed by nothing.
- MIN_QUOTES = 2
- THIN_ACCESS_PREFIXES = ("partial", "closed", "paywalled", "abstract-only")
- THIN_SHARE_WARN = 0.25
- ACCESS_RE = re.compile(r"^access:\s*(.+?)\s*$", re.MULTILINE)
- SOURCE_ID_COLUMNS = ("sources", "source_ids", "source", "source_id")
- SOURCE_ID_SPLIT_RE = re.compile(r"[;,|\s\[\]()]+")
- # Real runs use both `NN_slug.md` and `sNN_slug.md`; the id is the same either way.
- SOURCE_ID_FILE_RE = re.compile(r"^s?(\d+)_.+\.md$")
- def _norm_source_id(token: str) -> str | None:
- """`s01`, `[s01]`, `01`, `1` -> canonical `1`. Anything else -> None."""
- tok = token.strip().strip("[]()").lower()
- if tok.startswith("s"):
- tok = tok[1:]
- if not tok.isdigit():
- return None
- return tok.lstrip("0") or "0"
- def _split_frontmatter(text: str) -> tuple[str, str]:
- if text.startswith("---"):
- parts = text.split("---", 2)
- if len(parts) == 3:
- return parts[1], parts[2]
- return "", text
- def source_is_thin(text: str) -> bool:
- front, body = _split_frontmatter(text)
- m = ACCESS_RE.search(front)
- if m:
- access = m.group(1).strip().strip("\"'").lower()
- if access.startswith(THIN_ACCESS_PREFIXES):
- return True
- quotes = sum(1 for line in body.splitlines() if line.lstrip().startswith(">"))
- return quotes < MIN_QUOTES
- def collect_thin_sources(d: Path) -> tuple[dict[str, str], set[str]]:
- """-> ({normalized id: label}, {normalized id of every source file})."""
- src = d / "sources"
- thin: dict[str, str] = {}
- seen: set[str] = set()
- if not src.is_dir():
- return thin, seen
- for p in sorted(src.glob("*.md")):
- m = SOURCE_ID_FILE_RE.match(p.name)
- if not m:
- continue
- num = m.group(1)
- key = num.lstrip("0") or "0"
- seen.add(key)
- if source_is_thin(p.read_text(encoding="utf-8")):
- thin[key] = f"s{num}"
- return thin, seen
- # Breadth of the read set (references/phases/p4_search.md «Квоты»). Benchmark
- # 2026-10-28: 6 of 6 runs had 16–19 sources, 0 Academic, 3–4 types — the search
- # stopped at whatever the first queries returned.
- MIN_READ = {"medium": 25, "deep": 35}
- MIN_CANDIDATES = {"medium": 60, "deep": 100}
- TYPE_QUOTAS = {
- "academic": ({"academic"}, 2),
- "primary": ({"primary"}, 3),
- "media": ({"industry-media", "general-media"}, 2),
- "expert": ({"expert-blog"}, 2),
- "practice": ({"forum"}, 2),
- }
- MIN_CONTRADICTING = 3
- CANDIDATE_COLUMNS = ("url", "title", "snippet", "channel", "sphere", "stance", "vendor",
- "round", "target", "selected")
- ACADEMIC_CHANNELS = {"openalex", "semantic_scholar", "arxiv", "crossref"}
- MIN_TARGETED_ROUND = 5
- WAIVER_RE = re.compile(r"^quota_waiver:\s*(\w+)\s*[—-]\s*(\S.*)$", re.MULTILINE)
- # Depth and pool size are what the skill is for; N2 (2026-10-28) waived them in
- # plan.md before searching, citing the run budget.
- NON_WAIVABLE = {"count", "candidates", "round2"}
- BUDGET_REASON_RE = re.compile(r"бюджет|budget|\$\s?\d|деньг|дорог", re.IGNORECASE)
- FIELD_RE = re.compile(r"^(type|caveat):\s*(.+?)\s*$", re.MULTILINE)
- VERDICT_TABLE_RE = re.compile(r"^\|[^\n]*гипотез[^\n]*\|[^\n]*вердикт", re.IGNORECASE | re.MULTILINE)
- def check_verdict_table(d: Path, r: Report) -> None:
- """Benchmark 2026-10-28: explicit hypothesis verdicts in 2 of 6 reports."""
- dated = sorted(p for p in d.glob("*.md") if REPORT_RE.match(p.name))
- if dated and not VERDICT_TABLE_RE.search(dated[-1].read_text(encoding="utf-8")):
- r.err(f"{dated[-1].name}: no `| Гипотеза | Вердикт | Опора |` table — every "
- f"plan hypothesis needs an explicit verdict (phases/p6_synthesis.md)")
- # An unconditional pick: "по умолчанию", or an imperative with no "если" in the sentence.
- DEFAULT_RE = re.compile(r"(?<![«\w])(по умолчанию|берите|выбирайте|начинать стоит|начните с|рекомендуем)(?![»\w])",
- re.IGNORECASE)
- SENTENCE_SPLIT_RE = re.compile(r"(?<=[.!?])\s+|\n")
- def _default_sentences(text: str) -> list[str]:
- """Unconditional picks in «Ответ» / «Рекомендация» with no [sNN] behind them."""
- sections = re.split(r"^## ", text, flags=re.MULTILINE)
- scope = [sec for sec in sections if sec.lower().startswith("ответ")]
- scope += re.findall(r"^\**Рекомендаци[яи][^\n]*", text, re.MULTILINE)
- hits = []
- for chunk in scope:
- for sent in SENTENCE_SPLIT_RE.split(chunk):
- if DEFAULT_RE.search(sent) and not re.search(r"\bне\b[^.]*по умолчанию", sent, re.IGNORECASE) \
- and not re.search(r"\bесли\b", sent, re.IGNORECASE) \
- and not re.search(r"\[s\d{2,3}\]", sent):
- hits.append(sent.strip())
- return hits
- def check_unmeasured_default(d: Path, r: Report) -> None:
- """Benchmark 2026-10-28: 7 of 8 runs said 'Fivetran по умолчанию' with no price found."""
- dated = sorted(p for p in d.glob("*.md") if REPORT_RE.match(p.name))
- targets = [p for p in [d / "readout.md", *dated[-1:]] if p.is_file()]
- for p in targets:
- for sent in _default_sentences(p.read_text(encoding="utf-8")):
- r.err(f"{p.name}: default recommendation without a measured source — «{sent[:90]}». "
- f"Give conditional branches + the cheapest way to measure (phases/p6_synthesis.md, Антибиас)")
- PRE_REMEDIATION = "findings/draft_pre_remediation.md"
- HIGH_ID_RE = re.compile(r"^\W*(?:HIGH|H)-(\d+)\b", re.MULTILINE)
- REVIEW_HEADING_RE = re.compile(r"adversarial review|ремедиац|red team", re.IGNORECASE)
- # N4/N6 (2026-10-28): fixes were appended after the body, TL;DR kept the refuted claims.
- APPENDIX_RE = re.compile(r"приоритет над текстом|заменяется на|поправки ниже|см\. ремедиац", re.IGNORECASE)
- # "- HIGH-1 → §Ответ (TL;DR), §Ветки: что изменено" — a target ends at ":", "§", "|" or ", §".
- FIX_TARGET_RE = re.compile(r"§\s*([^§|:\n]+?)\s*(?=,\s*§|[§|:\n]|$)")
- def _sections(text: str) -> dict[str, str]:
- parts = re.split(r"^(#{2,3} .+)$", text, flags=re.MULTILINE)
- return {h.lstrip("# ").strip().casefold(): body for h, body in zip(parts[1::2], parts[2::2])}
- def _same(a: str | None, b: str | None) -> bool:
- return a is not None and b is not None and a.split() == b.split()
- def _find_section(sections: dict[str, str], target: str) -> str | None:
- t = target.strip(" *`").casefold()
- return next((h for h in sections if h == t or h.startswith(t)), None)
- def check_remediation_in_body(d: Path, mode: str, r: Report) -> None:
- """Each HIGH from the red-team log is fixed in the body section it names, not in an appendix."""
- logs = sorted((d / "findings").glob("redteam_*.md")) if (d / "findings").is_dir() else []
- dated = sorted(p for p in d.glob("*.md") if REPORT_RE.match(p.name))
- if GATE_RANK[mode] < GATE_RANK["medium"] or not dated:
- return
- if not logs:
- # N6 (2026-10-28) kept no log, so nothing below could check its remediation.
- r.err("findings/redteam_<date>.md missing — red team triage log (adversarial_pass.md)")
- return
- highs = sorted(set(HIGH_ID_RE.findall(logs[-1].read_text(encoding="utf-8"))), key=int)
- if not highs:
- return
- report = dated[-1]
- text = report.read_text(encoding="utf-8")
- for m in APPENDIX_RE.finditer(text):
- r.err(f"{report.name}: «{m.group(0)}» — remediation overrides the body from an appendix; "
- f"edit the section itself (phases/p6_synthesis.md, шаг 6)")
- snap = d / PRE_REMEDIATION
- if not snap.is_file():
- r.err(f"{PRE_REMEDIATION} missing — copy the draft there before `remediate`, "
- f"otherwise body fixes cannot be told from an appended postscript")
- return
- now, before = _sections(text), _sections(snap.read_text(encoding="utf-8"))
- review = next((body for h, body in now.items() if REVIEW_HEADING_RE.search(h)), "")
- changed_any = False
- for hid in highs:
- line = next((ln for ln in review.splitlines() if re.search(rf"\b(?:HIGH|H)-{hid}\b", ln)), None)
- if line is None:
- r.err(f"HIGH-{hid} from {logs[-1].name} not listed in `## Adversarial review` "
- f"with `→ §<section>` or `→ Open Questions`")
- continue
- if "open questions" in line.casefold():
- continue
- targets = FIX_TARGET_RE.findall(line)
- if not targets:
- r.err(f"HIGH-{hid}: no `§<section>` it was fixed in — «{line.strip()[:90]}»")
- for t in targets:
- h = _find_section(now, t)
- if h is None:
- r.err(f"HIGH-{hid}: section «§{t}» not found in {report.name}")
- elif _same(now[h], before.get(h)):
- r.err(f"HIGH-{hid}: «§{t}» unchanged vs {PRE_REMEDIATION} — the fix is not in the body")
- else:
- changed_any = True
- head = next((h for h in now if h.startswith(("ответ", "tl;dr"))), None)
- if changed_any and head and _same(now[head], before.get(head)):
- r.warn(f"{report.name}: body fixed after red team but «{head}» unchanged — "
- f"check it no longer states the refuted claims")
- def check_source_breadth(d: Path, mode: str, r: Report) -> None:
- if GATE_RANK[mode] < GATE_RANK["medium"]:
- return
- plan = d / "plan.md"
- waived: set[str] = set()
- for key, reason in WAIVER_RE.findall(plan.read_text(encoding="utf-8")) if plan.is_file() else []:
- if key in NON_WAIVABLE:
- r.err(f"quota_waiver: {key} — not waivable: depth and candidate pool are the point of medium/deep")
- elif BUDGET_REASON_RE.search(reason):
- r.err(f"quota_waiver: {key} — budget is not a reason; waive only what the search "
- f"did not find (cite candidates.csv)")
- else:
- waived.add(key)
- def gate(key: str, msg: str) -> None:
- hint = "" if key in waived or key in NON_WAIVABLE else f" — or add `quota_waiver: {key} — <why>` to plan.md"
- (r.warn if key in waived else r.err)(msg + hint)
- files = sorted((d / "sources").glob("*.md")) if (d / "sources").is_dir() else []
- if len(files) < MIN_READ[mode]:
- gate("count", f"{len(files)} sources read, {mode} needs ≥{MIN_READ[mode]}")
- types: list[str] = []
- vendor = contradicting = 0
- for p in files:
- front, _ = _split_frontmatter(p.read_text(encoding="utf-8"))
- fields = dict(FIELD_RE.findall(front))
- types.append(fields.get("type", "").strip("\"'").lower())
- vendor += fields.get("caveat", "").strip("\"'").lower() == "vendor"
- contradicting += bool(re.search(r"^\s+H\d+:\s*contradicts\b", front, re.MULTILINE))
- for key, (kinds, need) in TYPE_QUOTAS.items():
- have = sum(t in kinds for t in types)
- if have < need:
- gate(key, f"{have} {key} sources (type {'/'.join(sorted(kinds))}), quota ≥{need}")
- if contradicting < MIN_CONTRADICTING:
- gate("against", f"{contradicting} sources contradict a hypothesis "
- f"(hypothesis_evidence: contradicts), quota ≥{MIN_CONTRADICTING}")
- if files and vendor * 2 > len(files):
- gate("vendor", f"{vendor} of {len(files)} sources are vendor material — "
- f"at most half; the rest must be vendor-independent")
- no_card = [p.name for p in files if "\n## Разбор" not in p.read_text(encoding="utf-8")]
- if no_card:
- r.warn(f"{len(no_card)} of {len(files)} sources have no `## Разбор` card "
- f"(claim type, method, scope, modality) — e.g. {', '.join(no_card[:3])}")
- check_verdict_table(d, r)
- check_unmeasured_default(d, r)
- cand = d / "candidates.csv"
- if not cand.is_file():
- return # absence is reported by the phase-4 artifact check
- with cand.open(encoding="utf-8") as fh:
- reader = csv.DictReader(fh)
- missing = [c for c in CANDIDATE_COLUMNS if c not in (reader.fieldnames or [])]
- rows = list(reader)
- if missing:
- r.err(f"candidates.csv: missing columns {', '.join(missing)}")
- return
- if len(rows) < MIN_CANDIDATES[mode]:
- gate("candidates", f"candidates.csv has {len(rows)} rows, {mode} needs ≥{MIN_CANDIDATES[mode]}")
- if not any(row["channel"].strip().lower() in ACADEMIC_CHANNELS for row in rows):
- gate("academic", "no candidate came from an academic API "
- f"({', '.join(sorted(ACADEMIC_CHANNELS))})")
- targeted = [row for row in rows if row["round"].strip() == "2" and row["target"].strip()]
- if len(targeted) < MIN_TARGETED_ROUND:
- gate("round2", f"{len(targeted)} round-2 candidates with a `target` — the second "
- f"round must aim at the decision variable, ≥{MIN_TARGETED_ROUND}")
- selected = sum(row["selected"].strip().upper() == "Y" for row in rows)
- if files and selected and selected != len(files):
- r.warn(f"candidates.csv marks {selected} selected, sources/ has {len(files)}")
- def check_source_quality(d: Path, mode: str, r: Report) -> None:
- """Thin sources silently backing `triangulated` claims.
- Measured on 12 real runs: 27% of source files carry <2 quotes and every one
- of them reports `fetch_tier: webfetch` — the ladder was never used. A claim
- standing only on such sources is not triangulated, it is one bad fetch.
- """
- thin, seen = collect_thin_sources(d)
- if not seen:
- return
- for row in load_claim_rows(d):
- if (row.get("status") or "").strip().lower() != "triangulated":
- continue
- cid = (row.get("claim_id") or "").strip() or "?"
- cell = next(
- (row[c] for c in SOURCE_ID_COLUMNS if (row.get(c) or "").strip()), ""
- )
- cited = []
- for tok in SOURCE_ID_SPLIT_RE.split(cell):
- key = _norm_source_id(tok)
- if key is not None and key in seen and key not in cited:
- cited.append(key)
- if not cited:
- continue
- bad = [thin[k] for k in cited if k in thin]
- if not bad:
- continue
- listed = ", ".join(bad)
- if len(bad) == len(cited):
- msg = (
- f"claim {cid} is triangulated on thin sources only ({listed}) — "
- f"thin = <{MIN_QUOTES} quotes or access PARTIAL/closed; refetch via "
- f"scripts/fetch_source.py or downgrade"
- )
- (r.err if GATE_RANK[mode] >= GATE_RANK["medium"] else r.warn)(msg)
- else:
- r.warn(
- f"claim {cid} is triangulated partly on thin sources ({listed}) — "
- f"refetch them via scripts/fetch_source.py or drop them as legs"
- )
- share = len(thin) / len(seen)
- if share > THIN_SHARE_WARN:
- r.warn(
- f"{len(thin)} of {len(seen)} sources are thin ({share * 100:.0f}%) — "
- f"the fetch ladder was not used; see subagents_v2.md §fetch"
- )
- OUTLINE_ROW_RE = re.compile(r"^\|(?P<cells>.+)\|\s*$")
- CLAIM_CELL_SPLIT_RE = re.compile(r"[;,\s]+")
- # Claims this strong are the run's product: reaching one and leaving it out of the
- # report is work thrown away silently, not an editorial choice.
- CARRYING_STATUSES = {"triangulated", "contested"}
- STATE_SECTIONS = ("## Known", "## Gaps", "## Next")
- STATE_SOFT_LIMIT = 6 * 1024
- STATE_HARD_LIMIT = 12 * 1024
- def parse_outline(text: str) -> dict[str, list[str]]:
- """Read the `| section | block | claims |` table -> {section: [claim_id, ...]}.
- The claims cell is tokenized rather than pattern-matched: claim ids are whatever
- claims.csv calls them, and a validator that hard-codes `CL\\d+` silently reads
- zero claims from a run that numbered them differently.
- """
- out: dict[str, list[str]] = {}
- for line in text.splitlines():
- m = OUTLINE_ROW_RE.match(line.strip())
- if not m:
- continue
- cells = [c.strip() for c in m.group("cells").split("|")]
- if len(cells) < 3:
- continue
- section = cells[0]
- if not section or section.lower() == "section" or set(section) <= set("-: "):
- continue # header or separator row
- tokens = [
- t for t in CLAIM_CELL_SPLIT_RE.split(cells[2]) if t and t not in {"-", "—"}
- ]
- out[section] = tokens
- return out
- def load_claim_rows(d: Path) -> list[dict[str, str]]:
- ledger = d / "claims.csv"
- if not ledger.is_file():
- return []
- return list(csv.DictReader(ledger.read_text(encoding="utf-8").splitlines()))
- def check_outline_coverage(d: Path, mode: str, r: Report) -> None:
- """The outline is the join between what was found and what got written.
- Three failures live here: a section with no claims (prose with no ledger behind
- it), a claim_id that resolves to nothing, and — the expensive one — a carrying
- claim that never reached any section. The last is the synthesis gap: retrieval
- worked, organization dropped it.
- """
- outline = d / "outline.md"
- if not outline.is_file():
- return # presence is the phase-gate's business, above
- sections = parse_outline(outline.read_text(encoding="utf-8"))
- if not sections:
- r.err(
- "outline.md has no `| section | block | claims |` table rows — "
- "the section->claim map is what makes per-section synthesis checkable"
- )
- return
- rows = load_claim_rows(d)
- known = {(row.get("claim_id") or "").strip() for row in rows} - {""}
- mapped: set[str] = set()
- for section, claims in sections.items():
- if not claims:
- r.err(f"outline.md: section '{section}' maps to no claim_id")
- continue
- for cid in claims:
- if known and cid not in known:
- r.err(
- f"outline.md: section '{section}' cites {cid}, absent from claims.csv"
- )
- mapped.add(cid)
- if not rows:
- return
- orphans = []
- for row in rows:
- status = (row.get("status") or "").strip().lower()
- cid = (row.get("claim_id") or "").strip()
- if status in CARRYING_STATUSES and cid and cid not in mapped:
- orphans.append(f"{cid} ({status})")
- if orphans:
- message = (
- f"claims reached but never placed in the report: {', '.join(orphans)} — "
- f"triangulated/contested work dropped at synthesis, not at search "
- f"(map it to a section or say in Open Questions why it is out)"
- )
- # On deep this is a blocker; on medium a loud warning — a medium run may
- # legitimately carry more than its outline holds.
- (r.err if mode == "deep" else r.warn)(message)
- def check_constructs(d: Path, r: Report) -> None:
- """Layer 4: an unsourced named construct in the memo/TL;DR blocks finish."""
- path = d / ".verify/constructs.json"
- if not path.is_file():
- return # presence is enforced by PHASE_ARTIFACTS for 6.5
- try:
- data = json.loads(path.read_text(encoding="utf-8"))
- except json.JSONDecodeError as exc:
- r.err(
- f".verify/constructs.json is not valid JSON ({exc.msg}) — Layer 4 cannot be read"
- )
- return
- results = data.get("results")
- if not isinstance(results, list):
- r.err(
- ".verify/constructs.json has no `results` list — Layer 4 produced nothing"
- )
- return
- for item in results:
- if not isinstance(item, dict):
- continue
- status = str(item.get("status", "")).strip().lower()
- name = item.get("name", "?")
- locations = [str(loc) for loc in item.get("locations", [])]
- if status == "unsourced":
- carried = [loc for loc in locations if loc in {"memo.md", "F1", "F9"}]
- if carried:
- r.err(
- f"construct '{name}' is unsourced and appears in {', '.join(carried)} — "
- f"source it, mark it as ours, or remove it (Layer 4 fail-closed)"
- )
- else:
- r.warn(
- f"construct '{name}' is unsourced (in {', '.join(locations) or '?'})"
- )
- elif status not in {"sourced", "author-construct"}:
- r.warn(
- f"construct '{name}': status '{status or 'empty'}' is not one of "
- f"sourced/author-construct/unsourced"
- )
- def check_state_window(d: Path, r: Report) -> None:
- """state.md is a rebuilt window, not a second transcript."""
- state = d / "state.md"
- if not state.is_file():
- return # presence is enforced by PHASE_ARTIFACTS for phase 4
- text = state.read_text(encoding="utf-8")
- missing = [s for s in STATE_SECTIONS if s not in text]
- if missing:
- r.err(
- f"state.md is missing section(s) {', '.join(missing)} — the next round is "
- f"planned from Known/Gaps/Next, so a missing one is a silently empty input"
- )
- size = len(text.encode("utf-8"))
- if size > STATE_HARD_LIMIT:
- r.err(
- f"state.md is {size // 1024} KB (limit {STATE_HARD_LIMIT // 1024} KB) — "
- f"it is being appended to, not rebuilt; the archive belongs in "
- f"sources/, deviations.md and plan.md §15"
- )
- elif size > STATE_SOFT_LIMIT:
- r.warn(
- f"state.md is {size // 1024} KB (soft limit {STATE_SOFT_LIMIT // 1024} KB) — "
- f"compress Known to subquestion statuses with claim_id/[sNN] references"
- )
- def self_check(phases: list[dict], r: Report) -> None:
- """Warn if phases.yaml gained a file-emitting phase this table does not cover."""
- known = set(PHASE_ARTIFACTS) | NO_FILE_PHASES
- for p in phases:
- if p["id"] not in known:
- r.warn(
- f"phase {p['id']} ({p['name_en']}) is in phases.yaml but not in this "
- f"validator's artifact table — add it to PHASE_ARTIFACTS or NO_FILE_PHASES"
- )
- def check_wiki_ingest(d: Path, r: Report) -> None:
- """The run must have been compiled into the cross-run wiki (finish-up step 0).
- Not in PHASE_ARTIFACTS because it belongs to no phase: ingest runs at finish-up, at
- every depth, and its real output lives outside the run directory. The receipt is
- what makes it checkable here at all — without a gate this is precisely the kind of
- step that keeps being declared and quietly stops running.
- """
- receipt = d / ".verify" / "wiki_ingest.json"
- if not receipt.is_file():
- r.err(
- "finish-up: нет .verify/wiki_ingest.json — прогон не попал в кросс-прогонную "
- "вики (python scripts/wiki_ingest.py --research-dir <dir>)"
- )
- return
- try:
- data = json.loads(receipt.read_text(encoding="utf-8"))
- except json.JSONDecodeError as exc:
- r.err(f"wiki_ingest.json нечитаем ({exc}) — перезапусти wiki_ingest.py")
- return
- if data.get("run") != d.name:
- r.err(
- f"wiki_ingest.json от прогона '{data.get('run')}', а папка '{d.name}' — "
- "квитанция скопирована из чужого ресёрча"
- )
- def validate(d: Path, mode: str, phases: list[dict], r: Report) -> None:
- self_check(phases, r)
- check_source_perimeter(d, r)
- check_schema_contract(d, r)
- check_source_quality(d, mode, r)
- check_source_breadth(d, mode, r)
- check_remediation_in_body(d, mode, r)
- check_state_window(d, r)
- check_outline_coverage(d, mode, r)
- check_constructs(d, r)
- check_wiki_ingest(d, r)
- gate_of = {p["id"]: p["depth_gate"] for p in phases}
- run_rank = GATE_RANK[mode]
- for phase_id, spec in PHASE_ARTIFACTS.items():
- gate = gate_of.get(phase_id)
- if gate is None:
- r.warn(f"phase {phase_id} is mapped here but not present in phases.yaml")
- continue
- if GATE_RANK[gate] <= run_rank: # mandatory for this mode
- check_phase(d, phase_id, spec, mode, r)
- def main() -> int:
- ap = argparse.ArgumentParser(
- description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
- )
- ap.add_argument("--research-dir", required=True, type=Path)
- ap.add_argument(
- "--mode",
- choices=MODES,
- help="run depth; frontmatter, else inferred from artifacts, if omitted",
- )
- ap.add_argument("--strict", action="store_true", help="Exit 1 if any error")
- ap.add_argument("--json", action="store_true")
- ap.add_argument("--verbose", action="store_true", help="print every warning, not the first few")
- args = ap.parse_args()
- d = args.research_dir
- if not d.is_dir():
- print(f"ERROR: not a directory: {d}")
- return 2
- r = Report()
- mode = resolve_mode(d, args.mode, r)
- phases = phases_manifest.load_phases(
- Path(__file__).resolve().parents[1] / "phases.yaml"
- )
- validate(d, mode, phases, r)
- if args.json:
- print(
- json.dumps(
- {
- "research_dir": str(d),
- "mode": mode,
- "errors": r.errors,
- "warnings": r.warnings,
- },
- indent=2,
- )
- )
- else:
- print(f"Validating phases: {d} (mode: {mode})")
- for line in capped((f" ERROR {m}" for m in r.errors), CAP_ERRORS, args.verbose):
- print(line)
- for line in capped((f" warn {m}" for m in r.warnings), CAP_WARNINGS, args.verbose):
- print(line)
- if not r.errors and not r.warnings:
- print(" OK — all mandatory phases produced their artifacts")
- elif not r.errors:
- print(f"\n{len(r.warnings)} warning(s), 0 errors.")
- else:
- print(f"\n{len(r.errors)} error(s), {len(r.warnings)} warning(s).")
- if args.strict and r.errors:
- return 1
- return 0
- if __name__ == "__main__":
- sys.exit(main())
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.