Skip to content
Back to search

cti-expert

Cyber threat intelligence and OSINT toolkit — structured investigations with sourced, trust-scored findings. Bundles the skill, the /cti command set, the typed intel MCP surface, and the RULE 1 / outbound-action safety hooks as one installable unit.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/7onez/cti-expert with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/7onez/cti-expert'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/7onez-cti-expert/badge.svg)](/listings/7onez-cti-expert)

HTML

<a href="/listings/7onez-cti-expert"><img src="/api/listings/7onez-cti-expert/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

> **Recommended:** Use **Claude Code CLI** — it gives you the full terminal workflow, persistent sessions, and direct skill invocation. [Download here](https://docs.anthropic.com/en/docs/claude-code/overview) or run `npm install -g @anthropic-ai/claude-code`.

Why Claude Code CLI?

The entire CTI Expert workflow is optimized for Claude Code CLI. The CLI gives you: - **Persistent sessions** — investigations survive terminal restarts via `/workspace save` - **Full tool access** — file writes, Python scripts, DOCX generation, all run natively - **Skill invocation** — type `/cti-expert` directly in the terminal, no browser required - **Background agents** — parallel enrichment via AgentFlow works best with the CLI

🖥️ Where to run it — the CLI is best for this skill

> [!IMPORTANT] > CTI Expert is **execution-heavy**: it runs `uv`/Python, installs OSINT tools, writes `.md`/`.json`/`.csv`/`.html`/`.docx`/`.pdf` reports + IOC bundles, reaches many external sites, and saves case workspaces. What matters is a **real local shell + persistent files + open network** — a **CLI or local desktop agent** gives you that; an ephemeral **cloud sandbox does not**. This applies equally to **Claude** and **Codex**.

| Environment | Running cases | Why | |---|---|---| | **Claude Code CLI** · **Codex CLI** | ✅ **Best** | Real shell, persistence, background tasks, open network — what the skill is built for | | **Claude Code Desktop** · **Codex IDE extension** | ✅ Great | Same local execution; nicest for **reading** rendered reports, charts & diagrams | | **claude.ai/code (web)** · **Codex cloud / ChatGPT web** | ⚠️ Limited | Reasoning & query generation work, but files don't persist to your disk and outbound network is often restricted |

> [!TIP] > **Run investigations in a CLI** (Claude Code or Codex); open the generated `.docx`/report in a Desktop/IDE window if you prefer reading there. Use web/cloud surfaces only for analyst-reasoning, not execution-heavy recon.

---

Step 1 &mdash; Install Claude Code CLI

npm install -g @anthropic-ai/claude-code

> Requires Node.js 18+. Full docs: [docs.anthropic.com/en/docs/claude-code/overview](https://docs.anthropic.com/en/docs/claude-code/overview)

---

Step 2 &mdash; Clone + All-in-One Installer

The installer handles everything: Python dependencies, system tools (`whois`, `dig`, `asn`, `jq`, `exiftool`), OSINT tools (`maigret`, `sherlock`, `holehe`, `h8mail`, and more), and optional headless browser + Go tools. It is **powered by [uv](https://docs.astral.sh/uv/)** (Astral's ultra-fast Rust package manager) — the script bootstraps uv, then uses `uv venv` / `uv pip` / `uv tool` for all Python installs, falling back to `pip`/`pipx`/`venv` only if uv can't be installed. Use `install.ps1` on Windows (PowerShell) or `install.sh` on macOS/Linux/Git Bash/WSL.

<table> <tr> <th>Platform</th> <th>Command</th> </tr> <tr> <td><b>Linux / macOS</b></td> <td>

git clone https://github.com/7onez/cti-expert.git ~/.claude/skills/cti-expert
bash ~/.claude/skills/cti-expert/scripts/install.sh

</td> </tr> <tr> <td><b>Windows (Git Bash or WSL)</b></td> <td>

git clone https://github.com/7onez/cti-expert.git ~/.claude/skills/cti-expert
bash ~/.claude/skills/cti-expert/scripts/install.sh

</td> </tr> <tr> <td><b>Windows (PowerShell — native)</b></td> <td>

git clone https://github.com/7onez/cti-expert.git "$env:USERPROFILE\.claude\skills\cti-expert"
powershell -ExecutionPolicy Bypass -File "$env:USERPROFILE\.claude\skills\cti-expert\scripts\install.ps1"

</td> </tr> </table>

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/plugin.json · Manifest · 27 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "cti-expert",
  3. "description": "Cyber threat intelligence and OSINT toolkit — structured investigations with sourced, trust-scored findings. Bundles the skill, the /cti command set, the typed intel MCP surface, and the RULE 1 / outbound-action safety hooks as one installable unit.",
  4. "version": "2.12.0",
  5. "author": {
  6. "name": "Hieu Ngo",
  7. "url": "https://github.com/7onez/cti-expert"
  8. },
  9. "homepage": "https://github.com/7onez/cti-expert",
  10. "license": "See LICENSE",
  11. "keywords": [
  12. "osint",
  13. "cti",
  14. "threat-intelligence",
  15. "investigation",
  16. "dfir"
  17. ],
  18. "commands": "./commands",
  19. "skills": "./",
  20. "hooks": "./hooks/hooks.json",
  21. "mcpServers": {
  22. "intel": {
  23. "command": "${CLAUDE_PLUGIN_ROOT}/intel_engine/harness/mcp-server",
  24. "args": []
  25. }
  26. }
  27. }
README.md · README · 1,499 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. # CTI Expert
  3. ### Cyber Threat Intelligence & OSINT Analysis Toolkit
  4. **Transform Claude into a trained intelligence analyst — 120+ commands, 58 techniques, zero API keys required for core functionality.**
  5. <br>
  6. <p>
  7. <a href="#installation">Installation</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="#quick-start">Quick Start</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="#command-reference">Commands</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="#contributing">Contribute</a>
  8. </p>
  9. <br>
  10. <!-- Feature Badges -->
  11. <p>
  12. <a href="https://github.com/7onez/cti-expert"><img src="https://img.shields.io/badge/version-2.12-0080ff?style=for-the-badge&logo=semver&logoColor=white" alt="Version 2.12"></a>&nbsp;
  13. <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-00c853?style=for-the-badge&logo=opensourceinitiative&logoColor=white" alt="License: MIT"></a>&nbsp;
  14. <a href="#command-reference"><img src="https://img.shields.io/badge/commands-120+-ff6d00?style=for-the-badge&logo=windowsterminal&logoColor=white" alt="120+ Commands"></a>&nbsp;
  15. <a href="#technique-catalog"><img src="https://img.shields.io/badge/techniques-58-aa00ff?style=for-the-badge&logo=hackthebox&logoColor=white" alt="58 Techniques"></a>&nbsp;
  16. <a href="#installation"><img src="https://img.shields.io/badge/API_keys-none_for_core-00bfa5?style=for-the-badge&logo=shield&logoColor=white" alt="No API Keys for Core"></a>
  17. </p>
  18. <!-- GitHub Stats -->
  19. <p>
  20. <a href="https://github.com/7onez/cti-expert/stargazers"><img src="https://img.shields.io/github/stars/7onez/cti-expert?style=flat-square&logo=github&label=Stars" alt="Stars"></a>&nbsp;
  21. <a href="https://github.com/7onez/cti-expert/network/members"><img src="https://img.shields.io/github/forks/7onez/cti-expert?style=flat-square&logo=github&label=Forks" alt="Forks"></a>&nbsp;
  22. <a href="https://github.com/7onez/cti-expert/releases"><img src="https://img.shields.io/github/downloads/7onez/cti-expert/total?style=flat-square&logo=github&label=Downloads&color=brightgreen" alt="Downloads"></a>&nbsp;
  23. <a href="https://github.com/7onez/cti-expert/issues"><img src="https://img.shields.io/github/issues/7onez/cti-expert?style=flat-square&logo=github&label=Issues" alt="Issues"></a>&nbsp;
  24. <a href="https://github.com/7onez/cti-expert/pulls"><img src="https://img.shields.io/github/issues-pr/7onez/cti-expert?style=flat-square&logo=github&label=PRs" alt="Pull Requests"></a>&nbsp;
  25. <a href="https://github.com/7onez/cti-expert/commits"><img src="https://img.shields.io/github/last-commit/7onez/cti-expert?style=flat-square&logo=github&label=Last%20Commit" alt="Last Commit"></a>&nbsp;
  26. <a href="https://github.com/7onez/cti-expert"><img src="https://img.shields.io/github/repo-size/7onez/cti-expert?style=flat-square&logo=github&label=Size" alt="Repo Size"></a>&nbsp;
  27. <a href="https://github.com/7onez/cti-expert/graphs/contributors"><img src="https://img.shields.io/github/contributors/7onez/cti-expert?style=flat-square&logo=github&label=Contributors" alt="Contributors"></a>
  28. </p>
  29. <!-- Language Selector -->
  30. <p>
  31. 🇬🇧 <a href="README.md"><b>English</b></a>&nbsp;&nbsp;·&nbsp;&nbsp;🇻🇳 <a href="README.vi.md">Tiếng Việt</a>&nbsp;&nbsp;·&nbsp;&nbsp;🇨🇳 <a href="README.zh-CN.md">中文</a>
  32. </p>
  33. <br>
  34. <sub>Built by <a href="https://www.linkedin.com/in/hieu-minh-ngo-hieupc/"><b>Hieu Ngo</b></a> &bull; <a href="mailto:hieu.ngo@chongluadao.vn">hieu.ngo@chongluadao.vn</a> &bull; <a href="https://chongluadao.vn">chongluadao.vn</a></sub>
  35. <br>
  36. <sub>Core contributor &bull; <a href="https://github.com/Zeroska"><b>Zeroska</b></a> &bull; <a href="mailto:khuong.nguyen@chongluadao.vn">khuong.nguyen@chongluadao.vn</a></sub>
  37. </div>
  38. <br>
  39. ---
  40. <br>
  41. ## 🤝 Sponsors &amp; Supporters
  42. <div align="center">
  43. **CTI Expert is built in the open. These organisations back the work — with data, tooling, and hard-won investigative tradecraft.**
  44. </div>
  45. | Supporter | What they bring | In the toolkit |
  46. |-----------|-----------------|----------------|
  47. | [**Rexxfield**](https://rexxfield.com) | Cybercrime investigation and victim-side casework since 2008 — the real-world tradecraft the case workflow and attribution standards are modelled on | Tradecraft &amp; methodology |
  48. | [**APWG**](https://apwg.org) | Anti-Phishing Working Group — the global cross-industry coalition against phishing and e-crime, and host of the [eCrime research symposium](https://ecrimeresearch.org) whose 2026 papers ground eight of this toolkit's techniques | Phishing detectors (ClickFix, domain survival, kit fingerprint, visibility-aware HTML, PhishTrace, APK permission scope) · `/threat-model` AAM overlay |
  49. | [**Hudson Rock**](https://www.hudsonrock.com) | Infostealer-infection intelligence — which machines leaked which credentials, and when | `/breach-deep` · `/stealer-log` |
  50. | [**ParanoidLab**](https://paranoidlab.com) | Dark-web, Initial-Access-Broker and infostealer-log monitoring across forums, markets and private Telegram | Dark-web collection &amp; review |
  51. | [**ANY.RUN**](https://any.run) | Interactive malware sandbox + **TI Lookup** — sandbox-observed C2 and real endpoints from packed samples | `/binary` · `/hash-id` |
  52. | [**ZETAlytics**](https://zetalytics.com) | Global passive DNS with rare geographic diversity — historical resolution and co-tenancy pivots | `/webpivot` · `/cti-pivot` |
  53. | [**IntelX**](https://intelx.io) | Intelligence X — paste sites, leaks, darknet and phonebook selector search | `/webpivot` · `/email-deep` |
  54. | [**Shodan**](https://www.shodan.io) | Internet-connected host &amp; service intelligence — open ports, banners, tags and known CVEs, passive-first via InternetDB | `/webpivot` · `/appliance-scan` · `/cert-pivot` |
  55. | [**Censys**](https://censys.com) | Internet-wide host &amp; certificate scanning — the server-side view; every host on an exact leaf certificate (works on the free plan) | `/censys` · `/cert-pivot` |
  56. | [**URLScan.io**](https://urlscan.io) | Passive website scanning — what a page served and who it talked to, captured without touching the target | `/webpivot` · `/impersonate` |
  57. | [**SerpApi**](https://serpapi.com) | Search-engine + Google Ads Transparency results API — who *paid* to send traffic, plus multi-engine dork results | `/serp` · `/search-pivot` |
  58. | [**GrayHatWarfare**](https://grayhatwarfare.com) | Open cloud-bucket &amp; exposed-file search (S3/Azure/GCS/Spaces), the link-shortener corpus (historical long URLs behind goo.gl/bit.ly links) and the public-git corpus (committers, token-in-code, leaked-secret files) — graded **exposure**, not a same-operator pivot | `/grayhatwarfare` · `/secrets` · `/docleak` · auto-fired per apex in `pipeline open` |
  59. | [**Social Links**](https://sociallinks.io) | OSINT investigation platform — 1000+ methods across social media, blockchain and the dark web (SL Professional / Crimewall, Maltego transforms) | OSINT methodology &amp; data |
  60. | [**Validin**](https://validin.com) | DNS + certificates + favicon + response-body hashes in one graph — passive DNS, subdomain enumeration, reverse-IP and host-response hash pivots on a free community key | Native in `/webpivot` (domain lookup, reputation, cert &amp; favicon hosts) · MO-neighbour source · `/cti-pivot` |
  61. | [**Netlas**](https://netlas.io) | Independent internet-asset index — DNS, scan responses, WHOIS and certificate collections behind one key; `domains a:<origin-ip>` reverses a non-CDN origin to every apex with dates | `/webpivot` MO-neighbour source · `intel.py netlas` · entitlement probe |
  62. > [!IMPORTANT]
  63. > **ANY.RUN lookups are read-only; case binary submissions are automatic when keyed.** `/cti` downloads APK/installer links to `cases/<case>/bin/`, runs static analysis and, when `ANYRUN_API_KEY` is set in a spending run, submits each complete file through the private-by-default sandbox client. `--free-only`, case `no_spend`, default `loop` (unless `--full`) and hostile/no-direct-contact posture block submission; plan checks and privacy read-back still apply. A public-only account is refused unless `ANYRUN_ALLOW_PUBLIC=1`; public tasks are world-readable and irreversible. Direct `anyrun_submit` remains ask-first and requires `HARNESS_ALLOW_SUBMIT=1`. See [`SKILL.md`](SKILL.md).
  64. <sub>Listing here reflects support for the project and does <b>not</b> imply affiliation, endorsement, or any verification of this tool by the organisations named. Integrations marked above are optional and key-gated — <b>every core technique still runs with zero API keys</b>. Always respect each provider's terms of service. The full list of open-source projects and free public-interest services this skill depends on is in <a href="#-acknowledgments--credits">Acknowledgments &amp; Credits</a>.</sub>
  65. <br>
  66. ---
  67. <br>
  68. ## What is CTI Expert?
  69. A **Claude Code skill** that transforms Claude into a trained cyber threat intelligence and open-source intelligence analyst. It runs structured intelligence collection using **120+ commands** across **58 techniques** — no API keys required for core functionality. To take full advantage, add your own **free *or* paid** API keys to the skill's `.env` — each is **auto-detected** and unlocks higher-tier access (e.g., Wigle, VirusTotal, URLScan.io, Shodan, Censys, SecurityTrails, WhoisXML).
  70. > [!TIP]
  71. > **Keyless by default, more powerful with your keys.** Every core technique runs with zero API keys. Add any free or paid keys to `.env` (or run `/apikeys set <service> <KEY>`) and the skill auto-detects them, unlocking higher-tier pivots: reverse favicon→host, passive DNS, certificate search, sibling-domain discovery. A missing or bad key never breaks a run — it just degrades to a note. Setup guide: [handbook/api-keys.md](handbook/api-keys.md).
  72. > [!TIP]
  73. > **One skill, two layers.** cti-expert is the *broad collector* — the wide net (`/sweep`, `/webpivot`, `/subdomain`, `/username`, `/email-deep`…). Built into the repo is a *deep pipeline* (`intel_engine/`) that turns raw collection into a real case: a persistent knowledge base, versioned cases, cross-case correlation, and calibrated assessment. The flow reads like a sentence — **collect broadly → "seen this operator before?" → cluster → filter false positives → assess.** No external setup: the backend resolves to `SELF`, and **as of v2.9 the bundled installer (`scripts/install.{sh,ps1}`) provisions the deep layer automatically** (or by hand: `uv venv && uv pip install -r requirements.txt`). Architecture: [connectors/intel-backend.md](connectors/intel-backend.md).
  74. <table>
  75. <tr>
  76. <td width="50%">
  77. **Core Capability**
  78. Multi-vector reconnaissance on any target type — person, domain, organization, username, email, IP, WiFi — with automated finding validation, exposure scoring, and structured intelligence delivery.
  79. </td>
  80. <td width="50%">
  81. **AEAD Workflow**
  82. **A**cquire raw data &rarr; **E**nrich with pivot expansion &rarr; **A**ssess findings &rarr; **D**eliver structured reports — the base bundle (Markdown + JSON + CSV + IOC bundle) always saves, then you pick a presentation report: PDF, DOCX, HTML, or all.
  83. </td>
  84. </tr>
  85. </table>
  86. <br>
  87. ---
  88. <br>
  89. ## What's New in v2.12
  90. > **The release where the premium keys start pulling their weight — and the report writes itself.** v2.11 shipped detectors. v2.12 turns an eight-finding audit of the paid vendors into behaviour (every metered leg gated, bought **once per case** instead of once per host), adds **Netlas** as an independent index, makes the editorial PDF/DOCX a **deterministic composition** from the case directory, and closes the last honesty gap in the ANY.RUN docs: detonation exists, and it is gated five ways.
  91. | Category | What's New | Details |
  92. |----------|-----------|---------|
  93. | **Cross-type auto-pivot loop** | Every selector type is a frontier node; metered spend stays behind one digest-bound approval | Typed nodes (`selector_types` + `rungs.json` owner/kit/neighbour) → `case_state.propose` → required Plan phase (`PLAN_TOOLS`, read-only) → auto free/urlscan pivots with ≤ **N=25** hits, else `proposal.json` gated in `audit.gate` (`harness approve` · `case_approve` · TTY). Dispatcher ops `propose` + `corpus-search` (`corpus_search` tool). Assessment gains additive `kit_siblings` / `neighbours` / `mentions` — collected, never attributed. Rail: collecting ≠ attributing |
  94. | **MO-neighbour pivot** | Reverse a non-CDN origin to its co-tenants — and only ever seed on a registrant join key | `wp_mo_neighbours.discover` reverses the estate's non-CDN origin IP (also the MX origin) through **Netlas · Validin · urlscan**, WHOIS-verifies each candidate apex, and classifies it `same_registrant` (current registrant e-mail/phone join key only — a proxied record contributes **no** phone) / `same_mo` (reference-data policy) / `unrelated` / `unverifiable`. **Only `same_registrant` seeds the frontier**; same-MO personas render as a rung-10 *Related personas* table in the report (`--mask-personas` to aggregate). A bulk-hosting guard fires *before* any spend; per-origin lock + disk cache + a per-run WhoisXML ledger hold across collector subprocesses, so an origin is bought once per case. RULE 5 rails intact: facts-only KB ingest, never an edge, never an `operator_lead` |
  95. | **WHOIS eras · urlscan Pro lifecycle** | The timeline now knows who held the name *before* this operator | WHOIS history is an explicit `--whois-history purchase` scoped to seeds and cluster members (the render path never buys it); the house report renders a **Registrant eras** table (same-day flaps folded, registrar placeholders classed) and uses the current era's start as the archive-capture cutoff. urlscan **Pro** supplies the hostname lifecycle index — A/NS eras, CT and zonefile firsts, left-censored on a truncated walk — feeding the temporal view; verdict rows appear only on signal, and the API key is sent to urlscan.io only |
  96. | **Entitlement measured, not assumed** | A key measured *free* skips the calls that would 403 | `wp_capabilities.discover_plans` probes each vendor's **free account endpoint** (urlscan quotas, Netlas plan, SecurityTrails/DNSLytics usage, IntelX `/authenticate/info`, Validin `/api/paths`) once per case into `cases/<id>/capability_plans.json`; `enrich_live` reads the store and gates **only on a positive `free`** verdict — unknown keeps the productive call as the probe. Censys search records its own verdict. A failed probe is reported this run and never frozen, so the next process re-measures |
  97. | **Netlas + the rest of the vendor wiring** | Every registered key now has a pivot behind it | [`wp_netlas.py`](intel_engine/WebPivot/tools/wp_netlas.py) — Bearer client over the `domains` · `responses` · `whois_domains` · `whois_ip` · `certs` collections (search, count, facet, reverse-IP, plan), keyless query builder + web-UI links, ledgered, Cloudflare-safe UA; `intel.py netlas ip\|ns\|spf\|domain\|san\|title\|plan\|raw`. Live-measured: `domains a:<origin>` returned **32 apexes** on a 32-domain estate (15 members + same-MO siblings under other personas). Alongside: **SecurityTrails** DNS-history eras + DSL reverse-WHOIS (diffed against WhoisXML), **DNSLytics** reverse-IP under its own co-tenancy-routed key, **GrayHatWarfare** exposure lead + report section, **Shodan** cert/JARM search, once-per-case **Censys** cert search with a shared-cert fan-out guard, **IntelX** auto-fire in `pipeline open` (loop: `--full` only; role mailboxes excluded), and **Validin** wired natively (domain lookup, reputation, cert and favicon hosts). Frontier ranks owner-link candidates above lookalikes |
  98. | **The house report, composed deterministically** | The editorial PDF/DOCX no longer needs a model to write it | `intel.py house-report <CASE-ID>` composes the IntelReport document from the case dir: sections I–XI, both confidence scales + the **ICD-203 × Admiralty scatter**, relationship graph + **entity-relationship map**, attribution inference chain, temporal view + **registration heatmap** + **domain × shared-indicator matrix** (which honours the §2.5 false-positive control and carries the WHOIS join keys), a **landing-page capture** per estate host (proxy-gated egress; a page that will not render falls back to the newest public **web-scan**, then a **web-archive** snapshot — captioned, dated by the archive, and labelled a *previous owner's page* when it predates the current registration), per-domain **dossiers**, a glossary, Appendices **A–E**. Rule 12 scrub of tool/vendor/path names; third parties masked through one gate ([`scripts/cti_third_party_mask.py`](scripts/cti_third_party_mask.py)). One figure source ([`scripts/cti_report_figures.py`](scripts/cti_report_figures.py)) feeds both the dashboard DOCX and the house PDF, so the two deliverables cannot disagree. `--no-screenshots` is fully offline; `--no-archive-fallback` forbids the stand-ins |
  99. | **ANY.RUN: the gate is real, and the docs now say so** | Detonation exists — behind five gates, each of them code | SKILL.md, the README callouts, `.env.example` and the key registry claimed "no submit path". False since the gated layer landed. `anyrun_submit` is gated by: per-submission confirmation (a `confirm=false` **briefing** first — a data-driven `approval_briefing` exemption in `tool_policy.json` now lets that step through the MCP approval gate, fail-closed to fully gated if the file is unreadable); **private by default**, `public` refused unless `ANYRUN_ALLOW_PUBLIC=1` grants a *standing* authorization, and then only as an explicit, recorded downgrade; a **fail-closed plan check** (`/user` private quota — zero is denied and no attestation overrides it — else a prior private task, else `--i-have-a-paid-plan`); a **polled post-submit privacy read-back** that withdraws and flags a task that still landed public (`verify-privacy <uuid>` finishes the check if the task outlives the wait); and the harness deny unless `HARNESS_ALLOW_SUBMIT=1`. 103 stubbed checks in `test_intelx_anyrun` §7b–7c; `test_tool_gate` §2 pins briefing-allowed / confirm-denied / fail-closed |
  100. | **Repo hygiene** | The tree checks itself for pasted keys | `audit.sh` §5b greps every tracked file for vendor key shapes and `KEY=value` lines — proven to fire on a planted value and clean on the tree, so a key can only live in `.env`. [AGENTS.md](AGENTS.md) rewritten as the cross-agent *Repository Guidelines*. Key-alias registries reconciled with every tool's `_secret()` lookup and locked (`tests/test_key_alias_registry.py`); SecurityTrails / DNSLytics / CertSpotter registered so the capability banner stops going silent on keys it uses. The three workflow diagrams re-rendered from current sources — 79 `@tool`s, 9 commands, Netlas, gated ANY.RUN |
  101. | **Every documented command is real** | 26 SKILL.md commands that had no code behind them now resolve to a tool — or say what they are | 18 new keyless-first CLIs under [`scripts/osint/`](scripts/osint/) (`hash_id` — MD5 vs NTLM *before* you submit, `vuln_check`, `username_enum`, `phone_osint`, `github_osint`, `reputation_check`, `msft_recon`, `cn_recon`, `traffic_rank`, `sharelink_resolve`, `dork_builder`, `exposure_score`, `signature_scan`, `kb_crossref`, `case_drift`, `deep_profile`, `wifi_ssid` …) plus 8 existing `scripts/webpivot/` shims (`cert_pivot`, `email_hygiene`, `wayback_fetch/harvest`, `rank_relations`, `sensitive_paths`, `pivot_suggest`, `crypto_balance`) registered as `@tool`s and `intel.py` ops; every SKILL.md command row now names its **T2** CLI and **T1** tool, or is labelled `[model]` / `[unimplemented]`. **79 `@tool`s**, and the SDK allowlists are locked to the decorated set (`test_tool_registry.py`). Clustering enforces the **two-artifact rule**: each cluster carries `corroboration`, and one shared artifact is flagged `LEAD ONLY`, never an attributed estate. The vendored engine's own tests now run in `audit.sh` §9–10 and CI — they ran nowhere before |
  102. | **ChongLuaDao in the pipeline · subdomains · one timeout ceiling** | The first-party feeds grade every host, live subdomains join the case, and no call hangs a run | `wp_cld.py` runs per collected host inside `enrich_live` — denylist verdict + IoC analysis land as **reputation facts** (never a cluster edge) and CLD WHOIS is the **primary source for `.vn`** names. `subdomain_enum` unions subfinder (auto-keyed from `.env`), amass, assetfinder and findomain, DNS-verifies the names into `cases/<id>/subenum/<apex>.json`, and the frontier queues the **live** ones for the next round (hoster plumbing is a fact, not a seed). A single per-call ceiling — `CTI_CALL_TIMEOUT`, default **1800 s** from `WebPivot/references/timeouts.json` — floors every fetch, vendor call, collector and renderer subprocess; raw DNS/TLS probes keep their fail-fast bounds, and a running MCP server keeps its value until restarted |
  103. | **Registrable apexes · landlord IPs · kit binders** | The frontier and the KB agree on what an apex is and which IP is a landlord | A vendored **Public Suffix List** (`public_suffix_list.json`, `wp_psl_update.py`) drives one `_registrable()` reducer, so `shop.id.vn` and `panel.co.uk` fold to the right apex everywhere — frontier, ingest, report, Blueprint. `shared_hosting_max_cohosts` (12) is the **one** number that marks an IP as shared/bulk hosting: its `hosted_on` edges become facts, the cluster partition ignores them, and co-tenants are held back as leads. Expansion depth is anchored: a host owner-linked to a *previous hop* is `related_hosts`, never a member. Scraped social links resolve to the **account** — per-network reserved paths, second-segment routers and library/vendor credit accounts are reference data, and a bare platform apex (`t.me`, `facebook.com`) is never a handle |
  104. | **Blueprint folds to apex level** | A hundred-host estate no longer silently drops the HTML report's architecture tab | Archify's architecture type draws ≤ 12 nodes · 18 edges, so [`cti_archify.py`](scripts/cti_archify.py) now **collapses hosts under their registrable apex** (`Estate · N hosts`), folds the long tail into `+N more apexes`, ranks finding-bearing apexes first and places the operator hub mid-row with spokes above and below. `CTI_ARCHIFY=1` (auto: full graph if it fits, else the fold) · `force` (the widest 25-node grid) · `0`; `--plan` prints both outcomes offline, and every HTML export now confirms the Blueprint mode first (Step C — `--yolo` keeps Auto). Locked by `tests/test_archify_blueprint.py` |
  105. | **`/cti` on every coding agent** | One registration step wires the commands into every agent on the machine, not only Claude Code | [`scripts/register_agents.py`](scripts/register_agents.py) (run by `register.sh`) registers the 9 commands with **omp · pi · OpenCode · Gemini CLI · Qwen Code · Factory droid · Windsurf** and installs `/cti` as the shared `~/.agents/skills/cti` skill for **Cursor · Copilot CLI** (`/cti`), **Codex** (`$cti` — Codex removed custom prompts in 0.117, so the old `codex/cti-expert.md` is gone) and the **ChatGPT desktop app** (`@cti`). Every file is a pointer stub to this clone, so a `git pull` updates all agents at once. Install and `--uninstall` touch only what they wrote — marked stubs, and `~/.claude` symlinks that resolve into this repo — never a same-named command or skill of yours. Locked by `tests/test_register_agents.py` |
  106. | **Mermaid figures without a browser** | mermaid-cli + Puppeteer + headless Chrome replaced by one ~700 KB module | IntelGraph's `render_mermaid.py` now lays every Mermaid figure out with [Pele](https://github.com/obsidianmd/pele) (MIT) under Node 20+ or Bun — labels as real SVG `<text>`, no `<foreignObject>`, roughly 3–70 ms per figure. The npm release is pinned in `IntelGraph/references/pele.json`, sha512-verified and fetched into the gitignored `IntelGraph/vendor/pele/` (as `pele.mjs`, so every Node 20 loads it as ESM) by the installers (default install, no longer behind `--all`) or on first render. Pele's CSS variables are resolved to the house theme (`theme.PELE_VARS`) so the SVG renders the same in librsvg, cairosvg, pandoc and Word; PNG + vector PDF come from `rsvg-convert` (librsvg, now a default install) → headless Chrome/Edge → cairosvg. That order keeps the node type icons (🌐 💬 📧): cairosvg draws a label in one font and drops every glyph it lacks, so it is the last resort and warns. `references/diagram.css` and the mmdc flag shims are gone, and `render_mermaid.py` drops `--theme` / `--css` / `--no-css` / `--puppeteer-config` |
  107. <details>
  108. <summary><b>What's New in v2.11</b></summary>
  109. ## What's New in v2.11
  110. > **The eCrime-2026 follow-up batch — planned, then cooked.** Four deferred phases from the v2.10 research pass ship as keyless, offline, tested modules (planned in `plans/260826-1935-ecrime-2026-followup-analyzers/`, implemented via `/ak:cook`). Same contracts: zero-dep tests in `audit.sh`, attribution-safety (RULE 5). Full mapping: [docs/ecrime-research-integration.md](docs/ecrime-research-integration.md) §6.
  111. | Category | What's New | Details |
  112. |----------|-----------|---------|
  113. | **APK permission-scope scoring** | On-device-fraud capability from the manifest | [`scripts/apk_permission_scope.py`](scripts/apk_permission_scope.py) scores dangerous-permission **combinations** (accessibility+overlay+SMS = banking-trojan profile) — the combination is the signal, not any single grant, so banking/AV apps aren't blindly flagged; result is **capability, not guilt**. Handles plaintext manifests and binary AXML (UTF-8 **and** UTF-16 string pools; a zero-permission decode degrades to a note, never a false "clean"). Extends BinaryPivot. Grounded in *"The 'Allow' Reflex"* (Kandagadla Srinivasamurthy, Dupuis — UW). See [`techniques/apk-permission-scope.md`](techniques/apk-permission-scope.md) |
  114. | **Kit-template attribution** | Same-kit lineage across rotating hosts | [`scripts/kit_template_fingerprint.py`](scripts/kit_template_fingerprint.py) fingerprints DOM structure + harvest-form field-set + asset skeleton and grades similarity — but a **commodity template** (WordPress/Wix/Shopify) match is graded *noise*, not a same-operator link (§2.5 trap), and no grade ever auto-merges. Grounded in the Vicomtech *tree-structured attribution* paper. See [`techniques/kit-template-attribution.md`](techniques/kit-template-attribution.md) |
  115. | **Renderer-level confirmation** | Dynamic confirmation of the static ClickFix/visibility verdicts | [`scripts/render_confirm.py`](scripts/render_confirm.py) feeds renderer-captured **runtime clipboard writes** and **computed-style hidden elements** into the existing detectors (via new additive seams) and reconciles static vs rendered — promoting a JS-assembled payload the static pass missed, or weighing a static-only HIGH. Renderer (Playwright/agent-browser) is optional and **degrades to a note**. Grounded in *PasteJacked* + *Visibility-Aware HTML*. See [`techniques/renderer-confirmation.md`](techniques/renderer-confirmation.md) |
  116. | **PhishTrace dynamic features** | Characterize a page from its runtime trace | [`scripts/phishtrace_features.py`](scripts/phishtrace_features.py) turns a runtime trace (requests, redirects, form POSTs, cloaking) into a verdict + **exfil-endpoint IOCs** — where a thin/cloaked trace on a flagged page reads as **cloaked, never benign**, and exfil hosts are pivot leads, not attribution. Grounded in *"PhishTrace"* (Islam, Mannan). See [`techniques/phishtrace-dynamic-features.md`](techniques/phishtrace-dynamic-features.md) |
  117. | **AAM actor-modeling overlay** | Anticipate the next move, not just attribute the past | [`handbook/aam-actor-modeling.md`](handbook/aam-actor-modeling.md) adds an optional `/threat-model` overlay (OODA faces + Mirror/Twin/Opposite/Lever) alongside ACH. Grounded in ISECOM's *"Modeling Adversaries Through Chaos"* training. Docs-only |
  118. | **Tested & gated** | Four more zero-dep suites in `audit.sh` | `test_apk_permission_scope` (combo-vs-single + benign control + AXML zero-perm degrade), `test_kit_template_fingerprint` (commodity-trap RULE 5 guard + never-auto-merge), `test_render_confirm` (promotion + seam-additivity + no-renderer degrade), `test_phishtrace_features` (cloaked-not-benign) |
  119. </details>
  120. <details>
  121. <summary><b>What's New in v2.10</b></summary>
  122. ## What's New in v2.10
  123. > **The release that turns APWG eCrime 2026 research into working, tested detectors.** Three keyless, offline, deterministically-tested analyzers land at the Enrich→Assess boundary, each mapped to a named eCrime 2026 paper — closing the two audited gaps: page-content threat classification, and an explicit maliciously-registered-vs-compromised call. Full mapping + repo audit: [docs/ecrime-research-integration.md](docs/ecrime-research-integration.md).
  124. | Category | What's New | Details |
  125. |----------|-----------|---------|
  126. | **Phishing-domain survival profiling** | Registration/DNS strategy → maliciously-registered vs **compromised**, plus takedown-resilience | [`scripts/phish_domain_survival.py`](scripts/phish_domain_survival.py) turns collected WHOIS/DNS into a two-axis, fully auditable judgement (every signal carries a weight + reason; seed lists tunable via `--refs`). The compromised/malicious split is attribution-safety: a long-lived domain at an established registrar serving unrelated legitimate content is a **victim**, never named as the operator (RULE 5). Grounded in *"Built to Last? Registration and DNS Strategies in Phishing Domain Survival"* (Lim et al., eCrime 2026). See [`techniques/phishing-domain-survival.md`](techniques/phishing-domain-survival.md) |
  127. | **ClickFix / PasteJacking detection** | Clipboard-hijack lures + `-EncodedCommand` decode → C2 IOCs | [`scripts/clickfix_detect.py`](scripts/clickfix_detect.py) scores three co-occurring families (clipboard write · social lure · OS-command payload) so a coupon "copy" button or the word *powershell* in prose never reads as HIGH; it decodes PowerShell `-enc` UTF-16LE base64 to surface the hidden C2 URL, and maps to ATT&CK T1204.004/T1059. Grounded in *"PasteJacked: Detection and Characterization of Clipboard-Hijacking Attacks"* (Nabeel, Melicher, Starov — Palo Alto, eCrime 2026). See [`techniques/clickfix-clipboard-hijack.md`](techniques/clickfix-clipboard-hijack.md) |
  128. | **Visibility-aware HTML analysis** | Surface hidden credential forms / off-origin links / off-screen brand text | [`scripts/html_visibility_analysis.py`](scripts/html_visibility_analysis.py) catches inline + class-based hiding a naive parser misses, severity-ranked by concealed intent + origin (a hidden off-origin credential form is HIGH; a benign `type=hidden` CSRF field is **not** flagged). Static approximation with the renderer-level path (agent-browser/Playwright) noted as the upgrade. Grounded in *"Visibility-Aware HTML Analysis through Renderer-Level Extraction"* (Betts et al., Auckland, eCrime 2026). See [`techniques/visibility-aware-html.md`](techniques/visibility-aware-html.md) |
  129. | **Tested & gated** | Three zero-dep suites wired into `audit.sh` | `tests/test_phish_domain_survival.py`, `tests/test_clickfix_detect.py`, `tests/test_html_visibility_analysis.py` — with explicit false-positive guards (coupon-copy, prose mention, CSRF field) and the RULE 5 compromised/malicious split — run in [`scripts/audit.sh`](scripts/audit.sh) §6 |
  130. </details>
  131. <details>
  132. <summary><b>What's New in v2.9</b></summary>
  133. ## What's New in v2.9
  134. > **The release where "install it" and "install everything" become the same command — and the pivot loop stops leaving identifiers on the floor.** v2.8 moved the safety rails to where the harness runs. v2.9 fixes the layer *beneath* the collector: the bundled installer now provisions the whole two-layer stack, not just the light collector — and the last four auto-pivot dead-ends, where a collected identifier was typed but never chased, are wired shut.
  135. | Category | What's New | Details |
  136. |----------|-----------|---------|
  137. | **The installer provisions the deep layer** | "Install once" now installs the pipeline, not just the collector | Both `scripts/install.{sh,ps1}` installed **only** `scripts/requirements.txt` (the light collector) and never the **root** `requirements.txt` — so the IntelHarness Agent-SDK pipeline (`claude-agent-sdk`, `pydantic`, `rich`), the typed MCP server, and IntelGraph's `graphviz` render path were declared, imported, then **silently absent** on a fresh box. The installer now runs the root manifest unconditionally (small, pure-Python), so `/harness`, the MCP surface, and link-graph rendering work out of the box |
  138. | **`--all` = everything, including the heavy renderers** | The engines pandoc/IntelGraph shell out to are installable now | `dot` (Graphviz binary — IntelGraph link graphs) and `gh` (the GitHub CLI the whole `/github-osint` workflow calls) install on **every** platform now — `gh` was Windows-only before. `--headless` additionally installs **Playwright + Chromium** (post-JS render, screenshot evidence, Engage automation — previously an unguarded `ModuleNotFoundError`). `--all` adds the two heavy ones — **mermaid-cli** (`mmdc`, flow/kill-chain diagrams) and **xelatex** (IntelReport PDF) — behind the flag because each is hundreds of MB and DOCX/HTML need neither |
  139. | **Auto-pivot: the last dead-ends wired shut** | A typed identifier that never spawns a pivot is a silent hole in the spider-map | Four fixes in [`scripts/pivot_orchestrator.py`](scripts/pivot_orchestrator.py): (1) **altcoin wallets** — TRON/LTC/XMR are scraped by `pivot_extract` and traceable by `crypto_balance.py`, yet only BTC/ETH were typed; all three now classify, re-enter the frontier, and run on-chain flow. (2) **breach passwords** no longer mis-seed — a recovered password fell through to the username regex and got enumerated across 3000+ platforms; it is dropped from the breach yields. (3) **`/email-deep`** infrastructure (domain/IP) and (4) **`/gdoc`** share links now carry edge-matrix entries, so both feed discoveries back into the loop instead of ending at a leaf. Company names get a documented `type:"org"` tagging rule — `classify()` cannot tell an org from a person |
  140. | **46 → 52 MCP tools** | The typed intel surface grew; the changelog now says so | `intel_engine/harness/tools.py` registers **52** `@tool`s (24 collect + 28 analyze) — six past the v2.8 figure, added with the engine sync but never recorded. No new install step; they surface through the same `intel` MCP server |
  141. | **Command count corrected** | The badge undersold the toolkit | §3 now carries **120+** commands (measured: 145 rows, ~123 unique base commands) — the "75+" badge predated three releases of additions |
  142. </details>
  143. <details>
  144. <summary><b>What's New in v2.8</b></summary>
  145. ## What's New in v2.8
  146. > **The release where the engine caught up and the safety rails moved to where the harness can see them.** v2.7 landed the deep pipeline. v2.8 brings the vendored engine **~30 commits forward** — **24 → 46 MCP tools**, a new engagement skill, and a case loop that runs to convergence — and then fixes the layer underneath it: two safety properties were being enforced at a moment Claude Code never reaches. Both now fire where the work actually happens.
  147. | Category | What's New | Details |
  148. |----------|-----------|---------|
  149. | **Engine sync — 24 → 46 MCP tools** | A three-way merge, not a copy — and that distinction is the whole story | The vendored `intel_engine/` was ~30 commits behind. Rather than trust a remembered list of local patches, every vendored file was classified by **blob identity against all upstream history**: 114 pure copies, **15 deliberately patched**, **6 cti-expert-only**. A plain `rsync` would have silently reverted three real behaviours — `wp_common`'s extra `.env` depth (cti-expert nests one level deeper, so upstream's version resolves **every API key to empty**, and a keyless run then reports *"no siblings"* as a fact about the operator), `pivot_extract`'s reverse-WHOIS-on default, and the `collect_core` single-sourcing — plus destroyed `email_permute` and turned three RULE 4 shims back into copies. **48 → 69** CLI ops, all resolving |
  150. | **Engage — the authentication surface** | Find the login; then, only on explicit confirmation, get **inside** | Detection is passive and free: locate the login form, the password field and the registration page, and classify by **FIELDS** rather than by label — a confirm-password means *register*, an invite code is a **pivot, not an OTP**. Beyond that, `engage_account` creates a **synthetic-persona** account and reads the members area the public page hides (panel, deposit/withdraw flow, affiliate tree, support handles). It refuses a non-synthetic persona, refuses direct egress, and stops at a CAPTCHA. Account creation is outbound, attributable and irreversible — **gated exactly like a sandbox submission** |
  151. | **The case loop** | Judge the **cluster**, not the case — and run to convergence, not to an arbitrary depth | `/clusters` partitions a case into same-operator components *before* anything is judged, showing each binding indicator's **KB-wide prevalence** — so an indicator that binds 3 domains here but sits on 400 KB-wide reads as noise, not an owner link. `/frontier` reports the unresolved gaps: free next seeds already discovered, plus the **deferred metered leads** held for approval. `/loop` runs collect → assess until the case converges; `/reopen` re-opens a converged case on new seeds; `/scope` derives the intake — no-touch class, victim ownership, egress gate — up front instead of assuming it mid-run |
  152. | **Six new collection layers** | Each one closes a specific way the old answer was wrong | **`/liveness`** — a 200 parking/default/suspended/soft-404 page is **not** live and a 404/403/bot-wall is **not** dead; only NXDOMAIN reports dead, and every still-controlled name sets `reuse_watch`. **`/pssl`** — passive SSL runs the historical **cert → IP** direction that recovers an origin from behind a CDN, with the base-rate rail that keeps a shared CDN certificate (915 addresses in live measurement) out of the clustering. **`/paths`** — the URL *path* as an indicator (`path_kit:`) for an operator who rotates hosts and selects the template by directory; a generic path emits nothing. **`/serp`** — Ads Transparency identifies who **paid** (a verified, billed advertiser), with a cloaking probe that has a falsification control. **`/docmeta`** — PDF `/Info` + XMP, EXIF incl. GPS, PNG chunks. **`/victims`** — infer the **access vector** from the victim set |
  153. | **RULE 1 now fires at write time** | The leak gate was enforced at a moment an agent harness rarely reaches | `leakcheck.sh` ran only as a *git* pre-commit hook. Claude Code writes files continuously and commits rarely, so a leaked indicator could sit in the working tree all session — and `git commit --no-verify` skips the gate outright. [`hooks/leakguard.py`](hooks/leakguard.py) moves the check to **PreToolUse on `Write`/`Edit`**, where that flag does not exist. It **does not reimplement the patterns** — it shells out to `leakcheck.sh`, because a second copy would drift and a drifted guard reports clean. Scope is narrow on purpose: it denies only inside a cti-expert checkout on a path git does not ignore, so writing case data into `intel_engine/cases/` — the correct thing to do — is never blocked |
  154. | **The outbound gates moved above the vendored code** | A gate only a bad merge stands between is not a gate | `submit()` refuses without `confirm=True`; the Engage tools refuse a non-synthetic persona. Those gates are real — and they live in `intel_engine/`, which is **vendored**. Re-syncing it is a three-way merge over ~150 files where a deliberate local behaviour is reverted *silently*; three such reversions were caught by hand in this very release. [`hooks/actionguard.py`](hooks/actionguard.py) sits above the tools, in cti-expert's own tree, and fires on the tool **name**. It returns **ask** with a risk briefing, never a hard block — a rail you must disable to work is a rail that gets disabled. Dual-mode tools gate on the **flag**, not the tool: ordinary collection is silent, only `--submit` prompts |
  155. | **The stale-MCP failure has a name now** | A session was driving a four-week-old tool surface with no error anywhere | Claude Code resolves an MCP server's tool list when it **connects** and keeps it for the session. A session was found holding **17** tools while the engine on disk served **46** — and nothing said so; the model simply never saw the new tools and worked around their absence. [`hooks/sessionguard.py`](hooks/sessionguard.py) reports the resolved backend tier at SessionStart and warns when the `@tool` count has **changed since last session**, which is precisely when a cached registration went stale |
  156. | **Installable as a Claude Code plugin** | Skill + commands + MCP + hooks as **one** unit | `register.sh` symlinks the skill, the commands and the MCP server — but it cannot install **hooks**, and that is where the two rails above live. [`.claude-plugin/plugin.json`](.claude-plugin/plugin.json) bundles all four: `/plugin marketplace add <clone>` then `/plugin install cti-expert`. Hook paths use exec form with `${CLAUDE_PLUGIN_ROOT}`, so nothing is hardcoded to one machine and no path is shell-parsed. Both PreToolUse hooks **fail open** — a hook bug must never brick your repo; `audit.sh` and the git pre-commit hook remain the backstop |
  157. | **The OPSEC gate was met, not relaxed** | Upstream grew a submission path, so the test had to be satisfied honestly | ANY.RUN's API is mostly a *submission* API, and the upstream engine added a gated path to it — which failed `tests/test_no_sample_submission.py`, by design. The fix was not to weaken the assertion: the `REQUIRES_ANALYST_CONFIRMATION` marker was placed on `submit()` itself, the four submission-lifecycle endpoints were listed **explicitly** (an unreviewed new key still fails), and the test now demands the marker **and the refusal it claims** — so the marker cannot decay into a magic string. Verified by planting each failure: remove the marker → fail; remove the refusal → fail; restore → pass |
  158. | **The re-sync procedure was the one that breaks the repo** | Documented advice that fails *silently* is worse than none | [STRUCTURE.md](STRUCTURE.md) told the next person to "copy into `intel_engine/`, then re-apply the 5 shims". That is wrong, and its failure has no error message: the collectors keep running, they just stop finding things. It now documents the procedure that holds — classify by **blob identity**, three-way merge off the minimum-distance base, check for the duplicate `@tool` blocks a stale base produces — plus the `zsh` word-splitting trap that turns an unquoted `rsync` exclude list into **no exclusions at all**. The vendored engine's own **17 gates** now ship and run alongside cti-expert's 6, and `audit.sh` gained a check that every path `hooks.json` registers still resolves, because a renamed script disables its hook silently |
  159. </details>
  160. <details>
  161. <summary><b>What's New in v2.7</b></summary>
  162. ## What's New in v2.7
  163. > **The release where the deep pipeline landed.** v2.6 sharpened the collector. v2.7 makes cti-expert a **two-layer system** — a broad collector *plus* a built-in, self-contained intelligence pipeline with a persistent knowledge base — reachable from a cold prompt by one command, and guarded by a gate that checks the repo against its own rules on every push.
  164. | Category | What's New | Details |
  165. |----------|-----------|---------|
  166. | **One skill, two layers** | The deep pipeline is now **built in** — no external backend to stand up | `intel_engine/` vendors the whole **Collect → Correlate → Assess** pipeline: a persistent **knowledge base**, versioned cases, cross-case correlation, calibrated assessment and rendering (WebPivot · IntelAnalysis · IntelGraph · IntelReport · BinaryPivot). `/backend` resolves to **SELF** — nothing to configure, nothing to host. Install the deep-layer deps once with `uv venv && uv pip install -r requirements.txt`. The tree regrouped from **22 top-level directories to 14** behind a single `SKILL.md`. See [STRUCTURE.md](STRUCTURE.md) |
  167. | **8 registered commands** | `/cti` works **from a cold prompt, in any project** | Commands used to require the skill be loaded first. [`scripts/register.sh`](scripts/register.sh) symlinks the skill and `commands/*.md` into `~/.claude/` and writes the per-machine `.mcp.json`, so `/cti`, `/cti-recall`, `/cti-case`, `/cti-pivot`, `/cti-cluster`, `/cti-check`, `/cti-report` and `/cti-status` are available immediately. **There is now one command to remember — `/cti <target>`** — which routes by target type (domain · IP · email · username · phone · wallet · hash · APK) and runs the right chain. Everything else remains a convention command |
  168. | **`--deep` is genuinely parallel** | Sub-agent fan-out on **both** collection *and* assessment | `/cti --deep` spawns one sub-agent per discovered frontier seed — pruned through recall and false-positive control first, **≤6 concurrent, depth-capped at 2 hops**, with `--passive` propagating to every child — then converges them into one case. New here: when convergence yields **2+ clusters**, the *Assess* phase fans out too, one agent per cluster (ACH, confidence, risk, scoped to that cluster), while the **cross-cluster judgment stays central** in the orchestrator. Breadth in parallel; synthesis in one place |
  169. | **IntelX + ANY.RUN** | Leak/darknet selector search and sandbox-observed C2 — with the evidence **graded, not merged** | `intelx_search` reaches pastes, stealer logs, darknet and historical WHOIS. Critically, hits are **graded**: a breach-corpus or stealer-log sighting is **exposure evidence and explicitly not clusterable** — two addresses in one combolist share a *victim pool*, not an operator. Soft selectors are refused locally so a vague name never burns a query unit. `anyrun_lookup` answers what samples carrying an indicator actually *did* — the real endpoints a packed binary contacts — and is **read-only: this skill never submits a sample**, enforced by `tests/test_no_sample_submission.py` |
  170. | **Evidence archiving was silently off** | The wrapper was dropping **22 flags**, including `--archive-missing` | The vendored engine had been left half-migrated — the modular `wp_*` layer was in place but the live collector was still the pre-split 2,274-line monolith, so the harness's `--help` probe filtered out flags the collector no longer advertised. **Evidence archiving was therefore not running at all.** `collect_core` now drops **zero** flags and the supported surface went **19 → 42**. Dropped flags remain visible in the tool result by design: a silent drop is precisely the failure mode this class of bug hides in |
  171. | **Keyless answers stay honest** | Capability accounting — an absent key is never reported as a finding | `wp_capabilities` names **the evidence class each missing key costs**, so a keyless run that finds no siblings reports *"not queried"* — never *"no siblings exist."* Shipping alongside it: **Censys** (keyless CenQL builder, free-plan lookups, monthly credit guard), **asset discovery** (JS bundles, source maps, SPA routes, well-known files), **impersonation hunting**, **JARM** TLS-stack fingerprinting, and a multi-engine `search_pivot`. Every denylist, provider registry and permutation table moved out of code into analyst-tunable `references/*.json` |
  172. | **Nothing dead-ends** | Six identifier types were classified but had no pivot | The spider-map recognised `document`, `image`, `youtube_channel`, `coordinates`, `vin` and `ipv6` — then silently stopped on them. Now wired: **documents** → exiftool + oletools authorship → person/email/org; **images** → EXIF GPS → coordinates, with reverse-image and face search graded **LOW and held pending corroboration, never an auto-merge**; YouTube channels → about-panel links; **coordinates and VIN enrich only, deliberately producing no new seed**, so they cannot invent a false attribution; IPv6 → reverse/passive DNS + ASN, mirroring IPv4. Kept fixed by an invariant test: *every classifiable type must have at least one pivot* |
  173. | **The repo checks itself** | `audit.sh` + CI + a pre-commit leak scan | [`scripts/audit.sh`](scripts/audit.sh) is the gate: every `DISPATCH` op resolves to a real script, all five shared collectors are one canonical file + one re-export shim, the `@tool` count matches the contributor rules, modules byte-compile, tests pass. It runs in **GitHub Actions on every push and PR**, scanning only the PR's *added* lines so curated example values are never re-flagged. [`scripts/install-hooks.sh`](scripts/install-hooks.sh) wires the identifier leak scan as a **pre-commit hook**. Five zero-dependency suites ship with it — collection core, indicator classification, the false-positive ledger, no-sample-submission, and email-candidate containment |
  174. | **Every collection turn leads with a table** | Scannable yield, before the prose | Collection surfaced results only in prose plus the durable file exports; nothing guaranteed a per-domain summary in the conversation itself. A new output rule puts a markdown table **first** on every collection turn — **Resolves · Top pivots · Risk · Cluster · Seen-before** — so you see the yield at a glance instead of reading for it |
  175. | **Portable & framework-free** | No assistant-framework coupling left in the skill | The mandatory voice-notification block is gone and the customization directory moved from a framework-specific path to a neutral `~/.config/cti-expert/` (repo/cwd `.env` still wins). Also in this release: a **Sponsors &amp; Supporters** section — Rexxfield · Hudson Rock · ParanoidLab · ANY.RUN · ZETAlytics · IntelX — and the workflow diagrams rebuilt as **SVG**, including a new end-to-end tool-and-skill sequence diagram |
  176. </details>
  177. <details>
  178. <summary><b>What's New in v2.6</b></summary>
  179. ## What's New in v2.6
  180. | Category | What's New | Details |
  181. |----------|-----------|---------|
  182. | **`/case` runs unattended** | Pivot loop defaults to **`autonomy=auto`**; the new recon commands auto-fire | The spider-map now **expands to closure without approval prompts** — the confidence gate, not a human prompt, is what keeps expansion tight (exact-match links auto-pursue, weak links held, dedup + depth caps unchanged). Depth summaries still print, so the run stays auditable. And the v2.6 recon commands are in the pipeline with **no flags**: `/icp` on every domain/URL/org target, `/cn-corp` on any company name or USCC found, `/iban` on any payment detail, `/hash-id` on every hash (before `/hash`) — and all three discovery-driven ones feed their yields **back into the loop as new seeds**. `/redact` stays **opt-in** (`--redact`): a redacted report is a weaker artifact, so producing one should be a deliberate call. Narrow with `--checkpoint`, `--no-cn`, `--reach balanced\|focused`, `--depth N` |
  183. | **China / Sinophone recon** | `/icp` + `/cn-corp` — the attribution layer Western registries can't reach | **ICP filing (工信部备案)** maps a domain to its registered PRC entity, and the **licence serial** reverse-pivots to every sibling site under the same filing — a same-operator link as strong as a shared GA ID. Then the registry chain: **GSXT** (ground truth) → TianYanCha/QCC/Aiqicha → **信用中国** blacklist → UBO, with USCC validation and revoked-status flags. Adds **Quake (360)** and **ZoomEye** as independent cyberspace indexes, a **Baidu tier** for `/dork-sweep` (tiers 1–4 index almost no CN content), and **CJK variant generation** — pinyin, Simplified↔Traditional and company-name stems — as a new `/pivot-suggest` axis. See [`techniques/china-recon.md`](techniques/china-recon.md) |
  184. | **Fiat payment rails** | `/iban` — bank accounts become selectors, like wallets already were | Most victims never touch crypto — they make a bank transfer. [`iban_analyze.py`](scripts/iban_analyze.py) runs **ISO 7064 mod-97** validation (proving a "bank account" on a payment page is fabricated *without contacting anyone*), decomposes the BBAN into bank/branch/account, and flags **jurisdiction mismatch** — the classic beneficiary-abroad mule pattern. Validated accounts export as `financial/iban` IOCs; invalid ones are recorded as behavioural findings. Covers **VN/SEA non-IBAN rails** too: VietQR/NAPAS BIN, card BIN, e-wallets, BIC. See [`techniques/fiat-payment-osint.md`](techniques/fiat-payment-osint.md) |
  185. | **Shareable reports** | `/redact` — reversible PII redaction | [`redact.py`](scripts/redact.py) replaces PII with **stable numbered placeholders** (`[EMAIL_1]` means one address across the whole case) and writes a **reversible JSON map**, so a report can leave the organisation and still be reconstituted for evidence. Handles `.md`/`.json`/`.csv`; round-trip is byte-exact. Infrastructure is *not* redacted by default — in a CTI report the actor's domains are the analysis, not incidental PII |
  186. | **Analytic rigor** | Probability-anchored likelihood + 5W1H + ACH | Judgments now carry **likelihood terms with probability bands** (*almost no chance* → *almost certain*) reported alongside evidence confidence, because "MODERATE" alone means a 30-point-different thing to writer and reader. `/coverage` gains a **5W1H pass** — a technique matrix measures effort, so a case could score 96% while answering no **Why** or **How**. `/threat-model` now requires an **ACH matrix** for attribution: rival hypotheses scored by *inconsistency*, runner-up named, and the evidence that would change the ranking stated. See [`handbook/analytic-standards.md`](handbook/analytic-standards.md) |
  187. | **Hash typing** | `/hash-id` — before any hash lookup | 32 hex is MD5 **or NTLM** — one is a file hash, the other is credential material, and querying the wrong service returns a confident "unknown sample" that reads as exculpatory. Routes file hashes to MalwareBazaar/VT and credential hashes to `/breach-deep`, never a public cracking service |
  188. </details>
  189. <details>
  190. <summary><b>What's New in v2.5</b></summary>
  191. ## What's New in v2.5
  192. | Category | What's New | Details |
  193. |----------|-----------|---------|
  194. | **Recursive pivoting** | `/case` is a **spider-map** — expands the whole network | `/case` now runs a recursive BFS pivot engine ([`pivot_orchestrator.py`](scripts/pivot_orchestrator.py) + [`engine/pivot-orchestration.md`](engine/pivot-orchestration.md)): every discovered identifier (email/domain/IP/username/wallet/…) becomes a new seed and the relationship graph expands hop-by-hop **until the frontier is exhausted**. Confidence-gated (exact-match links auto-pursue, weak/PII links held), cycle-safe (dedup + depth caps), with **per-depth checkpoints**. Defaults: active · exhaustive · checkpoint-per-depth |
  195. | **Archive IOC harvest** | `/webpivot --harvest` — every selector the site ever exposed | [`wayback_harvest.py`](scripts/webpivot/wayback_harvest.py) runs the full extractor over a domain's **entire Wayback history**, merging **emails, phones, crypto wallets, tracking/verification IDs, SaaS-operator IDs and socials** with first-seen/last-seen — recovering selectors a network later scrubbed. Emits case-schema `indicators[]` straight into the IOC bundle; auto-runs in `/case` for domain/URL targets. `/webpivot` now also extracts **phone numbers** (`tel:` + formatted) as ranked pivot leads |
  196. | **Archive access** | Fetch archived pages Claude Code's WebFetch can't reach | WebFetch is blocked from `web.archive.org` (robots.txt at the fetch layer). [`wayback_fetch.py`](scripts/webpivot/wayback_fetch.py) routes around it — CDX lookup → nearest-snapshot resolve → raw `id_` fetch, with retry/backoff (`--near`, `--list`, `--url-only`, `--json`) |
  197. | **Web pivoting** | `/webpivot` — map the infra behind a page | Favicon **mmh3**, GA/GTM/AdSense IDs, wallets & SaaS-operator tokens from a page's DOM → ranked pivots; same-operator correlation via `/rank-relations` (weighted scoring + noise denylist), `/cert-pivot`, `/pivot-suggest`, `/crypto-balance`, `/email-hygiene`, `/sensitive-paths`. Auto-runs in `/case` for domain/URL targets |
  198. | **Keyless by default** | 100% free — no key, no signup | crt.sh (certificate transparency) + passive DNS + anonymous urlscan **always run**; full pivoting at zero cost, nothing to configure |
  199. | **Premium auto-detect** | Drop in a key → it upgrades itself | `/webpivot` **auto-detects** any premium key you've set (Shodan, Censys, FOFA, Hunter.how, DNSLytics, SecurityTrails, urlscan-PRO, WhoisXML) and unlocks its higher tier — no flag, no re-run; a missing/bad key degrades to a note, never breaks the run. Manage keys with `/apikeys` |
  200. | **Attack surface** | `/appliance-scan` — edge/VPN appliance → KEV mapping | Passive-first fingerprint of internet-facing Citrix/F5/Cisco/Ivanti/Forti/Palo Alto/Exchange appliances (Shodan InternetDB/Censys) → matched **CISA KEV/CVE** list; feeds `/vuln-check` + `/threat-model` |
  201. | **Identity fabric** | `/saas-map` — SaaS tenancy + IdP surface | DNS-TXT tenancy tokens (Google/Atlassian/Zscaler/Salesforce/Workday…), non-Microsoft IdP fingerprint (Okta/Auth0/OneLogin/Ping/Keycloak/ADFS), unauthenticated API/GraphQL/OpenAPI-spec discovery |
  202. | **Credentials** | Read-only liveness validation | A discovered key is confirmed live via identity-only endpoints (AWS STS, GitHub scopes, Slack `auth.test`, `…/v1/models`) — never a mutating call — upgrading it to CRITICAL with account/scope evidence |
  203. | **Integrity** | Evidence-gated analysis | every asserted claim cites a resolvable finding; untrusted collected data is tagged, never executed |
  204. | **Recon** | Native `asn` command | Keyless IP/ASN/domain lookups (ipwho.is + RDAP) on Windows; full nitefood/asn auto-installed on Linux/macOS/WSL |
  205. | **System tools** | `whois` + `dig` + `asn` auto-install on Windows | winget `Microsoft.Sysinternals.Whois` + `ISC.Bind`; previously manual steps |
  206. | **Reliability** | Windows PowerShell 5.1 hardening | Fixes native-stderr script aborts, the `OSArchitecture` probe crash, and maigret via `uv tool --force`; installs clean on WinPS 5.1 |
  207. | **Packaging** | Auto-PATH for CLI tools | `~/.local/bin` (uv tools + `asn`) added to PATH automatically — current session **and** persistent |
  208. </details>
  209. <details>
  210. <summary><b>What's New in v2.4</b></summary>
  211. ## What's New in v2.4
  212. | Category | What's New | Details |
  213. |----------|-----------|---------|
  214. | **Platform** | Cross-platform OS detection (Windows/macOS/Linux) | OS-aware auto-install; self-healing DOCX (UTF-8 + auto-located pandoc) |
  215. | **Packaging** | uv-first toolchain | `uv venv` / `uv pip` / `uv tool`; PEP 723 `uv run` zero-setup scripts; pip/pipx/venv fallback |
  216. | **Portability** | Cross-agent support | Runs in Claude Code **and** OpenAI Codex via `AGENTS.md` + a ready-to-copy `/cti-expert` Codex prompt |
  217. | **CTI** | Infostealer-log analyzer (`/stealer-log`) | Family ID, victim-vs-operator profiling, cross-log actor correlation, IOC + raw-artifact extraction |
  218. | **Recon** | Admin / sensitive-endpoint detection | Subdomain-prefix + path + CJK classifier (`admin`, `adm`, `kef`, `ador`, `panel`…) |
  219. | **Collection** | agent-browser integration | Primary interactive browser ([vercel-labs](https://github.com/vercel-labs/agent-browser)): CDP, accessibility-tree snapshots, screenshots; complementary to Scrapling, no API key for core |
  220. | **Collection** | Media & vision analysis | Image/A-V evidence is content-analyzed: OCR + sign/landmark/logo/face read and A/V transcription, with FFmpeg keyframe/audio preprocessing; extracted text/GPS/entities re-enter the pivot loop. **Keyless by default** — the agent's own vision + `tesseract` OCR + local Whisper; a `GEMINI_API_KEY` upgrades quality/scene-reading. Standalone (`npx` multix + `ffmpeg`). See [`techniques/media-vision-analysis.md`](techniques/media-vision-analysis.md) |
  221. | **Reliability** | Fresh-VPS install hardening + CI | root/sudo + prereq bootstrap; smoke test + GitHub Actions on a minimal root Ubuntu container |
  222. <details>
  223. <summary><b>What's New in v2.3</b></summary>
  224. ## What's New in v2.3
  225. | Category | What's New | Details |
  226. |----------|-----------|---------|
  227. | **WHOIS** | Universal WHOIS for all TLDs | whoisdomain + CLI + Whoxy API; .vn, .th, .sg, .kr, 27+ ccTLD servers |
  228. | **WHOIS** | Reverse & historical WHOIS (free) | Whoxy reverse API, historical lookup, ViewDNS |
  229. | **Web Collection** | Scrapling adaptive scraping | 3-tier: static → anti-bot → JS rendering; headless auto-open |
  230. | **Web Collection** | Headless browser auto-open default | JS-heavy sites auto-detected and rendered via DynamicFetcher |
  231. | **Orchestration** | AgentFlow parallel enrichment | DAG-based parallel pivot expansion for 3+ subjects |
  232. | **Performance** | HTML parsing ~2ms | Scrapling parser replaces slow HTTP scraping |
  233. | **Platform** | Python 3.10+ minimum | Required by Scrapling and AgentFlow |
  234. <details>
  235. <summary><b>What's New in v2.2</b></summary>
  236. ## What's New in v2.2
  237. | Category | What's New | Details |
  238. |----------|-----------|---------|
  239. | **Image Forensics** | Face search, reverse image, manipulation detection, AI geolocation | FaceCheck.id, TinEye, FotoForensics, Forensically, picarta.ai, GeoSpy, Pic2Map |
  240. | **Blockchain** | Crypto wallet tracing, transaction graphs, scam detection | Blockchair, Etherscan, WalletExplorer, OXT.me, Chainabuse, Breadcrumbs |
  241. | **Transport** | Aircraft tracking (unfiltered), vessel AIS, vehicle VIN lookup | ADS-B Exchange, Flightradar24, Marine Traffic, VesselFinder, NICB VINCheck |
  242. | **Darknet** | Tor search, ransomware monitoring, onion service discovery | Ahmia.fi, onionsearch, DarknetLive, ransomwatch |
  243. | **Social Media** | Reddit, Instagram, TikTok, Telegram investigation | Osintgram, instaloader, toutatis, RedditMetis, TGStat, TelegramDB, Bellingcat TikTok Timestamp |
  244. | **People Search** | US people search engines, free reverse lookups | TruePeopleSearch, FastPeopleSearch, IDCrawl, That's Them |
  245. | **Mega-Dorks** | 11 cross-platform Google dork templates covering 73 unique domains | Social, Telegram ecosystem, dev platforms, forums, paste sites, darknet, breach DBs, business, image, messaging, jobs |
  246. | **IoT** | Webcam directories, IoT device search | Insecam, Thingful |
  247. <details>
  248. <summary><b>What's New in v2.1</b></summary>
  249. | Category | New Commands | What It Does |
  250. |----------|-------------|--------------|
  251. | **Intelligence** | `/render threat-path`, `/render attack-surface` | Attack path flow + infrastructure exposure visualization |
  252. | **Intelligence** | `/snapshots`, `/diff` | Wayback Machine snapshots and version diffing |
  253. | **Intelligence** | `/drift`, `/report ioc` | Temporal risk tracking + IOC export (STIX 2.1) |
  254. | **UX** | `/onboard`, `/clarify`, `/quality` | First-time tutorial, finding explanation, quality scoring |
  255. | **UX** | `/blind-spots`, `/source-check` | Gap analysis + batch URL verification |
  256. | **UX** | `/workspace diff` | Compare two saved investigation sessions |
  257. | **Data Model** | Source Reliability A-F | Complements trust scores with source-level grading |
  258. | **Data Model** | 4 new entity types | Device, Image, Crypto Address, Custom |
  259. | **Data Model** | HIGH conflict severity | 4-level severity: CRITICAL/HIGH/NOTABLE/MINOR |
  260. </details>
  261. </details>
  262. </details>
  263. </details>
  264. <br>
  265. ---
  266. <br>
  267. ## Installation
  268. > **Recommended:** Use **Claude Code CLI** — it gives you the full terminal workflow, persistent sessions, and direct skill invocation. [Download here](https://docs.anthropic.com/en/docs/claude-code/overview) or run `npm install -g @anthropic-ai/claude-code`.
  269. ### Why Claude Code CLI?
  270. The entire CTI Expert workflow is optimized for Claude Code CLI. The CLI gives you:
  271. - **Persistent sessions** — investigations survive terminal restarts via `/workspace save`
  272. - **Full tool access** — file writes, Python scripts, DOCX generation, all run natively
  273. - **Skill invocation** — type `/cti-expert` directly in the terminal, no browser required
  274. - **Background agents** — parallel enrichment via AgentFlow works best with the CLI
  275. #### 🖥️ Where to run it — the CLI is best for this skill
  276. > [!IMPORTANT]
  277. > CTI Expert is **execution-heavy**: it runs `uv`/Python, installs OSINT tools, writes `.md`/`.json`/`.csv`/`.html`/`.docx`/`.pdf` reports + IOC bundles, reaches many external sites, and saves case workspaces. What matters is a **real local shell + persistent files + open network** — a **CLI or local desktop agent** gives you that; an ephemeral **cloud sandbox does not**. This applies equally to **Claude** and **Codex**.
  278. | Environment | Running cases | Why |
  279. |---|---|---|
  280. | **Claude Code CLI** · **Codex CLI** | ✅ **Best** | Real shell, persistence, background tasks, open network — what the skill is built for |
  281. | **Claude Code Desktop** · **Codex IDE extension** | ✅ Great | Same local execution; nicest for **reading** rendered reports, charts & diagrams |
  282. | **claude.ai/code (web)** · **Codex cloud / ChatGPT web** | ⚠️ Limited | Reasoning & query generation work, but files don't persist to your disk and outbound network is often restricted |
  283. > [!TIP]
  284. > **Run investigations in a CLI** (Claude Code or Codex); open the generated `.docx`/report in a Desktop/IDE window if you prefer reading there. Use web/cloud surfaces only for analyst-reasoning, not execution-heavy recon.
  285. ---
  286. ### Step 1 &mdash; Install Claude Code CLI
  287. ```bash
  288. npm install -g @anthropic-ai/claude-code
  289. ```
  290. > Requires Node.js 18+. Full docs: [docs.anthropic.com/en/docs/claude-code/overview](https://docs.anthropic.com/en/docs/claude-code/overview)
  291. ---
  292. ### Step 2 &mdash; Clone + All-in-One Installer
  293. The installer handles everything: Python dependencies, system tools (`whois`, `dig`, `asn`, `jq`, `exiftool`), OSINT tools (`maigret`, `sherlock`, `holehe`, `h8mail`, and more), and optional headless browser + Go tools. It is **powered by [uv](https://docs.astral.sh/uv/)** (Astral's ultra-fast Rust package manager) — the script bootstraps uv, then uses `uv venv` / `uv pip` / `uv tool` for all Python installs, falling back to `pip`/`pipx`/`venv` only if uv can't be installed. Use `install.ps1` on Windows (PowerShell) or `install.sh` on macOS/Linux/Git Bash/WSL.
  294. <table>
  295. <tr>
  296. <th>Platform</th>
  297. <th>Command</th>
  298. </tr>
  299. <tr>
  300. <td><b>Linux / macOS</b></td>
  301. <td>
  302. ```bash
  303. git clone https://github.com/7onez/cti-expert.git ~/.claude/skills/cti-expert
  304. bash ~/.claude/skills/cti-expert/scripts/install.sh
  305. ```
  306. </td>
  307. </tr>
  308. <tr>
  309. <td><b>Windows (Git Bash or WSL)</b></td>
  310. <td>
  311. ```bash
  312. git clone https://github.com/7onez/cti-expert.git ~/.claude/skills/cti-expert
  313. bash ~/.claude/skills/cti-expert/scripts/install.sh
  314. ```
  315. </td>
  316. </tr>
  317. <tr>
  318. <td><b>Windows (PowerShell — native)</b></td>
  319. <td>
  320. ```powershell
  321. git clone https://github.com/7onez/cti-expert.git "$env:USERPROFILE\.claude\skills\cti-expert"
  322. powershell -ExecutionPolicy Bypass -File "$env:USERPROFILE\.claude\skills\cti-expert\scripts\install.ps1"
  323. ```
  324. </td>
  325. </tr>
  326. </table>
  327. > **Windows users:** `install.ps1` is a **full native installer** (winget system tools + Python venv + OSINT tools) — no Git Bash or WSL required. It accepts the same `-Headless`, `-Go`, `-Telegram`, `-GHunt`, and `-All` flags (e.g. `install.ps1 -All`). Git Bash / WSL users can run `install.sh` instead. The DOCX generator self-heals UTF-8 output and auto-locates pandoc, so reports build on Windows with no extra environment setup. The skill itself detects the OS at runtime and installs any missing tool with the right manager (`winget` / `brew` / `apt`) — see `scripts/platform-setup.md`.
  328. ---
  329. ### Installer Options
  330. **macOS / Linux / Git Bash / WSL:**
  331. ```bash
  332. bash scripts/install.sh # Core: Python deps + system tools + OSINT tools
  333. bash scripts/install.sh --headless # + Scrapling headless browser (~200MB Chromium)
  334. bash scripts/install.sh --go # + Go tools (subfinder, amass, gau, gitleaks, httpx)
  335. bash scripts/install.sh --telegram # + Telegram stack (telethon, opentele2) — research account only
  336. bash scripts/install.sh --ghunt # + GHunt CLI (uv tool / pipx)
  337. bash scripts/install.sh --all # + Everything above
  338. ```
  339. **Windows (PowerShell):**
  340. ```powershell
  341. powershell -ExecutionPolicy Bypass -File scripts\install.ps1 # Core
  342. powershell -ExecutionPolicy Bypass -File scripts\install.ps1 -Headless # + Scrapling headless browser
  343. powershell -ExecutionPolicy Bypass -File scripts\install.ps1 -Go # + Go tools
  344. powershell -ExecutionPolicy Bypass -File scripts\install.ps1 -Telegram # + Telegram stack
  345. powershell -ExecutionPolicy Bypass -File scripts\install.ps1 -GHunt # + GHunt CLI
  346. powershell -ExecutionPolicy Bypass -File scripts\install.ps1 -All # + Everything above
  347. ```
  348. | Flag | What it installs | Size |
  349. |------|-----------------|------|
  350. | *(none)* | Python packages, whois, dig, asn, jq, exiftool, maigret, sherlock, holehe, h8mail, theHarvester, waymore, xeuledoc, agentflow | ~50 MB |
  351. | `--headless` | Scrapling StealthyFetcher + DynamicFetcher + Chromium, Playwright, agent-browser, CloakBrowser (then `cloakbrowser login`) | +200 MB |
  352. | `--go` | subfinder, amass, gau, gitleaks, httpx, trufflehog, phoneinfoga | +150 MB |
  353. | `--telegram` | telethon, opentele2, qrcode[pil], python-socks[asyncio] into the venv (then, in your terminal with a dedicated research account: `python3 scripts/backend/intel.py telegram login --qr --profile desktop`; TELEGRAM_API_ID/HASH optional with opentele2; against Telegram's API terms) | +10 MB |
  354. | `--ghunt` | GHunt as an isolated CLI (then `ghunt login` with a research Google account) | +30 MB |
  355. | `--all` | Everything (opt out per component: `NO_CLOAKBROWSER=1`, `NO_TELEGRAM=1`, `NO_GHUNT=1`) | ~450 MB |
  356. ---
  357. ### Step 3 &mdash; Register the commands with Claude Code and every other agent
  358. `install.sh` installs the OSINT *tools*. This one-time step wires the **skill, the 9 `/cti*` slash commands, and the MCP tools** into Claude Code so they work from a cold prompt in any project — it symlinks `commands/*.md` into `~/.claude/commands/` and writes the per-machine `.mcp.json`. It then registers `/cti` with every other coding agent it detects on the machine (see [Use in other agents](#use-in-other-agents-codex--chatgpt--cursor--gemini--omp--pi-)). It's idempotent, so it's safe to re-run after a `git pull`.
  359. ```bash
  360. # Register the skill + 9 commands + write the per-machine .mcp.json + other detected agents
  361. bash ~/.claude/skills/cti-expert/scripts/register.sh
  362. # Recommended: install the built-in deep-pipeline (intel_engine) deps once
  363. cd ~/.claude/skills/cti-expert && uv venv && uv pip install -r requirements.txt
  364. ```
  365. > **Windows (native PowerShell):** run `register.sh` from **Git Bash** — its `~` is `%USERPROFILE%`, where Windows-native Claude Code, Codex, Cursor, Gemini … read their config. **Not WSL:** inside WSL `~` is the Linux home, so a WSL run registers only agents that themselves run inside WSL. Git Bash's `ln -s` silently copies by default, so `register.sh` asks for real Windows symlinks (they need **Developer Mode** or an admin shell) and, when Windows refuses, writes pointer stubs instead — either way the commands follow `git pull`; enable Developer Mode and re-run to upgrade stubs to links. Stubs and the other agents need Python 3.10+ — the repo `.venv` (`.venv\Scripts\python.exe`), `python3`, `python` or the `py` launcher, tried in that order; if none runs, `register.sh` prints **Registration INCOMPLETE** and exits 1 instead of claiming success. The other-agent step alone also runs natively with `py scripts\register_agents.py`. Then, on every platform, **restart each agent** so the skill and commands load at startup.
  366. > **Uninstall:** `bash scripts/register.sh --uninstall` removes the Claude Code links that point into this clone and every stub `register_agents.py` wrote. A same-named command or skill you wrote yourself is never touched — on install or on uninstall.
  367. <details>
  368. <summary><b>Alternative — install as a Claude Code plugin (skill + commands + MCP + safety hooks in one unit)</b></summary>
  369. `register.sh` wires the skill, the commands and the MCP server, but it cannot install **hooks** — and two of cti-expert's safety properties are enforced there:
  370. | Hook | What it does |
  371. |---|---|
  372. | `hooks/leakguard.py` | **PreToolUse** on `Write`/`Edit` — scans the pending payload with `scripts/leakcheck.sh` and **denies** a write that would put case data into a *tracked* file. The git pre-commit hook only fires at commit time and `git commit --no-verify` skips it; this fires at write time, where that flag does not exist. Writes into the git-ignored case stores, and writes outside a cti-expert checkout, are untouched. |
  373. | `hooks/actionguard.py` | **PreToolUse** on the outbound actions — `engage_account`, `harvest_authenticated`, `anyrun_submit`, any `--submit` — returns **ask** with a risk briefing. The tools already refuse without confirmation, but that code is *vendored*; this gate lives in cti-expert's own tree, so a bad engine sync cannot delete it. Detection (`detect_login`, `url_paths`, `passive_ssl`) and ordinary collection are **not** gated. |
  374. | `hooks/sessionguard.py` | **SessionStart** — reports the resolved backend tier, and warns when the engine's `@tool` count has changed since the last session. Claude Code caches an MCP server's tool list at *connect* time, so a stale registration silently drives an old surface with no error. |
  375. ```bash
  376. # from a local clone
  377. claude
  378. /plugin marketplace add ~/.claude/skills/cti-expert
  379. /plugin install cti-expert
  380. ```
  381. Then restart Claude Code. Verify with `/hooks` (three entries) and `/mcp` (the `intel` server).
  382. Everything is behaviour-preserving: both `PreToolUse` hooks **fail open** on an internal error, so a hook bug can never brick your repo — `scripts/audit.sh` and the git pre-commit hook remain the backstop.
  383. </details>
  384. ---
  385. ### Verify Installation
  386. ```bash
  387. claude # open Claude Code CLI, then type:
  388. /cti-status # health check — backend tier, MCP tools, API-credit balances
  389. /cti example.com # …or just start investigating
  390. ```
  391. > `/cti-status` confirms the backend, MCP tools, and API-credit balances in one shot. If the `/cti*` commands aren't recognized, re-run **Step 3** (`register.sh`) and restart the agent; `python3 scripts/register_agents.py --list` shows which agents were detected and where each command lives. In Codex the command is `$cti`. You can also type `/cti-expert` (Claude Code) to load the skill directly, then describe your goal in plain English.
  392. ---
  393. ### Use in other agents (Codex · ChatGPT · Cursor · Gemini · omp · pi …)
  394. CTI Expert is portable: the analyst logic is plain Markdown and the scripts are OS-detecting Python/shell, so it runs in any [`AGENTS.md`](AGENTS.md)-aware coding agent, not just Claude Code. [`scripts/register_agents.py`](scripts/register_agents.py) (run by `register.sh`) registers `/cti` with every agent it detects:
  395. | Agent | Registered as | Invoke |
  396. |---|---|---|
  397. | oh-my-pi (omp) | `~/.omp/agent/commands/cti*.md` | `/cti example.com` |
  398. | pi | `~/.pi/agent/prompts/cti*.md` | `/cti example.com` |
  399. | OpenCode | `~/.config/opencode/commands/cti*.md` | `/cti example.com` |
  400. | Gemini CLI | `~/.gemini/commands/cti*.toml` | `/cti example.com` |
  401. | Qwen Code | `~/.qwen/commands/cti*.md` | `/cti example.com` |
  402. | Factory droid | `~/.factory/commands/cti*.md` | `/cti example.com` |
  403. | Windsurf | `~/.codeium/windsurf/global_workflows/cti*.md` | `/cti example.com` |
  404. | Cursor · GitHub Copilot CLI | agent skill `~/.agents/skills/cti/` | `/cti example.com` |
  405. | OpenAI Codex (CLI / IDE) | the same agent skill | `$cti example.com` — Codex removed custom prompts in 0.117 and invokes skills with `$`, never `/` |
  406. | ChatGPT desktop app | the same agent skill | `@cti example.com` |
  407. ```bash
  408. python3 scripts/register_agents.py --list # which agents are detected, and where each stub goes
  409. python3 scripts/register_agents.py --host all # register for every agent, installed or not
  410. python3 scripts/register_agents.py --host gemini,pi # just these
  411. python3 scripts/register_agents.py --uninstall # remove every stub this script wrote
  412. ```
  413. - Command-file agents get all 9 commands; the shared-skill agents get `/cti` (its single entry point). Each stub is a short pointer to this clone's `commands/<cmd>.md` and `SKILL.md` by absolute path, so a `git pull` updates every agent at once — re-run only after moving the clone or adding a command. A same-named file you wrote yourself is never overwritten or removed.
  414. - **[`AGENTS.md`](AGENTS.md)** is the cross-agent runtime contract (OS detection, uv, paths). Codex auto-concatenates it from the repo root; you can also reference it from `~/.codex/AGENTS.md`.
  415. - **MCP tools** for Codex and omp: `python3 scripts/backend/intel.py mcp --host codex|omp|all --write`. Without them every command still runs through the CLI dispatcher (Tier 2).
  416. - **Plain ChatGPT on the web (no code execution):** there is no slash-command registry; the reasoning, query generation, and report drafting all work (load `SKILL.md`/`AGENTS.md` as instructions or Custom-GPT knowledge); only local steps (DOCX build, CLI tool runs) need a code-capable harness.
  417. > Paths are resolved relative to the skill directory (the folder containing `SKILL.md`), so nothing assumes the Claude-specific `~/.claude/skills/` location.
  418. ---
  419. ### Alternative &mdash; Claude Code Desktop (macOS / Windows)
  420. > Download: [claude.ai/download](https://claude.ai/download) &mdash; available for **macOS** and **Windows**
  421. **Step-by-step (no terminal needed):**
  422. 1. **Install Claude Code Desktop** &mdash; Download from [claude.ai/download](https://claude.ai/download) and install the app
  423. 2. **Download CTI Expert** &mdash; Go to the [GitHub repository](https://github.com/7onez/cti-expert), click the green **"Code"** button, then select **"Download ZIP"**
  424. 3. **Extract to your skills folder** &mdash; Unzip the downloaded file, then move the extracted folder to your skills directory and rename it to `cti-expert`:
  425. | Platform | How to navigate |
  426. |----------|----------------|
  427. | **macOS** | Open **Finder** &rarr; Press **Shift + Cmd + G** &rarr; Type `~/.claude/skills/` &rarr; Press **Go** &rarr; Move the folder here |
  428. | **Windows** | Open **File Explorer** &rarr; Type `%USERPROFILE%\.claude\skills\` in the address bar &rarr; Press **Enter** &rarr; Move the folder here |
  429. > **Note:** If the `skills` folder does not exist, create it inside the `.claude` folder first.
  430. 4. **Run the installer + register** &mdash; Open the Claude Code Desktop terminal and run:
  431. ```bash
  432. bash ~/.claude/skills/cti-expert/scripts/install.sh # OSINT tools
  433. bash ~/.claude/skills/cti-expert/scripts/register.sh # skill + 9 commands + MCP + other agents
  434. ```
  435. Or on Windows PowerShell (Python deps + the other agents; run `register.sh` from Git Bash — not WSL — for the Claude Code links):
  436. ```powershell
  437. pip3 install -r "$env:USERPROFILE\.claude\skills\cti-expert\scripts\requirements.txt"
  438. py "$env:USERPROFILE\.claude\skills\cti-expert\scripts\register_agents.py" # /cti for the other agents
  439. ```
  440. 5. **Restart Claude Code Desktop** &mdash; Close and reopen the app
  441. 6. **Verify** &mdash; Type `/cti-status` in the chat to confirm the skill and commands loaded (or `/cti-expert` to load the skill directly)
  442. <details>
  443. <summary><b>System Requirements</b></summary>
  444. <br>
  445. | Requirement | Version | Purpose |
  446. |-------------|---------|---------|
  447. | [Claude Code CLI](https://docs.anthropic.com/en/docs/claude-code/overview) | Latest | **Recommended** terminal runtime |
  448. | [Claude Code Desktop](https://claude.ai/download) | Latest | GUI runtime (macOS/Windows) |
  449. | Node.js | 18+ | Required by Claude Code CLI |
  450. | [uv](https://docs.astral.sh/uv/) | Latest | **Recommended** — bootstrapped by the installer; manages Python, venv, packages & CLI tools |
  451. | Python | 3.10+ | DOCX report generation, Scrapling, AgentFlow (uv can install this for you) |
  452. | pip packages | See `requirements.txt` | Charts, diagrams, styling |
  453. | git | Any | Clone the repository |
  454. </details>
  455. <br>
  456. ---
  457. <br>
  458. ## Quick Start
  459. > [!TIP]
  460. > **New here, or on a fresh VPS?** The steps below take you from an empty box to a finished case in a couple of minutes. No API keys needed for core.
  461. ```bash
  462. # 1 — install the tools (Linux/macOS): Python + ffmpeg + Node + all OSINT tools
  463. git clone https://github.com/7onez/cti-expert.git ~/.claude/skills/cti-expert
  464. bash ~/.claude/skills/cti-expert/scripts/install.sh
  465. # 2 — register the skill + 9 /cti* commands + MCP with Claude Code and every other agent found, then RESTART them
  466. bash ~/.claude/skills/cti-expert/scripts/register.sh
  467. # (Windows: run register.sh from Git Bash, not WSL — WSL's ~ is the Linux home. Codex: the command is $cti.)
  468. # 3 — investigate anything (keyless, runs to closure):
  469. /cti example.com
  470. # 4 — optional: unlock reverse pivots + image vision
  471. /apikeys set shodan <KEY> # host / favicon reverse
  472. /apikeys set gemini <KEY> # image OCR, sign/landmark read, A-V transcription
  473. ```
  474. Windows, `--all`, and other-agent variants (Codex · ChatGPT · Cursor · Gemini · omp · pi …): see [Installation](#installation).
  475. ### How commands work — read this first
  476. There is **one command to remember: `/cti <target>`.** It looks at what you gave it — a domain, IP, email, username, phone, wallet, hash, or APK — and runs the right chain automatically. That's usually all you need.
  477. Under it sit **9 registered commands** that Claude Code and every command-file agent `register.sh` wired (omp · pi · OpenCode · Gemini CLI · Qwen Code · Factory droid · Windsurf) recognize from a cold prompt in any project, with no need to load the skill first. Cursor, Copilot CLI, Codex (`$cti`) and the ChatGPT desktop app (`@cti`) get the entry point `/cti` itself:
  478. | Command | What it does |
  479. |---------|--------------|
  480. | **`/cti <target>`** | **Entry point** — routes by target type and runs the whole chain |
  481. | `/cti-recall <seed>` | *"Have I seen this before?"* — check against every prior case. **Run this first.** |
  482. | `/cti-case <ID> <seeds>` | Full deterministic pipeline: collect → ingest → cluster → assess |
  483. | `/cti-pivot <url\|ip>` | Collect pivot artifacts from one target |
  484. | `/cti-cluster <domain>` | Expand & correlate an existing case |
  485. | `/cti-check <indicator>` | False-positive control — real operator link, or shared noise? |
  486. | `/cti-report <ID>` | Render the relationship graph + a polished PDF/DOCX |
  487. | `/cti-status` | Health check — backend, MCP tools, API-credit balances |
  488. | `/cti-proxy [op]` | Egress proxy / rotation pool for **all** outbound calls |
  489. Every *other* command on this page (`/case`, `/webpivot`, `/report`, `/sweep`…) is a **convention command**: shorthand that works once the skill is loaded — via `/cti`, or by typing `/cti-expert` to open the skill directly. At a cold prompt, reach for a registered command above, or just describe your goal in plain English — it works identically.
  490. ### First 60 seconds — turn on full power
  491. Before your first real case, run these once in Claude Code so you're not silently running keyless or degraded:
  492. ```bash
  493. /cti-expert # load the skill first — the four below are convention commands
  494. /onboard # interactive first-run guide
  495. /capabilities # which evidence classes are unavailable now + the free path that substitutes
  496. /backend # confirm Tier-1 typed MCP (103 tools) is live, not Tier-3 stateless
  497. /apikeys status # keyless works; add free/paid keys to unlock reverse pivots
  498. ```
  499. ### 1 &mdash; Investigate anything
  500. ```bash
  501. /cti example.com # domain → full pipeline
  502. /cti user@domain.com # email → breach + infrastructure + cross-platform
  503. /cti @username # handle → 3000+ platform enumeration, then pivot
  504. /cti 185.1.1.1 # IP → ASN, co-tenancy, open ports, passive DNS
  505. /cti ./trader.apk # file → static IOCs, clustered with the web infra
  506. ```
  507. > `/cti` picks the right techniques for the target, then expands the pivot graph **to closure — no approval prompts.** Add `--deep` for parallel sub-agent fan-out, `--quick` for a single pass, or `--passive` for hostile targets (no live contact). Default output: the base bundle (Markdown + JSON + CSV + IOC bundle) plus your chosen presentation report (PDF, DOCX, HTML, or all).
  508. ### 2 &mdash; Run a case end-to-end
  509. ```bash
  510. /cti-recall example.com # always first — have we seen this seed before?
  511. /cti-case CASE-0001 example.com # full pipeline on one or more seeds
  512. /cti-cluster CASE-0001 # expand: peers, shared indicators, TLS overlap
  513. /cti-report CASE-0001 # deliver: relationship graph + house report (PDF/DOCX)
  514. ```
  515. ### 3 &mdash; Guided Flows
  516. > The commands below are **convention commands** — type them once the skill is loaded.
  517. ```bash
  518. /flow person # Person investigation workflow
  519. /flow domain # Domain reconnaissance workflow
  520. /flow image # Image verification workflow
  521. ```
  522. ### 4 &mdash; Targeted Reconnaissance
  523. ```bash
  524. /sweep @username # Multi-vector recon on handle
  525. /query example.com # 12-15 advanced search queries
  526. /username johndoe # Platform enumeration (3000+)
  527. /email-deep user@domain.com # Deep email investigation
  528. /subdomain example.com # Certificate transparency + brute-force
  529. /github-osint github.com/org/repo # GitHub profiles, repos, code, commits, forks
  530. /threat-check 185.1.1.1 # IP/domain/URL threat intelligence
  531. /scam-check suspicious-site.xyz # Phishing/scam domain check
  532. /breach-deep user@domain.com # Multi-source breach lookup
  533. ```
  534. ### 5 &mdash; Analysis & Assessment
  535. ```bash
  536. /exposure domain.com # Composite risk score (0-100)
  537. /threat-model # Build threat model from findings
  538. /validate # Verify all findings
  539. /coverage # Check investigation completeness
  540. ```
  541. ### 6 &mdash; Reporting
  542. ```bash
  543. /report # Technical INTSUM report
  544. /report brief # Executive summary
  545. /brief # Plain-language summary
  546. /workspace save # Save case workspace state (resume later)
  547. ```
  548. ### 7 &mdash; Full pipeline & full harness (deepest mode)
  549. `/cti` is the everyday entry. For a **persisted, versioned case** that runs to convergence and correlates across every prior case, use the deep layer. Run `/backend` first — it reports the tier (Tier-1 typed MCP `intel-harness` = 103 tools → Tier-2 CLI → Tier-3 stateless); the bundled installer already provisions it (`SELF`, in-repo, no external setup).
  550. > **`/case` already persists a versioned case — with NO extra egress on the persist step.** When `/backend` is live (Tier 1/2), a `/case` run **reuses the pivots it just collected** (it does **not** re-fetch): it writes them under `$SKILL_DIR/intel_engine/cases/<CASE-ID>/raw/` and runs the deterministic pipeline in reuse mode — `intel.py pipeline open <CASE-ID> <seeds> --no-collect` — which skips the live fetch and runs the whole chain (ingest → recall → risk → clusters → `case_graph.json` → ICD-203 `assessment.md`) over that raw. The complete versioned case lands at `$SKILL_DIR/intel_engine/cases/<CASE-ID>/` (**not your CWD**) and is correlated across every prior case. **If the case has not converged** (`intel.py convergence <ID>` reports status ≠ `converged`, or `/frontier` still lists open leads) **and** posture is active, `/case` then **auto-escalates to the `/harness` deepening loop** — keyless-first (it uses the CLI's own model), egress **hard-gated** on hostile infra (the harness `audit.py` denies outbound collection on `hostile=True`), `--no-harness` opts out. Plain `/pipeline open` (without `--no-collect`) and `/cti-case` still **collect** — they re-fetch every seed directly — so run those by hand only for a **fresh** case with no prior collection, after setting the egress posture (`/scope`, `/cti-proxy`, or `--passive` for hostile infra). Tier 3 / no host seeds → `/case` skips the handoff silently.
  551. ```bash
  552. /cti-expert # load the skill first (the commands below are convention commands)
  553. /backend # confirm the deep layer is live (Tier-1 = 103 MCP tools)
  554. # A — full DETERMINISTIC pipeline (no LLM key, reproducible, persisted to cases/<ID>/)
  555. printf "example.com\nsibling.com\n" > seeds.txt
  556. /pipeline open CASE-0001 seeds.txt # collect → ingest → recall → risk → cluster → ICD-203 assessment
  557. # + MO-neighbour origin pivot · measured entitlement · IntelX auto-fire
  558. # --free-only: no metered credits · --whois-history purchase: registrant eras
  559. /clusters CASE-0001 # judge by same-operator cluster, with KB-wide prevalence
  560. /frontier CASE-0001 # unresolved gaps: free next seeds + deferred metered leads
  561. # B — full HARNESS (agent-driven; also auto-escalates inside /case when a case has not converged)
  562. /harness open CASE-0001 example.com sibling.com
  563. /harness continue CASE-0001 --depth 4 # Collect→Correlate→Assess, cross-case, until the frontier empties
  564. /harness status CASE-0001 # status needs no key
  565. /loop CASE-0001 # or: collect↔assess repeatedly until it converges
  566. /cti-report CASE-0001 # deliver: relationship graph + house report (editorial PDF/DOCX from the case dir)
  567. ```
  568. > **Pipeline vs harness:** `/pipeline` is the deterministic bread-and-butter chain — no LLM key, byte-reproducible. `/harness` is the **agent-driven** whole-case orchestration that reasons across cases to convergence (the deepest mode), and **auto-escalates inside `/case`** when the deterministic pipeline hasn't converged and posture is active (`--no-harness` opts out). **Keyless-first:** run interactively in Claude Code it uses the CLI's own model on your subscription (no separate LLM key); `HARNESS_BACKEND=local` (Ollama/vLLM/LM Studio, keyless) or an API key are needed only for unattended SDK `continue`. Both persist under `cases/<ID>/`; the registered cold-prompt alias `/cti-case <ID> <seeds>` runs the deterministic pipeline.
  569. <br>
  570. ---
  571. <br>
  572. ## Best Practices
  573. Habits that keep investigations fast, cheap, and correct — most are enforced by the skill itself,
  574. but knowing them helps.
  575. **Running a case**
  576. - **Start with `/cti <target>`.** It's the single entry point and routes by target type
  577. (domain, IP, email, username, phone, wallet, hash, APK). Don't hand-run collectors.
  578. - **Recall before you collect.** `/cti-recall <seed>` (or Step 0 of `/cti`) is the cheapest call in
  579. the toolkit — it tells you if a seed is already attributed, saving credits and avoiding
  580. contradicting a prior assessment.
  581. - **`--passive` for anything hostile.** The egress gate refuses a direct fetch of hostile
  582. infrastructure; passive mode works from Wayback/urlscan captures, so your IP never touches it.
  583. - **Screen indicators before clustering.** `/cti-check <indicator>` (false-positive control) —
  584. a false merge names an innocent party; a false split loses the case.
  585. - **`--deep` earns its cost at 3+ live seeds** (parallel sub-agent fan-out); a single seed runs
  586. inline.
  587. - **`/cti-status` when something behaves oddly** — backend tier, MCP tools, and API balances in one
  588. shot.
  589. **Cost & keys**
  590. - **Two separate ledgers:** model reasoning (`/cost`) vs third-party API credits (`api_usage`) —
  591. never the same number.
  592. - **Keyless by default; `/apikeys` upgrades.** Keys enrich pivots (Shodan/Censys/FOFA/…) but
  593. nothing requires them.
  594. **If you're developing the skill**
  595. - `bash scripts/install-hooks.sh` once per clone — wires the leak check as a pre-commit hook.
  596. - `bash scripts/audit.sh` before pushing — the drift/leak/test gate (also runs in CI on every PR).
  597. - A classification change ships **with** its test (RULE 5).
  598. <br>
  599. ---
  600. <br>
  601. ## Features
  602. <table>
  603. <tr>
  604. <td width="33%" valign="top">
  605. ### Identity & People
  606. - Person lookup — 50+ data points
  607. - Phone — carrier, reputation, associations
  608. - Email — accounts, breaches, infrastructure
  609. - Username — 3000+ platform enumeration
  610. - GitHub developer footprint — profiles, orgs, repos, commits, forks
  611. </td>
  612. <td width="33%" valign="top">
  613. ### Domain & Infrastructure
  614. - Subdomain enumeration via CT logs
  615. - CMS, CDN, analytics fingerprinting
  616. - DNS forensics & WHOIS deep/reverse
  617. - Traffic analysis & audience demographics
  618. - ICP filing &rarr; PRC entity + sibling-domain pivot
  619. - IBAN/bank-account validation & attribution
  620. </td>
  621. <td width="33%" valign="top">
  622. ### Analysis & Verification
  623. - Face search (FaceCheck.id) & reverse image (TinEye)
  624. - Image forensics (FotoForensics, Forensically)
  625. - AI photo geolocation (picarta.ai, GeoSpy)
  626. - Document/email metadata forensics
  627. - Google Docs identity extraction
  628. - 100+ paste sites & breach DBs
  629. - Likelihood-banded judgments, 5W1H coverage, ACH
  630. </td>
  631. </tr>
  632. <tr>
  633. <td width="33%" valign="top">
  634. ### WiFi, Geo & Transport
  635. - SSID/BSSID lookup via Wigle.net
  636. - W3W, Plus Codes, MGRS, Street View
  637. - Aircraft tracking (ADS-B Exchange, Flightradar24)
  638. - Vessel tracking (Marine Traffic, VesselFinder)
  639. - Vehicle VIN lookup & plate recognition
  640. </td>
  641. <td width="33%" valign="top">
  642. ### Security Auditing
  643. - Cloud audit (AWS/GCP/Azure)
  644. - OWASP Top 10 source code review
  645. - CVE & supply chain vulnerability checks
  646. - LLM/agent/MCP prompt injection audit
  647. </td>
  648. <td width="33%" valign="top">
  649. ### Reporting & Export
  650. - INTSUM, executive brief, plain-language
  651. - DOCX with charts, diagrams, timelines
  652. - Save/load case workspaces
  653. - Legal, journalist, HR, threat analyst formats
  654. - Reversible PII redaction for external sharing
  655. </td>
  656. </tr>
  657. </table>
  658. <br>
  659. ---
  660. <br>
  661. ## AEAD Case Lifecycle
  662. Every investigation follows four automated phases:
  663. ```
  664. ╭──────────────────────────────────────╮
  665. │ AEAD CASE LIFECYCLE │
  666. ╰──────────────────────────────────────╯
  667. ┌─── ACQUIRE ────────────────────────────────────────────────────────┐
  668. │ Collect raw data via /sweep, /query, /username, /phone, etc. │
  669. │ Database search, enumeration, collection gap logging │
  670. └────────────────────────────────┬───────────────────────────────────┘
  671. ▼
  672. ┌─── ENRICH ─────────────────────────────────────────────────────────┐
  673. │ Expand leads via /branch, /crossref, /link-subjects, /signatures │
  674. │ Shared identifier detection, relationship mapping │
  675. └────────────────────────────────┬───────────────────────────────────┘
  676. ▼
  677. ┌─── ASSESS ─────────────────────────────────────────────────────────┐
  678. │ Score & verify via /exposure, /threat-model, /validate, /coverage│
  679. │ Risk scoring, completeness check, evidence chains │
  680. └────────────────────────────────┬───────────────────────────────────┘
  681. ▼
  682. ┌─── DELIVER ────────────────────────────────────────────────────────┐
  683. │ Package output via /report, /brief, /render, /workspace save │
  684. │ Auto-saves .md, .html, .json, .csv + IOC set │
  685. └────────────────────────────────────────────────────────────────────┘
  686. ```
  687. > Run `/progress` at any point to see current phase and pending tasks.
  688. <br>
  689. ### Workflow diagrams
  690. **End-to-end tool &amp; skill flow** — one `/cti` call runs recall, intake, an optional Quarry reuse, collection, the cross-type enrich loop with per-round approval, assessment and delivery across both layers: 9 registered commands and 58 techniques in Layer 1, and 103 harness `@tool`s over the vendored `intel_engine` (claude, codex, omp or any OpenAI-compatible backend):
  691. <div align="center">
  692. <img src="assets/workflow-skills.svg" alt="cti-expert end-to-end tool and skill flow — sequence diagram across both layers" width="900">
  693. </div>
  694. **Full `/cti` · `/case` pipeline (AEAD)** — a recursive Acquire → Enrich → Assess → Deliver loop: every collector records a per-layer outcome (a failed layer is re-runnable, never "empty"), free pivots with 25 hits or fewer run on their own, metered or oversized pivots wait for approval in `proposal.json`, and only owner-linked results become new seeds:
  695. <div align="center">
  696. <img src="assets/workflow-case.svg" alt="cti-expert /case pipeline (AEAD)" width="820">
  697. </div>
  698. **`/webpivot` + correlation + premium API-key flow** — a keyless baseline always runs, plus the free-keyed ChongLuaDao, Validin and Webamon; metered engines join only when their key is set and `--free-only` / `no_spend` is off; usage thresholds warn but never block, so a vendor's HTTP 402/429 is the only limit:
  699. <div align="center">
  700. <img src="assets/workflow-apikeys.svg" alt="cti-expert /webpivot + correlation + API-key workflow" width="820">
  701. </div>
  702. <sub>Sources: <a href="workflow-skills.puml"><code>workflow-skills.puml</code></a> · <a href="workflow-case.puml"><code>workflow-case.puml</code></a> · <a href="workflow-apikeys.puml"><code>workflow-apikeys.puml</code></a> — rendered with <a href="https://plantuml.com">PlantUML</a> to <b>SVG</b> in <a href="assets/"><code>assets/</code></a> (vector, so it stays sharp at any zoom and needs no Git-LFS). Re-render after editing a source:</sub>
  703. ```bash
  704. plantuml -tsvg -o assets workflow-case.puml workflow-apikeys.puml workflow-skills.puml
  705. # add -tpng if you need raster copies for slides — note assets/*.png is Git-LFS tracked
  706. ```
  707. <sub>See also the <a href="handbook/api-keys.md">API-keys &amp; webpivot guide</a>.</sub>
  708. <br>
  709. ---
  710. <br>
  711. ## Command Reference
  712. > The tables below are **convention commands** — the full technique vocabulary, available once the skill is loaded (via `/cti` or `/cti-expert`). The 9 registered entry commands (`/cti`, `/cti-recall`, `/cti-case`…) are in [Quick Start](#quick-start) above. For the authoritative reference, see **[SKILL.md](SKILL.md)**.
  713. <details>
  714. <summary><b>Acquire</b> — Data collection commands</summary>
  715. <br>
  716. | Command | Purpose |
  717. |---------|---------|
  718. | `/case [target]` | Full pipeline — every applicable technique (**alias of `/cti`**; prefer `/cti`) |
  719. | `/sweep [target]` | Multi-vector recon (person/domain/org/username/email/IP) |
  720. | `/query [subject]` | 12-15 advanced search operator queries |
  721. | `/username [handle]` | 3000+ platform enumeration |
  722. | `/phone [number]` | Carrier lookup, reputation, associations |
  723. | `/email-deep [email]` | Accounts, breaches, infrastructure |
  724. | `/subdomain [domain]` | CT logs + passive enumeration |
  725. | `/github-osint [target]` | GitHub user/org/repo profiles, code, commits, forks |
  726. | `/cld [target]` ⭐ | ChongLuaDao first-party premium — URL/AI/IoC/breach/CVE verdicts; client talks only to CLD, CLD fetches server-side (needs key) |
  727. | `/threat-check [target]` | IP/domain/URL/hash threat intelligence |
  728. | `/breach-deep [email]` | Multi-source breach lookup |
  729. </details>
  730. <details>
  731. <summary><b>Enrich</b> — Lateral expansion commands</summary>
  732. <br>
  733. | Command | Purpose |
  734. |---------|---------|
  735. | `/branch [data]` | Lateral expansion (email&rarr;username, username&rarr;email, etc.) |
  736. | `/crossref` | Shared identifier detection across subjects |
  737. | `/link-subjects [A] [B]` | Define connection between subjects |
  738. | `/show-connections` | Display logged connections |
  739. | `/graph` | Full ASCII subject relationship map |
  740. </details>
  741. <details>
  742. <summary><b>Assess</b> — Scoring & verification commands</summary>
  743. <br>
  744. | Command | Purpose |
  745. |---------|---------|
  746. | `/exposure [target]` | Composite risk score (0-100) |
  747. | `/threat-model` | Build threat model from findings |
  748. | `/validate` | Verify finding evidence chains |
  749. | `/coverage` | Check investigation completeness |
  750. </details>
  751. <details>
  752. <summary><b>Deliver</b> — Report generation commands</summary>
  753. <br>
  754. | Command | Purpose |
  755. |---------|---------|
  756. | `/report` | Technical INTSUM report |
  757. | `/report brief` | Executive summary |
  758. | `/brief` | Plain-language summary |
  759. | `/workspace save` | Persist case workspace state (resume later) |
  760. </details>
  761. <details>
  762. <summary><b>Web-Infra Pivoting & Correlation</b> — infrastructure & same-operator analysis</summary>
  763. <br>
  764. | Command | Purpose |
  765. |---------|---------|
  766. | `/webpivot [url]` | Extract favicon/tracker/wallet/SaaS-operator artifacts &rarr; ranked pivot queries (Shodan/FOFA/urlscan). Flags: `--rank`, `--cert`, `--graph`, `--history`, `--whois` |
  767. | `/rank-relations` | Score + rank same-operator relations across pages (weighted signals, noise-filtered, clustered) |
  768. | `/cert-pivot [domain]` | Find other hosts serving the same TLS cert + SAN siblings (keyless; Shodan/Censys with keys) |
  769. | `/pivot-suggest` | Rank "what to pivot next" from findings (leet/variant/temporal/domain clusters, **CJK pinyin + Traditional + company-stem**) |
  770. | `/crypto-balance [addr]` | On-chain balance + lifetime flow for a wallet, valued at spot |
  771. | `/iban [value]` | Validate + decompose a bank account (mod-97, BBAN split, bank code, mule signals) |
  772. | `/email-hygiene [email]` | Grade an email domain 0-100 + A-F (disposable/MX/free/role) |
  773. | `/sensitive-paths [list]` | Classify a Wayback/URL list for exposed paths (.git/.env/backups/configs) |
  774. </details>
  775. <details>
  776. <summary><b>China / Sinophone Recon</b> — ICP filings, PRC registries, CN indexes</summary>
  777. <br>
  778. | Command | Purpose |
  779. |---------|---------|
  780. | `/icp [domain\|serial]` | ICP filing &rarr; registered PRC entity + licence no.; reverse the **licence serial** to sibling domains under one filing |
  781. | `/cn-corp [name\|USCC]` | GSXT &rarr; TianYanCha/QCC/Aiqicha &rarr; 信用中国 chain: officers, shareholders, subsidiaries, UBO, revoked-status flags |
  782. | `/dork-sweep [t] --baidu` | Baidu tier — tiers 1&ndash;4 (Google/Bing/DDG) index almost no CN-hosted content |
  783. | `/pivot-suggest --cjk` | Pinyin, Simplified&harr;Traditional and company-name-stem variants |
  784. Registries needing mainland egress (TianYanCha/QCC/Aiqicha) are logged as **collection gaps**, never blockers.
  785. </details>
  786. <details>
  787. <summary><b>Reporting Hygiene</b></summary>
  788. <br>
  789. | Command | Purpose |
  790. |---------|---------|
  791. | `/redact [file]` | Shareable report variant — stable `[EMAIL_1]` placeholders + reversible JSON map (`.md`/`.json`/`.csv`). Opt-in; the base data bundle stays unredacted |
  792. | `/hash-id [hash]` | Identify a hash's algorithm before lookup — file hash vs credential material |
  793. </details>
  794. <details>
  795. <summary><b>Deep Pipeline &amp; Knowledge Base</b> — built-in (vendored <code>intel_engine</code>)</summary>
  796. <br>
  797. Built into the skill under `intel_engine/` (`intel_engine/harness/`, `tools/`, `knowledge/`, `cases/`). `/backend` resolves to SELF — no setup. Install the deep-layer deps once: `uv venv && uv pip install -r requirements.txt`. See [connectors/intel-backend.md](connectors/intel-backend.md).
  798. | Command | Purpose |
  799. |---------|---------|
  800. | `/backend` | Detect the backend and report the tier — Tier 1 (typed MCP) → Tier 2 (CLI) → Tier 3 (stateless). `/backend check` shows the full resolution trail |
  801. | `/kb [query]` | Query the shared knowledge base — stats, entity/cluster/shared-indicator lookup, confirmed-operator ledger |
  802. | `/recall [seed]` | *"Have I seen this before?"* — check a seed against every prior case before collecting |
  803. | `/risk [case]` | Score a case's hosts for NRD / bulletproof-hosting / money-trail red flags |
  804. | `/reverse-whois [email\|name]` | Reverse-WHOIS a registrant → high-value pivots only (privacy/bulk filtered) |
  805. | `/cert-overlap [d1 d2 …]` | KB-aware TLS/SAN same-operator verdict across domains |
  806. | `/reference [check\|add\|list]` | False-positive control ledger — BENIGN vs SIGNAL fingerprints |
  807. | `/harness [open\|continue\|status]` | Agent-driven whole-case orchestration — persistent, versioned, cross-case to convergence; **auto-escalates inside `/case`** (keyless-first) |
  808. | `/graph --render` | IntelGraph publication-quality case-graph render → PNG/SVG |
  809. | `/report pdf` | IntelReport pandoc render of an assessment → polished PDF/DOCX |
  810. | `/binary [file\|url]` | Static IOC extraction from a scam APK/exe (signing cert, package, C2 hosts, wallets) → clusters with web infra |
  811. All backend commands dispatch through `scripts/backend/intel.py` at Tier 2 (or the typed MCP tool at Tier 1); absent → they degrade to a note.
  812. </details>
  813. <br>
  814. ---
  815. <br>
  816. ## Skill Tiers
  817. Output density and automation adapt to your proficiency. **Switch tiers any time — the output changes immediately:** `/novice` enters Novice, `/novice off` gives you Specialist, and Practitioner is the default in between.
  818. <table>
  819. <tr>
  820. <th width="33%">Novice</th>
  821. <th width="33%">Practitioner</th>
  822. <th width="33%">Specialist</th>
  823. </tr>
  824. <tr>
  825. <td valign="top">
  826. Low-jargon mode, step-by-step guidance, pre-built templates for due diligence, background checks, security reviews.
  827. **Switch:** `/novice`
  828. **Try:** `/flow person`, `/flow domain`, `/template list`
  829. </td>
  830. <td valign="top">
  831. Advanced search operators, manual pivot expansion, custom threat modeling, guided flows with explanation.
  832. **Switch:** default — no command
  833. **Try:** `/query [target]`, `/branch [data]`, `/crossref`, `/threat-model`
  834. </td>
  835. <td valign="top">
  836. Raw technique access, custom evidence weighting, CONTESTED finding resolution, direct database queries.
  837. **Switch:** `/novice off`
  838. **Try:** `/username [handle]`, `/email-deep [email]`, `/secrets [target]`, `/threat-check [target]`
  839. </td>
  840. </tr>
  841. </table>
  842. <br>
  843. ---
  844. <br>
  845. ## Technique Catalog
  846. <details>
  847. <summary><b>58 techniques</b> — click to expand full catalog</summary>
  848. <br>
  849. | Technique | Coverage | API Key Required? |
  850. |-----------|----------|-------------------|
  851. | `fx-metadata-parsing.md` | EXIF, email headers, document forensics | No |
  852. | `fx-image-verification.md` | Image authenticity, provenance, reverse search | No |
  853. | `media-vision-analysis.md` | Content of image / document / audio-video evidence — OCR, logos, landmarks, faces, transcripts → new seeds | Optional (vision + transcription need a Gemini key via `multix`) |
  854. | `fx-breach-discovery.md` | Breach database + paste site enumeration | Optional (HIBP bulk, DeHashed paid) |
  855. | `fx-http-fingerprint.md` | HTTP signature analysis, server fingerprinting | No |
  856. | `fx-leak-monitoring.md` | Leak and breach monitoring automation | Mixed (IntelligenceX/Shodan paid) |
  857. | `fx-dns-cert-history.md` | Historical DNS + SSL/TLS certificate timeline | No |
  858. | `fx-document-forensics.md` | PDF/Office authorship, creation chain, hidden content | No |
  859. | `fx-network-mapping.md` | Network topology, entity graph construction | No |
  860. | `username-osint.md` | 3000+ platform enumeration | No |
  861. | `phone-osint.md` | Carrier lookup, VoIP, FreeCNAM, WhoCalld | No |
  862. | `email-osint.md` | Deep email investigation, breach history | No |
  863. | `threat-intel.md` | GreyNoise, AbuseIPDB, OTX, VirusTotal, CIRCL CVE, NVD | Optional (VT/URLScan free keys) |
  864. | `web-traffic-analysis.md` | SimilarWeb, Semrush estimation | No |
  865. | `domain-advanced.md` | CT logs, Amass, Subfinder, passive enum | No |
  866. | `social-media-platforms.md` | Twitter/X, Discord, Strava, BlueSky, ShareTrace, Reddit, Instagram, TikTok, Telegram | Partial (Discord needs token) |
  867. | `image-forensics-and-face-search.md` | FaceCheck.id, TinEye, FotoForensics, Forensically, picarta.ai, GeoSpy, Pic2Map | No |
  868. | `blockchain-investigation.md` | Blockchair, Etherscan, WalletExplorer, OXT.me, Chainabuse, Breadcrumbs | Optional (Etherscan API for bulk) |
  869. | `fiat-payment-osint.md` | IBAN mod-97 + BBAN decomposition, BIC, VietQR/NAPAS BIN, card BIN, account-reuse pivot | No |
  870. | `china-recon.md` | ICP filings, GSXT/信用中国/TianYanCha/QCC/Aiqicha, USCC, Quake/ZoomEye/FOFA, Baidu dorking, CJK variants | Partial (CN indexes need free keys; aggregators need CN egress) |
  871. | `transport-tracking.md` | ADS-B Exchange, Flightradar24, Marine Traffic, VesselFinder, VIN decode | No |
  872. | `darknet-investigation.md` | Ahmia.fi, onionsearch, DarknetLive, ransomwatch | No |
  873. | `advanced-geolocation-techniques.md` | W3W, Plus Codes, MGRS, Overpass Turbo | No |
  874. | `wifi-ssid-osint.md` | Wigle.net SSID/BSSID geolocation | Free account (Wigle API) |
  875. | `web-dns-forensics.md` | Zone transfers, GitHub, Telegram, WHOIS | Optional (WHOIS API) |
  876. | `scam-check.md` | Phishing/scam domain verification | No |
  877. | `ioc-export.md` | IOC export (STIX 2.1, flat list) | No |
  878. | `cloud-audit.md` | AWS/GCP/Azure IAM, network, compute audit | No |
  879. | `dependency-audit.md` | CVE, supply chain, CI/CD security | No |
  880. | `disk-forensics.md` | Sleuth Kit, file carving, artifact recovery | No |
  881. | `incident-triage.md` | NIST 800-61, containment, IOC extraction | No |
  882. | `owasp-audit.md` | OWASP Top 10 source code review | No |
  883. | `prompt-injection-audit.md` | LLM/agent/MCP security assessment | No |
  884. | `fx-visitor-intelligence.md` | Visitor stats, tech stack, geo analysis | No |
  885. | `fx-social-topology.md` | Social graph construction and analysis | No |
  886. | `fx-geolocation.md` | GPS, W3W, Plus Codes, MGRS, Street View | No |
  887. | `secret-scanning.md` | Credential/secret detection in code | Optional (GitHub token for GitDorker) |
  888. | `github-osint.md` | GitHub profile, org, repo, code, commit, fork, and collaboration recon | Optional (GitHub token for higher API limits) |
  889. | `fx-email-header-analysis.md` | Email header analysis, SPF/DKIM | No |
  890. | `fx-edge-appliance-recon.md` | Edge/VPN appliance fingerprint → CISA KEV/CVE catalog + port-risk matrix | No (Shodan/Censys optional) |
  891. | `fx-saas-identity-recon.md` | SaaS tenancy (DNS-TXT) + IdP fingerprint + API/GraphQL/spec discovery | No |
  892. | `web-pivot.md` | Web-infra pivoting — favicon mmh3, tracker/wallet/SaaS-operator artifacts → ranked pivots | Optional (premium keys upgrade tiers) |
  893. | `whois-universal.md` | Universal multi-TLD WHOIS cascade — gTLD/ccTLD (.vn/.th/.sg/.kr), reverse & historical | Optional (Whoxy/WhoisXML for reverse/history) |
  894. | `web-collection-scrapling.md` | Adaptive web scraping — static → anti-bot → JS rendering | No |
  895. | `agent-browser.md` | Interactive browser collection — CDP, accessibility-tree snapshots, screenshot evidence | No (chat mode optional) |
  896. | `agentflow-enrichment.md` | Parallel DAG enrichment orchestration for 3+ subjects | No |
  897. | `microsoft-tenant-recon.md` | M365/Azure tenant recon — tenant ID, federation, MDI, SharePoint | No |
  898. | `stealer-log-analysis.md` | Infostealer-log triage — family ID, victim-vs-operator profiling, cross-log correlation, IOC | No |
  899. | `fx-dork-sweep.md` | Zero-auth dork sweep — Telegram ecosystem, doc-hosts, filetype families | No |
  900. | `fx-document-leak-hunt.md` | 18-platform document-leak hunt with severity classification | No |
  901. | `phishing-domain-survival.md` | Registration/DNS strategy → maliciously-registered vs compromised + survival outlook (eCrime 2026 "Built to Last?") | No |
  902. | `clickfix-clipboard-hijack.md` | ClickFix / PasteJacking clipboard-hijack detection + `-enc` decode → C2 IOCs (eCrime 2026 "PasteJacked") | No |
  903. | `visibility-aware-html.md` | Hidden credential forms / off-origin links / off-screen brand text (eCrime 2026 "Visibility-Aware HTML Analysis") | No |
  904. | `apk-permission-scope.md` | APK permission-scope risk scoring — on-device-fraud permission combos, capability≠guilt (eCrime 2026 "The 'Allow' Reflex") | No |
  905. | `kit-template-attribution.md` | Phishing kit/template structural fingerprint + similarity; commodity-template match graded as noise (eCrime 2026 tree-structured attribution) | No |
  906. | `renderer-confirmation.md` | Renderer-level confirmation of ClickFix + visibility (runtime clipboard / computed-hidden → reconcile); renderer optional (eCrime 2026 PasteJacked + Visibility-Aware HTML) | No |
  907. | `phishtrace-dynamic-features.md` | Runtime-trace phishing characterization → verdict + exfil IOCs; thin trace on a flagged page = cloaked (eCrime 2026 "PhishTrace") | No |
  908. </details>
  909. <br>
  910. ---
  911. <br>
  912. ## Report Formats
  913. You never have to ask for output. Every `/report`, `/brief`, and `/case` writes the full set automatically — one interactive web page to explore the case, plus machine-readable files for tooling and evidence. Need to share a report outside your team? Add `--redact` and PII is swapped for stable placeholders (and can be reversed later).
  914. <table>
  915. <tr>
  916. <td width="50%" valign="top">
  917. ### 🌐 Interactive HTML report — *the one you'll actually read*
  918. A single self-contained file — no internet, no server, opens in any browser.
  919. - **Dashboard** — KPIs, an exposure gauge, and pie / bar / donut charts
  920. - **Entity graph** — drag, zoom, and click any node to inspect it
  921. - **Infrastructure & timeline** — topology map plus an interactive event history
  922. - **Indicators & Selectors** — every IOC, contact, handle, and wallet, with actor ↔ victim attribution
  923. - **Navigation** — global search, category menus, dark / light theme, print-to-PDF
  924. </td>
  925. <td width="50%" valign="top">
  926. ### 📄 Markdown · JSON · CSV · IOC bundle — *for tooling & evidence*
  927. The same case, in formats other tools can read.
  928. - **Markdown** — the written report: INTSUM, executive brief, plain-language, or legal
  929. - **JSON** — structured case data to feed pipelines and other tools
  930. - **CSV** — findings and indicators, ready for a spreadsheet or SIEM
  931. - **IOC bundle** — STIX 2.1, flat list, CSV, and JSONL of every selector
  932. - **Word (.docx) / PDF** — pick at the prompt (choice **a**/**b**/**d**) or `/report legal`; cover page, table of contents, charts
  933. </td>
  934. </tr>
  935. </table>
  936. **Every report variant is one command** — the base data bundle (`.md` · `.json` · `.csv` · IOC bundle: `.stix.json`/`.txt`/`.csv`/`.jsonl`) saves automatically on every `/report`, `/brief`, and `/case`, then you're asked which presentation report to render: **(a) PDF · (b) DOCX · (c) HTML · (d) all**; when HTML is among them, one more confirmation shows what the Archify **Blueprint** would embed (`cti_archify.py --plan`) and lets you pick Auto / force / off before the file is written (`--yolo`/guided-auto skip both prompts and default to HTML with Blueprint in Auto). The variants below pick a specific format or audience:
  937. | Command | Format | Best for |
  938. |---------|--------|----------|
  939. | `/report html` · choice **c**/**d** | Interactive HTML *(primary human-facing deliverable)* | Everyone — analysts to execs |
  940. | `/report` | Technical INTSUM (Markdown) | Analysts, security teams |
  941. | `/report brief` | Executive brief | Decision-makers, management |
  942. | `/brief` | Plain-language summary | Non-technical stakeholders |
  943. | `/report legal` | Legal evidence format *(defaults to all: PDF + DOCX + HTML)* | Attorneys, compliance teams |
  944. | `/report journalist` | Source-citation-heavy | Reporters, media |
  945. | `/report json` · `/report csv` | JSON · CSV export | Pipelines, spreadsheets, SIEM |
  946. | `/report ioc` | IOC / selector bundle (STIX 2.1 · flat · CSV) | SIEM / TIP ingest, threat-intel sharing |
  947. | `/report docx` · `/report pdf` · choice **a**/**b**/**d** | Word / PDF document *(charts, cover, TOC)* | Formal sharing |
  948. | `/cti-report <ID>` | **House report** — editorial PDF/DOCX composed from the case dir *(charts, landing-page captures, dossiers, Appendices A–E)* | Polished, publication-grade case deliverable |
  949. <sub>Generated by <code>scripts/build_report_data.py</code> (report JSON) · <code>scripts/generate-cti-html.py</code> (HTML) · <code>scripts/generate-cti-iocs.py</code> (IOCs) · <code>scripts/generate-cti-docx-hybrid.py</code> (dashboard DOCX) · <code>intel_engine/tools/house_report.py</code> + <code>intel_engine/IntelReport</code> (house PDF/DOCX)</sub>
  950. <br>
  951. ---
  952. <br>
  953. ## Architecture
  954. <details>
  955. <summary><b>Project structure</b> — click to expand</summary>
  956. <br>
  957. cti-expert is **one skill, two layers** — a broad *collector* plus a vendored, self-contained *deep pipeline*. `STRUCTURE.md` is the authoritative map.
  958. ```
  959. cti-expert/
  960. ├── SKILL.md The single skill entrypoint — commands & tradecraft
  961. ├── README.md This file · README.vi.md · README.zh-CN.md
  962. ├── STRUCTURE.md Authoritative layout + anti-drift rules
  963. ├── AGENTS.md Cross-agent runtime contract (every AGENTS.md-aware coding agent)
  964. ├── CLAUDE.md Contributor rules (loads only when building the repo)
  965. │
  966. ├── commands/ 9 registered slash commands — work from a cold prompt
  967. │ ├── cti.md /cti — the entry point, routes by target type
  968. │ ├── cti-recall.md · cti-case.md · cti-pivot.md · cti-cluster.md
  969. │ └── cti-check.md · cti-report.md · cti-status.md · cti-proxy.md
  970. │
  971. │ ── LAYER 1 · Broad collector — cti-expert's own tools ──────────────
  972. ├── scripts/ Collectors, backend dispatcher, report generators
  973. │ ├── backend/ backend.py (tier resolver) · intel.py (T2 CLI dispatch)
  974. │ ├── webpivot/ pivot_extract · cert_pivot · wayback_* · rank_relations …
  975. │ ├── osint/ 18 keyless-first CLIs behind the SKILL.md commands — hash_id · vuln_check · username_enum · phone_osint · github_osint · reputation_check · cn_recon · kb_crossref · case_drift …
  976. │ ├── build_report_data.py Case dir → REPORT.json (deterministic report bridge)
  977. │ ├── generate-cti-html.py Interactive, offline, self-contained HTML report
  978. │ ├── generate-cti-iocs.py IOC / selector export (STIX 2.1 · flat · CSV · JSONL)
  979. │ ├── generate-cti-docx-hybrid.py Dashboard DOCX/PDF (charts, diagrams, cover page)
  980. │ ├── iban_analyze.py · redact.py · stealer_log_parse.py · pivot_orchestrator.py
  981. │ ├── register.sh · register_agents.py /cti for Claude Code · every other coding agent
  982. │ ├── install.sh · install.ps1 All-in-one cross-platform installer
  983. │ └── audit.sh · leakcheck.sh · install-hooks.sh Drift · leak · pre-commit gates
  984. │
  985. ├── techniques/ 58 collection techniques (the OSINT tradecraft)
  986. ├── handbook/ Pivot artifacts, API keys, operator queries, analytic standards
  987. ├── engine/ Case data-model design docs (schema, findings, pivot logic)
  988. ├── analysis/ · validation/ Pattern & exposure engines · QA + coverage matrices
  989. ├── experience/ UX — skill tiers, guided flows, case templates
  990. ├── workflows/ · guides/ Professional use-case guides · worked walkthroughs
  991. ├── connectors/ intel-backend · Maltego · Notion · Obsidian export
  992. │
  993. │ ── LAYER 2 · Deep pipeline — vendored, self-contained ──────────────
  994. └── intel_engine/ Collect → Correlate → Assess pipeline + knowledge base
  995. ├── harness/ Pipeline brain — orchestrator.py · mcp_server.py · tools.py (103 @tool)
  996. ├── tools/ intel.py (deterministic pipeline) · house_report*.py · case_state · kb/ (KB + correlation)
  997. ├── WebPivot/ Canonical collector — pivot_extract · wp_* vendor clients · enrich_live
  998. ├── IntelGraph/ Publication-quality case-graph rendering (PNG/SVG)
  999. ├── IntelReport/ Pandoc render of the composed house report → PDF/DOCX
  1000. ├── IntelAnalysis/ Correlation, attribution, confidence calibration
  1001. ├── IntelShare/ MISP export · stage · publish (two separate decisions)
  1002. ├── Engage/ Login detection · synthetic persona · gated members-area harvest
  1003. ├── BinaryPivot/ Static IOC extraction from a scam APK / exe · ANY.RUN lookup + gated submit
  1004. └── knowledge/ · cases/ Local runtime data — gitignored, never committed
  1005. ```
  1006. </details>
  1007. <br>
  1008. ---
  1009. <br>
  1010. ## Professional Workflows
  1011. | Workflow | Audience | File |
  1012. |----------|----------|------|
  1013. | **Journalist Source Verification** | Reporters, fact-checkers | `workflows/wf-journalist.md` |
  1014. | **HR Screening** | HR professionals, recruiters | `workflows/wf-hr-screening.md` |
  1015. | **Cyber Threat Intelligence** | Security analysts, IR teams | `workflows/wf-threat-analyst.md` |
  1016. | **Private Investigator** | Licensed PIs, legal teams | `workflows/wf-private-investigator.md` |
  1017. > Activate with `/flow [type]` for interactive guided prompts.
  1018. <br>
  1019. ---
  1020. <br>
  1021. ## Ethics & Responsible Use
  1022. > **This skill is for lawful research and professional security investigation only.**
  1023. <table>
  1024. <tr>
  1025. <th>Permitted</th>
  1026. <th>Prohibited</th>
  1027. </tr>
  1028. <tr>
  1029. <td valign="top">
  1030. - Journalist fact-checking & source verification
  1031. - HR background screening (with consent)
  1032. - Corporate security research & threat intelligence
  1033. - Authorized penetration testing & security audits
  1034. - Legal/compliance investigation
  1035. - Personal reputation monitoring (self-search)
  1036. </td>
  1037. <td valign="top">
  1038. - Doxxing, harassment, or stalking
  1039. - Unauthorized surveillance
  1040. - Social engineering or fraud
  1041. - Privacy violations
  1042. - Criminal activity
  1043. </td>
  1044. </tr>
  1045. </table>
  1046. **You are responsible for all use of this skill.** Comply with local laws, regulations, and platform terms of service. Always respect privacy and consent boundaries.
  1047. <br>
  1048. ---
  1049. <br>
  1050. ## Contributing
  1051. We welcome research contributions, new techniques, and workflow improvements.
  1052. <details>
  1053. <summary><b>Contribution guidelines</b></summary>
  1054. <br>
  1055. **Adding techniques:**
  1056. 1. Create `techniques/fx-[name].md` with method description, free tool lists, limitations
  1057. **Workflow improvements:**
  1058. 1. Document in `workflows/` with success criteria
  1059. **Pull request process:**
  1060. 1. Fork and create feature branch: `git checkout -b feature/technique-name`
  1061. 2. Document changes in SKILL.md and README.md
  1062. 3. Test on at least 3 real-world targets
  1063. 4. Submit PR with description
  1064. **Bug reports:** File issues with command output, environment, and target type.
  1065. </details>
  1066. <br>
  1067. ---
  1068. <br>
  1069. ## License
  1070. **MIT License** + Ethical Use Addendum
  1071. You are free to use, modify, and distribute this skill under the MIT license, provided that you include original attribution, comply with the ethical use guidelines above, and clearly mark any derivatives.
  1072. See [LICENSE](LICENSE) for full text.
  1073. <br>
  1074. ---
  1075. <br>
  1076. ## 🙏 Acknowledgments & Credits
  1077. CTI Expert stands on the shoulders of the open-source community and free, public-interest data providers. A huge thank-you to every project, vendor, and free API below — this skill simply would not exist without your work. *(Listing here does not imply affiliation or endorsement; always respect each provider's terms of service.)*
  1078. | Category | Projects & free services we're grateful to |
  1079. |----------|---------------------------------------------|
  1080. | **Agents & runtime** | [Anthropic — Claude Code](https://claude.com/claude-code) · [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-python) · [OpenAI — Codex](https://developers.openai.com/codex) · [Astral — uv](https://docs.astral.sh/uv/) · [Python](https://www.python.org) · [Node.js](https://nodejs.org) · [Bun](https://bun.sh) · [Rust](https://www.rust-lang.org) · [Pydantic](https://docs.pydantic.dev) · [Rich](https://github.com/Textualize/rich) · [python-socks](https://github.com/romis2012/python-socks) · local model backends [Ollama](https://ollama.com) · [vLLM](https://github.com/vllm-project/vllm) · [LM Studio](https://lmstudio.ai) · agents `/cti` also registers into: [OpenCode](https://opencode.ai) · [Gemini CLI](https://github.com/google-gemini/gemini-cli) · [Qwen Code](https://github.com/QwenLM/qwen-code) · [Cursor](https://cursor.com) · [GitHub Copilot CLI](https://github.com/features/copilot/cli) · [Windsurf](https://windsurf.com) · [Factory Droid](https://factory.ai) · omp · pi |
  1081. | **Browser & web collection** | [Moli — Lexmount](https://github.com/lexmount/moli) · [Playwright](https://playwright.dev) · [Chromium](https://www.chromium.org) · [agent-browser — Vercel Labs](https://github.com/vercel-labs/agent-browser) · [Scrapling](https://github.com/D4Vinci/Scrapling) · [Requests](https://requests.readthedocs.io) · [FlareSolverr](https://github.com/FlareSolverr/FlareSolverr) · [cloudscraper](https://github.com/venomous/cloudscraper) · [Brotli](https://github.com/google/brotli) · [CloakBrowser — CloakHQ](https://github.com/CloakHQ/CloakBrowser) |
  1082. | **Username, people & social** | [Maigret](https://github.com/soxoj/maigret) · [Sherlock](https://github.com/sherlock-project/sherlock) · [Blackbird](https://github.com/p1ngul1n0/blackbird) · [instaloader](https://github.com/instaloader/instaloader) · [Osintgram](https://github.com/Datalux/Osintgram) · [toutatis](https://github.com/megadose/toutatis) · [ShareTrace](https://github.com/7onez/sharetrace) · [Telethon](https://github.com/LonamiWebs/Telethon) · [opentele2 — DedInc](https://github.com/DedInc/opentele2) · [GHunt — mxrch](https://github.com/mxrch/GHunt) · [Telegram API](https://core.telegram.org/api) · [FamilySearch](https://www.familysearch.org) · [FreeBMD](https://www.freebmd.org.uk) |
  1083. | **Email & breach data** | [Holehe](https://github.com/megadose/holehe) · [h8mail](https://github.com/khast3x/h8mail) · [theHarvester](https://github.com/laramies/theHarvester) · [Have I Been Pwned](https://haveibeenpwned.com) · [Hudson Rock](https://www.hudsonrock.com) · [Intelligence X](https://intelx.io) · [OSINT Industries](https://www.osint.industries) · [LeakCheck](https://leakcheck.io) · [Lunar Domain Exposure](https://lunarcyber.com) · [keys.openpgp.org](https://keys.openpgp.org) · [Ubuntu keyserver](https://keyserver.ubuntu.com) · [Proton OpenPGP keyserver](https://proton.me) |
  1084. | **Domains, DNS, certificates & infrastructure** | [Subfinder](https://github.com/projectdiscovery/subfinder) · [Amass](https://github.com/owasp-amass/amass) · [assetfinder](https://github.com/tomnomnom/assetfinder) · [findomain](https://github.com/Findomain/Findomain) · [httpx](https://github.com/projectdiscovery/httpx) · [GAU](https://github.com/lc/gau) · [crt.sh](https://crt.sh) · [Shodan](https://www.shodan.io) (search · [InternetDB](https://internetdb.shodan.io) · [CTL](https://ctl.shodan.io)) · [Censys](https://censys.io) · [Netlas](https://netlas.io) · [Validin](https://validin.com) · [SecurityTrails](https://securitytrails.com) · [DNSlytics](https://dnslytics.com) · [ZETAlytics / ZoneCruncher](https://zetalytics.com) · [SSLMate CertSpotter](https://sslmate.com/certspotter/) · [crt.name](https://crt.name) · [agniops](https://app.agniops.in) · [CIRCL Passive DNS / Passive SSL](https://www.circl.lu/services/) · [WhoisXML API](https://whoisxmlapi.com) · [Whoxy](https://www.whoxy.com) · [whoisdomain](https://github.com/mboot-github/WhoisDomain) · [IPinfo](https://ipinfo.io) · [ipwho.is](https://ipwho.is) · [ViewDNS](https://viewdns.info) · [HackerTarget](https://hackertarget.com) · [Tranco](https://tranco-list.eu) · [Mozilla Public Suffix List](https://publicsuffix.org) · [CertKit](https://certkit.io) · [CertIndex](https://ctindex.io) · [IANA RDAP bootstrap](https://data.iana.org/rdap/) · [Hurricane Electric BGP Toolkit](https://bgp.he.net) · [ip-api](https://ip-api.com) · [JARM — Salesforce](https://github.com/salesforce/jarm) · [mmh3](https://github.com/hajimes/mmh3) · [BIND `dig` — ISC](https://www.isc.org/bind/) · [Sysinternals Whois](https://learn.microsoft.com/sysinternals/downloads/whois) · [psycopg](https://www.psycopg.org) (crt.sh PostgreSQL fallback) |
  1085. | **Threat intelligence & sandboxing** | [ChongLuaDao](https://chongluadao.vn) (first-party feeds, Shield & WHOIS) · [VirusTotal](https://www.virustotal.com) · [urlscan.io](https://urlscan.io) · [Webamon](https://webamon.com) · [ANY.RUN](https://any.run) · [GreyNoise](https://www.greynoise.io) · [AbuseIPDB](https://www.abuseipdb.com) · [AlienVault OTX](https://otx.alienvault.com) · [abuse.ch](https://abuse.ch) (URLhaus · ThreatFox · MalwareBazaar) · [CIRCL cve-search](https://www.circl.lu) · [NVD](https://nvd.nist.gov) · [ransomware.live](https://www.ransomware.live) · [MISP](https://www.misp-project.org) · [PublicWWW](https://publicwww.com) · [Spamhaus](https://www.spamhaus.org) |
  1086. | **Search engines & ads** | [SerpApi](https://serpapi.com) · [Bright Data SERP API](https://brightdata.com) · [Google Ads Transparency Center](https://adstransparency.google.com) · [Common Crawl](https://commoncrawl.org) · [Google sellers.json](https://iabtechlab.com/sellers-json/) (IAB Tech Lab) |
  1087. | **Secrets, code & exposure** | [TruffleHog](https://github.com/trufflesecurity/trufflehog) · [Gitleaks](https://github.com/gitleaks/gitleaks) · [GitHub CLI](https://cli.github.com) · [GrayHatWarfare](https://grayhatwarfare.com) · [Docker Hub](https://hub.docker.com) · [GitHub Container Registry](https://ghcr.io) · [Quay.io](https://quay.io) |
  1088. | **Phone** | [PhoneInfoga](https://github.com/sundowndev/phoneinfoga) · FreeCNAM · WhoCalld |
  1089. | **Geolocation & WiFi** | [OpenStreetMap](https://www.openstreetmap.org) · [what3words](https://what3words.com) · [Overpass Turbo](https://overpass-turbo.eu) · [WiGLE](https://wigle.net) |
  1090. | **Image, media & vision** | [ExifTool](https://exiftool.org) · [TinEye](https://tineye.com) · [FaceCheck.id](https://facecheck.id) · [FotoForensics](https://fotoforensics.com) · [picarta.ai](https://picarta.ai) · [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) · [FFmpeg](https://ffmpeg.org) · [ImageMagick](https://imagemagick.org) · [RMBG](https://huggingface.co/briaai/RMBG-1.4) · [rmbg-cli — mrgoonie](https://github.com/mrgoonie/rmbg) · [multix CLI — mrgoonie](https://github.com/mrgoonie/multix-cli) · [Google Gemini](https://ai.google.dev) · [pyzbar](https://github.com/NaturalHistoryMuseum/pyzbar) / [ZBar](https://github.com/mchehab/zbar) · [whisper-ctranslate2](https://github.com/Softcatala/whisper-ctranslate2) · [Yandex Images](https://yandex.com/images) · [Bing Visual Search](https://www.bing.com/visualsearch) · [Google Lens](https://lens.google) · [PimEyes](https://pimeyes.com) · [python-qrcode](https://github.com/lincolnloop/python-qrcode) |
  1091. | **Blockchain** | [Blockchair](https://blockchair.com) · [Etherscan](https://etherscan.io) · [Subscan](https://subscan.io) · [WalletExplorer](https://www.walletexplorer.com) · [Chainabuse](https://www.chainabuse.com) · [Blockstream Esplora](https://blockstream.info) · [Tronscan](https://tronscan.org) |
  1092. | **China / Sinophone recon** | [ENScan_GO](https://github.com/wgpsec/ENScan_GO) · [Kunyu](https://github.com/knownsec/Kunyu) · [fofax](https://github.com/xiecat/fofax) · [PyDork](https://github.com/blacknon/pydork) · [MediaCrawler](https://github.com/NanmiCoder/MediaCrawler) · [pypinyin](https://github.com/mozillazg/python-pinyin) · [OpenCC](https://github.com/BYVoid/OpenCC) · [jieba](https://github.com/fxsjy/jieba) · [FOFA](https://fofa.info) · [Quake (360)](https://quake.360.net) · [ZoomEye](https://www.zoomeye.ai) · [Hunter.how](https://hunter.how) · [GSXT](https://www.gsxt.gov.cn) · [信用中国](https://www.creditchina.gov.cn) · [Cninfo](http://www.cninfo.com.cn) |
  1093. | **Payment rails & hashes** | [ISO 13616 / ISO 7064](https://www.iso.org) (IBAN + mod-97 standards) · [NAPAS / VietQR](https://vietqr.vn) · [name-that-hash](https://github.com/HashPals/Name-That-Hash) |
  1094. | **Business & app registries** | [masothue.com](https://masothue.com) · [hsctvn.com](https://hsctvn.com) · [doanhnghiep.biz](https://doanhnghiep.biz) · [thuvienphapluat.vn](https://thuvienphapluat.vn) · [infocom.vn](https://infocom.vn) · [tratencongty.com](https://www.tratencongty.com) (Vietnamese tax-code / company registries) · [GLEIF](https://www.gleif.org) · [UK Companies House](https://find-and-update.company-information.service.gov.uk) · [OpenCorporates](https://opencorporates.com) · [recherche-entreprises (France)](https://recherche-entreprises.api.gouv.fr) · [Brønnøysund Register Centre (Norway)](https://data.brreg.no) · [PRH (Finland)](https://avoindata.prh.fi) · [SEC EDGAR](https://www.sec.gov/edgar) · [Google Play](https://play.google.com) (app developer legal name) |
  1095. | **Tradecraft & methodology** | [Rexxfield](https://rexxfield.com) · [ParanoidLab](https://paranoidlab.com) · [Social Links](https://sociallinks.io) · [SOsintOps — Speculator Project](https://github.com/SOsintOps/Speculator-Project) · [Wukong](https://github.com/SOsintOps/Wukong) (China-layer tool survey & access-reality matrix) · [Exploratores](https://github.com/SOsintOps/Exploratores) (reversible-redaction and IBAN-analysis *techniques* — independently reimplemented from its published documentation; that project is AGPL-3.0 and **no code was copied**) · [APWG eCrime research](https://ecrimeresearch.org) (the 2026 papers behind eight techniques) · [ISECOM](https://www.isecom.org) (Pete Herzog's adversary-modelling framework behind the AAM overlay) |
  1096. | **Transport tracking** | [ADS-B Exchange](https://www.adsbexchange.com) · [Flightradar24](https://www.flightradar24.com) · [MarineTraffic](https://www.marinetraffic.com) · [VesselFinder](https://www.vesselfinder.com) |
  1097. | **Darknet** | [Ahmia](https://ahmia.fi) · [OnionSearch](https://github.com/megadose/OnionSearch) · [ransomwatch](https://github.com/joshhighet/ransomwatch) |
  1098. | **Cloud & documents** | [MSFTRecon](https://github.com/Arcanum-Sec/msftrecon) · [Xeuledoc](https://github.com/Malfrats/xeuledoc) · [oletools](https://github.com/decalage2/oletools) · [poppler](https://poppler.freedesktop.org) · [qpdf](https://github.com/qpdf/qpdf) · [mat2](https://0xacab.org/jvoisin/mat2) · [The Sleuth Kit](https://www.sleuthkit.org) |
  1099. | **Web archives** | [Internet Archive — Wayback](https://web.archive.org) · [archive.today](https://archive.today) · [Common Crawl](https://commoncrawl.org) · [Waymore](https://github.com/xnl-h4ck3r/waymore) |
  1100. | **Reporting, diagrams & rendering** | [pandoc](https://pandoc.org) · [TeX Live / XeLaTeX](https://tug.org/texlive/) · [MiKTeX](https://miktex.org) · [LibreOffice](https://www.libreoffice.org) · [python-docx](https://github.com/python-openxml/python-docx) · [openpyxl](https://openpyxl.readthedocs.io) · [Matplotlib](https://matplotlib.org) · [NumPy](https://numpy.org) · [NetworkX](https://networkx.org) · [Graphviz](https://graphviz.org) · [Mermaid](https://mermaid.js.org) · [Pele — Obsidian](https://github.com/obsidianmd/pele) (the Mermaid → SVG renderer) · [librsvg](https://gitlab.gnome.org/GNOME/librsvg) · [Cytoscape.js](https://js.cytoscape.org) + [fCoSE · cose-base · layout-base — iVis@Bilkent](https://github.com/iVis-at-Bilkent/cytoscape.js-fcose) (interactive network graph) · [PlantUML](https://plantuml.com) · [diagrams](https://diagrams.mingrammer.com) · [CairoSVG](https://cairosvg.org) · [Pillow](https://python-pillow.org) · [Noto fonts](https://fonts.google.com/noto) · [DejaVu fonts](https://dejavu-fonts.github.io) · [jq](https://jqlang.github.io/jq/) · [ASN](https://github.com/nitefood/asn) |
  1101. | **Vendored (MIT, trimmed copies)** | [Archify — tt-a1i](https://github.com/tt-a1i/archify) (the HTML report's Blueprint diagram) · [Diagram Design — Cathryn Lavery](https://github.com/cathrynlavery/diagram-design) (editorial entity / topology SVG) · [Diagram AI Generator — carlosmgv02](https://github.com/carlosmgv02/diagram-ai-generator) (cloud-architecture figure) |
  1102. | **Standards & frameworks** | [OWASP](https://owasp.org) · [MITRE ATT&CK](https://attack.mitre.org) · [STIX 2.1 (OASIS)](https://oasis-open.github.io/cti-documentation/) · [ICD 203 — Analytic Standards (ODNI)](https://www.dni.gov/files/documents/ICD/ICD-203.pdf) · Admiralty / NATO source-reliability code · [FIRST TLP 2.0](https://www.first.org/tlp/) · [NIST SP 800-61](https://csrc.nist.gov/pubs/sp/800/61/r2/final) · [CWE](https://cwe.mitre.org) |
  1103. > Built something here we should credit, or want your project's listing changed/removed? Open an issue or PR — we'll fix it fast. 💙
  1104. <br>
  1105. ---
  1106. <br>
  1107. <div align="center">
  1108. ### Made with purpose by [Hieu Ngo](https://www.linkedin.com/in/hieu-minh-ngo-hieupc/)
  1109. <p>
  1110. <a href="https://www.linkedin.com/in/hieu-minh-ngo-hieupc/"><img src="https://img.shields.io/badge/LinkedIn-Hieu_Ngo-0A66C2?style=for-the-badge&logo=linkedin&logoColor=white" alt="LinkedIn"></a>&nbsp;
  1111. <a href="mailto:hieu.ngo@chongluadao.vn"><img src="https://img.shields.io/badge/Email-hieu.ngo%40chongluadao.vn-0080ff?style=for-the-badge&logo=gmail&logoColor=white" alt="Email"></a>&nbsp;
  1112. <a href="https://chongluadao.vn"><img src="https://img.shields.io/badge/Web-chongluadao.vn-00c853?style=for-the-badge&logo=googlechrome&logoColor=white" alt="Website"></a>&nbsp;
  1113. <a href="https://github.com/7onez"><img src="https://img.shields.io/badge/GitHub-7onez-181717?style=for-the-badge&logo=github&logoColor=white" alt="GitHub"></a>&nbsp;
  1114. <a href="https://github.com/Zeroska"><img src="https://img.shields.io/badge/Contributor-Zeroska-6f42c1?style=for-the-badge&logo=github&logoColor=white" alt="Contributor: Zeroska"></a>
  1115. </p>
  1116. <sub>If this tool helps your work, consider giving it a star. It helps others find it.</sub>
  1117. </div>
intel_engine/harness/backend_select.py · Bundled script · 416 lines · 1 finding marked
  1. """backend_select.py — pick the agent backend the harness runs its LLM phases on (stdlib only).
  2. WHY
  3. The harness used to assume the Anthropic Agent SDK + a logged-in Claude Code CLI. On a VPS that
  4. is often false (no login, an expired OAuth session, root), and inside Codex or OMP the natural
  5. model is the host's own subscription. An unusable backend then surfaced as "every phase ✓ at
  6. $0.00, no assessment produced". This module decides ONCE, before sdk_compat is imported, and
  7. says why.
  8. HARNESS_BACKEND=claude|codex|omp|openai|deepseek|kimi|local explicit — honoured as-is
  9. HARNESS_BACKEND unset / auto resolve() below
  10. auto order: the host agent this process runs inside (CODEX_THREAD_ID → codex, OMPCODE → omp,
  11. CLAUDECODE → claude; inner host first because the markers are inherited), then claude, codex,
  12. omp, then an OpenAI-compatible key. Each candidate must pass `available()` (binary/module/key
  13. present, no model call) and `probe()` (auth actually works). Probes are cheap and cached in the
  14. environment (HARNESS_BACKEND is exported), so child processes never re-probe.
  15. ROOT
  16. Claude Code refuses --dangerously-skip-permissions (the SDK's bypassPermissions) as euid 0 unless
  17. IS_SANDBOX=1. The harness never relies on that flag for safety — every tool call passes the
  18. PreToolUse gate (audit.gate) — so for euid 0 we set IS_SANDBOX=1 for the spawned CLI and say so.
  19. """
  20. from __future__ import annotations
  21. import importlib.util
  22. import json
  23. import os
  24. import re
  25. import shutil
  26. import subprocess
  27. import sys
  28. import tempfile
  29. import time
  30. CLI_BACKENDS = ("codex", "omp")
  31. OPENAI_BACKENDS = ("openai", "deepseek", "kimi", "moonshot", "local", "ollama", "vllm", "lmstudio")
  32. KNOWN = ("claude",) + CLI_BACKENDS + OPENAI_BACKENDS
  33. PROBE_TIMEOUT = int(os.environ.get("HARNESS_PROBE_TIMEOUT", "90"))
  34. def _log(msg: str) -> None:
  35. print(f"[backend] {msg}", file=sys.stderr, flush=True)
  36. def is_root() -> bool:
  37. return hasattr(os, "geteuid") and os.geteuid() == 0
  38. def ensure_root_sandbox_env() -> bool:
  39. """Set IS_SANDBOX=1 for euid 0 so the Claude CLI accepts bypassPermissions. Returns True when
  40. it set the variable (callers log it once)."""
  41. if is_root() and not os.environ.get("IS_SANDBOX"):
  42. os.environ["IS_SANDBOX"] = "1"
  43. return True
  44. return False
  45. def host_agent(env=None) -> str | None:
  46. env = os.environ if env is None else env
  47. if env.get("CODEX_THREAD_ID") or env.get("CODEX_SESSION_ID"):
  48. return "codex"
  49. if env.get("OMPCODE"):
  50. return "omp"
  51. if env.get("CLAUDECODE"):
  52. return "claude"
  53. return None
  54. def _openai_key(env=None) -> str:
  55. env = os.environ if env is None else env
  56. return env.get("OPENAI_API_KEY") or env.get("DEEPSEEK_API_KEY") or env.get("MOONSHOT_API_KEY") or ""
  57. def available(name: str) -> tuple[bool, str]:
  58. """Cheap presence check — no model call."""
  59. if name == "claude":
  60. if importlib.util.find_spec("claude_agent_sdk") is None:
  61. return False, "claude_agent_sdk not installed in this Python"
  62. if os.environ.get("ANTHROPIC_API_KEY"):
  63. return True, "ANTHROPIC_API_KEY set"
  64. if shutil.which("claude"):
  65. return True, "claude CLI on PATH (subscription login)"
  66. return False, "no ANTHROPIC_API_KEY and no claude CLI"
  67. if name in CLI_BACKENDS:
  68. exe = shutil.which(name)
  69. return (True, f"{name} CLI at {exe}") if exe else (False, f"{name} CLI not on PATH")
  70. if name in OPENAI_BACKENDS:
  71. if name in ("local", "ollama", "vllm", "lmstudio"):
  72. return (bool(os.environ.get("OPENAI_BASE_URL")),
  73. "OPENAI_BASE_URL set" if os.environ.get("OPENAI_BASE_URL") else "OPENAI_BASE_URL unset")
  74. return (True, "API key set") if _openai_key() else (False, "no OPENAI/DEEPSEEK/MOONSHOT key")
  75. return False, f"unknown backend {name!r}"
  76. # Model-provider hosts reach the model DIRECTLY, never through the research proxy pool — see
  77. # references/model_hosts.json. Fallback = the conservative core set (Anthropic + OpenAI).
  78. _MODEL_HOSTS_FALLBACK = ["api.anthropic.com", ".anthropic.com", "claude.ai", ".claude.ai",
  79. "api.openai.com", ".openai.com", "chatgpt.com", ".chatgpt.com"]
  80. def _model_hosts() -> list:
  81. try:
  82. with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "references",
  83. "model_hosts.json"), encoding="utf-8") as f:
  84. hosts = json.load(f).get("no_proxy_hosts") or []
  85. return [h for h in hosts if isinstance(h, str) and h] or list(_MODEL_HOSTS_FALLBACK)
  86. except (OSError, ValueError):
  87. print("[backend] WARNING: references/model_hosts.json unreadable — using the core host list",
  88. file=sys.stderr)
  89. return list(_MODEL_HOSTS_FALLBACK)
  90. MODEL_HOSTS = _model_hosts()
  91. def bypass_proxy_for_models(env: dict) -> dict:
  92. """Append MODEL_HOSTS to NO_PROXY/no_proxy in `env` (in place; returned for chaining). Any
  93. HTTPS_PROXY the dispatcher exported for collectors stays set — only model hosts go direct."""
  94. for var in ("NO_PROXY", "no_proxy"):
  95. cur = [h.strip() for h in (env.get(var) or "").split(",") if h.strip()]
  96. env[var] = ",".join(cur + [h for h in MODEL_HOSTS if h not in cur])
  97. return env
  98. def _run(cmd: list, *, env=None, timeout=PROBE_TIMEOUT, stdin_text: str | None = None) -> tuple[int, str]:
  99. try:
  100. # stdin is ALWAYS closed or fed: `claude -p` / `codex exec` wait on an inherited open pipe
  101. # (cron, CI, a backgrounded shell) until the timeout.
  102. env = bypass_proxy_for_models(dict(env if env is not None else os.environ))
  103. p = subprocess.run(cmd, input=stdin_text if stdin_text is not None else "", capture_output=True,
  104. text=True, timeout=timeout, env=env, cwd=os.path.expanduser("~"))
  105. return p.returncode, (p.stdout or "") + (p.stderr or "")
  106. except subprocess.TimeoutExpired:
  107. return 124, f"probe timed out after {timeout}s"
  108. except OSError as e:
  109. return 127, str(e)
  110. # --------------------------------------------------------------- failure classification
  111. # A phase that dies on one of these is the BACKEND's problem (login, plan credits, provider rate
  112. # limit), not the case's — so the run can resume on another backend with nothing lost. Anything
  113. # else (a tool fault, a schema miss) is a real failure and must not trigger a backend switch.
  114. _FAILURE_KINDS = (
  115. ("auth", re.compile(r"authenticat|oauth|not logged in|log ?in required|unauthori[sz]ed|\b401\b|"
  116. r"invalid api key|api key (?:is )?(?:missing|invalid)|session expired", re.I)),
  117. ("quota", re.compile(r"out of credits|insufficient (?:credit|balance|quota|funds)|credit balance|"
  118. r"quota|usage limit|billing|\b402\b|payment required|plan limit|add credits", re.I)),
  119. ("rate", re.compile(r"rate.?limit|\b429\b|too many requests|overloaded", re.I)),
  120. )
  121. # Exit code the orchestrator uses for "the backend became unusable mid-run" (EX_TEMPFAIL): the case
  122. # is intact and resumable, so harness/cli.py may offer the next backend.
  123. FAILOVER_EXIT = 75
  124. class BackendUnavailable(RuntimeError):
  125. """The LLM backend itself stopped answering (auth / credits / rate limit)."""
  126. def __init__(self, backend: str, reason: str, kind: str = "unavailable"):
  127. super().__init__(f"{backend} unavailable ({kind}): {reason}")
  128. self.backend, self.reason, self.kind = backend, reason, kind
  129. def classify_failure(text: str) -> str | None:
  130. """'auth' | 'quota' | 'rate' when `text` is a backend-level failure, else None."""
  131. for kind, rx in _FAILURE_KINDS:
  132. if rx.search(text or ""):
  133. return kind
  134. return None
  135. def write_failover(path: str | None, backend: str, reason: str, kind: str) -> None:
  136. """Hand the failure to the parent (harness/cli.py) — a file, so the live log stays on the tty."""
  137. if not path:
  138. return
  139. try:
  140. with open(path, "w", encoding="utf-8") as f:
  141. json.dump({"backend": backend, "reason": reason, "kind": kind}, f)
  142. except OSError:
  143. pass
  144. def read_failover(path: str | None) -> dict:
  145. try:
  146. with open(path or "", encoding="utf-8") as f:
  147. d = json.load(f)
  148. return d if isinstance(d, dict) else {}
  149. except (OSError, ValueError):
  150. return {}
  151. def _said_ok(text: str) -> bool:
  152. """The probe answer is exactly OK (a substring test would accept 'token' or 'look')."""
  153. lines = [ln.strip().strip(".!`*\"'").lower() for ln in (text or "").splitlines() if ln.strip()]
  154. return "ok" in lines # a whole line, anywhere: CLIs print progress ("Working...") around it
  155. def _codex_probe_home() -> str:
  156. """A throwaway CODEX_HOME for the probe: the user's login, none of their skills/MCP/hooks, no
  157. shell — so the probe is one small model turn, not a full agent session."""
  158. home = tempfile.mkdtemp(prefix="intel-codex-probe-")
  159. with open(os.path.join(home, "config.toml"), "w", encoding="utf-8") as f:
  160. f.write('approval_policy = "never"\nsandbox_mode = "read-only"\nweb_search = "disabled"\n'
  161. "[features]\nshell_tool = false\nunified_exec = false\nmulti_agent = false\n"
  162. "plugins = false\napps = false\nhooks = false\nbrowser_use = false\n")
  163. src = os.path.join(os.environ.get("CODEX_HOME") or os.path.expanduser("~/.codex"), "auth.json")
  164. if os.path.exists(src):
  165. os.symlink(src, os.path.join(home, "auth.json"))
  166. return home
  167. def probe(name: str) -> tuple[bool, str]:
  168. """Does this backend actually ANSWER right now? One tiny real request per CLI — a login-status
  169. call is not enough: a logged-in account with no credits left passes it and dies mid-run."""
  170. if name == "claude":
  171. if os.environ.get("ANTHROPIC_API_KEY"):
  172. return True, "ANTHROPIC_API_KEY set (not probed)"
  173. env = dict(os.environ)
  174. if is_root():
  175. env.setdefault("IS_SANDBOX", "1")
  176. rc, out = _run(["claude", "-p", "Reply with exactly: OK", "--max-turns", "1",
  177. "--output-format", "json"], env=env)
  178. try:
  179. data = json.loads(out.strip().splitlines()[-1]) if out.strip() else {}
  180. except (ValueError, IndexError):
  181. data = {}
  182. if rc == 0 and data and not data.get("is_error") and _said_ok(str(data.get("result") or "")):
  183. return True, "claude CLI answered"
  184. reason = (data.get("result") if isinstance(data, dict) else "") or out.strip()[-300:]
  185. return False, f"claude CLI unusable: {reason or f'exit {rc}'}"
  186. if name == "codex":
  187. rc, out = _run(["codex", "login", "status"], timeout=30)
  188. if rc != 0 or "not logged in" in out.lower():
  189. return False, f"codex not logged in ({out.strip()[-200:] or f'exit {rc}'})"
  190. home = _codex_probe_home()
  191. try:
  192. rc, out = _run(["codex", "exec", "--json", "--skip-git-repo-check", "-"],
  193. env={**os.environ, "CODEX_HOME": home}, stdin_text="Reply with exactly: OK")
  194. finally:
  195. shutil.rmtree(home, ignore_errors=True)
  196. text, errs = "", []
  197. for line in out.splitlines():
  198. try:
  199. e = json.loads(line)
  200. except ValueError:
  201. continue
  202. item = e.get("item") or {}
  203. if e.get("type") == "turn.failed":
  204. errs.append(str((e.get("error") or {}).get("message") or e))
  205. elif item.get("type") == "agent_message":
  206. text = item.get("text") or text
  207. elif item.get("type") == "error" and "skills context budget" not in str(item.get("message", "")).lower():
  208. errs.append(str(item.get("message") or ""))
  209. if _said_ok(text) and not errs:
  210. return True, "codex answered"
  211. return False, f"codex unusable: {('; '.join(errs) or out.strip()[-200:] or f'exit {rc}')[:300]}"
  212. if name == "omp":
  213. rc, out = _run(["omp", "-p", "--no-tools", "--no-session", "--no-skills", "--no-rules",
  214. "--no-extensions", "--no-title", "Reply with exactly: OK"])
  215. if rc == 0 and _said_ok(out):
  216. return True, "omp answered"
  217. return False, f"omp unusable: {out.strip()[-300:] or f'exit {rc}'}"
  218. if name in OPENAI_BACKENDS:
  219. return available(name)
  220. return False, f"unknown backend {name!r}"
  221. def candidates(env=None) -> list:
  222. host = host_agent(env)
  223. order = ([host] if host else []) + ["claude", "codex", "omp", "openai"]
  224. seen, out = set(), []
  225. for n in order:
  226. if n not in seen:
  227. seen.add(n)
  228. out.append(n)
  229. return out
  230. # ------------------------------------------------------------ cheap checks + probe cache
  231. # A per-run check must cost nothing: `harness open` runs often, and a real "reply OK" request is a
  232. # whole agent turn (Codex: ~45k input tokens even in an isolated home). So a run uses each CLI's
  233. # free login-PRESENCE command, which does not prove the login works (an expired session whose
  234. # refresh fails may still read as logged in). The real request lives in `harness doctor`, whose
  235. # result is cached so the next runs reuse it. Whatever the cheap check misses — a dead refresh,
  236. # credits gone before or during the round — fails the first affected phase in seconds and becomes a
  237. # failover (classify_failure → FAILOVER_EXIT → harness/cli.py).
  238. PROBE_CACHE = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
  239. "MEMORY", "backend_probe.json") # engine MEMORY/ — gitignored
  240. PASS_TTL = int(os.environ.get("HARNESS_PROBE_PASS_TTL") or 1800) # a doctor "answered" is trusted 30 min
  241. FAIL_TTL = int(os.environ.get("HARNESS_PROBE_FAIL_TTL") or 600) # a failure is skipped for 10 min
  242. def _cache_load() -> dict:
  243. try:
  244. with open(PROBE_CACHE, encoding="utf-8") as f:
  245. d = json.load(f)
  246. return d if isinstance(d, dict) else {}
  247. except (OSError, ValueError):
  248. return {}
  249. def _cache_put(name: str, ok: bool, why: str) -> None:
  250. d = _cache_load()
  251. d[name] = {"ok": bool(ok), "why": str(why)[:300], "ts": time.time()}
  252. try:
  253. os.makedirs(os.path.dirname(PROBE_CACHE), exist_ok=True)
  254. with open(PROBE_CACHE, "w", encoding="utf-8") as f:
  255. json.dump(d, f, indent=1)
  256. except OSError:
  257. pass
  258. def _cache_get(name: str):
  259. e = _cache_load().get(name)
  260. if not isinstance(e, dict):
  261. return None
  262. age = time.time() - float(e.get("ts") or 0)
  263. if age < (PASS_TTL if e.get("ok") else FAIL_TTL):
  264. return bool(e.get("ok")), f"{e.get('why')} (checked {int(age // 60)} min ago)"
  265. return None
  266. def mark_failed(name: str, why: str) -> None:
  267. """A backend that died mid-run is skipped by the next resolve (failover, next run) for FAIL_TTL."""
  268. _cache_put(name, False, f"failed mid-run: {why}")
  269. def quick_check(name: str) -> tuple[bool, str]:
  270. """Free login-PRESENCE checks — no model request, no tokens. NOT a verification: it reads the
  271. stored login, so an expired OAuth session whose refresh will fail (the Claude case we hit) or an
  272. account with no credits can still pass. That is acceptable only because such a backend fails the
  273. FIRST phase within seconds, which classify_failure turns into a failover. `doctor` (deep=True)
  274. is the real check."""
  275. if name == "claude":
  276. if os.environ.get("ANTHROPIC_API_KEY"):
  277. return True, "ANTHROPIC_API_KEY set"
  278. rc, out = _run(["claude", "auth", "status"], timeout=30)
  279. try:
  280. data = json.loads(out[out.index("{"):]) if "{" in out else {}
  281. except ValueError:
  282. data = {}
  283. if rc == 0 and data.get("loggedIn"):
  284. return True, (f"claude has a stored login ({data.get('authMethod') or 'unknown method'}) "
  285. f"— not verified; `harness doctor` tests it")
  286. return False, f"claude not logged in ({out.strip()[-160:] or f'exit {rc}'})"
  287. if name == "codex":
  288. rc, out = _run(["codex", "login", "status"], timeout=30)
  289. if rc == 0 and "not logged in" not in out.lower():
  290. return True, (f"codex has a stored login ({out.strip().splitlines()[-1][:60] if out.strip() else '?'}) "
  291. f"— not verified; `harness doctor` tests it")
  292. return False, f"codex not logged in ({out.strip()[-160:] or f'exit {rc}'})"
  293. # omp has no status command; an unusable login fails the first phase in seconds → failover.
  294. return available(name)
  295. def check(name: str, *, deep: bool = False) -> tuple[bool, str]:
  296. """deep=True (doctor): one real request, cached. deep=False (every run): a fresh cached result
  297. if there is one, else the free login-status check."""
  298. ok, why = available(name)
  299. if not ok:
  300. return ok, why
  301. if deep:
  302. ok, why = probe(name)
  303. if not ok and "timed out" in why:
  304. # A timeout says the PROBE did not finish (slow exit, network blip), not that the backend
  305. # failed — never cache it, or the next runs skip a backend that works.
  306. return False, f"{name} probe timed out — not cached; re-run `harness doctor` to retry"
  307. _cache_put(name, ok, why)
  308. return ok, why
  309. cached = _cache_get(name)
  310. return cached if cached is not None else quick_check(name)
  311. def resolve(*, do_probe: bool = True, exclude=(), deep: bool = False) -> tuple[str, str, list]:
  312. """(backend, why, tried). Raises SystemExit with every candidate's reason when none works.
  313. `exclude` skips backends that already failed this run (failover); it also ignores a pinned
  314. HARNESS_BACKEND, since the pinned one is the backend that just failed. `deep` sends each
  315. candidate one real request (doctor); the default is the free per-run check."""
  316. explicit = (os.environ.get("HARNESS_BACKEND") or "").strip().lower()
  317. if explicit and explicit != "auto" and not exclude:
  318. if explicit not in KNOWN:
  319. raise SystemExit(f"HARNESS_BACKEND={explicit!r} is not one of {', '.join(KNOWN)} or auto")
  320. return explicit, "HARNESS_BACKEND set explicitly", []
  321. skip = {e.lower() for e in exclude}
  322. tried = []
  323. for name in candidates():
  324. if name in skip:
  325. continue
  326. ok, why = check(name, deep=deep) if do_probe else available(name)
  327. tried.append((name, ok, why))
  328. if ok:
  329. return name, why, tried
  330. lines = "\n".join(f" - {n}: {w}" for n, _ok, w in tried)
  331. raise SystemExit(
  332. "no usable LLM backend for the harness:\n" + lines + "\n"
  333. "Fix one of: `claude /login` (or export ANTHROPIC_API_KEY) · `codex login` · configure omp "
  334. "(`omp` → /login) · export OPENAI_API_KEY/DEEPSEEK_API_KEY (+ OPENAI_BASE_URL). "
  335. "Or pin one with HARNESS_BACKEND=<name>.")
  336. def ensure_env() -> str:
  337. """Resolve once and export HARNESS_BACKEND so sdk_compat (and every child process) agree."""
  338. # Model traffic direct, collector traffic still on the research pool: every model CLI this run
  339. # spawns (the SDK's claude, codex/omp phases) inherits this process env.
  340. bypass_proxy_for_models(os.environ)
  341. name, why, tried = resolve()
  342. for n, ok, w in tried:
  343. if not ok:
  344. _log(f"skip {n}: {w}")
  345. os.environ["HARNESS_BACKEND"] = name
  346. if name == "claude" and ensure_root_sandbox_env():
  347. _log("running as root: set IS_SANDBOX=1 so the Claude CLI accepts headless permissions "
  348. "(every tool call is still gated by audit.gate)")
  349. _log(f"using {name} — {why}")
  350. return name
intel_engine/harness/cli_backend.py · Bundled script · 485 lines · 1 finding marked
  1. """cli_backend.py — run the harness phases on the Codex or OMP CLI (the host's own login).
  2. WHY
  3. On a VPS the Anthropic SDK path needs a Claude Code login or an API key. Codex and OMP users
  4. already have a subscription session on the box. This backend drives that CLI headlessly:
  5. every phase is one `codex exec --json` / `omp -p --mode json` process whose ONLY tools are the
  6. harness's own, served over stdio by harness/mcp_server.py — the same typed, audit-gated tool
  7. surface every other front-end uses (audit.gate runs inside the server on every call).
  8. ISOLATION (the security property; see the failure mode this was designed against)
  9. A stock Codex session with a refused MCP call routed around the refusal: it read the user's
  10. skills and shelled out to ANOTHER agent to make the call. So each phase runs in an isolated
  11. home (CODEX_HOME / OMP profile dir under cases/<case>/harness_home/): no user skills, hooks,
  12. plugins or AGENTS.md; shell / exec / browser / web-search / multi-agent tools disabled; only
  13. the `intel` MCP server, with its tools pre-approved. The event stream is the judge, never the
  14. final text: a phase FAILS if any command/shell execution item appears, or if a tool call was
  15. refused by the host's approval layer (that means the gate was bypassed, not that the tool said
  16. no — a harness denial arrives as an isError tool RESULT, which the model is meant to read).
  17. STRUCTURED OUTPUT
  18. Like openai_backend: the Assess phase gets an `emit_result` tool (served by the MCP server via
  19. HARNESS_EMIT_SCHEMA_FILE) and must call it; the server writes the payload to a file this backend
  20. reads back and validates against the required keys. Retries re-prompt in the same session.
  21. SESSIONS
  22. `resume` maps to `codex exec resume <thread_id>` / `omp -p --resume <session_id>` so
  23. Correlate → Verify → Assess share one transcript, as on the SDK path.
  24. CONFIG (env)
  25. HARNESS_BACKEND=codex|omp the switch (backend_select exports it)
  26. HARNESS_CLI_MODEL_MAP JSON {tier: model-id} (e.g. {"haiku":"gpt-5-codex-mini"})
  27. HARNESS_CLI_TIMEOUT seconds per phase process (default 1800)
  28. HARNESS_CLI_BIN override the executable path
  29. INTEL_PY Python for the MCP server launcher (see harness/mcp-server)
  30. """
  31. from __future__ import annotations
  32. import asyncio
  33. import json
  34. import os
  35. import re
  36. import shutil
  37. import subprocess
  38. import sys
  39. import tempfile
  40. import uuid
  41. from dataclasses import dataclass, field
  42. from typing import Any, Callable, Optional
  43. HERE = os.path.dirname(os.path.abspath(__file__))
  44. ROOT = os.path.dirname(HERE)
  45. sys.path.insert(0, HERE)
  46. import audit # noqa: E402 — the phase context the orchestrator bound (case/phase/hostile)
  47. from openai_backend import ( # noqa: E402 — reuse the shim's SDK-shaped surface, one implementation
  48. AgentDefinition,
  49. AssistantMessage,
  50. ClaudeAgentOptions,
  51. HookMatcher,
  52. ResultMessage,
  53. SdkMcpServer,
  54. SdkMcpTool,
  55. TextBlock,
  56. ToolAnnotations,
  57. ToolUseBlock,
  58. _env_json,
  59. _missing_required,
  60. create_sdk_mcp_server,
  61. tool,
  62. )
  63. __all__ = ["AgentDefinition", "AssistantMessage", "ClaudeAgentOptions", "HookMatcher", "ResultMessage",
  64. "ToolAnnotations", "ToolUseBlock", "TextBlock", "SdkMcpServer", "SdkMcpTool",
  65. "create_sdk_mcp_server", "tool", "query", "PhaseRun", "parse_codex_events",
  66. "parse_omp_events", "judge_events", "codex_config", "omp_mcp_config", "build_command"]
  67. BACKEND = (os.environ.get("HARNESS_BACKEND") or "codex").strip().lower()
  68. PHASE_TIMEOUT = int(os.environ.get("HARNESS_CLI_TIMEOUT", "1800"))
  69. STRUCT_RETRIES = int(os.environ.get("HARNESS_STRUCT_RETRIES", "3"))
  70. EMIT_NAME = "emit_result"
  71. SERVER_NAME = "intel"
  72. MCP_LAUNCHER = os.path.join(HERE, "mcp-server")
  73. # Tool-name normalisation: hosts prefix MCP tools differently (codex: server-qualified item
  74. # fields; omp: `mcp__intel_<tool>`), the ledger/policy use the bare name.
  75. _PREFIX_RE = re.compile(r"^(?:mcp__)?(?:%s)?[_:]*" % SERVER_NAME)
  76. def _bare(name: str) -> str:
  77. n = (name or "").split("__")[-1]
  78. return _PREFIX_RE.sub("", n) if n.startswith(SERVER_NAME) else n
  79. def _log(msg: str) -> None:
  80. print(msg, file=sys.stderr, flush=True)
  81. # ---------------------------------------------------------------- host configs
  82. def codex_config(*, mcp_launcher: str = MCP_LAUNCHER, timeout: int = PHASE_TIMEOUT,
  83. env: Optional[dict] = None) -> str:
  84. """The whole config.toml for an isolated CODEX_HOME: MCP-only, no shell, tools pre-approved."""
  85. env_lines = "".join(f'{k} = "{v}"\n' for k, v in (env or {}).items())
  86. return (
  87. 'approval_policy = "never"\n'
  88. 'sandbox_mode = "read-only"\n'
  89. 'web_search = "disabled"\n'
  90. "[features]\n"
  91. "shell_tool = false\nunified_exec = false\nmulti_agent = false\nbrowser_use = false\n"
  92. "computer_use = false\napps = false\nplugins = false\nimage_generation = false\n"
  93. "hooks = false\nskill_search = false\ntool_suggest = false\nview_image = false\n"
  94. "code_mode_host = true\n"
  95. f"[mcp_servers.{SERVER_NAME}]\n"
  96. f'command = "{mcp_launcher}"\n'
  97. f"tool_timeout_sec = {timeout}\nstartup_timeout_sec = 90\n"
  98. 'default_tools_approval_mode = "approve"\n'
  99. + (f"[mcp_servers.{SERVER_NAME}.env]\n{env_lines}" if env_lines else "")
  100. )
  101. def omp_mcp_config(*, mcp_launcher: str = MCP_LAUNCHER, timeout: int = PHASE_TIMEOUT,
  102. env: Optional[dict] = None) -> dict:
  103. srv: dict = {"command": mcp_launcher, "timeout": timeout * 1000}
  104. if env:
  105. srv["env"] = dict(env)
  106. return {"mcpServers": {SERVER_NAME: srv}}
  107. def _phase_home(case: Optional[str]) -> str:
  108. base = os.path.join(ROOT, "cases", case, "harness_home") if case else os.path.join(tempfile.gettempdir(), "intel-harness-home")
  109. os.makedirs(base, exist_ok=True)
  110. return base
  111. def _session_link(base: str, backend: str, sid: str) -> str:
  112. return os.path.join(base, f"{backend}_bysession", sid)
  113. def _prepare_home(backend: str, case: Optional[str], server_env: dict,
  114. resume: Optional[str] = None) -> tuple[str, dict, str, Optional[str]]:
  115. """Write an ISOLATED host config for one phase; return (workdir, extra process env, slot,
  116. sessions_dir).
  117. A fresh phase gets its OWN new slot (a uuid dir under cases/<case>/harness_home/<backend>_runs/)
  118. holding config.toml / .omp/mcp.json, so concurrent phases (parallel cluster judges) never share
  119. a mutable config file and one phase's HARNESS_EMIT_OUT_FILE / HARNESS_PHASE / INTEL_CASE can
  120. never land in another's. A RESUMED phase (Correlate -> Verify -> Assess of one cluster) reuses
  121. the slot that created its session — the host CLIs bind a session to the directory it was started
  122. in (omp refuses to resume a session whose directory is gone) — and rewrites that slot's config
  123. for the new phase. That is race-free because one chain's phases run strictly in sequence; other
  124. chains resume their own slots. `record_session_slot` links a session id to its slot after the
  125. first call; `sessions_dir` (one per case+backend) is the transcript store omp is pointed at with
  126. --session-dir and codex reads through a `sessions` symlink in each CODEX_HOME."""
  127. base = _phase_home(case)
  128. sessions = os.path.join(base, f"{backend}_sessions")
  129. os.makedirs(sessions, exist_ok=True)
  130. slot = ""
  131. if resume:
  132. link = _session_link(base, backend, resume)
  133. if os.path.isdir(link):
  134. slot = os.path.realpath(link)
  135. if not slot:
  136. slot = os.path.join(base, f"{backend}_runs", uuid.uuid4().hex)
  137. os.makedirs(slot, exist_ok=True)
  138. if backend == "codex":
  139. with open(os.path.join(slot, "config.toml"), "w", encoding="utf-8") as f:
  140. f.write(codex_config(env=server_env))
  141. auth_src = os.path.join(os.environ.get("CODEX_HOME") or os.path.expanduser("~/.codex"), "auth.json")
  142. for src, name in ((auth_src, "auth.json"), (sessions, "sessions")):
  143. dst = os.path.join(slot, name)
  144. if os.path.exists(src) and not os.path.lexists(dst):
  145. try:
  146. os.symlink(src, dst) # reuse the login / shared transcript store, never copy
  147. except OSError:
  148. pass
  149. work = os.path.join(slot, "work")
  150. os.makedirs(work, exist_ok=True)
  151. return work, {"CODEX_HOME": slot}, slot, None
  152. if backend == "omp":
  153. os.makedirs(os.path.join(slot, ".omp"), exist_ok=True)
  154. with open(os.path.join(slot, ".omp", "mcp.json"), "w", encoding="utf-8") as f:
  155. json.dump(omp_mcp_config(env=server_env), f, indent=1)
  156. return slot, {"OMP_MCP_TIMEOUT_MS": str(PHASE_TIMEOUT * 1000), "OMP_MCP_REQUIRE_READY": "1"}, slot, sessions
  157. raise SystemExit(f"cli_backend: unsupported backend {backend!r}")
  158. def record_session_slot(backend: str, case: Optional[str], sid: str, slot: str) -> None:
  159. """Link `sid` to the slot it was created in, so a later resume of that session starts in the
  160. same directory. Per-session symlink = atomic, no shared map file for concurrent phases to race."""
  161. if not sid or not slot:
  162. return
  163. link = _session_link(_phase_home(case), backend, sid)
  164. os.makedirs(os.path.dirname(link), exist_ok=True)
  165. try:
  166. os.symlink(slot, link)
  167. except OSError:
  168. pass
  169. def build_command(backend: str, *, model: Optional[str], resume: Optional[str],
  170. session_dir: Optional[str] = None) -> list:
  171. """The host CLI invocation. The prompt is NOT an argument — it is piped on stdin (a phase's
  172. system prompt is a whole skill file; Linux caps one argv element at 128 KiB)."""
  173. exe = os.environ.get("HARNESS_CLI_BIN") or shutil.which(backend) or backend
  174. if backend == "codex":
  175. cmd = [exe, "exec", "--json", "--skip-git-repo-check"]
  176. if model:
  177. cmd += ["-m", model]
  178. if resume:
  179. cmd += ["resume", resume]
  180. return cmd + ["-"]
  181. if backend == "omp":
  182. cmd = [exe, "-p", "--mode", "json", "--no-tools", "--no-skills", "--no-rules",
  183. "--no-extensions", "--no-title", "--approval-mode", "yolo"]
  184. if session_dir:
  185. cmd += ["--session-dir", session_dir]
  186. if model:
  187. cmd += ["--model", model]
  188. if resume:
  189. cmd += ["--resume", resume]
  190. return cmd
  191. raise SystemExit(f"cli_backend: unsupported backend {backend!r}")
  192. # ---------------------------------------------------------------- event parsing
  193. @dataclass
  194. class PhaseRun:
  195. session_id: str = ""
  196. tool_calls: list = field(default_factory=list) # [(name, args)]
  197. refused: list = field(default_factory=list) # host-level refusals (gate bypass risk)
  198. executions: list = field(default_factory=list) # shell/command items — must be empty
  199. final_text: str = ""
  200. usage: dict = field(default_factory=dict)
  201. errors: list = field(default_factory=list)
  202. def _events(stream: str):
  203. for line in stream.splitlines():
  204. line = line.strip()
  205. if not line.startswith("{"):
  206. continue
  207. try:
  208. yield json.loads(line)
  209. except ValueError:
  210. continue
  211. def parse_codex_events(stream: str) -> PhaseRun:
  212. run = PhaseRun()
  213. for e in _events(stream):
  214. t = e.get("type")
  215. if t == "thread.started":
  216. run.session_id = e.get("thread_id") or ""
  217. elif t == "turn.completed":
  218. run.usage = e.get("usage") or {}
  219. elif t == "turn.failed":
  220. run.errors.append(str((e.get("error") or {}).get("message") or e))
  221. elif t == "item.completed":
  222. item = e.get("item") or {}
  223. it = item.get("type")
  224. if it == "mcp_tool_call":
  225. if (item.get("server") or SERVER_NAME) != SERVER_NAME:
  226. run.executions.append(f"mcp:{item.get('server')}/{item.get('tool')}")
  227. continue
  228. run.tool_calls.append((_bare(item.get("tool") or ""), item.get("arguments") or {}))
  229. err = item.get("error") or {}
  230. if item.get("status") == "failed" and "approval" in str(err.get("message", "")).lower():
  231. run.refused.append(item.get("tool") or "")
  232. elif it in ("command_execution", "local_shell_call", "shell_call", "web_search",
  233. "file_change", "collab_tool_call"):
  234. run.executions.append(item.get("command") or it)
  235. elif it == "agent_message":
  236. run.final_text = item.get("text") or run.final_text
  237. elif it == "error":
  238. msg = str(item.get("message") or "")
  239. if "skills context budget" not in msg.lower():
  240. run.errors.append(msg)
  241. return run
  242. def parse_omp_events(stream: str) -> PhaseRun:
  243. run = PhaseRun()
  244. for e in _events(stream):
  245. t = e.get("type")
  246. if t == "session":
  247. run.session_id = e.get("id") or ""
  248. elif t == "tool_execution_start":
  249. name = e.get("toolName") or ""
  250. if name.startswith(f"mcp__{SERVER_NAME}_"):
  251. run.tool_calls.append((_bare(name), e.get("args") or {}))
  252. else: # built-ins that survive --no-tools, or ANOTHER MCP server
  253. run.executions.append(name)
  254. elif t == "tool_execution_end":
  255. res = e.get("result") or {}
  256. txt = " ".join(b.get("text", "") for b in (res.get("content") or []) if isinstance(b, dict))
  257. if "requires approval" in txt.lower() or "approval denied" in txt.lower():
  258. run.refused.append(e.get("toolName") or "")
  259. elif t in ("message_end", "turn_end"):
  260. m = e.get("message") or {}
  261. if m.get("role") == "assistant":
  262. for b in m.get("content") or []:
  263. if isinstance(b, dict) and b.get("type") == "text" and b.get("text"):
  264. run.final_text = b["text"]
  265. u = m.get("usage")
  266. if isinstance(u, dict):
  267. run.usage = u
  268. elif t == "error":
  269. run.errors.append(str(e.get("message") or e))
  270. return run
  271. def judge_events(run: PhaseRun) -> Optional[str]:
  272. """None when the phase is acceptable; else the reason it must be treated as failed."""
  273. if run.executions:
  274. return ("host executed non-MCP tools (shell/command/web) — the isolation contract is broken: "
  275. + ", ".join(str(x)[:80] for x in run.executions[:3]))
  276. if run.refused:
  277. return "host approval layer refused MCP tool(s) (gate bypassed): " + ", ".join(run.refused[:3])
  278. if run.errors:
  279. return "; ".join(run.errors[:3])
  280. return None
  281. # ---------------------------------------------------------------- the phase runner
  282. def _model_for(tier: Optional[str]) -> Optional[str]:
  283. table = _env_json("HARNESS_CLI_MODEL_MAP")
  284. return table.get(tier or "") or None # unmapped tier → the host's default model
  285. def _emit_file(case: Optional[str]) -> str:
  286. return os.path.join(_phase_home(case), f"emit_{uuid.uuid4().hex}.json")
  287. _SECRET_HINTS = ("KEY", "TOKEN", "SECRET", "PASSWORD", "PASSWD", "PROXY", "PROXIES", "AUTH", "COOKIE", "CREDENTIAL")
  288. def _looks_secret(name: str) -> bool:
  289. """A conservative name test for values that must never be written into a host config file."""
  290. n = name.upper()
  291. return any(h in n for h in _SECRET_HINTS)
  292. _HOST_MARKERS = ("CLAUDECODE", "OMPCODE", "CODEX_", "CLAUDE_CODE", "OMP_", "PI_")
  293. def _child_env(extra: dict) -> dict:
  294. """The phase process env: the caller's, minus host-agent markers (a nested run must not think
  295. it is inside Codex/OMP/Claude Code and inherit their session/config), plus `extra`."""
  296. env = {k: v for k, v in os.environ.items()
  297. if not any(k == m or k.startswith(m) for m in _HOST_MARKERS)}
  298. env.update(extra)
  299. return env
  300. async def _run_phase(cmd: list, *, cwd: str, env: dict, timeout: int,
  301. stdin_text: Optional[str] = None) -> tuple[int, str, str]:
  302. def _go():
  303. p = subprocess.run(cmd, cwd=cwd, env=env, capture_output=True, text=True, timeout=timeout,
  304. input=stdin_text if stdin_text is not None else "",
  305. **({} if stdin_text is not None else {"stdin": subprocess.DEVNULL}))
  306. return p.returncode, p.stdout or "", p.stderr or ""
  307. try:
  308. return await asyncio.to_thread(_go)
  309. except subprocess.TimeoutExpired as e:
  310. return 124, (e.stdout or "") if isinstance(e.stdout, str) else "", f"phase timed out after {timeout}s"
  311. def _phase_context() -> tuple[Optional[str], str, bool]:
  312. """(case, phase, hostile) as the orchestrator bound them via audit.set_context — the SDK path
  313. closes over these in its PreToolUse hook; here they must travel to the server process."""
  314. try:
  315. ctx = audit._CTX.get() or {}
  316. except Exception: # noqa: BLE001
  317. ctx = {}
  318. hostile = ctx.get("hostile")
  319. if hostile is None:
  320. hostile = os.environ.get("HARNESS_HOSTILE", "").strip().lower() in ("1", "true", "yes", "on")
  321. return ctx.get("case") or os.environ.get("INTEL_CASE") or None, str(ctx.get("phase") or "interactive"), bool(hostile)
  322. async def query(*, prompt: str, options: ClaudeAgentOptions):
  323. """Async generator mirroring claude_agent_sdk.query on top of a host CLI process."""
  324. backend = BACKEND
  325. case, phase, hostile = _phase_context()
  326. want_struct = bool(options.output_format)
  327. schema = (options.output_format or {}).get("schema") or {} if want_struct else {}
  328. allowed = sorted({_bare(t) for t in (options.allowed_tools or [])})
  329. # The server process is the harness for this phase: the HARNESS_* / INTEL_* knobs the operator
  330. # set (HARNESS_KB scratch KB, HARNESS_NO_ENRICH no-credit smoke, HARNESS_FORCE, …) must reach
  331. # it, or a "no-spend smoke" spends. Phase-specific values are layered on top and win. NEVER a
  332. # secret: this dict is written in clear into the host's config file under the case dir, so
  333. # proxy URLs (CTI_PROXY/CTI_PROXIES carry user:pass@host), API keys and tokens stay out — the
  334. # collectors re-load keys and the proxy pool from the repo .env themselves. CTI_API_KEYS_ENV
  335. # (a PATH to that file) is the one CTI_* value forwarded.
  336. server_env = {k: v for k, v in os.environ.items()
  337. if k.startswith(("HARNESS_", "INTEL_")) and k != "HARNESS_BACKEND"
  338. and not _looks_secret(k)}
  339. if os.environ.get("CTI_API_KEYS_ENV"):
  340. server_env["CTI_API_KEYS_ENV"] = os.environ["CTI_API_KEYS_ENV"]
  341. server_env.update({
  342. "HARNESS_ALLOWED_TOOLS": ",".join(allowed),
  343. "HARNESS_HOSTILE": "1" if hostile else "",
  344. "HARNESS_PHASE": phase,
  345. "HARNESS_BACKEND": backend,
  346. })
  347. if case:
  348. server_env["INTEL_CASE"] = case
  349. emit_path = ""
  350. if want_struct:
  351. emit_path = _emit_file(case)
  352. with open(emit_path + "schema.json", "w", encoding="utf-8") as f:
  353. json.dump(schema, f)
  354. server_env["HARNESS_EMIT_SCHEMA_FILE"] = emit_path + "schema.json"
  355. server_env["HARNESS_EMIT_OUT_FILE"] = emit_path
  356. cwd, extra_env, slot, sessions_dir = _prepare_home(backend, case, server_env, resume=options.resume)
  357. env = _child_env(extra_env)
  358. session_dir = sessions_dir if backend == "omp" else None
  359. system = options.system_prompt or ""
  360. user = prompt
  361. if want_struct:
  362. user += ("\n\nWhen finished, call the `emit_result` tool exactly once with the final structured "
  363. "result; it is the only accepted way to finish.")
  364. full_prompt = (f"<system>\n{system}\n</system>\n\n{user}" if system and not options.resume else user)
    • Note: Usual in code (tag names in paths and patterns): Contains a <system> block, a common wrapper for instructions hidden in a tool description
    • Line 424 on GitHub
  365. full_prompt += ("\n\nRULES: use only the provided `intel` MCP tools; you have no shell, files or web "
  366. "browsing, and must not try to obtain them.")
  367. model = _model_for(options.model)
  368. structured: Optional[dict] = None
  369. sid = options.resume or ""
  370. final_text = ""
  371. failure: Optional[str] = None
  372. usage: dict = {}
  373. tries = 0
  374. prompt_now = full_prompt
  375. while True:
  376. cmd = build_command(backend, model=model, resume=sid or None, session_dir=session_dir)
  377. rc, out, err = await _run_phase(cmd, cwd=cwd, env=env, timeout=PHASE_TIMEOUT, stdin_text=prompt_now)
  378. run = parse_codex_events(out) if backend == "codex" else parse_omp_events(out)
  379. sid = run.session_id or sid
  380. usage = run.usage or usage
  381. for name, args in run.tool_calls:
  382. yield AssistantMessage(content=[ToolUseBlock(name=name, input=args, id="")])
  383. failure = judge_events(run)
  384. if failure is None and rc != 0 and not run.tool_calls and not run.final_text:
  385. failure = f"{backend} exited {rc}: {(err or out).strip()[-300:]}"
  386. if failure is None and rc != 0 and "authenticat" in (err + out).lower():
  387. failure = f"{backend} auth failure: {(err or out).strip()[-200:]}"
  388. final_text = run.final_text or final_text
  389. if failure or not want_struct:
  390. break
  391. if os.path.exists(emit_path):
  392. try:
  393. structured = json.load(open(emit_path, encoding="utf-8"))
  394. except ValueError:
  395. structured = None
  396. if structured is not None and not _missing_required(schema, structured):
  397. break
  398. tries += 1
  399. if tries > STRUCT_RETRIES or not sid:
  400. break
  401. missing = _missing_required(schema, structured) if structured else ["<no emit_result call>"]
  402. prompt_now = ("Call the `emit_result` tool now with the COMPLETE final result matching the "
  403. f"required schema. Missing: {', '.join(missing)}.")
  404. if failure:
  405. subtype = "error"
  406. final_text = failure # the REASON, not the model's last message (that hid "out of credits")
  407. _log(f" ✗ phase failed: {failure}")
  408. elif want_struct:
  409. subtype = "success" if structured is not None else "error"
  410. else:
  411. subtype = "success"
  412. if emit_path:
  413. for p in (emit_path, emit_path + "schema.json"):
  414. try:
  415. os.remove(p)
  416. except OSError:
  417. pass
  418. if sid:
  419. record_session_slot(backend, case, sid, slot) # keep the slot: a later phase resumes here
  420. else:
  421. shutil.rmtree(slot, ignore_errors=True) # no session to resume — config dir is spent
  422. yield ResultMessage(subtype=subtype, session_id=sid, total_cost_usd=None,
  423. structured_output=structured, result=final_text)

Version history

  • v2.12.010/19/2026

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.