penthera
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/danoszz/penthera with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/danoszz/penthera'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/danoszz--penthera)HTML
<a href="/listings/danoszz--penthera"><img src="/api/listings/danoszz--penthera/badge.svg" alt="findsafeskills safety rating" /></a>Install
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 22 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- {
- "name": "penthera",
- "owner": {
- "name": "Daan van der Zwaag",
- "url": "https://github.com/danoszz"
- },
- "metadata": {
- "description": "Offline cyber security agent for your apps and AI coding tools. Open R&D: measured in public, improved every release.",
- "version": "2.4.0"
- },
- "plugins": [
- {
- "name": "penthera",
- "description": "Offline cyber security agent for your apps and AI coding tools. Open R&D: measured in public, improved every release.",
- "source": "./plugin",
- "strict": true,
- "skills": [
- "./skills/penthera"
- ]
- }
- ]
- }
README.md · README · 621 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <div align="center">
- <img src="site/media/social/og.png" alt="Your elite cyber security agent. Audit your app/site/code before you publish to protect yourself, and your users. Free, open source, offline. Penthera." width="840">
- <br/><br/>
- # Penthera: your open-source cyber security agent
- The security review you never had time for. Offline, rule-based, and you can verify every finding yourself.
- **[penthera.org](https://penthera.org)**: check your live site in a few seconds. Passive, nothing stored.
- Daily notes from Penthera: **[x.com/penthera_org](https://x.com/penthera_org)**
- **Who builds it:** [Daan van der Zwaag](docs/trust.md#who-builds-it), part-time and in the open.
- **How it compares:** [next to Snyk, Socket, Semgrep, ZAP and GitHub](#how-it-fits-with-other-tools).
- **New in 2.4:** `penthera --card` makes a security receipt of your repo to post, every surface counts one set of problems with the same words, and 13 ways to hide a finding or stall a scan are closed ([what changed](CHANGELOG.md)).
- **From your coding agent:** add Penthera ([one line per agent](#start-in-30-seconds)), then say "Hey Claude, run Penthera on this repo."
- **132** readable rules · **6** read-only MCP tools · **0** runtime dependencies · **0** requests past loopback in the benchmark · **MIT**, no telemetry
- <br/>
- [](https://www.npmjs.com/package/penthera)
- [](https://www.npmjs.com/package/penthera)
- [](https://github.com/danoszz/penthera/stargazers)
- 
- 
- [](https://github.com/danoszz/penthera/actions/workflows/ci.yml)
- [](https://scorecard.dev/viewer/?uri=github.com/danoszz/penthera)
- [](https://socket.dev/npm/package/penthera)
- [](https://glama.ai/mcp/servers/danoszz/penthera)
- <br/>
- <img src="assets/demo.gif" alt="A 12-second loop: npx penthera sets up a project, an offline scan runs in the terminal, the dashboard shows the verdict in Simple mode and the triage matrix in Advanced mode, and a fix is approved" width="760">
- <br/><br/>
- | `penthera ui`, Simple, dark | `penthera ui`, Advanced, light |
- |---|---|
- |  |  |
- </div>
- ---
- ## What it is
- Penthera checks your app, its dependencies and your AI coding tools on your own
- machine. It fixes what it can, with your approval. Every finding comes from a
- rule you can read, with its evidence at `file:line`. A model may propose; a rule
- decides.
- It is also an open R&D project. We measure every check in public and publish
- the results, good or bad, so you can see how far to trust it.
- It finds known classes of problems, not novel ones. It does not replace a human
- pentester.
- ## Start in 30 seconds
- You need Node.js 22 or later. No install, no account.
- Building with Cursor, Claude Code, Codex, VS Code, Windsurf, Gemini CLI, Zed,
- JetBrains or Kiro, or an app builder such as Lovable, Bolt, Base44, Replit or
- v0? [Penthera in your coding tool](docs/integrations.md) has the setup for
- each.
- **In your terminal.** Run this in your project folder:
- ```bash
- npx penthera
- ```
- It detects your setup offline, asks what to check, and gives you a grade per
- area and the three things to fix first. It scans your running app only after
- you confirm the app is yours. Nothing is committed or pushed
- ([guided start](docs/commands/init.md)).
- **In your coding agent.** Pick yours, then ask "scan my app for security
- issues". The MCP lines pin the version.
- | Agent | Run this |
- |---|---|
- | Claude Code, plugin (skill only, runs the CLI) | `claude plugin marketplace add danoszz/penthera`, then `claude plugin install penthera@penthera` |
- | Claude Code, MCP server only | `claude mcp add penthera -- npx -y penthera@2.4.0 mcp` |
- | Codex | `codex mcp add penthera -- npx -y penthera@2.4.0 mcp` |
- | GitHub Copilot CLI | `copilot plugin marketplace add danoszz/penthera`, then `copilot plugin install penthera@penthera` |
- | GitHub Copilot in VS Code | the `code --add-mcp` line in [mcp.md](docs/commands/mcp.md#github-copilot) |
- | Gemini CLI | `gemini extensions install https://github.com/danoszz/penthera` |
- | Cursor, Windsurf | the JSON in [mcp.md](docs/commands/mcp.md#cursor-and-windsurf) |
- | Any agent that reads skills | `npx skills add danoszz/penthera` |
- The skill tells your agent how to run Penthera; the MCP server gives it six
- read-only tools. More routes: [skill.md](docs/commands/skill.md),
- [mcp.md](docs/commands/mcp.md).
- **Where AI fits.** Rules decide every finding and every exit code. A model may
- only propose:
- | Role | Who | How |
- |---|---|---|
- | Decides | Penthera's rules, always | Every finding, every exit code |
- | Looks further (optional) | A model, with `--ai` | Proposes problems no rule describes; the rules check each one, and unchecked ones are listed apart and never fail CI ([AI mode](docs/ai-mode.md)) |
- | Fixes | Your own coding agent | Through the MCP server, the skill or `penthera fix --plan` |
- To run it in a container:
- `docker build -t penthera . && docker run --rm penthera https://your-app.example`.
- ## Public benchmarks
- We test Penthera on public datasets that other people built, next to the free
- open-source tools that do the same job, scored the same way. Each dataset was
- split in half before we looked: we improved Penthera on one half and scored it
- once on the other, sealed half. The rules for scoring and ranking were written
- down before that run ([protocol](docs/research/prereg/E11b-improvement-study.md)).
- <!-- public-benchmarks:start -->
- | Benchmark | Score | Penthera 1.9.0 | **Penthera 2.0.0** | Rank among free tools | Best other free tool |
- |---|---|---|---|---|---|
- | [Leaked secrets (Samsung CredData)](https://penthera.org/benchmarks/secrets) | F1 | 0.18 | **0.74** | **2nd of 10** | deepsecrets 0.78 |
- | [Malicious npm packages (DataDog)](https://penthera.org/benchmarks/npm-malware) | caught minus false alarms | 0.57 | **0.71** | **1st of 6, tied 1st to 3rd** ⁴ | malcontent 0.63 |
- | [Malicious PyPI packages (DataDog)](https://penthera.org/benchmarks/pypi-malware) | caught minus false alarms | 0.74 | **0.82** | **1st of 8, tied 1st to 3rd** ⁴ | GuardDog 0.74 |
- | [Injection bugs in JS libraries (SecBench.js)](https://penthera.org/benchmarks/js-injection) | found in the right file | 0.16 | **0.70** | **1st of 8, tied 1st to 4th** ⁴ | odgen 0.68 |
- ⁴ A tied range is a rank interval: the range Penthera's rank fell in, in 95% of 10,000 bootstrap redraws of each tool's counts (on SecBench.js, of its 64 test bugs). Penthera alone came out first in 76% of the redraws on npm, 82% on PyPI and 60% on SecBench.js. Exploratory: computed after the registered ranking.
- Leaked secrets: 2nd by F1, the registered score; 1st by F0.5, which weighs precision more (0.82 against deepsecrets 0.80); 2nd by F2, which weighs recall more (0.68 against deepsecrets 0.77).
- Some ranked tools are barely in the contest. Other ranked tools scoring below 0.10: 1 of 9 on CredData (trufflehog); 3 of 5 on npm (clamav, yara, ossgadget); 4 of 7 on PyPI (clamav, ossgadget, bandit4mal, aura); 1 of 7 on SecBench.js (njsscan).
- If 1 package in 1,000 were malicious, the share of Penthera's malware flags that would be right: npm, about 2.7% at the top of the false-alarm interval (it raised no false alarm on that sample); PyPI, about 1.6% at the measured false-alarm rate, 0.9% at the top of its interval. Exploratory, like the rank intervals.
- <details><summary>Every tool, every number (95% intervals)</summary>
- #### Leaked secrets (Samsung CredData)
- | Rank | Tool | Precision | Recall | F1 |
- |---:|---|---|---|---|
- | 1 | deepsecrets 2.2.0 | 0.80 [0.79, 0.82] | 0.77 [0.75, 0.78] | 0.78 |
- | 2 | **Penthera** 2.0.0 | 0.89 [0.87, 0.90] | 0.64 [0.62, 0.66] | 0.74 |
- | 3 | noseyparker 0.24.0 | 0.76 [0.74, 0.78] | 0.62 [0.60, 0.64] | 0.69 |
- | 4 | gitleaks 8.30.1 | 0.85 [0.84, 0.86] | 0.37 [0.35, 0.39] | 0.52 |
- | 5 | ripsecrets 0.1.11 | 0.65 [0.64, 0.67] | 0.35 [0.33, 0.37] | 0.45 |
- | 6 | kingfisher 2.14.0 | 0.96 [0.94, 0.97] | 0.29 [0.28, 0.31] | 0.45 |
- | 7 | detect-secrets 1.5.0 | 0.30 [0.29, 0.31] | 0.53 [0.51, 0.56] | 0.38 |
- | 8 | secretlint 13.0.7 | 0.92 [0.89, 0.94] | 0.19 [0.18, 0.21] | 0.32 |
- | 9 | whispers 2.4.0 | 0.46 [0.43, 0.50] | 0.09 [0.08, 0.10] | 0.15 |
- | 10 | trufflehog 3.99.3 | 0.40 [0.35, 0.45] | 0.05 [0.05, 0.07] | 0.10 |
- | – | credsweeper 1.19.4 ¹ | 1.00 [1.00, 1.00] | 0.98 [0.98, 0.99] | 0.99 |
- | – | Penthera 1.9.0 | 0.56 [0.51, 0.60] | 0.10 [0.09, 0.12] | 0.18 |
- | | Free tools, mean / median | 0.68 / 0.76 | 0.36 / 0.35 | 0.43 / 0.45 |
- #### Malicious npm packages (DataDog)
- | Rank | Tool | Caught | False alarms on benign packages | Caught minus false alarms |
- |---:|---|---|---|---|
- | 1 | **Penthera** 2.0.0 | 0.71 [0.62, 0.78] (79/112) | 0.00 [0.00, 0.02] (0/150) | 0.71 |
- | 2 | malcontent 1.27.1 (bf280ad6), yara-x 1.21.0 | 0.68 [0.59, 0.76] (76/112) | 0.05 [0.03, 0.10] (8/150) | 0.63 |
- | 3 | GuardDog 3.2.0 ² | 0.62 [0.52, 0.70] (69/112) | 0.03 [0.01, 0.07] (4/150) | 0.59 |
- | 4 | ossgadget 0.2.20 (414b6a59) | 0.94 [0.88, 0.97] (105/112) | 0.89 [0.83, 0.93] (133/150) | 0.05 |
- | 5 | clamav 1.5.4 | 0.01 [0.00, 0.05] (1/112) | 0.00 [0.00, 0.02] (0/150) | 0.01 |
- | 6 | yara yara-python 4.5.4 (YARA 4.5), YARA-Forge core 20261004 | 0.00 [0.00, 0.03] (0/112) | 0.00 [0.00, 0.02] (0/150) | 0.00 |
- | – | osv-scanner 2.6.0 ³ | 0.90 [0.83, 0.94] (101/112) | 0.01 [0.00, 0.04] (1/150) | 0.90 |
- | – | Penthera 1.9.0 | 0.57 [0.48, 0.66] (64/112) | 0.00 [0.00, 0.02] (0/150) | 0.57 |
- #### Malicious PyPI packages (DataDog)
- | Rank | Tool | Caught | False alarms on benign packages | Caught minus false alarms |
- |---:|---|---|---|---|
- | 1 | **Penthera** 2.0.0 | 0.87 [0.81, 0.92] (131/150) | 0.05 [0.03, 0.10] (8/150) | 0.82 |
- | 2 | GuardDog 3.2.0 ² | 0.85 [0.79, 0.90] (128/150) | 0.11 [0.07, 0.17] (16/147) | 0.74 |
- | 3 | malcontent 1.27.1 (bf280ad6), yara-x 1.21.0 | 0.93 [0.87, 0.96] (139/150) | 0.22 [0.16, 0.29] (32/147) | 0.71 |
- | 4 | yara yara-python 4.5.4 (YARA 4.5), YARA-Forge core 20261004 | 0.25 [0.18, 0.32] (37/150) | 0.00 [0.00, 0.03] (0/147) | 0.25 |
- | 5 | clamav 1.5.4 | 0.00 [0.00, 0.02] (0/150) | 0.00 [0.00, 0.03] (0/147) | 0.00 |
- | 6 | ossgadget 0.2.20 (414b6a59) | 0.77 [0.70, 0.83] (116/150) | 0.97 [0.93, 0.99] (143/147) | -0.20 |
- | 7 | aura 2.1 (SourceCode-AI/aura b40e6dd2) | 0.59 [0.51, 0.67] (88/149) | 0.83 [0.75, 0.88] (100/121) | -0.24 |
- | 8 | bandit4mal lyvd/bandit4mal 03ba95f3 (bandit 1.6.3 fork) | 0.59 [0.51, 0.67] (89/150) | 0.84 [0.78, 0.89] (124/147) | -0.25 |
- | – | Penthera 1.9.0 | 0.83 [0.77, 0.88] (125/150) | 0.09 [0.06, 0.15] (14/150) | 0.74 |
- | – | osv-scanner 2.6.0 ³ | 0.74 [0.66, 0.80] (110/149) | 0.00 [0.00, 0.03] (0/147) | 0.74 |
- #### Injection bugs in JavaScript libraries (SecBench.js)
- | Rank | Tool | Command injection | Code injection | Both |
- |---:|---|---|---|---|
- | 1 | **Penthera** 2.0.0 | 35/47 | 10/17 | 0.70 [0.58, 0.80] |
- | 2 | odgen c5157ee | 36/46 | 6/16 | 0.68 [0.55, 0.78] |
- | 3 | codeql 2.27.2 (used under the GitHub CodeQL terms, for research on open-source code) | 37/47 | 4/17 | 0.64 [0.52, 0.75] |
- | 4 | eslint-plugin-security 4.2.0 | 32/47 | 9/17 | 0.64 [0.52, 0.75] |
- | 5 | semgrep 1.180.0 | 15/47 | 9/17 | 0.38 [0.27, 0.50] |
- | 6 | graudit 849d7c7 | 12/47 | 10/17 | 0.34 [0.24, 0.47] |
- | 7 | bearer 2.1.1 | 14/47 | 0/17 | 0.22 [0.14, 0.33] |
- | 8 | njsscan 1.0.1 | 2/45 | 1/15 | 0.05 [0.02, 0.14] |
- | – | semgrep, every file 1.180.0 (another setting of semgrep, not ranked) | 16/47 | 9/17 | 0.39 [0.28, 0.51] |
- | – | Penthera 1.9.0 | 0/47 | 10/17 | 0.16 [0.09, 0.26] |
- A rank is shown only where at least 5 other free tools were measured; otherwise "–". The rank column orders tools by score, and close scores can swap places. Penthera's 95% rank interval (bootstrap, exploratory): 2nd on CredData, 1st to 3rd on npm, 1st to 3rd on PyPI, 1st to 4th on SecBench.js.
- Paid tools (GitGuardian, Socket, Snyk, Phylum, Semgrep Pro) publish no results on these benchmarks, so they are not in the tables.
- Published elsewhere, on other datasets (context, not comparable):
- - [A comparative study of software secrets reporting by secret detection tools (2023): nine tools on SecretBench, ggshield included](https://arxiv.org/abs/2307.00714)
- - [How effective are npm malicious package detectors? (2026): eleven detectors on their own benchmark](https://arxiv.org/abs/2603.27549)
- - [Cutting the Gordian knot: detecting malicious PyPI packages (USENIX Security 2026): GuardDog, Bandit4Mal and OSSGadget on PyPI](https://arxiv.org/abs/2601.16463)
- - [Study of JavaScript static analysis tools for vulnerability detection in Node.js packages (2023): nine tools incl. CodeQL, NodeJsScan, ODGen](https://www.researchgate.net/publication/367088656)
- ¹ Not ranked: trained on this dataset by its authors. ² Found most samples in this dataset, so its rate is inflated by construction. ³ Not ranked: a lookup of already-reported packages, not detection.
- </details>
- <!-- public-benchmarks:end -->
- What these numbers do not show: a benchmark is not your code. CredData replaces
- real secrets with look-alikes, which favors pattern rules like ours over tools
- that verify keys; the free tools ran on their defaults while Penthera was tuned
- on the other half; GuardDog helped build the malware dataset. Accuracy on real
- projects is experiment E1. To reproduce every table, get the sealed split with
- `npm run bench:public -- --fetch-split`, then run
- `npm run bench:public -- --split test --final` and
- `npm run bench:public -- --competitors --split test --final`.
- ## What it does
- ```mermaid
- flowchart LR
- U["You or<br/>your agent"] --> P(("Penthera<br/>on your machine"))
- P --> APP["Your app<br/>live and code"]
- P --> DEP["Dependencies<br/>npm and Python"]
- P --> AI["AI tooling<br/>skills, MCP, hooks"]
- APP & DEP & AI --> F[["Findings<br/>rules decide"]]
- F --> FIX["Fix<br/>asks first"]
- F --> R["Reports<br/>SARIF, ui, MCP"]
- F --> E["Evidence<br/>signed bundle"]
- FIX -. "re-scan" .-> F
- classDef core fill:#0000ed,stroke:#0000ed,color:#ffffff
- classDef box fill:#ffffff,stroke:#000000,color:#000000
- class P core
- class U,APP,DEP,AI,F,FIX,R,E box
- ```
- <details>
- <summary><b>Advanced: black box</b>, the running app over HTTP</summary>
- The default scan sends no attack payloads. `--deep`, `--fuzz` and `--adaptive`
- start only after you confirm the hostname, or pass `--confirm-target <host>` in
- CI. Limits are in [METHODOLOGY.md](docs/METHODOLOGY.md#under-the-hood).
- ```mermaid
- %%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 40}}}%%
- flowchart LR
- APP["Running app<br/>penthera URL"] --> DEF
- APP --> ACT
- subgraph DEF["Default: no attack payloads"]
- direction TB
- B1["TLS, headers<br/>cookies, CORS"]
- B2["Routes<br/>OpenAPI, files"]
- B3["Auth, JWT, OAuth<br/>IDOR: review"]
- B4["JS libraries<br/>bundle keys"]
- end
- subgraph ACT["Opt-in, gated"]
- direction TB
- A1["--deep<br/>SQLi, XSS, SSRF"]
- A2["--fuzz<br/>API bodies"]
- A3["--adaptive<br/>may send DELETE"]
- end
- DEF & ACT --> F[["Findings"]]
- classDef core fill:#0000ed,stroke:#0000ed,color:#ffffff
- classDef box fill:#ffffff,stroke:#000000,color:#000000
- class F core
- class APP,B1,B2,B3,B4,A1,A2,A3 box
- style DEF fill:none,stroke:#0000ed,stroke-width:2px
- style ACT fill:none,stroke:#0000ed,stroke-width:2px,stroke-dasharray:6 4
- ```
- </details>
- <details>
- <summary><b>Advanced: white box</b>, your code and config</summary>
- Code sinks use taint-lite within one file, with no call graph. Dependency
- capabilities are what code can do, not proof it is malicious.
- ```mermaid
- %%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 40}}}%%
- flowchart LR
- REPO["Code and config"] --> RS
- REPO --> OWN
- subgraph RS["penthera --repo ."]
- direction TB
- W1["Secrets<br/>key formats"]
- W2["Routes<br/>auth per route"]
- W3["Code sinks<br/>eval, vm, exec"]
- W4["CI, runtimes<br/>Actions, pins"]
- end
- subgraph OWN["Own commands"]
- direction TB
- W5["deps<br/>capabilities, IOCs"]
- W6["agents<br/>MCP, skills, hooks"]
- W7["install, hook<br/>before it lands"]
- end
- RS & OWN --> F[["Findings"]]
- classDef core fill:#0000ed,stroke:#0000ed,color:#ffffff
- classDef box fill:#ffffff,stroke:#000000,color:#000000
- class F core
- class REPO,W1,W2,W3,W4,W5,W6,W7 box
- style RS fill:none,stroke:#0000ed,stroke-width:2px
- style OWN fill:none,stroke:#0000ed,stroke-width:2px
- ```
- </details>
- <details>
- <summary><b>Advanced: engine and outputs</b>, how a match becomes a result</summary>
- Rules decide every finding. Exit code `0` means nothing critical or high, `1`
- critical or high, `2` an incomplete or refused scan. Under `--ai` a model only
- proposes.
- ```mermaid
- %%{init: {"flowchart": {"nodeSpacing": 30, "rankSpacing": 40}}}%%
- flowchart LR
- subgraph IN["Inputs"]
- direction LR
- R["Signed rule packs"]
- GI["--giants, opt-in"]
- AI["--ai, opt-in"]
- end
- IN --> FD[["Finding<br/>severity, confidence"]]
- FD --> EX{"Exit<br/>0, 1 or 2"}
- EX --> OUT
- subgraph OUT["Outputs"]
- direction LR
- O1["Reports, SARIF"]
- O2["penthera ui, MCP"]
- O3["Action, Vercel check"]
- O4["Evidence bundle"]
- end
- classDef core fill:#0000ed,stroke:#0000ed,color:#ffffff
- classDef box fill:#ffffff,stroke:#000000,color:#000000
- class FD core
- class R,GI,AI,EX,O1,O2,O3,O4 box
- style IN fill:none,stroke:#0000ed,stroke-width:2px
- style OUT fill:none,stroke:#0000ed,stroke-width:2px
- ```
- </details>
- ### Before you deploy
- | Job | What Penthera does | Command |
- |---|---|---|
- | Test the app | Scans your localhost or preview URL (TLS, headers, CORS, cookies, auth, JWT, IDOR, OAuth, OpenAPI; injection and fuzzing only when you ask, after a confirmation) and your repository (routes, trust boundaries, code-execution sinks, secrets including AI keys) | [`penthera <url> --repo .`](docs/commands/scan.md) |
- | Check your runtimes | Whether the PHP, nginx, Apache, Node.js, Python, OpenSSL or MongoDB version you run is affected by a known advisory, from the signed advisories pack | [Runtime advisories](docs/runtime-advisories.md) |
- | Fix and verify | Applies deterministic fix recipes on a local branch, re-scans to prove they worked, and remembers the result. Never pushes | [`penthera agent --loop`](docs/commands/fix.md) |
- | Block it at commit time | Stops secrets, risky agent config and dependency changes before they are committed | [`penthera hook install`](docs/commands/hook.md) |
- ### What you build with
- | Job | What Penthera does | Command |
- |---|---|---|
- | Vet dependencies | Shows what every npm and Python package can do (install scripts, shell, network, env, obfuscation), what an upgrade added, lookalike names and licenses | [`penthera deps --base main`](docs/commands/deps.md) |
- | Guard installs | Shows what a new package brings before any install script runs, and stops it under your policy | [`penthera install <pkg>`](docs/commands/install.md) |
- | Review pull requests | Comments the dependency capability diff on every PR that touches the lockfile | [GitHub Action](docs/commands/action.md) |
- | Audit your AI tooling | Checks skills, MCP servers, hooks, plugins and rule files for hidden instructions, unpinned servers, over-broad permissions and silent updates | [`penthera agents --home`](docs/commands/agents.md) |
- | Work from your agent | Six read-only MCP tools: scan the repo, check dependencies, audit AI tooling, runtime advisories, explain a finding, a passive URL scan | [`penthera mcp`](docs/commands/mcp.md) |
- ### After you ship
- | Job | What Penthera does | Command |
- |---|---|---|
- | Report and prove | Writes reports for developers, owners and auditors; a hashed, signable evidence bundle; NIS2 and CRA readiness reports and drafts | [`penthera <url> --audience auditor --evidence out/`](docs/evidence.md) |
- | See it all | A local dashboard on 127.0.0.1. Simple mode for non-programmers: a one-sentence verdict, problems in plain words, and a check-and-fix mode with Apply, Skip and Stop per change. Advanced adds dependencies, AI tooling, history and evidence | [`penthera ui`](docs/dashboard.md) |
- | Keep watching | Scheduled checks of production for drift: certificates, headers, new endpoints, keys in bundles, new CVEs in your SBOM | on the [roadmap](ROADMAP.md) |
- Every finding carries a severity, a confidence label (`confirmed`, `likely`,
- `potential`, `needs-human-review`), its evidence and the research behind it.
- Reports come out as Markdown, JSON and SARIF for GitHub code scanning. Exit
- code `1` means critical or high findings. Every command and flag is in
- [docs/commands/](docs/commands/README.md). Compliance output is readiness
- tooling, not a certification and not legal advice.
- **AI mode (opt-in).** `penthera --repo . --ai` lets a local model
- propose problems no rule describes. Penthera checks each proposal with its
- rules, and anything unconfirmed is labelled "model-proposed, unverified" and
- never fails CI. Whether it finds more than the rules alone is not measured yet
- ([AI mode](docs/ai-mode.md)).
- ### How far each part has been taken
- | Part | Status |
- |---|---|
- | Scans: URL, repo, dependencies, AI tooling, installs, pre-commit hook | **Stable**: in every release since 1.x, covered by the tests and the release gate |
- | Fix loop and local dashboard (`penthera ui`) | **Stable**; the dashboard was redesigned in 1.9.0 |
- | MCP server, Agent Skill, Claude Code plugin | **Stable**, since 1.9.0 |
- | Gemini CLI extension | **Stable**: it installs the newest release (2.4.0) |
- | Evidence bundles, NIS2 and CRA drafts | **Stable** as readiness tooling, not a certification |
- | AI mode (`--ai`) | **Experimental**: whether it finds more than the rules alone is not measured yet ([AI mode](docs/ai-mode.md)) |
- | Live check on [penthera.org](https://penthera.org) | **Beta**: passive only, [terms](docs/live-check-terms.md) |
- ## How it is measured
- Each feature goes through the same path: **method → evidence → tool.**
- - **Method.** The [methodology](docs/METHODOLOGY.md) states the threat model,
- what black-box (HTTP) and white-box (source) analysis can and cannot see, the
- confidence model, and the false-positive controls on each probe. Each design
- choice cites the research or standard it rests on: 158 verified references, in
- APA style, in [docs/references.md](docs/references.md).
- - **Evidence.** A benchmark runs on every release, and the release gate fails
- if precision or recall drops by more than 0.02. Experiments that decide what
- ships next are pre-registered in [docs/research/prereg/](docs/research/prereg/)
- before any data is collected, and the results are published either way.
- - **Tool.** What holds up ships in a reviewed, signed release. What does not
- hold up is dropped, and the rules stay.
- | Measure | Result | Sample |
- |---|---|---|
- | Precision, bundled benchmark | 0.92, 95% CI 0.74 to 0.98 | 22 true and 2 false positives |
- | Recall, bundled benchmark | 0.96, 95% CI 0.79 to 0.99 | 1 miss, [explained](bench/RESULTS.md) |
- | Requests past loopback during the benchmark | 0 | Enforced by a network guard |
- | Public benchmarks (E11b), sealed test half | 1st on npm and PyPI malware (tied 1st to 3rd), 1st on JS library injection bugs (tied 1st to 4th), 2nd on leaked secrets | 4 public datasets, up to 10 free tools each ([results](bench/public/RESULTS.md#e11b-final-the-sealed-test-half-scored-once-2026-10-25), [summary](#public-benchmarks)) |
- | E10 pilot, known-vulnerable apps | Precision 0.88, 95% CI 0.78 to 0.93. Recall 0.07 before fixes, 0.71 (CI 0.45 to 0.88) after | 14 root causes in Juice Shop and DVNA ([results](bench/known-vuln/RESULTS-after-fixes.md)). No run logs in, so it doesn't cover login or access rules |
- | Detection rules, all readable | 132 in this release: 16 URL scan templates, 24 secret and AI-key patterns, 15 dependency capabilities, 47 AI-tooling rules, 5 fix recipes, and 25 rules in the signed rule packs | [Rule packs](docs/rule-packs.md) |
- The bundled benchmark is 24 findings in small labs we wrote ourselves. It
- catches regressions; it does not measure real-world accuracy. The E10 pilot
- used one labeller and is not independent: the fixes were written after seeing
- the misses on the same two apps. On WebGoat, held out, recall stayed at 2 of 9.
- Accuracy on real projects is experiment E1, pre-registered and not yet run.
- Intervals are Wilson. What Penthera cannot see is listed under
- [Known limitations](docs/METHODOLOGY.md#known-limitations). Refresh the counts
- with `npm run docs:numbers` and `npm run bench`.
- **Cite this repository** with the button on GitHub or
- [CITATION.cff](CITATION.cff). APA and BibTeX forms are in the
- [methodology](docs/METHODOLOGY.md#how-to-cite).
- ## Why you can trust it
- - **Local and offline.** No account, no telemetry, no cloud. `--offline` limits
- network traffic to the target you named, and every report lists the hosts it
- contacted ([network use](docs/privacy.md)).
- - **Tested unplugged.** Repo scans, `agents`, `deps`, `brief`, evidence
- bundles and URL scans with `--offline` made no outside connections. They
- gave the same result twice, and the same with the network blocked, on macOS
- and Ubuntu ([E17 results](bench/offline/RESULTS.md)). The test can't watch
- `git`, which Penthera uses only for local commands.
- - **Deterministic.** Rules you can read, with `file:line` evidence. A model may
- propose a fix or a lead; a deterministic re-scan and your tests decide.
- - **Non-destructive by default.** Default scans send no attack payloads.
- Payload testing (`--deep`, `--fuzz`, `--adaptive`, `--all`) is opt-in, asks for a
- confirmation bound to the target, and is rate-limited. It can change data, so
- read [what active modes can change](docs/commands/scan.md#what-active-modes-can-change)
- first.
- - **Asks before it changes anything.** The fix loop works on a local branch,
- asks before every change by default, and never pushes or opens pull requests.
- - **No update channel to hijack.** Detection rules ship inside reviewed
- releases as signed, versioned [rule packs](docs/rule-packs.md). You can add
- your own by path (`--rules <dir>`). Nothing is downloaded and run at run time.
- - **Releases you can verify.** npm provenance through trusted publishing, a
- maintainer approval with 2FA for every version, SHA-pinned CI, CodeQL and
- OpenSSF Scorecard. Every release passes an
- [internal audit](docs/internal-audit.md) of eight gates
- ([latest report](audit/REPORT.md)).
- The [claim map](docs/claims.md) ties these claims to the tests that hold them.
- **Verify it yourself.** In an empty folder:
- ```bash
- npm install penthera@2.4.0 && npm audit signatures
- npx penthera@2.4.0 rules list
- ```
- The first checks npm's registry signature and the build provenance of the
- package you got. The second lists every bundled rule pack and the key that
- signed it; a pack whose signature does not verify is refused unless you pass
- `--allow-unsigned-rules`.
- Penthera finds known classes of problems, not novel ones, and it does not
- replace a human pentester or incident response. The decisions behind it, its
- limits, and how to challenge them are in the [manifesto and FAQ](docs/MANIFESTO.md).
- ## How it fits with other tools
- Penthera works next to the tools you already use; its SARIF output goes
- straight into GitHub code scanning, for example. Each comparison page says where
- the other tool goes further and how to run the two together.
- | | Focus | Where it runs | How it decides |
- |---|---|---|---|
- | **Penthera** | Your app, dependencies and AI tooling; fixes; EU evidence | Your machine or CI, offline | Readable deterministic rules, with evidence |
- | [Socket](docs/compare/socket.md) | Supply-chain risk across package registries | Cloud service, GitHub App, CLI | Registry-scale package analysis, including AI review |
- | [Snyk](docs/compare/snyk.md) | Known vulnerabilities in dependencies, code, containers, IaC | Cloud service, IDE, CLI | Curated vulnerability database plus static analysis |
- | [GitHub Dependabot and code scanning](docs/compare/github.md) | Known-CVE alerts, update PRs, CodeQL | GitHub | Advisory database, CodeQL queries |
- | [OWASP ZAP](docs/compare/zap.md) | Dynamic testing of web apps | Your machine or CI | Active and passive scan rules |
- | [Semgrep](docs/compare/semgrep.md) | Static analysis of your own code | Your machine, CI, cloud platform | Pattern rules, plus a paid cross-file engine |
- All comparisons, including Cloudflare's security-audit skill: [docs/compare/](docs/compare/README.md).
- ### Run the free tools you already have: `--giants`
- `--giants` runs the free scanners installed on your
- machine after Penthera's own scan: gitleaks, TruffleHog, Semgrep with your
- local rules, osv-scanner, Trivy and Checkov for a repository, nuclei and the
- ZAP baseline for a URL. Their findings go into the same report and SARIF, each
- labelled with the tool, its version and its rule. Nothing is downloaded or
- uploaded, and an area that neither Penthera's rules nor another tool reached
- is reported as "not covered".
- ```bash
- penthera giants # which tools you have
- penthera --repo . --giants # Penthera, then each installed tool
- ```
- This runs the tools you have and merges their findings. It is not a claim that
- the combination finds more: experiment E14 measured that
- ([results](bench/public/RESULTS.md)). Details, safety rules and the policy key:
- [docs/commands/giants.md](docs/commands/giants.md).
- ## Start here
- | You want to know | Read |
- |---|---|
- | Is my app secure? | [I'm new](docs/index.md#1-im-new), then [scan](docs/commands/scan.md) |
- | What should my small team do about security now? | [A Practical Guide to Security for 2027](docs/security-guide-2027.md): 10 steps, each with the command and its limit |
- | Is this package or MCP server safe? | [deps](docs/commands/deps.md), [install](docs/commands/install.md), [agents](docs/commands/agents.md) |
- | How do I use it from my coding agent? | [MCP server](docs/commands/mcp.md), [skill](docs/commands/skill.md), [agents safely](docs/agents-safely.md) |
- | How do I put it in CI? | [The GitHub Action](docs/commands/action.md), [scan in CI](docs/commands/scan.md#ci-integration) |
- | What do I give my auditor? | [Evidence bundles](docs/evidence.md), [NIS2](docs/compliance-nis2.md), [CRA](docs/compliance-cra.md) |
- | How do I run the fix loop? | [Fix and verify](docs/commands/fix.md), [the local loop](docs/local-loop.md) |
- The full map, by situation and from first scan to research, is in
- [docs/index.md](docs/index.md).
- ## Ethics and authorization
- Penthera sends HTTP requests to the targets you give it, and some modes
- (`--deep`, `--fuzz`, `--adaptive`, `--all`) send attack payloads, after a
- confirmation.
- Scan only what you own, what you have written authorization to test, or local
- environments you control. Do not use it to probe third-party systems,
- exfiltrate data, disrupt services, or harm anyone. Unauthorized testing may
- break computer-misuse laws such as the US CFAA, the UK Computer Misuse Act, and
- their EU equivalents. When in doubt, do not scan.
- The software is provided "as is", without warranty. The author accepts no
- liability for misuse, and you are responsible for making sure every scan you
- run is lawful. The live check on penthera.org has its own
- [terms](docs/live-check-terms.md).
- ## Building in public
- What changed, what we are measuring, and what comes next, in one place.
- | | Where it stands | Read more |
- |---|---|---|
- | **Latest release** | 2.4.0 (2026-11-08) on npm `latest`: `penthera --card` makes a security receipt to post, and 13 ways to hide a finding or stall a scan, found by Vercel Security Review, are closed. Crafted files that took seconds to hours now take under a second, and a file too big to finish makes the scan incomplete (exit 2). 2.3.2 brought: hostile files can no longer hang a scan, secrets stay masked in what reports quote | [CHANGELOG](CHANGELOG.md), [roadmap](ROADMAP.md) |
- | **Research** | Public benchmarks (E11b) published: on a sealed half of four public datasets, 1st of the free tools on npm and PyPI malware (tied 1st to 3rd), 1st on JS library injection bugs (tied 1st to 4th), 2nd on leaked secrets ([summary](#public-benchmarks)). E10 pilot published: precision 0.88, recall 0.07 to 0.71 on two known-vulnerable apps after fixes (one labeller, not independent; a re-run is pending because default scans no longer send login checks). E1 on real code is pre-registered; E8 (agent guardrails) is a draft | [E11b results](bench/public/RESULTS.md), [E10 results](bench/known-vuln/RESULTS.md), [E1 pre-registration](docs/research/prereg/E1-real-world-accuracy.md), [agents safely](docs/agents-safely.md) |
- | **Next** | Real-world precision and recall with intervals (E1), published whatever the result | [Roadmap: next](ROADMAP.md#next), [research plan](docs/vision-2.0.md) |
- [](https://star-history.com/#danoszz/penthera&Date)
- <a href="assets/walkthrough.mp4"><img src="assets/walkthrough-poster.jpg" alt="Walkthrough video: setup, a scan, the dashboard in Simple and Advanced mode, one fix approved, the report" width="760"></a>
- [Watch the 80-second walkthrough](assets/walkthrough.mp4): setup, a scan, the dashboard, a fix approved, the report. Demo data, no sound, captions in the video.
- Releases are deliberately slow: a feature freeze, an independent review, the
- release checks, and a release candidate before any new minor version. Watch the
- repository for releases or join the
- [Discussions](https://github.com/danoszz/penthera/discussions).
- ## How Penthera is built
- Penthera is developed with substantial help from generative AI: 91% of the
- commits up to v1.7.0 were written with Claude (Anthropic) through Claude Code,
- directed and reviewed by the maintainer, and marked with an `AI-Assisted:` trailer.
- Generated code passes the same gates as any change: tests, CodeQL, an
- independent review, the internal audit, and a 2FA-approved release. Inside the
- product, models only ever propose; deterministic checks decide. The full
- disclosure, and the rules for AI-assisted contributions, are in
- [docs/genai.md](docs/genai.md).
- ## Contributing and support
- Clone the repository and run `npm install`, then `npm test` for the suite and
- `npm run bench` for precision and recall.
- [CONTRIBUTING.md](CONTRIBUTING.md) covers the project layout, the release
- cadence and the release checklist; [AGENTS.md](AGENTS.md) is the short version
- for coding agents. The [roadmap](ROADMAP.md) shows what ships next, and the
- [research plan](docs/vision-2.0.md) what we measure. Issues and pull requests
- are welcome.
- Penthera is free and stays free. If it caught something before you shipped it,
- you can [sponsor its maintenance](https://github.com/sponsors/danoszz).
- Sponsorship pays for maintainer time and buys no influence over findings
- ([tiers](docs/funding/sponsor-tiers.md)).
- ## License
- MIT. See [LICENSE](LICENSE). Report vulnerabilities in Penthera itself through
- [SECURITY.md](SECURITY.md).
bench/front-door/classify.mjs · Bundled script · 532 lines
- /**
- * E20-R response classifier: a fixed, deterministic label for one homepage
- * response. Protocol: docs/research/prereg/E20r-front-door-replication.md § 4.2
- * and Amendment 2. The signature lists are frozen before a run; a change after
- * that is a deviation. No model decides a label (AGENTS.md rule 4).
- *
- * challenge an edge network tests the client
- * block an edge network or firewall refuses the client
- * unrecognized-error-response an error no rule recognized (pass 1: origin-error)
- * candidate-page no marker found; not proof of a page (pass 1: page)
- * body-unavailable a compressed answer, whose body was never read, so nothing can be said of it
- *
- * Header markers come first: Cloudflare documents that the status code alone
- * does not identify a challenge. `classify` is the V2 rule set, written after
- * the V1 labels were read. `classifyRegistered` is the rule set of pass 1,
- * unchanged, so C15 and the comparison with pass 1 use one definition.
- */
- import { createHash } from "node:crypto";
- import { isIP, isIPv6 } from "node:net";
- export const CHALLENGE_HEADERS = Object.freeze([
- ["cf-mitigated", /challenge/i, "cloudflare"],
- ["x-amzn-waf-action", /challenge|captcha/i, "aws-waf"],
- ["x-vercel-mitigated", /challenge/i, "vercel"],
- ]);
- // Pass 1 rules, frozen at registration. Do not edit.
- export const CHALLENGE_BODIES = Object.freeze([
- [/\/cdn-cgi\/challenge-platform\/|<title>\s*Just a moment\.\.\.\s*<\/title>|cf-browser-verification|cf_chl_opt/i, "cloudflare"],
- [/captcha-delivery\.com/i, "datadome"],
- [/px-captcha|captcha\.px-cdn\.net/i, "human"],
- [/_Incapsula_Resource/i, "imperva"],
- [/_sec\/verify\?provider=|sec-if-cpt-container/i, "akamai"],
- [/Vercel Security Checkpoint/i, "vercel"],
- [/AwsWafIntegration|awswaf\.com/i, "aws-waf"],
- ]);
- export const BLOCK_BODIES = Object.freeze([
- [/Attention Required! \| Cloudflare|Sorry, you have been blocked|cf-error-details/i, "cloudflare"],
- [/errors\.edgesuite\.net|<H1>Access Denied<\/H1>/i, "akamai"],
- [/Generated by cloudfront \(CloudFront\)|The request could not be satisfied/i, "cloudfront"],
- [/Incapsula incident ID/i, "imperva"],
- ]);
- // V2. Hard signatures appear only on an interstitial page.
- export const HARD_CHALLENGE_BODIES = Object.freeze([
- [/<title>\s*Just a moment\.\.\.\s*<\/title>|cf-browser-verification|cf_chl_opt|\/cdn-cgi\/challenge-platform\/h\/[a-z]\/orchestrate\/chl_page/i, "cloudflare"],
- [/captcha-delivery\.com\/captcha/i, "datadome"],
- [/px-captcha/i, "human"],
- [/id=["']main-iframe["'][^>]*_Incapsula_Resource/i, "imperva"],
- [/_sec\/verify\?provider=|sec-if-cpt-container/i, "akamai"],
- [/Vercel Security Checkpoint/i, "vercel"],
- [/<title>\s*DDoS-Guard\s*<\/title>/i, "ddos-guard"],
- ]);
- // Soft signatures are scripts that also sit on normal pages as passive bot
- // detection. They count on an error status, or on a 2xx too small to be a page.
- export const SOFT_CHALLENGE_BODIES = Object.freeze([
- [/\/cdn-cgi\/challenge-platform\//i, "cloudflare"],
- [/captcha-delivery\.com/i, "datadome"],
- [/captcha\.px-cdn\.net/i, "human"],
- [/_Incapsula_Resource/i, "imperva"],
- [/AwsWafIntegration|awswaf\.com/i, "aws-waf"],
- ]);
- export const SOFT_MAX_CHARS = 8192;
- export const BLOCK_BODIES_V2 = Object.freeze([
- [/Attention Required! \| Cloudflare|Sorry, you have been blocked|cf-error-details/i, "cloudflare"],
- [/errors\.edgesuite\.net|<H1>Access Denied<\/H1>/i, "akamai"],
- [/Request blocked|configured to block access from your country/i, "cloudfront"],
- [/Incapsula incident ID/i, "imperva"],
- ]);
- // CloudFront's own error page, after the two lines above that mean a refusal.
- export const CLOUDFRONT_ERROR = /The request could not be satisfied/i;
- // Cloudflare's short text error. These four are DNS, tunnel and Worker errors, not refusals.
- const CF_ERROR = /^\s*error code:\s*(1\d{3})\b/i;
- const CF_NOT_REFUSAL = new Set(["1001", "1016", "1033", "1101"]);
- const EDGE_NAMES = Object.freeze([["cf-ray", "cloudflare"], ["x-amz-cf-id", "cloudfront"], ["x-akamai-transformed", "akamai"], ["akamai-grn", "akamai"]]);
- const REFUSAL_STATUS = new Set([401, 403, 406, 429]);
- // Header values kept verbatim, then scrubbed. Every other header keeps its name only.
- export const KEPT_VALUES = Object.freeze(["cf-mitigated", "x-amzn-waf-action", "x-vercel-mitigated", "server", "via", "x-cache", "cf-cache-status", "content-type"]);
- export const EXCERPT_BYTES = 4096;
- export const PAGE_TEXT_CHARS = 4096;
- // Site-controlled text is cut to this before any regex runs on it.
- export const INPUT_CAP_CHARS = 65_536;
- const lower = (headers = {}) => Object.fromEntries(Object.entries(headers).map(([k, v]) => [k.toLowerCase(), v]));
- const edgeOf = (h) => {
- for (const [name, vendor] of EDGE_NAMES) if (h[name] !== undefined) return vendor;
- const server = String(h.server ?? "");
- return /cloudflare/i.test(server) ? "cloudflare" : /cloudfront/i.test(server) ? "cloudfront" : /akamai/i.test(server) ? "akamai" : null;
- };
- /** The label of one response, with the vendor and the rule that decided it. */
- export function classify({ status, headers = {}, body = "" }) {
- const h = lower(headers);
- for (const [name, re, vendor] of CHALLENGE_HEADERS) {
- if (re.test(String(h[name] ?? ""))) return { label: "challenge", vendor, rule: `header:${name}` };
- }
- const whole = String(body);
- const text = whole.slice(0, INPUT_CAP_CHARS);
- for (const [re, vendor] of HARD_CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "body" };
- // A long 2xx page may mention any of these words. It counts on an error, or a body too small to be a page.
- if (status >= 400 || whole.length <= SOFT_MAX_CHARS) {
- for (const [re, vendor] of BLOCK_BODIES_V2) if (re.test(text)) return { label: "block", vendor, rule: "body" };
- for (const [re, vendor] of SOFT_CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "soft-body" };
- if (CLOUDFRONT_ERROR.test(text)) return { label: "unrecognized-error-response", vendor: "cloudfront", rule: "cloudfront:error" };
- }
- const edge = edgeOf(h);
- const code = CF_ERROR.exec(text)?.[1];
- if (status >= 400 && code && edge === "cloudflare" && !CF_NOT_REFUSAL.has(code)) return { label: "block", vendor: "cloudflare", rule: `cloudflare:${code}` };
- // No body to read, so a challenge and a block look the same: the edge refused.
- if (REFUSAL_STATUS.has(status) && !text.trim() && edge) return { label: "block", vendor: edge, rule: "empty-at-edge" };
- // A homepage that answers 403 refuses this client, whoever serves it. A view
- // of signature-backed refusals alone drops this rule.
- if (status === 403) return { label: "block", vendor: edge, rule: "unrecognized-403" };
- if (status >= 400) return { label: "unrecognized-error-response", vendor: null, rule: "status" };
- if (h["content-encoding"] && !/^identity$/i.test(h["content-encoding"]) && !text.trim()) return { label: "body-unavailable", vendor: null, rule: "content-encoding" };
- return { label: "candidate-page", vendor: null, rule: "status" };
- }
- /** Pass 1's rule set, unchanged, on the whole body, so its labels stay comparable. Each rule is a plain alternation. */
- export function classifyRegistered({ status, headers = {}, body = "" }) {
- const h = lower(headers);
- for (const [name, re, vendor] of CHALLENGE_HEADERS) {
- if (re.test(String(h[name] ?? ""))) return { label: "challenge", vendor, rule: `header:${name}` };
- }
- const text = String(body);
- for (const [re, vendor] of CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "body" };
- for (const [re, vendor] of BLOCK_BODIES) if (re.test(text)) return { label: "block", vendor, rule: "body" };
- if (status >= 400) return { label: "origin-error", vendor: null, rule: "status" };
- return { label: "page", vendor: null, rule: "status" };
- }
- const matches = (rules, text) => Object.fromEntries(rules.map(([re, vendor]) => [vendor, re.test(text)]));
- /** Every signature, for every rule set, as a boolean, so a label can be replayed under other rules. */
- export function signatureMatches(body) {
- const whole = String(body ?? "");
- const text = whole.slice(0, INPUT_CAP_CHARS);
- return {
- registeredChallenge: matches(CHALLENGE_BODIES, whole), registeredBlock: matches(BLOCK_BODIES, whole),
- hardChallenge: matches(HARD_CHALLENGE_BODIES, text), softChallenge: matches(SOFT_CHALLENGE_BODIES, text), block: matches(BLOCK_BODIES_V2, text),
- cloudfrontError: CLOUDFRONT_ERROR.test(text), cloudflareCode: CF_ERROR.exec(text)?.[1] ?? null,
- };
- }
- /**
- * What a value test of the CSP needs, from the full header, not the grader's
- * 160-character evidence. Directive names only. `controlsScripts`: a policy has
- * `script-src`, `script-src-elem` or `default-src`. `openScripts`: that source
- * list still allows inline code without a nonce or hash, or any host of a scheme.
- */
- export function cspEvidence(headers = {}) {
- const raw = [].concat(lower(headers)["content-security-policy"] ?? []).join(",").slice(0, INPUT_CAP_CHARS);
- const policies = raw.split(",").map((p) => new Map(p.split(";").map((d) => d.trim().split(/\s+/)).filter(([n]) => /^[a-z-]{1,64}$/i.test(n ?? "")).map(([n, ...v]) => [n.toLowerCase(), v.map((x) => x.toLowerCase())])));
- const script = policies.map((p) => p.get("script-src") ?? p.get("script-src-elem") ?? p.get("default-src")).filter(Boolean);
- const open = (v) => v.some((s) => ["*", "http:", "https:", "data:"].includes(s)) || (v.includes("'unsafe-inline'") && !v.some((s) => /^'(?:nonce-|sha(?:256|384|512)-)/.test(s)));
- return {
- present: raw.trim().length > 0, chars: raw.length,
- directives: [...new Set(policies.flatMap((p) => [...p.keys()]))].sort().slice(0, 40),
- controlsScripts: script.length > 0,
- openScripts: script.length > 0 && script.every(open),
- };
- }
- /** Cookie names only, never values. */
- export function cookieNames(headers = {}) {
- const raw = lower(headers)["set-cookie"];
- const list = Array.isArray(raw) ? raw : raw ? [raw] : [];
- // No = before the first ; means there is no name, and what stands there may be a value.
- // A name that looks like a token, an address or anything but a name isn't kept.
- const name = (c) => { const pair = String(c).split(";")[0]; return pair.includes("=") ? pair.split("=")[0].trim() : ""; };
- return list.map(name).filter((n) => /^[\w.-]{1,64}$/.test(n) && scrub(n) === n).sort();
- }
- // § 6, secondary: what kind of site this is, from the homepage already fetched.
- // Fixed rules on the HTML; only the booleans are kept, never a URL or a name.
- export const TRAIT_RULES = Object.freeze({
- login: /<a\b[^>]{0,2000}>\s{0,200}(?:<[^>]{1,500}>\s{0,200}){0,8}(log\s?in|sign\s?in|my account|inloggen|anmelden|connexion)\b/i,
- payments: /js\.stripe\.com|checkout\.stripe\.com|paypal\.com\/sdk|paypalobjects\.com|js\.braintreegateway\.com|checkoutshopper-[a-z]+\.adyen\.com|klarna(?:cdn|\.com)|mollie\.com/i,
- shop: /cdn\.shopify\.com|woocommerce|magento|\/cart\b|\/checkout\b|add[-_ ]to[-_ ]cart/i,
- // A public security or trust page: a stand-in for a team that owns security, not proof of a CISO.
- securityPage: /<a\b[^>]{0,2000}href="[^"]{0,500}(?:\/security\b|\/trust\b|\/trust-center|\/\/trust\.|\/\/security\.)[^"]{0,500}"/i,
- });
- /** Booleans for one page. An empty page (no 2xx HTML) gives null: unknown, not "no". */
- export function traitsOf(html) {
- if (!html) return null;
- const text = String(html).slice(0, INPUT_CAP_CHARS);
- return Object.fromEntries(Object.entries(TRAIT_RULES).map(([k, re]) => [k, re.test(text)]));
- }
- // Every repeat below is bounded, so no input of INPUT_CAP_CHARS can make these slow.
- const ZERO_WIDTH = /[-]/g;
- const NUMERIC_ENTITY = /&#(?:x0*([0-9a-f]{1,8})|0*(\d{1,8}));?/gi;
- const NAMED_ENTITY = /&(commat|period|lowbar|colon);?/gi;
- const ESCAPED = /\\(?:x|u00)([0-9a-f]{2})/gi;
- const PERCENT = /%(?:25)?([2-7][0-9a-f])/gi;
- // A bracketed at is an address, whatever stands around it: both neighbouring words go.
- const BRACKET_AT = /[^\s<>"']{0,64}\s{0,3}[[({]\s{0,2}at\s{0,2}[\])}]\s{0,3}[^\s<>"']{0,64}/gi;
- const AT_WORD = /\s{1,3}@\s{0,3}(?=[\p{L}\p{N}])|(?<=[\p{L}\p{N}\p{M}])\s{0,3}@\s{1,3}|[@﹫]/gu;
- const TAGGED_AT = /(?:<\/?[a-z]{1,8}>\s{0,2}){1,4}@(?:\s{0,2}<\/?[a-z]{1,8}>){1,4}/gi;
- const DASH_AT = /([\p{L}\p{N}+]{1,64})[-_.]at[-_.]([\p{L}\p{N}-]{1,63}\.\p{L}{2,24})\b/giu;
- const DOT_WORD = /\s{0,3}[[({]\s{0,2}(?:dot|\.)\s{0,2}[\])}]\s{0,3}/gi;
- // A bare `at` counts only before something with a dot, spaced or not.
- const SPELLED_EMAIL = /([\p{L}\p{N}._%+-]{1,64})\s{1,3}at\s{1,3}([\p{L}\p{N}_-]{1,63})(?:\s{1,3}dot\s{1,3}|\s{0,3}\.\s{0,3})(\p{L}{2,24})\b/giu;
- const CF_EMAIL = /(data-cfemail=["']?)[0-9a-f]{2,512}|(\/cdn-cgi\/l\/email-protection#)[0-9a-f]{2,512}/gi;
- // No TLD needed: a cut address has none, and an @ inside a word is no loss.
- const EMAIL = /[\p{L}\p{M}\p{N}._%+'\uFFFD-]{1,64}@[\p{L}\p{M}\p{N}_\uFFFD-]{1,63}(?:\.[\p{L}\p{M}\p{N}_-]{1,63}){0,6}/gu;
- const QUOTED_EMAIL = /"[^"\n]{1,64}"@[\p{L}\p{N}-]{1,63}(?:\.[\p{L}\p{N}-]{1,63}){0,6}/gu;
- const IPV6_RUN = /(?<!:)(?:[0-9a-f]{0,4}:){2,8}(?:\d{1,3}(?:\.\d{1,3}){3}|[0-9a-f]{0,4})(?![0-9a-f])/gi;
- const IPV4 = /(?<!\d)(?:\d{1,3}\.){3}\d{1,3}(?!\d)(?!\.\d)/g;
- const IPV4_DASHED = /(?<!\d)\d{1,3}-\d{1,3}-\d{1,3}-\d{1,3}(?![\d-])/g;
- const IPV4_HEX = /\b0x[0-9a-f]{8}\b/gi;
- const QUERY = /\?[^\s"'<>)\\#?]{1,4096}/gu;
- const FRAGMENT = /(?<=[\w/.-])#[^\s"'<>)\\]{1,4096}/gu;
- const JWT = /eyJ[\w-]{6,4096}\.[\w-]{2,4096}(?:\.[\w-]{0,4096})?/g;
- const BEARER = /\b(bearer|basic|token)\s{1,3}[\w.~+/=-]{8,4096}/gi;
- const UUID = /\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi;
- const NAMED = /\b([\w-]{0,32}(?:token|session[_-]?id|sessionid|sessid|session|sess|sid|csrf|xsrf|auth|secret|nonce|password|passwd|api[_-]?key|access[_-]?key|[_-]pass|[_-]pwd)|pass|pwd)(["']?\s{0,3}[:=]\s{0,3}["']?)[^\s"'<>&;,)]{3,4096}/gi;
- const TOKEN_RUN = /[A-Za-z0-9_-]{16,4096}/g;
- const LONG_NUMBER = /\+?\d[\d\s().-]{7,40}\d/g;
- const digits = (s) => s.replace(/\D/g, "").length;
- const isToken = (s) => /^[0-9a-f]{16,}$/i.test(s) || (digits(s) >= 3 && /[A-Za-z]/.test(s)) || (s.length >= 20 && /\d/.test(s) && /[a-z]/.test(s) && /[A-Z]/.test(s));
- // A run can start with one stray label: `IP:` or the `ip6:` of an SPF record.
- /** One rule for the scrub and its detector. `a::b` is C++: with no digit, an address needs some length. */
- export const looksIpv6 = (s) => isIPv6(s) && (/\d/.test(s) || s.replace(/:/g, "").length >= 6);
- const ipv6Out = (m) => (looksIpv6(m) || (!m.startsWith("::") && looksIpv6(m.slice(m.indexOf(":") + 1))) ? "<ip>" : m);
- const printable = (code, m) => (code >= 0x20 && code < 0x7f ? String.fromCharCode(code) : m);
- const NAMED_CHARS = Object.freeze({ commat: "@", period: ".", lowbar: "_", colon: ":" });
- const safeChar = (m, hex) => { const ch = String.fromCharCode(parseInt(hex, 16)); return /[\w@.:-]/.test(ch) ? ch : m; };
- /** Every way a page writes an address, back to plain text, so one pattern can see it. */
- const decodeForms = (s) => s
- .replace(ZERO_WIDTH, "")
- .replace(/&(?=#)/gi, "&")
- .replace(NUMERIC_ENTITY, (m, hex, dec) => printable(hex ? parseInt(hex, 16) : Number(dec), m))
- .replace(NAMED_ENTITY, (_m, name) => NAMED_CHARS[name.toLowerCase()])
- .replace(ESCAPED, safeChar).replace(/\\100/g, "@").replace(PERCENT, safeChar)
- .replace(TAGGED_AT, "@").replace(DOT_WORD, ".")
- .replace(BRACKET_AT, "<email>").replace(AT_WORD, "@")
- .replace(SPELLED_EMAIL, "$1@$2.$3")
- .replace(DASH_AT, "$1@$2");
- /** ASCII lower case, one for one. `toLowerCase` makes U+0130 two units, which shifts every index after it. */
- const asciiLower = (s) => s.replace(/[A-Z]+/g, (m) => m.toLowerCase());
- /** The end of a tag, quotes respected, or the end of the text. */
- function tagEnd(text, from) {
- let quote = "";
- for (let i = from; i < text.length; i++) {
- const ch = text[i];
- if (quote) { if (ch === quote) quote = ""; } else if (ch === '"' || ch === "'") quote = ch;
- else if (ch === ">") return i + 1;
- }
- return text.length;
- }
- /** Form fields out, whatever their size: a field's value can be a token or a person's details. */
- function stripFields(text) {
- const low = asciiLower(text);
- const field = /<(input|textarea|select)\b/g;
- let out = "", at = 0, m;
- while ((m = field.exec(low))) {
- out += `${text.slice(at, m.index)}<input>`;
- const close = m[1] === "input" ? -1 : low.indexOf(`</${m[1]}`, m.index);
- at = m[1] === "input" ? tagEnd(text, m.index) : close < 0 ? text.length : tagEnd(text, close);
- field.lastIndex = at;
- }
- return out + text.slice(at);
- }
- /** The cap, ending at a space: what the cut split is never kept. */
- function capped(text) {
- if (text.length <= INPUT_CAP_CHARS) return text;
- let end = INPUT_CAP_CHARS;
- while (end > 0 && !/\s/.test(text[end - 1])) end--;
- return text.slice(0, end);
- }
- /** IP addresses and emails out. */
- export const scrubAddresses = (text) => decodeForms(String(text ?? "")).replace(QUOTED_EMAIL, "<email>").replace(EMAIL, "<email>").replace(IPV6_RUN, ipv6Out).replace(IPV4, "<ip>").replace(IPV4_DASHED, "<ip>").replace(IPV4_HEX, "<ip>");
- /** For anything stored: addresses, form fields, query strings, fragments, tokens and long numbers. Run it before slicing. */
- export const scrub = (text) => scrubAddresses(stripFields(capped(String(text ?? ""))).replace(CF_EMAIL, (_m, a, b) => `${a ?? b}<email>`))
- .replace(QUERY, "?<query>").replace(FRAGMENT, "#<fragment>")
- .replace(JWT, "<token>").replace(BEARER, "$1 <token>").replace(UUID, "<id>")
- .replace(NAMED, "$1$2<token>")
- .replace(TOKEN_RUN, (m) => (isToken(m) ? "<token>" : m))
- .replace(LONG_NUMBER, (m) => (digits(m) >= 9 && !/^\d{4}-\d{2}-\d{2}/.test(m) ? "<number>" : m));
- // What a scrub leaves behind, listed here on its own so a change to the scrub can't hide in the detector.
- const OWN_TOKENS = Object.freeze(["<email>", "<ip>", "<token>", "<id>", "<number>", "<query>", "<fragment>", "<input>"]);
- /** Does this text open with one of our tokens, past an opening bracket or quote? Our output, so no address can start there. */
- const opensWithOwnToken = (text) => {
- let i = 0;
- while (i < text.length && "([{\"'".includes(text[i])) i++;
- return OWN_TOKENS.some((t) => text.startsWith(t, i));
- };
- // What can follow a bare `at` as an address: past an opening bracket or quote, a whole domain, walked to its
- // end. Each label starts with a letter or number and ends with one or a mark, with hyphens inside and Unicode
- // allowed. The last is two or more letters, or punycode, at a token's edge. `$29.`, `5:30.`, `Joe's.`, `100%.`
- // and paths hold characters no label can, and our own tokens and web URLs fail the same test.
- // The scrub closes `(info) @ x.y` up to `(info)@x.y`, which the token walk below can't read past the bracket.
- const BARE_AT = /(?: [([{<"'`“‘«*]*|[([{<]+)(?:at|@)(?= )|(?<=[)\]}>"'`”’»])@|(?<=[\p{L}\p{N}._%+-])@(?=[([{<"'`“‘«])/gu;
- const OWN_LOCAL = new RegExp(`(?:${[...OWN_TOKENS, "<redacted>"].join("|")})$`);
- const ADDRESS_LOCAL = /[\p{L}\p{N}._%+-]["'`”’»)\]}>*]*$/u;
- const DOMAIN_OPENING = /[([{<"'`“‘«*]*/y;
- const DOMAIN_LABEL = /[\p{L}\p{N}][\p{L}\p{N}\p{M}-]*/uy;
- // Dots that join labels, and a dot that may instead end a sentence.
- const DOMAIN_JOIN = /[.。.。](?=[\p{L}\p{N}_-])| [.。.。] ?| dot | ?[([{] ?dot ?[)\]}] ?/uy;
- const SENTENCE_DOT = /[.。.。] /y;
- const LAST_LABEL = /^(?:\p{L}[\p{L}\p{M}]+|xn--[a-z0-9-]*[a-z0-9])$/u;
- // A candidate is whole unless what follows shows it goes on as something else. A fire costs one record's text
- // and a miss keeps an address, so this lists what rejects and all else is an edge. Punctuation of a query, fragment,
- // port or sentence only counts up to a space or the end, so `x.com:443`, `x.com?a=b` and `x.com..y` aren't domains,
- // and neither is what the scrub leaves of them, `x.com?<query>` and `x.com#<fragment>`.
- const TOKEN_EDGE = /[.:?#。.。]*(?:$| )|[.:。.。]*[^\s\p{L}\p{N}\p{M}_\-/\\@=&%+~.:?#。.。<]|</uy;
- // A name holds 253 characters, and spelled dots or spaces around them add a few to each label.
- const MAX_DOMAIN_CHARS = 1024;
- /** Is this a whole domain, read to its end? One bad label or character anywhere in it means no. */
- function opensDomain(text, start) {
- DOMAIN_OPENING.lastIndex = start;
- let at = start + DOMAIN_OPENING.exec(text)[0].length;
- const limit = at + MAX_DOMAIN_CHARS;
- let labels = 0, tight = false;
- while (at < limit) {
- DOMAIN_LABEL.lastIndex = at;
- const label = DOMAIN_LABEL.exec(text)?.[0];
- if (!label || label.endsWith("-")) return false;
- at += label.length;
- labels++;
- DOMAIN_JOIN.lastIndex = at;
- const join = DOMAIN_JOIN.exec(text)?.[0];
- if (join) {
- tight ||= /^[.。.。]$/.test(join);
- at += join.length;
- continue;
- }
- const whole = labels > 1 && LAST_LABEL.test(label);
- TOKEN_EDGE.lastIndex = at;
- if (whole) return TOKEN_EDGE.test(text);
- // `night. Open` can be a spaced name. `3.99. Open` is a price and a sentence: a name doesn't turn loose after a tight dot.
- SENTENCE_DOT.lastIndex = at;
- const dot = !tight && SENTENCE_DOT.exec(text)?.[0];
- if (!dot) return false;
- at += dot.length;
- }
- return false;
- }
- // A JWT opens at a token's edge. `keyJ.value` holds the letters and isn't one.
- const JWT_AT_EDGE = /(?:^|[^A-Za-z0-9_-])eyJ[A-Za-z0-9_-]*\.[A-Za-z0-9_-]/;
- /**
- * What should never survive a scrub. This walks tokens with its own tests and
- * shares no pattern with `scrub`, so one regex fault can't hide from both.
- */
- export function scrubMisses(text) {
- const out = { ipv4: 0, ipv6: 0, email: 0, query: 0, fragment: 0, jwt: 0, longHex: 0, longNumber: 0 };
- const whole = String(text ?? "");
- const low = whole.toLowerCase();
- for (const mark of ["&commat", "\\x40", "\\u0040", "\\100", "%40", "%2540", "@", "﹫"]) if (low.includes(mark)) out.email++;
- // A bracketed at, with or without inner spaces, in its own walk over the text.
- for (let i = 0; i < low.length; i++) {
- if (!"[({".includes(low[i])) continue;
- let k = i + 1;
- while (low[k] === " ") k++;
- if (low.startsWith("at", k)) {
- k += 2;
- while (low[k] === " ") k++;
- if (!"])}".includes(low[k] ?? "?")) continue;
- k++;
- while (low[k] === " ") k++;
- if (!opensWithOwnToken(low.slice(k, k + 16))) out.email++;
- }
- }
- // A bare `at` or `@` before something that could be a domain. A bracketed or quoted domain counts, and so
- // does prose that looks like one. Invisible characters, wide forms and spaces spelled as entities don't break it.
- const spaced = low.normalize("NFKC").replace(/\p{Cf}/gu, "").replace(/ |�*160;|�*a0;/g, " ").replace(/\s+/g, " ");
- for (const m of spaced.matchAll(BARE_AT)) {
- const after = m.index + m[0].length;
- if (OWN_LOCAL.test(spaced.slice(Math.max(0, m.index - 16), m.index))) continue;
- if (ADDRESS_LOCAL.test(spaced.slice(Math.max(0, m.index - 16), m.index)) && opensDomain(spaced, spaced[after] === " " ? after + 1 : after)) out.email++;
- }
- // Ordinary pages hold many entities for quotes and dashes. Only ones that spell ASCII hide text.
- let spelled = 0;
- for (let i = low.indexOf("&#"); i >= 0; i = low.indexOf("&#", i + 2)) {
- const hex = low[i + 2] === "x";
- let end = i + (hex ? 3 : 2);
- while (end < low.length && end < i + 16 && (hex ? "0123456789abcdef" : "0123456789").includes(low[end])) end++;
- const code = Number.parseInt(low.slice(i + (hex ? 3 : 2), end), hex ? 16 : 10);
- if (code === 0x40) out.email++;
- if (code >= 0x20 && code < 0x7f) spelled++;
- }
- if (spelled >= 4) out.email++;
- const edge = new Set(`.,;:!?()[]{}<>"'=\``);
- const trim = (t) => {
- let a = 0, b = t.length;
- while (a < b && edge.has(t[a])) a++;
- while (b > a && edge.has(t[b - 1])) b--;
- return t.slice(a, b);
- };
- for (const raw of whole.split(/[\s"'`<>()\[\],;\\]+/)) {
- if (raw === "redacted" || raw === "query" || raw === "fragment") continue;
- const t = trim(raw);
- if (!t) continue;
- for (const piece of [raw, t, ...t.split("="), t.slice(t.indexOf(":") + 1)]) {
- const candidate = isIP(piece) ? piece : trim(piece);
- const v = isIP(candidate);
- if (v === 4) { out.ipv4++; break; }
- if (v === 6 && looksIpv6(candidate)) { out.ipv6++; break; }
- }
- // The @ needs an address on both sides: `}@media` in a stylesheet and `pkg@^1` on a CDN have none.
- const wordChar = (ch) => ch.toLowerCase() !== ch.toUpperCase() || (ch >= "0" && ch <= "9");
- const localEnd = (ch) => wordChar(ch) || "._+-%'".includes(ch);
- if (t.split(/[/?&=:]/).some((piece) => { const at = piece.indexOf("@"); return at > 0 && at < piece.length - 1 && localEnd(piece[at - 1]) && wordChar(piece[at + 1]); })) out.email++;
- const eq = t.indexOf("=");
- const q = t.indexOf("?"), hash = t.indexOf("#");
- if (q >= 0 && eq > q + 1) out.query++;
- if (hash >= 0 && eq > hash + 1) out.fragment++;
- if (JWT_AT_EDGE.test(t)) out.jwt++;
- let run = 0, numerals = 0;
- for (const ch of t.toLowerCase()) {
- run = "0123456789abcdef".includes(ch) ? run + 1 : 0;
- if (ch >= "0" && ch <= "9") numerals++;
- if (run === 32) { out.longHex++; }
- }
- if (numerals >= 12 && numerals === t.length) out.longNumber++;
- }
- return out;
- }
- const headerEvidence = (res) => {
- const h = lower(res.headers);
- return {
- status: res.status,
- headerNames: Object.keys(h).filter((n) => /^[\w-]{1,64}$/.test(n) && scrub(n) === n).sort(),
- values: Object.fromEntries(KEPT_VALUES.filter((k) => h[k] !== undefined).map((k) => [k, scrub(h[k]).slice(0, 120)])),
- cookieNames: cookieNames(h),
- };
- };
- const sha256 = (s) => createHash("sha256").update(s).digest("hex");
- const outside2xx = (status) => status < 200 || status >= 300;
- const SKIPPED_TAGS = Object.freeze(["script", "style", "noscript", "svg", "title", "template", "textarea", "select"]);
- /** The text outside markup, in one pass. An unclosed comment or script runs to the end of the input. */
- function visibleText(html) {
- const low = asciiLower(html);
- let out = "";
- let i = 0;
- while (i < html.length) {
- const lt = low.indexOf("<", i);
- if (lt < 0) { out += html.slice(i); break; }
- out += `${html.slice(i, lt)} `;
- if (low.startsWith("<!--", lt)) {
- const end = low.indexOf("-->", lt + 4);
- if (end < 0) break;
- i = end + 3;
- continue;
- }
- const tag = SKIPPED_TAGS.find((t) => low.startsWith(`<${t}`, lt) && !/[a-z0-9-]/.test(low[lt + t.length + 1] ?? " "));
- if (tag) {
- const end = low.indexOf(`</${tag}`, lt);
- const close = end < 0 ? -1 : low.indexOf(">", end);
- if (close < 0) break;
- i = close + 1;
- continue;
- }
- const gt = low.indexOf(">", lt);
- if (gt < 0) break;
- i = gt + 1;
- }
- return out;
- }
- function between(html, low, tag) {
- const open = low.indexOf(`<${tag}`);
- const start = open < 0 ? -1 : low.indexOf(">", open);
- const end = start < 0 ? -1 : low.indexOf(`</${tag}`, start);
- return end < 0 ? "" : html.slice(start + 1, end);
- }
- /**
- * The visible text of a 2xx response and its shape, enough to tell a page from
- * a JS challenge, a soft block or a custom interstitial. No markup is kept.
- */
- export function pageExcerpt(body) {
- const whole = String(body ?? "");
- const text = stripFields(capped(whole));
- const low = asciiLower(text);
- const flat = (s) => visibleText(s).replace(/&(?:nbsp|amp|quot|lt|gt|#39|#x27);/gi, " ").replace(/\s+/g, " ").trim();
- return {
- title: scrub(flat(between(text, low, "title"))).slice(0, 160),
- text: scrub(flat(visibleText(text))).slice(0, PAGE_TEXT_CHARS),
- noscript: scrub(flat(between(text, low, "noscript"))).slice(0, 300),
- chars: whole.length,
- scripts: text.match(/<script\b/gi)?.length ?? 0,
- forms: text.match(/<form\b/gi)?.length ?? 0,
- links: text.match(/<a\b/gi)?.length ?? 0,
- metaRefresh: /<meta\b[^>]{0,500}http-equiv=["']?refresh/i.test(text),
- soft: SOFT_CHALLENGE_BODIES.filter(([re]) => re.test(text)).map(([, vendor]) => vendor),
- hard: HARD_CHALLENGE_BODIES.filter(([re]) => re.test(text)).map(([, vendor]) => vendor),
- };
- }
- /**
- * What E20-R stores for one site (§ 4.1, Amendment 2). Each of the three
- * responses gets a body hash. Outside 2xx the first 4 KB is kept, scrubbed. A
- * 2xx homepage keeps only `pageExcerpt`.
- */
- export function evidenceOf(page, responses = []) {
- const body = String(page.body ?? "");
- const other = ({ kind, res }) => ({
- kind, ...headerEvidence(res),
- bodySha256: sha256(String(res.body ?? "")),
- ...(outside2xx(res.status) ? { excerpt: scrub(res.body).slice(0, EXCERPT_BYTES) } : {}),
- });
- return {
- ...headerEvidence(page),
- bodySha256: sha256(body),
- ...(outside2xx(page.status) ? { excerpt: scrub(body).slice(0, EXCERPT_BYTES) } : { pageExcerpt: pageExcerpt(body) }),
- class: classify(page),
- classRegistered: classifyRegistered(page),
- signatures: signatureMatches(body),
- csp: cspEvidence(page.headers),
- traits: traitsOf(page.html),
- others: responses.map(other),
- };
- }
bench/front-door/legacy/classify-7023106f.mjs · Bundled script · 464 lines
- /**
- * E20-R response classifier: a fixed, deterministic label for one homepage
- * response. Protocol: docs/research/prereg/E20r-front-door-replication.md § 4.2
- * and Amendment 2. The signature lists are frozen before a run; a change after
- * that is a deviation. No model decides a label (AGENTS.md rule 4).
- *
- * challenge an edge network tests the client
- * block an edge network or firewall refuses the client
- * unrecognized-error-response an error no rule recognized (pass 1: origin-error)
- * candidate-page no marker found; not proof of a page (pass 1: page)
- * body-unavailable a compressed answer, whose body was never read, so nothing can be said of it
- *
- * Header markers come first: Cloudflare documents that the status code alone
- * does not identify a challenge. `classify` is the V2 rule set, written after
- * the V1 labels were read. `classifyRegistered` is the rule set of pass 1,
- * unchanged, so C15 and the comparison with pass 1 use one definition.
- */
- import { createHash } from "node:crypto";
- import { isIP, isIPv6 } from "node:net";
- export const CHALLENGE_HEADERS = Object.freeze([
- ["cf-mitigated", /challenge/i, "cloudflare"],
- ["x-amzn-waf-action", /challenge|captcha/i, "aws-waf"],
- ["x-vercel-mitigated", /challenge/i, "vercel"],
- ]);
- // Pass 1 rules, frozen at registration. Do not edit.
- export const CHALLENGE_BODIES = Object.freeze([
- [/\/cdn-cgi\/challenge-platform\/|<title>\s*Just a moment\.\.\.\s*<\/title>|cf-browser-verification|cf_chl_opt/i, "cloudflare"],
- [/captcha-delivery\.com/i, "datadome"],
- [/px-captcha|captcha\.px-cdn\.net/i, "human"],
- [/_Incapsula_Resource/i, "imperva"],
- [/_sec\/verify\?provider=|sec-if-cpt-container/i, "akamai"],
- [/Vercel Security Checkpoint/i, "vercel"],
- [/AwsWafIntegration|awswaf\.com/i, "aws-waf"],
- ]);
- export const BLOCK_BODIES = Object.freeze([
- [/Attention Required! \| Cloudflare|Sorry, you have been blocked|cf-error-details/i, "cloudflare"],
- [/errors\.edgesuite\.net|<H1>Access Denied<\/H1>/i, "akamai"],
- [/Generated by cloudfront \(CloudFront\)|The request could not be satisfied/i, "cloudfront"],
- [/Incapsula incident ID/i, "imperva"],
- ]);
- // V2. Hard signatures appear only on an interstitial page.
- export const HARD_CHALLENGE_BODIES = Object.freeze([
- [/<title>\s*Just a moment\.\.\.\s*<\/title>|cf-browser-verification|cf_chl_opt|\/cdn-cgi\/challenge-platform\/h\/[a-z]\/orchestrate\/chl_page/i, "cloudflare"],
- [/captcha-delivery\.com\/captcha/i, "datadome"],
- [/px-captcha/i, "human"],
- [/id=["']main-iframe["'][^>]*_Incapsula_Resource/i, "imperva"],
- [/_sec\/verify\?provider=|sec-if-cpt-container/i, "akamai"],
- [/Vercel Security Checkpoint/i, "vercel"],
- [/<title>\s*DDoS-Guard\s*<\/title>/i, "ddos-guard"],
- ]);
- // Soft signatures are scripts that also sit on normal pages as passive bot
- // detection. They count on an error status, or on a 2xx too small to be a page.
- export const SOFT_CHALLENGE_BODIES = Object.freeze([
- [/\/cdn-cgi\/challenge-platform\//i, "cloudflare"],
- [/captcha-delivery\.com/i, "datadome"],
- [/captcha\.px-cdn\.net/i, "human"],
- [/_Incapsula_Resource/i, "imperva"],
- [/AwsWafIntegration|awswaf\.com/i, "aws-waf"],
- ]);
- export const SOFT_MAX_CHARS = 8192;
- export const BLOCK_BODIES_V2 = Object.freeze([
- [/Attention Required! \| Cloudflare|Sorry, you have been blocked|cf-error-details/i, "cloudflare"],
- [/errors\.edgesuite\.net|<H1>Access Denied<\/H1>/i, "akamai"],
- [/Request blocked|configured to block access from your country/i, "cloudfront"],
- [/Incapsula incident ID/i, "imperva"],
- ]);
- // CloudFront's own error page, after the two lines above that mean a refusal.
- export const CLOUDFRONT_ERROR = /The request could not be satisfied/i;
- // Cloudflare's short text error. These four are DNS, tunnel and Worker errors, not refusals.
- const CF_ERROR = /^\s*error code:\s*(1\d{3})\b/i;
- const CF_NOT_REFUSAL = new Set(["1001", "1016", "1033", "1101"]);
- const EDGE_NAMES = Object.freeze([["cf-ray", "cloudflare"], ["x-amz-cf-id", "cloudfront"], ["x-akamai-transformed", "akamai"], ["akamai-grn", "akamai"]]);
- const REFUSAL_STATUS = new Set([401, 403, 406, 429]);
- // Header values kept verbatim, then scrubbed. Every other header keeps its name only.
- export const KEPT_VALUES = Object.freeze(["cf-mitigated", "x-amzn-waf-action", "x-vercel-mitigated", "server", "via", "x-cache", "cf-cache-status", "content-type"]);
- export const EXCERPT_BYTES = 4096;
- export const PAGE_TEXT_CHARS = 4096;
- // Site-controlled text is cut to this before any regex runs on it.
- export const INPUT_CAP_CHARS = 65_536;
- const lower = (headers = {}) => Object.fromEntries(Object.entries(headers).map(([k, v]) => [k.toLowerCase(), v]));
- const edgeOf = (h) => {
- for (const [name, vendor] of EDGE_NAMES) if (h[name] !== undefined) return vendor;
- const server = String(h.server ?? "");
- return /cloudflare/i.test(server) ? "cloudflare" : /cloudfront/i.test(server) ? "cloudfront" : /akamai/i.test(server) ? "akamai" : null;
- };
- /** The label of one response, with the vendor and the rule that decided it. */
- export function classify({ status, headers = {}, body = "" }) {
- const h = lower(headers);
- for (const [name, re, vendor] of CHALLENGE_HEADERS) {
- if (re.test(String(h[name] ?? ""))) return { label: "challenge", vendor, rule: `header:${name}` };
- }
- const whole = String(body);
- const text = whole.slice(0, INPUT_CAP_CHARS);
- for (const [re, vendor] of HARD_CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "body" };
- // A long 2xx page may mention any of these words. It counts on an error, or a body too small to be a page.
- if (status >= 400 || whole.length <= SOFT_MAX_CHARS) {
- for (const [re, vendor] of BLOCK_BODIES_V2) if (re.test(text)) return { label: "block", vendor, rule: "body" };
- for (const [re, vendor] of SOFT_CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "soft-body" };
- if (CLOUDFRONT_ERROR.test(text)) return { label: "unrecognized-error-response", vendor: "cloudfront", rule: "cloudfront:error" };
- }
- const edge = edgeOf(h);
- const code = CF_ERROR.exec(text)?.[1];
- if (status >= 400 && code && edge === "cloudflare" && !CF_NOT_REFUSAL.has(code)) return { label: "block", vendor: "cloudflare", rule: `cloudflare:${code}` };
- // No body to read, so a challenge and a block look the same: the edge refused.
- if (REFUSAL_STATUS.has(status) && !text.trim() && edge) return { label: "block", vendor: edge, rule: "empty-at-edge" };
- // A homepage that answers 403 refuses this client, whoever serves it. A view
- // of signature-backed refusals alone drops this rule.
- if (status === 403) return { label: "block", vendor: edge, rule: "unrecognized-403" };
- if (status >= 400) return { label: "unrecognized-error-response", vendor: null, rule: "status" };
- if (h["content-encoding"] && !/^identity$/i.test(h["content-encoding"]) && !text.trim()) return { label: "body-unavailable", vendor: null, rule: "content-encoding" };
- return { label: "candidate-page", vendor: null, rule: "status" };
- }
- /** Pass 1's rule set, unchanged, on the whole body, so its labels stay comparable. Each rule is a plain alternation. */
- export function classifyRegistered({ status, headers = {}, body = "" }) {
- const h = lower(headers);
- for (const [name, re, vendor] of CHALLENGE_HEADERS) {
- if (re.test(String(h[name] ?? ""))) return { label: "challenge", vendor, rule: `header:${name}` };
- }
- const text = String(body);
- for (const [re, vendor] of CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "body" };
- for (const [re, vendor] of BLOCK_BODIES) if (re.test(text)) return { label: "block", vendor, rule: "body" };
- if (status >= 400) return { label: "origin-error", vendor: null, rule: "status" };
- return { label: "page", vendor: null, rule: "status" };
- }
- const matches = (rules, text) => Object.fromEntries(rules.map(([re, vendor]) => [vendor, re.test(text)]));
- /** Every signature, for every rule set, as a boolean, so a label can be replayed under other rules. */
- export function signatureMatches(body) {
- const whole = String(body ?? "");
- const text = whole.slice(0, INPUT_CAP_CHARS);
- return {
- registeredChallenge: matches(CHALLENGE_BODIES, whole), registeredBlock: matches(BLOCK_BODIES, whole),
- hardChallenge: matches(HARD_CHALLENGE_BODIES, text), softChallenge: matches(SOFT_CHALLENGE_BODIES, text), block: matches(BLOCK_BODIES_V2, text),
- cloudfrontError: CLOUDFRONT_ERROR.test(text), cloudflareCode: CF_ERROR.exec(text)?.[1] ?? null,
- };
- }
- /**
- * What a value test of the CSP needs, from the full header, not the grader's
- * 160-character evidence. Directive names only. `controlsScripts`: a policy has
- * `script-src`, `script-src-elem` or `default-src`. `openScripts`: that source
- * list still allows inline code without a nonce or hash, or any host of a scheme.
- */
- export function cspEvidence(headers = {}) {
- const raw = [].concat(lower(headers)["content-security-policy"] ?? []).join(",").slice(0, INPUT_CAP_CHARS);
- const policies = raw.split(",").map((p) => new Map(p.split(";").map((d) => d.trim().split(/\s+/)).filter(([n]) => /^[a-z-]{1,64}$/i.test(n ?? "")).map(([n, ...v]) => [n.toLowerCase(), v.map((x) => x.toLowerCase())])));
- const script = policies.map((p) => p.get("script-src") ?? p.get("script-src-elem") ?? p.get("default-src")).filter(Boolean);
- const open = (v) => v.some((s) => ["*", "http:", "https:", "data:"].includes(s)) || (v.includes("'unsafe-inline'") && !v.some((s) => /^'(?:nonce-|sha(?:256|384|512)-)/.test(s)));
- return {
- present: raw.trim().length > 0, chars: raw.length,
- directives: [...new Set(policies.flatMap((p) => [...p.keys()]))].sort().slice(0, 40),
- controlsScripts: script.length > 0,
- openScripts: script.length > 0 && script.every(open),
- };
- }
- /** Cookie names only, never values. */
- export function cookieNames(headers = {}) {
- const raw = lower(headers)["set-cookie"];
- const list = Array.isArray(raw) ? raw : raw ? [raw] : [];
- // No = before the first ; means there is no name, and what stands there may be a value.
- // A name that looks like a token, an address or anything but a name isn't kept.
- const name = (c) => { const pair = String(c).split(";")[0]; return pair.includes("=") ? pair.split("=")[0].trim() : ""; };
- return list.map(name).filter((n) => /^[\w.-]{1,64}$/.test(n) && scrub(n) === n).sort();
- }
- // § 6, secondary: what kind of site this is, from the homepage already fetched.
- // Fixed rules on the HTML; only the booleans are kept, never a URL or a name.
- export const TRAIT_RULES = Object.freeze({
- login: /<a\b[^>]{0,2000}>\s{0,200}(?:<[^>]{1,500}>\s{0,200}){0,8}(log\s?in|sign\s?in|my account|inloggen|anmelden|connexion)\b/i,
- payments: /js\.stripe\.com|checkout\.stripe\.com|paypal\.com\/sdk|paypalobjects\.com|js\.braintreegateway\.com|checkoutshopper-[a-z]+\.adyen\.com|klarna(?:cdn|\.com)|mollie\.com/i,
- shop: /cdn\.shopify\.com|woocommerce|magento|\/cart\b|\/checkout\b|add[-_ ]to[-_ ]cart/i,
- // A public security or trust page: a stand-in for a team that owns security, not proof of a CISO.
- securityPage: /<a\b[^>]{0,2000}href="[^"]{0,500}(?:\/security\b|\/trust\b|\/trust-center|\/\/trust\.|\/\/security\.)[^"]{0,500}"/i,
- });
- /** Booleans for one page. An empty page (no 2xx HTML) gives null: unknown, not "no". */
- export function traitsOf(html) {
- if (!html) return null;
- const text = String(html).slice(0, INPUT_CAP_CHARS);
- return Object.fromEntries(Object.entries(TRAIT_RULES).map(([k, re]) => [k, re.test(text)]));
- }
- // Every repeat below is bounded, so no input of INPUT_CAP_CHARS can make these slow.
- const ZERO_WIDTH = /[-]/g;
- const NUMERIC_ENTITY = /&#(?:x0*([0-9a-f]{1,8})|0*(\d{1,8}));?/gi;
- const NAMED_ENTITY = /&(commat|period|lowbar|colon);?/gi;
- const ESCAPED = /\\(?:x|u00)([0-9a-f]{2})/gi;
- const PERCENT = /%(?:25)?([2-7][0-9a-f])/gi;
- // A bracketed at is an address, whatever stands around it: both neighbouring words go.
- const BRACKET_AT = /[^\s<>"']{0,64}\s{0,3}[[({]\s{0,2}at\s{0,2}[\])}]\s{0,3}[^\s<>"']{0,64}/gi;
- const AT_WORD = /\s{1,3}@\s{0,3}(?=[\p{L}\p{N}])|(?<=[\p{L}\p{N}\p{M}])\s{0,3}@\s{1,3}|[@﹫]/gu;
- const TAGGED_AT = /(?:<\/?[a-z]{1,8}>\s{0,2}){1,4}@(?:\s{0,2}<\/?[a-z]{1,8}>){1,4}/gi;
- const DASH_AT = /([\p{L}\p{N}+]{1,64})[-_.]at[-_.]([\p{L}\p{N}-]{1,63}\.\p{L}{2,24})\b/giu;
- const DOT_WORD = /\s{0,3}[[({]\s{0,2}(?:dot|\.)\s{0,2}[\])}]\s{0,3}/gi;
- // A bare `at` counts only before something with a dot, spaced or not.
- const SPELLED_EMAIL = /([\p{L}\p{N}._%+-]{1,64})\s{1,3}at\s{1,3}([\p{L}\p{N}_-]{1,63})(?:\s{1,3}dot\s{1,3}|\s{0,3}\.\s{0,3})(\p{L}{2,24})\b/giu;
- const CF_EMAIL = /(data-cfemail=["']?)[0-9a-f]{2,512}|(\/cdn-cgi\/l\/email-protection#)[0-9a-f]{2,512}/gi;
- // No TLD needed: a cut address has none, and an @ inside a word is no loss.
- const EMAIL = /[\p{L}\p{M}\p{N}._%+'\uFFFD-]{1,64}@[\p{L}\p{M}\p{N}_\uFFFD-]{1,63}(?:\.[\p{L}\p{M}\p{N}_-]{1,63}){0,6}/gu;
- const QUOTED_EMAIL = /"[^"\n]{1,64}"@[\p{L}\p{N}-]{1,63}(?:\.[\p{L}\p{N}-]{1,63}){0,6}/gu;
- const IPV6_RUN = /(?<!:)(?:[0-9a-f]{0,4}:){2,8}(?:\d{1,3}(?:\.\d{1,3}){3}|[0-9a-f]{0,4})(?![0-9a-f])/gi;
- const IPV4 = /(?<!\d)(?:\d{1,3}\.){3}\d{1,3}(?!\d)(?!\.\d)/g;
- const IPV4_DASHED = /(?<!\d)\d{1,3}-\d{1,3}-\d{1,3}-\d{1,3}(?![\d-])/g;
- const IPV4_HEX = /\b0x[0-9a-f]{8}\b/gi;
- const QUERY = /\?[^\s"'<>)\\#?]{1,4096}/gu;
- const FRAGMENT = /(?<=[\w/.-])#[^\s"'<>)\\]{1,4096}/gu;
- const JWT = /eyJ[\w-]{6,4096}\.[\w-]{2,4096}(?:\.[\w-]{0,4096})?/g;
- const BEARER = /\b(bearer|basic|token)\s{1,3}[\w.~+/=-]{8,4096}/gi;
- const UUID = /\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi;
- const NAMED = /\b([\w-]{0,32}(?:token|session[_-]?id|sessionid|sessid|session|sess|sid|csrf|xsrf|auth|secret|nonce|password|passwd|api[_-]?key|access[_-]?key|[_-]pass|[_-]pwd)|pass|pwd)(["']?\s{0,3}[:=]\s{0,3}["']?)[^\s"'<>&;,)]{3,4096}/gi;
- const TOKEN_RUN = /[A-Za-z0-9_-]{16,4096}/g;
- const LONG_NUMBER = /\+?\d[\d\s().-]{7,40}\d/g;
- const digits = (s) => s.replace(/\D/g, "").length;
- const isToken = (s) => /^[0-9a-f]{16,}$/i.test(s) || (digits(s) >= 3 && /[A-Za-z]/.test(s)) || (s.length >= 20 && /\d/.test(s) && /[a-z]/.test(s) && /[A-Z]/.test(s));
- // A run can start with one stray label: `IP:` or the `ip6:` of an SPF record.
- /** One rule for the scrub and its detector. `a::b` is C++: with no digit, an address needs some length. */
- export const looksIpv6 = (s) => isIPv6(s) && (/\d/.test(s) || s.replace(/:/g, "").length >= 6);
- const ipv6Out = (m) => (looksIpv6(m) || (!m.startsWith("::") && looksIpv6(m.slice(m.indexOf(":") + 1))) ? "<ip>" : m);
- const printable = (code, m) => (code >= 0x20 && code < 0x7f ? String.fromCharCode(code) : m);
- const NAMED_CHARS = Object.freeze({ commat: "@", period: ".", lowbar: "_", colon: ":" });
- const safeChar = (m, hex) => { const ch = String.fromCharCode(parseInt(hex, 16)); return /[\w@.:-]/.test(ch) ? ch : m; };
- /** Every way a page writes an address, back to plain text, so one pattern can see it. */
- const decodeForms = (s) => s
- .replace(ZERO_WIDTH, "")
- .replace(/&(?=#)/gi, "&")
- .replace(NUMERIC_ENTITY, (m, hex, dec) => printable(hex ? parseInt(hex, 16) : Number(dec), m))
- .replace(NAMED_ENTITY, (_m, name) => NAMED_CHARS[name.toLowerCase()])
- .replace(ESCAPED, safeChar).replace(/\\100/g, "@").replace(PERCENT, safeChar)
- .replace(TAGGED_AT, "@").replace(DOT_WORD, ".")
- .replace(BRACKET_AT, "<email>").replace(AT_WORD, "@")
- .replace(SPELLED_EMAIL, "$1@$2.$3")
- .replace(DASH_AT, "$1@$2");
- /** ASCII lower case, one for one. `toLowerCase` makes U+0130 two units, which shifts every index after it. */
- const asciiLower = (s) => s.replace(/[A-Z]+/g, (m) => m.toLowerCase());
- /** The end of a tag, quotes respected, or the end of the text. */
- function tagEnd(text, from) {
- let quote = "";
- for (let i = from; i < text.length; i++) {
- const ch = text[i];
- if (quote) { if (ch === quote) quote = ""; } else if (ch === '"' || ch === "'") quote = ch;
- else if (ch === ">") return i + 1;
- }
- return text.length;
- }
- /** Form fields out, whatever their size: a field's value can be a token or a person's details. */
- function stripFields(text) {
- const low = asciiLower(text);
- const field = /<(input|textarea|select)\b/g;
- let out = "", at = 0, m;
- while ((m = field.exec(low))) {
- out += `${text.slice(at, m.index)}<input>`;
- const close = m[1] === "input" ? -1 : low.indexOf(`</${m[1]}`, m.index);
- at = m[1] === "input" ? tagEnd(text, m.index) : close < 0 ? text.length : tagEnd(text, close);
- field.lastIndex = at;
- }
- return out + text.slice(at);
- }
- /** The cap, ending at a space: what the cut split is never kept. */
- function capped(text) {
- if (text.length <= INPUT_CAP_CHARS) return text;
- let end = INPUT_CAP_CHARS;
- while (end > 0 && !/\s/.test(text[end - 1])) end--;
- return text.slice(0, end);
- }
- /** IP addresses and emails out. */
- export const scrubAddresses = (text) => decodeForms(String(text ?? "")).replace(QUOTED_EMAIL, "<email>").replace(EMAIL, "<email>").replace(IPV6_RUN, ipv6Out).replace(IPV4, "<ip>").replace(IPV4_DASHED, "<ip>").replace(IPV4_HEX, "<ip>");
- /** For anything stored: addresses, form fields, query strings, fragments, tokens and long numbers. Run it before slicing. */
- export const scrub = (text) => scrubAddresses(stripFields(capped(String(text ?? ""))).replace(CF_EMAIL, (_m, a, b) => `${a ?? b}<email>`))
- .replace(QUERY, "?<query>").replace(FRAGMENT, "#<fragment>")
- .replace(JWT, "<token>").replace(BEARER, "$1 <token>").replace(UUID, "<id>")
- .replace(NAMED, "$1$2<token>")
- .replace(TOKEN_RUN, (m) => (isToken(m) ? "<token>" : m))
- .replace(LONG_NUMBER, (m) => (digits(m) >= 9 && !/^\d{4}-\d{2}-\d{2}/.test(m) ? "<number>" : m));
- /**
- * What should never survive a scrub. This walks tokens with its own tests and
- * shares no pattern with `scrub`, so one regex fault can't hide from both.
- */
- export function scrubMisses(text) {
- const out = { ipv4: 0, ipv6: 0, email: 0, query: 0, fragment: 0, jwt: 0, longHex: 0, longNumber: 0 };
- const whole = String(text ?? "");
- const low = whole.toLowerCase();
- for (const mark of ["&commat", "\\x40", "\\u0040", "\\100", "%40", "%2540", "@", "﹫"]) if (low.includes(mark)) out.email++;
- // A bracketed at, with or without inner spaces, in its own walk over the text.
- for (let i = 0; i < low.length; i++) {
- if (!"[({".includes(low[i])) continue;
- let k = i + 1;
- while (low[k] === " ") k++;
- if (low.startsWith("at", k)) {
- k += 2;
- while (low[k] === " ") k++;
- if ("])}".includes(low[k] ?? "?")) out.email++;
- }
- }
- // A bare `at` before something with a dot.
- const words = low.split(/\s+/);
- for (let i = 1; i < words.length; i++) {
- if (words[i] !== "at" || !/[\p{L}\p{N}._%+-]$/u.test(words[i - 1])) continue;
- if (/^[^\s.]+(?:\s*\.\s*|\s+dot\s+)[a-z]{2,}\b/.test(words.slice(i + 1, i + 5).join(" "))) out.email++;
- }
- // Ordinary pages hold many entities for quotes and dashes. Only ones that spell ASCII hide text.
- let spelled = 0;
- for (let i = low.indexOf("&#"); i >= 0; i = low.indexOf("&#", i + 2)) {
- const hex = low[i + 2] === "x";
- let end = i + (hex ? 3 : 2);
- while (end < low.length && end < i + 16 && (hex ? "0123456789abcdef" : "0123456789").includes(low[end])) end++;
- const code = Number.parseInt(low.slice(i + (hex ? 3 : 2), end), hex ? 16 : 10);
- if (code === 0x40) out.email++;
- if (code >= 0x20 && code < 0x7f) spelled++;
- }
- if (spelled >= 4) out.email++;
- const edge = new Set(`.,;:!?()[]{}<>"'=\``);
- const trim = (t) => {
- let a = 0, b = t.length;
- while (a < b && edge.has(t[a])) a++;
- while (b > a && edge.has(t[b - 1])) b--;
- return t.slice(a, b);
- };
- for (const raw of whole.split(/[\s"'`<>()\[\],;\\]+/)) {
- if (raw === "redacted" || raw === "query" || raw === "fragment") continue;
- const t = trim(raw);
- if (!t) continue;
- for (const piece of [raw, t, ...t.split("="), t.slice(t.indexOf(":") + 1)]) {
- const candidate = isIP(piece) ? piece : trim(piece);
- const v = isIP(candidate);
- if (v === 4) { out.ipv4++; break; }
- if (v === 6 && looksIpv6(candidate)) { out.ipv6++; break; }
- }
- // The @ needs an address on both sides: `}@media` in a stylesheet and `pkg@^1` on a CDN have none.
- const wordChar = (ch) => ch.toLowerCase() !== ch.toUpperCase() || (ch >= "0" && ch <= "9");
- const localEnd = (ch) => wordChar(ch) || "._+-%'".includes(ch);
- if (t.split(/[/?&=:]/).some((piece) => { const at = piece.indexOf("@"); return at > 0 && at < piece.length - 1 && localEnd(piece[at - 1]) && wordChar(piece[at + 1]); })) out.email++;
- const eq = t.indexOf("=");
- const q = t.indexOf("?"), hash = t.indexOf("#");
- if (q >= 0 && eq > q + 1) out.query++;
- if (hash >= 0 && eq > hash + 1) out.fragment++;
- if (t.includes("eyJ") && t.split(".").length >= 2) out.jwt++;
- let run = 0, numerals = 0;
- for (const ch of t.toLowerCase()) {
- run = "0123456789abcdef".includes(ch) ? run + 1 : 0;
- if (ch >= "0" && ch <= "9") numerals++;
- if (run === 32) { out.longHex++; }
- }
- if (numerals >= 12 && numerals === t.length) out.longNumber++;
- }
- return out;
- }
- const headerEvidence = (res) => {
- const h = lower(res.headers);
- return {
- status: res.status,
- headerNames: Object.keys(h).filter((n) => /^[\w-]{1,64}$/.test(n) && scrub(n) === n).sort(),
- values: Object.fromEntries(KEPT_VALUES.filter((k) => h[k] !== undefined).map((k) => [k, scrub(h[k]).slice(0, 120)])),
- cookieNames: cookieNames(h),
- };
- };
- const sha256 = (s) => createHash("sha256").update(s).digest("hex");
- const outside2xx = (status) => status < 200 || status >= 300;
- const SKIPPED_TAGS = Object.freeze(["script", "style", "noscript", "svg", "title", "template", "textarea", "select"]);
- /** The text outside markup, in one pass. An unclosed comment or script runs to the end of the input. */
- function visibleText(html) {
- const low = asciiLower(html);
- let out = "";
- let i = 0;
- while (i < html.length) {
- const lt = low.indexOf("<", i);
- if (lt < 0) { out += html.slice(i); break; }
- out += `${html.slice(i, lt)} `;
- if (low.startsWith("<!--", lt)) {
- const end = low.indexOf("-->", lt + 4);
- if (end < 0) break;
- i = end + 3;
- continue;
- }
- const tag = SKIPPED_TAGS.find((t) => low.startsWith(`<${t}`, lt) && !/[a-z0-9-]/.test(low[lt + t.length + 1] ?? " "));
- if (tag) {
- const end = low.indexOf(`</${tag}`, lt);
- const close = end < 0 ? -1 : low.indexOf(">", end);
- if (close < 0) break;
- i = close + 1;
- continue;
- }
- const gt = low.indexOf(">", lt);
- if (gt < 0) break;
- i = gt + 1;
- }
- return out;
- }
- function between(html, low, tag) {
- const open = low.indexOf(`<${tag}`);
- const start = open < 0 ? -1 : low.indexOf(">", open);
- const end = start < 0 ? -1 : low.indexOf(`</${tag}`, start);
- return end < 0 ? "" : html.slice(start + 1, end);
- }
- /**
- * The visible text of a 2xx response and its shape, enough to tell a page from
- * a JS challenge, a soft block or a custom interstitial. No markup is kept.
- */
- export function pageExcerpt(body) {
- const whole = String(body ?? "");
- const text = stripFields(capped(whole));
- const low = asciiLower(text);
- const flat = (s) => visibleText(s).replace(/&(?:nbsp|amp|quot|lt|gt|#39|#x27);/gi, " ").replace(/\s+/g, " ").trim();
- return {
- title: scrub(flat(between(text, low, "title"))).slice(0, 160),
- text: scrub(flat(visibleText(text))).slice(0, PAGE_TEXT_CHARS),
- noscript: scrub(flat(between(text, low, "noscript"))).slice(0, 300),
- chars: whole.length,
- scripts: text.match(/<script\b/gi)?.length ?? 0,
- forms: text.match(/<form\b/gi)?.length ?? 0,
- links: text.match(/<a\b/gi)?.length ?? 0,
- metaRefresh: /<meta\b[^>]{0,500}http-equiv=["']?refresh/i.test(text),
- soft: SOFT_CHALLENGE_BODIES.filter(([re]) => re.test(text)).map(([, vendor]) => vendor),
- hard: HARD_CHALLENGE_BODIES.filter(([re]) => re.test(text)).map(([, vendor]) => vendor),
- };
- }
- /**
- * What E20-R stores for one site (§ 4.1, Amendment 2). Each of the three
- * responses gets a body hash. Outside 2xx the first 4 KB is kept, scrubbed. A
- * 2xx homepage keeps only `pageExcerpt`.
- */
- export function evidenceOf(page, responses = []) {
- const body = String(page.body ?? "");
- const other = ({ kind, res }) => ({
- kind, ...headerEvidence(res),
- bodySha256: sha256(String(res.body ?? "")),
- ...(outside2xx(res.status) ? { excerpt: scrub(res.body).slice(0, EXCERPT_BYTES) } : {}),
- });
- return {
- ...headerEvidence(page),
- bodySha256: sha256(body),
- ...(outside2xx(page.status) ? { excerpt: scrub(body).slice(0, EXCERPT_BYTES) } : { pageExcerpt: pageExcerpt(body) }),
- class: classify(page),
- classRegistered: classifyRegistered(page),
- signatures: signatureMatches(body),
- csp: cspEvidence(page.headers),
- traits: traitsOf(page.html),
- others: responses.map(other),
- };
- }
install.sh · Bundled script · 83 lines · 1 finding marked
- #!/usr/bin/env bash
- #
- # Penthera installer — clones the repo, links the CLI, and installs the
- # Agent Skill into Cursor and Claude Code if it finds them.
- #
- # curl -fsSL https://raw.githubusercontent.com/danoszz/penthera/main/install.sh | bash
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 6 on GitHub
- #
- # This script is intentionally short and readable — review it before running.
- # It uses no sudo, makes no network calls beyond `git clone` + `npm install`,
- # and only writes to ~/.penthera and your agent skill folders.
- #
- set -euo pipefail
- REPO="https://github.com/danoszz/penthera.git"
- DIR="${PENTHERA_DIR:-$HOME/.penthera}"
- BLUE='\033[1;38;2;0;0;237m'; DIM='\033[2m'; GREEN='\033[32m'; YELLOW='\033[33m'; RED='\033[31m'; OFF='\033[0m'
- say() { printf "%b\n" "$1"; }
- ok() { printf " %b+%b %s\n" "$GREEN" "$OFF" "$1"; }
- warn() { printf " %b!%b %s\n" "$YELLOW" "$OFF" "$1"; }
- die() { printf " %bx%b %s\n" "$RED" "$OFF" "$1" >&2; exit 1; }
- say ""
- say " ${BLUE}P E N T H E R A${OFF} ${DIM}installer${OFF}"
- say ""
- # ── Prerequisites ───────────────────────────────────────────────────────────
- command -v git >/dev/null 2>&1 || die "git is required. Install it and re-run."
- command -v node >/dev/null 2>&1 || die "Node.js 22+ is required: https://nodejs.org"
- NODE_MAJOR="$(node -p 'process.versions.node.split(".")[0]')"
- [ "$NODE_MAJOR" -ge 22 ] || die "Node 22+ required (found $(node -v))."
- ok "Node $(node -v)"
- # ── Clone or update ─────────────────────────────────────────────────────────
- if [ -d "$DIR/.git" ]; then
- git -C "$DIR" pull --ff-only --quiet && ok "Updated $DIR"
- else
- git clone --depth 1 --quiet "$REPO" "$DIR" && ok "Cloned to $DIR"
- fi
- # ── Install dependencies + link CLI ─────────────────────────────────────────
- cd "$DIR"
- # Always from the lockfile: `npm ci` installs exactly the reviewed tree and
- # checks every tarball against its recorded integrity hash.
- [ -f package-lock.json ] || die "package-lock.json missing in $DIR; refusing an unpinned install."
- npm ci --omit=dev --silent
- ok "Installed dependencies"
- if npm link >/dev/null 2>&1; then
- ok "Linked CLI — run: penthera --version"
- else
- warn "Could not link globally (permission?). Run from $DIR: node bin/penthera.js"
- fi
- # ── Install the Agent Skill into detected agents ────────────────────────────
- install_skill() {
- local target="$1" name="$2"
- mkdir -p "$target"
- rm -rf "$target/penthera"
- cp -R "$DIR/skills/penthera" "$target/penthera"
- ok "Installed skill for $name → $target/penthera"
- }
- INSTALLED_ANY=0
- [ -d "$HOME/.cursor" ] && { install_skill "$HOME/.cursor/skills" "Cursor"; INSTALLED_ANY=1; }
- [ -d "$HOME/.claude" ] && { install_skill "$HOME/.claude/skills" "Claude Code"; INSTALLED_ANY=1; }
- if [ "$INSTALLED_ANY" -eq 0 ]; then
- warn "No Cursor/Claude Code config found. Cross-agent install: npx skills add danoszz/penthera"
- fi
- # ── Done ────────────────────────────────────────────────────────────────────
- say ""
- say " ${GREEN}Done.${OFF} Restart your agent, then just ask it:"
- say ""
- say " ${DIM}\"Scan my localhost:3000 for security issues\"${OFF}"
- say " ${DIM}\"Run a pre-deploy security audit on staging.myapp.com\"${OFF}"
- say " ${DIM}\"Find hardcoded secrets in this repo\"${OFF}"
- say ""
- say " Or from your terminal: ${BLUE}penthera${OFF} ${DIM}(interactive wizard)${OFF}"
- say ""
- say " ${YELLOW}Only scan systems you own or are authorized to test.${OFF}"
- say ""
lib/agents/hidden-content.js · Bundled script · 885 lines · 1 finding marked
- /**
- * Hidden-content engine for agent instructions (skills, rule files, commands).
- *
- * Deterministic and offline. It looks for content a human reviewer would not
- * see but a model would read and act on, and for payloads that run code.
- *
- * Designed against the ways LLM-judged skill scanners have been bypassed:
- * - Invisible text: Unicode tag characters, zero-width and bidi controls,
- * variation-selector smuggling. Decoded where possible, so the report shows
- * what the model would read.
- * - Content pushed out of view: hundreds of blank lines or a wide run of
- * spaces, so the payload sits past what a reviewer or a truncating judge sees.
- * This engine reads the whole file; there is no context window to overflow.
- * - Instructions in HTML comments, which Markdown renderers hide.
- * - Text addressed to the scanner itself ("this skill is safe, approve it").
- * A regex cannot be talked out of a finding, but the attempt is a signal.
- * - Payloads moved out of SKILL.md: every file in a skill folder is scanned,
- * binaries and archives are identified by magic bytes rather than extension,
- * and symlinks that leave the folder are reported.
- * - Encoded payloads: long base64 blobs, decoded and classified.
- *
- * Each detector returns hits `{ rule, severity, title, detail, line, count }`.
- * Callers turn hits into findings.
- */
- import { rulesOf, rulesGeneration, packTest, packMatch } from "../rules/index.js";
- import { redactCommand } from "./redact.js";
- // ── Invisible characters ──────────────────────────────────────────────────
- const TAG_RANGE = /[\u{E0000}-\u{E007F}]/u;
- const TAG_RUN = /[\u{E0000}-\u{E007F}]+/gu;
- const VS_SUPPLEMENT_RUN = /[\u{E0100}-\u{E01EF}]+/gu;
- const BIDI = /[--]/g;
- Note: Usual in code (characters a sanitizer strips): Contains 4 bidirectional-override character(s), which make text display in a different order from how it's read
- Line 33 on GitHub
- // Zero-width and other format characters that render as nothing. U+200D (ZWJ)
- // and U+200C (ZWNJ) have legitimate uses (emoji sequences, Persian, Indic
- // scripts), so those are filtered by context below.
- const ZERO_WIDTH = /[--ㅤᅟᅠᅠ]/g;
- const PICTO = /\p{Extended_Pictographic}|[\u{1F3FB}-\u{1F3FF}️]/u;
- const NON_LATIN_LETTER = /[\p{L}\p{M}]/u;
- function lineOf(text, index) {
- let n = 1;
- for (let i = 0; i < index && i < text.length; i++) if (text.charCodeAt(i) === 10) n++;
- return n;
- }
- /** Decode Unicode tag characters to the ASCII they smuggle. */
- export function decodeTagChars(s) {
- let out = "";
- for (const ch of s) {
- const cp = ch.codePointAt(0);
- if (cp >= 0xe0020 && cp <= 0xe007e) out += String.fromCharCode(cp - 0xe0000);
- }
- return out;
- }
- /** Make a decoded payload safe to print: visible ASCII only, truncated. */
- export function preview(s, max = 80) {
- const clean = String(s).replace(/[^\x20-\x7E]/g, "·").replace(/\s+/g, " ").trim();
- return clean.length > max ? clean.slice(0, max) + "…" : clean;
- }
- /**
- * Text from a scanned file as a finding quotes it: credentials hidden
- * (redactCommand), then made safe and shortened (preview). `context` is the
- * text it was cut from; a secret there is hidden in the quote too. Only the
- * first 4 × max characters are redacted: the rest is cut anyway.
- */
- export function quote(text, max = 80, context) {
- const s = String(text ?? "");
- const part = s.length > max * 4 ? s.slice(0, max * 4) : s;
- return preview(redactCommand(part, context ?? s), max);
- }
- /** The line around `text[index, index + length)`, at most 8 KiB either side: the context of a quoted match. */
- function lineAround(text, index, length) {
- const from = Math.max(text.lastIndexOf("\n", index) + 1, index - 8192);
- let to = text.indexOf("\n", index + length);
- if (to < 0) to = text.length;
- return text.slice(from, Math.min(to, index + length + 8192));
- }
- function detectInvisible(text) {
- const hits = [];
- const tagRuns = [...text.matchAll(TAG_RUN)];
- if (tagRuns.length) {
- const decoded = tagRuns.map((m) => decodeTagChars(m[0])).join(" ").trim();
- const count = tagRuns.reduce((n, m) => n + [...m[0]].length, 0);
- hits.push({
- rule: "unicode-tags",
- severity: "critical",
- title: "Hidden text in Unicode tag characters",
- detail: `${count} invisible tag character(s) (U+E0000–E007F). A model reads them as text; a reviewer sees nothing.` +
- (decoded ? ` Decoded: "${quote(decoded)}"` : ""),
- line: lineOf(text, tagRuns[0].index),
- count,
- });
- }
- const vsRuns = [...text.matchAll(VS_SUPPLEMENT_RUN)];
- if (vsRuns.length) {
- const count = vsRuns.reduce((n, m) => n + [...m[0]].length, 0);
- hits.push({
- rule: "variation-selectors",
- severity: "high",
- title: "Data hidden in variation selectors",
- detail: `${count} supplementary variation selector(s) (U+E0100–E01EF). These are invisible and can encode arbitrary bytes ("emoji smuggling").`,
- line: lineOf(text, vsRuns[0].index),
- count,
- });
- }
- const bidi = [...text.matchAll(BIDI)];
- if (bidi.length) {
- hits.push({
- rule: "bidi-controls",
- severity: "high",
- title: "Bidirectional control characters",
- detail: `${bidi.length} bidi override/isolate character(s) (U+202A–202E, U+2066–2069). They reorder how text displays, so what a reviewer reads differs from what the model reads (Trojan Source).`,
- line: lineOf(text, bidi[0].index),
- count: bidi.length,
- });
- }
- const zw = [];
- for (const m of text.matchAll(ZERO_WIDTH)) {
- const ch = m[0];
- const i = m.index;
- if (ch === "" && i === 0) continue; // byte-order mark
- // A zero-width space at the start of a line before ``` or ~~~ is the usual
- // trick for showing a code fence inside a fenced block.
- if (ch === "" && (i === 0 || text[i - 1] === "\n") && /^(?:```|~~~)/.test(text.slice(i + 1, i + 4))) continue;
- const prev = text.slice(Math.max(0, i - 2), i);
- const next = text.slice(i + 1, i + 3);
- if (ch === "" && PICTO.test(prev) && PICTO.test(next)) continue; // emoji ZWJ sequence
- if (ch === "" || ch === "") {
- const p = [...prev].pop() || "";
- const n = [...next][0] || "";
- // ZWNJ/ZWJ between letters of a non-Latin script is ordinary orthography.
- if (NON_LATIN_LETTER.test(p) && NON_LATIN_LETTER.test(n) && /[^\x00-ɏ]/.test(p + n)) continue;
- }
- zw.push(i);
- }
- if (zw.length) {
- hits.push({
- rule: "zero-width",
- severity: zw.length >= 3 ? "high" : "medium",
- title: "Zero-width or invisible characters",
- detail: `${zw.length} zero-width/invisible character(s). They can split keywords to dodge filters or carry a hidden binary encoding.`,
- line: lineOf(text, zw[0]),
- count: zw.length,
- });
- }
- return hits;
- }
- // ── Whitespace inflation ──────────────────────────────────────────────────
- export const DEFAULTS = {
- blankLines: 40, // consecutive blank lines before more content
- horizontalRun: 200, // consecutive spaces/tabs on one line before more content
- base64MinLength: 200, // characters in one base64 run
- };
- // Linear scans, not regexes: a nested whitespace quantifier backtracks
- // quadratically on exactly the input an attacker would write here.
- function detectWhitespaceInflation(text, opts) {
- const hits = [];
- const lines = text.split("\n");
- let run = 0;
- let longestV = 0;
- let firstV = null;
- let countV = 0;
- let longestH = 0;
- let firstH = null;
- let countH = 0;
- for (let i = 0; i < lines.length; i++) {
- const ln = lines[i];
- if (/^[ \t\r]*$/.test(ln)) { run++; continue; }
- if (run >= opts.blankLines) {
- countV++;
- longestV = Math.max(longestV, run);
- if (firstV === null) firstV = i + 1;
- }
- run = 0;
- let ws = 0;
- for (let j = 0; j < ln.length; j++) {
- const ch = ln.charCodeAt(j);
- if (ch === 32 || ch === 9 || ch === 0xa0) { ws++; continue; }
- if (ws >= opts.horizontalRun && ch !== 13) {
- countH++;
- longestH = Math.max(longestH, ws);
- if (firstH === null) firstH = i + 1;
- }
- ws = 0;
- }
- }
- if (countV) {
- hits.push({
- rule: "whitespace-vertical",
- severity: "high",
- title: "Content hidden below a wall of blank lines",
- detail: `${longestV} consecutive blank lines, then more content. This pushes text past where a reviewer stops scrolling, and past what a truncating scanner reads.`,
- line: firstV,
- count: countV,
- });
- }
- if (countH) {
- hits.push({
- rule: "whitespace-horizontal",
- severity: "high",
- title: "Content hidden off-screen to the right",
- detail: `A run of ${longestH} spaces, then more text on the same line. Editors and diff views without wrapping never show it.`,
- line: firstH,
- count: countH,
- });
- }
- return hits;
- }
- // ── Instructions in hidden places ─────────────────────────────────────────
- export const INSTRUCTION_PATTERNS = [
- /\b(?:ignore|disregard|forget|override)\b[^.\n]{0,40}\b(?:previous|prior|above|earlier|all|any|other)\b[^.\n]{0,20}\b(?:instructions?|rules?|prompts?|guidelines?|directions?)\b/i,
- /\b(?:ignore|disregard|forget)\b[^.\n]{0,30}\bthe user\b/i,
- /\b(?:you|assistant|claude|agent|model|ai|llm)\b[^.\n]{0,30}\b(?:must|should|shall|will|are to|need to|have to)\b/i,
- /\bdo(?:n'?t| not) (?:tell|inform|mention|reveal|show|alert|ask|notify)\b/i,
- /\bwithout (?:asking|telling|informing|confirm\w*|permission|the user)/i,
- /\b(?:system prompt|developer message|new instructions?)\b/i,
- /\b(?:run|execute|exec|invoke)\b[^.\n]{0,40}\b(?:command|script|bash|shell|curl|wget|powershell)\b/i,
- /\b(?:send|upload|post|exfiltrat\w*|transmit|forward|copy)\b[^\n]{0,60}(?:https?:\/\/|\btokens?\b|\bsecrets?\b|\bapi keys?\b|\bcredentials?\b|\.env\b|\benv(?:ironment)?\s+(?:files?|var(?:iable)?s?)\b|\bssh\b|\bpasswords?\b)/i,
- // Credential paths: lib/rules/core/agent-instructions.json.
- ...rulesOf("agent-instruction", { pack: "core" }).map((r) => r.pattern),
- ];
- const packInstructions = () => rulesOf("agent-instruction", { exceptPack: "core" });
- // Text addressed to an automated reviewer. Legitimate skills have no reason to
- // talk to a security scanner.
- const SCANNER_DIRECTED = /\b(?:security |safety |skill |code )?(?:scanners?|reviewers?|auditors?|judges?|classifiers?|analy[sz]ers?|moderators?)\b[^.\n]{0,60}\b(?:is|as|are|mark|classify|report|treat|consider|rate|label)\b[^.\n]{0,30}\b(?:safe|benign|harmless|clean|trusted|approved?|pass(?:ed)?|low[- ]risk|false positive)\b/i;
- export function looksLikeInstruction(s) {
- return INSTRUCTION_PATTERNS.some((re) => re.test(s)) || packInstructions().some((r) => packTest(r, s));
- }
- function detectHiddenInstructions(text) {
- const hits = [];
- const comments = [...text.matchAll(/<!--([\s\S]*?)-->/g)];
- const bad = comments.filter((m) => looksLikeInstruction(m[1]));
- if (bad.length) {
- hits.push({
- rule: "html-comment-instructions",
- severity: "high",
- title: "Instructions hidden in an HTML comment",
- detail: `${bad.length} HTML comment(s) contain instructions. Markdown renderers hide comments, the model still reads them. First: "${quote(bad[0][1])}"`,
- line: lineOf(text, bad[0].index),
- count: bad.length,
- });
- }
- const scanner = text.match(SCANNER_DIRECTED);
- if (scanner) {
- hits.push({
- rule: "scanner-directed",
- severity: "high",
- title: "Text addressed to a security scanner",
- detail: `Tries to talk an automated reviewer into a verdict: "${quote(scanner[0], 80, lineAround(text, scanner.index, scanner[0].length))}". This is how LLM-judged scanners get bypassed.`,
- line: lineOf(text, scanner.index),
- count: 1,
- });
- }
- return hits;
- }
- // ── Code execution patterns ───────────────────────────────────────────────
- // A shell as a pipe's target. Two readers, and a hit from either counts
- // (HEURISTICS R12, bypass checklist row 16): the stage parser below, and
- // main's old regex beside it (legacyShellAfterPipe), so a form the old regex
- // caught can never be lost to the rewrite.
- //
- // The stage parser reads the pipeline stage after the "|" as shell words:
- // - quotes and escapes are read as the shell reads them, so `FOO="a b" sh`
- // is one assignment and one command word, and `ba""sh` is bash;
- // - redirections and their targets (`>`, `>>`, `2>`, `&>`, `<`, `>|`,
- // `2>&1`, attached or spaced) are taken out of the words, so
- // `bash>/var/log/x` has the command word `bash`, not `x`;
- // - the stage ends at the next unquoted |, ;, &, ), backtick or line end
- // (a "\" at the end of a line continues it, as does a line that ends in
- // "|"), and a "(" or "{" that opens a group is passed over;
- // - wrappers that run the next word as a command (sudo, doas, env, nohup,
- // exec, command, builtin, time, nice, setsid, stdbuf, timeout), by name
- // or by any path, are skipped with their options, the values that take
- // one (`env -u NAME`, `sudo -u USER`, `nice -n 5`, `timeout 10`, …), and
- // every word that starts with "-" or has "=" in it;
- // - `env -S STRING` and `--split-string=STRING` run STRING, so its words
- // are read next, from a stack of positions in the strings already read:
- // no word is copied or spliced in, so repeated -S stays linear;
- // - the first word left decides: its name after the last "/" must be a
- // shell (sh, bash, zsh, dash, ksh, fish, csh, tcsh).
- // There is no cap on the words: padding past a cap would hide the shell.
- const SHELL_BASE = /^(?:ba|z|da|k|fi|c|tc)?sh\b/i;
- const wrapper = (short = "", long = [], positional = 0) => ({ short, long: new Set(long), positional });
- const WRAPPERS = {
- env: { ...wrapper("uCSP", ["--unset", "--chdir", "--split-string"]), split: true },
- sudo: wrapper("CDghpRrTtUu", ["--close-from", "--chdir", "--group", "--host", "--prompt", "--chroot", "--role", "--command-timeout", "--type", "--other-user", "--user"]),
- doas: wrapper("Cu"),
- nohup: wrapper(),
- exec: wrapper("a"),
- command: wrapper(),
- builtin: wrapper(),
- time: wrapper("fo", ["--format", "--output"]),
- nice: wrapper("n", ["--adjustment"]),
- setsid: wrapper(),
- stdbuf: wrapper("ioe", ["--input", "--output", "--error"]),
- timeout: wrapper("sk", ["--signal", "--kill-after"], 1),
- };
- const NO_OPTS = wrapper();
- const ASSIGNMENT = /^[A-Za-z_][A-Za-z0-9_]*=/;
- const isStageEnd = (c) => c === "|" || c === ";" || c === "&" || c === ")" || c === "`" || c === "\n";
- const isBlank = (c) => c === " " || c === "\t" || c === "\r" || c === "\f" || c === "\v";
- const isSpace = (c) => isBlank(c) || c === "\n";
- const isRedirect = (text, i) => text[i] === ">" || text[i] === "<" || (text[i] === "&" && text[i + 1] === ">");
- const unquote = (w) => { let a = 0, b = w.length; while (a < b && "'\"\\".includes(w[a])) a++; while (b > a && "'\"\\".includes(w[b - 1])) b--; return w.slice(a, b); };
- /**
- * One shell word from `i`, with quotes and escapes taken out: { lit, i, quoted }.
- * Stops at an unquoted blank, stage end or redirection. A quote with no end
- * runs to the end of the text. Each character is read once.
- */
- function readWord(text, i, prose = false) {
- let lit = "";
- let quoted = false;
- while (i < text.length) {
- const c = text[i];
- // The prose reading: an apostrophe inside a word ("don't") is not a quote. It
- // is dropped, as quote removal would drop it, so `b'a'sh` still reads as bash.
- if (prose && c === "'" && i > 0 && isLetter(text[i - 1]) && isLetter(text[i + 1] || "")) { quoted = true; i++; continue; }
- if (c === "\\") {
- if (text[i + 1] === "\n") { i += 2; continue; }
- if (i + 1 < text.length) lit += text[i + 1];
- quoted = true;
- i += 2;
- continue;
- }
- // A quote ends at its closing quote or at a line break (an unclosed quote
- // in prose, "don't", stops at the end of its line), as in pipeBoundaries.
- if (c === "'") {
- let j = i + 1;
- while (j < text.length && text[j] !== "'" && text[j] !== "\n") j++;
- lit += text.slice(i + 1, j);
- quoted = true;
- i = text[j] === "'" ? j + 1 : j;
- continue;
- }
- if (c === '"') {
- let j = i + 1;
- while (j < text.length && text[j] !== '"' && text[j] !== "\n") {
- if (text[j] === "\\" && j + 1 < text.length) {
- const n = text[j + 1];
- if (n !== "\n") lit += n === '"' || n === "\\" || n === "$" || n === "`" ? n : "\\" + n;
- j += 2;
- continue;
- }
- lit += text[j++];
- }
- quoted = true;
- i = text[j] === '"' ? j + 1 : j;
- continue;
- }
- if (isBlank(c) || isStageEnd(c) || c === "<" || c === ">") break;
- lit += c;
- i++;
- }
- return { lit, i: Math.min(i, text.length), quoted };
- }
- /** Past a redirection operator at `i` and its target word. */
- function skipRedirect(text, i) {
- if (text[i] === "&") i++; // &> and &>>
- const op = text[i++];
- if (text[i] === op) i++; // >> <<
- if (op === "<" && text[i] === "<") i++; // <<<
- if (text[i] === "|" || text[i] === "&" || (op === "<" && (text[i] === "-" || text[i] === ">"))) i++; // >| >& <& <<- <>
- while (i < text.length && isBlank(text[i])) i++;
- if (i < text.length && !isStageEnd(text[i]) && !isRedirect(text, i)) i = readWord(text, i).i;
- return i;
- }
- /** The words of the stage that starts after `from`, as [literal, end] pairs, without redirections. */
- function stageWords(text, from, prose = false) {
- let i = from;
- // Blank space and line breaks right after the pipe: `curl … |` then `sh` on the next line.
- while (i < text.length && (isSpace(text[i]) || (text[i] === "\\" && text[i + 1] === "\n"))) i += text[i] === "\\" ? 2 : 1;
- // Read lazily, one word per call: a stage of millions of short words was
- // held whole as an array (1.7 GB at the 16 MiB hook-script cap, Reviewer).
- return () => {
- for (;;) {
- while (i < text.length && (isBlank(text[i]) || (text[i] === "\\" && text[i + 1] === "\n"))) i += text[i] === "\\" ? 2 : 1;
- if (i >= text.length || (!isRedirect(text, i) && isStageEnd(text[i]))) return null;
- if (isRedirect(text, i)) { i = skipRedirect(text, i); continue; }
- if (text[i] === "(") { i++; continue; } // a subshell: ( sh )
- const w = readWord(text, i, prose);
- i = w.i;
- // `2>file`: a number right before a redirection is its file descriptor.
- if (!w.quoted && /^\d+$/.test(w.lit) && (text[i] === ">" || text[i] === "<")) { i = skipRedirect(text, i); continue; }
- return [w.lit, i];
- }
- };
- }
- /** The stage parser's answer: the end of the shell's word, or -1. */
- function parsedShellAfterPipe(text, pipe, prose = false) {
- // Frames: the stage's words (read one at a time), then the strings env -S runs ({ s, i, lim, end }).
- const stack = [{ next: stageWords(text, pipe + 1, prose) }];
- let opts = NO_OPTS;
- let skipValue = false;
- let skipPositional = 0;
- let splitNext = false;
- while (stack.length) {
- const top = stack[stack.length - 1];
- let s, a, lim, end, f = null;
- if (top.next) {
- const w = top.next();
- if (!w) { stack.pop(); continue; }
- [s, end] = w;
- a = 0;
- lim = s.length;
- if (splitNext && !skipValue) { splitNext = false; stack.push({ s, i: 0, lim, end }); continue; }
- } else {
- while (top.i < top.lim && isSpace(top.s[top.i])) top.i++;
- if (top.i >= top.lim) { stack.pop(); continue; }
- ({ s, lim, end } = top);
- a = top.i;
- f = top;
- splitNext = false; // inside a split string, -S's value is simply the next word
- }
- // A stage word is the whole literal; a word of a split string ends at a space.
- const inWord = (j) => j < lim && (!f || !isSpace(s[j]));
- const wordEnd = (j) => { while (inWord(j)) j++; return j; };
- const consume = (j) => { if (f) f.i = j; };
- // Run a split string from j: the rest of a split-string word, or a new frame over the stage word.
- const split = (j) => { if (f) f.i = j; else stack.push({ s, i: j, lim, end }); };
- if (skipValue) { skipValue = false; consume(wordEnd(a)); continue; }
- if (s[a] === "-") {
- if (s[a + 1] === "-") {
- let j = a;
- while (inWord(j) && s[j] !== "=") j++;
- const name = s.slice(a, j);
- if (inWord(j)) { // --name=value
- if (opts.split && name === "--split-string") split(j + 1);
- else consume(wordEnd(j));
- continue;
- }
- if (opts.long.has(name)) { if (opts.split && name === "--split-string") splitNext = true; else skipValue = true; }
- consume(j);
- continue;
- }
- let c = a + 1;
- for (; inWord(c); c++) {
- if (!opts.short.includes(s[c])) continue;
- // -S runs its value: read it next. Any other value is skipped.
- if (opts.split && s[c] === "S") { if (inWord(c + 1)) split(c + 1); else { splitNext = true; consume(c + 1); } }
- else if (inWord(c + 1)) consume(wordEnd(c + 1));
- else { skipValue = true; consume(c + 1); }
- break;
- }
- if (!inWord(c)) consume(c);
- continue;
- }
- const b = wordEnd(a);
- consume(b);
- const w = f ? unquote(s.slice(a, b)) : s;
- if (!w || w === "{" || w === "!") continue;
- // An assignment (NAME=value, by the shell grammar) is never the command,
- // and is read before any basename: `X=/usr/bin/timeout bash` must not pick
- // the timeout wrapper, whose positional skip would eat `bash` (Vercel R).
- if (ASSIGNMENT.test(w)) continue;
- const base = w.slice(w.lastIndexOf("/") + 1);
- if (Object.hasOwn(WRAPPERS, base)) { opts = WRAPPERS[base]; skipPositional = opts.positional; continue; }
- if (skipPositional > 0) { skipPositional--; continue; }
- if (w.includes("=")) continue;
- return SHELL_BASE.test(base) ? end : -1;
- }
- return -1;
- }
- /**
- * Main's regex for a pipe into a shell, kept beside the parser (R12). Written
- * so its whitespace runs cannot split one run between them: main's
- * `env\s+\S*\s*` was quadratic on `| env` and long blank space. The same
- * strings match. Tried at one "|" (sticky), so the work per pipe is the blank
- * space and the one word after it, and those of different pipes don't overlap.
- */
- // Leading assignments (`| X=1 sh`, `| X=/usr/bin/timeout bash`) are taken
- // too, so this regex alone covers every form main's did, and more. A value
- // stops at a blank, "|", ";" or "&", so the work stays inside one stage.
- const LEGACY_SHELL = /\|\s*(?:[A-Za-z_]\w*=[^\s|;&]*\s+)*(?:sudo\s+)?(?:env\s+(?:\S+\s*)?)?(?:ba|z|da|k|fi|c|tc)?sh\b/iy;
- export function legacyShellAfterPipe(text, pipe) {
- LEGACY_SHELL.lastIndex = pipe;
- const m = LEGACY_SHELL.exec(text);
- return m ? pipe + m[0].length : -1;
- }
- /**
- * The pipeline boundaries of `text`: the "|" characters outside quotes and not
- * escaped, found in one forward pass with one quote and escape state (Vercel
- * S: a parse started at every raw "|", escaped ones too, read on past later
- * pipes, quadratic on a run of `\|`). Quotes end at a line break, as in
- * readWord, so the two agree: a stage parsed from a boundary never reads past
- * the next one, and each character is read by one stage parse.
- */
- function pipeBoundaries(text) {
- const marks = new Uint8Array(text.length);
- boundaryPass(text, marks, false);
- // Prose: an apostrophe inside a word ("it's", "don't") is not a quote. A
- // second pass reads it as a plain character, and a pipe that either pass
- // finds is a boundary, so an apostrophe that pairs with a later real quote
- // can't hide the pipes between them (Reviewer, fifth pass).
- if (/[A-Za-z]'[A-Za-z]/.test(text)) boundaryPass(text, marks, true);
- return marks;
- }
- const isLetter = (c) => (c >= "a" && c <= "z") || (c >= "A" && c <= "Z");
- /**
- * One pass of pipeBoundaries: marks the "|" outside quotes and escapes. Inside
- * double quotes, `$(...)` and backticks are code again (bash runs them), so a
- * pipe there is a boundary too; a stack keeps the quote to return to.
- * `prose`: a letter'letter apostrophe is plain.
- */
- function boundaryPass(text, marks, prose) {
- let q = "";
- let qAt = -1;
- const subs = []; // substitutions opened inside "...": { depth, tick }
- for (let i = 0; ; i++) {
- const c = i < text.length ? text[i] : "\n";
- if (c === "\n") {
- // A quote still open at the line's end never closed: read the line again
- // from just after it, as if it were a plain character, so the pipes
- // after it are boundaries. No quote of that kind follows it on the line,
- // so a line is read at most three times: still linear.
- if (q) { i = qAt; q = ""; qAt = -1; subs.length = 0; continue; }
- subs.length = 0;
- if (i >= text.length) break;
- continue;
- }
- if (q === "'") { if (c === "'") q = ""; continue; }
- if (c === "\\") { i++; continue; }
- if (q === '"') {
- if (c === '"') q = "";
- else if (c === "$" && text[i + 1] === "(") { subs.push({ depth: 0, tick: false }); q = ""; i++; }
- else if (c === "`") { subs.push({ depth: 0, tick: true }); q = ""; }
- continue;
- }
- // subs[-1] on an empty stack is a slow property lookup in V8, per character.
- const top = subs.length ? subs[subs.length - 1] : null;
- if (top) {
- if (top.tick ? c === "`" : c === ")" && top.depth === 0) { subs.pop(); q = '"'; continue; }
- if (!top.tick && c === "(") top.depth++;
- else if (!top.tick && c === ")") top.depth--;
- }
- if (c === "'" && prose && i > 0 && isLetter(text[i - 1]) && isLetter(text[i + 1] || "")) continue;
- if (c === "'" || c === '"') { q = c; qAt = i; }
- else if (c === "|") marks[i] |= prose ? 2 : 1;
- }
- }
- /**
- * A reader for one text: (pipe) -> the end of the shell's word, or -1. The
- * boundaries are found once. The parser runs at a boundary; main's regex runs
- * at every raw "|" (it is linear per pipe), so a quoted or escaped pipe is
- * still read as main read it. Either hit counts.
- */
- export function shellPipeReader(text) {
- let marks = null;
- return (pipe) => {
- marks ||= pipeBoundaries(text);
- // Each boundary is read with the reading that found it: a pipe only the prose
- // pass marks is parsed with the prose reading too, so a word glued by an
- // intra-word apostrophe is not re-read to the line's end per pipe (Vercel, PR 281).
- const end = marks[pipe] & 1 ? parsedShellAfterPipe(text, pipe) : marks[pipe] & 2 ? parsedShellAfterPipe(text, pipe, true) : -1;
- return end !== -1 ? end : legacyShellAfterPipe(text, pipe);
- };
- }
- /** Does the stage after the "|" at `pipe` run a shell? One-off form of shellPipeReader. */
- export function shellAfterPipe(text, pipe) {
- return shellPipeReader(text)(pipe);
- }
- /** The parser alone, for the differential test. */
- export const _parsedShellAfterPipe = parsedShellAfterPipe;
- /** Index of the first "|" whose stage runs a shell, or -1. */
- export function firstShellPipe(text) {
- const at = shellPipeReader(text);
- for (let p = text.indexOf("|"); p !== -1; p = text.indexOf("|", p + 1)) if (at(p) !== -1) return p;
- return -1;
- }
- // The rule's pattern as a regex, kept for the rule list and pack exports.
- // Matching uses shellAfterPipe.
- const SHELL_DOC = String.raw`(?:sudo\s+)?(?:[\w.~-]*/)*(?:env\s+)?(?:ba|z|da|k|fi|c|tc)?sh\b`;
- const INTERPRETER = String.raw`(?:sudo\s+)?(?:python[0-9.]*|node|perl|ruby|php|deno|bun)\b`;
- const PS_DOWNLOAD = String.raw`\b(?:iwr|irm|Invoke-WebRequest|Invoke-RestMethod|DownloadString)\b`;
- // The patterns of the form `<source> ... | <sink>` are matched by the helpers
- // below rather than by `re` itself (kept as the definition). As one regex
- // they restart the scan at every <source> on a line, which is quadratic on a
- // line full of them; the helpers return the same match in linear time.
- function matchAt(text, start, end) {
- const m = [text.slice(start, end)];
- m.index = start;
- return m;
- }
- /**
- * `source[^\n|]*\|\s*sink`: the source sits in the pipeline stage just before
- * the pipe. `sinkFor(text)` gives a reader (pipe) -> the sink's end or -1,
- * made once per text (shellPipeReader finds the boundaries once).
- */
- const regexAtPipe = (sink) => {
- const re = new RegExp(String.raw`\|\s*${sink}`, "iy");
- return (text) => (p) => { re.lastIndex = p; const m = re.exec(text); return m ? p + m[0].length : -1; };
- };
- function pipedFrom(source, sinkFor) {
- const sourceG = new RegExp(source.source, source.flags.replace(/[gy]/g, "") + "g");
- return (text) => {
- // The last newline before the pipe, tracked forward: a backward search
- // per pipe re-read the whole prefix of a line with no newline, which was
- // quadratic on "|sh|sh|sh..." (about 45 minutes at the 5 MB cap). The
- // previous "|" is at most one match back, so lastIndexOf stays linear.
- let lastNl = -1;
- let nextNl = text.indexOf("\n");
- let prevPipe = -1;
- // The source's matches, found once for the whole text and walked with a
- // pointer: matching it in a slice per pipe cost a copy per pipe, which V8
- // can make slow on a large text (83 s for 5 MiB of `| env -S x`). A source
- // match never holds "|" or a line break, so it lies inside one stage, and
- // the first one at or after the stage's start is the stage's leftmost.
- const sources = [...text.matchAll(sourceG)].map((m) => [m.index, m.index + m[0].length]);
- const sinkAt = sinkFor(text);
- let k = 0;
- for (let p = text.indexOf("|"); p !== -1 && k < sources.length; prevPipe = p, p = text.indexOf("|", p + 1)) {
- while (nextNl !== -1 && nextNl < p) { lastNl = nextNl; nextNl = text.indexOf("\n", nextNl + 1); }
- const start = Math.max(prevPipe, lastNl) + 1;
- while (k < sources.length && sources[k][0] < start) k++;
- // The source must be in the stage just before this pipe; only then is the sink read.
- if (k >= sources.length || sources[k][1] > p) continue;
- const end = sinkAt(p);
- if (end !== -1) return matchAt(text, sources[k][0], end);
- }
- return null;
- };
- }
- /**
- * `first[^\n]*second`: leftmost `first`, extended to the last `second` that
- * starts on its line. A `second` that begins with `\|` may run past the line
- * end (`\|\s*sh` spans a newline), so it is tried at each pipe on the line.
- */
- function thenOnLine(first, second) {
- const firstG = new RegExp(first, "gi");
- // `second` is a regex source, or a reader factory (text) -> (pipe) -> end | -1 tried at each pipe.
- const atPipeFn = typeof second === "function" ? second : null;
- const secondG = atPipeFn ? null : new RegExp(second, "gi");
- const atPipe = atPipeFn || second.startsWith("\\|");
- const sinkFor = atPipeFn || (atPipe ? regexAtPipe(second.slice(2)) : null);
- const lastOnLine = (text, from, eol, sinkAt) => {
- let last = null;
- if (atPipe) {
- for (let p = text.indexOf("|", from); p >= 0 && p < eol; p = text.indexOf("|", p + 1)) {
- const end = sinkAt(p);
- if (end !== -1) last = { 0: text.slice(p, end), index: p };
- }
- } else {
- for (const b of text.slice(from, eol).matchAll(secondG)) last = { 0: b[0], index: from + b.index };
- }
- return last;
- };
- return (text) => {
- let skipTo = 0;
- const sinkAt = sinkFor ? sinkFor(text) : null;
- for (const a of text.matchAll(firstG)) {
- if (a.index < skipTo) continue;
- const aEnd = a.index + a[0].length;
- let eol = text.indexOf("\n", aEnd);
- if (eol < 0) eol = text.length;
- // Only the first `first` on a line needs checking: any `second` after a
- // later one is also after this one.
- skipTo = eol;
- const last = lastOnLine(text, aEnd, eol, sinkAt);
- if (last) return matchAt(text, a.index, last.index + last[0].length);
- }
- return null;
- };
- }
- function earliest(...finders) {
- return (text) => {
- let best = null;
- for (const f of finders) {
- const m = f(text);
- if (m && (!best || m.index < best.index)) best = m;
- }
- return best;
- };
- }
- const JS_EVAL_REMOTE = /\b(?:eval|Function)\s*\(\s*(?:await\s+)?(?:atob|Buffer\.from|fetch|require\(\s*["']child_process|\(await fetch)/;
- const evalAsCallback = thenOnLine(String.raw`\bfetch\s*\(`, String.raw`\.then\(\s*(?:(?:globalThis|window|self)\.)?eval\s*\)`);
- export const EXEC_PATTERNS = [
- {
- rule: "curl-pipe-shell", label: "download piped to a shell",
- re: new RegExp(String.raw`\b(?:curl|wget|fetch|iwr)\b[^\n|]*\|\s*${SHELL_DOC}`, "i"),
- match: pipedFrom(/\b(?:curl|wget|fetch|iwr)\b/i, shellPipeReader),
- },
- {
- rule: "curl-pipe-interpreter", label: "download piped to an interpreter",
- re: /\b(?:curl|wget)\b[^\n|]*\|\s*(?:sudo\s+)?(?:python[0-9.]*|node|perl|ruby|php|deno|bun)\b/i,
- match: pipedFrom(/\b(?:curl|wget)\b/i, regexAtPipe(INTERPRETER)),
- },
- // Process and command substitution: lib/rules/core/agent-exec.json.
- ...rulesOf("agent-exec", { pack: "core" }).map((r) => ({ rule: r.id, label: r.label, re: r.pattern })),
- {
- rule: "base64-pipe-shell", label: "decoded base64 piped to a shell",
- re: new RegExp(String.raw`base64\s+(?:-d|--decode|-D)\b[^\n]*\|\s*${SHELL_DOC}`, "i"),
- match: thenOnLine(String.raw`base64\s+(?:-d|--decode|-D)\b`, shellPipeReader),
- },
- { rule: "shell-eval", label: "eval of command output", re: /\beval\s+["']?\$\(/ },
- {
- rule: "js-eval-remote", label: "eval of fetched or decoded content",
- re: JS_EVAL_REMOTE,
- // Also eval passed as the callback of a fetch: fetch(url).then((r) => r.text()).then(eval).
- match: (text) => text.match(JS_EVAL_REMOTE) || evalAsCallback(text),
- },
- { rule: "py-exec-remote", label: "exec of fetched or decoded content", re: /\bexec\s*\(\s*(?:base64\.b64decode|__import__|compile|urllib|requests\.get|zlib\.decompress|marshal\.loads|bytes\.fromhex)/ },
- {
- rule: "powershell-iex", label: "PowerShell Invoke-Expression of a download",
- re: /\b(?:iex|Invoke-Expression)\b[^\n]*\b(?:iwr|irm|Invoke-WebRequest|Invoke-RestMethod|DownloadString)\b|\b(?:iwr|irm|Invoke-WebRequest|Invoke-RestMethod|DownloadString)\b[^\n]*\|\s*(?:iex|Invoke-Expression)\b/i,
- match: earliest(
- thenOnLine(String.raw`\b(?:iex|Invoke-Expression)\b`, PS_DOWNLOAD),
- thenOnLine(PS_DOWNLOAD, String.raw`\|\s*(?:iex|Invoke-Expression)\b`),
- ),
- },
- ];
- /** Find remote-code-execution patterns in a string (a file, a hook command). */
- export function findExecPatterns(text) {
- const out = [];
- for (const p of execPatterns()) {
- const m = p.pack ? packMatch(p.pack, text) : p.match ? p.match(text) : text.match(p.re);
- if (m) out.push({ rule: p.rule, label: p.label, match: m[0], index: m.index });
- }
- return out;
- }
- // Built-ins, then rules from other packs (bundled or --rules).
- let _exec = null;
- let _execGen = -1;
- function execPatterns() {
- if (_execGen !== rulesGeneration()) {
- _exec = [...EXEC_PATTERNS, ...rulesOf("agent-exec", { exceptPack: "core" }).map((r) => ({ rule: r.id, label: r.label, re: r.pattern, pack: r }))];
- _execGen = rulesGeneration();
- }
- return _exec;
- }
- function detectExec(text) {
- return findExecPatterns(text).map((m) => ({
- rule: m.rule,
- severity: "high",
- title: `Remote code execution pattern: ${m.label}`,
- detail: `\`${quote(m.match, 120, lineAround(text, m.index, m.match.length))}\`. An agent following this runs code nobody reviewed, fetched at run time.`,
- line: lineOf(text, m.index),
- count: 1,
- }));
- }
- // ── Base64 blobs ──────────────────────────────────────────────────────────
- const MAGIC = [
- { name: "ELF executable", test: (b) => b[0] === 0x7f && b[1] === 0x45 && b[2] === 0x4c && b[3] === 0x46 },
- { name: "Windows executable", test: (b) => b[0] === 0x4d && b[1] === 0x5a },
- { name: "Mach-O executable", test: (b) => {
- const m = b.readUInt32BE?.(0);
- return [0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe, 0xcafebabe].includes(m);
- } },
- { name: "zip archive", test: (b) => b[0] === 0x50 && b[1] === 0x4b && (b[2] === 0x03 || b[2] === 0x05 || b[2] === 0x07) },
- { name: "gzip archive", test: (b) => b[0] === 0x1f && b[1] === 0x8b },
- { name: "bzip2 archive", test: (b) => b[0] === 0x42 && b[1] === 0x5a && b[2] === 0x68 },
- { name: "xz archive", test: (b) => b[0] === 0xfd && b[1] === 0x37 && b[2] === 0x7a && b[3] === 0x58 },
- { name: "7z archive", test: (b) => b[0] === 0x37 && b[1] === 0x7a && b[2] === 0xbc && b[3] === 0xaf },
- { name: "rar archive", test: (b) => b[0] === 0x52 && b[1] === 0x61 && b[2] === 0x72 && b[3] === 0x21 },
- { name: "WebAssembly module", test: (b) => b[0] === 0x00 && b[1] === 0x61 && b[2] === 0x73 && b[3] === 0x6d },
- // CPython bytecode: a 2-byte version magic followed by \r\n.
- { name: "Python bytecode (.pyc)", test: (b) => b.length >= 16 && b[2] === 0x0d && b[3] === 0x0a && b[1] < 0x10 },
- { name: "tar archive", test: (b) => b.length > 262 && b.toString("latin1", 257, 262) === "ustar" },
- ];
- /** Name the executable/archive format of a buffer, or null. */
- export function sniffBinary(buf) {
- if (!buf || buf.length < 4) return null;
- const hit = MAGIC.find((m) => m.test(buf));
- return hit ? hit.name : null;
- }
- function isPrintable(buf) {
- if (!buf.length) return false;
- let ok = 0;
- for (const b of buf) if (b === 9 || b === 10 || b === 13 || (b >= 32 && b < 127)) ok++;
- return ok / buf.length > 0.95;
- }
- function detectBase64(text, opts) {
- const re = new RegExp(`[A-Za-z0-9+/_-]{${opts.base64MinLength},}={0,2}`, "g");
- const blobs = [];
- for (const m of text.matchAll(re)) {
- const s = m[0];
- // Real base64 mixes cases and digits; hex digests, identifiers and long
- // words do not.
- if (!/[A-Z]/.test(s) || !/[a-z]/.test(s) || !/[0-9]/.test(s)) continue;
- if (/^[0-9a-f]+$/i.test(s)) continue;
- const before = text.slice(Math.max(0, m.index - 40), m.index);
- const image = /data:image\/[a-z+.-]+;base64,$/i.test(before);
- const buf = Buffer.from(s.replace(/-/g, "+").replace(/_/g, "/"), "base64");
- const binary = sniffBinary(buf);
- let decodedText = null;
- if (!binary && isPrintable(buf)) decodedText = buf.toString("utf-8");
- const execInside = decodedText ? findExecPatterns(decodedText) : [];
- const instructionInside = decodedText ? looksLikeInstruction(decodedText) : false;
- blobs.push({ index: m.index, length: s.length, image, binary, decodedText, execInside, instructionInside });
- }
- if (!blobs.length) return [];
- const worst = blobs.find((b) => b.binary) || blobs.find((b) => b.execInside.length) ||
- blobs.find((b) => b.instructionInside) || blobs.find((b) => !b.image) || blobs[0];
- let severity = "medium";
- let what = "an opaque encoded payload a reviewer cannot read";
- if (worst.binary) { severity = "high"; what = `a ${worst.binary}`; }
- else if (worst.execInside.length) { severity = "high"; what = `a ${worst.execInside[0].label} (\`${quote(worst.execInside[0].match, 80, worst.decodedText)}\`)`; }
- else if (worst.instructionInside) { severity = "high"; what = `hidden instructions ("${quote(worst.decodedText)}")`; }
- else if (worst.image) { severity = "low"; what = "an inline image"; }
- else if (worst.decodedText) what = `encoded text ("${quote(worst.decodedText, 60)}")`;
- return [{
- rule: "base64-blob",
- severity,
- title: "Large base64 blob",
- detail: `${blobs.length} base64 run(s) of ${opts.base64MinLength}+ characters; the largest is ${Math.max(...blobs.map((b) => b.length))}. It decodes to ${what}.`,
- line: lineOf(text, worst.index),
- count: blobs.length,
- }];
- }
- // ── Public API ────────────────────────────────────────────────────────────
- /**
- * Scan one text file for hidden content.
- * @param {string} text
- * @param {object} [opts] { blankLines, horizontalRun, base64MinLength, markdown }
- */
- export function scanHiddenContent(text, opts = {}) {
- const o = { ...DEFAULTS, ...opts };
- return [
- ...detectInvisible(text),
- ...detectWhitespaceInflation(text, o),
- ...detectHiddenInstructions(text),
- ...detectExec(text),
- ...detectBase64(text, o),
- ];
- }
- export function hasTagChars(text) {
- return TAG_RANGE.test(text);
- }
- // Extensions that are binary or archives no matter what the bytes say.
- export const BINARY_EXTENSIONS = /\.(?:pyc|pyo|pyd|so|dylib|dll|exe|bin|jar|class|whl|egg|zip|tar|tgz|gz|bz2|xz|7z|rar|wasm|node|deb|rpm|dmg|pkg|msi|apk)$/i;
- // Media that ships in skills legitimately.
- export const MEDIA_EXTENSIONS = /\.(?:png|jpe?g|gif|webp|ico|bmp|avif|woff2?|ttf|otf|eot|mp3|mp4|wav|ogg|webm|mov|pdf)$/i;
lib/fix/apply.js · Bundled script · 386 lines · 1 finding marked
- /**
- * applyDiff: the one place fix recipes' diffs get written to disk.
- *
- * Planning never writes. Applying is a separate, explicit step that refuses to
- * write outside the repo, into any .git (any case, any depth, any NTFS alias),
- * to a protected file (tests, test config, Penthera's own files, CI, hooks,
- * lockfiles, .gitmodules: see PROTECTED_FILES), through a symlink anywhere in the path, or into a dirty git
- * tree (unless the caller passes allowDirty). It refuses added lines carrying
- * control characters a terminal would act on, and it is all-or-nothing: every
- * hunk must apply, and if a write fails the files already written are put back.
- *
- * Each file is written to a temp file in its directory and moved into place, so
- * a hard link is replaced rather than written through, and an edit is refused
- * if the file changed since it was read.
- */
- import { createHash, randomBytes } from "node:crypto";
- import {
- existsSync, readFileSync, mkdirSync, realpathSync, lstatSync, fstatSync, openSync, writeSync, closeSync,
- renameSync, linkSync, unlinkSync, rmdirSync, chmodSync, constants,
- } from "node:fs";
- import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
- import { createGit, isOwnReport, parseStatusZ } from "../../src/agent/git.js";
- import { parsePatch, applyHunks } from "./diff.js";
- import { trimEndChars } from "../linear-text.js";
- export class FixError extends Error {
- constructor(message, code) {
- super(message);
- this.name = "FixError";
- this.code = code;
- }
- }
- function inside(root, p) {
- const rel = relative(root, p);
- return rel === "" || (!rel.startsWith(`..${sep}`) && rel !== ".." && !isAbsolute(rel));
- }
- export const sha256 = (buf) => createHash("sha256").update(buf).digest("hex");
- // Code points HFS+ ignores in names, so ".git" is ".git" on macOS (CVE-2014-9390).
- const IGNORABLE = /[----]/g;
- Note: Usual in code (characters a sanitizer strips): Contains 2 bidirectional-override character(s), which make text display in a different order from how it's read
- Line 42 on GitHub
- /**
- * The name a path segment resolves to on case-insensitive file systems
- * (macOS, Windows): case folded, ignorable code points dropped, and the
- * trailing dots and spaces Windows strips removed.
- */
- export function canonicalSegment(seg) {
- return trimEndChars(String(seg).normalize("NFC").replace(IGNORABLE, ""), ". ").toLowerCase();
- }
- /** True for a segment that names the git directory: .git, .GIT, .Git., or the 8.3 alias GIT~1. */
- export function isGitDirSegment(seg) {
- const c = canonicalSegment(seg);
- return c === ".git" || /^git~\d+$/.test(c);
- }
- const LOCKFILES = new Set([
- "package-lock.json", "npm-shrinkwrap.json", "yarn.lock", "pnpm-lock.yaml", "bun.lock", "bun.lockb", "deno.lock",
- "poetry.lock", "pipfile.lock", "uv.lock", "pdm.lock", "cargo.lock", "go.sum", "gemfile.lock", "composer.lock",
- ]);
- const TEST_CONFIG = new Set(["package.json", "pytest.ini", "tox.ini", "setup.cfg", "pyproject.toml"]);
- /**
- * Paths a fix diff may never touch even though they live in the work tree: the
- * judge that verifies a fix (tests, test runner config, Penthera's own policy,
- * suppressions, baseline and memory), CI, git hooks and lockfiles. A fix may not
- * pass by editing its own checker. Each entry is [what, test(segments, basename)]
- * on canonical segments, so case and Windows aliases do not slip past.
- */
- const PROTECTED_FILES = [
- ["Windows 8.3 short names (they can alias a protected file)", (s) => s.some((d) => /~\d+(\.[^.]*)?$/.test(d))],
- [".gitmodules (it can point git at other repositories)", (s, b) => b === ".gitmodules"],
- ["tests", (s, b) => s.some((d) => d === "test" || d === "tests" || d === "__tests__") || /\.(test|spec)\./.test(b) || b === "conftest.py"],
- ["test runner config or the test script", (s, b) => /^(vitest|jest|playwright)\./.test(b) || TEST_CONFIG.has(b)],
- ["Penthera's own policy, suppressions, baseline, lock or memory", (s, b) => b.startsWith("penthera.") || s.includes(".penthera") || s.join("/").endsWith("audit/baseline.json")],
- ["CI workflows", (s) => s.some((d, i) => d === ".github" && s[i + 1] === "workflows")],
- ["git hooks", (s, b) => s.includes(".husky") || s.includes(".githooks") || /^(lefthook\.ya?ml|\.pre-commit-config\.ya?ml)$/.test(b)],
- ["lockfiles", (s, b) => LOCKFILES.has(b)],
- ];
- /** What protected kind of file `file` (repo-relative) is, or null if a fix may edit it. */
- export function protectedReason(file) {
- const segs = String(file).split(/[\\/]/).filter((x) => x && x !== ".").map(canonicalSegment);
- const hit = PROTECTED_FILES.find(([, test]) => test(segs, segs[segs.length - 1] || ""));
- return hit ? hit[0] : null;
- }
- // C0 controls (tab aside), DEL and C1 controls: a terminal acts on them instead of showing them.
- const CONTROL = /[\u0000-\u0008\u000a-\u001f\u007f-\u009f]/;
- /**
- * Throws unless no component of `target` below `root` is a symlink, and no
- * component above the file itself is something other than a directory. Every
- * existing component is lstat-ed, so neither a symlinked directory nor a
- * dangling symlink at the file itself can redirect a write.
- */
- function assertNoSymlinks(root, target, file) {
- let cur = root;
- const segs = relative(root, target).split(sep);
- for (let i = 0; i < segs.length; i++) {
- cur = join(cur, segs[i]);
- let st;
- try {
- st = lstatSync(cur);
- } catch (e) {
- if (e.code === "ENOENT") return;
- throw e;
- }
- if (st.isSymbolicLink()) {
- throw new FixError(`Refusing to write ${file}: ${relative(root, cur)} is a symlink, and fix diffs never write through symlinks`, "SYMLINK");
- }
- if (i < segs.length - 1 && !st.isDirectory()) {
- throw new FixError(`Refusing to write ${file}: ${relative(root, cur)} is not a directory`, "CONFLICT");
- }
- }
- }
- /** Resolve a diff path to an absolute path inside the repo, or throw. */
- function safeTarget(root, file) {
- if (!file || isAbsolute(file) || CONTROL.test(file) || file.includes("\0") || /^[a-zA-Z]:/.test(file)) {
- throw new FixError(`Refusing to write ${JSON.stringify(file)}: paths in a diff must be relative to the repo`, "OUTSIDE_REPO");
- }
- const segs = file.split(/[\\/]/);
- if (segs.includes("..")) throw new FixError(`Refusing to write ${file}: it is outside the repo`, "OUTSIDE_REPO");
- // NTFS alternate data streams (".git::$INDEX_ALLOCATION") name the same directory or file.
- if (segs.some((s) => s.includes(":"))) throw new FixError(`Refusing to write ${JSON.stringify(file)}: path segments may not contain ":"`, "OUTSIDE_REPO");
- // Any .git at any depth, in any case: on macOS .GIT/hooks/pre-commit is .git/hooks/pre-commit.
- if (segs.some(isGitDirSegment)) throw new FixError(`Refusing to write ${file}: it is inside .git`, "OUTSIDE_REPO");
- const why = protectedReason(file);
- if (why) {
- throw new FixError(`Refusing to write ${file}: fix diffs never edit ${why}, so the checks that verify a fix stay as they were. If the real fix needs this file changed, make that change by hand and review it.`, "PROTECTED");
- }
- const target = resolve(root, file);
- if (!inside(root, target) || target === root) throw new FixError(`Refusing to write ${file}: it is outside the repo`, "OUTSIDE_REPO");
- assertNoSymlinks(root, target, file);
- return target;
- }
- /**
- * Throws unless repoPath is in a git work tree with no uncommitted changes
- * under it, Penthera's own untracked reports aside (isOwnReport).
- * @returns {string[]} those reports, relative to repoPath
- */
- export function assertCleanTree(repoPath) {
- let entries;
- try {
- // The hardened runner: no fsmonitor, no hooks, no filters, no transports (src/agent/git.js).
- const git = createGit(repoPath);
- git.run("rev-parse", "--is-inside-work-tree");
- const prefix = git.run("rev-parse", "--show-prefix").replace(/\n$/, "");
- entries = parseStatusZ(git.run("status", "--porcelain=v1", "-z", "--untracked-files=all", "--", "."), prefix);
- } catch {
- throw new FixError(`${repoPath} is not a git work tree, so a fix could not be reviewed or reverted. Commit it to git first, or pass --allow-dirty.`, "NOT_GIT");
- }
- if (entries.some((e) => !isOwnReport(e))) {
- throw new FixError(`${repoPath} has uncommitted changes. Commit or stash them first so the fix is its own reviewable change, or pass --allow-dirty.`, "DIRTY");
- }
- return entries.map((e) => e.path);
- }
- const NOFOLLOW = constants.O_NOFOLLOW || 0;
- /** Current bytes of a regular file, null if it does not exist; throws on anything else. */
- function readCurrent(target, file) {
- let st;
- try {
- st = lstatSync(target);
- } catch (e) {
- if (e.code === "ENOENT") return null;
- throw e;
- }
- if (!st.isFile()) throw new FixError(`Refusing to write ${file}: it is not a regular file`, "CONFLICT");
- const fd = openSync(target, constants.O_RDONLY | NOFOLLOW);
- try {
- return readFileSync(fd);
- } finally {
- closeSync(fd);
- }
- }
- function assertDirInside(root, dir, file) {
- const real = realpathSync(dir);
- if (!inside(root, real)) throw new FixError(`Refusing to write ${file}: its directory resolves outside the repo`, "OUTSIDE_REPO");
- }
- /**
- * Put `data` at `target`, but only if the file currently holds `expect`
- * (a Buffer), or does not exist (expect === null). The bytes go to a temp file
- * in the same directory first, then are moved into place: rename for an edit
- * (which replaces a hard link instead of writing through it), link for a new
- * file (which fails if something appeared there meanwhile).
- *
- * @returns {{ madeDir: string|undefined }} the first directory mkdir created, if any
- */
- function replaceFile(root, file, target, data, expect) {
- assertNoSymlinks(root, target, file);
- const dir = dirname(target);
- const madeDir = mkdirSync(dir, { recursive: true });
- assertNoSymlinks(root, target, file);
- assertDirInside(root, dir, file);
- const current = readCurrent(target, file);
- const unchanged = expect === null ? current === null : current !== null && current.equals(expect);
- if (!unchanged) throw new FixError(`${file} changed on disk since the fix was planned; nothing was written to it`, "CHANGED");
- const mode = current === null ? 0o644 : lstatSync(target).mode & 0o7777;
- const tmp = join(dir, `.${basename(target)}.penthera-${randomBytes(6).toString("hex")}.tmp`);
- const fd = openSync(tmp, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | NOFOLLOW, 0o600);
- try {
- let off = 0;
- while (off < data.length) off += writeSync(fd, data, off, data.length - off);
- // The temp file we wrote must be the one at that path, in a directory inside the repo.
- const a = fstatSync(fd);
- const b = lstatSync(tmp);
- if (a.dev !== b.dev || a.ino !== b.ino) throw new FixError(`Refusing to write ${file}: its directory changed while writing`, "RACE");
- } catch (e) {
- closeSync(fd);
- try { unlinkSync(tmp); } catch { /* best effort */ }
- throw e;
- }
- closeSync(fd);
- try {
- chmodSync(tmp, mode);
- assertDirInside(root, dir, file);
- if (expect === null) {
- try {
- linkSync(tmp, target);
- } catch (e) {
- if (e.code === "EEXIST") throw new FixError(`${file} appeared on disk while the fix was being written; nothing was written to it`, "CHANGED");
- // File systems without hard links: exclusive create, never through a symlink.
- const out = openSync(target, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | NOFOLLOW, mode);
- try {
- let off = 0;
- while (off < data.length) off += writeSync(out, data, off, data.length - off);
- } finally {
- closeSync(out);
- }
- }
- } else {
- const again = readCurrent(target, file);
- if (!again || !again.equals(expect)) throw new FixError(`${file} changed on disk while the fix was being written; nothing was written to it`, "CHANGED");
- renameSync(tmp, target);
- }
- } finally {
- try { unlinkSync(tmp); } catch { /* already renamed */ }
- }
- return { madeDir };
- }
- /** Remove the directories a write created (deepest first), as long as they are empty. */
- function removeMadeDirs(target, madeDir) {
- if (!madeDir) return;
- for (let d = dirname(target); d.length >= madeDir.length; d = dirname(d)) {
- try { rmdirSync(d); } catch { return; }
- if (d === madeDir) return;
- }
- }
- /**
- * Put files written by applyDiff back the way they were, one by one, never
- * over somebody else's edit: a file is restored (or, if the fix created it,
- * removed) only while it still holds exactly what applyDiff wrote. A file that
- * changed since is left alone and reported as a conflict. Every file is tried
- * even if an earlier one fails.
- *
- * @param {string} repoPath
- * @param {{ path: string, created: boolean, before: Buffer|null, sha256: string }[]} files - applyDiff's result
- * @returns {{ restored: string[], conflicts: string[], errors: { path: string, message: string }[] }}
- */
- export function restoreFiles(repoPath, files) {
- const root = realpathSync(resolve(repoPath));
- const out = { restored: [], conflicts: [], errors: [] };
- for (const f of files) {
- try {
- const target = safeTarget(root, f.path);
- const current = readCurrent(target, f.path);
- const original = f.created ? null : f.before;
- if (current === null ? original === null : original !== null && current.equals(original)) {
- out.restored.push(f.path); // already as it was
- } else if (current === null || sha256(current) !== f.sha256) {
- out.conflicts.push(f.path);
- } else if (original === null) {
- unlinkSync(target);
- out.restored.push(f.path);
- } else {
- replaceFile(root, f.path, target, original, current);
- out.restored.push(f.path);
- }
- } catch (e) {
- if (e.code === "CHANGED") out.conflicts.push(f.path);
- else out.errors.push({ path: f.path, message: e.message });
- }
- }
- return out;
- }
- /** Throws if an added line carries a control character (a trailing CR of a CRLF line is fine). */
- function assertPrintable(f) {
- for (const h of f.hunks) {
- for (const op of h.ops) {
- if (op.t !== "+") continue;
- const line = op.line.replace(/\u0000$/, "").replace(/\r$/, "");
- if (CONTROL.test(line)) {
- throw new FixError(`Refusing to write ${f.newPath}: an added line contains a control character (${JSON.stringify(line.match(CONTROL)[0])}), which can hide code in a terminal`, "CONTROL_CHARS");
- }
- }
- }
- }
- /** Throws on two entries for one file, or one file that is another's directory. */
- function assertDistinctPaths(parsed) {
- const keys = parsed.map((f) => String(f.newPath).split(/[\\/]/).map(canonicalSegment).join("/"));
- for (let i = 0; i < keys.length; i++) {
- for (let j = 0; j < keys.length; j++) {
- if (i === j) continue;
- if (keys[i] === keys[j]) throw new FixError(`Refusing the diff: it changes ${parsed[i].newPath} more than once`, "DUPLICATE");
- if (keys[j].startsWith(`${keys[i]}/`)) throw new FixError(`Refusing the diff: ${parsed[i].newPath} is both a file and the directory of ${parsed[j].newPath}`, "CONFLICT");
- }
- }
- }
- /**
- * Apply a unified diff to files under repoPath.
- *
- * @param {string} repoPath
- * @param {string} diff
- * @param {{ dryRun?: boolean, allowDirty?: boolean }} [opts]
- * @returns {{ applied: boolean, files: { path: string, created: boolean, before: Buffer|null, sha256: string }[] }}
- * before is the file's bytes before the write; sha256 is the hash of what was written.
- */
- export function applyDiff(repoPath, diff, opts = {}) {
- const root = realpathSync(resolve(repoPath));
- const parsed = parsePatch(diff);
- if (parsed.length === 0) throw new FixError("The diff contains no file changes", "EMPTY");
- const ownReports = opts.allowDirty ? [] : assertCleanTree(root);
- const writes = [];
- for (const f of parsed) {
- if (ownReports.includes(f.newPath)) throw new FixError(`Refusing to write ${f.newPath}: it is an uncommitted scan report`, "DIRTY");
- if (!f.newPath) throw new FixError(`Refusing to delete ${f.oldPath}: fix diffs never delete files`, "DELETE");
- if (f.oldPath && f.oldPath !== f.newPath) throw new FixError(`Refusing to rename ${f.oldPath}: fix diffs never rename files`, "RENAME");
- assertPrintable(f);
- }
- assertDistinctPaths(parsed);
- for (const f of parsed) {
- const target = safeTarget(root, f.newPath);
- const exists = existsSync(target);
- if (exists && !lstatSync(target).isFile()) throw new FixError(`Refusing to write ${f.newPath}: it is not a regular file`, "CONFLICT");
- if (!f.oldPath && exists) throw new FixError(`${f.newPath} already exists; the plan expected to create it`, "CONFLICT");
- if (f.oldPath && !exists) throw new FixError(`${f.newPath} does not exist; the plan expected to edit it`, "CONFLICT");
- const before = exists ? readCurrent(target, f.newPath) : null;
- let after;
- try {
- after = Buffer.from(applyHunks(before === null ? null : before.toString("utf-8"), f.hunks), "utf-8");
- } catch (e) {
- throw new FixError(`${f.newPath}: ${e.message}`, "CONFLICT");
- }
- writes.push({ path: f.newPath, target, before, after, created: !exists });
- }
- if (!opts.dryRun) {
- const done = [];
- try {
- for (const w of writes) {
- const { madeDir } = replaceFile(root, w.path, w.target, w.after, w.before);
- done.push({ ...w, madeDir });
- }
- } catch (e) {
- // All or nothing: put back what was written, newest first.
- for (const w of done.reverse()) {
- try {
- if (w.created) unlinkSync(w.target);
- else replaceFile(root, w.path, w.target, w.before, w.after);
- removeMadeDirs(w.target, w.madeDir);
- } catch { /* reported through the original error */ }
- }
- throw e;
- }
- }
- return {
- applied: !opts.dryRun,
- files: writes.map((w) => ({ path: w.path, created: w.created, before: w.before, sha256: sha256(w.after) })),
- };
- }
lib/rules/pack.js · Bundled script · 470 lines · 2 findings marked
- /**
- * Rule-pack verification: how new detection knowledge reaches Penthera without a runtime
- * fetch or an update channel. See docs/rule-packs.md for the trust model.
- *
- * A pack is a directory:
- * pack.json name, version, created_at, min_penthera, rules[] index
- * ({ file, kind, sha256 } per rule file)
- * pack.json.sig `ssh-keygen -Y sign -n penthera-rules` over pack.json
- * *.json rule files, one kind each. Data only: regexes, string
- * lists, IOC lists. Never code.
- *
- * Loading verifies, in order: the signature over pack.json against
- * allowed_signers, every rule file's SHA-256 against pack.json, the schema of
- * every rule, and every regex against the ReDoS guard. Anything that fails is
- * refused whole; there is no partial load.
- *
- * Packs arrive two ways only: bundled in lib/rules/ (they ship inside the npm
- * release, staged and approved with 2FA), or by path with `--rules <dir>`.
- * Nothing in this module touches the network.
- */
- import { readFileSync, readdirSync, lstatSync, existsSync } from "node:fs";
- import { join, dirname, resolve } from "node:path";
- import { fileURLToPath } from "node:url";
- import { createHash } from "node:crypto";
- import { checkRegex } from "./regex-guard.js";
- import { PRODUCTS, PRODUCT_IDS } from "../advisories/products.js";
- import { parseVersion, compareVersions } from "../advisories/versions.js";
- import { parseAllowedSigners, verifyWithSigners } from "./sshsig.js";
- export const NAMESPACE = "penthera-rules";
- export const RULES_DIR = dirname(fileURLToPath(import.meta.url));
- export const BUILTIN_SIGNERS = join(RULES_DIR, "allowed_signers");
- const PKG_VERSION = JSON.parse(readFileSync(join(RULES_DIR, "..", "..", "package.json"), "utf-8")).version;
- const MAX_MANIFEST_BYTES = 1024 * 1024;
- const MAX_RULE_FILE_BYTES = 4 * 1024 * 1024;
- const MAX_RULES_PER_FILE = 5000;
- // Pack regexes run on at most this much input per call. Bundled packs are
- // part of the release and keep the engines' own behaviour (no cap).
- export const PACK_INPUT_CAP = 1024 * 1024;
- // ── Schema ────────────────────────────────────────────────────────────────
- const ID = /^[a-z0-9][a-z0-9._-]{0,63}$/;
- const PACK_NAME = /^[a-z0-9][a-z0-9-]{0,63}$/;
- const SEMVER = /^\d+\.\d+\.\d+$/;
- const DATE = /^\d{4}-\d{2}-\d{2}$/;
- const FILE_NAME = /^[a-z0-9][a-z0-9._-]{0,63}\.json$/;
- const SHA256 = /^[a-f0-9]{64}$/;
- const REF_KEY = /^[A-Za-z0-9_:.-]{1,64}$/;
- // Control characters, DEL, C1 and bidi overrides: a pack's strings reach the
- // terminal, Markdown and SARIF, so none of them may carry escapes.
- const UNSAFE_TEXT = /[\u0000-\u001f\u007f-\u009f--]/;
- Note: Usual in code (characters a sanitizer strips): Contains 4 bidirectional-override character(s), which make text display in a different order from how it's read
- Line 53 on GitHub
- const CONFIDENCE = new Set(["confirmed", "likely", "potential"]);
- const SEVERITY = new Set(["critical", "high", "medium", "low", "info"]);
- const NPM_NAME = /^(?:@[a-z0-9][a-z0-9._~-]*\/)?[a-z0-9][a-z0-9._~-]*$/;
- const VERSION_OR_STAR = /^(?:\*|\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?)$/;
- // PyPI: names normalized as in PEP 503, versions as PEP 440 public versions.
- const PYPI_NAME = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
- const PEP440_OR_STAR = /^(?:\*|\d+(?:\.\d+)*(?:(?:a|b|rc)\d+)?(?:\.post\d+)?(?:\.dev\d+)?)$/;
- const HOSTNAME = /^(?=.{1,253}$)(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;
- const REL_PATH = /^(?!\/)(?!.*(?:^|\/)\.\.(?:\/|$))[A-Za-z0-9._@/-]{1,200}$/;
- const HTTP_METHODS = new Set(["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"]);
- const CVE_ID = /^CVE-\d{4}-\d{4,7}$/;
- const GHSA_ID = /^GHSA(?:-[23456789cfghjmpqrvwx]{4}){3}$/;
- const CVSS_VERSIONS = new Set(["3.0", "3.1", "4.0"]);
- class PackError extends Error {}
- const fail = (msg) => { throw new PackError(msg); };
- function text(v, where, { max = 300, optional = false } = {}) {
- if (v === undefined && optional) return undefined;
- if (typeof v !== "string" || v.length === 0) fail(`${where}: expected a non-empty string`);
- if (v.length > max) fail(`${where}: longer than ${max} characters`);
- if (UNSAFE_TEXT.test(v)) fail(`${where}: contains control or bidi characters`);
- return v;
- }
- function list(v, where, { max = 5000, item }) {
- if (!Array.isArray(v) || v.length === 0) fail(`${where}: expected a non-empty array`);
- if (v.length > max) fail(`${where}: more than ${max} entries`);
- return v.map((x, i) => item(x, `${where}[${i}]`));
- }
- function regex(pattern, flags, where) {
- text(pattern, `${where}.pattern`, { max: 512 });
- if (flags !== undefined && typeof flags !== "string") fail(`${where}.flags: expected a string`);
- const r = checkRegex(pattern, flags || "");
- if (!r.ok) fail(`${where}.pattern rejected by the ReDoS guard: ${r.reason}`);
- return new RegExp(pattern, flags || "");
- }
- /** Fields every rule carries: where the knowledge came from, and how sure it is. */
- function provenance(r, where) {
- text(r.id, `${where}.id`);
- if (!ID.test(r.id)) fail(`${where}.id: must match ${ID}`);
- text(r.source, `${where}.source`, { max: 500 });
- let url;
- try { url = new URL(r.source); } catch { fail(`${where}.source: not a URL`); }
- if (url.protocol !== "https:") fail(`${where}.source: must be an https URL`);
- if (typeof r.date !== "string" || !DATE.test(r.date) || Number.isNaN(Date.parse(r.date))) fail(`${where}.date: expected YYYY-MM-DD`);
- if (!CONFIDENCE.has(r.confidence)) fail(`${where}.confidence: one of ${[...CONFIDENCE].join(", ")}`);
- const references = r.references === undefined ? [] : list(r.references, `${where}.references`, {
- max: 20,
- item: (k, w) => { if (typeof k !== "string" || !REF_KEY.test(k)) fail(`${w}: not a references.bib key`); return k; },
- });
- return {
- id: r.id,
- source: r.source,
- date: r.date,
- confidence: r.confidence,
- references,
- description: text(r.description, `${where}.description`, { max: 1000, optional: true }),
- };
- }
- function severity(v, where, dflt) {
- if (v === undefined && dflt) return dflt;
- if (!SEVERITY.has(v)) fail(`${where}: one of ${[...SEVERITY].join(", ")}`);
- return v;
- }
- /** Nuclei-compatible template, the subset lib/templates.js executes. */
- function template(t, where) {
- if (!t || typeof t !== "object" || Array.isArray(t)) fail(`${where}: expected an object`);
- text(t.id, `${where}.id`, { max: 100 });
- const info = t.info || fail(`${where}.info: required`);
- const out = {
- id: t.id,
- info: {
- name: text(info.name, `${where}.info.name`),
- severity: severity(info.severity, `${where}.info.severity`),
- tags: info.tags === undefined ? [] : list(info.tags, `${where}.info.tags`, { max: 20, item: (x, w) => text(x, w, { max: 40 }) }),
- ...(info.description !== undefined ? { description: text(info.description, `${where}.info.description`, { max: 1000 }) } : {}),
- },
- http: list(t.http, `${where}.http`, {
- max: 10,
- item: (req, w) => {
- const method = req.method === undefined ? "GET" : req.method;
- if (!HTTP_METHODS.has(method)) fail(`${w}.method: not an allowed HTTP method`);
- text(req.path, `${w}.path`, { max: 500 });
- // Only paths on the scan target: a pack can never point the scanner at another host.
- const p = req.path.replace(/^\{\{BaseURL\}\}/, "");
- if (!p.startsWith("/") || p.startsWith("//") || /[\s\\]/.test(p)) fail(`${w}.path: must be a path on the target ("/x" or "{{BaseURL}}/x")`);
- const matchers = list(req.matchers, `${w}.matchers`, {
- max: 10,
- item: (m, mw) => {
- const type = m.type;
- if (type === "status") return { type, status: list(m.status, `${mw}.status`, { max: 20, item: (s, sw) => { if (!Number.isInteger(s) || s < 100 || s > 599) fail(`${sw}: HTTP status`); return s; } }) };
- if (type === "word" || type === "negative-word") {
- return { type, words: list(m.words, `${mw}.words`, { max: 50, item: (x, xw) => text(x, xw, { max: 200 }) }), ...(m.condition ? { condition: m.condition === "and" ? "and" : "or" } : {}), ...(m.part === "header" ? { part: "header" } : {}) };
- }
- if (type === "regex" || type === "negative-regex") {
- const pats = list(m.regex, `${mw}.regex`, { max: 10, item: (x, xw) => { regex(x, m.flags, xw); return x; } });
- return { type, regex: pats, ...(m.flags ? { flags: m.flags } : {}), ...(m.condition ? { condition: m.condition === "and" ? "and" : "or" } : {}), ...(m.part === "header" ? { part: "header" } : {}) };
- }
- return fail(`${mw}.type: one of status, word, negative-word, regex, negative-regex`);
- },
- });
- const cond = req["matchers-condition"];
- if (cond !== undefined && cond !== "and" && cond !== "or") fail(`${w}.matchers-condition: "and" or "or"`);
- // Dropping these would turn a guarded check back into a status-only one.
- for (const k of ["confirm", "mustChangeAnswer"]) if (req[k] !== undefined) fail(`${w}.${k}: built-in templates only`);
- return { method, path: req.path, matchers, ...(cond ? { "matchers-condition": cond } : {}) };
- },
- }),
- };
- return out;
- }
- function product(v, where) {
- if (!PRODUCT_IDS.includes(v)) fail(`${where}: one of ${PRODUCT_IDS.join(", ")}`);
- return v;
- }
- /** A release line of `prod` ("8.3", "20", "1.1.1"), checked against how that product numbers them. */
- function branchName(prod, v, where) {
- text(v, where, { max: 20 });
- const parsed = parseVersion(v);
- if (!parsed || parsed.suffix || parsed.pre || PRODUCTS[prod].branchOf(parsed) !== v) fail(`${where}: not a ${PRODUCTS[prod].name} release line`);
- return v;
- }
- /** An exact upstream version of `prod` on release line `branch`. */
- function exactVersion(prod, branch, v, where) {
- text(v, where, { max: 30 });
- const parsed = parseVersion(v);
- if (!parsed || parsed.suffix || parsed.pre || parsed.patch === null) fail(`${where}: expected an exact upstream version like 8.3.27`);
- if (PRODUCTS[prod].branchOf(parsed) !== branch) fail(`${where}: ${v} is not on release line ${branch}`);
- return v;
- }
- /**
- * Rule kinds and the engine each one plugs into. `compile` validates one rule
- * and returns what the engine consumes.
- */
- export const KINDS = {
- // lib/whitebox/secrets.js
- secret: (r, w) => ({
- ...provenance(r, w),
- name: text(r.name, `${w}.name`, { max: 100 }),
- pattern: regex(r.pattern, r.flags, w),
- severity: severity(r.severity, `${w}.severity`),
- skipPlaceholders: r.skip_placeholders === true,
- }),
- // lib/agents/hidden-content.js: instructions hidden in comments, decoded blobs
- "agent-instruction": (r, w) => ({ ...provenance(r, w), pattern: regex(r.pattern, r.flags, w) }),
- // lib/agents/hidden-content.js + rules.js: remote-code-execution shapes in skills, hooks, MCP commands
- "agent-exec": (r, w) => ({ ...provenance(r, w), label: text(r.label, `${w}.label`, { max: 100 }), pattern: regex(r.pattern, r.flags, w) }),
- // lib/deps: known-malicious package versions (lockfile names)
- "ioc-package": (r, w) => ({
- ...provenance(r, w),
- campaign: text(r.campaign, `${w}.campaign`, { max: 100 }),
- severity: severity(r.severity, `${w}.severity`, "critical"),
- packages: list(r.packages, `${w}.packages`, {
- item: (p, pw) => {
- if (!p || typeof p.name !== "string" || p.name.length > 214 || !NPM_NAME.test(p.name)) fail(`${pw}.name: not an npm package name`);
- const versions = list(p.versions, `${pw}.versions`, { max: 200, item: (v, vw) => { if (typeof v !== "string" || !VERSION_OR_STAR.test(v)) fail(`${vw}: exact version or "*"`); return v; } });
- return { name: p.name, versions };
- },
- }),
- }),
- // lib/deps/python: known-malicious PyPI versions (normalized names, PEP 440 versions)
- "ioc-pypi-package": (r, w) => ({
- ...provenance(r, w),
- campaign: text(r.campaign, `${w}.campaign`, { max: 100 }),
- severity: severity(r.severity, `${w}.severity`, "critical"),
- packages: list(r.packages, `${w}.packages`, {
- item: (p, pw) => {
- if (!p || typeof p.name !== "string" || p.name.length > 214 || !PYPI_NAME.test(p.name)) fail(`${pw}.name: not a normalized PyPI name (PEP 503: lowercase, runs of -_. as one -)`);
- const versions = list(p.versions, `${pw}.versions`, { max: 200, item: (v, vw) => { if (typeof v !== "string" || !PEP440_OR_STAR.test(v)) fail(`${vw}: exact PEP 440 version or "*"`); return v; } });
- return { name: p.name, versions };
- },
- }),
- }),
- // lib/deps: exfiltration endpoints found in package code. `hosts` are
- // attacker-owned domains; `urls` pin an endpoint on a shared service
- // (webhook.site) where the bare host would be a false positive.
- Note: Quoted as data in code: Mentions a known data-relay/testing domain (webhook.site) without linking or sending anything to it
- Line 239 on GitHub
- "ioc-host": (r, w) => {
- if (r.hosts === undefined && r.urls === undefined) fail(`${w}: needs hosts or urls`);
- return {
- ...provenance(r, w),
- campaign: text(r.campaign, `${w}.campaign`, { max: 100 }),
- severity: severity(r.severity, `${w}.severity`, "critical"),
- hosts: r.hosts === undefined ? [] : list(r.hosts, `${w}.hosts`, { max: 500, item: (h, hw) => { if (typeof h !== "string" || !HOSTNAME.test(h)) fail(`${hw}: lowercase hostname`); return h; } }),
- urls: r.urls === undefined ? [] : list(r.urls, `${w}.urls`, {
- max: 500,
- item: (u, uw) => {
- text(u, uw, { max: 300 });
- let url;
- try { url = new URL(u); } catch { fail(`${uw}: not a URL`); }
- if (!/^https?:$/.test(url.protocol) || url.pathname.length < 2) fail(`${uw}: http(s) URL with a path`);
- return u;
- },
- }),
- };
- },
- // lib/deps: files a campaign drops into packages (by path, optionally pinned by hash)
- "ioc-file": (r, w) => ({
- ...provenance(r, w),
- campaign: text(r.campaign, `${w}.campaign`, { max: 100 }),
- severity: severity(r.severity, `${w}.severity`, "critical"),
- files: list(r.files, `${w}.files`, {
- max: 500,
- item: (f, fw) => {
- if (!f || typeof f.path !== "string" || !REL_PATH.test(f.path)) fail(`${fw}.path: relative path inside a package`);
- if (f.sha256 !== undefined && (typeof f.sha256 !== "string" || !SHA256.test(f.sha256))) fail(`${fw}.sha256: hex SHA-256`);
- return { path: f.path, ...(f.sha256 ? { sha256: f.sha256 } : {}) };
- },
- }),
- }),
- // lib/deps/typosquat.js: extra names worth protecting from look-alikes
- "typosquat-target": (r, w) => ({
- ...provenance(r, w),
- names: list(r.names, `${w}.names`, { max: 5000, item: (n, nw) => { if (typeof n !== "string" || n.length > 214 || !NPM_NAME.test(n)) fail(`${nw}: npm package name`); return n; } }),
- }),
- // lib/advisories: affected and fixed versions of a runtime or server, per release line
- "runtime-advisory": (r, w) => {
- const prod = product(r.product, `${w}.product`);
- const seen = new Set();
- const out = {
- ...provenance(r, w),
- product: prod,
- title: text(r.title, `${w}.title`, { max: 200 }),
- severity: severity(r.severity, `${w}.severity`, "critical"),
- cves: r.cves === undefined ? [] : list(r.cves, `${w}.cves`, { max: 50, item: (c, cw) => { if (typeof c !== "string" || !CVE_ID.test(c)) fail(`${cw}: CVE-YYYY-NNNN`); return c; } }),
- ghsa: r.ghsa === undefined ? [] : list(r.ghsa, `${w}.ghsa`, { max: 50, item: (g, gw) => { if (typeof g !== "string" || !GHSA_ID.test(g)) fail(`${gw}: GHSA-xxxx-xxxx-xxxx`); return g; } }),
- // What has to be true besides the version (an OS, a SAPI, a module).
- // A rule with a condition never yields more than `potential`.
- condition: text(r.condition, `${w}.condition`, { max: 300, optional: true }),
- fixed: list(r.fixed, `${w}.fixed`, {
- max: 50,
- item: (f, fw) => {
- if (!f || typeof f !== "object") fail(`${fw}: expected { branch, version }`);
- const branch = branchName(prod, f.branch, `${fw}.branch`);
- if (seen.has(branch)) fail(`${fw}.branch: ${branch} listed twice`);
- seen.add(branch);
- const version = exactVersion(prod, branch, f.version, `${fw}.version`);
- const introduced = f.introduced === undefined ? undefined : exactVersion(prod, branch, f.introduced, `${fw}.introduced`);
- if (introduced && compareVersions(introduced, version) >= 0) fail(`${fw}.introduced: must be below the fixed version`);
- return { branch, version, ...(introduced ? { introduced } : {}) };
- },
- }),
- };
- // Date the CVE was added to CISA's Known Exploited Vulnerabilities catalog (CC0).
- if (r.kev !== undefined) {
- if (typeof r.kev !== "string" || !DATE.test(r.kev) || Number.isNaN(Date.parse(r.kev))) fail(`${w}.kev: expected YYYY-MM-DD`);
- out.kev = r.kev;
- }
- if (r.cvss !== undefined) {
- const c = r.cvss;
- if (!c || !CVSS_VERSIONS.has(c.version) || typeof c.score !== "number" || c.score < 0 || c.score > 10) fail(`${w}.cvss: { version: "3.1" | "4.0", score: 0-10 }`);
- out.cvss = { version: c.version, score: c.score };
- }
- return out;
- },
- // lib/advisories: release lines past their vendor's end of support. A line
- // counts as end of life when its `eol` date is on or before the rule's
- // `date`, so the result depends on the pack, never on the scan's clock.
- "runtime-eol": (r, w) => {
- const prod = product(r.product, `${w}.product`);
- const seen = new Set();
- return {
- ...provenance(r, w),
- product: prod,
- branches: list(r.branches, `${w}.branches`, {
- max: 100,
- item: (b, bw) => {
- if (!b || typeof b !== "object") fail(`${bw}: expected { branch, eol }`);
- const branch = branchName(prod, b.branch, `${bw}.branch`);
- if (seen.has(branch)) fail(`${bw}.branch: ${branch} listed twice`);
- seen.add(branch);
- if (typeof b.eol !== "string" || !DATE.test(b.eol) || Number.isNaN(Date.parse(b.eol))) fail(`${bw}.eol: expected YYYY-MM-DD`);
- return { branch, eol: b.eol };
- },
- }),
- };
- },
- // lib/templates.js
- "nuclei-template": (r, w) => ({ ...provenance(r, w), template: template(r.template, `${w}.template`) }),
- };
- // ── Reading and verifying ─────────────────────────────────────────────────
- function cmpSemver(a, b) {
- const pa = a.split(".").map(Number);
- const pb = b.split(".").map(Number);
- for (let i = 0; i < 3; i++) if (pa[i] !== pb[i]) return pa[i] < pb[i] ? -1 : 1;
- return 0;
- }
- function readRegular(path, max, what) {
- let st;
- try { st = lstatSync(path); } catch { fail(`${what} not found`); }
- if (!st.isFile()) fail(`${what} is not a regular file (symlinks are refused)`);
- if (st.size > max) fail(`${what} is larger than ${max} bytes`);
- return readFileSync(path);
- }
- function validateManifest(m) {
- if (!m || typeof m !== "object" || Array.isArray(m)) fail("pack.json: expected an object");
- if (m.schema !== 1) fail("pack.json: schema must be 1");
- if (typeof m.name !== "string" || !PACK_NAME.test(m.name)) fail(`pack.json: name must match ${PACK_NAME}`);
- if (typeof m.version !== "string" || !SEMVER.test(m.version)) fail("pack.json: version must be x.y.z");
- if (typeof m.min_penthera !== "string" || !SEMVER.test(m.min_penthera)) fail("pack.json: min_penthera must be x.y.z");
- if (typeof m.created_at !== "string" || Number.isNaN(Date.parse(m.created_at))) fail("pack.json: created_at must be an ISO 8601 timestamp");
- text(m.description, "pack.json: description", { max: 1000, optional: true });
- const seen = new Set();
- list(m.rules, "pack.json: rules", {
- max: 100,
- item: (e, w) => {
- if (!e || typeof e.file !== "string" || !FILE_NAME.test(e.file) || e.file === "pack.json") fail(`${w}.file: a plain file name like secrets.json`);
- if (seen.has(e.file)) fail(`${w}.file: listed twice`);
- seen.add(e.file);
- if (!Object.hasOwn(KINDS, e.kind)) fail(`${w}.kind: one of ${Object.keys(KINDS).join(", ")}`);
- if (typeof e.sha256 !== "string" || !SHA256.test(e.sha256)) fail(`${w}.sha256: hex SHA-256`);
- return e;
- },
- });
- return m;
- }
- /** Load allowed_signers entries from the built-in file plus any extra files. */
- export function loadSigners(extra = []) {
- const out = [];
- for (const f of [BUILTIN_SIGNERS, ...[].concat(extra)]) {
- if (!f) continue;
- if (!existsSync(f)) {
- if (f === BUILTIN_SIGNERS) continue;
- throw new Error(`allowed_signers file not found: ${f}`);
- }
- out.push(...parseAllowedSigners(readFileSync(f, "utf-8")));
- }
- return out;
- }
- /**
- * Verify a pack directory without activating it.
- * @param {string} dir
- * @param {{ signers?: object[], now?: number, version?: string }} [opts]
- * @returns {{ ok: boolean, dir: string, name?: string, version?: string,
- * signature: "valid"|"invalid"|"missing", signer?: string, signatureError?: string,
- * errors: string[], warnings: string[], manifest?: object, rules?: Record<string, object[]>, count?: number }}
- */
- export function verifyPack(dir, opts = {}) {
- const abs = resolve(dir);
- const res = { ok: false, dir: abs, signature: "missing", errors: [], warnings: [] };
- try {
- let st;
- try { st = lstatSync(abs); } catch { fail(`not found: ${dir}`); }
- if (!st.isDirectory()) fail(`not a directory: ${dir}`);
- const raw = readRegular(join(abs, "pack.json"), MAX_MANIFEST_BYTES, "pack.json");
- let manifest;
- try { manifest = JSON.parse(raw.toString("utf-8")); } catch (e) { fail(`pack.json: invalid JSON (${e.message})`); }
- validateManifest(manifest);
- res.name = manifest.name;
- res.version = manifest.version;
- res.manifest = manifest;
- // 1. Signature over the exact bytes of pack.json.
- const sigPath = join(abs, "pack.json.sig");
- if (existsSync(sigPath)) {
- const sig = readRegular(sigPath, 64 * 1024, "pack.json.sig").toString("utf-8");
- const v = verifyWithSigners(raw, sig, NAMESPACE, opts.signers || loadSigners(), { now: opts.now });
- if (v.ok) { res.signature = "valid"; res.signer = v.principal; }
- else { res.signature = "invalid"; res.signatureError = v.reason; }
- }
- if (cmpSemver(opts.version || PKG_VERSION, manifest.min_penthera) < 0) {
- fail(`needs Penthera ${manifest.min_penthera} or later (this is ${opts.version || PKG_VERSION})`);
- }
- // 2. Every rule file against its hash in pack.json, then 3. its contents.
- const rules = Object.fromEntries(Object.keys(KINDS).map((k) => [k, []]));
- const ids = new Set();
- let count = 0;
- for (const entry of manifest.rules) {
- const buf = readRegular(join(abs, entry.file), MAX_RULE_FILE_BYTES, entry.file);
- const digest = createHash("sha256").update(buf).digest("hex");
- if (digest !== entry.sha256) fail(`${entry.file}: SHA-256 does not match pack.json (file changed after the pack was built)`);
- let data;
- try { data = JSON.parse(buf.toString("utf-8")); } catch (e) { fail(`${entry.file}: invalid JSON (${e.message})`); }
- if (!data || data.kind !== entry.kind) fail(`${entry.file}: kind is "${data?.kind}", pack.json says "${entry.kind}"`);
- list(data.rules, `${entry.file}: rules`, { max: MAX_RULES_PER_FILE, item: (r) => r });
- data.rules.forEach((r, i) => {
- const where = `${entry.file}: rules[${i}]`;
- if (!r || typeof r !== "object" || Array.isArray(r)) fail(`${where}: expected an object`);
- const compiled = KINDS[entry.kind](r, where);
- if (ids.has(compiled.id)) fail(`${where}.id: "${compiled.id}" used twice in this pack`);
- ids.add(compiled.id);
- rules[entry.kind].push({ ...compiled, kind: entry.kind, pack: manifest.name, packVersion: manifest.version });
- count++;
- });
- }
- const listed = new Set(manifest.rules.map((e) => e.file));
- for (const f of readdirSync(abs)) {
- if (f === "pack.json" || f === "pack.json.sig" || listed.has(f)) continue;
- res.warnings.push(`${f} is not listed in pack.json and is ignored`);
- }
- res.rules = rules;
- res.count = count;
- res.ok = true;
- } catch (e) {
- res.errors.push(e instanceof PackError ? e.message : `unexpected error: ${e.message}`);
- }
- return res;
- }
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.