Skip to content
Back to search

penthera

MarketplaceWorks with Claude CodeGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/danoszz/penthera with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/danoszz/penthera'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/danoszz--penthera/badge.svg)](/listings/danoszz--penthera)

HTML

<a href="/listings/danoszz--penthera"><img src="/api/listings/danoszz--penthera/badge.svg" alt="findsafeskills safety rating" /></a>

Install

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 22 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "penthera",
  3. "owner": {
  4. "name": "Daan van der Zwaag",
  5. "url": "https://github.com/danoszz"
  6. },
  7. "metadata": {
  8. "description": "Offline cyber security agent for your apps and AI coding tools. Open R&D: measured in public, improved every release.",
  9. "version": "2.4.0"
  10. },
  11. "plugins": [
  12. {
  13. "name": "penthera",
  14. "description": "Offline cyber security agent for your apps and AI coding tools. Open R&D: measured in public, improved every release.",
  15. "source": "./plugin",
  16. "strict": true,
  17. "skills": [
  18. "./skills/penthera"
  19. ]
  20. }
  21. ]
  22. }
README.md · README · 621 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. <img src="site/media/social/og.png" alt="Your elite cyber security agent. Audit your app/site/code before you publish to protect yourself, and your users. Free, open source, offline. Penthera." width="840">
  3. <br/><br/>
  4. # Penthera: your open-source cyber security agent
  5. The security review you never had time for. Offline, rule-based, and you can verify every finding yourself.
  6. **[penthera.org](https://penthera.org)**: check your live site in a few seconds. Passive, nothing stored.
  7. Daily notes from Penthera: **[x.com/penthera_org](https://x.com/penthera_org)**
  8. **Who builds it:** [Daan van der Zwaag](docs/trust.md#who-builds-it), part-time and in the open.
  9. **How it compares:** [next to Snyk, Socket, Semgrep, ZAP and GitHub](#how-it-fits-with-other-tools).
  10. **New in 2.4:** `penthera --card` makes a security receipt of your repo to post, every surface counts one set of problems with the same words, and 13 ways to hide a finding or stall a scan are closed ([what changed](CHANGELOG.md)).
  11. **From your coding agent:** add Penthera ([one line per agent](#start-in-30-seconds)), then say "Hey Claude, run Penthera on this repo."
  12. **132** readable rules · **6** read-only MCP tools · **0** runtime dependencies · **0** requests past loopback in the benchmark · **MIT**, no telemetry
  13. <br/>
  14. [![npm](https://img.shields.io/npm/v/penthera?color=0000ed)](https://www.npmjs.com/package/penthera)
  15. [![npm downloads](https://img.shields.io/npm/dm/penthera?color=0000ed&label=downloads)](https://www.npmjs.com/package/penthera)
  16. [![GitHub stars](https://img.shields.io/github/stars/danoszz/penthera?color=0000ed)](https://github.com/danoszz/penthera/stargazers)
  17. ![node](https://img.shields.io/badge/node-%E2%89%A522-0000ed)
  18. ![license](https://img.shields.io/badge/license-MIT-0000ed)
  19. [![CI](https://github.com/danoszz/penthera/actions/workflows/ci.yml/badge.svg)](https://github.com/danoszz/penthera/actions/workflows/ci.yml)
  20. [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/danoszz/penthera/badge)](https://scorecard.dev/viewer/?uri=github.com/danoszz/penthera)
  21. [![Socket](https://socket.dev/api/badge/npm/package/penthera)](https://socket.dev/npm/package/penthera)
  22. [![Glama MCP server score](https://glama.ai/mcp/servers/danoszz/penthera/badges/score.svg)](https://glama.ai/mcp/servers/danoszz/penthera)
  23. <br/>
  24. <img src="assets/demo.gif" alt="A 12-second loop: npx penthera sets up a project, an offline scan runs in the terminal, the dashboard shows the verdict in Simple mode and the triage matrix in Advanced mode, and a fix is approved" width="760">
  25. <br/><br/>
  26. | `penthera ui`, Simple, dark | `penthera ui`, Advanced, light |
  27. |---|---|
  28. | ![The dashboard in Simple mode, dark: the verdict, problems in plain words and one problem open](assets/dashboard/ui-dark.jpg) | ![The dashboard in Advanced mode, light: a project's tabs, the triage matrix and a finding open](assets/dashboard/ui-light.jpg) |
  29. </div>
  30. ---
  31. ## What it is
  32. Penthera checks your app, its dependencies and your AI coding tools on your own
  33. machine. It fixes what it can, with your approval. Every finding comes from a
  34. rule you can read, with its evidence at `file:line`. A model may propose; a rule
  35. decides.
  36. It is also an open R&D project. We measure every check in public and publish
  37. the results, good or bad, so you can see how far to trust it.
  38. It finds known classes of problems, not novel ones. It does not replace a human
  39. pentester.
  40. ## Start in 30 seconds
  41. You need Node.js 22 or later. No install, no account.
  42. Building with Cursor, Claude Code, Codex, VS Code, Windsurf, Gemini CLI, Zed,
  43. JetBrains or Kiro, or an app builder such as Lovable, Bolt, Base44, Replit or
  44. v0? [Penthera in your coding tool](docs/integrations.md) has the setup for
  45. each.
  46. **In your terminal.** Run this in your project folder:
  47. ```bash
  48. npx penthera
  49. ```
  50. It detects your setup offline, asks what to check, and gives you a grade per
  51. area and the three things to fix first. It scans your running app only after
  52. you confirm the app is yours. Nothing is committed or pushed
  53. ([guided start](docs/commands/init.md)).
  54. **In your coding agent.** Pick yours, then ask "scan my app for security
  55. issues". The MCP lines pin the version.
  56. | Agent | Run this |
  57. |---|---|
  58. | Claude Code, plugin (skill only, runs the CLI) | `claude plugin marketplace add danoszz/penthera`, then `claude plugin install penthera@penthera` |
  59. | Claude Code, MCP server only | `claude mcp add penthera -- npx -y penthera@2.4.0 mcp` |
  60. | Codex | `codex mcp add penthera -- npx -y penthera@2.4.0 mcp` |
  61. | GitHub Copilot CLI | `copilot plugin marketplace add danoszz/penthera`, then `copilot plugin install penthera@penthera` |
  62. | GitHub Copilot in VS Code | the `code --add-mcp` line in [mcp.md](docs/commands/mcp.md#github-copilot) |
  63. | Gemini CLI | `gemini extensions install https://github.com/danoszz/penthera` |
  64. | Cursor, Windsurf | the JSON in [mcp.md](docs/commands/mcp.md#cursor-and-windsurf) |
  65. | Any agent that reads skills | `npx skills add danoszz/penthera` |
  66. The skill tells your agent how to run Penthera; the MCP server gives it six
  67. read-only tools. More routes: [skill.md](docs/commands/skill.md),
  68. [mcp.md](docs/commands/mcp.md).
  69. **Where AI fits.** Rules decide every finding and every exit code. A model may
  70. only propose:
  71. | Role | Who | How |
  72. |---|---|---|
  73. | Decides | Penthera's rules, always | Every finding, every exit code |
  74. | Looks further (optional) | A model, with `--ai` | Proposes problems no rule describes; the rules check each one, and unchecked ones are listed apart and never fail CI ([AI mode](docs/ai-mode.md)) |
  75. | Fixes | Your own coding agent | Through the MCP server, the skill or `penthera fix --plan` |
  76. To run it in a container:
  77. `docker build -t penthera . && docker run --rm penthera https://your-app.example`.
  78. ## Public benchmarks
  79. We test Penthera on public datasets that other people built, next to the free
  80. open-source tools that do the same job, scored the same way. Each dataset was
  81. split in half before we looked: we improved Penthera on one half and scored it
  82. once on the other, sealed half. The rules for scoring and ranking were written
  83. down before that run ([protocol](docs/research/prereg/E11b-improvement-study.md)).
  84. <!-- public-benchmarks:start -->
  85. | Benchmark | Score | Penthera 1.9.0 | **Penthera 2.0.0** | Rank among free tools | Best other free tool |
  86. |---|---|---|---|---|---|
  87. | [Leaked secrets (Samsung CredData)](https://penthera.org/benchmarks/secrets) | F1 | 0.18 | **0.74** | **2nd of 10** | deepsecrets 0.78 |
  88. | [Malicious npm packages (DataDog)](https://penthera.org/benchmarks/npm-malware) | caught minus false alarms | 0.57 | **0.71** | **1st of 6, tied 1st to 3rd** ⁴ | malcontent 0.63 |
  89. | [Malicious PyPI packages (DataDog)](https://penthera.org/benchmarks/pypi-malware) | caught minus false alarms | 0.74 | **0.82** | **1st of 8, tied 1st to 3rd** ⁴ | GuardDog 0.74 |
  90. | [Injection bugs in JS libraries (SecBench.js)](https://penthera.org/benchmarks/js-injection) | found in the right file | 0.16 | **0.70** | **1st of 8, tied 1st to 4th** ⁴ | odgen 0.68 |
  91. ⁴ A tied range is a rank interval: the range Penthera's rank fell in, in 95% of 10,000 bootstrap redraws of each tool's counts (on SecBench.js, of its 64 test bugs). Penthera alone came out first in 76% of the redraws on npm, 82% on PyPI and 60% on SecBench.js. Exploratory: computed after the registered ranking.
  92. Leaked secrets: 2nd by F1, the registered score; 1st by F0.5, which weighs precision more (0.82 against deepsecrets 0.80); 2nd by F2, which weighs recall more (0.68 against deepsecrets 0.77).
  93. Some ranked tools are barely in the contest. Other ranked tools scoring below 0.10: 1 of 9 on CredData (trufflehog); 3 of 5 on npm (clamav, yara, ossgadget); 4 of 7 on PyPI (clamav, ossgadget, bandit4mal, aura); 1 of 7 on SecBench.js (njsscan).
  94. If 1 package in 1,000 were malicious, the share of Penthera's malware flags that would be right: npm, about 2.7% at the top of the false-alarm interval (it raised no false alarm on that sample); PyPI, about 1.6% at the measured false-alarm rate, 0.9% at the top of its interval. Exploratory, like the rank intervals.
  95. <details><summary>Every tool, every number (95% intervals)</summary>
  96. #### Leaked secrets (Samsung CredData)
  97. | Rank | Tool | Precision | Recall | F1 |
  98. |---:|---|---|---|---|
  99. | 1 | deepsecrets 2.2.0 | 0.80 [0.79, 0.82] | 0.77 [0.75, 0.78] | 0.78 |
  100. | 2 | **Penthera** 2.0.0 | 0.89 [0.87, 0.90] | 0.64 [0.62, 0.66] | 0.74 |
  101. | 3 | noseyparker 0.24.0 | 0.76 [0.74, 0.78] | 0.62 [0.60, 0.64] | 0.69 |
  102. | 4 | gitleaks 8.30.1 | 0.85 [0.84, 0.86] | 0.37 [0.35, 0.39] | 0.52 |
  103. | 5 | ripsecrets 0.1.11 | 0.65 [0.64, 0.67] | 0.35 [0.33, 0.37] | 0.45 |
  104. | 6 | kingfisher 2.14.0 | 0.96 [0.94, 0.97] | 0.29 [0.28, 0.31] | 0.45 |
  105. | 7 | detect-secrets 1.5.0 | 0.30 [0.29, 0.31] | 0.53 [0.51, 0.56] | 0.38 |
  106. | 8 | secretlint 13.0.7 | 0.92 [0.89, 0.94] | 0.19 [0.18, 0.21] | 0.32 |
  107. | 9 | whispers 2.4.0 | 0.46 [0.43, 0.50] | 0.09 [0.08, 0.10] | 0.15 |
  108. | 10 | trufflehog 3.99.3 | 0.40 [0.35, 0.45] | 0.05 [0.05, 0.07] | 0.10 |
  109. | – | credsweeper 1.19.4 ¹ | 1.00 [1.00, 1.00] | 0.98 [0.98, 0.99] | 0.99 |
  110. | – | Penthera 1.9.0 | 0.56 [0.51, 0.60] | 0.10 [0.09, 0.12] | 0.18 |
  111. | | Free tools, mean / median | 0.68 / 0.76 | 0.36 / 0.35 | 0.43 / 0.45 |
  112. #### Malicious npm packages (DataDog)
  113. | Rank | Tool | Caught | False alarms on benign packages | Caught minus false alarms |
  114. |---:|---|---|---|---|
  115. | 1 | **Penthera** 2.0.0 | 0.71 [0.62, 0.78] (79/112) | 0.00 [0.00, 0.02] (0/150) | 0.71 |
  116. | 2 | malcontent 1.27.1 (bf280ad6), yara-x 1.21.0 | 0.68 [0.59, 0.76] (76/112) | 0.05 [0.03, 0.10] (8/150) | 0.63 |
  117. | 3 | GuardDog 3.2.0 ² | 0.62 [0.52, 0.70] (69/112) | 0.03 [0.01, 0.07] (4/150) | 0.59 |
  118. | 4 | ossgadget 0.2.20 (414b6a59) | 0.94 [0.88, 0.97] (105/112) | 0.89 [0.83, 0.93] (133/150) | 0.05 |
  119. | 5 | clamav 1.5.4 | 0.01 [0.00, 0.05] (1/112) | 0.00 [0.00, 0.02] (0/150) | 0.01 |
  120. | 6 | yara yara-python 4.5.4 (YARA 4.5), YARA-Forge core 20261004 | 0.00 [0.00, 0.03] (0/112) | 0.00 [0.00, 0.02] (0/150) | 0.00 |
  121. | – | osv-scanner 2.6.0 ³ | 0.90 [0.83, 0.94] (101/112) | 0.01 [0.00, 0.04] (1/150) | 0.90 |
  122. | – | Penthera 1.9.0 | 0.57 [0.48, 0.66] (64/112) | 0.00 [0.00, 0.02] (0/150) | 0.57 |
  123. #### Malicious PyPI packages (DataDog)
  124. | Rank | Tool | Caught | False alarms on benign packages | Caught minus false alarms |
  125. |---:|---|---|---|---|
  126. | 1 | **Penthera** 2.0.0 | 0.87 [0.81, 0.92] (131/150) | 0.05 [0.03, 0.10] (8/150) | 0.82 |
  127. | 2 | GuardDog 3.2.0 ² | 0.85 [0.79, 0.90] (128/150) | 0.11 [0.07, 0.17] (16/147) | 0.74 |
  128. | 3 | malcontent 1.27.1 (bf280ad6), yara-x 1.21.0 | 0.93 [0.87, 0.96] (139/150) | 0.22 [0.16, 0.29] (32/147) | 0.71 |
  129. | 4 | yara yara-python 4.5.4 (YARA 4.5), YARA-Forge core 20261004 | 0.25 [0.18, 0.32] (37/150) | 0.00 [0.00, 0.03] (0/147) | 0.25 |
  130. | 5 | clamav 1.5.4 | 0.00 [0.00, 0.02] (0/150) | 0.00 [0.00, 0.03] (0/147) | 0.00 |
  131. | 6 | ossgadget 0.2.20 (414b6a59) | 0.77 [0.70, 0.83] (116/150) | 0.97 [0.93, 0.99] (143/147) | -0.20 |
  132. | 7 | aura 2.1 (SourceCode-AI/aura b40e6dd2) | 0.59 [0.51, 0.67] (88/149) | 0.83 [0.75, 0.88] (100/121) | -0.24 |
  133. | 8 | bandit4mal lyvd/bandit4mal 03ba95f3 (bandit 1.6.3 fork) | 0.59 [0.51, 0.67] (89/150) | 0.84 [0.78, 0.89] (124/147) | -0.25 |
  134. | – | Penthera 1.9.0 | 0.83 [0.77, 0.88] (125/150) | 0.09 [0.06, 0.15] (14/150) | 0.74 |
  135. | – | osv-scanner 2.6.0 ³ | 0.74 [0.66, 0.80] (110/149) | 0.00 [0.00, 0.03] (0/147) | 0.74 |
  136. #### Injection bugs in JavaScript libraries (SecBench.js)
  137. | Rank | Tool | Command injection | Code injection | Both |
  138. |---:|---|---|---|---|
  139. | 1 | **Penthera** 2.0.0 | 35/47 | 10/17 | 0.70 [0.58, 0.80] |
  140. | 2 | odgen c5157ee | 36/46 | 6/16 | 0.68 [0.55, 0.78] |
  141. | 3 | codeql 2.27.2 (used under the GitHub CodeQL terms, for research on open-source code) | 37/47 | 4/17 | 0.64 [0.52, 0.75] |
  142. | 4 | eslint-plugin-security 4.2.0 | 32/47 | 9/17 | 0.64 [0.52, 0.75] |
  143. | 5 | semgrep 1.180.0 | 15/47 | 9/17 | 0.38 [0.27, 0.50] |
  144. | 6 | graudit 849d7c7 | 12/47 | 10/17 | 0.34 [0.24, 0.47] |
  145. | 7 | bearer 2.1.1 | 14/47 | 0/17 | 0.22 [0.14, 0.33] |
  146. | 8 | njsscan 1.0.1 | 2/45 | 1/15 | 0.05 [0.02, 0.14] |
  147. | – | semgrep, every file 1.180.0 (another setting of semgrep, not ranked) | 16/47 | 9/17 | 0.39 [0.28, 0.51] |
  148. | – | Penthera 1.9.0 | 0/47 | 10/17 | 0.16 [0.09, 0.26] |
  149. A rank is shown only where at least 5 other free tools were measured; otherwise "–". The rank column orders tools by score, and close scores can swap places. Penthera's 95% rank interval (bootstrap, exploratory): 2nd on CredData, 1st to 3rd on npm, 1st to 3rd on PyPI, 1st to 4th on SecBench.js.
  150. Paid tools (GitGuardian, Socket, Snyk, Phylum, Semgrep Pro) publish no results on these benchmarks, so they are not in the tables.
  151. Published elsewhere, on other datasets (context, not comparable):
  152. - [A comparative study of software secrets reporting by secret detection tools (2023): nine tools on SecretBench, ggshield included](https://arxiv.org/abs/2307.00714)
  153. - [How effective are npm malicious package detectors? (2026): eleven detectors on their own benchmark](https://arxiv.org/abs/2603.27549)
  154. - [Cutting the Gordian knot: detecting malicious PyPI packages (USENIX Security 2026): GuardDog, Bandit4Mal and OSSGadget on PyPI](https://arxiv.org/abs/2601.16463)
  155. - [Study of JavaScript static analysis tools for vulnerability detection in Node.js packages (2023): nine tools incl. CodeQL, NodeJsScan, ODGen](https://www.researchgate.net/publication/367088656)
  156. ¹ Not ranked: trained on this dataset by its authors. ² Found most samples in this dataset, so its rate is inflated by construction. ³ Not ranked: a lookup of already-reported packages, not detection.
  157. </details>
  158. <!-- public-benchmarks:end -->
  159. What these numbers do not show: a benchmark is not your code. CredData replaces
  160. real secrets with look-alikes, which favors pattern rules like ours over tools
  161. that verify keys; the free tools ran on their defaults while Penthera was tuned
  162. on the other half; GuardDog helped build the malware dataset. Accuracy on real
  163. projects is experiment E1. To reproduce every table, get the sealed split with
  164. `npm run bench:public -- --fetch-split`, then run
  165. `npm run bench:public -- --split test --final` and
  166. `npm run bench:public -- --competitors --split test --final`.
  167. ## What it does
  168. ```mermaid
  169. flowchart LR
  170. U["You or<br/>your agent"] --> P(("Penthera<br/>on your machine"))
  171. P --> APP["Your app<br/>live and code"]
  172. P --> DEP["Dependencies<br/>npm and Python"]
  173. P --> AI["AI tooling<br/>skills, MCP, hooks"]
  174. APP & DEP & AI --> F[["Findings<br/>rules decide"]]
  175. F --> FIX["Fix<br/>asks first"]
  176. F --> R["Reports<br/>SARIF, ui, MCP"]
  177. F --> E["Evidence<br/>signed bundle"]
  178. FIX -. "re-scan" .-> F
  179. classDef core fill:#0000ed,stroke:#0000ed,color:#ffffff
  180. classDef box fill:#ffffff,stroke:#000000,color:#000000
  181. class P core
  182. class U,APP,DEP,AI,F,FIX,R,E box
  183. ```
  184. <details>
  185. <summary><b>Advanced: black box</b>, the running app over HTTP</summary>
  186. The default scan sends no attack payloads. `--deep`, `--fuzz` and `--adaptive`
  187. start only after you confirm the hostname, or pass `--confirm-target <host>` in
  188. CI. Limits are in [METHODOLOGY.md](docs/METHODOLOGY.md#under-the-hood).
  189. ```mermaid
  190. %%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 40}}}%%
  191. flowchart LR
  192. APP["Running app<br/>penthera URL"] --> DEF
  193. APP --> ACT
  194. subgraph DEF["Default: no attack payloads"]
  195. direction TB
  196. B1["TLS, headers<br/>cookies, CORS"]
  197. B2["Routes<br/>OpenAPI, files"]
  198. B3["Auth, JWT, OAuth<br/>IDOR: review"]
  199. B4["JS libraries<br/>bundle keys"]
  200. end
  201. subgraph ACT["Opt-in, gated"]
  202. direction TB
  203. A1["--deep<br/>SQLi, XSS, SSRF"]
  204. A2["--fuzz<br/>API bodies"]
  205. A3["--adaptive<br/>may send DELETE"]
  206. end
  207. DEF & ACT --> F[["Findings"]]
  208. classDef core fill:#0000ed,stroke:#0000ed,color:#ffffff
  209. classDef box fill:#ffffff,stroke:#000000,color:#000000
  210. class F core
  211. class APP,B1,B2,B3,B4,A1,A2,A3 box
  212. style DEF fill:none,stroke:#0000ed,stroke-width:2px
  213. style ACT fill:none,stroke:#0000ed,stroke-width:2px,stroke-dasharray:6 4
  214. ```
  215. </details>
  216. <details>
  217. <summary><b>Advanced: white box</b>, your code and config</summary>
  218. Code sinks use taint-lite within one file, with no call graph. Dependency
  219. capabilities are what code can do, not proof it is malicious.
  220. ```mermaid
  221. %%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 40}}}%%
  222. flowchart LR
  223. REPO["Code and config"] --> RS
  224. REPO --> OWN
  225. subgraph RS["penthera --repo ."]
  226. direction TB
  227. W1["Secrets<br/>key formats"]
  228. W2["Routes<br/>auth per route"]
  229. W3["Code sinks<br/>eval, vm, exec"]
  230. W4["CI, runtimes<br/>Actions, pins"]
  231. end
  232. subgraph OWN["Own commands"]
  233. direction TB
  234. W5["deps<br/>capabilities, IOCs"]
  235. W6["agents<br/>MCP, skills, hooks"]
  236. W7["install, hook<br/>before it lands"]
  237. end
  238. RS & OWN --> F[["Findings"]]
  239. classDef core fill:#0000ed,stroke:#0000ed,color:#ffffff
  240. classDef box fill:#ffffff,stroke:#000000,color:#000000
  241. class F core
  242. class REPO,W1,W2,W3,W4,W5,W6,W7 box
  243. style RS fill:none,stroke:#0000ed,stroke-width:2px
  244. style OWN fill:none,stroke:#0000ed,stroke-width:2px
  245. ```
  246. </details>
  247. <details>
  248. <summary><b>Advanced: engine and outputs</b>, how a match becomes a result</summary>
  249. Rules decide every finding. Exit code `0` means nothing critical or high, `1`
  250. critical or high, `2` an incomplete or refused scan. Under `--ai` a model only
  251. proposes.
  252. ```mermaid
  253. %%{init: {"flowchart": {"nodeSpacing": 30, "rankSpacing": 40}}}%%
  254. flowchart LR
  255. subgraph IN["Inputs"]
  256. direction LR
  257. R["Signed rule packs"]
  258. GI["--giants, opt-in"]
  259. AI["--ai, opt-in"]
  260. end
  261. IN --> FD[["Finding<br/>severity, confidence"]]
  262. FD --> EX{"Exit<br/>0, 1 or 2"}
  263. EX --> OUT
  264. subgraph OUT["Outputs"]
  265. direction LR
  266. O1["Reports, SARIF"]
  267. O2["penthera ui, MCP"]
  268. O3["Action, Vercel check"]
  269. O4["Evidence bundle"]
  270. end
  271. classDef core fill:#0000ed,stroke:#0000ed,color:#ffffff
  272. classDef box fill:#ffffff,stroke:#000000,color:#000000
  273. class FD core
  274. class R,GI,AI,EX,O1,O2,O3,O4 box
  275. style IN fill:none,stroke:#0000ed,stroke-width:2px
  276. style OUT fill:none,stroke:#0000ed,stroke-width:2px
  277. ```
  278. </details>
  279. ### Before you deploy
  280. | Job | What Penthera does | Command |
  281. |---|---|---|
  282. | Test the app | Scans your localhost or preview URL (TLS, headers, CORS, cookies, auth, JWT, IDOR, OAuth, OpenAPI; injection and fuzzing only when you ask, after a confirmation) and your repository (routes, trust boundaries, code-execution sinks, secrets including AI keys) | [`penthera <url> --repo .`](docs/commands/scan.md) |
  283. | Check your runtimes | Whether the PHP, nginx, Apache, Node.js, Python, OpenSSL or MongoDB version you run is affected by a known advisory, from the signed advisories pack | [Runtime advisories](docs/runtime-advisories.md) |
  284. | Fix and verify | Applies deterministic fix recipes on a local branch, re-scans to prove they worked, and remembers the result. Never pushes | [`penthera agent --loop`](docs/commands/fix.md) |
  285. | Block it at commit time | Stops secrets, risky agent config and dependency changes before they are committed | [`penthera hook install`](docs/commands/hook.md) |
  286. ### What you build with
  287. | Job | What Penthera does | Command |
  288. |---|---|---|
  289. | Vet dependencies | Shows what every npm and Python package can do (install scripts, shell, network, env, obfuscation), what an upgrade added, lookalike names and licenses | [`penthera deps --base main`](docs/commands/deps.md) |
  290. | Guard installs | Shows what a new package brings before any install script runs, and stops it under your policy | [`penthera install <pkg>`](docs/commands/install.md) |
  291. | Review pull requests | Comments the dependency capability diff on every PR that touches the lockfile | [GitHub Action](docs/commands/action.md) |
  292. | Audit your AI tooling | Checks skills, MCP servers, hooks, plugins and rule files for hidden instructions, unpinned servers, over-broad permissions and silent updates | [`penthera agents --home`](docs/commands/agents.md) |
  293. | Work from your agent | Six read-only MCP tools: scan the repo, check dependencies, audit AI tooling, runtime advisories, explain a finding, a passive URL scan | [`penthera mcp`](docs/commands/mcp.md) |
  294. ### After you ship
  295. | Job | What Penthera does | Command |
  296. |---|---|---|
  297. | Report and prove | Writes reports for developers, owners and auditors; a hashed, signable evidence bundle; NIS2 and CRA readiness reports and drafts | [`penthera <url> --audience auditor --evidence out/`](docs/evidence.md) |
  298. | See it all | A local dashboard on 127.0.0.1. Simple mode for non-programmers: a one-sentence verdict, problems in plain words, and a check-and-fix mode with Apply, Skip and Stop per change. Advanced adds dependencies, AI tooling, history and evidence | [`penthera ui`](docs/dashboard.md) |
  299. | Keep watching | Scheduled checks of production for drift: certificates, headers, new endpoints, keys in bundles, new CVEs in your SBOM | on the [roadmap](ROADMAP.md) |
  300. Every finding carries a severity, a confidence label (`confirmed`, `likely`,
  301. `potential`, `needs-human-review`), its evidence and the research behind it.
  302. Reports come out as Markdown, JSON and SARIF for GitHub code scanning. Exit
  303. code `1` means critical or high findings. Every command and flag is in
  304. [docs/commands/](docs/commands/README.md). Compliance output is readiness
  305. tooling, not a certification and not legal advice.
  306. **AI mode (opt-in).** `penthera --repo . --ai` lets a local model
  307. propose problems no rule describes. Penthera checks each proposal with its
  308. rules, and anything unconfirmed is labelled "model-proposed, unverified" and
  309. never fails CI. Whether it finds more than the rules alone is not measured yet
  310. ([AI mode](docs/ai-mode.md)).
  311. ### How far each part has been taken
  312. | Part | Status |
  313. |---|---|
  314. | Scans: URL, repo, dependencies, AI tooling, installs, pre-commit hook | **Stable**: in every release since 1.x, covered by the tests and the release gate |
  315. | Fix loop and local dashboard (`penthera ui`) | **Stable**; the dashboard was redesigned in 1.9.0 |
  316. | MCP server, Agent Skill, Claude Code plugin | **Stable**, since 1.9.0 |
  317. | Gemini CLI extension | **Stable**: it installs the newest release (2.4.0) |
  318. | Evidence bundles, NIS2 and CRA drafts | **Stable** as readiness tooling, not a certification |
  319. | AI mode (`--ai`) | **Experimental**: whether it finds more than the rules alone is not measured yet ([AI mode](docs/ai-mode.md)) |
  320. | Live check on [penthera.org](https://penthera.org) | **Beta**: passive only, [terms](docs/live-check-terms.md) |
  321. ## How it is measured
  322. Each feature goes through the same path: **method → evidence → tool.**
  323. - **Method.** The [methodology](docs/METHODOLOGY.md) states the threat model,
  324. what black-box (HTTP) and white-box (source) analysis can and cannot see, the
  325. confidence model, and the false-positive controls on each probe. Each design
  326. choice cites the research or standard it rests on: 158 verified references, in
  327. APA style, in [docs/references.md](docs/references.md).
  328. - **Evidence.** A benchmark runs on every release, and the release gate fails
  329. if precision or recall drops by more than 0.02. Experiments that decide what
  330. ships next are pre-registered in [docs/research/prereg/](docs/research/prereg/)
  331. before any data is collected, and the results are published either way.
  332. - **Tool.** What holds up ships in a reviewed, signed release. What does not
  333. hold up is dropped, and the rules stay.
  334. | Measure | Result | Sample |
  335. |---|---|---|
  336. | Precision, bundled benchmark | 0.92, 95% CI 0.74 to 0.98 | 22 true and 2 false positives |
  337. | Recall, bundled benchmark | 0.96, 95% CI 0.79 to 0.99 | 1 miss, [explained](bench/RESULTS.md) |
  338. | Requests past loopback during the benchmark | 0 | Enforced by a network guard |
  339. | Public benchmarks (E11b), sealed test half | 1st on npm and PyPI malware (tied 1st to 3rd), 1st on JS library injection bugs (tied 1st to 4th), 2nd on leaked secrets | 4 public datasets, up to 10 free tools each ([results](bench/public/RESULTS.md#e11b-final-the-sealed-test-half-scored-once-2026-10-25), [summary](#public-benchmarks)) |
  340. | E10 pilot, known-vulnerable apps | Precision 0.88, 95% CI 0.78 to 0.93. Recall 0.07 before fixes, 0.71 (CI 0.45 to 0.88) after | 14 root causes in Juice Shop and DVNA ([results](bench/known-vuln/RESULTS-after-fixes.md)). No run logs in, so it doesn't cover login or access rules |
  341. | Detection rules, all readable | 132 in this release: 16 URL scan templates, 24 secret and AI-key patterns, 15 dependency capabilities, 47 AI-tooling rules, 5 fix recipes, and 25 rules in the signed rule packs | [Rule packs](docs/rule-packs.md) |
  342. The bundled benchmark is 24 findings in small labs we wrote ourselves. It
  343. catches regressions; it does not measure real-world accuracy. The E10 pilot
  344. used one labeller and is not independent: the fixes were written after seeing
  345. the misses on the same two apps. On WebGoat, held out, recall stayed at 2 of 9.
  346. Accuracy on real projects is experiment E1, pre-registered and not yet run.
  347. Intervals are Wilson. What Penthera cannot see is listed under
  348. [Known limitations](docs/METHODOLOGY.md#known-limitations). Refresh the counts
  349. with `npm run docs:numbers` and `npm run bench`.
  350. **Cite this repository** with the button on GitHub or
  351. [CITATION.cff](CITATION.cff). APA and BibTeX forms are in the
  352. [methodology](docs/METHODOLOGY.md#how-to-cite).
  353. ## Why you can trust it
  354. - **Local and offline.** No account, no telemetry, no cloud. `--offline` limits
  355. network traffic to the target you named, and every report lists the hosts it
  356. contacted ([network use](docs/privacy.md)).
  357. - **Tested unplugged.** Repo scans, `agents`, `deps`, `brief`, evidence
  358. bundles and URL scans with `--offline` made no outside connections. They
  359. gave the same result twice, and the same with the network blocked, on macOS
  360. and Ubuntu ([E17 results](bench/offline/RESULTS.md)). The test can't watch
  361. `git`, which Penthera uses only for local commands.
  362. - **Deterministic.** Rules you can read, with `file:line` evidence. A model may
  363. propose a fix or a lead; a deterministic re-scan and your tests decide.
  364. - **Non-destructive by default.** Default scans send no attack payloads.
  365. Payload testing (`--deep`, `--fuzz`, `--adaptive`, `--all`) is opt-in, asks for a
  366. confirmation bound to the target, and is rate-limited. It can change data, so
  367. read [what active modes can change](docs/commands/scan.md#what-active-modes-can-change)
  368. first.
  369. - **Asks before it changes anything.** The fix loop works on a local branch,
  370. asks before every change by default, and never pushes or opens pull requests.
  371. - **No update channel to hijack.** Detection rules ship inside reviewed
  372. releases as signed, versioned [rule packs](docs/rule-packs.md). You can add
  373. your own by path (`--rules <dir>`). Nothing is downloaded and run at run time.
  374. - **Releases you can verify.** npm provenance through trusted publishing, a
  375. maintainer approval with 2FA for every version, SHA-pinned CI, CodeQL and
  376. OpenSSF Scorecard. Every release passes an
  377. [internal audit](docs/internal-audit.md) of eight gates
  378. ([latest report](audit/REPORT.md)).
  379. The [claim map](docs/claims.md) ties these claims to the tests that hold them.
  380. **Verify it yourself.** In an empty folder:
  381. ```bash
  382. npm install penthera@2.4.0 && npm audit signatures
  383. npx penthera@2.4.0 rules list
  384. ```
  385. The first checks npm's registry signature and the build provenance of the
  386. package you got. The second lists every bundled rule pack and the key that
  387. signed it; a pack whose signature does not verify is refused unless you pass
  388. `--allow-unsigned-rules`.
  389. Penthera finds known classes of problems, not novel ones, and it does not
  390. replace a human pentester or incident response. The decisions behind it, its
  391. limits, and how to challenge them are in the [manifesto and FAQ](docs/MANIFESTO.md).
  392. ## How it fits with other tools
  393. Penthera works next to the tools you already use; its SARIF output goes
  394. straight into GitHub code scanning, for example. Each comparison page says where
  395. the other tool goes further and how to run the two together.
  396. | | Focus | Where it runs | How it decides |
  397. |---|---|---|---|
  398. | **Penthera** | Your app, dependencies and AI tooling; fixes; EU evidence | Your machine or CI, offline | Readable deterministic rules, with evidence |
  399. | [Socket](docs/compare/socket.md) | Supply-chain risk across package registries | Cloud service, GitHub App, CLI | Registry-scale package analysis, including AI review |
  400. | [Snyk](docs/compare/snyk.md) | Known vulnerabilities in dependencies, code, containers, IaC | Cloud service, IDE, CLI | Curated vulnerability database plus static analysis |
  401. | [GitHub Dependabot and code scanning](docs/compare/github.md) | Known-CVE alerts, update PRs, CodeQL | GitHub | Advisory database, CodeQL queries |
  402. | [OWASP ZAP](docs/compare/zap.md) | Dynamic testing of web apps | Your machine or CI | Active and passive scan rules |
  403. | [Semgrep](docs/compare/semgrep.md) | Static analysis of your own code | Your machine, CI, cloud platform | Pattern rules, plus a paid cross-file engine |
  404. All comparisons, including Cloudflare's security-audit skill: [docs/compare/](docs/compare/README.md).
  405. ### Run the free tools you already have: `--giants`
  406. `--giants` runs the free scanners installed on your
  407. machine after Penthera's own scan: gitleaks, TruffleHog, Semgrep with your
  408. local rules, osv-scanner, Trivy and Checkov for a repository, nuclei and the
  409. ZAP baseline for a URL. Their findings go into the same report and SARIF, each
  410. labelled with the tool, its version and its rule. Nothing is downloaded or
  411. uploaded, and an area that neither Penthera's rules nor another tool reached
  412. is reported as "not covered".
  413. ```bash
  414. penthera giants # which tools you have
  415. penthera --repo . --giants # Penthera, then each installed tool
  416. ```
  417. This runs the tools you have and merges their findings. It is not a claim that
  418. the combination finds more: experiment E14 measured that
  419. ([results](bench/public/RESULTS.md)). Details, safety rules and the policy key:
  420. [docs/commands/giants.md](docs/commands/giants.md).
  421. ## Start here
  422. | You want to know | Read |
  423. |---|---|
  424. | Is my app secure? | [I'm new](docs/index.md#1-im-new), then [scan](docs/commands/scan.md) |
  425. | What should my small team do about security now? | [A Practical Guide to Security for 2027](docs/security-guide-2027.md): 10 steps, each with the command and its limit |
  426. | Is this package or MCP server safe? | [deps](docs/commands/deps.md), [install](docs/commands/install.md), [agents](docs/commands/agents.md) |
  427. | How do I use it from my coding agent? | [MCP server](docs/commands/mcp.md), [skill](docs/commands/skill.md), [agents safely](docs/agents-safely.md) |
  428. | How do I put it in CI? | [The GitHub Action](docs/commands/action.md), [scan in CI](docs/commands/scan.md#ci-integration) |
  429. | What do I give my auditor? | [Evidence bundles](docs/evidence.md), [NIS2](docs/compliance-nis2.md), [CRA](docs/compliance-cra.md) |
  430. | How do I run the fix loop? | [Fix and verify](docs/commands/fix.md), [the local loop](docs/local-loop.md) |
  431. The full map, by situation and from first scan to research, is in
  432. [docs/index.md](docs/index.md).
  433. ## Ethics and authorization
  434. Penthera sends HTTP requests to the targets you give it, and some modes
  435. (`--deep`, `--fuzz`, `--adaptive`, `--all`) send attack payloads, after a
  436. confirmation.
  437. Scan only what you own, what you have written authorization to test, or local
  438. environments you control. Do not use it to probe third-party systems,
  439. exfiltrate data, disrupt services, or harm anyone. Unauthorized testing may
  440. break computer-misuse laws such as the US CFAA, the UK Computer Misuse Act, and
  441. their EU equivalents. When in doubt, do not scan.
  442. The software is provided "as is", without warranty. The author accepts no
  443. liability for misuse, and you are responsible for making sure every scan you
  444. run is lawful. The live check on penthera.org has its own
  445. [terms](docs/live-check-terms.md).
  446. ## Building in public
  447. What changed, what we are measuring, and what comes next, in one place.
  448. | | Where it stands | Read more |
  449. |---|---|---|
  450. | **Latest release** | 2.4.0 (2026-11-08) on npm `latest`: `penthera --card` makes a security receipt to post, and 13 ways to hide a finding or stall a scan, found by Vercel Security Review, are closed. Crafted files that took seconds to hours now take under a second, and a file too big to finish makes the scan incomplete (exit 2). 2.3.2 brought: hostile files can no longer hang a scan, secrets stay masked in what reports quote | [CHANGELOG](CHANGELOG.md), [roadmap](ROADMAP.md) |
  451. | **Research** | Public benchmarks (E11b) published: on a sealed half of four public datasets, 1st of the free tools on npm and PyPI malware (tied 1st to 3rd), 1st on JS library injection bugs (tied 1st to 4th), 2nd on leaked secrets ([summary](#public-benchmarks)). E10 pilot published: precision 0.88, recall 0.07 to 0.71 on two known-vulnerable apps after fixes (one labeller, not independent; a re-run is pending because default scans no longer send login checks). E1 on real code is pre-registered; E8 (agent guardrails) is a draft | [E11b results](bench/public/RESULTS.md), [E10 results](bench/known-vuln/RESULTS.md), [E1 pre-registration](docs/research/prereg/E1-real-world-accuracy.md), [agents safely](docs/agents-safely.md) |
  452. | **Next** | Real-world precision and recall with intervals (E1), published whatever the result | [Roadmap: next](ROADMAP.md#next), [research plan](docs/vision-2.0.md) |
  453. [![Star history](https://api.star-history.com/svg?repos=danoszz/penthera&type=Date)](https://star-history.com/#danoszz/penthera&Date)
  454. <a href="assets/walkthrough.mp4"><img src="assets/walkthrough-poster.jpg" alt="Walkthrough video: setup, a scan, the dashboard in Simple and Advanced mode, one fix approved, the report" width="760"></a>
  455. [Watch the 80-second walkthrough](assets/walkthrough.mp4): setup, a scan, the dashboard, a fix approved, the report. Demo data, no sound, captions in the video.
  456. Releases are deliberately slow: a feature freeze, an independent review, the
  457. release checks, and a release candidate before any new minor version. Watch the
  458. repository for releases or join the
  459. [Discussions](https://github.com/danoszz/penthera/discussions).
  460. ## How Penthera is built
  461. Penthera is developed with substantial help from generative AI: 91% of the
  462. commits up to v1.7.0 were written with Claude (Anthropic) through Claude Code,
  463. directed and reviewed by the maintainer, and marked with an `AI-Assisted:` trailer.
  464. Generated code passes the same gates as any change: tests, CodeQL, an
  465. independent review, the internal audit, and a 2FA-approved release. Inside the
  466. product, models only ever propose; deterministic checks decide. The full
  467. disclosure, and the rules for AI-assisted contributions, are in
  468. [docs/genai.md](docs/genai.md).
  469. ## Contributing and support
  470. Clone the repository and run `npm install`, then `npm test` for the suite and
  471. `npm run bench` for precision and recall.
  472. [CONTRIBUTING.md](CONTRIBUTING.md) covers the project layout, the release
  473. cadence and the release checklist; [AGENTS.md](AGENTS.md) is the short version
  474. for coding agents. The [roadmap](ROADMAP.md) shows what ships next, and the
  475. [research plan](docs/vision-2.0.md) what we measure. Issues and pull requests
  476. are welcome.
  477. Penthera is free and stays free. If it caught something before you shipped it,
  478. you can [sponsor its maintenance](https://github.com/sponsors/danoszz).
  479. Sponsorship pays for maintainer time and buys no influence over findings
  480. ([tiers](docs/funding/sponsor-tiers.md)).
  481. ## License
  482. MIT. See [LICENSE](LICENSE). Report vulnerabilities in Penthera itself through
  483. [SECURITY.md](SECURITY.md).
bench/front-door/classify.mjs · Bundled script · 532 lines
  1. /**
  2. * E20-R response classifier: a fixed, deterministic label for one homepage
  3. * response. Protocol: docs/research/prereg/E20r-front-door-replication.md § 4.2
  4. * and Amendment 2. The signature lists are frozen before a run; a change after
  5. * that is a deviation. No model decides a label (AGENTS.md rule 4).
  6. *
  7. * challenge an edge network tests the client
  8. * block an edge network or firewall refuses the client
  9. * unrecognized-error-response an error no rule recognized (pass 1: origin-error)
  10. * candidate-page no marker found; not proof of a page (pass 1: page)
  11. * body-unavailable a compressed answer, whose body was never read, so nothing can be said of it
  12. *
  13. * Header markers come first: Cloudflare documents that the status code alone
  14. * does not identify a challenge. `classify` is the V2 rule set, written after
  15. * the V1 labels were read. `classifyRegistered` is the rule set of pass 1,
  16. * unchanged, so C15 and the comparison with pass 1 use one definition.
  17. */
  18. import { createHash } from "node:crypto";
  19. import { isIP, isIPv6 } from "node:net";
  20. export const CHALLENGE_HEADERS = Object.freeze([
  21. ["cf-mitigated", /challenge/i, "cloudflare"],
  22. ["x-amzn-waf-action", /challenge|captcha/i, "aws-waf"],
  23. ["x-vercel-mitigated", /challenge/i, "vercel"],
  24. ]);
  25. // Pass 1 rules, frozen at registration. Do not edit.
  26. export const CHALLENGE_BODIES = Object.freeze([
  27. [/\/cdn-cgi\/challenge-platform\/|<title>\s*Just a moment\.\.\.\s*<\/title>|cf-browser-verification|cf_chl_opt/i, "cloudflare"],
  28. [/captcha-delivery\.com/i, "datadome"],
  29. [/px-captcha|captcha\.px-cdn\.net/i, "human"],
  30. [/_Incapsula_Resource/i, "imperva"],
  31. [/_sec\/verify\?provider=|sec-if-cpt-container/i, "akamai"],
  32. [/Vercel Security Checkpoint/i, "vercel"],
  33. [/AwsWafIntegration|awswaf\.com/i, "aws-waf"],
  34. ]);
  35. export const BLOCK_BODIES = Object.freeze([
  36. [/Attention Required! \| Cloudflare|Sorry, you have been blocked|cf-error-details/i, "cloudflare"],
  37. [/errors\.edgesuite\.net|<H1>Access Denied<\/H1>/i, "akamai"],
  38. [/Generated by cloudfront \(CloudFront\)|The request could not be satisfied/i, "cloudfront"],
  39. [/Incapsula incident ID/i, "imperva"],
  40. ]);
  41. // V2. Hard signatures appear only on an interstitial page.
  42. export const HARD_CHALLENGE_BODIES = Object.freeze([
  43. [/<title>\s*Just a moment\.\.\.\s*<\/title>|cf-browser-verification|cf_chl_opt|\/cdn-cgi\/challenge-platform\/h\/[a-z]\/orchestrate\/chl_page/i, "cloudflare"],
  44. [/captcha-delivery\.com\/captcha/i, "datadome"],
  45. [/px-captcha/i, "human"],
  46. [/id=["']main-iframe["'][^>]*_Incapsula_Resource/i, "imperva"],
  47. [/_sec\/verify\?provider=|sec-if-cpt-container/i, "akamai"],
  48. [/Vercel Security Checkpoint/i, "vercel"],
  49. [/<title>\s*DDoS-Guard\s*<\/title>/i, "ddos-guard"],
  50. ]);
  51. // Soft signatures are scripts that also sit on normal pages as passive bot
  52. // detection. They count on an error status, or on a 2xx too small to be a page.
  53. export const SOFT_CHALLENGE_BODIES = Object.freeze([
  54. [/\/cdn-cgi\/challenge-platform\//i, "cloudflare"],
  55. [/captcha-delivery\.com/i, "datadome"],
  56. [/captcha\.px-cdn\.net/i, "human"],
  57. [/_Incapsula_Resource/i, "imperva"],
  58. [/AwsWafIntegration|awswaf\.com/i, "aws-waf"],
  59. ]);
  60. export const SOFT_MAX_CHARS = 8192;
  61. export const BLOCK_BODIES_V2 = Object.freeze([
  62. [/Attention Required! \| Cloudflare|Sorry, you have been blocked|cf-error-details/i, "cloudflare"],
  63. [/errors\.edgesuite\.net|<H1>Access Denied<\/H1>/i, "akamai"],
  64. [/Request blocked|configured to block access from your country/i, "cloudfront"],
  65. [/Incapsula incident ID/i, "imperva"],
  66. ]);
  67. // CloudFront's own error page, after the two lines above that mean a refusal.
  68. export const CLOUDFRONT_ERROR = /The request could not be satisfied/i;
  69. // Cloudflare's short text error. These four are DNS, tunnel and Worker errors, not refusals.
  70. const CF_ERROR = /^\s*error code:\s*(1\d{3})\b/i;
  71. const CF_NOT_REFUSAL = new Set(["1001", "1016", "1033", "1101"]);
  72. const EDGE_NAMES = Object.freeze([["cf-ray", "cloudflare"], ["x-amz-cf-id", "cloudfront"], ["x-akamai-transformed", "akamai"], ["akamai-grn", "akamai"]]);
  73. const REFUSAL_STATUS = new Set([401, 403, 406, 429]);
  74. // Header values kept verbatim, then scrubbed. Every other header keeps its name only.
  75. export const KEPT_VALUES = Object.freeze(["cf-mitigated", "x-amzn-waf-action", "x-vercel-mitigated", "server", "via", "x-cache", "cf-cache-status", "content-type"]);
  76. export const EXCERPT_BYTES = 4096;
  77. export const PAGE_TEXT_CHARS = 4096;
  78. // Site-controlled text is cut to this before any regex runs on it.
  79. export const INPUT_CAP_CHARS = 65_536;
  80. const lower = (headers = {}) => Object.fromEntries(Object.entries(headers).map(([k, v]) => [k.toLowerCase(), v]));
  81. const edgeOf = (h) => {
  82. for (const [name, vendor] of EDGE_NAMES) if (h[name] !== undefined) return vendor;
  83. const server = String(h.server ?? "");
  84. return /cloudflare/i.test(server) ? "cloudflare" : /cloudfront/i.test(server) ? "cloudfront" : /akamai/i.test(server) ? "akamai" : null;
  85. };
  86. /** The label of one response, with the vendor and the rule that decided it. */
  87. export function classify({ status, headers = {}, body = "" }) {
  88. const h = lower(headers);
  89. for (const [name, re, vendor] of CHALLENGE_HEADERS) {
  90. if (re.test(String(h[name] ?? ""))) return { label: "challenge", vendor, rule: `header:${name}` };
  91. }
  92. const whole = String(body);
  93. const text = whole.slice(0, INPUT_CAP_CHARS);
  94. for (const [re, vendor] of HARD_CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "body" };
  95. // A long 2xx page may mention any of these words. It counts on an error, or a body too small to be a page.
  96. if (status >= 400 || whole.length <= SOFT_MAX_CHARS) {
  97. for (const [re, vendor] of BLOCK_BODIES_V2) if (re.test(text)) return { label: "block", vendor, rule: "body" };
  98. for (const [re, vendor] of SOFT_CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "soft-body" };
  99. if (CLOUDFRONT_ERROR.test(text)) return { label: "unrecognized-error-response", vendor: "cloudfront", rule: "cloudfront:error" };
  100. }
  101. const edge = edgeOf(h);
  102. const code = CF_ERROR.exec(text)?.[1];
  103. if (status >= 400 && code && edge === "cloudflare" && !CF_NOT_REFUSAL.has(code)) return { label: "block", vendor: "cloudflare", rule: `cloudflare:${code}` };
  104. // No body to read, so a challenge and a block look the same: the edge refused.
  105. if (REFUSAL_STATUS.has(status) && !text.trim() && edge) return { label: "block", vendor: edge, rule: "empty-at-edge" };
  106. // A homepage that answers 403 refuses this client, whoever serves it. A view
  107. // of signature-backed refusals alone drops this rule.
  108. if (status === 403) return { label: "block", vendor: edge, rule: "unrecognized-403" };
  109. if (status >= 400) return { label: "unrecognized-error-response", vendor: null, rule: "status" };
  110. if (h["content-encoding"] && !/^identity$/i.test(h["content-encoding"]) && !text.trim()) return { label: "body-unavailable", vendor: null, rule: "content-encoding" };
  111. return { label: "candidate-page", vendor: null, rule: "status" };
  112. }
  113. /** Pass 1's rule set, unchanged, on the whole body, so its labels stay comparable. Each rule is a plain alternation. */
  114. export function classifyRegistered({ status, headers = {}, body = "" }) {
  115. const h = lower(headers);
  116. for (const [name, re, vendor] of CHALLENGE_HEADERS) {
  117. if (re.test(String(h[name] ?? ""))) return { label: "challenge", vendor, rule: `header:${name}` };
  118. }
  119. const text = String(body);
  120. for (const [re, vendor] of CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "body" };
  121. for (const [re, vendor] of BLOCK_BODIES) if (re.test(text)) return { label: "block", vendor, rule: "body" };
  122. if (status >= 400) return { label: "origin-error", vendor: null, rule: "status" };
  123. return { label: "page", vendor: null, rule: "status" };
  124. }
  125. const matches = (rules, text) => Object.fromEntries(rules.map(([re, vendor]) => [vendor, re.test(text)]));
  126. /** Every signature, for every rule set, as a boolean, so a label can be replayed under other rules. */
  127. export function signatureMatches(body) {
  128. const whole = String(body ?? "");
  129. const text = whole.slice(0, INPUT_CAP_CHARS);
  130. return {
  131. registeredChallenge: matches(CHALLENGE_BODIES, whole), registeredBlock: matches(BLOCK_BODIES, whole),
  132. hardChallenge: matches(HARD_CHALLENGE_BODIES, text), softChallenge: matches(SOFT_CHALLENGE_BODIES, text), block: matches(BLOCK_BODIES_V2, text),
  133. cloudfrontError: CLOUDFRONT_ERROR.test(text), cloudflareCode: CF_ERROR.exec(text)?.[1] ?? null,
  134. };
  135. }
  136. /**
  137. * What a value test of the CSP needs, from the full header, not the grader's
  138. * 160-character evidence. Directive names only. `controlsScripts`: a policy has
  139. * `script-src`, `script-src-elem` or `default-src`. `openScripts`: that source
  140. * list still allows inline code without a nonce or hash, or any host of a scheme.
  141. */
  142. export function cspEvidence(headers = {}) {
  143. const raw = [].concat(lower(headers)["content-security-policy"] ?? []).join(",").slice(0, INPUT_CAP_CHARS);
  144. const policies = raw.split(",").map((p) => new Map(p.split(";").map((d) => d.trim().split(/\s+/)).filter(([n]) => /^[a-z-]{1,64}$/i.test(n ?? "")).map(([n, ...v]) => [n.toLowerCase(), v.map((x) => x.toLowerCase())])));
  145. const script = policies.map((p) => p.get("script-src") ?? p.get("script-src-elem") ?? p.get("default-src")).filter(Boolean);
  146. const open = (v) => v.some((s) => ["*", "http:", "https:", "data:"].includes(s)) || (v.includes("'unsafe-inline'") && !v.some((s) => /^'(?:nonce-|sha(?:256|384|512)-)/.test(s)));
  147. return {
  148. present: raw.trim().length > 0, chars: raw.length,
  149. directives: [...new Set(policies.flatMap((p) => [...p.keys()]))].sort().slice(0, 40),
  150. controlsScripts: script.length > 0,
  151. openScripts: script.length > 0 && script.every(open),
  152. };
  153. }
  154. /** Cookie names only, never values. */
  155. export function cookieNames(headers = {}) {
  156. const raw = lower(headers)["set-cookie"];
  157. const list = Array.isArray(raw) ? raw : raw ? [raw] : [];
  158. // No = before the first ; means there is no name, and what stands there may be a value.
  159. // A name that looks like a token, an address or anything but a name isn't kept.
  160. const name = (c) => { const pair = String(c).split(";")[0]; return pair.includes("=") ? pair.split("=")[0].trim() : ""; };
  161. return list.map(name).filter((n) => /^[\w.-]{1,64}$/.test(n) && scrub(n) === n).sort();
  162. }
  163. // § 6, secondary: what kind of site this is, from the homepage already fetched.
  164. // Fixed rules on the HTML; only the booleans are kept, never a URL or a name.
  165. export const TRAIT_RULES = Object.freeze({
  166. login: /<a\b[^>]{0,2000}>\s{0,200}(?:<[^>]{1,500}>\s{0,200}){0,8}(log\s?in|sign\s?in|my account|inloggen|anmelden|connexion)\b/i,
  167. payments: /js\.stripe\.com|checkout\.stripe\.com|paypal\.com\/sdk|paypalobjects\.com|js\.braintreegateway\.com|checkoutshopper-[a-z]+\.adyen\.com|klarna(?:cdn|\.com)|mollie\.com/i,
  168. shop: /cdn\.shopify\.com|woocommerce|magento|\/cart\b|\/checkout\b|add[-_ ]to[-_ ]cart/i,
  169. // A public security or trust page: a stand-in for a team that owns security, not proof of a CISO.
  170. securityPage: /<a\b[^>]{0,2000}href="[^"]{0,500}(?:\/security\b|\/trust\b|\/trust-center|\/\/trust\.|\/\/security\.)[^"]{0,500}"/i,
  171. });
  172. /** Booleans for one page. An empty page (no 2xx HTML) gives null: unknown, not "no". */
  173. export function traitsOf(html) {
  174. if (!html) return null;
  175. const text = String(html).slice(0, INPUT_CAP_CHARS);
  176. return Object.fromEntries(Object.entries(TRAIT_RULES).map(([k, re]) => [k, re.test(text)]));
  177. }
  178. // Every repeat below is bounded, so no input of INPUT_CAP_CHARS can make these slow.
  179. const ZERO_WIDTH = /[​-‍⁠­]/g;
  180. const NUMERIC_ENTITY = /&#(?:x0*([0-9a-f]{1,8})|0*(\d{1,8}));?/gi;
  181. const NAMED_ENTITY = /&(commat|period|lowbar|colon);?/gi;
  182. const ESCAPED = /\\(?:x|u00)([0-9a-f]{2})/gi;
  183. const PERCENT = /%(?:25)?([2-7][0-9a-f])/gi;
  184. // A bracketed at is an address, whatever stands around it: both neighbouring words go.
  185. const BRACKET_AT = /[^\s<>"']{0,64}\s{0,3}[[({]\s{0,2}at\s{0,2}[\])}]\s{0,3}[^\s<>"']{0,64}/gi;
  186. const AT_WORD = /\s{1,3}@\s{0,3}(?=[\p{L}\p{N}])|(?<=[\p{L}\p{N}\p{M}])\s{0,3}@\s{1,3}|[@﹫]/gu;
  187. const TAGGED_AT = /(?:<\/?[a-z]{1,8}>\s{0,2}){1,4}@(?:\s{0,2}<\/?[a-z]{1,8}>){1,4}/gi;
  188. const DASH_AT = /([\p{L}\p{N}+]{1,64})[-_.]at[-_.]([\p{L}\p{N}-]{1,63}\.\p{L}{2,24})\b/giu;
  189. const DOT_WORD = /\s{0,3}[[({]\s{0,2}(?:dot|\.)\s{0,2}[\])}]\s{0,3}/gi;
  190. // A bare `at` counts only before something with a dot, spaced or not.
  191. const SPELLED_EMAIL = /([\p{L}\p{N}._%+-]{1,64})\s{1,3}at\s{1,3}([\p{L}\p{N}_-]{1,63})(?:\s{1,3}dot\s{1,3}|\s{0,3}\.\s{0,3})(\p{L}{2,24})\b/giu;
  192. const CF_EMAIL = /(data-cfemail=["']?)[0-9a-f]{2,512}|(\/cdn-cgi\/l\/email-protection#)[0-9a-f]{2,512}/gi;
  193. // No TLD needed: a cut address has none, and an @ inside a word is no loss.
  194. const EMAIL = /[\p{L}\p{M}\p{N}._%+'\uFFFD-]{1,64}@[\p{L}\p{M}\p{N}_\uFFFD-]{1,63}(?:\.[\p{L}\p{M}\p{N}_-]{1,63}){0,6}/gu;
  195. const QUOTED_EMAIL = /"[^"\n]{1,64}"@[\p{L}\p{N}-]{1,63}(?:\.[\p{L}\p{N}-]{1,63}){0,6}/gu;
  196. const IPV6_RUN = /(?<!:)(?:[0-9a-f]{0,4}:){2,8}(?:\d{1,3}(?:\.\d{1,3}){3}|[0-9a-f]{0,4})(?![0-9a-f])/gi;
  197. const IPV4 = /(?<!\d)(?:\d{1,3}\.){3}\d{1,3}(?!\d)(?!\.\d)/g;
  198. const IPV4_DASHED = /(?<!\d)\d{1,3}-\d{1,3}-\d{1,3}-\d{1,3}(?![\d-])/g;
  199. const IPV4_HEX = /\b0x[0-9a-f]{8}\b/gi;
  200. const QUERY = /\?[^\s"'<>)\\#?]{1,4096}/gu;
  201. const FRAGMENT = /(?<=[\w/.-])#[^\s"'<>)\\]{1,4096}/gu;
  202. const JWT = /eyJ[\w-]{6,4096}\.[\w-]{2,4096}(?:\.[\w-]{0,4096})?/g;
  203. const BEARER = /\b(bearer|basic|token)\s{1,3}[\w.~+/=-]{8,4096}/gi;
  204. const UUID = /\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi;
  205. const NAMED = /\b([\w-]{0,32}(?:token|session[_-]?id|sessionid|sessid|session|sess|sid|csrf|xsrf|auth|secret|nonce|password|passwd|api[_-]?key|access[_-]?key|[_-]pass|[_-]pwd)|pass|pwd)(["']?\s{0,3}[:=]\s{0,3}["']?)[^\s"'<>&;,)]{3,4096}/gi;
  206. const TOKEN_RUN = /[A-Za-z0-9_-]{16,4096}/g;
  207. const LONG_NUMBER = /\+?\d[\d\s().-]{7,40}\d/g;
  208. const digits = (s) => s.replace(/\D/g, "").length;
  209. const isToken = (s) => /^[0-9a-f]{16,}$/i.test(s) || (digits(s) >= 3 && /[A-Za-z]/.test(s)) || (s.length >= 20 && /\d/.test(s) && /[a-z]/.test(s) && /[A-Z]/.test(s));
  210. // A run can start with one stray label: `IP:` or the `ip6:` of an SPF record.
  211. /** One rule for the scrub and its detector. `a::b` is C++: with no digit, an address needs some length. */
  212. export const looksIpv6 = (s) => isIPv6(s) && (/\d/.test(s) || s.replace(/:/g, "").length >= 6);
  213. const ipv6Out = (m) => (looksIpv6(m) || (!m.startsWith("::") && looksIpv6(m.slice(m.indexOf(":") + 1))) ? "<ip>" : m);
  214. const printable = (code, m) => (code >= 0x20 && code < 0x7f ? String.fromCharCode(code) : m);
  215. const NAMED_CHARS = Object.freeze({ commat: "@", period: ".", lowbar: "_", colon: ":" });
  216. const safeChar = (m, hex) => { const ch = String.fromCharCode(parseInt(hex, 16)); return /[\w@.:-]/.test(ch) ? ch : m; };
  217. /** Every way a page writes an address, back to plain text, so one pattern can see it. */
  218. const decodeForms = (s) => s
  219. .replace(ZERO_WIDTH, "")
  220. .replace(/&amp;(?=#)/gi, "&")
  221. .replace(NUMERIC_ENTITY, (m, hex, dec) => printable(hex ? parseInt(hex, 16) : Number(dec), m))
  222. .replace(NAMED_ENTITY, (_m, name) => NAMED_CHARS[name.toLowerCase()])
  223. .replace(ESCAPED, safeChar).replace(/\\100/g, "@").replace(PERCENT, safeChar)
  224. .replace(TAGGED_AT, "@").replace(DOT_WORD, ".")
  225. .replace(BRACKET_AT, "<email>").replace(AT_WORD, "@")
  226. .replace(SPELLED_EMAIL, "$1@$2.$3")
  227. .replace(DASH_AT, "$1@$2");
  228. /** ASCII lower case, one for one. `toLowerCase` makes U+0130 two units, which shifts every index after it. */
  229. const asciiLower = (s) => s.replace(/[A-Z]+/g, (m) => m.toLowerCase());
  230. /** The end of a tag, quotes respected, or the end of the text. */
  231. function tagEnd(text, from) {
  232. let quote = "";
  233. for (let i = from; i < text.length; i++) {
  234. const ch = text[i];
  235. if (quote) { if (ch === quote) quote = ""; } else if (ch === '"' || ch === "'") quote = ch;
  236. else if (ch === ">") return i + 1;
  237. }
  238. return text.length;
  239. }
  240. /** Form fields out, whatever their size: a field's value can be a token or a person's details. */
  241. function stripFields(text) {
  242. const low = asciiLower(text);
  243. const field = /<(input|textarea|select)\b/g;
  244. let out = "", at = 0, m;
  245. while ((m = field.exec(low))) {
  246. out += `${text.slice(at, m.index)}<input>`;
  247. const close = m[1] === "input" ? -1 : low.indexOf(`</${m[1]}`, m.index);
  248. at = m[1] === "input" ? tagEnd(text, m.index) : close < 0 ? text.length : tagEnd(text, close);
  249. field.lastIndex = at;
  250. }
  251. return out + text.slice(at);
  252. }
  253. /** The cap, ending at a space: what the cut split is never kept. */
  254. function capped(text) {
  255. if (text.length <= INPUT_CAP_CHARS) return text;
  256. let end = INPUT_CAP_CHARS;
  257. while (end > 0 && !/\s/.test(text[end - 1])) end--;
  258. return text.slice(0, end);
  259. }
  260. /** IP addresses and emails out. */
  261. export const scrubAddresses = (text) => decodeForms(String(text ?? "")).replace(QUOTED_EMAIL, "<email>").replace(EMAIL, "<email>").replace(IPV6_RUN, ipv6Out).replace(IPV4, "<ip>").replace(IPV4_DASHED, "<ip>").replace(IPV4_HEX, "<ip>");
  262. /** For anything stored: addresses, form fields, query strings, fragments, tokens and long numbers. Run it before slicing. */
  263. export const scrub = (text) => scrubAddresses(stripFields(capped(String(text ?? ""))).replace(CF_EMAIL, (_m, a, b) => `${a ?? b}<email>`))
  264. .replace(QUERY, "?<query>").replace(FRAGMENT, "#<fragment>")
  265. .replace(JWT, "<token>").replace(BEARER, "$1 <token>").replace(UUID, "<id>")
  266. .replace(NAMED, "$1$2<token>")
  267. .replace(TOKEN_RUN, (m) => (isToken(m) ? "<token>" : m))
  268. .replace(LONG_NUMBER, (m) => (digits(m) >= 9 && !/^\d{4}-\d{2}-\d{2}/.test(m) ? "<number>" : m));
  269. // What a scrub leaves behind, listed here on its own so a change to the scrub can't hide in the detector.
  270. const OWN_TOKENS = Object.freeze(["<email>", "<ip>", "<token>", "<id>", "<number>", "<query>", "<fragment>", "<input>"]);
  271. /** Does this text open with one of our tokens, past an opening bracket or quote? Our output, so no address can start there. */
  272. const opensWithOwnToken = (text) => {
  273. let i = 0;
  274. while (i < text.length && "([{\"'".includes(text[i])) i++;
  275. return OWN_TOKENS.some((t) => text.startsWith(t, i));
  276. };
  277. // What can follow a bare `at` as an address: past an opening bracket or quote, a whole domain, walked to its
  278. // end. Each label starts with a letter or number and ends with one or a mark, with hyphens inside and Unicode
  279. // allowed. The last is two or more letters, or punycode, at a token's edge. `$29.`, `5:30.`, `Joe's.`, `100%.`
  280. // and paths hold characters no label can, and our own tokens and web URLs fail the same test.
  281. // The scrub closes `(info) @ x.y` up to `(info)@x.y`, which the token walk below can't read past the bracket.
  282. const BARE_AT = /(?: [([{<"'`“‘«*]*|[([{<]+)(?:at|@)(?= )|(?<=[)\]}>"'`”’»])@|(?<=[\p{L}\p{N}._%+-])@(?=[([{<"'`“‘«])/gu;
  283. const OWN_LOCAL = new RegExp(`(?:${[...OWN_TOKENS, "<redacted>"].join("|")})$`);
  284. const ADDRESS_LOCAL = /[\p{L}\p{N}._%+-]["'`”’»)\]}>*]*$/u;
  285. const DOMAIN_OPENING = /[([{<"'`“‘«*]*/y;
  286. const DOMAIN_LABEL = /[\p{L}\p{N}][\p{L}\p{N}\p{M}-]*/uy;
  287. // Dots that join labels, and a dot that may instead end a sentence.
  288. const DOMAIN_JOIN = /[.。.。](?=[\p{L}\p{N}_-])| [.。.。] ?| dot | ?[([{] ?dot ?[)\]}] ?/uy;
  289. const SENTENCE_DOT = /[.。.。] /y;
  290. const LAST_LABEL = /^(?:\p{L}[\p{L}\p{M}]+|xn--[a-z0-9-]*[a-z0-9])$/u;
  291. // A candidate is whole unless what follows shows it goes on as something else. A fire costs one record's text
  292. // and a miss keeps an address, so this lists what rejects and all else is an edge. Punctuation of a query, fragment,
  293. // port or sentence only counts up to a space or the end, so `x.com:443`, `x.com?a=b` and `x.com..y` aren't domains,
  294. // and neither is what the scrub leaves of them, `x.com?<query>` and `x.com#<fragment>`.
  295. const TOKEN_EDGE = /[.:?#。.。]*(?:$| )|[.:。.。]*[^\s\p{L}\p{N}\p{M}_\-/\\@=&%+~.:?#。.。<]|</uy;
  296. // A name holds 253 characters, and spelled dots or spaces around them add a few to each label.
  297. const MAX_DOMAIN_CHARS = 1024;
  298. /** Is this a whole domain, read to its end? One bad label or character anywhere in it means no. */
  299. function opensDomain(text, start) {
  300. DOMAIN_OPENING.lastIndex = start;
  301. let at = start + DOMAIN_OPENING.exec(text)[0].length;
  302. const limit = at + MAX_DOMAIN_CHARS;
  303. let labels = 0, tight = false;
  304. while (at < limit) {
  305. DOMAIN_LABEL.lastIndex = at;
  306. const label = DOMAIN_LABEL.exec(text)?.[0];
  307. if (!label || label.endsWith("-")) return false;
  308. at += label.length;
  309. labels++;
  310. DOMAIN_JOIN.lastIndex = at;
  311. const join = DOMAIN_JOIN.exec(text)?.[0];
  312. if (join) {
  313. tight ||= /^[.。.。]$/.test(join);
  314. at += join.length;
  315. continue;
  316. }
  317. const whole = labels > 1 && LAST_LABEL.test(label);
  318. TOKEN_EDGE.lastIndex = at;
  319. if (whole) return TOKEN_EDGE.test(text);
  320. // `night. Open` can be a spaced name. `3.99. Open` is a price and a sentence: a name doesn't turn loose after a tight dot.
  321. SENTENCE_DOT.lastIndex = at;
  322. const dot = !tight && SENTENCE_DOT.exec(text)?.[0];
  323. if (!dot) return false;
  324. at += dot.length;
  325. }
  326. return false;
  327. }
  328. // A JWT opens at a token's edge. `keyJ.value` holds the letters and isn't one.
  329. const JWT_AT_EDGE = /(?:^|[^A-Za-z0-9_-])eyJ[A-Za-z0-9_-]*\.[A-Za-z0-9_-]/;
  330. /**
  331. * What should never survive a scrub. This walks tokens with its own tests and
  332. * shares no pattern with `scrub`, so one regex fault can't hide from both.
  333. */
  334. export function scrubMisses(text) {
  335. const out = { ipv4: 0, ipv6: 0, email: 0, query: 0, fragment: 0, jwt: 0, longHex: 0, longNumber: 0 };
  336. const whole = String(text ?? "");
  337. const low = whole.toLowerCase();
  338. for (const mark of ["&commat", "\\x40", "\\u0040", "\\100", "%40", "%2540", "@", "﹫"]) if (low.includes(mark)) out.email++;
  339. // A bracketed at, with or without inner spaces, in its own walk over the text.
  340. for (let i = 0; i < low.length; i++) {
  341. if (!"[({".includes(low[i])) continue;
  342. let k = i + 1;
  343. while (low[k] === " ") k++;
  344. if (low.startsWith("at", k)) {
  345. k += 2;
  346. while (low[k] === " ") k++;
  347. if (!"])}".includes(low[k] ?? "?")) continue;
  348. k++;
  349. while (low[k] === " ") k++;
  350. if (!opensWithOwnToken(low.slice(k, k + 16))) out.email++;
  351. }
  352. }
  353. // A bare `at` or `@` before something that could be a domain. A bracketed or quoted domain counts, and so
  354. // does prose that looks like one. Invisible characters, wide forms and spaces spelled as entities don't break it.
  355. const spaced = low.normalize("NFKC").replace(/\p{Cf}/gu, "").replace(/&nbsp;|&#0*160;|&#x0*a0;/g, " ").replace(/\s+/g, " ");
  356. for (const m of spaced.matchAll(BARE_AT)) {
  357. const after = m.index + m[0].length;
  358. if (OWN_LOCAL.test(spaced.slice(Math.max(0, m.index - 16), m.index))) continue;
  359. if (ADDRESS_LOCAL.test(spaced.slice(Math.max(0, m.index - 16), m.index)) && opensDomain(spaced, spaced[after] === " " ? after + 1 : after)) out.email++;
  360. }
  361. // Ordinary pages hold many entities for quotes and dashes. Only ones that spell ASCII hide text.
  362. let spelled = 0;
  363. for (let i = low.indexOf("&#"); i >= 0; i = low.indexOf("&#", i + 2)) {
  364. const hex = low[i + 2] === "x";
  365. let end = i + (hex ? 3 : 2);
  366. while (end < low.length && end < i + 16 && (hex ? "0123456789abcdef" : "0123456789").includes(low[end])) end++;
  367. const code = Number.parseInt(low.slice(i + (hex ? 3 : 2), end), hex ? 16 : 10);
  368. if (code === 0x40) out.email++;
  369. if (code >= 0x20 && code < 0x7f) spelled++;
  370. }
  371. if (spelled >= 4) out.email++;
  372. const edge = new Set(`.,;:!?()[]{}<>"'=\``);
  373. const trim = (t) => {
  374. let a = 0, b = t.length;
  375. while (a < b && edge.has(t[a])) a++;
  376. while (b > a && edge.has(t[b - 1])) b--;
  377. return t.slice(a, b);
  378. };
  379. for (const raw of whole.split(/[\s"'`<>()\[\],;\\]+/)) {
  380. if (raw === "redacted" || raw === "query" || raw === "fragment") continue;
  381. const t = trim(raw);
  382. if (!t) continue;
  383. for (const piece of [raw, t, ...t.split("="), t.slice(t.indexOf(":") + 1)]) {
  384. const candidate = isIP(piece) ? piece : trim(piece);
  385. const v = isIP(candidate);
  386. if (v === 4) { out.ipv4++; break; }
  387. if (v === 6 && looksIpv6(candidate)) { out.ipv6++; break; }
  388. }
  389. // The @ needs an address on both sides: `}@media` in a stylesheet and `pkg@^1` on a CDN have none.
  390. const wordChar = (ch) => ch.toLowerCase() !== ch.toUpperCase() || (ch >= "0" && ch <= "9");
  391. const localEnd = (ch) => wordChar(ch) || "._+-%'".includes(ch);
  392. if (t.split(/[/?&=:]/).some((piece) => { const at = piece.indexOf("@"); return at > 0 && at < piece.length - 1 && localEnd(piece[at - 1]) && wordChar(piece[at + 1]); })) out.email++;
  393. const eq = t.indexOf("=");
  394. const q = t.indexOf("?"), hash = t.indexOf("#");
  395. if (q >= 0 && eq > q + 1) out.query++;
  396. if (hash >= 0 && eq > hash + 1) out.fragment++;
  397. if (JWT_AT_EDGE.test(t)) out.jwt++;
  398. let run = 0, numerals = 0;
  399. for (const ch of t.toLowerCase()) {
  400. run = "0123456789abcdef".includes(ch) ? run + 1 : 0;
  401. if (ch >= "0" && ch <= "9") numerals++;
  402. if (run === 32) { out.longHex++; }
  403. }
  404. if (numerals >= 12 && numerals === t.length) out.longNumber++;
  405. }
  406. return out;
  407. }
  408. const headerEvidence = (res) => {
  409. const h = lower(res.headers);
  410. return {
  411. status: res.status,
  412. headerNames: Object.keys(h).filter((n) => /^[\w-]{1,64}$/.test(n) && scrub(n) === n).sort(),
  413. values: Object.fromEntries(KEPT_VALUES.filter((k) => h[k] !== undefined).map((k) => [k, scrub(h[k]).slice(0, 120)])),
  414. cookieNames: cookieNames(h),
  415. };
  416. };
  417. const sha256 = (s) => createHash("sha256").update(s).digest("hex");
  418. const outside2xx = (status) => status < 200 || status >= 300;
  419. const SKIPPED_TAGS = Object.freeze(["script", "style", "noscript", "svg", "title", "template", "textarea", "select"]);
  420. /** The text outside markup, in one pass. An unclosed comment or script runs to the end of the input. */
  421. function visibleText(html) {
  422. const low = asciiLower(html);
  423. let out = "";
  424. let i = 0;
  425. while (i < html.length) {
  426. const lt = low.indexOf("<", i);
  427. if (lt < 0) { out += html.slice(i); break; }
  428. out += `${html.slice(i, lt)} `;
  429. if (low.startsWith("<!--", lt)) {
  430. const end = low.indexOf("-->", lt + 4);
  431. if (end < 0) break;
  432. i = end + 3;
  433. continue;
  434. }
  435. const tag = SKIPPED_TAGS.find((t) => low.startsWith(`<${t}`, lt) && !/[a-z0-9-]/.test(low[lt + t.length + 1] ?? " "));
  436. if (tag) {
  437. const end = low.indexOf(`</${tag}`, lt);
  438. const close = end < 0 ? -1 : low.indexOf(">", end);
  439. if (close < 0) break;
  440. i = close + 1;
  441. continue;
  442. }
  443. const gt = low.indexOf(">", lt);
  444. if (gt < 0) break;
  445. i = gt + 1;
  446. }
  447. return out;
  448. }
  449. function between(html, low, tag) {
  450. const open = low.indexOf(`<${tag}`);
  451. const start = open < 0 ? -1 : low.indexOf(">", open);
  452. const end = start < 0 ? -1 : low.indexOf(`</${tag}`, start);
  453. return end < 0 ? "" : html.slice(start + 1, end);
  454. }
  455. /**
  456. * The visible text of a 2xx response and its shape, enough to tell a page from
  457. * a JS challenge, a soft block or a custom interstitial. No markup is kept.
  458. */
  459. export function pageExcerpt(body) {
  460. const whole = String(body ?? "");
  461. const text = stripFields(capped(whole));
  462. const low = asciiLower(text);
  463. const flat = (s) => visibleText(s).replace(/&(?:nbsp|amp|quot|lt|gt|#39|#x27);/gi, " ").replace(/\s+/g, " ").trim();
  464. return {
  465. title: scrub(flat(between(text, low, "title"))).slice(0, 160),
  466. text: scrub(flat(visibleText(text))).slice(0, PAGE_TEXT_CHARS),
  467. noscript: scrub(flat(between(text, low, "noscript"))).slice(0, 300),
  468. chars: whole.length,
  469. scripts: text.match(/<script\b/gi)?.length ?? 0,
  470. forms: text.match(/<form\b/gi)?.length ?? 0,
  471. links: text.match(/<a\b/gi)?.length ?? 0,
  472. metaRefresh: /<meta\b[^>]{0,500}http-equiv=["']?refresh/i.test(text),
  473. soft: SOFT_CHALLENGE_BODIES.filter(([re]) => re.test(text)).map(([, vendor]) => vendor),
  474. hard: HARD_CHALLENGE_BODIES.filter(([re]) => re.test(text)).map(([, vendor]) => vendor),
  475. };
  476. }
  477. /**
  478. * What E20-R stores for one site (§ 4.1, Amendment 2). Each of the three
  479. * responses gets a body hash. Outside 2xx the first 4 KB is kept, scrubbed. A
  480. * 2xx homepage keeps only `pageExcerpt`.
  481. */
  482. export function evidenceOf(page, responses = []) {
  483. const body = String(page.body ?? "");
  484. const other = ({ kind, res }) => ({
  485. kind, ...headerEvidence(res),
  486. bodySha256: sha256(String(res.body ?? "")),
  487. ...(outside2xx(res.status) ? { excerpt: scrub(res.body).slice(0, EXCERPT_BYTES) } : {}),
  488. });
  489. return {
  490. ...headerEvidence(page),
  491. bodySha256: sha256(body),
  492. ...(outside2xx(page.status) ? { excerpt: scrub(body).slice(0, EXCERPT_BYTES) } : { pageExcerpt: pageExcerpt(body) }),
  493. class: classify(page),
  494. classRegistered: classifyRegistered(page),
  495. signatures: signatureMatches(body),
  496. csp: cspEvidence(page.headers),
  497. traits: traitsOf(page.html),
  498. others: responses.map(other),
  499. };
  500. }
bench/front-door/legacy/classify-7023106f.mjs · Bundled script · 464 lines
  1. /**
  2. * E20-R response classifier: a fixed, deterministic label for one homepage
  3. * response. Protocol: docs/research/prereg/E20r-front-door-replication.md § 4.2
  4. * and Amendment 2. The signature lists are frozen before a run; a change after
  5. * that is a deviation. No model decides a label (AGENTS.md rule 4).
  6. *
  7. * challenge an edge network tests the client
  8. * block an edge network or firewall refuses the client
  9. * unrecognized-error-response an error no rule recognized (pass 1: origin-error)
  10. * candidate-page no marker found; not proof of a page (pass 1: page)
  11. * body-unavailable a compressed answer, whose body was never read, so nothing can be said of it
  12. *
  13. * Header markers come first: Cloudflare documents that the status code alone
  14. * does not identify a challenge. `classify` is the V2 rule set, written after
  15. * the V1 labels were read. `classifyRegistered` is the rule set of pass 1,
  16. * unchanged, so C15 and the comparison with pass 1 use one definition.
  17. */
  18. import { createHash } from "node:crypto";
  19. import { isIP, isIPv6 } from "node:net";
  20. export const CHALLENGE_HEADERS = Object.freeze([
  21. ["cf-mitigated", /challenge/i, "cloudflare"],
  22. ["x-amzn-waf-action", /challenge|captcha/i, "aws-waf"],
  23. ["x-vercel-mitigated", /challenge/i, "vercel"],
  24. ]);
  25. // Pass 1 rules, frozen at registration. Do not edit.
  26. export const CHALLENGE_BODIES = Object.freeze([
  27. [/\/cdn-cgi\/challenge-platform\/|<title>\s*Just a moment\.\.\.\s*<\/title>|cf-browser-verification|cf_chl_opt/i, "cloudflare"],
  28. [/captcha-delivery\.com/i, "datadome"],
  29. [/px-captcha|captcha\.px-cdn\.net/i, "human"],
  30. [/_Incapsula_Resource/i, "imperva"],
  31. [/_sec\/verify\?provider=|sec-if-cpt-container/i, "akamai"],
  32. [/Vercel Security Checkpoint/i, "vercel"],
  33. [/AwsWafIntegration|awswaf\.com/i, "aws-waf"],
  34. ]);
  35. export const BLOCK_BODIES = Object.freeze([
  36. [/Attention Required! \| Cloudflare|Sorry, you have been blocked|cf-error-details/i, "cloudflare"],
  37. [/errors\.edgesuite\.net|<H1>Access Denied<\/H1>/i, "akamai"],
  38. [/Generated by cloudfront \(CloudFront\)|The request could not be satisfied/i, "cloudfront"],
  39. [/Incapsula incident ID/i, "imperva"],
  40. ]);
  41. // V2. Hard signatures appear only on an interstitial page.
  42. export const HARD_CHALLENGE_BODIES = Object.freeze([
  43. [/<title>\s*Just a moment\.\.\.\s*<\/title>|cf-browser-verification|cf_chl_opt|\/cdn-cgi\/challenge-platform\/h\/[a-z]\/orchestrate\/chl_page/i, "cloudflare"],
  44. [/captcha-delivery\.com\/captcha/i, "datadome"],
  45. [/px-captcha/i, "human"],
  46. [/id=["']main-iframe["'][^>]*_Incapsula_Resource/i, "imperva"],
  47. [/_sec\/verify\?provider=|sec-if-cpt-container/i, "akamai"],
  48. [/Vercel Security Checkpoint/i, "vercel"],
  49. [/<title>\s*DDoS-Guard\s*<\/title>/i, "ddos-guard"],
  50. ]);
  51. // Soft signatures are scripts that also sit on normal pages as passive bot
  52. // detection. They count on an error status, or on a 2xx too small to be a page.
  53. export const SOFT_CHALLENGE_BODIES = Object.freeze([
  54. [/\/cdn-cgi\/challenge-platform\//i, "cloudflare"],
  55. [/captcha-delivery\.com/i, "datadome"],
  56. [/captcha\.px-cdn\.net/i, "human"],
  57. [/_Incapsula_Resource/i, "imperva"],
  58. [/AwsWafIntegration|awswaf\.com/i, "aws-waf"],
  59. ]);
  60. export const SOFT_MAX_CHARS = 8192;
  61. export const BLOCK_BODIES_V2 = Object.freeze([
  62. [/Attention Required! \| Cloudflare|Sorry, you have been blocked|cf-error-details/i, "cloudflare"],
  63. [/errors\.edgesuite\.net|<H1>Access Denied<\/H1>/i, "akamai"],
  64. [/Request blocked|configured to block access from your country/i, "cloudfront"],
  65. [/Incapsula incident ID/i, "imperva"],
  66. ]);
  67. // CloudFront's own error page, after the two lines above that mean a refusal.
  68. export const CLOUDFRONT_ERROR = /The request could not be satisfied/i;
  69. // Cloudflare's short text error. These four are DNS, tunnel and Worker errors, not refusals.
  70. const CF_ERROR = /^\s*error code:\s*(1\d{3})\b/i;
  71. const CF_NOT_REFUSAL = new Set(["1001", "1016", "1033", "1101"]);
  72. const EDGE_NAMES = Object.freeze([["cf-ray", "cloudflare"], ["x-amz-cf-id", "cloudfront"], ["x-akamai-transformed", "akamai"], ["akamai-grn", "akamai"]]);
  73. const REFUSAL_STATUS = new Set([401, 403, 406, 429]);
  74. // Header values kept verbatim, then scrubbed. Every other header keeps its name only.
  75. export const KEPT_VALUES = Object.freeze(["cf-mitigated", "x-amzn-waf-action", "x-vercel-mitigated", "server", "via", "x-cache", "cf-cache-status", "content-type"]);
  76. export const EXCERPT_BYTES = 4096;
  77. export const PAGE_TEXT_CHARS = 4096;
  78. // Site-controlled text is cut to this before any regex runs on it.
  79. export const INPUT_CAP_CHARS = 65_536;
  80. const lower = (headers = {}) => Object.fromEntries(Object.entries(headers).map(([k, v]) => [k.toLowerCase(), v]));
  81. const edgeOf = (h) => {
  82. for (const [name, vendor] of EDGE_NAMES) if (h[name] !== undefined) return vendor;
  83. const server = String(h.server ?? "");
  84. return /cloudflare/i.test(server) ? "cloudflare" : /cloudfront/i.test(server) ? "cloudfront" : /akamai/i.test(server) ? "akamai" : null;
  85. };
  86. /** The label of one response, with the vendor and the rule that decided it. */
  87. export function classify({ status, headers = {}, body = "" }) {
  88. const h = lower(headers);
  89. for (const [name, re, vendor] of CHALLENGE_HEADERS) {
  90. if (re.test(String(h[name] ?? ""))) return { label: "challenge", vendor, rule: `header:${name}` };
  91. }
  92. const whole = String(body);
  93. const text = whole.slice(0, INPUT_CAP_CHARS);
  94. for (const [re, vendor] of HARD_CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "body" };
  95. // A long 2xx page may mention any of these words. It counts on an error, or a body too small to be a page.
  96. if (status >= 400 || whole.length <= SOFT_MAX_CHARS) {
  97. for (const [re, vendor] of BLOCK_BODIES_V2) if (re.test(text)) return { label: "block", vendor, rule: "body" };
  98. for (const [re, vendor] of SOFT_CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "soft-body" };
  99. if (CLOUDFRONT_ERROR.test(text)) return { label: "unrecognized-error-response", vendor: "cloudfront", rule: "cloudfront:error" };
  100. }
  101. const edge = edgeOf(h);
  102. const code = CF_ERROR.exec(text)?.[1];
  103. if (status >= 400 && code && edge === "cloudflare" && !CF_NOT_REFUSAL.has(code)) return { label: "block", vendor: "cloudflare", rule: `cloudflare:${code}` };
  104. // No body to read, so a challenge and a block look the same: the edge refused.
  105. if (REFUSAL_STATUS.has(status) && !text.trim() && edge) return { label: "block", vendor: edge, rule: "empty-at-edge" };
  106. // A homepage that answers 403 refuses this client, whoever serves it. A view
  107. // of signature-backed refusals alone drops this rule.
  108. if (status === 403) return { label: "block", vendor: edge, rule: "unrecognized-403" };
  109. if (status >= 400) return { label: "unrecognized-error-response", vendor: null, rule: "status" };
  110. if (h["content-encoding"] && !/^identity$/i.test(h["content-encoding"]) && !text.trim()) return { label: "body-unavailable", vendor: null, rule: "content-encoding" };
  111. return { label: "candidate-page", vendor: null, rule: "status" };
  112. }
  113. /** Pass 1's rule set, unchanged, on the whole body, so its labels stay comparable. Each rule is a plain alternation. */
  114. export function classifyRegistered({ status, headers = {}, body = "" }) {
  115. const h = lower(headers);
  116. for (const [name, re, vendor] of CHALLENGE_HEADERS) {
  117. if (re.test(String(h[name] ?? ""))) return { label: "challenge", vendor, rule: `header:${name}` };
  118. }
  119. const text = String(body);
  120. for (const [re, vendor] of CHALLENGE_BODIES) if (re.test(text)) return { label: "challenge", vendor, rule: "body" };
  121. for (const [re, vendor] of BLOCK_BODIES) if (re.test(text)) return { label: "block", vendor, rule: "body" };
  122. if (status >= 400) return { label: "origin-error", vendor: null, rule: "status" };
  123. return { label: "page", vendor: null, rule: "status" };
  124. }
  125. const matches = (rules, text) => Object.fromEntries(rules.map(([re, vendor]) => [vendor, re.test(text)]));
  126. /** Every signature, for every rule set, as a boolean, so a label can be replayed under other rules. */
  127. export function signatureMatches(body) {
  128. const whole = String(body ?? "");
  129. const text = whole.slice(0, INPUT_CAP_CHARS);
  130. return {
  131. registeredChallenge: matches(CHALLENGE_BODIES, whole), registeredBlock: matches(BLOCK_BODIES, whole),
  132. hardChallenge: matches(HARD_CHALLENGE_BODIES, text), softChallenge: matches(SOFT_CHALLENGE_BODIES, text), block: matches(BLOCK_BODIES_V2, text),
  133. cloudfrontError: CLOUDFRONT_ERROR.test(text), cloudflareCode: CF_ERROR.exec(text)?.[1] ?? null,
  134. };
  135. }
  136. /**
  137. * What a value test of the CSP needs, from the full header, not the grader's
  138. * 160-character evidence. Directive names only. `controlsScripts`: a policy has
  139. * `script-src`, `script-src-elem` or `default-src`. `openScripts`: that source
  140. * list still allows inline code without a nonce or hash, or any host of a scheme.
  141. */
  142. export function cspEvidence(headers = {}) {
  143. const raw = [].concat(lower(headers)["content-security-policy"] ?? []).join(",").slice(0, INPUT_CAP_CHARS);
  144. const policies = raw.split(",").map((p) => new Map(p.split(";").map((d) => d.trim().split(/\s+/)).filter(([n]) => /^[a-z-]{1,64}$/i.test(n ?? "")).map(([n, ...v]) => [n.toLowerCase(), v.map((x) => x.toLowerCase())])));
  145. const script = policies.map((p) => p.get("script-src") ?? p.get("script-src-elem") ?? p.get("default-src")).filter(Boolean);
  146. const open = (v) => v.some((s) => ["*", "http:", "https:", "data:"].includes(s)) || (v.includes("'unsafe-inline'") && !v.some((s) => /^'(?:nonce-|sha(?:256|384|512)-)/.test(s)));
  147. return {
  148. present: raw.trim().length > 0, chars: raw.length,
  149. directives: [...new Set(policies.flatMap((p) => [...p.keys()]))].sort().slice(0, 40),
  150. controlsScripts: script.length > 0,
  151. openScripts: script.length > 0 && script.every(open),
  152. };
  153. }
  154. /** Cookie names only, never values. */
  155. export function cookieNames(headers = {}) {
  156. const raw = lower(headers)["set-cookie"];
  157. const list = Array.isArray(raw) ? raw : raw ? [raw] : [];
  158. // No = before the first ; means there is no name, and what stands there may be a value.
  159. // A name that looks like a token, an address or anything but a name isn't kept.
  160. const name = (c) => { const pair = String(c).split(";")[0]; return pair.includes("=") ? pair.split("=")[0].trim() : ""; };
  161. return list.map(name).filter((n) => /^[\w.-]{1,64}$/.test(n) && scrub(n) === n).sort();
  162. }
  163. // § 6, secondary: what kind of site this is, from the homepage already fetched.
  164. // Fixed rules on the HTML; only the booleans are kept, never a URL or a name.
  165. export const TRAIT_RULES = Object.freeze({
  166. login: /<a\b[^>]{0,2000}>\s{0,200}(?:<[^>]{1,500}>\s{0,200}){0,8}(log\s?in|sign\s?in|my account|inloggen|anmelden|connexion)\b/i,
  167. payments: /js\.stripe\.com|checkout\.stripe\.com|paypal\.com\/sdk|paypalobjects\.com|js\.braintreegateway\.com|checkoutshopper-[a-z]+\.adyen\.com|klarna(?:cdn|\.com)|mollie\.com/i,
  168. shop: /cdn\.shopify\.com|woocommerce|magento|\/cart\b|\/checkout\b|add[-_ ]to[-_ ]cart/i,
  169. // A public security or trust page: a stand-in for a team that owns security, not proof of a CISO.
  170. securityPage: /<a\b[^>]{0,2000}href="[^"]{0,500}(?:\/security\b|\/trust\b|\/trust-center|\/\/trust\.|\/\/security\.)[^"]{0,500}"/i,
  171. });
  172. /** Booleans for one page. An empty page (no 2xx HTML) gives null: unknown, not "no". */
  173. export function traitsOf(html) {
  174. if (!html) return null;
  175. const text = String(html).slice(0, INPUT_CAP_CHARS);
  176. return Object.fromEntries(Object.entries(TRAIT_RULES).map(([k, re]) => [k, re.test(text)]));
  177. }
  178. // Every repeat below is bounded, so no input of INPUT_CAP_CHARS can make these slow.
  179. const ZERO_WIDTH = /[​-‍⁠­]/g;
  180. const NUMERIC_ENTITY = /&#(?:x0*([0-9a-f]{1,8})|0*(\d{1,8}));?/gi;
  181. const NAMED_ENTITY = /&(commat|period|lowbar|colon);?/gi;
  182. const ESCAPED = /\\(?:x|u00)([0-9a-f]{2})/gi;
  183. const PERCENT = /%(?:25)?([2-7][0-9a-f])/gi;
  184. // A bracketed at is an address, whatever stands around it: both neighbouring words go.
  185. const BRACKET_AT = /[^\s<>"']{0,64}\s{0,3}[[({]\s{0,2}at\s{0,2}[\])}]\s{0,3}[^\s<>"']{0,64}/gi;
  186. const AT_WORD = /\s{1,3}@\s{0,3}(?=[\p{L}\p{N}])|(?<=[\p{L}\p{N}\p{M}])\s{0,3}@\s{1,3}|[@﹫]/gu;
  187. const TAGGED_AT = /(?:<\/?[a-z]{1,8}>\s{0,2}){1,4}@(?:\s{0,2}<\/?[a-z]{1,8}>){1,4}/gi;
  188. const DASH_AT = /([\p{L}\p{N}+]{1,64})[-_.]at[-_.]([\p{L}\p{N}-]{1,63}\.\p{L}{2,24})\b/giu;
  189. const DOT_WORD = /\s{0,3}[[({]\s{0,2}(?:dot|\.)\s{0,2}[\])}]\s{0,3}/gi;
  190. // A bare `at` counts only before something with a dot, spaced or not.
  191. const SPELLED_EMAIL = /([\p{L}\p{N}._%+-]{1,64})\s{1,3}at\s{1,3}([\p{L}\p{N}_-]{1,63})(?:\s{1,3}dot\s{1,3}|\s{0,3}\.\s{0,3})(\p{L}{2,24})\b/giu;
  192. const CF_EMAIL = /(data-cfemail=["']?)[0-9a-f]{2,512}|(\/cdn-cgi\/l\/email-protection#)[0-9a-f]{2,512}/gi;
  193. // No TLD needed: a cut address has none, and an @ inside a word is no loss.
  194. const EMAIL = /[\p{L}\p{M}\p{N}._%+'\uFFFD-]{1,64}@[\p{L}\p{M}\p{N}_\uFFFD-]{1,63}(?:\.[\p{L}\p{M}\p{N}_-]{1,63}){0,6}/gu;
  195. const QUOTED_EMAIL = /"[^"\n]{1,64}"@[\p{L}\p{N}-]{1,63}(?:\.[\p{L}\p{N}-]{1,63}){0,6}/gu;
  196. const IPV6_RUN = /(?<!:)(?:[0-9a-f]{0,4}:){2,8}(?:\d{1,3}(?:\.\d{1,3}){3}|[0-9a-f]{0,4})(?![0-9a-f])/gi;
  197. const IPV4 = /(?<!\d)(?:\d{1,3}\.){3}\d{1,3}(?!\d)(?!\.\d)/g;
  198. const IPV4_DASHED = /(?<!\d)\d{1,3}-\d{1,3}-\d{1,3}-\d{1,3}(?![\d-])/g;
  199. const IPV4_HEX = /\b0x[0-9a-f]{8}\b/gi;
  200. const QUERY = /\?[^\s"'<>)\\#?]{1,4096}/gu;
  201. const FRAGMENT = /(?<=[\w/.-])#[^\s"'<>)\\]{1,4096}/gu;
  202. const JWT = /eyJ[\w-]{6,4096}\.[\w-]{2,4096}(?:\.[\w-]{0,4096})?/g;
  203. const BEARER = /\b(bearer|basic|token)\s{1,3}[\w.~+/=-]{8,4096}/gi;
  204. const UUID = /\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi;
  205. const NAMED = /\b([\w-]{0,32}(?:token|session[_-]?id|sessionid|sessid|session|sess|sid|csrf|xsrf|auth|secret|nonce|password|passwd|api[_-]?key|access[_-]?key|[_-]pass|[_-]pwd)|pass|pwd)(["']?\s{0,3}[:=]\s{0,3}["']?)[^\s"'<>&;,)]{3,4096}/gi;
  206. const TOKEN_RUN = /[A-Za-z0-9_-]{16,4096}/g;
  207. const LONG_NUMBER = /\+?\d[\d\s().-]{7,40}\d/g;
  208. const digits = (s) => s.replace(/\D/g, "").length;
  209. const isToken = (s) => /^[0-9a-f]{16,}$/i.test(s) || (digits(s) >= 3 && /[A-Za-z]/.test(s)) || (s.length >= 20 && /\d/.test(s) && /[a-z]/.test(s) && /[A-Z]/.test(s));
  210. // A run can start with one stray label: `IP:` or the `ip6:` of an SPF record.
  211. /** One rule for the scrub and its detector. `a::b` is C++: with no digit, an address needs some length. */
  212. export const looksIpv6 = (s) => isIPv6(s) && (/\d/.test(s) || s.replace(/:/g, "").length >= 6);
  213. const ipv6Out = (m) => (looksIpv6(m) || (!m.startsWith("::") && looksIpv6(m.slice(m.indexOf(":") + 1))) ? "<ip>" : m);
  214. const printable = (code, m) => (code >= 0x20 && code < 0x7f ? String.fromCharCode(code) : m);
  215. const NAMED_CHARS = Object.freeze({ commat: "@", period: ".", lowbar: "_", colon: ":" });
  216. const safeChar = (m, hex) => { const ch = String.fromCharCode(parseInt(hex, 16)); return /[\w@.:-]/.test(ch) ? ch : m; };
  217. /** Every way a page writes an address, back to plain text, so one pattern can see it. */
  218. const decodeForms = (s) => s
  219. .replace(ZERO_WIDTH, "")
  220. .replace(/&amp;(?=#)/gi, "&")
  221. .replace(NUMERIC_ENTITY, (m, hex, dec) => printable(hex ? parseInt(hex, 16) : Number(dec), m))
  222. .replace(NAMED_ENTITY, (_m, name) => NAMED_CHARS[name.toLowerCase()])
  223. .replace(ESCAPED, safeChar).replace(/\\100/g, "@").replace(PERCENT, safeChar)
  224. .replace(TAGGED_AT, "@").replace(DOT_WORD, ".")
  225. .replace(BRACKET_AT, "<email>").replace(AT_WORD, "@")
  226. .replace(SPELLED_EMAIL, "$1@$2.$3")
  227. .replace(DASH_AT, "$1@$2");
  228. /** ASCII lower case, one for one. `toLowerCase` makes U+0130 two units, which shifts every index after it. */
  229. const asciiLower = (s) => s.replace(/[A-Z]+/g, (m) => m.toLowerCase());
  230. /** The end of a tag, quotes respected, or the end of the text. */
  231. function tagEnd(text, from) {
  232. let quote = "";
  233. for (let i = from; i < text.length; i++) {
  234. const ch = text[i];
  235. if (quote) { if (ch === quote) quote = ""; } else if (ch === '"' || ch === "'") quote = ch;
  236. else if (ch === ">") return i + 1;
  237. }
  238. return text.length;
  239. }
  240. /** Form fields out, whatever their size: a field's value can be a token or a person's details. */
  241. function stripFields(text) {
  242. const low = asciiLower(text);
  243. const field = /<(input|textarea|select)\b/g;
  244. let out = "", at = 0, m;
  245. while ((m = field.exec(low))) {
  246. out += `${text.slice(at, m.index)}<input>`;
  247. const close = m[1] === "input" ? -1 : low.indexOf(`</${m[1]}`, m.index);
  248. at = m[1] === "input" ? tagEnd(text, m.index) : close < 0 ? text.length : tagEnd(text, close);
  249. field.lastIndex = at;
  250. }
  251. return out + text.slice(at);
  252. }
  253. /** The cap, ending at a space: what the cut split is never kept. */
  254. function capped(text) {
  255. if (text.length <= INPUT_CAP_CHARS) return text;
  256. let end = INPUT_CAP_CHARS;
  257. while (end > 0 && !/\s/.test(text[end - 1])) end--;
  258. return text.slice(0, end);
  259. }
  260. /** IP addresses and emails out. */
  261. export const scrubAddresses = (text) => decodeForms(String(text ?? "")).replace(QUOTED_EMAIL, "<email>").replace(EMAIL, "<email>").replace(IPV6_RUN, ipv6Out).replace(IPV4, "<ip>").replace(IPV4_DASHED, "<ip>").replace(IPV4_HEX, "<ip>");
  262. /** For anything stored: addresses, form fields, query strings, fragments, tokens and long numbers. Run it before slicing. */
  263. export const scrub = (text) => scrubAddresses(stripFields(capped(String(text ?? ""))).replace(CF_EMAIL, (_m, a, b) => `${a ?? b}<email>`))
  264. .replace(QUERY, "?<query>").replace(FRAGMENT, "#<fragment>")
  265. .replace(JWT, "<token>").replace(BEARER, "$1 <token>").replace(UUID, "<id>")
  266. .replace(NAMED, "$1$2<token>")
  267. .replace(TOKEN_RUN, (m) => (isToken(m) ? "<token>" : m))
  268. .replace(LONG_NUMBER, (m) => (digits(m) >= 9 && !/^\d{4}-\d{2}-\d{2}/.test(m) ? "<number>" : m));
  269. /**
  270. * What should never survive a scrub. This walks tokens with its own tests and
  271. * shares no pattern with `scrub`, so one regex fault can't hide from both.
  272. */
  273. export function scrubMisses(text) {
  274. const out = { ipv4: 0, ipv6: 0, email: 0, query: 0, fragment: 0, jwt: 0, longHex: 0, longNumber: 0 };
  275. const whole = String(text ?? "");
  276. const low = whole.toLowerCase();
  277. for (const mark of ["&commat", "\\x40", "\\u0040", "\\100", "%40", "%2540", "@", "﹫"]) if (low.includes(mark)) out.email++;
  278. // A bracketed at, with or without inner spaces, in its own walk over the text.
  279. for (let i = 0; i < low.length; i++) {
  280. if (!"[({".includes(low[i])) continue;
  281. let k = i + 1;
  282. while (low[k] === " ") k++;
  283. if (low.startsWith("at", k)) {
  284. k += 2;
  285. while (low[k] === " ") k++;
  286. if ("])}".includes(low[k] ?? "?")) out.email++;
  287. }
  288. }
  289. // A bare `at` before something with a dot.
  290. const words = low.split(/\s+/);
  291. for (let i = 1; i < words.length; i++) {
  292. if (words[i] !== "at" || !/[\p{L}\p{N}._%+-]$/u.test(words[i - 1])) continue;
  293. if (/^[^\s.]+(?:\s*\.\s*|\s+dot\s+)[a-z]{2,}\b/.test(words.slice(i + 1, i + 5).join(" "))) out.email++;
  294. }
  295. // Ordinary pages hold many entities for quotes and dashes. Only ones that spell ASCII hide text.
  296. let spelled = 0;
  297. for (let i = low.indexOf("&#"); i >= 0; i = low.indexOf("&#", i + 2)) {
  298. const hex = low[i + 2] === "x";
  299. let end = i + (hex ? 3 : 2);
  300. while (end < low.length && end < i + 16 && (hex ? "0123456789abcdef" : "0123456789").includes(low[end])) end++;
  301. const code = Number.parseInt(low.slice(i + (hex ? 3 : 2), end), hex ? 16 : 10);
  302. if (code === 0x40) out.email++;
  303. if (code >= 0x20 && code < 0x7f) spelled++;
  304. }
  305. if (spelled >= 4) out.email++;
  306. const edge = new Set(`.,;:!?()[]{}<>"'=\``);
  307. const trim = (t) => {
  308. let a = 0, b = t.length;
  309. while (a < b && edge.has(t[a])) a++;
  310. while (b > a && edge.has(t[b - 1])) b--;
  311. return t.slice(a, b);
  312. };
  313. for (const raw of whole.split(/[\s"'`<>()\[\],;\\]+/)) {
  314. if (raw === "redacted" || raw === "query" || raw === "fragment") continue;
  315. const t = trim(raw);
  316. if (!t) continue;
  317. for (const piece of [raw, t, ...t.split("="), t.slice(t.indexOf(":") + 1)]) {
  318. const candidate = isIP(piece) ? piece : trim(piece);
  319. const v = isIP(candidate);
  320. if (v === 4) { out.ipv4++; break; }
  321. if (v === 6 && looksIpv6(candidate)) { out.ipv6++; break; }
  322. }
  323. // The @ needs an address on both sides: `}@media` in a stylesheet and `pkg@^1` on a CDN have none.
  324. const wordChar = (ch) => ch.toLowerCase() !== ch.toUpperCase() || (ch >= "0" && ch <= "9");
  325. const localEnd = (ch) => wordChar(ch) || "._+-%'".includes(ch);
  326. if (t.split(/[/?&=:]/).some((piece) => { const at = piece.indexOf("@"); return at > 0 && at < piece.length - 1 && localEnd(piece[at - 1]) && wordChar(piece[at + 1]); })) out.email++;
  327. const eq = t.indexOf("=");
  328. const q = t.indexOf("?"), hash = t.indexOf("#");
  329. if (q >= 0 && eq > q + 1) out.query++;
  330. if (hash >= 0 && eq > hash + 1) out.fragment++;
  331. if (t.includes("eyJ") && t.split(".").length >= 2) out.jwt++;
  332. let run = 0, numerals = 0;
  333. for (const ch of t.toLowerCase()) {
  334. run = "0123456789abcdef".includes(ch) ? run + 1 : 0;
  335. if (ch >= "0" && ch <= "9") numerals++;
  336. if (run === 32) { out.longHex++; }
  337. }
  338. if (numerals >= 12 && numerals === t.length) out.longNumber++;
  339. }
  340. return out;
  341. }
  342. const headerEvidence = (res) => {
  343. const h = lower(res.headers);
  344. return {
  345. status: res.status,
  346. headerNames: Object.keys(h).filter((n) => /^[\w-]{1,64}$/.test(n) && scrub(n) === n).sort(),
  347. values: Object.fromEntries(KEPT_VALUES.filter((k) => h[k] !== undefined).map((k) => [k, scrub(h[k]).slice(0, 120)])),
  348. cookieNames: cookieNames(h),
  349. };
  350. };
  351. const sha256 = (s) => createHash("sha256").update(s).digest("hex");
  352. const outside2xx = (status) => status < 200 || status >= 300;
  353. const SKIPPED_TAGS = Object.freeze(["script", "style", "noscript", "svg", "title", "template", "textarea", "select"]);
  354. /** The text outside markup, in one pass. An unclosed comment or script runs to the end of the input. */
  355. function visibleText(html) {
  356. const low = asciiLower(html);
  357. let out = "";
  358. let i = 0;
  359. while (i < html.length) {
  360. const lt = low.indexOf("<", i);
  361. if (lt < 0) { out += html.slice(i); break; }
  362. out += `${html.slice(i, lt)} `;
  363. if (low.startsWith("<!--", lt)) {
  364. const end = low.indexOf("-->", lt + 4);
  365. if (end < 0) break;
  366. i = end + 3;
  367. continue;
  368. }
  369. const tag = SKIPPED_TAGS.find((t) => low.startsWith(`<${t}`, lt) && !/[a-z0-9-]/.test(low[lt + t.length + 1] ?? " "));
  370. if (tag) {
  371. const end = low.indexOf(`</${tag}`, lt);
  372. const close = end < 0 ? -1 : low.indexOf(">", end);
  373. if (close < 0) break;
  374. i = close + 1;
  375. continue;
  376. }
  377. const gt = low.indexOf(">", lt);
  378. if (gt < 0) break;
  379. i = gt + 1;
  380. }
  381. return out;
  382. }
  383. function between(html, low, tag) {
  384. const open = low.indexOf(`<${tag}`);
  385. const start = open < 0 ? -1 : low.indexOf(">", open);
  386. const end = start < 0 ? -1 : low.indexOf(`</${tag}`, start);
  387. return end < 0 ? "" : html.slice(start + 1, end);
  388. }
  389. /**
  390. * The visible text of a 2xx response and its shape, enough to tell a page from
  391. * a JS challenge, a soft block or a custom interstitial. No markup is kept.
  392. */
  393. export function pageExcerpt(body) {
  394. const whole = String(body ?? "");
  395. const text = stripFields(capped(whole));
  396. const low = asciiLower(text);
  397. const flat = (s) => visibleText(s).replace(/&(?:nbsp|amp|quot|lt|gt|#39|#x27);/gi, " ").replace(/\s+/g, " ").trim();
  398. return {
  399. title: scrub(flat(between(text, low, "title"))).slice(0, 160),
  400. text: scrub(flat(visibleText(text))).slice(0, PAGE_TEXT_CHARS),
  401. noscript: scrub(flat(between(text, low, "noscript"))).slice(0, 300),
  402. chars: whole.length,
  403. scripts: text.match(/<script\b/gi)?.length ?? 0,
  404. forms: text.match(/<form\b/gi)?.length ?? 0,
  405. links: text.match(/<a\b/gi)?.length ?? 0,
  406. metaRefresh: /<meta\b[^>]{0,500}http-equiv=["']?refresh/i.test(text),
  407. soft: SOFT_CHALLENGE_BODIES.filter(([re]) => re.test(text)).map(([, vendor]) => vendor),
  408. hard: HARD_CHALLENGE_BODIES.filter(([re]) => re.test(text)).map(([, vendor]) => vendor),
  409. };
  410. }
  411. /**
  412. * What E20-R stores for one site (§ 4.1, Amendment 2). Each of the three
  413. * responses gets a body hash. Outside 2xx the first 4 KB is kept, scrubbed. A
  414. * 2xx homepage keeps only `pageExcerpt`.
  415. */
  416. export function evidenceOf(page, responses = []) {
  417. const body = String(page.body ?? "");
  418. const other = ({ kind, res }) => ({
  419. kind, ...headerEvidence(res),
  420. bodySha256: sha256(String(res.body ?? "")),
  421. ...(outside2xx(res.status) ? { excerpt: scrub(res.body).slice(0, EXCERPT_BYTES) } : {}),
  422. });
  423. return {
  424. ...headerEvidence(page),
  425. bodySha256: sha256(body),
  426. ...(outside2xx(page.status) ? { excerpt: scrub(body).slice(0, EXCERPT_BYTES) } : { pageExcerpt: pageExcerpt(body) }),
  427. class: classify(page),
  428. classRegistered: classifyRegistered(page),
  429. signatures: signatureMatches(body),
  430. csp: cspEvidence(page.headers),
  431. traits: traitsOf(page.html),
  432. others: responses.map(other),
  433. };
  434. }
install.sh · Bundled script · 83 lines · 1 finding marked
  1. #!/usr/bin/env bash
  2. #
  3. # Penthera installer — clones the repo, links the CLI, and installs the
  4. # Agent Skill into Cursor and Claude Code if it finds them.
  5. #
  6. # curl -fsSL https://raw.githubusercontent.com/danoszz/penthera/main/install.sh | bash
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 6 on GitHub
  7. #
  8. # This script is intentionally short and readable — review it before running.
  9. # It uses no sudo, makes no network calls beyond `git clone` + `npm install`,
  10. # and only writes to ~/.penthera and your agent skill folders.
  11. #
  12. set -euo pipefail
  13. REPO="https://github.com/danoszz/penthera.git"
  14. DIR="${PENTHERA_DIR:-$HOME/.penthera}"
  15. BLUE='\033[1;38;2;0;0;237m'; DIM='\033[2m'; GREEN='\033[32m'; YELLOW='\033[33m'; RED='\033[31m'; OFF='\033[0m'
  16. say() { printf "%b\n" "$1"; }
  17. ok() { printf " %b+%b %s\n" "$GREEN" "$OFF" "$1"; }
  18. warn() { printf " %b!%b %s\n" "$YELLOW" "$OFF" "$1"; }
  19. die() { printf " %bx%b %s\n" "$RED" "$OFF" "$1" >&2; exit 1; }
  20. say ""
  21. say " ${BLUE}P E N T H E R A${OFF} ${DIM}installer${OFF}"
  22. say ""
  23. # ── Prerequisites ───────────────────────────────────────────────────────────
  24. command -v git >/dev/null 2>&1 || die "git is required. Install it and re-run."
  25. command -v node >/dev/null 2>&1 || die "Node.js 22+ is required: https://nodejs.org"
  26. NODE_MAJOR="$(node -p 'process.versions.node.split(".")[0]')"
  27. [ "$NODE_MAJOR" -ge 22 ] || die "Node 22+ required (found $(node -v))."
  28. ok "Node $(node -v)"
  29. # ── Clone or update ─────────────────────────────────────────────────────────
  30. if [ -d "$DIR/.git" ]; then
  31. git -C "$DIR" pull --ff-only --quiet && ok "Updated $DIR"
  32. else
  33. git clone --depth 1 --quiet "$REPO" "$DIR" && ok "Cloned to $DIR"
  34. fi
  35. # ── Install dependencies + link CLI ─────────────────────────────────────────
  36. cd "$DIR"
  37. # Always from the lockfile: `npm ci` installs exactly the reviewed tree and
  38. # checks every tarball against its recorded integrity hash.
  39. [ -f package-lock.json ] || die "package-lock.json missing in $DIR; refusing an unpinned install."
  40. npm ci --omit=dev --silent
  41. ok "Installed dependencies"
  42. if npm link >/dev/null 2>&1; then
  43. ok "Linked CLI — run: penthera --version"
  44. else
  45. warn "Could not link globally (permission?). Run from $DIR: node bin/penthera.js"
  46. fi
  47. # ── Install the Agent Skill into detected agents ────────────────────────────
  48. install_skill() {
  49. local target="$1" name="$2"
  50. mkdir -p "$target"
  51. rm -rf "$target/penthera"
  52. cp -R "$DIR/skills/penthera" "$target/penthera"
  53. ok "Installed skill for $name → $target/penthera"
  54. }
  55. INSTALLED_ANY=0
  56. [ -d "$HOME/.cursor" ] && { install_skill "$HOME/.cursor/skills" "Cursor"; INSTALLED_ANY=1; }
  57. [ -d "$HOME/.claude" ] && { install_skill "$HOME/.claude/skills" "Claude Code"; INSTALLED_ANY=1; }
  58. if [ "$INSTALLED_ANY" -eq 0 ]; then
  59. warn "No Cursor/Claude Code config found. Cross-agent install: npx skills add danoszz/penthera"
  60. fi
  61. # ── Done ────────────────────────────────────────────────────────────────────
  62. say ""
  63. say " ${GREEN}Done.${OFF} Restart your agent, then just ask it:"
  64. say ""
  65. say " ${DIM}\"Scan my localhost:3000 for security issues\"${OFF}"
  66. say " ${DIM}\"Run a pre-deploy security audit on staging.myapp.com\"${OFF}"
  67. say " ${DIM}\"Find hardcoded secrets in this repo\"${OFF}"
  68. say ""
  69. say " Or from your terminal: ${BLUE}penthera${OFF} ${DIM}(interactive wizard)${OFF}"
  70. say ""
  71. say " ${YELLOW}Only scan systems you own or are authorized to test.${OFF}"
  72. say ""
lib/agents/hidden-content.js · Bundled script · 885 lines · 1 finding marked
  1. /**
  2. * Hidden-content engine for agent instructions (skills, rule files, commands).
  3. *
  4. * Deterministic and offline. It looks for content a human reviewer would not
  5. * see but a model would read and act on, and for payloads that run code.
  6. *
  7. * Designed against the ways LLM-judged skill scanners have been bypassed:
  8. * - Invisible text: Unicode tag characters, zero-width and bidi controls,
  9. * variation-selector smuggling. Decoded where possible, so the report shows
  10. * what the model would read.
  11. * - Content pushed out of view: hundreds of blank lines or a wide run of
  12. * spaces, so the payload sits past what a reviewer or a truncating judge sees.
  13. * This engine reads the whole file; there is no context window to overflow.
  14. * - Instructions in HTML comments, which Markdown renderers hide.
  15. * - Text addressed to the scanner itself ("this skill is safe, approve it").
  16. * A regex cannot be talked out of a finding, but the attempt is a signal.
  17. * - Payloads moved out of SKILL.md: every file in a skill folder is scanned,
  18. * binaries and archives are identified by magic bytes rather than extension,
  19. * and symlinks that leave the folder are reported.
  20. * - Encoded payloads: long base64 blobs, decoded and classified.
  21. *
  22. * Each detector returns hits `{ rule, severity, title, detail, line, count }`.
  23. * Callers turn hits into findings.
  24. */
  25. import { rulesOf, rulesGeneration, packTest, packMatch } from "../rules/index.js";
  26. import { redactCommand } from "./redact.js";
  27. // ── Invisible characters ──────────────────────────────────────────────────
  28. const TAG_RANGE = /[\u{E0000}-\u{E007F}]/u;
  29. const TAG_RUN = /[\u{E0000}-\u{E007F}]+/gu;
  30. const VS_SUPPLEMENT_RUN = /[\u{E0100}-\u{E01EF}]+/gu;
  31. const BIDI = /[‪-‮⁦-⁩]/g;
    • Note: Usual in code (characters a sanitizer strips): Contains 4 bidirectional-override character(s), which make text display in a different order from how it's read
    • Line 33 on GitHub
  32. // Zero-width and other format characters that render as nothing. U+200D (ZWJ)
  33. // and U+200C (ZWNJ) have legitimate uses (emoji sequences, Persian, Indic
  34. // scripts), so those are filtered by context below.
  35. const ZERO_WIDTH = /[​-‍⁠-⁤᠎ㅤᅟᅠᅠ]/g;
  36. const PICTO = /\p{Extended_Pictographic}|[\u{1F3FB}-\u{1F3FF}️]/u;
  37. const NON_LATIN_LETTER = /[\p{L}\p{M}]/u;
  38. function lineOf(text, index) {
  39. let n = 1;
  40. for (let i = 0; i < index && i < text.length; i++) if (text.charCodeAt(i) === 10) n++;
  41. return n;
  42. }
  43. /** Decode Unicode tag characters to the ASCII they smuggle. */
  44. export function decodeTagChars(s) {
  45. let out = "";
  46. for (const ch of s) {
  47. const cp = ch.codePointAt(0);
  48. if (cp >= 0xe0020 && cp <= 0xe007e) out += String.fromCharCode(cp - 0xe0000);
  49. }
  50. return out;
  51. }
  52. /** Make a decoded payload safe to print: visible ASCII only, truncated. */
  53. export function preview(s, max = 80) {
  54. const clean = String(s).replace(/[^\x20-\x7E]/g, "·").replace(/\s+/g, " ").trim();
  55. return clean.length > max ? clean.slice(0, max) + "…" : clean;
  56. }
  57. /**
  58. * Text from a scanned file as a finding quotes it: credentials hidden
  59. * (redactCommand), then made safe and shortened (preview). `context` is the
  60. * text it was cut from; a secret there is hidden in the quote too. Only the
  61. * first 4 × max characters are redacted: the rest is cut anyway.
  62. */
  63. export function quote(text, max = 80, context) {
  64. const s = String(text ?? "");
  65. const part = s.length > max * 4 ? s.slice(0, max * 4) : s;
  66. return preview(redactCommand(part, context ?? s), max);
  67. }
  68. /** The line around `text[index, index + length)`, at most 8 KiB either side: the context of a quoted match. */
  69. function lineAround(text, index, length) {
  70. const from = Math.max(text.lastIndexOf("\n", index) + 1, index - 8192);
  71. let to = text.indexOf("\n", index + length);
  72. if (to < 0) to = text.length;
  73. return text.slice(from, Math.min(to, index + length + 8192));
  74. }
  75. function detectInvisible(text) {
  76. const hits = [];
  77. const tagRuns = [...text.matchAll(TAG_RUN)];
  78. if (tagRuns.length) {
  79. const decoded = tagRuns.map((m) => decodeTagChars(m[0])).join(" ").trim();
  80. const count = tagRuns.reduce((n, m) => n + [...m[0]].length, 0);
  81. hits.push({
  82. rule: "unicode-tags",
  83. severity: "critical",
  84. title: "Hidden text in Unicode tag characters",
  85. detail: `${count} invisible tag character(s) (U+E0000–E007F). A model reads them as text; a reviewer sees nothing.` +
  86. (decoded ? ` Decoded: "${quote(decoded)}"` : ""),
  87. line: lineOf(text, tagRuns[0].index),
  88. count,
  89. });
  90. }
  91. const vsRuns = [...text.matchAll(VS_SUPPLEMENT_RUN)];
  92. if (vsRuns.length) {
  93. const count = vsRuns.reduce((n, m) => n + [...m[0]].length, 0);
  94. hits.push({
  95. rule: "variation-selectors",
  96. severity: "high",
  97. title: "Data hidden in variation selectors",
  98. detail: `${count} supplementary variation selector(s) (U+E0100–E01EF). These are invisible and can encode arbitrary bytes ("emoji smuggling").`,
  99. line: lineOf(text, vsRuns[0].index),
  100. count,
  101. });
  102. }
  103. const bidi = [...text.matchAll(BIDI)];
  104. if (bidi.length) {
  105. hits.push({
  106. rule: "bidi-controls",
  107. severity: "high",
  108. title: "Bidirectional control characters",
  109. detail: `${bidi.length} bidi override/isolate character(s) (U+202A–202E, U+2066–2069). They reorder how text displays, so what a reviewer reads differs from what the model reads (Trojan Source).`,
  110. line: lineOf(text, bidi[0].index),
  111. count: bidi.length,
  112. });
  113. }
  114. const zw = [];
  115. for (const m of text.matchAll(ZERO_WIDTH)) {
  116. const ch = m[0];
  117. const i = m.index;
  118. if (ch === "" && i === 0) continue; // byte-order mark
  119. // A zero-width space at the start of a line before ``` or ~~~ is the usual
  120. // trick for showing a code fence inside a fenced block.
  121. if (ch === "​" && (i === 0 || text[i - 1] === "\n") && /^(?:```|~~~)/.test(text.slice(i + 1, i + 4))) continue;
  122. const prev = text.slice(Math.max(0, i - 2), i);
  123. const next = text.slice(i + 1, i + 3);
  124. if (ch === "‍" && PICTO.test(prev) && PICTO.test(next)) continue; // emoji ZWJ sequence
  125. if (ch === "‌" || ch === "‍") {
  126. const p = [...prev].pop() || "";
  127. const n = [...next][0] || "";
  128. // ZWNJ/ZWJ between letters of a non-Latin script is ordinary orthography.
  129. if (NON_LATIN_LETTER.test(p) && NON_LATIN_LETTER.test(n) && /[^\x00-ɏ]/.test(p + n)) continue;
  130. }
  131. zw.push(i);
  132. }
  133. if (zw.length) {
  134. hits.push({
  135. rule: "zero-width",
  136. severity: zw.length >= 3 ? "high" : "medium",
  137. title: "Zero-width or invisible characters",
  138. detail: `${zw.length} zero-width/invisible character(s). They can split keywords to dodge filters or carry a hidden binary encoding.`,
  139. line: lineOf(text, zw[0]),
  140. count: zw.length,
  141. });
  142. }
  143. return hits;
  144. }
  145. // ── Whitespace inflation ──────────────────────────────────────────────────
  146. export const DEFAULTS = {
  147. blankLines: 40, // consecutive blank lines before more content
  148. horizontalRun: 200, // consecutive spaces/tabs on one line before more content
  149. base64MinLength: 200, // characters in one base64 run
  150. };
  151. // Linear scans, not regexes: a nested whitespace quantifier backtracks
  152. // quadratically on exactly the input an attacker would write here.
  153. function detectWhitespaceInflation(text, opts) {
  154. const hits = [];
  155. const lines = text.split("\n");
  156. let run = 0;
  157. let longestV = 0;
  158. let firstV = null;
  159. let countV = 0;
  160. let longestH = 0;
  161. let firstH = null;
  162. let countH = 0;
  163. for (let i = 0; i < lines.length; i++) {
  164. const ln = lines[i];
  165. if (/^[ \t\r]*$/.test(ln)) { run++; continue; }
  166. if (run >= opts.blankLines) {
  167. countV++;
  168. longestV = Math.max(longestV, run);
  169. if (firstV === null) firstV = i + 1;
  170. }
  171. run = 0;
  172. let ws = 0;
  173. for (let j = 0; j < ln.length; j++) {
  174. const ch = ln.charCodeAt(j);
  175. if (ch === 32 || ch === 9 || ch === 0xa0) { ws++; continue; }
  176. if (ws >= opts.horizontalRun && ch !== 13) {
  177. countH++;
  178. longestH = Math.max(longestH, ws);
  179. if (firstH === null) firstH = i + 1;
  180. }
  181. ws = 0;
  182. }
  183. }
  184. if (countV) {
  185. hits.push({
  186. rule: "whitespace-vertical",
  187. severity: "high",
  188. title: "Content hidden below a wall of blank lines",
  189. detail: `${longestV} consecutive blank lines, then more content. This pushes text past where a reviewer stops scrolling, and past what a truncating scanner reads.`,
  190. line: firstV,
  191. count: countV,
  192. });
  193. }
  194. if (countH) {
  195. hits.push({
  196. rule: "whitespace-horizontal",
  197. severity: "high",
  198. title: "Content hidden off-screen to the right",
  199. detail: `A run of ${longestH} spaces, then more text on the same line. Editors and diff views without wrapping never show it.`,
  200. line: firstH,
  201. count: countH,
  202. });
  203. }
  204. return hits;
  205. }
  206. // ── Instructions in hidden places ─────────────────────────────────────────
  207. export const INSTRUCTION_PATTERNS = [
  208. /\b(?:ignore|disregard|forget|override)\b[^.\n]{0,40}\b(?:previous|prior|above|earlier|all|any|other)\b[^.\n]{0,20}\b(?:instructions?|rules?|prompts?|guidelines?|directions?)\b/i,
  209. /\b(?:ignore|disregard|forget)\b[^.\n]{0,30}\bthe user\b/i,
  210. /\b(?:you|assistant|claude|agent|model|ai|llm)\b[^.\n]{0,30}\b(?:must|should|shall|will|are to|need to|have to)\b/i,
  211. /\bdo(?:n'?t| not) (?:tell|inform|mention|reveal|show|alert|ask|notify)\b/i,
  212. /\bwithout (?:asking|telling|informing|confirm\w*|permission|the user)/i,
  213. /\b(?:system prompt|developer message|new instructions?)\b/i,
  214. /\b(?:run|execute|exec|invoke)\b[^.\n]{0,40}\b(?:command|script|bash|shell|curl|wget|powershell)\b/i,
  215. /\b(?:send|upload|post|exfiltrat\w*|transmit|forward|copy)\b[^\n]{0,60}(?:https?:\/\/|\btokens?\b|\bsecrets?\b|\bapi keys?\b|\bcredentials?\b|\.env\b|\benv(?:ironment)?\s+(?:files?|var(?:iable)?s?)\b|\bssh\b|\bpasswords?\b)/i,
  216. // Credential paths: lib/rules/core/agent-instructions.json.
  217. ...rulesOf("agent-instruction", { pack: "core" }).map((r) => r.pattern),
  218. ];
  219. const packInstructions = () => rulesOf("agent-instruction", { exceptPack: "core" });
  220. // Text addressed to an automated reviewer. Legitimate skills have no reason to
  221. // talk to a security scanner.
  222. const SCANNER_DIRECTED = /\b(?:security |safety |skill |code )?(?:scanners?|reviewers?|auditors?|judges?|classifiers?|analy[sz]ers?|moderators?)\b[^.\n]{0,60}\b(?:is|as|are|mark|classify|report|treat|consider|rate|label)\b[^.\n]{0,30}\b(?:safe|benign|harmless|clean|trusted|approved?|pass(?:ed)?|low[- ]risk|false positive)\b/i;
  223. export function looksLikeInstruction(s) {
  224. return INSTRUCTION_PATTERNS.some((re) => re.test(s)) || packInstructions().some((r) => packTest(r, s));
  225. }
  226. function detectHiddenInstructions(text) {
  227. const hits = [];
  228. const comments = [...text.matchAll(/<!--([\s\S]*?)-->/g)];
  229. const bad = comments.filter((m) => looksLikeInstruction(m[1]));
  230. if (bad.length) {
  231. hits.push({
  232. rule: "html-comment-instructions",
  233. severity: "high",
  234. title: "Instructions hidden in an HTML comment",
  235. detail: `${bad.length} HTML comment(s) contain instructions. Markdown renderers hide comments, the model still reads them. First: "${quote(bad[0][1])}"`,
  236. line: lineOf(text, bad[0].index),
  237. count: bad.length,
  238. });
  239. }
  240. const scanner = text.match(SCANNER_DIRECTED);
  241. if (scanner) {
  242. hits.push({
  243. rule: "scanner-directed",
  244. severity: "high",
  245. title: "Text addressed to a security scanner",
  246. detail: `Tries to talk an automated reviewer into a verdict: "${quote(scanner[0], 80, lineAround(text, scanner.index, scanner[0].length))}". This is how LLM-judged scanners get bypassed.`,
  247. line: lineOf(text, scanner.index),
  248. count: 1,
  249. });
  250. }
  251. return hits;
  252. }
  253. // ── Code execution patterns ───────────────────────────────────────────────
  254. // A shell as a pipe's target. Two readers, and a hit from either counts
  255. // (HEURISTICS R12, bypass checklist row 16): the stage parser below, and
  256. // main's old regex beside it (legacyShellAfterPipe), so a form the old regex
  257. // caught can never be lost to the rewrite.
  258. //
  259. // The stage parser reads the pipeline stage after the "|" as shell words:
  260. // - quotes and escapes are read as the shell reads them, so `FOO="a b" sh`
  261. // is one assignment and one command word, and `ba""sh` is bash;
  262. // - redirections and their targets (`>`, `>>`, `2>`, `&>`, `<`, `>|`,
  263. // `2>&1`, attached or spaced) are taken out of the words, so
  264. // `bash>/var/log/x` has the command word `bash`, not `x`;
  265. // - the stage ends at the next unquoted |, ;, &, ), backtick or line end
  266. // (a "\" at the end of a line continues it, as does a line that ends in
  267. // "|"), and a "(" or "{" that opens a group is passed over;
  268. // - wrappers that run the next word as a command (sudo, doas, env, nohup,
  269. // exec, command, builtin, time, nice, setsid, stdbuf, timeout), by name
  270. // or by any path, are skipped with their options, the values that take
  271. // one (`env -u NAME`, `sudo -u USER`, `nice -n 5`, `timeout 10`, …), and
  272. // every word that starts with "-" or has "=" in it;
  273. // - `env -S STRING` and `--split-string=STRING` run STRING, so its words
  274. // are read next, from a stack of positions in the strings already read:
  275. // no word is copied or spliced in, so repeated -S stays linear;
  276. // - the first word left decides: its name after the last "/" must be a
  277. // shell (sh, bash, zsh, dash, ksh, fish, csh, tcsh).
  278. // There is no cap on the words: padding past a cap would hide the shell.
  279. const SHELL_BASE = /^(?:ba|z|da|k|fi|c|tc)?sh\b/i;
  280. const wrapper = (short = "", long = [], positional = 0) => ({ short, long: new Set(long), positional });
  281. const WRAPPERS = {
  282. env: { ...wrapper("uCSP", ["--unset", "--chdir", "--split-string"]), split: true },
  283. sudo: wrapper("CDghpRrTtUu", ["--close-from", "--chdir", "--group", "--host", "--prompt", "--chroot", "--role", "--command-timeout", "--type", "--other-user", "--user"]),
  284. doas: wrapper("Cu"),
  285. nohup: wrapper(),
  286. exec: wrapper("a"),
  287. command: wrapper(),
  288. builtin: wrapper(),
  289. time: wrapper("fo", ["--format", "--output"]),
  290. nice: wrapper("n", ["--adjustment"]),
  291. setsid: wrapper(),
  292. stdbuf: wrapper("ioe", ["--input", "--output", "--error"]),
  293. timeout: wrapper("sk", ["--signal", "--kill-after"], 1),
  294. };
  295. const NO_OPTS = wrapper();
  296. const ASSIGNMENT = /^[A-Za-z_][A-Za-z0-9_]*=/;
  297. const isStageEnd = (c) => c === "|" || c === ";" || c === "&" || c === ")" || c === "`" || c === "\n";
  298. const isBlank = (c) => c === " " || c === "\t" || c === "\r" || c === "\f" || c === "\v";
  299. const isSpace = (c) => isBlank(c) || c === "\n";
  300. const isRedirect = (text, i) => text[i] === ">" || text[i] === "<" || (text[i] === "&" && text[i + 1] === ">");
  301. const unquote = (w) => { let a = 0, b = w.length; while (a < b && "'\"\\".includes(w[a])) a++; while (b > a && "'\"\\".includes(w[b - 1])) b--; return w.slice(a, b); };
  302. /**
  303. * One shell word from `i`, with quotes and escapes taken out: { lit, i, quoted }.
  304. * Stops at an unquoted blank, stage end or redirection. A quote with no end
  305. * runs to the end of the text. Each character is read once.
  306. */
  307. function readWord(text, i, prose = false) {
  308. let lit = "";
  309. let quoted = false;
  310. while (i < text.length) {
  311. const c = text[i];
  312. // The prose reading: an apostrophe inside a word ("don't") is not a quote. It
  313. // is dropped, as quote removal would drop it, so `b'a'sh` still reads as bash.
  314. if (prose && c === "'" && i > 0 && isLetter(text[i - 1]) && isLetter(text[i + 1] || "")) { quoted = true; i++; continue; }
  315. if (c === "\\") {
  316. if (text[i + 1] === "\n") { i += 2; continue; }
  317. if (i + 1 < text.length) lit += text[i + 1];
  318. quoted = true;
  319. i += 2;
  320. continue;
  321. }
  322. // A quote ends at its closing quote or at a line break (an unclosed quote
  323. // in prose, "don't", stops at the end of its line), as in pipeBoundaries.
  324. if (c === "'") {
  325. let j = i + 1;
  326. while (j < text.length && text[j] !== "'" && text[j] !== "\n") j++;
  327. lit += text.slice(i + 1, j);
  328. quoted = true;
  329. i = text[j] === "'" ? j + 1 : j;
  330. continue;
  331. }
  332. if (c === '"') {
  333. let j = i + 1;
  334. while (j < text.length && text[j] !== '"' && text[j] !== "\n") {
  335. if (text[j] === "\\" && j + 1 < text.length) {
  336. const n = text[j + 1];
  337. if (n !== "\n") lit += n === '"' || n === "\\" || n === "$" || n === "`" ? n : "\\" + n;
  338. j += 2;
  339. continue;
  340. }
  341. lit += text[j++];
  342. }
  343. quoted = true;
  344. i = text[j] === '"' ? j + 1 : j;
  345. continue;
  346. }
  347. if (isBlank(c) || isStageEnd(c) || c === "<" || c === ">") break;
  348. lit += c;
  349. i++;
  350. }
  351. return { lit, i: Math.min(i, text.length), quoted };
  352. }
  353. /** Past a redirection operator at `i` and its target word. */
  354. function skipRedirect(text, i) {
  355. if (text[i] === "&") i++; // &> and &>>
  356. const op = text[i++];
  357. if (text[i] === op) i++; // >> <<
  358. if (op === "<" && text[i] === "<") i++; // <<<
  359. if (text[i] === "|" || text[i] === "&" || (op === "<" && (text[i] === "-" || text[i] === ">"))) i++; // >| >& <& <<- <>
  360. while (i < text.length && isBlank(text[i])) i++;
  361. if (i < text.length && !isStageEnd(text[i]) && !isRedirect(text, i)) i = readWord(text, i).i;
  362. return i;
  363. }
  364. /** The words of the stage that starts after `from`, as [literal, end] pairs, without redirections. */
  365. function stageWords(text, from, prose = false) {
  366. let i = from;
  367. // Blank space and line breaks right after the pipe: `curl … |` then `sh` on the next line.
  368. while (i < text.length && (isSpace(text[i]) || (text[i] === "\\" && text[i + 1] === "\n"))) i += text[i] === "\\" ? 2 : 1;
  369. // Read lazily, one word per call: a stage of millions of short words was
  370. // held whole as an array (1.7 GB at the 16 MiB hook-script cap, Reviewer).
  371. return () => {
  372. for (;;) {
  373. while (i < text.length && (isBlank(text[i]) || (text[i] === "\\" && text[i + 1] === "\n"))) i += text[i] === "\\" ? 2 : 1;
  374. if (i >= text.length || (!isRedirect(text, i) && isStageEnd(text[i]))) return null;
  375. if (isRedirect(text, i)) { i = skipRedirect(text, i); continue; }
  376. if (text[i] === "(") { i++; continue; } // a subshell: ( sh )
  377. const w = readWord(text, i, prose);
  378. i = w.i;
  379. // `2>file`: a number right before a redirection is its file descriptor.
  380. if (!w.quoted && /^\d+$/.test(w.lit) && (text[i] === ">" || text[i] === "<")) { i = skipRedirect(text, i); continue; }
  381. return [w.lit, i];
  382. }
  383. };
  384. }
  385. /** The stage parser's answer: the end of the shell's word, or -1. */
  386. function parsedShellAfterPipe(text, pipe, prose = false) {
  387. // Frames: the stage's words (read one at a time), then the strings env -S runs ({ s, i, lim, end }).
  388. const stack = [{ next: stageWords(text, pipe + 1, prose) }];
  389. let opts = NO_OPTS;
  390. let skipValue = false;
  391. let skipPositional = 0;
  392. let splitNext = false;
  393. while (stack.length) {
  394. const top = stack[stack.length - 1];
  395. let s, a, lim, end, f = null;
  396. if (top.next) {
  397. const w = top.next();
  398. if (!w) { stack.pop(); continue; }
  399. [s, end] = w;
  400. a = 0;
  401. lim = s.length;
  402. if (splitNext && !skipValue) { splitNext = false; stack.push({ s, i: 0, lim, end }); continue; }
  403. } else {
  404. while (top.i < top.lim && isSpace(top.s[top.i])) top.i++;
  405. if (top.i >= top.lim) { stack.pop(); continue; }
  406. ({ s, lim, end } = top);
  407. a = top.i;
  408. f = top;
  409. splitNext = false; // inside a split string, -S's value is simply the next word
  410. }
  411. // A stage word is the whole literal; a word of a split string ends at a space.
  412. const inWord = (j) => j < lim && (!f || !isSpace(s[j]));
  413. const wordEnd = (j) => { while (inWord(j)) j++; return j; };
  414. const consume = (j) => { if (f) f.i = j; };
  415. // Run a split string from j: the rest of a split-string word, or a new frame over the stage word.
  416. const split = (j) => { if (f) f.i = j; else stack.push({ s, i: j, lim, end }); };
  417. if (skipValue) { skipValue = false; consume(wordEnd(a)); continue; }
  418. if (s[a] === "-") {
  419. if (s[a + 1] === "-") {
  420. let j = a;
  421. while (inWord(j) && s[j] !== "=") j++;
  422. const name = s.slice(a, j);
  423. if (inWord(j)) { // --name=value
  424. if (opts.split && name === "--split-string") split(j + 1);
  425. else consume(wordEnd(j));
  426. continue;
  427. }
  428. if (opts.long.has(name)) { if (opts.split && name === "--split-string") splitNext = true; else skipValue = true; }
  429. consume(j);
  430. continue;
  431. }
  432. let c = a + 1;
  433. for (; inWord(c); c++) {
  434. if (!opts.short.includes(s[c])) continue;
  435. // -S runs its value: read it next. Any other value is skipped.
  436. if (opts.split && s[c] === "S") { if (inWord(c + 1)) split(c + 1); else { splitNext = true; consume(c + 1); } }
  437. else if (inWord(c + 1)) consume(wordEnd(c + 1));
  438. else { skipValue = true; consume(c + 1); }
  439. break;
  440. }
  441. if (!inWord(c)) consume(c);
  442. continue;
  443. }
  444. const b = wordEnd(a);
  445. consume(b);
  446. const w = f ? unquote(s.slice(a, b)) : s;
  447. if (!w || w === "{" || w === "!") continue;
  448. // An assignment (NAME=value, by the shell grammar) is never the command,
  449. // and is read before any basename: `X=/usr/bin/timeout bash` must not pick
  450. // the timeout wrapper, whose positional skip would eat `bash` (Vercel R).
  451. if (ASSIGNMENT.test(w)) continue;
  452. const base = w.slice(w.lastIndexOf("/") + 1);
  453. if (Object.hasOwn(WRAPPERS, base)) { opts = WRAPPERS[base]; skipPositional = opts.positional; continue; }
  454. if (skipPositional > 0) { skipPositional--; continue; }
  455. if (w.includes("=")) continue;
  456. return SHELL_BASE.test(base) ? end : -1;
  457. }
  458. return -1;
  459. }
  460. /**
  461. * Main's regex for a pipe into a shell, kept beside the parser (R12). Written
  462. * so its whitespace runs cannot split one run between them: main's
  463. * `env\s+\S*\s*` was quadratic on `| env` and long blank space. The same
  464. * strings match. Tried at one "|" (sticky), so the work per pipe is the blank
  465. * space and the one word after it, and those of different pipes don't overlap.
  466. */
  467. // Leading assignments (`| X=1 sh`, `| X=/usr/bin/timeout bash`) are taken
  468. // too, so this regex alone covers every form main's did, and more. A value
  469. // stops at a blank, "|", ";" or "&", so the work stays inside one stage.
  470. const LEGACY_SHELL = /\|\s*(?:[A-Za-z_]\w*=[^\s|;&]*\s+)*(?:sudo\s+)?(?:env\s+(?:\S+\s*)?)?(?:ba|z|da|k|fi|c|tc)?sh\b/iy;
  471. export function legacyShellAfterPipe(text, pipe) {
  472. LEGACY_SHELL.lastIndex = pipe;
  473. const m = LEGACY_SHELL.exec(text);
  474. return m ? pipe + m[0].length : -1;
  475. }
  476. /**
  477. * The pipeline boundaries of `text`: the "|" characters outside quotes and not
  478. * escaped, found in one forward pass with one quote and escape state (Vercel
  479. * S: a parse started at every raw "|", escaped ones too, read on past later
  480. * pipes, quadratic on a run of `\|`). Quotes end at a line break, as in
  481. * readWord, so the two agree: a stage parsed from a boundary never reads past
  482. * the next one, and each character is read by one stage parse.
  483. */
  484. function pipeBoundaries(text) {
  485. const marks = new Uint8Array(text.length);
  486. boundaryPass(text, marks, false);
  487. // Prose: an apostrophe inside a word ("it's", "don't") is not a quote. A
  488. // second pass reads it as a plain character, and a pipe that either pass
  489. // finds is a boundary, so an apostrophe that pairs with a later real quote
  490. // can't hide the pipes between them (Reviewer, fifth pass).
  491. if (/[A-Za-z]'[A-Za-z]/.test(text)) boundaryPass(text, marks, true);
  492. return marks;
  493. }
  494. const isLetter = (c) => (c >= "a" && c <= "z") || (c >= "A" && c <= "Z");
  495. /**
  496. * One pass of pipeBoundaries: marks the "|" outside quotes and escapes. Inside
  497. * double quotes, `$(...)` and backticks are code again (bash runs them), so a
  498. * pipe there is a boundary too; a stack keeps the quote to return to.
  499. * `prose`: a letter'letter apostrophe is plain.
  500. */
  501. function boundaryPass(text, marks, prose) {
  502. let q = "";
  503. let qAt = -1;
  504. const subs = []; // substitutions opened inside "...": { depth, tick }
  505. for (let i = 0; ; i++) {
  506. const c = i < text.length ? text[i] : "\n";
  507. if (c === "\n") {
  508. // A quote still open at the line's end never closed: read the line again
  509. // from just after it, as if it were a plain character, so the pipes
  510. // after it are boundaries. No quote of that kind follows it on the line,
  511. // so a line is read at most three times: still linear.
  512. if (q) { i = qAt; q = ""; qAt = -1; subs.length = 0; continue; }
  513. subs.length = 0;
  514. if (i >= text.length) break;
  515. continue;
  516. }
  517. if (q === "'") { if (c === "'") q = ""; continue; }
  518. if (c === "\\") { i++; continue; }
  519. if (q === '"') {
  520. if (c === '"') q = "";
  521. else if (c === "$" && text[i + 1] === "(") { subs.push({ depth: 0, tick: false }); q = ""; i++; }
  522. else if (c === "`") { subs.push({ depth: 0, tick: true }); q = ""; }
  523. continue;
  524. }
  525. // subs[-1] on an empty stack is a slow property lookup in V8, per character.
  526. const top = subs.length ? subs[subs.length - 1] : null;
  527. if (top) {
  528. if (top.tick ? c === "`" : c === ")" && top.depth === 0) { subs.pop(); q = '"'; continue; }
  529. if (!top.tick && c === "(") top.depth++;
  530. else if (!top.tick && c === ")") top.depth--;
  531. }
  532. if (c === "'" && prose && i > 0 && isLetter(text[i - 1]) && isLetter(text[i + 1] || "")) continue;
  533. if (c === "'" || c === '"') { q = c; qAt = i; }
  534. else if (c === "|") marks[i] |= prose ? 2 : 1;
  535. }
  536. }
  537. /**
  538. * A reader for one text: (pipe) -> the end of the shell's word, or -1. The
  539. * boundaries are found once. The parser runs at a boundary; main's regex runs
  540. * at every raw "|" (it is linear per pipe), so a quoted or escaped pipe is
  541. * still read as main read it. Either hit counts.
  542. */
  543. export function shellPipeReader(text) {
  544. let marks = null;
  545. return (pipe) => {
  546. marks ||= pipeBoundaries(text);
  547. // Each boundary is read with the reading that found it: a pipe only the prose
  548. // pass marks is parsed with the prose reading too, so a word glued by an
  549. // intra-word apostrophe is not re-read to the line's end per pipe (Vercel, PR 281).
  550. const end = marks[pipe] & 1 ? parsedShellAfterPipe(text, pipe) : marks[pipe] & 2 ? parsedShellAfterPipe(text, pipe, true) : -1;
  551. return end !== -1 ? end : legacyShellAfterPipe(text, pipe);
  552. };
  553. }
  554. /** Does the stage after the "|" at `pipe` run a shell? One-off form of shellPipeReader. */
  555. export function shellAfterPipe(text, pipe) {
  556. return shellPipeReader(text)(pipe);
  557. }
  558. /** The parser alone, for the differential test. */
  559. export const _parsedShellAfterPipe = parsedShellAfterPipe;
  560. /** Index of the first "|" whose stage runs a shell, or -1. */
  561. export function firstShellPipe(text) {
  562. const at = shellPipeReader(text);
  563. for (let p = text.indexOf("|"); p !== -1; p = text.indexOf("|", p + 1)) if (at(p) !== -1) return p;
  564. return -1;
  565. }
  566. // The rule's pattern as a regex, kept for the rule list and pack exports.
  567. // Matching uses shellAfterPipe.
  568. const SHELL_DOC = String.raw`(?:sudo\s+)?(?:[\w.~-]*/)*(?:env\s+)?(?:ba|z|da|k|fi|c|tc)?sh\b`;
  569. const INTERPRETER = String.raw`(?:sudo\s+)?(?:python[0-9.]*|node|perl|ruby|php|deno|bun)\b`;
  570. const PS_DOWNLOAD = String.raw`\b(?:iwr|irm|Invoke-WebRequest|Invoke-RestMethod|DownloadString)\b`;
  571. // The patterns of the form `<source> ... | <sink>` are matched by the helpers
  572. // below rather than by `re` itself (kept as the definition). As one regex
  573. // they restart the scan at every <source> on a line, which is quadratic on a
  574. // line full of them; the helpers return the same match in linear time.
  575. function matchAt(text, start, end) {
  576. const m = [text.slice(start, end)];
  577. m.index = start;
  578. return m;
  579. }
  580. /**
  581. * `source[^\n|]*\|\s*sink`: the source sits in the pipeline stage just before
  582. * the pipe. `sinkFor(text)` gives a reader (pipe) -> the sink's end or -1,
  583. * made once per text (shellPipeReader finds the boundaries once).
  584. */
  585. const regexAtPipe = (sink) => {
  586. const re = new RegExp(String.raw`\|\s*${sink}`, "iy");
  587. return (text) => (p) => { re.lastIndex = p; const m = re.exec(text); return m ? p + m[0].length : -1; };
  588. };
  589. function pipedFrom(source, sinkFor) {
  590. const sourceG = new RegExp(source.source, source.flags.replace(/[gy]/g, "") + "g");
  591. return (text) => {
  592. // The last newline before the pipe, tracked forward: a backward search
  593. // per pipe re-read the whole prefix of a line with no newline, which was
  594. // quadratic on "|sh|sh|sh..." (about 45 minutes at the 5 MB cap). The
  595. // previous "|" is at most one match back, so lastIndexOf stays linear.
  596. let lastNl = -1;
  597. let nextNl = text.indexOf("\n");
  598. let prevPipe = -1;
  599. // The source's matches, found once for the whole text and walked with a
  600. // pointer: matching it in a slice per pipe cost a copy per pipe, which V8
  601. // can make slow on a large text (83 s for 5 MiB of `| env -S x`). A source
  602. // match never holds "|" or a line break, so it lies inside one stage, and
  603. // the first one at or after the stage's start is the stage's leftmost.
  604. const sources = [...text.matchAll(sourceG)].map((m) => [m.index, m.index + m[0].length]);
  605. const sinkAt = sinkFor(text);
  606. let k = 0;
  607. for (let p = text.indexOf("|"); p !== -1 && k < sources.length; prevPipe = p, p = text.indexOf("|", p + 1)) {
  608. while (nextNl !== -1 && nextNl < p) { lastNl = nextNl; nextNl = text.indexOf("\n", nextNl + 1); }
  609. const start = Math.max(prevPipe, lastNl) + 1;
  610. while (k < sources.length && sources[k][0] < start) k++;
  611. // The source must be in the stage just before this pipe; only then is the sink read.
  612. if (k >= sources.length || sources[k][1] > p) continue;
  613. const end = sinkAt(p);
  614. if (end !== -1) return matchAt(text, sources[k][0], end);
  615. }
  616. return null;
  617. };
  618. }
  619. /**
  620. * `first[^\n]*second`: leftmost `first`, extended to the last `second` that
  621. * starts on its line. A `second` that begins with `\|` may run past the line
  622. * end (`\|\s*sh` spans a newline), so it is tried at each pipe on the line.
  623. */
  624. function thenOnLine(first, second) {
  625. const firstG = new RegExp(first, "gi");
  626. // `second` is a regex source, or a reader factory (text) -> (pipe) -> end | -1 tried at each pipe.
  627. const atPipeFn = typeof second === "function" ? second : null;
  628. const secondG = atPipeFn ? null : new RegExp(second, "gi");
  629. const atPipe = atPipeFn || second.startsWith("\\|");
  630. const sinkFor = atPipeFn || (atPipe ? regexAtPipe(second.slice(2)) : null);
  631. const lastOnLine = (text, from, eol, sinkAt) => {
  632. let last = null;
  633. if (atPipe) {
  634. for (let p = text.indexOf("|", from); p >= 0 && p < eol; p = text.indexOf("|", p + 1)) {
  635. const end = sinkAt(p);
  636. if (end !== -1) last = { 0: text.slice(p, end), index: p };
  637. }
  638. } else {
  639. for (const b of text.slice(from, eol).matchAll(secondG)) last = { 0: b[0], index: from + b.index };
  640. }
  641. return last;
  642. };
  643. return (text) => {
  644. let skipTo = 0;
  645. const sinkAt = sinkFor ? sinkFor(text) : null;
  646. for (const a of text.matchAll(firstG)) {
  647. if (a.index < skipTo) continue;
  648. const aEnd = a.index + a[0].length;
  649. let eol = text.indexOf("\n", aEnd);
  650. if (eol < 0) eol = text.length;
  651. // Only the first `first` on a line needs checking: any `second` after a
  652. // later one is also after this one.
  653. skipTo = eol;
  654. const last = lastOnLine(text, aEnd, eol, sinkAt);
  655. if (last) return matchAt(text, a.index, last.index + last[0].length);
  656. }
  657. return null;
  658. };
  659. }
  660. function earliest(...finders) {
  661. return (text) => {
  662. let best = null;
  663. for (const f of finders) {
  664. const m = f(text);
  665. if (m && (!best || m.index < best.index)) best = m;
  666. }
  667. return best;
  668. };
  669. }
  670. const JS_EVAL_REMOTE = /\b(?:eval|Function)\s*\(\s*(?:await\s+)?(?:atob|Buffer\.from|fetch|require\(\s*["']child_process|\(await fetch)/;
  671. const evalAsCallback = thenOnLine(String.raw`\bfetch\s*\(`, String.raw`\.then\(\s*(?:(?:globalThis|window|self)\.)?eval\s*\)`);
  672. export const EXEC_PATTERNS = [
  673. {
  674. rule: "curl-pipe-shell", label: "download piped to a shell",
  675. re: new RegExp(String.raw`\b(?:curl|wget|fetch|iwr)\b[^\n|]*\|\s*${SHELL_DOC}`, "i"),
  676. match: pipedFrom(/\b(?:curl|wget|fetch|iwr)\b/i, shellPipeReader),
  677. },
  678. {
  679. rule: "curl-pipe-interpreter", label: "download piped to an interpreter",
  680. re: /\b(?:curl|wget)\b[^\n|]*\|\s*(?:sudo\s+)?(?:python[0-9.]*|node|perl|ruby|php|deno|bun)\b/i,
  681. match: pipedFrom(/\b(?:curl|wget)\b/i, regexAtPipe(INTERPRETER)),
  682. },
  683. // Process and command substitution: lib/rules/core/agent-exec.json.
  684. ...rulesOf("agent-exec", { pack: "core" }).map((r) => ({ rule: r.id, label: r.label, re: r.pattern })),
  685. {
  686. rule: "base64-pipe-shell", label: "decoded base64 piped to a shell",
  687. re: new RegExp(String.raw`base64\s+(?:-d|--decode|-D)\b[^\n]*\|\s*${SHELL_DOC}`, "i"),
  688. match: thenOnLine(String.raw`base64\s+(?:-d|--decode|-D)\b`, shellPipeReader),
  689. },
  690. { rule: "shell-eval", label: "eval of command output", re: /\beval\s+["']?\$\(/ },
  691. {
  692. rule: "js-eval-remote", label: "eval of fetched or decoded content",
  693. re: JS_EVAL_REMOTE,
  694. // Also eval passed as the callback of a fetch: fetch(url).then((r) => r.text()).then(eval).
  695. match: (text) => text.match(JS_EVAL_REMOTE) || evalAsCallback(text),
  696. },
  697. { rule: "py-exec-remote", label: "exec of fetched or decoded content", re: /\bexec\s*\(\s*(?:base64\.b64decode|__import__|compile|urllib|requests\.get|zlib\.decompress|marshal\.loads|bytes\.fromhex)/ },
  698. {
  699. rule: "powershell-iex", label: "PowerShell Invoke-Expression of a download",
  700. re: /\b(?:iex|Invoke-Expression)\b[^\n]*\b(?:iwr|irm|Invoke-WebRequest|Invoke-RestMethod|DownloadString)\b|\b(?:iwr|irm|Invoke-WebRequest|Invoke-RestMethod|DownloadString)\b[^\n]*\|\s*(?:iex|Invoke-Expression)\b/i,
  701. match: earliest(
  702. thenOnLine(String.raw`\b(?:iex|Invoke-Expression)\b`, PS_DOWNLOAD),
  703. thenOnLine(PS_DOWNLOAD, String.raw`\|\s*(?:iex|Invoke-Expression)\b`),
  704. ),
  705. },
  706. ];
  707. /** Find remote-code-execution patterns in a string (a file, a hook command). */
  708. export function findExecPatterns(text) {
  709. const out = [];
  710. for (const p of execPatterns()) {
  711. const m = p.pack ? packMatch(p.pack, text) : p.match ? p.match(text) : text.match(p.re);
  712. if (m) out.push({ rule: p.rule, label: p.label, match: m[0], index: m.index });
  713. }
  714. return out;
  715. }
  716. // Built-ins, then rules from other packs (bundled or --rules).
  717. let _exec = null;
  718. let _execGen = -1;
  719. function execPatterns() {
  720. if (_execGen !== rulesGeneration()) {
  721. _exec = [...EXEC_PATTERNS, ...rulesOf("agent-exec", { exceptPack: "core" }).map((r) => ({ rule: r.id, label: r.label, re: r.pattern, pack: r }))];
  722. _execGen = rulesGeneration();
  723. }
  724. return _exec;
  725. }
  726. function detectExec(text) {
  727. return findExecPatterns(text).map((m) => ({
  728. rule: m.rule,
  729. severity: "high",
  730. title: `Remote code execution pattern: ${m.label}`,
  731. detail: `\`${quote(m.match, 120, lineAround(text, m.index, m.match.length))}\`. An agent following this runs code nobody reviewed, fetched at run time.`,
  732. line: lineOf(text, m.index),
  733. count: 1,
  734. }));
  735. }
  736. // ── Base64 blobs ──────────────────────────────────────────────────────────
  737. const MAGIC = [
  738. { name: "ELF executable", test: (b) => b[0] === 0x7f && b[1] === 0x45 && b[2] === 0x4c && b[3] === 0x46 },
  739. { name: "Windows executable", test: (b) => b[0] === 0x4d && b[1] === 0x5a },
  740. { name: "Mach-O executable", test: (b) => {
  741. const m = b.readUInt32BE?.(0);
  742. return [0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe, 0xcafebabe].includes(m);
  743. } },
  744. { name: "zip archive", test: (b) => b[0] === 0x50 && b[1] === 0x4b && (b[2] === 0x03 || b[2] === 0x05 || b[2] === 0x07) },
  745. { name: "gzip archive", test: (b) => b[0] === 0x1f && b[1] === 0x8b },
  746. { name: "bzip2 archive", test: (b) => b[0] === 0x42 && b[1] === 0x5a && b[2] === 0x68 },
  747. { name: "xz archive", test: (b) => b[0] === 0xfd && b[1] === 0x37 && b[2] === 0x7a && b[3] === 0x58 },
  748. { name: "7z archive", test: (b) => b[0] === 0x37 && b[1] === 0x7a && b[2] === 0xbc && b[3] === 0xaf },
  749. { name: "rar archive", test: (b) => b[0] === 0x52 && b[1] === 0x61 && b[2] === 0x72 && b[3] === 0x21 },
  750. { name: "WebAssembly module", test: (b) => b[0] === 0x00 && b[1] === 0x61 && b[2] === 0x73 && b[3] === 0x6d },
  751. // CPython bytecode: a 2-byte version magic followed by \r\n.
  752. { name: "Python bytecode (.pyc)", test: (b) => b.length >= 16 && b[2] === 0x0d && b[3] === 0x0a && b[1] < 0x10 },
  753. { name: "tar archive", test: (b) => b.length > 262 && b.toString("latin1", 257, 262) === "ustar" },
  754. ];
  755. /** Name the executable/archive format of a buffer, or null. */
  756. export function sniffBinary(buf) {
  757. if (!buf || buf.length < 4) return null;
  758. const hit = MAGIC.find((m) => m.test(buf));
  759. return hit ? hit.name : null;
  760. }
  761. function isPrintable(buf) {
  762. if (!buf.length) return false;
  763. let ok = 0;
  764. for (const b of buf) if (b === 9 || b === 10 || b === 13 || (b >= 32 && b < 127)) ok++;
  765. return ok / buf.length > 0.95;
  766. }
  767. function detectBase64(text, opts) {
  768. const re = new RegExp(`[A-Za-z0-9+/_-]{${opts.base64MinLength},}={0,2}`, "g");
  769. const blobs = [];
  770. for (const m of text.matchAll(re)) {
  771. const s = m[0];
  772. // Real base64 mixes cases and digits; hex digests, identifiers and long
  773. // words do not.
  774. if (!/[A-Z]/.test(s) || !/[a-z]/.test(s) || !/[0-9]/.test(s)) continue;
  775. if (/^[0-9a-f]+$/i.test(s)) continue;
  776. const before = text.slice(Math.max(0, m.index - 40), m.index);
  777. const image = /data:image\/[a-z+.-]+;base64,$/i.test(before);
  778. const buf = Buffer.from(s.replace(/-/g, "+").replace(/_/g, "/"), "base64");
  779. const binary = sniffBinary(buf);
  780. let decodedText = null;
  781. if (!binary && isPrintable(buf)) decodedText = buf.toString("utf-8");
  782. const execInside = decodedText ? findExecPatterns(decodedText) : [];
  783. const instructionInside = decodedText ? looksLikeInstruction(decodedText) : false;
  784. blobs.push({ index: m.index, length: s.length, image, binary, decodedText, execInside, instructionInside });
  785. }
  786. if (!blobs.length) return [];
  787. const worst = blobs.find((b) => b.binary) || blobs.find((b) => b.execInside.length) ||
  788. blobs.find((b) => b.instructionInside) || blobs.find((b) => !b.image) || blobs[0];
  789. let severity = "medium";
  790. let what = "an opaque encoded payload a reviewer cannot read";
  791. if (worst.binary) { severity = "high"; what = `a ${worst.binary}`; }
  792. else if (worst.execInside.length) { severity = "high"; what = `a ${worst.execInside[0].label} (\`${quote(worst.execInside[0].match, 80, worst.decodedText)}\`)`; }
  793. else if (worst.instructionInside) { severity = "high"; what = `hidden instructions ("${quote(worst.decodedText)}")`; }
  794. else if (worst.image) { severity = "low"; what = "an inline image"; }
  795. else if (worst.decodedText) what = `encoded text ("${quote(worst.decodedText, 60)}")`;
  796. return [{
  797. rule: "base64-blob",
  798. severity,
  799. title: "Large base64 blob",
  800. detail: `${blobs.length} base64 run(s) of ${opts.base64MinLength}+ characters; the largest is ${Math.max(...blobs.map((b) => b.length))}. It decodes to ${what}.`,
  801. line: lineOf(text, worst.index),
  802. count: blobs.length,
  803. }];
  804. }
  805. // ── Public API ────────────────────────────────────────────────────────────
  806. /**
  807. * Scan one text file for hidden content.
  808. * @param {string} text
  809. * @param {object} [opts] { blankLines, horizontalRun, base64MinLength, markdown }
  810. */
  811. export function scanHiddenContent(text, opts = {}) {
  812. const o = { ...DEFAULTS, ...opts };
  813. return [
  814. ...detectInvisible(text),
  815. ...detectWhitespaceInflation(text, o),
  816. ...detectHiddenInstructions(text),
  817. ...detectExec(text),
  818. ...detectBase64(text, o),
  819. ];
  820. }
  821. export function hasTagChars(text) {
  822. return TAG_RANGE.test(text);
  823. }
  824. // Extensions that are binary or archives no matter what the bytes say.
  825. export const BINARY_EXTENSIONS = /\.(?:pyc|pyo|pyd|so|dylib|dll|exe|bin|jar|class|whl|egg|zip|tar|tgz|gz|bz2|xz|7z|rar|wasm|node|deb|rpm|dmg|pkg|msi|apk)$/i;
  826. // Media that ships in skills legitimately.
  827. export const MEDIA_EXTENSIONS = /\.(?:png|jpe?g|gif|webp|ico|bmp|avif|woff2?|ttf|otf|eot|mp3|mp4|wav|ogg|webm|mov|pdf)$/i;
lib/fix/apply.js · Bundled script · 386 lines · 1 finding marked
  1. /**
  2. * applyDiff: the one place fix recipes' diffs get written to disk.
  3. *
  4. * Planning never writes. Applying is a separate, explicit step that refuses to
  5. * write outside the repo, into any .git (any case, any depth, any NTFS alias),
  6. * to a protected file (tests, test config, Penthera's own files, CI, hooks,
  7. * lockfiles, .gitmodules: see PROTECTED_FILES), through a symlink anywhere in the path, or into a dirty git
  8. * tree (unless the caller passes allowDirty). It refuses added lines carrying
  9. * control characters a terminal would act on, and it is all-or-nothing: every
  10. * hunk must apply, and if a write fails the files already written are put back.
  11. *
  12. * Each file is written to a temp file in its directory and moved into place, so
  13. * a hard link is replaced rather than written through, and an edit is refused
  14. * if the file changed since it was read.
  15. */
  16. import { createHash, randomBytes } from "node:crypto";
  17. import {
  18. existsSync, readFileSync, mkdirSync, realpathSync, lstatSync, fstatSync, openSync, writeSync, closeSync,
  19. renameSync, linkSync, unlinkSync, rmdirSync, chmodSync, constants,
  20. } from "node:fs";
  21. import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
  22. import { createGit, isOwnReport, parseStatusZ } from "../../src/agent/git.js";
  23. import { parsePatch, applyHunks } from "./diff.js";
  24. import { trimEndChars } from "../linear-text.js";
  25. export class FixError extends Error {
  26. constructor(message, code) {
  27. super(message);
  28. this.name = "FixError";
  29. this.code = code;
  30. }
  31. }
  32. function inside(root, p) {
  33. const rel = relative(root, p);
  34. return rel === "" || (!rel.startsWith(`..${sep}`) && rel !== ".." && !isAbsolute(rel));
  35. }
  36. export const sha256 = (buf) => createHash("sha256").update(buf).digest("hex");
  37. // Code points HFS+ ignores in names, so ".g‌it" is ".git" on macOS (CVE-2014-9390).
  38. const IGNORABLE = /[​-‏‪-‮⁠-⁤-]/g;
    • Note: Usual in code (characters a sanitizer strips): Contains 2 bidirectional-override character(s), which make text display in a different order from how it's read
    • Line 42 on GitHub
  39. /**
  40. * The name a path segment resolves to on case-insensitive file systems
  41. * (macOS, Windows): case folded, ignorable code points dropped, and the
  42. * trailing dots and spaces Windows strips removed.
  43. */
  44. export function canonicalSegment(seg) {
  45. return trimEndChars(String(seg).normalize("NFC").replace(IGNORABLE, ""), ". ").toLowerCase();
  46. }
  47. /** True for a segment that names the git directory: .git, .GIT, .Git., or the 8.3 alias GIT~1. */
  48. export function isGitDirSegment(seg) {
  49. const c = canonicalSegment(seg);
  50. return c === ".git" || /^git~\d+$/.test(c);
  51. }
  52. const LOCKFILES = new Set([
  53. "package-lock.json", "npm-shrinkwrap.json", "yarn.lock", "pnpm-lock.yaml", "bun.lock", "bun.lockb", "deno.lock",
  54. "poetry.lock", "pipfile.lock", "uv.lock", "pdm.lock", "cargo.lock", "go.sum", "gemfile.lock", "composer.lock",
  55. ]);
  56. const TEST_CONFIG = new Set(["package.json", "pytest.ini", "tox.ini", "setup.cfg", "pyproject.toml"]);
  57. /**
  58. * Paths a fix diff may never touch even though they live in the work tree: the
  59. * judge that verifies a fix (tests, test runner config, Penthera's own policy,
  60. * suppressions, baseline and memory), CI, git hooks and lockfiles. A fix may not
  61. * pass by editing its own checker. Each entry is [what, test(segments, basename)]
  62. * on canonical segments, so case and Windows aliases do not slip past.
  63. */
  64. const PROTECTED_FILES = [
  65. ["Windows 8.3 short names (they can alias a protected file)", (s) => s.some((d) => /~\d+(\.[^.]*)?$/.test(d))],
  66. [".gitmodules (it can point git at other repositories)", (s, b) => b === ".gitmodules"],
  67. ["tests", (s, b) => s.some((d) => d === "test" || d === "tests" || d === "__tests__") || /\.(test|spec)\./.test(b) || b === "conftest.py"],
  68. ["test runner config or the test script", (s, b) => /^(vitest|jest|playwright)\./.test(b) || TEST_CONFIG.has(b)],
  69. ["Penthera's own policy, suppressions, baseline, lock or memory", (s, b) => b.startsWith("penthera.") || s.includes(".penthera") || s.join("/").endsWith("audit/baseline.json")],
  70. ["CI workflows", (s) => s.some((d, i) => d === ".github" && s[i + 1] === "workflows")],
  71. ["git hooks", (s, b) => s.includes(".husky") || s.includes(".githooks") || /^(lefthook\.ya?ml|\.pre-commit-config\.ya?ml)$/.test(b)],
  72. ["lockfiles", (s, b) => LOCKFILES.has(b)],
  73. ];
  74. /** What protected kind of file `file` (repo-relative) is, or null if a fix may edit it. */
  75. export function protectedReason(file) {
  76. const segs = String(file).split(/[\\/]/).filter((x) => x && x !== ".").map(canonicalSegment);
  77. const hit = PROTECTED_FILES.find(([, test]) => test(segs, segs[segs.length - 1] || ""));
  78. return hit ? hit[0] : null;
  79. }
  80. // C0 controls (tab aside), DEL and C1 controls: a terminal acts on them instead of showing them.
  81. const CONTROL = /[\u0000-\u0008\u000a-\u001f\u007f-\u009f]/;
  82. /**
  83. * Throws unless no component of `target` below `root` is a symlink, and no
  84. * component above the file itself is something other than a directory. Every
  85. * existing component is lstat-ed, so neither a symlinked directory nor a
  86. * dangling symlink at the file itself can redirect a write.
  87. */
  88. function assertNoSymlinks(root, target, file) {
  89. let cur = root;
  90. const segs = relative(root, target).split(sep);
  91. for (let i = 0; i < segs.length; i++) {
  92. cur = join(cur, segs[i]);
  93. let st;
  94. try {
  95. st = lstatSync(cur);
  96. } catch (e) {
  97. if (e.code === "ENOENT") return;
  98. throw e;
  99. }
  100. if (st.isSymbolicLink()) {
  101. throw new FixError(`Refusing to write ${file}: ${relative(root, cur)} is a symlink, and fix diffs never write through symlinks`, "SYMLINK");
  102. }
  103. if (i < segs.length - 1 && !st.isDirectory()) {
  104. throw new FixError(`Refusing to write ${file}: ${relative(root, cur)} is not a directory`, "CONFLICT");
  105. }
  106. }
  107. }
  108. /** Resolve a diff path to an absolute path inside the repo, or throw. */
  109. function safeTarget(root, file) {
  110. if (!file || isAbsolute(file) || CONTROL.test(file) || file.includes("\0") || /^[a-zA-Z]:/.test(file)) {
  111. throw new FixError(`Refusing to write ${JSON.stringify(file)}: paths in a diff must be relative to the repo`, "OUTSIDE_REPO");
  112. }
  113. const segs = file.split(/[\\/]/);
  114. if (segs.includes("..")) throw new FixError(`Refusing to write ${file}: it is outside the repo`, "OUTSIDE_REPO");
  115. // NTFS alternate data streams (".git::$INDEX_ALLOCATION") name the same directory or file.
  116. if (segs.some((s) => s.includes(":"))) throw new FixError(`Refusing to write ${JSON.stringify(file)}: path segments may not contain ":"`, "OUTSIDE_REPO");
  117. // Any .git at any depth, in any case: on macOS .GIT/hooks/pre-commit is .git/hooks/pre-commit.
  118. if (segs.some(isGitDirSegment)) throw new FixError(`Refusing to write ${file}: it is inside .git`, "OUTSIDE_REPO");
  119. const why = protectedReason(file);
  120. if (why) {
  121. throw new FixError(`Refusing to write ${file}: fix diffs never edit ${why}, so the checks that verify a fix stay as they were. If the real fix needs this file changed, make that change by hand and review it.`, "PROTECTED");
  122. }
  123. const target = resolve(root, file);
  124. if (!inside(root, target) || target === root) throw new FixError(`Refusing to write ${file}: it is outside the repo`, "OUTSIDE_REPO");
  125. assertNoSymlinks(root, target, file);
  126. return target;
  127. }
  128. /**
  129. * Throws unless repoPath is in a git work tree with no uncommitted changes
  130. * under it, Penthera's own untracked reports aside (isOwnReport).
  131. * @returns {string[]} those reports, relative to repoPath
  132. */
  133. export function assertCleanTree(repoPath) {
  134. let entries;
  135. try {
  136. // The hardened runner: no fsmonitor, no hooks, no filters, no transports (src/agent/git.js).
  137. const git = createGit(repoPath);
  138. git.run("rev-parse", "--is-inside-work-tree");
  139. const prefix = git.run("rev-parse", "--show-prefix").replace(/\n$/, "");
  140. entries = parseStatusZ(git.run("status", "--porcelain=v1", "-z", "--untracked-files=all", "--", "."), prefix);
  141. } catch {
  142. throw new FixError(`${repoPath} is not a git work tree, so a fix could not be reviewed or reverted. Commit it to git first, or pass --allow-dirty.`, "NOT_GIT");
  143. }
  144. if (entries.some((e) => !isOwnReport(e))) {
  145. throw new FixError(`${repoPath} has uncommitted changes. Commit or stash them first so the fix is its own reviewable change, or pass --allow-dirty.`, "DIRTY");
  146. }
  147. return entries.map((e) => e.path);
  148. }
  149. const NOFOLLOW = constants.O_NOFOLLOW || 0;
  150. /** Current bytes of a regular file, null if it does not exist; throws on anything else. */
  151. function readCurrent(target, file) {
  152. let st;
  153. try {
  154. st = lstatSync(target);
  155. } catch (e) {
  156. if (e.code === "ENOENT") return null;
  157. throw e;
  158. }
  159. if (!st.isFile()) throw new FixError(`Refusing to write ${file}: it is not a regular file`, "CONFLICT");
  160. const fd = openSync(target, constants.O_RDONLY | NOFOLLOW);
  161. try {
  162. return readFileSync(fd);
  163. } finally {
  164. closeSync(fd);
  165. }
  166. }
  167. function assertDirInside(root, dir, file) {
  168. const real = realpathSync(dir);
  169. if (!inside(root, real)) throw new FixError(`Refusing to write ${file}: its directory resolves outside the repo`, "OUTSIDE_REPO");
  170. }
  171. /**
  172. * Put `data` at `target`, but only if the file currently holds `expect`
  173. * (a Buffer), or does not exist (expect === null). The bytes go to a temp file
  174. * in the same directory first, then are moved into place: rename for an edit
  175. * (which replaces a hard link instead of writing through it), link for a new
  176. * file (which fails if something appeared there meanwhile).
  177. *
  178. * @returns {{ madeDir: string|undefined }} the first directory mkdir created, if any
  179. */
  180. function replaceFile(root, file, target, data, expect) {
  181. assertNoSymlinks(root, target, file);
  182. const dir = dirname(target);
  183. const madeDir = mkdirSync(dir, { recursive: true });
  184. assertNoSymlinks(root, target, file);
  185. assertDirInside(root, dir, file);
  186. const current = readCurrent(target, file);
  187. const unchanged = expect === null ? current === null : current !== null && current.equals(expect);
  188. if (!unchanged) throw new FixError(`${file} changed on disk since the fix was planned; nothing was written to it`, "CHANGED");
  189. const mode = current === null ? 0o644 : lstatSync(target).mode & 0o7777;
  190. const tmp = join(dir, `.${basename(target)}.penthera-${randomBytes(6).toString("hex")}.tmp`);
  191. const fd = openSync(tmp, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | NOFOLLOW, 0o600);
  192. try {
  193. let off = 0;
  194. while (off < data.length) off += writeSync(fd, data, off, data.length - off);
  195. // The temp file we wrote must be the one at that path, in a directory inside the repo.
  196. const a = fstatSync(fd);
  197. const b = lstatSync(tmp);
  198. if (a.dev !== b.dev || a.ino !== b.ino) throw new FixError(`Refusing to write ${file}: its directory changed while writing`, "RACE");
  199. } catch (e) {
  200. closeSync(fd);
  201. try { unlinkSync(tmp); } catch { /* best effort */ }
  202. throw e;
  203. }
  204. closeSync(fd);
  205. try {
  206. chmodSync(tmp, mode);
  207. assertDirInside(root, dir, file);
  208. if (expect === null) {
  209. try {
  210. linkSync(tmp, target);
  211. } catch (e) {
  212. if (e.code === "EEXIST") throw new FixError(`${file} appeared on disk while the fix was being written; nothing was written to it`, "CHANGED");
  213. // File systems without hard links: exclusive create, never through a symlink.
  214. const out = openSync(target, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | NOFOLLOW, mode);
  215. try {
  216. let off = 0;
  217. while (off < data.length) off += writeSync(out, data, off, data.length - off);
  218. } finally {
  219. closeSync(out);
  220. }
  221. }
  222. } else {
  223. const again = readCurrent(target, file);
  224. if (!again || !again.equals(expect)) throw new FixError(`${file} changed on disk while the fix was being written; nothing was written to it`, "CHANGED");
  225. renameSync(tmp, target);
  226. }
  227. } finally {
  228. try { unlinkSync(tmp); } catch { /* already renamed */ }
  229. }
  230. return { madeDir };
  231. }
  232. /** Remove the directories a write created (deepest first), as long as they are empty. */
  233. function removeMadeDirs(target, madeDir) {
  234. if (!madeDir) return;
  235. for (let d = dirname(target); d.length >= madeDir.length; d = dirname(d)) {
  236. try { rmdirSync(d); } catch { return; }
  237. if (d === madeDir) return;
  238. }
  239. }
  240. /**
  241. * Put files written by applyDiff back the way they were, one by one, never
  242. * over somebody else's edit: a file is restored (or, if the fix created it,
  243. * removed) only while it still holds exactly what applyDiff wrote. A file that
  244. * changed since is left alone and reported as a conflict. Every file is tried
  245. * even if an earlier one fails.
  246. *
  247. * @param {string} repoPath
  248. * @param {{ path: string, created: boolean, before: Buffer|null, sha256: string }[]} files - applyDiff's result
  249. * @returns {{ restored: string[], conflicts: string[], errors: { path: string, message: string }[] }}
  250. */
  251. export function restoreFiles(repoPath, files) {
  252. const root = realpathSync(resolve(repoPath));
  253. const out = { restored: [], conflicts: [], errors: [] };
  254. for (const f of files) {
  255. try {
  256. const target = safeTarget(root, f.path);
  257. const current = readCurrent(target, f.path);
  258. const original = f.created ? null : f.before;
  259. if (current === null ? original === null : original !== null && current.equals(original)) {
  260. out.restored.push(f.path); // already as it was
  261. } else if (current === null || sha256(current) !== f.sha256) {
  262. out.conflicts.push(f.path);
  263. } else if (original === null) {
  264. unlinkSync(target);
  265. out.restored.push(f.path);
  266. } else {
  267. replaceFile(root, f.path, target, original, current);
  268. out.restored.push(f.path);
  269. }
  270. } catch (e) {
  271. if (e.code === "CHANGED") out.conflicts.push(f.path);
  272. else out.errors.push({ path: f.path, message: e.message });
  273. }
  274. }
  275. return out;
  276. }
  277. /** Throws if an added line carries a control character (a trailing CR of a CRLF line is fine). */
  278. function assertPrintable(f) {
  279. for (const h of f.hunks) {
  280. for (const op of h.ops) {
  281. if (op.t !== "+") continue;
  282. const line = op.line.replace(/\u0000$/, "").replace(/\r$/, "");
  283. if (CONTROL.test(line)) {
  284. throw new FixError(`Refusing to write ${f.newPath}: an added line contains a control character (${JSON.stringify(line.match(CONTROL)[0])}), which can hide code in a terminal`, "CONTROL_CHARS");
  285. }
  286. }
  287. }
  288. }
  289. /** Throws on two entries for one file, or one file that is another's directory. */
  290. function assertDistinctPaths(parsed) {
  291. const keys = parsed.map((f) => String(f.newPath).split(/[\\/]/).map(canonicalSegment).join("/"));
  292. for (let i = 0; i < keys.length; i++) {
  293. for (let j = 0; j < keys.length; j++) {
  294. if (i === j) continue;
  295. if (keys[i] === keys[j]) throw new FixError(`Refusing the diff: it changes ${parsed[i].newPath} more than once`, "DUPLICATE");
  296. if (keys[j].startsWith(`${keys[i]}/`)) throw new FixError(`Refusing the diff: ${parsed[i].newPath} is both a file and the directory of ${parsed[j].newPath}`, "CONFLICT");
  297. }
  298. }
  299. }
  300. /**
  301. * Apply a unified diff to files under repoPath.
  302. *
  303. * @param {string} repoPath
  304. * @param {string} diff
  305. * @param {{ dryRun?: boolean, allowDirty?: boolean }} [opts]
  306. * @returns {{ applied: boolean, files: { path: string, created: boolean, before: Buffer|null, sha256: string }[] }}
  307. * before is the file's bytes before the write; sha256 is the hash of what was written.
  308. */
  309. export function applyDiff(repoPath, diff, opts = {}) {
  310. const root = realpathSync(resolve(repoPath));
  311. const parsed = parsePatch(diff);
  312. if (parsed.length === 0) throw new FixError("The diff contains no file changes", "EMPTY");
  313. const ownReports = opts.allowDirty ? [] : assertCleanTree(root);
  314. const writes = [];
  315. for (const f of parsed) {
  316. if (ownReports.includes(f.newPath)) throw new FixError(`Refusing to write ${f.newPath}: it is an uncommitted scan report`, "DIRTY");
  317. if (!f.newPath) throw new FixError(`Refusing to delete ${f.oldPath}: fix diffs never delete files`, "DELETE");
  318. if (f.oldPath && f.oldPath !== f.newPath) throw new FixError(`Refusing to rename ${f.oldPath}: fix diffs never rename files`, "RENAME");
  319. assertPrintable(f);
  320. }
  321. assertDistinctPaths(parsed);
  322. for (const f of parsed) {
  323. const target = safeTarget(root, f.newPath);
  324. const exists = existsSync(target);
  325. if (exists && !lstatSync(target).isFile()) throw new FixError(`Refusing to write ${f.newPath}: it is not a regular file`, "CONFLICT");
  326. if (!f.oldPath && exists) throw new FixError(`${f.newPath} already exists; the plan expected to create it`, "CONFLICT");
  327. if (f.oldPath && !exists) throw new FixError(`${f.newPath} does not exist; the plan expected to edit it`, "CONFLICT");
  328. const before = exists ? readCurrent(target, f.newPath) : null;
  329. let after;
  330. try {
  331. after = Buffer.from(applyHunks(before === null ? null : before.toString("utf-8"), f.hunks), "utf-8");
  332. } catch (e) {
  333. throw new FixError(`${f.newPath}: ${e.message}`, "CONFLICT");
  334. }
  335. writes.push({ path: f.newPath, target, before, after, created: !exists });
  336. }
  337. if (!opts.dryRun) {
  338. const done = [];
  339. try {
  340. for (const w of writes) {
  341. const { madeDir } = replaceFile(root, w.path, w.target, w.after, w.before);
  342. done.push({ ...w, madeDir });
  343. }
  344. } catch (e) {
  345. // All or nothing: put back what was written, newest first.
  346. for (const w of done.reverse()) {
  347. try {
  348. if (w.created) unlinkSync(w.target);
  349. else replaceFile(root, w.path, w.target, w.before, w.after);
  350. removeMadeDirs(w.target, w.madeDir);
  351. } catch { /* reported through the original error */ }
  352. }
  353. throw e;
  354. }
  355. }
  356. return {
  357. applied: !opts.dryRun,
  358. files: writes.map((w) => ({ path: w.path, created: w.created, before: w.before, sha256: sha256(w.after) })),
  359. };
  360. }
lib/rules/pack.js · Bundled script · 470 lines · 2 findings marked
  1. /**
  2. * Rule-pack verification: how new detection knowledge reaches Penthera without a runtime
  3. * fetch or an update channel. See docs/rule-packs.md for the trust model.
  4. *
  5. * A pack is a directory:
  6. * pack.json name, version, created_at, min_penthera, rules[] index
  7. * ({ file, kind, sha256 } per rule file)
  8. * pack.json.sig `ssh-keygen -Y sign -n penthera-rules` over pack.json
  9. * *.json rule files, one kind each. Data only: regexes, string
  10. * lists, IOC lists. Never code.
  11. *
  12. * Loading verifies, in order: the signature over pack.json against
  13. * allowed_signers, every rule file's SHA-256 against pack.json, the schema of
  14. * every rule, and every regex against the ReDoS guard. Anything that fails is
  15. * refused whole; there is no partial load.
  16. *
  17. * Packs arrive two ways only: bundled in lib/rules/ (they ship inside the npm
  18. * release, staged and approved with 2FA), or by path with `--rules <dir>`.
  19. * Nothing in this module touches the network.
  20. */
  21. import { readFileSync, readdirSync, lstatSync, existsSync } from "node:fs";
  22. import { join, dirname, resolve } from "node:path";
  23. import { fileURLToPath } from "node:url";
  24. import { createHash } from "node:crypto";
  25. import { checkRegex } from "./regex-guard.js";
  26. import { PRODUCTS, PRODUCT_IDS } from "../advisories/products.js";
  27. import { parseVersion, compareVersions } from "../advisories/versions.js";
  28. import { parseAllowedSigners, verifyWithSigners } from "./sshsig.js";
  29. export const NAMESPACE = "penthera-rules";
  30. export const RULES_DIR = dirname(fileURLToPath(import.meta.url));
  31. export const BUILTIN_SIGNERS = join(RULES_DIR, "allowed_signers");
  32. const PKG_VERSION = JSON.parse(readFileSync(join(RULES_DIR, "..", "..", "package.json"), "utf-8")).version;
  33. const MAX_MANIFEST_BYTES = 1024 * 1024;
  34. const MAX_RULE_FILE_BYTES = 4 * 1024 * 1024;
  35. const MAX_RULES_PER_FILE = 5000;
  36. // Pack regexes run on at most this much input per call. Bundled packs are
  37. // part of the release and keep the engines' own behaviour (no cap).
  38. export const PACK_INPUT_CAP = 1024 * 1024;
  39. // ── Schema ────────────────────────────────────────────────────────────────
  40. const ID = /^[a-z0-9][a-z0-9._-]{0,63}$/;
  41. const PACK_NAME = /^[a-z0-9][a-z0-9-]{0,63}$/;
  42. const SEMVER = /^\d+\.\d+\.\d+$/;
  43. const DATE = /^\d{4}-\d{2}-\d{2}$/;
  44. const FILE_NAME = /^[a-z0-9][a-z0-9._-]{0,63}\.json$/;
  45. const SHA256 = /^[a-f0-9]{64}$/;
  46. const REF_KEY = /^[A-Za-z0-9_:.-]{1,64}$/;
  47. // Control characters, DEL, C1 and bidi overrides: a pack's strings reach the
  48. // terminal, Markdown and SARIF, so none of them may carry escapes.
  49. const UNSAFE_TEXT = /[\u0000-\u001f\u007f-\u009f‪-‮⁦-⁩]/;
    • Note: Usual in code (characters a sanitizer strips): Contains 4 bidirectional-override character(s), which make text display in a different order from how it's read
    • Line 53 on GitHub
  50. const CONFIDENCE = new Set(["confirmed", "likely", "potential"]);
  51. const SEVERITY = new Set(["critical", "high", "medium", "low", "info"]);
  52. const NPM_NAME = /^(?:@[a-z0-9][a-z0-9._~-]*\/)?[a-z0-9][a-z0-9._~-]*$/;
  53. const VERSION_OR_STAR = /^(?:\*|\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?)$/;
  54. // PyPI: names normalized as in PEP 503, versions as PEP 440 public versions.
  55. const PYPI_NAME = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
  56. const PEP440_OR_STAR = /^(?:\*|\d+(?:\.\d+)*(?:(?:a|b|rc)\d+)?(?:\.post\d+)?(?:\.dev\d+)?)$/;
  57. const HOSTNAME = /^(?=.{1,253}$)(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;
  58. const REL_PATH = /^(?!\/)(?!.*(?:^|\/)\.\.(?:\/|$))[A-Za-z0-9._@/-]{1,200}$/;
  59. const HTTP_METHODS = new Set(["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"]);
  60. const CVE_ID = /^CVE-\d{4}-\d{4,7}$/;
  61. const GHSA_ID = /^GHSA(?:-[23456789cfghjmpqrvwx]{4}){3}$/;
  62. const CVSS_VERSIONS = new Set(["3.0", "3.1", "4.0"]);
  63. class PackError extends Error {}
  64. const fail = (msg) => { throw new PackError(msg); };
  65. function text(v, where, { max = 300, optional = false } = {}) {
  66. if (v === undefined && optional) return undefined;
  67. if (typeof v !== "string" || v.length === 0) fail(`${where}: expected a non-empty string`);
  68. if (v.length > max) fail(`${where}: longer than ${max} characters`);
  69. if (UNSAFE_TEXT.test(v)) fail(`${where}: contains control or bidi characters`);
  70. return v;
  71. }
  72. function list(v, where, { max = 5000, item }) {
  73. if (!Array.isArray(v) || v.length === 0) fail(`${where}: expected a non-empty array`);
  74. if (v.length > max) fail(`${where}: more than ${max} entries`);
  75. return v.map((x, i) => item(x, `${where}[${i}]`));
  76. }
  77. function regex(pattern, flags, where) {
  78. text(pattern, `${where}.pattern`, { max: 512 });
  79. if (flags !== undefined && typeof flags !== "string") fail(`${where}.flags: expected a string`);
  80. const r = checkRegex(pattern, flags || "");
  81. if (!r.ok) fail(`${where}.pattern rejected by the ReDoS guard: ${r.reason}`);
  82. return new RegExp(pattern, flags || "");
  83. }
  84. /** Fields every rule carries: where the knowledge came from, and how sure it is. */
  85. function provenance(r, where) {
  86. text(r.id, `${where}.id`);
  87. if (!ID.test(r.id)) fail(`${where}.id: must match ${ID}`);
  88. text(r.source, `${where}.source`, { max: 500 });
  89. let url;
  90. try { url = new URL(r.source); } catch { fail(`${where}.source: not a URL`); }
  91. if (url.protocol !== "https:") fail(`${where}.source: must be an https URL`);
  92. if (typeof r.date !== "string" || !DATE.test(r.date) || Number.isNaN(Date.parse(r.date))) fail(`${where}.date: expected YYYY-MM-DD`);
  93. if (!CONFIDENCE.has(r.confidence)) fail(`${where}.confidence: one of ${[...CONFIDENCE].join(", ")}`);
  94. const references = r.references === undefined ? [] : list(r.references, `${where}.references`, {
  95. max: 20,
  96. item: (k, w) => { if (typeof k !== "string" || !REF_KEY.test(k)) fail(`${w}: not a references.bib key`); return k; },
  97. });
  98. return {
  99. id: r.id,
  100. source: r.source,
  101. date: r.date,
  102. confidence: r.confidence,
  103. references,
  104. description: text(r.description, `${where}.description`, { max: 1000, optional: true }),
  105. };
  106. }
  107. function severity(v, where, dflt) {
  108. if (v === undefined && dflt) return dflt;
  109. if (!SEVERITY.has(v)) fail(`${where}: one of ${[...SEVERITY].join(", ")}`);
  110. return v;
  111. }
  112. /** Nuclei-compatible template, the subset lib/templates.js executes. */
  113. function template(t, where) {
  114. if (!t || typeof t !== "object" || Array.isArray(t)) fail(`${where}: expected an object`);
  115. text(t.id, `${where}.id`, { max: 100 });
  116. const info = t.info || fail(`${where}.info: required`);
  117. const out = {
  118. id: t.id,
  119. info: {
  120. name: text(info.name, `${where}.info.name`),
  121. severity: severity(info.severity, `${where}.info.severity`),
  122. tags: info.tags === undefined ? [] : list(info.tags, `${where}.info.tags`, { max: 20, item: (x, w) => text(x, w, { max: 40 }) }),
  123. ...(info.description !== undefined ? { description: text(info.description, `${where}.info.description`, { max: 1000 }) } : {}),
  124. },
  125. http: list(t.http, `${where}.http`, {
  126. max: 10,
  127. item: (req, w) => {
  128. const method = req.method === undefined ? "GET" : req.method;
  129. if (!HTTP_METHODS.has(method)) fail(`${w}.method: not an allowed HTTP method`);
  130. text(req.path, `${w}.path`, { max: 500 });
  131. // Only paths on the scan target: a pack can never point the scanner at another host.
  132. const p = req.path.replace(/^\{\{BaseURL\}\}/, "");
  133. if (!p.startsWith("/") || p.startsWith("//") || /[\s\\]/.test(p)) fail(`${w}.path: must be a path on the target ("/x" or "{{BaseURL}}/x")`);
  134. const matchers = list(req.matchers, `${w}.matchers`, {
  135. max: 10,
  136. item: (m, mw) => {
  137. const type = m.type;
  138. if (type === "status") return { type, status: list(m.status, `${mw}.status`, { max: 20, item: (s, sw) => { if (!Number.isInteger(s) || s < 100 || s > 599) fail(`${sw}: HTTP status`); return s; } }) };
  139. if (type === "word" || type === "negative-word") {
  140. return { type, words: list(m.words, `${mw}.words`, { max: 50, item: (x, xw) => text(x, xw, { max: 200 }) }), ...(m.condition ? { condition: m.condition === "and" ? "and" : "or" } : {}), ...(m.part === "header" ? { part: "header" } : {}) };
  141. }
  142. if (type === "regex" || type === "negative-regex") {
  143. const pats = list(m.regex, `${mw}.regex`, { max: 10, item: (x, xw) => { regex(x, m.flags, xw); return x; } });
  144. return { type, regex: pats, ...(m.flags ? { flags: m.flags } : {}), ...(m.condition ? { condition: m.condition === "and" ? "and" : "or" } : {}), ...(m.part === "header" ? { part: "header" } : {}) };
  145. }
  146. return fail(`${mw}.type: one of status, word, negative-word, regex, negative-regex`);
  147. },
  148. });
  149. const cond = req["matchers-condition"];
  150. if (cond !== undefined && cond !== "and" && cond !== "or") fail(`${w}.matchers-condition: "and" or "or"`);
  151. // Dropping these would turn a guarded check back into a status-only one.
  152. for (const k of ["confirm", "mustChangeAnswer"]) if (req[k] !== undefined) fail(`${w}.${k}: built-in templates only`);
  153. return { method, path: req.path, matchers, ...(cond ? { "matchers-condition": cond } : {}) };
  154. },
  155. }),
  156. };
  157. return out;
  158. }
  159. function product(v, where) {
  160. if (!PRODUCT_IDS.includes(v)) fail(`${where}: one of ${PRODUCT_IDS.join(", ")}`);
  161. return v;
  162. }
  163. /** A release line of `prod` ("8.3", "20", "1.1.1"), checked against how that product numbers them. */
  164. function branchName(prod, v, where) {
  165. text(v, where, { max: 20 });
  166. const parsed = parseVersion(v);
  167. if (!parsed || parsed.suffix || parsed.pre || PRODUCTS[prod].branchOf(parsed) !== v) fail(`${where}: not a ${PRODUCTS[prod].name} release line`);
  168. return v;
  169. }
  170. /** An exact upstream version of `prod` on release line `branch`. */
  171. function exactVersion(prod, branch, v, where) {
  172. text(v, where, { max: 30 });
  173. const parsed = parseVersion(v);
  174. if (!parsed || parsed.suffix || parsed.pre || parsed.patch === null) fail(`${where}: expected an exact upstream version like 8.3.27`);
  175. if (PRODUCTS[prod].branchOf(parsed) !== branch) fail(`${where}: ${v} is not on release line ${branch}`);
  176. return v;
  177. }
  178. /**
  179. * Rule kinds and the engine each one plugs into. `compile` validates one rule
  180. * and returns what the engine consumes.
  181. */
  182. export const KINDS = {
  183. // lib/whitebox/secrets.js
  184. secret: (r, w) => ({
  185. ...provenance(r, w),
  186. name: text(r.name, `${w}.name`, { max: 100 }),
  187. pattern: regex(r.pattern, r.flags, w),
  188. severity: severity(r.severity, `${w}.severity`),
  189. skipPlaceholders: r.skip_placeholders === true,
  190. }),
  191. // lib/agents/hidden-content.js: instructions hidden in comments, decoded blobs
  192. "agent-instruction": (r, w) => ({ ...provenance(r, w), pattern: regex(r.pattern, r.flags, w) }),
  193. // lib/agents/hidden-content.js + rules.js: remote-code-execution shapes in skills, hooks, MCP commands
  194. "agent-exec": (r, w) => ({ ...provenance(r, w), label: text(r.label, `${w}.label`, { max: 100 }), pattern: regex(r.pattern, r.flags, w) }),
  195. // lib/deps: known-malicious package versions (lockfile names)
  196. "ioc-package": (r, w) => ({
  197. ...provenance(r, w),
  198. campaign: text(r.campaign, `${w}.campaign`, { max: 100 }),
  199. severity: severity(r.severity, `${w}.severity`, "critical"),
  200. packages: list(r.packages, `${w}.packages`, {
  201. item: (p, pw) => {
  202. if (!p || typeof p.name !== "string" || p.name.length > 214 || !NPM_NAME.test(p.name)) fail(`${pw}.name: not an npm package name`);
  203. const versions = list(p.versions, `${pw}.versions`, { max: 200, item: (v, vw) => { if (typeof v !== "string" || !VERSION_OR_STAR.test(v)) fail(`${vw}: exact version or "*"`); return v; } });
  204. return { name: p.name, versions };
  205. },
  206. }),
  207. }),
  208. // lib/deps/python: known-malicious PyPI versions (normalized names, PEP 440 versions)
  209. "ioc-pypi-package": (r, w) => ({
  210. ...provenance(r, w),
  211. campaign: text(r.campaign, `${w}.campaign`, { max: 100 }),
  212. severity: severity(r.severity, `${w}.severity`, "critical"),
  213. packages: list(r.packages, `${w}.packages`, {
  214. item: (p, pw) => {
  215. if (!p || typeof p.name !== "string" || p.name.length > 214 || !PYPI_NAME.test(p.name)) fail(`${pw}.name: not a normalized PyPI name (PEP 503: lowercase, runs of -_. as one -)`);
  216. const versions = list(p.versions, `${pw}.versions`, { max: 200, item: (v, vw) => { if (typeof v !== "string" || !PEP440_OR_STAR.test(v)) fail(`${vw}: exact PEP 440 version or "*"`); return v; } });
  217. return { name: p.name, versions };
  218. },
  219. }),
  220. }),
  221. // lib/deps: exfiltration endpoints found in package code. `hosts` are
  222. // attacker-owned domains; `urls` pin an endpoint on a shared service
  223. // (webhook.site) where the bare host would be a false positive.
    • Note: Quoted as data in code: Mentions a known data-relay/testing domain (webhook.site) without linking or sending anything to it
    • Line 239 on GitHub
  224. "ioc-host": (r, w) => {
  225. if (r.hosts === undefined && r.urls === undefined) fail(`${w}: needs hosts or urls`);
  226. return {
  227. ...provenance(r, w),
  228. campaign: text(r.campaign, `${w}.campaign`, { max: 100 }),
  229. severity: severity(r.severity, `${w}.severity`, "critical"),
  230. hosts: r.hosts === undefined ? [] : list(r.hosts, `${w}.hosts`, { max: 500, item: (h, hw) => { if (typeof h !== "string" || !HOSTNAME.test(h)) fail(`${hw}: lowercase hostname`); return h; } }),
  231. urls: r.urls === undefined ? [] : list(r.urls, `${w}.urls`, {
  232. max: 500,
  233. item: (u, uw) => {
  234. text(u, uw, { max: 300 });
  235. let url;
  236. try { url = new URL(u); } catch { fail(`${uw}: not a URL`); }
  237. if (!/^https?:$/.test(url.protocol) || url.pathname.length < 2) fail(`${uw}: http(s) URL with a path`);
  238. return u;
  239. },
  240. }),
  241. };
  242. },
  243. // lib/deps: files a campaign drops into packages (by path, optionally pinned by hash)
  244. "ioc-file": (r, w) => ({
  245. ...provenance(r, w),
  246. campaign: text(r.campaign, `${w}.campaign`, { max: 100 }),
  247. severity: severity(r.severity, `${w}.severity`, "critical"),
  248. files: list(r.files, `${w}.files`, {
  249. max: 500,
  250. item: (f, fw) => {
  251. if (!f || typeof f.path !== "string" || !REL_PATH.test(f.path)) fail(`${fw}.path: relative path inside a package`);
  252. if (f.sha256 !== undefined && (typeof f.sha256 !== "string" || !SHA256.test(f.sha256))) fail(`${fw}.sha256: hex SHA-256`);
  253. return { path: f.path, ...(f.sha256 ? { sha256: f.sha256 } : {}) };
  254. },
  255. }),
  256. }),
  257. // lib/deps/typosquat.js: extra names worth protecting from look-alikes
  258. "typosquat-target": (r, w) => ({
  259. ...provenance(r, w),
  260. names: list(r.names, `${w}.names`, { max: 5000, item: (n, nw) => { if (typeof n !== "string" || n.length > 214 || !NPM_NAME.test(n)) fail(`${nw}: npm package name`); return n; } }),
  261. }),
  262. // lib/advisories: affected and fixed versions of a runtime or server, per release line
  263. "runtime-advisory": (r, w) => {
  264. const prod = product(r.product, `${w}.product`);
  265. const seen = new Set();
  266. const out = {
  267. ...provenance(r, w),
  268. product: prod,
  269. title: text(r.title, `${w}.title`, { max: 200 }),
  270. severity: severity(r.severity, `${w}.severity`, "critical"),
  271. cves: r.cves === undefined ? [] : list(r.cves, `${w}.cves`, { max: 50, item: (c, cw) => { if (typeof c !== "string" || !CVE_ID.test(c)) fail(`${cw}: CVE-YYYY-NNNN`); return c; } }),
  272. ghsa: r.ghsa === undefined ? [] : list(r.ghsa, `${w}.ghsa`, { max: 50, item: (g, gw) => { if (typeof g !== "string" || !GHSA_ID.test(g)) fail(`${gw}: GHSA-xxxx-xxxx-xxxx`); return g; } }),
  273. // What has to be true besides the version (an OS, a SAPI, a module).
  274. // A rule with a condition never yields more than `potential`.
  275. condition: text(r.condition, `${w}.condition`, { max: 300, optional: true }),
  276. fixed: list(r.fixed, `${w}.fixed`, {
  277. max: 50,
  278. item: (f, fw) => {
  279. if (!f || typeof f !== "object") fail(`${fw}: expected { branch, version }`);
  280. const branch = branchName(prod, f.branch, `${fw}.branch`);
  281. if (seen.has(branch)) fail(`${fw}.branch: ${branch} listed twice`);
  282. seen.add(branch);
  283. const version = exactVersion(prod, branch, f.version, `${fw}.version`);
  284. const introduced = f.introduced === undefined ? undefined : exactVersion(prod, branch, f.introduced, `${fw}.introduced`);
  285. if (introduced && compareVersions(introduced, version) >= 0) fail(`${fw}.introduced: must be below the fixed version`);
  286. return { branch, version, ...(introduced ? { introduced } : {}) };
  287. },
  288. }),
  289. };
  290. // Date the CVE was added to CISA's Known Exploited Vulnerabilities catalog (CC0).
  291. if (r.kev !== undefined) {
  292. if (typeof r.kev !== "string" || !DATE.test(r.kev) || Number.isNaN(Date.parse(r.kev))) fail(`${w}.kev: expected YYYY-MM-DD`);
  293. out.kev = r.kev;
  294. }
  295. if (r.cvss !== undefined) {
  296. const c = r.cvss;
  297. if (!c || !CVSS_VERSIONS.has(c.version) || typeof c.score !== "number" || c.score < 0 || c.score > 10) fail(`${w}.cvss: { version: "3.1" | "4.0", score: 0-10 }`);
  298. out.cvss = { version: c.version, score: c.score };
  299. }
  300. return out;
  301. },
  302. // lib/advisories: release lines past their vendor's end of support. A line
  303. // counts as end of life when its `eol` date is on or before the rule's
  304. // `date`, so the result depends on the pack, never on the scan's clock.
  305. "runtime-eol": (r, w) => {
  306. const prod = product(r.product, `${w}.product`);
  307. const seen = new Set();
  308. return {
  309. ...provenance(r, w),
  310. product: prod,
  311. branches: list(r.branches, `${w}.branches`, {
  312. max: 100,
  313. item: (b, bw) => {
  314. if (!b || typeof b !== "object") fail(`${bw}: expected { branch, eol }`);
  315. const branch = branchName(prod, b.branch, `${bw}.branch`);
  316. if (seen.has(branch)) fail(`${bw}.branch: ${branch} listed twice`);
  317. seen.add(branch);
  318. if (typeof b.eol !== "string" || !DATE.test(b.eol) || Number.isNaN(Date.parse(b.eol))) fail(`${bw}.eol: expected YYYY-MM-DD`);
  319. return { branch, eol: b.eol };
  320. },
  321. }),
  322. };
  323. },
  324. // lib/templates.js
  325. "nuclei-template": (r, w) => ({ ...provenance(r, w), template: template(r.template, `${w}.template`) }),
  326. };
  327. // ── Reading and verifying ─────────────────────────────────────────────────
  328. function cmpSemver(a, b) {
  329. const pa = a.split(".").map(Number);
  330. const pb = b.split(".").map(Number);
  331. for (let i = 0; i < 3; i++) if (pa[i] !== pb[i]) return pa[i] < pb[i] ? -1 : 1;
  332. return 0;
  333. }
  334. function readRegular(path, max, what) {
  335. let st;
  336. try { st = lstatSync(path); } catch { fail(`${what} not found`); }
  337. if (!st.isFile()) fail(`${what} is not a regular file (symlinks are refused)`);
  338. if (st.size > max) fail(`${what} is larger than ${max} bytes`);
  339. return readFileSync(path);
  340. }
  341. function validateManifest(m) {
  342. if (!m || typeof m !== "object" || Array.isArray(m)) fail("pack.json: expected an object");
  343. if (m.schema !== 1) fail("pack.json: schema must be 1");
  344. if (typeof m.name !== "string" || !PACK_NAME.test(m.name)) fail(`pack.json: name must match ${PACK_NAME}`);
  345. if (typeof m.version !== "string" || !SEMVER.test(m.version)) fail("pack.json: version must be x.y.z");
  346. if (typeof m.min_penthera !== "string" || !SEMVER.test(m.min_penthera)) fail("pack.json: min_penthera must be x.y.z");
  347. if (typeof m.created_at !== "string" || Number.isNaN(Date.parse(m.created_at))) fail("pack.json: created_at must be an ISO 8601 timestamp");
  348. text(m.description, "pack.json: description", { max: 1000, optional: true });
  349. const seen = new Set();
  350. list(m.rules, "pack.json: rules", {
  351. max: 100,
  352. item: (e, w) => {
  353. if (!e || typeof e.file !== "string" || !FILE_NAME.test(e.file) || e.file === "pack.json") fail(`${w}.file: a plain file name like secrets.json`);
  354. if (seen.has(e.file)) fail(`${w}.file: listed twice`);
  355. seen.add(e.file);
  356. if (!Object.hasOwn(KINDS, e.kind)) fail(`${w}.kind: one of ${Object.keys(KINDS).join(", ")}`);
  357. if (typeof e.sha256 !== "string" || !SHA256.test(e.sha256)) fail(`${w}.sha256: hex SHA-256`);
  358. return e;
  359. },
  360. });
  361. return m;
  362. }
  363. /** Load allowed_signers entries from the built-in file plus any extra files. */
  364. export function loadSigners(extra = []) {
  365. const out = [];
  366. for (const f of [BUILTIN_SIGNERS, ...[].concat(extra)]) {
  367. if (!f) continue;
  368. if (!existsSync(f)) {
  369. if (f === BUILTIN_SIGNERS) continue;
  370. throw new Error(`allowed_signers file not found: ${f}`);
  371. }
  372. out.push(...parseAllowedSigners(readFileSync(f, "utf-8")));
  373. }
  374. return out;
  375. }
  376. /**
  377. * Verify a pack directory without activating it.
  378. * @param {string} dir
  379. * @param {{ signers?: object[], now?: number, version?: string }} [opts]
  380. * @returns {{ ok: boolean, dir: string, name?: string, version?: string,
  381. * signature: "valid"|"invalid"|"missing", signer?: string, signatureError?: string,
  382. * errors: string[], warnings: string[], manifest?: object, rules?: Record<string, object[]>, count?: number }}
  383. */
  384. export function verifyPack(dir, opts = {}) {
  385. const abs = resolve(dir);
  386. const res = { ok: false, dir: abs, signature: "missing", errors: [], warnings: [] };
  387. try {
  388. let st;
  389. try { st = lstatSync(abs); } catch { fail(`not found: ${dir}`); }
  390. if (!st.isDirectory()) fail(`not a directory: ${dir}`);
  391. const raw = readRegular(join(abs, "pack.json"), MAX_MANIFEST_BYTES, "pack.json");
  392. let manifest;
  393. try { manifest = JSON.parse(raw.toString("utf-8")); } catch (e) { fail(`pack.json: invalid JSON (${e.message})`); }
  394. validateManifest(manifest);
  395. res.name = manifest.name;
  396. res.version = manifest.version;
  397. res.manifest = manifest;
  398. // 1. Signature over the exact bytes of pack.json.
  399. const sigPath = join(abs, "pack.json.sig");
  400. if (existsSync(sigPath)) {
  401. const sig = readRegular(sigPath, 64 * 1024, "pack.json.sig").toString("utf-8");
  402. const v = verifyWithSigners(raw, sig, NAMESPACE, opts.signers || loadSigners(), { now: opts.now });
  403. if (v.ok) { res.signature = "valid"; res.signer = v.principal; }
  404. else { res.signature = "invalid"; res.signatureError = v.reason; }
  405. }
  406. if (cmpSemver(opts.version || PKG_VERSION, manifest.min_penthera) < 0) {
  407. fail(`needs Penthera ${manifest.min_penthera} or later (this is ${opts.version || PKG_VERSION})`);
  408. }
  409. // 2. Every rule file against its hash in pack.json, then 3. its contents.
  410. const rules = Object.fromEntries(Object.keys(KINDS).map((k) => [k, []]));
  411. const ids = new Set();
  412. let count = 0;
  413. for (const entry of manifest.rules) {
  414. const buf = readRegular(join(abs, entry.file), MAX_RULE_FILE_BYTES, entry.file);
  415. const digest = createHash("sha256").update(buf).digest("hex");
  416. if (digest !== entry.sha256) fail(`${entry.file}: SHA-256 does not match pack.json (file changed after the pack was built)`);
  417. let data;
  418. try { data = JSON.parse(buf.toString("utf-8")); } catch (e) { fail(`${entry.file}: invalid JSON (${e.message})`); }
  419. if (!data || data.kind !== entry.kind) fail(`${entry.file}: kind is "${data?.kind}", pack.json says "${entry.kind}"`);
  420. list(data.rules, `${entry.file}: rules`, { max: MAX_RULES_PER_FILE, item: (r) => r });
  421. data.rules.forEach((r, i) => {
  422. const where = `${entry.file}: rules[${i}]`;
  423. if (!r || typeof r !== "object" || Array.isArray(r)) fail(`${where}: expected an object`);
  424. const compiled = KINDS[entry.kind](r, where);
  425. if (ids.has(compiled.id)) fail(`${where}.id: "${compiled.id}" used twice in this pack`);
  426. ids.add(compiled.id);
  427. rules[entry.kind].push({ ...compiled, kind: entry.kind, pack: manifest.name, packVersion: manifest.version });
  428. count++;
  429. });
  430. }
  431. const listed = new Set(manifest.rules.map((e) => e.file));
  432. for (const f of readdirSync(abs)) {
  433. if (f === "pack.json" || f === "pack.json.sig" || listed.has(f)) continue;
  434. res.warnings.push(`${f} is not listed in pack.json and is ignored`);
  435. }
  436. res.rules = rules;
  437. res.count = count;
  438. res.ok = true;
  439. } catch (e) {
  440. res.errors.push(e instanceof PackError ? e.message : `unexpected error: ${e.message}`);
  441. }
  442. return res;
  443. }

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.