Skip to content
Back to search

open-web-intel-mcp

Zero-proxy, SSRF-hardened TypeScript MCP server (stdio & Streamable HTTP) and CLI for Markdown extraction, llms.txt, sitemaps, ATS jobs, e-commerce catalogs, DNS enrichment, and podcast RSS.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/pie527645-cloud--open-web-intel-mcp/badge.svg)](/listings/pie527645-cloud--open-web-intel-mcp)

HTML

<a href="/listings/pie527645-cloud--open-web-intel-mcp"><img src="/api/listings/pie527645-cloud--open-web-intel-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/pie527645-cloud/open-web-intel-mcp.git
cd open-web-intel-mcp
npm install
npm run build

# Optional: link `open-web-intel` and `open-web-intel-mcp` binaries globally
npm link

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 209 lines
  1. # open-web-intel-mcp
  2. **Zero-Proxy, Zero-Browser Local TypeScript MCP Server & CLI for SSRF-Safe Web Intelligence.**
  3. `open-web-intel-mcp` gives AI coding agents, desktop assistants, and developer pipelines fast, direct access to structured public web data—without headless browsers, paid proxy pools, or API keys. Every outbound HTTP request passes through a shared defense-in-depth SSRF guard with DNS resolution checks, manual redirect hop re-validation, strict timeouts, and streaming byte caps.
  4. ---
  5. ## Key Capabilities
  6. - **Zero Headless Browser Overhead**: Queries clean public JSON, XML, RSS, DNS, and HTML endpoints directly on Node.js `>=20.18.1` (Node 20.18+ / 22).
  7. - **Defense-in-Depth SSRF Protection**: Blocks private RFC1918 networks, loopback, CGNAT, cloud metadata (`169.254.169.254`), link-local, unique-local IPv6, NAT64/6to4/Teredo, and IPv4-mapped IPv6 addresses across every DNS record and `redirect: 'manual'` hop.
  8. - **Token-Lean 7-Tool Schema**: Concise, read-only tool definitions (~1,460 tokens across all 7 tools combined) designed for minimal LLM context overhead and zero prompt-injection surface.
  9. - **Dual Entrypoints**:
  10. - `open-web-intel-mcp`: Model Context Protocol server over `stdio` (default) or stateless Streamable HTTP (`--http --port 3000`).
  11. - `open-web-intel`: Standalone terminal CLI with human-readable and `--json` output modes.
  12. ---
  13. ## Architecture
  14. ```mermaid
  15. flowchart LR
  16. Client["MCP Host / Terminal CLI"] --> Entry["Dual Entrypoints (dist/mcp.js | dist/cli.js)"]
  17. Entry --> Tools["7 Web Intelligence Tools"]
  18. Tools --> Guard["SSRF Guard (src/lib/safe-fetch.ts)"]
  19. Tools --> DNS["DNS Resolver (MX / SPF / DMARC)"]
  20. Guard --> Check["URL & Multi-Family DNS IP Validator"]
  21. Check --> Hop["Manual Redirect Loop (<= 5 Hops, Re-Validated)"]
  22. Hop --> Stream["Streaming Byte Cap & Timeout Reader"]
  23. Stream --> Web["Public JSON / XML / RSS / HTML Endpoints"]
  24. ```
  25. ---
  26. ## Installation & Quick Start
  27. ```bash
  28. git clone https://github.com/pie527645-cloud/open-web-intel-mcp.git
  29. cd open-web-intel-mcp
  30. npm install
  31. npm run build
  32. # Optional: link `open-web-intel` and `open-web-intel-mcp` binaries globally
  33. npm link
  34. ```
  35. ---
  36. ## MCP Client Configuration
  37. Point your MCP client directly at the built `dist/mcp.js` entrypoint:
  38. ### Claude Desktop (`claude_desktop_config.json`)
  39. ```json
  40. {
  41. "mcpServers": {
  42. "open-web-intel": {
  43. "command": "node",
  44. "args": ["/absolute/path/to/open-web-intel-mcp/dist/mcp.js"]
  45. }
  46. }
  47. }
  48. ```
  49. ### Cursor (`.cursor/mcp.json`)
  50. ```json
  51. {
  52. "mcpServers": {
  53. "open-web-intel": {
  54. "command": "node",
  55. "args": ["/absolute/path/to/open-web-intel-mcp/dist/mcp.js"]
  56. }
  57. }
  58. }
  59. ```
  60. ### Antigravity (`mcp_config.json`)
  61. ```json
  62. {
  63. "mcpServers": {
  64. "open-web-intel": {
  65. "command": "node",
  66. "args": ["/absolute/path/to/open-web-intel-mcp/dist/mcp.js"]
  67. }
  68. }
  69. }
  70. ```
  71. > **Note:** On Windows, pass a forward-slash or escaped path (for example `"C:/path/to/open-web-intel-mcp/dist/mcp.js"`). Once published to npm, you can also invoke a pinned release via `"command": "npx", "args": ["open-web-intel-mcp@1.0.0"]`.
  72. ### Stateless Streamable HTTP Mode
  73. Start the server locally bound to `127.0.0.1` (default) with `Host` and `Origin` DNS-rebinding protection enabled:
  74. ```bash
  75. node dist/mcp.js --http --port 3000
  76. # Health check: GET http://127.0.0.1:3000/health
  77. # MCP endpoint: POST http://127.0.0.1:3000/mcp
  78. # Optional custom Host / Origin allowlists for reverse-proxy or container setups:
  79. node dist/mcp.js --http --host 0.0.0.0 --port 3000 --allowed-hosts mcp.internal:3000 --allowed-origins https://app.internal
  80. ```
  81. ---
  82. ## The 7 MCP Tools
  83. | Tool Name | Purpose | Key Inputs |
  84. | :--- | :--- | :--- |
  85. | `fetch_markdown` | Fetches a public URL, strips scripts/styles/navigation boilerplate, converts to clean Markdown with `title` & `approxTokens`, and optionally probes `/llms.txt`. | `url`, `includeLlmsTxt`, `maxChars` |
  86. | `crawl_sitemap` | Reads `robots.txt` and crawls XML sitemaps (`urlset` & `sitemapindex`) with bounded recursion and URL count limits. | `url`, `maxUrls`, `maxDepth`, `pathFilter` |
  87. | `search_ats_jobs` | Queries direct public JSON/XML career APIs for `greenhouse`, `lever` (US & EU), `ashby`, `workable`, `personio`, and `recruitee`. | `provider`, `company`, `keywords`, `location`, `remoteOnly`, `leverRegion`, `maxJobs` |
  88. | `search_remote_jobs` | Aggregates and deduplicates remote tech listings across `remotive`, `jobicy`, `himalayas`, `remoteok`, and `weworkremotely` with graceful partial-failure handling. | `keywords`, `sources`, `maxJobs` |
  89. | `inspect_ecommerce_store` | Queries public Shopify (`/products.json`, `/meta.json`) and WooCommerce (`/wp-json/wc/store/v1/products`) catalogs plus storefront app/pixel signals. | `url`, `maxProducts` |
  90. | `enrich_domain_intel` | Extracts public contact emails, social profile links, `/llms.txt` status, DNS `MX`/`SPF`/`DMARC` posture, and tech stack signatures. | `domainOrUrl`, `checkContactPage` |
  91. | `search_podcasts_rss` | Queries the Apple Podcasts Search API (`itunes.apple.com/search`) or a direct RSS URL and parses show metadata, owner `<itunes:email>`, and recent episodes. | `query`, `maxShows`, `maxEpisodesPerShow`, `country` |
  92. ---
  93. ## CLI Usage (`open-web-intel`)
  94. Every tool is available directly from the terminal via `open-web-intel` (or `node dist/cli.js`). Pass `--json` on any subcommand for structured JSON output.
  95. ```bash
  96. # Show CLI help and all flags
  97. open-web-intel --help
  98. # 1. Fetch clean Markdown + probe /llms.txt
  99. open-web-intel fetch https://modelcontextprotocol.io/introduction --llms-txt --max-chars 20000
  100. # 2. Discover robots.txt and crawl XML sitemaps
  101. open-web-intel sitemap https://modelcontextprotocol.io --max-urls 100 --filter /docs --json
  102. # 3. Search public ATS job boards (greenhouse | lever | ashby | workable | personio | recruitee)
  103. open-web-intel ats greenhouse stripe --keywords "typescript" --remote-only --max-jobs 15 --json
  104. open-web-intel ats lever spotify --region auto --max-jobs 10
  105. # 4. Aggregate & deduplicate remote jobs across 5 public feeds
  106. open-web-intel remote --keywords "backend typescript" --sources remotive,himalayas,weworkremotely --max-jobs 20 --json
  107. # 5. Inspect a Shopify or WooCommerce storefront
  108. open-web-intel store https://allbirds.com --max-products 10 --json
  109. # 6. Enrich a domain with emails, socials, DNS MX/SPF/DMARC, llms.txt & tech stack
  110. open-web-intel enrich cloudflare.com --json
  111. # 7. Search podcasts & parse RSS feeds for host/owner email and recent episodes
  112. open-web-intel podcasts "software engineering" --max-shows 3 --max-episodes 2 --country US --json
  113. ```
  114. ---
  115. ## Security Design (`src/lib/safe-fetch.ts`)
  116. 1. **Scheme & Credential Enforcement**: Permits only `http:` and `https:` URLs without embedded `username:password` credentials.
  117. 2. **Hostname & Full Multi-Family DNS Validation**: Rejects `localhost`, `.localhost`, `.local`, `.internal`, `metadata.google.internal`, and IPv6 zone IDs (`%`). Resolves all `A` and `AAAA` records (`dns.lookup` with `{ all: true, verbatim: true }`) and blocks the request if **any** returned IP falls in:
  118. - Loopback (`127.0.0.0/8`, `::1`), unspecified (`0.0.0.0/8`, `::`)
  119. - RFC1918 private (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`)
  120. - Carrier-Grade NAT (`100.64.0.0/10`)
  121. - Link-local & Cloud Metadata (`169.254.0.0/16`, `fe80::/10`)
  122. - Unique local IPv6 (`fc00::/7`), site-local (`fec0::/10`), multicast (`224.0.0.0/4`, `ff00::/8`)
  123. - IPv4-mapped/compatible/translated IPv6 (`::ffff:0:0/96`, `::/96`), NAT64 (`64:ff9b::/96`), 6to4 (`2002::/16`), and Teredo (`2001:0000::/32`)
  124. 3. **Socket-Level Connect Pinning (`undici.Agent`) & Manual Redirect Re-Validation**: Eliminates DNS-rebinding TOCTOU by validating all resolved IPs inside a custom `undici.Agent` `connect.lookup` callback and pinning the TCP/TLS socket directly to the validated public IP. Uses `redirect: 'manual'` exclusively (<= 5 hops), re-validating every `Location` target before connecting and stripping sensitive headers across origins.
  125. 4. **Resource & Parser Bounds**: Enforces per-hop `AbortSignal.timeout` deadlines, cancels response streams immediately when cumulative chunk bytes exceed `maxBytes`, caps HTML parser input at 512 KiB, and uses linear `O(n)` `indexOf` tag scanning immune to unclosed-tag ReDoS.
  126. 5. **Streamable HTTP Origin & Host Protection**: Validates `Host` and `Origin` headers on `/mcp` and `/health` before instantiating transports (configurable via `--allowed-hosts` / `ALLOWED_HOSTS` and `--allowed-origins` / `ALLOWED_ORIGINS`).
  127. ---
  128. ## Docker
  129. ```bash
  130. docker build -t open-web-intel-mcp .
  131. docker run --rm -p 127.0.0.1:3000:3000 open-web-intel-mcp --http --host 0.0.0.0 --port 3000
  132. ```
  133. ---
  134. ## Development & Testing
  135. ```bash
  136. npm install
  137. npm run lint # Strict TypeScript check across src/ and tests/
  138. npm run build # Compile ESM output + declarations to dist/
  139. npm test # Run offline SSRF, tool fixture, CLI, and live stdio/HTTP MCP tests
  140. ```
  141. ---
  142. ## Need Managed Cloud Scale, Scheduling & Pay-Per-Event Apify Actors?
  143. Running high-volume scheduled crawls, webhook pipelines, persistent datasets, or multi-company batch jobs without keeping your local machine online? Check out the companion cloud Actors and automation templates:
  144. | Use Case | Managed Cloud Actor / Resource | Highlights |
  145. | :--- | :--- | :--- |
  146. | **Multi-Company ATS Career Feeds** | [ATS Career Jobs Scraper](https://apify.com/tsun_tsun/ats-career-jobs-scraper?fpr=cn0u9m) | Batch-scrape Greenhouse, Lever, Ashby, Workable & SmartRecruiters across dozens of companies on a schedule. |
  147. | **Bulk Shopify Store Intelligence** | [Shopify Store Intel](https://apify.com/tsun_tsun/shopify-store-intel?fpr=cn0u9m) | Large-scale catalog, pricing, app, pixel, and storefront intelligence exported to JSON/CSV/Webhook. |
  148. | **B2B Domain & Tech Stack Enrichment** | [Website Lead & Tech Enricher](https://apify.com/tsun_tsun/website-lead-tech-enricher?fpr=cn0u9m) | Enrich hundreds of domains with public contacts, DNS MX/SPF/DMARC posture, `llms.txt`, and 70+ tech signatures. |
  149. | **Newsletter & Content Archives** | [Substack Intelligence Scraper](https://apify.com/tsun_tsun/substack-intelligence-scraper?fpr=cn0u9m) | Full newsletter archive extraction with clean Markdown conversion and outbound link graphs. |
  150. | **Workflow & n8n Templates Hub** | [Apify Automation Toolkit](https://pie527645-cloud.github.io/apify-automation-toolkit/) | Ready-to-import automation blueprints, scheduling guides, and integration recipes. |
  151. ---
  152. ## License
  153. [MIT](./LICENSE)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.