Skip to content
Back to search

web-insights-mcp

MCP server for web research and SEO: fetch, crawl, audit, extract and compare pages. stdio + Streamable HTTP, SSRF-safe.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nqcthedev--web-insights-mcp/badge.svg)](/listings/nqcthedev--web-insights-mcp)

HTML

<a href="/listings/nqcthedev--web-insights-mcp"><img src="/api/listings/nqcthedev--web-insights-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires **Node.js 22.19+**.

> The package is prepared for npm (`bin`, `files`, `prepublishOnly`) but is **not published yet**. Until it is, use the > "from source" variants below; after publishing, the `npx -y web-insights-mcp` variants work as written.

git clone <this repo> web-insights-mcp && cd web-insights-mcp
pnpm install && pnpm build          # produces dist/cli.js
node dist/cli.js --help

Claude Code

# stdio (recommended)
claude mcp add web-insights -- npx -y web-insights-mcp
# from source
claude mcp add web-insights -- node /absolute/path/to/web-insights-mcp/dist/cli.js
# available in every project
claude mcp add --scope user web-insights -- npx -y web-insights-mcp
# with settings
claude mcp add web-insights -e WEB_INSIGHTS_TIMEOUT_MS=20000 -- npx -y web-insights-mcp
# over HTTP (start the server first: npx web-insights-mcp --http)
claude mcp add --transport http web-insights http://127.0.0.1:3400/mcp

Claude Desktop

Settings → Developer → Edit Config (`claude_desktop_config.json`), then restart Claude Desktop:

{
  "mcpServers": {
    "web-insights": {
      "command": "npx",
      "args": ["-y", "web-insights-mcp"]
    }
  }
}

From source, use `"command": "node", "args": ["/absolute/path/to/web-insights-mcp/dist/cli.js"]`. See [`examples/claude_desktop_config.json`](examples/claude_desktop_config.json) for a version with environment settings.

Cursor

`~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project) — same shape as above ([`examples/cursor-mcp.json`](examples/cursor-mcp.json)); for HTTP use `{"url": "http://127.0.0.1:3400/mcp"}` ([`examples/cursor-mcp-http.json`](examples/cursor-mcp-http.json)).

Streamable HTTP

npx web-insights-mcp --http                    # http://127.0.0.1:3400/mcp, GET /healthz for probes
npx web-insights-mcp --http --port 8080
WEB_INSIGHTS_HTTP_TOKEN=change-me npx web-insights-mcp --http --host 0.0.0.0   # public bind requires a token

The HTTP transport is stateless (one MCP session per request; cache and limits are shared), so it scales horizontally behind a load balancer.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 223 lines
  1. # web-insights-mcp
  2. **An MCP server that gives Claude (and any MCP client) safe, structured access to the web for research and SEO work.**
  3. Read any page as clean Markdown, map a site, run on-page SEO audits with concrete fixes, pull out structured data and
  4. compare a page against a competitor — with SSRF protection, timeouts, size limits and polite crawling built in.
  5. ![CI](https://img.shields.io/badge/tests-89%20passing-2ea44f) ![TypeScript](https://img.shields.io/badge/TypeScript-strict-3178c6) ![MCP](https://img.shields.io/badge/MCP-stdio%20%2B%20Streamable%20HTTP-7c5cff) ![License](https://img.shields.io/badge/license-MIT-blue)
  6. ![seo_audit result](docs/screenshots/seo-audit.png)
  7. ## Why
  8. LLM agents are good at reasoning about web content but bad at getting it: raw HTML wastes the context window, ad-hoc
  9. `fetch` tools happily request `http://169.254.169.254/` (cloud metadata) or your intranet, and "SEO audits" written
  10. by the model itself are inconsistent from run to run. This server does the deterministic parts in code and hands the
  11. model compact, typed results it can act on:
  12. - **Clean input for the model** — Mozilla Readability (the Firefox Reader View algorithm) plus Turndown turns a page into
  13. Markdown with absolute links and GFM tables — usually a small fraction of the raw HTML size.
  14. - **Repeatable audits** — the same page always gets the same checks, severities and score.
  15. - **Safe by default** — private, loopback, link-local and metadata addresses are blocked after DNS resolution, on every
  16. redirect hop and again at connect time.
  17. ## Tools
  18. | Tool | What it does | Typical prompt |
  19. |---|---|---|
  20. | `fetch_page` | URL → main content as Markdown + title, description, language, canonical, byline, word count. `article` or `full` mode, `max_chars` cap, optional link list. | "Summarise this article and quote the key numbers." |
  21. | `crawl_site` | Discover pages via sitemap.xml (incl. sitemap indexes) or same-origin BFS. Page/depth caps, path prefix, robots.txt + Crawl-delay honoured. Returns URL, status, title, redirects. | "List every blog post on this site and flag broken pages." |
  22. | `seo_audit` | 25+ on-page checks: status/HTTPS/redirect chains, title & meta description length, H1 count, heading order, canonical, meta robots + `X-Robots-Tag`, viewport, lang, charset, Open Graph, Twitter card, image alt, JSON-LD presence/validity/`@context`/`@type`, internal links, empty and generic anchors, thin content. Each issue has severity + fix; 0-100 score. | "Audit our pricing page and give me a prioritised fix list." |
  23. | `site_seo_audit` | Crawl + audit every page, then aggregate: most common issues (with page counts), duplicate titles/descriptions, broken pages, worst pages, average score. | "Do an SEO health check of the first 30 pages." |
  24. | `extract_structured` | JSON-LD (parsed, with errors), microdata (nested items), Open Graph/Twitter, contacts (emails, phones, social profiles, contact pages), heading outline, HTML tables → JSON rows. | "Get the product prices and ratings from this page." |
  25. | `compare_pages` | Two URLs side by side: title, description, H1, word count, sub-topics only in A / only in B, top keywords and phrases, structured-data types, link counts, plain-English observations. | "What does the competitor's guide cover that ours doesn't?" |
  26. Every tool has zod-validated input, a JSON **output schema** (`structuredContent`) for programmatic clients, a readable
  27. text rendering for chat clients, and `readOnlyHint`/`openWorldHint` annotations. Errors come back as tool errors
  28. (`isError: true`) with a message the model can act on — never as a crashed session.
  29. ## Demo
  30. All output below was recorded by `pnpm demo`, which spawns the **built server over stdio** with the official MCP
  31. client and runs it against a fictional fixture site served locally. Full, unedited transcript:
  32. [`docs/demo/transcript.md`](docs/demo/transcript.md).
  33. | | |
  34. |---|---|
  35. | ![tools/list](docs/screenshots/tools.png) | ![fetch_page](docs/screenshots/fetch-page.png) |
  36. | ![compare_pages](docs/screenshots/compare-pages.png) | ![seo_audit](docs/screenshots/seo-audit.png) |
  37. Excerpt (`site_seo_audit`, text content as the model sees it):
  38. ```text
  39. ## Site SEO audit: http://127.0.0.1:3401/
  40. 8 page(s) audited (sitemap) · average score **79/100** · 1 broken.
  41. ### Most common issues
  42. - [error] title — 1 page(s)
  43. - [error] canonical — 1 page(s)
  44. - [error] viewport — 1 page(s)
  45. - [error] json-ld-valid — 1 page(s)
  46. - [warning] https — 8 page(s)
  47. - [warning] open-graph — 4 page(s)
  48. - [warning] image-alt — 2 page(s)
  49. ...
  50. ### Broken pages
  51. - http://127.0.0.1:3401/discontinued-product — 404
  52. ```
  53. And the SSRF guard, from the same session:
  54. ```text
  55. fetch_page(http://169.254.169.254/latest/meta-data/) failed: Blocked by SSRF protection:
  56. 169.254.169.254 is a linkLocal address, not a public internet address.
  57. This URL cannot be fetched by design; do not retry it.
  58. ```
  59. ## Install
  60. Requires **Node.js 22.19+**.
  61. > The package is prepared for npm (`bin`, `files`, `prepublishOnly`) but is **not published yet**. Until it is, use the
  62. > "from source" variants below; after publishing, the `npx -y web-insights-mcp` variants work as written.
  63. ```bash
  64. git clone <this repo> web-insights-mcp && cd web-insights-mcp
  65. pnpm install && pnpm build # produces dist/cli.js
  66. node dist/cli.js --help
  67. ```
  68. ### Claude Code
  69. ```bash
  70. # stdio (recommended)
  71. claude mcp add web-insights -- npx -y web-insights-mcp
  72. # from source
  73. claude mcp add web-insights -- node /absolute/path/to/web-insights-mcp/dist/cli.js
  74. # available in every project
  75. claude mcp add --scope user web-insights -- npx -y web-insights-mcp
  76. # with settings
  77. claude mcp add web-insights -e WEB_INSIGHTS_TIMEOUT_MS=20000 -- npx -y web-insights-mcp
  78. # over HTTP (start the server first: npx web-insights-mcp --http)
  79. claude mcp add --transport http web-insights http://127.0.0.1:3400/mcp
  80. ```
  81. ### Claude Desktop
  82. Settings → Developer → Edit Config (`claude_desktop_config.json`), then restart Claude Desktop:
  83. ```json
  84. {
  85. "mcpServers": {
  86. "web-insights": {
  87. "command": "npx",
  88. "args": ["-y", "web-insights-mcp"]
  89. }
  90. }
  91. }
  92. ```
  93. From source, use `"command": "node", "args": ["/absolute/path/to/web-insights-mcp/dist/cli.js"]`.
  94. See [`examples/claude_desktop_config.json`](examples/claude_desktop_config.json) for a version with environment settings.
  95. ### Cursor
  96. `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project) — same shape as above
  97. ([`examples/cursor-mcp.json`](examples/cursor-mcp.json)); for HTTP use `{"url": "http://127.0.0.1:3400/mcp"}`
  98. ([`examples/cursor-mcp-http.json`](examples/cursor-mcp-http.json)).
  99. ### Streamable HTTP
  100. ```bash
  101. npx web-insights-mcp --http # http://127.0.0.1:3400/mcp, GET /healthz for probes
  102. npx web-insights-mcp --http --port 8080
  103. WEB_INSIGHTS_HTTP_TOKEN=change-me npx web-insights-mcp --http --host 0.0.0.0 # public bind requires a token
  104. ```
  105. The HTTP transport is stateless (one MCP session per request; cache and limits are shared), so it scales horizontally
  106. behind a load balancer.
  107. ## Configuration
  108. All optional, via environment variables:
  109. | Variable | Default | Purpose |
  110. |---|---|---|
  111. | `WEB_INSIGHTS_TIMEOUT_MS` | `15000` | Per-request timeout (whole request incl. redirects and body). |
  112. | `WEB_INSIGHTS_MAX_BYTES` | `5242880` | Max decoded response size; larger bodies are aborted mid-stream. |
  113. | `WEB_INSIGHTS_MAX_REDIRECTS` | `5` | Redirect hops per request. |
  114. | `WEB_INSIGHTS_CONCURRENCY` | `4` | Max outbound requests in flight, shared by all tools and sessions. |
  115. | `WEB_INSIGHTS_CACHE_ENTRIES` / `_CACHE_TTL_MS` | `200` / `300000` | In-memory LRU cache (0 disables). |
  116. | `WEB_INSIGHTS_MAX_CRAWL_PAGES` | `200` | Hard cap for `crawl_site` (and 100 for `site_seo_audit`). |
  117. | `WEB_INSIGHTS_USER_AGENT` | `web-insights-mcp/0.1.0 (+…; polite research bot)` | Identify your bot honestly. |
  118. | `WEB_INSIGHTS_ALLOW_LOCALHOST` | off | `1` allows **loopback only** (for testing against a local dev server). |
  119. | `WEB_INSIGHTS_HTTP_TOKEN` | – | Require `Authorization: Bearer <token>` on `/mcp`. |
  120. | `WEB_INSIGHTS_ALLOWED_HOSTS` | – | Extra `Host` header values to accept (reverse proxy / public hostname). |
  121. ## Security notes
  122. - **SSRF protection.** Only `http:`/`https:` URLs without embedded credentials are fetched. Hostnames are resolved and
  123. the request is refused if *any* answer is not a globally routable unicast address: RFC 1918 private ranges, loopback,
  124. link-local (incl. `169.254.169.254` cloud metadata), CGNAT, unique-local IPv6, multicast, unspecified (`0.0.0.0`),
  125. IPv4-mapped/NAT64/6to4 addresses that embed one of those, and odd IPv4 spellings like `2130706433` or `0x7f000001`.
  126. - **No TOCTOU / DNS rebinding.** The same check runs inside the socket `lookup` used by the HTTP agent, so the address
  127. actually connected to is the one that was validated. Every redirect hop is re-validated.
  128. - **Resource limits.** Timeouts, streamed size cap (checked on decompressed bytes, so gzip bombs are cut off),
  129. redirect cap, global concurrency limit, crawl page/depth caps, content-type allow-list (HTML/XML/text only).
  130. - **Polite crawling.** robots.txt is always honoured (no override), Crawl-delay is respected (capped at 5 s, crawl drops
  131. to 1 request at a time), tracking parameters are stripped to avoid duplicate fetches, and the User-Agent identifies the bot.
  132. - **HTTP transport.** Binds to `127.0.0.1` by default with `Host`/`Origin` validation against DNS rebinding;
  133. refuses to bind to a non-loopback interface unless a bearer token is set; token comparison is constant-time.
  134. - **Read-only.** Tools only issue `GET` requests and never write anywhere. Page content is untrusted input: like any web
  135. tool, a page can contain text aimed at manipulating the model (prompt injection), so keep a human in the loop for
  136. consequential actions.
  137. ## Development
  138. ```bash
  139. pnpm install
  140. pnpm typecheck # tsc --noEmit, strict
  141. pnpm test # build + 89 vitest tests (unit, tools, SSRF, stdio e2e, HTTP e2e)
  142. pnpm demo # regenerate docs/demo/transcript.md and docs/screenshots (needs Playwright + Chromium)
  143. ```
  144. Tests never touch the internet: `test/fixtures/server.ts` serves a fictional coffee-roaster site (sitemap index,
  145. robots.txt, redirects, a deliberately broken legacy page, gzip/latin-1 pages, slow/huge/hostile routes) on a random
  146. local port. The e2e suites spawn `dist/cli.js` exactly as an MCP client would — over stdio with the SDK's
  147. `StdioClientTransport`, and over HTTP with `StreamableHTTPClientTransport`.
  148. ```
  149. src/
  150. cli.ts CLI entry (stdio | --http)
  151. server.ts McpServer factory, tool registration
  152. http.ts Streamable HTTP transport, host/origin/token checks
  153. net/ssrf.ts IP classification, DNS validation, guarded lookup
  154. net/fetcher.ts undici client: redirects, limits, cache, concurrency
  155. net/crawl.ts robots.txt, sitemaps, BFS crawler
  156. parse/ cheerio facts, Readability → Markdown, keyword stats
  157. seo/audit.ts on-page checks, scoring, site aggregation
  158. tools/ one file per MCP tool (schemas + handlers)
  159. ```
  160. ## Limitations (honest list)
  161. - **No JavaScript rendering.** Pages are fetched as HTML; content that only appears after client-side rendering (some
  162. SPAs) is not seen. Server-rendered sites, blogs, docs and most marketing sites work well.
  163. - **On-page SEO only.** No Core Web Vitals/Lighthouse, backlinks, rankings, search volume or SERP data — those need
  164. paid APIs or a browser. Keyword statistics are frequency-based (stopwords removed), not semantic.
  165. - **Heuristic extraction.** Readability can misjudge unusual layouts; use `mode: "full"` when the article view drops
  166. content. Microdata support covers the common cases, not every edge of the spec.
  167. - **Crawl scope.** Same origin only; `.xml.gz` sitemaps are not decompressed; disallowed pages cannot be audited.
  168. - **Networking.** Requests go direct (system HTTP proxies are not used). Loopback can be enabled for testing, but other
  169. private networks cannot — auditing an intranet site is intentionally out of scope.
  170. - **Verification scope.** The automated tests run against local fixtures only (CI has no live-internet tests), so
  171. real-world sites with unusual markup may expose parser edge cases. Bug reports with a URL are welcome.
  172. - **HTTP auth** is a static bearer token, not OAuth. Put it behind your own gateway for multi-tenant use.
  173. ## License
  174. MIT © Cuong Nguyen
  175. ---
  176. Built by NQC The Dev — AI developer. Available for hire on Fiverr: https://www.fiverr.com/cuongnqxx

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.