web-insights-mcp
MCP server for web research and SEO: fetch, crawl, audit, extract and compare pages. stdio + Streamable HTTP, SSRF-safe.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/nqcthedev--web-insights-mcp)HTML
<a href="/listings/nqcthedev--web-insights-mcp"><img src="/api/listings/nqcthedev--web-insights-mcp/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Requires **Node.js 22.19+**.
> The package is prepared for npm (`bin`, `files`, `prepublishOnly`) but is **not published yet**. Until it is, use the > "from source" variants below; after publishing, the `npx -y web-insights-mcp` variants work as written.
git clone <this repo> web-insights-mcp && cd web-insights-mcp pnpm install && pnpm build # produces dist/cli.js node dist/cli.js --help
Claude Code
# stdio (recommended) claude mcp add web-insights -- npx -y web-insights-mcp # from source claude mcp add web-insights -- node /absolute/path/to/web-insights-mcp/dist/cli.js # available in every project claude mcp add --scope user web-insights -- npx -y web-insights-mcp # with settings claude mcp add web-insights -e WEB_INSIGHTS_TIMEOUT_MS=20000 -- npx -y web-insights-mcp # over HTTP (start the server first: npx web-insights-mcp --http) claude mcp add --transport http web-insights http://127.0.0.1:3400/mcp
Claude Desktop
Settings → Developer → Edit Config (`claude_desktop_config.json`), then restart Claude Desktop:
{
"mcpServers": {
"web-insights": {
"command": "npx",
"args": ["-y", "web-insights-mcp"]
}
}
}From source, use `"command": "node", "args": ["/absolute/path/to/web-insights-mcp/dist/cli.js"]`. See [`examples/claude_desktop_config.json`](examples/claude_desktop_config.json) for a version with environment settings.
Cursor
`~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project) — same shape as above ([`examples/cursor-mcp.json`](examples/cursor-mcp.json)); for HTTP use `{"url": "http://127.0.0.1:3400/mcp"}` ([`examples/cursor-mcp-http.json`](examples/cursor-mcp-http.json)).
Streamable HTTP
npx web-insights-mcp --http # http://127.0.0.1:3400/mcp, GET /healthz for probes npx web-insights-mcp --http --port 8080 WEB_INSIGHTS_HTTP_TOKEN=change-me npx web-insights-mcp --http --host 0.0.0.0 # public bind requires a token
The HTTP transport is stateless (one MCP session per request; cache and limits are shared), so it scales horizontally behind a load balancer.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 223 lines
- # web-insights-mcp
- **An MCP server that gives Claude (and any MCP client) safe, structured access to the web for research and SEO work.**
- Read any page as clean Markdown, map a site, run on-page SEO audits with concrete fixes, pull out structured data and
- compare a page against a competitor — with SSRF protection, timeouts, size limits and polite crawling built in.
-    
- 
- ## Why
- LLM agents are good at reasoning about web content but bad at getting it: raw HTML wastes the context window, ad-hoc
- `fetch` tools happily request `http://169.254.169.254/` (cloud metadata) or your intranet, and "SEO audits" written
- by the model itself are inconsistent from run to run. This server does the deterministic parts in code and hands the
- model compact, typed results it can act on:
- - **Clean input for the model** — Mozilla Readability (the Firefox Reader View algorithm) plus Turndown turns a page into
- Markdown with absolute links and GFM tables — usually a small fraction of the raw HTML size.
- - **Repeatable audits** — the same page always gets the same checks, severities and score.
- - **Safe by default** — private, loopback, link-local and metadata addresses are blocked after DNS resolution, on every
- redirect hop and again at connect time.
- ## Tools
- | Tool | What it does | Typical prompt |
- |---|---|---|
- | `fetch_page` | URL → main content as Markdown + title, description, language, canonical, byline, word count. `article` or `full` mode, `max_chars` cap, optional link list. | "Summarise this article and quote the key numbers." |
- | `crawl_site` | Discover pages via sitemap.xml (incl. sitemap indexes) or same-origin BFS. Page/depth caps, path prefix, robots.txt + Crawl-delay honoured. Returns URL, status, title, redirects. | "List every blog post on this site and flag broken pages." |
- | `seo_audit` | 25+ on-page checks: status/HTTPS/redirect chains, title & meta description length, H1 count, heading order, canonical, meta robots + `X-Robots-Tag`, viewport, lang, charset, Open Graph, Twitter card, image alt, JSON-LD presence/validity/`@context`/`@type`, internal links, empty and generic anchors, thin content. Each issue has severity + fix; 0-100 score. | "Audit our pricing page and give me a prioritised fix list." |
- | `site_seo_audit` | Crawl + audit every page, then aggregate: most common issues (with page counts), duplicate titles/descriptions, broken pages, worst pages, average score. | "Do an SEO health check of the first 30 pages." |
- | `extract_structured` | JSON-LD (parsed, with errors), microdata (nested items), Open Graph/Twitter, contacts (emails, phones, social profiles, contact pages), heading outline, HTML tables → JSON rows. | "Get the product prices and ratings from this page." |
- | `compare_pages` | Two URLs side by side: title, description, H1, word count, sub-topics only in A / only in B, top keywords and phrases, structured-data types, link counts, plain-English observations. | "What does the competitor's guide cover that ours doesn't?" |
- Every tool has zod-validated input, a JSON **output schema** (`structuredContent`) for programmatic clients, a readable
- text rendering for chat clients, and `readOnlyHint`/`openWorldHint` annotations. Errors come back as tool errors
- (`isError: true`) with a message the model can act on — never as a crashed session.
- ## Demo
- All output below was recorded by `pnpm demo`, which spawns the **built server over stdio** with the official MCP
- client and runs it against a fictional fixture site served locally. Full, unedited transcript:
- [`docs/demo/transcript.md`](docs/demo/transcript.md).
- | | |
- |---|---|
- |  |  |
- |  |  |
- Excerpt (`site_seo_audit`, text content as the model sees it):
- ```text
- ## Site SEO audit: http://127.0.0.1:3401/
- 8 page(s) audited (sitemap) · average score **79/100** · 1 broken.
- ### Most common issues
- - [error] title — 1 page(s)
- - [error] canonical — 1 page(s)
- - [error] viewport — 1 page(s)
- - [error] json-ld-valid — 1 page(s)
- - [warning] https — 8 page(s)
- - [warning] open-graph — 4 page(s)
- - [warning] image-alt — 2 page(s)
- ...
- ### Broken pages
- - http://127.0.0.1:3401/discontinued-product — 404
- ```
- And the SSRF guard, from the same session:
- ```text
- fetch_page(http://169.254.169.254/latest/meta-data/) failed: Blocked by SSRF protection:
- 169.254.169.254 is a linkLocal address, not a public internet address.
- This URL cannot be fetched by design; do not retry it.
- ```
- ## Install
- Requires **Node.js 22.19+**.
- > The package is prepared for npm (`bin`, `files`, `prepublishOnly`) but is **not published yet**. Until it is, use the
- > "from source" variants below; after publishing, the `npx -y web-insights-mcp` variants work as written.
- ```bash
- git clone <this repo> web-insights-mcp && cd web-insights-mcp
- pnpm install && pnpm build # produces dist/cli.js
- node dist/cli.js --help
- ```
- ### Claude Code
- ```bash
- # stdio (recommended)
- claude mcp add web-insights -- npx -y web-insights-mcp
- # from source
- claude mcp add web-insights -- node /absolute/path/to/web-insights-mcp/dist/cli.js
- # available in every project
- claude mcp add --scope user web-insights -- npx -y web-insights-mcp
- # with settings
- claude mcp add web-insights -e WEB_INSIGHTS_TIMEOUT_MS=20000 -- npx -y web-insights-mcp
- # over HTTP (start the server first: npx web-insights-mcp --http)
- claude mcp add --transport http web-insights http://127.0.0.1:3400/mcp
- ```
- ### Claude Desktop
- Settings → Developer → Edit Config (`claude_desktop_config.json`), then restart Claude Desktop:
- ```json
- {
- "mcpServers": {
- "web-insights": {
- "command": "npx",
- "args": ["-y", "web-insights-mcp"]
- }
- }
- }
- ```
- From source, use `"command": "node", "args": ["/absolute/path/to/web-insights-mcp/dist/cli.js"]`.
- See [`examples/claude_desktop_config.json`](examples/claude_desktop_config.json) for a version with environment settings.
- ### Cursor
- `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project) — same shape as above
- ([`examples/cursor-mcp.json`](examples/cursor-mcp.json)); for HTTP use `{"url": "http://127.0.0.1:3400/mcp"}`
- ([`examples/cursor-mcp-http.json`](examples/cursor-mcp-http.json)).
- ### Streamable HTTP
- ```bash
- npx web-insights-mcp --http # http://127.0.0.1:3400/mcp, GET /healthz for probes
- npx web-insights-mcp --http --port 8080
- WEB_INSIGHTS_HTTP_TOKEN=change-me npx web-insights-mcp --http --host 0.0.0.0 # public bind requires a token
- ```
- The HTTP transport is stateless (one MCP session per request; cache and limits are shared), so it scales horizontally
- behind a load balancer.
- ## Configuration
- All optional, via environment variables:
- | Variable | Default | Purpose |
- |---|---|---|
- | `WEB_INSIGHTS_TIMEOUT_MS` | `15000` | Per-request timeout (whole request incl. redirects and body). |
- | `WEB_INSIGHTS_MAX_BYTES` | `5242880` | Max decoded response size; larger bodies are aborted mid-stream. |
- | `WEB_INSIGHTS_MAX_REDIRECTS` | `5` | Redirect hops per request. |
- | `WEB_INSIGHTS_CONCURRENCY` | `4` | Max outbound requests in flight, shared by all tools and sessions. |
- | `WEB_INSIGHTS_CACHE_ENTRIES` / `_CACHE_TTL_MS` | `200` / `300000` | In-memory LRU cache (0 disables). |
- | `WEB_INSIGHTS_MAX_CRAWL_PAGES` | `200` | Hard cap for `crawl_site` (and 100 for `site_seo_audit`). |
- | `WEB_INSIGHTS_USER_AGENT` | `web-insights-mcp/0.1.0 (+…; polite research bot)` | Identify your bot honestly. |
- | `WEB_INSIGHTS_ALLOW_LOCALHOST` | off | `1` allows **loopback only** (for testing against a local dev server). |
- | `WEB_INSIGHTS_HTTP_TOKEN` | – | Require `Authorization: Bearer <token>` on `/mcp`. |
- | `WEB_INSIGHTS_ALLOWED_HOSTS` | – | Extra `Host` header values to accept (reverse proxy / public hostname). |
- ## Security notes
- - **SSRF protection.** Only `http:`/`https:` URLs without embedded credentials are fetched. Hostnames are resolved and
- the request is refused if *any* answer is not a globally routable unicast address: RFC 1918 private ranges, loopback,
- link-local (incl. `169.254.169.254` cloud metadata), CGNAT, unique-local IPv6, multicast, unspecified (`0.0.0.0`),
- IPv4-mapped/NAT64/6to4 addresses that embed one of those, and odd IPv4 spellings like `2130706433` or `0x7f000001`.
- - **No TOCTOU / DNS rebinding.** The same check runs inside the socket `lookup` used by the HTTP agent, so the address
- actually connected to is the one that was validated. Every redirect hop is re-validated.
- - **Resource limits.** Timeouts, streamed size cap (checked on decompressed bytes, so gzip bombs are cut off),
- redirect cap, global concurrency limit, crawl page/depth caps, content-type allow-list (HTML/XML/text only).
- - **Polite crawling.** robots.txt is always honoured (no override), Crawl-delay is respected (capped at 5 s, crawl drops
- to 1 request at a time), tracking parameters are stripped to avoid duplicate fetches, and the User-Agent identifies the bot.
- - **HTTP transport.** Binds to `127.0.0.1` by default with `Host`/`Origin` validation against DNS rebinding;
- refuses to bind to a non-loopback interface unless a bearer token is set; token comparison is constant-time.
- - **Read-only.** Tools only issue `GET` requests and never write anywhere. Page content is untrusted input: like any web
- tool, a page can contain text aimed at manipulating the model (prompt injection), so keep a human in the loop for
- consequential actions.
- ## Development
- ```bash
- pnpm install
- pnpm typecheck # tsc --noEmit, strict
- pnpm test # build + 89 vitest tests (unit, tools, SSRF, stdio e2e, HTTP e2e)
- pnpm demo # regenerate docs/demo/transcript.md and docs/screenshots (needs Playwright + Chromium)
- ```
- Tests never touch the internet: `test/fixtures/server.ts` serves a fictional coffee-roaster site (sitemap index,
- robots.txt, redirects, a deliberately broken legacy page, gzip/latin-1 pages, slow/huge/hostile routes) on a random
- local port. The e2e suites spawn `dist/cli.js` exactly as an MCP client would — over stdio with the SDK's
- `StdioClientTransport`, and over HTTP with `StreamableHTTPClientTransport`.
- ```
- src/
- cli.ts CLI entry (stdio | --http)
- server.ts McpServer factory, tool registration
- http.ts Streamable HTTP transport, host/origin/token checks
- net/ssrf.ts IP classification, DNS validation, guarded lookup
- net/fetcher.ts undici client: redirects, limits, cache, concurrency
- net/crawl.ts robots.txt, sitemaps, BFS crawler
- parse/ cheerio facts, Readability → Markdown, keyword stats
- seo/audit.ts on-page checks, scoring, site aggregation
- tools/ one file per MCP tool (schemas + handlers)
- ```
- ## Limitations (honest list)
- - **No JavaScript rendering.** Pages are fetched as HTML; content that only appears after client-side rendering (some
- SPAs) is not seen. Server-rendered sites, blogs, docs and most marketing sites work well.
- - **On-page SEO only.** No Core Web Vitals/Lighthouse, backlinks, rankings, search volume or SERP data — those need
- paid APIs or a browser. Keyword statistics are frequency-based (stopwords removed), not semantic.
- - **Heuristic extraction.** Readability can misjudge unusual layouts; use `mode: "full"` when the article view drops
- content. Microdata support covers the common cases, not every edge of the spec.
- - **Crawl scope.** Same origin only; `.xml.gz` sitemaps are not decompressed; disallowed pages cannot be audited.
- - **Networking.** Requests go direct (system HTTP proxies are not used). Loopback can be enabled for testing, but other
- private networks cannot — auditing an intranet site is intentionally out of scope.
- - **Verification scope.** The automated tests run against local fixtures only (CI has no live-internet tests), so
- real-world sites with unusual markup may expose parser edge cases. Bug reports with a URL are welcome.
- - **HTTP auth** is a static bearer token, not OAuth. Put it behind your own gateway for multi-tenant use.
- ## License
- MIT © Cuong Nguyen
- ---
- Built by NQC The Dev — AI developer. Available for hire on Fiverr: https://www.fiverr.com/cuongnqxx
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.