Skip to content
Back to search

fetcher-mcp

MCP server for fetch web page content using Playwright headless browser.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/jae-jae--fetcher-mcp/badge.svg)](/listings/jae-jae--fetcher-mcp)

HTML

<a href="/listings/jae-jae--fetcher-mcp"><img src="/api/listings/jae-jae--fetcher-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Run directly with npx:

npx -y fetcher-mcp

First time setup - install the required browser by running the following command in your terminal:

npx playwright install chromium

HTTP and SSE Transport

Use the `--transport=http` parameter to start both Streamable HTTP endpoint and SSE endpoint services simultaneously:

npx -y fetcher-mcp --log --transport=http --host=0.0.0.0 --port=3000

After startup, the server provides the following endpoints:

- `/mcp` - Streamable HTTP endpoint (modern MCP protocol) - `/sse` - SSE endpoint (legacy MCP protocol)

Clients can choose which method to connect based on their needs.

Debug Mode

Run with the `--debug` option to show the browser window for debugging:

npx -y fetcher-mcp --debug

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 305 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. <img src="https://raw.githubusercontent.com/jae-jae/fetcher-mcp/refs/heads/main/icon.svg" width="100" height="100" alt="Fetcher MCP Icon" />
  3. </div>
  4. [中文](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=zh) |
  5. [Deutsch](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=de) |
  6. [Español](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=es) |
  7. [français](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=fr) |
  8. [日本語](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=ja) |
  9. [한국어](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=ko) |
  10. [Português](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=pt) |
  11. [Русский](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=ru)
  12. # Fetcher MCP
  13. MCP server for fetch web page content using Playwright headless browser.
  14. > 🌟 **Recommended**: [OllaMan](https://ollaman.com/) - Powerful Ollama AI Model Manager.
  15. ## Advantages
  16. - **JavaScript Support**: Unlike traditional web scrapers, Fetcher MCP uses Playwright to execute JavaScript, making it capable of handling dynamic web content and modern web applications.
  17. - **Intelligent Content Extraction**: Built-in Readability algorithm automatically extracts the main content from web pages, removing ads, navigation, and other non-essential elements.
  18. - **Flexible Output Format**: Supports both HTML and Markdown output formats, making it easy to integrate with various downstream applications.
  19. - **Parallel Processing**: The `fetch_urls` tool enables concurrent fetching of multiple URLs, significantly improving efficiency for batch operations.
  20. - **Resource Optimization**: Automatically blocks unnecessary resources (images, stylesheets, fonts, media) to reduce bandwidth usage and improve performance.
  21. - **Robust Error Handling**: Comprehensive error handling and logging ensure reliable operation even when dealing with problematic web pages.
  22. - **Configurable Parameters**: Fine-grained control over timeouts, content extraction, and output formatting to suit different use cases.
  23. ## Quick Start
  24. Run directly with npx:
  25. ```bash
  26. npx -y fetcher-mcp
  27. ```
  28. First time setup - install the required browser by running the following command in your terminal:
  29. ```bash
  30. npx playwright install chromium
  31. ```
  32. ### HTTP and SSE Transport
  33. Use the `--transport=http` parameter to start both Streamable HTTP endpoint and SSE endpoint services simultaneously:
  34. ```bash
  35. npx -y fetcher-mcp --log --transport=http --host=0.0.0.0 --port=3000
  36. ```
  37. After startup, the server provides the following endpoints:
  38. - `/mcp` - Streamable HTTP endpoint (modern MCP protocol)
  39. - `/sse` - SSE endpoint (legacy MCP protocol)
  40. - `/openapi.json` - OpenAPI 3.1 specification for the REST tool API (HTTP mode only)
  41. - `POST /tools/{name}` - REST endpoint for each tool (`fetch_url`, `fetch_urls`, `browser_install`)
  42. Clients can choose which method to connect based on their needs. Stdio mode does not serve the HTTP, SSE, or REST routes.
  43. > **No authentication.** `/openapi.json` and `POST /tools/{name}` do not authenticate callers. That includes `browser_install`, which installs a browser on the host. Use these endpoints only on localhost or a trusted internal network. Do not expose them on an untrusted network (for example by binding `--host=0.0.0.0` where the port is reachable from outside that network).
  44. ### Debug Mode
  45. Run with the `--debug` option to show the browser window for debugging:
  46. ```bash
  47. npx -y fetcher-mcp --debug
  48. ```
  49. ## Configuration MCP
  50. Configure this MCP server in Claude Desktop:
  51. On MacOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
  52. On Windows: `%APPDATA%/Claude/claude_desktop_config.json`
  53. ```json
  54. {
  55. "mcpServers": {
  56. "fetcher": {
  57. "command": "npx",
  58. "args": ["-y", "fetcher-mcp"]
  59. }
  60. }
  61. }
  62. ```
  63. ## Docker Deployment
  64. ### Running with Docker
  65. ```bash
  66. docker run -p 3000:3000 ghcr.io/jae-jae/fetcher-mcp:latest
  67. ```
  68. ### Deploying with Docker Compose
  69. Create a `docker-compose.yml` file:
  70. ```yaml
  71. version: "3.8"
  72. services:
  73. fetcher-mcp:
  74. image: ghcr.io/jae-jae/fetcher-mcp:latest
  75. container_name: fetcher-mcp
  76. restart: unless-stopped
  77. ports:
  78. - "3000:3000"
  79. environment:
  80. - NODE_ENV=production
  81. # Using host network mode on Linux hosts can improve browser access efficiency
  82. # network_mode: "host"
  83. volumes:
  84. # For Playwright, may need to share certain system paths
  85. - /tmp:/tmp
  86. # Health check
  87. healthcheck:
  88. test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000"]
  89. interval: 30s
  90. timeout: 10s
  91. retries: 3
  92. ```
  93. Then run:
  94. ```bash
  95. docker-compose up -d
  96. ```
  97. ## Features
  98. - `fetch_url` - Retrieve web page content from a specified URL
  99. - Uses Playwright headless browser to parse JavaScript
  100. - Supports intelligent extraction of main content and conversion to Markdown
  101. - Supports the following parameters:
  102. - `url`: The URL of the web page to fetch (required parameter)
  103. - `timeout`: Page loading timeout in milliseconds, default is 30000 (30 seconds)
  104. - `waitUntil`: Specifies when navigation is considered complete, options: 'load', 'domcontentloaded', 'networkidle', 'commit', default is 'load'
  105. - `extractContent`: Whether to intelligently extract the main content, default is true
  106. - `maxLength`: Maximum length of returned content (in characters), default is no limit
  107. - `returnHtml`: Whether to return HTML content instead of Markdown, default is false
  108. - `returnText`: Whether to return plain text (`innerText` of the page body) instead of Markdown. Skips Readability extraction and Markdown conversion. Default is false. When omitted, the response is the same Markdown output as before
  109. - `waitForNavigation`: Whether to wait for additional navigation after initial page load (useful for sites with anti-bot verification), default is false
  110. - `navigationTimeout`: Maximum time to wait for additional navigation in milliseconds, default is 10000 (10 seconds)
  111. - `disableMedia`: Whether to disable media resources (images, stylesheets, fonts, media), default is true
  112. - `debug`: Whether to enable debug mode (showing browser window), overrides the --debug command line flag if specified
  113. - `fetch_urls` - Batch retrieve web page content from multiple URLs in parallel
  114. - Uses multi-tab parallel fetching for improved performance
  115. - Returns combined results with clear separation between webpages
  116. - Supports the following parameters:
  117. - `urls`: Array of URLs to fetch (required parameter)
  118. - Other parameters are the same as `fetch_url`
  119. - `browser_install` - Install Playwright Chromium browser binary automatically
  120. - Installs required Chromium browser binary when not available
  121. - Automatically suggested when browser installation errors occur
  122. - Supports the following parameters:
  123. - `withDeps`: Install system dependencies required by Chromium browser, default is false
  124. - `force`: Force installation even if Chromium is already installed, default is false
  125. ## Tips
  126. ### Handling Special Website Scenarios
  127. #### Dealing with Anti-Crawler Mechanisms
  128. - **Wait for Complete Loading**: For websites using CAPTCHA, redirects, or other verification mechanisms, include in your prompt:
  129. ```
  130. Please wait for the page to fully load
  131. ```
  132. This will use the `waitForNavigation: true` parameter.
  133. - **Increase Timeout Duration**: For websites that load slowly:
  134. ```
  135. Please set the page loading timeout to 60 seconds
  136. ```
  137. This adjusts both `timeout` and `navigationTimeout` parameters accordingly.
  138. #### Content Retrieval Adjustments
  139. - **Preserve Original HTML Structure**: When content extraction might fail:
  140. ```
  141. Please preserve the original HTML content
  142. ```
  143. Sets `extractContent: false` and `returnHtml: true`.
  144. - **Fetch Complete Page Content**: When extracted content is too limited:
  145. ```
  146. Please fetch the complete webpage content instead of just the main content
  147. ```
  148. Sets `extractContent: false`.
  149. - **Return Content as HTML**: When HTML format is needed instead of default Markdown:
  150. ```
  151. Please return the content in HTML format
  152. ```
  153. Sets `returnHtml: true`.
  154. - **Return Visible Text Only**: When Markdown markup is unnecessary:
  155. ```
  156. Please return only the visible text
  157. ```
  158. Sets `returnText: true`.
  159. ### Request targets
  160. `fetch_url` and `fetch_urls` accept only `http:` and `https:` URLs. Loopback, private, link-local, and cloud-metadata targets are blocked by default, including numeric IP spellings (decimal, hex, octal, and short forms) and names such as `localhost` and `metadata.google.internal`. Set `FETCHER_MCP_ALLOW_PRIVATE_IPS=true` to allow those targets, including blocked hostnames.
  161. A hostname that is not itself on the block list is not looked up before the browser fetches it, and redirects are not checked again. A name that resolves to a private address, or a redirect onto an internal address, can still reach that address.
  162. ### Debugging and Authentication
  163. #### Enabling Debug Mode
  164. - **Dynamic Debug Activation**: To display the browser window during a specific fetch operation:
  165. ```
  166. Please enable debug mode for this fetch operation
  167. ```
  168. This sets `debug: true` even if the server was started without the `--debug` flag.
  169. #### Using Custom Cookies for Authentication
  170. - **Manual Login**: To login using your own credentials:
  171. ```
  172. Please run in debug mode so I can manually log in to the website
  173. ```
  174. Sets `debug: true` or uses the `--debug` flag, keeping the browser window open for manual login.
  175. - **Interacting with Debug Browser**: When debug mode is enabled:
  176. 1. The browser window remains open
  177. 2. You can manually log into the website using your credentials
  178. 3. After login is complete, content will be fetched with your authenticated session
  179. - **Enable Debug for Specific Requests**: Even if the server is already running, you can enable debug mode for a specific request:
  180. ```
  181. Please enable debug mode for this authentication step
  182. ```
  183. Sets `debug: true` for this specific request only, opening the browser window for manual login.
  184. ## Development
  185. ### Install Dependencies
  186. ```bash
  187. npm install
  188. ```
  189. ### Install Playwright Browser
  190. Install the browsers needed for Playwright:
  191. ```bash
  192. npm run install-browser
  193. ```
  194. ### Build the Server
  195. ```bash
  196. npm run build
  197. ```
  198. ## Debugging
  199. Use MCP Inspector for debugging:
  200. ```bash
  201. npm run inspector
  202. ```
  203. You can also enable visible browser mode for debugging:
  204. ```bash
  205. node build/index.js --debug
  206. ```
  207. ## Related Projects
  208. - [g-search-mcp](https://github.com/jae-jae/g-search-mcp): A powerful MCP server for Google search that enables parallel searching with multiple keywords simultaneously. Perfect for batch search operations and data collection.
  209. ## License
  210. Licensed under the [MIT License](https://choosealicense.com/licenses/mit/)
  211. [![Powered by DartNode](https://dartnode.com/branding/DN-Open-Source-sm.png)](https://dartnode.com "Powered by DartNode - Free VPS for Open Source")

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.