Skip to content
Back to search

searxNcrawl

MCP Server and CLI Tools for searxing and fetching websites

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/DasDigitaleMomentum/searxNcrawl with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/DasDigitaleMomentum/searxNcrawl'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/dasdigitalemomentum--searxncrawl/badge.svg)](/listings/dasdigitalemomentum--searxncrawl)

HTML

<a href="/listings/dasdigitalemomentum--searxncrawl"><img src="/api/listings/dasdigitalemomentum--searxncrawl/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Pick your setup:

Docker Compose

MCP server with Playwright/Chromium, ready in one command. SearXNG required separately for search.

cp .env.example .env          # set SEARXNG_URL to your SearXNG instance
docker compose up --build

➜ MCP server at `http://localhost:9555/mcp`

pip (standalone)

CLI tools, Python API, and MCP server. SearXNG required for search.

python -m venv .venv && source .venv/bin/activate
pip install -e .
playwright install chromium

uv (standalone)

Same capabilities as pip.

uv sync
uv run playwright install chromium

What you get

| Feature | Docker Compose | pip / uv | | ----------------------- | -------------- | --------- | | MCP Server (STDIO) | — | ✅ | | MCP Server (HTTP) | ✅ | ✅ | | Web Crawl | ✅ | ✅ | | Web Search | ✅¹ | ✅¹ | | CLI Tools | via `exec`² | ✅ | | Python API | — | ✅ | | CORS (HTTP) | ✅ | ✅ |

¹ Requires a SearXNG instance. ² `docker compose exec searxncrawl crawl ...`

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 269 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <p align="center">
  2. <a href="https://www.searxncrawl.io/">
  3. <picture>
  4. <source media="(prefers-color-scheme: dark)" srcset="docs/assets/banner-dark.png">
  5. <img src="docs/assets/banner-light.png" alt="searxNcrawl: search and crawl the web for your AI agents. Open-source MCP server for web search and clean Markdown crawling." width="100%">
  6. </picture>
  7. </a>
  8. </p>
  9. <p align="center">
  10. <a href="https://pypi.org/project/searxncrawl/"><img alt="PyPI" src="https://img.shields.io/pypi/v/searxncrawl?style=flat-square&label=PyPI&labelColor=16161a&color=3046ff"></a>
  11. <a href="https://pypi.org/project/searxncrawl/"><img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-3046ff?style=flat-square&labelColor=16161a"></a>
  12. <a href="https://github.com/DasDigitaleMomentum/searxNcrawl/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/DasDigitaleMomentum/searxNcrawl/ci.yml?branch=main&style=flat-square&label=CI&labelColor=16161a"></a>
  13. <a href="LICENSE"><img alt="MIT licence" src="https://img.shields.io/badge/licence-MIT-3046ff?style=flat-square&labelColor=16161a"></a>
  14. <a href="https://modelcontextprotocol.io/"><img alt="MCP server" src="https://img.shields.io/badge/MCP-server-3046ff?style=flat-square&labelColor=16161a"></a>
  15. </p>
  16. <p align="center">
  17. <a href="https://www.searxncrawl.io/"><b>Website</b></a> ·
  18. <a href="#quick-start">Quick start</a> ·
  19. <a href="docs/overview.md">Docs</a> ·
  20. <a href="https://www.searxncrawl.io/developers/">Developers</a> ·
  21. <a href="CHANGELOG.md">Changelog</a>
  22. </p>
  23. <p align="center">
  24. <img src="docs/assets/demo.gif" alt="An MCP client starts uvx searxncrawl, calls search for crawl4ai quick start, then crawls the top result and receives clean Markdown" width="100%">
  25. <br>
  26. <sub>A real MCP client session (fastmcp) against <code>uvx searxncrawl</code> 0.31.0 and a local SearXNG: <code>search</code>, then <code>crawl</code> the top hit into Markdown. Output is shortened for display. Reproduce it with <a href="docs/assets/demo/demo.tape"><code>docs/assets/demo/</code></a>.</sub>
  27. </p>
  28. **searxNcrawl** is an open-source MCP server and CLI toolkit that gives AI agents web search and clean Markdown crawling, built on [Crawl4AI](https://github.com/unclecode/crawl4ai). Search queries the configured providers, [SearXNG](https://github.com/searxng/searxng) and/or [You.com](https://you.com), in parallel and merges the results. It runs on your own hardware, needs no API key with SearXNG and succeeds `searxng-mcp`.
  29. <table>
  30. <tr>
  31. <td width="33%" valign="top"><b>Web search</b><br>Your own SearXNG instance and/or the You.com API, queried in parallel; results are merged and attributed to their providers, filtered by language and time range.</td>
  32. <td width="33%" valign="top"><b>Page crawl</b><br>One or many URLs to Markdown or JSON, with repeated blocks removed and links optionally stripped.</td>
  33. <td width="33%" valign="top"><b>Site crawl</b><br>Whole sites with depth and page limits, tuned for documentation.</td>
  34. </tr>
  35. <tr>
  36. <td valign="top"><b>Authenticated crawls</b><br>Capture a login session once with <code>crawl-capture</code>, then crawl behind the login.</td>
  37. <td valign="top"><b>STDIO and HTTP</b><br>Plug into local MCP clients, or run an HTTP endpoint with opt-in Host/Origin checks and configurable CORS.</td>
  38. <td valign="top"><b>CLI and Python API</b><br><code>crawl</code>, <code>search</code> and <code>crawl-capture</code> on the command line, or import it in Python.</td>
  39. </tr>
  40. </table>
  41. ```bash
  42. SEARXNG_URL=http://your-searxng:8888 uvx searxncrawl
  43. ```
  44. <!-- mcp-name: io.github.DasDigitaleMomentum/searxncrawl -->
  45. ## Quick start
  46. Connect your MCP client directly to searxNcrawl over STDIO or its HTTP endpoint.
  47. You need [uv](https://docs.astral.sh/uv/) and, for the `search` tool, at least one search provider: a SearXNG instance with JSON output enabled and/or a You.com API key (`YDC_API_KEY`). No SearXNG yet? [Start one with Docker](docs/usage/installation.md#run-searxng-locally). Crawling works without either.
  48. ### Add it to your agent
  49. **One step for every agent on your machine**
  50. ```bash
  51. uvx searxncrawl setup
  52. ```
  53. `setup` detects Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and, optionally, a You.com API key (hidden input, never printed), shows each config change and applies it only after you confirm. Every file it touches is backed up first, and `setup --remove` undoes it. More options are in [MCP server](docs/usage/mcp-server.md#register-with-searxncrawl-setup).
  54. **Or per agent**
  55. <p>
  56. <a href="https://cursor.com/en/install-mcp?name=searxncrawl&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJzZWFyeG5jcmF3bCJdLCJlbnYiOnsiU0VBUlhOR19VUkwiOiJodHRwOi8vbG9jYWxob3N0Ojg4ODgifX0%3D"><img src="https://cursor.com/deeplink/mcp-install-dark.svg" alt="Add searxNcrawl to Cursor" height="32"></a>
  57. <a href="https://insiders.vscode.dev/redirect/mcp/install?name=searxncrawl&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22searxncrawl%22%5D%2C%22env%22%3A%7B%22SEARXNG_URL%22%3A%22http%3A%2F%2Flocalhost%3A8888%22%7D%7D"><img src="https://img.shields.io/badge/VS_Code-Install_searxNcrawl-0098FF?style=for-the-badge&logo=visualstudiocode&logoColor=white" alt="Install searxNcrawl in VS Code" height="32"></a>
  58. </p>
  59. The buttons and commands below assume SearXNG at `http://localhost:8888`; change `SEARXNG_URL` if yours runs elsewhere, or add `YDC_API_KEY` for You.com.
  60. **Claude Code**
  61. ```bash
  62. claude mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl
  63. ```
  64. **Codex**
  65. ```bash
  66. codex mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl
  67. ```
  68. **VS Code**
  69. ```bash
  70. code --add-mcp '{"name":"searxncrawl","command":"uvx","args":["searxncrawl"],"env":{"SEARXNG_URL":"http://localhost:8888"}}'
  71. ```
  72. **Cursor (`~/.cursor/mcp.json`), Claude Desktop and other clients that use the `mcpServers` format**
  73. ```json
  74. {
  75. "mcpServers": {
  76. "searxncrawl": {
  77. "command": "uvx",
  78. "args": ["searxncrawl"],
  79. "env": { "SEARXNG_URL": "http://localhost:8888" }
  80. }
  81. }
  82. }
  83. ```
  84. The first crawl downloads Playwright's Chromium once (about 550 MB). For Docker, HTTP transport and other setups see [Installation](docs/usage/installation.md) and [MCP server](docs/usage/mcp-server.md).
  85. - **Claude Desktop bundle:** download `searxncrawl-<version>.mcpb` from [GitHub Releases](https://github.com/DasDigitaleMomentum/searxNcrawl/releases/latest) and open it in Claude Desktop.
  86. - **Smithery users:** use the direct installation above or the native bundle; see the [Smithery transition](docs/registry.md#smithery-transition).
  87. ### Ask your agent
  88. ```text
  89. Search for the Crawl4AI quick start, crawl the official page and summarise
  90. how to run a first crawl.
  91. ```
  92. Your agent calls `search`, picks a result and calls `crawl`. It gets back clean Markdown instead of raw HTML.
  93. ### Use the terminal
  94. ```bash
  95. uvx --from searxncrawl crawl https://modelcontextprotocol.io/docs/getting-started/intro
  96. SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "model context protocol" --max-results 5
  97. SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "Python documentation" --max-results 3 --include-content
  98. ```
  99. ### Choose how to connect
  100. | Mode | Setup | Hosting |
  101. | --- | --- | --- |
  102. | Native MCP over STDIO (default) | `uvx searxncrawl` or `uvx searxncrawl setup` | Runs alongside your MCP client |
  103. | Native MCP over HTTP | `uvx searxncrawl --transport http` or Docker Compose | You operate the server; see [HTTP configuration](docs/usage/mcp-server.md#start-the-server) |
  104. | Arcade gateway (optional) | Separate [Arcade adapter](deploy/arcade/README.md) | Requires your own Playwright-compatible worker host and authenticated connection to Arcade |
  105. - Standard installations and releases do not require Arcade.
  106. - For shared HTTP access, use an authenticated TLS proxy. Enable [Host/Origin checks](docs/usage/mcp-server.md#http-host-and-cors-guards) with `FASTMCP_HTTP_HOST_ORIGIN_PROTECTION=true` before startup; allowlists alone do not enable checks or authenticate callers.
  107. ## How it works
  108. ```mermaid
  109. flowchart LR
  110. agent["AI agent<br/>Claude Code · Codex<br/>Cursor · VS Code"]
  111. server["searxNcrawl<br/>MCP server"]
  112. searxng["SearXNG (your instance)<br/>and/or You.com API"]
  113. browser["Crawl4AI +<br/>Playwright Chromium"]
  114. results["Merged results<br/>attributed per provider"]
  115. markdown["Deduplicated<br/>Markdown or JSON"]
  116. agent -- "MCP · STDIO or HTTP" --> server
  117. server -- "search" --> searxng --> results
  118. server -- "crawl · crawl_site" --> browser --> markdown
  119. ```
  120. Results return to the agent as the tool response. Search and crawling are also available as the `search` and `crawl` CLI commands, crawling also as a Python API.
  121. searxNcrawl exposes three MCP tools:
  122. | Tool | What it does |
  123. | --- | --- |
  124. | `search` | Queries the configured providers (SearXNG, You.com) in parallel and returns one merged, deduplicated list with the source of each result. Filters: language, time range, safe search, paging; categories and engines for SearXNG. With `include_content`, also returns the Markdown of each result page in the same call. |
  125. | `crawl` | Renders one or many URLs in Chromium and returns Markdown or JSON with metadata and references. |
  126. | `crawl_site` | Follows links from a seed URL within depth, page and timeout limits, tuned for documentation sites. |
  127. Both crawl tools drop sections that repeat within a page's Markdown (`dedup_mode="exact"`), can strip links for a smaller context (`remove_links`), and accept a Playwright `storage_state` for pages behind a login. Full parameters are in the [MCP tools reference](docs/usage/mcp-tools.md).
  128. ### Private SearXNG instances
  129. - `SEARXNG_EXTRA_HEADERS` adds headers to SearXNG requests in both CLI and MCP, including access-gateway credentials and a custom User-Agent.
  130. - Values such as `@~/.config/searxncrawl/cf_secret` load secrets from files; absolute and `~/` paths are supported.
  131. - Invalid header configuration disables SearXNG with a value-free warning; a configured You.com provider remains available.
  132. - Setup and JSON examples: [SearXNG configuration](docs/usage/installation.md).
  133. ## Why searxNcrawl
  134. Hosted search and crawl APIs are quick to start with, but every call goes through someone else's service and is billed per request or credit. searxNcrawl runs on your own machine or server.
  135. | | searxNcrawl | [Firecrawl](https://github.com/firecrawl/firecrawl) | [Tavily](https://github.com/tavily-ai/tavily-mcp) | [Exa](https://github.com/exa-labs/exa-mcp-server) |
  136. | --- | --- | --- | --- | --- |
  137. | Search and crawl run on your hardware | ✅ Python package or Docker; You.com queries go to You.com | ✅ Self-host stack (Docker Compose), or hosted API | ❌ Hosted API | ❌ Hosted API |
  138. | API key | Not needed with SearXNG; You.com needs one | Hosted API: needed. Self-hosted: not by default | Needed | Optional, anonymous use is rate-limited |
  139. | Cost per call | None with SearXNG; You.com per its plans | Hosted: free monthly credits, then paid credits. Self-hosted: none | Free monthly credits, then paid credits | Free monthly credits, then paid per request |
  140. | Search backend | Your SearXNG (the engines you enable) and/or You.com | Hosted: Firecrawl's search. Self-hosted: SearXNG if configured, otherwise DuckDuckGo | Tavily's search API | Exa's search index |
  141. | Licence | MIT | AGPL-3.0 | MCP server MIT, API proprietary | MCP server MIT, API proprietary |
  142. Self-hosted Firecrawl can search through SearXNG too; it runs as a multi-service stack (API, Playwright service, Redis, RabbitMQ, PostgreSQL and more), while searxNcrawl is a single Python process with Chromium, plus your SearXNG instance if you use one.
  143. Where the others are a better fit: if you do not want to run a search provider and a browser yourself, a hosted API is less work, and Exa's semantic search index or Firecrawl's extraction features go beyond what searxNcrawl does today. Comparison based on each project's public repository and documentation in October 2026; corrections are welcome. Firecrawl is a trademark of its owner; searxNcrawl is not affiliated with it.
  144. ## FAQ
  145. <details>
  146. <summary><b>Do I need SearXNG?</b></summary>
  147. Only for the `search` tool, and only if you do not use You.com (`YDC_API_KEY`). `crawl` and `crawl_site` work without any search provider. SearXNG serves HTML only by default, so JSON output must be enabled (`search.formats` in `settings.yml`). Public instances may not enable it and often rate-limit automated requests, so run your own. [Installation](docs/usage/installation.md#run-searxng-locally) has a Docker one-liner.
  148. </details>
  149. <details>
  150. <summary><b>Does it handle JavaScript-heavy pages?</b></summary>
  151. Yes. Pages are rendered in Playwright's Chromium through Crawl4AI before they are converted to Markdown, so client-side content is captured.
  152. </details>
  153. <details>
  154. <summary><b>Can it crawl pages behind a login?</b></summary>
  155. Yes. Capture a session once with `crawl-capture`, then pass the saved `storage_state` file to `crawl` or `crawl_site`. See [Session capture](docs/usage/session-capture.md).
  156. </details>
  157. <details>
  158. <summary><b>Can I run it remotely or share it with a team?</b></summary>
  159. - Start with `--transport http` (port 8000 by default) or Docker Compose (port 9555).
  160. - Put shared access behind an authenticated TLS proxy and connect clients to its `/mcp` endpoint.
  161. - Enable header protection and configure the externally visible [Host and Origin allowlists](docs/usage/mcp-server.md#http-host-and-cors-guards).
  162. </details>
  163. <details>
  164. <summary><b>Why is the first crawl slow?</b></summary>
  165. The first crawl downloads Playwright's Chromium (about 550 MB on disk). Later runs reuse it. Set `PLAYWRIGHT_AUTO_INSTALL=false` where browsers are provisioned separately.
  166. </details>
  167. <details>
  168. <summary><b>I used <code>searxng-mcp</code>. What changed?</b></summary>
  169. [`searxng-mcp`](https://github.com/tisDDM/searxng-mcp) is deprecated in favour of searxNcrawl. It only searched SearXNG and could pick a random public instance; searxNcrawl adds page and site crawling but needs your own `SEARXNG_URL` (or a You.com key), as random public-instance selection is not included.
  170. </details>
  171. <details>
  172. <summary><b>Do I need Arcade?</b></summary>
  173. - No. Connect directly over STDIO or HTTP for the standard setup.
  174. - The [optional adapter](deploy/arcade/README.md) connects a separately hosted worker to Arcade's authentication and gateway.
  175. - Arcade Cloud's current runner cannot install Playwright; the worker needs a compatible host.
  176. </details>
  177. ## Documentation
  178. | Guide | Contents |
  179. | --- | --- |
  180. | [Installation](docs/usage/installation.md) | uvx, Docker Compose, pip, uv, search providers and environment variables |
  181. | [MCP server](docs/usage/mcp-server.md) | STDIO and HTTP transports, client configuration, Host and CORS guards |
  182. | [MCP tools](docs/usage/mcp-tools.md) | Parameters for `crawl`, `crawl_site` and `search` |
  183. | [CLI](docs/usage/cli.md) | `crawl`, `search` and `crawl-capture` |
  184. | [Python API](docs/usage/python-api.md) | `crawl_page`, `crawl_site` and crawler configuration overrides |
  185. | [Output formats](docs/usage/output-formats.md) | Markdown and JSON structure, `CrawledDocument` |
  186. | [Session capture](docs/usage/session-capture.md) | Authenticated crawling with a saved browser session |
  187. | [MCP Registry](docs/registry.md) | `server.json` and the publish steps |
  188. | [Arcade adapter (optional)](deploy/arcade/README.md) | Shared authentication and routing through Arcade; separate deployment |
  189. | [Architecture](docs/overview.md) | Modules and features for contributors |
  190. ## Contributing
  191. Bug reports, feature ideas and pull requests are welcome. Read the [contributing guide](https://github.com/DasDigitaleMomentum/.github/blob/main/.github/CONTRIBUTING.md) and open an [issue](https://github.com/DasDigitaleMomentum/searxNcrawl/issues/new/choose) first for larger changes. Coding agents find project conventions in [AGENTS.md](AGENTS.md). Report vulnerabilities privately as described in the [security policy](https://github.com/DasDigitaleMomentum/.github/blob/main/.github/SECURITY.md).
  192. ## License
  193. MIT — © 2026 DDM – Das Digitale Momentum GmbH & Co KG
  194. ---
  195. <p align="center">
  196. <sub>Maintained by <a href="https://www.das-digitale-momentum.de/en/open-source/#searxncrawl"><b>DDM – Das Digitale Momentum GmbH &amp; Co. KG</b></a>, Much, Germany · <a href="https://www.searxncrawl.io/">www.searxncrawl.io</a> · <a href="https://github.com/DasDigitaleMomentum">All our open-source projects</a></sub>
  197. </p>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.