Skip to content
Back to search

mcp-crawl4ai-rag

Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/coleam00--mcp-crawl4ai-rag/badge.svg)](/listings/coleam00--mcp-crawl4ai-rag)

HTML

<a href="/listings/coleam00--mcp-crawl4ai-rag"><img src="/api/listings/coleam00--mcp-crawl4ai-rag/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Using Docker (Recommended)

1. Clone this repository:

   git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
   cd mcp-crawl4ai-rag

2. Build the Docker image:

   docker build -t mcp/crawl4ai-rag --build-arg PORT=8051 .

3. Create a `.env` file based on the configuration section below

Using uv directly (no Docker)

1. Clone this repository:

   git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
   cd mcp-crawl4ai-rag

2. Install uv if you don't have it:

   pip install uv

3. Create and activate a virtual environment:

   uv venv
   .venv\Scripts\activate
   # on Mac/Linux: source .venv/bin/activate

4. Install dependencies:

   uv pip install -e .
   crawl4ai-setup

5. Create a `.env` file based on the configuration section below

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 455 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <h1 align="center">Crawl4AI RAG MCP Server</h1>
  2. <p align="center">
  3. <em>Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants</em>
  4. </p>
  5. A powerful implementation of the [Model Context Protocol (MCP)](https://modelcontextprotocol.io) integrated with [Crawl4AI](https://crawl4ai.com) and [Supabase](https://supabase.com/) for providing AI agents and AI coding assistants with advanced web crawling and RAG capabilities.
  6. With this MCP server, you can <b>scrape anything</b> and then <b>use that knowledge anywhere</b> for RAG.
  7. The primary goal is to bring this MCP server into [Archon](https://github.com/coleam00/Archon) as I evolve it to be more of a knowledge engine for AI coding assistants to build AI agents. This first version of the Crawl4AI/RAG MCP server will be improved upon greatly soon, especially making it more configurable so you can use different embedding models and run everything locally with Ollama.
  8. Consider this GitHub repository a testbed, hence why I haven't been super actively address issues and pull requests yet. I certainly will though as I bring this into Archon V2!
  9. ## Overview
  10. This MCP server provides tools that enable AI agents to crawl websites, store content in a vector database (Supabase), and perform RAG over the crawled content. It follows the best practices for building MCP servers based on the [Mem0 MCP server template](https://github.com/coleam00/mcp-mem0/) I provided on my channel previously.
  11. The server includes several advanced RAG strategies that can be enabled to enhance retrieval quality:
  12. - **Contextual Embeddings** for enriched semantic understanding
  13. - **Hybrid Search** combining vector and keyword search
  14. - **Agentic RAG** for specialized code example extraction
  15. - **Reranking** for improved result relevance using cross-encoder models
  16. - **Knowledge Graph** for AI hallucination detection and repository code analysis
  17. See the [Configuration section](#configuration) below for details on how to enable and configure these strategies.
  18. ## Vision
  19. The Crawl4AI RAG MCP server is just the beginning. Here's where we're headed:
  20. 1. **Integration with Archon**: Building this system directly into [Archon](https://github.com/coleam00/Archon) to create a comprehensive knowledge engine for AI coding assistants to build better AI agents.
  21. 2. **Multiple Embedding Models**: Expanding beyond OpenAI to support a variety of embedding models, including the ability to run everything locally with Ollama for complete control and privacy.
  22. 3. **Advanced RAG Strategies**: Implementing sophisticated retrieval techniques like contextual retrieval, late chunking, and others to move beyond basic "naive lookups" and significantly enhance the power and precision of the RAG system, especially as it integrates with Archon.
  23. 4. **Enhanced Chunking Strategy**: Implementing a Context 7-inspired chunking approach that focuses on examples and creates distinct, semantically meaningful sections for each chunk, improving retrieval precision.
  24. 5. **Performance Optimization**: Increasing crawling and indexing speed to make it more realistic to "quickly" index new documentation to then leverage it within the same prompt in an AI coding assistant.
  25. ## Features
  26. - **Smart URL Detection**: Automatically detects and handles different URL types (regular webpages, sitemaps, text files)
  27. - **Recursive Crawling**: Follows internal links to discover content
  28. - **Parallel Processing**: Efficiently crawls multiple pages simultaneously
  29. - **Content Chunking**: Intelligently splits content by headers and size for better processing
  30. - **Vector Search**: Performs RAG over crawled content, optionally filtering by data source for precision
  31. - **Source Retrieval**: Retrieve sources available for filtering to guide the RAG process
  32. ## Tools
  33. The server provides essential web crawling and search tools:
  34. ### Core Tools (Always Available)
  35. 1. **`crawl_single_page`**: Quickly crawl a single web page and store its content in the vector database
  36. 2. **`smart_crawl_url`**: Intelligently crawl a full website based on the type of URL provided (sitemap, llms-full.txt, or a regular webpage that needs to be crawled recursively)
  37. 3. **`get_available_sources`**: Get a list of all available sources (domains) in the database
  38. 4. **`perform_rag_query`**: Search for relevant content using semantic search with optional source filtering
  39. ### Conditional Tools
  40. 5. **`search_code_examples`** (requires `USE_AGENTIC_RAG=true`): Search specifically for code examples and their summaries from crawled documentation. This tool provides targeted code snippet retrieval for AI coding assistants.
  41. ### Knowledge Graph Tools (requires `USE_KNOWLEDGE_GRAPH=true`, see below)
  42. 6. **`parse_github_repository`**: Parse a GitHub repository into a Neo4j knowledge graph, extracting classes, methods, functions, and their relationships for hallucination detection
  43. 7. **`check_ai_script_hallucinations`**: Analyze Python scripts for AI hallucinations by validating imports, method calls, and class usage against the knowledge graph
  44. 8. **`query_knowledge_graph`**: Explore and query the Neo4j knowledge graph with commands like `repos`, `classes`, `methods`, and custom Cypher queries
  45. ## Prerequisites
  46. - [Docker/Docker Desktop](https://www.docker.com/products/docker-desktop/) if running the MCP server as a container (recommended)
  47. - [Python 3.12+](https://www.python.org/downloads/) if running the MCP server directly through uv
  48. - [Supabase](https://supabase.com/) (database for RAG)
  49. - [OpenAI API key](https://platform.openai.com/api-keys) (for generating embeddings)
  50. - [Neo4j](https://neo4j.com/) (optional, for knowledge graph functionality) - see [Knowledge Graph Setup](#knowledge-graph-setup) section
  51. ## Installation
  52. ### Using Docker (Recommended)
  53. 1. Clone this repository:
  54. ```bash
  55. git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
  56. cd mcp-crawl4ai-rag
  57. ```
  58. 2. Build the Docker image:
  59. ```bash
  60. docker build -t mcp/crawl4ai-rag --build-arg PORT=8051 .
  61. ```
  62. 3. Create a `.env` file based on the configuration section below
  63. ### Using uv directly (no Docker)
  64. 1. Clone this repository:
  65. ```bash
  66. git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
  67. cd mcp-crawl4ai-rag
  68. ```
  69. 2. Install uv if you don't have it:
  70. ```bash
  71. pip install uv
  72. ```
  73. 3. Create and activate a virtual environment:
  74. ```bash
  75. uv venv
  76. .venv\Scripts\activate
  77. # on Mac/Linux: source .venv/bin/activate
  78. ```
  79. 4. Install dependencies:
  80. ```bash
  81. uv pip install -e .
  82. crawl4ai-setup
  83. ```
  84. 5. Create a `.env` file based on the configuration section below
  85. ## Database Setup
  86. Before running the server, you need to set up the database with the pgvector extension:
  87. 1. Go to the SQL Editor in your Supabase dashboard (create a new project first if necessary)
  88. 2. Create a new query and paste the contents of `crawled_pages.sql`
  89. 3. Run the query to create the necessary tables and functions
  90. ## Knowledge Graph Setup (Optional)
  91. To enable AI hallucination detection and repository analysis features, you need to set up Neo4j.
  92. Also, the knowledge graph implementation isn't fully compatible with Docker yet, so I would recommend right now running directly through uv if you want to use the hallucination detection within the MCP server!
  93. For installing Neo4j:
  94. ### Local AI Package (Recommended)
  95. The easiest way to get Neo4j running locally is with the [Local AI Package](https://github.com/coleam00/local-ai-packaged) - a curated collection of local AI services including Neo4j:
  96. 1. **Clone the Local AI Package**:
  97. ```bash
  98. git clone https://github.com/coleam00/local-ai-packaged.git
  99. cd local-ai-packaged
  100. ```
  101. 2. **Start Neo4j**:
  102. Follow the instructions in the Local AI Package repository to start Neo4j with Docker Compose
  103. 3. **Default connection details**:
  104. - URI: `bolt://localhost:7687`
  105. - Username: `neo4j`
  106. - Password: Check the Local AI Package documentation for the default password
  107. ### Manual Neo4j Installation
  108. Alternatively, install Neo4j directly:
  109. 1. **Install Neo4j Desktop**: Download from [neo4j.com/download](https://neo4j.com/download/)
  110. 2. **Create a new database**:
  111. - Open Neo4j Desktop
  112. - Create a new project and database
  113. - Set a password for the `neo4j` user
  114. - Start the database
  115. 3. **Note your connection details**:
  116. - URI: `bolt://localhost:7687` (default)
  117. - Username: `neo4j` (default)
  118. - Password: Whatever you set during creation
  119. ## Configuration
  120. Create a `.env` file in the project root with the following variables:
  121. ```
  122. # MCP Server Configuration
  123. HOST=0.0.0.0
  124. PORT=8051
  125. TRANSPORT=sse
  126. # OpenAI API Configuration
  127. OPENAI_API_KEY=your_openai_api_key
  128. # LLM for summaries and contextual embeddings
  129. MODEL_CHOICE=gpt-4.1-nano
  130. # RAG Strategies (set to "true" or "false", default to "false")
  131. USE_CONTEXTUAL_EMBEDDINGS=false
  132. USE_HYBRID_SEARCH=false
  133. USE_AGENTIC_RAG=false
  134. USE_RERANKING=false
  135. USE_KNOWLEDGE_GRAPH=false
  136. # Supabase Configuration
  137. SUPABASE_URL=your_supabase_project_url
  138. SUPABASE_SERVICE_KEY=your_supabase_service_key
  139. # Neo4j Configuration (required for knowledge graph functionality)
  140. NEO4J_URI=bolt://localhost:7687
  141. NEO4J_USER=neo4j
  142. NEO4J_PASSWORD=your_neo4j_password
  143. ```
  144. ### RAG Strategy Options
  145. The Crawl4AI RAG MCP server supports four powerful RAG strategies that can be enabled independently:
  146. #### 1. **USE_CONTEXTUAL_EMBEDDINGS**
  147. When enabled, this strategy enhances each chunk's embedding with additional context from the entire document. The system passes both the full document and the specific chunk to an LLM (configured via `MODEL_CHOICE`) to generate enriched context that gets embedded alongside the chunk content.
  148. - **When to use**: Enable this when you need high-precision retrieval where context matters, such as technical documentation where terms might have different meanings in different sections.
  149. - **Trade-offs**: Slower indexing due to LLM calls for each chunk, but significantly better retrieval accuracy.
  150. - **Cost**: Additional LLM API calls during indexing.
  151. #### 2. **USE_HYBRID_SEARCH**
  152. Combines traditional keyword search with semantic vector search to provide more comprehensive results. The system performs both searches in parallel and intelligently merges results, prioritizing documents that appear in both result sets.
  153. - **When to use**: Enable this when users might search using specific technical terms, function names, or when exact keyword matches are important alongside semantic understanding.
  154. - **Trade-offs**: Slightly slower search queries but more robust results, especially for technical content.
  155. - **Cost**: No additional API costs, just computational overhead.
  156. #### 3. **USE_AGENTIC_RAG**
  157. Enables specialized code example extraction and storage. When crawling documentation, the system identifies code blocks (≥300 characters), extracts them with surrounding context, generates summaries, and stores them in a separate vector database table specifically designed for code search.
  158. - **When to use**: Essential for AI coding assistants that need to find specific code examples, implementation patterns, or usage examples from documentation.
  159. - **Trade-offs**: Significantly slower crawling due to code extraction and summarization, requires more storage space.
  160. - **Cost**: Additional LLM API calls for summarizing each code example.
  161. - **Benefits**: Provides a dedicated `search_code_examples` tool that AI agents can use to find specific code implementations.
  162. #### 4. **USE_RERANKING**
  163. Applies cross-encoder reranking to search results after initial retrieval. Uses a lightweight cross-encoder model (`cross-encoder/ms-marco-MiniLM-L-6-v2`) to score each result against the original query, then reorders results by relevance.
  164. - **When to use**: Enable this when search precision is critical and you need the most relevant results at the top. Particularly useful for complex queries where semantic similarity alone might not capture query intent.
  165. - **Trade-offs**: Adds ~100-200ms to search queries depending on result count, but significantly improves result ordering.
  166. - **Cost**: No additional API costs - uses a local model that runs on CPU.
  167. - **Benefits**: Better result relevance, especially for complex queries. Works with both regular RAG search and code example search.
  168. #### 5. **USE_KNOWLEDGE_GRAPH**
  169. Enables AI hallucination detection and repository analysis using Neo4j knowledge graphs. When enabled, the system can parse GitHub repositories into a graph database and validate AI-generated code against real repository structures. (NOT fully compatible with Docker yet, I'd recommend running through uv)
  170. - **When to use**: Enable this for AI coding assistants that need to validate generated code against real implementations, or when you want to detect when AI models hallucinate non-existent methods, classes, or incorrect usage patterns.
  171. - **Trade-offs**: Requires Neo4j setup and additional dependencies. Repository parsing can be slow for large codebases, and validation requires repositories to be pre-indexed.
  172. - **Cost**: No additional API costs for validation, but requires Neo4j infrastructure (can use free local installation or cloud AuraDB).
  173. - **Benefits**: Provides three powerful tools: `parse_github_repository` for indexing codebases, `check_ai_script_hallucinations` for validating AI-generated code, and `query_knowledge_graph` for exploring indexed repositories.
  174. You can now tell the AI coding assistant to add a Python GitHub repository to the knowledge graph like:
  175. "Add https://github.com/pydantic/pydantic-ai.git to the knowledge graph"
  176. Make sure the repo URL ends with .git.
  177. You can also have the AI coding assistant check for hallucinations with scripts it just created, or you can manually run the command:
  178. ```
  179. python knowledge_graphs/ai_hallucination_detector.py [full path to your script to analyze]
  180. ```
  181. ### Recommended Configurations
  182. **For general documentation RAG:**
  183. ```
  184. USE_CONTEXTUAL_EMBEDDINGS=false
  185. USE_HYBRID_SEARCH=true
  186. USE_AGENTIC_RAG=false
  187. USE_RERANKING=true
  188. ```
  189. **For AI coding assistant with code examples:**
  190. ```
  191. USE_CONTEXTUAL_EMBEDDINGS=true
  192. USE_HYBRID_SEARCH=true
  193. USE_AGENTIC_RAG=true
  194. USE_RERANKING=true
  195. USE_KNOWLEDGE_GRAPH=false
  196. ```
  197. **For AI coding assistant with hallucination detection:**
  198. ```
  199. USE_CONTEXTUAL_EMBEDDINGS=true
  200. USE_HYBRID_SEARCH=true
  201. USE_AGENTIC_RAG=true
  202. USE_RERANKING=true
  203. USE_KNOWLEDGE_GRAPH=true
  204. ```
  205. **For fast, basic RAG:**
  206. ```
  207. USE_CONTEXTUAL_EMBEDDINGS=false
  208. USE_HYBRID_SEARCH=true
  209. USE_AGENTIC_RAG=false
  210. USE_RERANKING=false
  211. USE_KNOWLEDGE_GRAPH=false
  212. ```
  213. ## Running the Server
  214. ### Using Docker
  215. ```bash
  216. docker run --env-file .env -p 8051:8051 mcp/crawl4ai-rag
  217. ```
  218. ### Using Python
  219. ```bash
  220. uv run src/crawl4ai_mcp.py
  221. ```
  222. The server will start and listen on the configured host and port.
  223. ## Integration with MCP Clients
  224. ### SSE Configuration
  225. Once you have the server running with SSE transport, you can connect to it using this configuration:
  226. ```json
  227. {
  228. "mcpServers": {
  229. "crawl4ai-rag": {
  230. "transport": "sse",
  231. "url": "http://localhost:8051/sse"
  232. }
  233. }
  234. }
  235. ```
  236. > **Note for Windsurf users**: Use `serverUrl` instead of `url` in your configuration:
  237. > ```json
  238. > {
  239. > "mcpServers": {
  240. > "crawl4ai-rag": {
  241. > "transport": "sse",
  242. > "serverUrl": "http://localhost:8051/sse"
  243. > }
  244. > }
  245. > }
  246. > ```
  247. >
  248. > **Note for Docker users**: Use `host.docker.internal` instead of `localhost` if your client is running in a different container. This will apply if you are using this MCP server within n8n!
  249. > **Note for Claude Code users**:
  250. ```
  251. claude mcp add-json crawl4ai-rag '{"type":"http","url":"http://localhost:8051/sse"}' --scope user
  252. ```
  253. ### Stdio Configuration
  254. Add this server to your MCP configuration for Claude Desktop, Windsurf, or any other MCP client:
  255. ```json
  256. {
  257. "mcpServers": {
  258. "crawl4ai-rag": {
  259. "command": "python",
  260. "args": ["path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"],
  261. "env": {
  262. "TRANSPORT": "stdio",
  263. "OPENAI_API_KEY": "your_openai_api_key",
  264. "SUPABASE_URL": "your_supabase_url",
  265. "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
  266. "USE_KNOWLEDGE_GRAPH": "false",
  267. "NEO4J_URI": "bolt://localhost:7687",
  268. "NEO4J_USER": "neo4j",
  269. "NEO4J_PASSWORD": "your_neo4j_password"
  270. }
  271. }
  272. }
  273. }
  274. ```
  275. ### Docker with Stdio Configuration
  276. ```json
  277. {
  278. "mcpServers": {
  279. "crawl4ai-rag": {
  280. "command": "docker",
  281. "args": ["run", "--rm", "-i",
  282. "-e", "TRANSPORT",
  283. "-e", "OPENAI_API_KEY",
  284. "-e", "SUPABASE_URL",
  285. "-e", "SUPABASE_SERVICE_KEY",
  286. "-e", "USE_KNOWLEDGE_GRAPH",
  287. "-e", "NEO4J_URI",
  288. "-e", "NEO4J_USER",
  289. "-e", "NEO4J_PASSWORD",
  290. "mcp/crawl4ai"],
  291. "env": {
  292. "TRANSPORT": "stdio",
  293. "OPENAI_API_KEY": "your_openai_api_key",
  294. "SUPABASE_URL": "your_supabase_url",
  295. "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
  296. "USE_KNOWLEDGE_GRAPH": "false",
  297. "NEO4J_URI": "bolt://localhost:7687",
  298. "NEO4J_USER": "neo4j",
  299. "NEO4J_PASSWORD": "your_neo4j_password"
  300. }
  301. }
  302. }
  303. }
  304. ```
  305. ## Knowledge Graph Architecture
  306. The knowledge graph system stores repository code structure in Neo4j with the following components:
  307. ### Core Components (`knowledge_graphs/` folder):
  308. - **`parse_repo_into_neo4j.py`**: Clones and analyzes GitHub repositories, extracting Python classes, methods, functions, and imports into Neo4j nodes and relationships
  309. - **`ai_script_analyzer.py`**: Parses Python scripts using AST to extract imports, class instantiations, method calls, and function usage
  310. - **`knowledge_graph_validator.py`**: Validates AI-generated code against the knowledge graph to detect hallucinations (non-existent methods, incorrect parameters, etc.)
  311. - **`hallucination_reporter.py`**: Generates comprehensive reports about detected hallucinations with confidence scores and recommendations
  312. - **`query_knowledge_graph.py`**: Interactive CLI tool for exploring the knowledge graph (functionality now integrated into MCP tools)
  313. ### Knowledge Graph Schema:
  314. The Neo4j database stores code structure as:
  315. **Nodes:**
  316. - `Repository`: GitHub repositories
  317. - `File`: Python files within repositories
  318. - `Class`: Python classes with methods and attributes
  319. - `Method`: Class methods with parameter information
  320. - `Function`: Standalone functions
  321. - `Attribute`: Class attributes
  322. **Relationships:**
  323. - `Repository` -[:CONTAINS]-> `File`
  324. - `File` -[:DEFINES]-> `Class`
  325. - `File` -[:DEFINES]-> `Function`
  326. - `Class` -[:HAS_METHOD]-> `Method`
  327. - `Class` -[:HAS_ATTRIBUTE]-> `Attribute`
  328. ### Workflow:
  329. 1. **Repository Parsing**: Use `parse_github_repository` tool to clone and analyze open-source repositories
  330. 2. **Code Validation**: Use `check_ai_script_hallucinations` tool to validate AI-generated Python scripts
  331. 3. **Knowledge Exploration**: Use `query_knowledge_graph` tool to explore available repositories, classes, and methods
  332. ## Building Your Own Server
  333. This implementation provides a foundation for building more complex MCP servers with web crawling capabilities. To build your own:
  334. 1. Add your own tools by creating methods with the `@mcp.tool()` decorator
  335. 2. Create your own lifespan function to add your own dependencies
  336. 3. Modify the `utils.py` file for any helper functions you need
  337. 4. Extend the crawling capabilities by adding more specialized crawlers

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.