Skip to content
Back to search

scrapegraph-mcp

ScapeGraph MCP Server

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/scrapegraphai--scrapegraph-mcp/badge.svg)](/listings/scrapegraphai--scrapegraph-mcp)

HTML

<a href="/listings/scrapegraphai--scrapegraph-mcp"><img src="/api/listings/scrapegraphai--scrapegraph-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Get Your API Key

Sign up and get your API key from the [ScrapeGraph Dashboard](https://dashboard.scrapegraphai.com)

2. Install with Smithery (Recommended)

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

3. Start Using

Ask Claude or Cursor: - "Convert https://scrapegraphai.com to markdown" - "Extract all product prices from this e-commerce page" - "Research the latest AI developments and summarize findings"

That's it! The server is now available to your AI assistant.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 732 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # ScrapeGraph MCP Server
  2. <p align="center">
  3. <img src="./assets/scrapegraphAI.svg" width="250" alt="ScrapegraphAI Logo">
  4. </p>
  5. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
  6. [![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)
  7. [![smithery badge](https://smithery.ai/badge/@ScrapeGraphAI/scrapegraph-mcp)](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp)
  8. A production-ready [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) server that provides seamless integration with the [ScrapeGraph AI](https://scrapegraphai.com) API. This server enables language models to leverage advanced AI-powered web scraping capabilities with enterprise-grade reliability.
  9. ## Table of Contents
  10. - [Key Features](#key-features)
  11. - [Quick Start](#quick-start)
  12. - [Available Tools](#available-tools)
  13. - [Setup Instructions](#setup-instructions)
  14. - [Remote Server Usage](#remote-server-usage)
  15. - [Local Usage](#local-usage)
  16. - [Google ADK Integration](#google-adk-integration)
  17. - [Example Use Cases](#example-use-cases)
  18. - [Error Handling](#error-handling)
  19. - [Common Issues](#common-issues)
  20. - [Development](#development)
  21. - [Contributing](#contributing)
  22. - [Documentation](#documentation)
  23. - [Technology Stack](#technology-stack)
  24. - [License](#license)
  25. ## API v2
  26. This MCP server targets **ScrapeGraph API v2** (`https://v2-api.scrapegraphai.com/api`), aligned 1:1 with
  27. [scrapegraph-py PR #84](https://github.com/ScrapeGraphAI/scrapegraph-py/pull/84). Auth uses the
  28. `SGAI-APIKEY` header. Environment variables mirror the Python SDK:
  29. - **`SGAI_API_URL`** — override the base URL (default `https://v2-api.scrapegraphai.com/api`)
  30. - **`SGAI_TIMEOUT`** — request timeout in seconds (default `120`)
  31. - **`SGAI_API_KEY`** — API key (can also be passed via MCP `scrapegraphApiKey` or `X-API-Key` header)
  32. > Legacy aliases (still honored): `SCRAPEGRAPH_API_BASE_URL` for `SGAI_API_URL`, `SGAI_TIMEOUT_S` for `SGAI_TIMEOUT`.
  33. ## Key Features
  34. - **Scrape & extract**: `scrape` (POST /scrape, multi-format), `extract` (POST /extract, URL + prompt)
  35. - **Search**: `search` (POST /search; `num_results` clamped 3–20)
  36. - **Crawl**: Async multi-page crawl with `crawl_start` / `crawl_get_status` / `crawl_stop` / `crawl_resume`
  37. - **Schema**: `schema` (POST /schema) — generate or augment a JSON Schema from a prompt
  38. - **Monitors**: Scheduled jobs via `monitor_create`, `monitor_list`, `monitor_get`, pause/resume/delete, `monitor_activity` (paginated tick history)
  39. - **Account**: `credits`, `history`
  40. - **Easy integration**: Claude Desktop, Cursor, Smithery, HTTP transport
  41. - **Developer docs**: `.agent/` folder
  42. ## Migration: v2 → v3
  43. v3 renames every MCP tool that diverged from the v2 API docs. **Hard rename, no aliases.**
  44. | v2 (old) | v3 (new) |
  45. |---|---|
  46. | `smartscraper` | `extract` |
  47. | `searchscraper` | `search` |
  48. | `smartcrawler_initiate` | `crawl_start` |
  49. | `smartcrawler_fetch_results` | `crawl_get_status` |
  50. | `sgai_history` | `history` |
  51. | `generate_schema` | `schema` |
  52. | `markdownify` | **removed** — use `scrape` with `output_format="markdown"` |
  53. ## Quick Start
  54. ### 1. Get Your API Key
  55. Sign up and get your API key from the [ScrapeGraph Dashboard](https://dashboard.scrapegraphai.com)
  56. ### 2. Install with Smithery (Recommended)
  57. ```bash
  58. npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude
  59. ```
  60. ### 3. Start Using
  61. Ask Claude or Cursor:
  62. - "Convert https://scrapegraphai.com to markdown"
  63. - "Extract all product prices from this e-commerce page"
  64. - "Research the latest AI developments and summarize findings"
  65. That's it! The server is now available to your AI assistant.
  66. ## Available Tools
  67. | Tool | Role |
  68. |------|------|
  69. | `scrape` | POST /scrape (`output_format`: markdown, html, screenshot, branding, links, images, summary) |
  70. | `extract` | POST /extract (requires `website_url` + `user_prompt`; optional `output_schema`) |
  71. | `search` | POST /search (`num_results` 1–20; supports `country_search`, `time_range`, `output_schema`) |
  72. | `crawl_start` | POST /crawl — `extraction_mode` markdown / html / links / images / summary / branding / screenshot |
  73. | `crawl_get_status` | GET /crawl/:id (poll until `status: completed`) |
  74. | `crawl_stop`, `crawl_resume` | POST /crawl/:id/stop \| resume |
  75. | `schema` | POST /schema (generate or augment a JSON Schema from a prompt) |
  76. | `credits` | GET /credits |
  77. | `history` | GET /history (paginated, `service` filter) |
  78. | `monitor_create`, `monitor_list`, `monitor_get`, `monitor_pause`, `monitor_resume`, `monitor_delete` | /monitor API |
  79. | `monitor_activity` | GET /monitor/:id/activity (paginated tick history: `id`, `createdAt`, `status`, `changed`, `elapsedMs`, `diffs`) |
  80. **Removed:** `sitemap`, `agentic_scrapper`, async-status polling, and (in v3) `markdownify` — use `scrape` with `output_format="markdown"`.
  81. ## Setup Instructions
  82. To utilize this server, you'll need a ScrapeGraph API key. Follow these steps to obtain one:
  83. 1. Navigate to the [ScrapeGraph Dashboard](https://dashboard.scrapegraphai.com)
  84. 2. Create an account and generate your API key
  85. ### Automated Installation via Smithery
  86. For automated installation of the ScrapeGraph API Integration Server using [Smithery](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp):
  87. ```bash
  88. npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude
  89. ```
  90. ### Claude Desktop Configuration
  91. Update your Claude Desktop configuration file with the following settings (located on the top rigth of the Cursor page):
  92. (remember to add your API key inside the config)
  93. ```json
  94. {
  95. "mcpServers": {
  96. "@ScrapeGraphAI-scrapegraph-mcp": {
  97. "command": "npx",
  98. "args": [
  99. "-y",
  100. "@smithery/cli@latest",
  101. "run",
  102. "@ScrapeGraphAI/scrapegraph-mcp",
  103. "--config",
  104. "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
  105. ]
  106. }
  107. }
  108. }
  109. ```
  110. The configuration file is located at:
  111. - Windows: `%APPDATA%/Claude/claude_desktop_config.json`
  112. - macOS: `~/Library/Application\ Support/Claude/claude_desktop_config.json`
  113. ### Cursor Integration
  114. Add the ScrapeGraphAI MCP server on the settings:
  115. ![Cursor MCP Integration](assets/cursor_mcp.png)
  116. ## Remote Server Usage
  117. Connect to our hosted MCP server - no local installation required!
  118. > [!WARNING]
  119. > The legacy MCP endpoint at `https://mcp.scrapegraphai.com/mcp` will be
  120. > deprecated soon. New integrations should use the replacement MCP endpoint:
  121. > `https://sgai-mcp-main.onrender.com`.
  122. ### Claude Desktop Configuration (Remote)
  123. Add this to your Claude Desktop config (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS):
  124. ```json
  125. {
  126. "mcpServers": {
  127. "scrapegraph-mcp": {
  128. "command": "npx",
  129. "args": [
  130. "mcp-remote@0.1.25",
  131. "https://sgai-mcp-main.onrender.com",
  132. "--header",
  133. "X-API-Key:YOUR_API_KEY"
  134. ]
  135. }
  136. }
  137. }
  138. ```
  139. ### Cursor Configuration (Remote)
  140. Cursor supports native HTTP MCP connections. Add to your Cursor MCP settings (`~/.cursor/mcp.json`):
  141. ```json
  142. {
  143. "mcpServers": {
  144. "scrapegraph-mcp": {
  145. "url": "https://sgai-mcp-main.onrender.com",
  146. "headers": {
  147. "X-API-Key": "YOUR_API_KEY"
  148. }
  149. }
  150. }
  151. }
  152. ```
  153. ### Benefits of Remote Server
  154. - **No local setup** - Just configure and start using
  155. - **Always up-to-date** - Automatically receives latest updates
  156. - **Cross-platform** - Works on any OS with Node.js
  157. ## Local Usage
  158. To run the MCP server locally for development or testing, follow these steps:
  159. ### Prerequisites
  160. - Python 3.13 or higher
  161. - pip or uv package manager
  162. - ScrapeGraph API key
  163. ### Installation
  164. 1. **Clone the repository** (if you haven't already):
  165. ```bash
  166. git clone https://github.com/ScrapeGraphAI/scrapegraph-mcp
  167. cd scrapegraph-mcp
  168. ```
  169. 2. **Install the package**:
  170. ```bash
  171. # Using pip
  172. pip install -e .
  173. # Or using uv (faster)
  174. uv pip install -e .
  175. ```
  176. 3. **Set your API key**:
  177. ```bash
  178. # macOS/Linux
  179. export SGAI_API_KEY=your-api-key-here
  180. # Windows (PowerShell)
  181. $env:SGAI_API_KEY="your-api-key-here"
  182. # Windows (CMD)
  183. set SGAI_API_KEY=your-api-key-here
  184. ```
  185. ### Running the Server Locally
  186. You can run the server directly:
  187. ```bash
  188. # Using the installed command
  189. scrapegraph-mcp
  190. # Or using Python module
  191. python -m scrapegraph_mcp.server
  192. ```
  193. The server will start and communicate via stdio (standard input/output), which is the standard MCP transport method.
  194. ### Testing with MCP Inspector
  195. Test your local server using the MCP Inspector tool:
  196. ```bash
  197. npx @modelcontextprotocol/inspector python -m scrapegraph_mcp.server
  198. ```
  199. This provides a web interface to test all available tools interactively.
  200. ### Configuring Claude Desktop for Local Server
  201. To use your locally running server with Claude Desktop, update your configuration file:
  202. **macOS/Linux** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
  203. ```json
  204. {
  205. "mcpServers": {
  206. "scrapegraph-mcp-local": {
  207. "command": "python",
  208. "args": [
  209. "-m",
  210. "scrapegraph_mcp.server"
  211. ],
  212. "env": {
  213. "SGAI_API_KEY": "your-api-key-here"
  214. }
  215. }
  216. }
  217. }
  218. ```
  219. **Windows** (`%APPDATA%\Claude\claude_desktop_config.json`):
  220. ```json
  221. {
  222. "mcpServers": {
  223. "scrapegraph-mcp-local": {
  224. "command": "python",
  225. "args": [
  226. "-m",
  227. "scrapegraph_mcp.server"
  228. ],
  229. "env": {
  230. "SGAI_API_KEY": "your-api-key-here"
  231. }
  232. }
  233. }
  234. }
  235. ```
  236. **Note**: Make sure Python is in your PATH. You can verify by running `python --version` in your terminal.
  237. ### Configuring Cursor for Local Server
  238. In Cursor's MCP settings, add a new server with:
  239. - **Command**: `python`
  240. - **Args**: `["-m", "scrapegraph_mcp.server"]`
  241. - **Environment Variables**: `{"SGAI_API_KEY": "your-api-key-here"}`
  242. ### Troubleshooting Local Setup
  243. **Server not starting:**
  244. - Verify Python is installed: `python --version`
  245. - Check that the package is installed: `pip list | grep scrapegraph-mcp`
  246. - Ensure API key is set: `echo $SGAI_API_KEY` (macOS/Linux) or `echo %SGAI_API_KEY%` (Windows)
  247. **Tools not appearing:**
  248. - Check Claude Desktop logs:
  249. - macOS: `~/Library/Logs/Claude/`
  250. - Windows: `%APPDATA%\Claude\Logs\`
  251. - Verify the server starts without errors when run directly
  252. - Check that the configuration JSON is valid
  253. **Import errors:**
  254. - Reinstall the package: `pip install -e . --force-reinstall`
  255. - Verify dependencies: `pip install -r requirements.txt` (if available)
  256. ## Google ADK Integration
  257. The ScrapeGraph MCP server can be integrated with [Google ADK (Agent Development Kit)](https://github.com/google/adk) to create AI agents with web scraping capabilities.
  258. ### Prerequisites
  259. - Python 3.13 or higher
  260. - Google ADK installed
  261. - ScrapeGraph API key
  262. ### Installation
  263. 1. **Install Google ADK** (if not already installed):
  264. ```bash
  265. pip install google-adk
  266. ```
  267. 2. **Set your API key**:
  268. ```bash
  269. export SGAI_API_KEY=your-api-key-here
  270. ```
  271. ### Basic Integration Example
  272. Create an agent file (e.g., `agent.py`) with the following configuration:
  273. ```python
  274. import os
  275. from google.adk.agents import LlmAgent
  276. from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset
  277. from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams
  278. from mcp import StdioServerParameters
  279. # Path to the scrapegraph-mcp server directory
  280. SCRAPEGRAPH_MCP_PATH = "/path/to/scrapegraph-mcp"
  281. # Path to the server.py file
  282. SERVER_SCRIPT_PATH = os.path.join(
  283. SCRAPEGRAPH_MCP_PATH,
  284. "src",
  285. "scrapegraph_mcp",
  286. "server.py"
  287. )
  288. root_agent = LlmAgent(
  289. model='gemini-2.0-flash',
  290. name='scrapegraph_assistant_agent',
  291. instruction='Help the user with web scraping and data extraction using ScrapeGraph AI. '
  292. 'You can convert webpages to markdown, extract structured data using AI, '
  293. 'perform web searches, crawl multiple pages, and automate complex scraping workflows.',
  294. tools=[
  295. MCPToolset(
  296. connection_params=StdioConnectionParams(
  297. server_params=StdioServerParameters(
  298. command='python3',
  299. args=[
  300. SERVER_SCRIPT_PATH,
  301. ],
  302. env={
  303. 'SGAI_API_KEY': os.getenv('SGAI_API_KEY'),
  304. },
  305. ),
  306. timeout=300.0,)
  307. ),
  308. # Optional: Filter which tools from the MCP server are exposed
  309. # tool_filter=['scrape', 'extract', 'search']
  310. )
  311. ],
  312. )
  313. ```
  314. ### Configuration Options
  315. **Timeout Settings:**
  316. - Default timeout is 5 seconds, which may be too short for web scraping operations
  317. - Recommended: Set `timeout=300.0
  318. - Adjust based on your use case (crawling operations may need even longer timeouts)
  319. **Tool Filtering:**
  320. - By default, all registered MCP tools are exposed to the agent (see [Available Tools](#available-tools))
  321. - Use `tool_filter` to limit which tools are available:
  322. ```python
  323. tool_filter=['scrape', 'extract', 'search']
  324. ```
  325. **API Key Configuration:**
  326. - Set via environment variable: `export SGAI_API_KEY=your-key`
  327. - Or pass directly in `env` dict: `'SGAI_API_KEY': 'your-key-here'`
  328. - Environment variable approach is recommended for security
  329. ### Usage Example
  330. Once configured, your agent can use natural language to interact with web scraping tools:
  331. ```python
  332. # The agent can now handle queries like:
  333. # - "Convert https://example.com to markdown"
  334. # - "Extract all product prices from this e-commerce page"
  335. # - "Search for recent AI research papers and summarize them"
  336. # - "Crawl this documentation site and extract all API endpoints"
  337. ```
  338. For more information about Google ADK, visit the [official documentation](https://github.com/google/adk).
  339. ## Example Use Cases
  340. The server enables sophisticated queries across various scraping scenarios:
  341. ### Single Page Scraping
  342. - **Markdownify**: "Convert the ScrapeGraph documentation page to markdown"
  343. - **Extract**: "Extract all product names, prices, and ratings from this e-commerce page"
  344. - **Extract with scrolling**: "Scrape this infinite scroll page with 5 scrolls and extract all items"
  345. - **Basic Scrape**: "Fetch the HTML content of this JavaScript-heavy page with full rendering"
  346. ### Search and Research
  347. - **Search**: "Research and summarize recent developments in AI-powered web scraping"
  348. - **Search**: "Search for the top 5 articles about machine learning frameworks and extract key insights"
  349. - **Search**: "Find recent news about GPT-4 and provide a structured summary"
  350. - **Search**: v2 does not apply `time_range`; phrase queries to bias recency in natural language instead
  351. ### Website analysis
  352. - Use **`crawl_start`** plus **`crawl_get_status`** to map and capture multi-page content; there is no separate **sitemap** tool on v2.
  353. ### Multi-page crawling
  354. - **Crawl**: "Crawl the blog in markdown mode and poll until complete"
  355. - For structured fields per page, run **`extract`** on individual URLs (or **`monitor_create`** on a schedule)
  356. ### Monitors and account
  357. - **Monitor**: "Run this extract prompt on https://example.com every day at 9am" (`monitor_create` with interval)
  358. - **Credits / history**: `credits`, `history`
  359. - **Agentic Scraper**: "Execute a complex workflow: login, navigate to reports, download data, and extract summary statistics"
  360. ## Error Handling
  361. The server implements robust error handling with detailed, actionable error messages for:
  362. - API authentication issues
  363. - Malformed URL structures
  364. - Network connectivity failures
  365. - Rate limiting and quota management
  366. ## Common Issues
  367. ### Windows-Specific Connection
  368. When running on Windows systems, you may need to use the following command to connect to the MCP server:
  369. ```bash
  370. C:\Windows\System32\cmd.exe /c npx -y @smithery/cli@latest run @ScrapeGraphAI/scrapegraph-mcp --config "{\"scrapegraphApiKey\":\"YOUR-SGAI-API-KEY\"}"
  371. ```
  372. This ensures proper execution in the Windows environment.
  373. ### Other Common Issues
  374. **"ScrapeGraph client not initialized"**
  375. - **Cause**: Missing API key
  376. - **Solution**: Set `SGAI_API_KEY` environment variable or provide via `--config`
  377. **"Error 401: Unauthorized"**
  378. - **Cause**: Invalid API key
  379. - **Solution**: Verify your API key at the [ScrapeGraph Dashboard](https://dashboard.scrapegraphai.com)
  380. **"Error 402: Payment Required"**
  381. - **Cause**: Insufficient credits
  382. - **Solution**: Add credits to your ScrapeGraph account
  383. **Crawl not returning results**
  384. - **Cause**: Still processing (asynchronous operation)
  385. - **Solution**: Keep polling `crawl_get_status()` until status is "completed"
  386. **Tools not appearing in Claude Desktop**
  387. - **Cause**: Server not starting or configuration error
  388. - **Solution**: Check Claude logs at `~/Library/Logs/Claude/` (macOS) or `%APPDATA%\Claude\Logs\` (Windows)
  389. For detailed troubleshooting, see the [.agent documentation](.agent/README.md).
  390. ## Development
  391. ### Prerequisites
  392. - Python 3.13 or higher
  393. - pip or uv package manager
  394. - ScrapeGraph API key
  395. ### Installation from Source
  396. ```bash
  397. # Clone the repository
  398. git clone https://github.com/ScrapeGraphAI/scrapegraph-mcp
  399. cd scrapegraph-mcp
  400. # Install dependencies
  401. pip install -e ".[dev]"
  402. # Set your API key
  403. export SGAI_API_KEY=your-api-key
  404. # Run the server
  405. scrapegraph-mcp
  406. # or
  407. python -m scrapegraph_mcp.server
  408. ```
  409. ### Testing with MCP Inspector
  410. Test your server locally using the MCP Inspector tool:
  411. ```bash
  412. npx @modelcontextprotocol/inspector scrapegraph-mcp
  413. ```
  414. This provides a web interface to test all available tools.
  415. ### Code Quality
  416. **Linting:**
  417. ```bash
  418. ruff check src/
  419. ```
  420. **Type Checking:**
  421. ```bash
  422. mypy src/
  423. ```
  424. **Format Checking:**
  425. ```bash
  426. ruff format --check src/
  427. ```
  428. ### Project Structure
  429. ```
  430. scrapegraph-mcp/
  431. ├── src/
  432. │ └── scrapegraph_mcp/
  433. │ ├── __init__.py # Package initialization
  434. │ └── server.py # Main MCP server (all code in one file)
  435. ├── .agent/ # Developer documentation
  436. │ ├── README.md # Documentation index
  437. │ └── system/ # System architecture docs
  438. ├── assets/ # Images and badges
  439. ├── pyproject.toml # Project metadata & dependencies
  440. ├── smithery.yaml # Smithery deployment config
  441. └── README.md # This file
  442. ```
  443. ## Contributing
  444. We welcome contributions! Here's how you can help:
  445. ### Adding a New Tool
  446. 1. **Add method to `ScapeGraphClient` class** in [server.py](src/scrapegraph_mcp/server.py):
  447. ```python
  448. def new_tool(self, param: str) -> Dict[str, Any]:
  449. """Tool description."""
  450. url = f"{self.BASE_URL}/new-endpoint"
  451. data = {"param": param}
  452. response = self.client.post(url, headers=self.headers, json=data)
  453. if response.status_code != 200:
  454. raise Exception(f"Error {response.status_code}: {response.text}")
  455. return response.json()
  456. ```
  457. 2. **Add MCP tool decorator**:
  458. ```python
  459. @mcp.tool()
  460. def new_tool(param: str) -> Dict[str, Any]:
  461. """
  462. Tool description for AI assistants.
  463. Args:
  464. param: Parameter description
  465. Returns:
  466. Dictionary containing results
  467. """
  468. if scrapegraph_client is None:
  469. return {"error": "ScrapeGraph client not initialized. Please provide an API key."}
  470. try:
  471. return scrapegraph_client.new_tool(param)
  472. except Exception as e:
  473. return {"error": str(e)}
  474. ```
  475. 3. **Test with MCP Inspector**:
  476. ```bash
  477. npx @modelcontextprotocol/inspector scrapegraph-mcp
  478. ```
  479. 4. **Update documentation**:
  480. - Add tool to this README
  481. - Update [.agent documentation](.agent/README.md)
  482. 5. **Submit a pull request**
  483. ### Development Workflow
  484. 1. Fork the repository
  485. 2. Create a feature branch (`git checkout -b feature/amazing-feature`)
  486. 3. Make your changes
  487. 4. Run linting and type checking
  488. 5. Test with MCP Inspector and Claude Desktop
  489. 6. Update documentation
  490. 7. Commit your changes (`git commit -m 'Add amazing feature'`)
  491. 8. Push to the branch (`git push origin feature/amazing-feature`)
  492. 9. Open a Pull Request
  493. ### Code Style
  494. - **Line length**: 100 characters
  495. - **Type hints**: Required for all functions
  496. - **Docstrings**: Google-style docstrings
  497. - **Error handling**: Return error dicts, don't raise exceptions in tools
  498. - **Python version**: Target 3.13+
  499. For detailed development guidelines, see the [.agent documentation](.agent/README.md).
  500. ## Documentation
  501. For comprehensive developer documentation, see:
  502. - **[.agent/README.md](.agent/README.md)** - Complete developer documentation index
  503. - **[.agent/system/project_architecture.md](.agent/system/project_architecture.md)** - System architecture and design
  504. - **[.agent/system/mcp_protocol.md](.agent/system/mcp_protocol.md)** - MCP protocol integration details
  505. ## Technology Stack
  506. ### Core Framework
  507. - **Python 3.13+** - Modern Python with type hints
  508. - **FastMCP** - Lightweight MCP server framework
  509. - **httpx 0.24.0+** - Modern async HTTP client
  510. ### Development Tools
  511. - **Ruff** - Fast Python linter and formatter
  512. - **mypy** - Static type checker
  513. - **Hatchling** - Modern build backend
  514. ### Deployment
  515. - **Smithery** - Automated MCP server deployment
  516. - **Docker** - Container support with Alpine Linux
  517. - **stdio transport** - Standard MCP communication
  518. ### API Integration
  519. - **ScrapeGraph AI API** - Enterprise web scraping service
  520. - **Base URL**: `https://v2-api.scrapegraphai.com/api`
  521. - **Authentication**: API key-based
  522. ## License
  523. This project is distributed under the MIT License. For detailed terms and conditions, please refer to the LICENSE file.
  524. ## Acknowledgments
  525. Special thanks to [tomekkorbak](https://github.com/tomekkorbak) for his implementation of [oura-mcp-server](https://github.com/tomekkorbak/oura-mcp-server), which served as starting point for this repo.
  526. ## Resources
  527. ### Official Links
  528. - [ScrapeGraph AI Homepage](https://scrapegraphai.com)
  529. - [ScrapeGraph Dashboard](https://dashboard.scrapegraphai.com) - Get your API key
  530. - [ScrapeGraph API Documentation](https://api.scrapegraphai.com/docs)
  531. - [GitHub Repository](https://github.com/ScrapeGraphAI/scrapegraph-mcp)
  532. ### MCP Resources
  533. - [Model Context Protocol](https://modelcontextprotocol.io/) - Official MCP specification
  534. - [FastMCP Framework](https://github.com/jlowin/fastmcp) - Framework used by this server
  535. - [MCP Inspector](https://github.com/modelcontextprotocol/inspector) - Testing tool
  536. - [Smithery](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp) - MCP server distribution
  537. - mcp-name: io.github.ScrapeGraphAI/scrapegraph-mcp
  538. ### AI Assistant Integration
  539. - [Claude Desktop](https://claude.ai/desktop) - Desktop app with MCP support
  540. - [Cursor](https://cursor.sh/) - AI-powered code editor
  541. ### Support
  542. - [GitHub Issues](https://github.com/ScrapeGraphAI/scrapegraph-mcp/issues) - Report bugs or request features
  543. - [Developer Documentation](.agent/README.md) - Comprehensive dev docs
  544. ---
  545. Made with ❤️ by [ScrapeGraphAI](https://scrapegraphai.com) Team

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.