Skip to content
Back to search

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/d4vinci/scrapling with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/d4vinci/scrapling'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/d4vinci-scrapling/badge.svg)](/listings/d4vinci-scrapling)

HTML

<a href="/listings/d4vinci-scrapling"><img src="/api/listings/d4vinci-scrapling/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Let's give you a quick glimpse of what Scrapling can do without deep diving.

Basic Usage

HTTP requests with session support

from scrapling.fetchers import Fetcher, FetcherSession

with FetcherSession(impersonate='chrome') as session:  # Use latest version of Chrome's TLS fingerprint
    page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
    quotes = page.css('.quote .text::text').getall()

# Or use one-off requests
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()

Advanced stealth mode

from scrapling.fetchers import StealthyFetcher, StealthySession

with StealthySession(headless=True, solve_cloudflare=True) as session:  # Keep the browser open until you finish
    page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
    data = page.css('#padded_content a').getall()

# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
data = page.css('#padded_content a').getall()

Full browser automation

from scrapling.fetchers import DynamicFetcher, DynamicSession

with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session:  # Keep the browser open until you finish
    page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
    data = page.xpath('//span[@class="text"]/text()').getall()  # XPath selector if you prefer it

# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()

Spiders

Build full crawlers with concurrent requests, multiple session types, and pause/resume:

from scrapling.spiders import Spider, Request, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10

    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }

        next_page = response.css('.next a')
        if next_page:
            yield response.follow(next_page[0].attrib['href'])

result = QuotesSpider().start()
print(f"Scraped {len(result.items)} quotes")
result.items.to_json("quotes.json")

Use multiple session types in a single spider:

from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession

class MultiSessionSpider(Spider):
    name = "multi"
    start_urls = ["https://example.com/"]

    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)

    async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            # Route protected pages through the stealth session
            if "protected" in link:
                yield Request(link, sid="stealth")
            else:
                yield Request(link, sid="fast", callback=self.parse)  # explicit callback

Pause and resume long crawls with checkpoints by running the spider like this:

QuotesSpider(crawldir="./crawl_data").start()

Press Ctrl+C to pause gracefully - progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped.

Or skip writing the crawling logic altogether with the ready-made templates, like pulling an entire Shopify store's catalog:

from scrapling.spiders import ShopifySpider

class MyStore(ShopifySpider):
    target_website = "example.com"

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 619 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <!-- mcp-name: io.github.D4Vinci/Scrapling -->
  2. <h1 align="center">
  3. <a href="https://scrapling.readthedocs.io">
  4. <picture>
  5. <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/cover_dark.svg?sanitize=true">
  6. <img alt="Scrapling Poster" src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/cover_light.svg?sanitize=true">
  7. </picture>
  8. </a>
  9. <br>
  10. <small>Effortless Web Scraping for the Modern Web</small>
  11. </h1>
  12. <p align="center">
  13. <a href="https://trendshift.io/repositories/14244" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14244" alt="D4Vinci%2FScrapling | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
  14. <a href="https://www.star-history.com/d4vinci/scrapling"><img src="https://api.star-history.com/badge?repo=D4Vinci/Scrapling&type=rank" alt="Star History Rank" style="height: 55px;" height="55"/></a>
  15. <br/>
  16. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_AR.md"><img alt="README بالعربية" title="README بالعربية" src="https://img.shields.io/badge/Arabic-DFE0E5"></a>
  17. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-DFE0E5"></a>
  18. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_PT_BR.md"><img alt="README em Português (Brasil)" src="https://img.shields.io/badge/Português%20(Brasil)-DFE0E5"></a>
  19. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-DFE0E5"></a>
  20. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_DE.md"><img alt="README auf Deutsch" src="https://img.shields.io/badge/Deutsch-DFE0E5"></a>
  21. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-DFE0E5"></a>
  22. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_JP.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-DFE0E5"></a>
  23. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_RU.md"><img alt="Русская версия README" src="https://img.shields.io/badge/Русский-DFE0E5"></a>
  24. <a href="https://github.com/D4Vinci/Scrapling/blob/main/docs/README_KR.md"><img alt="한국어 README" src="https://img.shields.io/badge/한국어-DFE0E5"></a>
  25. <br/>
  26. <a href="https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml" alt="Tests">
  27. <img alt="Tests" src="https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml/badge.svg"></a>
  28. <a href="https://badge.fury.io/py/Scrapling" alt="PyPI version">
  29. <img alt="PyPI version" src="https://badge.fury.io/py/Scrapling.svg"></a>
  30. <a href="https://hub.docker.com/r/pyd4vinci/scrapling" target="_blank">
  31. <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/pyd4vinci/scrapling?labelColor=%20%23FDB062&logo=Docker&labelColor=%20%23528bff"></a>
  32. <a href="https://clickpy.clickhouse.com/dashboard/scrapling" rel="nofollow"><img src="https://img.shields.io/pypi/dm/scrapling" alt="PyPI package downloads"></a>
  33. <a href="https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html" alt="AI Agent Skill">
  34. <img alt="Static Badge" src="https://img.shields.io/badge/Skill-black?style=flat&label=Agent&link=https%3A%2F%2Fscrapling.readthedocs.io%2Fen%2Flatest%2Fai%2Fagent-skill.html"></a>
  35. <a href="https://clawhub.ai/D4Vinci/scrapling-official" alt="OpenClaw Skill">
  36. <img alt="OpenClaw Skill" src="https://img.shields.io/badge/Clawhub-darkred?style=flat&label=OpenClaw&link=https%3A%2F%2Fclawhub.ai%2FD4Vinci%2Fscrapling-official"></a>
  37. <br/>
  38. <a href="https://discord.gg/EMgGbDceNQ" alt="Discord" target="_blank">
  39. <img alt="Discord" src="https://img.shields.io/discord/1360786381042880532?style=social&logo=discord&link=https%3A%2F%2Fdiscord.gg%2FEMgGbDceNQ">
  40. </a>
  41. <a href="https://x.com/Scrapling_dev" alt="X (formerly Twitter)">
  42. <img alt="X (formerly Twitter) Follow" src="https://img.shields.io/twitter/follow/Scrapling_dev?style=social&logo=x&link=https%3A%2F%2Fx.com%2FScrapling_dev">
  43. </a>
  44. <br/>
  45. <a href="https://pypi.org/project/scrapling/" alt="Supported Python versions">
  46. <img alt="Supported Python versions" src="https://img.shields.io/pypi/pyversions/scrapling.svg"></a>
  47. </p>
  48. <p align="center">
  49. <a href="https://scrapling.readthedocs.io/en/latest/parsing/selection.html"><strong>Selection methods</strong></a>
  50. &middot;
  51. <a href="https://scrapling.readthedocs.io/en/latest/fetching/choosing.html"><strong>Fetchers</strong></a>
  52. &middot;
  53. <a href="https://scrapling.readthedocs.io/en/latest/spiders/architecture.html"><strong>Spiders</strong></a>
  54. &middot;
  55. <a href="https://scrapling.readthedocs.io/en/latest/spiders/proxy-blocking.html"><strong>Proxy Rotation</strong></a>
  56. &middot;
  57. <a href="https://scrapling.readthedocs.io/en/latest/cli/overview.html"><strong>CLI</strong></a>
  58. &middot;
  59. <a href="https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html"><strong>MCP</strong></a>
  60. </p>
  61. Scrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library.
  62. Scrape static pages, render JavaScript, interact with websites, and extract data from the API calls behind them. Build crawlers with reusable sessions, proxy rotation, automatic retries, pause/resume, and built-in data exports.
  63. Use it from Python, the terminal, or your AI agent through its MCP server and Agent Skill. From extracting a few fields to collecting a whole website as clean Markdown, Scrapling covers the full scraping workflow.
  64. ## What you can do with Scrapling
  65. - **[Fetch & interact](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
  66. - **[Parse & extract](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
  67. - **[Crawl & export](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
  68. - **[Scrape with AI or from your terminal](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.
  69. Start with a single page:
  70. ```python
  71. from scrapling.fetchers import Fetcher
  72. page = Fetcher.get('https://quotes.toscrape.com/')
  73. quotes = page.css('.quote .text::text').getall()
  74. print(quotes)
  75. ```
  76. Or scale up with a spider:
  77. ```python
  78. from scrapling.spiders import Spider, Response
  79. class QuotesSpider(Spider):
  80. name = "quotes"
  81. start_urls = ["https://quotes.toscrape.com/"]
  82. async def parse(self, response: Response):
  83. for quote in response.css('.quote'):
  84. yield {
  85. "text": quote.css('.text::text').get(),
  86. "author": quote.css('.author::text').get(),
  87. }
  88. result = QuotesSpider().start()
  89. print(len(result.items))
  90. result.items.to_json("quotes.json")
  91. ```
  92. This prints `10` and saves the quotes and authors to `quotes.json`.
  93. <p align="center">
  94. <a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
  95. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
  96. </a>
  97. </p>
  98. # Platinum Sponsors
  99. <table>
  100. <tr>
  101. <td width="200">
  102. <a href="https://go.nodemaven.com/scraplingoctober" target="_blank" title="Proxies with the Highest IP Scores">
  103. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/NodeMaven.jpg" width="240" height="100">
  104. </a>
  105. </td>
  106. <td>
  107. <a href="https://go.nodemaven.com/scraplingoctober" target="_blank">NodeMaven</a> - The most efficient proxy provider for Web Scraping and Automation with the Highest Quality IP on the market. SCRAPLING2X - 2X traffic for Black Friday
  108. </td>
  109. </tr>
  110. <tr>
  111. <td width="200">
  112. <a href="https://hypersolutions.co/?utm_source=github&utm_medium=readme&utm_campaign=scrapling" target="_blank" title="Bot Protection Bypass API for Akamai, DataDome, Incapsula & Kasada">
  113. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/HyperSolutions.png">
  114. </a>
  115. </td>
  116. <td> Scrapling handles Cloudflare Turnstile. For enterprise-grade protection, <a href="https://hypersolutions.co?utm_source=github&utm_medium=readme&utm_campaign=scrapling">
  117. <b>Hyper Solutions</b>
  118. </a> provides API endpoints that generate valid antibot tokens for <b>Akamai</b>, <b>DataDome</b>, <b>Kasada</b>, and <b>Incapsula</b>. Simple API calls, no browser automation required. </td>
  119. </tr>
  120. <tr>
  121. <td width="200">
  122. <a href="https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling" target="_blank" title="Evomi is your Swiss Quality Proxy Provider, starting at $0.49/GB">
  123. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/evomi.png">
  124. </a>
  125. </td>
  126. <td>
  127. <a href="https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling">
  128. <b>Evomi</b>
  129. </a>: residential proxies from $0.49/GB. Scraping browser with fully spoofed Chromium, residential IPs, auto CAPTCHA solving, and anti-bot bypass. </br>
  130. <b>Scraper API for hassle-free results. MCP and N8N integrations are available.</b>
  131. </td>
  132. </tr>
  133. <tr>
  134. <td width="200">
  135. <a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank" title="Unlock the Power of Social Media Data & AI">
  136. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/TikHub.jpg">
  137. </a>
  138. </td>
  139. <td>
  140. <a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets. <br /> Also offers <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">DISCOUNTED AI models</a> - Claude, GPT, GEMINI & more up to 71% off.
  141. </td>
  142. </tr>
  143. <tr>
  144. <td width="200">
  145. <a href="https://substack.thewebscraping.club/p/scrapling-hands-on-guide?utm_source=github&utm_medium=repo&utm_campaign=scrapling" target="_blank" title="The #1 newsletter dedicated to Web Scraping">
  146. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/TWSC.png">
  147. </a>
  148. </td>
  149. <td>
  150. Read a full review of <a href="https://substack.thewebscraping.club/p/scrapling-hands-on-guide?utm_source=github&utm_medium=repo&utm_campaign=scrapling" target="_blank">Scrapling on The Web Scraping Club</a> (Nov 2025), the #1 newsletter dedicated to Web Scraping.
  151. </td>
  152. </tr>
  153. <tr>
  154. <td width="200">
  155. <a href="https://byteful.com/?utm_source=github&utm_medium=github-sponsor&utm_campaign=scrapling_github_sponsor" target="_blank" title="Ethically sourced residential proxy infrastructure powering web scraping and AI agents">
  156. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/byteful.png">
  157. </a>
  158. </td>
  159. <td>
  160. <a href="https://byteful.com/?utm_source=github&utm_medium=github-sponsor&utm_campaign=scrapling_github_sponsor" target="_blank">Byteful</a> supplies ethical residential, mobile, ISP, and datacenter proxies, plus scraping and AI automation tools. Use code SCRAPLING10: 10% off residential proxies.
  161. </td>
  162. </tr>
  163. <tr>
  164. <td width="200">
  165. <a href="https://proxylane.dev/scrapling?utm_source=scrapling&utm_medium=affiliate&utm_campaign=scrapling_platinum_202609&utm_content=github_readme" target="_blank" title="90% fewer CAPTCHAs on your Scrapling runs.">
  166. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/proxylane.png">
  167. </a>
  168. </td>
  169. <td>
  170. <a href="https://proxylane.dev/scrapling?utm_source=scrapling&utm_medium=affiliate&utm_campaign=scrapling_platinum_202609&utm_content=github_readme" target="_blank">ProxyLane</a>: 90% fewer CAPTCHAs on your Scrapling runs. 28M real P2P residential devices, 98.7% clean IPs, 195 countries. City/ISP/ZIP targeting, sticky/rotating sessions, traffic never expires. Try 350 MB for $1.95. Code <strong>SCRAPEANDTAKE</strong>: <em>25% off</em>.
  171. </td>
  172. </tr>
  173. <tr>
  174. <td width="200">
  175. <a href="https://www.lundproxy.com/unlimited-residential-bandwidth.html?utm_source=github&utm_medium=sponsorship&utm_campaign=scrapling&utm_content=banner" target="_blank" title="Unlimited residential traffic, sized only by bandwidth.">
  176. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/lundproxy.png">
  177. </a>
  178. </td>
  179. <td>
  180. <a href="https://www.lundproxy.com?utm_source=github&utm_medium=sponsorship&utm_campaign=scrapling&utm_content=text" target="_blank">LundProxy</a>: Stop paying per GB. Unlimited residential traffic, 100M+ IPs, 190+ countries. Pay by bandwidth. Request a free 50 Mbps residential proxies trial.
  181. </td>
  182. </tr>
  183. <tr>
  184. <td width="200">
  185. <a href="https://resolvr.com/?ref=scrapling" target="_blank" title="Captcha and Browser challenges, solved on real devices loading the page.">
  186. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/resolvr.jpg">
  187. </a>
  188. </td>
  189. <td>
  190. <a href="https://resolvr.com/?ref=scrapling" target="_blank">Resolvr</a>: Scraping at scale means CAPTCHAs at scale. Resolvr solves reCAPTCHA and Cloudflare challenges on real devices through one API. Free beta access is available now.
  191. </td>
  192. </tr>
  193. <tr>
  194. <td width="200">
  195. <a href="https://nonecap.com/solutions/scrapling/?ref=scrapling" target="_blank" title="hCaptcha solver as an API.">
  196. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/Nonecap.png">
  197. </a>
  198. </td>
  199. <td>
  200. hCaptcha in your Scrapling runs? <a href="https://nonecap.com/solutions/scrapling/?ref=scrapling" target="_blank">NoneCap</a> returns valid hCaptcha tokens, Enterprise sitekeys included, through one API. 1,300 free credits on signup.
  201. </td>
  202. </tr>
  203. <tr>
  204. <td width="200">
  205. <a href="https://dataimpulse.com/?utm_source=website&utm_medium=github_readme&utm_campaign=scrapling_platinum" target="_blank" title="Buy Pay-as-you-go proxies from $1/GB">
  206. <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse_Platinum.png">
  207. </a>
  208. </td>
  209. <td>
  210. $1/GB at <a href="https://dataimpulse.com/?utm_source=website&utm_medium=github_readme&utm_campaign=scrapling_platinum" target="_blank">DataImpulse</a>. Ethical first-party proxies. 90M+ IPs, 195+ countries. Residential, datacenter, mobile. Pay-as-you-go = no subscription. Traffic never expires.
  211. </td>
  212. </tr>
  213. </table>
  214. <i><sub>Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=586646)</sub></i>
  215. # Sponsors
  216. <!-- sponsors -->
  217. <a href="https://cloro.dev/?utm_source=referral&utm_medium=scrapling" target="_blank" title="The search API for the AI era"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/cloro.jpg"></a>
  218. <br/>
  219. <a href="https://serpapi.com/?utm_source=scrapling" target="_blank" title="Scrape Google and other search engines with SerpApi"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/SerpApi.png"></a>
  220. <a href="https://visit.decodo.com/Dy6W0b" target="_blank" title="Try the Most Efficient Residential Proxies for Free"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/decodo.png"></a>
  221. <a href="https://proxyempire.io/?ref=scrapling&utm_source=github&utm_medium=paid_referral&utm_campaign=open_source_sponsorship&utm_content=scrapling" target="_blank" title="Collect The Data Your Project Needs with the Best Residential Proxies"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/ProxyEmpire.png"></a>
  222. <a href="https://www.webshare.io/?referral_code=48r2m2cd5uz1" target="_blank" title="The Most Reliable Proxy with Unparalleled Performance"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/webshare.png"></a>
  223. <a href="https://proxiware.com/?ref=scrapling" target="_blank" title="Collect Any Data. At Any Scale."><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/proxiware.png"></a>
  224. <a href="https://singaporemobileproxy.com/?utm_source=scrapling&utm_medium=sponsor&utm_campaign=paid_test_q4" target="_blank" title="Dedicated 4G/5G mobile proxies on real Singapore SIMs"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/SingaporeMobileProxy.png"></a>
  225. <a href="http://olostep.com/" target="_blank" title="Web Data Infrastructure for AI"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/olostep.png"></a>
  226. <!-- /sponsors -->
  227. <i><sub>Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci) and choose the tier that suits you!</sub></i>
  228. ---
  229. ## Key Features
  230. ### Spiders - A Full Crawling Framework
  231. - 🕷️ **Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects.
  232. - ⚡ **Concurrent Crawling**: Configurable concurrency limits, per-domain throttling, and download delays.
  233. - 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
  234. - 💾 **Pause & Resume**: Save pending requests and seen URLs with `crawldir`. Press Ctrl+C to stop gracefully and restart with the same directory.
  235. - 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.
  236. - 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.
  237. - 🚦 **AutoThrottle**: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what `Retry-After` asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops.
  238. - 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.
  239. - 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.
  240. - 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, `XMLFeedSpider`/`CSVFeedSpider` for iterating XML/RSS and CSV feeds, and `ShopifySpider` to collect products from public Shopify stores, one item per variant.
  241. - 🔗 **Link Extraction**: A standalone `LinkExtractor` primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.
  242. - 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with `result.items.to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()`.
  243. ### Advanced Websites Fetching with Session Support
  244. - **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
  245. - **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `StealthyFetcher` class supporting Playwright's Chromium and Google's Chrome.
  246. - **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
  247. - **Session Management**: Persistent session support with `FetcherSession` and `StealthySession` classes for cookie and state management across requests.
  248. - **Proxy Rotation**: Built-in `ProxyRotator` with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
  249. - **Domain & Ad Blocking**: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
  250. - **DNS Leak Prevention**: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
  251. - **Remote Browsers**: Instead of launching a browser locally, connect to one that's already running through CDP with `cdp_url`, whether it's on the same machine, another host, or a managed browser provider. You can also point any browser fetcher at your own Chromium build with `executable_path`.
  252. - **Background API Capture**: Enable `record_requests=True` on a browser session, then search its saved responses with `session.network.search(limit=None)`. Read API data and HTTP errors across page loads without sending requests again. History is bounded and saves text response bodies.
  253. - **Async Support**: Complete async support across all fetchers and dedicated async session classes.
  254. ### Parsing and Data Extraction
  255. - 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
  256. - 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
  257. - 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
  258. ### AI Features
  259. - 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with HTTP requests, browser actions, screenshots, saved login state, and background API responses. Uses our stealthy browser with the right tools to enable suitable AI models to bypass any CAPTCHA is faces while working (not to mention our Cloudflare solver). It uses a lot of methods to reduce the content sent to the model, block ads, and protect it against Prompt-Injection. Remote browsers over CDP and HTTP transport with authentication options are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
  260. - 🧠 **Agent Skill**: A ready-to-install [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
  261. - 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `ContentSpider` template, all without an LLM in the loop. ([docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
  262. ### High-Performance & battle-tested Architecture
  263. - 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
  264. - 🔋 **Memory Efficient**: Optimized data structures and lazy loading for a minimal memory footprint.
  265. - ⚡ **Fast JSON Serialization**: 10x faster than the standard library.
  266. - 🏗️ **Battle tested**: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.
  267. ### Developer/Web Scraper Friendly Experience
  268. - 🎯 **Interactive Web Scraping Shell**: Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.
  269. - 🚀 **Use it directly from the Terminal**: Optionally, you can use Scrapling to scrape a URL without writing a single line of code!
  270. - 🛠️ **Rich Navigation API**: Advanced DOM traversal with parent, sibling, and child navigation methods.
  271. - 🧬 **Enhanced Text Processing**: Built-in regex, cleaning methods, and optimized string operations.
  272. - 📝 **Auto Selector Generation**: Generate robust CSS/XPath selectors for any element.
  273. - 🔌 **Familiar API**: Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel.
  274. - 🤝 **Drop-in Scrapy Integration**: Already invested in Scrapy? Decorate any callback with `scrapling_response` to parse the responses you already fetch with Scrapling's parser, no rewrite needed.
  275. - 📘 **Complete Type Coverage**: Full type hints for excellent IDE support and code completion. The entire codebase is automatically scanned with **PyRight** and **MyPy** with each change.
  276. - 🔋 **Ready Docker image**: With each release, a Docker image containing all browsers is automatically built and pushed.
  277. ## Getting Started
  278. Let's give you a quick glimpse of what Scrapling can do without deep diving.
  279. ### Basic Usage
  280. HTTP requests with session support
  281. ```python
  282. from scrapling.fetchers import Fetcher, FetcherSession
  283. with FetcherSession(impersonate='chrome') as session: # Use latest version of Chrome's TLS fingerprint
  284. page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
  285. quotes = page.css('.quote .text::text').getall()
  286. # Or use one-off requests
  287. page = Fetcher.get('https://quotes.toscrape.com/')
  288. quotes = page.css('.quote .text::text').getall()
  289. ```
  290. Advanced stealth mode
  291. ```python
  292. from scrapling.fetchers import StealthyFetcher, StealthySession
  293. with StealthySession(headless=True, solve_cloudflare=True) as session: # Keep the browser open until you finish
  294. page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
  295. data = page.css('#padded_content a').getall()
  296. # Or use one-off request style, it opens the browser for this request, then closes it after finishing
  297. page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
  298. data = page.css('#padded_content a').getall()
  299. ```
  300. Full browser automation
  301. ```python
  302. from scrapling.fetchers import StealthyFetcher, StealthySession
  303. with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # Keep the browser open until you finish
  304. page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
  305. data = page.xpath('//span[@class="text"]/text()').getall() # XPath selector if you prefer it
  306. # Or use one-off request style, it opens the browser for this request, then closes it after finishing
  307. page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
  308. data = page.css('.quote .text::text').getall()
  309. ```
  310. ### Spiders
  311. Build full crawlers with concurrent requests, multiple session types, and pause/resume:
  312. ```python
  313. from scrapling.spiders import Spider, Response
  314. class QuotesSpider(Spider):
  315. name = "quotes"
  316. start_urls = ["https://quotes.toscrape.com/"]
  317. concurrent_requests = 10
  318. async def parse(self, response: Response):
  319. for quote in response.css('.quote'):
  320. yield {
  321. "text": quote.css('.text::text').get(),
  322. "author": quote.css('.author::text').get(),
  323. }
  324. next_page = response.css('.next a')
  325. if next_page:
  326. yield response.follow(next_page[0].attrib['href'])
  327. result = QuotesSpider().start()
  328. print(f"Scraped {len(result.items)} quotes")
  329. result.items.to_json("quotes.json")
  330. ```
  331. Use multiple session types in a single spider:
  332. ```python
  333. from scrapling.spiders import Spider, Request, Response
  334. from scrapling.fetchers import FetcherSession, AsyncStealthySession
  335. class MultiSessionSpider(Spider):
  336. name = "multi"
  337. start_urls = ["https://example.com/"]
  338. def configure_sessions(self, manager):
  339. manager.add("fast", FetcherSession(impersonate="chrome"))
  340. manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
  341. async def parse(self, response: Response):
  342. for link in response.css('a::attr(href)').getall():
  343. # Route protected pages through the stealth session
  344. if "protected" in link:
  345. yield Request(link, sid="stealth")
  346. else:
  347. yield Request(link, sid="fast", callback=self.parse) # explicit callback
  348. ```
  349. Pause and resume long crawls with checkpoints by running the spider like this:
  350. ```python
  351. QuotesSpider(crawldir="./crawl_data").start()
  352. ```
  353. With `crawldir` set, press Ctrl+C to save pending requests and seen URLs. Restart with the same `crawldir` to resume. Save scraped items separately.
  354. Or skip writing the crawling logic altogether with the ready-made templates, like pulling an entire Shopify store's catalog:
  355. ```python
  356. from scrapling.spiders import ShopifySpider
  357. class MyStore(ShopifySpider):
  358. target_website = "example.com"
  359. result = MyStore().start() # Every product in the store, one item per variant
  360. ```
  361. The template reads products exposed through public collection endpoints, one item per variant.
  362. ### Advanced Parsing & Navigation
  363. ```python
  364. from scrapling.fetchers import Fetcher
  365. # Rich element selection and navigation
  366. page = Fetcher.get('https://quotes.toscrape.com/')
  367. # Get quotes with multiple selection methods
  368. quotes = page.css('.quote') # CSS selector
  369. quotes = page.xpath('//div[@class="quote"]') # XPath
  370. quotes = page.find_all('div', {'class': 'quote'}) # BeautifulSoup-style
  371. # Same as
  372. quotes = page.find_all('div', class_='quote')
  373. quotes = page.find_all(['div'], class_='quote')
  374. quotes = page.find_all(class_='quote') # and so on...
  375. # Find element by text content
  376. quotes = page.find_by_text('Albert Einstein', first_match=False)
  377. # Advanced navigation
  378. quote_text = page.css('.quote')[0].css('.text::text').get()
  379. quote_texts = page.css('.quote').css('.text::text').getall() # Chained selectors
  380. first_quote = page.css('.quote')[0]
  381. author = first_quote.css('.author::text').get()
  382. next_quote = first_quote.next
  383. parent_container = first_quote.parent
  384. # Element relationships and similarity
  385. similar_elements = first_quote.find_similar()
  386. below_elements = first_quote.below_elements
  387. ```
  388. You can use the parser right away if you don't want to fetch websites like below:
  389. ```python
  390. from scrapling.parser import Selector
  391. page = Selector("<html>...</html>")
  392. ```
  393. And it works precisely the same way!
  394. ### Async Session Management Examples
  395. ```python
  396. import asyncio
  397. from scrapling.fetchers import FetcherSession, AsyncStealthySession
  398. async def main():
  399. async with FetcherSession(http3=True) as http_session: # `FetcherSession` is context-aware and can work in both sync/async patterns
  400. page1 = await http_session.get('https://quotes.toscrape.com/')
  401. page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
  402. print(len(page1.css('.quote')), len(page2.css('.quote')))
  403. async with AsyncStealthySession(max_pages=2) as browser_session:
  404. urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
  405. pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
  406. print([len(page.css('.quote')) for page in pages])
  407. asyncio.run(main())
  408. ```
  409. ## CLI & Interactive Shell
  410. Scrapling includes a powerful command-line interface:
  411. [![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
  412. Launch the interactive Web Scraping shell
  413. ```bash
  414. pip install "scrapling[shell]"
  415. scrapling shell
  416. ```
  417. Extract pages to a file directly without programming. Use `--css-selector` to pick the content, or `--ai-targeted` to select and sanitize the `body` content. Use `.txt` for text, `.md` for Markdown, or `.html` for HTML.
  418. ```bash
  419. scrapling extract get 'https://quotes.toscrape.com/' quotes.md
  420. scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
  421. scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
  422. scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
  423. ```
  424. > [!NOTE]
  425. > There are many additional features, but we want to keep this page concise, including the MCP server and the interactive Web Scraping Shell. Check out the full documentation [here](https://scrapling.readthedocs.io/en/latest/)
  426. ## Performance Benchmarks
  427. Scrapling isn't just powerful-it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.
  428. ### Text Extraction Speed Test (5000 nested elements)
  429. | # | Library | Time (ms) | vs Scrapling |
  430. |---|:-----------------:|:---------:|:------------:|
  431. | 1 | Scrapling | 1.99 | 1.0x |
  432. | 2 | Parsel/Scrapy | 2.06 | 1.035 |
  433. | 3 | Raw Lxml | 2.56 | 1.286 |
  434. | 4 | PyQuery | 23.98 | ~12x |
  435. | 5 | Selectolax | 197.02 | ~99x |
  436. | 6 | MechanicalSoup | 1545.15 | ~776.5x |
  437. | 7 | BS4 with Lxml | 1562.1 | ~785.0x |
  438. | 8 | BS4 with html5lib | 3412.73 | ~1714.9x |
  439. ### Element Similarity & Text Search Performance
  440. Scrapling's adaptive element finding capabilities significantly outperform alternatives:
  441. | Library | Time (ms) | vs Scrapling |
  442. |-------------|:---------:|:------------:|
  443. | Scrapling | 2.3 | 1.0x |
  444. | AutoScraper | 12.58 | 5.47x |
  445. > All benchmarks represent averages of 100+ runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology.
  446. ## Installation
  447. Scrapling requires Python 3.10 or higher:
  448. ```bash
  449. pip install scrapling
  450. ```
  451. > [!IMPORTANT]
  452. > This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies. So importing anything from `scrapling.fetchers` or `scrapling.spiders`, like in the examples above, will raise `ModuleNotFoundError` with this installation alone. If you are going to use any of the fetchers or spiders, install the fetchers' dependencies first as shown below.
  453. ### Optional Dependencies
  454. 1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:
  455. ```bash
  456. pip install "scrapling[fetchers]"
  457. scrapling install # normal install
  458. scrapling install --force # force reinstall
  459. ```
  460. This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
  461. Or you can install them from the code instead of running a command like this:
  462. ```python
  463. from scrapling.cli import install
  464. install([], standalone_mode=False) # normal install
  465. install(["--force"], standalone_mode=False) # force reinstall
  466. ```
  467. 2. Extra features:
  468. - Install the MCP server feature:
  469. ```bash
  470. pip install "scrapling[ai]"
  471. ```
  472. - Install dependencies for ([building RAG systems](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html)):
  473. ```bash
  474. pip install "scrapling[rag]"
  475. ```
  476. - Install shell features (Web Scraping shell and the `extract` command):
  477. ```bash
  478. pip install "scrapling[shell]"
  479. ```
  480. - Install everything:
  481. ```bash
  482. pip install "scrapling[all]"
  483. ```
  484. Run `scrapling install` when you need browser fetchers. HTTP requests do not need a browser installation.
  485. ### Docker
  486. You can also install a Docker image with all extras and browsers with the following command from DockerHub:
  487. ```bash
  488. docker pull pyd4vinci/scrapling
  489. ```
  490. Or download it from the GitHub registry:
  491. ```bash
  492. docker pull ghcr.io/d4vinci/scrapling:latest
  493. ```
  494. This image is automatically built and pushed using GitHub Actions and the repository's main branch.
  495. ## Contributing
  496. We welcome contributions! Please read our [contributing guidelines](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md) before getting started.
  497. ## Disclaimer
  498. > [!CAUTION]
  499. > This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect the terms of service of websites and robots.txt files.
  500. ## 🎓 Citations
  501. If you have used our library for research purposes please quote us with the following reference:
  502. ```text
  503. @misc{scrapling,
  504. author = {Karim Shoair},
  505. title = {Scrapling},
  506. year = {2024},
  507. url = {https://github.com/D4Vinci/Scrapling},
  508. note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
  509. }
  510. ```
  511. ## License
  512. This work is licensed under the BSD-3-Clause License.
  513. ## Acknowledgments
  514. This project includes code adapted from:
  515. - Parsel (BSD License)-Used for [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) submodule
  516. ---
  517. <div align="center"><small>Designed & crafted with ❤️ by Karim Shoair.</small></div><br>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.