Skip to content
Back to search

PyScrappy

Adaptive Python web scraping toolkit + MCP server for AI agents. Self-healing selectors that survive site changes, TLS-fingerprint stealth to bypass anti-bot filters, CSS/XPath parsing, and 24 built-in scrapers, clean, structured, LLM-ready data from any URL.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mldsveda--pyscrappy/badge.svg)](/listings/mldsveda--pyscrappy)

HTML

<a href="/listings/mldsveda--pyscrappy"><img src="/api/listings/mldsveda--pyscrappy/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pip install pyscrappy

**Optional extras:**

# Browser support (for JS-rendered pages)
pip install 'pyscrappy[browser]'
playwright install chromium

# DataFrame support
pip install 'pyscrappy[dataframe]'

# MCP server (use PyScrappy's scrapers as AI-agent tools)
pip install 'pyscrappy[mcp]'

# Stealth (TLS-fingerprint impersonation to bypass anti-bot filters)
pip install 'pyscrappy[stealth]'

# Parquet / Excel export (ScrapeResult.to_parquet() / .to_excel())
pip install 'pyscrappy[parquet]'
pip install 'pyscrappy[excel]'

# Everything
pip install 'pyscrappy[all]'

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 635 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src="https://raw.githubusercontent.com/mldsveda/PyScrappy/main/public/logo.png" alt="PyScrappy" width="480">
  3. </p>
  4. <h2 align="center">Adaptive Python web scraping toolkit (self-healing, stealth)<br>+ MCP server for AI agents</h2>
  5. [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
  6. [![PyPI Latest Release](https://img.shields.io/pypi/v/PyScrappy.svg)](https://pypi.org/project/PyScrappy/)
  7. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/mldsveda/PyScrappy/blob/main/LICENSE)
  8. [![Downloads](https://static.pepy.tech/badge/pyscrappy)](https://pepy.tech/project/pyscrappy)
  9. [![Glama quality](https://glama.ai/mcp/servers/mldsveda/PyScrappy/badges/score.svg)](https://glama.ai/mcp/servers/mldsveda/PyScrappy)
  10. [![Documentation](https://img.shields.io/badge/docs-pyscrappy-117866.svg)](https://pyscrappy.vercel.app)
  11. [![MCP Toplist](https://mcptoplist.com/badge/io.github.mldsveda%2Fpyscrappy.svg)](https://mcptoplist.com/server/io.github.mldsveda%2Fpyscrappy)
  12. <!-- mcp-name: io.github.mldsveda/pyscrappy -->
  13. PyScrappy is an AI-native web scraping toolkit that turns websites into structured, LLM-ready data. Use it as a Python library or expose it as an MCP server for AI agents.
  14. 📖 **Documentation:** [pyscrappy.vercel.app](https://pyscrappy.vercel.app)
  15. ## Key features
  16. - **Generic scraper** — give it any URL, get back structured text, links, images, tables, and metadata
  17. - **LLM-ready output** — `.to_markdown()` turns any result into clean Markdown; also `.to_json()` and `.to_dataframe()`
  18. - **MCP server** — expose the scrapers as tools for AI agents (Claude, Cursor, local LLMs, …)
  19. - **JS rendering** — optional Playwright backend for JavaScript-heavy sites
  20. - **Custom selectors** — pass CSS selectors to extract exactly what you need
  21. - **Chainable `Selector`** — navigate HTML directly with CSS/XPath, `find_all`, `find_by_text`, and `find_similar` (Scrapy/BeautifulSoup-style)
  22. - **Adaptive (self-healing) selectors** — remember an element and relocate it by similarity when a site changes its markup, so scrapers don't silently break
  23. - **Concurrent scraping** — `scrape_many` / `scrape_all` run scrapes in parallel
  24. - **Sitemap crawling** — enumerate and scrape a whole site from its `sitemap.xml` (index + gzip aware)
  25. - **Proxy & scraping-API support** — route through a proxy or ScraperAPI/ScrapeOps for blocked sites
  26. - **TLS-fingerprint impersonation** — `impersonate="chrome"` gets past anti-bot filters that block plain clients (optional `curl_cffi` backend)
  27. - **Command-line extract** — `pyscrappy extract <url> out.md` scrapes a URL straight to a file, no code
  28. - **Retry & rate-limiting** — built-in exponential backoff and per-domain rate limiting
  29. - **Type-safe** — full type hints, `py.typed` marker
  30. - **20+ built-in scrapers** — Wikipedia, IMDB, stocks, news, GitHub, Amazon/IKEA, YouTube, and [more](#built-in-scrapers)
  31. ## Installation
  32. ```sh
  33. pip install pyscrappy
  34. ```
  35. **Optional extras:**
  36. ```sh
  37. # Browser support (for JS-rendered pages)
  38. pip install 'pyscrappy[browser]'
  39. playwright install chromium
  40. # DataFrame support
  41. pip install 'pyscrappy[dataframe]'
  42. # MCP server (use PyScrappy's scrapers as AI-agent tools)
  43. pip install 'pyscrappy[mcp]'
  44. # Stealth (TLS-fingerprint impersonation to bypass anti-bot filters)
  45. pip install 'pyscrappy[stealth]'
  46. # Parquet / Excel export (ScrapeResult.to_parquet() / .to_excel())
  47. pip install 'pyscrappy[parquet]'
  48. pip install 'pyscrappy[excel]'
  49. # Everything
  50. pip install 'pyscrappy[all]'
  51. ```
  52. ## For AI agents
  53. PyScrappy ships an [MCP server](#mcp-server-use-pyscrappy-from-an-ai-agent) that
  54. exposes its scrapers as tools, so an agent (Claude, Cursor, an OpenAI agent, a
  55. local LLM) can pull structured web data from any URL and hand it straight to the
  56. model:
  57. ```text
  58. AI agent ──MCP tool call──▶ PyScrappy ──fetch + extract──▶ Any website
  59. ▲ │
  60. └────────────── clean Markdown / JSON ◀───────────────────────┘
  61. ```
  62. ```sh
  63. pip install 'pyscrappy[mcp]'
  64. claude mcp add pyscrappy pyscrappy-mcp
  65. ```
  66. Then just ask: *"use pyscrappy to summarize the latest headlines from bbc.com."*
  67. See [MCP server](#mcp-server-use-pyscrappy-from-an-ai-agent) for the full setup
  68. and tool list.
  69. ### Local models (Ollama), no MCP host needed
  70. Ollama can't talk MCP on its own, so normally you'd run a host (Goose, Cline, …)
  71. in between. PyScrappy skips that with a built-in agent that talks to Ollama
  72. directly and lets a local model call the scrapers as tools:
  73. ```sh
  74. pip install 'pyscrappy[mcp]' # needs Python 3.10+
  75. pyscrappy chat --model qwen2.5 "what's the current AAPL quote?"
  76. ```
  77. It exposes the same 22 tools as the MCP server. The only requirement is a model
  78. that supports **tool calling** (Llama 3.1, Qwen 2.5, Mistral, …); how well it
  79. *picks* the right tool is up to the model. Point it at a remote Ollama with
  80. `--host`, and pass `-v` to see each tool call.
  81. ## MCP server (use PyScrappy from an AI agent)
  82. PyScrappy ships an optional [Model Context Protocol](https://modelcontextprotocol.io)
  83. server, so an AI agent (e.g. Claude) can call PyScrappy's scrapers as tools and
  84. get structured web data back.
  85. <a href="https://glama.ai/mcp/servers/mldsveda/PyScrappy">
  86. <img width="380" height="200" src="https://glama.ai/mcp/servers/mldsveda/PyScrappy/badges/card.svg" alt="PyScrappy MCP server" />
  87. </a>
  88. ```sh
  89. pip install 'pyscrappy[mcp]'
  90. ```
  91. The MCP extra installs the standalone `fastmcp` package and requires Python 3.10
  92. or newer. On Python 3.9 the core scraping library still works, but the MCP server
  93. is unavailable.
  94. This installs the `pyscrappy-mcp` command. It uses stdio by default for local MCP
  95. clients; Streamable HTTP and legacy SSE are available for remote deployments:
  96. ```sh
  97. pyscrappy-mcp # stdio (default)
  98. pyscrappy-mcp --http # Streamable HTTP
  99. pyscrappy-mcp --sse # legacy SSE
  100. ```
  101. You can also run the stdio server with `python -m pyscrappy.mcp`.
  102. ### Register with Claude Code
  103. ```sh
  104. claude mcp add pyscrappy pyscrappy-mcp
  105. ```
  106. ### Register with Claude Desktop
  107. Add to your `claude_desktop_config.json` and restart the app:
  108. ```json
  109. {
  110. "mcpServers": {
  111. "pyscrappy": {
  112. "command": "pyscrappy-mcp"
  113. }
  114. }
  115. }
  116. ```
  117. > **Tip:** Claude Desktop does not inherit your shell `PATH`. If `pyscrappy-mcp`
  118. > is not found, use the absolute path to the command (e.g. the one printed by
  119. > `which pyscrappy-mcp`).
  120. ### Available tools
  121. The server exposes **20+ tools**. The most common ones are **`scrape_url`** (any
  122. URL → text, links, images, tables, metadata), **`scrape_wikipedia`**,
  123. **`scrape_stock`**, **`scrape_news`**, and **`search_github`** — plus many more
  124. covering image/YouTube/LinkedIn/Hacker News/book search, weather, crypto,
  125. currency, dictionary, Amazon/Newegg/IKEA/SoundCloud, IMDB, and Zomato/Uber Eats.
  126. To see the full, live list, ask the agent to call the **`list_available_scrapers`**
  127. tool, or from a shell:
  128. ```sh
  129. python -c "from pyscrappy import list_scrapers; print(', '.join(sorted(list_scrapers())))"
  130. ```
  131. The `lookup_movie` tool needs a free [OMDb](https://www.omdbapi.com/apikey.aspx) API
  132. key. Pass it to the server through your MCP client config, e.g. for Claude Desktop:
  133. ```json
  134. {
  135. "mcpServers": {
  136. "pyscrappy": {
  137. "command": "pyscrappy-mcp",
  138. "env": { "OMDB_API_KEY": "your-key" }
  139. }
  140. }
  141. }
  142. ```
  143. Once registered, just ask the agent naturally, e.g. *"use pyscrappy to get the
  144. latest headlines from bbc.co.uk and the AAPL stock quote."*
  145. ## Built-in scrapers
  146. PyScrappy ships **24 built-in scrapers**, and every one that works without a
  147. proxy is also exposed as an [MCP tool](#mcp-server-use-pyscrappy-from-an-ai-agent).
  148. A few of them:
  149. - **`GenericScraper`** — scrape any URL with auto-extraction (text, links, images, tables, metadata)
  150. - **Data / research** — **`WikipediaScraper`**, **`StockScraper`** (Yahoo Finance), **`NewsScraper`** (RSS/Atom), **`GitHubScraper`**, **`HackerNewsScraper`**, plus weather, crypto, currency, dictionary, image, LinkedIn-jobs, and book search
  151. - **E-commerce** — **`AmazonScraper`**, `NeweggScraper`, `IKEAScraper`
  152. - **Social / media / food** — **`YouTubeScraper`**, SoundCloud, Zomato, Uber Eats (Instagram / Twitter / Spotify also ship, but are blocked and need a proxy)
  153. …and many more. To see the full, live list:
  154. ```sh
  155. python -c "from pyscrappy import list_scrapers; print(', '.join(sorted(list_scrapers())))"
  156. ```
  157. **`IMDBScraper`** (`lookup_movie`) is the one exception that needs a key — a free
  158. [OMDb](https://www.omdbapi.com/apikey.aspx) `OMDB_API_KEY` (see the
  159. [MCP config](#available-tools) above for how to pass it).
  160. ## Plugins
  161. PyScrappy is extensible: you can add your own scrapers, and third parties can
  162. ship them as standalone `pyscrappy-<name>` packages. A registered scraper works
  163. everywhere a built-in does, including the MCP server and the `pyscrappy chat`
  164. agent, with no change to PyScrappy core.
  165. **In your own code** — register with the decorator:
  166. ```python
  167. from pyscrappy import BaseScraper, register_scraper, get_scraper
  168. from pyscrappy.core.models import ScrapeResult, ScrapeMetadata
  169. @register_scraper("reddit")
  170. class RedditScraper(BaseScraper):
  171. def scrape(self, subreddit: str, **kwargs) -> ScrapeResult:
  172. data = self.fetch_and_parse(f"https://old.reddit.com/r/{subreddit}/.json")
  173. # ... build a list of dicts ...
  174. return ScrapeResult(data=[...], metadata=ScrapeMetadata(scraper="reddit"))
  175. get_scraper("reddit")().scrape(subreddit="python")
  176. ```
  177. **As a distributable package** — advertise an entry point in your
  178. `pyproject.toml`, and PyScrappy discovers it once your package is installed:
  179. ```toml
  180. [project.entry-points."pyscrappy.scrapers"]
  181. reddit = "pyscrappy_reddit:RedditScraper"
  182. ```
  183. After `pip install pyscrappy-reddit`, the scraper shows up in
  184. `list_scrapers()`, and an AI agent can call it via the `scrape_with` MCP tool —
  185. no core change required.
  186. **First-class MCP tools (optional).** Add an `mcp_tools` mapping and your scraper
  187. becomes a dedicated, typed MCP tool instead of only being reachable through the
  188. generic `scrape_with` — its schema is derived from the method signature, so
  189. agents get proper named arguments:
  190. ```python
  191. @register_scraper("reddit")
  192. class RedditScraper(BaseScraper):
  193. mcp_tools = {"search_reddit": "scrape"} # tool name -> method
  194. def scrape(self, subreddit: str, sort: str = "hot") -> ScrapeResult:
  195. ...
  196. ```
  197. See the [plugin template](plugin-template/) for a complete, copyable starting
  198. point, and the [plugin guide](https://pyscrappy.vercel.app/docs/plugins/) for
  199. the full walkthrough.
  200. ## Quick start
  201. ### Scrape any URL → clean, LLM-ready Markdown
  202. ```python
  203. from pyscrappy import scrape
  204. result = scrape("https://en.wikipedia.org/wiki/Web_scraping")
  205. print(result.to_markdown()) # feed straight to an LLM
  206. # ...or result.to_json() / result.to_dataframe()
  207. # Write to a file — format inferred from the extension:
  208. result.save("out.json") # .json .csv .md .ndjson .yaml .parquet .xlsx
  209. # (.parquet needs pyscrappy[parquet]; .xlsx needs pyscrappy[excel])
  210. ```
  211. Prefer raw fields? Every result is a `ScrapeResult` with `.data` (a list of
  212. dicts):
  213. ```python
  214. print(result.data[0]["metadata"]["title"])
  215. print(result.data[0]["text"]["word_count"])
  216. ```
  217. ### Custom CSS selectors
  218. ```python
  219. from pyscrappy import GenericScraper
  220. with GenericScraper() as gs:
  221. result = gs.scrape(
  222. url="https://news.ycombinator.com",
  223. selectors={"title": ".titleline a", "score": ".score"},
  224. )
  225. for item in result.data:
  226. print(item["title"], item.get("score", ""))
  227. ```
  228. ### Navigate HTML with `Selector`
  229. When you want to traverse markup directly (Scrapy/BeautifulSoup-style) rather than
  230. get back structured dicts, use `Selector`:
  231. ```python
  232. from pyscrappy import Selector
  233. page = Selector(html) # or navigate any HTML string
  234. page.css(".title::text").getall() # CSS with ::text / ::attr(name)
  235. page.xpath("//a/@href").getall() # XPath (elements, text(), @attr)
  236. page.find_all("h2", class_="title") # BeautifulSoup-style search
  237. page.find_by_text("Add to cart", tag="button") # search by text content
  238. first = page.css(".product")[0]
  239. first.css(".price::text").get() # chainable
  240. first.find_similar() # sibling elements shaped like this one
  241. ```
  242. `css()` / `xpath()` return a `SelectorList` with `.get()` / `.getall()` / `.text()`.
  243. `find_similar()` locates elements with the same tag and overlapping classes, handy
  244. for pulling every card/row once you've found one.
  245. ### Adaptive (self-healing) selectors
  246. A hard-coded CSS selector silently breaks the day a site changes its markup.
  247. Adaptive selectors survive that: save a fingerprint of the element the first time,
  248. and if the selector later matches nothing, relocate it by structural and textual
  249. similarity instead of returning empty.
  250. ```python
  251. from pyscrappy import Selector
  252. # First run: match normally and remember this element under an id.
  253. page = Selector(html_v1, url="https://shop.example.com")
  254. price = page.css(".price", auto_save=True, adaptive_id="price").get()
  255. # Later, after a redesign renamed ".price" — heal instead of breaking.
  256. # `expect` is an optional contract: the relocated element must satisfy it,
  257. # so a good structural score can't smuggle in the wrong field.
  258. page = Selector(html_v2, url="https://shop.example.com")
  259. result = page.css(
  260. ".price",
  261. adaptive=True,
  262. adaptive_id="price",
  263. expect=lambda s: s.text().startswith("$"),
  264. )
  265. print(result.get(), "→ confidence:", result.adaptive_confidence)
  266. ```
  267. How the relocation decides — and where it's stronger than a naive similarity match:
  268. - **Weighted signals, not a flat average.** A stable `id` / `data-*` hook counts
  269. far more than a sibling-tag list, so weak signals can't outvote strong ones.
  270. - **Anchor-relative.** It remembers the nearest stable ancestor (an id'd / `data-*`
  271. container) and depth, so it survives layout reshuffles that move absolute positions.
  272. - **Volatility-aware text.** Prices, dates, and counts are down-weighted, so
  273. healing stays reliable on exactly the fields that change most between scrapes.
  274. - **Confidence-scored.** `SelectorList.adaptive_confidence` (0-100) tells you how
  275. sure the relocation was; `threshold=` sets the minimum to accept.
  276. - **Contract-enforced (opt-in).** Pass `expect=<callable>` to require the healed
  277. element to satisfy an invariant (e.g. "text looks like a price"). A heal that
  278. clears the threshold but fails the contract is rejected, so structural
  279. similarity alone never redefines what a field means.
  280. A heal is a change to what a selector resolves to, so **every accepted heal is
  281. recorded**. The store keeps an append-only audit log (`adaptive.heal.ndjson`
  282. beside the fingerprint store) with the confidence, the runner-up gap, and the
  283. before/after fingerprint, readable via `store.heal_log()` — so drift stays
  284. observable instead of being silently absorbed. For an at-a-glance summary,
  285. `store.heal_report()` aggregates the log into one row per selector (heal count,
  286. latest/lowest/average confidence, when it last healed), sorted most-healed first
  287. — so the selectors that have drifted the most, and the shakiest relocations
  288. (lowest confidence), surface at the top for a human to review.
  289. Fingerprints persist in a small JSON store (`~/.pyscrappy/adaptive.json` by
  290. default, or `$PYSCRAPPY_HOME`), namespaced by site so the same `adaptive_id` on
  291. two sites never collides. Adaptive is entirely opt-in: without `adaptive=True`, a
  292. broken selector still just returns empty, exactly as before.
  293. ### Site-specific scrapers
  294. Every built-in scraper follows the same pattern — instantiate, `scrape(...)`,
  295. read `result.data` (or `.to_dataframe()` / `.to_markdown()`):
  296. ```python
  297. from pyscrappy import WikipediaScraper
  298. with WikipediaScraper() as ws:
  299. result = ws.scrape(query="Python (programming language)", mode="summary")
  300. print(result.data[0]["text"])
  301. ```
  302. Each scraper has its own arguments (Wikipedia, stocks, IMDB, news, YouTube,
  303. Amazon/Newegg/IKEA, Uber Eats, and more — see the [full list](#built-in-scrapers)).
  304. For per-scraper arguments and examples, see the
  305. [documentation](https://pyscrappy.vercel.app/docs/scrapers/).
  306. ### From the command line
  307. Scrape a URL straight to a file without writing any code — the output format is
  308. inferred from the file extension:
  309. ```sh
  310. pyscrappy extract https://example.com out.md # clean Markdown
  311. pyscrappy extract https://example.com out.json # structured JSON
  312. pyscrappy extract https://example.com out.txt # extracted page text
  313. pyscrappy extract https://example.com out.html # raw fetched HTML
  314. # Narrow to elements matching a CSS selector, or render JS first:
  315. pyscrappy extract https://example.com items.txt --css-selector ".product"
  316. pyscrappy extract https://example.com page.md --render-js
  317. ```
  318. ## Configuration
  319. ```python
  320. from pyscrappy import ScraperConfig, GenericScraper
  321. config = ScraperConfig(
  322. timeout=20.0, # request timeout in seconds
  323. max_retries=3, # retry failed requests
  324. retry_jitter=True, # spread exponential retries to avoid lockstep traffic
  325. rate_limit=2.0, # seconds between requests per domain
  326. proxy="http://...", # proxy URL, or a list to rotate through
  327. scraper_api=None, # route via a scraping-API service (see below)
  328. headless=True, # browser runs headless
  329. render_js="auto", # auto-detect if JS rendering is needed
  330. cache_ttl=0, # response cache TTL in seconds (0 = disabled)
  331. cache_dir=None, # also persist the cache to disk (survives restarts)
  332. cache_dir_max_size=512, # max live entries kept on disk before oldest are pruned
  333. impersonate=None, # e.g. "chrome" to spoof a browser's TLS fingerprint (see below)
  334. )
  335. with GenericScraper(config) as gs:
  336. result = gs.scrape(url="https://example.com")
  337. ```
  338. ### Proxies and blocked sites
  339. Some sites (e.g. eBay, Instagram, Twitter/X, Spotify) block direct automated
  340. requests. PyScrappy supports two ways to get through them.
  341. **A proxy** (or a rotating list) — applies to both the HTTP and browser backends:
  342. ```python
  343. from pyscrappy import ScraperConfig, AmazonScraper
  344. # Single proxy
  345. config = ScraperConfig(proxy="http://user:pass@host:port")
  346. # Rotating list (one picked per request)
  347. config = ScraperConfig(proxy=["http://p1:8080", "http://p2:8080"])
  348. ```
  349. **A scraping-API service** (ScraperAPI, ScrapeOps, ScrapingBee) — routes requests
  350. through the service, which handles proxies and anti-bot challenges for you:
  351. ```python
  352. config = ScraperConfig(scraper_api={
  353. "provider": "scraperapi", # or "scrapeops", "scrapingbee"
  354. "api_key": "YOUR_KEY",
  355. "render_js": True, # optional
  356. })
  357. # Now any scraper works through the service, unchanged:
  358. with AmazonScraper(config) as scraper:
  359. result = scraper.scrape(query="laptop")
  360. ```
  361. This is the reliable way to use the scrapers marked "needs proxy" above.
  362. **TLS-fingerprint impersonation** — many anti-bot systems block a plain HTTP
  363. client by its TLS/JA3 fingerprint before serving any content. Set `impersonate`
  364. to mimic a real browser's fingerprint and get past that class of block without a
  365. headless browser:
  366. ```python
  367. from pyscrappy import ScraperConfig, GenericScraper
  368. # needs the optional extra: pip install 'pyscrappy[stealth]'
  369. config = ScraperConfig(impersonate="chrome") # or "chrome124", "safari", "firefox"
  370. with GenericScraper(config) as gs:
  371. result = gs.scrape("https://example.com")
  372. ```
  373. Impersonation works on **both the sync and async paths** (async uses
  374. `curl_cffi`'s `AsyncSession`), so you can combine stealth with high-throughput
  375. async scraping. All the usual retry, rate-limiting, caching, and robots handling
  376. still apply.
  377. ```python
  378. import asyncio
  379. from pyscrappy import scrape_async, ScraperConfig
  380. async def main():
  381. cfg = ScraperConfig(impersonate="chrome")
  382. return await scrape_async("https://example.com", config=cfg)
  383. asyncio.run(main())
  384. ```
  385. ### Concurrent scraping
  386. Scraping is I/O-bound, so running several scrapes at once parallelizes the
  387. network waits. `scrape_many` runs one scraper over many inputs; `scrape_all`
  388. runs a mix of scrapers together. Both preserve input order.
  389. ```python
  390. from pyscrappy import scrape_many, scrape_all, AmazonScraper, WikipediaScraper, NewsScraper
  391. # One scraper, many queries, concurrently:
  392. results = scrape_many(AmazonScraper, [{"query": "laptop"}, {"query": "phone"}])
  393. # Different scrapers at once:
  394. results = scrape_all([
  395. lambda: WikipediaScraper().scrape(query="Python"),
  396. lambda: NewsScraper().scrape(feed_url="https://rss.nytimes.com/services/xml/rss/nyt/World.xml"),
  397. ])
  398. ```
  399. ### Sitemap crawling
  400. Pagination follows next-page links; a **sitemap** enumerates a whole site's URLs
  401. directly. `GenericScraper` can read `/sitemap.xml` (discovered from `robots.txt`
  402. `Sitemap:` directives, or the conventional path), follow a `<sitemapindex>` into
  403. its child sitemaps, and scrape every listed page.
  404. ```python
  405. from pyscrappy import GenericScraper
  406. with GenericScraper() as gs:
  407. # Just enumerate the URLs:
  408. urls = gs.sitemap_urls("https://example.com") # -> list[str]
  409. # Or fetch + extract each, concurrently, into one result:
  410. result = gs.scrape_sitemap("https://example.com", max_urls=100)
  411. print(len(result.data), "pages scraped")
  412. ```
  413. Handles `<urlset>` leaves and `<sitemapindex>` files (recursing one level),
  414. gzip-compressed sitemaps (`.xml.gz`), and de-duplicates URLs. Fetches go through
  415. the usual rate-limiting, caching, proxy, and stealth machinery, and the fan-out
  416. reuses `scrape_all`. `max_urls` caps the crawl (a sitemap can list tens of
  417. thousands of URLs, so it's required for `scrape_sitemap`).
  418. ### Response caching
  419. Set `cache_ttl` to a positive number of seconds to cache successful GET
  420. responses. Repeated requests for the same URL (and query params) within the TTL
  421. are served from cache, skipping both the network and the rate limiter. Caching
  422. is **disabled by default** (`cache_ttl=0`).
  423. ```python
  424. from pyscrappy import WikipediaScraper
  425. from pyscrappy import ScraperConfig
  426. config = ScraperConfig(cache_ttl=300) # cache for 5 minutes
  427. with WikipediaScraper(config) as ws:
  428. ws.scrape(query="Python") # fetched over the network
  429. ws.scrape(query="Python") # served from cache
  430. ```
  431. The cache is in memory and shared across scraper instances in the same process
  432. (so it also speeds up repeated calls through the MCP server), and is cleared
  433. when the process exits. Call `HttpClient.clear_cache()` to empty it manually.
  434. It is **LRU-bounded**: at most `cache_max_size` live entries (default `512`),
  435. with the least-recently-used entry evicted once the cap is reached. So a
  436. long-running process (e.g. the MCP server) that fetches many distinct URLs stays
  437. bounded rather than growing until restart. Raise or lower the cap as needed:
  438. ```python
  439. config = ScraperConfig(cache_ttl=300, cache_max_size=2000)
  440. ```
  441. **Persistent (on-disk) cache.** Set `cache_dir` to also persist responses to
  442. disk, so cache hits survive across process restarts and separate runs — useful
  443. for re-running a scrape or a CLI job without re-fetching. The in-memory cache
  444. still fronts it for speed; a disk hit is promoted back into memory.
  445. ```python
  446. config = ScraperConfig(cache_ttl=3600, cache_dir="~/.cache/pyscrappy")
  447. ```
  448. The on-disk cache is bounded too: each write prunes expired entries and trims the
  449. oldest past `cache_dir_max_size` (default `512`), so a `cache_dir` doesn't grow
  450. one file per distinct URL forever.
  451. `clear_cache()` empties the in-memory cache; the on-disk cache persists by
  452. design — delete its `cache_dir` to clear it.
  453. ### Observability hooks
  454. For long crawls, pass lightweight callbacks to watch requests live (progress
  455. bars, metrics) without turning on logging:
  456. ```python
  457. config = ScraperConfig(
  458. on_request=lambda url: print("GET", url), # before a network fetch
  459. on_retry=lambda url, attempt, delay, err: print("retry", attempt, url),
  460. on_cache_hit=lambda url: print("cached", url), # served from cache
  461. )
  462. ```
  463. - `on_request(url)` fires once before a URL is fetched (not on a cache hit).
  464. - `on_retry(url, attempt, delay, error)` fires before each backoff sleep.
  465. - `on_cache_hit(url)` fires when a request is served from cache.
  466. All three are best-effort: a callback that raises is logged at debug and never
  467. breaks the scrape. They fire on both the sync and async paths.
  468. ## Dependencies
  469. **Required:** `httpx`, `beautifulsoup4`, `lxml`
  470. **Optional:** `playwright` (JS rendering), `pandas` (DataFrames), `fastmcp`
  471. (MCP server, Python 3.10+)
  472. ## License
  473. [MIT](https://github.com/mldsveda/PyScrappy/blob/main/LICENSE)
  474. ## Contributing
  475. All contributions welcome. See [Issues](https://github.com/mldsveda/PyScrappy/issues).
  476. **This package is for educational and research purposes.**

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.