anansi
A self-healing scraper for hostile sites: broken selectors repair themselves, browser rendering kicks in when needed, and a coherent identity layer (Chrome TLS fingerprints, matched personas, vendor-aware Cloudflare/Akamai/DataDome handling) works to slip past bot detection. Ships with an MCP server so any LLM can drive a full crawl by conversation.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/mdowis--anansi)HTML
<a href="/listings/mdowis--anansi"><img src="/api/listings/mdowis--anansi/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
# Core install pip install "git+https://github.com/mdowis/anansi" # For browser-based fetching (Cloudflare bypass, JS rendering): playwright install chromium # For TLS-fingerprint mimicry (curl-cffi impersonation): pip install "anansi-scraper[tls] @ git+https://github.com/mdowis/anansi" # Optional: the Patchright stealth browser engine (opt-in via ANANSI_BROWSER_ENGINE=patchright): pip install "anansi-scraper[patchright] @ git+https://github.com/mdowis/anansi" patchright install chromium
Full install matrix, extras, and Windows notes are in [Getting started](docs/getting-started.md#install).
---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 143 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <img src="https://repository-images.githubusercontent.com/1238896536/d711cc76-8358-4a4a-9160-341131498877">
- > *The spider that learns.*
- [](LICENSE)
- [](https://python.org)
- [](docs/mcp.md)
- **A self-healing web scraper for hostile sites — and it's driveable by any LLM.**
- Every scraper starts working. The question is how long before it breaks. Anansi is built on a
- different assumption: the web is adversarial and unstable, and your scraper should handle that
- without your involvement.
- When a site changes its layout, Anansi finds the data anyway and **remembers the fix**. When a
- page needs a browser to render, it **switches to one silently**. When bot detection gets in the
- way, it presents a **coherent identity** — matched TLS fingerprint, persona, and headers — that
- works to slip past detection instead of tripping it. And when you re-crawl, unchanged pages are
- skipped before a request is even made. The result is a crawler that survives redesigns, handles
- hostile sites, and gets better the longer it runs.
- Ships with an **[MCP server](docs/mcp.md)** so any LLM can drive a full crawl through conversation.
- ---
- ## Highlights
- - **Selectors that repair themselves** — CSS selectors carry confidence scores; when one breaks, four healing strategies compete and the winner is persisted. [How it works →](docs/how-it-works.md)
- - **A browser only when you need one** — every response is checked for JS shells and silently retried in a stealth Playwright browser, cached per domain. [How it works →](docs/how-it-works.md)
- - **A coherent anti-bot identity** — matched TLS/HTTP-2 fingerprint, persona, and headers, with vendor-aware handling of Cloudflare, Akamai, and DataDome, plus an opt-in [Patchright](docs/anti-bot.md#patchright-browser-engine-opt-in) engine for CDP-level stealth. [Anti-bot & identity →](docs/anti-bot.md)
- - **Re-crawls that skip the unchanged** — ETag, Last-Modified, content hashing, and sitemap `<lastmod>` filtering skip pages before the request goes out. [Capabilities →](docs/features.md)
- - **Data you can trust** — attach a Pydantic `item_schema` and every scraped item is validated and coerced before it hits your database. [Getting started →](docs/getting-started.md)
- - **Driveable by any LLM** — a FastMCP server exposes 17 tools so an agent can fetch, extract, crawl, and screenshot through conversation. [MCP server →](docs/mcp.md)
- See the full [capability reference](docs/features.md) for everything else — proxy rotation, adaptive rate limiting, URL canonicalization, JS interaction, network capture, and more.
- ---
- ## Install
- ```bash
- # Core install
- pip install "git+https://github.com/mdowis/anansi"
- # For browser-based fetching (Cloudflare bypass, JS rendering):
- playwright install chromium
- # For TLS-fingerprint mimicry (curl-cffi impersonation):
- pip install "anansi-scraper[tls] @ git+https://github.com/mdowis/anansi"
- # Optional: the Patchright stealth browser engine (opt-in via ANANSI_BROWSER_ENGINE=patchright):
- pip install "anansi-scraper[patchright] @ git+https://github.com/mdowis/anansi"
- patchright install chromium
- ```
- Full install matrix, extras, and Windows notes are in [Getting started](docs/getting-started.md#install).
- ---
- ## Quickstart
- ```python
- from pydantic import BaseModel
- from anansi import Crawler
- from anansi.core import Item, Request, Response
- from anansi.spider.spider import Spider
- class ProductItem(BaseModel):
- title: str
- price: float # "49.99" strings are auto-coerced
- sku: str | None = None
- class ShopSpider(Spider):
- name = "shop"
- start_urls = ["https://shop.example.com/products"]
- item_schema = ProductItem # validate every yielded item against this model
- async def parse(self, response: Response):
- for link in response.css("a.product-link"):
- yield Request(response.urljoin(link["href"]), callback="parse_product")
- async def parse_product(self, response: Response):
- yield Item({"title": response.css("h1")[0].get_text(), "url": response.url})
- # Self-healing, browser auto-upgrade, adaptive rate limiting, and incremental
- # re-crawls are all on by default.
- crawler = Crawler(ShopSpider, concurrency=10, max_pages=1000)
- async for item in crawler.run():
- print(item.data)
- ```
- More examples — structured-data extraction, pausing and resuming, proxies, exporting — in
- [Getting started](docs/getting-started.md).
- ---
- ## Drive it from any LLM
- Anansi ships a [FastMCP](docs/mcp.md) server exposing 17 scraping tools over stdio or SSE, so an
- LLM agent can run a full crawl through conversation:
- ```bash
- # Register with Claude Code
- claude mcp add anansi -- anansi-mcp
- ```
- Works with Claude Code, Claude Desktop, Cursor, Windsurf, ChatGPT, LangChain, and the OpenAI
- Agents SDK. Setup for each is in the [MCP server guide](docs/mcp.md).
- ---
- ## Documentation
- - **[Getting started](docs/getting-started.md)** — install, first fetch and crawl, the CLI.
- - **[Capabilities](docs/features.md)** — the full feature reference.
- - **[How it works](docs/how-it-works.md)** — self-healing extraction, browser auto-upgrade, and adaptive rate limiting internals.
- - **[Anti-bot & identity](docs/anti-bot.md)** — TLS fingerprint mimicry, coherent personas, crawler impersonation, vendor-aware escalation, sticky sessions, proxy scoring, CAPTCHA, operator controls.
- - **[MCP server](docs/mcp.md)** — run the server and drive Anansi from any LLM (all client configs).
- - **[Architecture](docs/architecture.md)** — package layout at a glance.
- ---
- ## Legal / Acceptable Use
- Anansi is a powerful scraping tool. **You are solely responsible for how you use it.** Before
- scraping any site, ensure you have the right to access and use the data and that you comply with
- the site's Terms of Service, its `robots.txt`, applicable rate limits, and all relevant laws
- (including computer-misuse statutes such as the CFAA and data-protection law such as GDPR/CCPA).
- The anti-bot, TLS-fingerprint-impersonation, and Cloudflare-handling features are intended for
- **authorized** testing, research, and scraping of content you have the right to access — not for
- circumventing access controls without permission. See [`DISCLAIMER.md`](DISCLAIMER.md) for the
- full statement and [Anti-bot & identity](docs/anti-bot.md#operator-controls) for operator
- controls.
- ---
- ## License
- Licensed under the Apache License, Version 2.0 — see [`LICENSE`](LICENSE) and
- [`NOTICE`](NOTICE). Use of this software is additionally subject to the acceptable-use terms in
- [`DISCLAIMER.md`](DISCLAIMER.md).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.