Skip to content
Back to search

anansi

A self-healing scraper for hostile sites: broken selectors repair themselves, browser rendering kicks in when needed, and a coherent identity layer (Chrome TLS fingerprints, matched personas, vendor-aware Cloudflare/Akamai/DataDome handling) works to slip past bot detection. Ships with an MCP server so any LLM can drive a full crawl by conversation.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mdowis--anansi/badge.svg)](/listings/mdowis--anansi)

HTML

<a href="/listings/mdowis--anansi"><img src="/api/listings/mdowis--anansi/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# Core install
pip install "git+https://github.com/mdowis/anansi"

# For browser-based fetching (Cloudflare bypass, JS rendering):
playwright install chromium

# For TLS-fingerprint mimicry (curl-cffi impersonation):
pip install "anansi-scraper[tls] @ git+https://github.com/mdowis/anansi"

# Optional: the Patchright stealth browser engine (opt-in via ANANSI_BROWSER_ENGINE=patchright):
pip install "anansi-scraper[patchright] @ git+https://github.com/mdowis/anansi"
patchright install chromium

Full install matrix, extras, and Windows notes are in [Getting started](docs/getting-started.md#install).

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 143 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <img src="https://repository-images.githubusercontent.com/1238896536/d711cc76-8358-4a4a-9160-341131498877">
  2. > *The spider that learns.*
  3. [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
  4. [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://python.org)
  5. [![MCP-ready](https://img.shields.io/badge/MCP-ready-8A2BE2.svg)](docs/mcp.md)
  6. **A self-healing web scraper for hostile sites — and it's driveable by any LLM.**
  7. Every scraper starts working. The question is how long before it breaks. Anansi is built on a
  8. different assumption: the web is adversarial and unstable, and your scraper should handle that
  9. without your involvement.
  10. When a site changes its layout, Anansi finds the data anyway and **remembers the fix**. When a
  11. page needs a browser to render, it **switches to one silently**. When bot detection gets in the
  12. way, it presents a **coherent identity** — matched TLS fingerprint, persona, and headers — that
  13. works to slip past detection instead of tripping it. And when you re-crawl, unchanged pages are
  14. skipped before a request is even made. The result is a crawler that survives redesigns, handles
  15. hostile sites, and gets better the longer it runs.
  16. Ships with an **[MCP server](docs/mcp.md)** so any LLM can drive a full crawl through conversation.
  17. ---
  18. ## Highlights
  19. - **Selectors that repair themselves** — CSS selectors carry confidence scores; when one breaks, four healing strategies compete and the winner is persisted. [How it works →](docs/how-it-works.md)
  20. - **A browser only when you need one** — every response is checked for JS shells and silently retried in a stealth Playwright browser, cached per domain. [How it works →](docs/how-it-works.md)
  21. - **A coherent anti-bot identity** — matched TLS/HTTP-2 fingerprint, persona, and headers, with vendor-aware handling of Cloudflare, Akamai, and DataDome, plus an opt-in [Patchright](docs/anti-bot.md#patchright-browser-engine-opt-in) engine for CDP-level stealth. [Anti-bot & identity →](docs/anti-bot.md)
  22. - **Re-crawls that skip the unchanged** — ETag, Last-Modified, content hashing, and sitemap `<lastmod>` filtering skip pages before the request goes out. [Capabilities →](docs/features.md)
  23. - **Data you can trust** — attach a Pydantic `item_schema` and every scraped item is validated and coerced before it hits your database. [Getting started →](docs/getting-started.md)
  24. - **Driveable by any LLM** — a FastMCP server exposes 17 tools so an agent can fetch, extract, crawl, and screenshot through conversation. [MCP server →](docs/mcp.md)
  25. See the full [capability reference](docs/features.md) for everything else — proxy rotation, adaptive rate limiting, URL canonicalization, JS interaction, network capture, and more.
  26. ---
  27. ## Install
  28. ```bash
  29. # Core install
  30. pip install "git+https://github.com/mdowis/anansi"
  31. # For browser-based fetching (Cloudflare bypass, JS rendering):
  32. playwright install chromium
  33. # For TLS-fingerprint mimicry (curl-cffi impersonation):
  34. pip install "anansi-scraper[tls] @ git+https://github.com/mdowis/anansi"
  35. # Optional: the Patchright stealth browser engine (opt-in via ANANSI_BROWSER_ENGINE=patchright):
  36. pip install "anansi-scraper[patchright] @ git+https://github.com/mdowis/anansi"
  37. patchright install chromium
  38. ```
  39. Full install matrix, extras, and Windows notes are in [Getting started](docs/getting-started.md#install).
  40. ---
  41. ## Quickstart
  42. ```python
  43. from pydantic import BaseModel
  44. from anansi import Crawler
  45. from anansi.core import Item, Request, Response
  46. from anansi.spider.spider import Spider
  47. class ProductItem(BaseModel):
  48. title: str
  49. price: float # "49.99" strings are auto-coerced
  50. sku: str | None = None
  51. class ShopSpider(Spider):
  52. name = "shop"
  53. start_urls = ["https://shop.example.com/products"]
  54. item_schema = ProductItem # validate every yielded item against this model
  55. async def parse(self, response: Response):
  56. for link in response.css("a.product-link"):
  57. yield Request(response.urljoin(link["href"]), callback="parse_product")
  58. async def parse_product(self, response: Response):
  59. yield Item({"title": response.css("h1")[0].get_text(), "url": response.url})
  60. # Self-healing, browser auto-upgrade, adaptive rate limiting, and incremental
  61. # re-crawls are all on by default.
  62. crawler = Crawler(ShopSpider, concurrency=10, max_pages=1000)
  63. async for item in crawler.run():
  64. print(item.data)
  65. ```
  66. More examples — structured-data extraction, pausing and resuming, proxies, exporting — in
  67. [Getting started](docs/getting-started.md).
  68. ---
  69. ## Drive it from any LLM
  70. Anansi ships a [FastMCP](docs/mcp.md) server exposing 17 scraping tools over stdio or SSE, so an
  71. LLM agent can run a full crawl through conversation:
  72. ```bash
  73. # Register with Claude Code
  74. claude mcp add anansi -- anansi-mcp
  75. ```
  76. Works with Claude Code, Claude Desktop, Cursor, Windsurf, ChatGPT, LangChain, and the OpenAI
  77. Agents SDK. Setup for each is in the [MCP server guide](docs/mcp.md).
  78. ---
  79. ## Documentation
  80. - **[Getting started](docs/getting-started.md)** — install, first fetch and crawl, the CLI.
  81. - **[Capabilities](docs/features.md)** — the full feature reference.
  82. - **[How it works](docs/how-it-works.md)** — self-healing extraction, browser auto-upgrade, and adaptive rate limiting internals.
  83. - **[Anti-bot & identity](docs/anti-bot.md)** — TLS fingerprint mimicry, coherent personas, crawler impersonation, vendor-aware escalation, sticky sessions, proxy scoring, CAPTCHA, operator controls.
  84. - **[MCP server](docs/mcp.md)** — run the server and drive Anansi from any LLM (all client configs).
  85. - **[Architecture](docs/architecture.md)** — package layout at a glance.
  86. ---
  87. ## Legal / Acceptable Use
  88. Anansi is a powerful scraping tool. **You are solely responsible for how you use it.** Before
  89. scraping any site, ensure you have the right to access and use the data and that you comply with
  90. the site's Terms of Service, its `robots.txt`, applicable rate limits, and all relevant laws
  91. (including computer-misuse statutes such as the CFAA and data-protection law such as GDPR/CCPA).
  92. The anti-bot, TLS-fingerprint-impersonation, and Cloudflare-handling features are intended for
  93. **authorized** testing, research, and scraping of content you have the right to access — not for
  94. circumventing access controls without permission. See [`DISCLAIMER.md`](DISCLAIMER.md) for the
  95. full statement and [Anti-bot & identity](docs/anti-bot.md#operator-controls) for operator
  96. controls.
  97. ---
  98. ## License
  99. Licensed under the Apache License, Version 2.0 — see [`LICENSE`](LICENSE) and
  100. [`NOTICE`](NOTICE). Use of this software is additionally subject to the acceptable-use terms in
  101. [`DISCLAIMER.md`](DISCLAIMER.md).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.