Skip to content
Back to search

WebReaper

AI-native web scraper. Single binary with a bundled Claude Code skill. MIT-licensed alternative to Firecrawl.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/alex-on-ai--webreaper/badge.svg)](/listings/alex-on-ai--webreaper)

HTML

<a href="/listings/alex-on-ai--webreaper"><img src="/api/listings/alex-on-ai--webreaper/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

**macOS / Linux (Homebrew):**

brew install alex-on-ai/webreaper/webreaper

**Any POSIX shell (install.sh):**

curl -fsSL https://raw.githubusercontent.com/alex-on-ai/WebReaper/master/scripts/install.sh | sh

**.NET library:**

dotnet add package WebReaper

Windows binaries are on the [GitHub Releases page](https://github.com/alex-on-ai/WebReaper/releases/latest); `winget` and `Scoop` are on the v10.1 roadmap.

**Updating:** `brew upgrade webreaper` (Homebrew), or re-run the install.sh line with `--upgrade` appended (`| sh -s -- --upgrade`), or `dotnet add package WebReaper` (library). When a newer release exists, `scrape` / `crawl` / `map` print a one-line upgrade hint on stderr (interactive terminals only, never in a pipe or CI); disable that check with `WEBREAPER_NO_UPDATE_CHECK=1`.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 549 lines · 1 finding marked

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ![logo](https://user-images.githubusercontent.com/6662454/221978697-3f35564a-f442-46e6-9182-f2604a17e1f6.png)
  2. # WebReaper
  3. [![NuGet](https://img.shields.io/nuget/v/WebReaper)](https://www.nuget.org/packages/WebReaper)
  4. [![CI](https://github.com/alex-on-ai/WebReaper/actions/workflows/CI.yml/badge.svg)](https://github.com/alex-on-ai/WebReaper/actions/workflows/CI.yml)
  5. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE.txt)
  6. AI-native web scraper. Single binary with a bundled Claude Code skill.
  7. Built by [HighCraft.io](https://highcraft.io).
  8. ## Install
  9. **macOS / Linux (Homebrew):**
  10. ```bash
  11. brew install alex-on-ai/webreaper/webreaper
  12. ```
  13. **Any POSIX shell (install.sh):**
  14. ```bash
  15. curl -fsSL https://raw.githubusercontent.com/alex-on-ai/WebReaper/master/scripts/install.sh | sh
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 24 on GitHub
  16. ```
  17. **.NET library:**
  18. ```bash
  19. dotnet add package WebReaper
  20. ```
  21. Windows binaries are on the [GitHub Releases page](https://github.com/alex-on-ai/WebReaper/releases/latest); `winget` and `Scoop` are on the v10.1 roadmap.
  22. **Updating:** `brew upgrade webreaper` (Homebrew), or re-run the install.sh line with `--upgrade` appended (`| sh -s -- --upgrade`), or `dotnet add package WebReaper` (library). When a newer release exists, `scrape` / `crawl` / `map` print a one-line upgrade hint on stderr (interactive terminals only, never in a pipe or CI); disable that check with `WEBREAPER_NO_UPDATE_CHECK=1`.
  23. ## 30-second demo
  24. ```bash
  25. $ webreaper scrape https://news.ycombinator.com
  26. # Hacker News
  27. - [Show HN: ...](https://news.ycombinator.com/item?id=...)
  28. - [Ask HN: ...](https://news.ycombinator.com/item?id=...)
  29. ...
  30. $ webreaper init
  31. Wrote WebReaper Agent Skill to .claude/skills/webreaper/SKILL.md
  32. Try it out:
  33. webreaper scrape https://example.com
  34. webreaper map https://example.com
  35. ```
  36. After `webreaper init`, the next Claude Code session picks up the skill and routes scraping intents (*"give me the markdown of X"*, *"what blog posts are on Y"*, *"scrape the top 5 articles"*) to `webreaper` automatically.
  37. ## Table of contents
  38. - [Why WebReaper](#why-webreaper)
  39. - [Quick start](#quick-start)
  40. - [Bot protection that just works](#bot-protection-that-just-works)
  41. - [Power your agent](#power-your-agent)
  42. - [AI features](#ai-features)
  43. - [Use cases](#use-cases)
  44. - [Packages](#packages)
  45. - [Compared to Firecrawl, Crawl4AI, and Crawlee](#compared-to-firecrawl-crawl4ai-and-crawlee)
  46. - [API overview](#api-overview)
  47. - [Architecture and interfaces](docs/architecture.md)
  48. - [Repository structure](#repository-structure)
  49. - [License](#license)
  50. ## Why WebReaper
  51. <table>
  52. <tr>
  53. <td width="33%" valign="top"><strong>🪶 Drop on PATH, run.</strong><br><br>No Docker, no Postgres, no signup. ~12 MB binary.</td>
  54. <td width="33%" valign="top"><strong>🤖 AI-native by composition.</strong><br><br>Markdown by default. Schema extraction, LLM fallback, self-healing selectors, autonomous agent. Stack with <code>.With…()</code>.</td>
  55. <td width="33%" valign="top"><strong>🔌 Bring any LLM.</strong><br><br>OpenAI, Anthropic, Ollama, Azure OpenAI, llamafile, via <code>Microsoft.Extensions.AI</code>.</td>
  56. </tr>
  57. <tr>
  58. <td width="33%" valign="top"><strong>🛡 Bot-checks handled automatically.</strong><br><br>A blocked page climbs HTTP → browser → stealth on its own, per page and host-sticky. Challenge pages are dropped, never returned as data. No flag needed for the browser fallback.</td>
  59. <td width="33%" valign="top"><strong>📡 Distributed when needed.</strong><br><br>Swap scheduler, tracker, sink to Redis, MongoDB, SQLite, Azure Service Bus, Cosmos. Same code.</td>
  60. <td width="33%" valign="top"><strong>📜 MIT, not AGPL.</strong><br><br>Embed in commercial software, fork, modify, redistribute. Firecrawl's AGPL requires open-sourcing your service or paying for a commercial license.</td>
  61. </tr>
  62. </table>
  63. ## Quick start
  64. ### CLI
  65. ```bash
  66. # One page as Markdown
  67. webreaper scrape https://example.com
  68. # Save Markdown to a file
  69. webreaper scrape https://example.com --output page.md
  70. # Discover URLs on a site
  71. webreaper map https://example.com --search /blog/ --max-urls 50
  72. # Crawl a whole site recursively (every on-domain page) to JSON Lines
  73. webreaper crawl https://example.com > pages.jsonl
  74. # Structured fields with a JSON schema (output: JSON; multi-page: JSON Lines)
  75. webreaper scrape https://example.com --schema schema.json
  76. # Schema-free extraction with an LLM (bring your own OpenAI-compatible endpoint)
  77. webreaper scrape https://example.com --prompt "title and author" \
  78. --model gpt-4o-mini --llm-url https://api.openai.com/v1
  79. # Whole site, fields, cheaply: infer a schema once, then extract the rest
  80. webreaper crawl https://example.com --infer "product name and price" \
  81. --model gpt-4o-mini --llm-url https://api.openai.com/v1 --output-dir ./out
  82. # JS-rendered single-page app
  83. webreaper scrape https://example.com --browser
  84. # Bot-protected site: a plain scrape already auto-climbs HTTP -> browser on a
  85. # block; --stealth starts at a stealth backend (--auto-stealth = no prompt, for CI)
  86. webreaper scrape https://example.com --stealth
  87. # Install the Claude Code skill
  88. webreaper init
  89. ```
  90. The CLI is built Native-AOT (ADR-0043), ships as a single binary on every tagged GitHub release across six RIDs (`linux-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`, `win-x64`, `win-arm64`), and is block-aware with automatic browser/stealth escalation (ADR-0083). The macOS binaries are Apple codesigned and notarized (ADR-0071); Homebrew installs run without Gatekeeper warnings on a clean machine.
  91. ### Library
  92. ```csharp
  93. using WebReaper.Builders;
  94. var engine = await ScraperEngineBuilder
  95. .Crawl("https://news.ycombinator.com")
  96. .AsMarkdown()
  97. .WriteToConsole()
  98. .BuildAsync();
  99. await engine.RunAsync();
  100. ```
  101. That is HTTP-only, no extra packages, no schema. For structured fields, swap `AsMarkdown()` for `Extract(schema)`; for JS-rendered pages, `Crawl` for `CrawlWithBrowser` plus a transport satellite ([`WebReaper.Playwright`](#packages) or [`WebReaper.Cdp`](#packages)). The full surface is in the [API overview](#api-overview).
  102. ## Bot protection that just works
  103. Most scrapers make you opt into a browser up front and guess when a site is blocking you. WebReaper detects the block and escalates on its own, one page at a time:
  104. ```
  105. HTTP -> browser (Chromium) -> stealth (CloakBrowser)
  106. (climbs only when a page actually looks blocked)
  107. ```
  108. - **Automatic.** A plain `scrape` or `crawl` starts on a fast HTTP fetch and climbs to a real browser only when a page looks blocked (a challenge status, response header, or body marker). No flag needed for the browser fallback.
  109. - **Per page, host-sticky.** The first confirmed block on a host lifts that host's floor, so the rest of a whole-site crawl starts at the working tier instead of re-paying the failed one. You pay for the climb once, not per page.
  110. - **No garbage in your data.** A page still blocked at the top tier is dropped, never written to your output, and the run exits non-zero so an unattended job knows. Clean data or a clear signal, never a challenge page masquerading as content.
  111. - **You decide on stealth.** `--stealth` starts at the stealth backend; `--auto-stealth` (or `WEBREAPER_AUTO_STEALTH=1`) enables it unattended; `--no-auto-stealth` caps the climb at a vanilla browser. The ~220 MB stealth backend downloads only if you opt in.
  112. Self-hosted, single binary, no cloud round-trip ([ADR-0083](docs/adr/0083-escalating-page-loader.md)).
  113. ## Power your agent
  114. Onboarding an agent from zero? Hand it [docs/AI-ONBOARDING.md](docs/AI-ONBOARDING.md) — install → verify → choose-your-path on one page, no credentials anywhere.
  115. ### Claude Code skill
  116. ```bash
  117. webreaper init
  118. ```
  119. Writes a polished `SKILL.md` to `.claude/skills/webreaper/`. Claude Code loads it on the next session and routes scraping intents to the CLI: *"scrape the top 5 stories on HN and summarize each"*, *"give me the markdown of this article"*, *"this Cloudflare-protected site is blocking me"*. The skill describes when to prefer `webreaper` over the built-in `WebFetch` (artifacts and structured data go through `webreaper`; conversational answers stay on `WebFetch`).
  120. ### MCP server
  121. Two MCP servers expose the same tools (`scrape`, `map`, `extract`, `extract_with_prompt`, `extract_inferred`, `crawl`) for clients that speak the Model Context Protocol:
  122. - **[`WebReaper.Mcp`](https://www.nuget.org/packages/WebReaper.Mcp)** (ADR-0049) speaks **stdio**, for local clients that spawn a process: Cursor, Claude Desktop, Copilot Studio.
  123. - **[`WebReaper.Mcp.AspNetCore`](https://www.nuget.org/packages/WebReaper.Mcp.AspNetCore)** (ADR-0086) speaks **Streamable HTTP**, for clients that connect to a URL. The headline case is **n8n**, whose MCP Client node is URL-only and cannot reach a stdio server. Run the Chromium-baked image, point n8n's MCP Client node at the URL with a bearer token, and call the tools from a workflow:
  124. ```bash
  125. docker run -p 8080:8080 -e WEBREAPER_MCP_TOKEN=your-secret \
  126. ghcr.io/alex-on-ai/webreaper-mcp-http:latest
  127. ```
  128. See the [n8n quickstart](docs/mcp-http-quickstart.md).
  129. Both are thin facades over the library API; the primary agent surface remains the CLI.
  130. ### CLI inside any agent harness
  131. The single binary works inside any shell-spawning agent: LangChain `ShellTool`, OpenAI Assistants code-interpreter, GitHub Actions, internal scripts. Zero runtime to install; one syscall to invoke.
  132. ## AI features
  133. ### 1. LLM-ready Markdown, no schema
  134. ```csharp
  135. await ScraperEngineBuilder
  136. .Crawl("https://example.com")
  137. .AsMarkdown() // ADR-0040 + ADR-0063
  138. .WriteToConsole()
  139. .BuildAsync()
  140. .Result.RunAsync();
  141. ```
  142. The Markdown extractor (`HtmlToMarkdown` primitive, ADR-0063; `MarkdownContentExtractor` adapter, ADR-0040) emits `{title, markdown}` per page. Pass the output straight into a follow-up LLM prompt.
  143. ### 2. Source-gen schemas with compile-time guards
  144. ```csharp
  145. using WebReaper.Extraction.Attributes;
  146. [ScrapeSchema]
  147. public partial class Article
  148. {
  149. [ScrapeField("h1")] public string? Title { get; set; }
  150. [ScrapeField(".views", Type = SchemaFieldType.Integer)] public int Views { get; set; }
  151. [ScrapeField(".tag", IsList = true)] public List<string> Tags { get; set; } = new();
  152. }
  153. // Emitted at compile time, reflection-free, AOT-clean:
  154. // public static Schema Schema { get; }
  155. // public static Article Materialize(JsonObject json)
  156. var engine = await ScraperEngineBuilder
  157. .Crawl("https://example.com/post")
  158. .Extract(Article.Schema)
  159. .Subscribe(p => HandleArticle(Article.Materialize(p.Data)))
  160. .BuildAsync();
  161. ```
  162. The [`WebReaper.Extraction.Generators`](https://www.nuget.org/packages/WebReaper.Extraction.Generators) Roslyn analyzer (ADR-0045) emits the schema and a `Materialize` function. Schema typos are compile errors; the generated path uses no reflection so it's AOT-clean.
  163. ### 3. LLM safety net for deterministic extraction
  164. Three composable patterns where the LLM only fires when the deterministic path can't deliver. Same proposer-validator shape across all three.
  165. ```csharp
  166. using WebReaper.AI;
  167. // (a) Fire LLM only when a field returns empty (ADR-0046)
  168. .WithLlmFallback(chatClient)
  169. // (b) Repair a broken selector once per (Schema, field), cache forever (ADR-0047)
  170. .WithLlmSelfHealing(chatClient)
  171. // (c) No schema at all: infer it from the URL, re-infer on validator failure (ADR-0067 + ADR-0069)
  172. .UseAi(chatClient, AiPolicyMode.Inferred)
  173. ```
  174. Stable pages cost zero LLM calls; broken pages cost one call per (page, field) and cache. Schema inference is one call per site (cached for the run). The `WebReaper.AI` satellite is built on `Microsoft.Extensions.AI` so any `IChatClient` works.
  175. ### 4. Autonomous agent: page selection by goal
  176. ```csharp
  177. using WebReaper.AI;
  178. var result = await LlmAgent.RunAsync(
  179. "https://example.com",
  180. goal: "Find the contact email and phone number for the support team.",
  181. chatClient);
  182. ```
  183. `AgentEngine` (ADR-0051) runs a sequential `decide → persist → execute` loop over a closed-sum `AgentDecision` (`Extract | Follow | Act | Stop`). The brain picks each step from the bounded `AgentState` view; the engine validates (visited-link enforcement, MaxSteps cap), persists (`IAgentRunStore`), and dispatches. Durable resume across process restarts.
  184. ### 5. Semantic page actions
  185. ```csharp
  186. ScraperEngineBuilder
  187. .CrawlWithBrowser(url, actions => actions
  188. .Do(PageAction.SemanticAct("click 'sign in'")) // ADR-0050
  189. .Do(PageAction.WaitForNetworkIdle())
  190. .Build())
  191. .WithLlmActionResolver(chatClient)
  192. // ...
  193. ```
  194. A natural-language `PageAction.SemanticAct(intent)` is one of the ten closed-sum arms (ADR-0050 added it; ADR-0074 added `Fill` / `Press` / `ScrollIntoView` for form interactions). The transport resolves it once via the registered `IActionResolver` (LLM-backed by default), dispatches the concrete arm, and caches the resolution per crawl by intent string. First page pays the LLM, subsequent same-intent pages dispatch the cached arm with no LLM call.
  195. ### Runnable end-to-end demo
  196. [`Examples/WebReaper.AiNativeShowcase`](Examples/WebReaper.AiNativeShowcase/Program.cs) wires every feature in this section:
  197. ```bash
  198. dotnet run --project Examples/WebReaper.AiNativeShowcase -- markdown
  199. dotnet run --project Examples/WebReaper.AiNativeShowcase -- sourcegen
  200. dotnet run --project Examples/WebReaper.AiNativeShowcase -- llm
  201. dotnet run --project Examples/WebReaper.AiNativeShowcase -- router
  202. dotnet run --project Examples/WebReaper.AiNativeShowcase -- changetrack
  203. ```
  204. ## Use cases
  205. - **Build LLM context from blog or docs sites.** `webreaper map` plus `webreaper scrape` per URL, piped into a prompt or a vector DB.
  206. - **Monitor competitor pricing or status pages for changes.** Schedule the CLI with cron or a worker, store records in MongoDB or SQLite, plug in `.WithChangeTracking()` (ADR-0048) so the sink fires only on diff. Hash-based dedup; cron-friendly.
  207. - **Run an autonomous research agent.** `LlmAgent.RunAsync(url, goal, chatClient)` decides which links to follow until the goal is met. Durable resume across restarts.
  208. - **Scrape Cloudflare-protected catalogs.** A plain `scrape` auto-climbs HTTP to a browser on a block; add `--stealth` (or `--auto-stealth` for unattended runs) to escalate to a stealth backend. Blocked pages are dropped, never emitted as challenge-page garbage.
  209. - **Generate clean datasets from semi-structured pages.** `[ScrapeSchema]` POCO plus the source generator; reflection-free, AOT-compiles into a native binary.
  210. - **Embed a scraping primitive in your own app.** `dotnet add package WebReaper`; the public registration seam lets you plug Redis, Cosmos DB, your own sink.
  211. ## Packages
  212. The release ships fifteen packages (one core, fourteen satellites), all versioned in lockstep. The core stays dependency-light and Native-AOT-publishable with zero warnings; satellites bring their own SDK dependencies and quarantine them off the core graph (ADR-0009).
  213. | Package | Add it for | Key builder calls |
  214. |---|---|---|
  215. | **WebReaper** | Core. HTTP crawl and parse, in-memory and file scheduler / visited-link tracker / cookie and config storage, Console / CSV / JSON-Lines sinks, Markdown extractor, schema fold. Dependency-light, Native-AOT-ready, Newtonsoft-free. | `Crawl` `Extract` `AsMarkdown` `Follow` `Paginate` `Sweep` `WriteToJsonFile` `WriteToCsvFile` `WriteToConsole` `Subscribe` |
  216. | **WebReaper.Cdp** | Raw CDP `IPageLoadTransport` (ADR-0052). AOT-clean (no PuppeteerSharp / Playwright dependency); System.Net.WebSockets plus System.Text.Json source-gen. Bedrock for the stealth pattern. | `.WithCdpPageLoader(cdpUrl)` (BYO) or `.WithCdpPageLoader(CdpLaunchOptions)` (launch managed Chromium) |
  217. | **WebReaper.Playwright** | Microsoft.Playwright-backed transport (ADR-0053). Multi-browser (Chromium default; Firefox / WebKit opt-in). All ten `PageAction` arms supported. Use for modern multi-browser needs; pair with `WebReaper.Cdp` for AOT or stealth. | `.WithPlaywrightPageLoader()` |
  218. | **WebReaper.Stealth.CloakBrowser** | First stealth-backend satellite (ADR-0054). Auto-downloads CloakBrowser on first use; composes on `WebReaper.Cdp`. Disposable via the ADR-0058 engine teardown chain. | `.WithCloakBrowser()` |
  219. | **WebReaper.AI** | LLM extraction, LLM action resolver, LLM brain, LLM self-healing, LLM schema inferrer (ADR-0044 / 0050 / 0051 / 0067). Built on `Microsoft.Extensions.AI`; bring your own `IChatClient`. | `.WithLlmFallback` `.WithLlmSelfHealing` `.WithLlmExtractor` `.WithLlmAgentBrain` `.WithLlmActionResolver` `.WithLlmSchemaInferrer` `.UseAi(client)` |
  220. | **WebReaper.AI.Http** | AOT-safe OpenAI-compatible `IChatClient` (ADR-0084): raw `HttpClient` plus System.Text.Json source-gen, no provider SDK. The bring-your-own client the AOT CLI and the MCP servers use for prompt extraction. | `new OpenAiCompatibleChatClient(baseUrl, model, apiKey)` |
  221. | **WebReaper.Extraction.Attributes** | The `[ScrapeSchema]` / `[ScrapeField]` marker types. Standalone, no runtime cost. | `[ScrapeSchema]` `[ScrapeField("selector")]` |
  222. | **WebReaper.Extraction.Generators** | Roslyn source generator that emits `static Schema` plus reflection-free `static Materialize(JsonObject)` (ADR-0045). `DevelopmentDependency=true`; does not propagate at runtime. | compile-time only |
  223. | **WebReaper.Mcp** | MCP server `Exe` exposing scrape / map / extract / extract_with_prompt / extract_inferred / crawl as MCP tools over **stdio** (ADR-0049). Interop adapter for local MCP clients (Cursor, Claude Desktop). | the package _is_ the executable |
  224. | **WebReaper.Mcp.AspNetCore** | MCP server over **Streamable HTTP** (ADR-0086) for URL-based clients like n8n; same tools as `WebReaper.Mcp`, plus bearer-token auth and a `WEBREAPER_CDP_URL` browser sidecar. Also ships as a Chromium-baked container image. | the package _is_ the executable; or `docker run ghcr.io/alex-on-ai/webreaper-mcp-http` |
  225. | **WebReaper.Mongo** | MongoDB result sink and MongoDB-backed config / cookie storage. | `.WriteToMongoDb(...)` `.WithMongoDbConfigStorage(...)` `.WithMongoDbCookieStorage(...)` |
  226. | **WebReaper.Redis** | Redis scheduler, visited-link tracker, result sink, config / cookie storage. | `.WithRedisScheduler(...)` `.TrackVisitedLinksInRedis(...)` `.WriteToRedis(...)` `.WithRedisConfigStorage(...)` `.WithRedisCookieStorage(...)` |
  227. | **WebReaper.AzureServiceBus** | Distributed scheduler over an Azure Service Bus queue. | `.WithAzureServiceBusScheduler(...)` |
  228. | **WebReaper.Cosmos** | Azure Cosmos DB result sink. | `.WriteToCosmosDb(...)` |
  229. | **WebReaper.Sqlite** | Local **durable** scheduler and visited-link tracker on an embedded SQLite store; resume is a query, no position file. Opt-in robust-local tier (no server, unlike Redis). | `.WithSqliteScheduler(...)` `.TrackVisitedLinksInSqlite(...)` |
  230. `WebReaper.Cli` (the AOT single-binary; ADR-0043) is not a NuGet package; it ships as platform binaries on every GitHub release (Native-AOT plus `dotnet tool install` are mutually incompatible on one target). Install via Homebrew or `install.sh`, or build from source.
  231. ## Compared to Firecrawl, Crawl4AI, and Crawlee
  232. | | WebReaper | Firecrawl | Crawl4AI | WebFetch (Claude) |
  233. |---|---|---|---|---|
  234. | **License** | MIT | AGPL-3.0 (plus commercial) | Apache 2.0 | bundled with Claude |
  235. | **Install** | one binary, ~12 MB | Docker + Postgres + Redis (self-host) or hosted | Docker + Python + Playwright | nothing to install |
  236. | **Cost** | free | metered API plus free tier | free | included with Claude |
  237. | **BYO LLM** | any `IChatClient` | no (their model) | yes (LiteLLM) | Claude only |
  238. | **Autonomous agent** | `Agent.RunAsync()` durable, in-process | `/agent` endpoint (cloud only) | code it yourself | not available |
  239. | **Whole-site crawl** | `webreaper crawl` / `.Sweep()`: recursive, on-domain, sitemap-seeded, streams JSON Lines | `crawl` (cloud or self-host) | deep-crawl strategies (code it yourself) | no (single fetch) |
  240. | **Page actions** | 10 declarative arms: `Click`, `Wait`, `Fill`, `Press`, `ScrollToEnd`, `ScrollIntoView`, `WaitForSelector`, `WaitForNetworkIdle`, `EvaluateExpression`, `SemanticAct` (natural-language) | 9 actions: `wait`, `click`, `write`, `press`, `scroll`, `executeJavascript`, plus 3 observation (`screenshot`, `pdf`, `scrape`) | JS hooks; no closed-sum vocabulary | none (single-fetch only) |
  241. | **Bot-protected** | automatic HTTP → browser → stealth climb, per page, host-sticky, self-hosted | cloud yes; self-host degraded (no Fire-engine) | BYO | no |
  242. | **Claude Code skill** | `webreaper init` bundled | community `firecrawl-claude-code-skill` wraps the cloud API | none official | not applicable |
  243. [Crawlee](https://github.com/apify/crawlee) (Apify's Node/Python library) is also worth knowing; it covers similar ground to the WebReaper library API but doesn't ship a binary, a Claude Code skill, or a built-in LLM safety net. Use it if you're already in the Apify ecosystem.
  244. The closest reference is Firecrawl: same AI-native positioning, opposite distribution shape. Firecrawl optimises for the hosted-API flow; WebReaper optimises for the local-binary flow. If you want a managed cloud with someone else's proxies and infra, Firecrawl is the buy. If you want a binary that runs locally with your own LLM key and no metering, WebReaper is the build.
  245. ## API overview
  246. The library is a fluent builder over a small set of seams. For the deep seam-by-seam reference (interfaces, main entities, custom sinks), see [`docs/architecture.md`](docs/architecture.md).
  247. ### Schema extraction
  248. ```csharp
  249. using WebReaper.Builders;
  250. var engine = await ScraperEngineBuilder
  251. .Crawl("https://www.alexpavlov.dev/blog")
  252. .Extract(new()
  253. {
  254. new("title", ".text-3xl.font-bold"),
  255. new("text", ".max-w-max.prose.prose-dark")
  256. })
  257. .Follow("a.text-gray-900.transition")
  258. .WriteToJsonFile("output.json")
  259. .PageCrawlLimit(10)
  260. .WithParallelismDegree(30)
  261. .LogToConsole()
  262. .BuildAsync();
  263. await engine.RunAsync();
  264. ```
  265. Each `new("field", "css-selector")` is a leaf; nest schemas for objects, set `IsList = true` for arrays, set `Attr = "href"` to read an HTML attribute instead of inner text.
  266. ### Collect records in-process
  267. Get the scraped records back in your own process, no file and no custom sink. Pass `Subscribe` (ADR-0038) the `Add` of a thread-safe collection, then read them after the run:
  268. ```csharp
  269. using System.Collections.Concurrent;
  270. using WebReaper.Builders;
  271. using WebReaper.Sinks.Models;
  272. var records = new ConcurrentBag<ParsedData>();
  273. var engine = await ScraperEngineBuilder
  274. .Crawl("https://news.ycombinator.com")
  275. .AsMarkdown()
  276. .Subscribe(records.Add) // called concurrently; collect into a thread-safe type
  277. .BuildAsync();
  278. await engine.RunAsync();
  279. foreach (var record in records)
  280. Console.WriteLine($"{record.Url}: {record.Data["markdown"]?.GetValue<string>()?.Length ?? 0} chars");
  281. ```
  282. The Crawl driver fans out to sinks concurrently, so a `ConcurrentBag` is the right collector, not a bare `List`. For structured fields, read your schema keys out of `record.Data` instead of `"markdown"`.
  283. ### Parsing dynamic pages (SPA)
  284. For JS-rendered pages, swap `Crawl` for `CrawlWithBrowser` and register a browser transport. Two satellites are available.
  285. **[`WebReaper.Playwright`](https://www.nuget.org/packages/WebReaper.Playwright)** is the modern default (ADR-0053): multi-browser, all ten `PageAction` arms.
  286. ```csharp
  287. using WebReaper.Builders;
  288. using WebReaper.Playwright;
  289. await ScraperEngineBuilder
  290. .CrawlWithBrowser("https://example.com")
  291. .Extract(new() { new("title", "h1") })
  292. .WithPlaywrightPageLoader()
  293. .BuildAsync();
  294. ```
  295. **[`WebReaper.Cdp`](https://www.nuget.org/packages/WebReaper.Cdp)** is the AOT-clean alternative (ADR-0052): raw CDP over System.Net.WebSockets, AOT-publishable. Use for AOT consumers or as the base for stealth backends.
  296. ```csharp
  297. using WebReaper.Cdp;
  298. .WithCdpPageLoader(new CdpLaunchOptions { Headless = true })
  299. // or .WithCdpPageLoader(cdpUrl: "http://localhost:9222") for BYO browser
  300. ```
  301. For bot-protected sites, layer **[`WebReaper.Stealth.CloakBrowser`](https://www.nuget.org/packages/WebReaper.Stealth.CloakBrowser)** (ADR-0054) on top of `WebReaper.Cdp`:
  302. ```csharp
  303. using WebReaper.Stealth.CloakBrowser;
  304. .WithCloakBrowser() // auto-downloads CloakBrowser on first use, ~220 MB
  305. ```
  306. For visible-browser debugging, add `.HeadlessMode(false)`.
  307. ### Running JavaScript and page actions
  308. Drive the page as it loads. Pass an actions lambda.
  309. ```csharp
  310. using WebReaper.Builders;
  311. using WebReaper.Playwright;
  312. using WebReaper.Domain.PageActions;
  313. await ScraperEngineBuilder
  314. .CrawlWithBrowser("https://www.reddit.com/r/dotnet/", actions => actions
  315. .ScrollToEnd()
  316. .Build())
  317. .Extract(new() { new("title", "h1") })
  318. .WithPlaywrightPageLoader()
  319. .BuildAsync();
  320. ```
  321. `PageActionBuilder` exposes `Click`, `Wait`, `ScrollToEnd`, `ScrollIntoView`, `WaitForSelector`, `WaitForNetworkIdle`, `EvaluateExpression`, `Fill`, `Press`, `SemanticAct`, `Repeat` / `RepeatWithDelay`, and `Build()`. `Fill(selector, value)` / `Press(key)` / `ScrollIntoView(selector)` (ADR-0074) carry an implicit 30 s auto-wait and use the React-friendly native-setter trick on the CDP transport, so controlled components in React / Vue / Svelte observe the change. `SemanticAct` (ADR-0050) accepts a natural-language intent and resolves it via the registered `IActionResolver` (see [AI features §5](#5-semantic-page-actions)).
  322. ### Persist progress locally
  323. Survive `kill -9` and resume across restarts. Two adapters: file-backed (zero deps, in core) and SQLite-backed (durable, opt-in via `WebReaper.Sqlite`).
  324. ```csharp
  325. using WebReaper.Builders;
  326. using WebReaper.Sqlite;
  327. await ScraperEngineBuilder
  328. .Crawl("https://example.com")
  329. .Extract(new() { new("name", "h1") })
  330. .Follow(".forumlink>a")
  331. .Paginate("a.torTopic", ".pg")
  332. .WriteToJsonFile("result.json")
  333. .WithSqliteScheduler("crawl/state.db") // resume is a query, not a position file
  334. .TrackVisitedLinksInSqlite("crawl/state.db") // the table is the set
  335. .BuildAsync();
  336. ```
  337. Pass `dataCleanupOnStart: true` to any sink, tracker, or scheduler method to wipe its store at start (note: `WriteToJsonFile` defaults this to `true`; the others default to `false`).
  338. ### Authorization
  339. If the site needs cookies, call `SetCookies` and fill the container. You perform the login yourself.
  340. ```csharp
  341. using System.Net;
  342. using WebReaper.Builders;
  343. await ScraperEngineBuilder
  344. .Crawl("https://example.com/protected")
  345. .Extract(new() { new("name", "h1") })
  346. .SetCookies(cookies =>
  347. {
  348. cookies.Add(new Cookie("AuthToken", "123"));
  349. })
  350. .BuildAsync();
  351. ```
  352. ### Distributed and serverless
  353. Swap the scheduler, config storage, and link tracker to Redis or Azure Service Bus; multiple workers or serverless functions share one crawl. [`Examples/WebReaper.AzureFuncs`](Examples/WebReaper.AzureFuncs) shows the serverless shape (two functions: `StartScraping` seeds the work, `WebReaperSpider` is the distributed Crawl driver). [`Examples/WebReaper.DistributedScraperWorkerService`](Examples/WebReaper.DistributedScraperWorkerService) shows the worker-service shape.
  354. `DistributedSpiderBuilder.BuildSpider()` returns a bare `ISpider` without a Crawl seed (ADR-0009 / ADR-0025: "two seams, not one bug" split). The worker's config is persisted separately by the start endpoint.
  355. ### Storage and scheduler backends
  356. Every backend is a swappable seam. In-memory is the default; file-backed lives in core; the rest come from satellites.
  357. | Seam | Core (in-memory default + file) | Satellite options |
  358. |---|---|---|
  359. | Scheduler | in-memory, `WithTextFileScheduler` | `WithSqliteScheduler`, `WithRedisScheduler`, `WithAzureServiceBusScheduler` |
  360. | Visited-link tracker | in-memory, `TrackVisitedLinksInFile` | `TrackVisitedLinksInSqlite`, `TrackVisitedLinksInRedis` |
  361. | Config storage | in-memory, `WithFileConfigStorage` | `WithMongoDbConfigStorage`, `WithRedisConfigStorage` |
  362. | Cookie storage | in-memory, `WithFileCookieStorage` | `WithMongoDbCookieStorage`, `WithRedisCookieStorage` |
  363. | Agent run store | in-memory, file (ADR-0051) | `WithSqliteAgentRunStore`, `WithRedisAgentRunStore`, `WithMongoAgentRunStore`, `WithCosmosAgentRunStore` |
  364. | Result sink | `WriteToConsole`, `WriteToCsvFile`, `WriteToJsonFile` | `WriteToMongoDb`, `WriteToRedis`, `WriteToCosmosDb` |
  365. | Page loader transport | HTTP (default) | `WithPlaywrightPageLoader`, `WithCdpPageLoader`, `WithCloakBrowser` |
  366. Custom sinks, the full interface index, and the main domain entities live in [`docs/architecture.md`](docs/architecture.md).
  367. ## Repository structure
  368. | Project | Description |
  369. |---|---|
  370. | `WebReaper` | The core library (the `WebReaper` NuGet package). |
  371. | `WebReaper.Cdp` | Raw CDP transport satellite (ADR-0052). |
  372. | `WebReaper.Playwright` | Microsoft.Playwright transport satellite (ADR-0053). |
  373. | `WebReaper.Stealth.CloakBrowser` | First stealth-backend satellite (ADR-0054). |
  374. | `WebReaper.AI` | LLM extraction, action resolver, agent brain, self-healing, schema inferrer. |
  375. | `WebReaper.Extraction.Attributes` | `[ScrapeSchema]` / `[ScrapeField]` marker types (ADR-0045). |
  376. | `WebReaper.Extraction.Generators` | Roslyn source generator (ADR-0045). |
  377. | `WebReaper.Mcp` | MCP server satellite (ADR-0049). |
  378. | `WebReaper.Mongo` | MongoDB sink plus config / cookie storage. |
  379. | `WebReaper.Redis` | Redis scheduler, tracker, sink, config / cookie storage. |
  380. | `WebReaper.AzureServiceBus` | Azure Service Bus distributed scheduler. |
  381. | `WebReaper.Cosmos` | Azure Cosmos DB sink. |
  382. | `WebReaper.Sqlite` | Local durable scheduler and visited-link tracker over embedded SQLite. |
  383. | `WebReaper.Cli` | AOT single-binary CLI (ADR-0043). |
  384. | `Examples/WebReaper.ConsoleApplication` | Using WebReaper in a console application. |
  385. | `Examples/WebReaper.AiNativeShowcase` | Runnable demos for every AI feature in this README. |
  386. | `Examples/WebReaper.SchemaInferenceShowcase` | Demos for ADR-0067 / 0068 / 0069 schema inference. |
  387. | `Examples/WebReaper.ScraperWorkerService` | Using WebReaper in a .NET Worker Service. |
  388. | `Examples/WebReaper.DistributedScraperWorkerService` | Distributed crawl across workers sharing crawl state. |
  389. | `Examples/WebReaper.AzureFuncs` | Serverless crawl with Azure Functions plus Azure Service Bus. |
  390. | `Examples/BrownsfashionScraper` | A real-world e-commerce scraper example. |
  391. | `Misc/WebReaper.ProxyProviders` | Example proxy-provider implementations. |
  392. ## License
  393. WebReaper is MIT-licensed (ADR-0017). All NuGet packages plus the `WebReaper.Cli` binary ship under the same terms. Use it commercially, embed it in proprietary software, fork it, modify it, redistribute it; the only ask is that you keep the copyright notice.
  394. Prior to the 10.0.0 wave, WebReaper was GPL-3.0-or-later. The relicense is strictly more permissive: every existing user is unaffected; new users who couldn't embed under GPL now can. Historical contributors are credited in [`CONTRIBUTORS.md`](CONTRIBUTORS.md). See [`docs/adr/0017-relicense-gpl-mit.md`](docs/adr/0017-relicense-gpl-mit.md) for the analysis and contributor consent path.
  395. Contributions are welcome under the same MIT terms; sign-off via DCO ([`CONTRIBUTING.md`](CONTRIBUTING.md)).
  396. ---
  397. Built by [HighCraft.io](https://highcraft.io) — a software and AI engineering team.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.