Skip to content
Back to search

frankensurf

MarketplaceWorks with Claude CodeGitHub

Frankensurf: a web reader and search for AI agents.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/yail259--frankensurf/badge.svg)](/listings/yail259--frankensurf)

HTML

<a href="/listings/yail259--frankensurf"><img src="/api/listings/yail259--frankensurf/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Python 3.12+, on Linux, macOS, or Windows through WSL. No account or key needed.

pip install "frankensurf[mcp]"
frankensurf install        # every free tool: browsers, stealth browsers, local search, Crawl4AI
frankensurf read "https://www.walmart.com/search?q=air+fryer" --explain

The read climbs until it gets the real page:

https://www.walmart.com/search?q=air+fryer
  ✗ http                   CAPTCHA                0.4s
  ✗ local                  CAPTCHA                1.5s
  ✓ camoufox               got the page           15.6s

air fryer - Walmart.com
24,192 characters via camoufox, complete (107 results), free

`frankensurf install` works like `playwright install`: name just the tools you want (`frankensurf install search chromium`), add `--with-deps` for the browsers' system libraries on Linux, `--list` to see what's installed where, and `frankensurf uninstall all` to remove it. No Docker needed. Each release is installed from PyPI on fresh Ubuntu and macOS machines and checked.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 15 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "frankensurf",
  3. "owner": {
  4. "name": "yail259",
  5. "url": "https://github.com/yail259"
  6. },
  7. "description": "Frankensurf: a web reader and search for AI agents.",
  8. "plugins": [
  9. {
  10. "name": "frankensurf",
  11. "source": "./plugin",
  12. "description": "Read any web page and search the web: 28 web tools behind one read() call, cheapest first, every page checked."
  13. }
  14. ]
  15. }
README.md · README · 377 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <!-- mcp-name: io.github.yail259/frankensurf -->
  2. <p align="center">
  3. <picture>
  4. <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/yail259/frankensurf/main/site/src/assets/stitched-wave-mint.svg">
  5. <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/yail259/frankensurf/main/site/src/assets/stitched-wave-ink.svg">
  6. <img src="https://raw.githubusercontent.com/yail259/frankensurf/main/site/src/assets/stitched-wave-ink.svg" alt="Frankensurf" width="120">
  7. </picture>
  8. </p>
  9. <h1 align="center">Frankensurf</h1>
  10. <p align="center">
  11. <strong>Access 96% of the web, cheaply and accurately.</strong><br>
  12. A free, self-hosted alternative to Firecrawl: one container, the same API, and every page checked before your agent reads it.
  13. </p>
  14. <p align="center">
  15. <a href="https://pypi.org/project/frankensurf/"><img alt="PyPI" src="https://img.shields.io/pypi/v/frankensurf?style=flat-square&logo=pypi&logoColor=white&label=pypi&color=blue"></a>
  16. <a href="https://pypi.org/project/frankensurf/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/frankensurf?style=flat-square&logo=python&logoColor=white"></a>
  17. <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-green?style=flat-square"></a>
  18. <a href="https://github.com/yail259/frankensurf/stargazers"><img alt="GitHub stars" src="https://img.shields.io/github/stars/yail259/frankensurf?style=flat-square&logo=github&color=lightgrey"></a>
  19. <a href="https://frankensurf.dev/docs/agents/"><img alt="MCP server" src="https://img.shields.io/badge/MCP-server-8A2BE2?style=flat-square"></a>
  20. </p>
  21. <p align="center">
  22. <a href="https://frankensurf.dev"><strong>Website</strong></a> ·
  23. <a href="https://frankensurf.dev/docs/"><strong>Docs</strong></a> ·
  24. <a href="#quick-start"><strong>Quick start</strong></a> ·
  25. <a href="#benchmark"><strong>Benchmark</strong></a> ·
  26. <a href="https://frankensurf.dev/blog/i-just-wanted-a-cheap-film-camera/"><strong>Why I built it</strong></a>
  27. </p>
  28. ---
  29. Most scrapers count a 200 as a success, so your agent ends up reading challenge pages, empty app
  30. shells and menus, and answering anyway. Frankensurf reads each page with the cheapest tool that
  31. works, checks it's the real page, and climbs to stealth browsers (or paid APIs, if you allow them)
  32. only when a site pushes back. It speaks Firecrawl's API, so anything with a Firecrawl URL setting
  33. can point at it.
  34. <table>
  35. <tr>
  36. <td width="33%" valign="top"><strong>Gets past walls</strong><br><sub>41 web tools behind one call: plain HTTP, Jina Reader, browsers, free stealth browsers, then paid APIs only if you allow them.</sub></td>
  37. <td width="33%" valign="top"><strong>Every page checked</strong><br><sub>A challenge page, a login wall or a search page whose results never loaded counts as a failure, with a receipt saying why.</sub></td>
  38. <td width="33%" valign="top"><strong>Drop-in for Firecrawl</strong><br><sub>Same API, one Docker container. Check it on your own URLs first with <code>frankensurf compare</code>.</sub></td>
  39. </tr>
  40. </table>
  41. <p align="center">
  42. <img src="https://frankensurf.dev/blog/head-to-head-set7.png" alt="Share of 150 sites read correctly: Frankensurf 96%, Zyte 86%, Firecrawl 85%, free tools only 88%" width="820">
  43. </p>
  44. On 150 sites picked blind: 96% read right with paid fallbacks against 85% for Firecrawl's cloud on its
  45. own, and 88% with free tools only. On a harder blind set of 148 less-known sites: 88.5% against 70.3%,
  46. and 81.8% free. Against a self-hosted Firecrawl on the same sites, 126 read against
  47. 89 ([details](#against-self-hosted-firecrawl), losses included).
  48. ## Quick start
  49. **As a Firecrawl-compatible server** (one container, free tools inside, no account needed):
  50. ```bash
  51. docker run -d -p 3002:3002 --shm-size=1g -e FRANKENSURF_API_KEY=pick-a-key ghcr.io/yail259/frankensurf
  52. ```
  53. Then point your tool's Firecrawl URL at `http://localhost:3002` with that key
  54. ([which tools, and how](https://frankensurf.dev/docs/firecrawl-compatible/)).
  55. **As a Python package and CLI** (Python 3.12+, on Linux, macOS, or Windows through WSL):
  56. ```bash
  57. pip install "frankensurf[mcp]"
  58. frankensurf install # every free tool: browsers, stealth browsers, local search, Crawl4AI
  59. frankensurf read "https://www.walmart.com/search?q=air+fryer" --explain
  60. ```
  61. The read climbs until it gets the real page:
  62. ```text
  63. https://www.walmart.com/search?q=air+fryer
  64. ✗ http CAPTCHA 0.4s
  65. ✗ local CAPTCHA 1.5s
  66. ✓ camoufox got the page 15.6s
  67. air fryer - Walmart.com
  68. 24,192 characters via camoufox, complete (107 results), free
  69. ```
  70. `frankensurf install` works like `playwright install`: name just the tools you want
  71. (`frankensurf install search chromium`), add `--with-deps` for the browsers' system libraries on
  72. Linux, `--list` to see what's installed where, and `frankensurf uninstall all` to remove it. No
  73. Docker needed. Each release is installed from PyPI on fresh Ubuntu and macOS machines and checked.
  74. ## Use it
  75. **From an agent**, as an MCP server ([setup for each client](https://frankensurf.dev/docs/agents/)):
  76. ```bash
  77. claude mcp add frankensurf -- frankensurf-mcp
  78. ```
  79. **From Python:**
  80. ```python
  81. import frankensurf as fs
  82. page = fs.read("https://example.com/pricing")
  83. if page.ok:
  84. print(page.title, page.text[:500])
  85. else:
  86. print("couldn't read it:", page.failure) # {"code": "CAPTCHA", "message": ...}
  87. pages = fs.batch(urls, links=True) # paced per site, in order
  88. hits = fs.search("playwright python tutorial", recency="month")
  89. page = await fs.aread(url) # aread, abatch, asearch inside async code
  90. ```
  91. For a long-running service, keep one `Runtime` open: `async with Runtime() as web: await web.read(url)`.
  92. **Map and crawl:** `fs.map(url)` lists a site's URLs; `fs.crawl(url, limit=50)` reads a section of it as markdown.
  93. **Agent mode** (`pip install "frankensurf[agent]"`, any LiteLLM model): `fs.ask("question")` searches, reads and
  94. answers with sources, and says what it couldn't read; `fs.extract(urls, schema=...)` and `fs.summarize(url)`.
  95. [Agent mode](https://frankensurf.dev/docs/agent-mode/).
  96. **In Docker**, one install for your whole network: `docker run -p 3002:3002 -e FRANKENSURF_API_KEY=... ghcr.io/yail259/frankensurf`
  97. ([Docker](https://frankensurf.dev/docs/docker/)).
  98. **From any tool that takes a Firecrawl URL** (GPT Researcher, Hermes Agent, OpenClaw, Open WebUI, Dify, LibreChat, n8n, LangChain, LlamaIndex and others):
  99. `frankensurf serve`, then point it at `http://127.0.0.1:3002` ([setup](https://frankensurf.dev/docs/firecrawl-compatible/)).
  100. **From the command line:**
  101. ```bash
  102. frankensurf read https://example.com --markdown
  103. frankensurf search playwright python tutorial --recency week
  104. frankensurf watch https://news.ycombinator.com/ --link-pattern 'item\?id=\d+'
  105. ```
  106. **In Claude Code as a plugin** (the MCP server plus the skill that teaches Claude to use it):
  107. ```
  108. /plugin marketplace add yail259/frankensurf
  109. /plugin install frankensurf@frankensurf
  110. ```
  111. **From LangChain, the OpenAI Agents SDK or CrewAI** (`pip install "frankensurf[langchain]"`, and so on):
  112. ```python
  113. from frankensurf.agent_tools import langchain_tools # or openai_agents_tools, crewai_tools
  114. tools = langchain_tools() # frankensurf_read, frankensurf_search
  115. ```
  116. You get raw page data, not answers: text or markdown, JSON-LD, the page's own JSON, images, and
  117. the receipt. Your agent decides what the page means. If it isn't what it wanted, it asks again with
  118. `retry_of=<trace_id>` and Frankensurf skips every tool it already tried.
  119. ## Benchmark
  120. **Cost per 1,000 correct pages:** Frankensurf's free tools $0 in fees, about $0.01 of a $5/month
  121. server (87–88% right); with paid fallbacks $0.22 (96%); Firecrawl on its own $0.71 on its Scale plan,
  122. $0.98 on Standard (85%).
  123. [How that's worked out](https://frankensurf.dev/docs/more/cost/).
  124. `scripts/toolbench.py` runs each tool on its own, then Frankensurf, on the same pages with the same
  125. content checks. The newest set: 148 less-known sites in 15 categories, picked before any was read,
  126. every tool on its own, paid ones included (28 October 2026, rows and corrections in
  127. [`benchmarks/2026-10-28-heldout9/`](benchmarks/2026-10-28-heldout9/)).
  128. | Tool | Sites read | Median | Cost per 1k |
  129. | --- | ---: | ---: | ---: |
  130. | Plain HTTP only | 41.9% | 0.9 s | free |
  131. | Jina Reader only | 68.2% | 3.3 s | free |
  132. | Firecrawl only | 70.3% | 4.3 s | $5.22 |
  133. | Zyte only | 70.3% | 9.5 s | not reported |
  134. | Frankensurf, free tools only | 81.8% | 4.8 s | $0 |
  135. | **Frankensurf, with paid fallbacks** | **88.5%** | **4.9 s** | **$0.96** |
  136. - Frankensurf read 28 sites Firecrawl missed; Firecrawl read 1 it missed.
  137. - Every tool's reads put together come to 84.5%; Frankensurf reads more than that, through retries
  138. and rescues.
  139. - Its slowest tenth takes 33.0 s against Firecrawl's 10.9 s.
  140. - This set is harder than the last: every tool read 10 to 20 points less. A rerun the same evening
  141. gave 87.2% and 79.7%.
  142. The set before (150 sites, 11 October 2026, rows in [`benchmarks/2026-10-11/`](benchmarks/2026-10-11/)):
  143. | Tool | Sites read | Median | Cost per 1k |
  144. | --- | ---: | ---: | ---: |
  145. | Plain HTTP only | 48.7% | 1.2 s | free |
  146. | Camoufox only | 68.7% | 10.9 s | free |
  147. | Jina Reader only | 73.3% | 1.5 s | free |
  148. | Firecrawl only | 84.7% | 4.6 s | $4.28 |
  149. | Zyte only | 86.0% | 10.7 s | not reported |
  150. | Frankensurf, free tools only | 88.0% | 3.6 s | $0 |
  151. | **Frankensurf, with paid fallbacks** | **96.0%** | **2.9 s** | **$0.98** |
  152. - Frankensurf read 17 sites Firecrawl missed; Firecrawl read none it missed.
  153. - With no paid tools at all it is level with the best paid tool on its own (88.0% against 86.0% and
  154. 84.7%; at 150 sites that gap is within the noise).
  155. - Its slowest reads are slower: the slowest tenth takes 25.7 s against Firecrawl's 10.6 s, when a
  156. read climbs several tools.
  157. - Not a one-off: 95.3% vs 85.3% on the set the day before; 89.5% vs 80.3% and 88.5% vs 74.1% on
  158. earlier fresh sets.
  159. <details>
  160. <summary>Every tool on the 11 October set</summary>
  161. | Tool | Sites read | Median | Cost per 1k |
  162. | --- | ---: | ---: | ---: |
  163. | Patchright only | 38.7% | 4.1 s | free |
  164. | Headless Chromium only | 40.7% | 2.7 s | free |
  165. | Plain HTTP only | 48.7% | 1.2 s | free |
  166. | Camoufox only | 68.7% | 10.9 s | free |
  167. | Scrapling only | 68.7% | 8.5 s | free |
  168. | Jina Reader only | 73.3% | 1.5 s | free |
  169. | Firecrawl only | 84.7% | 4.6 s | $4.28 |
  170. | Zyte only | 86.0% | 10.7 s | not reported |
  171. | Frankensurf, free tools only | 88.0% | 3.6 s | $0 |
  172. | Frankensurf, with paid fallbacks | 96.0% | 2.9 s | $0.98 |
  173. Medians are over pages read; cost is per 1,000 pages read right. ZenRows and Scrapfly are left out
  174. (their plans ran out of credits). From a fresh cloud VM, whose data-centre address sites wall more
  175. often: free tools 82.0%, lightning mode 84.0% at a 1.2 s median.
  176. See [Benchmarks](https://frankensurf.dev/docs/more/benchmarks/).
  177. </details>
  178. ### Against self-hosted Firecrawl
  179. Firecrawl is open source too, so the fair question for anyone running their own is how the Firecrawl
  180. you'd self-host does. Its docs say self-hosted instances don't get Fire-engine, the anti-bot layer of
  181. its cloud. Same 150 sites, 23 October 2026, both servers in Docker on one machine, each site read by
  182. both with the first reader alternating site by site:
  183. | | Self-hosted Firecrawl 2.11.604 | Frankensurf 0.52.0 (free tools) |
  184. | --- | ---: | ---: |
  185. | Pages read right | 89 of 150 | **126 of 150** |
  186. | Wrong page passed on as the page | 18 | **5** |
  187. | Nothing back | 43 | 19 |
  188. | Memory while reading | about 4.2 GB, six containers | **1.4 GB, one container** |
  189. | Median, pages both read | **4.2 s** | 4.8 s |
  190. | Slowest tenth, pages both read | **9.8 s** | 17.8 s |
  191. | Time to give up on a page (median) | **1.5 s** | 16.5 s |
  192. - Frankensurf read 38 sites the self-hosted Firecrawl didn't; Firecrawl read 1 it didn't.
  193. - The slower tail and slower give-ups are the cost of the extra pages: where self-hosted Firecrawl
  194. tries once and stops, Frankensurf climbs to stealth browsers and checks each page before handing it
  195. back.
  196. - Rows, scripts and setup: [`benchmarks/2026-10-23-self-hosted-firecrawl/`](benchmarks/2026-10-23-self-hosted-firecrawl/).
  197. Run it on your own pages with `frankensurf compare urls.txt --against http://localhost:3002`.
  198. ## Search
  199. `search` uses free sources first and checks every result: on topic, the right kind of page, inside
  200. the date window. Exa, Brave, Tavily, Serper and other paid sources join only when you allow paid
  201. tools. Judged blind (results pooled and shuffled, which tool found them hidden), useful results in
  202. the top five:
  203. | Blind set 7 (120 queries) | Useful results (of 5) | Cost per 1k |
  204. | --- | ---: | ---: |
  205. | Frankensurf, paid allowed | 4.67 | $9.38 |
  206. | Exa only | 4.66 | $9.50 |
  207. | Frankensurf, free (with the local search engine) | 3.83 | $0 |
  208. Free, it beats Firecrawl's search: better on 35 queries, worse on 5. With paid allowed it is level
  209. with Exa. The local search engine (`frankensurf install search`, SearXNG without Docker) makes the
  210. free tier much stronger. v0.41 improved free search again: on a fresh blind set of 120
  211. queries it beat v0.40 on 41 and lost on 12 (3.42 against 2.71 useful results of 5,
  212. [`benchmarks/2026-10-14-search-v041/`](benchmarks/2026-10-14-search-v041/README.md)). Details in
  213. [`benchmarks/2026-10-13-search/`](benchmarks/2026-10-13-search/README.md).
  214. ## What else is in the box
  215. <details>
  216. <summary><strong>Reading</strong>: quality grades, main content, racing slow reads, PDFs</summary>
  217. - Every read carries `receipt.quality`: `good`, `partial` or `poor`, with flags such as `paywall`,
  218. `menus` or `placeholder`, so an agent knows when a page didn't really come through.
  219. - `--schema product.json` (MCP `schema`) fills a JSON Schema from the page's JSON-LD, microdata,
  220. meta tags and own JSON into `extracted`, and climbs to a stronger tool while a required field is
  221. missing ([Structured fields](https://frankensurf.dev/docs/read/structured/)).
  222. - `--main` (MCP `main_content=True`) adds `main_text`: the article without menus, footers and
  223. banners (best with `pip install "frankensurf[main]"`).
  224. - A read with no page after 3 seconds starts two more free tools beside it and keeps the first
  225. complete page (`--lightning` races from the start).
  226. - A URL that answers "not found" is looked up with the site's own search; `receipt.rescued` says so.
  227. - Still walled? Allow a real Chrome on your machine (`--real-browser`, with its own profile, never
  228. yours) or a second connection (`FRANKENSURF_EGRESS`).
  229. </details>
  230. <details>
  231. <summary><strong>Search</strong>: verticals, boolean queries, local engine</summary>
  232. - `--site`, `--exclude-domain`, `--recency` and `--region` on every source, and a `--vertical`
  233. (news, reference, discussions, qa, code, papers, books) backed by official free APIs.
  234. - Monitoring-style queries (`"Acme" OR "Acme Corp" (acquisition OR layoffs)`) are judged by the
  235. name they're about, and a weak answer is retried once with just the name.
  236. - `--mode merge` asks every source at once and fuses the results.
  237. - `frankensurf search-engine start|stop|status` manages the local SearXNG; search restarts it after
  238. a reboot.
  239. </details>
  240. <details>
  241. <summary><strong>Agents</strong>: skill, profiles, handoff, site modules</summary>
  242. - A bundled skill teaches agents the tools: the MCP server sends it as its instructions, and
  243. `frankensurf skill install` adds it to `~/.claude/skills`.
  244. - **Profiles:** log in once and any tool can use the session; the model never sees cookies or keys.
  245. - **Human handoff:** for a CAPTCHA or 2FA only a person can clear, it opens the page for you, then
  246. carries on.
  247. - **Site modules:** save what your agent learns about a site as data. `frankensurf module discover
  248. URL --query "desk lamp" --save` finds the site's search, reads the results and saves a module;
  249. after that, searches on that site come back as structured `items`.
  250. - **Shared site knowledge:** `frankensurf share sync` starts from which tool reads which site, as the
  251. community learned it; `share export` writes yours to review and contribute (opt in both ways).
  252. - `frankensurf watch-sites URL… --since DATE` lists new pages from the feeds and sitemaps sites
  253. publish.
  254. </details>
  255. <details>
  256. <summary><strong>Paid tools and budgets</strong></summary>
  257. Everything works with no keys. Paid tools are opt-in and only come in once a site has walled a free
  258. tool, cheapest likely success first. Set their keys (for example `FIRECRAWL_API_KEY`) and pass
  259. `--paid`; `--max-cost USD` caps a read. From Python: `allow_paid_fallbacks` and `max_cost_usd`.
  260. See [Paid tools and budgets](https://frankensurf.dev/docs/blocked/paid-tools/).
  261. </details>
  262. ## How it works
  263. Every way to fetch a page is a rung: plain HTTP and markdown negotiation, browsers, stealth
  264. browsers and signed requests, hosted browsers, paid APIs, and finally a person clearing the wall in
  265. a visible browser. A read starts at the cheapest rung and climbs only when a page pushes back. Every
  266. tool is a plugin, so new ones slot in as another rung.
  267. <p align="center">
  268. <img src="https://frankensurf.dev/blog/fallback-chain.png" alt="One read of Crunchbase: plain HTTP, headless Chrome, Steel and Camoufox were blocked; Firecrawl got through" width="760">
  269. </p>
  270. See [How escalation works](https://frankensurf.dev/docs/blocked/escalation/) and the
  271. [full list of integrations](https://frankensurf.dev/docs/integrations/).
  272. ## Known walls
  273. It's an alpha, and some sites still win:
  274. - **Need the paid tools:** in a free-only spot check, Etsy, Home Depot, Yelp, Crunchbase and G2
  275. blocked every free tool.
  276. - **Beat everything so far:** Shopee, Temu and Idealista.
  277. - **Social sites without a login:** LinkedIn, X, YouTube, Threads, Bluesky and Reddit read fine.
  278. Instagram, TikTok, Facebook and Pinterest show only the public shell; the posts need a login.
  279. - **Logged-in pages** (your feeds, your orders) need a
  280. [profile](https://frankensurf.dev/docs/read/profiles/).
  281. - **A slow tail:** most reads take a few seconds, but one in ten climbs several tools and takes 25 s
  282. or more.
  283. Found a site it can't read?
  284. [Report it](https://github.com/yail259/frankensurf/issues/new?template=site-it-cant-read.yml); every
  285. report becomes a test case.
  286. ## Develop
  287. ```bash
  288. python3 -m venv .venv
  289. .venv/bin/pip install -e '.[test,mcp]'
  290. .venv/bin/frankensurf install
  291. PYTHONPATH=.:src .venv/bin/pytest -q
  292. ```
  293. The product design is in `SPEC.rst` and contributor rules in `AGENTS.md`. New tools are plugins,
  294. never branches in Core: see [Writing a plugin](https://frankensurf.dev/docs/more/plugins/) and the
  295. example in `examples/frankensurf-plugin-example`. The website and docs live in `site/`.
  296. ## Licence
  297. Apache-2.0. See [`LICENSE`](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.