Skip to content
Back to search

jevcrawl

MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/rayenmlayeh--jevcrawl/badge.svg)](/listings/rayenmlayeh--jevcrawl)

HTML

<a href="/listings/rayenmlayeh--jevcrawl"><img src="/api/listings/rayenmlayeh--jevcrawl/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

You need a Jev API key from [console.typesafe.ai](https://console.typesafe.ai/) and [uv](https://docs.astral.sh/uv/).

**Claude Code**

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl

**Claude Desktop, Cursor and other MCP clients** (`mcpServers` in the client's config file)

{
  "mcpServers": {
    "jevcrawl": {
      "command": "uvx",
      "args": ["jevcrawl"],
      "env": { "TYPESAFE_API_KEY": "<your key>" }
    }
  }
}

Then ask your agent something like *"Use jevcrawl to find which Python version added the walrus operator, starting from docs.python.org"*. Check the setup with `uvx jevcrawl doctor`. Agents that load skills can also use [`skills/jevcrawl/SKILL.md`](skills/jevcrawl/SKILL.md).

**Pages that need JavaScript** (optional). jevcrawl reads plain HTML. To also render pages that are empty without JavaScript, add the `js` extra and download Chromium once:

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx --from "jevcrawl[js]" jevcrawl
uvx --from "jevcrawl[js]" playwright install chromium

(In `mcpServers`, use `"args": ["--from", "jevcrawl[js]", "jevcrawl"]`.) Only pages detected as JavaScript shells are opened in the browser, at most five per call, and they are listed in `stats.js_rendered_urls`. Everything else is still fetched as plain HTML.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 255 lines · 1 finding marked

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. <picture>
  3. <source media="(prefers-color-scheme: dark)" srcset="assets/logo-dark.svg">
  4. <img alt="jevcrawl" src="assets/logo-light.svg" width="400">
  5. </picture>
  6. ### The web crawler for AI agents that finds the answer, not just pages.
  7. A small decision model picks every link and every passage, so your agent's LLM reads
  8. **only what answers its question** at **a few cents per crawl**.
  9. [![PyPI](https://img.shields.io/pypi/v/jevcrawl?color=6D5DFC)](https://pypi.org/project/jevcrawl/)
  10. [![CI](https://github.com/RayenMlayeh/jevcrawl/actions/workflows/ci.yml/badge.svg)](https://github.com/RayenMlayeh/jevcrawl/actions/workflows/ci.yml)
  11. [![License](https://img.shields.io/badge/license-Apache--2.0-6D5DFC)](LICENSE)
  12. [![Python](https://img.shields.io/badge/python-3.11%2B-3776AB?logo=python&logoColor=white)](pyproject.toml)
  13. [![MCP server](https://img.shields.io/badge/MCP-server-22C3EE)](https://modelcontextprotocol.io)
  14. [![Benchmark](https://img.shields.io/badge/WebWalkerQA-40%25_vs_30%25_for_Claude-6D5DFC)](#benchmarks)
  15. [![Cost](https://img.shields.io/badge/cost-%240.025%2Ftask-2EA043)](#cost-and-privacy)
  16. [Quick start](#quick-start) · [Benchmarks](#benchmarks) · [Tools](#tools) ·
  17. [How it works](#how-it-works) · [Configuration](#configuration)
  18. </div>
  19. <br>
  20. <p align="center">
  21. <img alt="Demo: jevcrawl crawls docs.python.org for 'What does the walrus operator := do and which Python version added it?' and returns the What's New In Python 3.8 passage. 6 pages, 80,034 tokens reduced to 1,873, $0.0069, 8.3 seconds." src="assets/demo.svg" width="100%">
  22. </p>
  23. <picture>
  24. <source media="(prefers-color-scheme: dark)" srcset="assets/scoreboard-dark.svg">
  25. <img alt="On 50 WebWalkerQA tasks jevcrawl answered 40% correctly and found the answer page 54% of the time for $0.025 per task, against 30%, 33% and $0.194 when Claude picks every link and 8%, 11% and $0.228 for a breadth-first crawl." src="assets/scoreboard-light.svg" width="100%">
  26. </picture>
  27. ## Why jevcrawl
  28. Agents that browse a site have two bad options. They can **read everything** a crawler fetches,
  29. which buries the answer in navigation and boilerplate and burns tokens. Or they can **let the LLM
  30. pick each link**, which is slow and costs dollars per question. jevcrawl does neither:
  31. - 🎯 **Finds the answer page more often.** 54% of tasks vs 33% when Claude picks the links, and
  32. 2x as often as [Crawl4AI](#jevcrawl-vs-crawl4ai)'s goal-directed crawl.
  33. - 💸 **About 8x cheaper per task, 10x per correct answer.** The crawl decisions cost about a cent.
  34. - ⚡ **2x faster than an LLM-driven crawl.** Four pages at a time, one decision request per page.
  35. - 📜 **Verbatim, with sources.** Every passage is copied from the page with its URL, heading path
  36. and character offsets. The decision model only scores, so it cannot invent text.
  37. - 🛡️ **Safe against prompt injection by design.** Page text is only ever scored, never followed.
  38. - 🔌 **Works everywhere MCP does:** Claude Code, Claude Desktop, Cursor, or the CLI.
  39. ## Quick start
  40. You need a Jev API key from [console.typesafe.ai](https://console.typesafe.ai/) and
  41. [uv](https://docs.astral.sh/uv/).
  42. **Claude Code**
  43. ```bash
  44. claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl
  45. ```
  46. **Claude Desktop, Cursor and other MCP clients** (`mcpServers` in the client's config file)
  47. ```json
  48. {
  49. "mcpServers": {
  50. "jevcrawl": {
  51. "command": "uvx",
  52. "args": ["jevcrawl"],
  53. "env": { "TYPESAFE_API_KEY": "<your key>" }
  54. }
  55. }
  56. }
  57. ```
  58. Then ask your agent something like *"Use jevcrawl to find which Python version added the walrus
  59. operator, starting from docs.python.org"*. Check the setup with
  60. `uvx jevcrawl doctor`. Agents that load skills
  61. can also use [`skills/jevcrawl/SKILL.md`](skills/jevcrawl/SKILL.md).
  62. **Pages that need JavaScript** (optional). jevcrawl reads plain HTML. To also render pages that
  63. are empty without JavaScript, add the `js` extra and download Chromium once:
  64. ```bash
  65. claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx --from "jevcrawl[js]" jevcrawl
  66. uvx --from "jevcrawl[js]" playwright install chromium
  67. ```
  68. (In `mcpServers`, use `"args": ["--from", "jevcrawl[js]", "jevcrawl"]`.) Only pages detected as
  69. JavaScript shells are opened in the browser, at most five per call, and they are listed in
  70. `stats.js_rendered_urls`. Everything else is still fetched as plain HTML.
  71. ## Benchmarks
  72. Every system starts on a site's home page with the same 30-page budget and the same question.
  73. What each one collects goes to the **same answering LLM**, and the **same judge** grades the
  74. answer, so the crawler is the only thing that changes.
  75. <picture>
  76. <source media="(prefers-color-scheme: dark)" srcset="assets/cost-dark.svg">
  77. <img alt="Cost of one correct answer: jevcrawl $0.06, Claude picking every link $0.65, breadth-first crawl $2.85." src="assets/cost-light.svg" width="100%">
  78. </picture>
  79. ### jevcrawl vs Crawl4AI
  80. [Crawl4AI](https://github.com/unclecode/crawl4ai) is one of the most popular open-source crawlers
  81. for LLMs. Its `BestFirstCrawlingStrategy` with a keyword scorer is its way to crawl toward a question
  82. without an LLM, so we compared the two on whether they fetch the page that holds the answer.
  83. <picture>
  84. <source media="(prefers-color-scheme: dark)" srcset="assets/crawl4ai-dark.svg">
  85. <img alt="On 20 tasks jevcrawl found an answer page 10 times and Crawl4AI 5 times; every answer page 5 vs 2; median crawl time 29 s vs 43 s; sites that failed to load 0 vs 3. jevcrawl found 6 answer pages Crawl4AI missed, Crawl4AI found 1 jevcrawl missed." src="assets/crawl4ai-light.svg" width="100%">
  86. </picture>
  87. ### Where the numbers come from
  88. | | |
  89. |---|---|
  90. | **Tasks** | [WebWalkerQA](https://huggingface.co/datasets/callanwu/WebWalkerQA) from *WebWalker: Benchmarking LLMs in Web Traversal* (Wu et al., Alibaba Tongyi Lab, [ACL 2025](https://arxiv.org/abs/2501.07572)): questions whose answers sit several clicks deep in real websites (conferences, universities, organizations, games), in English and Chinese. |
  91. | **Task selection** | Each task was re-checked live before the run: the site and answer page still load and still contain the reference answer. Tasks looked at while developing are excluded ([`bench/tasks/seen-ids.txt`](bench/tasks/seen-ids.txt)). |
  92. | **Accuracy run** | 50 tasks, 2026-10-12. Answering model `claude-opus-5-5`, judge `claude-sonnet-5-5`. |
  93. | **Crawl4AI run** | 20 different tasks, 2026-10-13, Crawl4AI 0.9.4. Crawl only, no LLM: did the crawler fetch the gold page? In the accuracy run, whether a system fetched the gold page closely tracked whether it answered correctly. |
  94. | **Baselines** | *Claude picks every link*: Claude reads each page and chooses the next link. *Breadth-first*: every page to depth 2. All systems share jevcrawl's fetcher and remove navigation and footers. |
  95. | **Costs** | Jev cost is billed and exact. LLM costs are token counts at list price. |
  96. | **Caveats** | 50 and 20 tasks are small samples: the cost gap is large and stable, the accuracy gap less so. Crawl4AI drives a real browser; the benchmark ran jevcrawl without JavaScript rendering. |
  97. The benchmark code and how to reproduce every number: [`bench/`](bench/README.md).
  98. ## Tools
  99. | Tool | Use it for |
  100. |---|---|
  101. | `crawl(goal, start_url, max_pages=30, max_seconds=120, return_tokens=4000, max_decision_usd=0.05)` | The answer may be several pages deep. Returns the relevant passages. |
  102. | `scrape(url, goal=None)` | One known page. With a goal, only the relevant passages; without one, all content and no Jev call. |
  103. | `map(start_url, goal, limit=50, max_pages=10)` | Ranked URLs only, to decide what to `scrape`. |
  104. <details>
  105. <summary><b>A real result</b> (docs.python.org, "What does the walrus operator := do and which Python version added it?")</summary>
  106. ```json
  107. {"blocks": [{"url": "https://docs.python.org/3/whatsnew/3.8.html",
  108. "heading_path": "What’s New In Python 3.8 > New Features > Assignment expressions",
  109. "text": "There is new syntax := that assigns values to variables as part of a larger expression. ...",
  110. "char_start": 307, "char_end": 580, "relevance": 0.97}],
  111. "uncertain": [],
  112. "pages": [{"url": "https://docs.python.org/3/", "title": "3.15.0 Documentation", "lang": "en",
  113. "text_sha256": "b624ce93..."}],
  114. "stats": {"pages_fetched": 6, "tokens_raw": 80034, "tokens_returned": 1873, "jev_calls": 11,
  115. "cache_hits": 0, "jev_input_tokens": 164139, "jev_cost_usd": 0.0069,
  116. "model": "jev-1.13.0", "degraded": false, "degraded_reason": null,
  117. "needs_js_urls": [], "stopped_because": "answered", "elapsed_seconds": 8.3}}
  118. ```
  119. Trimmed to one block and one page. 80,034 tokens of pages became 1,873 tokens of answer, for
  120. $0.0069, in 8.3 seconds.
  121. - `blocks` are the relevant passages, best first; `uncertain` are borderline ones worth reading
  122. before concluding something is missing.
  123. - `char_start`/`char_end` index into the page text whose sha256 is in `pages`, so a quote can be
  124. checked later.
  125. - `stopped_because` is `answered`, `frontier_empty`, `max_pages`, `max_seconds`, `budget` or
  126. `auth_error`.
  127. - Expected failures come back as `{"error": "<code>", "message": "<what to do>"}`; the tools
  128. never raise into the agent's session.
  129. </details>
  130. The same operations work from the command line (`uv tool install jevcrawl` or
  131. `pip install jevcrawl` puts `jevcrawl` on your path):
  132. ```bash
  133. jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" [--json]
  134. jevcrawl scrape https://example.com/help/returns [--goal "..."] [--json]
  135. jevcrawl map https://example.com --goal "..." [--limit 20]
  136. jevcrawl doctor
  137. ```
  138. ## How it works
  139. <picture>
  140. <source media="(prefers-color-scheme: dark)" srcset="assets/how-dark.svg">
  141. <img alt="A crawl in six steps: seed from the sitemap, fetch the best 4 pages, segment them into passages, filter boilerplate for free, Jev scores every passage and link in one request per page, Jev checks whether the kept passages answer the goal. If not, it follows the next best links. The docs.python.org example turned 80,034 tokens of pages into 1,873 tokens for $0.0069 in 8.3 s." src="assets/how-light.svg" width="100%">
  142. </picture>
  143. - **Jev decides, it does not write.** [Jev](https://typesafe.ai) returns a probability per
  144. yes/no question, so it cannot produce text. Everything you get back is copied from the page.
  145. - **Page text is data, never instructions.** It travels only in data fields of a question; every
  146. question sentence is a fixed string written by jevcrawl. An injected "ignore previous
    • Dismissed after review: Quotes instruction-override phrasing (in quotes, code, a table or an example): "ignore previous instructions"
    • Line 189 on GitHub
  147. instructions" can at worst bias a score.
  148. - **Links below the priority floor are kept in reserve**, not dropped, so a crawl never stops
  149. early while it still has page budget.
  150. - **Failures degrade, they do not hide.** After three failed Jev requests the crawl continues
  151. breadth-first, returns every block unscored in `uncertain`, and sets `stats.degraded`.
  152. - **Repeats are free.** Jev's answers are cached locally (`~/.jevcrawl/cache`), so the same goal
  153. on the same pages costs nothing.
  154. ## Configuration
  155. | Environment variable | Meaning | Default |
  156. |---|---|---|
  157. | `TYPESAFE_API_KEY` | Jev API key (required) | none |
  158. | `JEVCRAWL_BASE_URL` | Jev-compatible endpoint | `https://api.typesafe.ai` |
  159. | `JEVCRAWL_MODEL` | model id | `jev-1.13.0` (pinned: thresholds are calibrated per version) |
  160. | `JEVCRAWL_CACHE_DIR` | answer cache | `~/.jevcrawl/cache` |
  161. | `JEVCRAWL_RESPECT_ROBOTS` | skip URLs the site's `robots.txt` disallows (`0` turns it off) | `1` |
  162. | `JEVCRAWL_RENDER_JS` | render JavaScript-only pages when the `js` extra is installed (`0` turns it off) | `1` |
  163. Everything else (thresholds, links per page, concurrency) has a measured default in
  164. [`jevcrawl/config.py`](jevcrawl/config.py), and Python callers can pass their own `Config`.
  165. ## Cost and privacy
  166. Jev charges for input tokens only, $0.042 per million for `jev-1.13.0`; in the benchmark a crawl
  167. cost $0.011 on average. `max_decision_usd` is a hard limit: a request that could pass it is never
  168. sent, and the crawl stops with `budget`. Every result reports `jev_cost_usd`, and `tokens_raw`
  169. (what a plain crawl would hand your LLM) against `tokens_returned`.
  170. **Page text is sent to TypeSafe** (blocks, link labels and URLs) to be scored. The API key is
  171. read from the environment and never written to disk or logs.
  172. ## Crawling responsibly
  173. jevcrawl follows each site's `robots.txt` by default (set `JEVCRAWL_RESPECT_ROBOTS=0` to turn
  174. that off) and does not rate-limit itself beyond fetching four pages at a time. You are
  175. responsible for the sites you crawl, their terms of service and the law that applies to you. It
  176. never logs in and does not try to get past paywalls or CAPTCHAs. Without the `js` extra it does
  177. not run JavaScript (pages that need it are listed in `stats.needs_js_urls`). Every call is
  178. bounded by `max_pages`, `max_seconds`, `max_decision_usd` and `return_tokens`.
  179. ## Contributing
  180. ```bash
  181. uv sync
  182. uv run ruff check . && uv run ruff format --check . && uv run mypy --strict jevcrawl
  183. uv run pytest # offline: all HTTP is mocked
  184. TYPESAFE_API_KEY=... uv run pytest -m live -s # one tiny real request, a fraction of a cent
  185. ```
  186. Tests never touch the network or spend money; anything that does is marked `live`. The README
  187. charts are drawn by [`assets/build.py`](assets/build.py) from the measured numbers. Issues and
  188. pull requests are welcome; please report security problems as described in
  189. [SECURITY.md](SECURITY.md).
  190. ## Acknowledgments
  191. - [WebWalkerQA](https://github.com/Alibaba-NLP/WebAgent) by Alibaba's Tongyi Lab, for a
  192. benchmark of questions that really need crawling.
  193. - [Crawl4AI](https://github.com/unclecode/crawl4ai), the baseline we measure against.
  194. - [Model Context Protocol](https://modelcontextprotocol.io), [selectolax](https://github.com/rushter/selectolax)
  195. and [httpx](https://www.python-httpx.org/).
  196. ## License
  197. [Apache-2.0](LICENSE)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.