crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/fastcrw/crw with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/fastcrw/crw'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/fastcrw--crw)HTML
<a href="/listings/fastcrw--crw"><img src="/api/listings/fastcrw--crw/badge.svg" alt="findsafeskills safety rating" /></a>Install
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 17 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- {
- "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
- "name": "fastcrw",
- "metadata": {
- "description": "fastCRW agent skills — scrape, crawl, map, search, parse, extract, and change-track the web with the open-source Firecrawl alternative."
- },
- "owner": {
- "name": "us"
- },
- "plugins": [
- {
- "name": "fastcrw",
- "source": "./",
- "category": "developer-tools"
- }
- ]
- }
README.md · README · 235 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <p align="center">
- <img src="docs/logo-animation.gif" alt="fastCRW" width="180" />
- </p>
- <h1 align="center">fastCRW</h1>
- <p align="center">
- Turn URLs into clean <strong>markdown</strong> or structured
- <strong>JSON</strong> with one engine for search, scrape, map, crawl, and extract.
- </p>
- <p align="center">
- Run it locally as a small Rust binary or use the managed API.
- </p>
- <p align="center">
- <a href="https://fastcrw.com/register?ref=gh-hero"><strong>Get 1000 free credits →</strong></a> ·
- <a href="#one-command-install"><strong>Install</strong></a> ·
- <a href="https://docs.fastcrw.com"><strong>Docs</strong></a>
- </p>
- <p align="center">
- <sub>No credit card. Continue with GitHub.</sub>
- </p>
- <p align="center">
- <a href="https://crates.io/crates/crw-server"><img src="https://img.shields.io/crates/v/crw-server.svg" alt="crates.io"></a>
- <a href="https://pypi.org/project/crw/"><img src="https://img.shields.io/pypi/v/crw.svg?label=pypi" alt="PyPI"></a>
- <a href="https://www.npmjs.com/package/crw-mcp"><img src="https://img.shields.io/npm/v/crw-mcp.svg?label=npm%20mcp" alt="npm crw-mcp"></a>
- <a href="https://github.com/fastcrw/crw/actions/workflows/ci.yml"><img src="https://github.com/fastcrw/crw/actions/workflows/ci.yml/badge.svg?branch=main&event=push" alt="CI"></a>
- <a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue.svg" alt="License"></a>
- <a href="https://github.com/fastcrw/crw/stargazers"><img src="https://img.shields.io/github/stars/fastcrw/crw?style=social" alt="GitHub Stars"></a>
- </p>
- ## One-command install
- ```bash
- curl -fsSL https://fastcrw.com/install | sh
- ```
- Runs local and free, no account needed. To use the Cloud, paste your key into
- the same command and it installs the binary, connects the key, and registers the
- MCP server with the AI coding tools you already have:
- ```bash
- curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... sh
- ```
- ```bash
- crw search "rust tutorials"
- ```
- Claude Code, Cursor, Codex, Gemini CLI, OpenCode and Windsurf are picked up
- automatically when they are already set up; nothing else is touched, and your
- key stays in `~/.config/crw/config.toml` rather than being copied into each
- tool. Add `CRW_NO_AGENTS=1` to skip that step, or run `crw setup` on its own to
- choose interactively.
- **1000 free credits, no credit card.** Managed proxies, JS rendering and search,
- with nothing to run or keep up to date.
- [Get my free key →](https://fastcrw.com/register?ref=gh-quickstart)
- macOS and Linux, Intel and ARM.
- [More install options →](https://docs.fastcrw.com/installation/)
- ## What it does
- | Operation | Outcome |
- |---|---|
- | **Scrape** | One URL to markdown, HTML, links, screenshots, or schema JSON |
- | **Crawl** | Follow a bounded site crawl and collect its pages |
- | **Map** | Discover URLs without scraping every page |
- | **Search** | Search the web and optionally scrape selected results |
- | **Extract** | Produce structured fields from one or many URLs |
- [See the full API →](https://docs.fastcrw.com/#rest-api)
- ## Why fastCRW
- On Firecrawl's own public 1,000-URL dataset, fastCRW recovered more truth than
- Crawl4AI and Firecrawl, matched the fastest median latency, and idled at
- **~14 MB RAM**.
- <p align="center">
- <a href="BENCHMARKS.md">
- <img src=".github/benchmarks/bench-radar.svg" alt="fastCRW compared with Crawl4AI and Firecrawl on truth-recall, unique recoveries, median latency, download size, and recall depth" width="100%">
- </a>
- </p>
- <p align="center"><sub><a href="BENCHMARKS.md">Methodology, full numbers, and how to reproduce it</a></sub></p>
- On a different benchmark entirely, answer accuracy rather than scrape recall,
- fastCRW answers **90.0%** of the 600 AA-Omniscience questions correctly. Every
- product listed on the Artificial Analysis Search Index sits below it.
- <p align="center">
- <a href="https://fastcrw.com/benchmarks/aa-omniscience">
- <img src=".github/benchmarks/bench-omniscience.svg" alt="AA-Omniscience answer accuracy. fastCRW 90.0 percent, ahead of every product on the Artificial Analysis Search Index: Firecrawl 73, Exa 70, You.com 69, Parallel 68, Tavily 64, and 38 with no search at all." width="100%">
- </a>
- </p>
- <p align="center"><sub><a href="https://fastcrw.com/benchmarks/aa-omniscience">The full 14-product comparison, the control run, and how to reproduce it</a></sub></p>
- ## Choose how you use it
- ### CLI
- ```bash
- crw https://example.com # scrape, works right after install
- crw search "rust async runtime" # search, after `crw setup`
- ```
- ### Python SDK
- Using Cloud? [Get an API key](https://fastcrw.com/register?ref=gh-sdk), then export it once:
- ```bash
- export CRW_API_KEY="crw_live_..."
- pip install crw
- ```
- ```python
- from crw import CrwClient
- client = CrwClient()
- page = client.scrape("https://example.com", formats=["markdown"])
- print(page["markdown"])
- ```
- <details>
- <summary><strong>Node.js</strong></summary>
- ```bash
- npm install crw-sdk
- ```
- ```javascript
- import { CrwClient } from "crw-sdk";
- const client = new CrwClient();
- const page = await client.scrape("https://example.com", {
- formats: ["markdown"],
- });
- console.log(page.markdown);
- ```
- </details>
- [Local mode and more SDK examples →](https://docs.fastcrw.com/sdk-examples/) ·
- [REST API →](https://docs.fastcrw.com/#rest-api)
- ### MCP for AI agents
- ```bash
- npx -y crw-mcp@latest install
- ```
- Installs the CRW skill and MCP server in your detected AI tools. `crw setup` can
- also do this step, so either path is enough.
- [Manual setup →](https://docs.fastcrw.com/mcp-clients/)
- ## Choose where it runs
- | | Managed API | Local / self-hosted |
- |---|---|---|
- | Best for | Zero infrastructure and managed scaling | Data control, private networks, or custom infrastructure |
- | Start | [Create an API key](https://fastcrw.com/register?ref=gh-table), then `crw setup` | Install and run `crw <URL>` |
- | Operations | Managed proxies, billing, and hosted capabilities | You choose renderers, search, auth, proxies, and capacity |
- Capabilities and response shapes can differ by deployment:
- [`/v1/capabilities`](https://docs.fastcrw.com/capabilities/) · [response shapes](https://docs.fastcrw.com/response-shapes/)
- [Self-hosting guide →](https://docs.fastcrw.com/self-hosting/)
- ## Learn more
- - [Quickstart](https://docs.fastcrw.com/quick-start/)
- - [API reference](https://docs.fastcrw.com/#rest-api)
- - [Benchmarks](BENCHMARKS.md)
- - [Firecrawl migration](https://docs.fastcrw.com/migrate-from-firecrawl/)
- - [Self-hosting](https://docs.fastcrw.com/self-hosting/)
- ## Contributing
- The workspace requires Rust 1.85 or newer:
- ```bash
- git clone https://github.com/fastcrw/crw
- cd crw
- make check-fast
- ```
- [Read the contributor guide →](CONTRIBUTING.md)
- ### Contributors
- crw is better because of these people. Thank you, all of you:
- <!-- contributors:start -->
- <p align="center">
- <a href="https://github.com/us" title="us"><img src="https://github.com/us.png?size=96" width="48" height="48" alt="us"/></a>
- <a href="https://github.com/swackhamer" title="swackhamer"><img src="https://github.com/swackhamer.png?size=96" width="48" height="48" alt="swackhamer"/></a>
- <a href="https://github.com/paoloantinori" title="paoloantinori"><img src="https://github.com/paoloantinori.png?size=96" width="48" height="48" alt="paoloantinori"/></a>
- <a href="https://github.com/behramcelen" title="behramcelen"><img src="https://github.com/behramcelen.png?size=96" width="48" height="48" alt="behramcelen"/></a>
- <a href="https://github.com/WhiteNoise0000" title="WhiteNoise0000"><img src="https://github.com/WhiteNoise0000.png?size=96" width="48" height="48" alt="WhiteNoise0000"/></a>
- <a href="https://github.com/rqi14" title="rqi14"><img src="https://github.com/rqi14.png?size=96" width="48" height="48" alt="rqi14"/></a>
- <a href="https://github.com/santhreal" title="santhreal"><img src="https://github.com/santhreal.png?size=96" width="48" height="48" alt="santhreal"/></a>
- <a href="https://github.com/AsheTheWings" title="AsheTheWings"><img src="https://github.com/AsheTheWings.png?size=96" width="48" height="48" alt="AsheTheWings"/></a>
- <a href="https://github.com/gjija" title="gjija"><img src="https://github.com/gjija.png?size=96" width="48" height="48" alt="gjija"/></a>
- <a href="https://github.com/adambenhassen" title="adambenhassen"><img src="https://github.com/adambenhassen.png?size=96" width="48" height="48" alt="adambenhassen"/></a>
- <a href="https://github.com/atirna" title="atirna"><img src="https://github.com/atirna.png?size=96" width="48" height="48" alt="atirna"/></a>
- <a href="https://github.com/cdebled" title="cdebled"><img src="https://github.com/cdebled.png?size=96" width="48" height="48" alt="cdebled"/></a>
- <a href="https://github.com/VIVAAN-DHAWAN" title="VIVAAN-DHAWAN"><img src="https://github.com/VIVAAN-DHAWAN.png?size=96" width="48" height="48" alt="VIVAAN-DHAWAN"/></a>
- <a href="https://github.com/alkaz-nodemaven" title="alkaz-nodemaven"><img src="https://github.com/alkaz-nodemaven.png?size=96" width="48" height="48" alt="alkaz-nodemaven"/></a>
- <a href="https://github.com/mj520" title="mj520"><img src="https://github.com/mj520.png?size=96" width="48" height="48" alt="mj520"/></a>
- </p>
- <!-- contributors:end -->
- Engine and MCP server: [AGPL-3.0](LICENSE). Python and TypeScript SDKs: MIT.
- Embedding license: hello@fastcrw.com.
- ## Star History
- <a href="https://www.star-history.com/?repos=us%2Fcrw&type=date&legend=top-left">
- <picture>
- <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=us/crw&type=date&theme=dark&legend=top-left&sealed_token=Pe6pRWL7lqTM-St9eo-Cmpk5kYNyuyun0krw9eVZQFIrm3g_R2h46IW6wfNalPXquMsWSNCgKqiar1YVo9MGy2IZmN5Lz6rjZcjBCw6bCcRHORKORFRi9A" />
- <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=us/crw&type=date&legend=top-left&sealed_token=Pe6pRWL7lqTM-St9eo-Cmpk5kYNyuyun0krw9eVZQFIrm3g_R2h46IW6wfNalPXquMsWSNCgKqiar1YVo9MGy2IZmN5Lz6rjZcjBCw6bCcRHORKORFRi9A" />
- <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=us/crw&type=date&legend=top-left&sealed_token=Pe6pRWL7lqTM-St9eo-Cmpk5kYNyuyun0krw9eVZQFIrm3g_R2h46IW6wfNalPXquMsWSNCgKqiar1YVo9MGy2IZmN5Lz6rjZcjBCw6bCcRHORKORFRi9A" />
- </picture>
- </a>
- <sub>Please respect website policies. Crawl and map follow `robots.txt` by default.</sub>
install.sh · Bundled script · 222 lines · 2 findings marked
- #!/bin/sh
- # CRW installer. Downloads the latest release binary for your platform.
- # Usage:
- # curl -fsSL https://fastcrw.com/install | sh
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 4 on GitHub
- # wget -qO- https://fastcrw.com/install | sh
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 5 on GitHub
- #
- # Options (environment variables):
- # CRW_VERSION=v0.3.0 Install a specific version instead of latest
- # CRW_INSTALL_DIR=~/.local/bin Custom install directory
- # CRW_BINARY=crw-mcp Install crw-mcp (MCP server) or crw-server instead of
- # the default crw CLI
- # CRW_API_KEY=crw_live_… Connect to CRW Cloud and register the MCP server
- # with every detected AI coding tool, in one command
- # CRW_NO_AGENTS=1 With CRW_API_KEY, skip the AI-tool registration
- set -eu
- main() {
- REPO="fastcrw/crw"
- INSTALL_DIR="${CRW_INSTALL_DIR:-/usr/local/bin}"
- BINARY="${CRW_BINARY:-crw}"
- # --- helpers ----------------------------------------------------------------
- BOLD="$(tput bold 2>/dev/null || printf '')"
- BLUE="$(tput setaf 4 2>/dev/null || printf '')"
- GREEN="$(tput setaf 2 2>/dev/null || printf '')"
- RED="$(tput setaf 1 2>/dev/null || printf '')"
- RESET="$(tput sgr0 2>/dev/null || printf '')"
- info() { printf '%s==>%s %s\n' "${BLUE}${BOLD}" "${RESET}" "$*"; }
- success() { printf '%s==>%s %s\n' "${GREEN}${BOLD}" "${RESET}" "$*"; }
- err() { printf '%serror:%s %s\n' "${RED}${BOLD}" "${RESET}" "$*" >&2; exit 1; }
- need() {
- command -v "$1" >/dev/null 2>&1 || err "'$1' is required but not found"
- }
- # Print a binary's version line, or nothing. Only the `crw` CLI has a --version
- # flag; crw-mcp and crw-server reject it, so callers must treat an empty result
- # as "unknown" and fall back to the plain binary name.
- version_of() {
- "$1" --version 2>/dev/null | head -1
- }
- # --- detect downloader ------------------------------------------------------
- download() {
- if command -v curl >/dev/null 2>&1; then
- curl --fail --location --silent --show-error \
- --proto '=https' --tlsv1.2 \
- --output "$2" "$1"
- elif command -v wget >/dev/null 2>&1; then
- wget --https-only --quiet --output-document="$2" "$1"
- else
- err "curl or wget is required"
- fi
- }
- # --- detect platform --------------------------------------------------------
- detect_platform() {
- OS="$(uname -s)"
- ARCH="$(uname -m)"
- case "$OS" in
- Darwin) PLATFORM="darwin" ;;
- Linux) PLATFORM="linux" ;;
- MINGW*|MSYS*|CYGWIN*) PLATFORM="win32" ;;
- *) err "Unsupported OS: $OS. Try: cargo install $BINARY" ;;
- esac
- # Rosetta 2 detection: uname returns x86_64 under Rosetta on Apple Silicon
- if [ "$PLATFORM" = "darwin" ] && [ "$ARCH" = "x86_64" ]; then
- if sysctl -n sysctl.proc_translated 2>/dev/null | grep -q '^1$'; then
- info "Rosetta 2 detected, installing native arm64 binary"
- ARCH="arm64"
- fi
- fi
- case "$ARCH" in
- x86_64|amd64) ARCH_LABEL="x64" ;;
- aarch64|arm64) ARCH_LABEL="arm64" ;;
- *) err "Unsupported architecture: $ARCH. Try: cargo install $BINARY" ;;
- esac
- # Linux binaries are static musl builds, so they run on ANY libc (glibc and
- # musl/Alpine alike), so no libc gate is needed.
- if [ "$PLATFORM" = "win32" ]; then
- ASSET="${BINARY}-${PLATFORM}-${ARCH_LABEL}.zip"
- else
- ASSET="${BINARY}-${PLATFORM}-${ARCH_LABEL}.tar.gz"
- fi
- }
- # --- pick the download URL ----------------------------------------------------
- # Unpinned, we hand the download straight to GitHub's own
- # `releases/latest/download/<asset>` redirect instead of first asking the REST
- # API which tag is latest. That API allows 60 unauthenticated requests per hour
- # per IP, so on any shared address (CI runners, an office NAT, a cloud VM) the
- # tag lookup was what failed, never the download itself. The redirect is not
- # rate limited and needs no token.
- pick_url() {
- if [ -n "${CRW_VERSION:-}" ]; then
- VERSION="$CRW_VERSION"
- URL="https://github.com/${REPO}/releases/download/${VERSION}/${ASSET}"
- else
- # Empty means "whatever the redirect gives us"; messages say "latest" and
- # the real tag is read back off the binary once it is installed.
- VERSION=""
- URL="https://github.com/${REPO}/releases/latest/download/${ASSET}"
- fi
- }
- # --- download & install -----------------------------------------------------
- install() {
- CRW_TMPDIR="$(mktemp -d)"
- trap 'rm -rf "$CRW_TMPDIR"' EXIT
- # Check for existing installation
- if command -v "$BINARY" >/dev/null 2>&1; then
- INSTALLED="$(version_of "$BINARY")"
- info "Upgrading ${INSTALLED:-$BINARY} to ${VERSION:-latest}..."
- else
- info "Downloading ${BINARY} ${VERSION:-latest} (${PLATFORM}/${ARCH_LABEL})..."
- fi
- download "$URL" "${CRW_TMPDIR}/${ASSET}"
- info "Extracting..."
- if [ "$PLATFORM" = "win32" ]; then
- need unzip
- unzip -o "${CRW_TMPDIR}/${ASSET}" -d "$CRW_TMPDIR" >/dev/null
- else
- tar xzf "${CRW_TMPDIR}/${ASSET}" -C "$CRW_TMPDIR"
- fi
- # Verify the binary was extracted
- [ -f "${CRW_TMPDIR}/${BINARY}" ] || err "Archive did not contain '${BINARY}'"
- # Create install directory if needed
- if [ ! -d "$INSTALL_DIR" ]; then
- if [ -w "$(dirname "$INSTALL_DIR")" ]; then
- mkdir -p "$INSTALL_DIR"
- else
- sudo mkdir -p "$INSTALL_DIR"
- fi
- fi
- info "Installing to ${INSTALL_DIR}/${BINARY}..."
- if [ -w "$INSTALL_DIR" ]; then
- mv "${CRW_TMPDIR}/${BINARY}" "${INSTALL_DIR}/${BINARY}"
- chmod +x "${INSTALL_DIR}/${BINARY}"
- else
- sudo mv "${CRW_TMPDIR}/${BINARY}" "${INSTALL_DIR}/${BINARY}"
- sudo chmod +x "${INSTALL_DIR}/${BINARY}"
- fi
- # With the `latest` redirect the tag is never known up front, so report the
- # version the installed binary actually is rather than the label we asked for.
- INSTALLED_VERSION="$(version_of "${INSTALL_DIR}/${BINARY}")"
- success "${INSTALLED_VERSION:-$BINARY} installed to ${INSTALL_DIR}/${BINARY}"
- echo ""
- # Check if install dir is in PATH
- case ":$PATH:" in
- *":${INSTALL_DIR}:"*) ;;
- *) echo " Note: ${INSTALL_DIR} is not in your PATH. Add it with:"
- echo " export PATH=\"${INSTALL_DIR}:\$PATH\""
- echo "" ;;
- esac
- # First-run onboarding (crw CLI only). Basic local scraping needs no setup,
- # so installation must never drop users into an unsolicited wizard. An API
- # key supplied explicitly is still a deliberate one-command cloud connect;
- # otherwise print the runnable first command and keep setup clearly optional.
- if [ "$BINARY" = "crw" ]; then
- if [ -n "${CRW_API_KEY:-}" ]; then
- # One-command cloud connect: `curl … | CRW_API_KEY=crw_live_… sh`.
- # Non-interactive: validates the key and writes config.toml, no /dev/tty
- # needed (works in CI too).
- echo ""
- info "Connecting to CRW Cloud with your API key…"
- echo ""
- # Registration into detected AI tools is part of the same command; the
- # pasted CRW_API_KEY is the consent a piped installer cannot prompt for.
- # CRW_NO_AGENTS=1 opts out.
- if [ -n "${CRW_NO_AGENTS:-}" ]; then
- "${INSTALL_DIR}/${BINARY}" setup --api-key "${CRW_API_KEY}" --no-agents \
- || info "Cloud connect failed. Run 'crw setup --api-key <key>' to retry."
- else
- "${INSTALL_DIR}/${BINARY}" setup --api-key "${CRW_API_KEY}" \
- || info "Cloud connect failed. Run 'crw setup --api-key <key>' to retry."
- fi
- else
- echo ""
- echo " Run: crw https://example.com"
- echo " Optional: crw setup # connect Cloud or add local JS/search"
- echo " Help: crw --help"
- echo ""
- fi
- else
- echo ""
- echo " Run: ${BINARY} --help"
- echo ""
- fi
- }
- # --- run --------------------------------------------------------------------
- detect_platform
- pick_url
- install
- }
- # main() wrapper ensures the entire script is downloaded before execution
- main
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.