Skip to content
Back to search

crawlie

Fast, free, open-source technical SEO + GEO crawler: built for humans and agents.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/spronta--crawlie/badge.svg)](/listings/spronta--crawlie)

HTML

<a href="/listings/spronta--crawlie"><img src="/api/listings/spronta--crawlie/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**The easy way — npm** (installs the `crawlie` CLI and the `crawlie-mcp` server):

npm i -g crawlie

**The macOS app** — grab the signed `.dmg` from [Releases](https://github.com/spronta/crawlie/releases).

**From source** — needs [Rust](https://rustup.rs) (engine/CLI/MCP) and, for the desktop app, [pnpm](https://pnpm.io) + Node:

git clone https://github.com/spronta/crawlie
cd crawlie
cargo build --release
# → target/release/crawlie  and  target/release/crawlie-mcp

# or install onto your PATH:
cargo install --path crates/crawlie-cli      # installs `crawlie`
cargo install --path crates/crawlie-mcp      # installs `crawlie-mcp`

> **How it ships:** the **CLI + MCP** come *only* through npm — the right native binary installs automatically as a platform package (nothing to download or unblock). The **desktop app** is the only direct download: a signed, notarized `.dmg` on [Releases](https://github.com/spronta/crawlie/releases).

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 15 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "spronta",
  3. "owner": {
  4. "name": "Spronta",
  5. "url": "https://spronta.com"
  6. },
  7. "description": "Open-source SEO tooling by Spronta.",
  8. "plugins": [
  9. {
  10. "name": "crawlie",
  11. "source": "./",
  12. "description": "Turn your agent into an SEO team: crawl any site, catch broken links, redirect chains, missing metadata, thin content, and AI-search (GEO) gaps — then get a prioritized fix list and diff crawls to prove the fixes landed. Runs on Crawlie Cloud: sign in and go, nothing to install."
  13. }
  14. ]
  15. }
README.md · README · 316 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. # crawlie
  3. **The fast, free, open-source technical SEO + GEO crawler — built for humans and agents.**
  4. Crawl any site for broken links, redirects, missing metadata, and 40+ SEO & Generative-Engine checks — with plain-English guidance on every fix. Runs locally, ships a CLI and an MCP server, and costs nothing.
  5. [![npm](https://img.shields.io/npm/v/crawlie?color=cb3837&logo=npm&label=crawlie)](https://www.npmjs.com/package/crawlie)
  6. [![CI](https://github.com/spronta/crawlie/actions/workflows/ci.yml/badge.svg)](https://github.com/spronta/crawlie/actions/workflows/ci.yml)
  7. [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
  8. <p>
  9. <a href="#setup">Setup</a> ·
  10. <a href="#how-to-use-cli">CLI</a> ·
  11. <a href="#use-with-agents-mcp">MCP &amp; agents</a> ·
  12. <a href="https://crawlie.dev/changelog" target="_blank">Changelog</a> ·
  13. <a href="#use-cases">Use cases</a> ·
  14. <a href="#why-i-built-this">Why I built this</a> ·
  15. <a href="#desktop-app">Desktop app</a> ·
  16. <a href="#what-it-checks">Checks</a> ·
  17. <a href="#how-it-compares">Compare</a> ·
  18. <a href="#architecture">Architecture</a>
  19. </p>
  20. *[Quickstart and CLI guide →](#setup)*
  21. [💡Share feedback & suggestions here](https://github.com/spronta/crawlie/discussions/2)
  22. </div>
  23. ![Showcase of the example app demonstrating a report](https://cdn.spronta.com/spronta-8d32a2/pretty_snap_2026_5_18_23_58%20(2).png)
  24. ### Built by Spronta
  25. Crawlie helps you audit whether your content can be discovered. **Spronta helps visitors find it once they reach your website**: Search, Answers and Content Analytics for your existing blog, docs or resource library.
  26. [**Try free site search on your content →**](https://www.spronta.com/free-search/?utm_source=github&utm_medium=readme&utm_campaign=crawlie-to-spronta&utm_content=repo-intro) Connect a public website and test the demo. Search includes 10,000 requests per workspace each month, with no credit card required. Answers and Analytics are separate paid products.
  27. Crawlie runs locally under the MIT license. Spronta is an optional hosted service.
  28. ---
  29. ## Setup
  30. **The easy way — npm** (installs the `crawlie` CLI and the `crawlie-mcp` server):
  31. ```bash
  32. npm i -g crawlie
  33. ```
  34. **The macOS app** — grab the signed `.dmg` from [Releases](https://github.com/spronta/crawlie/releases).
  35. **From source** — needs [Rust](https://rustup.rs) (engine/CLI/MCP) and, for the desktop app, [pnpm](https://pnpm.io) + Node:
  36. ```bash
  37. git clone https://github.com/spronta/crawlie
  38. cd crawlie
  39. cargo build --release
  40. # → target/release/crawlie and target/release/crawlie-mcp
  41. # or install onto your PATH:
  42. cargo install --path crates/crawlie-cli # installs `crawlie`
  43. cargo install --path crates/crawlie-mcp # installs `crawlie-mcp`
  44. ```
  45. > **How it ships:** the **CLI + MCP** come *only* through npm — the right native binary installs automatically as a platform package (nothing to download or unblock). The **desktop app** is the only direct download: a signed, notarized `.dmg` on [Releases](https://github.com/spronta/crawlie/releases).
  46. ---
  47. ## How to use (CLI)
  48. ```bash
  49. # Crawl a whole site (respects robots.txt, seeds from sitemap.xml)
  50. crawlie crawl https://example.com --format pretty
  51. # Audit a single page, or a specific set of pages
  52. crawlie audit https://example.com/pricing
  53. crawlie audit https://example.com/a https://example.com/b
  54. # Save a shareable, self-contained HTML report
  55. crawlie crawl https://example.com --format html -o report.html
  56. # Clean JSON on stdout (perfect for piping / scripting / agents)
  57. crawlie crawl https://example.com --format json -o report.json
  58. # Learn why any finding matters and how to fix it
  59. crawlie explain geo-not-answerable
  60. ```
  61. **Output formats:** `pretty` (terminal), `json` (machine-readable, the default), `csv` (issues), `html` (shareable file).
  62. **Common flags:**
  63. | Flag | What it does |
  64. |---|---|
  65. | `--max-pages <n>` | Cap pages fetched (default 500) |
  66. | `--max-depth <n>` | Max click depth from the seed |
  67. | `--concurrency <n>` | Parallel requests (default 16) |
  68. | `--include <glob>` / `--exclude <glob>` | Scope the crawl by URL pattern |
  69. | `--no-robots` / `--no-sitemap` / `--no-external` | Turn off robots.txt, sitemap seeding, external link checks |
  70. | `--severity error\|warning\|notice` | Only output findings at/above a level |
  71. | `--save` | Save to local report history (`crawlie reports`, `crawlie report <id>`) |
  72. | `--fail-on error\|warning` | Non-zero exit code for CI gating |
  73. Every crawl returns three scores: a **Health** score (technical SEO), a **GEO** score (AI-search readiness), and an **Accessibility** score (WCAG conformance) — each reported separately so one kind of problem never hides another.
  74. ---
  75. ## Use with agents (MCP)
  76. crawlie ships a [Model Context Protocol](https://modelcontextprotocol.io) server so an LLM agent can run a full audit and act on it — no human in the loop. This is the part most SEO tools don't have.
  77. ### Connect it
  78. After `npm i -g crawlie`, `crawlie-mcp` is on your `PATH`. For **Claude Desktop**, edit `claude_desktop_config.json`:
  79. ```jsonc
  80. {
  81. "mcpServers": {
  82. "crawlie": {
  83. "command": "crawlie-mcp"
  84. }
  85. }
  86. }
  87. ```
  88. For **Claude Code**:
  89. ```bash
  90. claude mcp add crawlie crawlie-mcp
  91. ```
  92. (If you built from source instead, use the absolute path to `target/release/crawlie-mcp`.)
  93. (Any MCP-compatible client works — Cursor, Cline, your own agent. It speaks JSON-RPC over stdio.)
  94. ### Hosted: the Crawlie Cloud MCP (no install)
  95. Prefer not to install anything, or want crawls to run on our infrastructure? Point any MCP client at the hosted endpoint and authenticate with a Crawlie API key (create one in the dashboard under Settings, API keys). It speaks MCP Streamable HTTP.
  96. **Endpoint:** `https://crawlie.app/mcp`
  97. For **Claude Code**:
  98. ```bash
  99. claude mcp add --transport http crawlie-cloud https://crawlie.app/mcp \
  100. --header "Authorization: Bearer crw_your_key"
  101. ```
  102. For **Claude Desktop** (or any client that takes a JSON config):
  103. ```jsonc
  104. {
  105. "mcpServers": {
  106. "crawlie-cloud": {
  107. "type": "http",
  108. "url": "https://crawlie.app/mcp",
  109. "headers": { "Authorization": "Bearer crw_your_key" }
  110. }
  111. }
  112. }
  113. ```
  114. Hosted crawls run on the same engine as the dashboard, are scoped to your team, and are metered against your plan. The tools mirror the local server (`crawl_site`, `audit_url`, `top_fixes`, `geo_gaps`, `affected_urls`, `diff_reports`, plus `crawl_status` to poll a long crawl and `get_report` / `list_reports` over your saved cloud reports). Every crawl returns a `reportId` you can re-slice later without re-crawling.
  115. ### One-step install: the Claude Code plugin
  116. The fastest path. The [`crawlie` plugin](.claude-plugin/plugin.json) bundles the MCP server **and** a set of skills (audit playbooks) in a single install — the MCP server auto-runs via `npx`, so you don't even pre-install the binary:
  117. ```bash
  118. # add this repo as a marketplace, then install the plugin
  119. claude plugin marketplace add spronta/crawlie
  120. claude plugin install crawlie@spronta
  121. ```
  122. ### Skills (works with *any* agent, even without the MCP)
  123. The [`skills/`](skills/) folder holds standalone [Agent Skills](https://agentskills.io) that teach an agent how to run real audits — full-site SEO + GEO, broken-link fixes, pre-launch gates, and AI-search readiness. Each is **self-contained**: it needs neither this repo nor a pre-installed crawlie. Missing the binary? The skill runs it on demand via `npx -y -p crawlie …` (the install *is* the run), and automatically uses the MCP tools when they're present. See [skills/README.md](skills/README.md).
  124. ### Tools exposed
  125. | Tool | Purpose |
  126. |---|---|
  127. | `crawl_site` | Crawl + audit a whole site (SEO + GEO), returns scores, issues, per-page data |
  128. | `audit_url` | Audit a single page |
  129. | `audit_urls` | Audit an explicit list of pages |
  130. | `explain_issue` | Why a rule matters + how to fix it |
  131. | `list_rules` | The full catalogue of checks |
  132. | `list_reports` / `get_report` | Read saved crawl history |
  133. ### Example agent prompts
  134. > *"Crawl crawlie.dev, then give me the top 5 fixes that would most improve my GEO score, with the exact change for each."*
  135. > *"Audit these three landing pages and tell me which is least ready to be cited by AI search, and why."*
  136. > *"Run a crawl with `--fail-on error` semantics — are there any broken links or 5xx pages blocking launch?"*
  137. The agent calls `crawl_site`, reads the structured issues, and uses `explain_issue` to turn findings into a prioritized, actionable plan.
  138. ---
  139. ## Use cases
  140. - **Pre-launch QA** — catch broken links, redirects, 4xx/5xx, and missing metadata before you ship.
  141. - **GEO optimization** — make pages citable by AI search: structured data, semantic HTML, answer-ready content, authorship/E-E-A-T.
  142. - **Agent workflows** — let a marketing/SEO agent audit a site and propose fixes autonomously via MCP.
  143. - **CI/CD gating** — `crawlie crawl … --fail-on error` in a pipeline to block regressions.
  144. - **Client reporting** — generate a polished, shareable HTML report in one command.
  145. - **Auditing AI-generated sites** — verify that the site your agent just built is actually built for search.
  146. ---
  147. ## Why I built this
  148. I'm **Sean Ryan**. I've spent 6+ years as a Lead Marketing Engineer, and on the side I build AI tooling for marketers.
  149. With AI, it's faster than ever to ship a marketing site — but most of what gets generated is slop that was never built to be found. And the tools meant to catch that fall short: most SEO auditors cost money, don't play nicely with your agents, or tell you *what's* wrong without telling you *how to actually rank* for SEO **and** GEO (Generative Engine Optimization — being cited by AI search like ChatGPT, Perplexity, and Google AI Overviews).
  150. crawlie fixes that. It's free, it's local-first, it's agent-native, and every issue it finds comes with *why it matters* and *how to fix it*.
  151. **If this is useful to you, [connect with me on LinkedIn →](https://linkedin.com/in/sean-exe)** — I share what I'm learning building AI for marketers and SEO/GEO tooling, and I'd love to hear how you're using crawlie.
  152. ---
  153. ## Desktop app
  154. A beautiful Tauri + React app (Geist design, light/dark, seamless window chrome):
  155. ```bash
  156. cd apps/desktop
  157. pnpm install
  158. pnpm tauri dev # live native crawls
  159. pnpm dev # preview the UI in a browser (demo data, no backend)
  160. ```
  161. Whole-site / single-page / URL-list modes, live progress, **Health**, **GEO** & **Accessibility** score rings, issues with built-in *why-it-matters* guidance, a sortable pages table, a per-page drawer (GEO signals, headers, schema, hreflang…), auto-saved report history, and one-click shareable HTML export.
  162. > First run, generate the icon set: `cd src-tauri/icons && python3 generate.py && cd .. && pnpm tauri icon icons/source.png`
  163. ---
  164. ## What it checks
  165. *57 rules and counting.*
  166. **Technical SEO** — broken links · 4xx/5xx · redirects & chains · titles & meta descriptions (missing / duplicate / length) · H1s · canonicals · noindex / nofollow / X-Robots-Tag · robots.txt blocking · images missing alt · thin & duplicate content · orphan & deep pages
  167. **Performance & security** — slow responses · large pages · missing compression · HTTPS · mixed content · HSTS
  168. **Accessibility (WCAG)** — links & buttons without an accessible name · form controls without a label · iframes missing a title · zoom-blocking viewport · positive `tabindex` · skipped heading levels
  169. **Mobile, international & social** — viewport · `lang` · hreflang · Open Graph · Twitter cards · structured data
  170. **Structured-data validation** — parses JSON-LD and checks each item against Google's rich-result requirements: invalid markup, missing required fields, and missing recommended fields (Product, Article, Recipe, Event, FAQ, Breadcrumb, and more)
  171. **JavaScript rendering** — crawl with `--render` to audit each page's post-JavaScript DOM via headless Chrome, so client-rendered content (React/Next/Vue) is seen, and `content-requires-js` flags pages whose content only exists after JS runs
  172. **GEO — Generative Engine Optimization** — structured data, semantic HTML, answer-readiness, authorship/E-E-A-T, dated content, question-style headings, and extractable blocks, rolled into a per-page **GEO score**.
  173. Every finding links to plain-English guidance: **why it matters**, **how to fix it**, and **what happens if you ignore it**.
  174. ---
  175. ## How it compares
  176. | | **crawlie** | Screaming Frog | Sitebulb |
  177. |---|:---:|:---:|:---:|
  178. | Price | **Free & open-source** | £259/yr to unlock | from £13.50/mo |
  179. | Engine | **Rust, async, tiny binary** | Java (JVM) | .NET |
  180. | CLI with JSON output | ✅ | partial | ❌ |
  181. | JavaScript rendering | ✅ headless Chrome | ✅ | ✅ |
  182. | **MCP server (agent-native)** | ✅ | ❌ | ❌ |
  183. | **GEO — AI/answer-engine audit** | ✅ | ❌ | ❌ |
  184. | **"Why it matters" built in** | ✅ every issue | ❌ | partial |
  185. | Shareable HTML report | ✅ | paid | ✅ |
  186. | Source you can read & extend | ✅ | ❌ | ❌ |
  187. ---
  188. ## Architecture
  189. ```
  190. crates/
  191. crawlie-core # the engine — crawl, audit, score, knowledge base, reports
  192. crawlie-cli # `crawlie` — JSON / pretty / CSV / HTML output
  193. crawlie-mcp # `crawlie-mcp` — Model Context Protocol server (stdio)
  194. apps/
  195. desktop # Tauri v2 + React (Geist) desktop app
  196. ```
  197. `crawlie-core` has zero host dependencies — the same audited engine drops straight into a cloud worker (it already targets `wasm32`). One engine, every surface, identical results.
  198. ---
  199. ## Roadmap
  200. - Cloud workers (shared Rust core) for scheduled/remote crawls
  201. - JavaScript rendering for SPA-heavy sites
  202. - Crawl-to-crawl comparison & regression alerts
  203. - Internal-link graph visualization
  204. ---
  205. ## License & author
  206. **MIT** © **[Spronta Ltd](https://crawlie.dev)** — the crawler engine, CLI, MCP
  207. server, and desktop app: everything in this repository is MIT. Crawlie Cloud (the
  208. hosted crawl service and the marketing site) is a separate, closed-source product
  209. and is not part of this repository. Pull requests to the open engine, CLI, and
  210. desktop app are very welcome.
  211. Built by Sean Ryan — Lead Marketing Engineer at Pendo.io, building AI for marketers on the side. **[Connect on LinkedIn →](https://linkedin.com/in/sean-exe)**
  212. If crawlie saves you time, a ⭐ on the repo and a hello on LinkedIn mean a lot.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.