Skip to content
Back to search

doc-scraper

Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data).

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Sriram-PR/doc-scraper with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Sriram-PR/doc-scraper'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/sriram-pr--doc-scraper/badge.svg)](/listings/sriram-pr--doc-scraper)

HTML

<a href="/listings/sriram-pr--doc-scraper"><img src="/api/listings/sriram-pr--doc-scraper/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Prerequisites

- **Go:** Version 1.26 or later - **Git:** For cloning the repository - **Disk Space:** Sufficient for storing crawled content and state database

Installation

**Option 1: Release Binaries**

Download the archive for your OS and architecture (Linux, macOS, Windows; amd64 and arm64) from the [Releases page](https://github.com/Sriram-PR/doc-scraper/releases/latest). Each release includes a `checksums.txt` to verify the download:

sha256sum --check --ignore-missing checksums.txt

Extract the archive and put `doc-scraper` on your `PATH`. The archive also ships a sample `config.yaml`.

**Option 2: Docker**

docker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/data ghcr.io/sriram-pr/doc-scraper:latest crawl -site rust_cli_book

The image (linux/amd64 and linux/arm64) uses `/data` as its working directory and `doc-scraper` as its entrypoint, so the arguments after the image name are the subcommand. Mount a directory containing `config.yaml` at `/data` and point `output_base_dir` and `state_dir` at relative paths so crawl output lands in the mounted directory. The image runs as a non-root user, so `--user` makes the bind mount writable. Versioned tags (for example `2.9.2`) are published alongside `latest`.

**Option 3: Claude Desktop Extension**

Download `doc-scraper.mcpb` from the [latest release](https://github.com/Sriram-PR/doc-scraper/releases/latest) and open it in Claude Desktop. It prompts for the path to your `config.yaml` and runs the bundled binary as an MCP server (see [MCP Server Mode](#mcp-server-mode)).

**Option 4: Go Install**

Install the latest version directly from GitHub:

go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest

This installs the `doc-scraper` binary to your `GOPATH/bin` directory (usually `~/go/bin` or `%USERPROFILE%\go\bin`). Make sure this directory is in your `PATH`.

**Option 5: Clone and Build**

1. **Clone the repository:**

   git clone https://github.com/Sriram-PR/doc-scraper.git
   cd doc-scraper

2. **Install Dependencies:**

   go mod tidy

3. **Build the Binary:**

   make build
   # or: go build -o doc-scraper ./cmd/doc-scraper

This creates an executable named `doc-scraper` in the project root.

Quick Start

Create a minimal `config.yaml` in the project root:

output_base_dir: "./crawled_docs"
state_dir: "./crawler_state"
enable_jsonl_output: true
sites:
  rust_cli_book:
    start_urls:
      - "https://rust-cli.github.io/book/index.html"
    allowed_domain: "rust-cli.github.io"
    allowed_path_prefix: "/book/"
    content_selector: "#content, main"
    max_depth: 2          # seed plus one level; set 0 for the whole book

Run the crawl:

./doc-scraper crawl -site rust_cli_book -loglevel info

The Markdown, plus `pages.jsonl`, `llms.txt`, and `llms-full.txt`, lands under `./crawled_docs/rust_cli_book/` (output is organized by site key). A small book like this finishes in a few seconds; large sites can take minutes, so start with a low `max_depth` to gauge size before removing the bound.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 89 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # LLM Documentation Scraper (`doc-scraper`)
  2. [![CI](https://github.com/Sriram-PR/doc-scraper/actions/workflows/ci.yml/badge.svg)](https://github.com/Sriram-PR/doc-scraper/actions/workflows/ci.yml)
  3. [![Go Version](https://img.shields.io/github/go-mod/go-version/Sriram-PR/doc-scraper)](https://golang.org/)
  4. [![Go Reference](https://pkg.go.dev/badge/github.com/Sriram-PR/doc-scraper/v2.svg)](https://pkg.go.dev/github.com/Sriram-PR/doc-scraper/v2)
  5. [![License](https://img.shields.io/github/license/Sriram-PR/doc-scraper)](https://github.com/Sriram-PR/doc-scraper/blob/main/LICENSE)
  6. [![Glama score](https://glama.ai/mcp/servers/Sriram-PR/doc-scraper/badges/score.svg)](https://glama.ai/mcp/servers/Sriram-PR/doc-scraper)
  7. > A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.
  8. ![doc-scraper crawling a docs site and answering search queries offline](demo/demo.gif)
  9. doc-scraper crawls documentation sites into clean Markdown and serves them to coding agents over MCP. Your agent searches and reads the docs you chose, offline, instead of guessing from training data or a live web search. Point `doc-scraper add` at any docs page and it detects the framework (30+ docs generators, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook), validates the content selector, and drafts the config for you.
  10. **Documentation: https://sriram-pr.github.io/doc-scraper/**
  11. ## Features
  12. - **Offline search for agents:** BM25 full-text search (`search_docs`) and page reads (`read_page`) over the stored corpus, with no network access
  13. - **Framework detection:** 30+ docs generators recognized, each selector validated on the page, Readability fallback, JavaScript-only shells reported instead of crawled empty
  14. - **Clean output:** GitHub-Flavored Markdown per page with frontmatter, plus `pages.jsonl`, `llms.txt`, and `llms-full.txt`
  15. - **Polite and resumable:** robots.txt and sitemaps, per-host rate limits, retries with backoff, resumable crawls, and staged output that never clobbers the last good copy
  16. - **Stays fresh:** incremental re-crawls, watch mode, and crawl history (`get_freshness`, `diff_crawl`)
  17. - **Many sites at once:** parallel crawls with shared limits
  18. ## Install
  19. Download a release binary (Linux, macOS, Windows; amd64 and arm64) from the [Releases page](https://github.com/Sriram-PR/doc-scraper/releases/latest), or:
  20. ```bash
  21. go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest
  22. ```
  23. ```bash
  24. docker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/data ghcr.io/sriram-pr/doc-scraper:latest crawl -site rust_cli_book
  25. ```
  26. Claude Desktop users can install `doc-scraper.mcpb` from the latest release instead. Details: [Install](https://sriram-pr.github.io/doc-scraper/getting-started/install/).
  27. ## Quick start
  28. ```bash
  29. doc-scraper add -site rust_cli_book https://rust-cli.github.io/book/index.html # detect, preview, confirm
  30. doc-scraper crawl -site rust_cli_book # Markdown lands in ./crawled_docs/rust_cli_book/
  31. doc-scraper search -site rust_cli_book "error handling"
  32. ```
  33. Then give it to Claude Code:
  34. ```bash
  35. claude mcp add --transport stdio doc-scraper -- /path/to/doc-scraper mcp-server -config /path/to/config.yaml
  36. ```
  37. See [Claude Code setup](https://sriram-pr.github.io/doc-scraper/mcp/claude-code/) for scopes and a first prompt.
  38. ## Docs
  39. - [Getting started](https://sriram-pr.github.io/doc-scraper/getting-started/install/): install, quick start, adding a site
  40. - [Use with agents (MCP)](https://sriram-pr.github.io/doc-scraper/mcp/overview/): Claude Code, Claude Desktop, any stdio client, and the 13 tools
  41. - [Docs frameworks](https://sriram-pr.github.io/doc-scraper/frameworks/overview/): how detection works, with guides for Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook
  42. - [Configuration](https://sriram-pr.github.io/doc-scraper/reference/configuration/) and [CLI](https://sriram-pr.github.io/doc-scraper/reference/cli/) reference
  43. - [Detection benchmark](https://sriram-pr.github.io/doc-scraper/benchmark/)
  44. ## Contributing
  45. Contributions are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) before starting: it covers setup, the testing rules CI enforces, how to claim an issue, and the policy on AI-assisted contributions. Issues labeled [`good first issue`](https://github.com/Sriram-PR/doc-scraper/labels/good%20first%20issue) are a good place to begin.
  46. Report security problems privately as described in [SECURITY.md](SECURITY.md). Everyone taking part follows the [Code of Conduct](CODE_OF_CONDUCT.md).
  47. ## Privacy Policy
  48. doc-scraper collects nothing: no telemetry, no analytics, no accounts. All output and state stays on your machine, and the only network requests it makes are the crawls and fetches you explicitly ask for. Full policy: [PRIVACY.md](https://github.com/Sriram-PR/doc-scraper/blob/main/PRIVACY.md).
  49. ## License
  50. This project is licensed under the [Apache-2.0 License](https://github.com/Sriram-PR/doc-scraper/blob/main/LICENSE).
  51. ## Acknowledgements
  52. - [GoQuery](https://github.com/PuerkitoBio/goquery) for HTML parsing
  53. - [html-to-markdown](https://github.com/JohannesKaufmann/html-to-markdown) for conversion
  54. - [BadgerDB](https://github.com/dgraph-io/badger) for state persistence
  55. - [mcp-go](https://github.com/mark3labs/mcp-go) for MCP server implementation
  56. - [go-readability](https://github.com/go-shiori/go-readability) for content extraction fallback
  57. - [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) for the pure-Go crawl-history index
  58. - [robotstxt](https://github.com/temoto/robotstxt) for `robots.txt` parsing
  59. - [yaml.v3](https://github.com/go-yaml/yaml) for configuration parsing
  60. - [google/uuid](https://github.com/google/uuid) for crawl job IDs
  61. - [x/sync](https://pkg.go.dev/golang.org/x/sync) for the weighted semaphores that cap concurrency

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.