doc-scraper
Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data).
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Sriram-PR/doc-scraper with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Sriram-PR/doc-scraper'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/sriram-pr--doc-scraper)HTML
<a href="/listings/sriram-pr--doc-scraper"><img src="/api/listings/sriram-pr--doc-scraper/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Prerequisites
- **Go:** Version 1.26 or later - **Git:** For cloning the repository - **Disk Space:** Sufficient for storing crawled content and state database
Installation
**Option 1: Release Binaries**
Download the archive for your OS and architecture (Linux, macOS, Windows; amd64 and arm64) from the [Releases page](https://github.com/Sriram-PR/doc-scraper/releases/latest). Each release includes a `checksums.txt` to verify the download:
sha256sum --check --ignore-missing checksums.txt
Extract the archive and put `doc-scraper` on your `PATH`. The archive also ships a sample `config.yaml`.
**Option 2: Docker**
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/data ghcr.io/sriram-pr/doc-scraper:latest crawl -site rust_cli_book
The image (linux/amd64 and linux/arm64) uses `/data` as its working directory and `doc-scraper` as its entrypoint, so the arguments after the image name are the subcommand. Mount a directory containing `config.yaml` at `/data` and point `output_base_dir` and `state_dir` at relative paths so crawl output lands in the mounted directory. The image runs as a non-root user, so `--user` makes the bind mount writable. Versioned tags (for example `2.9.2`) are published alongside `latest`.
**Option 3: Claude Desktop Extension**
Download `doc-scraper.mcpb` from the [latest release](https://github.com/Sriram-PR/doc-scraper/releases/latest) and open it in Claude Desktop. It prompts for the path to your `config.yaml` and runs the bundled binary as an MCP server (see [MCP Server Mode](#mcp-server-mode)).
**Option 4: Go Install**
Install the latest version directly from GitHub:
go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest
This installs the `doc-scraper` binary to your `GOPATH/bin` directory (usually `~/go/bin` or `%USERPROFILE%\go\bin`). Make sure this directory is in your `PATH`.
**Option 5: Clone and Build**
1. **Clone the repository:**
git clone https://github.com/Sriram-PR/doc-scraper.git cd doc-scraper
2. **Install Dependencies:**
go mod tidy
3. **Build the Binary:**
make build # or: go build -o doc-scraper ./cmd/doc-scraper
This creates an executable named `doc-scraper` in the project root.
Quick Start
Create a minimal `config.yaml` in the project root:
output_base_dir: "./crawled_docs"
state_dir: "./crawler_state"
enable_jsonl_output: true
sites:
rust_cli_book:
start_urls:
- "https://rust-cli.github.io/book/index.html"
allowed_domain: "rust-cli.github.io"
allowed_path_prefix: "/book/"
content_selector: "#content, main"
max_depth: 2 # seed plus one level; set 0 for the whole bookRun the crawl:
./doc-scraper crawl -site rust_cli_book -loglevel info
The Markdown, plus `pages.jsonl`, `llms.txt`, and `llms-full.txt`, lands under `./crawled_docs/rust_cli_book/` (output is organized by site key). A small book like this finishes in a few seconds; large sites can take minutes, so start with a low `max_depth` to gauge size before removing the bound.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 89 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # LLM Documentation Scraper (`doc-scraper`)
- [](https://github.com/Sriram-PR/doc-scraper/actions/workflows/ci.yml)
- [](https://golang.org/)
- [](https://pkg.go.dev/github.com/Sriram-PR/doc-scraper/v2)
- [](https://github.com/Sriram-PR/doc-scraper/blob/main/LICENSE)
- [](https://glama.ai/mcp/servers/Sriram-PR/doc-scraper)
- > A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.
- 
- doc-scraper crawls documentation sites into clean Markdown and serves them to coding agents over MCP. Your agent searches and reads the docs you chose, offline, instead of guessing from training data or a live web search. Point `doc-scraper add` at any docs page and it detects the framework (30+ docs generators, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook), validates the content selector, and drafts the config for you.
- **Documentation: https://sriram-pr.github.io/doc-scraper/**
- ## Features
- - **Offline search for agents:** BM25 full-text search (`search_docs`) and page reads (`read_page`) over the stored corpus, with no network access
- - **Framework detection:** 30+ docs generators recognized, each selector validated on the page, Readability fallback, JavaScript-only shells reported instead of crawled empty
- - **Clean output:** GitHub-Flavored Markdown per page with frontmatter, plus `pages.jsonl`, `llms.txt`, and `llms-full.txt`
- - **Polite and resumable:** robots.txt and sitemaps, per-host rate limits, retries with backoff, resumable crawls, and staged output that never clobbers the last good copy
- - **Stays fresh:** incremental re-crawls, watch mode, and crawl history (`get_freshness`, `diff_crawl`)
- - **Many sites at once:** parallel crawls with shared limits
- ## Install
- Download a release binary (Linux, macOS, Windows; amd64 and arm64) from the [Releases page](https://github.com/Sriram-PR/doc-scraper/releases/latest), or:
- ```bash
- go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest
- ```
- ```bash
- docker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/data ghcr.io/sriram-pr/doc-scraper:latest crawl -site rust_cli_book
- ```
- Claude Desktop users can install `doc-scraper.mcpb` from the latest release instead. Details: [Install](https://sriram-pr.github.io/doc-scraper/getting-started/install/).
- ## Quick start
- ```bash
- doc-scraper add -site rust_cli_book https://rust-cli.github.io/book/index.html # detect, preview, confirm
- doc-scraper crawl -site rust_cli_book # Markdown lands in ./crawled_docs/rust_cli_book/
- doc-scraper search -site rust_cli_book "error handling"
- ```
- Then give it to Claude Code:
- ```bash
- claude mcp add --transport stdio doc-scraper -- /path/to/doc-scraper mcp-server -config /path/to/config.yaml
- ```
- See [Claude Code setup](https://sriram-pr.github.io/doc-scraper/mcp/claude-code/) for scopes and a first prompt.
- ## Docs
- - [Getting started](https://sriram-pr.github.io/doc-scraper/getting-started/install/): install, quick start, adding a site
- - [Use with agents (MCP)](https://sriram-pr.github.io/doc-scraper/mcp/overview/): Claude Code, Claude Desktop, any stdio client, and the 13 tools
- - [Docs frameworks](https://sriram-pr.github.io/doc-scraper/frameworks/overview/): how detection works, with guides for Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs, and mdBook
- - [Configuration](https://sriram-pr.github.io/doc-scraper/reference/configuration/) and [CLI](https://sriram-pr.github.io/doc-scraper/reference/cli/) reference
- - [Detection benchmark](https://sriram-pr.github.io/doc-scraper/benchmark/)
- ## Contributing
- Contributions are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) before starting: it covers setup, the testing rules CI enforces, how to claim an issue, and the policy on AI-assisted contributions. Issues labeled [`good first issue`](https://github.com/Sriram-PR/doc-scraper/labels/good%20first%20issue) are a good place to begin.
- Report security problems privately as described in [SECURITY.md](SECURITY.md). Everyone taking part follows the [Code of Conduct](CODE_OF_CONDUCT.md).
- ## Privacy Policy
- doc-scraper collects nothing: no telemetry, no analytics, no accounts. All output and state stays on your machine, and the only network requests it makes are the crawls and fetches you explicitly ask for. Full policy: [PRIVACY.md](https://github.com/Sriram-PR/doc-scraper/blob/main/PRIVACY.md).
- ## License
- This project is licensed under the [Apache-2.0 License](https://github.com/Sriram-PR/doc-scraper/blob/main/LICENSE).
- ## Acknowledgements
- - [GoQuery](https://github.com/PuerkitoBio/goquery) for HTML parsing
- - [html-to-markdown](https://github.com/JohannesKaufmann/html-to-markdown) for conversion
- - [BadgerDB](https://github.com/dgraph-io/badger) for state persistence
- - [mcp-go](https://github.com/mark3labs/mcp-go) for MCP server implementation
- - [go-readability](https://github.com/go-shiori/go-readability) for content extraction fallback
- - [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) for the pure-Go crawl-history index
- - [robotstxt](https://github.com/temoto/robotstxt) for `robots.txt` parsing
- - [yaml.v3](https://github.com/go-yaml/yaml) for configuration parsing
- - [google/uuid](https://github.com/google/uuid) for crawl job IDs
- - [x/sync](https://pkg.go.dev/golang.org/x/sync) for the weighted semaphores that cap concurrency
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.