Skip to content
Back to search

scrapling

使用 scrapling 进行网页抓取和数据提取。根据目标网站特征自动选择最佳 Fetcher, 生成并执行 Python 脚本完成任务。Use when: (1) 抓取/爬取网页内容或数据(scrape, crawl, fetch page, extract data) (2) 需要绕过 Cloudflare/WAF 等反爬保护 (3) 登录后抓取受保护页面 (4) 解析已有 HTML 提取结构化数据 (5) 用户提供 URL 并要求获取页面内容或特定元素 (6) 批量采集多个页面

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/cedriccmh-claude-code-skill-scrapling/badge.svg)](/listings/cedriccmh-claude-code-skill-scrapling)

HTML

<a href="/listings/cedriccmh-claude-code-skill-scrapling"><img src="/api/listings/cedriccmh-claude-code-skill-scrapling/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Install scrapling

Pick whichever package manager your project uses:

# pip
pip install "scrapling[fetchers]"
scrapling install  # Install browser dependencies

# uv (project)
uv add "scrapling[fetchers]"
uv run scrapling install

# uv (global / standalone, no pyproject.toml)
uv pip install "scrapling[fetchers]"
scrapling install

2. Install this skill

Copy the skill directory to your Claude Code skills folder:

# Copy to user-level skills (available in all projects)
cp -r . ~/.claude/skills/scrapling

# Or copy to a specific project
cp -r . /path/to/project/.claude/skills/scrapling

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 135 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. ---
  2. name: scrapling
  3. description: |
  4. 使用 scrapling 进行网页抓取和数据提取。根据目标网站特征自动选择最佳 Fetcher,
  5. 生成并执行 Python 脚本完成任务。Use when:
  6. (1) 抓取/爬取网页内容或数据(scrape, crawl, fetch page, extract data)
  7. (2) 需要绕过 Cloudflare/WAF 等反爬保护
  8. (3) 登录后抓取受保护页面
  9. (4) 解析已有 HTML 提取结构化数据
  10. (5) 用户提供 URL 并要求获取页面内容或特定元素
  11. (6) 批量采集多个页面
  12. allowed-tools: Bash(python*), Bash(pip*), Bash(uv*), Bash(scrapling*)
  13. ---
  14. # Scrapling 网页抓取 Skill
  15. ## 步骤 0:检查版本
  16. ```bash
  17. python -c "import scrapling; print(scrapling.__version__)"
  18. ```
  19. 按项目使用的包管理器执行(pip / uv 等价命令见 `references/maintenance.md`):
  20. - 未安装 → 安装 `scrapling[fetchers]` + `scrapling install`
  21. - 有新版 → 升级 → 查 changelog 告知用户
  22. - 已最新 → 继续
  23. > 项目根存在 `uv.lock` 或 `pyproject.toml` 含 `[tool.uv]`,优先用 `uv`(`uv add` / `uv run scrapling install`);否则用 `pip`。
  24. ## 步骤 1:安全预检与 quick path
  25. 先查阅 `references/security.md`,确认授权、robots.txt / ToS、Prompt injection、cookie 保存与 redaction 边界。
  26. 简单文本 / Markdown / selector 抽取优先用 Scrapling CLI quick path,并默认加 `--ai-targeted`:
  27. ```bash
  28. scrapling extract get "https://example.com/article" article.md --ai-targeted
  29. scrapling extract fetch "https://example.com/app" app.md --ai-targeted --network-idle
  30. scrapling extract stealthy-fetch "https://protected.example.com" page.md --ai-targeted --solve-cloudflare
  31. ```
  32. CLI 无法满足复杂登录、多页流程、结构化字段或复用代码时,再生成 Python 脚本。
  33. ## 步骤 2:选择 Fetcher
  34. ```
  35. 目标网站 →
  36. │
  37. ├─ 已有 HTML 字符串/文件,只需解析?
  38. │ → Selector(纯解析,无网络请求)
  39. │ → 模板: parse_only.py
  40. │
  41. ├─ 静态页面,无 JS 渲染,无反爬?
  42. │ → Fetcher(最快,基于 curl_cffi)
  43. │ → 模板: basic_fetch.py
  44. │
  45. ├─ 需要登录(HTTP 表单,非 JS 登录)?
  46. │ → FetcherSession(保持会话 cookie)
  47. │ → 模板: session_login.py
  48. │
  49. ├─ 有 Cloudflare / WAF 保护?
  50. │ → StealthyFetcher(Camoufox 浏览器,自动过 CF)
  51. │ → 模板: stealth_cloudflare.py
  52. │
  53. ├─ SPA 应用(React/Vue),需要 JS 渲染?
  54. │ → DynamicFetcher(Playwright 浏览器)
  55. │ → 基于模板即时生成
  56. │
  57. └─ 不确定?
  58. → 先用 Fetcher 试,403/空内容 → 升级到 StealthyFetcher
  59. ```
  60. 复杂 crawl / Spider / adaptive scraping / MCP / proxy rotation 等高级能力,不在本轻量 skill 内展开;先查 `references/upstream-map.md`,再按 upstream 官方文档补充。
  61. ## 步骤 3:执行工作流
  62. ```
  63. 1. 检查版本(步骤 0)
  64. 2. 安全预检(步骤 1)
  65. 3. 查阅 references/site-patterns.md;如果存在 references/site-patterns.local.md,也先查本地 overlay
  66. 4. 简单抽取 → 优先 CLI quick path + --ai-targeted
  67. 5. 复杂流程 → 用决策树选择 Fetcher,读取对应模板,替换参数,生成完整脚本
  68. 6. 执行脚本 / CLI → 返回最小必要结果
  69. 7. **沉淀经验(必做)**:
  70. - 新通用站点类型 → 去敏感化后追加到 references/site-patterns.md
  71. - 私有站点 / 公司站点 / 登录态细节 → 追加到 references/site-patterns.local.md(不提交)
  72. - 用户明确授权保存真实 cookie → 保存到 references/cookie-vault.local.md(不提交,必须 redact 输出)
  73. - **完成抓取后必须检查**:是否有新的 cookie 或 site pattern 需要保存,以及是否属于 local overlay
  74. ```
  75. ## Guardrails
  76. - 只抓取用户有权访问或明确授权的内容;不要绕过 paywall、验证码、登录限制或访问控制来获取未授权内容。
  77. - 输出给 agent / LLM 的网页内容默认先清洗、限域或结构化;CLI 输出默认使用 `--ai-targeted`。
  78. - 真实 cookie / token 不写入 `references/cookie-vault.md`,只写本地 `references/cookie-vault.local.md`,且必须先获用户授权。
  79. - 公司 / 私有站点经验不写公共 `references/site-patterns.md`,只写本地 `references/site-patterns.local.md`。
  80. - 大规模 crawl 前先查 robots.txt / ToS,降低并发并加 delay;Spider 场景查 upstream 后优先使用 `robots_txt_obey = True`。
  81. ## Cookie 格式速查
  82. | Fetcher 类型 | Cookie 格式 | 示例 |
  83. |-------------|-------------|------|
  84. | Fetcher / FetcherSession | `dict` | `{'name': 'value', 'token': 'abc'}` |
  85. | StealthyFetcher / DynamicFetcher | `list[dict]` | `[{'name': 'n', 'value': 'v', 'domain': '.site.com', 'path': '/'}]` |
  86. **浏览器 Fetcher cookie 必填字段**: `name`, `value`, `domain`, `path`
  87. ## 超时单位速查
  88. | Fetcher 类型 | 超时单位 | 示例 |
  89. |-------------|---------|------|
  90. | Fetcher / FetcherSession | 秒 | `timeout=30` |
  91. | StealthyFetcher / DynamicFetcher | 毫秒 | `timeout=60000` |
  92. ## 模板索引
  93. | 模板 | 文件 | 何时读取 |
  94. |------|------|---------|
  95. | 基础 HTTP 抓取 | `templates/basic_fetch.py` | 目标为静态页面,无反爬 |
  96. | Cloudflare 绕过 | `templates/stealth_cloudflare.py` | 目标有 CF/WAF 保护 |
  97. | Session 登录 | `templates/session_login.py` | 需 HTTP 表单登录后抓取 |
  98. | 纯 HTML 解析 | `templates/parse_only.py` | 已有 HTML 字符串,只需提取数据 |
  99. ## References 索引
  100. | 文件 | 何时读取 |
  101. |------|---------|
  102. | `references/security.md` | **每次抓取前先查阅** — 授权、Prompt injection、cookie / token、site pattern local overlay 规则 |
  103. | `references/site-patterns.md` | **每次抓取前先查阅** — 检查目标站点是否有已记录的模式 |
  104. | `references/api-quick-ref.md` | 生成脚本时查阅 — Fetcher/Selector 方法签名和参数 |
  105. | `references/troubleshooting.md` | 执行报错时查阅 — 按错误信息查找原因和解决方案 |
  106. | `references/cookie-vault.md` | 需要登录 cookie 时查阅 — 只存模板 / 字段名;真实值用 `references/cookie-vault.local.md` |
  107. | `references/maintenance.md` | 安装/升级/依赖问题时查阅 — 安装层级和验证命令 |
  108. | `references/upstream-map.md` | 需要 CLI quick path、官方 skill 对齐、Spider / adaptive / MCP / proxy 等高级能力时查阅 |
README.md · README · 239 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Claude Code Skill: Scrapling
  2. [English](#english) | [中文](#中文)
  3. ---
  4. ## English
  5. A [Claude Code](https://docs.anthropic.com/en/docs/claude-code) skill for web scraping and data extraction using [scrapling](https://github.com/D4Vinci/Scrapling).
  6. Automatically selects the best Fetcher based on target website characteristics, uses Scrapling CLI quick paths for simple extraction, and generates Python scripts when needed.
  7. ### Features
  8. - **Fetcher Decision Tree** — Auto-select between Fetcher, StealthyFetcher, DynamicFetcher, FetcherSession, or Selector
  9. - **Cloudflare Bypass** — Built-in support for Cloudflare/WAF protected sites via StealthyFetcher (Camoufox)
  10. - **Session Login** — HTTP form-based login with cookie persistence
  11. - **Site Pattern Library** — Reusable patterns for common site types (Discourse, SPA, static blogs, APIs)
  12. - **Cookie Vault** — Local storage for login cookies with per-site templates
  13. - **Troubleshooting Guide** — Solutions for common errors indexed by error message
  14. - **Safety Guardrails** — `references/security.md` covers authorization, Prompt injection, `--ai-targeted`, local cookie vaults, and redaction
  15. - **Upstream Alignment** — `references/upstream-map.md` keeps this lightweight skill aligned with `D4Vinci/Scrapling` and `scrapling-official`
  16. ### Installation
  17. #### 1. Install scrapling
  18. Pick whichever package manager your project uses:
  19. ```bash
  20. # pip
  21. pip install "scrapling[fetchers]"
  22. scrapling install # Install browser dependencies
  23. # uv (project)
  24. uv add "scrapling[fetchers]"
  25. uv run scrapling install
  26. # uv (global / standalone, no pyproject.toml)
  27. uv pip install "scrapling[fetchers]"
  28. scrapling install
  29. ```
  30. #### 2. Install this skill
  31. Copy the skill directory to your Claude Code skills folder:
  32. ```bash
  33. # Copy to user-level skills (available in all projects)
  34. cp -r . ~/.claude/skills/scrapling
  35. # Or copy to a specific project
  36. cp -r . /path/to/project/.claude/skills/scrapling
  37. ```
  38. ### Structure
  39. ```
  40. .
  41. ├── SKILL.md # Skill definition (entry point)
  42. ├── references/
  43. │ ├── api-quick-ref.md # Fetcher/Selector API cheat sheet
  44. │ ├── cookie-vault.md # Cookie storage template
  45. │ ├── maintenance.md # Installation & upgrade guide
  46. │ ├── security.md # Authorization, prompt-injection, cookie/token guardrails
  47. │ ├── site-patterns.md # Site-specific scraping patterns
  48. │ ├── troubleshooting.md # Error solutions
  49. │ └── upstream-map.md # Official Scrapling docs/skill sync map
  50. └── templates/
  51. ├── basic_fetch.py # Static page scraping
  52. ├── stealth_cloudflare.py # Cloudflare bypass
  53. ├── session_login.py # Login + multi-page scraping
  54. └── parse_only.py # HTML parsing without network
  55. ```
  56. ### Usage
  57. Once installed, Claude Code will automatically activate this skill when you ask it to:
  58. - Scrape or crawl a website
  59. - Extract data from a URL
  60. - Bypass Cloudflare protection
  61. - Parse HTML content
  62. - Login and scrape protected pages
  63. For simple text/Markdown extraction, prefer Scrapling CLI with `--ai-targeted` before writing Python:
  64. ```bash
  65. scrapling extract get "https://example.com/article" article.md --ai-targeted
  66. scrapling extract fetch "https://example.com/app" app.md --ai-targeted --network-idle
  67. scrapling extract stealthy-fetch "https://protected.example.com" page.md --ai-targeted --solve-cloudflare
  68. ```
  69. #### Examples
  70. ```
  71. > Scrape the title and content from https://example.com/blog
  72. > Extract all product prices from this page: https://shop.example.com
  73. > This site has Cloudflare, scrape it anyway: https://protected.example.com
  74. > I have this HTML, extract all links from it
  75. ```
  76. ### Cookie Vault
  77. The `references/cookie-vault.md` file is a **template**. For actual use:
  78. 1. Copy it to `cookie-vault.local.md` (or keep it in your local skill installation)
  79. 2. Fill in real cookie values from your browser's DevTools
  80. 3. **Never commit real cookie values to version control**
  81. ### Local overlays
  82. - Put real cookies/tokens only in `references/cookie-vault.local.md`.
  83. - Put private company site patterns, internal API details, CSRF notes, and non-public domains only in `references/site-patterns.local.md`.
  84. - Both local overlay files are ignored by `.gitignore`.
  85. - Keep public `references/site-patterns.md` generic and scrubbed of secrets.
  86. ---
  87. ## 中文
  88. 基于 [scrapling](https://github.com/D4Vinci/Scrapling) 的 [Claude Code](https://docs.anthropic.com/en/docs/claude-code) 网页抓取技能。
  89. 根据目标网站特征自动选择最佳 Fetcher;简单抽取优先用 Scrapling CLI quick path,复杂流程再生成并执行 Python 脚本。
  90. ### 功能特性
  91. - **Fetcher 决策树** — 自动选择 Fetcher、StealthyFetcher、DynamicFetcher、FetcherSession 或 Selector
  92. - **Cloudflare 绕过** — 通过 StealthyFetcher (Camoufox) 内置支持 Cloudflare/WAF 防护站点
  93. - **Session 登录** — 基于 HTTP 表单的登录,自动保持 cookie 会话
  94. - **站点模式库** — 常见站点类型的可复用抓取模式(Discourse 论坛、SPA、静态博客、API)
  95. - **Cookie 保险库** — 按站点模板存储登录 cookie
  96. - **故障排查指南** — 按错误信息索引的常见问题解决方案
  97. - **安全边界** — `references/security.md` 说明授权、Prompt injection、`--ai-targeted`、本地 cookie vault 与 redaction
  98. - **上游同步** — `references/upstream-map.md` 说明如何与 `D4Vinci/Scrapling` 和 `scrapling-official` 保持轻量对齐
  99. ### 安装
  100. #### 1. 安装 scrapling
  101. 按项目所用的包管理器选一种即可:
  102. ```bash
  103. # pip
  104. pip install "scrapling[fetchers]"
  105. scrapling install # 安装浏览器依赖
  106. # uv(项目内)
  107. uv add "scrapling[fetchers]"
  108. uv run scrapling install
  109. # uv(全局 / 独立,无 pyproject.toml)
  110. uv pip install "scrapling[fetchers]"
  111. scrapling install
  112. ```
  113. #### 2. 安装此技能
  114. 将技能目录复制到 Claude Code 的 skills 文件夹:
  115. ```bash
  116. # 复制到用户级 skills(所有项目可用)
  117. cp -r . ~/.claude/skills/scrapling
  118. # 或复制到特定项目
  119. cp -r . /path/to/project/.claude/skills/scrapling
  120. ```
  121. ### 目录结构
  122. ```
  123. .
  124. ├── SKILL.md # 技能定义(入口文件)
  125. ├── references/
  126. │ ├── api-quick-ref.md # Fetcher/Selector API 速查表
  127. │ ├── cookie-vault.md # Cookie 存储模板
  128. │ ├── maintenance.md # 安装与升级指南
  129. │ ├── security.md # 授权、prompt-injection、cookie/token 安全边界
  130. │ ├── site-patterns.md # 站点专用抓取模式
  131. │ ├── troubleshooting.md # 错误解决方案
  132. │ └── upstream-map.md # 官方 Scrapling 文档 / skill 同步地图
  133. └── templates/
  134. ├── basic_fetch.py # 静态页面抓取
  135. ├── stealth_cloudflare.py # Cloudflare 绕过
  136. ├── session_login.py # 登录 + 多页抓取
  137. └── parse_only.py # 纯 HTML 解析(无需网络)
  138. ```
  139. ### 使用方式
  140. 安装后,当你向 Claude Code 提出以下需求时,技能会自动激活:
  141. - 抓取或爬取网站
  142. - 从 URL 提取数据
  143. - 绕过 Cloudflare 防护
  144. - 解析 HTML 内容
  145. - 登录后抓取受保护页面
  146. 简单文本 / Markdown 抽取优先使用 Scrapling CLI,并默认加 `--ai-targeted`:
  147. ```bash
  148. scrapling extract get "https://example.com/article" article.md --ai-targeted
  149. scrapling extract fetch "https://example.com/app" app.md --ai-targeted --network-idle
  150. scrapling extract stealthy-fetch "https://protected.example.com" page.md --ai-targeted --solve-cloudflare
  151. ```
  152. #### 示例
  153. ```
  154. > 抓取 https://example.com/blog 的标题和正文
  155. > 提取这个页面的所有商品价格:https://shop.example.com
  156. > 这个站点有 Cloudflare,帮我绕过抓取:https://protected.example.com
  157. > 我有这段 HTML,提取里面所有链接
  158. ```
  159. ### Cookie 保险库
  160. `references/cookie-vault.md` 是一个**模板文件**。实际使用时:
  161. 1. 复制为 `cookie-vault.local.md`(或在本地 skill 安装目录中保存)
  162. 2. 从浏览器 DevTools 填入真实 cookie 值
  163. 3. **切勿将真实 cookie 值提交到版本控制**
  164. ### 本地 overlay
  165. - 真实 cookie / token 只放 `references/cookie-vault.local.md`。
  166. - 公司内站点模式、内部 API、CSRF 细节、非公开域名只放 `references/site-patterns.local.md`。
  167. - 这两个 local overlay 文件都已被 `.gitignore` 忽略。
  168. - 公共 `references/site-patterns.md` 只保留去敏感化后的通用经验。
  169. ## License
  170. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.