Skip to content
Back to search

web-content-fetcher

Extract article content from any URL as clean Markdown. Uses Scrapling script as primary method (with auto fast→stealth fallback), Jina Reader as alternative for simple pages. Preserves headings, links, images, lists, and code blocks. Use this skill whenever the user wants to fetch, read, extract, scrape, or summarize content from a URL — including blog posts, news articles, WeChat articles (微信公众号), documentation pages, or any web page. Also trigger when the user says things like "帮我读一下这篇文章", "抓取这个网页", "提取正文", or "read this page for me".

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/shirenchuang-web-content-fetcher/badge.svg)](/listings/shirenchuang-web-content-fetcher)

HTML

<a href="/listings/shirenchuang-web-content-fetcher"><img src="/api/listings/shirenchuang-web-content-fetcher/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

方式一:一键安装(推荐)

访问 [skills.sh](https://skills.sh/shirenchuang/web-content-fetcher/web-content-fetcher) 页面,按提示一键安装。

方式二:命令行安装

npx skills add https://github.com/shirenchuang/web-content-fetcher --skill web-content-fetcher

方式三:手动安装

# Clone
git clone https://github.com/shirenchuang/web-content-fetcher.git

# Copy to Claude Code skills directory
cp -r web-content-fetcher ~/.claude/skills/

安装 Python 依赖

pip install scrapling html2text

> **注意**:在系统管理的 Python (macOS/Linux) 上,加 `--break-system-packages` 或使用 venv。

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 94 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. ---
  2. name: web-content-fetcher
  3. description: >
  4. Extract article content from any URL as clean Markdown.
  5. Uses Scrapling script as primary method (with auto fast→stealth fallback),
  6. Jina Reader as alternative for simple pages.
  7. Preserves headings, links, images, lists, and code blocks.
  8. Use this skill whenever the user wants to fetch, read, extract, scrape, or summarize
  9. content from a URL — including blog posts, news articles, WeChat articles (微信公众号),
  10. documentation pages, or any web page. Also trigger when the user says things like
  11. "帮我读一下这篇文章", "抓取这个网页", "提取正文", or "read this page for me".
  12. ---
  13. # Web Content Fetcher
  14. Given a URL, return its main content as clean Markdown — headings, links, images, lists, code blocks all preserved.
  15. ## Extraction Strategy
  16. Always try **one method per URL** — don't cascade blindly. Pick the right one upfront.
  17. ```
  18. URL
  19. │
  20. ├─ 1. Scrapling script (preferred)
  21. │ Run fetch.py — check the domain routing table to decide fast vs --stealth.
  22. │ Works for most sites. Returns clean Markdown directly.
  23. │
  24. └─ 2. Jina Reader (fallback — only if Scrapling fails or dependencies not installed)
  25. web_fetch("https://r.jina.ai/<url>")
  26. Free tier: 200 req/day. Fast (~1-2s), good Markdown output.
  27. Does NOT work for: WeChat (403), some Chinese platforms.
  28. ```
  29. ### Scrapling script
  30. ```bash
  31. python3 <SKILL_DIR>/scripts/fetch.py "<url>" [max_chars] [--stealth]
  32. ```
  33. `<SKILL_DIR>` is the directory where this SKILL.md lives. Resolve it before calling the script.
  34. The script has two modes built in:
  35. - **Default (fast):** HTTP fetch, ~1-3s, works for most sites
  36. - **`--stealth`:** Headless browser, ~5-15s, for JS-rendered or anti-scraping sites
  37. When run without `--stealth`, the script automatically falls back to stealth if the fast result has too little content. So you rarely need to specify `--stealth` manually — the only reason to force it is when you already know the site needs it (see routing table), which saves the initial fast attempt.
  38. ## Domain Routing
  39. Use this table to pick the right mode on the first call:
  40. | Domain | Command | Why |
  41. |--------|---------|-----|
  42. | `mp.weixin.qq.com` | `fetch.py <url> --stealth` | JS-rendered content |
  43. | `zhuanlan.zhihu.com` | `fetch.py <url> --stealth` | Anti-scraping + JS |
  44. | `juejin.cn` | `fetch.py <url> --stealth` | JS-rendered SPA |
  45. | `sspai.com` | `fetch.py <url>` | Static HTML |
  46. | `blog.csdn.net` | `fetch.py <url>` | Static HTML |
  47. | `ruanyifeng.com` | `fetch.py <url>` | Static blog |
  48. | `openai.com` | `fetch.py <url>` | Static HTML |
  49. | `blog.google` | `fetch.py <url>` | Static HTML |
  50. | Everything else | `fetch.py <url>` | Auto-fallback handles it |
  51. ## Script Options
  52. ```bash
  53. # Basic — auto-selects fast or stealth
  54. python3 <SKILL_DIR>/scripts/fetch.py "https://sspai.com/post/73145"
  55. # Force stealth for known JS-heavy sites
  56. python3 <SKILL_DIR>/scripts/fetch.py "https://mp.weixin.qq.com/s/xxx" --stealth
  57. # Limit output to 15000 characters (default: 30000)
  58. python3 <SKILL_DIR>/scripts/fetch.py "https://example.com/article" 15000
  59. # JSON output with metadata (url, mode, selector, content_length)
  60. python3 <SKILL_DIR>/scripts/fetch.py "https://example.com" --json
  61. ```
  62. ## Install Dependencies
  63. First use only — the script checks and tells you if anything is missing:
  64. ```bash
  65. pip install scrapling html2text
  66. ```
  67. If on system-managed Python (macOS/Linux), add `--break-system-packages` or use a venv.
  68. ## Failure Rules
  69. - Same URL fails once → give up, tell the user "unable to extract content from this URL"
  70. - Do not retry — each failed call wastes context tokens
README.md · README · 214 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. # Web Content Fetcher
  3. **网页正文提取 · 永久免费 · 支持微信公众号**
  4. [![Python](https://img.shields.io/badge/Python-3.10%2B-blue?logo=python&logoColor=white)](https://www.python.org/)
  5. [![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
  6. </div>
  7. ---
  8. ## 简介
  9. Web Content Fetcher 是一个轻量级的网页正文提取工具,能够自动将任意网页转换为干净的 Markdown 格式,保留标题、链接、图片和列表结构。
  10. **核心优势:**
  11. - Scrapling 优先提取,内置 fast / stealth 双模式,自动降级
  12. - Jina Reader 作为二级备选
  13. - 完美支持微信公众号、掘金、CSDN 等国内平台
  14. - 返回标准 Markdown 格式,便于后续处理
  15. - 零配置,开箱即用
  16. ---
  17. ## 安装
  18. ### 方式一:一键安装(推荐)
  19. 访问 [skills.sh](https://skills.sh/shirenchuang/web-content-fetcher/web-content-fetcher) 页面,按提示一键安装。
  20. ### 方式二:命令行安装
  21. ```bash
  22. npx skills add https://github.com/shirenchuang/web-content-fetcher --skill web-content-fetcher
  23. ```
  24. ### 方式三:手动安装
  25. ```bash
  26. # Clone
  27. git clone https://github.com/shirenchuang/web-content-fetcher.git
  28. # Copy to Claude Code skills directory
  29. cp -r web-content-fetcher ~/.claude/skills/
  30. ```
  31. ### 安装 Python 依赖
  32. ```bash
  33. pip install scrapling html2text
  34. ```
  35. > **注意**:在系统管理的 Python (macOS/Linux) 上,加 `--break-system-packages` 或使用 venv。
  36. ---
  37. ## 使用方式
  38. ### 在 Claude Code 中使用
  39. 直接告诉 AI 你要读取的 URL,会自动选择最佳方案:
  40. ```
  41. 帮我读取这篇文章:https://mp.weixin.qq.com/s/EwVItQH4JUsONqv_Fmi4wQ
  42. Extract the content from https://openai.com/blog/gpt-4o
  43. ```
  44. ### 命令行单独使用
  45. ```bash
  46. # 基础用法(自动选择 fast 或 stealth 模式)
  47. python3 scripts/fetch.py https://sspai.com/post/73145
  48. # 强制 stealth 模式(用于 JS 渲染页面)
  49. python3 scripts/fetch.py https://mp.weixin.qq.com/s/xxx --stealth
  50. # 限制输出字符数(默认 30000)
  51. python3 scripts/fetch.py https://example.com/article 15000
  52. # JSON 输出(含 url, mode, selector, content_length)
  53. python3 scripts/fetch.py https://example.com --json
  54. # 输出到文件
  55. python3 scripts/fetch.py https://example.com/article > output.md
  56. ```
  57. ---
  58. ## 提取策略
  59. ```
  60. URL 输入
  61. │
  62. ▼
  63. ┌─────────────────────────────────────┐
  64. │ 1. Scrapling(首选) │
  65. │ · fast 模式:~1-3s,大部分网站 │
  66. │ · stealth 模式:~5-15s,JS 渲染 │
  67. │ · 内容太少时自动 fast → stealth │
  68. └─────────────────────────────────────┘
  69. │ 失败 / 未安装依赖
  70. ▼
  71. ┌─────────────────────────────────────┐
  72. │ 2. Jina Reader(备选) │
  73. │ · 速度快(~1-2s),格式干净 │
  74. │ · 免费额度:200次/天 │
  75. │ · 不支持:微信公众号、部分国内站 │
  76. └─────────────────────────────────────┘
  77. ```
  78. ### 域名路由
  79. | 域名 | 模式 | 说明 |
  80. |------|------|------|
  81. | `mp.weixin.qq.com` | `--stealth` | JS 渲染内容 |
  82. | `zhuanlan.zhihu.com` | `--stealth` | 反爬 + JS |
  83. | `juejin.cn` | `--stealth` | JS 渲染 SPA |
  84. | `sspai.com` | fast | 静态 HTML |
  85. | `blog.csdn.net` | fast | 静态 HTML |
  86. | 其他 | fast | 自动降级 |
  87. ---
  88. ## 支持平台
  89. ### 国内平台
  90. | 平台 | 模式 | 状态 | 说明 |
  91. |------|------|:----:|------|
  92. | 微信公众号 (mp.weixin.qq.com) | fast | ✅ | 正文完整提取 |
  93. | 掘金 (juejin.cn) | stealth (auto) | ✅ | 自动降级到 stealth |
  94. | CSDN (blog.csdn.net) | fast | ✅ | 正文精准提取 |
  95. | 少数派 (sspai.com) | fast | ✅ | article 选择器命中 |
  96. | 博客园 (cnblogs.com) | fast | ✅ | 文章列表和正文 |
  97. | 知乎 (zhihu.com) | stealth | ✅ | 需有效 URL |
  98. | 36氪 (36kr.com) | fast | ✅ | 需有效文章 URL |
  99. | 今日头条 (toutiao.com) | stealth | ✅ | JS 渲染,需有效文章 URL |
  100. | InfoQ 中文 (infoq.cn) | stealth (auto) | ✅ | 需有效文章 URL |
  101. | 网易 (163.com) | fast | ✅ | 需有效文章 URL |
  102. | 小红书 | - | ❌ | 需登录态 |
  103. ### 海外平台
  104. | 平台 | 模式 | 状态 | 说明 |
  105. |------|------|:----:|------|
  106. | OpenAI Blog | fast | ✅ | article 选择器命中 |
  107. | Google Blog | fast | ✅ | article 选择器命中 |
  108. | Nature | fast | ✅ | 论文摘要完整 |
  109. | arXiv | fast | ✅ | 标题/作者/摘要 |
  110. | GitHub | fast | ✅ | README 完整提取 |
  111. | Next.js Blog | fast | ✅ | article 选择器命中 |
  112. | React Docs (react.dev) | fast | ✅ | 文档正文清晰 |
  113. | MDN Web Docs | fast | ✅ | main 选择器命中 |
  114. | Python Docs | fast | ✅ | 目录和正文 |
  115. | Paul Graham Essays | fast | ✅ | 经典静态页 |
  116. | 阮一峰博客 | fast | ✅ | 周刊完整提取 |
  117. | Claude Code Docs | fast | ✅ | 文档正文 |
  118. | Product Hunt | stealth | ⚠️ | Cloudflare 验证拦截 |
  119. | more... | | | |
  120. ---
  121. ## 输出格式
  122. 返回标准 Markdown,自动保留:
  123. - **标题层级**:`# ## ###`
  124. - **超链接**:`[文字](url)`
  125. - **图片**:`![alt](url)`(data-src 懒加载自动处理)
  126. - **列表、代码块、引用块**
  127. ---
  128. ## 相关项目
  129. ### [Kuaifa(快发)](https://github.com/shirenchuang/kuaifa) — 公众号一键排版发布
  130. 如果你需要将 Markdown 文章发布到微信公众号,推荐使用 **Kuaifa**:
  131. - 一键 Markdown 排版,支持多种主题
  132. - 自动上传图片到 CDN
  133. - 一键创建公众号草稿
  134. - 支持预览和发布
  135. ```bash
  136. pip install kuaifa
  137. kuaifa publish your-article.md
  138. ```
  139. ---
  140. ## 作者
  141. <div align="center">
  142. **石臻说AI**
  143. AI科技博主 · 10+年大厂AI提效专家
  144. 专注于个人提效、超级个体、AI 资讯
  145. <img src="qrcode_for_shizhen.jpg" width="200" alt="公众号二维码"/>
  146. *扫码关注公众号*
  147. </div>
  148. ---
  149. ## License
  150. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.