Skip to content
Back to search

qiaomu-markdown-proxy

"Read, fetch, extract, parse, or convert a URL or link into clean Markdown. Use FIRST whenever a user asks to read source content from a webpage, especially WeChat/微信公众号 mp.weixin.qq.com, Feishu/Lark docs, X/Twitter, PDFs, arXiv papers, or a URL that will feed a later summary, rewrite, article, podcast, or analysis. For WeChat and Feishu, prefer this specialist extractor over generic web open or generic content parsers. Exclude YouTube, pure web search, already-pasted text, and local non-PDF files."

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/joeseesun--qiaomu-markdown-proxy/badge.svg)](/listings/joeseesun--qiaomu-markdown-proxy)

HTML

<a href="/listings/joeseesun--qiaomu-markdown-proxy"><img src="/api/listings/joeseesun--qiaomu-markdown-proxy/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

npx skills add joeseesun/qiaomu-markdown-proxy

Verify:

ls ~/.agents/skills/qiaomu-markdown-proxy/SKILL.md

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 151 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: qiaomu-markdown-proxy
  3. description: "Read, fetch, extract, parse, or convert a URL or link into clean Markdown. Use FIRST whenever a user asks to read source content from a webpage, especially WeChat/微信公众号 mp.weixin.qq.com, Feishu/Lark docs, X/Twitter, PDFs, arXiv papers, or a URL that will feed a later summary, rewrite, article, podcast, or analysis. For WeChat and Feishu, prefer this specialist extractor over generic web open or generic content parsers. Exclude YouTube, pure web search, already-pasted text, and local non-PDF files."
  4. version: 2.1.0
  5. ---
  6. # Markdown Proxy - URL to Markdown
  7. 将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。
  8. ## Trigger Priority(先触发,再分流)
  9. - 用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时,优先使用本 Skill。
  10. - 用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时,先用本 Skill 取回原文,再把 Markdown 交给下游 Skill;不要因为最终任务是写作而跳过抓取。
  11. - `mp.weixin.qq.com` 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。
  12. - 用户已经贴出全文时不触发。YouTube 交给 `qiaomu-youtube-download`,普通搜索问题交给搜索工具。
  13. ## URL Routing (先判断再执行)
  14. 收到 URL 后,先判断类型,不同类型走不同通道:
  15. | URL Pattern | Route To | Reason |
  16. |-------------|----------|--------|
  17. | `mp.weixin.qq.com` | `scripts/fetch_weixin.sh` | 先代理,验证码页自动回退 Playwright |
  18. | `feishu.cn/docx/` `feishu.cn/wiki/` `larksuite.com/docx/` | `scripts/fetch_feishu.py` | 需飞书 API 认证 |
  19. | `youtube.com` `youtu.be` | `qiaomu-youtube-download` skill | YouTube 有专用工具链 |
  20. | `huggingface.co/papers/` | 提取 arXiv ID → `scripts/extract_tex.py` | HuggingFace 论文页实际是 arXiv 镜像,先找到 arXiv 链接再走 LaTeX 提取 |
  21. | `arxiv.org/abs/` `arxiv.org/pdf/` | `scripts/extract_tex.py` | 从 LaTeX 源码提取结构化内容 (章节/图表/公式) |
  22. | `.pdf` (URL or local path) | `scripts/extract_pdf.sh` | PDF 专用提取 |
  23. | All other URLs | `scripts/fetch.sh` | 代理级联自动 fallback |
  24. ## Workflow
  25. ### Step 1: Route by URL Type
  26. ```
  27. if URL contains "mp.weixin.qq.com":
  28. → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_weixin.sh "URL"
  29. → Done
  30. if URL contains "feishu.cn/docx/" or "feishu.cn/wiki/" or "larksuite.com/docx/":
  31. → python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_feishu.py "URL"
  32. → Done
  33. if URL contains "huggingface.co/papers/":
  34. → First fetch the page (WebFetch) to find the arXiv URL
  35. → Then python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py "{arxiv_url}"
  36. → Done
  37. if URL contains "arxiv.org/abs/" or "arxiv.org/pdf/":
  38. → python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py "URL"
  39. → Done
  40. if URL contains "youtube.com" or "youtu.be":
  41. → Call qiaomu-youtube-download skill
  42. → Done
  43. if URL ends with ".pdf" or is local PDF path:
  44. if remote URL:
  45. → Try: curl -sL "https://r.jina.ai/{url}"
  46. → If fails: download + extract_pdf.sh
  47. if local path:
  48. → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_pdf.sh "PATH"
  49. → Done
  50. else:
  51. → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh "URL"
  52. → Done
  53. ```
  54. ### Step 2: Display Content
  55. After fetching, show to user:
  56. ```
  57. Title: {title}
  58. Author: {author} (if available)
  59. Source: {platform} (公众号 / 飞书文档 / 网页 / PDF)
  60. URL: {original_url}
  61. Summary
  62. {3-5 sentence summary}
  63. Content
  64. {full Markdown, truncated at 200 lines if long}
  65. ```
  66. ### Step 3: Continue or Save
  67. - **Composite request**(“读取后写稿/总结/分析”):把提取结果直接交给下游任务,在同一轮继续;除非用户要求,不必额外保存源文件。
  68. - **Extraction-only request**(“只读取/转 Markdown”):保存到 `~/Downloads/{title}.md`,使用 YAML frontmatter。
  69. - Filename: use article title, remove special characters.
  70. - Format: YAML frontmatter (title, author, date, url, source) + Markdown body.
  71. - Tell the user the saved path.
  72. - Skip saving if the user says “just preview” or “don't save”.
  73. Only stop after extraction when extraction was the complete request. If the user asked for a downstream deliverable, continue to that deliverable.
  74. ## Examples
  75. ### General URL
  76. ```bash
  77. bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh "https://example.com/article"
  78. ```
  79. ### X/Twitter Post
  80. ```bash
  81. bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh "https://x.com/username/status/1234567890"
  82. ```
  83. ### WeChat Article
  84. ```bash
  85. bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_weixin.sh "https://mp.weixin.qq.com/s/abc123"
  86. ```
  87. ### Feishu Document
  88. ```bash
  89. python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_feishu.py "https://xxx.feishu.cn/docx/xxxxxxxx"
  90. ```
  91. ### arXiv LaTeX Source
  92. ```bash
  93. python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py "https://arxiv.org/abs/1706.03762"
  94. ```
  95. ### PDF (Remote)
  96. ```bash
  97. curl -sL "https://r.jina.ai/https://example.com/paper.pdf"
  98. ```
  99. ### PDF (Local)
  100. ```bash
  101. bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_pdf.sh "/path/to/paper.pdf"
  102. ```
  103. ### With Custom Proxy
  104. ```bash
  105. bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh "https://example.com" "http://127.0.0.1:7890"
  106. ```
  107. ## Notes
  108. - r.jina.ai and defuddle.md require no API key
  109. - `fetch.sh` handles proxy cascade with automatic fallback
  110. - Content validation: filters error, login-wall, and WeChat verification pages; requires >5 lines
  111. - WeChat wrapper prefers dependency-free proxies, then uses local Playwright; when Python packages are missing and `uv` is available, it runs them in an isolated environment
  112. - Playwright fallback requires a Chromium runtime; install once with `python3 -m playwright install chromium` if absent
  113. - Feishu script requires: `FEISHU_APP_ID` + `FEISHU_APP_SECRET` env vars
  114. - PDF extraction tries: marker-pdf → pdftotext → pypdf
  115. - For detailed method documentation, see `references/methods.md`
README.md · README · 246 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # qiaomu-markdown-proxy
  2. > 给 Agent 一个链接,它会先选对读取通道,再把完整原文交给后续的总结、写作、翻译或分析任务。
  3. 微信、飞书、X/Twitter、PDF、arXiv 和普通网页走不同的专用路线。遇到公众号验证码页会自动回退到浏览器提取,不会把“环境异常”误当正文。
  4. **[English](#english) | [中文](#中文)**
  5. ---
  6. <!-- qiaomu-profile:start -->
  7. ## 关于向阳乔木
  8. 向阳乔木(乔向阳 / Joe)是一位实践型 AI 产品与内容创作者,长期把前沿 AI 变化转译成可复用的工作流、产品判断、AI 编程实践、AI 搜索实践和 GEO/AI 营销方法。
  9. - 个人网站: https://qiaomu.ai
  10. - 博客: https://blog.qiaomu.ai
  11. - X: https://x.com/vista8
  12. - GitHub: https://github.com/joeseesun/
  13. - 微信公众号: 向阳乔木推荐看
  14. ### 支持与关注
  15. | 打赏支持 | 微信公众号 |
  16. |---|---|
  17. | <img src="assets/qiaomu-profile/qiaomu_reward_qr.png" alt="向阳乔木打赏二维码" width="180" /> | <img src="assets/qiaomu-profile/qiaomu_wechat_public_account_qr.jpg" alt="向阳乔木推荐看公众号二维码" width="180" /> |
  18. | 感谢支持乔木持续分享 AI 实践 | 扫码关注「向阳乔木推荐看」 |
  19. <!-- qiaomu-profile:end -->
  20. <a name="english"></a>
  21. ## English
  22. ### Features
  23. Send any URL to Claude, and it automatically fetches the full content as Markdown. Six content types have dedicated extraction:
  24. | URL Type | Method | Why |
  25. |----------|--------|-----|
  26. | WeChat Articles (`mp.weixin.qq.com`) | Proxy-first wrapper with Playwright fallback | Rejects verification walls before returning content |
  27. | Feishu/Lark Docs (`feishu.cn`, `larksuite.com`) | Built-in Feishu API script | Requires API authentication, auto-converts to Markdown |
  28. | YouTube | Dedicated YouTube skill | Video content has its own toolchain |
  29. | arXiv / Hugging Face Papers | Built-in LaTeX source extractor | Preserves sections, equations, figures, and tables |
  30. | PDF (remote or local) | Built-in PDF extraction (`extract_pdf.sh`) | Three-method cascade: marker-pdf → pdftotext → pypdf |
  31. | All other URLs | Proxy cascade via `fetch.sh`: r.jina.ai → defuddle.md → agent-fetch | Free, no API key, content validation built-in |
  32. ### Prerequisites
  33. - [ ] [Claude Code](https://docs.anthropic.com/en/docs/claude-code) installed
  34. - [ ] **curl** (built-in on macOS/Linux)
  35. - [ ] (Optional - WeChat scraping) Python 3.8+ with playwright
  36. ```bash
  37. python3 -m pip install playwright beautifulsoup4 lxml
  38. python3 -m playwright install chromium
  39. ```
  40. - [ ] (Optional - PDF extraction) One of:
  41. - **marker-pdf** (best quality): `pip install marker-pdf`
  42. - **pdftotext** (fast): `brew install poppler`
  43. - **pypdf** (fallback): `pip install pypdf`
  44. - [ ] (Optional - Proxy fallback) [agent-fetch](https://github.com/teng-lin/agent-fetch)
  45. ```bash
  46. npx agent-fetch --help # No pre-install needed, npx auto-downloads
  47. ```
  48. - [ ] (Optional - Feishu docs) Environment variables `FEISHU_APP_ID` and `FEISHU_APP_SECRET`
  49. ```bash
  50. echo $FEISHU_APP_ID # Verify configured
  51. ```
  52. ### Installation
  53. ```bash
  54. npx skills add joeseesun/qiaomu-markdown-proxy
  55. ```
  56. Verify:
  57. ```bash
  58. ls ~/.agents/skills/qiaomu-markdown-proxy/SKILL.md
  59. ```
  60. ### Usage
  61. Just send Claude a URL:
  62. - "Read this article: https://example.com/post"
  63. - "Fetch this tweet: https://x.com/user/status/123456"
  64. - "Read this WeChat article: https://mp.weixin.qq.com/s/abc123"
  65. - "Convert this Feishu doc to Markdown: https://xxx.feishu.cn/docx/xxxxxxxx"
  66. - "Read this WeChat post, then rewrite it in first person: https://mp.weixin.qq.com/s/abc123"
  67. ### Proxy Priority
  68. 1. **r.jina.ai** — Most complete content, preserves image links
  69. 2. **defuddle.md** — Cleaner output with YAML frontmatter
  70. 3. **[agent-fetch](https://github.com/teng-lin/agent-fetch)** — Local tool, no network proxy needed
  71. 4. **defuddle CLI** — Local CLI, good for standard web pages
  72. ### Feishu/Lark Document Support
  73. Built-in `fetch_feishu.py` script fetches documents via Feishu Open API and auto-converts to Markdown:
  74. - Supports new docs (docx), legacy docs (doc), and wiki pages
  75. - Auto-parses document blocks into Markdown format
  76. - Supports headings, lists, code blocks, quotes, todos, equations, images, etc.
  77. - Requires `FEISHU_APP_ID` and `FEISHU_APP_SECRET` environment variables
  78. - App needs `docx:document:readonly` permission
  79. ### Troubleshooting
  80. | Issue | Solution |
  81. |-------|----------|
  82. | WeChat scraping fails | Run `playwright install chromium` to install browser |
  83. | WeChat returns `环境异常` | Use `scripts/fetch_weixin.sh`; it rejects verification pages and falls back to Playwright |
  84. | Python packages are missing | Install `uv`, or run `python3 -m pip install playwright beautifulsoup4 lxml` |
  85. | Feishu returns permission error | Check `FEISHU_APP_ID` and `FEISHU_APP_SECRET` env vars, confirm app has document read permission |
  86. | Feishu wiki page fails | Confirm app has `wiki:wiki:readonly` permission |
  87. | r.jina.ai returns empty | Auto-falls back to defuddle.md (no action needed) |
  88. | All proxies fail | URL may have strict auth restrictions, try `npx agent-fetch` |
  89. ### Credits
  90. - [r.jina.ai](https://r.jina.ai) — Free URL-to-Markdown proxy by Jina AI
  91. - [defuddle.md](https://defuddle.md) — Clean article extraction service
  92. - [agent-fetch](https://github.com/teng-lin/agent-fetch) — Local URL content extraction tool
  93. - [Playwright](https://playwright.dev/) — Browser automation for WeChat scraping
  94. - [Feishu Open Platform](https://open.feishu.cn/) — Feishu Document API
  95. ---
  96. <a name="中文"></a>
  97. ## 中文
  98. ### 功能
  99. 给 Claude 发一个 URL,自动抓取完整内容并转为 Markdown。支持六种内容类型的专用抓取:
  100. 如果最终目标是“读取后写稿、总结、翻译或分析”,本 Skill 负责先拿到可信原文,再把内容交给下游任务,同一轮继续完成,不会停在抓取结果。
  101. | URL 类型 | 抓取方式 | 原因 |
  102. |----------|---------|------|
  103. | 微信公众号 (`mp.weixin.qq.com`) | 代理优先、Playwright 回退的专用入口 | 验证码页不会被误判为正文 |
  104. | 飞书文档 (`feishu.cn/docx/`, `/wiki/`, `/docs/`) | 内置飞书 API 脚本 | 需要 API 认证,自动转 Markdown |
  105. | YouTube | 专用 YouTube skill | 视频内容有专用工具链 |
  106. | arXiv / Hugging Face 论文 | 内置 LaTeX 源码提取 | 保留章节、公式、图片和表格 |
  107. | PDF(远程 URL 或本地文件) | 内置 PDF 提取(`extract_pdf.sh`) | 三级 fallback:marker-pdf → pdftotext → pypdf |
  108. | 其他所有 URL | 代理级联 `fetch.sh`:r.jina.ai → defuddle.md → agent-fetch | 免费、无需 API key、内置内容验证 |
  109. ### 前置条件
  110. - [ ] 已安装 [Claude Code](https://docs.anthropic.com/en/docs/claude-code)
  111. - [ ] **curl**(macOS/Linux 自带)
  112. - [ ] (可选 - 公众号抓取)Python 3.8+ 及 playwright
  113. ```bash
  114. python3 -m pip install playwright beautifulsoup4 lxml
  115. python3 -m playwright install chromium
  116. ```
  117. - [ ] (可选 - PDF 提取)以下任一:
  118. - **marker-pdf**(最佳质量):`pip install marker-pdf`
  119. - **pdftotext**(速度快):`brew install poppler`
  120. - **pypdf**(兜底):`pip install pypdf`
  121. - [ ] (可选 - 代理降级)[agent-fetch](https://github.com/teng-lin/agent-fetch)
  122. ```bash
  123. npx agent-fetch --help # 无需预装,npx 自动下载
  124. ```
  125. - [ ] (可选 - 飞书抓取)环境变量 `FEISHU_APP_ID` 和 `FEISHU_APP_SECRET`
  126. ```bash
  127. echo $FEISHU_APP_ID # 验证已配置
  128. ```
  129. ### 安装
  130. ```bash
  131. npx skills add joeseesun/qiaomu-markdown-proxy
  132. ```
  133. 验证:
  134. ```bash
  135. ls ~/.agents/skills/qiaomu-markdown-proxy/SKILL.md
  136. ```
  137. ### 你可以直接这样说
  138. 直接给 Claude 发 URL:
  139. - "帮我读一下这篇文章:https://example.com/post"
  140. - "抓取这条推文:https://x.com/user/status/123456"
  141. - "读一下这篇公众号:https://mp.weixin.qq.com/s/abc123"
  142. - "读取这篇公众号,然后用第一人称重写:https://mp.weixin.qq.com/s/abc123"
  143. - "把这个飞书文档转成 Markdown:https://xxx.feishu.cn/docx/xxxxxxxx"
  144. - "读一下这个飞书知识库页面:https://xxx.feishu.cn/wiki/xxxxxxxx"
  145. - "提取这个 PDF:https://example.com/paper.pdf"
  146. - "转换本地 PDF:/path/to/document.pdf"
  147. 维护者验证:
  148. ```bash
  149. python3 ~/.agents/skills/qiaomu-meta-skill/scripts/validate_skill.py ~/.agents/skills/qiaomu-markdown-proxy
  150. python3 -m unittest discover -s ~/.agents/skills/qiaomu-markdown-proxy/tests -p 'test_*.py'
  151. ```
  152. ### 代理优先级
  153. 1. **r.jina.ai** — 内容最完整,保留图片链接
  154. 2. **defuddle.md** — 输出更干净,带 YAML frontmatter
  155. 3. **[agent-fetch](https://github.com/teng-lin/agent-fetch)** — 本地工具,无需网络代理
  156. 4. **defuddle CLI** — 本地 CLI,适合普通网页
  157. ### 飞书文档支持
  158. 内置 `fetch_feishu.py` 脚本,通过飞书开放 API 抓取文档内容并自动转为 Markdown:
  159. - 支持新版文档(docx)、旧版文档(doc)、知识库页面(wiki)
  160. - 自动解析文档 blocks 并转换为 Markdown 格式
  161. - 支持标题、列表、代码块、引用、待办、公式、图片等
  162. - 需要飞书应用的 `FEISHU_APP_ID` 和 `FEISHU_APP_SECRET` 环境变量
  163. - 应用需要 `docx:document:readonly` 权限
  164. ### 常见问题
  165. | 问题 | 解决方法 |
  166. |------|----------|
  167. | 公众号返回“环境异常” | 使用 `scripts/fetch_weixin.sh`,它会识别验证页并回退 Playwright |
  168. | 公众号浏览器抓取失败 | 运行 `python3 -m playwright install chromium` 安装浏览器 |
  169. | 缺少 Python 依赖 | 安装 `uv`,或运行 `python3 -m pip install playwright beautifulsoup4 lxml` |
  170. | 飞书文档返回权限错误 | 检查 `FEISHU_APP_ID` 和 `FEISHU_APP_SECRET` 环境变量,确认应用有文档读取权限 |
  171. | 飞书知识库页面抓取失败 | 确认应用有 `wiki:wiki:readonly` 权限 |
  172. | PDF 提取失败 | 安装任一工具:`pip install marker-pdf`、`brew install poppler`、`pip install pypdf` |
  173. | r.jina.ai 返回空内容 | 自动降级到 defuddle.md(无需手动操作) |
  174. | 所有代理都失败 | URL 可能有严格认证限制,尝试 `npx agent-fetch` |
  175. ### 致谢
  176. - [r.jina.ai](https://r.jina.ai) — Jina AI 提供的免费 URL 转 Markdown 代理
  177. - [defuddle.md](https://defuddle.md) — 干净的文章提取服务
  178. - [agent-fetch](https://github.com/teng-lin/agent-fetch) — 本地 URL 内容提取工具
  179. - [Playwright](https://playwright.dev/) — 微信公众号抓取的浏览器自动化
  180. - [飞书开放平台](https://open.feishu.cn/) — 飞书文档 API
  181. ## 许可与边界
  182. - 抓取会访问用户提供的外部 URL,请遵守目标网站的访问规则与内容版权。
  183. - 飞书读取需要用户自己的应用凭据,仓库不会保存任何 token 或 cookie。
  184. - 默认只读取内容;保存源文件、后续发布或账号操作由用户请求和下游工具决定。
  185. ## License
  186. MIT. Copyright (c) 向阳乔木.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.