Skip to content
Back to search

digcrawl

Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/originaleric--digcrawl/badge.svg)](/listings/originaleric--digcrawl)

HTML

<a href="/listings/originaleric--digcrawl"><img src="/api/listings/originaleric--digcrawl/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

**Install with npx** (macOS / Linux / Windows; Node.js 20.19+):

npx digcrawl-skill install

Installs the program, complete Skill, and default profile. Configures Codex and installs the Claude Code Skill when detected.

**Or use the install script** (macOS / Linux; no Node.js or Go required):

curl -fsSL https://raw.githubusercontent.com/originaleric/digcrawl/main/scripts/install.sh | sh

Both methods verify downloads. Restart your agent and open a new chat after installation.

<details> <summary>Choose an agent, install locations, and alternatives</summary>

npx digcrawl-skill install codex
npx digcrawl-skill install claude-code
npx digcrawl-skill status

- Configures Codex automatically and installs the Claude Code Skill when detected. See [MCP setup](./.agents/skills/digcrawl/mcp/README.md) for other hosts. - Installs to `~/.local/bin`; add it to `PATH` if the command is not found. - Windows users can also download a [Release](https://github.com/originaleric/digcrawl/releases). See [installation](./USAGE.md#1-安装) for version pinning and source builds.

</details>

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 418 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <h1 align="center">DigCrawl</h1>
  2. <p align="center"><strong>Multi-platform content retrieval for AI agents.</strong></p>
  3. <p align="center">Retrieve pages, text, images, and video from public and user-authorized signed-in sources.</p>
  4. <p align="center">
  5. <a href="./README.zh-CN.md">中文</a> ·
  6. <a href="#quick-start">Quick start</a> ·
  7. <a href="#supported-sources">Sources</a> ·
  8. <a href="#collect-export-and-watch">Collect &amp; watch</a> ·
  9. <a href="#documentation">Docs</a> ·
  10. <a href="https://github.com/originaleric/digcrawl/releases">Releases</a>
  11. </p>
  12. <p align="center">
  13. <a href="./LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-blue.svg"></a>
  14. <img alt="Local-first" src="https://img.shields.io/badge/runtime-local--first-111111">
  15. <img alt="CLI, MCP, HTTP" src="https://img.shields.io/badge/interfaces-CLI%20%7C%20MCP%20%7C%20HTTP-0071e3">
  16. </p>
  17. <p align="center">
  18. <img src="./docs/assets/readme/hero.en.svg" width="100%" alt="DigCrawl retrieves pages, posts, images and video for your agent using user-authorized sessions">
  19. </p>
  20. ## DigCrawl Skill: Run It in Your Agent
  21. **Paste into your agent chat** (Codex, Grok Build, Claude Code, or similar):
  22. ```text
  23. Install the DigCrawl Skill for me: https://github.com/originaleric/digcrawl
  24. ```
  25. Follow the instructions to restart your agent or open a new chat, then send a link:
  26. ```text
  27. Read and summarize this link: <URL>
  28. ```
  29. ## Quick Start
  30. With Node.js 20.19+:
  31. ```bash
  32. npx digcrawl-skill@latest install
  33. ```
  34. Or install with the shell script on macOS / Linux:
  35. **Install in your terminal** (macOS / Linux, no Go required):
  36. ```bash
  37. curl -fsSL https://raw.githubusercontent.com/originaleric/digcrawl/main/scripts/install.sh | sh
  38. ```
  39. The installer verifies downloads and installs the program, complete Skill, and default profile.
  40. <details>
  41. <summary>Install location, other hosts, and Windows</summary>
  42. - Configures Codex automatically and installs the Claude Code Skill when detected. See [MCP setup](./.agents/skills/digcrawl/mcp/README.md) for other hosts.
  43. - Installs to `~/.local/bin`; add it to `PATH` if the command is not found.
  44. - For Windows, download a [Release](https://github.com/originaleric/digcrawl/releases). See [installation](./USAGE.md#1-installation) for version pinning and source builds.
  45. </details>
  46. ## What You Can Do
  47. | Use case | Capability |
  48. | --- | --- |
  49. | Content retrieval | Read pages, articles, posts, and source images |
  50. | Video understanding | Retrieve available captions; optionally transcribe speech and recognize on-screen text |
  51. | Ongoing collection | Process URL batches, monitor updates, and notify |
  52. | Agent reading | Supply source content for summaries, comparisons, and research |
  53. ## Supported Sources
  54. - **Posts and communities**: WeChat Official Accounts, Xiaohongshu, Weibo, Zhihu, X, Reddit, Threads, Instagram.
  55. - **Video and content platforms**: Douyin, Toutiao, Bilibili, Kuaishou, YouTube.
  56. - **Web and feeds**: Web pages, RSS, Atom, sitemaps, platform lists, and profiles.
  57. See [source details](./USAGE.md#supported-content-sources) for platform capabilities and login requirements.
  58. ## Collect, Export, and Watch
  59. Save retrieved content as Markdown, HTML, JSON, and optional media files, or monitor sources continuously with Watchlist.
  60. <details>
  61. <summary>Common CLI commands: pages, batches, video, and watchlists</summary>
  62. ```bash
  63. # Single pages and batches
  64. digcrawl collect "https://mp.weixin.qq.com/s/..." --format md,html,json
  65. digcrawl collect --file urls.txt --out ./collect-output
  66. # Analyze local video using the configured processors
  67. digcrawl video ./meeting.mp4
  68. # Run once or monitor continuously
  69. digcrawl watchlist run --once --config config/config.yaml
  70. digcrawl watchlist serve --config config/config.yaml
  71. ```
  72. Transcription and OCR require the relevant dependencies; see [video analysis](./USAGE.md#direct-video-analysis) and [Collect](./collect/README.md).
  73. </details>
  74. ## Use with AI Agents
  75. Original pages beside actual retrieval results (2026-10-17). WeChat uses a source excerpt; video examples did not run speech transcription.
  76. ### Article · WeChat
  77. ```text
  78. Read this agent article and extract its text and sections. https://mp.weixin.qq.com/s/mbU0bE6BV5YfAsTmoY4H3A
  79. ```
  80. <p align="center">
  81. <img src="./docs/assets/readme/examples/wechat.en.svg?v=c65d6772" width="100%" alt="WeChat: success">
  82. </p>
  83. [Source](https://mp.weixin.qq.com/s/mbU0bE6BV5YfAsTmoY4H3A) · Retrieved the article text, sections, and 7 image references.
  84. <details>
  85. <summary>View returned field excerpts</summary>
  86. ```json
  87. {
  88. "status": "success",
  89. "title": "别再只卷Prompt 了,真正拉开Agent差距的是Context Engineering",
  90. "author": "腾讯技术工程",
  91. "media_references": {
  92. "image": 7
  93. },
  94. "excerpt": "Prompt Engineering → ReAct → Context Engineering\nContext 的七类构成要素\n长任务的三大核心技术"
  95. }
  96. ```
  97. [Verification data](./docs/assets/readme/examples/wechat.json)
  98. </details>
  99. ### Image Note · Xiaohongshu
  100. ```text
  101. Read this agent image note and extract its text and images. <paste the full Xiaohongshu share link>
  102. ```
  103. <p align="center">
  104. <img src="./docs/assets/readme/examples/xiaohongshu.en.svg?v=63ceefe9" width="100%" alt="Xiaohongshu: success">
  105. </p>
  106. [Source](https://www.xiaohongshu.com/explore/6ac20976000000001c00ca12) · [Original screenshot](./docs/assets/readme/examples/originals/xiaohongshu.jpg) · Used an authorized local session to retrieve the note text and 2 image references. Use the full share link.
  107. <details>
  108. <summary>View returned field excerpts</summary>
  109. ```json
  110. {
  111. "status": "success",
  112. "title": "2026 Agent新物种地图|20款近期高热 Agent",
  113. "author": "略懂AI的Asher",
  114. "media_references": {
  115. "image": 2
  116. },
  117. "excerpt": "能力范式转向;组织形态重组;负责周期跨越",
  118. "image_observation": [
  119. "Personal",
  120. "Coding",
  121. "Workflow",
  122. "Research",
  123. "Creative"
  124. ]
  125. }
  126. ```
  127. [Verification data](./docs/assets/readme/examples/xiaohongshu.json)
  128. </details>
  129. ### Video
  130. #### Bilibili · Agent Fundamentals
  131. ```text
  132. Read this agent tutorial and extract its title, description, and parts. https://www.bilibili.com/video/BV1z3ZFYUE6K/
  133. ```
  134. <p align="center">
  135. <img src="./docs/assets/readme/examples/bilibili.en.svg?v=bb1ce02b" width="100%" alt="Bilibili: success">
  136. </p>
  137. [Source](https://www.bilibili.com/video/BV1z3ZFYUE6K/) · [Original screenshot](./docs/assets/readme/examples/originals/bilibili.jpg) · Retrieved video metadata and media references; transcription was not requested.
  138. <details>
  139. <summary>View returned field excerpts</summary>
  140. ```json
  141. {
  142. "status": "success",
  143. "title": "李宏毅:一堂课搞懂AI Agent原理【生成式AI的机器学习(2025)】大模型LLM",
  144. "author": "AI算法工程师笔记-",
  145. "media_references": {
  146. "image": 1,
  147. "video": 2
  148. },
  149. "excerpt": "分P1: 第二讲:一堂课搞懂AI Agent原理\n分P2: 大模型路线图-\n分P4: ML2025Agent"
  150. }
  151. ```
  152. [Verification data](./docs/assets/readme/examples/bilibili.json)
  153. </details>
  154. #### X · Operator Demo
  155. ```text
  156. Read this Operator video post and extract its text and video information. https://x.com/OpenAI/status/1882509286439637448
  157. ```
  158. <p align="center">
  159. <img src="./docs/assets/readme/examples/x.en.svg?v=02a9c970" width="100%" alt="X: success">
  160. </p>
  161. [Source](https://x.com/OpenAI/status/1882509286439637448) · [Original screenshot](./docs/assets/readme/examples/originals/x.jpg) · Retrieved the post text, video reference, and cover reference; no transcription was requested.
  162. <details>
  163. <summary>View returned field excerpts</summary>
  164. ```json
  165. {
  166. "status": "success",
  167. "title": "A research preview of Operator, an agent that can use its own browser to perform...",
  168. "author": "OpenAI",
  169. "media_references": {
  170. "video": 1,
  171. "image": 1
  172. },
  173. "excerpt": "A research preview of Operator, an agent that can use its own browser to perform tasks for you."
  174. }
  175. ```
  176. [Verification data](./docs/assets/readme/examples/x.json)
  177. </details>
  178. #### YouTube · Agent Building Advice
  179. ```text
  180. Read this agent video and extract its description and chapters. https://www.youtube.com/watch?v=LP5OCa20Zpg
  181. ```
  182. <p align="center">
  183. <img src="./docs/assets/readme/examples/youtube.en.svg?v=ceb49958" width="100%" alt="YouTube: original video page and retrieved content">
  184. </p>
  185. [Source](https://www.youtube.com/watch?v=LP5OCa20Zpg) · [Original video screenshot](./docs/assets/readme/examples/originals/youtube-after-login.jpg) · Retrieved the description, chapters, and cover after sign-in; caption text was not retrieved.
  186. <details>
  187. <summary>View fields before and after sign-in</summary>
  188. ```json
  189. {
  190. "before_sign_in": {
  191. "status": "error",
  192. "error": {
  193. "code": "LOGIN_REQUIRED",
  194. "message": "Sign in to confirm you’re not a bot"
  195. }
  196. },
  197. "after_sign_in": {
  198. "status": "success",
  199. "title": "Tips for building AI agents",
  200. "author": "Anthropic",
  201. "media_references": {
  202. "image": 1
  203. },
  204. "captions_available": true,
  205. "warnings": [
  206. "YouTube captions were advertised but could not be read"
  207. ],
  208. "excerpt": "00:26 Defining AI agents and workflows\n10:47 Coding agents: Potential and challenges\n16:26 Advice for developers exploring agents"
  209. }
  210. }
  211. ```
  212. [After sign-in evidence](./docs/assets/readme/examples/youtube.json) · [Before sign-in evidence](./docs/assets/readme/examples/youtube-before-login.json)
  213. </details>
  214. ### YouTube Sign-In
  215. On `LOGIN_REQUIRED`, ask your agent to open the dedicated Chrome for Testing sign-in window:
  216. ```text
  217. Open the DigCrawl sign-in window for YouTube.
  218. ```
  219. Sign in to Google yourself, return to YouTube, then tell the agent “I’ve finished signing in.” The agent saves the local session and reads the original video again.
  220. <p align="center">
  221. <img src="./docs/assets/readme/login/youtube-flow.en.svg?v=fb10ec86" width="100%" alt="Five-step local YouTube sign-in flow with original video screenshots before and after session synchronization">
  222. </p>
  223. <details>
  224. <summary>YouTube manual sign-in and session details</summary>
  225. ```bash
  226. digcrawl browser login --config "<profile path used by MCP>" --platform youtube
  227. ```
  228. Sign in through the dedicated Chrome window and return to YouTube, then press Enter in the original terminal. DigCrawl closes the window and synchronizes the session. On Windows, close all dedicated Chrome windows before pressing Enter.
  229. Use the same profile as MCP; your everyday browser session is not copied automatically. Credentials and cookies stay on your machine. Preserve any Google sign-in rejection; see [sign-in guidance](./USAGE.md#platform-sign-in-and-health-checks).
  230. </details>
  231. ### When Sign-In Is Required · Xiaohongshu Example
  232. On `LOGIN_REQUIRED`, ask your agent to open DigCrawl’s dedicated sign-in window:
  233. ```text
  234. Open the DigCrawl sign-in window for Xiaohongshu.
  235. ```
  236. Sign in yourself, then tell the agent “I’ve finished signing in.” The agent saves the local session and reads the original link again.
  237. <p align="center">
  238. <img src="./docs/assets/readme/login/flow.en.svg?v=b147dc73" width="100%" alt="Local Xiaohongshu sign-in flow with actual platform screenshots">
  239. </p>
  240. <details>
  241. <summary>Manual sign-in and session details</summary>
  242. ```bash
  243. digcrawl browser login --config "<profile path used by MCP>" --platform xiaohongshu
  244. ```
  245. When running the command yourself, press Enter in that terminal after signing in. Use the same profile as MCP; your everyday browser session is not copied automatically. Credentials and cookies stay on your machine.
  246. See [sign-in guidance](./USAGE.md#platform-sign-in-and-health-checks) for other platforms and troubleshooting.
  247. </details>
  248. <details>
  249. <summary>Manual Skill and MCP configuration</summary>
  250. Install the complete [Skill directory](./.agents/skills/digcrawl/), then configure the local server for your host:
  251. - [Codex example](./.agents/skills/digcrawl/codex/config.toml.example)
  252. - [Claude Code example](./.agents/skills/digcrawl/claude-code/.mcp.json.example)
  253. - [Other MCP hosts](./.agents/skills/digcrawl/mcp/README.md)
  254. Use absolute program and profile paths, and copy the whole Skill rather than only `SKILL.md`. See [setup](./USAGE.md#6-ide-integration-mcp) for profile initialization.
  255. </details>
  256. ## Local-First Security
  257. - Runs on your machine; HTTP Gateway listens on localhost by default.
  258. - The default Skill needs no API key and does not expose cookies to the agent.
  259. - Signed-in sources use user-authorized local sessions; users handle login and captchas.
  260. - Access failures and rate limits are reported explicitly; error pages are not treated as content.
  261. ### Browser recovery and diagnostics
  262. Background rendering uses a dedicated headless runtime; sign-in uses a separately identified headed browser. Dependency failures never fall back to daily Chrome. Source installations must run `digcrawl browser install`; migrate legacy configurations and embedded copies using the [upgrade instructions](./USAGE.md#browser-capabilities-optional).
  263. Follow the returned error guidance or see [browser troubleshooting](./USAGE.md#browser-timeouts-or-repeated-failures).
  264. <details>
  265. <summary>Check the local environment</summary>
  266. ```bash
  267. digcrawl doctor --config "$HOME/.config/digcrawl/digcrawl-skill.yaml"
  268. ```
  269. See the [MCP documentation](./.agents/skills/digcrawl/mcp/README.md) for upgrade reconnection and Bridge lifecycle details.
  270. </details>
  271. ## Architecture and Interfaces
  272. | Interface | Use |
  273. | --- | --- |
  274. | CLI / Watchlist | Local retrieval, batches, and scheduled monitoring |
  275. | Skill / MCP | Content reading tools for AI agents |
  276. | HTTP / stdio / Go SDK | Integration with applications and data workflows |
  277. See [Runtime](./runtime/README.md), [Transport](./transport/README.md), and [Protocol](./protocol/README.md) for internal modules and tool contracts.
  278. ## Documentation
  279. **For users**
  280. - [Installation and usage](./USAGE.md) · [Product guide](./docs/DigCrawl功能与使用手册.md) · [Browser troubleshooting](./USAGE.md#browser-timeouts-or-repeated-failures)
  281. - [Agent Skill](./.agents/skills/digcrawl/) · [MCP setup](./.agents/skills/digcrawl/mcp/README.md)
  282. **For developers**
  283. - [CI](./.github/workflows/test.yml) runs only through manual `workflow_dispatch`; the Test workflow covers Linux, macOS, and Windows. Release publishing remains triggered by version tags.
  284. - [HTTP / stdio](./transport/README.md) · [Go SDK](./client/README.md) · [Configuration examples](./config/)
  285. - [Collect](./collect/README.md) · [Runtime](./runtime/README.md) · [Protocol](./protocol/README.md)
  286. ## Contributing
  287. Issues and PRs are welcome. Include your use case, reproduction steps, or tests. Never commit credentials, cookies, browser data, or private content.
  288. ## Open Source Notes
  289. Respect source terms, permissions, and applicable law. Do not bypass login, captchas, or access restrictions.
  290. ## License
  291. [MIT License](./LICENSE)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.