Skip to content
Back to search

abx-dl

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/ArchiveBox/abx-dl with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/ArchiveBox/abx-dl'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/archivebox--abx-dl/badge.svg)](/listings/archivebox--abx-dl)

HTML

<a href="/listings/archivebox--abx-dl"><img src="/api/listings/archivebox--abx-dl/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

cd "$(mktemp -d)"
exec >stdout.log
uv tool install abx-dl
abx-dl version
cd "$(mktemp -d)"
exec >stdout.log
uvx abx-dl version
abx-dl install wget title

Docker

The image includes the downloader plugins and their dependencies. Mount an output directory at `/out` to keep the downloaded files:

set -Eeuo pipefail
output_dir="$(mktemp -d)"
image="${ABXDL_IMAGE:-archivebox/abx-dl:latest}"
trap 'rm -rf "$output_dir"' EXIT
docker run --rm \
  --volume "$output_dir:/out" \
  "$image" \
  --no-install --plugins=title,wget 'https://example.com'
test -s "$output_dir/index.jsonl"
test -s "$output_dir/title/title.txt"
test -s "$output_dir/wget/example.com/index.html"
grep -q 'Example Domain' "$output_dir/title/title.txt"
grep -q 'Example Domain' "$output_dir/wget/example.com/index.html"

To persist browser personas, also mount their directory at `/data/personas`. The image does not create an anonymous persona volume.

<br/>

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md Β· README Β· 581 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # ⬇️ `abx-dl`
  2. > A simple all-in-one CLI tool to auto-detect and download *everything* available from a URL.
  3. <!--
  4. ```bash
  5. cd "$(mktemp -d)"
  6. exec >stdout.log
  7. ```
  8. -->
  9. <!--pytest-codeblocks:cont-->
  10. ```bash
  11. uvx abx-dl 'https://example.com'
  12. ```
  13. <!--pytest-codeblocks:cont-->
  14. <!--
  15. ```bash
  16. test -s index.jsonl
  17. test -s title/title.txt
  18. test -s wget/example.com/index.html
  19. ```
  20. -->
  21. ---
  22. ✨ *Ever wish you could `yt-dlp`, `gallery-dl`, `wget`, `curl`, `puppeteer`, etc. all in one command?*
  23. `abx-dl` is an all-in-one CLI tool for downloading URLs "by any means necessary".
  24. It's useful for scraping, downloading, OSINT, digital preservation, and more.
  25. `abx-dl` provides a simpler one-shot CLI interface to the [ArchiveBox plugin ecosystem](https://plugins.archivebox.io/).
  26. <img width="1000" height="1082" alt="Screenshot 2026-03-11 at 6 53 03β€―AM" src="https://github.com/user-attachments/assets/4e19d985-1a93-4f65-9970-2565be16b718" />
  27. ---
  28. <br/>
  29. #### 🍜 What does it save?
  30. <!--
  31. ```bash
  32. set -Eeuo pipefail
  33. trap 'status=$?; printf "README crawl failed: %s (exit %s)\n" "$BASH_COMMAND" "$status" >failure.log; find . -maxdepth 3 -type f | sort >>failure.log; exit "$status"' ERR
  34. scratch_dir="${ABX_DL_DOCS_OUTPUT_DIR:-$(mktemp -d)}"
  35. mkdir -p "$scratch_dir"
  36. cd "$scratch_dir"
  37. exec >stdout.log 2>stderr.log
  38. ```
  39. -->
  40. <!--pytest-codeblocks:cont-->
  41. ```bash
  42. abx-dl --plugins=wget,title,screenshot,pdf,readability 'https://example.com'
  43. ```
  44. <!--pytest-codeblocks:cont-->
  45. <!--
  46. ```bash
  47. test -s index.jsonl
  48. test -s title/title.txt
  49. test -s wget/example.com/index.html
  50. test -s screenshot/screenshot.png
  51. test -s pdf/output.pdf
  52. test -s readability/content.html
  53. grep -q 'Example Domain' title/title.txt
  54. grep -q 'Example Domain' wget/example.com/index.html
  55. # Readability extracts article body, which may omit the heading. The live
  56. # example.com DOM now does; title extraction is checked separately above.
  57. grep -Fq 'This domain is for use in documentation examples without needing permission.' readability/content.txt
  58. grep -q '"plugin": "wget".*"status": "succeeded"' index.jsonl
  59. grep -q '"plugin": "screenshot".*"status": "succeeded"' index.jsonl
  60. grep -q '"plugin": "pdf".*"status": "succeeded"' index.jsonl
  61. grep -q '"plugin": "title".*"status": "succeeded"' index.jsonl
  62. grep -q '"plugin": "readability".*"status": "succeeded"' index.jsonl
  63. ```
  64. -->
  65. `abx-dl` runs all plugins by default (and auto installs dependencies). You can specify `--plugins=wget,favicon,title` or filters like `--output=html,pdf,ico,text/` to limit plugin selection.
  66. - HTML, JS, CSS, images, etc. rendered with a headless browser
  67. - title, favicon, headers, outlinks, and other metadata
  68. - audio, video, subtitles, playlists, comments
  69. - snapshot of the page as a PDF, screenshot, and [Singlefile](https://github.com/gildas-lormeau/single-file-cli) HTML
  70. - article text, `git` source code
  71. - [and much more](https://plugins.archivebox.io/)...
  72. <br/>
  73. #### 🧩 How does it work?
  74. `abx-dl` uses the **[Plugin Library](https://plugins.archivebox.io/)** (shared with [ArchiveBox](https://github.com/ArchiveBox/ArchiveBox)) to run a collection of downloading and scraping tools.
  75. Plugins are loaded from the installed [`abx-plugins`](https://pypi.org/project/abx-plugins/) package (or from `ABX_PLUGINS_DIR` if you override it) and execute in distinct phases:
  76. 1. **Install phase** runner reads plugins `config.json`: `required_binaries` and emits `BinaryRequestEvent`s for `abxpkg.binary_service.BinaryService`, which resolves or installs binaries using built-in providers such as env, pip, npm, brew, apt, cargo, and browser-specific providers. `abxpkg` owns the persistent binary cache; `abx-dl` only projects resolved paths into the current run's in-memory config.
  77. 2. **CrawlSetup hooks** (`on_CrawlSetup__*`) launch/configure expensive crawl-scoped processes like chrome, or trigger side effects. background hooks use their first stdout line as the readiness boundary and emit no stdout JSONL records.
  78. 3. **Snapshot hooks** (`on_Snapshot__*`) run per URL to extract content. background hooks use their first stdout line as the readiness boundary; JSONL records after that are `ArchiveResult`, `Snapshot`, and `Tag`.
  79. Applications embedding the runtime use the same framework-free interfaces as
  80. the CLI: `PluginCatalog` for inventory, `PluginConfigResolver` for config,
  81. the service classes for explicit listener composition, typed events for phase
  82. dispatch, `execute_hook()` for a single finite hook, and `OutputManifest` for
  83. output metadata. These APIs accept plain mappings, filesystem paths,
  84. environment variables, and CLI arguments; they do not depend on Django or an
  85. application database.
  86. Standalone `download()` attaches both `CrawlService` (plugin crawl hooks) and
  87. `CrawlLifecycleService` (phase sequencing). Embedders can attach only the
  88. listener suites whose behavior they want and dispatch the corresponding typed
  89. events directly.
  90. For database-backed queues, pass an async `run_snapshots(CrawlStartEvent)`
  91. callback to `CrawlLifecycleService`. The callback admits and awaits snapshot
  92. work, draining already-admitted tasks in its `finally` block if scheduling
  93. fails. The service still owns setup, cleanup, and completion; embedders must
  94. not duplicate those phases. Omitting the callback runs the usual single
  95. snapshot. An explicit user abort completes after cleanup; an execution
  96. takeover (`CrawlAbortEvent(user_initiated=False)`) leaves completion to its new
  97. owner.
  98. If a background crawl-setup resource required by another selected plugin fails
  99. after readiness, the owner stops new snapshot admissions through `CrawlAbortEvent`
  100. with `user_initiated=False` and its `error`. Already-started captures clean up and
  101. complete with that failure; queued URLs are not attempted against the dead resource.
  102. An empty error with `user_initiated=False` remains ownership takeover and emits no
  103. terminal completion. Optional monitors and snapshot-level background downloads keep
  104. their independent failure behavior; this does not restart or recover a browser.
  105. Before destroying an embedded event bus, await `ProcessService.stop()` to stop
  106. its remaining hooks and deliver their process-completion facts. Keep per-snapshot
  107. services registered until this finishes, before calling their `close()` methods:
  108. their takeover and cleanup consumers are still needed during the drain. This
  109. uses the same takeover control path and does not complete interrupted snapshots
  110. or crawls. Abrupt process death can still interrupt shutdown; the next owner
  111. starts a fresh snapshot attempt rather than depending on this drain for recovery.
  112. `SnapshotCompletedEvent` means an attempt has settled, not that every plugin
  113. succeeded. Its `error` contains a handled execution or persistence failure;
  114. existing hook failures retain their own ArchiveResults. Completion is emitted
  115. only after actual hook processes stop, and never for task cancellation or
  116. ownership takeover. A handled snapshot failure allows independent snapshots
  117. to finish and produces a nonzero final crawl outcome. `SnapshotCleanupEvent.error`
  118. carries the owner's failure context to later final-metadata consumers before
  119. the terminal event is emitted. Preparation consumers run before the runtime
  120. owner on the serial `SnapshotEvent`; a recorded preparation exception prevents
  121. hooks from starting. Final-metadata consumers register after runtime cleanup
  122. on serial `SnapshotCleanupEvent`, and act only when `finalize_snapshot=True`.
  123. `CrawlCompletedEvent.error` likewise records a handled failure after the crawl's
  124. processes stop. The original exception still reaches the caller after completion
  125. consumers finish. This fact ends one execution; an embedder may still have queued
  126. snapshots to run. Cancellation, takeover, or live writers prevent completion.
  127. Interactive Ctrl+C remains a live-session control: it can interrupt a hook,
  128. wait for the user's choice, then retry that hook, skip it, or abort the capture.
  129. That in-memory prompt does not persist a paused snapshot. A dead or replaced
  130. owner cannot continue a half-finished snapshot; a later attempt starts fresh.
  131. Hook preparation failures (before a process starts) retain their failed result
  132. and allow independent hooks to finish. After cleanup and completion consumers
  133. finish, the crawl raises `SnapshotAttemptError` so CLI callers do not report a
  134. false success. Ordinary extractor exits remain per-hook outcomes.
  135. Failed foreground crawl-setup hooks are likewise reported after independent
  136. work finishes; optional setup hooks express best-effort behavior in their own
  137. exit status.
  138. `parse_input(source_text, catalog, output_dir)` is the framework-free import
  139. path for pasted text and bookmark/feed exports. It writes
  140. `staticfile/stdin.txt`, runs only plugins declaring
  141. `x-accepts-internal-input`, and returns metadata-preserving `Snapshot` facts at
  142. depth zero. It does not create a crawl, database row, or synthetic URL.
  143. <br/>
  144. #### βš™οΈ Configuration
  145. Configuration is handled via environment variables plus a user config file under the platformdirs user config path (`<user-config>/abx/config.env`). Resolved binary paths are projected into the current run in memory; persistent binary state lives only in the abxpkg library cache:
  146. <!--
  147. ```bash
  148. cd "$(mktemp -d)"
  149. exec >stdout.log
  150. ```
  151. -->
  152. <!--pytest-codeblocks:cont-->
  153. ```bash
  154. abx-dl config # show all config (global + per-plugin)
  155. abx-dl config --get WGET_TIMEOUT # get a specific value
  156. abx-dl config --set TIMEOUT=120 # set persistently (resolves aliases)
  157. ```
  158. Output is grouped by section:
  159. ```text
  160. # GLOBAL
  161. TIMEOUT=60
  162. USER_AGENT="Mozilla/5.0 ..."
  163. ...
  164. # plugins/wget
  165. WGET_BINARY="wget"
  166. WGET_TIMEOUT=60
  167. ...
  168. # plugins/chrome
  169. CHROME_BINARY="chromium"
  170. ...
  171. ```
  172. Common options:
  173. - `TIMEOUT=60` - default timeout for hooks
  174. - `MEMORY_LIMIT` - best-effort per-hook process-tree RSS limit in bytes, checked once per second; `0` or absent disables it. The base plugin config supplies a 2GiB default and `CHROME_MEMORY_LIMIT=0` exempts shared Chrome. `{PLUGIN}_MEMORY_LIMIT` overrides it. Exceeding the limit kills that hook tree and records a failure while preserving saved files. This is not a hard allocation limit: sudden growth between checks can still exhaust memory, and RSS counts shared pages more than once.
  175. - `MEMORY_MIN_FREE` - best-effort host RAM reserve, checked by the same process monitor. The base plugin supplies a 512MiB default; `0` disables this reserve. Below it, owned hook trees using at least 256MiB RSS are stopped through the same failure path, including Chrome with its fixed cap disabled. This protects against other host services consuming memory after admission; it cannot guarantee protection against sudden allocations between checks.
  176. - `USER_AGENT` - optional HTTP user agent override; empty uses Chrome's detected identity after browser setup. yt-dlp keeps its native identity and its own argument overrides.
  177. - `{PLUGIN}_BINARY` - path or name of the binary to use (e.g. `WGET_BINARY=wget` or `CHROME_BINARY=/usr/bin/chromium`)
  178. - `{PLUGIN}_ENABLED=True/False` - enable/disable specific plugins
  179. - `{PLUGIN}_TIMEOUT=120` - per-plugin timeout overrides
  180. Aliases are automatically resolved (e.g. `--set USE_WGET=false` saves as `WGET_ENABLED=false`).
  181. One-off config is easy via env vars or CLI args:
  182. <!--
  183. ```bash
  184. cd "$(mktemp -d)"
  185. exec >stdout.log
  186. ```
  187. -->
  188. <!--pytest-codeblocks:cont-->
  189. ```bash
  190. env \
  191. TIMEOUT=120 \
  192. WGET_TIMEOUT=120 \
  193. abx-dl \
  194. --dir=./config-example \
  195. --plugins=title,wget \
  196. --timeout=90 \
  197. 'https://example.com'
  198. ```
  199. <!--pytest-codeblocks:cont-->
  200. <!--
  201. ```bash
  202. test -s config-example/index.jsonl
  203. test -s config-example/title/title.txt
  204. test -s config-example/wget/example.com/index.html
  205. grep -q 'Example Domain' config-example/title/title.txt
  206. grep -q '"plugin": "wget".*"status": "succeeded"' config-example/index.jsonl
  207. grep -q '"plugin": "title".*"status": "succeeded"' config-example/index.jsonl
  208. ```
  209. -->
  210. <br/>
  211. ---
  212. <br/>
  213. ### πŸ“¦ Install
  214. <!--
  215. ```bash
  216. cd "$(mktemp -d)"
  217. exec >stdout.log
  218. ```
  219. -->
  220. <!--pytest-codeblocks:cont-->
  221. ```bash
  222. uv tool install abx-dl
  223. abx-dl version
  224. ```
  225. <!--
  226. ```bash
  227. cd "$(mktemp -d)"
  228. exec >stdout.log
  229. ```
  230. -->
  231. <!--pytest-codeblocks:cont-->
  232. ```bash
  233. uvx abx-dl version
  234. ```
  235. ```bash
  236. abx-dl install wget title
  237. ```
  238. #### Docker
  239. `archivebox/abx-dl:main-unstable` and the universal `:sha-<12-character-commit>` image are published after both Docker build/check jobs pass, before the remaining CI finishes. They use the exact `main` checkout and its locked dependencies. After full CI passes, the same image is promoted to `:main` and the release tags without rebuilding. All tags include amd64 and arm64; ordinary installations should use the fully tested release channels.
  240. The image includes the downloader plugins and their dependencies. Mount an
  241. output directory at `/out` to keep the downloaded files:
  242. <!--pytest.mark.docker_required-->
  243. ```bash
  244. set -Eeuo pipefail
  245. output_dir="$(mktemp -d)"
  246. image="${ABXDL_IMAGE:-archivebox/abx-dl:latest}"
  247. trap 'rm -rf "$output_dir"' EXIT
  248. docker run --rm \
  249. --volume "$output_dir:/out" \
  250. "$image" \
  251. --no-install --plugins=title,wget 'https://example.com'
  252. test -s "$output_dir/index.jsonl"
  253. test -s "$output_dir/title/title.txt"
  254. test -s "$output_dir/wget/example.com/index.html"
  255. grep -q 'Example Domain' "$output_dir/title/title.txt"
  256. grep -q 'Example Domain' "$output_dir/wget/example.com/index.html"
  257. ```
  258. To persist browser personas, also mount their directory at `/data/personas`.
  259. The image does not create an anonymous persona volume.
  260. <br/>
  261. ### πŸ”  Usage
  262. <!--
  263. ```bash
  264. cd "$(mktemp -d)"
  265. exec >stdout.log
  266. ```
  267. -->
  268. <!--pytest-codeblocks:cont-->
  269. ```bash
  270. abx-dl --plugins=title,wget --dir=./downloads --timeout=120 'https://example.com'
  271. ```
  272. <!--pytest-codeblocks:cont-->
  273. <!--
  274. ```bash
  275. test -s downloads/index.jsonl
  276. test -s downloads/title/title.txt
  277. test -s downloads/wget/example.com/index.html
  278. ```
  279. -->
  280. ```text
  281. # Default command - a bare URL archives with all enabled plugins:
  282. abx-dl 'https://example.com'
  283. # Select plugins by output type (mimetypes, categories, or file extensions):
  284. abx-dl --output=html,pdf,video/ 'https://example.com'
  285. abx-dl -o text -o image -o mp4 'https://example.com'
  286. # Limit work to a subset of plugins by name:
  287. abx-dl --plugins=wget,title,screenshot,pdf 'https://example.com'
  288. # Skip auto-installing missing dependencies (emit warnings instead):
  289. abx-dl --no-install 'https://example.com'
  290. # Specify output directory (default is current working dir):
  291. abx-dl --dir=./downloads 'https://example.com'
  292. # Set timeout:
  293. abx-dl --timeout=120 'https://example.com'
  294. ```
  295. #### Commands
  296. ```text
  297. abx-dl <url> # Download URL (default shorthand)
  298. abx-dl plugins # Check + show info for all plugins
  299. abx-dl plugins wget ytdlp git # Check + show info for specific plugins
  300. abx-dl install wget ytdlp git # Pre-install plugin dependencies
  301. abx-dl config # Show all config values
  302. abx-dl config --get TIMEOUT # Get a specific config value
  303. abx-dl config --set TIMEOUT=120 # Set a config value persistently
  304. ```
  305. #### Installing Dependencies
  306. Many plugins require external binaries (e.g., `wget`, `chrome`, `yt-dlp`, `single-file`).
  307. By default, `abx-dl` lazily installs missing dependencies as needed when you download a URL.
  308. Use `--no-install` to skip plugins with missing dependencies instead. `install` runs only the pre-run dependency pipeline (`required_binaries` β†’ `BinaryRequestEvent` β†’ `BinaryEvent`) without starting crawl setup or snapshot extraction:
  309. <!--
  310. ```bash
  311. cd "$(mktemp -d)"
  312. exec >stdout.log
  313. ```
  314. -->
  315. <!--pytest-codeblocks:cont-->
  316. ```bash
  317. abx-dl install wget title
  318. abx-dl plugins wget title
  319. ```
  320. ```text
  321. abx-dl 'https://example.com' # checks and installs missing deps before hooks run
  322. abx-dl --no-install 'https://example.com' # skips plugins with missing deps and emits warnings
  323. abx-dl install wget singlefile ytdlp # installs dependencies for specific plugins only
  324. abx-dl plugins # checks which dependencies are available/missing
  325. ```
  326. Every preflight request is resolved through `abxpkg`. Compatible host binaries are selected first and projected into `ABXPKG_LIB_DIR/env/bin`; otherwise the configured managed provider installs and projects the dependency. Hook subprocesses then use the resolved Python or Node interpreter and projected runtime environment directly.
  327. During capture, a dependency failure leaves that plugin's real hook to report its failure while unrelated plugins continue. Explicit `abx-dl install` commands still fail when a requested dependency cannot be installed. Crawl setup readiness failures remain fatal because later hooks may require those shared services.
  328. The normal runtime flow after dependency preflight is:
  329. - `CrawlEvent` (internal lifecycle root)
  330. - `CrawlSetupEvent` β†’ plugin `on_CrawlSetup__*` hooks
  331. - `CrawlStartEvent` β†’ `SnapshotEvent`
  332. - `SnapshotEvent` β†’ plugin `on_Snapshot__*` hooks
  333. - `SnapshotCleanupEvent` / `CrawlCleanupEvent`
  334. Hook output contract:
  335. - `EXTRA_CONTEXT` is opaque correlation data reflected into output records only. Hooks must never inspect it. Snapshot hooks receive `--url`, `--snapshot-id`, and `--depth` as explicit arguments; archived titles/tags stay in filesystem data. Embedders provide `download(snapshot=Snapshot(...))` for an existing snapshot, not identity or depth hidden in configuration.
  336. - binary preflight is driven by plugin `required_binaries` and handled by `abxpkg`, not by plugin hooks
  337. - `on_CrawlSetup__*` background hooks emit a first stdout readiness line, but no stdout JSONL records
  338. - `on_Snapshot__*` background hooks emit a first stdout readiness line; hook JSONL records after that are only `ArchiveResult`, `Snapshot`, and `Tag`
  339. - the TUI and services consume structured events derived from those hook records
  340. Dependencies are installed to `<user-config>/abx/lib/{arch}/` using the appropriate package manager:
  341. - **pip packages** β†’ `<user-config>/abx/lib/{arch}/pip/venv/`
  342. - **npm packages** β†’ `<user-config>/abx/lib/{arch}/npm/`
  343. - **brew/apt packages** β†’ system locations
  344. You can override the install location with `ABXPKG_LIB_DIR=/path/to/lib abx-dl install wget`.
  345. <br/>
  346. ---
  347. <br/>
  348. ### Output Structure
  349. By default, `abx-dl` writes results into the current working directory. Each run creates an `index.jsonl` manifest plus one subdirectory per plugin that produced output. If you want to keep runs isolated, `cd` into a scratch directory first or pass `--dir=/path/to/run`.
  350. Hook working directories are created for execution, then removed at the end of
  351. snapshot/crawl cleanup if empty. Plugins returning `noresults` leave no directory
  352. when they wrote nothing; `.DS_Store` is discarded, while captures and failure logs are preserved. Process
  353. commands are recorded in `index.jsonl` and active process metadata, without
  354. generating per-hook shell scripts. Plugin-owned files remain unchanged.
  355. Default runtime personas (including Chrome profiles and writable extension
  356. copies) live under `TMP_DIR/personas`, isolated by output directory. An existing
  357. host-prepared `.persona` symlink is preserved. Explicit `PERSONAS_DIR` settings
  358. remain supported; keep runtime storage local, outside archive output mounts.
  359. Active hook stdout, stderr, and process-control PID files live under
  360. `TMP_DIR/process` (the system temporary directory when `TMP_DIR` is unset).
  361. Keep `TMP_DIR` on local storage even when captured output uses a remote mount.
  362. Successful hooks remove these temporary files; failed or cancelled hooks retain
  363. their stdout and stderr logs in the plugin output directory.
  364. ```bash
  365. mkdir -p /tmp/abx-run && cd /tmp/abx-run
  366. uvx --from abx-dl abx-dl --plugins=title,wget 'https://example.com'
  367. ```
  368. ```
  369. ./
  370. β”œβ”€β”€ index.jsonl # Snapshot metadata and results (JSONL format)
  371. β”œβ”€β”€ title/
  372. β”‚ └── title.txt
  373. β”œβ”€β”€ favicon/
  374. β”‚ └── favicon.ico
  375. β”œβ”€β”€ screenshot/
  376. β”‚ └── screenshot.png
  377. β”œβ”€β”€ pdf/
  378. β”‚ └── output.pdf
  379. β”œβ”€β”€ dom/
  380. β”‚ └── output.html
  381. β”œβ”€β”€ wget/
  382. β”‚ └── example.com/
  383. β”‚ └── index.html
  384. β”œβ”€β”€ singlefile/
  385. β”‚ └── output.html
  386. └── ...
  387. ```
  388. <br/>
  389. ### All Outputs
  390. - `index.jsonl` - snapshot metadata and plugin results (JSONL format, ArchiveBox-compatible)
  391. - `title/title.txt` - page title
  392. - `favicon/favicon.ico` - site favicon
  393. - `screenshot/screenshot.png` - full page screenshot (Chrome)
  394. - `pdf/output.pdf` - page as PDF (Chrome)
  395. - `dom/output.html` - rendered DOM (Chrome)
  396. - `wget/example.com/...` - mirrored site files
  397. - `singlefile/output.html` - single-file HTML snapshot
  398. - ... and more via plugin library ...
  399. ---
  400. ### Available Plugins
  401. See the [`abx-plugins` marketplace](https://github.com/ArchiveBox/abx-plugins).
  402. #### Snapshot / Extraction Plugins
  403. - `ytdlp` - downloads media plus sidecars: audio, video, images/thumbnails, subtitles (`.srt`, `.vtt`), JSON metadata, and text descriptions.
  404. - `gallerydl` - downloads gallery/media sets as images, videos, JSON sidecars, text sidecars, and ZIP archives.
  405. - `forumdl` - exports forum/thread archives as JSONL, WARC, and mailbox-style message archives.
  406. - `git` - clones repository contents including text, binaries, images, audio, video, fonts, and other tracked files.
  407. - `wget` - mirrors pages and requisites as HTML, WARC, images, CSS, JavaScript, fonts, audio, and video.
  408. - `archivedotorg` - saves a Wayback Machine archive link as plain text.
  409. - `favicon` - saves site favicons and touch icons as image files.
  410. - `modalcloser` - setup helper only; no direct archive files.
  411. - `consolelog` - saves browser console events as JSONL.
  412. - `dns` - saves observed DNS activity as JSONL.
  413. - `ssl` - saves TLS certificate/connection metadata as JSONL.
  414. - `responses` - saves HTTP response metadata as JSONL and can record referenced text, images, audio, video, apps, and fonts.
  415. - `redirects` - saves redirect chains as JSONL.
  416. - `staticfile` - saves non-HTML direct file responses such as PDF, EPUB, images, audio, video, JSON, XML, CSV, ZIP, and generic binary files.
  417. - `headers` - saves main-document HTTP headers as JSON.
  418. - `chrome` - manages shared browser state and emits plain-text and JSON runtime metadata.
  419. - `seo` - saves SEO metadata such as meta tags and Open Graph fields as JSON.
  420. - `accessibility` - saves the browser accessibility tree as JSON.
  421. - `infiniscroll` - page-expansion helper only; no direct archive files.
  422. - `claudechrome` - saves Claude-computer-use interaction results as JSON plus PNG screenshots.
  423. - `singlefile` - saves a full self-contained page snapshot as HTML.
  424. - `screenshot` - saves rendered page screenshots as PNG.
  425. - `pdf` - saves rendered pages as PDF.
  426. - `dom` - saves fully rendered DOM output as HTML.
  427. - `title` - saves the final page title as plain text.
  428. - `readability` - extracts article HTML, plain text, and JSON metadata.
  429. - `defuddle` - extracts cleaned article HTML, plain text, and JSON metadata.
  430. - `mercury` - extracts article HTML, plain text, and JSON metadata.
  431. - `claudecodeextract` - generates cleaned Markdown from other extractor outputs.
  432. - `htmltotext` - converts archived HTML into plain text.
  433. - `trafilatura` - extracts article content as plain text, Markdown, HTML, CSV, JSON, and XML/TEI.
  434. - `papersdl` - downloads academic papers as PDF.
  435. - `parse_html_urls` - emits discovered links from HTML as JSONL records.
  436. - `parse_txt_urls` - emits discovered links from text files as JSONL records.
  437. - `parse_rss_urls` - emits discovered feed entry URLs from RSS/Atom as JSONL records.
  438. - `parse_netscape_urls` - emits discovered bookmark URLs from Netscape bookmark exports as JSONL records.
  439. - `parse_jsonl_urls` - emits discovered bookmark URLs from JSONL exports as JSONL records.
  440. - `parse_dom_outlinks` - emits crawlable rendered-DOM outlinks as JSONL records.
  441. - `search_backend_sqlite` - writes a searchable SQLite FTS index database.
  442. - `search_backend_sonic` - pushes content into Sonic search; no local archive files declared.
  443. - `claudecodecleanup` - writes cleanup/deduplication results as plain text.
  444. - `hashes` - writes file hash manifests as JSON.
  445. - and more via the [`abx-plugins` marketplace](https://github.com/ArchiveBox/abx-plugins)...
  446. ---
  447. ### AI Skill
  448. This repo includes an `abx-dl` skill for coding agents that need to run the standalone ArchiveBox extractor pipeline without a full ArchiveBox install.
  449. - Skill source: [`skills/abx-dl/SKILL.md`](./skills/abx-dl/SKILL.md)
  450. - skills.sh page: https://skills.sh/archivebox/abx-dl/abx-dl
  451. ---
  452. ### Architecture
  453. `abx-dl` is built on these components:
  454. - **`abx_dl/catalog.py`** - Plugin discovery, selection, and config resolution from `abx-plugins` or `ABX_PLUGINS_DIR`
  455. - **`abx_dl/executor.py`** - Hook execution engine with config propagation
  456. - **`abx_dl/config.py`** - Environment variable configuration
  457. - **`abx_dl/cli.py`** - Rich CLI with live progress display
  458. ### Related Projects
  459. - `abxbus` https://abxbus.archivebox.io https://github.com/archiveBox/abxbus
  460. - `abxpkg` https://abxpkg.archivebox.io https://github.com/archiveBox/abxpkg
  461. - `abx-plugins` https://plugins.archivebox.io https://github.com/ArchiveBox/abx-plugins
  462. - `archivebox` https://archivebox.io https://github.com/ArchiveBox/ArchiveBox
  463. - And lots more...
  464. - https://github.com/stars/pirate/lists/internet-archiving
  465. - https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community
  466. ---
  467. For more advanced use with collections, parallel downloading, a Web UI + REST API, etc.
  468. See: [`ArchiveBox/ArchiveBox`](https://github.com/ArchiveBox/ArchiveBox)
  469. `abx-dl` runs one snapshot at a time. `--snapshot-max-size` stops starting more
  470. hooks once reported output reaches the budget; cleanup still runs to preserve
  471. recordings. Accounting lives only in memory for that run, so retrying an output
  472. directory starts fresh and preserves partial artifacts until hooks overwrite them.
  473. Crawl-wide URL, size, and time limits belong to ArchiveBox and its database.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.