Skip to content
Back to search

ArchiveBox

šŸ—ƒ Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/ArchiveBox/ArchiveBox with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/ArchiveBox/ArchiveBox'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/archivebox--archivebox/badge.svg)](/listings/archivebox--archivebox)

HTML

<a href="/listings/archivebox--archivebox"><img src="/api/listings/archivebox--archivebox/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

uv tool install --python 3.13 --prerelease explicit --upgrade 'archivebox>=0.9.0rc0,<0.10' mkdir -p ~/archivebox/data && cd ~/archivebox/data archivebox init archivebox install

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md Ā· README Ā· 1,568 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center" style="text-align: center; width: 100%">
  2. <img src="https://archivebox.io/icon.png" height="90px"/>
  3. <h1>ArchiveBox<br/><sub>Open-source self-hosted web archiving.</sub></h1>
  4. <br/>
  5. ā–¶ļø <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart">Quickstart</a> | <a href="https://demo.archivebox.io">Demo</a> | <a href="https://github.com/ArchiveBox/ArchiveBox">GitHub</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Documentation</a> | <a href="#background--motivation">Info & Motivation</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/rewiki/Web-Archiving-Community">Community</a>
  6. <br/>
  7. <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/LICENSE"><img src="https://img.shields.io/badge/Open_source-MIT-green.svg?logo=git&logoColor=green"/></a> <a href="https://github.com/ArchiveBox/ArchiveBox/commits/dev"><img src="https://img.shields.io/github/last-commit/ArchiveBox/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/></a> &nbsp; <a href="https://github.com/ArchiveBox/ArchiveBox"><img src="https://img.shields.io/github/stars/ArchiveBox/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/></a> &nbsp; <a href="https://hub.docker.com/r/archivebox/archivebox"><img src="https://img.shields.io/docker/pulls/archivebox/archivebox.svg?label=Docker+Pulls"/></a> <a href="https://pypi.org/project/archivebox/"><img src="https://img.shields.io/pypi/dm/archivebox?label=PyPI%20Installs&color=%235f7dae"/></a> <a href="https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj"><img src="https://img.shields.io/chrome-web-store/users/habonpimjphpdnmcfkaockjnffodikoj?label=Chrome%20Store&color=%231973e8"/></a>
  8. </div>
  9. <hr/>
  10. <br/>
  11. **ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.**
  12. We aim to make your data immediately useful, and kept in formats that other programs can read directly. As output, we save standard HTML, PNG, PDF, TXT, JSON, WARC, SQLite, all guaranteed to be readable for decades to come. ArchiveBox also has a CLI, REST API, and webhooks so you can set up integrations with other services.
  13. Without active preservation effort, everything on the internet eventually disappears or degrades.
  14. *ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...*
  15. <br/>
  16. > āž”ļø Get ArchiveBox with `pip install archivebox` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ā­ļø on Linux/macOS.
  17. *Once installed, you can interact with it through the: [Browser Extension](https://github.com/ArchiveBox/archivebox-browser-extension), [CLI](#usage), [self-hosted web interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive), [Python API](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage), or [filesystem](#static-archive-exporting).*
  18. <br/>
  19. <hr/>
  20. <br/>
  21. šŸ“„ **You can feed ArchiveBox URLs one at a time, or schedule regular imports** from your bookmarks or history, social media feeds or RSS, link-saving services like Pocket/Pinboard, our [Browser Extension](https://github.com/ArchiveBox/archivebox-browser-extension), and more.
  22. <sub>See <a href="#input-formats">Input Formats</a> for a full list of supported input formats...</sub>
  23. <br/>
  24. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/90f1ce3c-75bb-401d-88ed-6297694b76ae" alt="snapshot detail page" align="right" width="190px" style="float: right"/>
  25. **It saves snapshots of the URLs you feed it in several redundant formats.**
  26. It also detects any content featured *inside* pages & extracts it out into a folder:
  27. - 🌐 **HTML**/**Any websites** āž”ļø `original HTML+CSS+JS`, `singlefile HTML`, `screenshot PNG`, `PDF`, `WARC`, `title`, `article text`, `favicon`, `headers`, ...
  28. - šŸŽ„ **Social Media**/**News** āž”ļø `post content TXT`, `comments`, `title`, `author`, `images`, ...
  29. - šŸŽ¬ **YouTube**/**SoundCloud**/etc. āž”ļø `MP3/MP4`s, `subtitles`, `metadata`, `thumbnail`, ...
  30. - šŸ’¾ **Github**/**Gitlab**/etc. links āž”ļø `clone of GIT source code`, `README`, `images`, ...
  31. - šŸ“‘ **Google Docs**/**Dropbox**/**Scientific Papers** āž”ļø `PDF`, `DOCX`, `CSV`, `PPTX`, ...
  32. - ✨ *and more, see [Output Formats](#output-formats) below...*
  33. You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the `uv`-installed CLI, Python API, and SQLite3 APIs.
  34. All the ways of using it are equivalent, and provide matching features like adding tags, scheduling regular crawls, viewing logs, and more...
  35. <br/>
  36. <hr/>
  37. šŸ› ļø ArchiveBox uses [standard tools](#dependencies) like Chrome, [`wget`](https://www.gnu.org/software/wget/), & [`yt-dlp`](https://github.com/yt-dlp/yt-dlp), and stores data in [ordinary files & folders](#archive-layout).
  38. *(no complex proprietary formats, all data is readable without needing to run ArchiveBox)*
  39. The goal is to sleep soundly knowing the part of the internet you care about will be automatically preserved in durable, easily accessible formats [for decades](#background--motivation) after it goes down.
  40. <hr/>
  41. <br/>
  42. **šŸ“¦&nbsp; Install ArchiveBox using your preferred method: `docker` / `uv` / `brew` / `apt` / etc. ([see full Quickstart below](#quickstart)).**
  43. <div align="center" style="text-align: center">
  44. <br/><br/><a href="https://trendshift.io/repositories/6883?utm_source=repository-badge&amp;utm_medium=badge&amp;utm_campaign=badge-repository-6883" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/6883" alt="ArchiveBox%2FArchiveBox Stats" width="250" height="55"/></a>
  45. <br/>
  46. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic"> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/b2765a33-0d1e-4019-a1db-920c7e00e20e" height="75px" alt="logo" align="top"/> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic">
  47. <br/><br/>
  48. <small><a href="https://demo.archivebox.io">Demo</a> | <a href="https://archivebox.io/screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
  49. <br/>
  50. <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
  51. <br/><br/>
  52. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/8d67382c-e0ce-4286-89f7-7915f09b930c" width="22%" alt="cli init screenshot" align="top">
  53. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/dad2bc51-e7e5-484e-bb26-f956ed692d16" width="22%" alt="cli init screenshot" align="top">
  54. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e8e0b6f8-8fdf-4b7f-8124-c10d8699bdb2" width="22%" alt="server snapshot admin screenshot" align="top">
  55. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ace0954a-ddac-4520-9d18-1c77b1ec50b2" width="28.6%" alt="server snapshot details page screenshot" align="top"/>
  56. <br/><br/>
  57. </div>
  58. ## Key Features
  59. - [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/dev/LICENSE), own your own data & maintain your privacy by self-hosting
  60. - [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage) with [modular dependencies](#dependencies) and [support for Google Drive/NFS/SMB/S3/B2/etc.](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage)
  61. - [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
  62. - [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (yt-dlp), articles (readability), code (git), etc.](#output-formats)
  63. - [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
  64. - [**Uses standard, durable, long-term formats**](#output-formats) like HTML, JSON, PDF, PNG, MP4, TXT, and WARC
  65. - [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage), [Python API](https://docs.archivebox.io/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://electron.archivebox.io/)
  66. - [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archivedotorg_enabled) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archivedotorg_enabled) for local-only mode)
  67. - Support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
  68. - Support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [infiniscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
  69. <br/>
  70. <div align="center" style="text-align: center">
  71. <br/>
  72. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0db52ea7-4a2c-441d-b47f-5553a5d8fe96" width="49%" alt="grass"/><img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0db52ea7-4a2c-441d-b47f-5553a5d8fe96" width="49%" alt="grass"/>
  73. </div>
  74. <a name="install"></a>
  75. # Quickstart
  76. **šŸ–„&nbsp; [Supported OSs](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#supported-systems):** Ubuntu, macOS, Docker &nbsp; **šŸ‘¾&nbsp; CPUs:** `amd64` (`x86_64`), `arm64`<br/>
  77. <br/>
  78. #### āœ³ļø&nbsp; Easy Setup
  79. <details>
  80. <summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
  81. <br/>
  82. <ol>
  83. <li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
  84. <li>Set up an admin account and generate an API Key.</li>
  85. <li>Use the Web UI or <a href="https://app.archivebox.io/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://app.archivebox.io/">mobile app</a>, the <a href="https://extension.archivebox.io/">browser extension</a>, and other integrations to your server using your API Key.</li>
  86. </ol>
  87. <br/>
  88. </details>
  89. <details>
  90. <summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code></b> (macOS/Linux) &nbsp; <b>šŸ‘ˆ&nbsp; recommended</b> &nbsp; <i>(click to expand)</i></summary>
  91. <br/>
  92. <i>šŸ‘ Docker Compose is recommended for the easiest install/update UX + best security + all <a href="#dependencies">extras</a> out-of-the-box.</i>
  93. <br/><br/>
  94. <ol>
  95. <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
  96. <li>Download the <a href="https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/docker-compose.yml" download><code>docker-compose.yml</code></a> file into a new empty directory (can be anywhere).
  97. <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox
  98. curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
  99. docker compose pull
  100. </code></pre></li>
  101. <li>Start the server, which initializes a new collection automatically.
  102. <pre lang="bash"><code style="white-space: pre-line">docker compose up -d --wait
  103. </code></pre></li>
  104. <li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
  105. <pre lang="bash"><code style="white-space: pre-line">
  106. # run CLI commands inside the server container started above
  107. docker compose exec archivebox archivebox add 'https://example.com'
  108. docker compose exec archivebox archivebox help
  109. </code></pre>
  110. <i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. āž”ļø</i>
  111. </li>
  112. </ol>
  113. See <a href="#cli-usage">below</a> for more usage examples using the CLI, Web UI, or <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#sql-shell-usage">filesystem/SQL/Python</a> to manage your archive.
  114. <br/><br/>
  115. </details>
  116. <details>
  117. <summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker run</code></b> (macOS/Linux)</summary>
  118. <br/>
  119. <ol>
  120. <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
  121. <li>Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere).
  122. <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
  123. docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev
  124. </code></pre>
  125. </li>
  126. <li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
  127. <pre lang="bash"><code style="white-space: pre-line">
  128. # completely optional, CLI can always be used without running a server
  129. # docker exec archivebox archivebox [subcommand] [--help]
  130. docker exec archivebox archivebox help
  131. </code></pre>
  132. <i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. āž”ļø</i>
  133. </li>
  134. </ol>
  135. See <a href="#cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
  136. <br/><br/>
  137. </details>
  138. <details>
  139. <summary><b><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh uv install shortcut" height="28px" align="top"/> <code>bash</code> uv install shortcut</b> (macOS/Linux/BSD)</summary>
  140. <br/>
  141. <ol>
  142. <li>Run the shortcut for the <code>uv</code> install method. It installs <code>uv</code> first when needed.
  143. <pre lang="bash"><code style="white-space: pre-line">curl -fsSL 'https://get.archivebox.io' | bash</code></pre>
  144. <i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-b-uv-install-shortcut">Install: Bare Metal</a> in the Wiki. āž”ļø</i>
  145. </li>
  146. </ol>
  147. See <a href="#cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
  148. See <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/setup.sh"><code>setup.sh</code></a> for the source code of the <code>uv</code> install shortcut.<br/>
  149. See <a href="https://docs.sweeting.me/s/against-curl-sh">"Against curl | sh as an install method"</a> blog post for my thoughts on the shortcomings of this install method.
  150. <br/><br/>
  151. </details>
  152. <br/>
  153. #### šŸ› &nbsp; Package Manager Setup
  154. <a name="Manual-Setup"></a>
  155. <details>
  156. <summary><b><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="uv" height="28px" align="top"/> <code>uv</code></b> (macOS/Linux/BSD)</summary>
  157. <br/>
  158. <ol>
  159. <li>Install <a href="https://docs.astral.sh/uv/getting-started/installation/">uv</a> on your system (if not already installed).</li>
  160. <li>Install the ArchiveBox package using <code>uv</code>.
  161. <pre lang="bash"><code style="white-space: pre-line">uv tool install --python 3.13 --prerelease explicit --upgrade 'archivebox>=0.9.0rc0,<0.10'
  162. archivebox version
  163. </code></pre>
  164. <i>See the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install">Install: Bare Metal</a> Wiki for full install instructions for each OS...</i>
  165. </li>
  166. <li>Create a new empty directory and initialize your collection (can be anywhere).
  167. <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data # for example
  168. archivebox init # initialize a new collection
  169. archivebox install # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
  170. </code></pre>
  171. </li>
  172. <li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
  173. <pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
  174. # completely optional, CLI can always be used without running a server
  175. # archivebox [subcommand] [--help]
  176. archivebox help
  177. </code></pre>
  178. </li>
  179. </ol>
  180. See <a href="#cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
  181. <br/>
  182. <sub>See the <a href="https://docs.astral.sh/uv/guides/tools/"><code>uv tool</code> documentation</a> for more details about this installation method.</sub>
  183. <br/><br/>
  184. </details>
  185. <details>
  186. <summary><b><img src="https://user-images.githubusercontent.com/511499/117448075-49597580-af0c-11eb-91ba-f34fff10096b.png" alt="aptitude" height="28px" align="top"/> <code>apt</code></b> (Ubuntu/Debian/etc.)</summary>
  187. <br/>
  188. <ol>
  189. <li>Add the third-party ArchiveBox apt repo and install <code>archivebox</code>.
  190. <pre lang="bash"><code style="white-space: pre-line">echo 'deb [trusted=yes] https://archivebox.github.io/debian-archivebox dev main' | sudo tee /etc/apt/sources.list.d/archivebox.list
  191. sudo apt update
  192. sudo apt install archivebox
  193. (cd /tmp && archivebox version) # make sure the package is installed
  194. </code></pre>
  195. </li>
  196. <li>Create a new empty directory and initialize your collection (can be anywhere).
  197. <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data
  198. cd ~/archivebox/data
  199. archivebox init
  200. sudo "$(command -v archivebox)" install
  201. archivebox add 'https://example.com'
  202. </code></pre>
  203. <br/>
  204. </li>
  205. <li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
  206. <pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
  207. # completely optional, CLI can always be used without running a server
  208. # archivebox [subcommand] [--help]
  209. archivebox help
  210. </code></pre>
  211. </li>
  212. </ol>
  213. See <a href="#cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
  214. <sub>The apt package is a thin dev-channel wrapper around the normal Python install flow. <code>sudo "$(command -v archivebox)" install</code> uses apt for missing system dependencies while preserving ownership of the user-owned collection. See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for details.</sub>
  215. <br/><br/>
  216. </details>
  217. <details>
  218. <summary><b><img src="https://user-images.githubusercontent.com/511499/117447803-f2ec3700-af0b-11eb-87d3-671d114f011d.png" alt="homebrew" height="28px" align="top"/> <code>brew</code></b> (macOS and Linux)</summary>
  219. <br/>
  220. Run Homebrew as your normal non-root user on both macOS and Linux; do not use <code>sudo brew</code>.
  221. <ol>
  222. <li>Install <a href="https://brew.sh/#install">Homebrew</a> on your system (if not already installed).</li>
  223. <li>Install the ArchiveBox package using <code>brew</code>.
  224. <pre lang="bash"><code style="white-space: pre-line">brew tap archivebox/archivebox
  225. brew trust archivebox/archivebox
  226. brew install archivebox
  227. archivebox version # verify the installed version
  228. </code></pre>
  229. <i>See the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-c-bare-metal-setup">Install: Bare Metal</a> Wiki for more granular instructions for macOS... āž”ļø</i>
  230. </li>
  231. <li>Create a new empty directory and initialize your collection (can be anywhere).
  232. <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
  233. archivebox init
  234. archivebox install
  235. </code></pre>
  236. </li>
  237. <li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
  238. <pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
  239. # completely optional, CLI can always be used without running a server
  240. # archivebox [subcommand] [--help]
  241. archivebox help
  242. </code></pre><br/>
  243. </li>
  244. </ol>
  245. See <a href="#cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
  246. <sub>See the <a href="https://github.com/ArchiveBox/homebrew-archivebox"><code>homebrew-archivebox</code></a> repo for more details about this distribution.</sub>
  247. <br/><br/>
  248. </details>
  249. <details>
  250. <summary><img src="https://user-images.githubusercontent.com/511499/118077361-f0616580-b381-11eb-973c-ee894a3349fb.png" alt="Arch" height="28px" align="top"/> <code>pacman</code> / <img src="https://user-images.githubusercontent.com/511499/118077946-29e6a080-b383-11eb-94f0-d4871da08c3f.png" alt="FreeBSD" height="28px" align="top"/> <code>pkg</code> / <img src="https://user-images.githubusercontent.com/511499/118077861-002d7980-b383-11eb-86a7-5936fad9190f.png" alt="Nix" height="28px" align="top"/> <code>nix</code> (Arch/FreeBSD/NixOS/more)</summary>
  251. <br/>
  252. > *Warning: These are contributed by external volunteers and may lag behind the official `uv` and Docker channels.*
  253. <ul>
  254. <li>Arch: <a href="https://aur.archlinux.org/packages/archivebox/"><code>yay -S archivebox</code></a> (contributed by <a href="https://github.com/imlonghao"><code>@imlonghao</code></a>, maintained by <a href="https://github.com/jasongodev"><code>@jasongodev</code></a>)</li>
  255. <li>FreeBSD: <a href="https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup"><code>curl -fsSL 'https://get.archivebox.io' | bash</code></a> (uses <code>uv</code>)</li>
  256. <li>Nix: <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/misc/archivebox/default.nix"><code>nix-env --install archivebox</code></a> (contributed by <a href="https://github.com/siraben"><code>@siraben</code></a>)</li>
  257. <li>Guix: <a href="https://packages.guix.gnu.org/packages/archivebox/"><code>guix install archivebox</code></a> (contributed by <a href="https://github.com/rakino"><code>@rakino</code></a>)</li>
  258. <li>More: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/new"><i>contribute another distribution...!</i></a></li>
  259. </ul>
  260. See <a href="#cli-usage">below</a> for usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
  261. <br/><br/>
  262. </details>
  263. <br/>
  264. #### šŸŽ—&nbsp; Other Options
  265. <details>
  266. <summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker</code> + <img src="https://user-images.githubusercontent.com/511499/117447263-4316c980-af0b-11eb-928d-eaf1292ac646.png" alt="Electron" height="28px" align="top"/> <code>electron</code> Desktop App</b> (macOS/Linux/Windows)</summary>
  267. <br/>
  268. <ol>
  269. <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
  270. <li><a href="https://electron.archivebox.io/#get-started">Download the latest installer for Windows, Linux, or Mac</a>.</li>
  271. <li>Install and open ArchiveBox Desktop, then create your local administrator account to start archiving.</li>
  272. </ol>
  273. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/575ef92f-bb3e-4a7c-a4ba-986c1fd76ecf" width="320px">
  274. <br/>
  275. <a href="https://electron.archivebox.io/">Desktop app guide</a> Ā· <a href="https://electron.archivebox.io/screenshots/">Screenshots</a>
  276. <br/>
  277. </details>
  278. <details>
  279. <summary><img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0c46e949-00fe-49c8-a613-ee14501c014c" alt="Self-hosting Platforms" height="28px" align="top"/><b> TrueNAS / UNRAID / YunoHost / Cloudron / etc.</b> (self-hosting solutions)</summary>
  280. <br/>
  281. > *Warning: These are contributed by external volunteers and may lag behind the official Docker and `uv` channels.*
  282. <ul>
  283. <li><a href="https://apps.truenas.com/catalog/archivebox_community/">TrueNAS</a></li>
  284. <li><a href="https://unraid.net/community/apps?q=archivebox#r">UnRaid</a></li>
  285. <li><a href="https://community-scripts.github.io/ProxmoxVE/scripts?id=archivebox">Proxmox</a></li>
  286. <li><a href="https://github.com/YunoHost-Apps/archivebox_ynh">Yunohost</a></li>
  287. <li><a href="https://www.cloudron.io/store/io.archivebox.cloudronapp.html">Cloudron</a></li>
  288. <li><a href="https://docs.saltbox.dev/sandbox/apps/archivebox/">Saltbox</a></li>
  289. <li><a href="https://portainer-templates.as93.net/archivebox">Portainer</a></li>
  290. <li><a href="https://github.com/ArchiveBox/ArchiveBox/pull/922/files#diff-00f0606e18b2618c3cc1667ca7c2b703b537af690ca71eba1330633587dcb1ee">AppImage</a></li>
  291. <li><a href="https://runtipi.io/docs/apps-available#:~:text=for%20AI%20Chats.-,ArchiveBox,Open%20source%20self%2Dhosted%20web%20archiving.,-Atuin%20Server">Runtipi</a></li>
  292. <li><a href="https://github.com/ArchiveBox/ArchiveBox/issues/986">Umbrel</a> (need contributors...)</li>
  293. <li>More: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/new"><i>contribute another distribution...!</i></a></li>
  294. </ul>
  295. See <a href="#cli-usage">below</a> for usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
  296. <br/><br/>
  297. </details>
  298. <details>
  299. <summary><img src="https://user-images.githubusercontent.com/511499/117448723-1663b180-af0d-11eb-837f-d43959227810.png" alt="paid" height="27px" align="top"/> Third-party paid hosting options (cloud VPS)</summary>
  300. <br/>
  301. <ul>
  302. <li><a href="https://elest.io/open-source/archivebox"><img src="https://img.shields.io/badge/Managed_Hosting-Elest.io-%23193f7e.svg?style=flat" height="22px"/></a></li>
  303. <li><a href="https://www.stellarhosted.com/archivebox/"><img src="https://img.shields.io/badge/Semi_Managed_Hosting-StellarHosted.com-%23193f7e.svg?style=flat" height="22px"/></a> (USD $29-250/mo, <a href="https://www.stellarhosted.com/archivebox/#pricing">pricing</a>)</li>
  304. <li><a href="https://www.pikapods.com/pods?run=archivebox"><img src="https://img.shields.io/badge/Semi_Managed_Hosting-PikaPods.com-%2343a047.svg?style=flat" height="22px"/></a> (from USD $2.6/mo)</li>
  305. <li><a href="https://pethost.dev/templates/archivebox/"><img src="https://img.shields.io/badge/Semi_Managed_Hosting-Pethost.dev-%230e0e0e.svg?style=flat" height="22px"/></a> (from EUR €9.99/mo)</li>
  306. <li><a href="https://m.do.co/c/cbc4c0c17840">
  307. <img src="https://img.shields.io/badge/Unmanaged_VPS-DigitalOcean.com-%232f7cf7.svg?style=flat" height="22px"/>
  308. </a> (USD $5-50+/mo, <a href="https://m.do.co/c/cbc4c0c17840">šŸŽ—&nbsp; referral link</a>, <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-compose-on-ubuntu-20-04">instructions</a>)</li>
  309. <li><a href="https://www.vultr.com/?ref=7130289">
  310. <img src="https://img.shields.io/badge/Unmanaged_VPS-Vultr.com-%232337a8.svg?style=flat" height="22px"/>
  311. </a> (USD $2.5-50+/mo, <a href="https://www.vultr.com/?ref=7130289">šŸŽ—&nbsp; referral link</a>, <a href="https://www.vultr.com/docs/install-docker-compose-on-ubuntu-20-04">instructions</a>)</li>
  312. <li><a href="https://fly.io/">
  313. <img src="https://img.shields.io/badge/Unmanaged_App-Fly.io-%239a2de6.svg?style=flat" height="22px"/>
  314. </a> (USD $10-50+/mo, <a href="https://fly.io/docs/hands-on/start/">instructions</a>)</li>
  315. <li><a href="https://railway.app/template/2Vvhmy">
  316. <img src="https://img.shields.io/badge/Unmanaged_App-Railway-%23A11BE6.svg?style=flat" height="22px"/>
  317. </a> (USD $1-5+/mo)</li>
  318. <li><a href="https://aws.amazon.com/marketplace/pp/Linnovate-Open-Source-Innovation-Support-For-Archi/B08RVW6MJ2"><img src="https://img.shields.io/badge/Unmanaged_VPS-AWS-%23ee8135.svg?style=flat" height="22px"/></a> (USD $60-200+/mo)</li>
  319. <li><a href="https://azuremarketplace.microsoft.com/en-us/marketplace/apps/meanio.archivebox?ocid=gtmrewards_whatsnewblog_archivebox_vol118"><img src="https://img.shields.io/badge/Unmanaged_VPS-Azure-%237cb300.svg?style=flat" height="22px"/></a> (USD $60-200+/mo)</li>
  320. <br/>
  321. <sub><i>Referral links marked šŸŽ— provide $5-10 of free credit for new users and help pay for our <a href="https://demo.archivebox.io">demo server</a> hosting costs.</i></sub>
  322. </ul>
  323. For more discussion on third-party hosting options see here: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/531">Issue #531</a>.
  324. </details>
  325. <br/>
  326. #### āž”ļø&nbsp; Next Steps
  327. - Import URLs from some of the supported [Input Formats](#input-formats) or view the supported [Output Formats](#output-formats)...
  328. - (Optional) [Import browser cookies and settings into a persona](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docs/Chromium-Install.md#import-an-existing-browser-profile).
  329. - Tweak your UI or archiving behavior [Configuration](#configuration), read about some of the [Caveats](#caveats), or [Troubleshoot](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
  330. - Read about the [Dependencies](#dependencies) used for archiving, the [Upgrading Process](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives), or the [Archive Layout](#archive-layout) on disk...
  331. - Or check out our full [Documentation](#documentation) or [Community Wiki](#internet-archiving-ecosystem)...
  332. <br/>
  333. ### Usage
  334. <a id="cli-usage"></a>
  335. #### āš”ļø&nbsp; <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage">CLI Usage</a>
  336. ArchiveBox commands can be run in a terminal [directly on your host](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage), or via [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1)/[Docker Compose](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage).
  337. <sup>(depending on how you chose to install it above)</sup>
  338. ```bash
  339. mkdir -p ~/archivebox/data # create a new data dir anywhere
  340. cd ~/archivebox/data # IMPORTANT: cd into the directory
  341. # archivebox [subcommand] [--help]
  342. archivebox version
  343. archivebox help
  344. # equivalent: docker compose run --rm archivebox [subcommand] [--help]
  345. docker compose run --rm archivebox help
  346. # equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
  347. docker run -it -v $PWD:/data archivebox/archivebox:dev help
  348. ```
  349. #### ArchiveBox Subcommands
  350. - `archivebox` `help`/`version` to see the list of available subcommands / currently installed version info
  351. - `archivebox` `setup`/`init`/`config`/`status`/`shell`/`manage` to administer your collection
  352. - `archivebox` `add`/`schedule` to pull in fresh URLs from [bookmarks/history/RSS/etc.](#input-formats)
  353. - `archivebox` `list`/`update`/`remove` to manage existing Snapshots in your collection
  354. <br/>
  355. <details>
  356. <summary><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" height="22px" align="top"/> <b>CLI Usage Examples: non-Docker</b></summary>
  357. <br/>
  358. <pre lang="bash"><code style="white-space: pre-line">
  359. # make sure you have uv-installed ArchiveBox and it's available in your $PATH first
  360. <br/>
  361. # archivebox [subcommand] [--help]
  362. mkdir -p ~/archivebox/data && cd ~/archivebox/data
  363. archivebox init
  364. archivebox install
  365. archivebox add 'https://example.com'
  366. archivebox version # get archivebox version info + check dependencies
  367. archivebox help # get list of archivebox subcommands that can be run
  368. </code></pre>
  369. <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage">Usage: CLI Usage</a> wiki. āž”ļø</i>
  370. </details>
  371. <br/>
  372. <details>
  373. <summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="22px" align="top"/> <b>CLI Usage Examples: Docker Compose</b></summary>
  374. <br/>
  375. <pre lang="bash"><code style="white-space: pre-line">
  376. # make sure you have `docker-compose.yml` from the Quickstart instructions first
  377. <br/>
  378. # docker compose exec archivebox archivebox [subcommand] [--help]
  379. docker compose exec archivebox archivebox version
  380. docker compose exec archivebox archivebox help
  381. docker compose exec archivebox archivebox add 'https://example.com'
  382. # to start webserver: docker compose up
  383. </code></pre>
  384. <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage">Usage: Docker Compose CLI</a> wiki. āž”ļø</i>
  385. </details>
  386. <br/>
  387. <details>
  388. <summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="22px" align="top"/> <b>CLI Usage Examples: Docker</b></summary>
  389. <br/>
  390. <pre lang="bash"><code style="white-space: pre-line">
  391. # make sure the `archivebox` server container from the Quickstart is running first
  392. <br/>
  393. # docker exec archivebox archivebox [subcommand] [--help]
  394. docker exec archivebox archivebox version
  395. docker exec archivebox archivebox help
  396. docker exec archivebox archivebox add 'https://example.com'
  397. </code></pre>
  398. <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1">Usage: Docker CLI</a> wiki. āž”ļø</i>
  399. </details>
  400. <br/>
  401. <details>
  402. <summary><b>šŸ—„&nbsp; SQL/Python/Filesystem Usage</b></summary>
  403. <pre lang="bash"><code style="white-space: pre-line">
  404. archivebox shell # explore the Python library API in a REPL
  405. sqlite3 ./index.sqlite3 # run SQL queries directly on your index
  406. find ./archive/users -path '*/snapshots/*/*/*/index.html' # inspect snapshot data directly
  407. </code></pre>
  408. <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage">Python Shell</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#sql-shell-usage">SQL API</a>, and <a href="https://github.com/ArchiveBox/ArchiveBox#archive-layout">Disk Layout</a> wikis. āž”ļø</i>
  409. </details>
  410. <br/>
  411. <details>
  412. <summary><b>šŸ–„&nbsp; Web UI & API Usage</b></summary>
  413. <pre lang="bash"><code style="white-space: pre-line">
  414. # Start the server on bare metal (uv/apt/brew):
  415. archivebox server 0.0.0.0:5797 # start the server
  416. <br/>
  417. # Or with Docker Compose:
  418. docker compose up # start the server
  419. <br/>
  420. # Or with a Docker container:
  421. docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev
  422. </code></pre>
  423. <sup>Open <a href="http://admin.archivebox.localhost:5797/admin/"><code>http://admin.archivebox.localhost:5797/admin/</code></a> to create the first admin user.</sup><br/>
  424. <br/><br/>
  425. <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage">Usage: Web UI</a> wiki. āž”ļø</i>
  426. <br/><br/>
  427. <b>Optional: Change permissions to allow non-logged-in users</b>
  428. <pre lang="bash"><code style="white-space: pre-line">
  429. archivebox config --set PUBLIC_ADD_VIEW=True # allow guests to submit URLs
  430. archivebox config --set PERMISSIONS=public # make newly added snapshots public
  431. archivebox config --set PUBLIC_INDEX=True # allow guests to see list of all snapshots
  432. # or
  433. docker compose run --rm archivebox config --set PERMISSIONS=public
  434. # restart the server to apply any config changes
  435. </code></pre>
  436. </details>
  437. <br/>
  438. <details>
  439. <summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧&nbsp; Desktop / Mobile / Extension Usage</b></summary>
  440. <br/>
  441. - šŸ“± [<code>ArchiveBox</code> iOS Mobile App](https://app.archivebox.io/)
  442. - šŸ–„ļø [<code>ArchiveBox.app</code> macOS Desktop App](https://app.archivebox.io/)
  443. - 🧩 [ArchiveBox Browser Extension](https://extension.archivebox.io/)
  444. - šŸ’» [Linux/Windows Desktop App (Requires Docker Desktop)](https://electron.archivebox.io/)
  445. <br/>
  446. </details>
  447. <br/>
  448. <br/>
  449. > [!TIP]
  450. > Whether in Docker or not, ArchiveBox commands work the same way, and can be used to access the same data on-disk.
  451. > For example, you could run the Web UI in Docker Compose, and run one-off commands with `uv`-installed ArchiveBox.
  452. <details>
  453. <summary><i>Expand to show comparison...</i></summary><br/>
  454. <pre lang="bash"><code style="white-space: pre-line">
  455. archivebox add --depth=1 'https://example.com' # add a URL with uv-installed archivebox on the host
  456. docker compose run --rm archivebox add --depth=1 'https://example.com' # or w/ Docker Compose
  457. docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://example.com' # or w/ Docker, all equivalent
  458. </code></pre>
  459. <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker">Docker</a> wiki. āž”ļø</i>
  460. </details>
  461. <br/>
  462. <div align="center" style="text-align: center">
  463. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/65f82532-18dd-49c5-86f1-02b1f3100e1e" width="49%" alt="grass"/><img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/65f82532-18dd-49c5-86f1-02b1f3100e1e" width="49%" alt="grass"/>
  464. </div>
  465. <br/>
  466. <div align="center" style="text-align: center">
  467. <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
  468. <br/><br/>
  469. <a href="https://demo.archivebox.io">DEMO: <code>https://demo.archivebox.io</code></a><br/>
  470. <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration">Configuration</a> | <a href="#caveats">Caveats</a>
  471. <br/>
  472. </div>
  473. <br/>
  474. ---
  475. <div align="center" style="text-align: center">
  476. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ac1f897a-8baa-4f8b-8ee8-7443611f258b" width="96%" alt="lego"/>
  477. </div>
  478. <br/>
  479. # Overview
  480. <a name="input-formats"></a>
  481. ## Input Formats: How to pass URLs into ArchiveBox for saving
  482. - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>
  483. <i>Provides realtime archiving of browsing history or selected pages from Chrome/Brave/Firefox/Safari/Edge browsers.</i>
  484. - <img src="https://app.archivebox.io/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official ArchiveBox <a href="https://app.archivebox.io/">macOS Desktop App</a>, <a href="https://app.archivebox.io/">iOS App</a>, or <a href="https://android.archivebox.io">Android App</a></b><br/>
  485. <i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
  486. - šŸ’¬ <b>From <a href="https://github.com/ArchiveBox/archivebox-chat-bot">ArchiveBox Bot</a> in your chat apps</b><br/>
  487. <i>Save links from DMs, @mentions, and groups; use ArchiveBox AI Bot for multi-step archive tasks. <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Integrating-with-Slack">Setup & screenshots →</a></i>
  488. - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files
  489. <i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#import-a-list-of-urls-from-a-text-file).</i>
  490. - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/32b494e6-4de1-4984-8d88-dc02f18e5c34" height="22px"/> From manually exported [browser history](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) or [browser bookmarks](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) (in Netscape format)
  491. <i>Instructions: <a href="https://support.google.com/chrome/answer/96816?hl=en">Chrome</a>, <a href="https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer">Firefox</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/assets/511499/24ad068e-0fa6-41f4-a7ff-4c26fc91f71a">Safari</a>, <a href="https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows">IE</a>, <a href="https://help.opera.com/en/latest/features/#bookmarks:~:text=Click%20the%20import/-,export%20button,-on%20the%20bottom">Opera</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a></i>
  492. - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/4f7bd318-265c-4235-ad25-38be89946b12" height="22px"/> From URLs visited through a [MITM Proxy](https://mitmproxy.org/) with [`archivebox-proxy`](https://github.com/ArchiveBox/archivebox-proxy)
  493. <i>Provides [realtime archiving](https://github.com/ArchiveBox/ArchiveBox/issues/577) of all traffic from any device going through the proxy.</i>
  494. - <img src="https://getpocket.com/favicon.ico" height="22px"/> From bookmarking services or social media (e.g. Twitter bookmarks, Reddit saved posts, etc.)
  495. <i>Instructions: <a href="https://getpocket.com/export">Pocket</a>, <a href="https://pinboard.in/export/">Pinboard</a>, <a href="https://www.instapaper.com/user">Instapaper</a>, <a href="https://shaarli.readthedocs.io/en/master/Usage/#importexport">Shaarli</a>, <a href="https://www.groovypost.com/howto/howto/export-delicious-bookmarks-xml/">Delicious</a>, <a href="https://github.com/csu/export-saved-reddit">Reddit Saved</a>, <a href="https://doc.wallabag.org/en/user/import/wallabagv2.html">Wallabag</a>, <a href="http://help.unmark.it/import-export">Unmark.it</a>, <a href="https://www.addictivetips.com/web/onetab-save-close-all-chrome-tabs-to-restore-export-or-import/">OneTab</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/issues/648">Firefox Sync</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a></i>
  496. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e1e5bd78-b0b6-45dc-914c-e1046fee4bc4" width="330px" align="right" style="float: right"/>
  497. ```bash
  498. # archivebox add --help
  499. archivebox add 'https://example.com/some/page'
  500. archivebox add --depth=1 --plugins=parse_rss_urls "file://$HOME/Downloads/some_feed.xml"
  501. archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12'
  502. echo 'http://example.com' | archivebox add
  503. echo 'any text with <a href="https://example.com">urls</a> in it' | archivebox add
  504. # if using Docker, add -i when piping stdin:
  505. # echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox:dev add
  506. # if using Docker Compose, add -T when piping stdin / stdout:
  507. # echo 'https://example.com' | docker compose run --rm -T archivebox add
  508. ```
  509. See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage) page for documentation and examples.
  510. Use `archivebox add --parser=jsonl --plugins=wget < bookmarks.jsonl` for structured imports, or set `"parser": "jsonl"` in a `POST /api/v1/cli/add` request whose `urls` list contains the import text. `json` also accepts a JSON object or array. These modes take each record's URL field as a complete URL, preserving its metadata without extracting additional URLs from its path or query. `parser` selects how the input is read; `plugins` independently selects how each resulting page is captured. Text/auto parsing still discovers embedded URLs in ambiguous input. Duplicate records for the same URL retain the richest supplied metadata and combine complementary tags.
  511. It also includes a built-in scheduled import feature with `archivebox schedule`, handled by the same orchestrator that powers `archivebox server`, so you can pull in URLs from RSS feeds and websites regularly without a separate cron container.
  512. Recursive schedules (`--depth=1` or greater) refresh their explicit seed URLs on each run so they can discover new links. With `ONLY_NEW=True`, previously archived discovered URLs are still skipped. Nonrecursive schedules and ordinary `add --only-new` calls continue to skip existing URLs. Schedules preserve explicitly submitted filters, plugin selection, limits, and custom config; each new run resolves current defaults for settings left unspecified.
  513. `CRAWL_DISCOVERY_PLUGINS=parse_html_urls` restricts child URL discovery to the named plugin's saved `urls.jsonl`; comma-separated names select multiple sources, and the default empty value accepts every source. Other plugins still run and preserve their outputs. This setting leaves explicit seeds unchanged and combines with `URL_ALLOWLIST`, `URL_DENYLIST`, depth limits, and `ONLY_NEW` when admitting children.
  514. <br/>
  515. <a name="output-formats"></a>
  516. ## Output Formats: What ArchiveBox saves for each URL
  517. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ace0954a-ddac-4520-9d18-1c77b1ec50b2" width="330px" align="right" style="float: right"/>
  518. For each web page added, ArchiveBox creates a Snapshot folder and preserves its content as ordinary files inside the folder (e.g. HTML, PDF, PNG, JSON, etc.).
  519. It uses all available methods out-of-the-box, but you can disable extractors and fine-tune the [configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) as-needed.
  520. <br/>
  521. <details>
  522. <summary><i>Expand to see the full list of ways it saves each page...</i></summary>
  523. <code>data/archive/users/{username}/snapshots/{YYYYMMDD}/{domain}/{Snapshot.id}/</code><br/>
  524. <ul>
  525. <li><strong>Index:</strong> <code>index.html</code> &amp; <code>index.json</code> HTML and JSON index files containing metadata and details</li>
  526. <li><strong>Title</strong>, <strong>Favicon</strong>, <strong>Headers</strong> Response headers, site favicon, and parsed site title</li>
  527. <li><strong>SingleFile:</strong> <code>singlefile.html</code> HTML snapshot rendered with headless Chrome using SingleFile</li>
  528. <li><strong>Wget Clone:</strong> <code>example.com/page-name.html</code> wget clone of the site with <code>warc/TIMESTAMP.gz</code></li>
  529. <li>Chrome Headless <ul>
  530. <li><strong>PDF:</strong> <code>output.pdf</code> Printed PDF of site using headless chrome</li>
  531. <li><strong>Screenshot:</strong> <code>screenshot.png</code> 1440x900 screenshot of site using headless chrome</li>
  532. <li><strong>DOM Dump:</strong> <code>output.html</code> DOM Dump of the HTML after rendering using headless chrome</li>
  533. </ul></li>
  534. <li><strong>Article Text:</strong> <code>article.html/json</code> Article text extraction using Readability &amp; Mercury</li>
  535. <li><strong>Archive.org Permalink:</strong> <code>archive.org.txt</code> A link to the saved site on archive.org</li>
  536. <li><strong>Audio &amp; Video:</strong> <code>media/</code> all audio/video files + playlists, including subtitles &amp; metadata w/ <code>yt-dlp</code></li>
  537. <li><strong>Source Code:</strong> <code>git/</code> clone of any repository found on GitHub, Bitbucket, or GitLab links</li>
  538. <li><em>More coming soon! See the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap">Roadmap</a>...</em></li>
  539. </ul>
  540. </details>
  541. <br/>
  542. ## Configuration
  543. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ea672e6b-4df5-49d8-b550-7f450951fd27" width="330px" align="right" style="float: right"/>
  544. ArchiveBox can be configured via environment variables, by using the `archivebox config` CLI, or by editing `./ArchiveBox.conf`.
  545. <br/>
  546. <details>
  547. <summary><i>Expand to see examples...</i></summary>
  548. <pre lang="bash"><code style="white-space: pre-line">archivebox config # view the entire config
  549. archivebox config --get CHROME_BINARY # view a specific value
  550. <br/>
  551. archivebox config --set CHROME_BINARY=chromium # persist a config using CLI
  552. # OR edit ArchiveBox.conf and add this under its existing [ARCHIVING_CONFIG] section:
  553. CHROME_BINARY=chromium
  554. # OR
  555. env CHROME_BINARY=chromium archivebox version # run with a one-off config
  556. </code></pre>
  557. <sub>These methods also work the same way when run inside Docker, see the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration">Docker Configuration</a> wiki page for details.</sub>
  558. </details><br/>
  559. The configuration is documented here: **[Configuration Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration)**, and loaded from: [`archivebox/config/`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config/).
  560. <a name="most-common-options-to-tweak"></a>
  561. <details>
  562. <summary><i>Expand to see the most common options to tweak...</i></summary>
  563. <pre lang="bash"><code style="white-space: pre-line">
  564. # e.g. archivebox config --set TIMEOUT=120
  565. # or docker compose run --rm archivebox config --set TIMEOUT=120
  566. <br/>
  567. TIMEOUT=240 # default: 60 add more seconds on slower networks
  568. CHECK_SSL_VALIDITY=False # default: True False = allow saving URLs w/ bad SSL
  569. <br/>
  570. PUBLIC_INDEX=True # default: True whether anon users can view index
  571. PERMISSIONS=public # default: public visibility for newly added snapshots
  572. PUBLIC_ADD_VIEW=False # default: False whether anon users can add new URLs
  573. <br/>
  574. USER_AGENT="Mozilla/5.0 ..." # change this to get around bot blocking
  575. </code></pre>
  576. </details>
  577. <br/>
  578. ## Dependencies
  579. To achieve high-fidelity archives in as many situations as possible, ArchiveBox depends on a variety of 3rd-party libraries and tools that specialize in extracting different types of content.
  580. > Under-the-hood, ArchiveBox uses [Django](https://www.djangoproject.com/start/overview/) to power its [Web UI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage), [Django Ninja](https://django-ninja.dev/) for the REST API, and [SQlite](https://www.sqlite.org/locrsf.html) + the filesystem to provide [fast & durable metadata storage](https://www.sqlite.org/locrsf.html) w/ [deterministic upgrades](https://stackoverflow.com/a/39976321/2156113).
  581. ArchiveBox bundles industry-standard tools like [Google Chrome](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install), [`wget`, `yt-dlp`, `readability`, etc.](#dependencies) internally, and its operation can be [tuned, secured, and extended](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) as-needed for many different applications.
  582. <br/>
  583. <details>
  584. <summary><i>Expand to learn more about ArchiveBox's internals & dependencies...</i></summary><br/>
  585. <blockquote>
  586. <p><em>TIP: For better security while running ArchiveBox, and to avoid polluting your host system with a bunch of sub-dependencies that you need to keep up-to-date,<strong>it is strongly recommended to use the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker">ā­ļø official Docker image</a></strong> which provides everything in an easy container with simple one-liner upgrades.</em></p>
  587. </blockquote>
  588. <ul>
  589. <li>Language: Python <code>&gt;=3.13</code></li>
  590. <li>Backend: <a href="https://www.djangoproject.com/">Django</a> + <a href="https://django-ninja.dev/">Django-Ninja</a> for REST API</li>
  591. <li>Frontend: <a href="https://docs.djangoproject.com/en/6.0/ref/contrib/admin/">Django Admin</a> + Vanilla HTML, CSS, JS</li>
  592. <li>Web Server: <a href="https://www.djangoproject.com/">Django</a> + <a href="https://uvicorn.dev/"><code>uvicorn</code></a> (ASGI; Daphne for <code>--debug</code>)</li>
  593. <li>Database: <a href="https://docs.djangoproject.com/en/6.0/ref/databases/#sqlite-notes">Django ORM</a> saving to <a href="https://www.sqlite.org/mostdeployed.html">SQLite3</a> <code>./data/index.sqlite3</code></li>
  594. <li>Job Queue: Custom orchestrator using <code>supervisord</code> for worker management</li>
  595. <li>Build/test/lint: <a href="https://github.com/astral-sh/uv"><code>uv</code></a> / <code>pyright</code>+<code>ty</code>+<code>pytest</code> / <code>ruff</code></li>
  596. <li>Subdependencies: <a href="https://github.com/ArchiveBox/abxpkg"><code>abxpkg</code></a> installs apt/brew/pip/npm pkgs at runtime (e.g. <code>yt-dlp</code>, <code>singlefile</code>, <code>readability</code>, <code>git</code>)</li>
  597. </ul>
  598. These optional subdependencies used for archiving sites include:
  599. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/62a02155-05d7-4f3e-8de5-75a50a145c4f" alt="archivebox --version CLI output screenshot showing dependencies installed" width="330px" align="right" style="max-width: 100%;">
  600. <ul>
  601. <li><code>chromium</code> / <code>chrome</code> (for screenshots, PDF, DOM HTML, and headless JS scripts)</li>
  602. <li><code>node</code> &amp; <code>npm</code> (for readability, mercury, and singlefile)</li>
  603. <li><code>wget</code> (for plain HTML, static files, and WARC saving)</li>
  604. <li><code>curl</code> (for fetching headers, favicon, and posting to Archive.org)</li>
  605. <li><code>yt-dlp</code> or <code>youtube-dl</code> (for audio, video, and subtitles)</li>
  606. <li><code>git</code> (for cloning git repos)</li>
  607. <li><code>singlefile</code> (for saving into a self-contained html file)</li>
  608. <li><code>postlight/parser</code> (for discussion threads, forums, and articles)</li>
  609. <li><code>readability</code> (for articles and long text content)</li>
  610. <li>and more as we grow...</li>
  611. </ul>
  612. You don't need to install every dependency by hand. ArchiveBox resolves every extractor dependency through <code>abxpkg</code>: it uses a compatible host installation when one is already available, and otherwise installs and manages the dependency for you.
  613. <pre lang="bash"><code style="white-space: pre-line"># install uv + archivebox first (see Quickstart instructions above)
  614. <br/>
  615. archivebox install # auto install all the extractors and extras
  616. archivebox --version # see info and check validity of installed dependencies
  617. </code></pre>
  618. Installing directly on <strong>Windows without Docker or WSL/WSL2/Cygwin is not officially supported</strong> (I cannot respond to Windows support tickets), but some advanced users have reported getting it working.
  619. <h4>Learn More</h4>
  620. <ul>
  621. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#3-install-runtime-dependencies-using-archivebox-install">Wiki: Install (Dependencies)</a></li>
  622. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install">Wiki: Chromium Install</a></li>
  623. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives">Wiki: Upgrading or Merging Archives</a></li>
  624. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#installing">Wiki: Troubleshooting (Installing)</a></li>
  625. </ul>
  626. </details>
  627. <br/>
  628. ## Archive Layout
  629. All of ArchiveBox's state (SQLite DB, content, config, logs, etc.) is stored in a single folder per collection.
  630. <br/>
  631. <details>
  632. <summary><i>Expand to learn more about the layout of Archivebox's data on-disk...</i></summary><br/>
  633. Data folders can be created anywhere (`~/archivebox/data` or `$PWD/data` as seen in our examples), and you can create as many data folders as you want to hold different collections.
  634. All <code>archivebox</code> CLI commands are designed to be run from inside an ArchiveBox data folder, starting with <code>archivebox init</code> to initialize a new collection inside an empty directory.
  635. <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data # just an example, can be anywhere
  636. archivebox init</code></pre>
  637. The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard <code>index.sqlite3</code> database in the root of the data folder (it can also be <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html">exported as static JSON/HTML</a>). Snapshot data is organized by user, date, domain, and UUID under <code>data/archive/users/</code>.
  638. <img src="https://user-images.githubusercontent.com/511499/117453293-c7b91600-af12-11eb-8a3f-aa48b0f9da3c.png" width="400px" align="right" style="float: right"/>
  639. <pre lang="bash"><code style="white-space: pre-line">data/
  640. index.sqlite3
  641. ArchiveBox.conf
  642. archive/
  643. 1617687755 -> users/admin/snapshots/20210406/example.com/SNAPSHOT_UUID/ # legacy style dirs have been moved to a new layout
  644. users/
  645. admin/
  646. snapshots/
  647. 20210406/
  648. example.com/
  649. SNAPSHOT_UUID/
  650. index.html
  651. index.jsonl
  652. screenshot/screenshot.png
  653. ytdlp/media/some_video.mp4
  654. wget/warc/example.com.warc.gz
  655. git/somerepo.git
  656. ...
  657. </code></pre>
  658. Each snapshot subfolder includes static metadata and plain extractor output files. Current releases do not create top-level <code>data/archive/TIMESTAMP</code> projections; legacy timestamp directories are migrated into the user-scoped tree by <code>archivebox update --migrate-only</code>.
  659. <h4>Learn More</h4>
  660. <ul>
  661. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage">Wiki: Setting Up Storage (SMB, NFS, S3, B2, Google Drive, etc.)</a></li>
  662. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout">Wiki: Usage (Disk Layout)</a></li>
  663. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#large-archives">Wiki: Usage (Large Archives)</a></li>
  664. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#output-folder">Wiki: Security Overview (Output Folder)</a></li>
  665. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
  666. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives">Wiki: Upgrading or Merging Archives</a></li>
  667. </ul>
  668. </details>
  669. <br/>
  670. ## Static Archive Exporting
  671. You can export your index as static HTML using `archivebox list` (so you can view it without an ArchiveBox server).
  672. <br/>
  673. <details>
  674. <summary><i>Expand to learn how to export your ArchiveBox collection...</i></summary><br/>
  675. <blockquote>
  676. <p><em>NOTE: These exports are not paginated, exporting many URLs or the entire archive at once may be slow. Use the filtering CLI flags on the <code>archivebox list</code> command to export specific Snapshots or ranges.</em></p>
  677. </blockquote>
  678. <pre lang="bash"><code style="white-space: pre-line"># archivebox list --help
  679. archivebox list --html --with-headers > index.html # export to static html table
  680. archivebox list --json --with-headers > index.json # export to json blob
  681. archivebox list --csv=timestamp,url,title > index.csv # export to csv spreadsheet
  682. # (if using Docker Compose, add the -T flag when piping)
  683. # docker compose run --rm -T archivebox list --html 'https://example.com' > index.html
  684. </code></pre>
  685. The paths in the static exports are relative, make sure to keep them next to your `./archive` folder when backing them up or viewing them.
  686. <h4>Learn More</h4>
  687. <ul>
  688. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html">Wiki: Publishing Your Archive (Exporting as Static HTML)</a></li>
  689. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing">Wiki: Security Overview (Publishing)</a></li>
  690. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#permissions">Wiki: Configuration (<code>PERMISSIONS</code>, <code>PUBLIC_INDEX</code>, <code>PUBLIC_ADD_VIEW</code>)</a></li>
  691. </ul>
  692. </details>
  693. <br/>
  694. <div align="center" style="text-align: center">
  695. <img src="https://docs.monadical.com/uploads/upload_b6900afc422ae699bfefa2dcda3306f3.png" width="100%" alt="security graphic"/>
  696. </div>
  697. ## Caveats
  698. ### Archiving Private Content
  699. <a id="archiving-private-urls"></a>
  700. If you're importing pages with private content or URLs containing secret tokens you don't want public (e.g Google Docs, paywalled content, unlisted videos, etc.), **you may want to disable some of the extractor methods to avoid leaking that content to 3rd party APIs or the public**.
  701. <br/>
  702. <details>
  703. <summary><i>Expand to learn about privacy, permissions, and user accounts...</i></summary>
  704. <pre lang="bash"><code style="white-space: pre-line"># don't save private content to ArchiveBox, e.g.:
  705. archivebox add 'https://docs.google.com/document/d/12345somePrivateDocument'
  706. archivebox add 'https://vimeo.com/somePrivateVideo'
  707. # restrict the main index, Snapshot content, and Add Page to authenticated users as-needed:
  708. archivebox config --set PUBLIC_INDEX=False
  709. archivebox config --set PERMISSIONS=private
  710. archivebox config --set PUBLIC_ADD_VIEW=False
  711. archivebox manage createsuperuser
  712. </code></pre>
  713. <blockquote>
  714. <p><em>CAUTION: Assume anyone <em>viewing</em> your archives will be able to see any cookies, session tokens, or private URLs passed to ArchiveBox during archiving.</em>
  715. <em>Make sure to secure your ArchiveBox data and don't share snapshots with others without stripping out sensitive headers and content first.</em></p>
  716. </blockquote>
  717. <h4>Learn More</h4>
  718. <ul>
  719. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
  720. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
  721. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">Wiki: Chromium Install (Setting Up a User Profile)</a></li>
  722. <li><a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/docs/Chromium-Install.md#import-an-existing-browser-profile">Wiki: Chrome profiles and cookies</a></li>
  723. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#default_persona">Wiki: Configuration (<code>DEFAULT_PERSONA</code>)</a></li>
  724. </ul>
  725. </details>
  726. <br/>
  727. ### Security Risks of Viewing Archived JS
  728. Archived JavaScript is untrusted content. The default <code>SERVER_SECURITY_MODE=auto</code> uses isolated subdomains with full replay on <code>*.localhost</code>, and a one-domain no-JS replay policy on ordinary public or LAN hostnames. Choose <code>safe-subdomains-fullreplay</code> only when wildcard DNS and TLS are configured. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) and [Issue #239](https://github.com/ArchiveBox/ArchiveBox/issues/239) for details.
  729. <br/>
  730. <details>
  731. <summary><i>Expand to see risks and mitigations...</i></summary>
  732. <pre lang="bash"><code style="white-space: pre-line"># Explicit wildcard mode: full replay on isolated snapshot subdomains
  733. archivebox config --set SERVER_SECURITY_MODE=safe-subdomains-fullreplay
  734. # Alternative for deployments without wildcard subdomains: disable JS replay
  735. archivebox config --set SERVER_SECURITY_MODE=safe-onedomain-nojsreplay
  736. </code></pre>
  737. <blockquote>
  738. <p><em>NOTE: Only the <code>wget</code> &amp; <code>dom</code> extractor methods execute archived JS when viewing snapshots, all other archive methods produce static output that does not execute JS on viewing.</em><br/>
  739. <em>If you do not need JavaScript-capable replay at all, you can also disable those extractors with:<br/> <code>archivebox config --set WGET_ENABLED=False DOM_ENABLED=False</code>.</em></p>
  740. </blockquote>
  741. <h4>Learn More</h4>
  742. <ul>
  743. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
  744. <li><a href="https://github.com/ArchiveBox/ArchiveBox/issues/239">ArchiveBox Github Issue: #239</a></li>
  745. <li><a href="https://github.com/ArchiveBox/ArchiveBox/security/advisories/GHSA-cr45-98w9-gwqx">Security Advisory: <code>CVE-2023-45815</code></a></li>
  746. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing">Wiki: Security Overview (Publishing)</a></li>
  747. </ul>
  748. </details>
  749. <br/>
  750. ### Working Around Sites that Block Archiving
  751. For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) actively block archiving or bots in general. There are a number of approaches to work around this.
  752. <br/>
  753. <details>
  754. <summary><i>Click to learn how to set up user agents, cookies, and site logins...</i></summary>
  755. <br/>
  756. <ul>
  757. <li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/docs/Chromium-Install.md#import-an-existing-browser-profile">importing a chrome profile into a persona</a></li>
  758. <li>Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g.<br>
  759. <code>reddit.com/some/url</code> -&gt; <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
  760. </ul>
  761. </details>
  762. <br/>
  763. ### Saving Multiple Snapshots of a Single URL
  764. ArchiveBox can preserve multiple snapshots of the same URL. The default <code>ONLY_NEW=True</code> skips URLs already in the collection; use <code>--no-only-new</code> when you intentionally want another snapshot.
  765. <br/>
  766. <details>
  767. <summary><i>Click to learn how the <code>Re-Snapshot</code> feature works...</i></summary>
  768. <br/>
  769. Each re-archive creates a distinct Snapshot row for the same URL:
  770. <pre lang="bash"><code style="white-space: pre-line">archivebox add 'https://example.com'
  771. archivebox add --no-only-new 'https://example.com'
  772. </code></pre>
  773. The <img src="https://user-images.githubusercontent.com/511499/115942091-73c02300-a476-11eb-958e-5c1fc04da488.png" alt="Re-Snapshot Button" height="24px"/> button in the Admin UI performs the same explicit re-archive.
  774. <code>Archive Now</code> creates a new capture and preserves the selected historical
  775. snapshot. <code>Redo</code> reruns extraction on the selected snapshot in place.
  776. <h4>Learn More</h4>
  777. <ul>
  778. <li><a href="https://github.com/ArchiveBox/ArchiveBox/issues/179">ArchiveBox Issues: #179</a></li>
  779. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#explanation-of-buttons-in-the-web-ui---admin-snapshots-list">Wiki: Usage (Explanation of Web UI Buttons)</a></li>
  780. </ul>
  781. </details>
  782. <br/>
  783. ### Storage Requirements
  784. Because ArchiveBox is designed to ingest a large volume of URLs with multiple copies of each URL stored by different 3rd-party tools, it can be quite disk-space intensive. There are also some special requirements when using filesystems like NFS/SMB/FUSE.
  785. <br/>
  786. <details>
  787. <summary><i>Click to learn more about ArchiveBox's filesystem and hosting requirements...</i></summary>
  788. <br/>
  789. <ul>
  790. <li><strong>ArchiveBox can use anywhere from ~1gb per 1000 Snapshots, to ~50gb per 1000 Snapshots</strong>, mostly dependent on whether you're saving video/audio using <code>YTDLP_ENABLED=True</code> and whether you lower <code>YTDLP_MAX_SIZE=750m</code>.</li>
  791. <li>Disk usage can be reduced by using a compressed/<a href="https://www.ixsystems.com/blog/ixsystems-and-klara-systems-celebrate-valentines-day-with-a-heartfelt-donation-of-fast-dedupe-to-openzfs-and-truenas/">deduplicated</a> filesystem like <a href="https://www.reddit.com/r/zfs/comments/t9cexx/a_simple_real_world_zfs_compression_speed_an/">ZFS</a>/BTRFS, or by turning off extractors methods you don't need. You can also deduplicate content with a tool like <a href="https://github.com/adrianlopezroche/fdupes"><code>fdupes</code></a> or <a href="https://github.com/pauldreik/rdfind"><code>rdfind</code></a>.
  792. </li>
  793. <li><strong>Don't store large collections on older filesystems like EXT3/FAT</strong> as they may not be able to handle more than 50k directory entries in the <code>data/archive/</code> folder.
  794. </li>
  795. <li><strong>Keep the <code>data/index.sqlite3</code> file on a reliable local drive (not a network mount)</strong> or SSD when using SQLite; the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
  796. <li>Do not synchronize a live SQLite database or its <code>-wal</code>, <code>-shm</code>, or <code>-journal</code> files with Syncthing or similar file-sync tools. SQLite coordinates its own readers and writers; copying or replacing those files externally bypasses that coordination.</li>
  797. <li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, configure the server-side UID/GID mapping or ACL so ArchiveBox's non-root user can create and remove files. <code>root_squash</code> is compatible when the selected non-root identity has write access; do not rely on container root to override server-side permissions.
  798. </li>
  799. </ul>
  800. <h4>Learn More</h4>
  801. <ul>
  802. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout">Wiki: Usage (Disk Layout)</a></li>
  803. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#output-folder">Wiki: Security Overview (Output-Folder)</a></li>
  804. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#large-archives">Wiki: Usage (Large Archives)</a></li>
  805. <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#run-archivebox-as-an-unprivileged-user">Wiki: Security Overview (Privilege Dropping)</a></li>
  806. </ul>
  807. </details>
  808. <br/>
  809. ---
  810. <br/>
  811. ## Screenshots
  812. <div align="center" width="80%">
  813. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0362bcd1-1dfd-43c6-b4ec-db5e78da07b2" width="80%"/>
  814. <table>
  815. <tbody>
  816. <tr>
  817. <td>
  818. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e9fdcb16-344e-48c8-8be0-efa48ec155d5" alt="brew install archivebox" width="210px"><br/>
  819. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e235c9d8-fda9-499d-a6a5-59b0e6a0efce" alt="archivebox version" width="210px">
  820. </td>
  821. <td>
  822. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0e3da0c7-d2c2-4a71-b096-6caedafd6ef7" alt="archivebox init" width="210px"><br/>
  823. </td>
  824. <td>
  825. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/31c47440-ec14-4a02-99a3-aae8a9078d46" alt="archivebox add" width="210px">
  826. </td>
  827. <td>
  828. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ebcdc21d-e2af-4bf8-ad4b-bc4f3151bbef" alt="archivebox data dir" width="210px">
  829. </td>
  830. </tr>
  831. <tr>
  832. <td>
  833. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ea672e6b-4df5-49d8-b550-7f450951fd27" alt="archivebox server" width="210px">
  834. </td>
  835. <td>
  836. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e1e5bd78-b0b6-45dc-914c-e1046fee4bc4" alt="archivebox server add" width="210px">
  837. </td>
  838. <td>
  839. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e8e0b6f8-8fdf-4b7f-8124-c10d8699bdb2" alt="archivebox server list" width="210px">
  840. </td>
  841. <td>
  842. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ace0954a-ddac-4520-9d18-1c77b1ec50b2" alt="archivebox server detail" width="210px">
  843. </td>
  844. </tr>
  845. </tbody>
  846. </table>
  847. </div>
  848. <br/>
  849. <br/>
  850. <div align="center" style="text-align: center">
  851. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ca85432e-a2df-40c6-968f-51a1ef99b24e" width="100%" alt="paisley graphic">
  852. </div>
  853. <a id="background--motivation"></a>
  854. # Background & Motivation
  855. ArchiveBox aims to enable more of the internet to be saved from deterioration by empowering people to self-host their own archives. The intent is for all the web content you care about to be viewable with common software in 50 - 100 years without needing to run ArchiveBox or other specialized software to replay it.
  856. <br/>
  857. <details>
  858. <summary><i>Click to read more about why archiving is important and how to do it ethically...</i></summary>
  859. <br/>
  860. Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity.
  861. Whether it's to resist censorship by saving news articles before they get taken down or edited, or just to save a collection of early 2010's flash games you loved to play, having the tools to archive internet content enables to you save the stuff you care most about before it disappears.
  862. <div align="center" style="text-align: center">
  863. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/71e36bc5-1c94-44e2-92b6-405fa898c734" width="40%"/><br/>
  864. <sup><i>Image from <a href="https://perma.cc/">Perma.cc</a>...</i><br/></sup>
  865. </div>
  866. The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. I don't think everything should be preserved in an automated fashion--making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about, just like libraries do. Without the work of archivists saving physical books, manuscrips, and paintings we wouldn't have any knowledge of our ancestors' history. I believe archiving the web is just as important to provide the same benefit to future generations.
  867. ArchiveBox's stance is that duplication of other people's content is only ethical if it:
  868. - A. doesn't deprive the original creators of revenue and
  869. - B. is responsibly curated by an individual/institution.
  870. In the U.S., <a href="https://guides.library.oregonstate.edu/copyright/libraries">libraries, researchers, and archivists</a> are allowed to duplicate copyrighted materials under <a href="https://libguides.ala.org/copyright/fairuse">"fair use"</a> for <a href="https://guides.cuny.edu/cunyfairuse/librarians#:~:text=One%20of%20these%20specified%20conditions,may%20be%20liable%20for%20copyright">private study, scholarship, or research</a>. Archive.org's non-profit preservation work is <a href="https://blog.archive.org/2024/03/01/fair-use-in-action-at-the-internet-archive/">covered under fair use</a> in the US, and they properly handle <a href="https://cardozoaelj.com/2015/03/20/use-of-copyright-law-to-take-down-revenge-porn/">unethical content</a>/<a href="https://help.archive.org/help/rights/">DMCA</a>/<a href="https://gdpr.eu/right-to-be-forgotten/#:~:text=An%20individual%20has%20the%20right,that%20individual%20withdraws%20their%20consent.">GDPR</a> removal requests to maintain good standing in the eyes of the law.
  871. As long as you A. don't try to profit off pirating copyrighted content and B. have processes in place to respond to removal requests, many countries allow you to use software like ArchiveBox to ethically and responsibly archive any web content you can view. That being said, ArchiveBox is not liable for how you choose to operate the software. You must research your own local laws and regulations, and get proper legal counsel if you plan to host a public instance (start by putting your DMCA/GDPR contact info in <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#footer_info"><code>FOOTER_INFO</code></a> and placing branding overrides in your collection's fixed <code>custom_templates/</code> directory).
  872. </details>
  873. <br/>
  874. ## Comparison to Other Projects
  875. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/4cac62a9-e8fb-425b-85a3-ca644aa6dd42" width="5%" align="right" alt="comparison" style="float: right"/>
  876. > **Check out our [community wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) for a list of alternative web archiving tools and orgs.**
  877. ArchiveBox gained momentum in the internet archiving industry because it uniquely combines 3 things:
  878. - **it's distributed:** users own their data instead of entrusting it to one big central provider
  879. - **it's future-proof:** saving in *multiple formats* and extracting out raw TXT, PNG, PDF, MP4, etc. files
  880. - **it's extensible:** with powerful APIs, flexible storage, and a big community adding new extractors regularly
  881. <br/>
  882. <details>
  883. <summary><i>Expand for a more direct comparison to Archive.org and specific open-source alternatives...</i></summary><br/>
  884. ArchiveBox tries to be a robust, set-and-forget archiving solution suitable for archiving RSS feeds, bookmarks, or your entire browsing history (beware, it may be too big to store), including private/authenticated content that you wouldn't otherwise share with a centralized service like Archive.org.
  885. <h3>Comparison With Centralized Public Archives</h3>
  886. Not all content is suitable to be archived on a centralized, publicly accessible platform. Archive.org doesn't offer the ability to save things behind login walls for good reason, as the content may not have been intended for a public audience. ArchiveBox exists to fill that gap by letting everyone save what they have access to on an individual basis, and to encourage decentralized archiving that's less succeptible to censorship or natural disasters.
  887. By having users store their content locally or within their organizations, we can also save much larger portions of the internet than a centralized service has the disk capacity to handle. The eventual goal is to work towards federated archiving where users can share portions of their collections with each other, and with central archives on a case-by-case basis.
  888. <h3>Comparison With Other Self-Hosted Archiving Options</h3>
  889. ArchiveBox differentiates itself from [similar self-hosted projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-projects) by providing both a comprehensive CLI interface for managing your archive, a Web UI that can be used either independently or together with the CLI, and a simple on-disk data format that can be used without either.
  890. *If you want better fidelity for very complex interactive pages with heavy JS/streams/API requests, check out [ArchiveWeb.page](https://archiveweb.page) and [ReplayWeb.page](https://replayweb.page).*
  891. *If you want more bookmark categorization and note-taking features, check out [Memex](https://github.com/WorldBrain/Memex), [Hoarder](https://github.com/hoarder-app/hoarder), [LinkWarden](https://github.com/linkwarden/linkwarden), [Archivy](https://archivy.github.io/), or [LinkAce](https://www.linkace.org/).*
  892. *If you need more advanced recursive spider/crawling ability beyond `--depth=1`, check out [Browsertrix](https://github.com/webrecorder/browsertrix-crawler), [Photon](https://github.com/s0md3v/Photon), or [Scrapy](https://scrapy.org/) and pipe the outputted URLs into ArchiveBox.*
  893. For more alternatives, see our [list here](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-projects)...
  894. ArchiveBox is neither the highest fidelity nor the simplest tool available for self-hosted archiving, rather it's a jack-of-all-trades that tries to do most things well by default. We encourage you to try these other tools made by our friends if ArchiveBox isn't suited to your needs.
  895. </details>
  896. <br/>
  897. ## Internet Archiving Ecosystem
  898. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/78d8a725-97f4-47f5-b983-1f62843ddc51" width="14%" align="right" style="float: right"/>
  899. <details>
  900. <summary><i>Our <b><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community">Community Wiki</a></b> strives to be a comprehensive index of the web archiving industry...</i></summary>
  901. <br/>
  902. - [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
  903. - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-projects)
  904. _List of ArchiveBox alternatives and open source projects in the internet archiving space._
  905. - [Awesome-Web-Archiving Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#the-master-lists)
  906. _Community-maintained indexes of archiving tools and institutions like `iipc/awesome-web-archiving`._
  907. - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#reading-list)
  908. _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._
  909. - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#communities)
  910. _A collection of the most active internet archiving communities and initiatives._
  911. - Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog)
  912. - Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://items.ssrc.org/parameters/on-the-importance-of-web-archiving/)" blog post.
  913. - Reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter
  914. </details>
  915. <br/>
  916. <div align="center" style="text-align: center">
  917. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/897f7a88-1265-4aab-b80c-b1640afaad1f" width="100%" alt="documentation graphic">
  918. </div>
  919. # Documentation
  920. <img src="https://read-the-docs-guidelines.readthedocs-hosted.com/_images/logo-dark.png" width="13%" align="right" style="float: right"/>
  921. We use the [ArchiveBox GitHub Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki) for documentation.
  922. <sub>There is also a mirror available on <a href="https://archivebox.readthedocs.io/en/latest/">Read the Docs</a> (though it's sometimes outdated).</sub>
  923. > āœļø You can submit docs changes & suggestions in our dedicated repo [`ArchiveBox/docs`](https://github.com/ArchiveBox/docs).
  924. ## Getting Started
  925. - [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart)
  926. - [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install)
  927. - [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker)
  928. - [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage)
  929. - [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration)
  930. - [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
  931. - [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox#output-formats)
  932. - [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving)
  933. ## Advanced
  934. - [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview)
  935. - [Cookies & Sessions Setup](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile) (archiving sites that require logins)
  936. - [Setting up the Search Backends](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search) (choosing ripgrep, Sonic, or FTS5)
  937. - [Setting up Local/Remote Storages](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Storage) (S3/B2/Google Drive/SMB/NFS/etc.)
  938. - [Native OIDC, OAuth, and SAML SSO](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#saml--oauth2--openid-authentication) (optional django-allauth integration)
  939. - [Setting up Authentication & Permissions](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) (SSO/LDAP/OAuth/API Keys/etc.)
  940. - [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) (sharing your archive server with others)
  941. - [Chromium Install Options](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) (installing and configuring ArchiveBox's Chrome)
  942. - [Upgrading or Merging Archives](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives)
  943. - [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
  944. New SSO accounts are administrators. The legacy `DEFAULT_USER_PERMISSIONS` setting is no longer supported, including `none`. Registration is disabled by default; when enabled, `REGISTRATION_MODE=approval` keeps new accounts inactive until an administrator approves them. `REGISTRATION_MODE=open` activates new administrator accounts immediately.
  945. ## Developers
  946. - [Developer Documentation](https://github.com/ArchiveBox/ArchiveBox#archivebox-development)
  947. - [Python API](https://docs.archivebox.io/)
  948. - [REST API](https://demo.archivebox.io/api) (alpha)
  949. ## More Info
  950. - [Bug Tracker](https://github.com/ArchiveBox/ArchiveBox/issues)
  951. - [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)
  952. - [Changelog](https://github.com/ArchiveBox/ArchiveBox/releases)
  953. - [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
  954. - [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation)
  955. - [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
  956. <br/>
  957. ---
  958. <div align="center" style="text-align: center">
  959. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e895e79f-5c7d-429b-ad8a-7df2cc183ca3" width="100%" alt="development">
  960. </div>
  961. # ArchiveBox Development
  962. All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap.
  963. For low hanging fruit / easy first tickets, see: <a href="https://github.com/ArchiveBox/ArchiveBox/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22help+wanted%22">ArchiveBox/Issues `#good first ticket` `#help wanted`</a>.
  964. **Python API Documentation:** https://docs.archivebox.io/dev/apidocs/
  965. **Internal Architecture Diagrams:** https://github.com/ArchiveBox/ArchiveBox/wiki/ArchiveBox-Architecture-Diagrams
  966. ### Setup the dev environment
  967. <details><summary><i>Click to expand...</i></summary>
  968. #### 1. Setup the monorepo
  969. First make sure you have `uv` installed: https://docs.astral.sh/uv/getting-started/installation/
  970. ```bash
  971. git clone https://github.com/ArchiveBox/monorepo
  972. cd monorepo
  973. ./bin/setup.sh
  974. # activate the monorepo venv, then cd into archivebox
  975. source .venv/bin/activate
  976. cd archivebox
  977. ```
  978. Repos included in monorepo setup:
  979. - https://github.com/ArchiveBox/abxbus
  980. - https://github.com/ArchiveBox/abxpkg
  981. - https://github.com/ArchiveBox/abx-plugins
  982. - https://github.com/ArchiveBox/abx-dl
  983. - https://github.com/ArchiveBox/ArchiveBox
  984. #### 2. Option A: Install the Python, JS, and system dependencies directly on your machine
  985. ```bash
  986. # Install ArchiveBox runtime dependencies
  987. mkdir -p data && cd data
  988. archivebox init
  989. archivebox install # detect and install all extractor dependencies
  990. # Run the development server w/ autoreloading (but no bg workers)
  991. archivebox server --debug --reload 0.0.0.0:5797
  992. # Run the production server (with bg workers but no autoreloading)
  993. archivebox server 0.0.0.0:5797
  994. ```
  995. #### 2. Option B: Build the docker container and use that for development instead
  996. ```bash
  997. # Optional: develop via docker by mounting the code dir into the container
  998. # if you edit e.g. ./archivebox/core/models.py on the docker host, runserver
  999. # inside the container will reload and pick up your changes
  1000. ./bin/build_docker.sh dev
  1001. # Run the development server w/ autoreloading (but no bg workers)
  1002. docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:5797
  1003. # Run the production server (with bg workers but no autoreloading)
  1004. docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init
  1005. # (remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)
  1006. # When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.
  1007. ```
  1008. </details>
  1009. ### Common development tasks
  1010. See the `./bin/` folder and read the source of the bash scripts within.
  1011. You can also run all these in Docker. For more examples see the GitHub Actions CI/CD tests that are run: `.github/workflows/*.yaml`.
  1012. #### Run in DEBUG mode
  1013. <details><summary><i>Click to expand...</i></summary>
  1014. ```bash
  1015. # set up persistent DEBUG=True for all runs
  1016. archivebox config --set DEBUG=True
  1017. # OR you can run a dev server with DEBUG=True in a few ways:
  1018. archivebox server --debug --reload 0.0.0.0:5797
  1019. # or
  1020. archivebox server --debug 0.0.0.0:5797
  1021. # or
  1022. env DEBUG=True daphne -b 0.0.0.0 -p 5797 archivebox.core.asgi:application
  1023. ```
  1024. https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running
  1025. </details>
  1026. #### Install and run a specific GitHub branch
  1027. <details><summary><i>Click to expand...</i></summary>
  1028. ##### Use a Pre-Built Image
  1029. If you're looking for the latest `dev` Docker image, it's often available pre-built on Docker Hub, simply pull and use `archivebox/archivebox:dev`.
  1030. ```bash
  1031. docker pull archivebox/archivebox:dev
  1032. docker run archivebox/archivebox:dev version
  1033. # verify the BUILD_TIME and COMMIT_HASH in the output are recent
  1034. ```
  1035. ##### Build Branch from Source
  1036. You can also build and run any branch yourself from source, for example to build & use `dev` locally:
  1037. ```bash
  1038. # docker-compose.yml:
  1039. services:
  1040. archivebox:
  1041. image: archivebox/archivebox:dev
  1042. build: 'https://github.com/ArchiveBox/ArchiveBox.git#dev'
  1043. ...
  1044. # or with plain Docker:
  1045. docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
  1046. docker run -it -v $PWD:/data -p 5797:5797 archivebox:dev
  1047. # or with uv:
  1048. uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
  1049. archivebox install
  1050. ```
  1051. </details>
  1052. #### Run the linters / tests
  1053. <details><summary><i>Click to expand...</i></summary>
  1054. ```bash
  1055. ./bin/lint.sh
  1056. ./bin/test.sh
  1057. ```
  1058. (uses `ruff`, `pyright`, `ty`, and `pytest -s`)
  1059. </details>
  1060. #### Make DB migrations, enter Django shell, other dev helper commands
  1061. <details><summary><i>Click to expand...</i></summary>
  1062. ```bash
  1063. # generate the database migrations after changes to models.py
  1064. archivebox manage makemigrations
  1065. # enter a python shell or a SQL shell
  1066. cd path/to/test/data/
  1067. archivebox shell
  1068. archivebox manage dbshell
  1069. # generate a graph of the ORM models
  1070. brew install graphviz
  1071. uv run --with pydot --with graphviz archivebox manage graph_models -a -o orm.png
  1072. open orm.png
  1073. # list all models with field db info and methods
  1074. archivebox manage list_model_info --all --signature --db-type --field-class
  1075. # print all django settings
  1076. archivebox manage print_settings
  1077. uv run --with pyyaml archivebox manage print_settings --format=yaml
  1078. # autogenerate an admin.py from given app models
  1079. archivebox manage admin_generator core > core/admin.py
  1080. # dump db data to a script that re-populates it
  1081. archivebox manage dumpscript core > scripts/testdata.py
  1082. archivebox manage reset core
  1083. archivebox manage runscript testdata
  1084. # resetdb and clear all data!
  1085. archivebox manage reset_db
  1086. # use django-tui to interactively explore commands
  1087. # ensure django-tui is in INSTALLED_APPS: core/settings.py
  1088. uv run --with django-tui archivebox manage tui
  1089. ```
  1090. <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/dc3e9f8c-9544-46e0-a7f0-30f571b72022" width="600px" alt="ArchiveBox ORM models relatinoship graph"/>
  1091. - https://django-extensions.readthedocs.io/en/latest/command_extensions.html
  1092. - https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running
  1093. - https://github.com/anze3db/django-tui (explore `manage.py` commands as TUI)
  1094. - https://github.com/bloomberg/memray (advanced python profiler)
  1095. - https://github.com/laixintao/flameshow (display flamegraphs in terminal)
  1096. - https://github.com/taliraj/django-migrations-tui (explore migrations as TUI)
  1097. </details>
  1098. #### Contributing a new extractor
  1099. Extractors are maintained in a separate repo here: https://github.com/ArchiveBox/abx-plugins (included in monorepo setup).
  1100. Copy a similar plugin as a template to modify, then open a new PR to add it in that repo.
  1101. #### Build the docs, pip package, and docker image
  1102. <details><summary><i>Click to expand...</i></summary>
  1103. (Normally CI takes care of this, but these scripts can be run to do it manually)
  1104. ```bash
  1105. ./bin/build.sh
  1106. # or individually:
  1107. ./bin/build_docs.sh
  1108. ./bin/build_pip.sh
  1109. ./bin/build_docker.sh
  1110. ```
  1111. </details>
  1112. ---
  1113. ## Further Reading
  1114. <img src="https://raw.githubusercontent.com/Monadical-SAS/redux-time/HEAD/examples/static/jeremy.jpg" width="100px" align="right"/>
  1115. - [ArchiveBox.io Website](https://archivebox.io) / [ArchiveBox Github (Source Code)](https://github.com/ArchiveBox/ArchiveBox) / [ArchiveBox Demo Server](https://demo.archivebox.io)
  1116. - [Documentation (Github Wiki)](https://github.com/ArchiveBox/ArchiveBox/wiki) / [API Reference Docs (ReadTheDocs)](https://docs.archivebox.io) / [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) / [Changelog](https://github.com/ArchiveBox/ArchiveBox/releases)
  1117. - [Bug Tracker (Github Issues)](https://github.com/ArchiveBox/ArchiveBox/issues) / [Discussions (Github Discussions)](https://github.com/ArchiveBox/ArchiveBox/discussions) / [Community Chat Forum (Zulip)](https://zulip.archivebox.io)
  1118. - Find us on social media: [Twitter `@ArchiveBoxApp`](https://twitter.com/ArchiveBoxApp), [LinkedIn](https://www.linkedin.com/company/archivebox/), [YouTube](https://www.youtube.com/@ArchiveBoxApp), [SaaSHub](https://www.saashub.com/archivebox), [Alternative.to](https://alternativeto.net/software/archivebox/about/), [Reddit](https://www.reddit.com/r/ArchiveBox/)
  1119. ---
  1120. <br/>
  1121. <div align="center" style="text-align: center">
  1122. <a href="https://hcb.hackclub.com/donations/start/archivebox"><img src="https://img.shields.io/badge/Donate-Directly-%13DE5D26.svg"/></a> &nbsp;
  1123. <a href="https://github.com/sponsors/pirate"><img src="https://img.shields.io/badge/Github_Sponsors-%23B7CDFE.svg"/></a> &nbsp;
  1124. <a href="https://www.patreon.com/theSquashSH"><img src="https://img.shields.io/badge/Patreon-%23DD5D76.svg"/></a> &nbsp;
  1125. <a href="https://paypal.me/NicholasSweeting"><img src="https://img.shields.io/badge/Paypal-%23FFD141.svg"/></a> &nbsp;
  1126. <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Donations"><img src="https://img.shields.io/badge/BTC%5CETH-%231a1a1a.svg"/></a> &nbsp;
  1127. <a href="https://archivebox-shop.fourthwall.com/"><img src="https://img.shields.io/badge/Merch-%23903851.svg"/></a>
  1128. <br/>
  1129. <sup><i>ArchiveBox operates as a US 501(c)(3) nonprofit <a href="https://en.wikipedia.org/wiki/Fiscal_sponsorship">FSP</a> (sponsored by <a href="https://hackclub.com/hcb?ref=donation">HCB</a>), <a href="https://hcb.hackclub.com/donations/start/archivebox">direct donations</a> are tax-deductible.</i></sup>
  1130. <br/><br/>
  1131. <a href="https://twitter.com/ArchiveBoxApp"><img src="https://img.shields.io/badge/Tweet-%40ArchiveBoxApp-blue.svg?style=flat"/></a>&nbsp;
  1132. <a href="https://github.com/ArchiveBox/ArchiveBox"><img src="https://img.shields.io/github/stars/ArchiveBox/ArchiveBox.svg?style=flat&label=Star+on+Github"/></a>&nbsp;
  1133. <a href="https://zulip.archivebox.io/"><img src="https://img.shields.io/badge/Join_Our_Community-Zulip_Forum-%23B7EDFE.svg"/></a><br/><br/>
  1134. <hr/>
  1135. <i>✨ Have spare CPU/disk/bandwidth after all your ē½‘ē«™å­˜ę”£ēˆ¬ and want to help the world?<br/>Check out our <a href="https://github.com/ArchiveBox/good-karma-kit">Good Karma Kit</a>...</i>
  1136. </div>
  1137. ### Folders, importers, and query exports
  1138. `archivebox folder` provides `create`, `get`, `list`, `update`, `delete`, `rules`,
  1139. and `move-snapshots`. Smart folders are native Folder records with a JSON `rules`
  1140. tree; `folder rules` describes the supported fields, operators, and ordering.
  1141. Updates validate cycles and rule references through the same model as the UI.
  1142. Deleting a folder preserves captures. Folder lists support `--format csv|tsv|json|jsonl`.
  1143. ```bash
  1144. archivebox folder create Reading
  1145. archivebox folder create News --rules '{"field":"domain","lookup":"exact","value":"news.ycombinator.com"}'
  1146. archivebox folder list --format csv
  1147. archivebox list --folder-id FOLDER_UUID --format jsonl
  1148. archivebox list --export-query 'folder=FOLDER_UUID&q=research&search_mode=meta' --format html_zip --output /absolute/path/export
  1149. archivebox list --export-query 'folder=FOLDER_UUID' --format wacz --output /absolute/path/wacz-export
  1150. ```
  1151. The authenticated REST API exposes `/api/v1/core/folders` (GET/POST),
  1152. `/api/v1/core/folder/{id}` (GET/PATCH/DELETE), `/api/v1/core/folders/rules` (GET),
  1153. and `/api/v1/core/folders/move-snapshots` (POST). Lists accept `format=csv|tsv|jsonl|json`.
  1154. Snapshot lists accept `folder_id` for regular and smart membership. Snapshot and
  1155. Crawl PATCH accept `folder_id`, including explicit `null` to clear it.
  1156. Snapshot metadata exports use `/api/v1/core/snapshots/export/{format}`;
  1157. POST to the same route starts a static ZIP or unified WACZ job. Export filters
  1158. accept the snapshot view query string (including folder, search, domain, tags,
  1159. and visibility); `folder_id`, `search`, and `status` also accept API spellings.
  1160. Job responses include status, download, and cancellation URLs. ZIP/WACZ exports
  1161. preserve the existing permission checks and native Process lifecycle.
  1162. CSV/JSONL snapshot records include folder, owner, and persona IDs alongside names.
  1163. Snapshot REST responses expose `persona_name`; `persona` contains the same value.
  1164. `archivebox importer` exposes source CRUD, catalog discovery, queued actions, and
  1165. run history. See [Importers](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/importers/README.md) for commands and REST
  1166. endpoints. MCP exposes `folder` and `importer` as native action tools; snapshot
  1167. export options are available through its existing `snapshot` list action.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.