Skip to content
Back to search

ref-downloader

Batch-download reference PDFs from a DOI or paper PDF using Crossref and your institutional Edge session.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/ltczding-gif/ref-downloader with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/ltczding-gif/ref-downloader'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ltczding-gif--ref-downloader/badge.svg)](/listings/ltczding-gif--ref-downloader)

HTML

<a href="/listings/ltczding-gif--ref-downloader"><img src="/api/listings/ltczding-gif--ref-downloader/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

The skill is self-contained under `skills/ref-downloader/`. Pick the install path for your agent framework:

git clone https://github.com/ltczding-gif/ref-downloader.git

# Pick ONE install destination for your agent framework:
#   Claude Code:        cp -r ref-downloader/skills/ref-downloader ~/.claude/skills/
#   Codex CLI:          cp -r ref-downloader/skills/ref-downloader ~/.codex/skills/
#   Copilot CLI / VSC:  cp -r ref-downloader/skills/ref-downloader .github/skills/
#   Project-local:      cp -r ref-downloader/skills/ref-downloader .agents/skills/

cd ~/.claude/skills/ref-downloader     # or wherever you copied it
pip install playwright pymupdf
playwright install msedge
cp config.example.toml config.local.toml      # then set [crossref].mailto

# In your agent: just describe the task; the skill triggers via its description.
# Direct CLI for testing: python scripts/run_ref_downloader.py 10.1021/jacs.5c05017

What you'll see: 30–80 refs discovered for a typical chemistry/physics paper, then a mix of `downloaded` (refs your institution covers), `manual_pending` (SSO bounce or paywall), and occasional `failed` (publisher quirk). Run on a DOI from a journal your institution actually subscribes to for the highest hit rate. Details below.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 313 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # ref-downloader
  2. > **Stop losing an afternoon to chasing dozens of reference PDFs by hand.**
  3. > One DOI in, every reference PDF out — using your existing institutional access.
  4. [![Version: 0.4.1](https://img.shields.io/badge/version-0.4.1-orange.svg)](CHANGELOG.md)
  5. [![Status: beta](https://img.shields.io/badge/status-beta-orange.svg)](#known-limitations)
  6. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
  7. [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
  8. ![Verified on Windows + Edge](https://img.shields.io/badge/verified%20on-Windows%20+%20Edge-success)
  9. [中文完整文档 / Full Chinese version](README.zh.md)
  10. > **Status: beta (v0.4.1).** Windows + Microsoft Edge verified path. macOS / Linux / Chromium untested. Expect rough edges around supplementary downloads and publisher-site changes. PR-worthy issues welcome.
  11. > **Heads up — not a paywall bypass.** ref-downloader uses _your_ institutional access. If your university or organization subscribes to a journal, those refs work. If they don't, those refs become `manual_pending` for you to follow up on by hand.
  12. ## Demo (30-second console preview)
  13. ```text
  14. $ python run_ref_downloader.py 10.1021/jacs.5c05017
  15. === Ref Downloader Wrapper ===
  16. DOI: 10.1021/jacs.5c05017
  17. PROJECT: jacs.5c05017
  18. Config: config.example.toml + config.local.toml
  19. >>> extract_refs.py
  20. Title: Designing Natural Cell-Inspired Heme-Spurred Membrane...
  21. References found: 38
  22. >>> validate_refs.py
  23. Total: 38 Verified: 38 Failed: 0 No DOI: 0
  24. >>> download_refs.py
  25. [ 1] downloaded (842 KB) Lee2016_NatEnergy.pdf
  26. [ 2] downloaded (1.2 MB) Wang2018_AdvMater.pdf
  27. [ 3] manual_pending (auth_redirect)
  28. [ 4] downloaded (655 KB) Chen2019_JACS.pdf
  29. [ 5] failed (challenge_timeout)
  30. [ 6] ignored (ignored_institution_access)
  31. ... 31 more refs processed ...
  32. [38] downloaded (956 KB) Park2024_JElectrochemSoc.pdf
  33. ========== Download report ==========
  34. Total references: 38
  35. Main PDFs: 33 downloaded · 3 manual_pending · 1 failed · 1 ignored
  36. SI files: 12 captured
  37. PDFs land in: ./jacs.5c05017_refs/jacs.5c05017/
  38. =====================================
  39. ```
  40. ## Contents
  41. - [What you get](#what-you-get)
  42. - [Why not Zotero, scihub, or generic scrapers?](#why-not-zotero-scihub-or-generic-scrapers)
  43. - [Quick start](#quick-start)
  44. - [Requirements](#requirements)
  45. - [Install](#install)
  46. - [Usage examples](#usage-examples)
  47. - [Configuration](#configuration)
  48. - [Architecture](#architecture)
  49. - [Supported publishers](#supported-publishers)
  50. - [Known limitations](#known-limitations)
  51. - [Contributing](#contributing)
  52. - [Security](#security)
  53. - [License](#license)
  54. ## What you get
  55. - **Paywalled refs work without setup.** _Drives your real Microsoft Edge profile, so any institutional login already in your browser carries through. No API keys, no proxies, no reverse engineering._
  56. - **One DOI in, every reference PDF out.** _Crossref-driven extraction + 17+ publisher-specific download paths (Wiley PDFDirect, Elsevier viewer, AIP loading-page wait — see [per-publisher reliability tier](docs/SUPPORTED_PUBLISHERS.md)), not generic scraping._
  57. - **Two routing modes from an agent** (v0.4.1+). _**Mode A**: hand the agent one paper, get all of its references. **Mode B**: hand the agent a custom list — DOIs / titles / arXiv-PMID IDs / BibTeX / abstract queries like "Wang 2024 Nature Energy papers" — and it resolves to DOIs and downloads them. See [SKILL.md](skills/ref-downloader/SKILL.md)._
  58. - **You always know which refs failed and why.** _`download_report.csv` gives every ref a status + reason (`manual_pending (auth_redirect)`, `failed (challenge_timeout)`, `ignored`); `events.jsonl` keeps the per-ref event trace._
  59. - **Pick up where you left off** after a VPN drop, browser crash, or `Ctrl+C`. _State persists per project; rerunning skips already-downloaded refs and retries only the failures._
  60. ## Why not Zotero, scihub, or generic scrapers?
  61. - **vs. Zotero's _Find Available PDF_** — walks one paper at a time and silently gives up at SSO redirects. ref-downloader walks the whole reference list at once and treats SSO as a configurable step instead of a dead end.
  62. - **vs. scihub-style tools** — don't carry your institutional license, so paywalled refs you _legitimately_ have access to just fail. ref-downloader uses your authenticated browser session, so subscriptions you already pay for actually count.
  63. - **vs. generic web scrapers** — don't know Wiley needs PDFDirect, Elsevier needs a viewer click, or AIP serves a Chinese loading page first. ref-downloader has 17+ publisher-specific paths plus Elsevier popup state machine + `--auto` mode retry queue (manual-pending refs get a second async attempt 60s later, hot-session preserved).
  64. - **vs. raw Playwright** — gets blocked on Cloudflare / Radware / Turnstile-heavy sites. Set `REF_DOWNLOADER_BROWSER=cloak` to swap in [cloakbrowser](https://pypi.org/project/cloakbrowser/)'s stealth Chromium with humanized input — no code changes, same pipeline. See [Configuration](#configuration).
  65. ## Quick start
  66. The skill is self-contained under `skills/ref-downloader/`. Pick the install path for your agent framework:
  67. ```powershell
  68. git clone https://github.com/ltczding-gif/ref-downloader.git
  69. # Pick ONE install destination for your agent framework:
  70. # Claude Code: cp -r ref-downloader/skills/ref-downloader ~/.claude/skills/
  71. # Codex CLI: cp -r ref-downloader/skills/ref-downloader ~/.codex/skills/
  72. # Copilot CLI / VSC: cp -r ref-downloader/skills/ref-downloader .github/skills/
  73. # Project-local: cp -r ref-downloader/skills/ref-downloader .agents/skills/
  74. cd ~/.claude/skills/ref-downloader # or wherever you copied it
  75. pip install playwright pymupdf
  76. playwright install msedge
  77. cp config.example.toml config.local.toml # then set [crossref].mailto
  78. # In your agent: just describe the task; the skill triggers via its description.
  79. # Direct CLI for testing: python scripts/run_ref_downloader.py 10.1021/jacs.5c05017
  80. ```
  81. What you'll see: 30–80 refs discovered for a typical chemistry/physics paper, then a mix of `downloaded` (refs your institution covers), `manual_pending` (SSO bounce or paywall), and occasional `failed` (publisher quirk). Run on a DOI from a journal your institution actually subscribes to for the highest hit rate. Details below.
  82. ## Requirements
  83. - **OS**: Windows 10/11 (verified). macOS / Linux untested — PRs welcome.
  84. - **Browser**: Microsoft Edge (Stable channel). The script claims your persistent Edge profile, so close all Edge windows before running.
  85. - **Python**: 3.11 or newer (uses stdlib `tomllib`).
  86. - **Optional**: A Zotero installation (auto-detects DOI from a PDF's filename via Zotero's SQLite database — much faster than text extraction).
  87. - **Optional**: PyMuPDF (`pip install pymupdf`) for DOI extraction from PDF text when Zotero lookup is unavailable.
  88. ## Install
  89. ### As an agent skill (recommended)
  90. Pick the install path for your agent framework:
  91. | Framework | Install command |
  92. |---|---|
  93. | Claude Code | `cp -r skills/ref-downloader ~/.claude/skills/` |
  94. | Claude Agent SDK | same (auto-discovers `~/.claude/skills/`) |
  95. | Codex CLI | `cp -r skills/ref-downloader ~/.codex/skills/` |
  96. | Copilot CLI / VS Code agent | `cp -r skills/ref-downloader .github/skills/` |
  97. | Any framework (project-local) | `cp -r skills/ref-downloader .agents/skills/` |
  98. Then install Python prereqs INSIDE the copied skill folder (the skill protocol doesn't manage Python deps):
  99. ```powershell
  100. cd ~/.claude/skills/ref-downloader # or wherever you copied it
  101. pip install playwright pymupdf # or use the source's requirements.txt
  102. playwright install msedge
  103. cp config.example.toml config.local.toml
  104. # Edit config.local.toml — at minimum set [crossref].mailto.
  105. # Windows: notepad config.local.toml
  106. # macOS / Linux: $EDITOR config.local.toml (or vim / nano / code / ...)
  107. ```
  108. ### As a Python tool (for developers)
  109. If you want to hack on the code, the skill folder _is_ a runnable Python project:
  110. ```powershell
  111. git clone https://github.com/ltczding-gif/ref-downloader.git
  112. cd ref-downloader
  113. pip install -r requirements.txt -r requirements-dev.txt
  114. playwright install msedge
  115. cp skills/ref-downloader/config.example.toml skills/ref-downloader/config.local.toml
  116. # Edit config.local.toml — at minimum set [crossref].mailto.
  117. # Run the offline test suite
  118. python -m pytest tests/ -v
  119. # Run the tool directly
  120. python skills/ref-downloader/scripts/run_ref_downloader.py 10.1021/jacs.5c05017
  121. ```
  122. ## Usage examples
  123. (After install — paths assume the skill is at `<SKILL_DIR>`, e.g. `~/.claude/skills/ref-downloader/`. In source, `<SKILL_DIR>` = `skills/ref-downloader/`.)
  124. ### Input: a DOI
  125. ```powershell
  126. python <SKILL_DIR>/scripts/run_ref_downloader.py 10.1021/jacs.5c05017
  127. ```
  128. Default output: `<cwd>/jacs.5c05017_refs/jacs.5c05017/`
  129. ### Input: a local PDF (with DOI in metadata or in PDF text)
  130. ```powershell
  131. python <SKILL_DIR>/scripts/run_ref_downloader.py "C:\path\to\your_paper.pdf"
  132. ```
  133. Default output: `<pdf_dir>/your_paper_refs/<doi-derived-name>/`
  134. ### Custom output directory
  135. ```powershell
  136. python <SKILL_DIR>/scripts/run_ref_downloader.py 10.1021/jacs.5c05017 --output-dir refs/
  137. ```
  138. ### Non-interactive (CI / batch)
  139. ```powershell
  140. python <SKILL_DIR>/scripts/run_ref_downloader.py 10.1021/jacs.5c05017 --yes --auto
  141. ```
  142. ### Alternate config file
  143. ```powershell
  144. python <SKILL_DIR>/scripts/run_ref_downloader.py 10.1021/jacs.5c05017 --config ./alt.toml
  145. ```
  146. ## Configuration
  147. All configuration lives in `config.local.toml` (gitignored). Copy `config.example.toml` to bootstrap.
  148. | Section | Key | Purpose |
  149. |---|---|---|
  150. | `[crossref]` | `mailto` | Your email — entry into Crossref polite pool |
  151. | `[zotero]` | `db_path` | Optional path to `zotero.sqlite` for DOI lookup from PDF filename |
  152. | `[browser]` | `edge_profile_dir` | Edge profile directory; empty = OS default |
  153. | `[browser]` | `disable_extensions` | Set `true` to launch with `--disable-extensions` |
  154. | `[institution]` | `auth_hosts` | Hostnames that mean "you got bounced to SSO" (e.g. `["sso.your-uni.edu"]`) |
  155. | `[institution]` | `auth_url_fragments` | URL substrings indicating SSO (e.g. `["oauth", "saml"]`) |
  156. | `[institution]` | `auth_page_titles` | `<title>` text for SSO pages (catches HTML served as PDF) |
  157. | `[institution]` | `auth_loading_titles` | Loading-page titles (also reused for AIP/AVS publisher loading detection) |
  158. | `[institution]` | `ignored_access_dois` | DOIs you know are paywalled at your institution; skipped without retry |
  159. Environment variables override file values:
  160. | Variable | Maps to |
  161. |---|---|
  162. | `REF_DOWNLOADER_MAILTO` | `crossref.mailto` |
  163. | `REF_DOWNLOADER_ZOTERO_DB` | `zotero.db_path` |
  164. | `REF_DOWNLOADER_EDGE_PROFILE` | `browser.edge_profile_dir` |
  165. | `REF_DOWNLOADER_DISABLE_EXTENSIONS` | `browser.disable_extensions` (`1`/`true` to enable) |
  166. | `REF_DOWNLOADER_CONFIG` | Path to alternate TOML file |
  167. See [`skills/ref-downloader/config.example.toml`](skills/ref-downloader/config.example.toml) for full documentation.
  168. ### Alternative backend: CloakBrowser (optional, for Cloudflare-heavy sites)
  169. **What it is.** [CloakBrowser](https://github.com/CloakHQ/CloakBrowser) is a third-party Python package by CloakHQ (MIT-licensed, available on [PyPI](https://pypi.org/project/cloakbrowser/) as `cloakbrowser`). It ships a patched Chromium build with source-level anti-fingerprint changes designed to look like a normal browser to common bot-detection layers (Cloudflare Turnstile, Radware, DataDome, FingerprintJS, etc). Its `launch_persistent_context_async()` API is intentionally compatible with Playwright's — that's what lets ref-downloader swap backends with a single env var instead of rewriting the download flow.
  170. **Not a dependency of ref-downloader.** If you don't run `pip install cloakbrowser` it's never imported. The default Edge backend is unchanged. When CloakBrowser IS the active backend, ref-downloader uses Chromium under a **separate persistent profile** at `~/.local/cloakbrowser/profiles/ref-downloader` (or `REF_DOWNLOADER_CLOAK_PROFILE`), so your Edge profile is not touched — Edge does NOT need to be closed.
  171. **When to use it.** Sites you'd reach for it on: CCS Chemistry (`10.31635`, Cloudflare-protected), some Elsevier paths gated by Radware, anything where the Edge backend keeps producing `manual_pending (radware_bot_manager)` or `failed (challenge_timeout)`. **Don't** reach for it as a default — the Edge backend is more reliable when your institutional access is the actual bottleneck, because Edge carries your authenticated cookies.
  172. **Caveats.** CloakBrowser is **beta** third-party software; install + use at your own discretion (review its [repo](https://github.com/CloakHQ/CloakBrowser) before pulling it). It is **not a captcha solver** — interactive challenges still need you. It also does not carry your institutional cookies (separate profile), so it's most useful for open-Cloudflare sites, less useful for paywalled-but-license-covered refs.
  173. ```powershell
  174. pip install cloakbrowser # one-time, separate from ref-downloader
  175. $env:REF_DOWNLOADER_BROWSER = "cloak"
  176. $env:REF_DOWNLOADER_CLOAK_HUMAN_PRESET = "careful" # optional: slower mouse/scroll
  177. python skills/ref-downloader/scripts/run_ref_downloader.py 10.31635/ccsorg...
  178. ```
  179. CloakBrowser env vars (all optional):
  180. | Variable | Default | Purpose |
  181. |---|---|---|
  182. | `REF_DOWNLOADER_BROWSER` | `edge` | Set to `cloak` (or `cloakbrowser`) to switch backend |
  183. | `REF_DOWNLOADER_CLOAK_PROFILE` | `~/.local/cloakbrowser/profiles/ref-downloader` | Persistent Chromium profile path |
  184. | `REF_DOWNLOADER_CLOAK_HUMANIZE` | `1` | `0`/`false` to disable humanized input |
  185. | `REF_DOWNLOADER_CLOAK_HUMAN_PRESET` | `default` | `default` or `careful` (slower) |
  186. | `REF_DOWNLOADER_CLOAK_PROXY` | _unset_ | HTTP/SOCKS proxy URL |
  187. | `REF_DOWNLOADER_CLOAK_GEOIP` | auto | `1` to force GeoIP rerouting (auto when proxy is set) |
  188. | `CLOAKBROWSER_PYTHONPATH` | _unset_ | sys.path hint for a local cloakbrowser source checkout |
  189. Notes:
  190. - **Edge does not need to be closed** when using the cloak backend — it uses its own Chromium.
  191. - A fresh cloak profile may still hit Cloudflare/security pages on first visit — warm it manually with that profile before batch downloads.
  192. - `human_preset=careful` reduces behavior-based detection but is **not** a captcha solver.
  193. - cloakbrowser is NOT a hard dependency of ref-downloader. If you never set `REF_DOWNLOADER_BROWSER=cloak`, it's not imported.
  194. ## Architecture
  195. Three-stage pipeline + a wrapper:
  196. ```
  197. skills/ref-downloader/
  198. ├── SKILL.md agent runbook (slim entry)
  199. ├── references/agent-runbook.md extended manual flow + DOI fallback
  200. ├── config.example.toml config schema (copy to config.local.toml)
  201. └── scripts/
  202. ├── run_ref_downloader.py entry — config + DOI resolution + sequencing
  203. │ └─> extract_refs.py (1) Crossref API: fetch parent's reference list
  204. │ └─> validate_refs.py (2) Crossref API: per-ref metadata + publisher classify
  205. │ └─> download_refs.py (3) Playwright/Edge: download main PDF + SI per publisher
  206. └── _config.py TOML + env-var loader
  207. ```
  208. You can also run the three scripts manually for debugging or partial restarts. See the agent runbook in [`skills/ref-downloader/references/agent-runbook.md`](skills/ref-downloader/references/agent-runbook.md) for the manual flow.
  209. Agent users can install or inspect the packaged skill at [`skills/ref-downloader/SKILL.md`](skills/ref-downloader/SKILL.md). The repository root remains the human-facing Python project; the skill bundle is kept separate so Codex does not treat README, changelog, tests, and source files as always-associated skill context.
  210. ## Supported publishers
  211. ACS, Nature, Science, Elsevier, Wiley, RSC, Springer, PNAS, ECS, IOP, AIP, AVS, IEEE, OSA, KPS, Beilstein, APS, Annual Reviews, Taylor & Francis, CCS Chemistry. Maturity varies — see [`docs/SUPPORTED_PUBLISHERS.md`](docs/SUPPORTED_PUBLISHERS.md) for the per-publisher tier table and known issues. CCS Chemistry sits behind Cloudflare; pair it with `REF_DOWNLOADER_BROWSER=cloak` for reliable access.
  212. ## Known limitations
  213. - **Windows + Microsoft Edge only**: that's the verified path. macOS / Linux / Chromium support has not been tested. If you try, please open an issue with results.
  214. - **Headed mode required**: empirically, `headless=True` yields empty results for Wiley / ACS supplementary downloads. The default is headed.
  215. - **Edge must be fully closed before running**: Playwright needs exclusive access to the persistent profile. Check Task Manager for any background `msedge.exe` processes.
  216. - **SSO redirects are detected, not solved**: when the script bounces to your institution's SSO, the ref becomes `manual_pending` so you can sign in interactively. Configure `[institution]` to teach it which redirects to recognize.
  217. - **SI download is the most fragile path**: main PDFs are reliable; SI lookup varies by publisher and is the area most likely to need a tweak when a publisher updates their site.
  218. - **Paywalled content needs institutional access**: this is not a bypass tool.
  219. - **Crossref dependency**: papers with no reference list deposited at Crossref can't be processed automatically.
  220. ## Contributing
  221. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidance on:
  222. - Adding a new publisher (DOI prefix → strategy)
  223. - Adding institutional SSO patterns
  224. - Reporting download failures with useful logs
  225. ## Security
  226. This tool launches your real Edge profile, with all your cookies and saved sessions. Read [SECURITY.md](SECURITY.md) before running it against a profile you also use for daily browsing.
  227. ## License
  228. MIT — see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.