Skip to content
Back to search

paper-preflight

MarketplaceWorks with Claude CodeGitHub

paper-preflight: pre-submission reference checks for LaTeX papers.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/amos689--paper-preflight/badge.svg)](/listings/amos689--paper-preflight)

HTML

<a href="/listings/amos689--paper-preflight"><img src="/api/listings/amos689--paper-preflight/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

With [uv](https://docs.astral.sh/uv/) nothing needs installing (or `pip install paper-preflight`):

uvx paper-preflight check path/to/paper

`path/to/paper` is the project directory, its main `.tex` file, or a single `.bib` file. A project that ships no `.bib`, as many arXiv sources do, is read from its compiled `.bbl` (checked, but never edited).

Writing in Word, Markdown or Typst? The manuscript is checked the same way:

uvx paper-preflight check paper.docx     # Word: Zotero, Mendeley or EndNote citations, or the typed list
uvx paper-preflight check paper.qmd      # Markdown, Quarto, R Markdown: [@key] against its bibliography
uvx paper-preflight check paper.typ      # Typst: @key against #bibliography(...), .bib or Hayagriva .yml

In a Word manuscript, citations inserted by Zotero, Mendeley or EndNote carry the reference manager's own record of each work (field codes), which is read first; Word's source manager next; else the reference list as typed, after its "References" heading. A bibliography may also be checked on its own as CSL-JSON (`.json`), RIS (`.ris`) or YAML (`.yml`, Hayagriva or CSL).

No manuscript at all? A reference list as plain text works too, in the common styles (APA, IEEE, ACM, Nature, Vancouver, Springer, Elsevier, MDPI, GOST, Chicago, MLA), one reference per line, per paragraph or numbered:

uvx paper-preflight check references.txt
pbpaste | uvx paper-preflight check -      # or from stdin

Any arXiv paper, by its ID: the source is downloaded to a temporary folder, checked, and deleted.

uvx paper-preflight check arxiv:2607.06922

Only the PDF? Its reference list is read too, with the `pdf` extra:

uvx --from 'paper-preflight[pdf]' paper-preflight check paper.pdf

| Option | Effect | |---|---| | `--format json` / `--format sarif` | Machine-readable output (SARIF works with GitHub code scanning) | | `--offline` | Never touch the network; use only answers already in the local cache | | `--refresh` | Ask every source again instead of using cached answers (after a correction, say) | | `--recheck` | Search again now for references found too new to be indexed on earlier runs (otherwise once a day) | | `--fail-on warning` | Make warnings fail the run too (the default is errors) | | `--details` | List every finding; by default a suggestion that applies to many entries (published preprints, available DOIs) is one line | | `--lang zh` | Chinese messages (also chosen automatically from your locale) |

Exit codes:

| Code | Meaning | |---|---| | 0 | Nothing at or above `--fail-on` was found | | 1 | Blocking findings | | 2 | No blocking findings, but a source was unavailable and nothing could answer in its place, so the paper cannot be called clean yet | | 3 | Usage error | | 4 | Internal error (please report it) |

The JSON report follows [a published JSON Schema](docs/schema/check-report.schema.json) (`schema_version` 0.1): fields may be added, but none is removed or changes meaning without a new `schema_version`. The tests check every report against it.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 17 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "paper-preflight",
  3. "owner": {
  4. "name": "amos689",
  5. "url": "https://github.com/amos689"
  6. },
  7. "description": "paper-preflight: pre-submission reference checks for LaTeX papers.",
  8. "plugins": [
  9. {
  10. "name": "paper-preflight",
  11. "source": "./plugins/paper-preflight",
  12. "description": "Verify every reference of a LaTeX paper against real scholarly records before submission: an MCP server plus a skill that runs the check and fixes what it proves wrong.",
  13. "category": "productivity",
  14. "tags": ["latex", "bibtex", "citations", "research"]
  15. }
  16. ]
  17. }
README.md · README · 565 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <!-- mcp-name: io.github.amos689/paper-preflight -->
  2. <div align="center">
  3. ![paper-preflight](docs/assets/brand/paper-preflight-logo.svg)
  4. **Check every reference of a paper against real scholarly records before you submit.<br>
  5. No LLM guessing, no false accusations.**
  6. [![CI](https://github.com/amos689/paper-preflight/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/amos689/paper-preflight/actions/workflows/ci.yml)
  7. [![PyPI version](https://img.shields.io/pypi/v/paper-preflight?label=PyPI&color=2f6fb0)](https://pypi.org/project/paper-preflight/)
  8. [![MIT license](docs/assets/badges/license.en.svg)](LICENSE)
  9. [![paper-preflight MCP server on Glama](https://glama.ai/mcp/servers/amos689/paper-preflight/badges/score.svg)](https://glama.ai/mcp/servers/amos689/paper-preflight)
  10. [![Python 3.11 to 3.14](docs/assets/badges/python.en.svg)](pyproject.toml)
  11. [![Input: LaTeX, Word, Markdown and PDF](docs/assets/badges/input.en.svg)](#what-it-reads)
  12. [![Checked against six scholarly databases](docs/assets/badges/sources.en.svg)](#how-it-works)
  13. [![No LLM in the verdicts](docs/assets/badges/verdicts.en.svg)](#design-principles)
  14. [![Read-only MCP tools](docs/assets/badges/mcp.en.svg)](docs/mcp.md)
  15. [![Tested on Windows, Linux and macOS](docs/assets/badges/platforms.en.svg)](https://github.com/amos689/paper-preflight/actions/workflows/ci.yml)
  16. [![English and Simplified Chinese](docs/assets/badges/languages.en.svg)](README.zh-CN.md)
  17. **English** · [简体中文](README.zh-CN.md) · [Quick start](#quick-start) ·
  18. [User guide](docs/README.md) · [Releases](https://github.com/amos689/paper-preflight/releases) ·
  19. [Feedback](https://github.com/amos689/paper-preflight/issues/new/choose)
  20. </div>
  21. ![paper-preflight checking the demo paper: errors for an undefined citation key, a DOI that belongs to another paper, a reference no source knows, a retracted paper and a duplicate entry key; warnings for a published preprint, two entries for the same work, a wrong year and a LaTeX-escaped DOI](https://raw.githubusercontent.com/amos689/paper-preflight/main/docs/demo/demo.gif)
  22. Language models invent references, and copy-pasted BibTeX carries wrong years, wrong authors
  23. and dead DOIs. paper-preflight reads your paper (LaTeX, Word, Markdown, Typst, a PDF or a plain
  24. reference list) and asks Crossref, dblp, arXiv, DataCite, PubMed and OpenAlex about every cited
  25. work, with Semantic Scholar too if you have a free key, Open Library for books without a DOI,
  26. and GitHub, PyPI, CRAN, Hugging Face and OpenML for software, models and datasets:
  27. - Does it exist?
  28. - Does it match what you wrote?
  29. - Has it been retracted?
  30. - Has the preprint you cite been published since?
  31. Every finding is backed by a record, or by every source that could know the work answering
  32. "no". When it cannot tell, it says "cannot determine" and why, instead of guessing. No language
  33. model takes part in a verdict.
  34. > **Status: 1.0.** On three weeks of real arXiv papers held out for the release: 0.79 false
  35. > alarms per 100 references, against 409 real problems found ([accuracy](#how-accurate-is-it)).
  36. > Rule IDs, the JSON report, settings, commands and the Python API are kept through 1.x
  37. > ([what is stable](docs/stability.md)). False positives are the bugs we most want to hear
  38. > about: please [open an issue](https://github.com/amos689/paper-preflight/issues).
  39. ## Quick start
  40. With [uv](https://docs.astral.sh/uv/) nothing needs installing:
  41. ```bash
  42. uvx paper-preflight check path/to/paper
  43. ```
  44. `path/to/paper` is the project's folder, its main `.tex` file or a `.bib` file. To install it
  45. for good (Python 3.11 or later):
  46. ```bash
  47. pip install paper-preflight
  48. ```
  49. | Extra | Adds | Install |
  50. |---|---|---|
  51. | `pdf` | Reading the reference list of a PDF | `pip install "paper-preflight[pdf]"` |
  52. | `mcp` | The MCP server for coding agents | `pip install "paper-preflight[mcp]"` |
  53. | `support` | The experimental evidence finder, with its local model | `pip install "paper-preflight[support]"` |
  54. The repository's [demo paper](examples/demo-paper) cites eleven works, several of them wrong on
  55. purpose. The animation above is a real run of `paper-preflight check examples/demo-paper`
  56. against the live sources. In it, the correct NeurIPS paper is verified through dblp although
  57. Crossref only holds fake copies of it, the book without an identifier is confirmed by Open
  58. Library, and the Chinese book, whose script is not supported yet, is "cannot determine", not
  59. "not found".
  60. <details>
  61. <summary>The same run as text</summary>
  62. ```text
  63. $ paper-preflight check examples/demo-paper
  64. paper-preflight 1.0.0 · main.tex · 12 entries, 12 cited keys
  65. error CIT001 main.tex:31
  66. Citation key 'nonexistent2023' is not defined in any bibliography file (1 use(s)).
  67. error REF001 refs.bib:43
  68. The doi of 'devlin2019bert' (10.1109/cvpr.2016.90) resolves to a different work in Crossref: "Deep Residual Learning for Image Recognition" (He et al., 2016).
  69. error REF003 refs.bib:66
  70. 'lindqvist2024quantum' was not found in Crossref, dblp and Semantic Scholar, and every source responded. Check that the work exists and that its title is correct.
  71. error REF004 refs.bib:73
  72. 'wakefield1998ileal' has been retracted (reported by Crossref, OpenAlex). Cite it only if the text discusses the retraction.
  73. error CIT002 refs.bib:128
  74. Entry key 'kingma2015adam' is already defined at line 47; BibTeX ignores this one.
  75. warning REF015 refs.bib:31
  76. 'he2015residual' cites a preprint that has been published in CVPR (2016), DOI 10.1109/cvpr.2016.90. Cite the published version and keep the eprint field.
  77. warning CIT004 refs.bib:37
  78. Entries 'devlin2019bert' and 'he2016deep' look like the same work (same DOI).
  79. warning REF013 refs.bib:51
  80. 'kingma2015adam' gives the year 2016, but dblp records 2014, 2015.
  81. warning REF017 refs.bib:112
  82. The doi of 'tacl2019example' contains LaTeX escapes: '10.1162/tacl\_a\_00276'. Write it as: 10.1162/tacl_a_00276
  83. info REF005 refs.bib:73
  84. 'wakefield1998ileal' has a published correction (reported by Crossref).
  85. info REF090 refs.bib:95
  86. 'zhou2016ml' could not be verified: non-Latin titles are not supported yet; grey literature without an identifier (book, report, software, web page).
  87. info CIT003 refs.bib:116
  88. Entry 'lecun1998gradient' is never cited.
  89. References: 7 verified · 1 metadata mismatch · 1 identifier conflict · 1 not found · 1 cannot determine
  90. 5 error(s) · 4 warning(s) · 3 info
  91. ```
  92. </details>
  93. ## What it reads
  94. | You have | Command |
  95. |---|---|
  96. | A LaTeX project: its folder, main `.tex` or `.bib`; a project that ships only its compiled `.bbl`, as many arXiv sources do, is read from it (checked, never edited) | `paper-preflight check path/to/paper` |
  97. | A Word manuscript: citations inserted by Zotero, Mendeley or EndNote, Word's own sources, or the reference list as typed | `paper-preflight check paper.docx` |
  98. | Markdown, Quarto or R Markdown: `[@key]` against its bibliography | `paper-preflight check paper.qmd` |
  99. | Typst: `@key` against `#bibliography(...)`, a `.bib` or Hayagriva `.yml` | `paper-preflight check paper.typ` |
  100. | A bibliography on its own: BibTeX, CSL-JSON (`.json`), RIS (`.ris`) or YAML (`.yml`) | `paper-preflight check refs.ris` |
  101. | A reference list as plain text, in the common styles (APA, IEEE, ACM, Nature, Vancouver, Springer, Elsevier, MDPI, GOST, Chicago, MLA) | `paper-preflight check references.txt`, or `-` for stdin |
  102. | Only the PDF (with the `pdf` extra) | `paper-preflight check paper.pdf` |
  103. | Any arXiv paper: its source is downloaded to a temporary folder, checked and deleted | `paper-preflight check arxiv:2607.06922` |
  104. What each input can and cannot tell, in detail: [docs/inputs.md](docs/inputs.md).
  105. ## What it checks
  106. Every reference gets one verdict:
  107. | Verdict | Means |
  108. |---|---|
  109. | verified | A record of the work was found and agrees with the entry |
  110. | metadata mismatch | The work was found, but the entry says something else about it: the findings name the field |
  111. | identifier conflict | The entry's DOI or arXiv ID belongs to another work |
  112. | not found | Every source that could know the work answered, and none has it |
  113. | cannot determine | It could not be decided, and the report says why |
  114. The findings say what is wrong:
  115. | Rule | Finding |
  116. |---|---|
  117. | REF001 | The DOI or arXiv ID points to a different paper |
  118. | REF002 | The DOI or arXiv ID does not exist |
  119. | REF003 | The work was not found in any source, and every source answered |
  120. | REF004 · REF005 | The work was retracted, or has an expression of concern or a correction |
  121. | REF010–REF014 | Authors, title, year or venue differ from the real record |
  122. | REF015 | A cited preprint has been formally published |
  123. | REF016 | The registry has a DOI the entry lacks (offered as a safe fix) |
  124. | REF017 | An identifier is written so that links break (`10.1162/tacl\_a\_00276`, `…v1`) |
  125. | REF018 | A cited arXiv preprint was withdrawn by its authors |
  126. | REF019 · REF021 | A linked repository, package or dataset does not exist; a linked web page is gone, with no archived copy |
  127. | REF020 | A preprint is cited with a venue no catalogue of journals and conferences has |
  128. | REF090 | Cannot determine, always with the reason (source unavailable, grey literature, …) |
  129. | CIT001–CIT008 | Undefined, duplicate, unused or near-duplicate citation keys; broken `.bib` syntax |
  130. | TEX001–TEX004 | Included files missing, unreadable or including each other; a remote bibliography left unchecked |
  131. | RUN001 · RUN002 · CFG001 | A run left incomplete by a source outage; a source answered in place of another; a suppression comment that silenced nothing |
  132. Every rule has [a page](docs/rules/README.md): what it checks, when it can be wrong, and what to
  133. do. `paper-preflight explain REF003` prints it in the terminal.
  134. ## How accurate is it?
  135. Four measurements, all against the live sources:
  136. | Measured on | Result |
  137. |---|---|
  138. | [Real papers](#real-papers): 60 arXiv papers from three weeks held out for 1.0, 2,714 references, every flag reviewed by hand | **0.79 false alarms per 100 references** (one every three papers), against 409 real problems |
  139. | [Hallucinations that passed peer review](#hallucinations-that-passed-peer-review): the 151 GPTZero found in NeurIPS 2025 and ICLR 2026 papers | **142 flagged (94%)**, 9 cannot determine, **none verified** |
  140. | [HALLMARK](#hallmark), a public benchmark: 831 entries never used during development | Any issue: precision 98.5%, recall 93.5%, false-positive rate 1.9% |
  141. | [Next to five other tools](#next-to-other-tools), on 104 references labelled by hand in a published study | Precision 72.5%, against 31–51% for the others, at 87.9% recall |
  142. Precision comes first: a reference is called fabricated only on positive evidence, and an
  143. unanswered or ambiguous lookup is reported as "cannot determine", never as "not found". The
  144. evaluation harness and every run's summary are in [`evals/`](evals/README.md).
  145. ### Real papers
  146. Each week is 20 arXiv papers (cs, stat, q-bio, quant-ph and astro-ph) chosen mechanically. For
  147. 1.0, three weeks never used before were committed together, before the changes they measure,
  148. and each was run once on the same code. The gate was at most 1.5 false alarms per 100
  149. references in each week and at most 1.0 on average. Above them are the more recent weeks that
  150. stayed under 1.0, each run once with the version of its time; every week measured, those over
  151. 1.0 too, is in [`evals/README.md`](evals/README.md#real-papers).
  152. | Papers first submitted | Version | References | Flags | Real problems | False alarms | Unclear | False alarms per 100 references |
  153. |---|---|---|---|---|---|---|---|
  154. | 2026-10-07..13 | 1.0 candidate, 1st try | 991 | 81 | 76 | 4 | 1 | 0.4 |
  155. | 2026-09-30..10-06 | 0.7.0 | 1,014 | 89 | 79 | 7 | 3 | 0.7 |
  156. | 2026-09-23..29 | 0.6.0 | 786 | 48 | 42 | 6 | 0 | 0.8 |
  157. | 2026-06-24..30 | 0.8.0 | 1,127 | 143 | 132 | 10 | 1 | 0.9 |
  158. | 2026-05-13..19 | 1.0 candidate, 1st try | 1,008 | 133 | 123 | 8 | 2 | 0.8 |
  159. | 2026-05-06..12 | 1.0 candidate, 2nd try | 950 | 133 | 126 | 7 | 0 | 0.7 |
  160. | 2026-04-08..14 | 1.0 candidate, 3rd try | 1,088 | 194 | 181 | 7 | 6 | 0.6 |
  161. | 2026-04-01..07 | 1.0 candidate, 3rd try | 763 | 86 | 80 | 6 | 0 | 0.8 |
  162. | 2026-03-25..31 | 1.0.0 | 809 | 220 | 209 | 8 | 3 | 1.0 |
  163. | 2026-03-18..24 | 1.0.0 | 1,007 | 110 | 105 | 5 | 0 | 0.5 |
  164. | 2026-03-11..17 | 1.0.0 | 898 | 104 | 95 | 8 | 1 | 0.9 |
  165. | **The three 1.0.0 weeks together** | **1.0.0** | **2,714** | **434** | **409** | **21** | **4** | **0.79** |
  166. - **The 409 real problems:** 220 cited preprints since published, 149 identifiers written so
  167. that links break, 30 entries with wrong authors, titles or years (NeurIPS 2025 papers dated
  168. 2026, invented co-authors, author fields BibTeX misreads), and 10 works that are not there as
  169. cited: 5 found nowhere, 3 identifiers that do not exist, 2 that belong to other works. A few
  170. papers' reference lists are largely invented (medical imaging, LLMs).
  171. - **The 21 false alarms:** registries' own errors (a cut or misspelt title, a garbled name, a
  172. dropped symbol) 5; groups credited as their makers ask and Zotero's "N other authors"
  173. placeholder 6; works no index holds or that a later version retitled 3; a conference paper
  174. compared in place of its journal version 2; APA's `[software]` and `[dataset]` labels after a
  175. title 2; three others. [When a finding is wrong](docs/false-positives.md) says what to do.
  176. - **The gate took four tries.** The first three averaged 1.08, 1.24 and 1.28 per 100
  177. references, each with one week over 1.5; each try's false alarms were fixed before the next,
  178. on new weeks. The first release, 0.1.0, measured 4.5. Every week measured since is in
  179. [`evals/README.md`](evals/README.md#real-papers).
  180. ### Hallucinations that passed peer review
  181. GPTZero published 151 hallucinated references it found in
  182. [NeurIPS 2025 papers](https://gptzero.me/news/neurips/) and
  183. [ICLR 2026 submissions](https://gptzero.me/news/iclr-2026/), each confirmed by its staff.
  184. Pasted as plain text, as the papers printed them:
  185. | References | Flagged | Cannot determine | Verified |
  186. |---|---|---|---|
  187. | 151 | **142 (94%)** | 9 | **0** |
  188. - **None of them is verified.** The 9 left undecided are web pages and blog posts, a workshop
  189. no source indexes, real titles given with invented authors where several works share the
  190. title, and two references too garbled to search. Each is listed with its reason in
  191. [`evals/results/gptzero.md`](evals/results/gptzero.md).
  192. - GPTZero's own tool found these, so they are the hallucinations a search can find; recall on
  193. every kind of hallucination is lower (see HALLMARK below).
  194. ### HALLMARK
  195. [HALLMARK](https://github.com/rpatrik96/hallmark) (v1.2.3) is a public benchmark of real and
  196. hallucinated BibTeX entries. *Fabrication* counts a wrong identifier, a work not found and no
  197. author in common; *any issue* also counts wrong authors, title, year or venue.
  198. | Split | Mode | Precision | Recall | False-positive rate | Coverage |
  199. |---|---|---|---|---|---|
  200. | `test_public`: 831 entries, never used during development | Any issue | 98.5% | 93.5% | 1.9% | 97.7% |
  201. | | Fabrication | 99.0% | 50.2% | 0.6% | 97.7% |
  202. | `dev_public`: 1,119 entries, used during development | Any issue | 97.7% | 94.3% | 2.1% | 98.9% |
  203. | | Fabrication | 98.2% | 53.7% | 1.0% | 98.9% |
  204. `test_public` was run with 1.0.0, `dev_public` with 0.8.0; no rule was tuned on `test_public`.
  205. - **Most of the false positives are disputed labels.** Every flag on a `dev_public` entry
  206. labelled VALID was checked by hand: the 11 that remain are not correct citations (DOIs that
  207. belong to other papers, author lists naming people who did not write the paper, a shifted
  208. year, a truncated title). Without them, both modes reach 100% precision. The list, each with
  209. a reason one lookup confirms: [`evals/hallmark_disputed.toml`](evals/hallmark_disputed.toml).
  210. - **What is still missed:** author lists cut short at the end, or checked only against a
  211. preprint whose later versions add people (named in an info), and invented entries that borrow
  212. a title several real works share. Every hallucination type is broken down in
  213. [`evals/results/`](evals/results/).
  214. ### Next to other tools
  215. [Badalova & Mayr (2026)](https://doi.org/10.5281/zenodo.21457492) checked 104 references by hand
  216. and published what five tools flagged. On the same references, with their labels:
  217. | Tool | Precision [95% CI] | Recall | False flags per 100 correct references |
  218. |---|---|---|---|
  219. | CheckIfExist | 47.7% [36.0%, 59.6%] | 93.9% | 47.9 |
  220. | HalluCiteChecker | 47.4% [32.5%, 62.7%] | 54.5% | 28.2 |
  221. | Hallucinator | 50.9% [38.3%, 63.4%] | 87.9% | 39.4 |
  222. | HalRef | 31.2% [21.9%, 42.2%] | 72.7% | 74.6 |
  223. | RefChecker | 47.1% [35.7%, 58.8%] | 97.0% | 50.7 |
  224. | **paper-preflight** | **72.5% [57.2%, 83.9%]** | 87.9% | **15.5** |
  225. The sample is small, so the intervals are wide. The study labels a reference correct when the
  226. work exists, so five of paper-preflight's "false" flags point at real errors (a wrong author, a
  227. broken DOI). Two causes of false flags found in this data were fixed before the run above (the
  228. first run measured 62.8%); see [`evals/results/badalova-mayr.md`](evals/results/badalova-mayr.md).
  229. ## Limits
  230. - **Titles in scripts other than Latin** (Chinese, Japanese, Cyrillic, …) are "cannot
  231. determine" for now, and so is a Chinese-language work cited by a translated English title,
  232. which the open indexes rarely hold.
  233. - **Grey literature without an identifier** (reports, theses, manuals, standards) is often
  234. "cannot determine": the open registries hold little of it. Books are asked of Open Library,
  235. software of GitHub, PyPI and CRAN, models and datasets of Hugging Face and OpenML.
  236. - **Web pages are never judged true or false.** A linked page that is gone, with no archived
  237. copy, is reported (REF021).
  238. - **A work no open index holds** is "cannot determine" when its venue is unknown to the
  239. catalogues too, but at a venue they know it can be reported as not found: read
  240. [REF003's page](docs/rules/REF003.md) before acting on that finding.
  241. - **Not caught:** an invented reference that borrows a real title several works share, an
  242. author list cut short at the end (an info), and a citation of a real paper for a claim it
  243. does not make (only the experimental [`support`](#experimental-find-the-passage-behind-each-citation)
  244. reads the claims).
  245. - **It needs the network** on a first run: about 0.4 s per reference on a cold cache, so a
  246. 50-reference paper takes about 20 seconds. Answers are cached, and `--offline` works from the
  247. cache. A source that is down, with nothing to answer in its place, makes the run incomplete
  248. (exit code 2), never "clean".
  249. ## Fetch verified BibTeX
  250. Instead of writing an entry from memory, ask for it by DOI, arXiv ID or title. Every field
  251. comes from the registry record, which a comment above the entry names:
  252. ```bash
  253. paper-preflight bib fetch 1810.04805
  254. ```
  255. ```bibtex
  256. % Verified with paper-preflight against dblp (conf/naacl/DevlinCLT19), 2026-10-03
  257. @inproceedings{devlin2019bert,
  258. title = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language Understanding},
  259. author = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  260. booktitle = {NAACL-HLT (1)},
  261. year = {2019},
  262. doi = {10.18653/v1/n19-1423},
  263. eprint = {1810.04805},
  264. archivePrefix = {arXiv},
  265. }
  266. ```
  267. - **Preprints:** an arXiv preprint that has been published comes back as the published version,
  268. with its `eprint` kept (`--prefer preprint` for the preprint itself).
  269. - **Titles:** `--title` (with `--author`/`--year` if needed) lists the candidates instead of
  270. choosing when several works match.
  271. - **Retractions:** a retracted work comes with a warning.
  272. - **Scripts and agents:** `--format json`.
  273. ## Fix the bibliography
  274. `bib fix` turns findings into edits of your `.bib` files, taken from the verified records. It
  275. prints a diff and changes nothing until you add `--apply`:
  276. ```bash
  277. paper-preflight bib fix path/to/paper --level unsafe
  278. ```
  279. ```diff
  280. --- a/refs.bib
  281. +++ b/refs.bib
  282. @@ -48,7 +47,7 @@
  283. title = {Adam: A Method for Stochastic Optimization},
  284. author = {Kingma, Diederik P. and Ba, Jimmy},
  285. booktitle = {International Conference on Learning Representations (ICLR)},
  286. - year = {2016},
  287. + year = {2015},
  288. }
  289. ```
  290. - `--level safe` (the default) only fixes what cannot change which work is cited: identifiers
  291. written so that links break, and DOIs the registry has but the entry lacks.
  292. - `--level unsafe` also rewrites authors, title, year and venue from the record, removes
  293. identifiers that point to another work, and cites a preprint as its published version (the
  294. entry type, venue, year, volume, pages and DOI; the eprint is kept). Review the diff first.
  295. - Only the affected fields change; comments, formatting, line endings and encoding are kept.
  296. A reference nobody could find is never "fixed": only you can say what was meant.
  297. ## Options and exit codes
  298. | Option | Effect |
  299. |---|---|
  300. | `--format json` / `--format sarif` | Machine-readable output (SARIF works with GitHub code scanning); `-o` writes it to a file |
  301. | `--fail-on warning` | Make warnings fail the run too (the default is errors; `never` for neither) |
  302. | `--offline` | Never touch the network; use only answers already in the local cache |
  303. | `--refresh` | Ask every source again instead of using cached answers (after a correction, say) |
  304. | `--recheck` | Search again now for references found too new to be indexed on earlier runs (otherwise once a day) |
  305. | `--details` | List every finding; by default a suggestion that applies to many entries (published preprints, available DOIs) is one line |
  306. | `--hide-info` | Leave out infos |
  307. | `--lang zh` | Chinese messages (also chosen automatically from your locale) |
  308. | Exit code | Meaning |
  309. |---|---|
  310. | 0 | Nothing at or above `--fail-on` was found |
  311. | 1 | Blocking findings |
  312. | 2 | No blocking findings, but a source was unavailable and nothing could answer in its place, so the paper cannot be called clean yet |
  313. | 3 | Usage error |
  314. | 4 | Internal error (please report it) |
  315. The JSON report follows [a published JSON Schema](docs/schema/check-report.schema.json)
  316. (`schema_version` 1.0): fields may be added, but none is removed or changes meaning within 1.x.
  317. Every option, the cache and its lifetimes: [docs/configuration.md](docs/configuration.md).
  318. ## Project settings
  319. For a whole project, put the settings in `paper-preflight.toml` next to the paper (or in a
  320. folder above it, up to the repository's root), or in `[tool.paper-preflight]` of
  321. `pyproject.toml`:
  322. ```toml
  323. ignore-rules = ["REF016"] # never report these rules
  324. ignore-keys = ["internal2024*"] # nor anything about these entries
  325. severity = { REF015 = "info" } # report published preprints as infos
  326. disable-sources = ["s2"] # optional sources only
  327. fail-on = "warning" # as --fail-on; the command line wins
  328. ```
  329. `check --config <path>` names the file instead. The optional sources are `s2`, `openlibrary`,
  330. `github`, `pypi`, `cran`, `huggingface`, `openml`, `web` (links to web pages) and `wayback`;
  331. the registries that judge a reference cannot be turned off. A mistake in the file (an unknown
  332. rule or setting) is an error, never silently ignored. The JSON report names the file in
  333. `run.notes`, and the MCP server reads it too.
  334. ### Silence a finding you have checked
  335. A comment directly above an entry silences rules for that entry, with an optional reason:
  336. ```bibtex
  337. % preflight: ignore[REF003] reason="internal technical report, not indexed anywhere"
  338. @techreport{lab2024internal,
  339. ...
  340. }
  341. ```
  342. The verdict stays in the JSON report; only the finding is dropped. A suppression that silenced
  343. nothing is reported as CFG001 (info), so stale comments do not pile up (for reference rules,
  344. only after a complete online run, since offline answers and outages may leave them unrun).
  345. ### Better results with free credentials
  346. paper-preflight works without any account. These optional environment variables make it faster
  347. and more complete; their values are never printed or logged.
  348. | Variable | Effect |
  349. |---|---|
  350. | `PAPER_PREFLIGHT_EMAIL` | Crossref's polite pool: faster, more reliable lookups |
  351. | `OPENALEX_API_KEY` | A larger OpenAlex budget for retraction checks |
  352. | `S2_API_KEY` | Semantic Scholar as a rescue source for references nobody else found |
  353. | `GITHUB_TOKEN` | GitHub's larger budget (5,000 requests an hour, not 60) for references to repositories |
  354. | `NCBI_API_KEY` | PubMed's larger budget (10 requests a second, not 3) for PMIDs and PMCIDs |
  355. `paper-preflight doctor` shows which are set and whether each source answers right now.
  356. ## Use it in your workflow
  357. **Claude Code:** install the plugin. It bundles an MCP server and a skill that makes Claude
  358. check the references before calling a paper finished, fix only what is proven wrong, and never
  359. invent a reference.
  360. ```bash
  361. claude plugin marketplace add amos689/paper-preflight
  362. ```
  363. ```bash
  364. claude plugin install paper-preflight@paper-preflight
  365. ```
  366. **Codex, Gemini CLI, Copilot, Cursor and other agents:** install the same skill
  367. (`gh skill install amos689/paper-preflight paper-preflight` works too). It runs the CLI when no
  368. MCP server is configured:
  369. ```bash
  370. npx skills add amos689/paper-preflight
  371. ```
  372. **MCP clients (Codex, Cursor, VS Code, …):** run `paper-preflight mcp` (the `mcp` extra). The
  373. tools are read-only and confined to your workspace; see [docs/mcp.md](docs/mcp.md).
  374. **GitHub Actions:** `uses: amos689/paper-preflight@main` checks the paper on every push, with
  375. the report in the job summary and optional code-scanning alerts; see
  376. [docs/github-action.md](docs/github-action.md).
  377. **pre-commit:** check citation keys and cached verdicts on every commit in seconds; see
  378. [docs/pre-commit.md](docs/pre-commit.md).
  379. **Python:**
  380. ```python
  381. import paper_preflight
  382. report = paper_preflight.check_paper("paper/")
  383. for ref in report.references:
  384. print(ref.key, ref.verdict, [f.rule for f in ref.findings])
  385. ```
  386. The results are plain, frozen dataclasses; see [docs/python-api.md](docs/python-api.md).
  387. ## Experimental: find the passage behind each citation
  388. `support` looks in each cited work for a passage that says what the citing sentence claims. It
  389. reads the work's text (the arXiv source, an open-access full text or PDF, or else the abstract)
  390. and a small local model (HHEM-2.1-open, 0.4 GB) scores the passages ranked best for the claim.
  391. It needs the `support` extra.
  392. ```bash
  393. paper-preflight support path/to/paper --download-model --all
  394. ```
  395. `--download-model` fetches the model's weights once; `--all` also lists the confirmed citations
  396. with their quotes. `arxiv:<id>` works as a target, as it does for `check`.
  397. - **What it says:** "confirmed", with the passage quoted word for word, or "could not
  398. confirm", with the reason (no text, only the abstract, or no passage close enough).
  399. - **It never calls a citation wrong.** On a gold set of 298 citations, 93% of its
  400. confirmations were right [95% CI 82%, 98%], but it confirms only about one real citation in
  401. six, and a low score pointed at a mis-citation less than half the time. The gold set's labels
  402. were made by AI models, not experts; see [`evals/results/support.md`](evals/results/support.md).
  403. - **Or let your agent judge.** The MCP tool `preflight_cited_passages` returns each claim with
  404. the cited work's best passages, for the agent to judge by the same rules, without the model.
  405. On 100 gold-set citations, a Claude agent judging from it confirmed 39% of the real ones,
  406. against HHEM's 9%, every confirmation at least partly supported (the labels come from the
  407. same model family).
  408. - **What leaves your machine:** only the cited works' identifiers, to fetch their text; the
  409. claims are scored locally.
  410. ## How it works
  411. 1. **Source-first.** It reads the LaTeX project as LaTeX sees it: comments, `\iffalse` blocks and
  412. `\includeonly` are respected, `.aux` files are used when they are fresh, and the first
  413. definition of a duplicated key wins, as in BibTeX.
  414. 2. **Identifier-first routing.** DOIs go to their registration agency (doi.org tells which:
  415. Crossref, DataCite, …). arXiv IDs go to arXiv, with DataCite as a fallback, and PMIDs and
  416. PMCIDs to PubMed, which also marks retracted articles. Entries without identifiers are
  417. searched by title in dblp and Crossref; dblp through its SPARQL endpoint, since its search
  418. API is behind a bot challenge. Which source is asked what, and how politely:
  419. [docs/sources.md](docs/sources.md).
  420. 3. **Field-by-field matching with guards.** Titles (including earlier arXiv versions'), authors
  421. (tolerating transcriptions such as Reiß/Reis), year and venue are compared. A search result
  422. is used only when enough of these agree and no other work fits as well; known fake DOI copies
  423. are skipped.
  424. 4. **One verdict per reference**, as in [What it checks](#what-it-checks). "Not found" needs
  425. every required source to answer "no".
  426. 5. **Cached locally.** Answers are kept in a SQLite file, so re-runs are fast and `--offline`
  427. works.
  428. ### Design principles
  429. - **Positive confirmation or abstain.** Rate limits, outages and unindexed works lead to "cannot
  430. determine", never to "not found".
  431. - **No LLM in the verdicts.** Every verdict comes from records and fixed rules, so the same
  432. records give the same report.
  433. - **Neutral wording.** Findings state observations ("not found in Crossref, dblp and Semantic
  434. Scholar, and every source responded"), never accusations.
  435. - **Local-first, no telemetry.** Only the metadata of the cited works (DOIs, titles, authors) is
  436. sent to the public scholarly APIs above. Your manuscript never leaves your machine.
  437. ### What it will never do
  438. Help evade plagiarism or AI-text detection, scrape paywalled or bot-protected sites, recommend
  439. or "complete" references from memory, or name and shame authors.
  440. ## Roadmap
  441. - **Next: references in Chinese script**, measured on held-out papers before anything ships.
  442. - **Fewer references left undecided** among grey literature (reports, theses, manuals) without
  443. an identifier.
  444. What each release brought is in the [changelog](CHANGELOG.md); day-to-day progress in
  445. [docs/PROGRESS.md](docs/PROGRESS.md) (in Chinese).
  446. ## Contributing
  447. Bug reports with a reproducible `.bib` entry are the most valuable contribution, especially
  448. false positives. See [CONTRIBUTING.md](CONTRIBUTING.md); security issues:
  449. [SECURITY.md](SECURITY.md).
  450. ## License
  451. [MIT](LICENSE). See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for adapted code.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.