Skip to content
Back to search

Anti-Autoresearch

Don't trust an autoresearch paper at face value. Reviewer-side integrity forensics (self-consistency + fabrication), deterministic verdict. 61 signals: 46 integrity hack-patterns (families A–H, verdict-bearing) + 13 zero-weight AI writing-style impressions (AIS) + 2 advisory. Not an opaque AI-text classifier. The dual of ARIS.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/wanshuiyin/Anti-Autoresearch with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/wanshuiyin/Anti-Autoresearch'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/wanshuiyin--anti-autoresearch/badge.svg)](/listings/wanshuiyin--anti-autoresearch)

HTML

<a href="/listings/wanshuiyin--anti-autoresearch"><img src="/api/listings/wanshuiyin--anti-autoresearch/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Agent workflow (normal use)

Anti-Autoresearch runs as a Claude Code skill workflow — the Python tools are the deterministic spine *inside* that workflow, not the usual interface.

# 1) Install the skills + workflow (global, or pass a project's .claude/skills dir)
git clone https://github.com/wanshuiyin/Anti-Autoresearch.git
./Anti-Autoresearch/tools/install_anti_autoresearch.sh              # → ~/.claude/skills
# project-local instead: ./Anti-Autoresearch/tools/install_anti_autoresearch.sh ./.claude/skills

# 2) Wire the cross-model reviewer (end state: Claude Code exposes mcp__codex__codex)
claude mcp add codex -- codex mcp-server
claude mcp list

# 3) Audit a paper
claude
> /anti-autoresearch ~/papers/submission

The run writes `REPORT.md` + `report.json` + `claims.json` + per-skill `*.findings.json` into the paper directory. Put the code/result artifacts alongside the paper to unlock L2 checks; PDF/source-only runs are observability-limited by design.

Zero verdict weight — the AIS + advisory tracks (reported, never moves the verdict)

Three skills produce outputs that are **reported but carry zero weight on the integrity verdict** — the non-integrity categories that round out a report: the **AIS** writing-style track and the **advisory** memos. They matter to a human reviewer (a style impression, the worst-case rejection paragraph, prior-art overlap), so the report shows them in their own section — but the deterministic verdict stays driven **only** by the 46 integrity patterns. A paper can be `CLEAN_GIVEN_EVIDENCE` while listing many. `/anti-autoresearch` runs them automatically; to run one standalone, build the ledger first (next section) and invoke it like any auditor.

| Skill | What it writes | |-------|----------------| | `/ai-style-impressions` | *(AIS · separate report section)* AI writing-style impressions: defensive hedging, LLM phrasing tics, clause-then-formula walls, bullet/bold spam, invented codenames, single-style figures | | `/adversarial-case-builder` | *(memo, no verdict)* the single strongest evidence-bound rejection paragraph a hostile reviewer would write | | `/novelty-duplication-advisory` | *(memo, no verdict)* prior-work overlap: trivial-combination ("缝合 / stapling") and duplicate-publication candidates, laid out for a human to weigh |

Single-skill use

Every auditor is also a standalone skill — the installer drops all of them plus the workflow, so you can run just the axis you care about. They share one contract, so run it in order:

claude
# 1) Build the evidence ledger ONCE — the spine every auditor anchors to. Skip it and
#    any auditor stops with:  NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST
> /evidence-ledger ~/papers/submission        # → claims.json + observability level (L0/L1/L2)

# 2) Then run any auditor below against that ledger → <skill>.findings.json

**The verdict-bearing auditors** — each takes the paper dir, reads the ledger, and proposes span-anchored findings the deterministic adjudicator turns into the verdict (the zero-weight AIS + advisory skills are in the section above):

| Skill | What it catches | |-------|-----------------| | `/consistency-audit` | the paper against itself: inflated / mismatched numbers, method & scope drift, appendix-vs-body contradictions | | `/citation-forensics` | citations: hallucinated references, and real papers cited for a claim they don't make | | `/baseline-comparison-audit` | the missing / weak / mistuned baselines hiding behind a "SOTA" or "outperforms" claim | | `/experiment-forensics` | *(L2 — needs code+results)* fake / derived ground truth, score self-normalization, phantom results, placeholder data, code output ≠ reported numbers | | `/proof-derivation-forensics` | *(L1 — needs LaTeX source)* the written proof: skipped obligations, circularity, invalid steps, symbol drift, smuggled assumptions |

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 522 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Anti-Autoresearch 🛡️
  2. [![Join Community](https://img.shields.io/badge/💬_Join-Community-7C3AED?style=flat)](#community) · [![Cite](https://img.shields.io/badge/📖_Cite-BibTeX-2E7D32?style=flat)](#citation) · [![Parent ARIS stars](https://img.shields.io/github/stars/wanshuiyin/Auto-claude-code-research-in-sleep?style=flat&logo=github&logoColor=white&color=gold&label=Parent%20ARIS%20%E2%98%85)](https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep/stargazers) · [![ARIS Report arXiv:2605.03042](https://img.shields.io/badge/ARIS%20Report-arXiv%3A2605.03042-b31b1b?style=flat&logo=arxiv)](https://arxiv.org/abs/2605.03042) · [![ARIS · HF Daily #1](https://img.shields.io/badge/ARIS%20%F0%9F%A4%97%20HF%20Daily-%231-ffcc4d?style=flat)](https://huggingface.co/papers/2605.03042) · [![ARIS on PaperWeekly](https://img.shields.io/badge/ARIS%20on-PaperWeekly-red?style=flat)](https://mp.weixin.qq.com/s/tDniVryVGjDkkkWl-5sTkQ) · [![ARIS in awesome-agent-skills](https://img.shields.io/badge/ARIS%20in-awesome--agent--skills-blue?style=flat&logo=github)](https://github.com/VoltAgent/awesome-agent-skills) · [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg?style=flat)](LICENSE) · [![README 中文](https://img.shields.io/badge/README-%E4%B8%AD%E6%96%87-blue?style=flat)](README_CN.md)
  3. <div align="center">
  4. ### 🔬 The field has tolerated unreliable autoresearch long enough — Anti-Autoresearch is the read that finally catches it.
  5. ***天下苦 autoresearch 久矣 —— Anti-Autoresearch 替研究者们一眼看穿不靠谱的工作。***
  6. </div>
  7. > 🏆 **Built on a battle-tested foundation: [ARIS](https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep)** (~12.5k★ · HuggingFace Daily Papers #1 · 78+ skills across 7+ platforms). Anti-Autoresearch points ARIS's production audit DNA (experiment-audit · paper-claim-audit · citation-audit · kill-argument) **outward** — auditing a third party's submission instead of your own.
  8. Autoresearch has gone mainstream, and a fast-growing share of what reaches the review
  9. pile is machine-generated — and a lot of it **doesn't hold up**: tables that don't
  10. match the text, baselines that aren't there, open-sourced code that won't reproduce
  11. its own paper. Reviewers, area chairs, and honest authors increasingly need to
  12. **verify** that, not just suspect it.
  13. > Regardless of *who or what* wrote a paper, does the science hold together and
  14. > reflect its own evidence? Anti-Autoresearch audits a submission for
  15. > **self-consistency** and **fabrication**, and produces a span-anchored,
  16. > reviewer-ready report. It is **not** an *opaque* AI-text classifier (no authorship
  17. > probabilities, no "AI-written" verdict) and does **not** judge misconduct — it surfaces
  18. > discrepancies a human reviewer should investigate. Separately, it lists transparent,
  19. > itemized **AI writing-style impressions** in a quarantined, **zero-verdict-weight**
  20. > section (a paper can be integrity-`CLEAN` while listing many), because reviewers react to them.
  21. ---
  22. ## 🧭 What's inside
  23. **46 integrity patterns across 8 families** — the coverage vocabulary every finding cites — plus **13 zero-weight AI writing-style impressions** and **2 advisories**:
  24. | | Family | Catches |
  25. |---|--------|---------|
  26. | **A** | Numeric self-consistency | 数值自洽:table vs text vs delta arithmetic that doesn't add up |
  27. | **B** | Method & scope | 方法与范围:the described method/scope ≠ what was actually done |
  28. | **C** | Baseline integrity | baseline 诚信:missing, weak, or unfairly configured comparisons |
  29. | **D** | Experiment integrity | 实验诚信:fake ground truth, phantom results, code ≠ numbers (needs code) |
  30. | **E** | Citation integrity | 引用诚信:fabricated, misattributed, or retracted references |
  31. | **F** | Presentation & surface | 表面信号:layout / prose / figure signals — capped at `minor` |
  32. | **G** | Proof & derivation | 证明诚信:skipped obligations, circular or invalid derivations |
  33. | **H** | Evaluation design & validity | 评测设计:data leakage, LLM-judge validity, selective reporting |
  34. Delivered as **11 skills + 1 orchestrating workflow** on a deterministic spine: a span-anchored, hashed evidence **ledger** → LLM auditors that only **propose** findings → a rules-only **reporter** that lists every proposal with what the auditor said about it, and summarizes — with 8 patterns eval-gated end-to-end (GRIM · GRIMMER · statcheck · delta arithmetic · hedge-density · …) and the whole gate in CI.
  35. ---
  36. ## 📰 News
  37. - **2026-07-10** — 🧮 **Math was invited to overturn false alarms — and honestly declined.** We built exact calculators to re-check the arithmetic behind numeric criticals, hoping to auto-clear false alarms like rounding differences. Adversarial review broke every version — the killer was "dropout of exactly 50%" vs a table's 50.4%: a calculator can't know whether a number is exact or rounded. The calculators stayed, demoted: they attach their full working to the report for the human, and **nothing gets auto-cleared**.
  38. - **2026-07-10** — 🛡️ **Criticals must now survive their own defense.** A critical has to state which innocent explanations it ruled out (blank → drops to major), face one fresh refutation thread whose only power is a ⚠️ CONTESTED marker for the human, and — if numeric — declare exactly which numbers it computes over. Reports stamp `adjudicator: deterministic-rules-v2`. ⚠️ Re-run the auditors on old findings files instead of re-adjudicating them.
  39. - **2026-06-28** — ✍️ **v0.5 — the AIS track.** 13 itemized AI writing-style impressions now live in a separate, **zero-verdict-weight** report section — a paper can be integrity-`CLEAN` while listing many. Transparent impressions, never an authorship verdict; the 5 pure-style patterns moved out of family F.
  40. <details>
  41. <summary><b>Earlier updates</b> (2026-06)</summary>
  42. - **2026-06-27** — 🧬 **v0.4 — the taxonomy grows to 8 families** (new: proof & derivation G, evaluation design H — the full map is in *What's inside* above). The deterministic eval gate grew 3→8 patterns (GRIM / GRIMMER / statcheck + a conservative hedge-density screen), CI landed, and prior-art acknowledgments were added.
  43. - **2026-06-26** — 🚀 **v0.1 — initial release.** Reviewer-side integrity forensics for autoresearch / AI-Scientist papers: evidence ledger, deterministic adjudicator, observability tiers. Not an AI-text detector.
  44. </details>
  45. ## 🚀 Quickstart
  46. ### Agent workflow (normal use)
  47. Anti-Autoresearch runs as a Claude Code skill workflow — the Python tools are the
  48. deterministic spine *inside* that workflow, not the usual interface.
  49. ```bash
  50. # 1) Install the skills + workflow (global, or pass a project's .claude/skills dir)
  51. git clone https://github.com/wanshuiyin/Anti-Autoresearch.git
  52. ./Anti-Autoresearch/tools/install_anti_autoresearch.sh # → ~/.claude/skills
  53. # project-local instead: ./Anti-Autoresearch/tools/install_anti_autoresearch.sh ./.claude/skills
  54. # 2) Wire the cross-model reviewer (end state: Claude Code exposes mcp__codex__codex)
  55. claude mcp add codex -- codex mcp-server
  56. claude mcp list
  57. # 3) Audit a paper
  58. claude
  59. > /anti-autoresearch ~/papers/submission
  60. ```
  61. The run writes `REPORT.md` + `report.json` + `claims.json` + per-skill
  62. `*.findings.json` into the paper directory. Put the code/result artifacts alongside
  63. the paper to unlock L2 checks; PDF/source-only runs are observability-limited by
  64. design.
  65. ### Zero verdict weight — the AIS + advisory tracks (reported, never moves the verdict)
  66. Three skills produce outputs that are **reported but carry zero weight on the integrity
  67. verdict** — the non-integrity categories that round out a report: the **AIS** writing-style
  68. track and the **advisory** memos. They matter to a human reviewer (a style impression, the
  69. worst-case rejection paragraph, prior-art overlap), so the report shows them in their own
  70. section — but the deterministic verdict stays driven **only** by the 46 integrity patterns. A
  71. paper can be `CLEAN_GIVEN_EVIDENCE` while listing many. `/anti-autoresearch` runs them
  72. automatically; to run one standalone, build the ledger first (next section) and invoke it like
  73. any auditor.
  74. | Skill | What it writes |
  75. |-------|----------------|
  76. | `/ai-style-impressions` | *(AIS · separate report section)* AI writing-style impressions: defensive hedging, LLM phrasing tics, clause-then-formula walls, bullet/bold spam, invented codenames, single-style figures |
  77. | `/adversarial-case-builder` | *(memo, no verdict)* the single strongest evidence-bound rejection paragraph a hostile reviewer would write |
  78. | `/novelty-duplication-advisory` | *(memo, no verdict)* prior-work overlap: trivial-combination ("缝合 / stapling") and duplicate-publication candidates, laid out for a human to weigh |
  79. ### Single-skill use
  80. Every auditor is also a standalone skill — the installer drops all of them plus the
  81. workflow, so you can run just the axis you care about. They share one contract, so
  82. run it in order:
  83. ```text
  84. claude
  85. # 1) Build the evidence ledger ONCE — the spine every auditor anchors to. Skip it and
  86. # any auditor stops with: NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST
  87. > /evidence-ledger ~/papers/submission # → claims.json + observability level (L0/L1/L2)
  88. # 2) Then run any auditor below against that ledger → <skill>.findings.json
  89. ```
  90. **The verdict-bearing auditors** — each takes the paper dir, reads the ledger, and proposes
  91. span-anchored findings the deterministic adjudicator turns into the verdict (the zero-weight
  92. AIS + advisory skills are in the section above):
  93. | Skill | What it catches |
  94. |-------|-----------------|
  95. | `/consistency-audit` | the paper against itself: inflated / mismatched numbers, method & scope drift, appendix-vs-body contradictions |
  96. | `/citation-forensics` | citations: hallucinated references, and real papers cited for a claim they don't make |
  97. | `/baseline-comparison-audit` | the missing / weak / mistuned baselines hiding behind a "SOTA" or "outperforms" claim |
  98. | `/experiment-forensics` | *(L2 — needs code+results)* fake / derived ground truth, score self-normalization, phantom results, placeholder data, code output ≠ reported numbers |
  99. | `/proof-derivation-forensics` | *(L1 — needs LaTeX source)* the written proof: skipped obligations, circularity, invalid steps, symbol drift, smuggled assumptions |
  100. | `/eval-design-forensics` | the evaluation's validity: train/test leakage, a conflicted or unvalidated LLM-judge metric, selective reporting (dropped conditions / switched metrics) |
  101. | `/presentation-signals` | *(capped at `minor` → at most SOFT)* checkable surface tells: duplicate tables, leftover pipeline/template strings, LLM-generated figures, page-padding — context, never a verdict |
  102. A single skill only **proposes** span-anchored findings — it never returns a verdict.
  103. To get one, feed the findings to the deterministic adjudicator (the
  104. `python3 tools/adjudicate_findings.py … --ledger …` command in the next section); the
  105. model never grades. Two more notes: `consistency-audit`, `presentation-signals`, and
  106. `ai-style-impressions` also write a `*.deterministic.findings.json` (works with no
  107. cross-model reviewer wired); and **`/anti-autoresearch`** runs every auditor above in one
  108. shot, adding ingest
  109. (arxiv-id / pdf → workdir + `pdftotext`), automatic observability, auto-selection of
  110. which auditors apply, and the final cross-dimension verdict + `REPORT.md`.
  111. ### Deterministic core (CI / offline / zero-dependency)
  112. This bypasses the agent layer and exercises only the eval-tested deterministic
  113. checks — use it for CI, regression tests, or environments with no cross-model
  114. reviewer (Python 3 stdlib, nothing to install):
  115. ```bash
  116. # Prove the pipeline on clean + corrupted fixtures (the regression gate)
  117. python3 eval/run_eval.py
  118. # clean / delta_inflate / dup_table / headline_inflate → all PASS
  119. # injected-defect recall: 100% (7 deterministic patterns) · clean FP: none
  120. python3 tests/test_adjudicator.py # gate unit tests (the anti-slop guarantee)
  121. # Or run the spine by hand on a real paper:
  122. python3 tools/build_claim_ledger.py --paper-id mypaper --latex main.tex sections/*.tex \
  123. --observability-level 1 --out claims.json
  124. python3 tools/check_numeric_consistency.py --ledger claims.json --out findings.json
  125. python3 tools/adjudicate_findings.py --findings findings.json --ledger claims.json \
  126. --paper-id mypaper --observability-level 1 --out report.json --md REPORT.md
  127. # --ledger is REQUIRED: a finding must quote a verbatim ledger span or it fails closed to info.
  128. ```
  129. ## 🎯 Why this exists
  130. Machine-generated papers and reviews are now a measurable share of the literature,
  131. and the failure that matters for an area chair is rarely *"was this text written by
  132. an LLM?"* (a human can write a dishonest paper; an LLM can write an honest one). It
  133. is: **does the paper contradict itself, and is it backed by its own evidence?**
  134. That is what autoresearch pipelines get wrong — they hallucinate *local* coherence:
  135. an abstract number that no table reports, a "16% improvement" that the operands say
  136. is 6%, a citation for a claim the cited paper never makes, a method described one
  137. way and evaluated another.
  138. Those are checkable under a declared observability level. Concretely, taxonomy v0.5
  139. names **46 integrity patterns across 8 families** (numeric self-consistency · method /
  140. scope · baseline integrity · experiment integrity · citation integrity ·
  141. presentation / surface signals · proof & derivation integrity · evaluation design &
  142. validity) — the repo's **coverage vocabulary**, not a detector benchmark — plus a
  143. **13-signal AI writing-style impression track (AIS)** that carries **zero verdict weight**.
  144. > **Shipped v0:** the deterministic spine and the **seven** ✓-marked patterns (across
  145. > the representative list below and the full catalog) are eval-tested; the other 39 integrity
  146. > patterns are agent-layer contracts (a cross-model reviewer proposes span-anchored findings, the
  147. > deterministic adjudicator scores or demotes them) — not bundled-eval detector claims.
  148. The full catalog, with detection signals and false-positive cases, lives in
  149. [the taxonomy](references/hack-pattern-taxonomy.md). A representative ten (✓ = gated
  150. by the deterministic eval today):
  151. - `HP-NUM-INFLATE` — abstract says 85.3%, but Table 2 never gets past 84.7%. ✓
  152. - `HP-DELTA-ERROR` — a "16% improvement" from 73.1 to 78.0 is really 6.7%. ✓
  153. - `HP-DUP-TABLE` — two tables carry the identical ordered numbers — usually copy-paste padding. ✓
  154. - `HP-METHOD-DRIFT` — the method section says "no labels"; the eval quietly uses gold-label calibration.
  155. - `HP-SCOPE-INFLATE` — "comprehensive" turns out to be two datasets, one domain, maybe one seed.
  156. - `HP-MISSING-BASELINE` — SOTA is claimed while the obvious recent baseline never appears in the table.
  157. - `HP-FAKE-GT` — (L2) the "reference" targets are model outputs, then reported as ground truth.
  158. - `HP-PHANTOM-RESULT` — (L2) a headline number points at a result file or metric key that isn't there.
  159. - `HP-PROOF-CIRCULARITY` — (L1) the "proof" restates the claim in different words and calls it done — it proves nothing.
  160. - `HP-CITE-HALLUC` — the DOI / arXiv id / venue / author list simply doesn't exist.
  161. <details>
  162. <summary><b>… the other 36 integrity patterns + the 13 AIS impressions, in full</b></summary>
  163. **A · Numeric self-consistency**
  164. - `HP-AGG-DRIFT` — they write "mean over seeds", but the number is really the best seed.
  165. - `HP-DENOM-DRIFT` — one table averages all tasks; the conclusion quietly uses the applicable-only subset.
  166. - `HP-UNIT-DIR-MISMATCH` — points silently become percent, or a lower-better metric is celebrated upward.
  167. - `HP-CAPTION-MISMATCH` — the caption promises N=5 and method B; the plot shows neither.
  168. - `HP-APPENDIX-CONTRA` — the appendix reruns the same quantity and disagrees with the main text.
  169. - `HP-GRANULARITY-IMPOSSIBLE` — "84.7% on 500 items" is arithmetically impossible — no integer k/500 rounds to it (GRIM). ✓
  170. - `HP-VARIANCE-IMPOSSIBLE` — a reported SD bigger than a bounded metric can have at that mean (e.g. SD 18% at mean 98% — cap ≈15.7%). ✓
  171. - `HP-STAT-INCONSISTENCY` — the reported p contradicts its own test statistic and overstates significance ("z=1.10, p=.036" → really p≈.27). ✓
  172. **B · Method & scope**
  173. - `HP-ABLATION-ATTRIB` — they credit component X, but every ablation keeps X bundled with Y.
  174. - `HP-THEOREM-SCOPE-DRIFT` — the abstract sells a general theorem; the assumptions do nearly all the work.
  175. - `HP-ARGUMENT-CHAIN-BREAK` — a substantive missing link: the problem motivated isn't the one the method addresses, or the experiments measure something the mechanism doesn't predict.
  176. - `HP-CAUSAL-EVIDENCE-LEAP` — a causal / equivalence relation is concluded that no experiment in the paper actually varies or tests.
  177. - `HP-RESOURCE-IDENTITY-MISMATCH` — a named dataset/model/benchmark described with a property its public record contradicts ("ImageNet-1k, 5,000 classes" — it's 1,000).
  178. - `HP-ACRONYM-DRIFT` — the same load-bearing component/term gets two incompatible names or acronym expansions across the paper.
  179. **C · Baseline integrity**
  180. - `HP-WEAK-BASELINE` — the new method gets tuning and compute the baseline plainly did not.
  181. - `HP-SIG-OVERLAP` — "outperforms" by crumbs, with overlapping error bars or no seeds shown.
  182. **D · Experiment integrity** (needs code/results — L2)
  183. - `HP-SELF-NORM` — (L2) the score nears 1.0 because it's divided by the model's own max.
  184. - `HP-DEAD-METRIC` — (L2) a metric function exists with no call site and no result, yet is discussed.
  185. - `HP-SUSPICIOUS-REGULARITY` — (L2) rows differ by a suspiciously clean offset — check the files before calling it fake.
  186. - `HP-PLACEHOLDER-DATA` — (L2) released code still ships placeholder/dummy/fake data feeding a reported figure or number.
  187. - `HP-RESULT-ARTIFACT-MISMATCH` — (L2) the released code / artifacts, run as written, produce numbers different from the paper's.
  188. - `HP-MISSING-REPRO-ARTIFACT` — (L2) an empirical paper ships neither code nor the prompts/configs its results depend on.
  189. **E · Citation integrity**
  190. - `HP-CITE-CONTEXT` — real paper, wrong job: cited for a claim it explicitly doesn't make (incl. semantic-hallucination + a support/contrast/mention intent label).
  191. - `HP-CITE-RETRACTED` — a load-bearing citation that resolves to a **retracted** paper, with no note of the retraction (Crossref / Retraction Watch).
  192. **F · Presentation & surface signals** (capped at `minor` — never a verdict)
  193. - `HP-THIN-FLOAT` — a "broad empirical study" somehow has two tables and one lonely figure.
  194. - `HP-LLM-FIGURE` — the "figure" is decorative model art, not a plot or a real diagram.
  195. - `HP-PAGE-PADDING` — oversized floats, repeated text, or empty prose doing page-count labor.
  196. - `HP-PIPELINE-ARTIFACT` — a leftover pipeline/template string ("As an AI language model", "regenerate response", "[INSERT X]") survives into the finished text. ✓ (exact-match, low-FP)
  197. **G · Proof & derivation integrity** (verdict-bearing at L1 — from the written math)
  198. - `HP-PROOF-OBLIGATION-GAP` — (L1) a required lemma / case / transition is skipped with "clearly" across a real gap.
  199. - `HP-DERIVATION-INVALID` — (L1) an algebra / probability / calculus step does not follow (a misapplied inequality, a wrong limit).
  200. - `HP-SYMBOL-SEMANTIC-DRIFT` — (L1) a symbol / operator / inequality direction changes meaning between definition, formula, and proof.
  201. - `HP-ASSUMPTION-SMUGGLE` — (L1) the proof relies on an assumption (independence, convexity, …) the theorem statement never lists.
  202. - `HP-UNDEFINED-NOTATION` — (L1) a load-bearing symbol is used in a key equation/proof but never defined and not inferable from standard convention.
  203. **H · Evaluation design & validity** (L0/L1 stated → L2 confirmed)
  204. - `HP-EVAL-LEAKAGE` — train/test leakage (preprocess-before-split, duplicates across splits, temporal leak, pretraining contamination) means the score may not measure generalization. Adopts the Kapoor–Narayanan leakage taxonomy.
  205. - `HP-JUDGE-VALIDITY` — the load-bearing metric is an LLM judge that's *conflicted* (same family as a compared system) or *unvalidated* (no human-agreement check).
  206. - `HP-SELECTIVE-REPORTING` — a condition the setup declared (a dataset / baseline / metric / seed-count) is dropped from the results, or the metric is switched to favor the method.
  207. **AIS · AI writing-style impressions** (zero verdict weight — reported in a separate section, NEVER move the verdict; impressions, not integrity findings)
  208. - `AIS-NARRATIVE-ARC-BREAK` — abrupt 1–2¶ intro / dump-like abstract; no background → contribution → evidence arc.
  209. - `AIS-LLM-PHRASE-TICS` — LLM phrasing tics ("it is worth noting", "not only … but also", clichéd em-dash/semicolon, flowery adverbs).
  210. - `AIS-DEFENSIVE-HEDGE` — pervasive "we do not claim … / not X but rather Y" instead of stating what was done (deterministic density screen).
  211. - `AIS-JARGON-STUFF` — dense term-stuffing with no surrounding content.
  212. - `AIS-INVENTED-CODENAME` — an undefined, internal-flavored run/experiment codename used as if defined.
  213. - `AIS-CLAUSE-FORMULA-WALL` — a short clause then a wall of formulas, repeated, no connective prose.
  214. - `AIS-GRATUITOUS-PSEUDOCODE` — pseudocode that just restates the prose / adds no operational content.
  215. - `AIS-BULLET-LIST-OVERUSE` — sequential logic flattened into parallel-looking bullets.
  216. - `AIS-BOLD-MODULE-SPAM` — verbose module names with excessive bolding.
  217. - `AIS-RESTATE-OVERCLAIM` — a rhetorical restatement loop ("we propose an X …" repeated).
  218. - `AIS-FOCUS-DRIFT` — high-level motivation pivots to a minor implementation detail.
  219. - `AIS-SINGLE-STYLE-FIGURES` — figures share a generic generated visual grammar.
  220. - `AIS-APPENDIX-DUMPING-GROUND` — the appendix reads like an unintegrated AI-trace dump.
  221. </details>
  222. **This is not hypothetical.** Paraphrased from a public reviewer account during the
  223. NeurIPS 2026 cycle (illustrative, not a citation), one batch maps almost one-to-one
  224. onto the taxonomy this repo encodes:
  225. > - *Paper 1* — "data tables don't match the text; several rows are misaligned;
  226. > there's an obvious add/subtract regularity across backbones — it doesn't look
  227. > like it was actually run." → consistency · `HP-SUSPICIOUS-REGULARITY`
  228. > - *Paper 2* — "two tables fill a page and are identical; the one figure is
  229. > LLM-generated; and it *still* didn't fill 9 pages." → `HP-DUP-TABLE` ·
  230. > presentation signals
  231. > - *Paper 3* — "formula derivations don't hold; the experiments look complete but
  232. > the math can't give those results." → proof-derivation-forensics · `HP-DERIVATION-INVALID`
  233. > - *Paper 4* — "open-sourced, beautifully written and drawn — but I ran the code
  234. > and it gives completely different results from the paper." → experiment-forensics (L2)
  235. The fourth case is this repo's thesis in one line: **surface polish is not integrity.**
  236. ## 🔒 How it stays honest (the anti-"LLM-slop" design)
  237. The obvious dismissal of any such tool is *"an LLM grading another LLM's paper is
  238. just noise."* Three structural defenses, not just a disclaimer:
  239. 1. **Evidence ledger.** One deterministic pass turns the paper into `claims.json` —
  240. span-anchored, hashed claims. Every finding must cite a `claim_id` + verbatim
  241. span. **No span → it cannot be a high-severity finding.**
  242. 2. **The LLM never grades.** Auditors *propose* findings; a **deterministic
  243. adjudicator** (`tools/adjudicate_findings.py`, pure rules) computes the verdict.
  244. Same findings → same summary, by a fixed rule. The severities in it are what the
  245. auditors proposed, annotated with what the computation could and could not confirm.
  246. 3. **Observability levels.** A run declares what it could see (L0 PDF-only → L2
  247. repo+results). A finding that needs code is **reported with the level it needs and
  248. a mark that this run did not have it**, so nothing shouts "fraud" from a PDF and
  249. nothing disappears either. See
  250. [references/observability-levels.md](references/observability-levels.md).
  251. **Surface signals and AI writing-style impressions have firewalls.** Family-F surface tells
  252. (duplicate tables, LLM-generated figures, page-padding, leftover pipeline strings) are
  253. reported only as *high-false-positive context*: the adjudicator hard-caps them at `minor`
  254. (`SURFACE_ONLY_SKILLS` / `SURFACE_PATTERNS`), so they reach at most `SOFT_FLAGS`. The pure
  255. **AI writing-style impressions** (the AIS track — defensive hedging, LLM phrasing tics, …) go
  256. further: they carry **zero verdict weight** — forced to `info`, excluded from
  257. `overall_verdict` / counts / dimensions, and rendered in a separate "NOT integrity" section.
  258. A paper can be `CLEAN_GIVEN_EVIDENCE` while listing many. Both caps are enforced in code
  259. (`_is_zero_weight` + the weight-1-only verdict in `tools/adjudicate_findings.py`), not just
  260. promised — and regression-tested.
  261. And an **eval harness** (`eval/`) proves the deterministic core on clean +
  262. synthetically-corrupted fixtures every change — measured false-positive / recall,
  263. not vibes.
  264. ## 🏗️ Architecture
  265. ```
  266. input (pdf | pdf+latex | pdf+repo+results)
  267. │
  268. ▼ [evidence-ledger] artifact_manifest.json (+ observability level) + claims.json ← deterministic
  269. │
  270. ▼ fan out auditors (each reads the ledger, emits span-anchored findings):
  271. consistency-audit flagship · paper vs itself · ARIS paper-claim-audit
  272. citation-forensics exists? correct? right context? · ARIS citation-audit
  273. baseline-comparison-audit missing/weak/mistuned baselines · ARIS paper-claim-audit
  274. experiment-forensics L2: fake GT / self-norm / phantom · ARIS experiment-audit
  275. proof-derivation-forensics L1: proof gap / circularity / invalid step · verdict-bearing · ARIS proof-checker
  276. eval-design-forensics L0/L1: data leakage / conflicted-or-unvalidated LLM judge / selective reporting
  277. presentation-signals checkable surface tells · auxiliary, capped at minor
  278. ai-style-impressions AI writing-style impressions · ZERO verdict weight · separate section
  279. adversarial-case-builder evidence-bound memo, no verdict · ARIS kill-argument
  280. novelty-duplication-advisory prior-work overlap memo, no verdict · ARIS novelty-check
  281. │
  282. ▼ [adjudicate_findings.py] rules, not a model → REPORT.md + report.json ← deterministic
  283. ```
  284. | Path | What |
  285. |------|------|
  286. | `skills/` | the eleven auditor / impression skills (LLM proposes findings, span-anchored) |
  287. | `workflows/anti-autoresearch/` | the end-to-end orchestrator |
  288. | `tools/` | deterministic spine: manifest/observability derivation · ledger builder · numeric checks · adjudicator |
  289. | `schemas/` | JSON contracts: claims · finding · report · artifact manifest |
  290. | `references/` | hack-pattern taxonomy (the core contribution) · observability levels · reviewer independence · forensics contract |
  291. | `eval/` | clean + synthetic-corruption fixtures + the regression harness |
  292. | `tests/` | gate unit tests for the adjudicator (the anti-slop invariants) |
  293. | `docs/` | positioning vs existing work · limitations |
  294. ## ⚠️ Honest limitations
  295. - **Forensics ≠ proof of misconduct.** Output is *flags for a human*, never an accusation.
  296. - **PDF-only (L0) catches inconsistency + tells, not all fabrication** — it cannot
  297. verify external ground truth or run code.
  298. - **False positives exist** (legitimate round numbers, single-seed pilots,
  299. deliberate scope choices). Hence tiers, FP-risk tags, and the eval harness — not
  300. a binary "guilty".
  301. - **The taxonomy is a living document.** Adversaries who know a signal can route
  302. around it; this is a safety net, not a guarantee. See [docs/limitations.md](docs/limitations.md).
  303. ## 🧬 Provenance: derived from ARIS
  304. [**ARIS — Auto Research in Sleep**](https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep)
  305. is an AI research-agent skill platform that runs end-to-end research pipelines
  306. (literature → idea → experiment → paper) — and does so **with integrity guardrails
  307. built in**, which is what makes it a credible base for the auditor:
  308. - 🛡️ **A three-layer audit stack** keeps ARIS's *own* output honest:
  309. `experiment-audit` (fake GT / score-normalization / phantom results),
  310. `result-to-claim` (is the claim scientifically supported?), and zero-context
  311. `paper-claim-audit` + `citation-audit` (do the reported numbers and references
  312. hold up?). Anti-Autoresearch is these same audits **pointed outward**.
  313. - 🔬 **Cross-model adversarial review** is the core doctrine: the executor and the
  314. reviewer must be different model families, so no LLM ever judges its own output.
  315. Anti-Autoresearch inherits this *and* hardens it — here the model only **proposes**
  316. findings; a deterministic adjudicator decides.
  317. **Two sides of one coin.** ARIS is how to do autoresearch *responsibly*;
  318. Anti-Autoresearch is how to flag autoresearch that wasn't. A generator that publishes
  319. its own audit stack knows precisely how these pipelines fail — because it engineered
  320. against those failures from the inside. That is the perspective this repo brings.
  321. 👉 **ARIS main repo**: https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep
  322. **How the skills map** — Anti-Autoresearch's skills are ARIS's audit skills, copied
  323. and reframed for a **third party auditing an unknown submission** rather than an
  324. author checking their own work: `consistency-audit` ← `paper-claim-audit`,
  325. `experiment-forensics` ← `experiment-audit`, `citation-forensics` ← `citation-audit`,
  326. `baseline-comparison-audit` ← `paper-claim-audit`, `proof-derivation-forensics` ←
  327. `proof-checker`, `adversarial-case-builder` ← `kill-argument`,
  328. `novelty-duplication-advisory` ← `novelty-check`, plus the new `evidence-ledger`
  329. spine, `presentation-signals`, `eval-design-forensics`, and the zero-weight
  330. `ai-style-impressions` (the AIS track).
  331. ## 🤝 Prior art & acknowledgments
  332. Anti-Autoresearch's design borrows ideas — and in places, taxonomy structure — from a
  333. body of integrity, reproducibility, and evaluation-hygiene work that predates it. We
  334. credit it explicitly. **Taxonomies and ideas are adapted with credit; no external code
  335. is vendored** — where a tool is GPL/AGPL we reimplemented the *method* from its paper
  336. rather than copying its source, and where a tool is proprietary we credit the concept
  337. only.
  338. **Deterministic self-consistency (the closest methodological cousins).**
  339. - **statcheck** — Nuijten & Epskamp. Recomputes reported NHST *p*-values from their
  340. test statistics; the canonical "paper against itself" check. *(GPL-3 — method
  341. reimplemented from the paper, source not vendored.)* Informs `HP-STAT-INCONSISTENCY`.
  342. - **GRIM / GRIMMER** — Brown & Heathers (GRIM); Anaya (GRIMMER). Tests whether reported
  343. means/SDs are arithmetically possible for the stated sample size. Informs
  344. `HP-GRANULARITY-IMPOSSIBLE` / `HP-VARIANCE-IMPOSSIBLE`.
  345. - **scrutiny** — Jung. An R toolkit packaging GRIM/GRIMMER-style consistency tests. *(MIT.)*
  346. These inform taxonomy family A (numeric self-consistency); `tools/check_stat_consistency.py`
  347. is an independent, pure-stdlib reimplementation.
  348. **Evaluation integrity & LLM-judge validity (why the model never grades).**
  349. - **Leakage taxonomy** — Kapoor & Narayanan, *Leakage and the Reproducibility Crisis in
  350. ML-based Science*. The priority reference behind the planned family-H eval-integrity patterns.
  351. - **LLM-as-judge validity** — Zheng et al. (judging LLM-as-a-judge), Panickssery et al.
  352. (evaluators favor their own generations), Wang et al. (position / identity bias). Precisely
  353. why our adjudicator is **deterministic** and the model only proposes findings.
  354. - **"Show Your Work"** — Dodge et al. Reporting-hygiene discipline behind the *planned* selective-reporting checks.
  355. **Citation & claim integrity.**
  356. - **Retraction Watch** — the retraction-record project; conceptual basis for citation-status awareness.
  357. - **Problematic Paper Screener** — Cabanac, Labbé, Magazinov. Corpus-scale screening for tell-tale
  358. strings; the basis for `HP-PIPELINE-ARTIFACT`.
  359. - **scite** — supporting / contrasting citation context. *(Proprietary — conceptual credit; informs HP-CITE-CONTEXT.)*
  360. - **SciFact** — Wadden et al. Scientific-claim verification dataset/model behind the claim–evidence framing.
  361. - **Fabricated-citation taxonomy** — Ansari. Informs `HP-CITE-HALLUC` / `HP-CITE-CONTEXT`.
  362. **Reproducibility detection** (prior art for *planned* L2 repro patterns — not yet implemented).
  363. - **ODDPub** — Riedel et al. Detects open-data / open-code statements. *(AGPL-3 — conceptual/method prior art; no code vendored; any implementation will be independent.)*
  364. - **RTransparent** — Serghiou et al. Large-scale data/code-sharing transparency detection. *(GPL-3 — same.)*
  365. - **SciScore** — automated methods-rigor / reproducibility checker. *(Proprietary — conceptual credit only.)*
  366. **Author-side self-audit (evaluated, scope-bounded).**
  367. - **academic-integrity-skill** — 1anj. An author-side, wet-lab/biomedical self-audit skill (image-forensics-heavy). We evaluated its non-image deterministic screens — raw-data terminal-digit / exact-duplicate forensics (last-digit / Benford tradition) and reported-vs-source reconciliation — and adopt neither as a check: digit forensics need raw per-sample tables our reviewer-side tiers rarely see, and reported-vs-source overlaps family D. Credited as prior art and the author-side counterpart to this reviewer-side toolkit. *(MIT.)*
  368. - **anti-defensive-writing** — Kiterlin. An author-side Codex skill that *revises* defensive writing (removes unnecessary caveats/hedges, strengthens prose). The clean **dual** of our `AIS-DEFENSIVE-HEDGE`: they fix it for the author, we flag it for the reviewer (zero-weight). We cross-referenced its discouraged-construction list to extend our deterministic hedge templates ("this is not to say", "this should not be taken to mean", "rather than arguing X, we argue Y"). *(MIT.)*
  369. Anti-Autoresearch's own contribution is not any single one of these checks but their
  370. **combination** into an autoresearch-specific taxonomy under a deterministic adjudicator
  371. and explicit observability tiers (see [docs/positioning.md](docs/positioning.md)).
  372. ## 🔭 Related projects
  373. Where Anti-Autoresearch sits relative to neighboring tools (stars / last update as
  374. gathered 2026-06-27; not a ranking).
  375. | Project | ★ | Updated | Relation to Anti-Autoresearch |
  376. |---------|---|---------|-------------------------------|
  377. | [SakanaAI/AI-Scientist](https://github.com/SakanaAI/AI-Scientist) | 14.1k | 2025-12 | A generator whose output we audit — the class of pipeline this repo is built to check. |
  378. | [karpathy/autoresearch](https://github.com/karpathy/autoresearch) | 88.8k | 2026-03 | A generator whose output we audit; the namesake of the failure surface. |
  379. | [scienceverse/metacheck](https://github.com/scienceverse/metacheck) | 45 | 2026-06 | Closest cousin: modular deterministic paper checks. We add an autoresearch taxonomy + observability tiers + cross-model proposers. |
  380. | [MicheleNuijten/statcheck](https://github.com/MicheleNuijten/statcheck) | 189 | 2026-03 | Deterministic self-consistency (NHST *p*-values) — narrow, but exactly our spirit; informs family A. |
  381. | [lhdjung/scrutiny](https://github.com/lhdjung/scrutiny) | 8 | 2026-05 | GRIM/GRIMMER consistency tests (R); same deterministic-self-consistency family. |
  382. | [allenai/scifact](https://github.com/allenai/scifact) | 265 | 2023-10 | Claim verification against evidence — the claim–evidence framing, applied to external literature rather than the paper's own. |
  383. | [DEFENSE-SEU/FactReview](https://github.com/DEFENSE-SEU/FactReview) | 70 | 2026-06 | Closest *framing* neighbor: an LLM reviewer that audits empirical claims and makes **no** accept/reject call. Differs by grounding against external literature + **executing the repo** (an L3 move we refuse) and model-produced claim statuses — vs our deterministic self-consistency + observability taxonomy. *(AGPL-3.0)* |
  384. | [1anj/academic-integrity-skill](https://github.com/1anj/academic-integrity-skill) | 51 | 2026-05 | Closest *sibling skill*, mirror stance: an **author-side** wet-lab/biomedical self-audit skill (image-forensics-heavy — blot/microscopy/flow, which we exclude). Authors self-check pre-submission vs us auditing third-party autoresearch output; its numeric/citation screens run on the author's raw data tables, not reviewer-side PDF/LaTeX. *(MIT)* |
  385. | [Kiterlin/anti-defensive-writing](https://github.com/Kiterlin/anti-defensive-writing) | 10 | 2026-06 | Author-side **dual** of our `AIS-DEFENSIVE-HEDGE`: a Codex skill that *revises* defensive writing (caveats/hedges → direct, claim-forward prose). They fix it pre-submission; we flag it reviewer-side at zero verdict weight. We cross-referenced its discouraged-construction list to extend our hedge templates. *(MIT)* |
  386. | [ahans30/Binoculars](https://github.com/ahans30/Binoculars) | 390 | 2024-05 | AI-text detector — **what we are NOT**: it answers "was this LLM-written?", a question orthogonal to integrity. |
  387. | [baoguangsheng/fast-detect-gpt](https://github.com/baoguangsheng/fast-detect-gpt) | 414 | 2026-02 | AI-text detector — same boundary; stylometry ≠ integrity. |
  388. A few framing-relevant efforts have **no open repository** and are credited by name
  389. only: Pangram, GPTZero, and the Problematic Paper Screener.
  390. <a id="community"></a>
  391. ## 💬 Community
  392. **The taxonomy grows with the community.** Caught an autoresearch / AI-Scientist paper
  393. pulling a trick that isn't in the [pattern catalog](references/hack-pattern-taxonomy.md)
  394. yet? That is the single most valuable contribution here — open an issue with the concrete
  395. example, or send a PR adding the pattern (with an eval fixture + a false-positive case so
  396. it doesn't over-fire). New auditor skills, adjudicator gates, and corruption fixtures are
  397. just as welcome. **[CONTRIBUTING.md](CONTRIBUTING.md)** explains how a pattern is
  398. structured and the honesty rules every flag must follow (describe a checkable discrepancy,
  399. never impute misconduct or authorship).
  400. Join the WeChat group (shared with the [ARIS](https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep)
  401. community) to swap autoresearch failure modes:
  402. <p align="center">
  403. <img src="docs/wechat_group.jpg" alt="WeChat group QR code (shared with the ARIS community)" width="300">
  404. </p>
  405. *(The group QR rotates weekly — if it's expired, open an issue and we'll post a fresh one.)*
  406. <a id="citation"></a>
  407. ## 📖 Citation
  408. Anti-Autoresearch is **derived from ARIS** and reuses its audit DNA. If this
  409. repository helped your research / paper / review, please cite the ARIS methodology
  410. paper:
  411. ```bibtex
  412. @article{yang2026aris,
  413. title={ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration},
  414. author={Yang, Ruofeng and Li, Yongcan and Li, Shuai},
  415. journal={arXiv preprint arXiv:2605.03042},
  416. year={2026}
  417. }
  418. ```
  419. ## ⚖️ License
  420. MIT — see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.