Skip to content
Back to search

sloptrim

A local detector for AI-writing patterns. Scores every prose file your agent saves. Python standard library only, no network, no model.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/seyedehsanhadi--sloptrim/badge.svg)](/listings/seyedehsanhadi--sloptrim)

HTML

<a href="/listings/seyedehsanhadi--sloptrim"><img src="/api/listings/seyedehsanhadi--sloptrim/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Paste into Claude Code, Codex, Cursor, or any coding agent:

Install the sloptrim plugin from https://github.com/seyedehsanhadi/sloptrim

Restart, then run `/sloptrim doctor`. It answers with four `[OK]` lines. They test the local install, not whether the host delivers hook events, so save a prose file and look for its entry in `/sloptrim show`.

<details> <summary>Explicit commands, and installing without the marketplace</summary>

/plugin marketplace add seyedehsanhadi/sloptrim
/plugin install sloptrim@sloptrim
git clone https://github.com/seyedehsanhadi/sloptrim.git ~/.claude/skills/sloptrim
mkdir -p ~/.claude/commands
cp ~/.claude/skills/sloptrim/install/sloptrim-command.md ~/.claude/commands/sloptrim.md

Do not skip the `mkdir`. On a fresh machine `~/.claude/commands` does not exist yet and the copy fails with "No such file or directory". In PowerShell the last two lines are `New-Item -ItemType Directory -Force $HOME/.claude/commands` and `Copy-Item`. The copy puts `/sloptrim` in the `/` menu, because Claude Code does not scan a skill folder's own `commands/`. A marketplace install needs no such step. Either way the router also answers to `/sloptrim:sloptrim`.

</details>

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 15 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "sloptrim",
  3. "owner": {
  4. "name": "Seyed Ehsan Hadi",
  5. "url": "https://github.com/seyedehsanhadi"
  6. },
  7. "description": "Prose-quality tooling for AI-assisted drafts. Ships sloptrim: a local detector for prose saved through supported file-edit tools and size limits.",
  8. "plugins": [
  9. {
  10. "name": "sloptrim",
  11. "source": "./",
  12. "description": "Scores the first 262,144 characters of extracted prose from files accepted up to 512 KB for plain text and 4 MB for supported archives, against a catalogue of 71 documented patterns, 62 of them with a detector behind them and 50 able to move the score. It names the ones that fired so the flagged spans get fixed before the file ships. The 0 to 100 score lands in one of five bands: clean, light tells, mixed, heavy tells, pervasive tells. Every band describes the prose and none of them names an author. Reads 20 formats including .docx and .epub. Deterministic, Python standard library only, no network, no model. Prose only; code, config and commits stay untouched."
  13. }
  14. ]
  15. }
README.md · README · 184 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. <h1>
  3. <picture>
  4. <source media="(prefers-color-scheme: dark)" srcset="assets/sloptrim-logo-dark.svg">
  5. <img src="assets/sloptrim-logo.svg" width="358" alt="sloptrim">
  6. </picture>
  7. </h1>
  8. **A local detector for AI-writing patterns. It scores the first 262,144 characters of extracted
  9. prose in files saved through supported file-edit tools, and asks for the flagged
  10. spans to be fixed. Plain text is accepted up to 512 KB, supported archives up to 4 MB.**
  11. Python standard library only, no network, no model. Prose only, never code.
  12. [![test](https://github.com/seyedehsanhadi/sloptrim/actions/workflows/test.yml/badge.svg)](https://github.com/seyedehsanhadi/sloptrim/actions/workflows/test.yml)
  13. [![Version](https://img.shields.io/badge/version-0.9.7-blue)](CHANGELOG.md)
  14. [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE.txt)
  15. [![Dependencies](https://img.shields.io/badge/dependencies-none-blue)](scripts/detect.py)
  16. [![Tests](https://img.shields.io/badge/tests-305-blue)](tests/)
  17. [![Python](https://img.shields.io/badge/python-3.9%2B-blue)](scripts/detect.py)
  18. [Install](#install) &middot; [What it does](#what-it-does) &middot; [Measured](#measured) &middot; [Limits](#what-it-cannot-do) &middot; [Patterns](references/patterns.md) &middot; [Ethics](ETHICS.md)
  19. </div>
  20. > [!IMPORTANT]
  21. > **This is a command-line tool and an agent plugin. There is no website and no hosted version.**
  22. > Nothing you write is uploaded, there is no account, and no text ever leaves your machine.
  23. > Any site offering a service under this name is unrelated to this project.
  24. <picture>
  25. <source media="(prefers-color-scheme: dark)" srcset="assets/demo-dark.svg">
  26. <img src="assets/demo-light.svg" width="800" alt="A recorded session. The contract arrives at session start and again inside a subagent, and each prompt gets a one-line reminder. A saved file scores 45 and its patterns are named. After the fix the next read is clean at 15. The same text saved as a .docx scores the same, read out of the zip, and inside a source file it is not scored. Every hook latency is measured.">
  27. </picture>
  28. <div align="center"><sub>A recorded session, not a mock-up. <a href="record/session_capture.py">record/session_capture.py</a> fires the real hooks with the payloads Claude Code sends and writes <a href="record/session.json">record/session.json</a>; <a href="record/anim.py">record/anim.py</a> draws it. The hook lines, the scores and the milliseconds come from that capture; the grey labels beside them are narration. The prose being scored is <a href="record/draft.md">record/draft.md</a>, a short piece written for the recording.</sub></div>
  29. ---
  30. ## Install
  31. Paste into Claude Code, Codex, Cursor, or any coding agent:
  32. ```text
  33. Install the sloptrim plugin from https://github.com/seyedehsanhadi/sloptrim
  34. ```
  35. Restart, then run `/sloptrim doctor`. It answers with four `[OK]` lines. They test
  36. the local install, not whether the host delivers hook events, so save a prose file
  37. and look for its entry in `/sloptrim show`.
  38. <details>
  39. <summary>Explicit commands, and installing without the marketplace</summary>
  40. ```
  41. /plugin marketplace add seyedehsanhadi/sloptrim
  42. /plugin install sloptrim@sloptrim
  43. ```
  44. ```bash
  45. git clone https://github.com/seyedehsanhadi/sloptrim.git ~/.claude/skills/sloptrim
  46. mkdir -p ~/.claude/commands
  47. cp ~/.claude/skills/sloptrim/install/sloptrim-command.md ~/.claude/commands/sloptrim.md
  48. ```
  49. Do not skip the `mkdir`. On a fresh machine `~/.claude/commands` does not exist yet
  50. and the copy fails with "No such file or directory". In PowerShell the last two lines
  51. are `New-Item -ItemType Directory -Force $HOME/.claude/commands` and `Copy-Item`.
  52. The copy puts `/sloptrim` in the `/` menu, because Claude Code does not scan a skill
  53. folder's own `commands/`. A marketplace install needs no such step. Either way the
  54. router also answers to `/sloptrim:sloptrim`.
  55. </details>
  56. ## What it does
  57. The score is 0-100 against 71 documented patterns. 62 of them have a detector; the
  58. other 9 need a reading and are worked during the rewrite. Of the 62, 50 can move the
  59. score and 12 are reported as writing advice and count for nothing: most of them because
  60. measurement showed they mark formal register rather than machine authorship, the rest
  61. because they are typographic habits.
  62. | | |
  63. |---|---|
  64. | Formats | 20, including `.docx`, `.pptx`, `.xlsx`, OpenDocument, `.epub`, `.ipynb`, LaTeX; the first 262,144 characters of extracted prose is scored |
  65. | Runs in | Claude Code, on save. Other agents via `/sloptrim init`, which writes the contract to `AGENTS.md`, and `.cursor/rules/` |
  66. | Needs | Node for the hooks, Python 3.9 or newer for the detector, nothing else |
  67. | Suite | 222 Python tests and 83 hook checks, with CI configured for Linux, Windows and macOS, against Python 3.9 and 3.13 (macOS on 3.13) |
  68. | Does not see | A file written by a `Bash` command, which reaches disk without passing `Write` or `Edit` |
  69. | Context cost | About 1,400 tokens at session start and for each subagent in `full` (1,120 in `lite`, 1,415 in `strict`), plus a one-line reminder per prompt. `SLOPTRIM_MODE=off` makes it zero for a run |
  70. | Command | Effect |
  71. |---|---|
  72. | `/sloptrim full` | Contract + guard *(default)* |
  73. | `/sloptrim strict` | Flags at 20 instead of 40, and asks for a character scrub |
  74. | `/sloptrim lite` / `off` | Contract only / nothing |
  75. | `/sloptrim check <file>` | Score a file, name the tells, no rewrite |
  76. | `/sloptrim init` | Write the contract to `./AGENTS.md` and a Cursor rule to `./.cursor/rules/` |
  77. | `/sloptrim doctor` | Diagnose the install |
  78. For one run only, set `SLOPTRIM_MODE` to `off`, `lite`, `full` or `strict`. It beats the saved level and is never written back, so a headless code job can run `SLOPTRIM_MODE=off claude -p ...` without paying for the prose contract (about 1,400 tokens at session start and the same for each subagent).
  79. ```bash
  80. python scripts/detect.py draft.docx # JSON: patterns, metrics, 0-100 score
  81. ```
  82. A score lands in one of five bands: `clean`, `light tells`, `mixed`, `heavy tells`,
  83. `pervasive tells`. The guard nudges above 40, or above 20 in strict mode.
  84. ## Measured
  85. Sloptrim's public matched benchmark uses five separate 30-human/30-machine arms
  86. from the [Human Detectors](https://github.com/jenna-russell/human_detectors)
  87. release, pinned at commit `afcf03d`. Each arm is matched by prompt and scored
  88. separately. AUC is a ranking measure, not accuracy at Sloptrim's guard threshold.
  89. | machine arm | ROC-AUC | bootstrap 95% CI | default TPR / FPR |
  90. |---|---:|---:|---:|
  91. | GPT-4o | **0.943** | 0.870–0.992 | 46.7% / 0% |
  92. | Claude 3.5 Sonnet | **0.842** | 0.729–0.936 | 3.3% / 0% |
  93. | o1-pro | **0.881** | 0.777–0.957 | 23.3% / 3.3% |
  94. | paraphrased GPT-4o | **0.832** | 0.728–0.925 | 6.7% / 3.3% |
  95. | humanized o1-pro | **0.783** | 0.668–0.889 | 0% / 3.3% |
  96. <picture>
  97. <source media="(prefers-color-scheme: dark)" srcset="assets/detection-dark.svg">
  98. <img src="assets/detection-light.svg" width="756" alt="Public matched benchmark ROC-AUC: GPT-4o 0.943, Claude 3.5 Sonnet 0.842, o1-pro 0.881, paraphrased GPT-4o 0.832, humanized o1-pro 0.783.">
  99. </picture>
  100. Across these arms, Sloptrim achieved ROC-AUC **0.783–0.943**. Confidence intervals
  101. use 10,000 paired prompt-cluster bootstrap resamples. The public
  102. [result record](docs/research/frontier-benchmark-results.json) pins the source hash;
  103. the harness refuses any other file. The benchmark texts are not redistributed here.
  104. ```bash
  105. git clone https://github.com/jenna-russell/human_detectors.git
  106. git -C human_detectors checkout afcf03d14d2da4a038d8d0fafa5ec779dd858181
  107. python scripts/benchmark_frontier.py PATH_TO_HUMAN_DETECTORS_JSON
  108. ```
  109. ## What it cannot do
  110. **Format support means reading selected text, not rebuilding every document.**
  111. DOCX/DOCM read main-body paragraphs and tables, excluding headers, footers and
  112. comments. Explicit code styles (`Code`, `SourceCode`, `CodeBlock`, `CodeChar`,
  113. `HTMLPreformatted`) are excluded; inherited styles and unmarked code are not
  114. resolved. Monospace fonts alone do not imply code. PPTX/PPTM read slide text,
  115. excluding speaker notes. XLSX/XLSM read cell
  116. text; OpenDocument reads `content.xml`; EPUB reads HTML/XHTML members; notebooks
  117. read Markdown cells only. Embedded images/OCR are unsupported. PDF and RTF are
  118. recorded as unreadable rather than scored. Plain formats use Markdown-style code
  119. masking; `.tex` paths additionally protect common math, comments, literal-code
  120. environments, citations and preambles. This is a lexical check: it does not expand
  121. custom macros, resolve includes, or parse every MDX/RST/Org/AsciiDoc construct.
  122. `--clean` prints text. It must never be redirected over an Office archive, EPUB
  123. or notebook. Use a format-aware editor for prose revisions and validate the saved
  124. document. Pass a `.tex` filename to preserve its syntax context; stdin is treated
  125. as Markdown/plain prose.
  126. **It cannot prove whether a model wrote something.** The public arms show that the
  127. score often ranks these machine samples above matched human samples. The threshold
  128. results show why that is not the same as a dependable yes/no classifier: sensitivity
  129. changes sharply with model, prompt, formatting and threshold.
  130. **Long Markdown documents written by people can score high.** Headings, bullet lists
  131. and bold labels are counted, and a long project README has many of each. In a sample
  132. of 102 READMEs written before 2022, 18 scored above 40. Read the named patterns
  133. before rewriting a document of that kind.
  134. **It is not an authorship classifier and must not be used as one.** A score says
  135. something about writing, never about a person. Read [ETHICS.md](ETHICS.md).
  136. ## Related
  137. [Privacyassay](https://privacyassay.com) measures what a website can read from your
  138. browser and scores how much of it your browser hides. Same idea applied to a different
  139. surface: run the check locally, upload nothing, report what the evidence supports and
  140. no more. Source at [seyedehsanhadi/privacyassay](https://github.com/seyedehsanhadi/privacyassay).
  141. ## License
  142. Apache-2.0 ([full text](LICENSE.txt), [NOTICE](NOTICE)). Cite with
  143. [CITATION.cff](CITATION.cff).
scripts/detect.py · Bundled script · 2,454 lines
  1. #!/usr/bin/env python3
  2. """detect.py: deterministic scan for sloptrim's AI-writing patterns. Reads stdin or a file argument, emits JSON, or scrubbed text with --clean."""
  3. from __future__ import annotations
  4. import json
  5. import os
  6. import collections
  7. import re
  8. import statistics
  9. import sys
  10. import unicodedata
  11. MAX_SAMPLES = 3
  12. # ----------------------------------------------------------- lexical patterns
  13. AI_VOCAB = (
  14. r"\b(?:additionally|align(?:s|ed|ing)? with|crucial|"
  15. r"delv(?:e|es|ed|ing)|tapestry|pivotal|vibrant|meticulous(?:ly)?|"
  16. r"testament|underscor(?:e|es|ed|ing)|intricate|intricacies|interplay|"
  17. r"garner(?:s|ed|ing)?|bolster(?:s|ed|ing)?|foster(?:s|ed|ing)?|"
  18. r"showcas(?:e|es|ed|ing)|emphasiz(?:e|es|ed|ing)|enduring|"
  19. r"enhanc(?:e|es|ed|ing)|leverag(?:e|es|ed|ing)|utiliz(?:e|es|ed|ing)|"
  20. r"facilitat(?:e|es|ed|ing)|encompass(?:es|ed|ing)?|harness(?:es|ed|ing)?|"
  21. r"holistic|paradigm|transformative|unprecedented|myriad|plethora|"
  22. r"robust|seamless(?:ly)?|navigat(?:e|es|ed|ing)|embark(?:s|ed|ing)?|"
  23. r"craft(?:s|ed|ing)?|landscape|journey|realm|multifaceted|"
  24. r"elevat(?:e|es|ed|ing)|empower(?:s|ed|ing)?|unlock(?:s|ed|ing)?|"
  25. r"captures? the (?:essence|spirit|imagination|heart|magic)|"
  26. r"capturing the (?:essence|spirit|imagination|heart|magic))\b"
  27. )
  28. ING_TAIL = (
  29. r",\s+(?:highlighting|reflecting|symbolizing|emphasizing|underscoring|"
  30. r"showcasing|fostering|cultivating|encompassing|signaling|demonstrating|"
  31. r"illustrating|representing|suggesting|indicating|reinforcing|leveraging|"
  32. r"aligning|embodying|exemplifying|epitomizing)\b"
  33. )
  34. PROMO = (
  35. r"\b(?:nestled|breathtaking|must-visit|stunning|picturesque|charming|"
  36. r"idyllic|vibrant community|rich cultural heritage|in the heart of|"
  37. r"world-class|top-tier|unparalleled|premier|leading|"
  38. r"esteemed organization|valuable asset|passion for excellence|"
  39. r"proven track record|results-oriented|wealth of experience|"
  40. r"dynamic team|fast-paced environment|deeply passionate|"
  41. r"perfectly situated|(?:close|tight)-knit team|"
  42. r"i am excited to (?:apply|bring)|uniquely positioned to)\b"
  43. )
  44. DIFF_ANCHORED = (
  45. r"\b(?:unlike the (?:previous|prior|old|earlier) (?:version|release|"
  46. r"implementation|approach)|"
  47. r"(?:the|this) (?:latest |new )?(?:update|release|version|change|patch|"
  48. r"design|redesign) "
  49. r"(?:adds|introduces|brings|delivers|includes|provides|improves)|"
  50. r"(?:has|have) been (?:updated|refactored|improved|enhanced|changed) to|"
  51. r"(?:previously|formerly),? (?:the|this|it|users?) [^.,\n]{0,40} "
  52. r"(?:now|but now)|"
  53. r"(?:wasn'?t|weren'?t) there before|"
  54. r"we(?:'ve| have) (?:now )?(?:added|removed|replaced|refactored|migrated))\b"
  55. )
  56. VAGUE_ATTR = (
  57. r"\b(?:experts (?:argue|believe|say|claim|note|suggest)|"
  58. r"industry (?:reports|observers|analysts)|"
  59. r"observers have (?:cited|noted)|some critics argue|"
  60. r"researchers have found|studies suggest|it is widely believed)\b"
  61. )
  62. CHALLENGES = (
  63. r"\b(?:despite (?:these|its|the) (?:challenges|obstacles)|"
  64. r"challenges? (?:and|&) (?:future )?(?:prospects?|outlook|legacy)|"
  65. r"continues? to thrive|typical of (?:emerging|urban|modern)|"
  66. r"faces (?:several|many|numerous) challenges)\b"
  67. )
  68. COPULA_AVOID = (
  69. r"\b(?:serves as|stands as|functions as|acts as|represents (?:a|the)|"
  70. r"marks (?:a|the)|boasts|"
  71. r"features (?:a|an|the|\d+|one|two|three|four|five|six|seven|eight|nine|"
  72. r"ten|several|multiple|numerous|over|more than)\b|"
  73. r"offers a|refers to)\b"
  74. )
  75. NEG_PARALLEL = (
  76. r"\b(?:it'?s not (?:just|merely) (?:about|a|an)|"
  77. r"not only [^.\n]{2,40}? but (?:also )?|"
  78. r"it doesn'?t just [^.,]{2,40} it)\b"
  79. )
  80. TAILING_NEG = r",\s+no (?:guessing|waste|wasted motion|hassle|fluff|stress|jargon)\b"
  81. # -------------------------------------------------------- formatting patterns
  82. BOLD = r"\*\*[^*\n]{1,60}\*\*"
  83. INLINE_HEADER = r"(?:^|\n)[ \t*\-]*\*\*[^*]+(?::\*\*|\*\*[ \t]*:)"
  84. TITLE_CASE_HEADING = re.compile(
  85. r"^(#{1,6})\s+([A-Z][^\n]+)$", re.MULTILINE
  86. )
  87. EMOJI = re.compile(r"[\U0001F1E6-\U0001F1FF\U0001F300-\U0001FAFF☀-➿]")
  88. CURLY = r"[‘’“”]"
  89. # -------------------------------------------------------- assistant artefacts
  90. CHATBOT = (
  91. r"\b(?:I hope this helps|certainly(?=!)|of course(?=!)|"
  92. r"let me know if you|would you like me to|"
  93. r"feel free to (?:ask|reach out)|happy to help(?=[!.]|$)|"
  94. r"(?:here|below) is (?:a|an|the)?\s*(?:overview|summary|rundown|recap|brief(?!\s+\w))|"
  95. r"here (?:are|'?s) (?:a few|three|two|some) (?:options|drafts|versions|variations)|"
  96. r"here (?:is|'?s) a draft(?: you can use)?|"
  97. r"depending on the (?:vibe|tone|style) (?:of|you)|"
  98. r"i hope (?:this email )?finds you well|"
  99. r"as an? (?:AI|LLM|AI assistant|AI language model|"
  100. r"large language model|language model))\b"
  101. )
  102. CUTOFF = (
  103. r"\b(?:as of my (?:last )?(?:training|knowledge)|up to my (?:last )?"
  104. r"(?:training|knowledge)|while specific details are (?:limited|scarce|"
  105. r"not (?:extensively|widely) documented)|based on (?:the )?available "
  106. r"information|as of my (?:knowledge )?cutoff)\b"
  107. )
  108. SYCOPHANTIC = (
  109. r"\b(?:you(?:'?re| are) absolutely right\b|"
  110. r"(?:great question|excellent point|"
  111. r"what a (?:thoughtful|great|wonderful|fantastic) question|"
  112. r"that'?s an? (?:great|wonderful|fantastic|brilliant|excellent) "
  113. r"(?:point|question))[!.,])"
  114. )
  115. FILLER = (
  116. r"\b(?:in order to|due to the fact that|at this point in time|"
  117. r"in the event that|has the ability to|it is important to note that|"
  118. r"it should be mentioned that|needless to say)\b"
  119. )
  120. HEDGES = (
  121. r"\b(?:may|might|could|possibly|potentially|arguably|perhaps|"
  122. r"seems?(?:\s+(?:to|like))?|appears?\s+to|somewhat|relatively|fairly|"
  123. r"kind of|sort of)\b"
  124. )
  125. GENERIC_END = (
  126. r"\b(?:the future looks bright|exciting times lie ahead|"
  127. r"journey toward excellence|step in the right direction|"
  128. r"(?:the )?(?:outlook|future)(?: for [^.\n]{0,50})? (?:is|looks|remains) bright|"
  129. r"well[ -]positioned for (?:whatever comes next|the future|the road ahead))\b"
  130. )
  131. HYPHEN_CLICHE = (
  132. r"\b(?:mission-critical|battle-tested|out-of-the-box|enterprise-grade|"
  133. r"best-of-breed|plug-and-play|industry-leading|purpose-built|"
  134. r"bleeding-edge|turn-key|value-added|first-class|laser-focused|"
  135. r"tried-and-true|ready-made|best-in-class|future-proof|cutting-edge|"
  136. r"results-driven|customer-centric|state-of-the-art|next-generation|"
  137. r"forward-thinking|user-friendly|data-driven|cross-functional|"
  138. r"client-facing|end-to-end)\b"
  139. )
  140. AUTHORITY_TROPE = (
  141. r"\b(?:here'?s the thing|make no mistake|the truth is|let'?s be honest|"
  142. r"the simple fact is|at its core|what really matters|more than anything|"
  143. r"at the end of the day|the bottom line is|in reality|fundamentally,|"
  144. r"the real question is|the heart of the matter|the deeper issue|"
  145. r"when you get down to it|when all is said and done)(?!\w)"
  146. )
  147. SIGNPOSTING = (
  148. r"\b(?:let'?s (?:dive|delve|explore)|let'?s break (?:this|it) down|"
  149. r"here'?s what you need to know|without further ado|now let'?s look at|"
  150. r"buckle up|stay with me|bear with me|let'?s unpack(?: this| that| it)?|"
  151. r"here'?s the rundown|we'?ll get to (?:that|this) in a moment|first, some context|"
  152. r"before we begin|what follows is|let'?s take a (?:closer )?look|"
  153. r"(?:are you )?ready to dive in(?:to)?)\b"
  154. )
  155. CATALOG_LEADIN = (
  156. r"\b(?:used|uses|employs?|employed|relied? on|utilizes?)\s+"
  157. r"(?:several|various|a\s+number\s+of|multiple|different|many|numerous)"
  158. r"\s+\w+"
  159. )
  160. CATALOG_PIVOT = (
  161. r"\bThese\s+(?:\w+\s+)?(?:provide|offer|give|reveal|show|allow|enable|help)"
  162. r"\s+(?:information|insight|understanding|details|data)\b"
  163. )
  164. EMPTY_PIVOT = (
  165. r"\b(?:it'?s worth noting that|it'?s worth mentioning that|"
  166. r"it bears (?:noting|mentioning) that|"
  167. r"one thing to (?:consider|keep in mind) is|"
  168. r"it is interesting to note that|a key consideration is|"
  169. r"a point to highlight is)\b"
  170. )
  171. COMPULSIVE_CONCLUSION = (
  172. r"(?:^|\n)[ \t]*(?:in conclusion|overall|in summary|to summarize|"
  173. r"to conclude|all in all|in essence|ultimately)[,\s]"
  174. )
  175. OUTCOME_SPECULATION = (
  176. r",\s+(?:raising questions about|prompting reflection on|"
  177. r"sparking debate about|with broader implications for|"
  178. r"reshaping our understanding of|paving the way for|"
  179. r"signaling a shift toward)\b"
  180. )
  181. SIMPLE_YET = (
  182. r"\bsimple yet (?:effective|powerful|profound|elegant|"
  183. r"transformative|sophisticated|beautiful)\b"
  184. )
  185. # Multi-word significance inflation (catalogue section 10). The single words
  186. # live in AI_VOCAB; these are the closed collocations, which age more slowly.
  187. # ------------------------------------------------ significance and notability
  188. SIGNIFICANCE_INFLATION = (
  189. r"\b(?:stands|serves|stood|served) as (?:a |an |the )?"
  190. r"(?:testament|reminder|symbol|beacon|cornerstone|bridge|gateway|hallmark)\b"
  191. r"|\b(?:underscor|highlight|emphasis)(?:es|ing|ed) (?:its|the|their|his|her) "
  192. r"(?:importance|significance|enduring|central|vital|crucial|pivotal|key) ?\w*\b"
  193. r"|\breflect(?:s|ing|ed)? (?:a )?broader (?:trend|shift|pattern|movement|change)\b"
  194. r"|\bmark(?:s|ing|ed) a (?:turning point|milestone|watershed|new (?:chapter|era))\b"
  195. r"|\bsymboliz(?:es|ing)|\bsymbolis(?:es|ing)\b(?=[^.]{0,40}\b(?:ongoing|enduring|lasting)\b)"
  196. r"|\bset(?:s|ting)? the stage for\b"
  197. r"|\bleft? an indelible mark\b"
  198. r"|\bplay(?:s|ed|ing)? (?:a |an )(?:crucial|pivotal|vital|key|central|significant) role\b"
  199. r"|\b(?:has|have) (?:shaped|sparked|come to (?:define|symbolize|symbolise))\b"
  200. )
  201. # Notability name-dropping (catalogue section 11): a credit verb plus two or
  202. # more named outlets in one breath, or a raw follower count as evidence.
  203. NOTABILITY_LIST = (
  204. r"\b(?:featured|profiled|covered|cited|mentioned|highlighted|showcased) "
  205. r"(?:in|by)\b[^.]{0,120}?,[^.]{0,120}?\band\b[^.]{0,60}?[A-Z]"
  206. r"|\bactive (?:social media|online|digital) presence\b"
  207. r"|\bstrong (?:digital|online|social) presence\b"
  208. r"|\b(?:more than |over |upwards of )?[\d,.]+[KkMm]? followers\b"
  209. r"|\bwidely (?:covered|reported|featured) (?:by|in)\b"
  210. r"|\bindependent coverage\b"
  211. )
  212. # Closed collocations that survive vocabulary rotation, grouped by the move they
  213. # make rather than by the words they use.
  214. # -------------------------------------------------------- closed collocations
  215. FAUX_INSIGHT = (
  216. r"\bhere(?:'s| is) what (?:nobody|no one|most people|they don't)\b"
  217. r"|\bwhat (?:most|so many) people get wrong\b"
  218. r"|\bthe (?:part|thing|bit) (?:that )?(?:everyone|most people|nobody) (?:misses|gets wrong|overlooks)\b"
  219. r"|\bnobody (?:talks|tells you) about\b"
  220. r"|\bthe (?:dirty )?secret (?:that )?(?:nobody|no one) (?:tells|mentions)\b"
  221. r"|\bhere(?:'s| is) the (?:uncomfortable|inconvenient|hard) (?:truth|part)\b"
  222. )
  223. DEFINITIONAL_METAPHOR = (
  224. r"\b(?:is|are|was|were|remains|becomes) (?:the |a )?"
  225. r"(?:language|currency|architecture|heartbeat|lifeblood|DNA|backbone|bedrock|"
  226. r"engine|scaffolding|connective tissue) of\b"
  227. )
  228. COLON_REVEAL = (
  229. # Line-local whitespace only. A \s* here would cross newlines and retry at
  230. # every position, which is quadratic on newline-heavy input.
  231. r"(?m)^[ \t]{0,8}(?:The (?:best part|kicker|catch|twist|result|"
  232. r"real (?:reason|problem|question))|Plot twist|Here(?:'s| is) the thing|"
  233. r"The bottom line)[ \t]{0,4}:"
  234. )
  235. CHAT_RESIDUE = (
  236. r"\bto (?:answer|address) your question\b"
  237. r"|\bin (?:response|answer) to your question\b"
  238. r"|\byour question (?:about|regarding|concerning)\b"
  239. r"|\bas (?:you )?(?:requested|asked)\b, here"
  240. r"|\bhere(?:'s| is) (?:a |the )?(?:draft|first pass|revised version) (?:of|for) your\b"
  241. )
  242. PRIVACY_FILLER = (
  243. r"\bmaintains? a (?:relatively )?low(?:-key)? (?:public )?profile\b"
  244. r"|\bkeeps? (?:his|her|their) personal life private\b"
  245. r"|\bprefers? to keep (?:his|her|their) (?:personal )?life out of the (?:public eye|spotlight)\b"
  246. r"|\blittle is publicly known about (?:his|her|their) personal life\b"
  247. )
  248. # False ranges (catalogue section 15): "from X to Y" where the poles are not on
  249. # a shared scale. Only the two closed forms, which is where the tell is safe.
  250. FALSE_RANGE = (
  251. r"\bfrom the (?:dawn|birth|earliest days|inception|advent|emergence|humble)\b"
  252. r"[^.]{0,80}?\bto the (?:modern|present|current|contemporary|digital|"
  253. r"future|mighty|vanished|towering)\b"
  254. r"|\bfrom (?:the )?humble\b[^.]{0,60}?\bto (?:the )?(?:mighty|towering|grand|sprawling)\b"
  255. )
  256. QA_RHETORICAL = (
  257. r"\b(?:is|are|was|were|will|does|do|did|can|could|should|would|"
  258. r"has|have|had)\s+[^.?!\n]{2,40}\?\s+(?:yes|no|absolutely|never|"
  259. r"definitely|certainly|maybe|probably|possibly|rarely|always|"
  260. r"hardly|nope|nah|sure|exactly|sometimes)[.!]"
  261. )
  262. SELF_THOROUGHNESS = (
  263. r"\b(?:this comprehensive (?:guide|overview|analysis)|"
  264. r"this in-depth (?:analysis|look|examination)|"
  265. r"this complete (?:guide|overview)|"
  266. r"a thorough examination of|everything you need to know about)\b"
  267. )
  268. EDITORIAL_INTERJECTION = (
  269. r"\b(?:it'?s important to (?:remember|note|understand|recognize)|"
  270. r"it is important to (?:remember|note|understand|recognize)|"
  271. r"one must note that|it is worth pointing out|"
  272. r"it cannot be overstated that|it must be emphasized that)\b"
  273. )
  274. # ------------------------------------------------------------ character layer
  275. _EXTRA_INVISIBLE: set = {
  276. 0x034F,
  277. 0x115F, 0x1160,
  278. 0x17B4, 0x17B5,
  279. 0x180B, 0x180C, 0x180D, 0x180F,
  280. 0x3164,
  281. 0xFFA0,
  282. 0x2065,
  283. }
  284. _EXTRA_INVISIBLE |= set(range(0xFFF0, 0xFFF9))
  285. _EXTRA_INVISIBLE |= set(range(0xE0100, 0xE01F0))
  286. # These three were kept unconditionally because they are load-bearing in Arabic,
  287. # Indic scripts and emoji. That also made them the one channel a payload could
  288. # ride through untouched, so they are now decided by context like everything
  289. # else: kept beside the scripts that need them, dropped between Latin letters,
  290. # where they do nothing at all.
  291. _CONTEXTUAL_INVISIBLE = {0x200C, 0x200D} | set(range(0xFE00, 0xFE10))
  292. # A codepoint is debris only where it has no job. The same character can be a
  293. # watermark in English and load-bearing three words later, so context decides.
  294. # Each rule below exists because removing it blind destroyed real text: the
  295. # England flag became a black flag, a Japanese name lost its variant, Arabic
  296. # changed reading order, Thai lost every line-break opportunity.
  297. _SPACELESS = (
  298. (0x0E00, 0x0E7F), # Thai
  299. (0x0E80, 0x0EFF), # Lao
  300. (0x1780, 0x17FF), # Khmer
  301. (0x1000, 0x109F), # Myanmar
  302. (0x3040, 0x30FF), # kana
  303. (0x3400, 0x4DBF), (0x4E00, 0x9FFF), (0xF900, 0xFAFF), # CJK
  304. )
  305. _RTL = ((0x0590, 0x08FF), (0xFB1D, 0xFDFF), (0xFE70, 0xFEFF),
  306. (0x10800, 0x10FFF), (0x1E800, 0x1EFFF))
  307. _MONGOLIAN = ((0x1800, 0x18AF),)
  308. _KHMER = ((0x1780, 0x17FF),)
  309. _CJK = ((0x3400, 0x4DBF), (0x4E00, 0x9FFF), (0xF900, 0xFAFF), (0x20000, 0x2FA1F))
  310. # Scripts where a joiner or non-joiner changes how a word is written.
  311. _JOINING = ((0x0590, 0x08FF), (0x0900, 0x0DFF), (0xFB1D, 0xFDFF), (0xFE70, 0xFEFF),
  312. (0x1000, 0x109F), (0x1780, 0x17FF))
  313. _SYMBOL = ((0x2190, 0x2BFF), (0x1F000, 0x1FAFF), (0xFE0F, 0xFE0F), (0x20E3, 0x20E3))
  314. def _emoji_ish(ch: str) -> bool:
  315. if not ch:
  316. return False
  317. return _in(ch, _SYMBOL) or unicodedata.category(ch) == "So"
  318. def _in(ch: str, ranges) -> bool:
  319. if not ch:
  320. return False
  321. cp = ord(ch)
  322. return any(a <= cp <= b for a, b in ranges)
  323. _RTL_RE = re.compile("[%s]" % "".join("\\U%08x-\\U%08x" % (a, b) for a, b in _RTL))
  324. def _has_rtl(text: str) -> bool:
  325. """Whether the document contains right-to-left script.
  326. A regex, because this is asked once per scan over the whole document and
  327. the Python loop it replaced put a 540 KB file within a second of the
  328. guard's 8-second timeout.
  329. """
  330. return _RTL_RE.search(text) is not None
  331. def _flag_tags(text: str) -> set:
  332. """Find flag tags once; walking backwards at every tag is quadratic."""
  333. return {i for m in re.finditer("\U0001F3F4[\U000E0020-\U000E007E]+\U000E007F", text)
  334. for i in range(m.start() + 1, m.end())}
  335. def _keeps_function(ch: str, prev: str, nxt: str, rtl_doc: bool,
  336. flag_tag: bool = False) -> bool:
  337. """True when this invisible character is doing a job here."""
  338. cp = ord(ch)
  339. # Emoji tag sequences: the subdivision flags are a base plus tag letters
  340. # ending in U+E007F. Stripping the tags leaves a plain black flag.
  341. if 0xE0020 <= cp <= 0xE007F:
  342. return flag_tag
  343. # Ideographic variation sequences select the right form of a kanji, which
  344. # is how names and places are written correctly.
  345. if 0xE0100 <= cp <= 0xE01EF:
  346. return _in(prev, _CJK)
  347. # Scripts written without spaces use ZWSP to mark where a line may break.
  348. if cp == 0x200B:
  349. return _in(prev, _SPACELESS) or _in(nxt, _SPACELESS)
  350. # A soft hyphen between two letters is a hyphenation point an author chose.
  351. if cp == 0x00AD:
  352. return bool(prev) and bool(nxt) and prev.isalpha() and nxt.isalpha()
  353. # Bidi controls order visible text; only meaningless where nothing is RTL.
  354. if cp in (0x200E, 0x200F, 0x061C) or 0x202A <= cp <= 0x202E or 0x2066 <= cp <= 0x2069:
  355. return rtl_doc
  356. if 0x180B <= cp <= 0x180F:
  357. return _in(prev, _MONGOLIAN)
  358. if cp in (0x17B4, 0x17B5):
  359. return _in(prev, _KHMER) or _in(nxt, _KHMER)
  360. # Joiner and non-joiner: real work in Arabic, Indic and emoji sequences,
  361. # nothing whatever between two Latin letters.
  362. if cp in (0x200C, 0x200D):
  363. return (_in(prev, _JOINING) or _in(nxt, _JOINING)
  364. or _emoji_ish(prev) or _emoji_ish(nxt))
  365. # A variation selector chooses a glyph form; with no glyph in front of it
  366. # there is no form to choose.
  367. if 0xFE00 <= cp <= 0xFE0F:
  368. # A keycap is digit + VS16 + U+20E3, so the base is behind and the
  369. # enclosing mark ahead; looking only backwards broke "press 1".
  370. return _emoji_ish(prev) or _in(prev, _CJK) or (nxt and ord(nxt) == 0x20E3)
  371. return False
  372. _SUSPECT = re.compile(r"[^\t\n\r\x20-\x7E]")
  373. def _is_invisible(ch: str) -> bool:
  374. if ch in "\t\n\r":
  375. return False
  376. cp = ord(ch)
  377. if cp in _CONTEXTUAL_INVISIBLE:
  378. return True
  379. if unicodedata.category(ch) in ("Cf", "Cc", "Zl", "Zp"):
  380. return True
  381. if cp in _EXTRA_INVISIBLE:
  382. return True
  383. return 0xE0000 <= cp <= 0xE0FFF
  384. def _is_nonstandard_space(ch: str) -> bool:
  385. return ch != " " and unicodedata.category(ch) == "Zs"
  386. def _char_name(cp: int) -> str:
  387. try:
  388. name = unicodedata.name(chr(cp))
  389. except (ValueError, OverflowError):
  390. name = ""
  391. if not name:
  392. if 0xE0000 <= cp <= 0xE0FFF:
  393. return "reserved tag/VS plane (invisible)"
  394. if cp <= 0x001F or 0x007F <= cp <= 0x009F:
  395. return "CONTROL"
  396. return "reserved / default-ignorable"
  397. return f"{name} [TAG block]" if 0xE0000 <= cp <= 0xE007F else name
  398. NOT_X_JUST_Y = (
  399. r"\b(?:"
  400. r"(?:no|not)\s+\w+,\s+(?:just|only|simply)\s+\w+"
  401. r"|(?:(?:is|are|was|were)n'?t|not)\s+(?:just|merely)\s+(?:a |an |the )?\w+\s*[,—–-]\s*"
  402. r"(?:it'?s|it is|it was|they'?re|they are|they were|that'?s|but)\b"
  403. r")"
  404. )
  405. COMPULSIVE_INTRO = (
  406. r"\b(?:in today['’]?s (?:[\w-]+ ){0,2}(?:world|landscape|environment|market)\b|"
  407. r"in a world where|in an era of|"
  408. r"more than ever before|we live in a (?:time|world) when|"
  409. r"(?:great|exciting|big) news[!:]|"
  410. r"ever (?:wanted|wondered|dreamed|wished) to [^?\n]{3,60}\?|"
  411. r"in an increasingly (?:interconnected|digital|competitive|complex) world)(?!\w)"
  412. )
  413. ARTICLE_TITLE_NOUN = (
  414. r"\bThe (?:List|Index|Compilation|Collection|Catalogue|Guide|Overview) of "
  415. r"[A-Z][\w\s]{3,60}\s+is (?:a|an|the)\b"
  416. )
  417. HYPHEN_RANGE = re.compile(
  418. r"(?<![\w-])(\d{4})-(\d{4})(?![\w-])"
  419. )
  420. _YEAR_MIN, _YEAR_MAX = 1500, 2999
  421. CONCRETE_EVIDENCE = (
  422. r"\b(?:without concrete evidence|do you have concrete evidence|"
  423. r"i (?:would )?need concrete (?:evidence|examples) to|"
  424. r"absent concrete proof|provide concrete evidence)\b"
  425. )
  426. ECOSYSTEM_PADDING = (
  427. r"\b(?:plays a (?:vital|crucial|key|significant) role in (?:its|the) "
  428. r"ecosystem|conservation efforts are (?:underway|ongoing|in place)|"
  429. r"serves as an indicator species|contributes to biodiversity)\b"
  430. )
  431. CANONICAL_SLOP = (
  432. r"unlock(?:ing)? (?:the |your |its )?full potential|"
  433. r"navigat(?:e|ing) the complexit(?:y|ies)|"
  434. r"a testament to the (?:\w+ ){0,2}(?:power|importance|spirit|resilience)\b|"
  435. r"stay(?:ing)? ahead of the curve|"
  436. r"turn(?:ing)? (?:challenges|obstacles) into opportunit(?:y|ies)|"
  437. r"the future is (?:full of possibilit(?:y|ies)|bright)|"
  438. r"something for everyone|"
  439. r"whether you (?:'?re|are) an? (?:beginner|novice|seasoned|professional|expert|pro|"
  440. r"student|newcomer|veteran)\b|"
  441. r"the key takeaway|"
  442. r"seamlessly integrat(?:e|es|ed|ing|ion)|"
  443. r"take (?:it|your \w+|things|productivity) to the next level|"
  444. r"work(?:ing)? smarter,? not harder|"
  445. r"more important than ever|"
  446. r"in an? (?:increasingly|ever-)\w|"
  447. r"drive(?:s|n)? (?:meaningful|real|tangible|transformative) (?:impact|change|results|value)|"
  448. r"stay(?:ing)? one step ahead|"
  449. r"harness(?:ing)? the power of|"
  450. r"pave(?:s|d)? the way for|"
  451. r"from startups to enterprises|"
  452. r"meet(?:s|ing)? you where(?:ver)? you are|"
  453. r"whether you'?re (?:a |an )?[\w-]+ or (?:just )?[\w-]+ing(?: out)?|"
  454. r"embark(?:s|ed|ing)? on an? (?:exciting|incredible|transformative) journey|"
  455. r"reimagin(?:e|es|ing) what'?s possible|"
  456. r"buckle up[,!]|"
  457. r"unlock(?:s|ing)? (?:new|endless|infinite) possibilit(?:y|ies)|"
  458. r"empower(?:s|ing)? (?:our|your|their) (?:\w+ )?(?:community|teams?|people|users)\b|"
  459. r"deeply passionate about|"
  460. r"wealth of (?:experience|knowledge|expertise)|"
  461. r"results?-(?:oriented|driven) (?:team|professional|approach|culture)|"
  462. r"cultivat(?:e|es|ing) a culture of|"
  463. r"culture of (?:relentless )?(?:excellence|innovation)|"
  464. r"nestled (?:in the heart of|at the intersection of)|"
  465. r"at the intersection of \w+ and \w+|"
  466. r"(?:bold|forward-thinking|shared) vision (?:of|for|to)\b|"
  467. r"working (?:hard )?behind the scenes|"
  468. r"packed with (?:powerful|exciting|amazing) (?:new )?features|"
  469. r"streamlines? your workflow|"
  470. r"(?:a|the) (?:brighter|better),? more \w+ (?:future|tomorrow|world)|"
  471. r"those who (?:dare to|embrace) \w+ will|"
  472. r"say (?:goodbye|hello) to|"
  473. r"stop [\w-]+ing[.,!]? (?:and )?start [\w-]+ing|"
  474. r"reclaim your [\w-]+|"
  475. r"transform your (?:life|business|workflow|day|home|routine) with|"
  476. r"power(?:s|ing)? the future of|"
  477. r"the ultimate (?:tool|solution|guide|companion|destination) for|"
  478. r"(?:start|begin) your [\w-]+ journey|"
  479. r"(?:^|[.!?]\s+)escape to (?:serenity|tranquility|the|our|this|your)\b|"
  480. r"blend(?:s|ing)? (?:[\w-]+ ){0,3}(?:charm|craftsmanship|tradition|simplicity|"
  481. r"heritage|flavors?) (?:with|and)|"
  482. r"we'?(?:re| are) (?:excited|thrilled|delighted|proud) to "
  483. r"(?:announce|introduce|launch|share|unveil)|"
  484. r"whether you'?re (?:aiming|looking|chasing|trying|hoping|seeking) to|"
  485. r"imagine a world where|"
  486. r"adventure awaits|"
  487. r"experience the magic of|"
  488. r"step into the world of|"
  489. r"gateway to (?:the world of|a world)|"
  490. r"nothing short of (?:amazing|extraordinary|explosive|remarkable|spectacular)|"
  491. r"it'?s time (?:we|to) (?:stop|rethink|embrace|move beyond)|"
  492. r"like never before|"
  493. r"you have the power to|"
  494. r"(?:the )?perfect blend of|"
  495. r"if you blinked[^.,\n]{0,24},? you (?:might have |probably )?missed"
  496. )
  497. PLACEHOLDER = (
  498. r"(?:\[(?:your|insert|add|enter|company name|client name|placeholder|todo|"
  499. r"tbd|tk|source url)\b[^\]\n]{0,40}\](?![(\[])|"
  500. r"\[(?:startup |company |client |journal |first |last |full )?name\](?![(\[])|"
  501. r"\[city, ?state\](?![(\[])|"
  502. r"\[(?:manuscript id|lead vc firm|editor'?s last name|date of submission)"
  503. r"(?: [^\]\n]{0,20})?\](?![(\[])|"
  504. # An all-caps bracket is a slot only when it names one, [COMPANY] or [XX].
  505. # Most are citation keys and labels: [RFC2119], [NII], [UNICODE]. Across
  506. # 13,000 documents the unrestricted form matched 194 human texts and no
  507. # machine slot, and the rule is decisive, so each was floored at 45.
  508. r"(?-i:\[(?=[A-Z][A-Z0-9 ._/-]{1,30}\])(?:[A-Z0-9 ._/-]*[ ._/-])?"
  509. r"(?:COMPANY|NAME|DATE|CITY|STATE|ADDRESS|EMAIL|PHONE|TITLE|POSITION|ROLE|"
  510. r"CLIENT|PRODUCT|AMOUNT|NUMBER|LINK|URL|TOPIC|RECIPIENT|MANAGER|CANDIDATE|"
  511. r"SENDER|SIGNATURE|SUBJECT|XX+)(?:[ ._/-][A-Z0-9 ._/-]*)?\])(?![(\[])|"
  512. # The double-brace form. Without it a rewrite could turn [Name] into
  513. # {{first_name}}, watch this rule stop firing, and read that as progress.
  514. r"\{\{ ?[a-z][a-z0-9 ._-]{1,30} ?\}\}|"
  515. r"\bx{3,}\b|\blorem ipsum\b)"
  516. )
  517. CHATBOT_REF_MARKUP = (
  518. r"(?:\bcite(?:turn|start|nav)\w*|\boai_citation\w*|\bcontentReference\b|"
  519. r"\bnavlist\b|【\d+†[^】]*】)"
  520. )
  521. AI_UTM = (
  522. r"utm_source=(?:chatgpt\.com|chat\.openai\.com|perplexity\.ai|claude\.ai|"
  523. r"gemini\.google\.com|copilot\.microsoft\.com|poe\.com)(?![\w]|\.\w)"
  524. )
  525. MODEL_DIALECT_CLAUDE = (
  526. r"\b(?:genuinely|fascinating|nuanced|thoughtfully|refreshingly|"
  527. r"strikingly|compellingly|quietly (?:powerful|profound|radical)|"
  528. r"deeply (?:human|personal|felt|unsettling)|resonates? deeply)\b"
  529. )
  530. MODEL_DIALECT_GPT = (
  531. r"\b(?:game-chang\w+|supercharge\w*|skyrocket\w*|turbocharge\w*|"
  532. r"revolutioniz\w+|double down on|dive deep(?:er)? into|"
  533. r"unlock(?:ing|s)? (?:the )?(?:power|potential|value)|"
  534. r"take[sn]? [^.,\n]{0,25} to the next level)\b"
  535. )
  536. PASSIVE_VOICE = re.compile(
  537. r"\b(?:is|are|was|were|been|being|be)\s+(?:\w+ly\s+)?(\w+(?:ed|en))\b",
  538. re.IGNORECASE,
  539. )
  540. _PASSIVE_STOP = {
  541. "red", "bed", "fed", "led", "wed", "sacred", "hundred", "kindred",
  542. "oxen", "wooden", "golden", "sudden", "hidden", "garden", "wicked",
  543. "naked", "linen", "kitchen", "sacred", "blessed", "rugged", "tired",
  544. }
  545. CONFUSABLES: dict[int, str] = {
  546. 0x0430: "a", 0x03F2: "c", 0x0441: "c", 0x0501: "d", 0x0435: "e",
  547. 0x0562: "g", 0x0456: "i", 0x0458: "j", 0x0578: "n",
  548. 0x03BF: "o", 0x043E: "o", 0x0440: "p", 0x0455: "s", 0x057E: "u",
  549. 0xFEAA: "v", 0x051D: "w", 0x0445: "x", 0x0443: "y", 0x04BB: "h",
  550. 0x0410: "A", 0x0412: "B", 0x0415: "E", 0x041A: "K", 0x041C: "M",
  551. 0x041D: "H", 0x041E: "O", 0x0420: "P", 0x0421: "C", 0x0422: "T",
  552. 0x0423: "Y", 0x0425: "X", 0x0406: "I", 0x0408: "J", 0x0405: "S",
  553. 0x0391: "A", 0x0392: "B", 0x0395: "E", 0x0396: "Z", 0x0397: "H",
  554. 0x0399: "I", 0x039A: "K", 0x039C: "M", 0x039D: "N", 0x039F: "O",
  555. 0x03A1: "P", 0x03A4: "T", 0x03A5: "Y", 0x03A7: "X",
  556. }
  557. _ALPHA_RUN = re.compile(r"[^\W\d_]+")
  558. # ---------------------------------------------------------- homoglyph folding
  559. def _native(ch: str) -> bool:
  560. """A letter that is not ASCII and not a lookalike: someone's real script."""
  561. return ch.isalpha() and ord(ch) > 127 and ord(ch) not in CONFUSABLES
  562. def _is_mixed_run(run: str) -> bool:
  563. has_latin = any("a" <= c.lower() <= "z" and ord(c) < 128 for c in run)
  564. has_confusable = any(ord(c) in CONFUSABLES for c in run)
  565. return has_latin and has_confusable and not any(_native(c) for c in run)
  566. def _fold_run(run: str) -> str:
  567. """Fold the lookalikes that are surrounded by Latin, and only those.
  568. `Сбер` glued to `Bank` used to come out as `Cбep`, because the run held
  569. both Latin and confusables. Three real Cyrillic letters were rewritten in a
  570. Russian word. A lookalike is only an attack when the letters beside it are
  571. ASCII; beside its own script it is simply that script.
  572. """
  573. if any(_native(c) for c in run):
  574. return run
  575. return "".join(CONFUSABLES.get(ord(c), c) for c in run)
  576. def _fold_homoglyphs(token: str) -> str:
  577. return _ALPHA_RUN.sub(
  578. lambda m: _fold_run(m.group(0)) if _is_mixed_run(m.group(0)) else m.group(0),
  579. token,
  580. )
  581. def _is_mixed_script_confusable(token: str) -> bool:
  582. return any(_is_mixed_run(m.group(0)) for m in _ALPHA_RUN.finditer(token))
  583. def detect_homoglyphs(text: str) -> list[str]:
  584. hits: list[str] = []
  585. for tok in re.findall(r"\S+", text):
  586. if _is_mixed_script_confusable(tok):
  587. hits.append(tok)
  588. return hits
  589. def fold_mixed_script(text: str) -> str:
  590. return re.sub(
  591. r"\S+",
  592. lambda m: _fold_homoglyphs(m.group(0))
  593. if _is_mixed_script_confusable(m.group(0))
  594. else m.group(0),
  595. text,
  596. )
  597. _INWORD_ZERO_WIDTH = re.compile(r"(?<=[A-Za-z])[​‌‍⁠]+(?=[A-Za-z])")
  598. def defuse_evasion(text: str) -> tuple:
  599. hits = _INWORD_ZERO_WIDTH.findall(text)
  600. return fold_mixed_script(_INWORD_ZERO_WIDTH.sub("", text)), sum(len(h) for h in hits)
  601. _QUOTE_MARK = re.compile(r"^[ \t]{0,3}>[ \t]?")
  602. def strip_quoted_spans(text: str) -> str:
  603. lines = text.split("\n")
  604. quoted = [bool(_QUOTE_MARK.match(ln)) for ln in lines]
  605. # A document that is mostly quotation is the deliverable, not evidence inside one.
  606. if 2 * sum(quoted) > sum(1 for ln in lines if ln.strip()):
  607. return text
  608. out = []
  609. for i, ln in enumerate(lines):
  610. run = quoted[i]
  611. out.append(" " * len(ln) if run else ln)
  612. return "\n".join(out)
  613. TRANSITION_OPENERS = re.compile(
  614. r"(?:^|\n|[.!?]\s+)(Additionally|Furthermore|Moreover|However|Nevertheless|"
  615. r"Consequently|Subsequently|Notably)\b",
  616. re.MULTILINE,
  617. )
  618. CONTRACTABLE = re.compile(
  619. r"\b(?:it is|do not|does not|did not|is not|are not|was not|were not|"
  620. r"will not|would not|could not|should not|cannot|i am|you are|"
  621. r"we are|they are|i have|you have|we have|they have|i will|you will)\b",
  622. re.IGNORECASE,
  623. )
  624. CONTRACTIONS = re.compile(
  625. r"\b(?:it'?s|don'?t|doesn'?t|didn'?t|isn'?t|aren'?t|wasn'?t|weren'?t|"
  626. r"won'?t|wouldn'?t|couldn'?t|shouldn'?t|can'?t|i'?m|you'?re|we'?re|"
  627. r"they'?re|i'?ve|you'?ve|we'?ve|they'?ve|i'?ll|you'?ll)\b",
  628. re.IGNORECASE,
  629. )
  630. FIRST_PERSON = re.compile(
  631. r"\b(?:i|me|my|mine|we|us|our|ours)\b",
  632. re.IGNORECASE,
  633. )
  634. ABBREV = {
  635. "mr.", "mrs.", "ms.", "dr.", "prof.", "vs.", "etc.", "e.g.", "i.e.",
  636. "u.s.", "u.k.", "fig.", "no.", "vol.", "pp.", "st.", "ed.", "eds.",
  637. "inc.", "ltd.", "corp.", "co.", "sr.", "jr.", "ave.", "blvd.", "rd.",
  638. }
  639. # ------------------------------------------------------------- text structure
  640. def split_sentences(text: str) -> list[str]:
  641. placeholder = ""
  642. text = text.replace(placeholder, "")
  643. protected = re.sub(r"(\d)\.(\d)", rf"\1{placeholder}\2", text)
  644. protected = re.sub(
  645. r"\b([a-z]{1,2})\.([a-z]{1,2})\.",
  646. lambda m: f"{m.group(1)}{placeholder}{m.group(2)}{placeholder}",
  647. protected,
  648. flags=re.IGNORECASE,
  649. )
  650. out: list[str] = []
  651. buf = ""
  652. for m in re.finditer(r"[^.!?]+(?:[.!?]+|$)", protected):
  653. chunk = m.group(0)
  654. if not chunk.strip():
  655. continue
  656. buf += chunk
  657. tail = chunk.strip().lower()
  658. base = ""
  659. if tail.endswith("."):
  660. toks = tail.split()
  661. if toks:
  662. base = re.sub(r"[^\w.]", "", toks[-1]).rstrip(".")
  663. if base and ((base + ".") in ABBREV or re.fullmatch(r"[a-z]", base)):
  664. buf += " "
  665. continue
  666. out.append(buf.strip())
  667. buf = ""
  668. if buf.strip():
  669. out.append(buf.strip())
  670. return [s.replace(placeholder, ".") for s in out]
  671. def count_words(s: str) -> int:
  672. return len(re.findall(r"\b[\w'-]+\b", s))
  673. def first_two_words(s: str) -> str:
  674. words = re.findall(r"\b[\w'-]+\b", s.lower())
  675. return " ".join(words[:2]) if len(words) >= 2 else ""
  676. def split_paragraphs(text: str) -> list[str]:
  677. return [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
  678. # A model that loses the thread repeats itself. Not the way a person repeats a
  679. # term of art, which is what academic prose does constantly, but by emitting the
  680. # same span until the window closes: "is caused in part by - at least in the US -
  681. # is caused in part by in part by a shortage of a". Nothing else in the catalogue
  682. # looks for it, and it is the only tell present in small-model continuations,
  683. # which carry none of the marketing vocabulary the rest of the rules are built
  684. # around.
  685. #
  686. # The measure is the share of six-gram positions whose six-gram occurs more than
  687. # once. It has to clear a high bar because legitimate repetition is common:
  688. # 15.1% of PubMed abstracts and 10.5% of textbook passages exceed 2%, and no
  689. # human corpus measured reaches 0.13 at the 99th percentile. Degeneration is an
  690. # order of magnitude worse, with machine SQuAD and writing-prompt continuations
  691. # at 0.62 and 0.63 at the 90th. At 0.20 the rule fires on 0.0% of PubMed,
  692. # textbook and every MAGE human arm, on 0.2% of arXiv abstracts, and on 18.6% of
  693. # machine SQuAD and 32.3% of machine writing prompts.
  694. _REP_WORD = re.compile(r"[A-Za-z0-9']+")
  695. _REP_N = 6
  696. _REP_MIN_WORDS = _REP_N * 3
  697. _REP_THRESHOLD = 0.20
  698. def detect_degenerate_repetition(text: str) -> dict:
  699. """Share of six-gram positions whose six-gram appears more than once."""
  700. words = [w.lower() for w in _REP_WORD.findall(text)]
  701. if len(words) < _REP_MIN_WORDS:
  702. return {"ratio": 0.0, "degenerate": False, "worst": None}
  703. grams = [tuple(words[i:i + _REP_N]) for i in range(len(words) - _REP_N + 1)]
  704. counts = collections.Counter(grams)
  705. dup = sum(c for g, c in counts.items() if c > 1)
  706. ratio = dup / float(len(grams))
  707. worst = max(counts.items(), key=lambda kv: kv[1]) if counts else None
  708. return {
  709. "ratio": round(ratio, 4),
  710. "degenerate": ratio > _REP_THRESHOLD,
  711. "worst": (" ".join(worst[0]), worst[1]) if worst and worst[1] > 1 else None,
  712. }
  713. def detect_opener_repetition(text: str) -> list[dict]:
  714. hits: list[dict] = []
  715. for p_i, para in enumerate(split_paragraphs(text)):
  716. sentences = split_sentences(para)
  717. if len(sentences) < 3:
  718. continue
  719. openers = [first_two_words(s) for s in sentences if s]
  720. counter: dict[str, int] = {}
  721. for o in openers:
  722. if not o:
  723. continue
  724. counter[o] = counter.get(o, 0) + 1
  725. for opener, count in counter.items():
  726. if count >= 3:
  727. hits.append({"paragraph": p_i + 1, "opener": opener, "repeats": count})
  728. return hits
  729. def detect_burstiness(text: str) -> dict:
  730. sentences = split_sentences(text)
  731. if len(sentences) < 3:
  732. return {"sentences": len(sentences), "cv": None, "monotonous": False, "has_short": True}
  733. lengths = [count_words(s) for s in sentences if count_words(s) > 0]
  734. if len(lengths) < 3:
  735. return {"sentences": len(sentences), "cv": None, "monotonous": False, "has_short": True}
  736. mean = statistics.mean(lengths)
  737. stdev = statistics.pstdev(lengths)
  738. cv = stdev / mean if mean else 0
  739. has_short = any(n < 10 for n in lengths)
  740. return {
  741. "sentences": len(sentences),
  742. "cv": round(cv, 3),
  743. "monotonous": cv < 0.35,
  744. "has_short": has_short,
  745. }
  746. def detect_paragraph_monotony(text: str) -> dict:
  747. paragraphs = split_paragraphs(text)
  748. if len(paragraphs) < 4:
  749. return {"paragraphs": len(paragraphs), "cv": None, "monotonous": False}
  750. lengths = [count_words(p) for p in paragraphs if count_words(p) > 0]
  751. if len(lengths) < 4:
  752. return {"paragraphs": len(paragraphs), "cv": None, "monotonous": False}
  753. mean = statistics.mean(lengths)
  754. stdev = statistics.pstdev(lengths)
  755. cv = stdev / mean if mean else 0
  756. return {
  757. "paragraphs": len(paragraphs),
  758. "cv": round(cv, 3),
  759. "monotonous": cv < 0.4,
  760. }
  761. def detect_hedge_stacking(text: str) -> list[dict]:
  762. hits: list[dict] = []
  763. pattern = re.compile(HEDGES)
  764. for i, sent in enumerate(split_sentences(text)):
  765. matches = pattern.findall(sent)
  766. if len(matches) >= 2:
  767. hits.append({
  768. "sentence_idx": i,
  769. "hedges": matches,
  770. "sample": sent[:120] + ("..." if len(sent) > 120 else ""),
  771. })
  772. return hits
  773. def detect_em_dash_overuse(text: str) -> list[dict]:
  774. hits: list[dict] = []
  775. for p_i, para in enumerate(split_paragraphs(text)):
  776. count = para.count("—")
  777. if count >= 2:
  778. hits.append({"paragraph": p_i + 1, "em_dash_count": count})
  779. return hits
  780. def _plural_forms(term: str) -> list[str]:
  781. parts = term.split(" ")
  782. last = parts[-1]
  783. if last.endswith("is") and len(last) > 2:
  784. plural_last = last[:-2] + "es"
  785. elif last.endswith("y") and len(last) > 1 and last[-2] not in "aeiou":
  786. plural_last = last[:-1] + "ies"
  787. elif last.endswith(("s", "ss", "sh", "ch", "x", "z")):
  788. plural_last = last + "es"
  789. else:
  790. plural_last = last + "s"
  791. plural = " ".join(parts[:-1] + [plural_last])
  792. return [term] if plural == term else [term, plural]
  793. def detect_synonym_cycling(text: str) -> list[list[str]]:
  794. CLUSTERS = [
  795. ["protagonist", "main character", "central figure", "hero", "heroine"],
  796. ["company", "firm", "organization", "enterprise", "business"],
  797. ["author", "writer", "novelist", "scribe"],
  798. ["report", "study", "analysis", "investigation", "examination"],
  799. ["technology", "tool", "system", "platform", "solution"],
  800. ["technique", "method", "approach", "strategy", "tactic"],
  801. ["challenge", "obstacle", "difficulty", "hurdle", "barrier"],
  802. ["benefit", "advantage", "upside", "strength"],
  803. ]
  804. lower = text.lower()
  805. hits: list[list[str]] = []
  806. for cluster in CLUSTERS:
  807. used: list[str] = []
  808. for term in cluster:
  809. forms = _plural_forms(term)
  810. term_pattern = r"\b(?:" + "|".join(re.escape(f) for f in forms) + r")\b"
  811. if re.search(term_pattern, lower):
  812. used.append(term)
  813. if len(used) >= 3:
  814. hits.append(used)
  815. return hits
  816. _CTA_VERBS = {
  817. "install", "join", "grab", "download", "try", "discover", "experience",
  818. "escape", "transform", "reclaim", "unlock", "elevate", "boost", "say",
  819. "imagine", "explore", "meet", "introducing", "upgrade", "claim", "unleash",
  820. "supercharge", "level", "stop", "start",
  821. }
  822. def detect_imperative_cta(text: str) -> list[str]:
  823. hits = []
  824. for s in split_sentences(text):
  825. w = re.match(r"[^\w]*([A-Za-z][\w'-]*)", s)
  826. if w and w.group(1).lower() in _CTA_VERBS:
  827. hits.append(s.strip()[:60])
  828. return hits if len(hits) >= 3 else []
  829. def detect_fragmented_headers(text: str) -> list[dict]:
  830. lines = text.split("\n")
  831. hits: list[dict] = []
  832. for i, line in enumerate(lines):
  833. m = re.match(r"^(#{1,6})\s+(.+)$", line)
  834. if not m:
  835. continue
  836. heading_words = set(re.findall(r"\w+", m.group(2).lower()))
  837. j = i + 1
  838. while j < len(lines) and not lines[j].strip():
  839. j += 1
  840. if j >= len(lines):
  841. continue
  842. next_line = lines[j].strip()
  843. if next_line.startswith("#"):
  844. continue
  845. if re.match(r"^(?:[-*+•]|\d+\.|\|)\s", next_line):
  846. continue
  847. if count_words(next_line) > 12:
  848. continue
  849. next_words = set(re.findall(r"\w+", next_line.lower()))
  850. if heading_words and len(heading_words & next_words) >= 1 and count_words(next_line) < 8:
  851. hits.append({
  852. "heading": m.group(2),
  853. "filler_line": next_line,
  854. "line_no": i + 1,
  855. })
  856. return hits
  857. def detect_title_case_headings(text: str) -> list[dict]:
  858. hits: list[dict] = []
  859. minor = {
  860. "a", "an", "the", "and", "or", "but", "of", "in", "on", "at",
  861. "to", "for", "with", "by", "as", "is", "it",
  862. }
  863. for m in TITLE_CASE_HEADING.finditer(text):
  864. heading = m.group(2).strip()
  865. words = heading.split()
  866. if len(words) < 3:
  867. continue
  868. if re.match(r"^[\d\W]", heading):
  869. continue
  870. major_words = [w for w in words if w.lower() not in minor]
  871. if len(major_words) < 2:
  872. continue
  873. all_cap = all(re.match(r"^[A-Z]", w) for w in major_words)
  874. if all_cap and not heading.isupper():
  875. hits.append({"heading": heading})
  876. return hits
  877. def detect_transition_clusters(text: str) -> list[dict]:
  878. hits: list[dict] = []
  879. for p_i, para in enumerate(split_paragraphs(text)):
  880. matches = TRANSITION_OPENERS.findall(para)
  881. if len(matches) >= 2:
  882. hits.append({
  883. "paragraph": p_i + 1,
  884. "transitions": matches,
  885. "count": len(matches),
  886. })
  887. return hits
  888. _ADJECTIVE_SUFFIXES = (
  889. "ive", "ous", "able", "ible", "ic", "ical", "ant", "ent", "ful",
  890. "less", "al", "ial", "ary", "ory", "ish", "ative", "itive",
  891. )
  892. _ADJECTIVE_AI_TELLS = {
  893. "innovative", "transformative", "groundbreaking", "robust",
  894. "seamless", "dynamic", "holistic", "scalable", "agile", "elegant",
  895. "vibrant", "powerful", "comprehensive", "cutting-edge",
  896. "forward-thinking", "next-generation", "world-class",
  897. }
  898. def detect_stacked_adjectives(text: str) -> list[str]:
  899. pattern = re.compile(
  900. r"\b(?:[a-z]+(?:,[ \t]+|[ \t]+and[ \t]+)){3,10}[a-z]+[ \t]+[a-z]+\b",
  901. re.IGNORECASE,
  902. )
  903. hits: list[str] = []
  904. for m in pattern.finditer(text):
  905. snippet = m.group(0)
  906. if not ("," in snippet and len(snippet.split()) <= 12):
  907. continue
  908. candidate_words = re.findall(r"[A-Za-z][A-Za-z'-]*", snippet)[:-1]
  909. has_adj_evidence = any(
  910. w.lower() in _ADJECTIVE_AI_TELLS
  911. or any(w.lower().endswith(s) for s in _ADJECTIVE_SUFFIXES)
  912. for w in candidate_words
  913. )
  914. if has_adj_evidence:
  915. hits.append(snippet)
  916. return hits[:MAX_SAMPLES * 2]
  917. # ------------------------------------------------------- structural detectors
  918. def detect_decorative_rules(text: str) -> dict:
  919. """Standalone horizontal rules used as section furniture. Markdown post-training
  920. leaves them behind; human markdown uses one or none."""
  921. rules = re.findall(r"(?m)^[ \t]{0,3}(?:-{3,}|\*{3,}|_{3,})[ \t]*$", text)
  922. lines = len(text.splitlines())
  923. if len(rules) < 3 or lines < 25:
  924. return {}
  925. return {"count": len(rules),
  926. "samples": ["%d standalone horizontal rules in %d lines" % (len(rules), lines)]}
  927. def detect_mid_essay_bullets(text: str) -> dict:
  928. structured = bool(re.search(r"^#{1,6}\s+\S", text, re.M))
  929. paragraphs = split_paragraphs(text)
  930. bullet_count = 0
  931. prose_count = 0
  932. for para in paragraphs:
  933. lines = [ln for ln in para.split("\n") if ln.strip()]
  934. if all(re.match(r"^\s*(?:[-*•]|\d+\.)\s+", ln) for ln in lines):
  935. bullet_count += 1
  936. else:
  937. prose_count += 1
  938. return {
  939. "bullet_paragraphs": bullet_count,
  940. "prose_paragraphs": prose_count,
  941. "structured": structured,
  942. "mixed": (not structured) and bullet_count >= 1 and prose_count >= 2,
  943. }
  944. def _codepoint_breakdown(matches: list[str]) -> dict[str, int]:
  945. counts: dict[str, int] = {}
  946. for c in matches:
  947. key = f"U+{ord(c):04X}"
  948. counts[key] = counts.get(key, 0) + 1
  949. return counts
  950. def _labelled_samples(counts: dict[str, int]) -> list[str]:
  951. out: list[str] = []
  952. for key, n in counts.items():
  953. cp = int(key[2:], 16)
  954. name = _char_name(cp)
  955. out.append(f"{key} {name}: {n}" if name else f"{key}: {n}")
  956. return out
  957. def detect_invisible_chars(text: str) -> dict:
  958. """Report only what the scrub would remove.
  959. Detection and cleaning have to agree. When they did not, a variation
  960. selector doing its job inside an emoji was counted as a watermark, and
  961. every document with an emoji gained points it had not earned.
  962. """
  963. # _SUSPECT finds the handful of non-ASCII positions in C, so the context
  964. # test runs on those and not on every character. Walking the whole string
  965. # in Python instead cost 8 seconds on a 540 KB document and tripped the
  966. # guard's own timeout.
  967. rtl_doc = _has_rtl(text)
  968. n = len(text)
  969. matches = []
  970. flag_tags = _flag_tags(text)
  971. for m in _SUSPECT.finditer(text):
  972. ch, i = m.group(0), m.start()
  973. if ch in "\t\n\r" or not _is_invisible(ch):
  974. continue
  975. prev = text[i - 1] if i else ""
  976. nxt = text[i + 1] if i + 1 < n else ""
  977. if not _keeps_function(ch, prev, nxt, rtl_doc, i in flag_tags):
  978. matches.append(ch)
  979. if not matches:
  980. return {"count": 0, "chars": {}}
  981. return {"count": len(matches), "chars": _codepoint_breakdown(matches)}
  982. def detect_nonstandard_spaces(text: str) -> dict:
  983. n = len(text)
  984. matches = []
  985. for m in _SUSPECT.finditer(text):
  986. ch, i = m.group(0), m.start()
  987. if not _is_nonstandard_space(ch):
  988. continue
  989. prev = text[i - 1] if i else ""
  990. nxt = text[i + 1] if i + 1 < n else ""
  991. if not _space_holds(ch, prev, nxt):
  992. matches.append(ch)
  993. if not matches:
  994. return {"count": 0, "chars": {}}
  995. return {"count": len(matches), "chars": _codepoint_breakdown(matches)}
  996. # A no-break space earns its place by holding something together: a number to
  997. # its unit, or French punctuation to the word before it. Between two ordinary
  998. # English words it is debris, and that is the common case, so it is normalised
  999. # there rather than everywhere.
  1000. _NOBREAK = {0x00A0, 0x202F}
  1001. _FRENCH_AFTER = set(":;!?»›%€")
  1002. _FRENCH_BEFORE = set("«‹")
  1003. def _space_holds(ch: str, prev: str, nxt: str) -> bool:
  1004. if ord(ch) not in _NOBREAK:
  1005. return False
  1006. if prev.isdigit() or nxt.isdigit():
  1007. return True
  1008. return nxt in _FRENCH_AFTER or prev in _FRENCH_BEFORE
  1009. def clean_text(text: str, syntax: str = "markdown") -> str:
  1010. # Hide the same code spans the prose detector skips. Tokens must be absent
  1011. # from the input, and restoration happens only after every cleanup pass.
  1012. code = []
  1013. prefix = "SLOPTRIMCODE"
  1014. while prefix in text:
  1015. prefix += "X"
  1016. def protect(span):
  1017. code.append(span)
  1018. return f"|{prefix}{len(code) - 1}END|"
  1019. text = (_map_latex if syntax == "latex" else _map_code)(text, protect)
  1020. out: list = []
  1021. rtl_doc = _has_rtl(text)
  1022. n = len(text)
  1023. flag_tags = _flag_tags(text)
  1024. for i, ch in enumerate(text):
  1025. if ch in "\t\n\r":
  1026. out.append(ch)
  1027. continue
  1028. cat = unicodedata.category(ch)
  1029. if cat in ("Zl", "Zp"):
  1030. out.append("\n")
  1031. continue
  1032. if _is_invisible(ch):
  1033. prev = text[i - 1] if i else ""
  1034. nxt = text[i + 1] if i + 1 < n else ""
  1035. if _keeps_function(ch, prev, nxt, rtl_doc, i in flag_tags):
  1036. out.append(ch)
  1037. continue
  1038. if cat == "Zs":
  1039. prev = text[i - 1] if i else ""
  1040. nxt = text[i + 1] if i + 1 < n else ""
  1041. out.append(ch if _space_holds(ch, prev, nxt) else " ")
  1042. continue
  1043. out.append(ch)
  1044. cleaned = "".join(out)
  1045. cleaned = fold_mixed_script(cleaned)
  1046. # Every run of whitespace at the end of a line goes, including the two spaces that
  1047. # write a Markdown hard break. Carving the break out was tried and reverted: it made
  1048. # the rule depend on how the file renders, which is not knowable from the text alone,
  1049. # and it spared the same two spaces in .txt and .rst where they are only debris.
  1050. cleaned = re.sub(r"(?<![ \t])[ \t]+(?=\r?\n)", "", cleaned)
  1051. # Blank leading lines go; the first line's own indentation stays, because
  1052. # four spaces in Markdown is a code block, not stray whitespace.
  1053. cleaned = re.sub(r"\A(?:[ \t]*\r?\n)+", "", cleaned)
  1054. cleaned = cleaned.rstrip(" \t\r\n")
  1055. return re.sub(r"\|" + re.escape(prefix) + r"(\d+)END\|",
  1056. lambda m: code[int(m[1])], cleaned)
  1057. def detect_trailing_whitespace(text: str) -> dict:
  1058. eof_run = text[len(text.rstrip(" \t\r\n")):]
  1059. trailing_newlines = eof_run.count("\n")
  1060. trailing_blanks = eof_run.count(" ") + eof_run.count("\t")
  1061. lead = re.match(r"[ \t\r\n]+", text)
  1062. leading_ws = len(lead.group(0)) if lead else 0
  1063. eol_spaces = len(re.findall(r"(?<![ \t])[ \t]+\r?\n", text))
  1064. artifact = (
  1065. trailing_newlines >= 2
  1066. or trailing_blanks > 0
  1067. or eol_spaces > 0
  1068. or leading_ws > 0
  1069. )
  1070. return {
  1071. "trailing_newlines": trailing_newlines,
  1072. "trailing_blanks": trailing_blanks,
  1073. "leading_whitespace": leading_ws,
  1074. "eol_trailing_spaces": eol_spaces,
  1075. "artifact": artifact,
  1076. }
  1077. def detect_hyphen_ranges(text: str) -> list[str]:
  1078. hits: list[str] = []
  1079. for m in HYPHEN_RANGE.finditer(text):
  1080. a, b = m.group(1), m.group(2)
  1081. try:
  1082. ai, bi = int(a), int(b)
  1083. if (
  1084. _YEAR_MIN <= ai <= _YEAR_MAX
  1085. and _YEAR_MIN <= bi <= _YEAR_MAX
  1086. and bi > ai
  1087. and bi - ai < 500
  1088. ):
  1089. hits.append(m.group(0))
  1090. except ValueError:
  1091. continue
  1092. return hits
  1093. def detect_punctuation_diversity(text: str) -> dict:
  1094. words = count_words(text)
  1095. if words < 500:
  1096. return {"words": words, "checked": False}
  1097. semicolons = text.count(";")
  1098. open_parens = text.count("(")
  1099. close_parens = text.count(")")
  1100. parentheticals = min(open_parens, close_parens)
  1101. per_1k_semi = round(semicolons / words * 1000, 2)
  1102. per_1k_paren = round(parentheticals / words * 1000, 2)
  1103. return {
  1104. "words": words,
  1105. "checked": True,
  1106. "semicolons": semicolons,
  1107. "parentheticals": parentheticals,
  1108. "semicolons_per_1k": per_1k_semi,
  1109. "parentheticals_per_1k": per_1k_paren,
  1110. "underused": semicolons == 0 and parentheticals < 2,
  1111. }
  1112. def _count_syllables(word: str) -> int:
  1113. word = word.lower()
  1114. count = 0
  1115. prev_vowel = False
  1116. for c in word:
  1117. is_vowel = c in "aeiouy"
  1118. if is_vowel and not prev_vowel:
  1119. count += 1
  1120. prev_vowel = is_vowel
  1121. if word.endswith("e") and count > 1:
  1122. count -= 1
  1123. return max(count, 1)
  1124. def _flesch_kincaid_grade(text: str) -> float | None:
  1125. sentences = split_sentences(text)
  1126. words = re.findall(r"[A-Za-z]+", text)
  1127. if not sentences or not words:
  1128. return None
  1129. syllables = sum(_count_syllables(w) for w in words)
  1130. words_per_sentence = len(words) / len(sentences)
  1131. syllables_per_word = syllables / len(words)
  1132. return 0.39 * words_per_sentence + 11.8 * syllables_per_word - 15.59
  1133. def detect_readability_consistency(text: str) -> dict:
  1134. paragraphs = [p for p in split_paragraphs(text) if count_words(p) >= 10]
  1135. if len(paragraphs) < 3:
  1136. return {"paragraphs": len(paragraphs), "fk_stdev": None, "uniform": False}
  1137. grades = [g for g in (_flesch_kincaid_grade(p) for p in paragraphs) if g is not None]
  1138. if len(grades) < 3:
  1139. return {"paragraphs": len(grades), "fk_stdev": None, "uniform": False}
  1140. sd = statistics.pstdev(grades)
  1141. return {"paragraphs": len(grades), "fk_stdev": round(sd, 2), "uniform": sd < 1.5}
  1142. def detect_cadence_alternation(text: str) -> dict:
  1143. sentences = split_sentences(text)
  1144. lengths = [count_words(s) for s in sentences if count_words(s) > 0]
  1145. if len(lengths) < 6:
  1146. return {"checked": False, "zigzag": None, "mechanical": False}
  1147. diffs = [b - a for a, b in zip(lengths, lengths[1:])]
  1148. signs = [1 if d > 0 else -1 if d < 0 else 0 for d in diffs]
  1149. pairs = list(zip(signs, signs[1:]))
  1150. if not pairs:
  1151. return {"checked": False, "zigzag": None, "mechanical": False}
  1152. flips = sum(1 for a, b in pairs if a and b and a == -b)
  1153. zigzag = flips / len(pairs)
  1154. mean = statistics.mean(lengths)
  1155. cv = statistics.pstdev(lengths) / mean if mean else 0.0
  1156. return {
  1157. "checked": True,
  1158. "zigzag": round(zigzag, 3),
  1159. "mechanical": zigzag >= 0.85 and cv >= 0.25,
  1160. "sentences": len(lengths),
  1161. }
  1162. def detect_passive_voice(text: str) -> dict:
  1163. sentences = split_sentences(text)
  1164. if not sentences:
  1165. return {"count": 0, "ratio": None}
  1166. count = 0
  1167. for s in sentences:
  1168. m = PASSIVE_VOICE.search(s)
  1169. if m and m.group(1).lower() not in _PASSIVE_STOP:
  1170. count += 1
  1171. return {"count": count, "ratio": round(count / len(sentences), 3)}
  1172. # -------------------------------------------------------------------- scoring
  1173. _SCORE_DECISIVE = {
  1174. "62_invisible_chars", "63_placeholder_text", "64_chatbot_ref_markup",
  1175. "65_ai_tracking_params", "66_homoglyphs", "13_article_title_noun",
  1176. "47_chatbot_artifacts", "50_cutoff_disclaimer", "48_sycophantic",
  1177. }
  1178. _SCORE_STRONG = {
  1179. "16_ing_tail", "20_empty_pivot_phrase", "21_outcome_speculation",
  1180. "24_self_thoroughness", "18_not_x_just_y", "27_compulsive_intro",
  1181. "30_diff_anchored", "5_simple_yet",
  1182. "69_canonical_slop", "71_degenerate_repetition",
  1183. }
  1184. # Keys listed here that are also demoted below never reach the weight, because the
  1185. # scoring loop tests the demoted sets first. Only rules that can score belong here.
  1186. _SCORE_SOFT = {
  1187. "37_transition_cluster", "57_emojis", "2_model_dialect",
  1188. "23_editorial_interjection",
  1189. }
  1190. # 17_negative_parallelism is report-only, declared once, below.
  1191. _SCORE_STYLE_ONLY = {"60_curly_quotes", "61_hyphen_for_en_dash"}
  1192. _SCORE_REPORT_ONLY = {
  1193. "4_hyphen_cliche", "19_filler_phrases", "22_authority_trope",
  1194. "41_mechanical_alternation", "55_contraction_absence",
  1195. "68_trailing_whitespace",
  1196. # Openers and hedge stacking are ordinary features of formal prose. Reporting
  1197. # them remains useful editing advice; scoring them would manufacture false
  1198. # positives from register rather than identify a prose-quality defect.
  1199. "42_opener_repetition", "54_hedge_stacking",
  1200. # Em dashes, negative parallelism, and sparse semicolon/parenthesis use also
  1201. # vary with house style and register. Keep the detections visible, but do not
  1202. # turn those typography and rhetoric choices into score weight.
  1203. "58_em_dash_overuse", "17_negative_parallelism", "46_punctuation_underuse",
  1204. }
  1205. _SCORE_RHYTHM = {"40_sentence_monotony", "55_contraction_absence", "41_mechanical_alternation"}
  1206. _SCORE_MIN_WORDS = 120
  1207. _ESCALATOR_FAMILIES = set(range(1, 40)) | {47, 48, 49, 50, 51, 54, 69}
  1208. _RHYTHM_CAP = 18.0
  1209. _CONTENT_CAP = 48.0
  1210. _VOCAB_CAP = 26.0
  1211. _VOCAB_FLOOR = 12.0
  1212. def _band(score: float) -> str:
  1213. if score <= 20:
  1214. return "clean"
  1215. if score <= 40:
  1216. return "light tells"
  1217. if score <= 60:
  1218. return "mixed"
  1219. if score <= 80:
  1220. return "heavy tells"
  1221. return "pervasive tells"
  1222. def _decisive_families(result: dict, text: str) -> set:
  1223. keys = {k for k in result if k in _SCORE_DECISIVE}
  1224. if "62_invisible_chars" in keys and result["62_invisible_chars"]["count"] < 3:
  1225. keys.discard("62_invisible_chars")
  1226. if "63_placeholder_text" in keys and all(
  1227. re.fullmatch(r"x{3,}", m, re.IGNORECASE)
  1228. for m in regex_count(text, PLACEHOLDER)):
  1229. keys.discard("63_placeholder_text")
  1230. return {k.split("_", 1)[0] for k in keys}
  1231. def ai_tell_score(result: dict, text: str, burst: dict, para_mono: dict, punct: dict,
  1232. readability: dict | None = None) -> dict:
  1233. words = max(count_words(text), 1)
  1234. eff = max(words, _SCORE_MIN_WORDS)
  1235. diversity = 0.0
  1236. content_load = 0.0
  1237. rhythm_flags = 0
  1238. # Density counts every hit; diversity counts each catalogue family once.
  1239. counted_families = set()
  1240. for key, val in result.items():
  1241. if key.startswith("_") or not isinstance(val, dict):
  1242. continue
  1243. if key in _SCORE_STYLE_ONLY or key in _SCORE_REPORT_ONLY or key == "1_ai_vocabulary":
  1244. continue
  1245. if key in _SCORE_RHYTHM:
  1246. rhythm_flags += 1
  1247. continue
  1248. count = min(val.get("count", 1), 5)
  1249. weight = 3.0 if key in _SCORE_DECISIVE else 1.0 if key in _SCORE_SOFT else 1.6
  1250. content_load += weight * count
  1251. family = key.split("_", 1)[0]
  1252. if family in counted_families:
  1253. continue
  1254. counted_families.add(family)
  1255. diversity += (
  1256. 16.0 if key in _SCORE_DECISIVE
  1257. else 10.0 if key in _SCORE_STRONG
  1258. else 2.5 if key in _SCORE_SOFT
  1259. else 5.0
  1260. )
  1261. density_bonus = min(10.0, content_load / eff * 1000 * 1.0)
  1262. content_term = min(_CONTENT_CAP, diversity + density_bonus)
  1263. cv = burst.get("cv")
  1264. burst_term = 0.0 if cv is None else max(0.0, min(1.0, (0.45 - cv) / 0.45)) * 10.0
  1265. pcv = para_mono.get("cv")
  1266. mono_term = 0.0 if pcv is None else max(0.0, min(1.0, (0.4 - pcv) / 0.4)) * 8.0
  1267. read_term = 3.0 if (readability or {}).get("uniform") else 0.0
  1268. rhythm_term = min(_RHYTHM_CAP, rhythm_flags * 4.0 + burst_term + mono_term + read_term)
  1269. vocab_count = result.get("1_ai_vocabulary", {}).get("count", 0)
  1270. vocab_rate = vocab_count / eff * 1000
  1271. vocab_term = min(_VOCAB_CAP, max(0.0, vocab_rate - _VOCAB_FLOOR) * 0.45)
  1272. # This fires on exactly the condition that raises 46_punctuation_underuse.
  1273. # When that rule is report-only the main loop skips it, and this term used to
  1274. # deliver its five points anyway. Demoting a rule has to close every route its
  1275. # evidence takes to the score, not just the obvious one.
  1276. punct_term = (5.0 if (punct.get("checked") and punct.get("underused")
  1277. and "46_punctuation_underuse" not in _SCORE_REPORT_ONLY)
  1278. else 0.0)
  1279. score = round(min(100.0, content_term + rhythm_term + vocab_term + punct_term))
  1280. scored = {int(k.split("_", 1)[0]) for k in result
  1281. if not k.startswith("_") and k not in _SCORE_REPORT_ONLY
  1282. and k not in _SCORE_STYLE_ONLY and k.split("_", 1)[0].isdigit()}
  1283. decisive_hits = len(_decisive_families(result, text))
  1284. thin = score_confidence(words, len(scored))["confidence"] in ("none", "low")
  1285. if decisive_hits >= 2:
  1286. # Two distinct decisive families corroborate each other.
  1287. score = max(score, 65)
  1288. elif decisive_hits == 1 and not thin:
  1289. # A lone marker needs the evidence behind it.
  1290. score = max(score, 45)
  1291. # _SCORE_STYLE_ONLY belongs here beside _SCORE_REPORT_ONLY. Without it,
  1292. # 60_curly_quotes counted toward an escalation floor while being excluded
  1293. # from the score itself, and it fires on 68.7% of human documents.
  1294. fams = {int(k.split("_", 1)[0]) for k in result
  1295. if not k.startswith("_") and k not in _SCORE_REPORT_ONLY
  1296. and k not in _SCORE_STYLE_ONLY
  1297. and k.split("_", 1)[0].isdigit()}
  1298. if words < _SCORE_MIN_WORDS:
  1299. content_fams = fams & _ESCALATOR_FAMILIES
  1300. if len(content_fams) >= 3:
  1301. score = max(score, 45)
  1302. elif len(content_fams) >= 2:
  1303. score = max(score, 25)
  1304. elif words >= 400:
  1305. if len(fams) >= 6:
  1306. score = max(score, 45)
  1307. elif len(fams) >= 5:
  1308. score = max(score, 25)
  1309. return {"score": score, "band": _band(score)}
  1310. def detect_contraction_absence(text: str) -> dict:
  1311. first_person = len(FIRST_PERSON.findall(text))
  1312. words = max(count_words(text), 1)
  1313. fp_ratio = first_person / words
  1314. if fp_ratio < 0.02:
  1315. return {"opinion_mode": False, "ratio": round(fp_ratio, 3)}
  1316. contractable = len(CONTRACTABLE.findall(text))
  1317. contractions = len(CONTRACTIONS.findall(text))
  1318. total = contractable + contractions
  1319. if total == 0:
  1320. return {"opinion_mode": True, "ratio": round(fp_ratio, 3), "absent": False}
  1321. absent_ratio = contractable / total
  1322. return {
  1323. "opinion_mode": True,
  1324. "first_person_ratio": round(fp_ratio, 3),
  1325. "contractable_forms": contractable,
  1326. "contractions": contractions,
  1327. "absence_ratio": round(absent_ratio, 3),
  1328. "absent": absent_ratio > 0.7,
  1329. }
  1330. def regex_count(text: str, pattern: str, flags: int = re.IGNORECASE) -> list[str]:
  1331. matches = re.findall(pattern, text, flags)
  1332. out: list[str] = []
  1333. for m in matches:
  1334. if isinstance(m, tuple):
  1335. out.append(next((x for x in m if x), ""))
  1336. else:
  1337. out.append(m)
  1338. return [m for m in out if m]
  1339. def samples(text: str, pattern: str, flags: int = re.IGNORECASE, k: int = MAX_SAMPLES) -> list[str]:
  1340. out: list[str] = []
  1341. for m in re.finditer(pattern, text, flags):
  1342. start = max(0, m.start() - 20)
  1343. end = min(len(text), m.end() + 20)
  1344. snippet = text[start:end].replace("\n", " ").strip()
  1345. out.append(snippet)
  1346. if len(out) >= k:
  1347. break
  1348. return out
  1349. _FENCE_OPEN = re.compile(r"^[ ]{0,3}(`{3,}|~{3,})([^\n]*)$")
  1350. _INLINE_CODE = re.compile(r"`[^`\n]+`")
  1351. _TABLE_ROW = re.compile(r"^[ \t]*\|.*\|[ \t]*$", re.M)
  1352. _BADGE_LINE = re.compile(r"^[ \t]*(?:\[!\[[^\]]*\]\([^)]*\)\]\([^)]*\)[ \t]*)+$", re.M)
  1353. _FRONTMATTER = re.compile(r"\A---\r?\n.*?\r?\n---\r?\n", re.S)
  1354. def strip_markdown_furniture(text: str) -> str:
  1355. def blank(m: re.Match) -> str:
  1356. return re.sub(r"[^\n]", " ", m.group(0))
  1357. return _BADGE_LINE.sub(blank, _TABLE_ROW.sub(blank, _FRONTMATTER.sub(blank, text)))
  1358. # ----------------------------------------------------------- pre-scan cleanup
  1359. def _map_code(text: str, replace) -> str:
  1360. """Apply one replacement to each recognized Markdown code span."""
  1361. out = []
  1362. fence = None
  1363. indented = False
  1364. blank_before = True
  1365. for line in text.split("\n"):
  1366. m = _FENCE_OPEN.match(line)
  1367. if fence is not None:
  1368. out.append(replace(line))
  1369. if m and m[1][0] == fence[0] and len(m[1]) >= len(fence) and not m[2].strip():
  1370. fence = None
  1371. elif m and (m[1][0] != '`' or '`' not in m[2]):
  1372. fence = m.group(1)
  1373. out.append(replace(line))
  1374. elif ((blank_before or indented) and line.startswith((' ', '\t'))
  1375. or indented and not line.strip()):
  1376. indented = True
  1377. out.append(replace(line))
  1378. else:
  1379. if line.strip():
  1380. indented = False
  1381. out.append(line)
  1382. blank_before = not line.strip()
  1383. blanked = "\n".join(out)
  1384. return _INLINE_CODE.sub(lambda m: replace(m.group(0)), blanked)
  1385. def strip_code(text: str) -> str:
  1386. return _map_code(text, lambda span: " " * len(span))
  1387. _TEX_TOKEN = re.compile(r"%[^\n]*|\\(?:begin|end)\{[^{}]*\}|"
  1388. r"\\verb\*?[^A-Za-z\s]|\\[A-Za-z@]+\*?|\\[^A-Za-z]|\$\$?")
  1389. _TEX_PARAGRAPH = re.compile(r"\n[ \t]*\n")
  1390. _TEX_LITERAL_ENVS = {"verbatim", "verbatim*", "Verbatim", "lstlisting", "minted",
  1391. "equation", "equation*", "align", "align*", "alignat", "alignat*",
  1392. "gather", "gather*", "multline", "multline*", "displaymath", "math",
  1393. "eqnarray", "eqnarray*", "tikzpicture"}
  1394. _TEX_ARGUMENTS = {"cite", "citep", "citet", "citeauthor", "citeyear", "parencite",
  1395. "textcite", "autocite", "label", "ref", "eqref", "pageref", "url",
  1396. "href", "includegraphics", "input", "include", "bibliography",
  1397. "bibliographystyle", "addbibresource"}
  1398. def _tex_group_end(text: str, start: int) -> int:
  1399. opening = text[start]
  1400. closing = "}" if opening == "{" else "]"
  1401. depth, i = 1, start + 1
  1402. while i < len(text):
  1403. if text[i] == "\\":
  1404. i += 2
  1405. continue
  1406. if text[i] == opening:
  1407. depth += 1
  1408. elif text[i] == closing:
  1409. depth -= 1
  1410. if not depth:
  1411. return i + 1
  1412. i += 1
  1413. return len(text)
  1414. def _map_latex(text: str, replace) -> str:
  1415. """Protect common TeX syntax without executing macros or reading includes.
  1416. ponytail: this is a bounded lexical view, not a TeX interpreter. Custom
  1417. macros/environments need a real parser if their rendered prose must be read.
  1418. """
  1419. out, end, limit, missed = [], 0, -1, {}
  1420. document = re.search(r"(?m)^[ \t]*\\begin\{document\}", text)
  1421. if document:
  1422. end = document.end()
  1423. out.append(replace(text[:end]))
  1424. for m in _TEX_TOKEN.finditer(text):
  1425. if m.start() < end:
  1426. continue
  1427. token, stop = m[0], m.end()
  1428. if token.startswith("\\begin{") and token[7:-1] in _TEX_LITERAL_ENVS:
  1429. closing = "\\end{" + token[7:-1] + "}"
  1430. found = text.find(closing, stop)
  1431. stop = len(text) if found < 0 else found + len(closing)
  1432. elif re.fullmatch(r"\\verb\*?[^A-Za-z\s]", token):
  1433. found = text.find(token[-1], stop)
  1434. stop = len(text) if found < 0 else found + 1
  1435. elif token in ("\\lstinline", "\\lstinline*", "\\mintinline"):
  1436. while stop < len(text) and text[stop].isspace():
  1437. stop += 1
  1438. if stop < len(text) and text[stop] == "[":
  1439. stop = _tex_group_end(text, stop)
  1440. if token == "\\mintinline":
  1441. while stop < len(text) and text[stop].isspace():
  1442. stop += 1
  1443. if stop < len(text) and text[stop] == "{":
  1444. stop = _tex_group_end(text, stop)
  1445. while stop < len(text) and text[stop].isspace():
  1446. stop += 1
  1447. if stop < len(text):
  1448. if text[stop] == "{":
  1449. stop = _tex_group_end(text, stop)
  1450. else:
  1451. found = text.find(text[stop], stop + 1)
  1452. stop = len(text) if found < 0 else found + 1
  1453. elif token in ("$", "$$", "\\(", "\\["):
  1454. closing = {"\\(": "\\)", "\\[": "\\]"}.get(token, token)
  1455. # TeX math cannot span a paragraph break, so an unclosed opener hides only itself.
  1456. if limit < stop:
  1457. para = _TEX_PARAGRAPH.search(text, stop)
  1458. limit = para.start() if para else len(text)
  1459. # A closer missing from this paragraph stays missing for every later opener in it.
  1460. found = -1 if missed.get(closing) == limit else text.find(closing, stop, limit)
  1461. while found >= 0:
  1462. back = found
  1463. while back > stop and text[back - 1] == "\\":
  1464. back -= 1
  1465. if (found - back) % 2 == 0:
  1466. break
  1467. found = text.find(closing, found + len(closing), limit)
  1468. if found < 0:
  1469. missed[closing] = limit
  1470. else:
  1471. stop = found + len(closing)
  1472. elif token == "\\end{document}":
  1473. stop = len(text)
  1474. elif token.startswith("\\"):
  1475. command = token[1:].rstrip("*")
  1476. groups = 2 if command in {"newcommand", "renewcommand", "providecommand", "DeclareRobustCommand"} else 1 if command in _TEX_ARGUMENTS else 0
  1477. while groups:
  1478. start = stop
  1479. while start < len(text) and text[start].isspace():
  1480. start += 1
  1481. if start >= len(text) or text[start] not in "[{":
  1482. break
  1483. stop = _tex_group_end(text, start)
  1484. if text[start] == "{":
  1485. groups -= 1
  1486. out.extend((text[end:m.start()], replace(text[m.start():stop])))
  1487. end = stop
  1488. out.append(text[end:])
  1489. return "".join(out)
  1490. _SCAN_CAP = 262144
  1491. def _reading_order(name: str) -> tuple:
  1492. """Sort chapter10 after chapter2. A plain string sort puts a book's tenth chapter
  1493. before its second, which shuffles the document the rhythm rules then read."""
  1494. parts = re.split(r"(\d+)", name)
  1495. return tuple((1, int(p)) if p.isdigit() else (0, p) for p in parts)
  1496. # ------------------------------------------------------------------ main scan
  1497. def scan(raw_text: str, syntax: str = "markdown") -> dict:
  1498. result: dict = {}
  1499. # One window for every layer.
  1500. capped = raw_text[:_SCAN_CAP]
  1501. # Non-whitespace markers avoid turning hidden code into stray whitespace.
  1502. # Keep line boundaries, but exclude code from character repair advice too.
  1503. mapper = _map_latex if syntax == "latex" else _map_code
  1504. prose_chars = mapper(capped, lambda span: "|CODE|")
  1505. prose = mapper(capped, lambda span: re.sub(r"[^\n]", " ", span))
  1506. if syntax == "latex":
  1507. prose = re.sub(r"[{}]", " ", prose)
  1508. text, evasion_defused = defuse_evasion(
  1509. strip_markdown_furniture(strip_quoted_spans(prose))
  1510. )
  1511. unfolded = text
  1512. text = text.replace("’", "'")
  1513. text = re.sub(r"(?<=\S)[ \t\xa0]{2,}(?=\S)|(?<=\S)[\t\xa0](?=\S)", " ", text)
  1514. simple = [
  1515. ("16_ing_tail", ING_TAIL, "Superficial -ing tail clauses"),
  1516. ("3_promotional", PROMO, "Promotional language"),
  1517. ("12_vague_attribution", VAGUE_ATTR, "Vague attributions"),
  1518. ("36_challenges_section", CHALLENGES, "Formulaic challenges sections"),
  1519. ("1_ai_vocabulary", AI_VOCAB, "AI vocabulary"),
  1520. ("51_copula_avoidance", COPULA_AVOID, "Copula avoidance"),
  1521. ("17_negative_parallelism", NEG_PARALLEL, "Negative parallelism"),
  1522. ("17_tailing_negation", TAILING_NEG, "Tailing negation"),
  1523. ("56_boldface", BOLD, "Boldface overuse"),
  1524. ("33_inline_header_list", INLINE_HEADER, "Inline-header list"),
  1525. ("60_curly_quotes", CURLY, "Curly quotes"),
  1526. ("47_chatbot_artifacts", CHATBOT, "Chatbot artifacts"),
  1527. ("50_cutoff_disclaimer", CUTOFF, "Knowledge-cutoff disclaimer"),
  1528. ("48_sycophantic", SYCOPHANTIC, "Sycophantic tone"),
  1529. ("19_filler_phrases", FILLER, "Filler phrases"),
  1530. ("39_generic_positive_end", GENERIC_END, "Generic positive conclusion"),
  1531. ("4_hyphen_cliche", HYPHEN_CLICHE, "Hyphenated cliché"),
  1532. ("22_authority_trope", AUTHORITY_TROPE, "Persuasive authority trope"),
  1533. ("31_signposting", SIGNPOSTING, "Signposting"),
  1534. ("32_catalog_leadin", CATALOG_LEADIN, "Cataloguing lead-in"),
  1535. ("32_catalog_pivot", CATALOG_PIVOT, "Cataloguing pivot"),
  1536. ("20_empty_pivot_phrase", EMPTY_PIVOT, "Empty pivot phrase"),
  1537. ("38_compulsive_conclusion", COMPULSIVE_CONCLUSION, "Compulsive conclusion phrase"),
  1538. ("21_outcome_speculation", OUTCOME_SPECULATION, "Outcome speculation tail"),
  1539. ("5_simple_yet", SIMPLE_YET, "Simple yet X cliché"),
  1540. ("10_significance_inflation", SIGNIFICANCE_INFLATION, "Significance inflation"),
  1541. ("11_notability_list", NOTABILITY_LIST, "Notability name-dropping"),
  1542. ("15_false_range", FALSE_RANGE, "False range"),
  1543. ("36_faux_insight", FAUX_INSIGHT, "Faux-insight setup"),
  1544. ("32_definitional_metaphor", DEFINITIONAL_METAPHOR, "Definitional metaphor"),
  1545. ("37_colon_reveal", COLON_REVEAL, "Colon reveal"),
  1546. ("47_chat_residue", CHAT_RESIDUE, "Chat residue in a deliverable"),
  1547. ("50_privacy_filler", PRIVACY_FILLER, "Speculative privacy filler"),
  1548. ("25_question_answer", QA_RHETORICAL, "Question-answer rhetorical pattern"),
  1549. ("24_self_thoroughness", SELF_THOROUGHNESS, "Self-thoroughness phrase"),
  1550. ("23_editorial_interjection", EDITORIAL_INTERJECTION, "Editorial interjection"),
  1551. ("13_article_title_noun", ARTICLE_TITLE_NOUN, "Article-title-as-proper-noun"),
  1552. ("18_not_x_just_y", NOT_X_JUST_Y, "Not X, just Y framing"),
  1553. ("27_compulsive_intro", COMPULSIVE_INTRO, "Compulsive intro hook"),
  1554. ("26_concrete_evidence", CONCRETE_EVIDENCE, "Concrete-evidence defense phrase"),
  1555. ("14_ecosystem_padding", ECOSYSTEM_PADDING, "Conservation / ecosystem padding"),
  1556. ("63_placeholder_text", PLACEHOLDER, "Placeholder / Mad-Libs text"),
  1557. ("64_chatbot_ref_markup", CHATBOT_REF_MARKUP, "Chatbot reference-markup leak"),
  1558. ("65_ai_tracking_params", AI_UTM, "AI tracking params (utm_source)"),
  1559. ("30_diff_anchored", DIFF_ANCHORED, "Diff-anchored writing"),
  1560. ("69_canonical_slop", CANONICAL_SLOP, "Canonical marketing-slop phrase"),
  1561. ]
  1562. flags_re = re.IGNORECASE
  1563. for key, pattern, label in simple:
  1564. src = unfolded if key == "60_curly_quotes" else text
  1565. matches = regex_count(src, pattern, flags_re)
  1566. if matches:
  1567. result[key] = {
  1568. "label": label,
  1569. "count": len(matches),
  1570. "samples": samples(src, pattern, flags_re),
  1571. }
  1572. emojis = EMOJI.findall(text)
  1573. if emojis:
  1574. result["57_emojis"] = {
  1575. "label": "Emojis",
  1576. "count": len(emojis),
  1577. "samples": list(dict.fromkeys(emojis))[:MAX_SAMPLES],
  1578. }
  1579. tc = detect_title_case_headings(text)
  1580. if tc:
  1581. result["59_title_case_heading"] = {
  1582. "label": "Title case in headings",
  1583. "count": len(tc),
  1584. "samples": [h["heading"] for h in tc[:MAX_SAMPLES]],
  1585. }
  1586. em = detect_em_dash_overuse(text)
  1587. if em:
  1588. result["58_em_dash_overuse"] = {
  1589. "label": "Em-dash overuse (>=2 per paragraph)",
  1590. "count": len(em),
  1591. "samples": em[:MAX_SAMPLES],
  1592. }
  1593. syn = detect_synonym_cycling(text)
  1594. if syn:
  1595. result["8_synonym_cycling"] = {
  1596. "label": "Synonym cycling",
  1597. "count": len(syn),
  1598. "samples": [", ".join(c) for c in syn[:MAX_SAMPLES]],
  1599. }
  1600. hedge = detect_hedge_stacking(text)
  1601. if hedge:
  1602. result["54_hedge_stacking"] = {
  1603. "label": "Hedge stacking",
  1604. "count": len(hedge),
  1605. "samples": [h["sample"] for h in hedge[:MAX_SAMPLES]],
  1606. }
  1607. cta = detect_imperative_cta(text)
  1608. if cta:
  1609. result["3_imperative_cta"] = {
  1610. "label": "Imperative marketing calls-to-action",
  1611. "count": len(cta),
  1612. "samples": cta[:MAX_SAMPLES],
  1613. }
  1614. frag = detect_fragmented_headers(text)
  1615. if frag:
  1616. result["35_fragmented_header"] = {
  1617. "label": "Fragmented headers",
  1618. "count": len(frag),
  1619. "samples": [f"{f['heading']!r} / {f['filler_line']!r}" for f in frag[:MAX_SAMPLES]],
  1620. }
  1621. rep = detect_degenerate_repetition(text)
  1622. if rep["degenerate"]:
  1623. result["71_degenerate_repetition"] = {
  1624. "label": "Degenerate repetition",
  1625. "count": 1,
  1626. "samples": ["%.0f%% of six-word spans repeat%s"
  1627. % (100 * rep["ratio"],
  1628. "; worst %r x%d" % rep["worst"] if rep["worst"] else "")],
  1629. }
  1630. op = detect_opener_repetition(text)
  1631. if op:
  1632. result["42_opener_repetition"] = {
  1633. "label": "Opener repetition",
  1634. "count": len(op),
  1635. "samples": [f"para {h['paragraph']}: '{h['opener']}' x{h['repeats']}" for h in op[:MAX_SAMPLES]],
  1636. }
  1637. trans = detect_transition_clusters(text)
  1638. if trans:
  1639. result["37_transition_cluster"] = {
  1640. "label": "Transition cluster overuse",
  1641. "count": len(trans),
  1642. "samples": [f"para {t['paragraph']}: {', '.join(t['transitions'])}" for t in trans[:MAX_SAMPLES]],
  1643. }
  1644. stacked = detect_stacked_adjectives(text)
  1645. if stacked:
  1646. result["6_stacked_adjectives"] = {
  1647. "label": "Stacked adjective chains",
  1648. "count": len(stacked),
  1649. "samples": stacked[:MAX_SAMPLES],
  1650. }
  1651. invisible = detect_invisible_chars(prose_chars)
  1652. if invisible["count"]:
  1653. result["62_invisible_chars"] = {
  1654. "label": "Invisible / zero-width characters (strip)",
  1655. "count": invisible["count"],
  1656. "samples": _labelled_samples(invisible["chars"]),
  1657. }
  1658. nbsp = detect_nonstandard_spaces(prose_chars)
  1659. if nbsp["count"]:
  1660. result["67_nonstandard_spaces"] = {
  1661. "label": "Non-standard spaces (normalize to U+0020)",
  1662. "count": nbsp["count"],
  1663. "samples": _labelled_samples(nbsp["chars"]),
  1664. }
  1665. trailing = detect_trailing_whitespace(prose_chars)
  1666. if trailing["artifact"]:
  1667. tsamples: list[str] = []
  1668. if trailing["leading_whitespace"]:
  1669. tsamples.append(f"{trailing['leading_whitespace']} leading whitespace char(s)")
  1670. if trailing["eol_trailing_spaces"]:
  1671. tsamples.append(f"{trailing['eol_trailing_spaces']} line(s) with trailing spaces/tabs")
  1672. if trailing["trailing_blanks"]:
  1673. tsamples.append(f"{trailing['trailing_blanks']} trailing space/tab char(s) at EOF")
  1674. if trailing["trailing_newlines"] >= 2:
  1675. tsamples.append(f"{trailing['trailing_newlines']} blank trailing newline(s) at EOF")
  1676. result["68_trailing_whitespace"] = {
  1677. "label": "Trailing / stray whitespace (trim)",
  1678. "count": len(tsamples),
  1679. "samples": tsamples,
  1680. }
  1681. rules = detect_decorative_rules(prose_chars)
  1682. if rules:
  1683. result["70_decorative_rules"] = {
  1684. "label": "Decorative horizontal rules",
  1685. "count": rules["count"],
  1686. "samples": rules["samples"],
  1687. }
  1688. bullets = detect_mid_essay_bullets(text)
  1689. if bullets["mixed"]:
  1690. result["34_mid_essay_bullets"] = {
  1691. "label": "Mid-essay bullet injection",
  1692. "count": bullets["bullet_paragraphs"],
  1693. "samples": [
  1694. f"{bullets['bullet_paragraphs']} bullet block(s) inside "
  1695. f"{bullets['prose_paragraphs']} prose paragraph(s)"
  1696. ],
  1697. }
  1698. para_mono = detect_paragraph_monotony(text)
  1699. if para_mono.get("monotonous"):
  1700. result["44_paragraph_monotony"] = {
  1701. "label": "Uniform paragraph length",
  1702. "count": 1,
  1703. "samples": [f"cv={para_mono['cv']} below 0.4 threshold across {para_mono['paragraphs']} paragraphs"],
  1704. }
  1705. contractions = detect_contraction_absence(text)
  1706. if contractions.get("opinion_mode") and contractions.get("absent"):
  1707. result["55_contraction_absence"] = {
  1708. "label": "Contraction absence (opinion mode)",
  1709. "count": contractions["contractable_forms"],
  1710. "samples": [
  1711. f"{contractions['contractable_forms']} full forms vs "
  1712. f"{contractions['contractions']} contractions "
  1713. f"(absence ratio {contractions['absence_ratio']})"
  1714. ],
  1715. }
  1716. hyphen_ranges = detect_hyphen_ranges(text)
  1717. if hyphen_ranges:
  1718. result["61_hyphen_for_en_dash"] = {
  1719. "label": "Hyphen-for-en-dash in numeric ranges",
  1720. "count": len(hyphen_ranges),
  1721. "samples": hyphen_ranges[:MAX_SAMPLES],
  1722. }
  1723. punct = detect_punctuation_diversity(text)
  1724. if punct.get("checked") and punct.get("underused"):
  1725. result["46_punctuation_underuse"] = {
  1726. "label": "Semicolon and parenthesis underuse",
  1727. "count": 1,
  1728. "samples": [
  1729. f"{punct['semicolons']} semicolons and {punct['parentheticals']} "
  1730. f"parentheticals across {punct['words']} words"
  1731. ],
  1732. }
  1733. homoglyphs = detect_homoglyphs(prose_chars)
  1734. if homoglyphs:
  1735. result["66_homoglyphs"] = {
  1736. "label": "Homoglyph / mixed-script confusables (fold to ASCII)",
  1737. "count": len(homoglyphs),
  1738. "samples": homoglyphs[:MAX_SAMPLES],
  1739. }
  1740. burst = detect_burstiness(text)
  1741. if burst["monotonous"]:
  1742. result["40_sentence_monotony"] = {
  1743. "label": "Sentence-length monotony",
  1744. "count": 1,
  1745. "samples": [f"cv={burst['cv']} below 0.35 threshold"],
  1746. }
  1747. cadence = detect_cadence_alternation(text)
  1748. if cadence["mechanical"]:
  1749. result["41_mechanical_alternation"] = {
  1750. "label": "Mechanical sentence-length alternation",
  1751. "count": 1,
  1752. "samples": [
  1753. f"zigzag={cadence['zigzag']} across {cadence['sentences']} sentences"
  1754. ],
  1755. }
  1756. dialect_claude = regex_count(text, MODEL_DIALECT_CLAUDE)
  1757. dialect_gpt = regex_count(text, MODEL_DIALECT_GPT)
  1758. if len(dialect_claude) + len(dialect_gpt) >= 2:
  1759. dsamples: list[str] = []
  1760. if dialect_claude:
  1761. dsamples.append("claude-dialect: " + ", ".join(dict.fromkeys(dialect_claude))[:100])
  1762. if dialect_gpt:
  1763. dsamples.append("gpt-dialect: " + ", ".join(dict.fromkeys(dialect_gpt))[:100])
  1764. result["2_model_dialect"] = {
  1765. "label": "Model-dialect vocabulary",
  1766. "count": len(dialect_claude) + len(dialect_gpt),
  1767. "samples": dsamples,
  1768. }
  1769. readability = detect_readability_consistency(text)
  1770. passive = detect_passive_voice(text)
  1771. score = ai_tell_score(result, text, burst, para_mono, punct, readability)
  1772. result["_metrics"] = {
  1773. "scanned_chars": min(len(raw_text), _SCAN_CAP),
  1774. "truncated": len(raw_text) > _SCAN_CAP,
  1775. "sentences": burst["sentences"],
  1776. "length_cv": burst["cv"],
  1777. "has_short_sentence": burst["has_short"],
  1778. "monotonous": burst["monotonous"],
  1779. "paragraph_cv": para_mono.get("cv"),
  1780. "paragraph_monotonous": para_mono.get("monotonous"),
  1781. "invisible_chars": invisible["count"],
  1782. "evasion_defused": evasion_defused,
  1783. "nonstandard_spaces": nbsp["count"],
  1784. "trailing_newlines": trailing["trailing_newlines"],
  1785. "eol_trailing_spaces": trailing["eol_trailing_spaces"],
  1786. "leading_whitespace": trailing["leading_whitespace"],
  1787. "first_person_ratio": contractions.get("first_person_ratio", contractions.get("ratio")),
  1788. "semicolons_per_1k": punct.get("semicolons_per_1k"),
  1789. "parentheticals_per_1k": punct.get("parentheticals_per_1k"),
  1790. "passive_voice_ratio": passive.get("ratio"),
  1791. "readability_fk_stdev": readability.get("fk_stdev"),
  1792. "readability_uniform": readability.get("uniform"),
  1793. "cadence_zigzag": cadence.get("zigzag"),
  1794. "homoglyphs": len(homoglyphs),
  1795. "ai_tell_score": score["score"],
  1796. "ai_tell_band": score["band"],
  1797. "patterns_flagged": sum(1 for k in result if not k.startswith("_")),
  1798. }
  1799. # Third place the same asymmetry appeared. Style-only rules are excluded
  1800. # from the score, so counting them here inflated the confidence attached to
  1801. # it: a document could be reported as having more independent evidence than
  1802. # any of it had reached the number.
  1803. scored_fams = {
  1804. int(k.split("_", 1)[0])
  1805. for k in result
  1806. if not k.startswith("_")
  1807. and k not in _SCORE_REPORT_ONLY
  1808. and k not in _SCORE_STYLE_ONLY
  1809. }
  1810. result["_metrics"].update(score_confidence(count_words(text), len(scored_fams)))
  1811. return result
  1812. _CONFIDENCE_FLOOR = 120
  1813. _CONFIDENCE_SOLID = 300
  1814. # ----------------------------------------------------------------- confidence
  1815. def score_confidence(words: int, families: int) -> dict:
  1816. """How much the score is worth. Short text gives the scorer too little to work
  1817. with: a 20-word snippet can cross a band on one phrase. Reported rather than
  1818. hidden, so a caller can decline to act on a number that has not earned it."""
  1819. if words < 40:
  1820. return {"confidence": "none",
  1821. "confidence_reason": "%d words: too short to score. Treat any "
  1822. "number here as noise." % words}
  1823. if words < _CONFIDENCE_FLOOR:
  1824. return {"confidence": "low",
  1825. "confidence_reason": "%d words: below the %d-word floor, so one "
  1826. "phrase can move the band."
  1827. % (words, _CONFIDENCE_FLOOR)}
  1828. if words < _CONFIDENCE_SOLID and families <= 1:
  1829. return {"confidence": "low",
  1830. "confidence_reason": "%d words and %d scored pattern: not enough "
  1831. "independent evidence." % (words, families)}
  1832. if words < _CONFIDENCE_SOLID:
  1833. return {"confidence": "moderate",
  1834. "confidence_reason": "%d words: usable, but %d or more is where "
  1835. "the score settles." % (words, _CONFIDENCE_SOLID)}
  1836. return {"confidence": "high", "confidence_reason": ""}
  1837. # ----------------------------------------------------------- document formats
  1838. _ZIP_DOC = {
  1839. ".docx": ("word/document.xml",), ".docm": ("word/document.xml",),
  1840. ".pptx": ("ppt/slides/slide",), ".pptm": ("ppt/slides/slide",),
  1841. ".xlsx": ("xl/sharedStrings.xml", "xl/worksheets/sheet"),
  1842. ".xlsm": ("xl/sharedStrings.xml", "xl/worksheets/sheet"),
  1843. ".odt": ("content.xml",), ".odp": ("content.xml",), ".ods": ("content.xml",),
  1844. ".epub": (".xhtml", ".html"),
  1845. }
  1846. _BLOCK_TAGS = {"p", "h", "br", "tab", "tr", "title", "caption"}
  1847. _TEXT_TAGS = {"t", "seg", "span"}
  1848. # Markup that carries no prose.
  1849. _EPUB_SKIP_TAGS = {"head", "script", "style", "nav", "svg", "template",
  1850. "noscript"}
  1851. _EPUB_MAX_DEPTH = 100
  1852. _ZIP_MEMBER_CAP = 32 * 1024 * 1024
  1853. _EPUB_BLOCK_TAGS = {
  1854. "address", "article", "aside", "blockquote", "br", "caption", "div",
  1855. "footer", "h1", "h2", "h3", "h4", "h5", "h6", "header", "li",
  1856. "main", "nav", "p", "section", "table", "td", "th", "title", "tr",
  1857. }
  1858. def _local(tag: str) -> str:
  1859. return tag.rsplit("}", 1)[-1]
  1860. _NAMED_ENTITY = re.compile(rb"&([A-Za-z][A-Za-z0-9]{1,30});")
  1861. _XML_ENTITY = {b"amp", b"lt", b"gt", b"quot", b"apos"}
  1862. def _numeric_entities(data: bytes) -> bytes:
  1863. """XML defines five entity names; EPUB and XHTML use the HTML set. An
  1864. &nbsp; or an &mdash; is a parse error, and the chapter it sits in used to
  1865. be dropped without a word, so the names are rewritten as numeric
  1866. references before the parser sees them."""
  1867. from html.entities import name2codepoint
  1868. def sub(m):
  1869. name = m.group(1)
  1870. if name in _XML_ENTITY:
  1871. return m.group(0)
  1872. cp = name2codepoint.get(name.decode("ascii"))
  1873. return b"&#%d;" % cp if cp else m.group(0)
  1874. return _NAMED_ENTITY.sub(sub, data)
  1875. def _strip_prefixed_attrs(blob: bytes) -> bytes:
  1876. """Drop namespace-prefixed attributes so an undeclared prefix cannot
  1877. take a whole chapter down. Attributes hold no prose."""
  1878. return re.sub(rb'''[ ]+[A-Za-z_][-\w.]*:[A-Za-z_][-\w.]*[ ]*=[ ]*("[^"]*"|'[^']*')''', b'', blob)
  1879. def _epub_text(el, buf: list, depth: int = 0) -> None:
  1880. """Pre-order text of an XHTML tree, skipping subtrees that carry no prose."""
  1881. tag = _local(el.tag)
  1882. if tag in _EPUB_SKIP_TAGS or depth > _EPUB_MAX_DEPTH:
  1883. if el.tail:
  1884. buf.append(el.tail)
  1885. return
  1886. if tag in _EPUB_BLOCK_TAGS and buf and buf[-1] != "\n":
  1887. buf.append("\n")
  1888. if el.text:
  1889. buf.append(el.text)
  1890. for child in el:
  1891. _epub_text(child, buf, depth + 1)
  1892. if el.tail:
  1893. buf.append(el.tail)
  1894. def extract_office_text(path: str) -> str:
  1895. """Pull the visible text out of a zip-based document. Standard library only."""
  1896. import xml.etree.ElementTree as ET
  1897. import zipfile
  1898. ext = os.path.splitext(path)[1].lower()
  1899. wanted = _ZIP_DOC.get(ext)
  1900. if not wanted:
  1901. return ""
  1902. epub = ext == ".epub"
  1903. spreadsheet = ext in {".xlsx", ".xlsm"}
  1904. shared_strings = []
  1905. out: list = []
  1906. with zipfile.ZipFile(path) as z:
  1907. names = [n for n in z.namelist() if any(w in n for w in wanted)]
  1908. names.sort(key=_reading_order)
  1909. for name in names:
  1910. # Bound what an archive member may expand to.
  1911. try:
  1912. if z.getinfo(name).file_size > _ZIP_MEMBER_CAP:
  1913. continue
  1914. except KeyError:
  1915. continue
  1916. try:
  1917. blob = z.read(name)
  1918. except (KeyError, zipfile.BadZipFile, EOFError):
  1919. continue
  1920. try:
  1921. root = ET.fromstring(_numeric_entities(blob))
  1922. except ET.ParseError:
  1923. # Prefixed attributes carry no prose; drop them and retry.
  1924. try:
  1925. root = ET.fromstring(_numeric_entities(_strip_prefixed_attrs(blob)))
  1926. except ET.ParseError:
  1927. continue
  1928. except KeyError:
  1929. continue
  1930. buf: list = []
  1931. if ext in {".docx", ".docm"}:
  1932. # ponytail: explicit code styles only; no Word style-cascade
  1933. # interpreter. Untagged code still needs editorial judgment.
  1934. w = "{http://schemas.openxmlformats.org/wordprocessingml/2006/main}"
  1935. code_styles = {"code", "sourcecode", "codeblock", "codechar", "htmlpreformatted"}
  1936. for el in root.iter():
  1937. if el.tag not in {w + "p", w + "r"}:
  1938. continue
  1939. props = el.find(w + ("pPr" if el.tag == w + "p" else "rPr"))
  1940. if props is None:
  1941. continue
  1942. style = props.find(w + ("pStyle" if el.tag == w + "p" else "rStyle"))
  1943. if style is not None and re.sub(r"\s+", "", style.get(w + "val", "")).lower() in code_styles:
  1944. for t in el.iter(w + "t"):
  1945. t.text = " "
  1946. if epub:
  1947. _epub_text(root, buf)
  1948. elif spreadsheet:
  1949. if name == "xl/sharedStrings.xml":
  1950. shared_strings = ["".join(el.text or "" for el in si.iter()
  1951. if _local(el.tag) == "t")
  1952. for si in root if _local(si.tag) == "si"]
  1953. continue
  1954. for cell in root.iter():
  1955. if _local(cell.tag) != "c":
  1956. continue
  1957. kind = cell.get("t")
  1958. value = next((el.text or "" for el in cell
  1959. if _local(el.tag) == "v"), "")
  1960. if kind == "s":
  1961. try:
  1962. index = int(value)
  1963. except ValueError:
  1964. continue
  1965. if not 0 <= index < len(shared_strings):
  1966. continue
  1967. value = shared_strings[index]
  1968. elif kind == "inlineStr":
  1969. value = "".join(el.text or "" for el in cell.iter()
  1970. if _local(el.tag) == "t")
  1971. elif kind != "str":
  1972. continue
  1973. if value:
  1974. buf.extend((value, "\n"))
  1975. elif ext in {".odt", ".odp", ".ods"}:
  1976. space_budget = _ZIP_MEMBER_CAP
  1977. for el in root.iter():
  1978. tag = _local(el.tag)
  1979. if tag == "s":
  1980. count = next((v for k, v in el.attrib.items() if _local(k) == "c"), "1")
  1981. try:
  1982. count = min(max(int(count), 1), space_budget)
  1983. except ValueError:
  1984. count = min(1, space_budget)
  1985. el.text = " " * count
  1986. space_budget -= count
  1987. elif tag in {"tab", "line-break"}:
  1988. el.text = "\t" if tag == "tab" else "\n"
  1989. for el in root.iter():
  1990. if _local(el.tag) in {"p", "h"}:
  1991. buf.extend(("".join(el.itertext()), "\n"))
  1992. else:
  1993. for el in root.iter():
  1994. tag = _local(el.tag)
  1995. if tag in _TEXT_TAGS and el.text:
  1996. buf.append(el.text)
  1997. elif tag in _BLOCK_TAGS:
  1998. if buf and buf[-1] != "\n":
  1999. buf.append("\n")
  2000. if el.text:
  2001. buf.append(el.text)
  2002. if el.tail and el.tail.strip():
  2003. buf.append(el.tail)
  2004. chunk = "".join(buf)
  2005. if epub:
  2006. chunk = re.sub(r"[ \t]+\n", "\n", chunk)
  2007. if chunk.strip():
  2008. out.append(chunk)
  2009. text = "\n\n".join(out)
  2010. return re.sub(r"\n{3,}", "\n\n", text).strip()
  2011. # ------------------------------------------------------------------ error bar
  2012. def score_interval(text: str, resamples: int = 60, seed: int = 11) -> dict:
  2013. """A confidence interval on the score, by resampling the document's own
  2014. sentences with replacement and rescoring. Answers a question no threshold can:
  2015. how much of this number is the document, and how much is which sentences
  2016. happened to be in it. Costs `resamples` extra scans, so it is opt-in."""
  2017. import random
  2018. sents = [s for s in split_sentences(text) if s.strip()]
  2019. if len(sents) < 4:
  2020. return {"score_ci": None,
  2021. "score_ci_note": "fewer than 4 sentences: no interval"}
  2022. rnd = random.Random(seed)
  2023. n = len(sents)
  2024. draws = []
  2025. for _ in range(resamples):
  2026. sample = " ".join(rnd.choice(sents) for _ in range(n))
  2027. draws.append(scan(sample)["_metrics"]["ai_tell_score"])
  2028. draws.sort()
  2029. lo = draws[int(0.025 * resamples)]
  2030. hi = draws[min(resamples - 1, int(0.975 * resamples))]
  2031. return {"score_ci": [lo, hi], "score_ci_resamples": resamples,
  2032. "score_ci_note": "2.5th to 97.5th percentile over %d sentence "
  2033. "resamples" % resamples}
  2034. def extract_notebook_text(path: str) -> str:
  2035. """Markdown cells out of a Jupyter notebook. Code cells and outputs are not
  2036. prose and are left alone, the same way the guard leaves source files alone."""
  2037. try:
  2038. with open(path, "r", encoding="utf-8-sig", errors="replace") as f:
  2039. nb = json.load(f)
  2040. except (OSError, ValueError):
  2041. return ""
  2042. if not isinstance(nb, dict) or not isinstance(nb.get("cells", []), list):
  2043. return ""
  2044. out = []
  2045. for cell in nb.get("cells", []):
  2046. if not isinstance(cell, dict):
  2047. return ""
  2048. if cell.get("cell_type") != "markdown":
  2049. continue
  2050. src = cell.get("source", "")
  2051. if isinstance(src, list) and all(isinstance(part, str) for part in src):
  2052. src = "".join(src)
  2053. if not isinstance(src, str):
  2054. return ""
  2055. out.append(src)
  2056. return re.sub(r"\n{3,}", "\n\n", "\n\n".join(out)).strip()
  2057. _UTF16_BOM = (b"\xff\xfe", b"\xfe\xff")
  2058. _UNREADABLE_EXT = {".pdf", ".rtf"}
  2059. def _decode_text(data: bytes) -> str:
  2060. """UTF-16 is what PowerShell redirection writes by default. Decoded as
  2061. UTF-8 it becomes one NUL per character, which the character layer then
  2062. reports as thousands of invisible characters in ordinary human prose.
  2063. Anything else holding a NUL byte is not text at all, and the control bytes
  2064. it decodes to would be reported the same invented way, so it reads as
  2065. nothing rather than as a document full of hidden characters."""
  2066. if data[:2] in _UTF16_BOM:
  2067. return data.decode("utf-16", errors="replace")
  2068. if b"\x00" in data:
  2069. return ""
  2070. return data.decode("utf-8-sig", errors="replace")
  2071. def read_source(paths: list) -> tuple:
  2072. """The text, plus the line ending it arrived with so --clean can put it
  2073. back. The scanner only ever sees LF, which is what every published number
  2074. was measured on."""
  2075. if paths:
  2076. ext = os.path.splitext(paths[0])[1].lower()
  2077. if ext in _ZIP_DOC:
  2078. return extract_office_text(paths[0]), "\n"
  2079. if ext == ".ipynb":
  2080. return extract_notebook_text(paths[0]), "\n"
  2081. if ext in _UNREADABLE_EXT:
  2082. return "", "\n"
  2083. with open(paths[0], "rb") as f:
  2084. text = _decode_text(f.read())
  2085. newline = "\r\n" if "\r\n" in text else "\n"
  2086. return text.replace("\r\n", "\n").replace("\r", "\n"), newline
  2087. return _decode_text(sys.stdin.buffer.read()), "\n"
  2088. def read_input(paths: list) -> str:
  2089. return read_source(paths)[0]
  2090. USAGE = """usage: detect.py [--clean] [--ci] [--] [FILE]
  2091. Scans FILE, or stdin when no file is given, and prints the findings as JSON.
  2092. --clean print the scrubbed text instead of the JSON report. For a .docx, .epub,
  2093. .odt or .ipynb this prints the extracted plain text, not a rebuilt
  2094. document, so redirecting it back over the original destroys the file.
  2095. --ci add a resampled interval to the score
  2096. -- stop reading options, so a FILE may begin with a dash
  2097. --help this message
  2098. """
  2099. def main() -> int:
  2100. import zipfile
  2101. for stream in (sys.stdin, sys.stdout):
  2102. if hasattr(stream, "reconfigure"):
  2103. # newline="" or Windows rewrites every LF in the cleaned text to
  2104. # CRLF, so no document could round-trip byte for byte.
  2105. stream.reconfigure(encoding="utf-8", errors="replace", newline="")
  2106. paths: list = []
  2107. flags: set = set()
  2108. literal = False
  2109. for arg in sys.argv[1:]:
  2110. if literal or not arg.startswith("-"):
  2111. paths.append(arg)
  2112. elif arg == "--":
  2113. literal = True
  2114. elif arg in ("--clean", "--ci"):
  2115. flags.add(arg)
  2116. elif arg in ("--help", "-h"):
  2117. sys.stdout.write(USAGE)
  2118. return 0
  2119. else:
  2120. print(json.dumps({"error": f"unknown option: {arg}. Try --help."}),
  2121. file=sys.stderr)
  2122. return 2
  2123. if len(paths) > 1:
  2124. # Scanning the first and dropping the rest in silence was the defect. Erroring
  2125. # would break a shell glob that used to work, so say so and carry on.
  2126. print("detect.py: %d files given; scanning %s only, one at a time is supported."
  2127. % (len(paths), paths[0]), file=sys.stderr)
  2128. del paths[1:]
  2129. try:
  2130. text, newline = read_source(paths)
  2131. except (OSError, zipfile.BadZipFile) as e:
  2132. reason = getattr(e, "strerror", None) or e
  2133. print(json.dumps({"error": f"cannot read input: {reason}"}), file=sys.stderr)
  2134. return 1
  2135. if not text.strip():
  2136. print(json.dumps({"error": "empty input"}), file=sys.stderr)
  2137. return 1
  2138. syntax = "latex" if paths and os.path.splitext(paths[0])[1].lower() == ".tex" else "markdown"
  2139. if "--clean" in flags:
  2140. cleaned = clean_text(text, syntax=syntax)
  2141. # A file that ended in a newline still should.
  2142. if text.endswith("\n") and not cleaned.endswith("\n"):
  2143. cleaned += "\n"
  2144. sys.stdout.write(cleaned.replace("\n", newline) if newline != "\n" else cleaned)
  2145. return 0
  2146. result = scan(text, syntax=syntax)
  2147. if "--ci" in flags:
  2148. interval_text = _map_latex(text, lambda span: " ") if syntax == "latex" else text
  2149. result["_metrics"].update(score_interval(interval_text))
  2150. print(json.dumps(result, indent=2, ensure_ascii=False))
  2151. return 0
  2152. if __name__ == "__main__":
  2153. sys.exit(main())

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.