Skip to content
Back to search

watermarks-remover

MarketplaceWorks with Claude CodeGitHub

Anti-detection hygiene skills: invisible Unicode, statistical text watermarks, and C2PA/EXIF/XMP provenance metadata.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/guillaumemeyer/watermarks-remover with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/guillaumemeyer/watermarks-remover'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/guillaumemeyer-watermarks-remover/badge.svg)](/listings/guillaumemeyer-watermarks-remover)

HTML

<a href="/listings/guillaumemeyer-watermarks-remover"><img src="/api/listings/guillaumemeyer-watermarks-remover/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

The skill ships **no code** — it calls the service over HTTP. Install the skill (markdown only) and start the service, then set `WATERMARKS_SERVICE_URL` if it is not `http://127.0.0.1:8765`.

In Claude Code, the fastest route is the bundled [plugin marketplace](#claude-code-plugin-marketplace) — no clone, and it updates in place. Everywhere else, one installer covers every supported host (Python 3.10+ stdlib, no dependencies):

python3 install_skill.py --skill remove-ai-marks --target claude-code

| Host | Target | Lands in | | --- | --- | --- | | Claude Code (personal) | `--target claude-code` | `~/.claude/skills/<skill>` (honors `CLAUDE_CONFIG_DIR`) | | Claude Code (project) | `--target claude-project --project-dir PATH` | `PATH/.claude/skills/<skill>` | | Cowork, claude.ai, cloud sessions, routines | `--target cowork` | `dist/<skill>.zip` to upload under **Customize → Skills** | | Cursor | `--target cursor` (default) | `~/.cursor/skills/<skill>` |

Shipped skills: `remove-ai-marks` (full, service-backed) and `clean-user-facing-text` (text only, self-contained). `--list` prints them. Existing installations are preserved unless you pass `--force`; replacement is staged first and the previous install is kept as a uniquely named backup. `--link` symlinks this checkout instead of copying, so edits are picked up live. On Windows, use `py install_skill.py ...`; the `install-skill.sh` wrapper is provided for macOS/Linux shells.

Before writing anything, the installer validates the skill against the [Agent Skills](https://agentskills.io) packaging rules that claude.ai uploads and the Skills API enforce: spec-only frontmatter (`name`, `description`, `license`, `compatibility`, `metadata`, `allowed-tools`), a lowercase hyphenated `name` of at most 64 characters matching the directory, a non-empty `description` of at most 1024 characters. The Cowork bundle additionally has to fit the 30 MB upload limit, which the packager enforces.

Automatic cleaning via hook (deterministic)

A skill is an instruction: the model decides whether to invoke it, and the model is the thing producing the marks. A **hook** is executed by the harness on every matching tool call, cooperation not required. That makes the hook the deterministic half of this workflow.

The plugin registers a `PostToolUse` hook on `Write|Edit|MultiEdit|NotebookEdit` that runs [`service/scripts/hook_written_file.py`](service/scripts/hook_written_file.py) against the file the agent just wrote. Two modes, matching the pre-commit convention of check-by-default:

| Mode | Behaviour | | --- | --- | | `check` (default) | Reports provenance marks, leaves the file alone. Findings go to the model (exit 2), so it can offer to clean them. | | `clean` | Strips the marks in place, then tells the model the file on disk changed. |

Set the mode from the plugin's settings (**Hook mode** in `/plugin manage`, read by the hook as `CLAUDE_PLUGIN_OPTION_HOOK_MODE`), or with `WATERMARKS_HOOK_MODE=clean` in the environment. The hook command deliberately does **not** interpolate `${user_config.hook_mode}`: Claude Code refuses to run a hook that references an option the user has never opened `/plugin manage` to set — a declared `default` does not satisfy it — so interpolating it would mean the hook silently never runs on a fresh install. Detection reuses `audit_lib`'s `scan_file` / `is_actionable`, so the hook, the pre-commit gate, and the CI SARIF export agree on what counts as actionable; cleaning shells out to `clean_file.py`, so no cleaning logic is duplicated. `clean` mode writes to a sibling temp file and swaps only on a real difference, so files that were already clean keep their mtime and don't retrigger file watchers.

Without the plugin, wire it in `~/.claude/settings.json` (or a project `.claude/settings.json`) yourself:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Write|Edit|MultiEdit|NotebookEdit",
        "hooks": [
          {

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 41 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "watermarks-remover",
  3. "owner": {
  4. "name": "watermarks-remover contributors",
  5. "url": "https://github.com/guillaumemeyer/watermarks-remover"
  6. },
  7. "description": "Anti-detection hygiene skills: invisible Unicode, statistical text watermarks, and C2PA/EXIF/XMP provenance metadata.",
  8. "plugins": [
  9. {
  10. "name": "watermarks-remover",
  11. "source": "./",
  12. "displayName": "Watermarks Remover",
  13. "description": "Remove multi-vendor AI provenance marks from text, images, documents, and media. Bundles the remove-ai-marks and clean-user-facing-text skills.",
  14. "version": "0.7.0",
  15. "author": {
  16. "name": "watermarks-remover contributors",
  17. "url": "https://github.com/guillaumemeyer/watermarks-remover"
  18. },
  19. "homepage": "https://github.com/guillaumemeyer/watermarks-remover#readme",
  20. "repository": "https://github.com/guillaumemeyer/watermarks-remover",
  21. "license": "MIT",
  22. "category": "content",
  23. "keywords": [
  24. "watermark",
  25. "c2pa",
  26. "content-credentials",
  27. "provenance",
  28. "metadata",
  29. "unicode",
  30. "synthid",
  31. "ai-detection"
  32. ],
  33. "tags": [
  34. "watermark",
  35. "c2pa",
  36. "metadata",
  37. "unicode"
  38. ]
  39. }
  40. ]
  41. }
README.md · README · 1,500 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. ```
  2. _ _ _ ____ ___ ____ ____ _ _ ____ ____ _ _ ____ ____ ____ _ _ ____ _ _ ____ ____
  3. | | | |__| | |___ |__/ |\/| |__| |__/ |_/ [__ __ |__/ |___ |\/| | | | | |___ |__/
  4. |_|_| | | | |___ | \ | | | | | \ | \_ ___] | \ |___ | | |__| \/ |___ | \
  5. ```
  6. # watermarks-remover
  7. <!-- logo: figlet -d .figlet -f cybermedium -w 120 "watermarks-remover" -->
  8. [![CI](https://github.com/guillaumemeyer/watermarks-remover/actions/workflows/ci.yml/badge.svg)](https://github.com/guillaumemeyer/watermarks-remover/actions/workflows/ci.yml)
  9. [![Release](https://img.shields.io/github/v/release/guillaumemeyer/watermarks-remover)](https://github.com/guillaumemeyer/watermarks-remover/releases)
  10. [![Stars](https://img.shields.io/github/stars/guillaumemeyer/watermarks-remover)](https://github.com/guillaumemeyer/watermarks-remover/stargazers)
  11. [![Forks](https://img.shields.io/github/forks/guillaumemeyer/watermarks-remover)](https://github.com/guillaumemeyer/watermarks-remover/forks)
  12. Agent skill + stdlib Python service to strip **multi-vendor AI provenance marks** from text and files — for privacy and hygiene on content **you own**. The skill is a thin client: it drives the machinery over HTTP, so the agent host needs no Python.
  13. | Layer | Target | How |
  14. | --- | --- | --- |
  15. | **A** | Invisible Unicode, exotic spaces, bidi, tag chars | Deterministic Python scripts |
  16. | **B** | Statistical (token-sampling) text watermarks | Agent rewrite + optional `rewrite_text.py` hook |
  17. | **Files** | C2PA / EXIF / XMP / doc props | PNG, JPEG, WebP, AVIF, HEIC, BMP, GIF, TIFF, SVG, PDF, DOCX, XLSX, PPTX, EPUB, ODT, HTML, Markdown, MP4/MOV/M4A/M4V, WAV, MP3, FLAC |
  18. Vendors / ecosystems (class-level): **Claude**, **Gemini / SynthID-Text**, **OpenAI** provenance surfaces, **open-LLM** Kirchenbauer-style (green-list) and keyed-Gumbel / EXP (Aaronson) marks.
  19. **Latest release:** [v0.7.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.7.0)
  20. Skill path: [`skills/remove-ai-marks/`](skills/remove-ai-marks/)
  21. Service path: [`service/`](service/)
  22. (migration: formerly `remove-claude-marks`; slash alias `/remove-claude-marks` still documented)
  23. ## Install (agent skill)
  24. The skill ships **no code** — it calls the service over HTTP. Install the skill (markdown only) and start the service, then set `WATERMARKS_SERVICE_URL` if it is not `http://127.0.0.1:8765`.
  25. In Claude Code, the fastest route is the bundled
  26. [plugin marketplace](#claude-code-plugin-marketplace) — no clone, and it updates
  27. in place. Everywhere else, one installer covers every supported host
  28. (Python 3.10+ stdlib, no dependencies):
  29. ```bash
  30. python3 install_skill.py --skill remove-ai-marks --target claude-code
  31. ```
  32. | Host | Target | Lands in |
  33. | --- | --- | --- |
  34. | Claude Code (personal) | `--target claude-code` | `~/.claude/skills/<skill>` (honors `CLAUDE_CONFIG_DIR`) |
  35. | Claude Code (project) | `--target claude-project --project-dir PATH` | `PATH/.claude/skills/<skill>` |
  36. | Codex (personal) | `--target codex` | `~/.agents/skills/<skill>` |
  37. | Codex (project) | `--target codex-project --project-dir PATH` | `PATH/.agents/skills/<skill>` |
  38. | Cowork, claude.ai, cloud sessions, routines | `--target cowork` | `dist/<skill>.zip` to upload under **Customize → Skills** |
  39. | Cursor | `--target cursor` (default) | `~/.cursor/skills/<skill>` |
  40. Shipped skills: `remove-ai-marks` (full, service-backed) and
  41. `clean-user-facing-text` (text only, self-contained). `--list` prints them.
  42. Existing installations are preserved unless you pass `--force`; replacement is
  43. staged first and the previous install is kept as a uniquely named backup.
  44. `--link` symlinks this checkout instead of copying, so edits are picked up
  45. live. On Windows, use `py install_skill.py ...`; the `install-skill.sh` wrapper
  46. is provided for macOS/Linux shells.
  47. Before writing anything, the installer validates the skill against the
  48. [Agent Skills](https://agentskills.io) packaging rules that claude.ai uploads
  49. and the Skills API enforce: spec-only frontmatter (`name`, `description`,
  50. `license`, `compatibility`, `metadata`, `allowed-tools`), a lowercase hyphenated
  51. `name` of at most 64 characters matching the directory, a non-empty
  52. `description` of at most 1024 characters. The Cowork bundle additionally has
  53. to fit the 30 MB upload limit, which the packager enforces.
  54. ### Codex terminal and desktop
  55. Install either shipped skill for your user, or install one into a particular
  56. project:
  57. ```bash
  58. python3 install_skill.py --skill remove-ai-marks --target codex
  59. python3 install_skill.py --skill clean-user-facing-text --target codex
  60. python3 install_skill.py --skill remove-ai-marks --target codex-project --project-dir /path/to/project
  61. ```
  62. On Windows, use `py` in place of `python3`. Open the project in Codex. In the
  63. terminal, use `/skills` or type `$` to select a skill; in the desktop app,
  64. select or mention the skill in your prompt. Restart Codex if the new skill does
  65. not appear. The project install is available when Codex runs in that project.
  66. For `remove-ai-marks`, start the existing HTTP service from this repository
  67. with `make serve` before using the skill (or on Windows without `make`, run
  68. `py service/scripts/server.py --host 127.0.0.1 --port 8765`). It uses
  69. `WATERMARKS_SERVICE_URL` (default `http://127.0.0.1:8765`), checks `/health`,
  70. and calls the existing `/inspect` and `/clean` endpoints. If the service
  71. requires an API key, set `WATERMARKS_SERVER_API_KEY` for the client.
  72. `clean-user-facing-text` is
  73. self-contained and does not need the service.
  74. ### Automatic cleaning via hook (deterministic)
  75. A skill is an instruction: the model decides whether to invoke it, and the
  76. model is the thing producing the marks. A **hook** is executed by the harness
  77. on every matching tool call, cooperation not required. That makes the hook the
  78. deterministic half of this workflow.
  79. The plugin registers a `PostToolUse` hook on `Write|Edit|MultiEdit|NotebookEdit`
  80. that runs [`service/scripts/hook_written_file.py`](service/scripts/hook_written_file.py)
  81. against the file the agent just wrote. Two modes, matching the pre-commit
  82. convention of check-by-default:
  83. | Mode | Behaviour |
  84. | --- | --- |
  85. | `check` (default) | Reports provenance marks, leaves the file alone. Findings go to the model (exit 2), so it can offer to clean them. |
  86. | `clean` | Strips the marks in place, then tells the model the file on disk changed. |
  87. Set the mode from the plugin's settings (**Hook mode** in `/plugin manage`,
  88. read by the hook as `CLAUDE_PLUGIN_OPTION_HOOK_MODE`), or with
  89. `WATERMARKS_HOOK_MODE=clean` in the environment. The hook command deliberately
  90. does **not** interpolate `${user_config.hook_mode}`: Claude Code refuses to run
  91. a hook that references an option the user has never opened `/plugin manage` to
  92. set — a declared `default` does not satisfy it — so interpolating it would mean
  93. the hook silently never runs on a fresh install. Detection reuses `audit_lib`'s
  94. `scan_file` / `is_actionable`, so the hook, the pre-commit gate, and the CI
  95. SARIF export agree on what counts as actionable; cleaning shells out to
  96. `clean_file.py`, so no cleaning logic is duplicated. `clean` mode writes to a
  97. sibling temp file and swaps only on a real difference, so files that were
  98. already clean keep their mtime and don't retrigger file watchers.
  99. Without the plugin, wire it in `~/.claude/settings.json` (or a project
  100. `.claude/settings.json`) yourself:
  101. ```json
  102. {
  103. "hooks": {
  104. "PostToolUse": [
  105. {
  106. "matcher": "Write|Edit|MultiEdit|NotebookEdit",
  107. "hooks": [
  108. {
  109. "type": "command",
  110. "command": "python3",
  111. "args": ["/path/to/watermarks-remover/service/scripts/hook_written_file.py",
  112. "--mode", "check"],
  113. "timeout": 30
  114. }
  115. ]
  116. }
  117. ]
  118. }
  119. }
  120. ```
  121. On Windows, replace `python3` with `py`.
  122. **What a hook cannot do.** No hook can rewrite the assistant's chat message
  123. before you read it. Claude Code's `Stop` hook receives `last_assistant_message`
  124. read-only, and there is no pre-send filter for final responses — the same limit
  125. this project already documents for Cursor rules. So the deterministic guarantee
  126. covers **files the agent writes**, plus the
  127. [pre-commit gate](#pre-commit-hook) for anything on its way into git. Text that
  128. only ever exists in the chat transcript still depends on the skill workflow,
  129. which is model-instruction-based and therefore best-effort.
  130. ### Claude Code plugin (marketplace)
  131. The repository is also a Claude Code **plugin** and a single-plugin
  132. **marketplace** (`.claude-plugin/`), so both skills install and update in two
  133. commands, no clone or script required:
  134. ```
  135. /plugin marketplace add guillaumemeyer/watermarks-remover
  136. /plugin install watermarks-remover@watermarks-remover
  137. ```
  138. The skills then load namespaced: `/watermarks-remover:remove-ai-marks` and
  139. `/watermarks-remover:clean-user-facing-text` (the bare `/remove-ai-marks` also
  140. works when nothing else claims the name). `/plugin marketplace update
  141. watermarks-remover` pulls later versions. The same works from the CLI with
  142. `claude plugin marketplace add …` / `claude plugin install …`, and from a local
  143. checkout by passing a path instead of `owner/repo`.
  144. Maintainers: `make plugin-validate` runs `claude plugin validate . --strict`
  145. against both manifests; `tests/test_plugin_manifest.py` covers the same files
  146. without needing the CLI.
  147. ### Claude Code
  148. ```bash
  149. # Personal — available in all your projects
  150. python3 install_skill.py --skill remove-ai-marks --target claude-code
  151. # or: make install-claude-code-skill
  152. # Project — commit .claude/skills/ to share it with the repo
  153. python3 install_skill.py --skill remove-ai-marks --target claude-project \
  154. --project-dir /path/to/project
  155. # or: make install-claude-project-skill PROJECT=/path/to/project
  156. ```
  157. Claude Code picks up personal and project skills without a restart; `/skills`
  158. lists what it loaded. Invoke with `/remove-ai-marks` or ask to “strip AI
  159. watermarks / C2PA / Claude marks / SynthID-class text.” A project install is
  160. also what [cloud sessions](https://code.claude.com/docs/en/cloud-environments)
  161. read, since they clone the repository and load its `.claude/skills/`.
  162. ### Cowork (and claude.ai, cloud sessions, routines)
  163. Cowork sessions do **not** read `~/.claude/skills` on your machine — they load
  164. the skills enabled for your claude.ai account, synced when the session starts.
  165. So install there by uploading a bundle:
  166. ```bash
  167. python3 install_skill.py --skill remove-ai-marks --target cowork
  168. # writes dist/remove-ai-marks.zip (make package-cowork-skill)
  169. ```
  170. Then, in the Claude Desktop app, open **Customize → Skills → Add** and upload
  171. the zip (the same skill settings on claude.ai work too). The bundle is
  172. reproducible and contains a single top-level `remove-ai-marks/` directory with
  173. `SKILL.md` at its root, which is the layout the upload expects.
  174. Service reachability matters more here than in a local install: the skill is a
  175. thin HTTP client, so the session must be able to reach `WATERMARKS_SERVICE_URL`.
  176. Cowork sessions that run locally on your machine reach a local `make serve`;
  177. cloud sessions and routines run remotely and need a service URL reachable from
  178. there (and `WATERMARKS_SERVER_API_KEY` set on it). If you want a skill with no
  179. service at all, upload `clean-user-facing-text` instead — it is text-only and
  180. ships its own scripts:
  181. ```bash
  182. python3 install_skill.py --skill clean-user-facing-text --target cowork
  183. ```
  184. ### Grok
  185. ```bash
  186. # Grok Build / project-local
  187. mkdir -p .grok/skills
  188. ln -sfn "$(pwd)/skills/remove-ai-marks" .grok/skills/remove-ai-marks
  189. # User-global Grok
  190. mkdir -p ~/.grok/skills
  191. ln -sfn "$(pwd)/skills/remove-ai-marks" ~/.grok/skills/remove-ai-marks
  192. ```
  193. ### Optional text-only skill
  194. [`skills/clean-user-facing-text/`](skills/clean-user-facing-text/) is a
  195. self-contained skill for authorized manuscripts, documentation, and web
  196. copy. It excludes image, C2PA, service, and external-model tooling, and runs
  197. its own vendored Layer A scripts instead of calling the service.
  198. ```bash
  199. python3 install_skill.py --skill clean-user-facing-text --target claude-code
  200. python3 install_skill.py --skill clean-user-facing-text --target cursor
  201. ```
  202. Skill invocation is model-selected. Projects that explicitly adopt this
  203. workflow in Cursor can also copy the optional rule:
  204. ```bash
  205. mkdir -p /path/to/project/.cursor/rules
  206. cp integrations/cursor/clean-user-facing-text.mdc \
  207. /path/to/project/.cursor/rules/clean-user-facing-text.mdc
  208. ```
  209. For all projects, put the same instruction in Cursor **User Rules** instead.
  210. Rules improve consistency but remain model instructions; Cursor does not expose
  211. a deterministic pre-send filter for final chat responses.
  212. ### Start the service
  213. The fastest path is a local HTTP server (Python 3.10+ stdlib only — no deps, no Docker):
  214. ```bash
  215. make serve # http://127.0.0.1:8765
  216. # or directly:
  217. python3 service/scripts/server.py --host 127.0.0.1 --port 8765
  218. ```
  219. ### Windows (no Docker)
  220. See [docs/windows-autostart.md](docs/windows-autostart.md) for auto-starting the service at Windows login without Docker.
  221. For the whole infra (core + optional harness/heavy backends), see [Docker / compose](#docker--compose) below.
  222. Optional system tools (auto-used when present — preinstalled in the core Docker image):
  223. | Tool | Role |
  224. | --- | --- |
  225. | [`c2patool`](https://github.com/contentauth/c2pa-rs/tree/main/cli) | Inspect C2PA manifests |
  226. | [`exiftool`](https://exiftool.org/) | Residual metadata strip (esp. **PDF**) |
  227. | [`qpdf`](https://qpdf.sourceforge.io/) | Structural PDF rebuild — **required** for a real PDF strip (see below) |
  228. Core scripts need **Python 3.10+** stdlib only. Layer B model calls are optional.
  229. ## Quick use (scripts)
  230. ```bash
  231. SCRIPTS=service/scripts
  232. # Unified inspect / clean
  233. python3 "$SCRIPTS/inspect_file.py" draft.md
  234. python3 "$SCRIPTS/clean_file.py" draft.md -o draft.cleaned.md
  235. python3 "$SCRIPTS/clean_file.py" photo.png -o photo.cleaned.png
  236. python3 "$SCRIPTS/clean_file.py" notes.docx -o notes.cleaned.docx
  237. # Text Layer A
  238. python3 "$SCRIPTS/inspect_text.py" draft.md
  239. python3 "$SCRIPTS/clean_text.py" draft.md -o draft.cleaned.md --stats
  240. # Layer B rewrite hook (default: print prompt only — no model required)
  241. python3 "$SCRIPTS/rewrite_text.py" draft.md --backend print-prompt --tactic paraphrase
  242. # Optional local Ollama (loopback only by default — remote endpoints require
  243. # WATERMARKS_REWRITE_ALLOW_REMOTE=1 or --allow-remote):
  244. # WATERMARKS_REWRITE_BACKEND=ollama WATERMARKS_REWRITE_MODEL=llama3.2 \
  245. # python3 "$SCRIPTS/rewrite_text.py" draft.md -o draft.rewritten.md
  246. # API keys are read from WATERMARKS_REWRITE_API_KEY only (never argv).
  247. # Images
  248. python3 "$SCRIPTS/inspect_image.py" shot.png
  249. python3 "$SCRIPTS/clean_image.py" shot.png -o shot.cleaned.png
  250. ```
  251. ### Text tools refuse binary input
  252. `inspect_text.py`, `clean_text.py` and `rewrite_text.py` operate on text. Pointed
  253. at a `.docx`, `.pdf` or image they used to decode the compressed bytes and report
  254. whatever codepoints fell out — noise that tracks the compression, not the
  255. content — and `clean_text.py` then wrote those mangled bytes back, destroying the
  256. file. They now refuse binary input and name the tool that handles it:
  257. ```bash
  258. python3 "$SCRIPTS/inspect_text.py" report.docx
  259. # refusing to treat report.docx as text: it looks like a ZIP container (DOCX, ODT, …).
  260. # Use inspect_file.py / clean_file.py, which route by format,
  261. # or pass --force-text to scan the raw bytes anyway.
  262. ```
  263. Detection is by magic number plus a control-byte ratio, so text in encodings
  264. other than UTF-8 keeps working. `--force-text` overrides it everywhere.
  265. ### Unrecognized formats are never auto-cleaned
  266. `classify()` labels bytes that match no supported text, image or container
  267. format as **`unknown`** — it no longer falls back to "text". In auto mode
  268. `clean_file.py` refuses such files (exit 2, no output written) instead of
  269. decoding them as UTF-8 and writing back mangled bytes; `--as text` or
  270. `--force-text` are the explicit opt-ins. `inspect_file.py` reports the file
  271. as `unknown` (exit 0), and the HTTP service answers `/inspect` with
  272. `kind: "unknown"` but rejects `/clean` of unknown formats (400 — send a
  273. filename with a known extension, e.g. `notes.txt`).
  274. ## HTTP service
  275. The same machinery runs as a stdlib HTTP service (`service/scripts/server.py`) — the interface the skill uses and the way any web app can integrate without vendoring:
  276. | Method | Path | Body | Returns |
  277. | --- | --- | --- | --- |
  278. | GET | `/health` | — | `{"ok": true, "version": ...}` |
  279. | GET | `/capabilities` | — | optional tools / backends usable (each tool is version-probed, not just found on `PATH`) |
  280. | GET | `/openapi.json` | — | dynamically generated OpenAPI 3.0.3 spec |
  281. | POST | `/inspect` | `{"file": "<base64>", "name": "notes.md"}` | `{"ok", "kind", "suspicious", "report"}` |
  282. | POST | `/detect` | `{"file": "<base64>", "name": "notes.txt"}` | `{"ok", "kind", "detections": [...]}` |
  283. | POST | `/clean` | `{"file": "<base64>", "name": "notes.md", "options": {...}}` | `{"ok", "kind", "cleaned": "<base64>", "report"}` |
  284. | POST | `/watermark` | `{"text": "...", "keys": [118, 504, ...], "options": {...}}` or `{"file": "<base64>", ...}` | `{"ok", "kind", "watermarked_text", "report": {"scheme_used", ...}}` |
  285. | POST | `/inspect/batch` | `{"files": [{"file": "<base64>", "name": "notes.md"}, ...]}` | `{"ok", "results": [{"name", "ok", "kind", "suspicious", "report"}, ...]}` |
  286. | POST | `/detect/batch` | `{"files": [{"file": "<base64>", "name": "notes.txt"}, ...]}` | `{"ok", "results": [{"name", "ok", "kind", "detections", "report"}, ...]}` |
  287. | POST | `/clean/batch` | `{"files": [{"file": "<base64>", "name": "notes.md", "options": {...}}, ...]}` | `{"ok", "results": [{"name", "ok", "kind", "cleaned", "report"}, ...]}` |
  288. | POST | `/watermark/batch` | `{"files": [{"text": "...", "keys": [...]}, {"file": "<base64>"}, ...]}` | `{"ok", "results": [{"name", "ok", "kind", "watermarked_text", "report": {"scheme_used", ...}}, ...]}` |
  289. Batch endpoints loop the same per-file pipeline as `/inspect`, `/detect`, `/clean`, and `/watermark`, capped at `WATERMARKS_MAX_BATCH_FILES` files per request (default 50). A malformed entry (bad base64, unknown option, unrecognized format) surfaces as that entry's `"ok": false` with an `"error"` string — it never aborts the rest of the batch.
  290. ```bash
  291. WM="http://127.0.0.1:8765"
  292. curl -s "$WM/health" # {"ok": true, "version": "..."}
  293. curl -s "$WM/openapi.json" # machine-readable OpenAPI 3.0.3 contract
  294. curl -s -X POST "$WM/clean" -H 'Content-Type: application/json' \
  295. -d "{\"file\": \"$(base64 < notes.md | tr -d '\n')\", \"name\": \"notes.md\"}"
  296. ```
  297. The service routes by filename extension then magic bytes, so text / image / container are auto-detected. Set `WATERMARKS_SERVER_API_KEY` to require `Authorization: Bearer <key>` on every request. Loopback-only bind by default (`--host` to override); intended for a trusted network.
  298. ### Watermark detection (`/detect` and `detect_before` / `detect_after`)
  299. Detection is a separate step from cleaning — the service never calls vendor
  300. APIs unless you ask it to:
  301. - **`POST /detect`** runs the configured watermark detectors on a file.
  302. Text → vendor detectors + stylometry; image → SynthID pixel score.
  303. - **`/inspect`** accepts an opt-in `"detect": true` flag that appends
  304. detector results to the text report (and can flip `suspicious`).
  305. - **`/clean`** accepts `"detect_before"` / `"detect_after"` options to
  306. score the input and the cleaned output, so you can measure what a clean
  307. actually changed.
  308. - **`/clean`** runs the Layer B text rewrite after Layer A **by default** (it
  309. is a required step for text). A **`"strategy"`** option (an ordered
  310. `tactic@intensity` list, e.g. `"paraphrase@0.8,mlm@0.2"`) overrides the
  311. default from the strategy config file (see below). When the rewrite
  312. backend/model for a step isn't configured, `/clean` returns a 400.
  313. Text detectors (see `/capabilities` → `text_detectors`):
  314. Text detectors (see `/capabilities` → `text_detectors`):
  315. | Detector | Activated by | Notes |
  316. | --- | --- | --- |
  317. | `markllm` | `MARKLLM_DIR` (host checkout) | Research harness (KGW / SynthID schemes), same-config-only — not a vendor oracle. |
  318. | `gumbel` | `WATERMARKS_GUMBEL_KEY` | Model-free same-key replay of the keyed-Gumbel (Aaronson EXP) scheme (see `detect_gumbel.py`), stdlib-only — self-hosted engines such as arbi-serve; same-key-only, not a vendor oracle. |
  319. | `claude-text` | — (placeholder) | Anthropic has announced a watermark detection API; this seam activates when it ships. |
  320. Image scoring: when `WATERMARKS_SYNTHID_SCORER_URL` is set, the service
  321. scores images through the `wr-synthid-score` sidecar (heavy profile); with a
  322. local `REVERSE_SYNTHID_DIR` it uses the checkout directly. Detection is
  323. fail-soft: unconfigured, timed-out, or errored detectors report
  324. `{"available": false, "error": ...}` and never block cleaning.
  325. ### Watermark generation (`/watermark` and `/watermark/batch`)
  326. Generates watermarked text for benchmark evaluation and round-trip testing.
  327. When `WATERMARKS_SYNTHID_TEXT_URL` is set, the service delegates generation to the
  328. `wr-synthid-text` sidecar (harness profile); with a local `MARKLLM_DIR` it uses the
  329. checkout directly. Like detection, generation is fail-soft: an unconfigured generator
  330. reports `{"ok": false, "error": ...}`.
  331. ## Docker / compose
  332. Published images (GHCR):
  333. | Image tag | Contents | Published? |
  334. | --- | --- | --- |
  335. | `ghcr.io/guillaumemeyer/watermarks-remover:<tag>` / `:latest` | Core HTTP service + all cleaners + exiftool / qpdf / c2patool | Yes |
  336. | `…:markllm-<tag>` / `:markllm-latest` | MarkLLM text-watermark harness (Apache-2.0 upstream) | Yes |
  337. | `…:markdiffusion-<tag>` / `:markdiffusion-latest` | MarkDiffusion image harness (Apache-2.0 upstream) | Yes |
  338. | `watermarks-remover-ctrlregen:local` | CtrlRegen pixel removal — **never published** (`noai-watermark` ships no LICENSE) | Local build only |
  339. | `watermarks-remover-synthid-scorer:local` | reverse-SynthID scorer — **never published** (non-commercial Research License) | Local build only (CLI scorer + optional `wr-synthid-score` HTTP sidecar under the `heavy` profile) |
  340. Build and run the core service:
  341. ```bash
  342. make docker-core-build
  343. docker run --rm -p 127.0.0.1:8765:8765 --read-only --tmpfs /tmp watermarks-remover
  344. # any CLI stays runnable by overriding the command:
  345. docker run --rm -v "$(pwd):/data" watermarks-remover \
  346. /app/scripts/clean_file.py /data/notes.md -o /data/notes.cleaned.md
  347. ```
  348. Whole-infra bring-up:
  349. ```bash
  350. docker compose up -d # core HTTP service only
  351. docker compose --profile harness up -d # + markllm / markdiffusion / wr-synthid-text sidecar
  352. docker compose --profile heavy up -d # + ctrlregen / synthid (local builds)
  353. docker compose --profile harness --profile heavy up -d # all services
  354. ```
  355. The compose stack maps the core service to `127.0.0.1:8765`. Persistent services run as background daemons (`wr-core` and the `wr-synthid-text` sidecar under the harness profile). The remaining harness/heavy services are one-shot CLIs — invoke with `docker compose run --rm <service> …` when you need verification or pixel work.
  356. Validate the running stack (exit code only, no output on success):
  357. ```bash
  358. make compose-check # or: ./compose-check.sh
  359. ```
  360. Checks `wr-core` via `GET /health` and runs each harness/heavy service with `--help`, requiring exit `0`.
  361. ### Configuration (env vars for docker compose)
  362. **Text cleaning requires Layer B configuration** — the Layer B rewrite is a
  363. required step for `POST /clean` on text, so the core service needs the rewrite
  364. backend set up, or text cleaning returns HTTP 400. Image/container metadata
  365. cleaning works out of the box. For text you must configure the Layer B strategy
  366. dependencies: `transformers` + `roberta-large` (for the default `mlm` step) and
  367. the `WATERMARKS_REWRITE_*` LLM config (for the `paraphrase` step):
  368. ```bash
  369. echo "Hello\u200bWorld\u00ad!" > /tmp/sample.txt
  370. curl -s -X POST http://127.0.0.1:8765/clean -H 'Content-Type: application/json' \
  371. -d "{\"file\": \"$(base64 < /tmp/sample.txt | tr -d '\n')\", \"name\": \"sample.txt\"}"
  372. ```
  373. Languages whose typography relies on a non-breaking space (French `« … »`, the
  374. space before `; : ! ?`) should pass `"options": {"normalize_spaces": false}`, the
  375. HTTP equivalent of `clean_text.py --no-normalize-spaces`. Invisible carriers are
  376. still removed; only the space rewrite is skipped.
  377. Everything else is optional and lives in a `.env` file at the repo root. `docker compose` **auto-loads `.env`** and interpolates the `${VAR}` references in `compose.yaml` from it (shell exports win over `.env` if both are set).
  378. ```bash
  379. cp .env.example .env # then edit
  380. docker compose up -d # picks up .env automatically
  381. ```
  382. `.env` is **gitignored** (deny-by-default) — never commit it. For host-side CLI runs (`rewrite_text.py`, the skill), export the same file into the environment:
  383. ```bash
  384. set -a; . ./.env; set +a; python3 service/scripts/rewrite_text.py /tmp/x.txt -o /tmp/x.rewritten.txt
  385. ```
  386. | Var | Reaches | Purpose |
  387. | --- | --- | --- |
  388. | `WATERMARKS_SERVER_API_KEY` | `wr-core` (via compose `environment`) | Require `Authorization: Bearer <key>` on the HTTP API |
  389. | `WATERMARKS_GEMINI_*` | — | Removed Aug 2026: Google retired SynthID text watermarking on the API (see `vendor-notes.md`) |
  390. | `WATERMARKS_SYNTHID_SCORER_URL` | `wr-core` | Point core at the `wr-synthid-score` sidecar for SynthID image scoring (e.g. `http://wr-synthid-score:8766` under the heavy profile) |
  391. | `WATERMARKS_SYNTHID_SCORER_API_KEY` | `wr-core` + `wr-synthid-score` | Shared bearer key for the scorer sidecar (empty = no auth) |
  392. | `WATERMARKS_SYNTHID_TEXT_URL` | `wr-core` | Point core at the `wr-synthid-text` sidecar for SynthID text watermarking (e.g. `http://wr-synthid-text:8767` under the harness profile) |
  393. | `WATERMARKS_SYNTHID_TEXT_API_KEY` | `wr-core` + `wr-synthid-text` | Shared bearer key for the text watermark sidecar (empty = no auth) |
  394. | `WATERMARKS_SYNTHID_TEXT_TIMEOUT` | `wr-core` | Seconds to wait for the `wr-synthid-text` sidecar (default 120) |
  395. | `WATERMARKS_MARKLLM_SCHEME` | `text_detectors.py` (host) | MarkLLM scheme for `/detect`: `kgw` (default) / `synthid` |
  396. | `HF_TOKEN` | harness/heavy services | Hugging Face token for gated models |
  397. | `WATERMARKS_SERVICE_URL` | client only (skill / curl) | Where to reach the service; default `http://127.0.0.1:8765` |
  398. | `WATERMARKS_REWRITE_BACKEND` | `rewrite_text.py` hook | `print-prompt` (default) / `ollama` / `openai-compatible` |
  399. | `WATERMARKS_REWRITE_MODEL` | `rewrite_text.py` hook | Model name (e.g. `deepseek-v4-flash`) |
  400. | `WATERMARKS_REWRITE_BASE_URL` | `rewrite_text.py` hook | API base (e.g. `https://api.deepseek.com`) |
  401. | `WATERMARKS_REWRITE_API_KEY` | `rewrite_text.py` hook | API key — env only, never on argv |
  402. | `WATERMARKS_REWRITE_ALLOW_REMOTE` | `rewrite_text.py` hook | `1` to allow non-loopback endpoints |
  403. | `WATERMARKS_REWRITE_REASONING_EFFORT` | `rewrite_text.py` hook | `none` (default) / `low` / `medium` / `high` / `off` |
  404. | `WATERMARKS_CLEAN_STRATEGY_FILE` | `server.py` `/clean` | Path to the Layer B strategy config JSON (default `config/clean_strategy.json`) |
  405. | `WATERMARKS_GUMBEL_KEY` | `detect_gumbel.py` / `text_detectors.py` | Secret key for keyed-Gumbel (EXP) same-key replay (e.g. `0x…`); preferred over argv — never logged |
  406. **Layer B is required for text cleaning.** `/clean` always applies the default
  407. strategy (from `config/clean_strategy.json`, `{"default_strategy": "paraphrase@0.8,mlm@0.2"}`) to a text file after Layer A, unless the request passes its own `"strategy"` option (an ordered `tactic@intensity` list). A strategy step is `tactic@intensity`; the `mlm` step needs `transformers` + `roberta-large`, and any LLM step (`paraphrase`, `humanize`, …) needs the `WATERMARKS_REWRITE_*` config. If the required backend/model isn't configured — or no strategy is available — `/clean` **rejects the request with a 400**. Precedence for the config path: `--strategy-config` CLI flag > `WATERMARKS_CLEAN_STRATEGY_FILE` env var > the default `config/clean_strategy.json`.
  408. Images publish automatically on `v*` tags via [`.github/workflows/release-images.yml`](.github/workflows/release-images.yml).
  409. ## Optional SynthID pixel scoring
  410. `inspect_image.py` and `clean_image.py` can report a pixel-domain SynthID
  411. confidence score when an external checkout of
  412. [`aloshdenny/reverse-SynthID`](https://github.com/aloshdenny/reverse-SynthID)
  413. is available. The scorer is **not bundled**: it is loaded at runtime from your
  414. checkout, and its code remains under the upstream project's non-commercial
  415. Research License.
  416. ### Option 1: one-command bootstrap (no Docker)
  417. ```bash
  418. SCRIPTS=service/scripts
  419. # Clones upstream, creates a venv, and installs scorer-only dependencies.
  420. "$SCRIPTS/setup_synthid.sh"
  421. # Score an image (default checkout: ~/reverse-SynthID).
  422. REVERSE_SYNTHID_DIR=~/reverse-SynthID \
  423. ~/reverse-SynthID/.venv/bin/python "$SCRIPTS/score_synthid.py" shot.png
  424. # Or surface the score from inspect / clean (same venv Python).
  425. REVERSE_SYNTHID_DIR=~/reverse-SynthID \
  426. ~/reverse-SynthID/.venv/bin/python "$SCRIPTS/inspect_image.py" shot.png
  427. ```
  428. `setup_synthid.sh` accepts `--dir PATH`, `--ref REF`, and `--full` (install the
  429. full upstream `requirements.txt`, which adds `torch`/`diffusers` for the
  430. upstream VAE bypass this project does not use).
  431. On Windows use `setup_synthid.ps1` (`-Dir`, `-Ref`, `-Full`), which creates the
  432. venv at `.venv\Scripts\` — the layout `image_meta.py` already looks for on
  433. `os.name == "nt"`.
  434. ### Option 2: local Docker build
  435. ```bash
  436. make docker-synthid-build
  437. # Run unprivileged and with a read-only rootfs; the scorer only needs to read
  438. # /data and write to stdout/tmp.
  439. docker run --rm \
  440. --user "$(id -u):$(id -g)" \
  441. --read-only --tmpfs /tmp \
  442. -v "$(pwd):/data" \
  443. watermarks-remover-synthid-scorer /data/shot.png
  444. ```
  445. The image is built locally from the upstream source at build time. It is not
  446. published, so it does not redistribute the upstream code.
  447. ### Option 3: HTTP scorer sidecar (docker compose)
  448. Under the `heavy` profile the compose stack also runs the scorer as an HTTP
  449. sidecar (`wr-synthid-score`) so the **published core service** can score
  450. images before/after cleaning without bundling the non-commercial upstream
  451. code. Point `wr-core` at it and share a bearer key (see `.env.example`):
  452. ```bash
  453. # .env
  454. WATERMARKS_SYNTHID_SCORER_URL=http://wr-synthid-score:8766
  455. WATERMARKS_SYNTHID_SCORER_API_KEY=change-me
  456. docker compose --profile heavy up -d
  457. ```
  458. Then `POST /clean` with `{"options": {"detect_before": true,
  459. "detect_after": true}}` returns `synthid_before` / `synthid_after` in the
  460. report, and `POST /detect` on an image returns the SynthID score. Fail-soft:
  461. if the sidecar is down or unconfigured, reports carry
  462. `{"available": false, "error": ...}` and cleaning still succeeds.
  463. V4 scoring uses `artifacts/spectral_codebook_v4.npz` from the upstream checkout
  464. (`220 MB). This is **detection/scoring only** — it does not remove pixel
  465. watermarks.
  466. ## Optional CtrlRegen pixel removal
  467. For **pixel-domain** image watermarks (SynthID-class, StegaStamp, Tree-Ring,
  468. StableSignature), an optional external backend runs the CtrlRegen pipeline
  469. (ControlNet + DINOv2 IP-Adapter controllable regeneration). The backend is
  470. [`mertizci/noai-watermark`](https://github.com/mertizci/noai-watermark), a
  471. maintained reimplementation of the ICLR 2025
  472. [CtrlRegen](https://arxiv.org/abs/2410.05470) method with automatic tiling.
  473. The backend is **not bundled** and ships no LICENSE file, so it is treated as
  474. all-rights-reserved: it is cloned at a pinned commit and loaded at runtime.
  475. Its research-era dependency pins (`requirements-ctrlregen.txt` — e.g.
  476. `transformers==4.37.2`, `diffusers==0.27.2`) carry published advisories and
  477. are intentionally not current, so they are only ever installed inside the
  478. dedicated venv this script creates and never into the main service image;
  479. `setup_ctrlregen.sh` also re-verifies the pinned commit on existing
  480. checkouts, not just fresh clones.
  481. ### Bootstrap
  482. ```bash
  483. SCRIPTS=service/scripts
  484. # Clones upstream (pinned commit), creates a venv, installs torch + deps.
  485. "$SCRIPTS/setup_ctrlregen.sh"
  486. # Standalone removal (default checkout: ~/noai-watermark).
  487. NOAI_WATERMARK_DIR=~/noai-watermark \
  488. ~/noai-watermark/.venv/bin/python "$SCRIPTS/clean_ctrlregen.py" shot.png -o shot.ctrlregen.png
  489. ```
  490. On Windows use `setup_ctrlregen.ps1` (same flags as `-Dir`, `-Ref`, `-Python`);
  491. the venv lands in `.venv\Scripts\`, which `clean_image.py` already resolves.
  492. It probes the published PyTorch wheel indices and picks the highest one at or
  493. below the CUDA version `nvidia-smi` prints that actually exists — that number
  494. is the maximum the *driver* supports, and drivers are backward compatible, so a
  495. driver reporting 13.1 (no published `cu131`) installs `cu130`. Below compute
  496. capability 7.5 it forces `cu126`, the last index whose wheels still carry
  497. Maxwell/Pascal/Volta kernels. It installs `torch` **and** `torchvision`
  498. together from that index so the dependency install cannot swap them for CPU
  499. builds from PyPI, then verifies after install that `torch.cuda.is_available()`
  500. is true — if a GPU was detected but torch ends up CPU-only, the script warns
  501. loudly and exits non-zero instead of pretending the setup succeeded.
  502. ### From `clean_image.py`
  503. ```bash
  504. NOAI_WATERMARK_DIR=~/noai-watermark \
  505. ~/noai-watermark/.venv/bin/python "$SCRIPTS/clean_image.py" shot.png \
  506. -o shot.cleaned.png --remove-pixel ctrlregen
  507. ```
  508. Order of operations: metadata strip first, then CtrlRegen pixel removal, then
  509. an optional reverse-SynthID before/after score (when `REVERSE_SYNTHID_DIR` is
  510. also set).
  511. **Intensity is conservative by default** (`--ctrlregen-intensity 0.25`), because
  512. higher intensity removes more watermark but regenerates more of the image.
  513. Documented presets: `0.15` minimal / `0.25` default / `0.35` balanced /
  514. `0.5` aggressive / `0.7` max (backend default is 0.5). `--ctrlregen-steps`
  515. defaults to 50 (effective denoising steps ≈ steps × intensity).
  516. ### Image size (512×512 native limit)
  517. CtrlRegen is a 512×512 Stable Diffusion 1.5 ControlNet. The backend resolves
  518. this for arbitrary inputs, so no extra tiling is exposed here:
  519. - **≤512 px:** single pass — center-crop/resize to 512, regenerate, resize back.
  520. - **>512 px:** automatic overlapping tiling (512 px tiles, 192 px overlap),
  521. width/height aligned to multiples of 8, then cosine-blended seams.
  522. - **Either path:** output is resized to the original size and color-matched to
  523. the original image.
  524. Very large images (e.g. 4K) produce many tiles, so runs scale with tile count
  525. (slower and higher VRAM). Pre-downscale large inputs when practical; tile size
  526. and overlap are hardcoded upstream and are not exposed as flags.
  527. ### Compute, gated models, and verification
  528. Expect ~10 GB of model downloads; a GPU is strongly recommended and CPU runs
  529. are slow. Some upstream models are gated, so export `HF_TOKEN` (env only —
  530. never argv). `clean_ctrlregen.py` refuses to auto-install dependencies; run
  531. `setup_ctrlregen.sh` first.
  532. There is no local detector for StegaStamp/Tree-Ring/StableSignature, so the
  533. only local signal is the reverse-SynthID score (a surrogate). When available,
  534. `clean_image.py --remove-pixel ctrlregen` reports that score before/after; the
  535. official Google SynthID check remains the final authority.
  536. ### Docker
  537. ```bash
  538. make docker-ctrlregen-build
  539. docker run --rm -e HF_TOKEN="$HF_TOKEN" \
  540. --user "$(id -u):$(id -g)" \
  541. -v "$(pwd):/data" \
  542. watermarks-remover-ctrlregen /data/shot.png -o /data/shot.ctrlregen.png
  543. ```
  544. ## Optional MarkLLM text-watermark verification
  545. For **controlled experiments**, an optional external harness wraps
  546. [`THU-BPM/MarkLLM`](https://github.com/THU-BPM/MarkLLM) (Apache-2.0) to
  547. watermark test text and re-detect it after a Layer B rewrite — e.g. prove that
  548. a KGW (Kirchenbauer, your "open-LLM" row) or SynthID-Text (Gemini row) mark
  549. disappears under your rewrite. It is a **verification harness, not an oracle**:
  550. MarkLLM detection is only valid against the *same* scheme config + keys used at
  551. generation, and it cannot certify a vendor detector will fail.
  552. The backend is **not bundled**. `setup_markllm.sh` clones upstream at a pinned
  553. commit, creates a venv, and installs pinned deps (torch + transformers); the
  554. scoring model (default `facebook/opt-1.3b`, Apache-2.0) downloads from Hugging
  555. Face on first run.
  556. ```bash
  557. SCRIPTS=service/scripts
  558. # Bootstrap (clones upstream, creates ~/MarkLLM/.venv, installs deps).
  559. "$SCRIPTS/setup_markllm.sh"
  560. # Generate watermarked + unwatermarked sample text under the KGW scheme.
  561. MARKLLM_DIR=~/MarkLLM \
  562. ~/MarkLLM/.venv/bin/python "$SCRIPTS/detect_text_watermark.py" watermark prompt.txt \
  563. --scheme kgw -o wm.txt -o2 plain.txt
  564. # Detect the scheme mark in a text file.
  565. MARKLLM_DIR=~/MarkLLM \
  566. ~/MarkLLM/.venv/bin/python "$SCRIPTS/detect_text_watermark.py" detect wm.txt --scheme kgw --json
  567. ```
  568. **Verification around a Layer B rewrite:** pass `--markllm-scheme` to
  569. `rewrite_text.py` (with `--markllm-dir`), and it records the MarkLLM detection
  570. before/after plus a `cleared` flag:
  571. ```bash
  572. export WATERMARKS_REWRITE_BACKEND=ollama WATERMARKS_REWRITE_MODEL=llama3.2
  573. MARKLLM_DIR=~/MarkLLM \
  574. python3 "$SCRIPTS/rewrite_text.py" wm.txt -o wm.rewritten.txt \
  575. --markllm-scheme kgw --markllm-dir "$HOME/MarkLLM" --json-stats
  576. ```
  577. **Detection-guided iterative rewriting:** Layer B now rewrites iteratively and
  578. stops as soon as an attempt passes evaluation. Each evaluation round generates
  579. `--candidates` variants (default **1**, `WATERMARKS_REWRITE_CANDIDATES`)
  580. and `--max-loops` caps how many rounds run before the best-effort variant is
  581. returned (default **1**, `WATERMARKS_REWRITE_LOOPS`). Each variant is one
  582. rewrite call plus one evaluation, and a round exits early on the first attempt
  583. the evaluator reports as not watermarked — so raising `--max-loops` retries
  584. new variants until an evaluation passes (a typical clean rewrite costs one
  585. attempt). The evaluator is chosen by priority:
  586. 1. **MarkLLM** — same-config research detection, when `--markllm-scheme` is
  587. passed (with `--markllm-dir`). A vendor-detector slot is reserved above
  588. MarkLLM for Google's SynthID-text detector, which Google retired on its API
  589. in Aug 2026 — a future vendor endpoint can plug in there.
  590. 2. **bigram-Jaccard lexical divergence** — when no detector is configured; no
  591. pass/fail verdict, so every attempt is generated and the most lexically
  592. diverged one is selected (the original behavior).
  593. `--json-stats` reports the evaluator, attempts made, pass/fail, and per-attempt
  594. records:
  595. ```json
  596. {
  597. "evaluator": "markllm",
  598. "candidates": 1,
  599. "max_loops": 2,
  600. "attempts_made": 2,
  601. "passed": true,
  602. "candidate_scores": [
  603. {
  604. "lexical_divergence": 0.91,
  605. "selection_score": 0.91,
  606. "selected": false,
  607. "passed": false,
  608. "evaluation": {"detector": "markllm", "available": true, "scheme": "kgw",
  609. "is_watermarked": true, "score": 4.3, "threshold": 3.0}
  610. },
  611. {
  612. "lexical_divergence": 0.84,
  613. "selection_score": 0.84,
  614. "selected": true,
  615. "passed": true,
  616. "evaluation": {"detector": "markllm", "available": true, "scheme": "kgw",
  617. "is_watermarked": false, "score": 1.7, "threshold": 3.0}
  618. }
  619. ],
  620. "markllm": {"scheme": "kgw", "before": {"...": "..."}, "after": {"...": "..."},
  621. "cleared": true, "note": "same-config only"}
  622. }
  623. ```
  624. A detector that is unconfigured, times out, or errors yields an
  625. `"available": false` entry with an `error` reason and never fails the
  626. rewrite — that attempt simply cannot pass, and the loop falls back to
  627. lexical-divergence selection. When the max is exhausted without a pass, the
  628. least-watermarked (lowest score) attempt is returned as best-effort with a
  629. note.
  630. If the backend is unconfigured or its deps are missing, the rewrite proceeds
  631. and the report notes verification was unavailable. A GPU is recommended; CPU
  632. runs work but are slow, and the model download is a few GB.
  633. Hardening knobs:
  634. - `--offline` on the adapter (or any MarkLLM run) loads the scoring model from
  635. the Hugging Face cache only — zero network egress; fails fast if not cached.
  636. Custom remote code is never executed (transformers `trust_remote_code` is
  637. never enabled).
  638. - `WATERMARKS_MARKLLM_RLIMIT_AS=<bytes>` (env, POSIX) applies an address-space
  639. limit to the MarkLLM detector subprocess. Off by default because torch/CUDA
  640. usually needs large address spaces.
  641. - Config files are capped at 1 MiB; the upstream checkout and the base image
  642. are pinned by SHA/digest.
  643. ### Docker
  644. ```bash
  645. make docker-markllm-build
  646. docker run --rm --user "$(id -u):$(id -g)" -v "$(pwd):/data" \
  647. watermarks-remover-markllm detect /data/wm.txt --scheme kgw --json
  648. ```
  649. ### Keyed-Gumbel (Aaronson EXP) same-key verification
  650. [ARBI's technical report](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/) describes the
  651. keyed-Gumbel ("exponential") text watermark — now shipping in the open-source
  652. arbi-serve engine (`ARBI_WATERMARK_KEY`) — where the sampler's noise is derived
  653. from a keyed hash of the last 4-token context window. Detection is a
  654. **model-free replay**: recompute `u = PRF(Hash(key, window), token)` from the
  655. text alone and test the Gamma tail, so it needs no GPU, model, or logits.
  656. This repo ships that detector as `detect_gumbel.py` (stdlib-only; the p-value
  657. is the exact Poisson-sum identity for an integer Gamma shape):
  658. ```bash
  659. # Text mode (deterministic word/run tokenizer) — quick checks and rewrite-loop
  660. # evaluation; exact replay against a real engine needs its tokenizer:
  661. python3 service/scripts/detect_gumbel.py draft.txt --key 0x... --json
  662. # Exact replay: pass the engine's token ids (JSON array or one per line).
  663. python3 service/scripts/detect_gumbel.py ids.json --tokens --key 0x... --json
  664. ```
  665. Same honesty caveat as MarkLLM: this is a **same-key replay** — valid only
  666. against the same key, tokenizer, and PRF layout used at generation, and a
  667. negative result establishes nothing. The HMAC-SHA256 layout here is an
  668. auditable instantiation, not bit-compatible with any specific engine kernel
  669. (see the module docstring for what to adapt for exact replay).
  670. **Detection-guided rewriting:** pass `--gumbel-key` to `rewrite_text.py`
  671. (env: `WATERMARKS_GUMBEL_KEY`, preferred) and the iterative rewrite loop is
  672. driven by the same-key Gumbel replay — evaluator priority becomes gumbel >
  673. MarkLLM > lexical divergence — with a `gumbel.before/after/cleared` report:
  674. ```bash
  675. export WATERMARKS_REWRITE_BACKEND=ollama WATERMARKS_REWRITE_MODEL=llama3.2
  676. export WATERMARKS_GUMBEL_KEY=0x...
  677. python3 "$SCRIPTS/rewrite_text.py" wm.txt -o wm.rewritten.txt --json-stats
  678. ```
  679. The key never appears in stats or logs. Self-hosted operators who hold their
  680. engine's key can verify a rewrite cleared a Gumbel mark; everyone else treats
  681. Layer B as best-effort only.
  682. ## Optional SynthID-text removal benchmark
  683. [`bench_synthid_text.py`](service/scripts/bench_synthid_text.py) measures how
  684. effectively a Layer B rewrite clears SynthID-text-class watermarks and at
  685. what cost. It generates watermarked + unwatermarked samples with the MarkLLM
  686. SynthID scheme (same-config detection, sanity-gated), runs your rewrite
  687. variants (tactic × max rewrite attempts; the loop stops early on pass) plus
  688. controls (no-removal, Layer-A-only, optional re-stamp check), and writes a
  689. shareable `report.md` /
  690. `results.json` / `results.csv`. Full guide:
  691. [`docs/synthid-text-benchmark.md`](docs/synthid-text-benchmark.md).
  692. Requires a MarkLLM checkout (`setup_markllm.sh` / `MARKLLM_DIR`) and a
  693. rewrite backend. **The rewriting model is an LLM you configure** — the same
  694. `rewrite_text.py` backend the skill uses. MarkLLM's default
  695. `facebook/opt-1.3b` (`--markllm-model`) is only the watermark
  696. generator/detector; it never rewrites. Configure the rewrite model via env
  697. vars or benchmark flags (they mirror the
  698. [config table](#configuration-env-vars-for-docker-compose) above):
  699. | Env var | Benchmark flag | Default | Meaning |
  700. | --- | --- | --- | --- |
  701. | `WATERMARKS_REWRITE_BACKEND` | `--rewrite-backend` | `ollama` | `ollama` or `openai-compatible` |
  702. | `WATERMARKS_REWRITE_MODEL` | `--rewrite-model` | *(required)* | The LLM that performs the rewrite (e.g. `llama3.2`, `deepseek-v4-flash`) |
  703. | `WATERMARKS_REWRITE_BASE_URL` | `--rewrite-base-url` | `http://127.0.0.1:11434` | Endpoint; the Ollama default is loopback |
  704. | `WATERMARKS_REWRITE_API_KEY` | `--rewrite-api-key` | — | API key (env-only in the child process, never argv) |
  705. | `WATERMARKS_REWRITE_ALLOW_REMOTE=1` | `--rewrite-allow-remote` | off | Required to send content to non-loopback endpoints |
  706. ```bash
  707. # Ollama (loopback):
  708. python3 service/scripts/bench_synthid_text.py --markllm-dir ~/MarkLLM \
  709. --rewrite-backend ollama --rewrite-model llama3.2
  710. # OpenAI-compatible API (remote):
  711. WATERMARKS_REWRITE_API_KEY=... python3 service/scripts/bench_synthid_text.py \
  712. --markllm-dir ~/MarkLLM --rewrite-backend openai-compatible \
  713. --rewrite-model deepseek-v4-flash --rewrite-base-url https://api.deepseek.com \
  714. --rewrite-allow-remote
  715. ```
  716. Use a **non-origin model** for rewriting (do not rewrite with the same
  717. watermarked model that generated the text) or the rewrite can re-stamp the
  718. output; `--restamp-control` measures this.
  719. ## Optional MarkDiffusion image-watermark harness
  720. For **controlled experiments on images**, an optional external harness wraps
  721. [`THU-BPM/MarkDiffusion`](https://github.com/THU-BPM/MarkDiffusion) (Apache-2.0),
  722. a *generative watermarking* toolkit for latent diffusion models (it embeds marks
  723. — it does not remove them). We use it for three things:
  724. 1. **Verification harness** (like MarkLLM, but for images): watermark a test
  725. image with a scheme, run removal, and re-detect with the *same* scheme config
  726. — e.g. prove a Tree-Ring-class mark clears under your pipeline. It is a
  727. **verification harness, not an oracle**: detection requires the generating
  728. model (and keys for key-based schemes), so it cannot certify a vendor
  729. detector will fail on an arbitrary image.
  730. 2. **Optional pixel-removal engine**: its `DiffusionPurification` regeneration
  731. attack is exposed as `clean_image.py --remove-pixel diffusion`, an
  732. alternative to CtrlRegen. It is **blind** regeneration (no ControlNet
  733. conditioning), so it drifts image content more than CtrlRegen — conservative
  734. intensity default (`0.3`), treated as a fallback/comparison, never a
  735. guarantee.
  736. 3. **Local same-scheme detector** for Tree-Ring-class marks, partially filling
  737. the "no local detector for StegaStamp/Tree-Ring/StableSignature" gap (it
  738. covers Tree-Ring/Ring-ID/Gaussian-Shading etc., not StegaStamp /
  739. StableSignature / SynthID-media).
  740. The backend is **not bundled**. `setup_markdiffusion.sh` creates a venv and
  741. installs `markdiffusion==1.0.2` from PyPI (pinned), with torch installed from
  742. the right platform index; `--checkout` installs an editable clone at a pinned
  743. commit instead. The Stable Diffusion model (default
  744. `huanzi05/stable-diffusion-2-1-base`) downloads from Hugging Face on first run.
  745. ```bash
  746. SCRIPTS=service/scripts
  747. # Bootstrap (PyPI pin default; creates ~/markdiffusion/.venv, installs deps).
  748. "$SCRIPTS/setup_markdiffusion.sh"
  749. # 1. Generate a Tree-Ring watermarked image (+ unwatermarked control).
  750. echo "a red fox in snow" > /tmp/prompt.txt
  751. MARKDIFFUSION_DIR=~/markdiffusion \
  752. ~/markdiffusion/.venv/bin/python "$SCRIPTS/markdiffusion_harness.py" watermark \
  753. /tmp/prompt.txt -o wm.png -o2 plain.png --scheme tr --json
  754. # 2. Remove with the DiffusionPurification regeneration attack.
  755. MARKDIFFUSION_DIR=~/markdiffusion \
  756. ~/markdiffusion/.venv/bin/python "$SCRIPTS/markdiffusion_harness.py" purify \
  757. wm.png -o wm.purified.png --purification-intensity 0.3 --json
  758. # 3. Re-detect with the SAME scheme config.
  759. MARKDIFFUSION_DIR=~/markdiffusion \
  760. ~/markdiffusion/.venv/bin/python "$SCRIPTS/markdiffusion_harness.py" detect \
  761. wm.purified.png --scheme tr --detector-type l1_distance --json
  762. ```
  763. Or run purification as part of the normal image pipeline:
  764. ```bash
  765. MARKDIFFUSION_DIR=~/markdiffusion \
  766. ~/markdiffusion/.venv/bin/python "$SCRIPTS/clean_image.py" shot.png \
  767. -o shot.cleaned.png --remove-pixel diffusion
  768. ```
  769. Hardening knobs mirror the MarkLLM harness: `--offline` loads the model from
  770. the Hugging Face cache only (zero network egress, no remote code), `HF_TOKEN`
  771. is env-only (never argv), algorithm configs are capped at 1 MiB, and the
  772. subprocess gets the same higher resource caps as CtrlRegen.
  773. ### Docker
  774. ```bash
  775. make docker-markdiffusion-build
  776. docker run --rm --user "$(id -u):$(id -g)" -v "$(pwd):/data" \
  777. watermarks-remover-markdiffusion detect /data/wm.png --scheme tr --json
  778. ```
  779. The image installs a CPU torch; CUDA users should run `setup_markdiffusion.sh`
  780. on the host instead. Model downloads still hit the HF hub on first run.
  781. ## Coverage matrix
  782. | Channel | Claude | Gemini/SynthID | OpenAI | Open-LLM |
  783. | --- | --- | --- | --- | --- |
  784. | Unicode / edit-based text | Layer A | Layer A | Layer A | Layer A |
  785. | **Statistical sampling text** | Layer B best-effort (Claude seam when Anthropic's detection API ships) | Layer B best-effort (+ MarkLLM same-config harness; Google retired the vendor detector Aug 2026) | Layer B if present | Layer B best-effort + optional MarkLLM harness |
  786. | C2PA / file metadata | Yes (listed formats) | Yes when present | Yes when present | Yes when present |
  787. | Pixel image marks | Out of scope | Optional SynthID score + CtrlRegen removal (external); optional MarkDiffusion same-scheme detect + DiffusionPurification removal (external) | Out of scope | Optional CtrlRegen / MarkDiffusion removal (external) |
  788. | Training backdoors | Out of scope | Out of scope | Out of scope | Out of scope |
  789. Details: [`skills/remove-ai-marks/references/vendor-notes.md`](skills/remove-ai-marks/references/vendor-notes.md), [`mark-classes.md`](skills/remove-ai-marks/references/mark-classes.md).
  790. ---
  791. ## How text marking works (short)
  792. Modern LLM watermarks often hide a signal in **which tokens are chosen** (generative / sampling bias), not only in invisible characters. Edit-based schemes inject Unicode or synonym rules. File schemes attach **C2PA** or generator metadata.
  793. - **Layer A** removes edit-based Unicode carriers (testable).
  794. - **Layer B** attacks sampling watermarks via heavy rewrite (best-effort; literature-standard attacks such as paraphrase / back-translation).
  795. - **File cleaners** strip C2PA/XMP/props from supported containers.
  796. Until vendors ship public detectors and keys, **no tool can honestly certify** “this fails the official check.” Reports must separate verifiable vs best-effort work.
  797. Prefer a **non-origin** model for Layer B (do not rewrite Claude text with Claude if you are trying to avoid re-stamping).
  798. ---
  799. ## Disclaimer: what removing a text watermark costs
  800. Text watermarks live in **the wording itself**: the signal is spread across token choices, so nearly every sentence carries a little of it. Two consequences follow, and they are why Layer B is honestly described as *best-effort* rather than a magic eraser.
  801. 1. **Removal means rewording, not restructuring.** Shuffling paragraphs, changing headings, or light touch-ups barely move the signal. Stripping a statistical mark requires rewriting a substantial fraction of the text — sentence by sentence, not section by section.
  802. 2. **Rewording degrades the copy.** Any rewrite replaces the original word choices with the rewriting model's, which flattens tone, voice, and precision. On production copy (SEO, marketing, client work) that degradation is real and often visible to the people who care most about the writing. It is like taking text from a top-tier model and asking a less capable model to rewrite it from scratch: the result cannot exceed the rewrite model's ceiling.
  803. Which leads to the honest full-circle question:
  804. > If the plan is to rewrite the text with a cheaper model anyway, why pay for a premium model in the first place? Generating directly with the cheaper model is simpler, cheaper, and produces the same — or better — end result.
  805. Layer B makes sense when you specifically want the premium model's **thinking and drafting** and accept a rewrite pass to satisfy a hygiene or privacy requirement — not as a cheap route to mark-free text.
  806. **When to skip Layer B:**
  807. - **Quality matters more than hygiene:** use the lossless path — Layer A Unicode scrub plus the file metadata cleaners — and keep the original prose.
  808. - **Rewriting anyway:** use a **non-origin** model (rewriting with the origin model can re-stamp the text), and remember residual risk remains — no tool can certify a vendor detector will fail.
  809. ---
  810. ## File formats
  811. | Format | Inspect | Clean |
  812. | --- | --- | --- |
  813. | PNG / JPEG / WebP | C2PA chunks / APP11 / RIFF `C2PA`, AI XMP hints | Drop metadata segments |
  814. | AVIF / HEIC | ISOBMFF `jumb` / XMP `uuid` boxes | Drop boxes |
  815. | BMP | Trailing non-image bytes (no standardized channel) | Truncate trailing metadata, fix file-size field |
  816. | GIF | Comment / XMP application extensions | Drop comment & XMP, keep `NETSCAPE2.0` loop |
  817. | TIFF (classic + BigTIFF) | IFD tags: XMP, EXIF, GPS, IPTC, MakerNote | Drop tags, zero payloads, keep strips |
  818. | SVG | `<metadata>`, XMP | Strip blocks |
  819. | PDF | Byte/XMP + optional tools | **exiftool** then **qpdf**, then **ghostscript** for metadata inside embedded images; each missing tool degrades a different layer (document strip, structural rewrite, embedded images) |
  820. | DOCX / XLSX / PPTX | docProps / customXml / Office web-extension properties and references | Scrub props, drop customXml and webextensions parts, remove their content-type overrides and dangling relationships; refuse cleaning while an Office owner file is present |
  821. | EPUB | OPF metadata, XHTML meta/JSON-LD, embedded media | Scrub OPF, strip XHTML meta, clean media + Layer A (skips encrypted parts) |
  822. | ODT | meta.xml | Drop generator / AI-ish meta |
  823. | HTML | meta, JSON-LD, data-ai*, `<!-- -->` comments naming an AI tool or marked AI-generated / C2PA / content credential | Strip tags/attrs/comments |
  824. | Markdown | YAML frontmatter AI keys, `<!-- -->` comments naming an AI tool or marked AI-generated / C2PA / content credential (outside code fences) | Drop keys and comments + Layer A body |
  825. | MP4 / MOV / M4A / M4V | ISOBMFF `jumb`/`uuid` boxes (same mechanism as AVIF/HEIC) + `moov/udta` generator tags | Drop boxes |
  826. | WAV | RIFF `C2PA` / `LIST INFO` chunks, embedded `id3\x20` chunk | Drop chunks |
  827. | MP3 | ID3v2 frames (v2.3/v2.4 per-frame; v2.2 whole-tag) | Drop matched frames or whole tag |
  828. | FLAC | C2PA manifest in an ID3v2 `GEOB` frame | Drop the matched frame or whole ID3v2 tag |
  829. FLAC support covers C2PA's standardized ID3v2 carrier. Native FLAC metadata blocks,
  830. Vorbis Comments, and waveform-domain watermarks are left untouched.
  831. #### Why PDF needs qpdf, not just exiftool
  832. ExifTool writes PDFs **incrementally**. `exiftool -all=` appends a
  833. `%BeginExifToolUpdate` block that frees the Info object and drops `/Info` from
  834. the trailer — but the original metadata bytes stay in the file verbatim, and
  835. exiftool itself can undo the edit with `-PDF-update:all=`. The command exits
  836. `0`, viewers show no metadata, and the file gets *larger*, which is the tell.
  837. For a provenance-stripping tool that is a silent leak, so `clean_pdf` follows
  838. the exiftool pass with `qpdf --linearize`, which re-serializes the document
  839. from its object graph and drops the now-unreferenced objects. Without `qpdf`
  840. installed the clean still runs, but it says so:
  841. ```
  842. warning: exiftool PDF edits are incremental — the original metadata bytes
  843. remain recoverable; install qpdf for a structural rewrite
  844. ```
  845. #### Why qpdf is not enough for images inside the PDF
  846. Both passes above work on the document: the Info dictionary, the XMP packet,
  847. the object graph. Neither descends into an image XObject, so a scan or a
  848. Photoshop export — a page that *is* one big JPEG — keeps whatever the image
  849. carries. On a real Photoshop-exported PDF that leaves 27 tags in place after a
  850. "successful" clean, `IFD0:Software`, the capture timestamps and a preview
  851. thumbnail among them; a C2PA manifest attached to the same image survives it
  852. too.
  853. So `clean_pdf` adds a third pass, `deep_images`, driven by Ghostscript's
  854. `pdfwrite`. It runs in two rungs and stops as soon as the file is clean:
  855. 1. **Lossless.** `pdfwrite` with pass-through rebuilds the document from the
  856. object graph while copying the compressed image data byte-for-byte — verified
  857. by hashing the streams before and after. This clears everything the PDF
  858. wrapped around the image. Pass-through covers the codecs Ghostscript supports
  859. for it, JPEG (DCTDecode) and JPEG2000 (JPXDecode); Flate, CCITT and LZW
  860. images are decoded and re-encoded, which is lossless in practice for those
  861. codecs but not byte-identical. `never` is the option for a document whose
  862. streams must survive untouched.
  863. 2. **Re-encode, only on evidence.** Anything living in the JPEG's own APPn
  864. segments — EXIF in APP1, a C2PA manifest in APP11, Photoshop resources in
  865. APP13 — travels with the bytes it is attached to, so pass-through preserves
  866. it. Rung 2 runs the same pass with pass-through off, and only when rung 1
  867. demonstrably left something behind: an AI/C2PA marker in any mode, or, under
  868. `always`, any surviving APPn metadata. APP0 (JFIF) and APP2 (ICC) are left
  869. alone — the first is structural and the second decides how the colours are
  870. read. Pixels are spent on evidence, never on suspicion.
  871. `deep_images` takes `auto` (default: rung 1 only when markers survived the
  872. document strip, then rung 2 if they survive that), `always` (rung 1 for every
  873. PDF, escalating to rung 2 for camera and editor EXIF too), `lossless` (rung 1
  874. only — never recompress, and report whatever survives through the usual
  875. `still_has_c2pa` / `post_findings` fields) and `never`. An unrecognised value is
  876. rejected rather than quietly treated as `auto`. The report says which rungs ran
  877. via `meta.deep_image_pass` and `meta.images_reencoded`, and when the pass is
  878. skipped it names the option that would go further:
  879. ```text
  880. deep image pass not needed for AI/C2PA markers; pass deep_images="always"
  881. to also clear non-AI EXIF inside images
  882. ```
  883. Without Ghostscript installed the clean still runs and says what it could not
  884. reach:
  885. ```text
  886. warning: metadata inside embedded images left in place; install ghostscript
  887. for the deep image pass
  888. ```
  889. Pixel-domain watermark **removal** is now available as an optional external CtrlRegen backend (see above); it is a regenerating remover, not a guarantee. **C2PA soft binding** (in-content watermark that can re-link a remote Content Credentials manifest after metadata is stripped) remains **out of scope**. Stripping hard-bound C2PA does **not** clear those channels.
  890. ### Residual risk after a clean
  891. This tool reports **verifiable** removals (Unicode counts, metadata actions) and **best-effort** Layer B rewrites. It cannot certify that vendor detectors will fail.
  892. To check residual signals yourself (optional, external):
  893. | Channel | What we remove | What may remain | External check (examples) |
  894. | --- | --- | --- | --- |
  895. | Hard-bound C2PA / EXIF / XMP | Yes | Soft-bound / pixel marks | [c2patool](https://github.com/contentauth/c2pa-rs/tree/main/cli), [Content Credentials verify](https://contentcredentials.org/verify) |
  896. | SynthID-class media | Optional pixel removal (external CtrlRegen); local score otherwise | Audio/video watermark; residual pixel watermark after removal | Provider tools (e.g. [Google SynthID](https://deepmind.google/science/synthid/) / Vertex detector where offered); optional local [reverse-SynthID](https://github.com/aloshdenny/reverse-SynthID) scorer |
  897. | Statistical text | Best-effort rewrite | Strong marks after light edit | No public universal detector; vendor tools when available |
  898. Industry two-layer context (C2PA + imperceptible watermark): [Institute of AI PM guide](https://www.institutepm.com/knowledge-hub/ai-content-provenance-watermarking).
  899. ---
  900. ## Watermark Detectors
  901. Vendor-provided checkers for verifying whether content carries AI provenance marks:
  902. - **[Claude: Check if a file was made with Claude](https://claude.com/check-content)** — reads C2PA content credentials in images, video, and audio to report whether Claude was involved in producing the file; runs in-browser. Claude's text-watermark detection API is currently in private preview.
  903. - **[OpenAI: Verify OpenAI-generated content](https://openai.com/research/verify/)** — upload an image or audio file and check for OpenAI provenance signals (C2PA metadata and SynthID watermarks). A programmatic API is also available.
  904. - **[Google DeepMind: SynthID](https://deepmind.google/models/synthid/)** — Google's watermarking technology for AI-generated images, audio, text, and video, with an overview of how the imperceptible marks are embedded and detected.
  905. - **[Gemini: Verify AI-generated images, videos, and audio](https://support.google.com/gemini/answer/16722517)** — Google's guide to verifying files in the Gemini app using SynthID watermarks and Content Credentials, including upload limits and how to read results.
  906. ---
  907. ## Removal options (summary)
  908. | Option | Removes | Notes |
  909. | --- | --- | --- |
  910. | Unicode scrub (Layer A) | ZWSP, bidi, tags, exotic spaces, … | Safe default for text |
  911. | Rewrite (Layer B) | Statistical token marks (best-effort) | Always offered by skill; costs style — see [Disclaimer](#disclaimer-what-removing-a-text-watermark-costs) |
  912. | Container/metadata strip | File provenance | See format table |
  913. | CtrlRegen pixel removal (optional) | Pixel-domain image marks (SynthID-class, StegaStamp, Tree-Ring, StableSignature) | External backend; heavy compute; conservative intensity default |
  914. | DiffusionPurification pixel removal (optional) | Pixel-domain image marks (Tree-Ring-class) | MarkDiffusion backend; blind regeneration (more drift than CtrlRegen); conservative intensity default |
  915. | Open-weight local models | Avoid re-stamping with origin model | Operational alternative |
  916. Matrix: [`skills/remove-ai-marks/references/removal-matrix.md`](skills/remove-ai-marks/references/removal-matrix.md).
  917. ## Ethics and disclaimer
  918. See [`skills/remove-ai-marks/references/ethics.md`](skills/remove-ai-marks/references/ethics.md). For privacy and research on **your** content — not academic fraud or false “human-written” claims.
  919. **Responsible use:** This project is for content you own or are authorized to process. Users must adhere to local regulations and use it responsibly. The developers disclaim any liability for potential misuse by users.
  920. ## Ecosystem
  921. Third-party projects that wrap or complement this repository, listed for discoverability only. **They are not maintained, endorsed, or supported by this project.** This project does not review their code, vouch for their behavior or guarantees, or take responsibility for anything you install or run from this list. Each project is governed by its own license, maintainers, and documentation — read those before using it.
  922. ### MetaClean — desktop GUI
  923. [MetaClean](https://github.com/Moresyl/metaclean) is an independent MIT-licensed Rust/Tauri desktop application (Windows, macOS, Linux) providing a packaged native GUI for drag-and-drop metadata cleaning, with a system tray and Explorer integration. It is a separate codebase: it does not call this repository's Python service, and its supported formats and cleaning guarantees differ from this project's. See its README for details.
  924. ### unmark-web — browser web UI
  925. [unmark-web](https://github.com/ivanusto/unmark-web) is an independent, MIT-licensed static web client. It removes invisible Unicode marks from text and strips provenance metadata from images entirely in the browser, and can optionally call this repository's HTTP service for the formats it does not handle locally. It is a separate codebase and is not affiliated with this project; see its README for scope and limits.
  926. ### DropMarks — macOS GUI
  927. [DropMarks](https://github.com/Nicktili72/dropmarks) is an independent MIT-licensed macOS SwiftUI application. It calls this repository's `inspect_file.py` / `clean_file.py` (and optionally `rewrite_text.py`) via a vendored snapshot of those stdlib scripts. It is a separate codebase and is not affiliated with this project; see its README for scope and limits.
  928. ### unmark-checker — measurement harness
  929. [unmark-checker](https://github.com/Yurakonoplya/unmark-checker) is an independent MIT-licensed Python tool that plants a statistical text watermark of the published SynthID-Text class with a key of your own and scores what a removal left behind. It ships a runner (`integrations/watermarks-remover/run.py`) that hands a marked sample to this repository, either through its `/clean` HTTP service or by calling `clean_text.py` and `rewrite_text.py` in a checkout, and reports the detector score next to how much of the meaning, the facts and the length survived. Which layers ran is part of the result: the `/clean` service runs both layers, while a checkout run always does layer A and only does layer B when `WATERMARKS_REWRITE_BACKEND` is set and `--layer-a-only` is absent. The runner labels every run with the layers it measured, so a layer-A-only score is never read as a full-pipeline one. It only measures; it never removes anything. It is a separate codebase and is not affiliated with this project; see its README for scope and limits.
  930. ### Simple Unmark: privacy-preserving SaaS
  931. [Simple Unmark](https://simpleunmark.com) makes `watermarks-remover` available through a simple web app, currently for text only, with no installation or server setup. The service preserves privacy by default without retaining submitted content. Confidential mode runs the remover in a trusted execution environment (TEE), adding cryptographic verification and hardware-backed protection against infrastructure operators accessing content during processing. An independent project with an [open-source core](https://github.com/SimpleUnmark/confidential). See the [privacy architecture](https://github.com/SimpleUnmark/confidential/blob/main/docs/architecture.md) for guarantees and scope.
  932. ### Adding a project
  933. To register a project here, open a PR adding a short entry — project name, what it wraps or adds, and a link to its own repository. Keep entries brief and factual; do not claim compatibility with, or endorsement by, this project. A listed project should build on or integrate this repository — for example, by calling its service or reusing its detection engine — rather than merely address the same problem independently. Please avoid names that start with or closely resemble `watermarks-remover` — look-alike names make it hard to tell which project is which.
  934. ## Pre-commit hook
  935. CI gating already exists (`audit_dir.py`'s SARIF export, see [Coverage matrix](#coverage-matrix) context) — the [pre-commit](https://pre-commit.com/) hooks below catch the same class of problem earlier, before a marked file is even committed. Both wrap the existing CLIs (`audit_dir.py` / `clean_file.py`) — no separate detection logic.
  936. ```yaml
  937. # .pre-commit-config.yaml
  938. repos:
  939. - repo: https://github.com/guillaumemeyer/watermarks-remover
  940. rev: v0.5.0 # pin to a tag/commit
  941. hooks:
  942. - id: watermarks-remover-check # fails the commit if marks are found
  943. # - id: watermarks-remover-clean # opt-in: cleans staged files in place instead
  944. ```
  945. `watermarks-remover-check` fails the commit and lists findings; `watermarks-remover-clean` is opt-in and rewrites staged files in place (exits 1 so you review the diff and re-stage — the same convention as auto-fixing hooks like `ruff --fix`). When the cleaner cannot process a file at all — it crashed, was killed, or produced no report — `watermarks-remover-clean` names that file and exits 3 instead, so a cleaner that failed is never mistaken for an already-clean file. Run either by hand with `python3 service/scripts/check_staged.py <files...>` / `clean_staged.py <files...>`.
  946. ## Tests
  947. ```bash
  948. python3 -m venv .venv && .venv/bin/pip install pytest
  949. .venv/bin/python -m pytest # or: make test
  950. make smoke # quick CLI smoke on fixtures
  951. ```
  952. ## Changelog
  953. ### [v0.7.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.7.0) — `/clean` Layer B rewrite, watermark-stealing module, audio/video watermark removal, and benchmark/tooling breadth
  954. v0.7.0 brings the Layer B statistical-mark rewrite into the `/clean` service itself, driven by a configurable, benchmark-tuned strategy (`paraphrase@0.8,mlm@0.2`). Alongside it: a black-box watermark-stealing module, destructive audio and per-frame video watermark removal, a substantially richer rewrite benchmark, and a stack of hardening, security, and tooling fixes.
  955. **Layer B rewriting in the service**
  956. - `/clean` runs the Layer B rewrite for text after Layer A. The default comes from `config/clean_strategy.json`; a per-request `options.strategy` overrides it, and `/clean` rejects with 400 when the required backend isn't configured (#315). Config precedence: `--strategy-config` > `WATERMARKS_CLEAN_STRATEGY_FILE` > `config/clean_strategy.json`.
  957. - New `mlm` rewrite tactic: mask a fraction of content words and infill with `roberta-large` — a non-autoregressive local edit, so the output mixes the original token stream with masked-LM predictions (#311).
  958. - The `humanize` tactic now applies the humanizer-skill pass deterministically (straight quotes, no em/en dashes, filler collapses, `utilize`→`use`) and names the human-writer rules in the prompt (#311). `rewrite_text.py` gained a `--strategy` CLI path.
  959. - Rewrite correctness: Unicode word tokenization in lexical divergence (#305); compare raw margins before rounding and record selection metadata / ranked p-values (#249).
  960. **Benchmark**
  961. - SynthID recipe search + robust measurement (#280); renamed rewrite vocabulary, cross-input search, and humanize-last ordering (#302); recommend only strategies that still clear after the humanize polish (#307).
  962. - Pangram bulk API as a human-likeness backend (#296); hardened minimal-rewrite-level benchmark with a 30-doc corpus (#257); validated weight grid + widened recipe search (#294); Polish benchmark corpus (#295).
  963. **Watermark stealing**
  964. - New black-box watermark-stealing module and prompt-corpus downloader (#303); clear stale state on start-over probe failure (#310).
  965. **Audio / video / image**
  966. - Destructive audio watermark removal chain for silentcipher/AudioSeal/WavMark (tempo + pitch + EQ + low-bitrate re-encode → M4A) (#266).
  967. - Per-frame TrustMark video purification that collapses the temporal vote (#265).
  968. - C2PA content-provenance `uuid` box recognized on MP4/MOV/AVIF/HEIC (#264).
  969. - Preserve truncated MP4 tails during stripping (#242); keep the audio re-encode dest distinct from the container-clean dest (#278).
  970. - Skip discarded exiftool output and redundant SynthID in the post-clean scan (#261); degrade cleanly when exiftool can't process a PDF (#281).
  971. - Cap decompressed PNG `zTXt`/`iTXt` at 1 MiB (#308); strip SVG XML DOCTYPE/ENTITY declarations (#288); keep DOCX binary members byte-safe (#314); preserve OOXML `AppVersion` (#289).
  972. **HTTP service & CLI**
  973. - `/clean` option to keep exotic spaces, mirroring the CLI (#274); `/inspect` exposes explicit evidence classes in the suspicious payload (#277); timestamps in HTTP request logs (#256); thread payload bytes into HTTP SynthID scoring and `inspect_*` to avoid a redundant read-back.
  974. - `clean_file.py` gained `-q`/`--quiet`/`--only-changed` (#254).
  975. **Skills, plugin & hooks**
  976. - Stylometry scoring and detector levers for `clean-user-facing-text` (#258); PostToolUse hook launcher made cross-platform (#255); pre-commit hook treats byte-identical clean non-text files as changed (#238).
  977. **Audit**
  978. - `audit_dir.py` scans source, docs, and i18n files the router walked past (#284); scans `.ts/.tsx/.jsx/.gd` and aligns space confidence across formats (#273); `audit_website.py --sarif` support (#194); harden in-place backups, clean-file status, SynthID verdict, truncated ID3v2, and zip routing (#201).
  979. **Security**
  980. - Remove polynomial ReDoS in data-URI and JSON-LD scans (#306); block HTTP redirects in the SynthID scorer to prevent SSRF (#252).
  981. **CI, tooling & docs**
  982. - CI fails when optional backend requirements can't resolve (#301); Docker image reports ffmpeg as usable and installs Ghostscript (#272); dependency bumps (cython #299, scipy #298, ruff #297, docker/setup-buildx-action #237).
  983. - Docs: Watermark Detectors section, ETH SRI "Probing SynthID" blog reference, Ecosystem policy (drop ClaudeWatermarks; require listed projects to use this repo) (#292).
  984. ### [v0.6.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.6.0) — wider format coverage, Layer A hardening, plugin & hook distribution, and detection-guided rewriting
  985. **Format & container coverage**
  986. - **AVIF / HEIC**: native stdlib metadata and C2PA stripping (#84, #85)
  987. - **BMP / GIF / TIFF**: stdlib detection, inspection, and metadata cleaning — GIF comment/XMP extensions are dropped while `NETSCAPE2.0` looping and other animation chunks are preserved; TIFF IFD metadata (XMP/EXIF/GPS/IPTC/MakerNote) is dropped with payloads zeroed and strip offsets kept, for both classic and BigTIFF; BMP trailing metadata is truncated with the file-size field rewritten (#107)
  988. - **EPUB**: stdlib container cleaning — OPF metadata and XHTML meta/JSON-LD scrubbed, embedded raster/SVG media stripped, Layer A applied to XHTML body text, marker-carrying metadata parts dropped, and OCF-encrypted parts passed through untouched (#107)
  989. - **XLSX / PPTX / DOCX (OOXML)**: native stdlib container metadata, text, and embedded media scrubbing; always empty DOCX `docProps` provenance fields; prune dangling relationships after `customXml` removal; run Layer A over DOCX/ODT body text; decode XML entities before Layer A scrub (#91, #100, #76, #83, #73, #80, #74, #81, #142)
  990. - **SGML/vector containers**: linear-time metadata stripping for SVG/ODT (GHSA-7vpp-96qp-j9wh) (#147); recursively inspect and clean embedded raster data URIs in SVGs, HTML, and Markdown (#87, #88)
  991. - **Audio / video**: AI/C2PA metadata stripping for MP4/MOV, WAV, and MP3 (#139); WAV RIFF C2PA chunk detection and removal; FLAC C2PA metadata support; reject partial ID3v2 frame parsing (#232); preserve MP4 media offsets when stripping metadata (#183)
  992. - **PDF**: reach metadata that lives inside embedded images and stop resizing the PDF to strip XMP; run the deep-image pass whether or not exiftool is installed; honour JPEG marker fill bytes and share one segment walker
  993. - **PNG**: detect AI generator product names in PNG text metadata; detect AI markers in compressed PNG text (#127); keep the truncated tail instead of dropping it in png/isobmff strips (#182)
  994. **Layer A (invisible Unicode) hardening**
  995. - **Consolidated Layer A hardening** (#133): strip reserved `Default_Ignorable` code points without a legitimate interchange use (`U+2065`, `U+FFF0`–`U+FFF8`, `U+E0000`, `U+E0080`–`U+E00FF`, `U+E01F0`–`U+E0FFF` — reported as `reserved_ignorable`), the 66 noncharacters (`U+FDD0`–`U+FDEF` plus `U+FFFE`/`U+FFFF` per plane — reported as `noncharacter`), and three blank-rendering Default_Ignorable carriers the `Cf` catch-all never saw (`U+180F`, `U+3164`, `U+FFA0`). Each has the same in-context preservation as its already-covered siblings, so partial-syllable text is not corrupted, and each is applied to both the service engine and the vendored lightweight-skill copy
  996. - **Stop stripping visible-layout format controls next to their own script**: Egyptian hieroglyph quadrat controls (`U+13430`–`U+1343F`), Duployan shorthand controls (`U+1BCA0`–`U+1BCA3`), and musical beam/tie/slur/phrase controls (`U+1D173`–`U+1D17A`) are now preserved when adjacent to their own script and still stripped (and flagged) when floating between unrelated text; `--strip-emoji-glue` paranoid mode still strips them everywhere
  997. - **Emoji / script polish**: preserve VS16 after emoji singletons outside the block ranges; preserve script joiners, flag emoji, and Arabic Cf marks; preserve multilingual Unicode during text cleanup (#34)
  998. **Layer B rewriting & watermark detection**
  999. - **Iterative, detection-guided Layer B rewriting**: each round generates `--candidates` variants (default 1, `WATERMARKS_REWRITE_CANDIDATES`) and `--max-loops` (default 1, `WATERMARKS_REWRITE_LOOPS`) caps the evaluation rounds, stopping as soon as an attempt passes detection. Evaluator priority: MarkLLM (`--markllm-scheme`) > bigram-Jaccard lexical divergence (fallback). `rewrite_text.py --json-stats` now reports `evaluator` / `max_loops` / `attempts_made` / `passed` and per-attempt `candidate_scores` (#153)
  1000. - **Keyed-Gumbel (Aaronson EXP) same-key verification**: new stdlib-only `detect_gumbel.py` implements the model-free replay test (u = PRF(Hash(key, window), token); exact Gamma-tail p-value; repeated-window masking) with no GPU, model, or logits. `rewrite_text.py --gumbel-key` (env `WATERMARKS_GUMBEL_KEY`, preferred) makes it the iterative-loop evaluator (priority: gumbel > markllm > lexical divergence) and it is exposed as `gumbel` in `/capabilities` and `/detect`. Same-key-only — not a vendor oracle; the key is never logged (#190)
  1001. - **Benchmarks**: multi-scheme MarkLLM text benchmark and detection (#188) and a reproducible SynthID-text removal benchmark (#145); default variants `paraphrase:3`; report and CSV carry attempts per document (`mean_attempts` / `att`, `attempts` / `evaluator` / `passed` columns); `--rewrite-loops` mirrors `--max-loops`
  1002. - **Detection**: vendor text-watermark detection (Gemini SynthID, Claude seam, MarkLLM) plus a SynthID image scorer sidecar (#109); new zero-LLM statistical and stylometric AI text detector for CI and audits (#68, #69)
  1003. **Distribution: plugin, hooks, and skill installs**
  1004. - **The repository is now a Claude Code plugin and a single-plugin marketplace** (`.claude-plugin/plugin.json` + `marketplace.json`), so both skills install with `/plugin marketplace add guillaumemeyer/watermarks-remover` then `/plugin install watermarks-remover@watermarks-remover`, and update in place. `make plugin-validate` runs `claude plugin validate . --strict`; `tests/test_plugin_manifest.py` checks the manifests without the CLI
  1005. - **`install_skill.py` grew a `--target`** (`claude-code`, `claude-project`, `cowork`, `cursor`) and a `--skill` selector covering both shipped skills, plus `--list`, `--link`, and `CLAUDE_CONFIG_DIR`. The `cowork` target builds a reproducible upload bundle (`dist/<skill>.zip`, single top-level skill directory); every target validates against the Agent Skills packaging rules and the 30 MB upload limit. New `make` targets: `install-claude-code-skill`, `install-claude-code-text-skill`, `install-claude-project-skill`, `package-cowork-skill`, `package-cowork-text-skill`
  1006. - **Deterministic auto-cleaning via a `PostToolUse` hook** (`hooks/hooks.json` + `service/scripts/hook_written_file.py`): after the agent writes a file the harness runs the hook whether or not the model cooperates. `check` (default) reports marks to the model; `clean` strips them in place and tells the model the file moved, swapping only on a real difference so clean files keep their mtime. Mode comes from the plugin's `hook_mode` setting or `WATERMARKS_HOOK_MODE`; detection reuses `audit_lib.scan_file` / `is_actionable`, so the hook, the pre-commit gate, and the CI SARIF export agree. A hook still cannot rewrite the assistant's chat message — no such hook point exists — so that path stays best-effort
  1007. - **Pre-commit hook integration** for staged-file checking/cleaning (#138); lightweight Cursor text skill (#35); `clean-user-facing-text`'s description no longer names Cursor as the only host
  1008. **HTTP service**
  1009. - Batch endpoints: `POST /clean/batch`, `/inspect/batch` (#137) and `POST /detect/batch` (#151)
  1010. - Preserve image format extensions in `/clean` and use safe writes in `av_meta` (#150); use portable base64 in the `/detect` curl example (and fix the macOS `realpath` portability in the bootstraps, #185)
  1011. **Audit / inspection & security**
  1012. - `audit_dir.py` gained multi-worker concurrency and SARIF 2.1.0 export (#101, #102)
  1013. - Route website binary formats to their real scanners (#177); refuse DTD/entity bombs in the sitemap parser (GHSA-pjg6-92pm-mmcf) (#146); a crashed cleaner blocks the commit instead of reading as clean (#179); an unreadable text file is a failed scan, not a clean one (#169)
  1014. **Reliability & correctness fixes**
  1015. - A second `--in-place` run preserves the original `.bak`; keep collected evidence when a later zip member fails to read (#175); truncated ISOBMFF containers still run the C2PA byte-scan fallback (#176); distinguish a failed cleaner from an already-clean file (#159, #161); treat a failed c2patool run as inconclusive rather than "no C2PA" (#156); validate clean option types (#111); never auto-select MPS device for text watermark detection (#99); macOS portability — pure `--json` stdout for the SynthID scorer and BSD `realpath` probe (#70); fix a Windows `subprocess_creationflags` path in `_ghostscript_usable` and stop child processes from opening a console window on Windows
  1016. - Behavioural hardening: preserve benign JPEG comments keep-mode; fix the `bench-synthid-text` swallowed flag; simplify flag passthrough for the Ghostscript probe and clean_text unneeded noqa (lint)
  1017. **CI / tooling / docs**
  1018. - Ruff linting and formatting with CI enforcement (#103); add macOS to the test matrix (#152); add a CodeRabbit config for automated PR reviews (#222); CODEOWNERS for CODE_OF_CONDUCT/LICENSE and main-review owners; attribute copyright to Guillaume Meyer and contributors (#228)
  1019. - Docs: voice-preserving rewrite guidance and protecting voice/accessibility choices; Ecosystem additions (ClaudeWatermarks, unmark-web) and a note discouraging look-alike names; arXiv 2402.14904 reference; Windows auto-start guide via Task Scheduler; portable base64 in curl examples; pin the vendored Cursor-skill text engine to the service copy (#96)
  1020. ### Unreleased
  1021. - **OOXML add-in metadata**: inspect Office web-extension vendor properties and references in DOCX, XLSX, and PPTX, then remove their package parts and dangling relationships; refuse cleaning while a matching Office owner file exists.
  1022. - Pre-commit clean hook (`watermarks-remover-clean` / `clean_staged.py`): use content digests (`SHA-256`) and active action detection so clean files on disk are recognized without demanding infinite re-staging (#173)
  1023. - **OOXML container preservation**: keep `<AppVersion>` intact in `docProps/app.xml` during DOCX, XLSX, and PPTX metadata cleaning to satisfy ECMA-376 schema constraints and avoid Microsoft Word/Office "unreadable content" errors (#283)
  1024. ### [v0.5.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.5.0) — service & Docker distribution, HTTP API, and verification harnesses
  1025. **Service / Docker distribution**
  1026. - **Skill/service split**: the skill (`skills/remove-ai-marks/`) is now a code-free remote client over HTTP; all implementation moved to `service/scripts/` and runs behind `server.py`, a stdlib HTTP entrypoint (`/health`, `/inspect`, `/clean`, `/capabilities`)
  1027. - **HTTP service**: `service/scripts/server.py` exposes the cleaning pipeline over JSON/base64; hardening mirrors the CLIs (size caps, binary guard, atomic writes, loopback default, optional `WATERMARKS_SERVER_API_KEY` bearer auth)
  1028. - **OpenAPI**: `GET /openapi.json` serves a dynamically generated OpenAPI 3.0.3 spec (built from the route table + live config, so it never drifts from the real endpoints); CI validates it with `openapi-spec-validator`
  1029. - **Core Docker image** (`service/Dockerfile`): full cleaning service with exiftool / qpdf / c2patool preinstalled; any CLI stays runnable by overriding the command
  1030. - **Docker / compose**: `compose.yaml` brings up the whole infra (`core` always; `markllm` / `markdiffusion` behind `profile: harness`; `ctrlregen` / `synthid` behind `profile: heavy` as local-only builds); services are prefixed `wr-`; harness/heavy services default to `command: ["--help"]` so `docker compose up --profile harness --profile heavy` exits cleanly (one-shot CLIs are run with `docker compose run`); new `make compose-check` / `compose-check.sh` validates the running stack (exit code only)
  1031. - **GHCR publishing**: `.github/workflows/release-images.yml` publishes `core`, `markllm`, `markdiffusion` images on `v*` tags; `ctrlregen` / `synthid` are never published (upstream licensing)
  1032. - **Env configuration**: `.env.example` + service configuration guide; `docker compose` auto-loads `.env`; `.env` is gitignored (deny-by-default)
  1033. - **Repo hygiene**: `.gitignore` and `service/.dockerignore` are now deny-by-default — only explicitly allowed paths can be committed or sent in a build context (image contexts only ship `service/scripts/`, which is all the Dockerfiles COPY)
  1034. - Tests: `tests/test_http_server.py` (13 cases) for the HTTP service; all suites re-pointed at `service/scripts/`
  1035. **MarkDiffusion image-watermark harness (optional)**
  1036. - New optional harness (external `THU-BPM/MarkDiffusion`, Apache-2.0): `markdiffusion_harness.py` with `watermark` / `detect` / `purify` subcommands for nine image schemes (Tree-Ring, Ring-ID, ROBIN, WIND, SFW, Gaussian-Shading, GaussMarker, PRC, SEAL)
  1037. - `clean_image.py --remove-pixel diffusion` runs the MarkDiffusion `DiffusionPurification` regeneration attack as an alternative pixel-removal engine (conservative intensity 0.3 default)
  1038. - `setup_markdiffusion.sh` bootstrap (PyPI pin `1.0.2`; `--checkout` editable clone at pinned commit) + `requirements-markdiffusion.txt` + `Dockerfile.markdiffusion` and Makefile `bootstrap-markdiffusion` / `smoke-markdiffusion` / `docker-markdiffusion-build` / `docker-markdiffusion-help`
  1039. - Mock-based tests (`tests/test_markdiffusion_harness.py`) — no torch in CI; `references/markdiffusion.md` reference doc
  1040. - Docs: same-scheme-only verification caveat (not a vendor-detector oracle) and blind-regeneration drift caveat in README, SKILL.md, `removal-matrix.md`, `markdiffusion.md`
  1041. **MarkLLM text-watermark harness (optional)**
  1042. - New optional harness (external `THU-BPM/MarkLLM` checkout, Apache-2.0): `detect_text_watermark.py` with `detect` / `watermark` subcommands for KGW and SynthID schemes
  1043. - `rewrite_text.py --markllm-scheme` runs before/after detection around a Layer B rewrite and per-candidate detection when `--candidates N>1` (env-gated; reports `cleared`)
  1044. - `setup_markllm.sh` bootstrap + `requirements-markllm.txt` (pinned deps) + `Dockerfile.markllm` and Makefile `bootstrap-markllm` / `smoke-markllm` / `docker-markllm-build` / `docker-markllm-help`
  1045. - Hardening: `--offline` cache-only model loading (no HF egress, no remote code), 1 MiB config cap, optional `WATERMARKS_MARKLLM_RLIMIT_AS` on the rewrite subprocess, pinned torch in the Dockerfile, and clone-SHA verification in `Dockerfile.markllm`
  1046. - Mock-based tests (`tests/test_markllm_detect.py`, 21 cases) — no torch in CI; verification-harness caveat (same-config-only, not a vendor-detector oracle) documented in README, SKILL.md, `removal-matrix.md`, `vendor-notes.md`
  1047. **Fixes and polish**
  1048. - **Layer B**: `rewrite_text.py` now sends `reasoning_effort: "none"` by default for `openai-compatible` backends (`--reasoning-effort` / `WATERMARKS_REWRITE_REASONING_EFFORT`; `off` omits it). Reasoning models like `deepseek-v4-flash` otherwise burn ~100s of chain-of-thought on a one-line rewrite (9,894 vs 12 completion tokens)
  1049. - **Fix markllm image build**: `requirements-markllm.txt` pinned `tokenizers==0.23.1`, which conflicts with `transformers==5.15.0` (caps `tokenizers<=0.23.0`; no 0.23.0 release exists) — now pinned `tokenizers==0.22.2`; torch moved to the CPU wheel index (`torch==2.13.0.*`) so the image is CPU-only like `Dockerfile.markdiffusion`
  1050. - **Fix ctrlregen image build**: the 2023-era research pins (`safetensors==0.4.3`, `transformers==4.37.2` → `tokenizers<0.19`) ship no Python 3.14 wheels, so the base image is now `python:3.11-slim` (digest-pinned, multi-arch)
  1051. - **Fix harness images at runtime**: `Dockerfile.markllm` and `Dockerfile.markdiffusion` never copied `common.py` into `/app` (pre-existing bug) — added
  1052. - **WebP**: stdlib-only inspection and metadata cleaning for RIFF `C2PA`, XMP, EXIF, and ICC profile chunks (#37)
  1053. - **BMP / GIF / TIFF**: stdlib-only detection, inspection, and metadata cleaning — GIF comment/XMP extensions are dropped while `NETSCAPE2.0` looping is preserved; TIFF IFD metadata (XMP/EXIF/GPS/IPTC/MakerNote) is dropped with payloads zeroed and strip offsets kept, for both classic and BigTIFF; BMP trailing metadata is truncated with the file-size field rewritten
  1054. - **EPUB**: stdlib-only container cleaning — OPF metadata and XHTML meta/JSON-LD scrubbed, embedded raster/SVG media stripped, Layer A applied to XHTML body text, marker-carrying metadata parts dropped, and OCF-encrypted parts passed through untouched
  1055. - **Filename sanitization**: HTTP service refuses unsafe client-supplied output names
  1056. - **Fix markdown frontmatter cleaner** crashing on and leaking nested AI keys (#25)
  1057. - **Text tools refuse binary input**; `--force-text` overrides (#24)
  1058. - **`--json` no longer suppresses the residual-signal exit code** (#30)
  1059. - **`inspect_file` prints the filename** in its output (#50)
  1060. - **Preserve mixed-case CMS generator meta tags** (#42)
  1061. - **Preserve load-bearing script invisibles, strip PUA** in Layer A (#38, #52)
  1062. - **Preserve script joiners, flag emoji, and Arabic Cf marks** in Layer A (#28)
  1063. - **Harden website audit against SSRF and gzip bombs** (#49)
  1064. - **SECURITY.md** only references the private advisories channel (#51)
  1065. - **Windows**: PowerShell ports of the setup bootstraps (#40)
  1066. - **Docs**: add stars/forks shields and drop star-history chart; add MarkLLM to README references; pull request template; plan for Docker CLI + API deployment
  1067. ### [v0.4.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.4.0) — pixel removal, finding confidence, Windows & false-positive fixes
  1068. **Optional CtrlRegen pixel removal (external backend)**
  1069. - Optional pixel-domain watermark removal via an external `mertizci/noai-watermark` checkout: `clean_ctrlregen.py` adapter + `setup_ctrlregen.sh` bootstrap (pinned commit, sparse checkout, venv, SHA verification), plus `Dockerfile.ctrlregen` and `make bootstrap-ctrlregen` / `docker-ctrlregen-build` / `smoke-ctrlregen`
  1070. - `clean_image.py --remove-pixel ctrlregen` runs metadata strip → CtrlRegen removal → optional reverse-SynthID before/after score; `inspect_image.py` hints at the flag on a high SynthID score
  1071. - Conservative default intensity `0.25` (presets 0.15/0.25/0.35/0.5/0.7); the 512×512-native pipeline is auto-tiled by the backend for larger images; the torch subprocess gets higher env-overridable resource caps
  1072. - Backend is never bundled: `noai-watermark` ships no LICENSE file (treated as all-rights-reserved), and its auto-install/restart code paths are bypassed by using `CtrlRegenEngine` directly
  1073. **Finding confidence and aggregate audits**
  1074. - Findings are now classified `confirmed` / `probable` / `informational` / `likely_false_positive`, exposed in text/image/container JSON and human reports
  1075. - New `audit_dir.py` (recursive tree) and `audit_website.py` (sitemap discovery + crawl) aggregate reports; documented in SKILL.md
  1076. **False-positive fixes**
  1077. - DOCX: scan only `docProps`/`customXml`, not the visible body (#14)
  1078. - Text Layer A: preserve emoji `VS16`/`ZWJ` after an emoji base; new `--strip-emoji-glue` paranoid flag (#22)
  1079. - HTML: treat CMS generator tags as informational, not AI metadata (#13)
  1080. - PDF: exclude stream payloads from the AI-marker byte scan (#13)
  1081. - Inspect reports note unsupported/best-effort paths
  1082. **Windows support**
  1083. - Gate POSIX-only `preexec_fn` and `os.fchmod` so writes and optional tools run on Windows (#15, #23)
  1084. - Reconfigure stdio to UTF-8 so redirected Windows streams no longer raise on invisible Unicode; Windows CI leg + CLI smoke run (#23)
  1085. **Docs and supply chain**
  1086. - README CtrlRegen section + research references (CtrlRegen, UnMarker, forensic-stealth caveat), responsible-use disclaimer; SKILL/matrix/vendor-notes/ethics updates
  1087. - Dependabot config + security-path CODEOWNERS; bump scipy/numpy/opencv-python/scikit-learn/pywavelets and the base image to Python 3.14-slim
  1088. - Mock-based CtrlRegen tests (no torch in CI)
  1089. ### [v0.3.2](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.3.2) — security hardening (safe writes, HTTP client, CI supply chain)
  1090. - **Safe, atomic output writes**: every cleaner now writes via temp-file + atomic rename (`safe_write_bytes` / `safe_write_text`), refuses symlinked destinations, and creates `.bak` backups through the same safe path — pre-placed symlinks (e.g. in `/tmp` or download dirs) can no longer redirect a clean write onto an arbitrary file
  1091. - **`rewrite_text.py` HTTP client hardening**: redirects are refused outright, so an API key in the `Authorization` header can never be re-sent to an unvalidated host; non-loopback endpoints are **denied by default** (opt in with `--allow-remote` or `WATERMARKS_REWRITE_ALLOW_REMOTE=1`); only http(s) schemes are accepted; `--api-key` was removed — keys are env-only via `WATERMARKS_REWRITE_API_KEY`
  1092. - **Resource caps**: default max input 1 GiB → 256 MiB, new 64 MiB stdin cap, DOCX/ODT zip budget 512 MiB → 128 MiB, and `RLIMIT_AS`/`RLIMIT_FSIZE` applied to exiftool/c2patool/SynthID subprocesses (all caps env-overridable)
  1093. - **Supply chain**: CI actions SHA-pinned with `permissions: contents: read`, pinned dev deps (`requirements-dev.txt`), a `pip-audit` step, and a new CodeQL workflow; the Docker image now runs as an unprivileged user with pip pinned
  1094. - **Scorer deps**: Pillow bumped 10.4.0 → 12.3.0 (24 known CVEs); API usage verified against the pinned upstream commit
  1095. - Tests: 18 new security regression tests (60 total, all passing)
  1096. ### [v0.3.1](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.3.1) — stronger Layer B statistical-watermark rewrite
  1097. - `rewrite_text.py` default paraphrase now performs an explicit **word-choice + syntax** attack (clause order, connectors, transition words, sentence boundaries, function words) rather than a generic rewrite
  1098. - New `--tactic humanize`: zero-shot "write like a human" pass targeting formulaic AI-style phrasing
  1099. - New `--tactic code`: rewrites comments, docstrings, and string literals, and renames local identifiers while preserving behavior and public API names
  1100. - Structural pass now emits "natural, varied human prose" instead of AI-typical "clear professional style"
  1101. - New `--temperature` (default `0.9`) for both Ollama and OpenAI-compatible backends
  1102. - New `--candidates N`: generates N rewrites and selects the most lexically diverged (bigram Jaccard distance) with a length-drift guard
  1103. - Stronger model hygiene: prefer local open-weight models and avoid any known-watermarked vendor, not just the suspected origin
  1104. - Residual-risk reporting now distinguishes short/highly predictable text (lower risk) from long, high-entropy prose (higher risk)
  1105. - Docs updated in `SKILL.md`, `removal-matrix.md`, and `vendor-notes.md`; tests cover new prompts, divergence scoring, and candidate selection
  1106. ### [v0.3.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.3.0) — optional SynthID pixel scoring
  1107. - Optional pixel-domain SynthID scorer via an external [`aloshdenny/reverse-SynthID`](https://github.com/aloshdenny/reverse-SynthID) checkout (`score_synthid.py`); surfaced in `inspect_image.py` / `clean_image.py` with `REVERSE_SYNTHID_DIR` or `--synthid-dir`
  1108. - `setup_synthid.sh` bootstrap (scorer-only dependencies; `--full` installs upstream requirements); `Dockerfile.synthid` plus `make docker-synthid-build` / `docker-synthid-help`
  1109. - Makefile `smoke-synthid` and `bootstrap-synthid` targets
  1110. - Tests for the scorer adapter, CLI unavailable path, JSON parsing, and runtime errors
  1111. - Docs: detection/scoring only (no pixel removal); upstream code is not bundled and remains under its non-commercial Research License
  1112. ### [v0.2.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.2.0) — c2patool false-positive fix
  1113. - `image_meta.py`: `has_manifest` no longer flags `Error: No claim found` / `No JUMBF data found` as a manifest (operator-precedence bug: the negative markers now veto every positive branch)
  1114. - New `tests/test_c2patool_report.py` (4 cases: no claim, no JUMBF, genuine manifest, tool absent)
  1115. - Docs: fixed `c2patool` links (repo moved to `contentauth/c2pa-rs`); added a disclaimer on the quality cost of text-watermark removal
  1116. ### [v0.1.0](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.1.0) — packaging polish + provenance honesty
  1117. - `Makefile` (`test` / `smoke` / `install-skill`) and `pytest.ini`
  1118. - Fixture samples for Markdown, HTML, SVG; PDF degraded-clean test
  1119. - Docs: industry **two-layer** model (hard-bound C2PA vs soft binding / SynthID-media)
  1120. - README residual-risk table + links to external verify tools
  1121. - Reference: Institute of AI PM C2PA/SynthID guide
  1122. - Soft-binding and pixel/audio/video watermarks explicitly out of scope in skill/matrix/ethics
  1123. ### [v0.0.1](https://github.com/guillaumemeyer/watermarks-remover/releases/tag/v0.0.1) — initial multi-vendor release
  1124. - Agent skill `remove-ai-marks` (replaces Claude-only `remove-claude-marks`)
  1125. - **Layer A:** invisible Unicode / bidi / tag chars / space homoglyphs (`inspect_text` / `clean_text`)
  1126. - **Layer B:** rewrite guidance + optional `rewrite_text.py` (print-prompt, Ollama, OpenAI-compatible)
  1127. - **Files:** C2PA/AI metadata strip for PNG, JPEG, SVG, PDF, DOCX, ODT, HTML, Markdown
  1128. - Unified `inspect_file.py` / `clean_file.py`
  1129. - Multi-vendor docs (Claude, Gemini/SynthID-class, OpenAI, open-LLM)
  1130. - Stdlib-first scripts; optional `c2patool` / `exiftool`
  1131. ## License
  1132. MIT — see [LICENSE](LICENSE).
  1133. ## Bibliography
  1134. - [How Claude marks AI-generated content](https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content) (Anthropic)
  1135. - Dathathri et al., [*Scalable watermarking for identifying large language model outputs*](https://www.nature.com/articles/s41586-024-08025-4) (SynthID-Text, Nature 2024)
  1136. - Google AI for Developers, [*SynthID safeguards*](https://ai.google.dev/responsible/docs/safeguards/synthid) (Gemini API docs)
  1137. - [C2PA](https://c2pa.org/) / [c2patool](https://github.com/contentauth/c2pa-rs/tree/main/cli)
  1138. - Kirchenbauer et al., [*A Watermark for Large Language Models*](https://arxiv.org/abs/2301.10226)
  1139. - Evseev, D. (Arbitration City), [*Accurate, Costless, and Invisible AI Text Watermarking for Self-Hosted AI Inference*](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/) (technical report, August 2026) — keyed-Gumbel watermarking shipped in the open-source arbi-serve engine, with exact-test detection and speculative-decoding support — [PDF](https://arbicity.com/news/ai-text-watermarking-for-self-hosted-ai/ARBI-Watermark-Technical-Paper.pdf)
  1140. - [THU-BPM/MarkLLM](https://github.com/THU-BPM/MarkLLM) (unified toolkit for evaluating LLM watermarking algorithms)
  1141. - Pan et al., [*MarkDiffusion: An Open-Source Toolkit for Generative Watermarking of Latent Diffusion Models*](https://arxiv.org/abs/2509.10569) (JMLR) — the embedding toolkit this repo's optional image-watermark harness wraps — [code](https://github.com/THU-BPM/MarkDiffusion), [docs](https://markdiffusion.readthedocs.io)
  1142. - Zhang et al., [*Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models*](https://arxiv.org/abs/2311.04378v5) (ICML 2024)
  1143. - Sander et al., [*Watermarking Makes Language Models Radioactive*](https://arxiv.org/abs/2402.14904) — watermarks survive fine-tuning and mark downstream models trained on watermarked data
  1144. - Pan et al., [*Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?*](https://arxiv.org/abs/2502.11598) — watermark-based provenance and protection against knowledge distillation
  1145. - [google-deepmind/synthid-text](https://github.com/google-deepmind/synthid-text) (research reference; not used for detection here)
  1146. - [aloshdenny/reverse-SynthID](https://github.com/aloshdenny/reverse-SynthID) (research reference)
  1147. - ETH Zurich SRI, [*Probing SynthID*](https://www.sri.inf.ethz.ch/blog/probingsynthid) (research blog on the detectability of SynthID watermarks)
  1148. - Liu et al., [*Image Watermarks are Removable Using Controllable Regeneration from Clean Noise*](https://arxiv.org/abs/2410.05470) (ICLR 2025) — the pixel-regeneration method the optional CtrlRegen backend implements — [code](https://github.com/yepengliu/CtrlRegen)
  1149. - Kassis & Hengartner, [*UnMarker: A Universal Attack on Defensive Image Watermarking*](https://arxiv.org/abs/2405.08363) (arXiv:2405.08363; IEEE S&P 2025) — a universal watermark attack compared on a different metric than CtrlRegen
  1150. - Goonatilake & Ateniese, [*Removing the Watermark Is Not Enough: Forensic Stealth in Generative-AI Watermark Removal*](https://arxiv.org/abs/2605.09203) (arXiv:2605.09203) — motivates the conservative-intensity default: removal can still leave forensic traces
  1151. - [mertizci/noai-watermark](https://github.com/mertizci/noai-watermark) (CLI/Python toolkit for SynthID/StableSignature/TreeRing removal and AI metadata stripping)
  1152. - [0xROOTPLS/DeSynth](https://github.com/0xROOTPLS/DeSynth) (SynthID removal for OpenAI/Google images)
  1153. - Institute of AI PM, [*AI Content Provenance and Watermarking: The PM's Guide to C2PA and SynthID*](https://www.institutepm.com/knowledge-hub/ai-content-provenance-watermarking) (two-layer industry model: C2PA + imperceptible watermark / soft binding; SB 942 / EU AI Act Art. 50 context)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.