Skip to content
Back to search

book-to-skill

SkillWorks with Claude CodeGitHub

"Converts books and documents (PDF, EPUB, DOCX, HTML, Markdown, plain text, RTF, MOBI/AZW with Calibre) into structured agent skills, extracting frameworks, mental models, principles, techniques, and anti-patterns. Use when the user wants to study a document through GitHub Copilot CLI, Amp, Claude Code, Hermes Agent, OpenCode, or OpenClaw, apply an author's frameworks while working, or build a reusable knowledge base from a file."

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/virgiliojr94-book-to-skill/badge.svg)](/listings/virgiliojr94-book-to-skill)

HTML

<a href="/listings/virgiliojr94-book-to-skill"><img src="/api/listings/virgiliojr94-book-to-skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# One command, any host — via the cross-agent skills CLI:
npx skills add virgiliojr94/book-to-skill

# Or manually — clone into your skills folder (registers /book-to-skill):
git clone https://github.com/virgiliojr94/book-to-skill.git ~/.claude/skills/book-to-skill
# (Copilot CLI: ~/.copilot/skills/ · Amp/cross-agent: ~/.agents/skills/)
# (Hermes Agent: ${HERMES_HOME:-$HOME/.hermes}/skills/<category>/)
# (OpenClaw: ${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/skills/; ~/.agents/skills/ only with default state)
# (OpenCode: ~/.agents/skills/ — it also reads ~/.config/opencode/skills/)

📥 **All hosts, optional extractors, and the standalone CLI → [docs/install.md](docs/install.md)**

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 880 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: book-to-skill
  3. description: "Converts books and documents (PDF, EPUB, DOCX, HTML, Markdown, plain text, RTF, MOBI/AZW with Calibre) into structured agent skills, extracting frameworks, mental models, principles, techniques, and anti-patterns. Use when the user wants to study a document through GitHub Copilot CLI, Amp, Claude Code, Hermes Agent, OpenCode, or OpenClaw, apply an author's frameworks while working, or build a reusable knowledge base from a file."
  4. ---
  5. <!--
  6. Cross-agent notes (informational; ignored by host agents):
  7. - Compatible skill roots: GitHub Copilot CLI (~/.copilot/skills, ~/.agents/skills,
  8. .github/skills, .claude/skills, .agents/skills), Amp (.agents/skills,
  9. ~/.config/agents/skills, ~/.config/amp/skills), Claude Code (~/.claude/skills),
  10. Hermes Agent ($HERMES_HOME/skills, .hermes/skills, .agents/skills),
  11. OpenClaw (${OPENCLAW_STATE_DIR:-~/.openclaw}/skills, .agents/skills, skills/;
  12. ~/.agents/skills only with the default state).
  13. OpenCode (~/.config/opencode/skills, ~/.claude/skills, .opencode/skills,
  14. ~/.agents/skills, .agents/skills).
  15. - `allowed-tools` is intentionally omitted to stay agent-neutral: Copilot CLI uses
  16. `shell`/MCP-server names, Claude uses `Bash`/`Read`/`Write`/`Glob`/`Grep`, Amp
  17. adds `shell_command`. The skill needs shell (to run extract.py) and file
  18. read/write — each host will prompt for those on first use.
  19. - Argument hint: <path-to-document-folder-or-glob>... [skill-name-slug]
  20. -->
  21. # Book-to-Skill Converter
  22. Transform written knowledge into actionable agent skills by extracting structure — not producing summaries.
  23. ## Philosophy
  24. Books contain crystallized expertise: frameworks, principles, and techniques that took years to develop. This skill extracts that knowledge into a format GitHub Copilot CLI, Amp, Claude Code, Hermes Agent, OpenCode, OpenClaw, or another compatible agent can leverage repeatedly.
  25. **Extract structure, not summaries.** A skill isn't a book report. It's a toolkit of:
  26. - Named frameworks (mental models with clear application)
  27. - Actionable principles (rules that guide decisions)
  28. - Techniques (step-by-step methods)
  29. - Anti-patterns (what to avoid and why)
  30. - Voice calibration (how the author thinks and communicates)
  31. **Preserve the author's precision.** Frameworks often have specific names for reasons. "The 5 Whys" isn't interchangeable with "ask why multiple times." Capture the exact formulation.
  32. **Layer depth appropriately.** Simple books → simple skills. Complex books with 10+ frameworks → skills with reference files and on-demand chapters.
  33. ---
  34. ## Modes of Operation
  35. Four paths available. Route based on what the user asks:
  36. ### 1. Full Conversion (Default)
  37. **Trigger:** User provides one or more document/directory/glob paths without special instructions
  38. **Action:** Run all steps below (Steps 0–9)
  39. **Output:** Complete skill with SKILL.md, chapters/, glossary, patterns, cheatsheet
  40. ### 2. Analyze Only
  41. **Trigger:** User says "analyze", "just extract", or "I want to review before generating"
  42. **Action:** Run Steps 0–3, then produce a structured extraction report (frameworks, principles, techniques found). Stop — do NOT generate skill files.
  43. **Output:** Analysis report for user review
  44. ### 3. Generate from Prior Analysis
  45. **Trigger:** User has existing analysis notes or previously ran analyze-only
  46. **Action:** Skip Steps 0–3, use the provided analysis as input, run Steps 4–9
  47. **Output:** Skill files from the provided analysis
  48. ### 4. Update / Fold-in (Existing Skill)
  49. **Trigger:** User provides one or more new source paths and indicates they want to update an existing skill (either by pointing to the existing skill folder, providing a skill slug that already exists in `SKILLS_HOME`, or explicitly requesting an update).
  50. **Action:** Run Step 0 (out-of-scope check), Step 1 (validate inputs), Step 1.5 (identify book type), and Step 2 (extract new files). Then skip to Step 5 (identify/detect existing skill path) and run the **Update / Fold-in Workflow** to merge the new content into the existing skill files.
  51. **Output:** Updated existing skill with new/revised chapter summaries and merged indexes/glossaries.
  52. ---
  53. ## Skill Locations
  54. This converter can run from multiple skill systems. When looking for this converter's helper script or writing the generated book skill, prefer these locations in order:
  55. 1. GitHub Copilot CLI personal skills: `~/.copilot/skills/`
  56. 2. Cross-agent personal skills (Copilot, Amp, Codex; OpenClaw with its default state): `~/.agents/skills/`
  57. 3. Claude Code personal skills: `~/.claude/skills/`
  58. 4. Project-local Copilot skills: `.github/skills/`
  59. 5. Project-local Claude skills: `.claude/skills/`
  60. 6. Project-local Amp / Copilot / OpenClaw skills: `.agents/skills/`
  61. 7. Amp global skills: `~/.config/agents/skills/`
  62. 8. Amp legacy global skills: `~/.config/amp/skills/`
  63. 9. Hermes Agent personal skills: `$HERMES_HOME/skills/` (defaults to `~/.hermes/skills/`)
  64. 10. Hermes Agent project skills: `.hermes/skills/` or `.agents/skills/`
  65. 11. OpenClaw personal skills: `${OPENCLAW_STATE_DIR:-~/.openclaw}/skills/` (active state; `~/.agents/skills/` is shared only with the default state)
  66. 12. OpenClaw project skills: `.agents/skills/` or `skills/`
  67. 11. OpenCode personal skills: `~/.config/opencode/skills/` or `~/.agents/skills/`
  68. 12. OpenCode project skills: `.opencode/skills/` or `.agents/skills/`
  69. For **generated** book skills, prefer the user-level cross-agent root `~/.agents/skills/` — one physical copy serves the cross-agent hosts and OpenClaw when it uses its default state. Copilot CLI and Amp discover it natively; Claude Code needs a symlink from `~/.claude/skills/<skill_name>` (created in Step 10, see Step 5 for the rules). Pick a host-private or project-local root only when the user explicitly asks for one. `BOOK_TO_SKILL_SCOPE=project` or `personal` can make that choice explicit for automation; do not ask a mandatory scope question merely because both scopes are available.
  70. ---
  71. ## Step 0 — Out-of-scope check
  72. If no arguments are provided, stop and respond:
  73. > "book-to-skill requires a supported document path, folder, or glob pattern. Usage: `book-to-skill <path-to-document-folder-or-glob>... [skill-name-slug]`"
  74. Throughout the workflow:
  75. - **Resolve the destination root first.** Before any existing-skill lookup, determine where this run would write: the personal or project-local root for the user's host, per the Step 5 table and selection rules — a project-local request uses its project-local root, never the personal root. Below, "the resolved root" means that destination.
  76. - Identify the input paths and the optional skill slug.
  77. - If the last argument is not a file, folder, or glob that exists or matches any files, and it looks like a skill slug (e.g. lowercase hyphens, alphanumeric), treat it as `SKILL_NAME`.
  78. - Treat all other arguments as the list of `INPUT_PATHS`.
  79. - If any input path is an existing skill directory (contains `SKILL.md` and a `chapters/` sub-folder), or if `SKILL_NAME` matches an existing skill slug in the resolved root, flag this run as an **Update/Fold-in** operation (Mode 4).
  80. - **Re-run guard.** Before starting extraction for a Full Conversion, derive the prospective skill slug under Step 5's naming options — the author-concept form, the by-title form, and any `SKILL_NAME` given — and check the resolved root for an existing match. On a match, STOP and ask the user: "`<skill-name>` already exists. Choose: (1) Update/Fold-in (Mode 4), (2) verify the existing skill is complete and stop, or (3) force full regeneration." Do not re-extract until the user chooses. On context recovery after an interrupted run (network cut-off, replayed or continued conversation), reuse the extraction work directory that run reported (`Workdir ->` in its output, or the path you set in `BOOK_SKILL_WORKDIR`) **only if it is still intact and still matches the current inputs and options** — the directory exists, its `metadata.json` is readable, the `sources` it lists match the files being converted now one-to-one on filename **and content fingerprint** (the extractor records a per-source `sha256` in `metadata.json`; recompute the fingerprint from each file as it exists now and require an exact match — `reuse_is_safe()` in `book_to_skill/utils.py` is the executable form of this check), and its recorded `extraction_mode` is the mode this run is using (Step 2's "confirm the extraction is the document you asked for" rule). If it is missing (temp cleanup), its `metadata.json` is gone, the sources no longer match (a different filename, or a changed content fingerprint), the recorded metadata has no `sha256` for a source (recorded before fingerprints existed, so freshness cannot be established), or the mode differs, say so and start a fresh extraction instead of resuming. When in doubt, ask the user before discarding or resuming.
  81. ---
  82. ## Step 1 — Validate input
  83. Verify that there is at least one supported file, directory, or glob pattern among the `INPUT_PATHS`.
  84. For directories and globs, expand them to find matching supported files (`.pdf`, `.epub`, `.docx`, `.txt`, `.md`, `.markdown`, `.rst`, `.adoc`, `.html`, `.htm`, `.rtf`, `.mobi`, `.azw`, `.azw3`).
  85. If no supported files are found, stop with a clear error message.
  86. ---
  87. ## Step 1.5 — Identify content type
  88. Before extracting, ask the user:
  89. > "What kind of content do these sources have? This helps me choose the best extraction method.
  90. >
  91. > 1. **Technical** — has code blocks, tables, formulas, diagrams (e.g. programming books, academic papers, architecture guides)
  92. > 2. **Text-heavy** — mostly prose, few or no tables/code (e.g. management, productivity, narrative non-fiction)
  93. > 3. **Not sure** — I'll use the fast method and warn you if quality seems limited"
  94. Store the answer as `BOOK_TYPE`:
  95. - Option 1 → `BOOK_TYPE=technical`
  96. - Option 2 → `BOOK_TYPE=text`
  97. - Option 3 → `BOOK_TYPE=text`
  98. **If `BOOK_TYPE=technical`**, inform the user before proceeding:
  99. > "📐 Technical mode selected — PDFs attempt Docling for layout-aware extraction. EPUBs and other formats use their existing text extractors; code, tables, and formulas may lose structure. Check each source's `extraction_method` and `fallback_reason` in metadata.json for the actual extraction path."
  100. **If `BOOK_TYPE=text`**, inform:
  101. > "📄 Text mode selected — using the fastest suitable extractor for each file type. Plain text/Markdown/HTML are usually ready in seconds; PDFs use pdftotext when available."
  102. ---
  103. ## Step 2 — Extract text from the source documents
  104. Run the extraction script, passing the input paths:
  105. ```bash
  106. SCRIPT_PATH=""
  107. HERMES_HOME_RESOLVED="${HERMES_HOME:-$HOME/.hermes}"
  108. OPENCLAW_STATE_DIR_RESOLVED="${OPENCLAW_STATE_DIR:-$HOME/.openclaw}"
  109. PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)"
  110. HERMES_PROJECT_TRUSTED=false
  111. if [ -n "$PROJECT_ROOT" ] && [ "${HERMES_AGENT:-}" = true ] && \
  112. command -v hermes >/dev/null 2>&1 && \
  113. command -v python3 >/dev/null 2>&1 && \
  114. hermes config get skills.trusted_project_dirs --json 2>/dev/null | PROJECT_ROOT="$PROJECT_ROOT" python3 -c 'import json, os, pathlib, sys; root=pathlib.Path(os.environ["PROJECT_ROOT"]).resolve(); sys.exit(not any(pathlib.Path(p).expanduser().resolve() == root for p in json.load(sys.stdin)))' 2>/dev/null
  115. then
  116. HERMES_PROJECT_TRUSTED=true
  117. fi
  118. CANDIDATES=(
  119. "$HOME/.copilot/skills/book-to-skill/scripts/extract.py"
  120. "$HOME/.agents/skills/book-to-skill/scripts/extract.py"
  121. "$HOME/.claude/skills/book-to-skill/scripts/extract.py"
  122. "${OPENCLAW_STATE_DIR_RESOLVED}/skills/book-to-skill/scripts/extract.py"
  123. "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/book-to-skill/scripts/extract.py
  124. "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/book-to-skill/scripts/extract.py
  125. "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/*/book-to-skill/scripts/extract.py
  126. "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/*/*/book-to-skill/scripts/extract.py
  127. "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/*/*/*/book-to-skill/scripts/extract.py
  128. "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/*/*/*/*/book-to-skill/scripts/extract.py
  129. "$HOME/.config/opencode/skills/book-to-skill/scripts/extract.py"
  130. "$HERMES_HOME_RESOLVED/skills/book-to-skill/scripts/extract.py"
  131. "$HERMES_HOME_RESOLVED"/skills/*/book-to-skill/scripts/extract.py
  132. )
  133. if [ "${HERMES_AGENT:-}" != true ]; then
  134. # Project-local roots are resolved against the git worktree, not the current
  135. # directory: an agent can be invoked from anywhere inside the project (e.g.
  136. # `src/nested`) and these roots still have to be found — that is how OpenCode
  137. # and the other hosts discover project skills. The CWD-relative forms are kept
  138. # so the probe also works outside a git repository.
  139. if [ -n "$PROJECT_ROOT" ]; then
  140. CANDIDATES+=(
  141. "$PROJECT_ROOT/.github/skills/book-to-skill/scripts/extract.py"
  142. "$PROJECT_ROOT/.claude/skills/book-to-skill/scripts/extract.py"
  143. "$PROJECT_ROOT/.agents/skills/book-to-skill/scripts/extract.py"
  144. "$PROJECT_ROOT/.opencode/skills/book-to-skill/scripts/extract.py"
  145. )
  146. fi
  147. CANDIDATES+=(
  148. ".github/skills/book-to-skill/scripts/extract.py"
  149. ".claude/skills/book-to-skill/scripts/extract.py"
  150. ".agents/skills/book-to-skill/scripts/extract.py"
  151. "skills/book-to-skill/scripts/extract.py"
  152. "skills"/*/book-to-skill/scripts/extract.py
  153. "skills"/*/*/book-to-skill/scripts/extract.py
  154. "skills"/*/*/*/book-to-skill/scripts/extract.py
  155. "skills"/*/*/*/*/book-to-skill/scripts/extract.py
  156. "skills"/*/*/*/*/*/book-to-skill/scripts/extract.py
  157. "skills"/*/*/*/*/*/*/book-to-skill/scripts/extract.py
  158. ".opencode/skills/book-to-skill/scripts/extract.py"
  159. )
  160. if [ -n "$PROJECT_ROOT" ]; then
  161. CANDIDATES+=(
  162. "$PROJECT_ROOT/skills/book-to-skill/scripts/extract.py"
  163. "$PROJECT_ROOT/skills"/*/book-to-skill/scripts/extract.py
  164. "$PROJECT_ROOT/skills"/*/*/book-to-skill/scripts/extract.py
  165. "$PROJECT_ROOT/skills"/*/*/*/book-to-skill/scripts/extract.py
  166. "$PROJECT_ROOT/skills"/*/*/*/*/book-to-skill/scripts/extract.py
  167. "$PROJECT_ROOT/skills"/*/*/*/*/*/book-to-skill/scripts/extract.py
  168. "$PROJECT_ROOT/skills"/*/*/*/*/*/*/book-to-skill/scripts/extract.py
  169. )
  170. fi
  171. fi
  172. CANDIDATES+=(
  173. "$HOME/.config/agents/skills/book-to-skill/scripts/extract.py"
  174. "$HOME/.config/amp/skills/book-to-skill/scripts/extract.py"
  175. )
  176. if [ "$HERMES_PROJECT_TRUSTED" = true ]; then
  177. CANDIDATES=(
  178. "$PROJECT_ROOT/.hermes/skills/book-to-skill/scripts/extract.py"
  179. "$PROJECT_ROOT/.hermes/skills"/*/book-to-skill/scripts/extract.py
  180. "$PROJECT_ROOT/.agents/skills/book-to-skill/scripts/extract.py"
  181. "$PROJECT_ROOT/.agents/skills"/*/book-to-skill/scripts/extract.py
  182. "${CANDIDATES[@]}"
  183. )
  184. fi
  185. for candidate in "${CANDIDATES[@]}"
  186. do
  187. if [ -f "$candidate" ]; then
  188. SCRIPT_PATH="$candidate"
  189. break
  190. fi
  191. done
  192. if [ -z "$SCRIPT_PATH" ]; then
  193. echo "Could not find scripts/extract.py for book-to-skill" >&2
  194. exit 1
  195. fi
  196. PYTHON_BIN="${PYTHON_BIN:-python3}"
  197. if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
  198. PYTHON_BIN="python"
  199. fi
  200. "$PYTHON_BIN" "$SCRIPT_PATH" $INPUT_PATHS --mode <BOOK_TYPE> --install-missing ask
  201. ```
  202. Before extraction, the script checks optional Python packages needed for the detected format. If a better extractor is missing, it prompts the user with the available fallback. Non-interactive sessions default to fallback unless install mode is explicitly `yes`.
  203. **Tip — preflight the environment:** run `"$PYTHON_BIN" "$SCRIPT_PATH" --check` to print a per-format report of which extractors are installed and the exact command to install whatever is missing, without processing any file. Useful when a user reports a setup or quality problem.
  204. This creates a **per-run** work directory — `<tempdir>/book_skill_work-<pid>/` by default, or exactly the path you set in `BOOK_SKILL_WORKDIR` — containing:
  205. - `full_text.txt` — combined extracted text of all sources with clear visually demarcated boundaries.
  206. - `metadata.json` — overall combined size, words, pages, token counts, dropped EPUB image counts, the resolved `workdir`, and a detailed list of individual processed `sources`.
  207. The run prints all three paths on completion (`Workdir ->`, `Text ->`, `Meta ->`). **Take the paths from that output (or from `metadata.json`'s own `workdir` field) rather than assuming a fixed location** — the directory name differs per run so that concurrent extractions on one machine cannot overwrite each other's results.
  208. Read that run's `metadata.json` to inspect the results.
  209. **Always confirm the extraction is the document you asked for** before generating anything: check `filename` / `source_file` in `metadata.json`, or the `SOURCE:` header on the first line of `full_text.txt`. If you are waiting on a background run, wait on *its* specific workdir — polling a shared path can surface a different run's output.
  210. ---
  211. ## Step 2.5 — Pre-flight cost estimate
  212. Read this run's `metadata.json` (the `Meta ->` path from the extraction output) and present the user with an estimate **before doing any generation**:
  213. ```
  214. 📖 Sources detected: <total_sources> source(s)
  215. <list each source filename and format from the sources metadata list>
  216. <if images_dropped > 5: warn that N source images were not read>
  217. 📄 Combined Pages/Sections: ~<N> | Words: ~<N> | Total tokens: ~<N>K
  218. 💰 Estimated token cost (Full Conversion / Update):
  219. Input (reading + prompts): ~<N>K tokens
  220. Output (skill files generated/updated): ~<N>K tokens
  221. Total: ~<N>K tokens
  222. Cost: multiply the token counts above by your model's current
  223. input/output per-1M-token rates (prices and model names change often —
  224. do not hardcode them; quote today's rate and label it as an estimate).
  225. ⏱ Estimated time: ~<N> minutes
  226. 📁 Files to be generated/updated:
  227. SKILL.md + chapter files + glossary + patterns + cheatsheet
  228. ➡ Proceed with Full Conversion / Update? (or type "analyze only" to preview first)
  229. ```
  230. **How to estimate:**
  231. - Input tokens ≈ `estimated_tokens` from metadata × 1.3 (prompts overhead per chapter pass)
  232. - Output tokens ≈ chapters × per-chapter budget + 4,000 (SKILL.md) + 4,500 (glossary + patterns + cheatsheet)
  233. - Per-chapter budget midpoint by `BOOK_TYPE` (DEPTH is decided later in Step 4 and can raise it): `text` ≈ 1,000, `technical` ≈ 1,800. If the user has already indicated reference-only vs deep study, use the matching row of the Step 7 matrix.
  234. - Cost: report the token counts and multiply by the user's current per-1M-token input/output rates. Do NOT hardcode dollar figures — model names and prices change; if you show one, label it an estimate and date it.
  235. Wait for the user to confirm before proceeding. If they say "analyze only", switch to Mode 2.
  236. ---
  237. ## Step 2.6 — REPL-style access for large books (> 50k tokens)
  238. Inspired by the Recursive Language Model (RLM) paradigm: treat `full_text.txt` as a queryable corpus, not a single read. Loading the whole file into context burns budget you will need later for generation.
  239. For books over ~50k tokens, prefer programmatic probes over `Read(full_text.txt)` without bounds:
  240. ```bash
  241. # Size check before any Read
  242. wc -w "$FULL_TEXT_PATH"
  243. # Find chapter offsets without loading the whole file
  244. grep -n -E "^\s*(Chapter|CHAPTER)\s+[0-9]+" "$FULL_TEXT_PATH" | head -40
  245. # Pull only the chapter you need (lines start..end inclusive)
  246. sed -n '<start>,<end>p' "$FULL_TEXT_PATH"
  247. # Verify a framework is actually mentioned before claiming it in SKILL.md
  248. grep -c -i "westrum\|dora" "$FULL_TEXT_PATH"
  249. # Targeted Read with offset/limit avoids dumping the full file
  250. # Read(file_path=full_text.txt, offset=<line>, limit=<lines>)
  251. ```
  252. Use this approach for Step 3 (structure analysis), Step 7 (per-chapter summaries), and Step 8 (glossary / patterns extraction). On books under 50k tokens, a single `Read` is fine.
  253. Why this matters: a 200-page book is ~75k tokens. Re-reading it once per chapter (28 passes) costs ~2M input tokens; using grep + sed to pull only relevant slices keeps generation cost proportional to the output, not the source.
  254. ---
  255. ## Step 3 — Analyze book structure
  256. Read the first 8,000 characters of the extracted `full_text.txt` to identify:
  257. - Book **title** and **author(s)**
  258. - **Chapter structure** (look for "Chapter N", "PART I", numbered headings, table of contents)
  259. - **Core themes** and subject domain
  260. - Approximate number of chapters
  261. Then read the Table of Contents section if present to map all chapters.
  262. **If mode is "Analyze Only":** produce the extraction report now and stop. Structure:
  263. ```
  264. ## Extraction Report — <Title>
  265. ### Author's Core Frameworks
  266. - **<Framework Name>**: <what it is and when to apply>
  267. ### Key Principles
  268. - <Principle>: <actionable rule>
  269. ### Techniques & Methods
  270. - <Technique>: <step-by-step or how-to>
  271. ### Anti-patterns
  272. - <What to avoid>: <why>
  273. ### Suggested Skill Name
  274. `{author-lastname}-{core-concept}` — e.g. `cialdini-influence`
  275. ### Chapters Detected
  276. | # | Title | Main Frameworks |
  277. ```
  278. ---
  279. ## Step 4 — Ask purpose (Full Conversion only)
  280. Before generating, ask the user:
  281. > "What should this skill help you do? (Pick one or more)
  282. > 1. Apply the author's frameworks while working
  283. > 2. Think with the author's mental models
  284. > 3. Reference specific chapters and concepts
  285. > 4. All of the above"
  286. Use the answer to weight what gets highlighted in the SKILL.md Core section.
  287. **Derive `DEPTH` from the answer (no extra prompt):**
  288. - Answer is **only** option 3 (reference) → `DEPTH=reference` — lean, fast-lookup chapters.
  289. - Answer includes option 1, 2, or 4 → `DEPTH=study` — deeper chapters with more worked detail, examples, and reasoning.
  290. `DEPTH` and `BOOK_TYPE` together set the per-chapter token budget in Step 7. Do **not** ask a separate "study vs reference" question — it is inferred here. (In Modes 2/3, where Step 4 is skipped, default `DEPTH=study`.)
  291. ---
  292. ## Step 5 — Determine skill name
  293. If `SKILL_NAME` was provided, use it as the skill slug.
  294. Otherwise, propose two options and let the user choose:
  295. - **By author-concept**: `{author-lastname}-{core-concept}` (e.g. `cialdini-influence`, `meadows-systems`)
  296. - **By title**: lowercase hyphens from book title (e.g. `designing-data-intensive-apps`)
  297. Default to author-concept format if the book has a strong methodological identity.
  298. Choose the destination skill root (`SKILLS_HOME`). First resolve **scope** from an explicit user request or `BOOK_TO_SKILL_SCOPE`, then probe **host**. A request for project-local/project output selects the project-local row; a request for personal/global output selects the personal row. If neither scope is requested, preserve the established personal default (`~/.agents/skills` for non-Hermes hosts). Do not ask a mandatory scope question solely because project-local roots exist. The selected root may still require host approval before writing.
  299. | Host agent | Personal skill root | Project-local root |
  300. |---|---|---|
  301. | **GitHub Copilot CLI** | `~/.agents/skills` (discovered natively) | `.github/skills` → `.claude/skills` → `.agents/skills` |
  302. | **Amp** | `~/.agents/skills` (discovered natively) | `.agents/skills` |
  303. | **OpenAI Codex** | `~/.agents/skills` (discovered natively; follows symlinks) | `.agents/skills` |
  304. | **Hermes Agent** | `$HERMES_HOME/skills/<category>` (defaults to `~/.hermes/skills/<category>`) | `.hermes/skills/<category>` → `.agents/skills` |
  305. | **Claude Code** | `~/.agents/skills` + symlink from `~/.claude/skills/<skill_name>` | `.claude/skills` |
  306. | **OpenClaw** | `${OPENCLAW_STATE_DIR:-~/.openclaw}/skills` (active state; `~/.agents/skills` only with default state) | `.agents/skills` → `skills/` |
  307. Hermes Agent is the one host that keeps its own personal root: it partitions personal skills by category and does not scan the cross-agent root. Use the active profile's `HERMES_HOME` and choose a category that matches the generated skill's subject. Do not construct profile paths manually. If the user selects a project-local Hermes root, run `hermes skills trust <project-root>` after generation and verify discovery with `hermes skills list`; project skills remain unavailable until the project is trusted.
  308. For OpenClaw, use the active state directory's `skills/` root: `${OPENCLAW_STATE_DIR:-~/.openclaw}/skills/`. The shared `~/.agents/skills` compatibility root is discoverable only when `OPENCLAW_STATE_DIR` is unset or the default `~/.openclaw`; with a non-default state, do not claim that OpenClaw will see a shared-root install. Verify discovery with `openclaw skills list` after generation.
  309. | **OpenCode** | `~/.agents/skills` (discovered natively; `~/.config/opencode/skills` and `~/.claude/skills` are read too) | `.opencode/skills` → `.agents/skills` → `.claude/skills` |
  310. Hermes Agent is the one host that keeps its own personal root: it partitions personal skills by category and does not scan the cross-agent root. Use the active profile's `HERMES_HOME` and choose a category that matches the generated skill's subject. Do not construct profile paths manually. If the user selects a project-local Hermes root, run `hermes skills trust <project-root>` after generation and verify discovery with `hermes skills list`; project skills remain unavailable until the project is trusted.
  311. OpenCode scans the cross-agent `~/.agents/skills` root natively, so the generated-skill default above already serves it — no symlink and no trust step. It also reads its own managed `~/.config/opencode/skills` root and the Claude compatibility root `~/.claude/skills`; use the OpenCode-managed root only when the user asks for it. Project skills are discovered by walking up from the working directory to the git worktree, so start a new session after generation if the skill does not appear. (`~/.cache/opencode/skills` is the remote-skill download cache from `skills.urls`, not an authoring root — never write a generated skill there.)
  312. Selection rules:
  313. 1. Personal install: set `SKILLS_HOME` to `~/.agents/skills` (create the directory if missing). One exception, so the default does not invent a convention in someone else's house: if `~/.agents/skills` does not exist **and** the host's private root already contains skills, use the private root instead and say why in the report.
  314. 2. **Claude Code does not scan `~/.agents/skills`** — after generation completes, Step 10 links the skill in with `ln -sfn "$HOME/.agents/skills/<skill_name>" "$HOME/.claude/skills/<skill_name>"`.
  315. 3. **Hermes Agent personal installs use the Hermes row above**, not the cross-agent root, and take no symlink.
  316. 4. If the user explicitly asks for a host-private root (`~/.copilot/skills`, `~/.claude/skills`, `~/.config/agents/skills`, `~/.config/amp/skills`), honor it and skip the symlink.
  317. 5. If the user explicitly asked for project-local output, use the project-local row for their host.
  318. 6. If the choice requires knowing the host (project-local output, the Hermes personal root, the OpenClaw state root, or the Claude Code symlink) and you cannot identify it, ask: "Which agent are you running in — OpenCode, OpenClaw, Hermes Agent, GitHub Copilot CLI, Amp, Codex, or Claude Code?"
  319. 7. For OpenClaw personal output, use `${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/skills`. The shared `~/.agents/skills` root is a valid OpenClaw destination only when `OPENCLAW_STATE_DIR` is unset or equals the default `$HOME/.openclaw`; otherwise use the active state root or a project/extra directory.
  320. 8. If the user explicitly asks for an OpenClaw-managed personal root, use the active state root and verify discovery with `openclaw skills list`.
  321. Set `SKILLS_HOME` to the selected root and check if `$SKILLS_HOME/<skill_name>/` already exists. On Claude Code, also check whether `~/.claude/skills/<skill_name>` exists as a **real directory** (not a symlink) — a previous install may live there; if so, offer to migrate it (move the directory into `~/.agents/skills/` and replace the original path with the symlink) before continuing.
  322. If the skill already exists, prompt the user to choose:
  323. 1. **Update / Fold-in** (Mode 4) — integrate new files/content into the existing skill components.
  324. 2. **Overwrite** — delete and regenerate the skill from scratch.
  325. 3. **Rename** — append `-2` or use a different custom slug.
  326. If the user selects **Update / Fold-in**, proceed immediately to the **Update / Fold-in Workflow** section after Step 2.5 (skipping Steps 3, 4, 6, 7, 8, 9).
  327. ---
  328. ## Step 6 — Create skill directory structure
  329. ```bash
  330. mkdir -p "$SKILLS_HOME/<skill_name>/chapters"
  331. ```
  332. ---
  333. ## Step 7 — Generate chapter summaries
  334. **TOKEN BUDGET RULE — CRITICAL (adaptive):**
  335. The per-chapter budget scales with `BOOK_TYPE` and `DEPTH`. Technical chapters need room for code and tables; study depth needs room for worked reasoning. Pick the budget from this matrix:
  336. | | `DEPTH=reference` | `DEPTH=study` |
  337. |---|---|---|
  338. | `BOOK_TYPE=text` | 800–1,200 tokens | 1,000–1,800 tokens |
  339. | `BOOK_TYPE=technical` | 1,200–1,800 tokens | 2,000–3,000 tokens |
  340. - These are per-file targets, not hard caps — a dense chapter may run over, a thin one under. Density still beats length (Quality Rule #3): never pad to hit a number.
  341. - Files are loaded on-demand, so a larger chapter only costs tokens when that chapter is actually read.
  342. - When in doubt between two cells (e.g. mixed-content book), use the lower budget and let depth come from precision, not volume.
  343. **`DEPTH=study` is earned with content, not a bigger number.** The standard section template (Core Idea → Connects To) naturally lands a dense prose chapter around 700–900 tokens. To reach the study budget *honestly* — not by padding — a study-depth chapter must add concrete material:
  344. - **Reproduce one worked example or artifact** from the chapter (e.g. the example press release, a sample dialogue, a filled-in template, a decision the author walks through) under a `## Worked Example` section. This is the single biggest lever and the main thing a learner returns for.
  345. - **Expand the "How" of each framework** into explicit steps or criteria, not a one-liner.
  346. - **Add a short "Why it works / failure mode" note** to the top 1–2 frameworks.
  347. If a chapter genuinely has no worked example and resists expansion, let it land below the study floor rather than padding — and note that the chapter is thin in its Core Idea. A `reference`-depth chapter, by contrast, deliberately omits worked examples and keeps only the decision-ready essentials.
  348. For EACH chapter/major section identified in Step 3:
  349. Read the corresponding section of the extracted `full_text.txt` using top-level `chapter_ranges` from `metadata.json` when present (`start_line`/`end_line` are 1-based inclusive, for `sed`/`Read`); otherwise grep for chapter headings. (`sources[].chapter_ranges` use the same fields but line numbers in that source's text only, not the consolidated `full_text.txt`.)
  350. Create `$SKILLS_HOME/<skill_name>/chapters/ch<NN>-<slug>.md` using the structure below.
  351. **Adapt emphasis based on `BOOK_TYPE`:**
  352. - `technical` → prioritize "Code Examples", "Reference Tables", and "Commands & APIs" sections; preserve exact syntax
  353. - `text` → prioritize "Frameworks Introduced", "Mental Models", and "Key Takeaways"; skip empty technical sections
  354. ```markdown
  355. # Chapter N: <Full Title>
  356. ## Core Idea
  357. <1–2 sentences: the single most important thing this chapter teaches>
  358. ## Frameworks Introduced
  359. - **<Framework Name>**: <exact formulation — preserve the author's naming>
  360. - When to use: <specific situation>
  361. - How: <steps or criteria>
  362. ## Key Concepts
  363. - **<Term>**: <precise definition in 1 sentence>
  364. (5–10 most important terms from this chapter)
  365. ## Mental Models
  366. <2–4 frameworks or thinking tools. Write as "Use X when Y" or "Think of X as Y">
  367. ## Anti-patterns
  368. - **<What to avoid>**: <why it fails>
  369. ## Code Examples *(technical books only — omit if BOOK_TYPE=text)*
  370. <!-- Copy the most instructive snippet from the chapter. Preserve indentation exactly. -->
  371. ```<language>
  372. <key code example from this chapter>
  373. ```
  374. - **What it demonstrates**: <one line>
  375. ## Reference Tables *(technical books only — omit if BOOK_TYPE=text)*
  376. <!-- Reproduce any comparison matrix, parameter table, or decision table from the chapter in markdown. -->
  377. ## Worked Example *(DEPTH=study only — omit for DEPTH=reference)*
  378. <!-- Reproduce or reconstruct one concrete example the author works through: a
  379. sample document, a dialogue, a filled-in template, a before/after, or a
  380. decision walked end-to-end. This is what makes a study chapter worth its
  381. budget. Keep it faithful to the source; never copy long raw passages —
  382. reconstruct the example compactly. -->
  383. ## Key Takeaways
  384. 1. <Actionable insight>
  385. 2. <Actionable insight>
  386. 3. <Actionable insight>
  387. (3–7 takeaways a practitioner must remember)
  388. ## Connects To
  389. - **Ch N**: <why this chapter relates>
  390. - **<Concept>**: <external concept or standard it connects with>
  391. ```
  392. ---
  393. ## Step 8 — Generate supporting files
  394. ### glossary.md
  395. Create `$SKILLS_HOME/<skill_name>/glossary.md`:
  396. - Every significant term from the book, alphabetically sorted
  397. - Format: `**Term** — definition (Ch N)`
  398. - Max 1,500 tokens
  399. ### patterns.md
  400. Create `$SKILLS_HOME/<skill_name>/patterns.md`:
  401. - All concrete techniques, design patterns, algorithms from the book
  402. - Format: `## Pattern Name\n**When to use**: ...\n**How**: ...\n**Trade-offs**: ...`
  403. - Max 2,000 tokens
  404. ### cheatsheet.md
  405. Create `$SKILLS_HOME/<skill_name>/cheatsheet.md`:
  406. **This is the most differentiated layer of the skill — treat it as a reasoning aid, not a keyword list.** Anyone can grep the glossary for a term. The cheatsheet captures the author's *judgment*: the decisions they'd make and why. It's the file that turns "I know the words" into "I'd act the way the author would".
  407. Prioritize, in order:
  408. 1. **Decision rules** — "When X, do Y, because Z." The if/then logic the author applies, stated so the reader can apply it without re-reading the book.
  409. 2. **Decision trees / flowcharts** (as nested bullets or a small table) — for choices with more than two branches.
  410. 3. **Trade-off matrices** — competing options scored on the dimensions the author cares about, so the reader can pick under their own constraints.
  411. 4. **Thresholds & defaults** — the specific numbers, ratios, or rules of thumb the author commits to (e.g. "keep functions under ~20 lines", "alert when error budget < 10%").
  412. 5. **Tells & smells** — fast heuristics for recognizing a situation ("if you see X, you're probably in trouble Y").
  413. Avoid: bare term→definition rows (that's the glossary), and prose paragraphs (that's the chapters). Every line should help the reader *decide* something.
  414. - Format mostly as compact tables and decision rules; the content you'd want on a single printed page kept beside you while working.
  415. - Max 1,200 tokens.
  416. ---
  417. ## Step 9 — Generate the master SKILL.md
  418. **CRITICAL TOKEN BUDGET: Keep SKILL.md body under 4,000 tokens.**
  419. Compaction truncates from the END — put the most important content FIRST.
  420. Create `$SKILLS_HOME/<skill_name>/SKILL.md`:
  421. ```markdown
  422. ---
  423. name: <skill_name>
  424. description: "Knowledge base from \"<Full Title>\" by <Author(s)>. Use when applying <author>'s frameworks for <key topics, 3–6 terms>, studying the book, or referencing its concepts."
  425. ---
  426. <!-- argument-hint: [topic, framework name, or chapter number] -->
  427. # <Full Title>
  428. **Author**: <Author(s)> | **Pages**: ~<N> | **Chapters**: <N> | **Generated**: <YYYY-MM-DD>
  429. ## How to Use This Skill
  430. - **Without arguments** — load core frameworks for reference
  431. - **With a topic** — ask about `replication`, `pricing`, or another indexed topic; I find and read the relevant chapter
  432. - **With chapter** — ask for `ch05`; I load that specific chapter
  433. - **Browse** — ask "what chapters do you have?" to see the full index
  434. When you ask about a topic not covered in Core Frameworks below, I will read
  435. the relevant chapter file before answering.
  436. ---
  437. ## Core Frameworks & Mental Models
  438. <!-- ~2,000 tokens: the author's most important named frameworks and principles.
  439. Preserve exact names. Write as "Use X when Y", "Prefer X over Y because Z".
  440. This is a toolkit, not a summary. -->
  441. <generate 2,000 tokens of the most critical frameworks and insights here>
  442. ---
  443. ## Chapter Index
  444. | # | Title | Key Frameworks |
  445. |---|-------|----------------|
  446. | [ch01](chapters/ch01-<slug>.md) | <Title> | <framework1>, <framework2> |
  447. | [ch02](chapters/ch02-<slug>.md) | <Title> | <framework1>, <framework2> |
  448. ...
  449. ## Topic Index
  450. <!-- Alphabetical. Major terms/frameworks → chapter(s) that cover them. -->
  451. - **<Term>** → ch<N>[, ch<N>]
  452. - **<Term>** → ch<N>
  453. ## Supporting Files
  454. - [glossary.md](glossary.md) — all key terms with definitions
  455. - [patterns.md](patterns.md) — all techniques and design patterns
  456. - [cheatsheet.md](cheatsheet.md) — quick reference tables and decision guides
  457. - Full text: `<output_text from metadata.json>` — grep it for depth the chapter files omit (local extraction, not shipped)
  458. ---
  459. ## Scope & Limits
  460. This skill covers the book content only. For hands-on implementation in your codebase,
  461. combine with project-specific tools. For topics beyond this book, check related skills
  462. or ask the agent directly.
  463. <if images_dropped > 5: state that N source images were not read>
  464. ```
  465. ---
  466. ## Step 9.5 — Scan the generated skill
  467. Before reporting success, loading the skill in another session, or publishing it, run the advisory security scan:
  468. ```bash
  469. SKILL_CONVERTER_ROOT="$(cd "$(dirname "$SCRIPT_PATH")/.." && pwd)"
  470. "$PYTHON_BIN" "$SKILL_CONVERTER_ROOT/tools/scan_generated_skill.py" "$SKILLS_HOME/<skill_name>"
  471. ```
  472. If the scanner exits non-zero, stop and ask a human to review its file/line findings. Do not silently rewrite the generated files, and do not load or publish the skill until the findings are resolved or explicitly accepted.
  473. ---
  474. ## Step 10 — Cleanup and report
  475. If the host is Claude Code and `SKILLS_HOME` is `~/.agents/skills` (the default personal install), expose the skill to Claude Code with a symlink — Claude Code only scans `~/.claude/skills`:
  476. ```bash
  477. mkdir -p "$HOME/.claude/skills"
  478. LINK="$HOME/.claude/skills/<skill_name>"
  479. TARGET="$HOME/.agents/skills/<skill_name>"
  480. if [ -d "$LINK" ] && [ ! -L "$LINK" ]; then
  481. CLAUDE_STATUS="skipped-realdir" # Step 5 migration declined; leave the old dir
  482. else
  483. ln -sfn "$TARGET" "$LINK" 2>/dev/null || true
  484. # Read the link back — do NOT trust that `ln` did what was asked. On Windows/MSYS
  485. # `ln -s` may COPY instead of link (or need Developer Mode / an elevated shell), and
  486. # PowerShell/cmd have no `ln` at all. The report must reflect what is on disk, not the
  487. # fact that the command ran.
  488. if [ -L "$LINK" ] && [ "$(readlink "$LINK")" = "$TARGET" ]; then
  489. CLAUDE_STATUS="linked"
  490. elif [ -e "$LINK" ]; then
  491. CLAUDE_STATUS="copy" # a real file/dir landed instead of a link
  492. else
  493. CLAUDE_STATUS="absent" # ln unavailable or refused
  494. fi
  495. fi
  496. ```
  497. The real-directory guard is required: `ln -sfn` into an existing real directory would nest the link *inside* it (`~/.claude/skills/<skill_name>/<skill_name>`), leaving Claude Code loading the stale copy. If the user declined the Step 5 migration, skip the symlink and say so in the report — Claude Code keeps using the old directory until it is migrated.
  498. **Read the link back before you report anything about it.** The symlink is a claim, not a fact: fill the "Discoverable by" line from `CLAUDE_STATUS` (what is actually on disk), never from "the command was issued". **Do not hard-fail when the link is missing or is a copy** — the skill exists at the hub and every other host still finds it; the honest report is "written to `~/.agents/skills/<skill_name>`; Claude Code will not see it until the link is created", not an abort. (Windows lead, unverified: a directory junction — `mklink /J` in an elevated `cmd`, or `New-Item -ItemType Junction` in PowerShell — needs neither Developer Mode nor a symlink privilege; if you attempt it, it does not change the read-back-then-report rule.)
  499. Skip this when the user chose a host-private or project-local root (Step 5, rules 3-4).
  500. Then clean up the extraction workdir:
  501. ```bash
  502. PYTHON_BIN="${PYTHON_BIN:-python3}"
  503. if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
  504. PYTHON_BIN="python"
  505. fi
  506. Remove **the work directory this run actually used** — the `Workdir ->` path from the
  507. extraction output, which is also stored as `workdir` in `metadata.json`. Never delete a
  508. directory you did not create: another extraction may be running beside yours.
  509. ```bash
  510. # WORKDIR is the path this run reported; quote it in case of spaces.
  511. rm -rf "$WORKDIR"
  512. ```
  513. Equivalently, if you still have the metadata file:
  514. ```bash
  515. "$PYTHON_BIN" - "$WORKDIR_METADATA_JSON" <<'PY'
  516. import json
  517. import shutil
  518. import sys
  519. from pathlib import Path
  520. meta_path = Path(sys.argv[1])
  521. workdir = json.loads(meta_path.read_text(encoding="utf-8")).get("workdir")
  522. if workdir:
  523. shutil.rmtree(workdir, ignore_errors=True)
  524. PY
  525. ```
  526. Older copies of this file removed a single fixed `book_skill_work` directory. That path is
  527. no longer used, so such a cleanup is now a harmless no-op rather than something that could
  528. delete a concurrent run's output.
  529. Then report to the user:
  530. ```
  531. ✅ Skill created: $SKILLS_HOME/<skill_name>/
  532. 📚 Book: <Full Title> — <Author>
  533. 📄 Pages: ~<N> | Chapters: <N>
  534. Files generated:
  535. SKILL.md — core frameworks + index (~X tokens)
  536. chapters/ — <N> chapter summaries (~X tokens each, ~X total)
  537. glossary.md — key terms (~X tokens)
  538. patterns.md — techniques & patterns (~X tokens)
  539. cheatsheet.md — quick reference (~X tokens)
  540. ─────────────────────────────────────────────────────
  541. Total skill size: ~X tokens (loaded on-demand, not all at once)
  542. 💡 Tip: check your agent's session cost/usage command to see actual token usage.
  543. Usage:
  544. Ask for <skill_name> → load core frameworks
  545. Ask <skill_name> about <topic> → find and explain a topic
  546. Ask <skill_name> for ch<N> → dive into a specific chapter
  547. Discoverable by: <only what is true for the chosen destination — see below>
  548. Somewhere else? mv ~/.agents/skills/<skill_name> <dest_root>/<skill_name> \
  549. && ln -sfn <dest_root>/<skill_name> ~/.claude/skills/<skill_name>
  550. Prompted for permission on every file? That is your host gating writes outside the
  551. working directory. Say "save it in this project" and re-run to write inside it.
  552. Reload (if your agent doesn't auto-detect new skills):
  553. GitHub Copilot CLI: /skills reload
  554. Claude Code: restart the session
  555. Amp: restart the session
  556. Hermes Agent: start a new session
  557. OpenClaw: openclaw skills list (new session if watcher disabled)
  558. OpenCode: start a new session
  559. Share this skill (optional):
  560. GitHub repo, installable on any host (Step 11): say "publish"
  561. Copilot ecosystem: gh skill publish $SKILLS_HOME/<skill_name>
  562. ```
  563. Fill the "Discoverable by" line from `CLAUDE_STATUS` (the read-back result), never from the fact that `ln` ran — for `~/.agents/skills` installs:
  564. - `linked` → "Copilot CLI, Amp, Codex (natively); Claude Code via symlink ~/.claude/skills/<skill_name>"
  565. - `skipped-realdir` → "Copilot CLI, Amp, Codex (natively); **NOT** Claude Code — migrate the real directory at ~/.claude/skills/<skill_name> first"
  566. - `copy` or `absent` → "Copilot CLI, Amp, Codex (natively); **NOT** Claude Code — the host could not create the symlink (a plain copy drifts on the next Update/Fold-in). Enable Developer Mode / create the link manually, or run the skill from ~/.agents/skills"
  567. - Hermes Agent personal root → "Hermes Agent (from `$HERMES_HOME/skills/<category>`)"; no symlink claim, and no cross-agent claim, because the other hosts do not scan the Hermes root
  568. - other host-private or project-local root → name only the host(s) that scan that root; no symlink claim
  569. The "Somewhere else?" relocation line must be correct for the path actually taken, so it never breaks the symlink the run just created. **`mv` always targets the final skill directory, `<dest_root>/<skill_name>`, never `<dest_root>` itself.** `mv ~/.agents/skills/mybook ~/.copilot/skills && ln -sfn ~/.copilot/skills ~/.claude/skills/mybook` reads as valid and is not: the skill lands at `~/.copilot/skills/mybook` while the link points one level up at the root, so Claude Code resolves to a directory with no `SKILL.md`, which is the exact breakage this line exists to avoid. Substitute the destination the user actually named, so the printed command carries real paths and there is nothing left to interpret:
  570. - `~/.agents/skills` + symlink → `mv ~/.agents/skills/<skill_name> <dest_root>/<skill_name> && ln -sfn <dest_root>/<skill_name> ~/.claude/skills/<skill_name>`
  571. - host-private root, Hermes Agent included → `mv <src_root>/<skill_name> <dest_root>/<skill_name>`
  572. - project-local root → `mv <project_root>/<skill_name> <dest_root>/<skill_name>`
  573. The "Prompted for permission on every file?" line is the answer to a host that gates writes outside the working directory (any personal-scope root is out-of-cwd): the destination was announced above, and the one-line fix — re-run asking for the project-local root — sits next to it. Keep it only for personal-scope installs; drop it when the user already chose project-local.
  574. ---
  575. ## Step 11 — Publish the generated skill to GitHub (optional)
  576. After the Step 10 report, offer once — and only if the Step 9.5 scan passed:
  577. > "Want me to publish this skill to GitHub so any Agent Skills host can install it with `npx skills add`? (yes / skip)"
  578. If the user declines, stop here. Requirements: the `gh` CLI, authenticated (check `gh auth status`). If `gh` is missing or unauthenticated, offer to set it up (`brew install gh` or https://cli.github.com, then `gh auth login`) — or use the no-`gh` path: the user creates an empty repo of the chosen visibility in the GitHub web UI, then you run the `git init`/`add`/`commit` commands below followed by `git remote add origin <repo-url> && git push -u origin main`. The visibility rule below applies to the web-created repo exactly the same.
  579. **Visibility is a separate closed question — never inferred, never read out of an earlier answer.** Once the user accepts, ask it on its own and require a one-word reply:
  580. > "Private or public repository? Reply with one word: `private` or `public`."
  581. **The reply must *be* `public`, not merely contain it — a hard rule, not a suggestion.** Run `gh repo create` with `--private` in every case except one: the answer to the visibility question is the bare word `public`. Substring matching is forbidden, because a sentence about **the source's licence is not a visibility answer** — "it's public domain", "the book is public domain", "it's publicly available" all describe the material, not the repository, and all resolve to `--private`. A paraphrase, a sentence, an ambiguous answer, silence, or your own inference is NOT consent: re-ask once, and if the reply is still not the bare word, use `--private` and say so in the report. A private repo can be flipped public later; a public push of book-derived content cannot be un-published.
  582. **Copyright gate — always apply before creating the repo:** chapter files are synthesized summaries, not raw text, but they still derive from the source material. Per the README's Copyright & fair use policy, skills generated from **third-party copyrighted books must stay private**; offer public only when the source is the user's own writing, openly licensed content, or material the user explicitly confirms they are authorized to redistribute publicly — and state which case applies. Having access to internal company material is not permission to disclose it: skills from internal docs stay **private** unless the user states they hold publication rights.
  583. If accepted:
  584. 1. Add a repo `README.md` inside `$SKILLS_HOME/<skill_name>/` (never overwrite an existing file) — the skill title, a one-paragraph description ("Agent skill generated from *<Title>* by <Author> with [book-to-skill](https://github.com/virgiliojr94/book-to-skill)"), the install command from step 3 below, the file inventory, and a note that the content is synthesized summaries, not the book text.
  585. 2. Initialize the skill folder as a git repository and create the remote (default repo name `<skill_name>`; let the user override — some prefer a `<skill_name>-skill` suffix). **Nested-repo guard:** first check whether the skill folder already sits inside a git repository (`git -C "$SKILLS_HOME/<skill_name>" rev-parse --show-toplevel` — always the case for project-local roots like `.claude/skills/`). If it does, do NOT `git init` in place: the outer repository would record the folder as an embedded repo (gitlink, mode 160000) without `.gitmodules`, and fresh clones of the outer project would silently omit the skill. Instead, copy the skill folder to a scratch directory, run the commands below from the copy, and tell the user the published repo — not the project-local folder — is the remote's working copy.
  586. ```bash
  587. cd "$SKILLS_HOME/<skill_name>"
  588. git init -b main
  589. git add -A
  590. git commit -m "Add <skill_name> skill"
  591. gh repo create <repo_name> --private --source . --push
  592. # --private is the default; substitute --public ONLY under the visibility rule above
  593. # (the visibility answer WAS the bare word "public" AND the copyright gate allows it)
  594. ```
  595. 3. Report the repo URL and the cross-host install command:
  596. ```
  597. ✅ Published: https://github.com/<owner>/<repo_name> (<private|public>)
  598. Install on any Agent Skills host:
  599. npx skills add https://github.com/<owner>/<repo_name> --skill <skill_name>
  600. ```
  601. When the nested-repo guard fired and the repo was published from a scratch copy, add one line — that local folder never gains a remote, so the Update/Fold-in push offer will never appear for it:
  602. ```
  603. ⚠️ Published from a copy: <skill folder> sits inside another git repository, so it has
  604. no remote of its own. To publish a later update, re-run Step 11, or clone
  605. https://github.com/<owner>/<repo_name> and fold new material into the clone.
  606. ```
  607. The root-level `SKILL.md` layout is exactly what the `skills` CLI detects, so the repo is installable as-is — no restructuring needed. Outside the nested-repo case the local folder stays the live install for this machine and is the remote's working copy, so later Update/Fold-in runs can commit and push their changes to the same remote.
  608. ---
  609. ## Update / Fold-in Workflow
  610. When performing an Update/Fold-in operation on an existing skill at `$SKILLS_HOME/<skill_name>/`:
  611. ### 1. Read Existing Skill Structure
  612. Read and parse the existing skill's files:
  613. - Read `$SKILLS_HOME/<skill_name>/SKILL.md` to parse the existing **Chapter Index**, **Topic Index**, metadata (author, total chapters), and **Core Frameworks**.
  614. - List all files in `$SKILLS_HOME/<skill_name>/chapters/` to find the highest chapter number (e.g. `ch12`).
  615. - Read `$SKILLS_HOME/<skill_name>/glossary.md`, `$SKILLS_HOME/<skill_name>/patterns.md`, and `$SKILLS_HOME/<skill_name>/cheatsheet.md` to see what terms and frameworks are already indexed.
  616. ### 2. Match Content & Identify Revisions vs. Additions
  617. Analyze the new extracted text in this run's `full_text.txt` (the `Text ->` path from the extraction output) to identify if the new content represents:
  618. - **Updates/Revisions to existing chapters**: If a section of the new content directly updates or expands an existing chapter's topic, read the existing chapter file, merge the new details into it, and rewrite the file.
  619. - **New additions**: If the content introduces new chapters, papers, or separate sections, create **new chapter summary files** under `chapters/`. Start numbering these files after the highest existing chapter number (e.g. if the existing chapters stop at `ch12`, create `ch13-*.md`, `ch14-*.md`, etc.).
  620. ### 3. Generate or Update Chapter Summary Files
  621. For each new or revised chapter:
  622. - Read the corresponding section of the extracted new text.
  623. - Follow the formatting guidelines in **Step 7** to build the summary.
  624. - Write/update the file in `$SKILLS_HOME/<skill_name>/chapters/`.
  625. ### 4. Merge Supporting Files
  626. - **Merge glossary.md**:
  627. - Read the existing `$SKILLS_HOME/<skill_name>/glossary.md`.
  628. - Extract all new terms and definitions from the new content (Step 8 glossary guidelines).
  629. - Combine and alphabetize the list of existing and new terms.
  630. - If a term already exists, append the new chapter/source references to it (e.g. `**Term** — definition (Ch 4, Ch 13)`).
  631. - Rewrite `$SKILLS_HOME/<skill_name>/glossary.md` with the fully merged, alphabetized list.
  632. - **Merge patterns.md**:
  633. - Read existing `$SKILLS_HOME/<skill_name>/patterns.md`.
  634. - Extract any new techniques, algorithms, or patterns from the new content.
  635. - Append the new patterns, ensuring consistent formatting, and keeping the total length concise (under 2,500 tokens).
  636. - **Merge cheatsheet.md**:
  637. - Read existing `$SKILLS_HOME/<skill_name>/cheatsheet.md`.
  638. - Extract new comparison rules, decision tables, or parameter guides.
  639. - Integrate them cleanly into the cheatsheet structure.
  640. ### 5. Re-generate the Master SKILL.md
  641. Update the master skill file `$SKILLS_HOME/<skill_name>/SKILL.md`:
  642. - **Metadata**: Increment the chapter count, update the estimated page count, and add the new source names if appropriate. Update the `Generated` date to the current date.
  643. - **Core Frameworks**: Fold in the most high-impact mental models or principles from the new content (ensuring the overall file remains under 4,000 tokens).
  644. - **Chapter Index**: Append the new chapters to the index table, linking to the newly created files.
  645. - **Topic Index**: Merge the new topics alphabetically. If an existing topic is also covered in the new chapters, append the new chapter links to its line (e.g. `- **Topic** → ch05, ch13`).
  646. ### 6. Scan, Cleanup, and Report
  647. Once the files are successfully written and merged, run **Step 9.5**, then proceed to **Step 10** to perform cleanup and print a custom update report summarizing the newly added chapters, merged glossary terms, and updated indices. If the skill folder is a git repository with a remote (published via **Step 11**), offer to commit the update and push it.
  648. ---
  649. ## Quality Rules
  650. 1. **Extract structure, not summaries** — capture named frameworks, exact formulations, anti-patterns; not chapter recaps
  651. 2. **Preserve the author's precision** — "The 5 Whys" ≠ "ask why multiple times"; keep exact naming
  652. 3. **Density over completeness** — a 1,000-token summary beats a 10,000-token excerpt
  653. 4. **Practitioner voice** — write "Use X when Y", not "The book explains X"
  654. 5. **Front-load SKILL.md** — compaction keeps the first 5,000 tokens; most important content comes first
  655. 6. **Chapter files are on-demand** — they don't count against skill budget until loaded
  656. 7. **Never copy raw book text** — always synthesize, summarize, extract signal
  657. 8. **Topic index is critical** — it's how the agent navigates to the right chapter file
README.md · README · 265 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src="docs/assets/banner.webp" alt="Booklin, the book-to-skill wizard, holding an open book whose pages scatter into sparkles that settle into an ordered grid" width="100%">
  3. </p>
  4. <h1 align="center">book-to-skill</h1>
  5. <p align="center">
  6. <a href="README.md"><strong>English</strong></a> ·
  7. <a href="README.ru.md">Русский</a> ·
  8. <a href="README.zh-CN.md">简体中文</a>
  9. </p>
  10. <p align="center">
  11. <strong>Turn any technical book, document folder, or collection of sources into a unified agent skill — ready to study, reference, and use while you work in GitHub Copilot CLI, Amp, Claude Code, Codex, Hermes Agent, OpenCode, or OpenClaw.</strong></p>
  12. <p align="center">
  13. <a href="https://github.com/virgiliojr94/book-to-skill/releases"><img src="https://img.shields.io/github/v/release/virgiliojr94/book-to-skill?style=for-the-badge&color=blueviolet" alt="Latest release"></a>
  14. <img src="https://img.shields.io/badge/Agent_Skills-Open_Standard-blueviolet?style=for-the-badge" alt="Agent Skills standard">
  15. <img src="https://img.shields.io/badge/PDF%20%E2%80%A2%20EPUB%20%E2%80%A2%20DOCX%20%E2%80%A2%20MD%20%E2%80%A2%20HTML%20%E2%80%A2%20RTF%20%E2%80%A2%20MOBI-supported-green?style=for-the-badge" alt="Formats supported">
  16. <img src="https://img.shields.io/badge/License-MIT-blue?style=for-the-badge" alt="MIT License">
  17. <a href="https://github.com/sponsors/virgiliojr94"><img src="https://img.shields.io/github/sponsors/virgiliojr94?style=for-the-badge&color=ea4aaa&logo=githubsponsors&logoColor=white" alt="Sponsor"></a>
  18. </p>
  19. <p align="center">
  20. <a href="https://trendshift.io/repositories/27038?utm_source=repository-badge&amp;utm_medium=badge&amp;utm_campaign=badge-repository-27038" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/27038" alt="virgiliojr94%2Fbook-to-skill | Trendshift" width="250" height="55"/></a>
  21. <a href="https://trendshift.io/repositories/27038?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-27038" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/27038/daily?language=Python" alt="virgiliojr94%2Fbook-to-skill | Trendshift (daily, Python)" width="250" height="55"/></a>
  22. <a href="https://www.star-history.com/virgiliojr94/book-to-skill"><img src="https://api.star-history.com/badge?repo=virgiliojr94/book-to-skill&type=rank" alt="Star History Rank" /></a>
  23. </p>
  24. <p align="center">
  25. <a href="#-why">Why</a> ·
  26. <a href="#-what-it-generates">What it generates</a> ·
  27. <a href="#-beyond-books">Beyond books</a> ·
  28. <a href="docs/how-it-works.md">How it works</a> ·
  29. <a href="docs/usage.md">Usage</a> ·
  30. <a href="docs/install.md">Install</a> ·
  31. <a href="docs/faq.md">FAQ</a> ·
  32. <a href="docs/performance.md">Performance</a> ·
  33. <a href="docs/architecture.md">Architecture</a> ·
  34. <a href="CHANGELOG.md">Changelog</a>
  35. </p>
  36. <p align="center">
  37. <strong>24×–51× fewer tokens than dumping the book into context</strong> to answer one question, measured on real books (<a href="docs/performance.md#the-discovery-loop-tax">how it's measured</a>).
  38. </p>
  39. **How it works, in 3 steps:**
  40. 1. **Point** it at a file, folder, or glob — `/book-to-skill ./my-book.pdf`
  41. 2. **It distills** the book into a skill — frameworks, decision rules, anti-patterns, and per-chapter files. Structure, not a summary.
  42. 3. **Your agent loads it on demand** — ask `/my-book replication` and it reads the right chapter and answers from the real content, no hallucination.
  43. ---
  44. ## 🤔 Why
  45. <img align="right" width="200" src="docs/assets/booklin.png" alt="Booklin — the book-to-skill mascot, a purple wizard holding a book">
  46. You buy a great technical book. You read it once. Three months later you can't remember chapter 7 existed.
  47. The usual workarounds don't help:
  48. - 📄 "Let me just search the PDF" → you get a list of pages, not answers
  49. - 🧠 "I'll ask the agent about this book" → it either hallucinates or says it doesn't have the content
  50. - 📝 "I'll take notes as I read" → you end up with a 200-line doc you never open again
  51. **book-to-skill solves this by turning the book into a structured skill your agent loads on demand.**
  52. Once installed, you just type `/your-book-slug replication` and the agent reads the right chapter and answers from the actual content. No hallucination. No digging through PDFs. The book becomes part of your workflow.
  53. Works with any host that supports the open [Agent Skills](https://github.com/agentskills/agentskills) standard — GitHub Copilot CLI, Amp, Claude Code, Codex, Hermes Agent, OpenCode, and OpenClaw all read the same `SKILL.md` format.
  54. ---
  55. ## 📦 What it generates
  56. Running `/book-to-skill your-book.pdf` (or a folder, glob, or list of files) creates a full skill in the user-level cross-agent skills directory `~/.agents/skills/<slug>/` — one copy that Copilot CLI, Amp, Codex, OpenCode, and OpenClaw discover natively (OpenCode also reads its own managed root `~/.config/opencode/skills/<slug>/`; OpenClaw also discovers it when using its default state directory, with a non-default `OPENCLAW_STATE_DIR` use the active state's `skills/` root instead). When run under Claude Code, the converter also attempts a verified symlink at `~/.claude/skills/<slug>/` — Claude Code sees the skill when the link read-back confirms it; otherwise the run report says so. Hermes Agent partitions its personal skills by category and does not scan the cross-agent root, so a Hermes install lands in `/skills/<category>/<slug>/` instead. OpenClaw-managed, OpenCode-managed, and project-local destinations remain available when explicitly requested:
  57. | File | Purpose | Size |
  58. |------|---------|------|
  59. | `SKILL.md` | Core mental models + chapter index | ~4,000 tokens |
  60. | `chapters/ch01-*.md` … | One file per chapter, loaded on-demand | ~1,000 tokens each |
  61. | `glossary.md` | Every key term, alphabetically sorted with chapter refs | ~1,500 tokens |
  62. | `patterns.md` | All techniques, algorithms, and design patterns | ~2,000 tokens |
  63. | `cheatsheet.md` | Decision tables and quick-reference rules | ~1,000 tokens |
  64. **Chapter files are loaded on-demand** — they don't count against the skill budget until you ask about that topic.
  65. ---
  66. ## 🏢 Beyond books
  67. The name says "book", but the input is any structured prose. The same extraction works on knowledge you own and re-read constantly:
  68. - **Internal documentation** — architecture decision records, runbooks, onboarding guides. Fold a whole `docs/` folder into one skill and ask it while you code.
  69. - **Brand & design systems** — voice guidelines, tone-of-voice docs, component principles. Turn a brand book into a skill your team queries instead of skimming a 60-page PDF.
  70. - **Research clusters** — a stack of papers plus your own notes, merged into a single unified skill and updated as new material lands (see [Update / fold-in](#-usage)).
  71. - **Specs & standards** — RFCs, API contracts, compliance docs you reference but never memorize.
  72. If you re-open a document often enough to wish you'd memorized it, it's a candidate.
  73. ---
  74. ## 🧾 The Discovery Loop Tax
  75. A PDF-reading agent doesn't just read — it *navigates*: it re-fetches the ToC, backtracks, and re-processes all of it on every turn. book-to-skill pays that structuring cost **once**, at conversion, so queries stay proportional to the answer — **24×–51× fewer tokens** than dumping the book into context, measured on real books.
  76. 📊 **Full methodology, numbers, and per-book tables → [docs/performance.md](docs/performance.md#the-discovery-loop-tax)**
  77. ---
  78. ## ⚙️ How it works
  79. Two halves: a deterministic Python **extractor** (document → clean text + metadata) and a spec-driven **generator** (your agent follows `SKILL.md` to turn that into a structured skill). On-demand chapter files keep the loaded skill small.
  80. 🔧 **Full walkthrough (Steps 0–10, extraction modes, token budgets) → [docs/how-it-works.md](docs/how-it-works.md)**
  81. ---
  82. ## 🚀 Usage
  83. `/book-to-skill <path|folder|glob> [skill-name]` — plus analyze-only, generate-from-analysis, and update/fold-in modes. After a conversion, the converter can publish the skill to GitHub (private by default) so any host installs it with `npx skills add`.
  84. ▶️ **All modes and examples → [docs/usage.md](docs/usage.md)**
  85. 💬 **In practice → [use cases](https://github.com/virgiliojr94/book-to-skill-use-cases)** — a DevEx book became a survey of 300+ engineers; a scanned PDF that stalled became [#130](https://github.com/virgiliojr94/book-to-skill/pull/130). Add yours: the account lives in your own Gist, the index takes a one-line PR.
  86. ---
  87. ## 📥 Install
  88. ```bash
  89. # One command, any host — via the cross-agent skills CLI:
  90. npx skills add virgiliojr94/book-to-skill
  91. # Or manually — clone into your skills folder (registers /book-to-skill):
  92. git clone https://github.com/virgiliojr94/book-to-skill.git ~/.claude/skills/book-to-skill
  93. # (Copilot CLI: ~/.copilot/skills/ · Amp/cross-agent: ~/.agents/skills/)
  94. # (Hermes Agent: ${HERMES_HOME:-$HOME/.hermes}/skills/<category>/)
  95. # (OpenClaw: ${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/skills/; ~/.agents/skills/ only with default state)
  96. # (OpenCode: ~/.agents/skills/ — it also reads ~/.config/opencode/skills/)
  97. ```
  98. 📥 **All hosts, optional extractors, and the standalone CLI → [docs/install.md](docs/install.md)**
  99. ---
  100. ## ❓ FAQ
  101. Common questions — "why not just dump the PDF?", cost, privacy, non-book inputs, multi-file books.
  102. ❓ **Answers → [docs/faq.md](docs/faq.md)**
  103. ---
  104. <details>
  105. <summary>🔧 <strong>Requirements</strong></summary>
  106. The extractor tries tools in order per format and uses the first available. If nothing is installed, it tells you which command to run. Plain text, Markdown, reStructuredText and AsciiDoc need no extra deps.
  107. > **Check your setup in one command:** `python3 scripts/extract.py --check` prints which extractors are installed for every format and the exact command to install anything missing — no file needed.
  108. **PDF — choose by book type:**
  109. | Book type | Tool | Install | Speed |
  110. |-----------|------|---------|-------|
  111. | Text-heavy (prose, few tables) | `pdftotext` (poppler) | `sudo apt install poppler-utils` | ⚡ instant |
  112. | Text-heavy fallback | `pypdf` | `pip3 install pypdf` | ⚡ instant |
  113. | Text-heavy fallback | `pdfminer.six` | `pip3 install pdfminer.six` | ⚡ instant |
  114. | **Technical (code, tables, formulas)** | **`docling`** | `pip3 install docling` | ~1.5s/page |
  115. > Before extraction begins, the skill asks you whether the book is **technical** or **text-heavy** and picks the right tool automatically. Docling preserves markdown tables and code blocks; pdftotext is faster for prose-only books.
  116. > **Scanned PDFs need OCR first.** A PDF that is page images with no text layer — a photographed or scanned book — has nothing for these tools to extract. The extractor checks the first pages and stops immediately with an explanation, rather than working through the whole book to produce an empty skill. Run OCR yourself, then convert the result:
  117. >
  118. > ```bash
  119. > ocrmypdf input.pdf output.pdf
  120. > ```
  121. **EPUB:**
  122. | Tool | Install | Quality |
  123. |------|---------|---------|
  124. | `ebooklib` + `beautifulsoup4` | `pip3 install ebooklib beautifulsoup4` | ⭐⭐⭐ Best |
  125. | stdlib `zipfile` | built-in — no install needed | ⭐⭐ Always available |
  126. **Other formats:**
  127. | Format | Tool | Install |
  128. |--------|------|---------|
  129. | DOCX | `python-docx` (fallback: stdlib ZIP/XML) | `pip3 install python-docx` |
  130. | HTML | `beautifulsoup4` (fallback: stdlib `html.parser`) | `pip3 install beautifulsoup4` |
  131. | RTF | `striprtf` (fallback: regex) | `pip3 install striprtf` |
  132. | MOBI / AZW / AZW3 | Calibre `ebook-convert` (external app, not pip) | https://calibre-ebook.com/download |
  133. | TXT / Markdown / reStructuredText / AsciiDoc | built-in | — |
  134. ---
  135. </details>
  136. <details>
  137. <summary>📁 <strong>Repository structure</strong></summary>
  138. ```
  139. book-to-skill/
  140. ├── SKILL.md # Skill definition + step-by-step instructions (the generator spec)
  141. ├── scripts/
  142. │ ├── extract.py # Thin entrypoint wrapper
  143. │ └── extractor/ # Modular extraction package
  144. │ ├── config.py # Extensions, paths, dependency constants
  145. │ ├── dependencies.py # optional-dep probing + --check
  146. │ ├── exceptions.py # ExtractionError (per-source failures, batch-safe)
  147. │ ├── utils.py # CLI parsing, multi-source resolution, chapter detection, runner
  148. │ └── parsers/ # Format-specific parsers (pdf, epub, docx, html, rtf, calibre, text)
  149. ├── tools/
  150. │ ├── discovery_tax.py # measures token cost vs context-dump / discovery loop
  151. │ └── validate_skill.py # checks a generated SKILL.md against host rules (--lens claude|copilot|amp|hermes|openclaw|opencode)
  152. ├── tests/ # pytest suite (extraction, detection, discovery tax)
  153. ├── docs/
  154. │ ├── performance.md # measured benchmarks, discovery tax, cost
  155. │ └── architecture.md # pipeline + component map
  156. ├── CHANGELOG.md # release history (semver)
  157. ├── CONTRIBUTING.md # dev setup, PR conventions, release process
  158. ├── SECURITY.md # vulnerability reporting
  159. └── README.md # This file
  160. ```
  161. ---
  162. </details>
  163. ---
  164. ## ⚖️ Copyright & fair use
  165. book-to-skill ships **no book content** — not a single page. It's a converter you point at files you already own.
  166. - **Processing is local.** Extraction and analysis run on your machine. Your files are never uploaded by this tool. (If your agent's model runs in the cloud, the text you feed it follows that provider's normal data terms — same as any prompt.)
  167. - **You use your own copy.** Bring a book you bought, docs your company owns, or papers you have the right to read.
  168. - **The output is your notes.** A generated skill is a structured, synthesized derivative — framework names, definitions, takeaways — not a reproduction of the text. The skill explicitly never copies raw passages (see Quality Rule #7). Treat it like handwritten study notes: yours, for personal use.
  169. - **Don't redistribute.** Publishing or sharing a generated skill of a copyrighted work can infringe the rights holder. Keep skills of third-party books private. Internal docs, your own writing, and openly-licensed material are fine to share within the bounds of their license.
  170. When in doubt, follow the license or terms of the source document. This project is a tool; how you use it is on you.
  171. ---
  172. ## 💖 Sponsors
  173. <img align="right" width="150" src="docs/assets/booklin-celebrating.png" alt="Booklin celebrating">
  174. book-to-skill is free and MIT-licensed, maintained on personal time. If it saves you tokens or study hours, consider sponsoring its upkeep: PR reviews, multilingual fixes, releases, and docs.
  175. **[Become a sponsor → github.com/sponsors/virgiliojr94](https://github.com/sponsors/virgiliojr94)**
  176. Every sponsor is listed in [BACKERS.md](BACKERS.md). Thank you for keeping open, privacy-first tooling alive. ✨
  177. ## License
  178. MIT — applies to the converter (code + skill definition) in this repository, **not** to any book or document you process with it.
book_to_skill/utils.py · Bundled script · 2,041 lines
  1. from __future__ import annotations
  2. import glob
  3. import hashlib
  4. import json
  5. import os
  6. import re
  7. import statistics
  8. import sys
  9. import tempfile
  10. from collections import Counter
  11. import shutil
  12. import zipfile
  13. from pathlib import Path
  14. from book_to_skill.exceptions import ExtractionError
  15. from book_to_skill.config import (
  16. OUTPUT_DIR,
  17. OUTPUT_TEXT,
  18. OUTPUT_META,
  19. WORDS_PER_TOKEN,
  20. CJK_CHARS_PER_TOKEN,
  21. SUPPORTED_EXTENSIONS,
  22. TEXT_EXTENSIONS,
  23. HTML_EXTENSIONS,
  24. CALIBRE_EBOOK_EXTENSIONS,
  25. supported_formats_message,
  26. )
  27. from book_to_skill.dependencies import (
  28. normalize_install_mode,
  29. prepare_dependencies,
  30. run_dependency_check,
  31. )
  32. from book_to_skill.parsers.text import read_text_file
  33. from book_to_skill.parsers.html import extract_html_file
  34. from book_to_skill.parsers.docx import extract_docx
  35. from book_to_skill.parsers.rtf import extract_rtf
  36. from book_to_skill.parsers.calibre import extract_with_ebook_convert
  37. from book_to_skill.parsers.pdf import (
  38. _PART_HEADER,
  39. extract_with_docling,
  40. extract_with_pdftotext,
  41. extract_with_pypdf,
  42. extract_with_pdfminer,
  43. extract_pypdf_page_records,
  44. looks_image_only,
  45. count_pages,
  46. pdf_outline_titles,
  47. pdf_outline_all_titles,
  48. )
  49. from book_to_skill.parsers.epub import (
  50. extract_with_ebooklib,
  51. extract_with_zipfile,
  52. count_epub_chapters,
  53. count_epub_images,
  54. )
  55. from book_to_skill.sanitize import sanitize_extracted_text
  56. # Covers and decorative assets are common in prose EPUBs, so only surface the
  57. # omission when the archive contains more than five images.
  58. _EPUB_IMAGE_NOTICE_THRESHOLD = 5
  59. # CJK codepoints: ideographs + extensions, kana, hangul, CJK punctuation, and
  60. # fullwidth forms. These are not whitespace-delimited, so counting "words" on a
  61. # Chinese/Japanese book collapses it to a handful of tokens; count them directly.
  62. #
  63. # The last range is Planes 2 and 3 (U+20000-U+3FFFF), the ideographic
  64. # supplementary planes, taken end to end rather than enumerated block by block
  65. # so a future extension does not silently fall through the way Extension H
  66. # (U+31350-U+323AF) did. Nothing non-ideographic lives up here: emoji,
  67. # mathematical alphanumerics and regional indicators are all in Plane 1, which
  68. # this range does not touch. Classical Chinese, Cantonese, Hong Kong and
  69. # Taiwan place/personal names, and Japanese 人名用漢字 all draw on it. Without it
  70. # those characters fell through to the whitespace-word branch, where a
  71. # space-less run of them counts as a single "word": the same ~1000x undercount
  72. # #103 fixed for the BMP, one plane up.
  73. # The Kangxi-radical range (U+2F00-U+2FDF) is included because some Chinese
  74. # ebooks render ordinary Han characters — 网 as ⽹ (U+2F79), 大 as ⼤
  75. # (U+2F24), 一 as ⼀ (U+2F00) — as radical forms throughout the whole text;
  76. # without it such a book still falls through to the whitespace-word branch.
  77. _CJK_RE = re.compile(
  78. r"[⼀-⿟ -〿぀-ヿ㐀-䶿一-鿿"
  79. r"가-힣豈-﫿＀-￯"
  80. r"\U00020000-\U0003FFFF]"
  81. )
  82. def estimate_tokens(text: str) -> int:
  83. """Estimate the token count of ``text`` with a deterministic heuristic.
  84. Latin / whitespace-delimited text is counted by words (``words /
  85. WORDS_PER_TOKEN`` — the project's long-standing ratio). CJK characters are
  86. counted directly against ``CJK_CHARS_PER_TOKEN`` because they carry little
  87. or no whitespace; without this a space-less Chinese/Japanese book estimates
  88. at a few tokens and the cost pre-flight under-reports by ~1000x. Kept
  89. dependency-free on purpose so the same book always yields the same number.
  90. """
  91. if not text:
  92. return 0
  93. cjk = len(_CJK_RE.findall(text))
  94. if not cjk:
  95. return int(len(text.split()) / WORDS_PER_TOKEN)
  96. latin_words = len(_CJK_RE.sub(" ", text).split())
  97. return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)
  98. # Explicit chapter heading: "Chapter 5", "Capítulo 5: ...", "Chapter 1. Intro".
  99. # Also French/German/Italian/Dutch/Vietnamese chapter words (chapitre/kapitel/
  100. # capitolo/hoofdstuk/chương), matching the ToC languages added alongside. "ch.?"
  101. # stays last so the longer words match in full. Captures the number (bounded to
  102. # 1..99 — drops years like "2025.") and whatever follows it on the line, so we
  103. # can reject prose.
  104. _EXPLICIT_CHAPTER = re.compile(
  105. r"^\s*(?:chapter|unit|lesson|module|lecture|part|chapitre|kapitel|cap[ií]tulo|capitolo|hoofdstuk|chương|ch\.?)\s*(?:(\d{1,2})|(?P<roman>[IVXLCDMivxlcdm]{1,7}))\b(?P<rest>.*)$",
  106. re.IGNORECASE,
  107. )
  108. # A heading's number is followed by end-of-line, punctuation (“. : - —“), or a
  109. # Capitalized title word. A lowercase continuation (“Chapter 6 explores...”,
  110. # “Chapter 8 are relevant...”) is prose / a cross-reference, not a heading.
  111. # The uppercase class is À-Þ so titles starting with Ü/Û (common in German, e.g. “Überblick”) are recognized.
  112. _HEADING_TAIL = re.compile(r"^\s*$|^\s*[.:\-—–]|^\s+(?![a-z])")
  113. # A tail that opens with a sentence period — "Chapter 4. References are
  114. # indicated by the `&` symbol…", "Chapter 17.) We have seen…" — is prose wrapped
  115. # at the column limit, unless it reads like a title. Two shapes, measured on the
  116. # Rust Book sources (rust-lang/book, MIT OR Apache-2.0, 112 files) where the
  117. # numeric scan accepted 13 lines and every one of them was prose or code:
  118. # * a clause of 10-14 words ("Chapter 4. References are indicated by the `&`
  119. # symbol and borrow the value they"), and
  120. # * a bare "Chapter 6." ending a sentence that began on the line above.
  121. # The longest period-tail heading in this repository's fixtures and tests is 5
  122. # words ("Chapter 1. Introduction to Building AI"). (Issue #237)
  123. _PERIOD_TAIL = re.compile(r"^\s*\.[)\"'”’\]]*(\s|$)")
  124. _TAIL_WORD = re.compile(r"[^\s]*[0-9A-Za-z][^\s]*")
  125. _TITLE_TAIL_MAX_WORDS = 8
  126. # A line that ends a sentence / paragraph. When the line above does NOT end like
  127. # this, the next line continues it instead of starting a heading.
  128. _SENTENCE_END = re.compile(r"[.!?…:;\"'”’)\]]\s*$")
  129. # Roman-numeral chapter heading: "I: Loomings", "II. The Carpet-Bag".
  130. # Uppercase alone at line start is safe — no common English word is a valid
  131. # uppercase Roman numeral. Lowercase ("i: Loomings") is only accepted inside
  132. # a markdown heading ("## i. introduction") to avoid false positives from
  133. # words that happen to be valid Roman numerals ("vi: the editor" → 6).
  134. _ROMAN_HEAD = re.compile(r"^\s*([IVXLCDM]+)\s*[:.]\s+[A-ZÀ-Þ0-9\"“(]")
  135. _LC_MD_ROMAN = re.compile(r"^\s*#{1,6}\s+([ivxlcdm]+)\s*[:.]\s+[A-Za-zÀ-Þ\"“(]")
  136. _ROMAN_VALUES = {"I": 1, "V": 5, "X": 10, "L": 50, "C": 100, "D": 500, "M": 1000}
  137. # Optional Markdown / AsciiDoc heading prefix ("## Chapter 1", "== Section").
  138. # Stripped in _chapter_number() as a second pass so the CJK/Thai/Korean
  139. # matchers (which already tolerate the prefix inline) are untouched. (Issue #91)
  140. _MD_HEADING_PREFIX = re.compile(r"^(#{1,6}|={1,6})\s+")
  141. # Chinese chapter headings. Two common styles:
  142. # 1. explicit "第N章" / "第 3 回" / "第十二节" / "第一讲" — 第 + numeral + a
  143. # chapter classifier (章回卷节篇讲);
  144. # 2. a Markdown heading led by a CJK ordinal and a separator, e.g.
  145. # "## 一 · 缘起" or "## 第一讲" — common in CJK ebooks and lecture notes.
  146. # Scoped to CJK numerals, so Latin/Roman detection above is completely unaffected
  147. # (e.g. "## 5 Setup" is still not treated as a heading here). detect_structure()
  148. # dedupes by number, so a "##" heading and a repeated "###" sub-ordinal collapse
  149. # to a single chapter.
  150. _CN_NUM_VALUES = {
  151. "〇": 0, "零": 0, "一": 1, "二": 2, "两": 2, "三": 3, "四": 4, "五": 5,
  152. "六": 6, "七": 7, "八": 8, "九": 9,
  153. }
  154. _CN_NUM_UNITS = {"十": 10, "百": 100, "千": 1000}
  155. _CN_NUM_CLASS = "〇零一二两三四五六七八九十百千"
  156. # Kangxi-radical numerals → CJK ideograph numerals. Some Chinese ebooks
  157. # (e.g. certain e-reader platforms) encode numerals as Kangxi radicals from
  158. # the U+2F00 block instead of CJK unified ideographs — "第⼀章" with
  159. # U+2F00 (⼀) rather than U+4E00 (一). NFKC does not map these, so normalize
  160. # them explicitly before chapter detection. Only numerals that exist as
  161. # Kangxi radicals are listed (三/四/五/六/七/九 have no radical form).
  162. _KANGXI_NUMERAL_TRANS = {
  163. 0x2F00: ord("一"), # ⼀ KANGXI RADICAL ONE
  164. 0x2F06: ord("二"), # ⼆ KANGXI RADICAL TWO
  165. 0x2F0B: ord("八"), # ⼋ KANGXI RADICAL EIGHT
  166. 0x2F17: ord("十"), # ⼗ KANGXI RADICAL TEN
  167. }
  168. # Full-width Arabic digits (U+FF10–U+FF19) are common in Japanese typesetting,
  169. # e.g. "第1章". int() already parses them (str.isdigit() is True), so only the
  170. # regex character classes need to accept them.
  171. _FW_DIGITS = "0-9"
  172. _CN_CHAPTER = re.compile(rf"^\s*第\s*([0-9{_FW_DIGITS}{_CN_NUM_CLASS}]+)\s*[章回卷节篇讲]")
  173. _MD_CN_HEADING = re.compile(rf"^#{{1,6}}\s+第?\s*([{_FW_DIGITS}{_CN_NUM_CLASS}]+)\s*[·、.::章回卷节篇讲]")
  174. # Thai chapter headings: "บทที่ 3", "บทที่ ๑๒", "ตอนที่ ๘๗", "ภาคที่ 2".
  175. # Thai digits (U+0E50-U+0E59) are positional like Arabic — unlike the Chinese
  176. # numerals above they need no unit composition, only a digit remap. Optional
  177. # Markdown "#" prefix so "## บทที่ ๑" is recognized in converted ebooks.
  178. _TH_DIGITS = "๐-๙"
  179. _TH_DIGIT_MAP = str.maketrans("๐๑๒๓๔๕๖๗๘๙", "0123456789")
  180. _TH_CHAPTER = re.compile(
  181. rf"^\s*(?:#{{1,6}}\s+)?(?:บทที่|ตอนที่|ภาคที่|บท|ตอน|ภาค)\s*([0-9{_TH_DIGITS}]+)\b"
  182. )
  183. # Hindi (Devanagari) chapter headings: "अध्याय 1", "अध्याय १", "## अध्याय 2".
  184. # अध्याय ("chapter") + a number. Devanagari digits (U+0966-U+096F) are positional
  185. # like Arabic, so — as with Thai — only a digit remap is needed, no composition.
  186. # Optional Markdown "#" prefix so "## अध्याय १" is recognized in converted ebooks.
  187. # Scoped to the digit form (not word ordinals like "पहला अध्याय") and requiring a
  188. # number keeps prose that merely uses the word अध्याय from matching.
  189. _HI_DIGITS = "०-९"
  190. _HI_DIGIT_MAP = str.maketrans("०१२३४५६७८९", "0123456789")
  191. _HI_CHAPTER = re.compile(
  192. rf"^\s*(?:#{{1,6}}\s+)?अध्याय\s*([0-9{_HI_DIGITS}]+)\b"
  193. )
  194. # Bengali chapter headings: "অধ্যায় 1", "অধ্যায় ১", "## অধ্যায় 2".
  195. # অধ্যায় ("chapter") + a number. Bengali digits (U+09E6-U+09EF) are positional
  196. # like the Hindi block above, so only a digit remap is needed. Optional Markdown
  197. # "#" prefix so "## অধ্যায় ১" is recognized in converted ebooks. Requiring a
  198. # number keeps prose that merely uses the word অধ্যায় from matching.
  199. _BN_DIGITS = "০-৯"
  200. _BN_DIGIT_MAP = str.maketrans("০১২৩৪৫৬৭৮৯", "0123456789")
  201. _BN_CHAPTER = re.compile(
  202. rf"^\s*(?:#{{1,6}}\s+)?অধ্যায়\s*([0-9{_BN_DIGITS}]+)\b"
  203. )
  204. # Odia chapter headings: "ଅଧ୍ୟାୟ 1", "ଅଧ୍ୟାୟ ୧", "## ଅଧ୍ୟାୟ 2".
  205. # ଅଧ୍ୟାୟ ("chapter") + a number. Grouped next to Bengali because both are
  206. # Eastern Brahmic and the words look alike, but the scripts occupy separate
  207. # blocks (Odia U+0B00-U+0B7F, Bengali U+0980-U+09FF), so the two patterns
  208. # cannot see each other's text. Odia digits (U+0B66-U+0B6F) are positional, so
  209. # only a digit remap is needed. Requiring a number keeps prose that merely uses
  210. # an inflected form ("ଅଧ୍ୟାୟରେ", a locative suffix) from matching.
  211. _OR_DIGITS = "୦-୯"
  212. _OR_DIGIT_MAP = str.maketrans("୦୧୨୩୪୫୬୭୮୯", "0123456789")
  213. _OR_CHAPTER = re.compile(
  214. rf"^\s*(?:#{{1,6}}\s+)?ଅଧ୍ୟାୟ\s*([0-9{_OR_DIGITS}]+)\b"
  215. )
  216. # Tamil chapter headings: "அத்தியாயம் 1", "அத்தியாயம் ௧", "## அத்தியாயம் 2".
  217. # அத்தியாயம் ("chapter") + a number. Tamil digits (U+0BE6-U+0BEF) are positional
  218. # like the Devanagari/Bengali blocks above, so only a digit remap is needed.
  219. # Optional Markdown "#" prefix. Requiring a number keeps prose that merely uses
  220. # the word (or an inflected form like "அத்தியாயத்தில்", which is a different stem)
  221. # from matching.
  222. _TA_DIGITS = "௦-௯"
  223. _TA_DIGIT_MAP = str.maketrans("௦௧௨௩௪௫௬௭௮௯", "0123456789")
  224. _TA_CHAPTER = re.compile(
  225. rf"^\s*(?:#{{1,6}}\s+)?அத்தியாயம்\s*([0-9{_TA_DIGITS}]+)\b"
  226. )
  227. # Sinhala chapter headings: "පරිච්ඡේදය 1", "අධ්‍යාය ෨", "## පරිච්ඡේදය 2".
  228. # Two words are used: පරිච්ඡේදය and the Sanskrit loan අධ්‍යාය. Grouped next to
  229. # Tamil as the other Sri Lankan script. Modern Sinhala prints numbers with ASCII
  230. # digits; the Lith digits (U+0DE6-U+0DEF) are included for historical texts,
  231. # remapped like the other Indic blocks.
  232. #
  233. # The joiner in අධ්‍යාය is matched optionally, and that is deliberate: the word
  234. # carries a ZERO WIDTH JOINER inside its ධ්‍ය conjunct, but
  235. # sanitize_extracted_text() strips U+200D before detect_structure() ever runs, so
  236. # the spelling that reaches this pattern has no joiner even though every real
  237. # source file has one. "\u200d?" matches both, which keeps the pattern working
  238. # for callers that reach _chapter_number() without sanitizing first.
  239. #
  240. # අධ්‍යාය shares its transliteration with Devanagari अध्याय, Gujarati અધ્યાય and
  241. # Gurmukhi ਅਧਿਆਇ, but Sinhala occupies U+0D80-U+0DFF and none of those blocks
  242. # overlap it, so the matchers cannot see each other's text. Requiring a number
  243. # keeps prose that merely inflects a word ("පරිච්ඡේදයේ", "අධ්‍යායේ") from matching.
  244. _SI_DIGITS = "෦-෯"
  245. _SI_DIGIT_MAP = str.maketrans("෦෧෨෩෪෫෬෭෮෯", "0123456789")
  246. _SI_CHAPTER = re.compile(
  247. rf"^\s*(?:#{{1,6}}\s+)?(?:පරිච්ඡේදය|අධ්\u200d?යාය)\s*([0-9{_SI_DIGITS}]+)\b"
  248. )
  249. # Telugu chapter headings: "అధ్యాయము 1", "అధ్యాయం ౧", "## అధ్యాయం 2".
  250. # The chapter word has two common spellings — అధ్యాయము (formal) and అధ్యాయం
  251. # (modern), sharing the stem అధ్యాయ — so both endings are matched. Telugu digits
  252. # (U+0C66-U+0C6F) are positional like the other Indic blocks above, so only a
  253. # digit remap is needed. Requiring the full word then a number keeps prose (or an
  254. # inflected form like "అధ్యాయంలో", a different suffix) from matching.
  255. _TE_DIGITS = "౦-౯"
  256. _TE_DIGIT_MAP = str.maketrans("౦౧౨౩౪౫౬౭౮౯", "0123456789")
  257. _TE_CHAPTER = re.compile(
  258. rf"^\s*(?:#{{1,6}}\s+)?అధ్యాయ(?:ము|ం)\s*([0-9{_TE_DIGITS}]+)\b"
  259. )
  260. # Kannada chapter headings: "ಅಧ್ಯಾಯ 1", "ಅಧ್ಯಾಯ ೧", "## ಅಧ್ಯಾಯ 2".
  261. # ಅಧ್ಯಾಯ ("chapter") + a number. Kannada digits (U+0CE6-U+0CEF) are positional
  262. # like the other Indic blocks above, so only a digit remap is needed. Requiring
  263. # the word then a number keeps prose that merely uses an inflected form
  264. # ("ಅಧ್ಯಾಯದಲ್ಲಿ", a case suffix) from matching.
  265. _KN_DIGITS = "೦-೯"
  266. _KN_DIGIT_MAP = str.maketrans("೦೧೨೩೪೫೬೭೮೯", "0123456789")
  267. _KN_CHAPTER = re.compile(
  268. rf"^\s*(?:#{{1,6}}\s+)?ಅಧ್ಯಾಯ\s*([0-9{_KN_DIGITS}]+)\b"
  269. )
  270. # Malayalam chapter headings: "അധ്യായം 1", "അധ്യായം ൧", "## അദ്ധ്യായം 2".
  271. # The chapter word has two common spellings — അധ്യായം (modern) and അദ്ധ്യായം
  272. # (older ദ്ധ orthography), differing only in that conjunct — so both stems are
  273. # matched. Malayalam digits (U+0D66-U+0D6F) are positional like the other Indic
  274. # blocks above, so only a digit remap is needed. Requiring the full word
  275. # (including the final ം) then a number keeps prose that merely uses an
  276. # inflected form like "അധ്യായത്തിൽ" from matching.
  277. _ML_DIGITS = "൦-൯"
  278. _ML_DIGIT_MAP = str.maketrans("൦൧൨൩൪൫൬൭൮൯", "0123456789")
  279. _ML_CHAPTER = re.compile(
  280. rf"^\s*(?:#{{1,6}}\s+)?അ(?:ധ്|ദ്ധ്)യായം\s*([0-9{_ML_DIGITS}]+)\b"
  281. )
  282. # Gujarati chapter headings: "પ્રકરણ 1", "અધ્યાય ૧", "## પ્રકરણ 2".
  283. # Gujarati uses two distinct words for "chapter" — પ્રકરણ (common in modern
  284. # books) and અધ્યાય (classical/religious) — so both are matched. અધ્યાય shares
  285. # its transliteration with Devanagari अध्याय (the Hindi block above) but not its
  286. # codepoints (U+0A85.. vs U+0905..), so the two scripts cannot collide. Gujarati
  287. # digits (U+0AE6-U+0AEF) are positional like the other Indic blocks above, so
  288. # only a digit remap is needed. Unlike Malayalam, an inflected form keeps the
  289. # whole word ("પ્રકરણમાં" is the word + માં), so requiring a number right after
  290. # it is what keeps prose from matching.
  291. _GU_DIGITS = "૦-૯"
  292. _GU_DIGIT_MAP = str.maketrans("૦૧૨૩૪૫૬૭૮૯", "0123456789")
  293. _GU_CHAPTER = re.compile(
  294. rf"^\s*(?:#{{1,6}}\s+)?(?:પ્રકરણ|અધ્યાય)\s*([0-9{_GU_DIGITS}]+)\b"
  295. )
  296. # Punjabi (Gurmukhi) chapter headings: "ਕਾਂਡ 1", "ਅਧਿਆਇ ੧", "## ਕਾਂਡ 2".
  297. # Like Gujarati above, Punjabi uses two distinct words for "chapter" — ਕਾਂਡ
  298. # (usual in books) and ਅਧਿਆਇ — so both are matched. ਅਧਿਆਇ resembles Gujarati
  299. # અધ્યાય and Devanagari अध्याय but the scripts are disjoint (Gurmukhi
  300. # U+0A00-U+0A7F, Gujarati U+0A80-U+0AFF, Devanagari U+0900-U+097F), so none of
  301. # the three patterns can see another's text. Gurmukhi digits (U+0A66-U+0A6F) are
  302. # positional, so only a digit remap is needed. Requiring a number right after
  303. # the word keeps prose from matching, whether it follows the word with a
  304. # postposition ("ਕਾਂਡ ਵਿੱਚ") or inflects it ("ਕਾਂਡਾਂ").
  305. _PA_DIGITS = "੦-੯"
  306. _PA_DIGIT_MAP = str.maketrans("੦੧੨੩੪੫੬੭੮੯", "0123456789")
  307. _PA_CHAPTER = re.compile(
  308. rf"^\s*(?:#{{1,6}}\s+)?(?:ਕਾਂਡ|ਅਧਿਆਇ)\s*([0-9{_PA_DIGITS}]+)\b"
  309. )
  310. # Russian (Cyrillic) chapter headings: "Глава 1", "ГЛАВА 12", "## Глава 2".
  311. # "Глава" ("chapter") + a number. Cyrillic uses ordinary Arabic digits, so —
  312. # unlike the Devanagari/Bengali blocks above — no digit remap is needed. A
  313. # dedicated matcher (rather than adding the word to _EXPLICIT_CHAPTER) is used
  314. # because that alternation is Latin-only and its number would still be read
  315. # there. Requiring whitespace then a number keeps prose that merely uses an
  316. # inflected form ("В этой главе…", "Главная страница") from matching.
  317. _RU_CHAPTER = re.compile(r"^\s*(?:#{1,6}\s+)?глава\s+([0-9]+)\b", re.IGNORECASE)
  318. # Greek chapter headings: "Κεφάλαιο 1", "ΚΕΦΑΛΑΙΟ 12", "## Κεφάλαιο 2".
  319. # "Κεφάλαιο" ("chapter") + a number. Greek uses ordinary Arabic digits, so — like
  320. # the Russian block above — no digit remap is needed. The 4th letter is written
  321. # [άα] because all-caps Greek drops the accent ("ΚΕΦΑΛΑΙΟ" has plain Α, not Ά),
  322. # and case-folding alone would not bridge Ά↔Α. Requiring whitespace then a number
  323. # keeps prose that merely uses the word (also "capital", "Το κεφάλαιο αυτό…") from
  324. # matching.
  325. _EL_CHAPTER = re.compile(r"^\s*(?:#{1,6}\s+)?κεφ[άα]λαιο\s+([0-9]+)\b", re.IGNORECASE)
  326. # Hebrew chapter headings: "פרק 1", "פרק 12", "## פרק 2".
  327. # פרק ("chapter") + a number. Hebrew is unicameral and uses ordinary Arabic
  328. # digits, so neither case folding nor a digit remap is needed. Being right-to-left
  329. # does not affect the pattern: the regex reads the logical order stored in the
  330. # file, which is פ-ר-ק then the number, and sanitize_extracted_text() removes any
  331. # bidi wrapper (RLM/RLE/PDF) before detect_structure() sees the line. Requiring
  332. # whitespace then a number keeps prose from matching — the word also means
  333. # "joint", and the prefixed and plural forms ("בפרק", "פרקים") are not followed by
  334. # a number. Gematria numbering ("פרק א"), common in religious texts, is a separate
  335. # letter-to-number mapping and is deliberately not handled here.
  336. _HE_CHAPTER = re.compile(r"^\s*(?:#{1,6}\s+)?פרק\s+([0-9]+)\b")
  337. # Armenian chapter headings: "Գլուխ 1", "ԳԼՈՒԽ 12", "## Գլուխ 2".
  338. # "Գլուխ" ("chapter") + a number, with ordinary Arabic digits so no remap is
  339. # needed. Armenian is bicameral and — unlike Greek above — its case mapping
  340. # round-trips exactly ("Գլուխ".upper() == "ԳԼՈՒԽ" letter for letter), so plain
  341. # re.IGNORECASE is enough and no letter class is required. The word also means
  342. # "head", so requiring *whitespace* then a number does the guarding: an
  343. # inflected form elides the ու stem vowel ("գլխում") and so shares no prefix,
  344. # and the plural keeps the word intact, which is why \s+ matters — it stops
  345. # "Գլուխներ 5" ("chapters 5") from reading as chapter 5.
  346. _HY_CHAPTER = re.compile(r"^\s*(?:#{1,6}\s+)?գլուխ\s+([0-9]+)\b", re.IGNORECASE)
  347. # Korean chapter headings: "제1장 총칙", "## 제4장 근로시간과 휴식", "제6장의2 …".
  348. # 제 + Arabic numeral + a classifier (장 chapter / 편 part / 절 section / 관
  349. # subsection), with an optional "의N" branch suffix that Korean statutes use for
  350. # inserted chapters (제6장의2). Modern Korean numbers chapters with Arabic digits,
  351. # so unlike the Chinese branch no numeral composition is needed. Optional Markdown
  352. # "#" prefix so "## 제1장" is recognized in converted ebooks.
  353. #
  354. # The trailing group is the Korean analogue of _HEADING_TAIL: Korean has no letter
  355. # case, so the existing "capitalized title word" test does not transfer.
  356. # Requiring end-of-line, punctuation, or whitespace-then-content is what separates
  357. # a heading from a prose cross-reference, because Korean particles attach directly
  358. # to the noun ("제5장에서", "제2장의") with no intervening space.
  359. _KO_CHAPTER = re.compile(
  360. r"^\s*(?:#{1,6}\s+)?제\s*([0-9]+)\s*[장편절관](?:\s*의\s*[0-9]+)?(?:\s*$|[.:\-]|\s+\S)"
  361. )
  362. # Persian chapter headings: "فصل ۱", "فصل اول", "بخش ۲: مفاهیم",
  363. # "فصل بیست و یکم", "فصل سی و چهارمخداحافظ…" (PDF glue on long forms).
  364. # Labels are فصل / بخش. Digits may be ASCII, Persian (U+06F0–U+06F9), or
  365. # Arabic-Indic (U+0660–U+0669); int() parses all three. Word numerals use a
  366. # small ordinal map (1–34) with longest-prefix matching so compounds
  367. # ("بیست و یکم") and teens ("یازدهم") stay maintainable. Markdown "#" prefixes
  368. # are handled by `_chapter_number`'s second pass (Issue #91).
  369. #
  370. # Trailing rules (Persian has no letter case for a Latin-style `_HEADING_TAIL`):
  371. # - digits: EOL / punctuation / spaced title (Korean-style);
  372. # - short word ordinals 1–10: require a separator (space, punct, ZWNJ, or EOL)
  373. # so "فصل اولویت‌ها" / "فصل اولیه" are not read as chapter 1;
  374. # - teens and compounds: also allow a glued title letter — PDF extractors
  375. # often drop the space, and a long ordinal is not a plausible word prefix.
  376. _FA_DIGITS = "۰-۹٠-٩" # Persian then Arabic-Indic
  377. _FA_ONES = (
  378. "اول", "دوم", "سوم", "چهارم", "پنجم", "ششم", "هفتم", "هشتم", "نهم", "دهم",
  379. )
  380. # Ones used after "بیست و" / "سی و" (یکم, not اول).
  381. _FA_COMPOUND_ONES = (
  382. "یکم", "دوم", "سوم", "چهارم", "پنجم", "ششم", "هفتم", "هشتم", "نهم",
  383. )
  384. _FA_TEENS = (
  385. "یازدهم", "دوازدهم", "سیزدهم", "چهاردهم", "پانزدهم",
  386. "شانزدهم", "هفدهم", "هجدهم", "نوزدهم",
  387. )
  388. _FA_ONES_SET = frozenset(_FA_ONES)
  389. # After a short (1–10) word ordinal: end, whitespace, punctuation, or ZWNJ.
  390. _FA_SHORT_ORDINAL_TAIL = re.compile(r"^(?:$|\s|[.:\-—–:]|\u200c)")
  391. def _fa_ordinal_map() -> dict[str, int]:
  392. """Persian chapter ordinals 1–34, including common spelling variants."""
  393. m: dict[str, int] = {}
  394. for i, w in enumerate(_FA_ONES, 1):
  395. m[w] = i
  396. for i, w in enumerate(_FA_TEENS, 11):
  397. m[w] = i
  398. m["هیجدهم"] = 18 # common alternate spelling of هجدهم
  399. # Fused "بیستم" only — "بیست ام" / "بیست‌ام" are not common spellings
  400. # (unlike "سی ام" / "سی‌ام" for 30), so they stay unmapped on purpose.
  401. m["بیستم"] = 20
  402. m["سی ام"] = 30
  403. m["سی‌ام"] = 30 # ZWNJ spelling common in Persian typography
  404. for i, w in enumerate(_FA_COMPOUND_ONES, 1):
  405. m[f"بیست و {w}"] = 20 + i
  406. m[f"سی و {w}"] = 30 + i
  407. # sanitize_extracted_text() strips U+200C (it is a prompt-injection spacer)
  408. # before detect_structure() runs, so a key spelled with a ZWNJ can never be
  409. # reached through the pipeline. Register the stripped spelling of every such
  410. # key as well, so both the typographic form and what detection actually
  411. # receives resolve to the same chapter number.
  412. for key, value in list(m.items()):
  413. m.setdefault(key.replace("\u200c", ""), value)
  414. return m
  415. _FA_ORDINALS = _fa_ordinal_map()
  416. # Longest first so "چهاردهم" wins over "چهارم", "بیست و یکم" over nothing shorter.
  417. _FA_ORDINAL_KEYS = sorted(_FA_ORDINALS, key=len, reverse=True)
  418. _FA_LABEL_REST = re.compile(r"^\s*(?:فصل|بخش)\s+(.*)$")
  419. _FA_DIGIT_HEAD = re.compile(rf"^([0-9{_FA_DIGITS}]+)(.*)$")
  420. # Digit form: same idea as the Korean trailing guard (no Latin case to lean on).
  421. _FA_DIGIT_TAIL = re.compile(r"^(?:\s*$|[.:\-—–:]|\s+\S)")
  422. def _fa_chapter_number(s: str) -> int | None:
  423. """Return a Persian chapter number (1–99 digits / 1–34 words) or None."""
  424. m = _FA_LABEL_REST.match(s)
  425. if not m:
  426. return None
  427. rest = m.group(1)
  428. dm = _FA_DIGIT_HEAD.match(rest)
  429. if dm:
  430. n = int(dm.group(1))
  431. if 1 <= n <= 99 and _FA_DIGIT_TAIL.match(dm.group(2)) is not None:
  432. return n
  433. return None
  434. for key in _FA_ORDINAL_KEYS:
  435. if not rest.startswith(key):
  436. continue
  437. tail = rest[len(key):]
  438. # Short 1–10 ordinals need a separator; teens/compounds may be PDF-glued.
  439. if key in _FA_ONES_SET and _FA_SHORT_ORDINAL_TAIL.match(tail) is None:
  440. return None
  441. return _FA_ORDINALS[key]
  442. return None
  443. # Table-of-contents header lines across common languages. Anchored to a whole
  444. # line (^\s*X\s*$) so an inline "the contents of this chapter" never matches.
  445. _TOC_HEADERS = (
  446. "table of contents", "contents", "índice", "sumário", # EN / ES / PT
  447. "sumario", # PT (no accent — OCR / accent-stripped, like indice below)
  448. "table des matières", # French
  449. "inhaltsverzeichnis", # German
  450. "indice", "sommario", # Italian (no accent — distinct from índice above)
  451. "inhoudsopgave", # Dutch
  452. )
  453. _TOC_CJK_PATTERN = r"目[ \t\u3000]*(?:录|錄|次)"
  454. _TOC_PATTERN = re.compile(
  455. r"^\s*(?:#{1,6}\s*)?(?:"
  456. + "|".join([*(re.escape(h) for h in _TOC_HEADERS), _TOC_CJK_PATTERN])
  457. + r")\s*$",
  458. re.IGNORECASE | re.MULTILINE,
  459. )
  460. # Structural extraction sees every heading at the selected depth, including
  461. # the book's framing sections. These whole-title guards remove only conventional
  462. # front/back matter; instructional titles such as "Indexing Strategies" and
  463. # "Glossary of Terms" must remain chapters.
  464. _FRONT_MATTER_HEADERS = (
  465. "cover",
  466. "preface",
  467. "foreword",
  468. "acknowledgements",
  469. "acknowledgments",
  470. "dedication",
  471. "copyright",
  472. "epigraph",
  473. "prologue",
  474. )
  475. _BACK_MATTER_HEADERS = (
  476. "bibliography",
  477. "references",
  478. "works cited",
  479. "further reading",
  480. "index",
  481. "glossary",
  482. "notes",
  483. "endnotes",
  484. "footnotes",
  485. "colophon",
  486. "about the author",
  487. "about the authors",
  488. )
  489. _NON_CHAPTER_HEADERS = frozenset((*_FRONT_MATTER_HEADERS, *_BACK_MATTER_HEADERS))
  490. _NON_CHAPTER_PREFIX = re.compile(r"^(?:appendix|appendices)\b", re.IGNORECASE)
  491. def _is_non_chapter_section(title: str) -> bool:
  492. """Return whether *title* is conventional book framing, not a chapter."""
  493. normalized = title.strip()
  494. return (
  495. normalized.casefold() in _NON_CHAPTER_HEADERS
  496. or bool(_TOC_PATTERN.fullmatch(normalized))
  497. or bool(_NON_CHAPTER_PREFIX.match(normalized))
  498. or bool(_PART_HEADER.match(normalized))
  499. )
  500. # ATX-style heading: "# Title", "## Section", AsciiDoc "= Title", "== Section".
  501. # The required space after the marker distinguishes an AsciiDoc "== X" from a
  502. # reStructuredText underline "=====" (no space) — the latter is intentionally
  503. # ignored (RST underline headings are out of scope).
  504. _ATX_HEADING = re.compile(r"^(#{1,6}|={1,6})\s+(.+?)\s*#*$")
  505. # Setext/RST underline: a full line of "=" (level 1) or "-" (level 2), length
  506. # >= 2. Marks the line directly above it as a heading title.
  507. _SETEXT_UNDERLINE = re.compile(r"^(={2,}|-{2,})$")
  508. # Opening or closing line of a fenced code block: three or more backticks or
  509. # tildes. The captured marker lets the closer be matched to its opener.
  510. _CODE_FENCE = re.compile(r"^(`{3,}|~{3,})")
  511. def _closed_fence_line_numbers(lines: list[str]) -> set[int]:
  512. """Line indices inside a fenced code block that is actually CLOSED.
  513. A fence that never closes is treated as ordinary text rather than swallowing
  514. everything after it. Extraction routinely loses a closing fence, and a book
  515. about Markdown can simply contain a stray one — and the old live-toggling
  516. scan then dropped every heading from that point to the end of the document.
  517. Counting a handful of code lines as prose is a far cheaper mistake than
  518. losing most of a book's structure.
  519. Per CommonMark, a closer must use the same character, be at least as long
  520. as its opener, and have no trailing info string. A longer fence can then
  521. safely contain examples of shorter fences without exposing sample headings.
  522. """
  523. inside: set[int] = set()
  524. opener: tuple[str, int] | None = None
  525. for index, line in enumerate(lines):
  526. stripped = line.strip()
  527. match = _CODE_FENCE.match(stripped)
  528. if not match:
  529. continue
  530. marker = match.group(1)
  531. if opener is None:
  532. opener = (marker, index)
  533. elif (
  534. marker[0] == opener[0][0]
  535. and len(marker) >= len(opener[0])
  536. and not stripped[match.end():]
  537. ):
  538. # Include both fence marker lines themselves.
  539. inside.update(range(opener[1], index + 1))
  540. opener = None
  541. return inside
  542. # A numbered heading is a chapter when the numbering is systematic AND the
  543. # sections carry a chapter's worth of text. Both are required, because neither
  544. # separates the two shapes alone: a three-step tutorial is also systematic and
  545. # also ascends from 1, while a single long section is not a numbering scheme.
  546. # Measured medians of body text per section: tutorial steps ~20 chars, doc
  547. # sections ~500, paper sections ~2,000, real book chapters ~5,000. The floor
  548. # sits an order of magnitude below the smallest real chapter seen and an order
  549. # above the largest tutorial step.
  550. _MIN_NUMBERED_TITLES = 3
  551. _MIN_NUMBERED_BODY_CHARS = 200
  552. def _numbered_titles_are_structural(
  553. entries: list[tuple[str, int, str]], heading_lines: list[int], lines: list[str]
  554. ) -> bool:
  555. """Decide whether digit-led titles at one depth are chapters or list items.
  556. Deliberately not based on the numbers themselves. An ascending run starting
  557. at 1 describes "Step 1 / Step 2 / Step 3" as accurately as it describes a
  558. paper's sections, and requiring the run to be unbroken would throw away a
  559. whole book when extraction drops one heading, a chapter list that starts at
  560. 0, or a multi-source corpus where the numbering restarts.
  561. """
  562. if len(entries) < _MIN_NUMBERED_TITLES:
  563. return False
  564. ordered = sorted(heading_lines)
  565. bodies = []
  566. for _, index, _ in entries:
  567. after = [ln for ln in ordered if ln > index]
  568. end = after[0] if after else len(lines)
  569. bodies.append(sum(len(ln) for ln in lines[index + 1:end]))
  570. return statistics.median(bodies) >= _MIN_NUMBERED_BODY_CHARS
  571. def _structural_chapter_headings(text: str) -> list[str]:
  572. """Return chapter-like structural headings in Markdown/AsciiDoc/RST sources.
  573. Recognizes ATX headings ("# Title", "== Section") and setext/RST underline
  574. headings (a title line directly above a row of "=" or "-"). Groups distinct
  575. (case-normalized) titles by depth and returns the count at the shallowest
  576. depth with >= 2 distinct titles — this selects the real chapter level in the
  577. common "# Book Title / ## Chapter" layout where the top level appears once.
  578. Guards against false positives: headings inside fenced code blocks are
  579. skipped; an ATX title starting with a bare digit ("## 5 Setup") or made only
  580. of punctuation ("=====" table borders) is rejected; a setext underline counts
  581. only when it sits directly under a non-blank title line at least as long as
  582. the underline (so thematic breaks, table borders, and front-matter "---" do
  583. not match).
  584. """
  585. lines = text.splitlines()
  586. # Map normalized titles to their original representation. Dict insertion
  587. # order makes the returned sample match the source order while preserving
  588. # the previous case-insensitive de-duplication behavior.
  589. levels: dict[int, dict[str, str]] = {}
  590. # Digit-led titles are held back and judged per depth at the end (see
  591. # _numbered_titles_are_structural): "## 1. Introduction" and "## 5 Setup"
  592. # are the same string shape, so the line alone cannot decide.
  593. numbered: dict[int, list[tuple[str, int, str]]] = {}
  594. heading_lines: list[int] = []
  595. fenced = _closed_fence_line_numbers(lines)
  596. prev = "" # previous non-fence line (stripped); a setext title candidate
  597. for index, line in enumerate(lines):
  598. if index in fenced:
  599. prev = ""
  600. continue
  601. s = line.strip()
  602. # Setext/RST underline: "=" (level 1) or "-" (level 2) directly under a
  603. # title line at least as long as the underline.
  604. if (
  605. _SETEXT_UNDERLINE.match(s)
  606. and prev
  607. and not _SETEXT_UNDERLINE.match(prev)
  608. and len(s) >= len(prev)
  609. # A title made only of punctuation is never a chapter. Two thematic
  610. # breaks in a row ("***" over "---"), an ASCII box rule, a row of
  611. # dots, or a table border sitting above an underline all reach this
  612. # point. The ATX branch below already rejects them with the same
  613. # test; the setext branch had no equivalent, so the identical string
  614. # counted as a heading here and not there.
  615. and re.search(r"\w", prev)
  616. ):
  617. depth = 1 if s[0] == "=" else 2
  618. heading_lines.append(index)
  619. if not _is_non_chapter_section(prev):
  620. levels.setdefault(depth, {}).setdefault(prev.casefold(), prev)
  621. prev = ""
  622. continue
  623. # ATX heading ("# Title", "== Section").
  624. m = _ATX_HEADING.match(s)
  625. if m:
  626. title = m.group(2).strip().lower()
  627. depth = len(m.group(1))
  628. # Reject empty and all-punctuation ("=====" table-border) titles.
  629. if title and re.search(r"\w", title):
  630. heading_lines.append(index)
  631. if not _is_non_chapter_section(title):
  632. if title[0].isdigit():
  633. numbered.setdefault(depth, []).append((title, index, s))
  634. else:
  635. levels.setdefault(depth, {}).setdefault(title, s)
  636. # An ATX heading line is not a setext title for the next line.
  637. prev = ""
  638. continue
  639. prev = s
  640. for depth, entries in numbered.items():
  641. if _numbered_titles_are_structural(entries, heading_lines, lines):
  642. level = levels.setdefault(depth, {})
  643. for title, _, sample in entries:
  644. level.setdefault(title, sample)
  645. if not levels:
  646. return []
  647. for depth in sorted(levels):
  648. if len(levels[depth]) >= 2:
  649. return list(levels[depth].values())
  650. # No level has >= 2 distinct headings: a thin doc (e.g. one heading per
  651. # level). Count them all — this path runs only as a fallback when numeric
  652. # chapter detection already found zero, so it cannot inflate real books.
  653. return [
  654. title
  655. for depth in sorted(levels)
  656. for title in levels[depth].values()
  657. ]
  658. def _structural_chapter_count(text: str) -> int:
  659. """Count chapter-like structural headings in Markdown/AsciiDoc/RST sources."""
  660. return len(_structural_chapter_headings(text))
  661. def _cn_numeral_to_int(s: str) -> int | None:
  662. """Parse a Chinese (or ASCII-digit) chapter numeral into an int (1..999)."""
  663. if s.isdigit():
  664. n = int(s)
  665. return n if 1 <= n <= 999 else None
  666. section = current = 0
  667. for ch in s:
  668. if ch in _CN_NUM_VALUES:
  669. current = _CN_NUM_VALUES[ch]
  670. elif ch in _CN_NUM_UNITS:
  671. section += (current or 1) * _CN_NUM_UNITS[ch]
  672. current = 0
  673. else:
  674. return None
  675. total = section + current
  676. return total if 1 <= total <= 999 else None
  677. def _int_to_roman(n: int) -> str:
  678. table = [(1000, "M"), (900, "CM"), (500, "D"), (400, "CD"), (100, "C"),
  679. (90, "XC"), (50, "L"), (40, "XL"), (10, "X"), (9, "IX"),
  680. (5, "V"), (4, "IV"), (1, "I")]
  681. out = []
  682. for val, sym in table:
  683. while n >= val:
  684. out.append(sym)
  685. n -= val
  686. return "".join(out)
  687. def _roman_to_int(s: str) -> int | None:
  688. """Convert a Roman numeral to int, returning None if it isn't canonical."""
  689. s = s.upper()
  690. total = prev = 0
  691. for ch in reversed(s):
  692. v = _ROMAN_VALUES.get(ch)
  693. if v is None:
  694. return None
  695. total += -v if v < prev else v
  696. prev = max(prev, v)
  697. if total == 0 or total > 200:
  698. return None
  699. # Reject non-canonical forms ("IIII", "VV") by round-tripping.
  700. return total if _int_to_roman(total) == s else None
  701. def _is_prose_period_tail(tail: str, prev_line: str | None) -> bool:
  702. """True when a period tail is a wrapped sentence, not a title.
  703. The context decides first: a line that stands on its own (nothing above it,
  704. or the line above ends a sentence) keeps its heading. When the paragraph
  705. continues into the line, the tail must read like a title — a bare
  706. "Chapter 6." and a clause of more than `_TITLE_TAIL_MAX_WORDS` words are the
  707. sentence that wrapped. (Issue #237)
  708. """
  709. if not _PERIOD_TAIL.match(tail):
  710. return False
  711. if not prev_line or _SENTENCE_END.search(prev_line):
  712. return False
  713. words = _TAIL_WORD.findall(tail)
  714. return not words or len(words) > _TITLE_TAIL_MAX_WORDS
  715. def _match_chapter_number(
  716. line: str, prev_line: str | None = None, heading_marked: bool = False
  717. ) -> int | None:
  718. """Return the chapter number if the line is a genuine chapter heading,
  719. with no Markdown/AsciiDoc heading prefix (the caller strips it first).
  720. `prev_line` is the line directly above (None when it is blank or inside a
  721. fenced block); it is consulted only for a bare period tail. `heading_marked`
  722. says the line carried an explicit "#"/"==" prefix before it was stripped —
  723. an explicit heading is trusted as it stands, so "## Chapter 4. References
  724. are indicated by the `&` symbol…" keeps counting. (Issue #237)
  725. """
  726. # Normalize Kangxi-radical numerals (⼀⼆⼋⼗) to ideographs so Chinese
  727. # ebooks that encode chapter numbers in the U+2F00 block are detected.
  728. s = line.strip().translate(_KANGXI_NUMERAL_TRANS)
  729. if len(s) > 80:
  730. return None
  731. # Plain numbered chapter headings used by many technical books,
  732. # e.g. "1 Introduction" or "12 Advanced Topics".
  733. #
  734. # Require at least two spaces after the chapter number. This avoids
  735. # treating ordinary numbered list items such as "1. Item" as chapters.
  736. #
  737. # A unified-diff hunk line starts with exactly that shape
  738. # ("1 + use crate::trpl::StreamExt;", "12 - use std::io::prelude::*;").
  739. # Its first non-space character is the diff marker, which no chapter title
  740. # starts with, so the marker decides. Fenced blocks are skipped by the
  741. # caller; this covers code that arrives without fences. (Issue #237)
  742. plain = re.match(r"^([1-9]\d{0,2})\s{2,}(\S)", s)
  743. if plain and plain.group(2) not in "+-":
  744. return int(plain.group(1))
  745. m = _EXPLICIT_CHAPTER.match(s)
  746. if m and _HEADING_TAIL.match(m.group("rest")):
  747. if not heading_marked and _is_prose_period_tail(m.group("rest"), prev_line):
  748. return None
  749. if m.group(1):
  750. return int(m.group(1))
  751. return _roman_to_int(m.group("roman").upper())
  752. rm = _ROMAN_HEAD.match(s) or _LC_MD_ROMAN.match(s)
  753. if rm:
  754. return _roman_to_int(rm.group(1))
  755. cm = _CN_CHAPTER.match(s) or _MD_CN_HEADING.match(s)
  756. if cm:
  757. return _cn_numeral_to_int(cm.group(1))
  758. tm = _TH_CHAPTER.match(s)
  759. if tm:
  760. return int(tm.group(1).translate(_TH_DIGIT_MAP))
  761. hm = _HI_CHAPTER.match(s)
  762. if hm:
  763. return int(hm.group(1).translate(_HI_DIGIT_MAP))
  764. bm = _BN_CHAPTER.match(s)
  765. if bm:
  766. return int(bm.group(1).translate(_BN_DIGIT_MAP))
  767. orm = _OR_CHAPTER.match(s)
  768. if orm:
  769. return int(orm.group(1).translate(_OR_DIGIT_MAP))
  770. tam = _TA_CHAPTER.match(s)
  771. if tam:
  772. return int(tam.group(1).translate(_TA_DIGIT_MAP))
  773. sim = _SI_CHAPTER.match(s)
  774. if sim:
  775. return int(sim.group(1).translate(_SI_DIGIT_MAP))
  776. tem = _TE_CHAPTER.match(s)
  777. if tem:
  778. return int(tem.group(1).translate(_TE_DIGIT_MAP))
  779. knm = _KN_CHAPTER.match(s)
  780. if knm:
  781. return int(knm.group(1).translate(_KN_DIGIT_MAP))
  782. mlm = _ML_CHAPTER.match(s)
  783. if mlm:
  784. return int(mlm.group(1).translate(_ML_DIGIT_MAP))
  785. gum = _GU_CHAPTER.match(s)
  786. if gum:
  787. return int(gum.group(1).translate(_GU_DIGIT_MAP))
  788. pam = _PA_CHAPTER.match(s)
  789. if pam:
  790. return int(pam.group(1).translate(_PA_DIGIT_MAP))
  791. rum = _RU_CHAPTER.match(s)
  792. if rum:
  793. return int(rum.group(1))
  794. elm = _EL_CHAPTER.match(s)
  795. if elm:
  796. return int(elm.group(1))
  797. hem = _HE_CHAPTER.match(s)
  798. if hem:
  799. return int(hem.group(1))
  800. hym = _HY_CHAPTER.match(s)
  801. if hym:
  802. return int(hym.group(1))
  803. km = _KO_CHAPTER.match(s)
  804. if km:
  805. return int(km.group(1))
  806. fa = _fa_chapter_number(s)
  807. if fa is not None:
  808. return fa
  809. return None
  810. def _chapter_number(line: str, prev_line: str | None = None) -> int | None:
  811. """Return the chapter number if the line is a genuine chapter heading.
  812. `prev_line` is the line directly above (None when it is blank or inside a
  813. fenced block); see `_match_chapter_number` for how it is used.
  814. Handles Arabic ("Chapter 5", "Capítulo 5: ..."), Roman-numeral
  815. ("I: Loomings", "## i. introduction", "II. The Carpet-Bag"),
  816. Chinese ("第三章 …", "## 一 · …", "## 第一讲"), Thai ("บทที่ 3",
  817. "## บทที่ ๑"), Hindi ("अध्याय 1", "अध्याय १", "## अध्याय 2"),
  818. Bengali ("অধ্যায় 1", "অধ্যায় ১", "## অধ্যায় 2"),
  819. Odia ("ଅଧ୍ୟାୟ 1", "ଅଧ୍ୟାୟ ୧", "## ଅଧ୍ୟାୟ 2"),
  820. Tamil ("அத்தியாயம் 1", "அத்தியாயம் ௧", "## அத்தியாயம் 2"),
  821. Sinhala ("පරිච්ඡේදය 1", "අධ්‍යාය ෨", "## පරිච්ඡේදය 2"),
  822. Telugu ("అధ్యాయము 1", "అధ్యాయం ౧", "## అధ్యాయం 2"),
  823. Kannada ("ಅಧ್ಯಾಯ 1", "ಅಧ್ಯಾಯ ೧", "## ಅಧ್ಯಾಯ 2"),
  824. Malayalam ("അധ്യായം 1", "അധ്യായം ൧", "## അദ്ധ്യായം 2"),
  825. Gujarati ("પ્રકરણ 1", "અધ્યાય ૧", "## પ્રકરણ 2"),
  826. Punjabi ("ਕਾਂਡ 1", "ਅਧਿਆਇ ੧", "## ਕਾਂਡ 2"),
  827. Russian ("Глава 1", "ГЛАВА 12", "## Глава 2"),
  828. Greek ("Κεφάλαιο 1", "ΚΕΦΑΛΑΙΟ 12", "## Κεφάλαιο 2"),
  829. Hebrew ("פרק 1", "פרק 12", "## פרק 2"),
  830. Armenian ("Գլուխ 1", "ԳԼՈՒԽ 12", "## Գլուխ 2"),
  831. Korean ("제1장 총칙", "## 제4장 근로시간과 휴식"), and
  832. Persian ("فصل ۱", "فصل اول", "فصل بیست و یکم", "بخش ۲: مفاهیم",
  833. "## فصل ۱: مقدمه", PDF-glued "فصل سی و چهارمخداحافظ…") heading styles — each
  834. optionally preceded by a Markdown/AsciiDoc heading marker
  835. ("## Chapter 1" is a chapter heading just like "Chapter 1").
  836. """
  837. match = _match_chapter_number(line, prev_line)
  838. if match is not None:
  839. return match
  840. # Second pass: a Markdown/AsciiDoc heading prefix ("## Chapter 1",
  841. # "== Section") hides the heading from the matchers above — the CJK
  842. # matchers tolerate the prefix inline but the Latin/Thai/Korean ones anchor
  843. # on the line start. Strip the prefix and retry so --mode technical
  844. # (Docling emits headings as Markdown) detects the same chapters as
  845. # plain-text extraction. (Issue #91)
  846. #
  847. # No previous line is passed and the line is marked as heading-marked: an
  848. # explicit "#"/"==" prefix is a structural claim, so a period tail there
  849. # ("## Chapter 6. Enums", "## Chapter 4. References are indicated by…") is
  850. # trusted as written.
  851. s = line.strip()
  852. md = _MD_HEADING_PREFIX.match(s)
  853. if md:
  854. return _match_chapter_number(s[md.end():], heading_marked=True)
  855. return None
  856. def _line_title_text(line: str) -> str:
  857. """Normalize a source line to its heading title, if any."""
  858. stripped = line.strip()
  859. match = _ATX_HEADING.match(stripped)
  860. return match.group(2).strip() if match else stripped
  861. _TOC_LEADER = re.compile(r"\.{3,}")
  862. _PAGE_CHAPTER_HEADER = re.compile(r"^\d+\s*\|\s*Chapter\s+\d", re.IGNORECASE)
  863. _CHAPTER_MARKER = re.compile(r"^\s*CHAPTER\s+\d+\s*$", re.IGNORECASE)
  864. _OUTLINE_CHAPTER_PREFIX = re.compile(r"^chapter\s+\d+[.:]?\s*", re.IGNORECASE)
  865. def _outline_bare_title(outline_title: str) -> str:
  866. return _OUTLINE_CHAPTER_PREFIX.sub("", outline_title.strip(), count=1).strip()
  867. def _normalized_heading(line: str) -> str:
  868. return _line_title_text(line).strip().rstrip(".").casefold()
  869. def _is_disqualified_chapter_line(line: str) -> bool:
  870. stripped = line.strip()
  871. if not stripped:
  872. return True
  873. if _PAGE_CHAPTER_HEADER.match(stripped):
  874. return True
  875. if _TOC_LEADER.search(stripped):
  876. return True
  877. return False
  878. def _chapter_start_at(lines: list[str], title_line: int) -> int:
  879. if title_line > 0 and _CHAPTER_MARKER.match(lines[title_line - 1].strip()):
  880. return title_line - 1
  881. return title_line
  882. def _line_matches_outline_title(lines: list[str], index: int, outline_title: str) -> int | None:
  883. """Return the 0-based start line if *outline_title* begins at *index*, else None."""
  884. if _is_disqualified_chapter_line(lines[index]):
  885. return None
  886. bare = _outline_bare_title(outline_title)
  887. full_cf = outline_title.strip().casefold()
  888. bare_cf = bare.casefold()
  889. line = lines[index].strip()
  890. heading_cf = _normalized_heading(lines[index])
  891. if heading_cf == bare_cf or heading_cf == full_cf or line.casefold() == bare_cf:
  892. return _chapter_start_at(lines, index)
  893. if index + 1 < len(lines) and not _is_disqualified_chapter_line(lines[index + 1]):
  894. combined = f"{line} {lines[index + 1].strip()}".casefold()
  895. if combined == bare_cf or combined == full_cf:
  896. return _chapter_start_at(lines, index)
  897. if _CHAPTER_MARKER.match(line):
  898. for look in range(index + 1, min(index + 4, len(lines))):
  899. if _is_disqualified_chapter_line(lines[look]):
  900. continue
  901. if _normalized_heading(lines[look]) == bare_cf:
  902. return index
  903. if look + 1 < len(lines) and not _is_disqualified_chapter_line(lines[look + 1]):
  904. pair = f"{lines[look].strip()} {lines[look + 1].strip()}".casefold()
  905. if pair == bare_cf:
  906. return index
  907. return None
  908. def _find_outline_chapter_start(lines: list[str], title: str, start: int) -> int | None:
  909. """0-based line index where the outline chapter begins in extracted text."""
  910. for index in range(start, len(lines)):
  911. found = _line_matches_outline_title(lines, index, title)
  912. if found is not None:
  913. return found
  914. return None
  915. def _is_isolated_heading_line(lines: list[str], index: int) -> bool:
  916. if not lines[index].strip():
  917. return False
  918. prev_blank = index == 0 or not lines[index - 1].strip()
  919. next_blank = index + 1 >= len(lines) or not lines[index + 1].strip()
  920. return prev_blank and next_blank
  921. def _is_chapter_content_boundary(
  922. lines: list[str], index: int, range_end_titles: frozenset[str]
  923. ) -> bool:
  924. """True when a book- or part-level framing heading ends the chapter body."""
  925. stripped = lines[index].strip()
  926. if not stripped or _PAGE_CHAPTER_HEADER.match(stripped):
  927. return False
  928. heading_cf = _normalized_heading(lines[index])
  929. if heading_cf in range_end_titles:
  930. return True
  931. title = _line_title_text(lines[index])
  932. if _PART_HEADER.match(title) or _NON_CHAPTER_PREFIX.match(title):
  933. return _is_isolated_heading_line(lines, index)
  934. return False
  935. def _first_chapter_boundary(
  936. lines: list[str], start: int, limit: int, range_end_titles: frozenset[str]
  937. ) -> int:
  938. """Earliest line in [start, limit) that ends the preceding chapter body."""
  939. for index in range(start, limit):
  940. if _is_chapter_content_boundary(lines, index, range_end_titles):
  941. return index
  942. return limit
  943. def _range_end_title_keys(titles: list[str]) -> frozenset[str]:
  944. return frozenset(_normalized_heading(title) for title in titles if title.strip())
  945. def chapter_line_ranges(
  946. text: str,
  947. chapter_titles: list[str],
  948. *,
  949. range_end_titles: list[str] | None = None,
  950. ) -> list[dict[str, int | str]]:
  951. """1-based inclusive line ranges for outline chapters in *text*."""
  952. lines = text.splitlines()
  953. end_keys = _range_end_title_keys(range_end_titles or [])
  954. starts: list[tuple[str, int]] = []
  955. cursor = 0
  956. for title in chapter_titles:
  957. line_index = _find_outline_chapter_start(lines, title, cursor)
  958. if line_index is None:
  959. print(
  960. f"WARNING: outline chapter not found in extracted text: {title!r}",
  961. file=sys.stderr,
  962. )
  963. continue
  964. starts.append((title, line_index))
  965. cursor = line_index + 1
  966. if not starts:
  967. return []
  968. ranges: list[dict[str, int | str]] = []
  969. for index, (title, start_line) in enumerate(starts):
  970. next_start = starts[index + 1][1] if index + 1 < len(starts) else len(lines)
  971. end_exclusive = _first_chapter_boundary(
  972. lines, start_line + 1, next_start, end_keys
  973. )
  974. ranges.append(
  975. {
  976. "title": title,
  977. "start_line": start_line + 1,
  978. "end_line": end_exclusive,
  979. }
  980. )
  981. return ranges
  982. def detect_structure(text: str) -> dict:
  983. """Detect chapter count and table of contents presence.
  984. Scans the whole text (not just the head) and counts DISTINCT chapter numbers
  985. from explicit "Chapter N"/"Capítulo N" headings, rejecting prose
  986. cross-references and numbered list items. Counting distinct numbers means a
  987. ToC entry and its body heading are not double-counted.
  988. Lines inside closed code fences are skipped, the same guard
  989. `_structural_chapter_count()` uses: a sample holding a diff
  990. ("1 + use crate::trpl::StreamExt;") otherwise matched the numeric scan and
  991. the count built from code lines won over the structural count. A period
  992. tail is judged against the line above it so a wrapped cross-reference
  993. ("...as we cover in / Chapter 6. Closures create types...") is not counted
  994. while a heading on its own line still is. (Issue #237)
  995. """
  996. lines = text.splitlines()
  997. fenced = _closed_fence_line_numbers(lines)
  998. headings = []
  999. numbers = set()
  1000. part_headings = []
  1001. part_numbers = set()
  1002. prev = "" # previous non-blank line; cleared by blanks and fences
  1003. for index, line in enumerate(lines):
  1004. if index in fenced:
  1005. prev = ""
  1006. continue
  1007. s = line.strip()
  1008. if not s:
  1009. prev = ""
  1010. continue
  1011. num = _chapter_number(line, prev)
  1012. if num is not None:
  1013. title = _MD_HEADING_PREFIX.sub("", s)
  1014. if _PART_HEADER.match(title):
  1015. part_numbers.add(num)
  1016. part_headings.append(s)
  1017. else:
  1018. numbers.add(num)
  1019. headings.append(s)
  1020. prev = s
  1021. numeric_count = len(numbers)
  1022. # Fall back to structural (Markdown/AsciiDoc) headings only when no numeric
  1023. # "Chapter N" headings were found, so books with real chapters are unaffected.
  1024. #
  1025. # Which branch answered is reported alongside the count. The two disagree
  1026. # often, and a wrong count is not visible in the output it produces: it
  1027. # becomes the plan in Step 3 and the chapter files of the generated skill.
  1028. # Every parser in this project already announces which method it used
  1029. # ("Trying python-docx... OK"); this decision had the same shape and was
  1030. # the only silent one.
  1031. if numeric_count >= 2:
  1032. chapters_detected = numeric_count
  1033. chapters_method = "numeric"
  1034. chapter_headings_sample = headings[:10]
  1035. else:
  1036. # A single stray number (e.g. a Roman numeral inside an example paper
  1037. # reproduced in the book, or a lone "Part 1") is not enough to suppress
  1038. # the structural (Markdown/AsciiDoc) heading count, so course-style
  1039. # books with "### Unit N" headings still get counted via max().
  1040. structural_headings = _structural_chapter_headings(text)
  1041. structural_count = len(structural_headings)
  1042. # Parts group chapters and must not win the numeric branch ahead of
  1043. # the structural chapters inside them. Retain the old Part-only
  1044. # fallback when the extraction contains no other chapter evidence.
  1045. if not numeric_count and not structural_count:
  1046. numeric_count = len(part_numbers)
  1047. headings = part_headings
  1048. chapters_detected = max(numeric_count, structural_count)
  1049. chapters_method = (
  1050. "structural" if structural_count > numeric_count
  1051. else "numeric" if numeric_count
  1052. else "none"
  1053. )
  1054. chapter_headings_sample = (
  1055. structural_headings[:10]
  1056. if chapters_method == "structural"
  1057. else headings[:10]
  1058. )
  1059. # Look for ToC indicators in the first ~30k chars (multilingual; see _TOC_PATTERN)
  1060. has_toc = bool(_TOC_PATTERN.search(text[:30000]))
  1061. return {
  1062. "chapters_detected": chapters_detected,
  1063. "chapters_method": chapters_method,
  1064. "chapter_headings_sample": chapter_headings_sample,
  1065. "has_toc": has_toc,
  1066. }
  1067. def apply_pdf_outline(structure: dict, pdf_path: str) -> dict:
  1068. """Prefer the PDF's embedded outline over the text heuristics.
  1069. The text scan treats any short line starting with a number and two spaces
  1070. ("1 Introduction") as a chapter heading, so page-layout lines such as
  1071. numbered figure/listing captions and ToC rows inflate a technical book to
  1072. hundreds of "chapters". A publisher outline is authoritative; the
  1073. heuristics remain the fallback when the PDF has none.
  1074. """
  1075. titles = [t for t in pdf_outline_titles(pdf_path) if not _is_non_chapter_section(t)]
  1076. if len(titles) < 2:
  1077. return structure
  1078. all_titles = pdf_outline_all_titles(pdf_path)
  1079. range_end_titles = [t for t in all_titles if _is_non_chapter_section(t)]
  1080. return {
  1081. **structure,
  1082. "chapters_detected": len(titles),
  1083. "chapters_method": "outline",
  1084. "chapter_headings_sample": titles[:10],
  1085. "chapter_titles": titles,
  1086. "chapter_range_end_titles": range_end_titles,
  1087. }
  1088. def parse_arguments(argv: list[str]) -> tuple[list[str], str, str]:
  1089. """Parse argv into (input_paths, extraction_mode, install_mode)."""
  1090. input_paths = []
  1091. extraction_mode = "text"
  1092. args = argv[1:]
  1093. i = 0
  1094. while i < len(args):
  1095. arg = args[i]
  1096. if arg == "--mode":
  1097. if i + 1 < len(args):
  1098. extraction_mode = args[i+1].lower()
  1099. i += 2
  1100. else:
  1101. i += 1
  1102. elif arg == "--install-missing":
  1103. if i + 1 < len(args) and not args[i+1].startswith("--"):
  1104. i += 2
  1105. else:
  1106. i += 1
  1107. elif arg == "--no-install-missing":
  1108. i += 1
  1109. elif arg == "--pdf-page-sidecar":
  1110. i += 1
  1111. elif arg.startswith("-"):
  1112. print(f"WARNING: Unknown flag '{arg}' — ignoring it.", file=sys.stderr)
  1113. i += 1
  1114. else:
  1115. input_paths.append(arg)
  1116. i += 1
  1117. install_mode = normalize_install_mode(argv)
  1118. if extraction_mode not in ("technical", "text"):
  1119. extraction_mode = "text"
  1120. return input_paths, extraction_mode, install_mode
  1121. def resolve_input_files(paths: list[str]) -> list[Path]:
  1122. """Resolve paths including files, directories, and glob patterns to Path objects.
  1123. User-given order is preserved for explicit file arguments. Expanded
  1124. results (directories, globs) are sorted deterministically so repeated
  1125. runs produce the same output.
  1126. A leading "~" is expanded here rather than relying on the shell: a glob has
  1127. to be quoted to reach us unexpanded ("~/books/*.epub"), and quoting stops
  1128. the shell expanding the tilde too. `glob.glob` and `Path` both treat "~" as
  1129. a literal directory name, so without this the pattern silently matches
  1130. nothing.
  1131. """
  1132. resolved = []
  1133. for raw_path in paths:
  1134. # Normalise "~" once, at the entry point, so both the glob branch and
  1135. # the file/directory branch below see a real path.
  1136. path_str = os.path.expanduser(raw_path)
  1137. # Check if it has glob wildcards
  1138. if not Path(path_str).exists() and any(
  1139. char in path_str for char in ("*", "?", "[")
  1140. ):
  1141. glob_matches = glob.glob(path_str, recursive=True)
  1142. # Sort expanded glob results deterministically
  1143. expanded = []
  1144. for match in glob_matches:
  1145. p = Path(match)
  1146. if p.is_file() and p.suffix.lower() in SUPPORTED_EXTENSIONS:
  1147. expanded.append(p.resolve())
  1148. expanded.sort(key=lambda x: str(x).lower())
  1149. resolved.extend(expanded)
  1150. else:
  1151. p = Path(path_str)
  1152. if p.is_dir():
  1153. # Sort expanded directory results deterministically
  1154. dir_files = []
  1155. for root, _, files in os.walk(p):
  1156. for file in files:
  1157. file_path = Path(root) / file
  1158. if file_path.suffix.lower() in SUPPORTED_EXTENSIONS:
  1159. dir_files.append(file_path.resolve())
  1160. dir_files.sort(key=lambda x: str(x).lower())
  1161. resolved.extend(dir_files)
  1162. else:
  1163. # Keep even if it doesn't exist so the error check can report it
  1164. resolved.append(p.resolve())
  1165. # Deduplicate while preserving insertion order (user order for explicit files)
  1166. seen = set()
  1167. unique_paths = []
  1168. for path in resolved:
  1169. resolved_path = path.resolve() if path.exists() else path
  1170. if resolved_path not in seen:
  1171. seen.add(resolved_path)
  1172. unique_paths.append(resolved_path)
  1173. return unique_paths
  1174. def _sha256_file(path: str) -> str:
  1175. """Streaming sha256 — constant memory for multi-GB sources."""
  1176. h = hashlib.sha256()
  1177. with open(path, "rb") as f:
  1178. for chunk in iter(lambda: f.read(1 << 20), b""):
  1179. h.update(chunk)
  1180. return h.hexdigest()
  1181. def reuse_is_safe(current_inputs, metadata, current_mode):
  1182. """Decide whether an interrupted run's extraction workdir is safe to resume.
  1183. current_inputs: list of (path, fingerprint) — fingerprint is the sha256
  1184. hexdigest of each file as it exists NOW, recomputed by the caller
  1185. (hashlib.sha256(path.read_bytes()).hexdigest(); _sha256_file is available
  1186. for streaming on large files). Size is deliberately NOT part of matching:
  1187. metadata.json only records file_size_mb rounded to 2 decimals, so an
  1188. exact-size comparison is unimplementable — the fingerprint subsumes size
  1189. entirely.
  1190. metadata: the recorded run's metadata dict (parsed metadata.json).
  1191. current_mode: the extraction mode THIS run intends ("technical" or "text").
  1192. Returns (ok, reason). ok=True only when ALL hold:
  1193. - metadata["workdir"] exists and is a directory
  1194. - metadata["extraction_mode"] == current_mode
  1195. - the recorded sources match the current inputs one-to-one on
  1196. filename AND sha256 — the recomputed fingerprints vs the recorded ones
  1197. Anything else: (False, reason) — the caller must start a fresh
  1198. extraction (or ask the user when unsure), never resume.
  1199. """
  1200. if not isinstance(metadata, dict):
  1201. return False, "metadata.json unreadable or not an object"
  1202. workdir = metadata.get("workdir")
  1203. if not workdir or not os.path.isdir(workdir):
  1204. return False, "workdir missing or unreadable"
  1205. if metadata.get("extraction_mode") != current_mode:
  1206. return False, (
  1207. f"mode changed (recorded {metadata.get('extraction_mode')!r}, "
  1208. f"current is {current_mode!r})"
  1209. )
  1210. recorded = metadata.get("sources") or []
  1211. if len(recorded) != len(current_inputs):
  1212. return False, "sources changed"
  1213. current_pairs = [(Path(path).name, fingerprint) for path, fingerprint in current_inputs]
  1214. current_by_name = {}
  1215. for filename, fingerprint in current_pairs:
  1216. current_by_name.setdefault(filename, []).append(fingerprint)
  1217. recorded_pairs = []
  1218. for src in recorded:
  1219. filename = src.get("filename")
  1220. if filename not in current_by_name:
  1221. return False, f"sources changed (recorded {filename!r} is not in this run)"
  1222. recorded_hash = src.get("sha256")
  1223. if not recorded_hash:
  1224. return False, (
  1225. f"no recorded fingerprint for {filename} — legacy metadata "
  1226. "(recorded before fingerprints existed)"
  1227. )
  1228. recorded_pairs.append((filename, recorded_hash))
  1229. if recorded_pairs != current_pairs:
  1230. if Counter(recorded_pairs) == Counter(current_pairs):
  1231. return False, "sources reordered"
  1232. recorded_name_counts = Counter(filename for filename, _ in recorded_pairs)
  1233. current_name_counts = Counter(filename for filename, _ in current_pairs)
  1234. if all(count == 1 for count in recorded_name_counts.values()) and all(
  1235. count == 1 for count in current_name_counts.values()
  1236. ):
  1237. for filename, recorded_hash in recorded_pairs:
  1238. if recorded_hash != current_by_name[filename][0]:
  1239. return False, f"content changed for {filename}"
  1240. return False, "sources changed"
  1241. return True, "workdir intact and all sources match the current inputs"
  1242. def extract_single_file(input_path: Path, extraction_mode: str, install_mode: str) -> dict:
  1243. """Extract text and metadata from a single file path."""
  1244. input_str = str(input_path)
  1245. if not input_path.exists():
  1246. raise ExtractionError(f"File not found: {input_str}")
  1247. ext = input_path.suffix.lower()
  1248. document_format = ext.lstrip(".")
  1249. # Sniff magic bytes if suffix is not supported.
  1250. #
  1251. # Every failure in this function has to surface as ExtractionError: the
  1252. # batch loop in main() catches only that, and anything else aborts the whole
  1253. # run — including the sources that would have extracted fine. An unreadable
  1254. # or unopenable file is a per-source problem, so translate it here. (The
  1255. # ZipFile branch below already does this for OSError.)
  1256. if ext not in SUPPORTED_EXTENSIONS:
  1257. try:
  1258. with open(input_str, "rb") as f:
  1259. header = f.read(8)
  1260. except OSError as exc:
  1261. raise ExtractionError(
  1262. f"Could not read {input_path.name}: {exc.strerror or exc}"
  1263. ) from exc
  1264. if header[:4] == b"%PDF":
  1265. ext = ".pdf"
  1266. document_format = "pdf"
  1267. elif header[:2] == b"PK":
  1268. try:
  1269. with zipfile.ZipFile(input_str) as zf:
  1270. names = set(zf.namelist())
  1271. if "mimetype" in names and zf.read("mimetype").startswith(b"application/epub"):
  1272. ext = ".epub"
  1273. document_format = "epub"
  1274. elif "word/document.xml" in names:
  1275. ext = ".docx"
  1276. document_format = "docx"
  1277. else:
  1278. raise ExtractionError(
  1279. f"Unsupported ZIP-based format '{input_path.name}'. Supported: {supported_formats_message()}"
  1280. )
  1281. except (zipfile.BadZipFile, KeyError, OSError):
  1282. raise ExtractionError(
  1283. f"Unsupported ZIP-based format '{input_path.name}'. Supported: {supported_formats_message()}"
  1284. )
  1285. else:
  1286. raise ExtractionError(
  1287. f"Unsupported format '{ext or '<none>'}'. Supported: {supported_formats_message()}"
  1288. )
  1289. prepare_dependencies(ext, extraction_mode, install_mode)
  1290. if ext in CALIBRE_EBOOK_EXTENSIONS and not shutil.which("ebook-convert"):
  1291. raise ExtractionError(
  1292. "MOBI/AZW/AZW3 extraction requires Calibre's ebook-convert command. "
  1293. "Install Calibre and ensure ebook-convert is on PATH, then rerun this command."
  1294. )
  1295. text = ""
  1296. method = ""
  1297. fallback_reason = None
  1298. pages = 0
  1299. pages_label = "sections"
  1300. images_dropped = None
  1301. if ext == ".epub":
  1302. print(f"Extracting EPUB: {input_str}")
  1303. if extraction_mode == "technical":
  1304. fallback_reason = "format_not_supported_in_technical_mode"
  1305. print(
  1306. "WARNING: Technical mode is not supported for EPUB; "
  1307. "using the EPUB text extraction chain. "
  1308. "Layout-aware extraction was not used.",
  1309. file=sys.stderr,
  1310. )
  1311. text = extract_with_ebooklib(input_str)
  1312. if text and text.strip():
  1313. method = "ebooklib"
  1314. else:
  1315. print("ebooklib not available")
  1316. print("Trying stdlib zipfile parser...", end=" ", flush=True)
  1317. text = extract_with_zipfile(input_str)
  1318. if text and text.strip():
  1319. print("OK")
  1320. method = "zipfile"
  1321. else:
  1322. print("FAILED")
  1323. raise ExtractionError(
  1324. "Could not extract text from EPUB.\n"
  1325. "Install ebooklib + beautifulsoup4 for best results:\n"
  1326. " pip3 install ebooklib beautifulsoup4"
  1327. )
  1328. pages = count_epub_chapters(input_str)
  1329. pages_label = "spine_items"
  1330. images_dropped = count_epub_images(input_str)
  1331. if images_dropped > _EPUB_IMAGE_NOTICE_THRESHOLD:
  1332. print(
  1333. f" [warn] {input_path.name} contains {images_dropped} image(s); "
  1334. "their content is not extracted",
  1335. file=sys.stderr,
  1336. )
  1337. elif ext == ".pdf":
  1338. print(f"Extracting PDF: {input_str}")
  1339. if looks_image_only(input_str):
  1340. raise ExtractionError(
  1341. f"{input_path.name} looks like a scanned (image-only) PDF: its first pages "
  1342. "contain no extractable text, only images.\n"
  1343. "Run OCR on it first, then retry:\n"
  1344. " ocrmypdf input.pdf output.pdf"
  1345. )
  1346. if extraction_mode == "technical":
  1347. print("Mode: technical — using Docling (layout-aware)...")
  1348. try:
  1349. text = extract_with_docling(input_str)
  1350. except Exception as exc:
  1351. fallback_reason = "docling_failed"
  1352. print(
  1353. f"WARNING: Docling failed ({type(exc).__name__}: {exc}); "
  1354. "using the PDF text fallback chain.",
  1355. file=sys.stderr,
  1356. )
  1357. text = None
  1358. if text and text.strip():
  1359. method = "docling"
  1360. print("OK")
  1361. else:
  1362. if fallback_reason is None:
  1363. if text is None:
  1364. fallback_reason = "docling_unavailable"
  1365. detail = "Docling is unavailable"
  1366. else:
  1367. fallback_reason = "docling_empty"
  1368. detail = "Docling produced no text"
  1369. print(
  1370. f"WARNING: {detail}; using the PDF text fallback chain.",
  1371. file=sys.stderr,
  1372. )
  1373. extraction_mode = "text"
  1374. if extraction_mode == "text" or not text:
  1375. print("Mode: text — using pdftotext...")
  1376. print("Trying pdftotext...", end=" ", flush=True)
  1377. text = extract_with_pdftotext(input_str)
  1378. if text and text.strip():
  1379. method = "pdftotext"
  1380. print("OK")
  1381. else:
  1382. print("not available")
  1383. print("Trying pypdf...", end=" ", flush=True)
  1384. text = extract_with_pypdf(input_str)
  1385. if text and text.strip():
  1386. method = "pypdf"
  1387. print("OK")
  1388. else:
  1389. print("not available")
  1390. print("Trying pdfminer.six...", end=" ", flush=True)
  1391. text = extract_with_pdfminer(input_str)
  1392. if text and text.strip():
  1393. method = "pdfminer"
  1394. print("OK")
  1395. else:
  1396. print("FAILED")
  1397. raise ExtractionError(
  1398. "Could not extract text from PDF.\n"
  1399. "Install one of: poppler-utils (pdftotext), pypdf, or pdfminer.six\n"
  1400. " sudo apt install poppler-utils\n"
  1401. " pip3 install pypdf\n"
  1402. " pip3 install pdfminer.six"
  1403. )
  1404. pages = count_pages(input_str)
  1405. pages_label = "pages"
  1406. elif ext in TEXT_EXTENSIONS:
  1407. print(f"Extracting text document: {input_str}")
  1408. text = read_text_file(input_str)
  1409. if text is None or not text.strip():
  1410. raise ExtractionError(f"Could not read text document: {input_path.name}")
  1411. method = "plain-text"
  1412. pages = 0
  1413. pages_label = "sections"
  1414. elif ext in HTML_EXTENSIONS:
  1415. print(f"Extracting HTML: {input_str}")
  1416. text = extract_html_file(input_str)
  1417. if text is None or not text.strip():
  1418. raise ExtractionError(f"Could not extract text from HTML: {input_path.name}")
  1419. method = "html-parser"
  1420. pages = 0
  1421. pages_label = "sections"
  1422. elif ext == ".docx":
  1423. print(f"Extracting DOCX: {input_str}")
  1424. text, method = extract_docx(input_str)
  1425. pages = 0
  1426. pages_label = "sections"
  1427. elif ext == ".rtf":
  1428. print(f"Extracting RTF: {input_str}")
  1429. text, method = extract_rtf(input_str)
  1430. pages = 0
  1431. pages_label = "sections"
  1432. elif ext in CALIBRE_EBOOK_EXTENSIONS:
  1433. print(f"Extracting ebook with Calibre: {input_str}")
  1434. text = extract_with_ebook_convert(input_str)
  1435. if text is None or not text.strip():
  1436. raise ExtractionError(
  1437. f"Could not extract text from {ext}. Install Calibre and ensure ebook-convert is on PATH."
  1438. )
  1439. method = "ebook-convert"
  1440. pages = 0
  1441. pages_label = "sections"
  1442. text, removed_invisible = sanitize_extracted_text(text)
  1443. if removed_invisible:
  1444. print(
  1445. f" [security] removed {removed_invisible} invisible Unicode "
  1446. f"code point(s) from {input_path.name}",
  1447. file=sys.stderr,
  1448. )
  1449. if not text.strip():
  1450. raise ExtractionError(
  1451. f"Extracted text from {input_path.name} contained no visible content "
  1452. "after Unicode sanitization."
  1453. )
  1454. tokens = estimate_tokens(text)
  1455. structure = detect_structure(text)
  1456. if ext == ".pdf":
  1457. structure = apply_pdf_outline(structure, input_str)
  1458. print(
  1459. f" chapters: {structure['chapters_detected']} "
  1460. f"({structure['chapters_method']})"
  1461. )
  1462. try:
  1463. file_size_mb = os.path.getsize(input_str) / (1024 * 1024)
  1464. file_sha256 = _sha256_file(input_str)
  1465. except OSError as exc:
  1466. raise ExtractionError(
  1467. f"Could not read file size for {input_path.name}: {exc}"
  1468. ) from exc
  1469. return {
  1470. "source_file": str(input_path.resolve()),
  1471. "filename": input_path.name,
  1472. "format": document_format,
  1473. "extraction_method": method,
  1474. "fallback_reason": fallback_reason,
  1475. "file_size_mb": round(file_size_mb, 2),
  1476. "sha256": file_sha256,
  1477. pages_label: pages,
  1478. "pages_label": pages_label,
  1479. "pages": pages,
  1480. "chars": len(text),
  1481. "words": len(text.split()),
  1482. "estimated_tokens": tokens,
  1483. "images_dropped": images_dropped,
  1484. "text": text,
  1485. **structure,
  1486. }
  1487. def prepare_output_dir(path: Path) -> None:
  1488. """Create the work directory, guarding against two shared-tmp risks:
  1489. a pre-planted symlink at a predictable path, and reusing a directory
  1490. another user already owns (either could expose or tamper with the
  1491. extracted document text, which may be sensitive).
  1492. """
  1493. if path.is_symlink():
  1494. raise ExtractionError(
  1495. f"Refusing to use {path}: it is a symbolic link, not a real "
  1496. "directory. Remove it or set BOOK_SKILL_WORKDIR to a private path."
  1497. )
  1498. if path.exists():
  1499. if not path.is_dir():
  1500. raise ExtractionError(f"Refusing to use {path}: it exists and is not a directory.")
  1501. if hasattr(os, "getuid"):
  1502. owner_uid = path.stat().st_uid
  1503. if owner_uid != os.getuid():
  1504. raise ExtractionError(
  1505. f"Refusing to use {path}: it is owned by a different user "
  1506. f"(uid {owner_uid}). Set BOOK_SKILL_WORKDIR to a private directory."
  1507. )
  1508. os.chmod(path, 0o700)
  1509. else:
  1510. path.mkdir(parents=True, mode=0o700)
  1511. def write_pdf_page_sidecar(sources: list[dict], output_dir: Path) -> Path | None:
  1512. """Write opt-in physical-page evidence without changing legacy outputs.
  1513. pypdf's page text is a separate text-layer view, not a character-offset map
  1514. into ``full_text.txt``. When the main extractor differs, mark that explicitly.
  1515. A source whose hash changes between extraction and this pass gets no page
  1516. evidence, so citations cannot silently refer to a different PDF revision.
  1517. """
  1518. entries = []
  1519. for source in sources:
  1520. if source["format"] != "pdf":
  1521. continue
  1522. entry = {
  1523. "source_file": source["source_file"],
  1524. "source_sha256": source["sha256"],
  1525. "full_text_extraction_method": source["extraction_method"],
  1526. "page_text_extraction_method": "pypdf",
  1527. "status": "unavailable",
  1528. "pages": [],
  1529. }
  1530. try:
  1531. if _sha256_file(source["source_file"]) != source["sha256"]:
  1532. entry["reason"] = "source_changed_since_extraction"
  1533. else:
  1534. records = extract_pypdf_page_records(source["source_file"])
  1535. if _sha256_file(source["source_file"]) != source["sha256"]:
  1536. entry["reason"] = "source_changed_during_page_extraction"
  1537. elif records is None:
  1538. entry["reason"] = "pypdf_unavailable_or_failed"
  1539. else:
  1540. entry["status"] = "ok"
  1541. if source["pages"] and len(records) != source["pages"]:
  1542. entry["low_confidence_reasons"] = ["page_count_mismatch"]
  1543. for record in records:
  1544. record["source_sha256"] = source["sha256"]
  1545. if source["extraction_method"] != "pypdf":
  1546. record["low_confidence_reasons"].append(
  1547. "different_extractor_from_full_text"
  1548. )
  1549. record["requires_manual_review"] = bool(
  1550. record["low_confidence_reasons"]
  1551. )
  1552. entry["pages"] = records
  1553. except OSError as exc:
  1554. entry["reason"] = f"source_unreadable: {type(exc).__name__}"
  1555. entries.append(entry)
  1556. payload = {"schema_version": 1, "sources": entries}
  1557. destination = output_dir / "pdf_pages.json"
  1558. temporary = None
  1559. try:
  1560. with tempfile.NamedTemporaryFile(
  1561. mode="w", encoding="utf-8", newline="\n", dir=output_dir,
  1562. prefix=".pdf_pages-", suffix=".tmp", delete=False,
  1563. ) as stream:
  1564. temporary = Path(stream.name)
  1565. json.dump(payload, stream, indent=2, ensure_ascii=False)
  1566. stream.write("\n")
  1567. os.replace(temporary, destination)
  1568. except OSError as exc:
  1569. print(f" [warn] could not write PDF page sidecar: {exc}", file=sys.stderr)
  1570. if temporary is not None:
  1571. temporary.unlink(missing_ok=True)
  1572. return None
  1573. return destination
  1574. def print_intro() -> None:
  1575. """Two lines of attribution at the start of every run.
  1576. Printed here rather than only in SKILL.md so it shows however the agent
  1577. invokes extraction. States who maintains the project without asking for
  1578. anything — the ask belongs at the end, after the work is delivered.
  1579. """
  1580. sys.stderr.write(
  1581. "book-to-skill · turns a document into a structured agent skill\n"
  1582. "free and MIT-licensed · maintained in personal time · "
  1583. "github.com/virgiliojr94/book-to-skill\n\n"
  1584. )
  1585. def print_support_note() -> None:
  1586. """One closing line about funding, printed only after a successful run.
  1587. Deliberately at the end and deliberately conditional: the reader has just
  1588. received something that worked, and the sentence says what the money is
  1589. for rather than asking for it. Never printed when extraction failed —
  1590. nobody should be asked to fund what just wasted their time.
  1591. Written to stdout, with the rest of the closing report: stderr is
  1592. unbuffered and stdout is not when the run is piped (which is how an agent
  1593. captures it), so mixing the two puts the closing line at the top.
  1594. """
  1595. print(
  1596. "\n book-to-skill is free, and maintained in personal time."
  1597. "\n If it saves you work, you can fund its upkeep: "
  1598. "github.com/sponsors/virgiliojr94"
  1599. )
  1600. def print_usage() -> None:
  1601. """Print standalone CLI usage."""
  1602. print(
  1603. "Usage: book-to-skill <path-to-document-folder-or-glob>... "
  1604. "[--mode technical|text] [--install-missing ask|yes|no] "
  1605. "[--pdf-page-sidecar]",
  1606. file=sys.stderr,
  1607. )
  1608. print(
  1609. " book-to-skill --check # report which extractors are installed",
  1610. file=sys.stderr,
  1611. )
  1612. print(f"Supported formats: {supported_formats_message()}", file=sys.stderr)
  1613. def main():
  1614. print_intro()
  1615. if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
  1616. print_usage()
  1617. sys.exit(0)
  1618. if "--check" in sys.argv[1:]:
  1619. sys.exit(run_dependency_check())
  1620. if len(sys.argv) < 2:
  1621. print_usage()
  1622. sys.exit(1)
  1623. raw_input_paths, extraction_mode, install_mode = parse_arguments(sys.argv)
  1624. if not raw_input_paths:
  1625. print("ERROR: No input document, folder, or glob pattern specified.", file=sys.stderr)
  1626. sys.exit(1)
  1627. input_files = resolve_input_files(raw_input_paths)
  1628. if not input_files:
  1629. print(f"ERROR: No supported files found matching: {', '.join(raw_input_paths)}", file=sys.stderr)
  1630. sys.exit(1)
  1631. prepare_output_dir(OUTPUT_DIR)
  1632. extracted_sources = []
  1633. combined_texts = []
  1634. errors = []
  1635. for file_path in input_files:
  1636. try:
  1637. res = extract_single_file(file_path, extraction_mode, install_mode)
  1638. except ExtractionError as exc:
  1639. print(f"WARNING: Skipping {file_path.name}: {exc}", file=sys.stderr)
  1640. errors.append((file_path, str(exc)))
  1641. continue
  1642. extracted_sources.append(res)
  1643. # Format the text with a clear boundary
  1644. separator = f"\n\n{'=' * 80}\nSOURCE: {res['filename']} (Path: {res['source_file']})\n{'=' * 80}\n\n"
  1645. combined_texts.append(separator + res["text"])
  1646. if not extracted_sources:
  1647. print(f"\nERROR: All {len(errors)} source(s) failed extraction:", file=sys.stderr)
  1648. for path, err in errors:
  1649. print(f" - {path.name}: {err}", file=sys.stderr)
  1650. sys.exit(1)
  1651. # Combine texts
  1652. consolidated_text = "".join(combined_texts).strip()
  1653. # Preserve existing source line endings. On Windows, the default newline
  1654. # translation turns an extracted CRLF into CRCRLF and breaks setext headings
  1655. # when the persisted corpus is read back.
  1656. with OUTPUT_TEXT.open("w", encoding="utf-8", newline="\n") as output_file:
  1657. output_file.write(consolidated_text)
  1658. # Consolidate metadata
  1659. total_file_size_mb = sum(src["file_size_mb"] for src in extracted_sources)
  1660. total_pages = sum(src["pages"] for src in extracted_sources)
  1661. total_chars = len(consolidated_text)
  1662. total_words = len(consolidated_text.split())
  1663. total_tokens = estimate_tokens(consolidated_text)
  1664. total_images_dropped = sum(
  1665. src["images_dropped"] or 0 for src in extracted_sources
  1666. )
  1667. # Detect structure from source content only. The generated SOURCE banners in
  1668. # full_text.txt use rows of "=", which can otherwise become phantom setext
  1669. # headings and make the result depend on the source-path length.
  1670. structure_text = "\n\n".join(src["text"] for src in extracted_sources)
  1671. consolidated_structure = detect_structure(structure_text)
  1672. if len(extracted_sources) == 1 and extracted_sources[0]["chapters_method"] == "outline":
  1673. # The outline came from the file, not the text; a re-scan would lose it.
  1674. src = extracted_sources[0]
  1675. consolidated_structure.update(
  1676. {k: src[k] for k in (
  1677. "chapters_detected",
  1678. "chapters_method",
  1679. "chapter_headings_sample",
  1680. "chapter_titles",
  1681. ) if k in src}
  1682. )
  1683. chapter_titles = src.get("chapter_titles")
  1684. if chapter_titles:
  1685. end_titles = src.get("chapter_range_end_titles")
  1686. consolidated_structure["chapter_ranges"] = chapter_line_ranges(
  1687. consolidated_text,
  1688. chapter_titles,
  1689. range_end_titles=end_titles,
  1690. )
  1691. structural_sources = [
  1692. src for src in extracted_sources
  1693. if src["chapters_method"] == "structural" and src["chapters_detected"] >= 2
  1694. ]
  1695. if (
  1696. len(extracted_sources) > 1
  1697. and structural_sources
  1698. and all(src["chapters_method"] != "numeric" for src in extracted_sources)
  1699. ):
  1700. # Each source has already selected its own chapter depth. Re-scanning
  1701. # the joined corpus selects the H1 source titles instead when there are
  1702. # multiple books with H2 chapters (issue #272). A lone heading in a
  1703. # source may only be its title, so it is not added to the chapter count.
  1704. consolidated_structure["chapters_detected"] = sum(
  1705. src["chapters_detected"] for src in structural_sources
  1706. )
  1707. consolidated_structure["chapter_headings_sample"] = [
  1708. heading
  1709. for src in structural_sources
  1710. for heading in src["chapter_headings_sample"]
  1711. ][:10]
  1712. consolidated_structure["chapters_method"] = "structural"
  1713. # has_toc is a per-source property, so it has to be combined per source
  1714. # rather than re-derived from the corpus. detect_structure only scans the
  1715. # first ~30k chars, because a table of contents sits in a book's front
  1716. # matter -- but on a consolidated corpus that window covers only the FIRST
  1717. # source, so a ToC in any later book was invisible and the answer flipped on
  1718. # input order alone. Each per-source result already scanned its own front
  1719. # matter, so OR them.
  1720. consolidated_structure["has_toc"] = any(
  1721. src["has_toc"] for src in extracted_sources
  1722. )
  1723. metadata = {
  1724. "source_file": "Consolidated from multiple sources" if len(extracted_sources) > 1 else extracted_sources[0]["source_file"],
  1725. "filename": "multi-source" if len(extracted_sources) > 1 else extracted_sources[0]["filename"],
  1726. "format": "mixed" if len(extracted_sources) > 1 else extracted_sources[0]["format"],
  1727. "extraction_method": "multi-method" if len(extracted_sources) > 1 else extracted_sources[0]["extraction_method"],
  1728. "extraction_mode": extraction_mode,
  1729. "file_size_mb": round(total_file_size_mb, 2),
  1730. "pages": total_pages,
  1731. "chars": total_chars,
  1732. "words": total_words,
  1733. "estimated_tokens": total_tokens,
  1734. "estimated_tokens_human": f"~{total_tokens // 1000}K",
  1735. "images_dropped": total_images_dropped,
  1736. # Self-describing so a consumer can clean up exactly the directory this
  1737. # run created, without having to reconstruct the per-run default path.
  1738. "workdir": str(OUTPUT_DIR),
  1739. "output_text": str(OUTPUT_TEXT),
  1740. "total_sources": len(extracted_sources),
  1741. "sources": [
  1742. {
  1743. "source_file": src["source_file"],
  1744. "filename": src["filename"],
  1745. "format": src["format"],
  1746. "extraction_method": src["extraction_method"],
  1747. "fallback_reason": src["fallback_reason"],
  1748. "file_size_mb": src["file_size_mb"],
  1749. "sha256": src["sha256"],
  1750. "pages": src["pages"],
  1751. "pages_label": src["pages_label"],
  1752. "chars": src["chars"],
  1753. "words": src["words"],
  1754. "estimated_tokens": src["estimated_tokens"],
  1755. "images_dropped": src["images_dropped"],
  1756. "chapters_detected": src["chapters_detected"],
  1757. "chapters_method": src["chapters_method"],
  1758. "chapter_headings_sample": src["chapter_headings_sample"],
  1759. **(
  1760. {"chapter_titles": src["chapter_titles"]}
  1761. if "chapter_titles" in src
  1762. else {}
  1763. ),
  1764. **(
  1765. {
  1766. "chapter_ranges": chapter_line_ranges(
  1767. src["text"],
  1768. src["chapter_titles"],
  1769. range_end_titles=src.get("chapter_range_end_titles"),
  1770. )
  1771. }
  1772. if "chapter_titles" in src
  1773. else {}
  1774. ),
  1775. "has_toc": src["has_toc"]
  1776. }
  1777. for src in extracted_sources
  1778. ],
  1779. **consolidated_structure,
  1780. }
  1781. # encoding="utf-8" is required, not cosmetic: the payload is dumped with
  1782. # ensure_ascii=False, so any non-ASCII chapter heading, filename or path
  1783. # reaches the encoder verbatim. Without it, write_text() falls back to the
  1784. # locale encoding and raises UnicodeEncodeError on a Windows cp1252 host or
  1785. # under LC_ALL=C — after every source has already been extracted.
  1786. OUTPUT_META.write_text(
  1787. json.dumps(metadata, indent=2, ensure_ascii=False), encoding="utf-8"
  1788. )
  1789. page_sidecar = None
  1790. if "--pdf-page-sidecar" in sys.argv[1:]:
  1791. page_sidecar = write_pdf_page_sidecar(extracted_sources, OUTPUT_DIR)
  1792. page_line = f" Total Pages: {total_pages}"
  1793. print("\nExtraction complete:")
  1794. print(f" Sources : {len(extracted_sources)} processed")
  1795. print(f" Size : {total_file_size_mb:.2f} MB")
  1796. print(page_line)
  1797. print(f" Words : {total_words:,}")
  1798. print(f" Tokens : ~{total_tokens // 1000}K")
  1799. print(
  1800. f" Chapters: {consolidated_structure['chapters_detected']} detected overall "
  1801. f"({consolidated_structure['chapters_method']})"
  1802. )
  1803. if consolidated_structure["chapters_method"] == "structural" and (
  1804. consolidated_structure["chapters_detected"] <= 1 and total_words > 5000
  1805. ):
  1806. # Numeric "Chapter N" headings found nothing and the structural fallback
  1807. # came back with one section for a document of real length. That pairing
  1808. # is a detection failure far more often than it is a one-chapter book,
  1809. # and it is invisible in the output it produces.
  1810. print(
  1811. " WARN : only one section found in a document this long — chapter "
  1812. "detection likely failed; check the headings before generating."
  1813. )
  1814. print(f" ToC : {'yes' if consolidated_structure['has_toc'] else 'not detected'}")
  1815. if not consolidated_structure["has_toc"]:
  1816. print(
  1817. " WARN : No table of contents detected — chapter mapping in Step 3 "
  1818. "will rely on heading scan only, which may miss or duplicate sections."
  1819. )
  1820. print(f"\n Workdir -> {OUTPUT_DIR}")
  1821. print(f" Text -> {OUTPUT_TEXT}")
  1822. print(f" Meta -> {OUTPUT_META}")
  1823. if page_sidecar is not None:
  1824. print(f" PDF pages -> {page_sidecar}")
  1825. if errors:
  1826. print(f"\n WARNING: {len(errors)} source(s) skipped due to errors:")
  1827. for path, err in errors:
  1828. print(f" - {path.name}: {err}")
  1829. else:
  1830. print_support_note()

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.