book-to-skill
"Converts books and documents (PDF, EPUB, DOCX, HTML, Markdown, plain text, RTF, MOBI/AZW with Calibre) into structured agent skills, extracting frameworks, mental models, principles, techniques, and anti-patterns. Use when the user wants to study a document through GitHub Copilot CLI, Amp, Claude Code, Hermes Agent, OpenCode, or OpenClaw, apply an author's frameworks while working, or build a reusable knowledge base from a file."
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/virgiliojr94-book-to-skill)HTML
<a href="/listings/virgiliojr94-book-to-skill"><img src="/api/listings/virgiliojr94-book-to-skill/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
# One command, any host — via the cross-agent skills CLI:
npx skills add virgiliojr94/book-to-skill
# Or manually — clone into your skills folder (registers /book-to-skill):
git clone https://github.com/virgiliojr94/book-to-skill.git ~/.claude/skills/book-to-skill
# (Copilot CLI: ~/.copilot/skills/ · Amp/cross-agent: ~/.agents/skills/)
# (Hermes Agent: ${HERMES_HOME:-$HOME/.hermes}/skills/<category>/)
# (OpenClaw: ${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/skills/; ~/.agents/skills/ only with default state)
# (OpenCode: ~/.agents/skills/ — it also reads ~/.config/opencode/skills/)📥 **All hosts, optional extractors, and the standalone CLI → [docs/install.md](docs/install.md)**
---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 880 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- ---
- name: book-to-skill
- description: "Converts books and documents (PDF, EPUB, DOCX, HTML, Markdown, plain text, RTF, MOBI/AZW with Calibre) into structured agent skills, extracting frameworks, mental models, principles, techniques, and anti-patterns. Use when the user wants to study a document through GitHub Copilot CLI, Amp, Claude Code, Hermes Agent, OpenCode, or OpenClaw, apply an author's frameworks while working, or build a reusable knowledge base from a file."
- ---
- <!--
- Cross-agent notes (informational; ignored by host agents):
- - Compatible skill roots: GitHub Copilot CLI (~/.copilot/skills, ~/.agents/skills,
- .github/skills, .claude/skills, .agents/skills), Amp (.agents/skills,
- ~/.config/agents/skills, ~/.config/amp/skills), Claude Code (~/.claude/skills),
- Hermes Agent ($HERMES_HOME/skills, .hermes/skills, .agents/skills),
- OpenClaw (${OPENCLAW_STATE_DIR:-~/.openclaw}/skills, .agents/skills, skills/;
- ~/.agents/skills only with the default state).
- OpenCode (~/.config/opencode/skills, ~/.claude/skills, .opencode/skills,
- ~/.agents/skills, .agents/skills).
- - `allowed-tools` is intentionally omitted to stay agent-neutral: Copilot CLI uses
- `shell`/MCP-server names, Claude uses `Bash`/`Read`/`Write`/`Glob`/`Grep`, Amp
- adds `shell_command`. The skill needs shell (to run extract.py) and file
- read/write — each host will prompt for those on first use.
- - Argument hint: <path-to-document-folder-or-glob>... [skill-name-slug]
- -->
- # Book-to-Skill Converter
- Transform written knowledge into actionable agent skills by extracting structure — not producing summaries.
- ## Philosophy
- Books contain crystallized expertise: frameworks, principles, and techniques that took years to develop. This skill extracts that knowledge into a format GitHub Copilot CLI, Amp, Claude Code, Hermes Agent, OpenCode, OpenClaw, or another compatible agent can leverage repeatedly.
- **Extract structure, not summaries.** A skill isn't a book report. It's a toolkit of:
- - Named frameworks (mental models with clear application)
- - Actionable principles (rules that guide decisions)
- - Techniques (step-by-step methods)
- - Anti-patterns (what to avoid and why)
- - Voice calibration (how the author thinks and communicates)
- **Preserve the author's precision.** Frameworks often have specific names for reasons. "The 5 Whys" isn't interchangeable with "ask why multiple times." Capture the exact formulation.
- **Layer depth appropriately.** Simple books → simple skills. Complex books with 10+ frameworks → skills with reference files and on-demand chapters.
- ---
- ## Modes of Operation
- Four paths available. Route based on what the user asks:
- ### 1. Full Conversion (Default)
- **Trigger:** User provides one or more document/directory/glob paths without special instructions
- **Action:** Run all steps below (Steps 0–9)
- **Output:** Complete skill with SKILL.md, chapters/, glossary, patterns, cheatsheet
- ### 2. Analyze Only
- **Trigger:** User says "analyze", "just extract", or "I want to review before generating"
- **Action:** Run Steps 0–3, then produce a structured extraction report (frameworks, principles, techniques found). Stop — do NOT generate skill files.
- **Output:** Analysis report for user review
- ### 3. Generate from Prior Analysis
- **Trigger:** User has existing analysis notes or previously ran analyze-only
- **Action:** Skip Steps 0–3, use the provided analysis as input, run Steps 4–9
- **Output:** Skill files from the provided analysis
- ### 4. Update / Fold-in (Existing Skill)
- **Trigger:** User provides one or more new source paths and indicates they want to update an existing skill (either by pointing to the existing skill folder, providing a skill slug that already exists in `SKILLS_HOME`, or explicitly requesting an update).
- **Action:** Run Step 0 (out-of-scope check), Step 1 (validate inputs), Step 1.5 (identify book type), and Step 2 (extract new files). Then skip to Step 5 (identify/detect existing skill path) and run the **Update / Fold-in Workflow** to merge the new content into the existing skill files.
- **Output:** Updated existing skill with new/revised chapter summaries and merged indexes/glossaries.
- ---
- ## Skill Locations
- This converter can run from multiple skill systems. When looking for this converter's helper script or writing the generated book skill, prefer these locations in order:
- 1. GitHub Copilot CLI personal skills: `~/.copilot/skills/`
- 2. Cross-agent personal skills (Copilot, Amp, Codex; OpenClaw with its default state): `~/.agents/skills/`
- 3. Claude Code personal skills: `~/.claude/skills/`
- 4. Project-local Copilot skills: `.github/skills/`
- 5. Project-local Claude skills: `.claude/skills/`
- 6. Project-local Amp / Copilot / OpenClaw skills: `.agents/skills/`
- 7. Amp global skills: `~/.config/agents/skills/`
- 8. Amp legacy global skills: `~/.config/amp/skills/`
- 9. Hermes Agent personal skills: `$HERMES_HOME/skills/` (defaults to `~/.hermes/skills/`)
- 10. Hermes Agent project skills: `.hermes/skills/` or `.agents/skills/`
- 11. OpenClaw personal skills: `${OPENCLAW_STATE_DIR:-~/.openclaw}/skills/` (active state; `~/.agents/skills/` is shared only with the default state)
- 12. OpenClaw project skills: `.agents/skills/` or `skills/`
- 11. OpenCode personal skills: `~/.config/opencode/skills/` or `~/.agents/skills/`
- 12. OpenCode project skills: `.opencode/skills/` or `.agents/skills/`
- For **generated** book skills, prefer the user-level cross-agent root `~/.agents/skills/` — one physical copy serves the cross-agent hosts and OpenClaw when it uses its default state. Copilot CLI and Amp discover it natively; Claude Code needs a symlink from `~/.claude/skills/<skill_name>` (created in Step 10, see Step 5 for the rules). Pick a host-private or project-local root only when the user explicitly asks for one. `BOOK_TO_SKILL_SCOPE=project` or `personal` can make that choice explicit for automation; do not ask a mandatory scope question merely because both scopes are available.
- ---
- ## Step 0 — Out-of-scope check
- If no arguments are provided, stop and respond:
- > "book-to-skill requires a supported document path, folder, or glob pattern. Usage: `book-to-skill <path-to-document-folder-or-glob>... [skill-name-slug]`"
- Throughout the workflow:
- - **Resolve the destination root first.** Before any existing-skill lookup, determine where this run would write: the personal or project-local root for the user's host, per the Step 5 table and selection rules — a project-local request uses its project-local root, never the personal root. Below, "the resolved root" means that destination.
- - Identify the input paths and the optional skill slug.
- - If the last argument is not a file, folder, or glob that exists or matches any files, and it looks like a skill slug (e.g. lowercase hyphens, alphanumeric), treat it as `SKILL_NAME`.
- - Treat all other arguments as the list of `INPUT_PATHS`.
- - If any input path is an existing skill directory (contains `SKILL.md` and a `chapters/` sub-folder), or if `SKILL_NAME` matches an existing skill slug in the resolved root, flag this run as an **Update/Fold-in** operation (Mode 4).
- - **Re-run guard.** Before starting extraction for a Full Conversion, derive the prospective skill slug under Step 5's naming options — the author-concept form, the by-title form, and any `SKILL_NAME` given — and check the resolved root for an existing match. On a match, STOP and ask the user: "`<skill-name>` already exists. Choose: (1) Update/Fold-in (Mode 4), (2) verify the existing skill is complete and stop, or (3) force full regeneration." Do not re-extract until the user chooses. On context recovery after an interrupted run (network cut-off, replayed or continued conversation), reuse the extraction work directory that run reported (`Workdir ->` in its output, or the path you set in `BOOK_SKILL_WORKDIR`) **only if it is still intact and still matches the current inputs and options** — the directory exists, its `metadata.json` is readable, the `sources` it lists match the files being converted now one-to-one on filename **and content fingerprint** (the extractor records a per-source `sha256` in `metadata.json`; recompute the fingerprint from each file as it exists now and require an exact match — `reuse_is_safe()` in `book_to_skill/utils.py` is the executable form of this check), and its recorded `extraction_mode` is the mode this run is using (Step 2's "confirm the extraction is the document you asked for" rule). If it is missing (temp cleanup), its `metadata.json` is gone, the sources no longer match (a different filename, or a changed content fingerprint), the recorded metadata has no `sha256` for a source (recorded before fingerprints existed, so freshness cannot be established), or the mode differs, say so and start a fresh extraction instead of resuming. When in doubt, ask the user before discarding or resuming.
- ---
- ## Step 1 — Validate input
- Verify that there is at least one supported file, directory, or glob pattern among the `INPUT_PATHS`.
- For directories and globs, expand them to find matching supported files (`.pdf`, `.epub`, `.docx`, `.txt`, `.md`, `.markdown`, `.rst`, `.adoc`, `.html`, `.htm`, `.rtf`, `.mobi`, `.azw`, `.azw3`).
- If no supported files are found, stop with a clear error message.
- ---
- ## Step 1.5 — Identify content type
- Before extracting, ask the user:
- > "What kind of content do these sources have? This helps me choose the best extraction method.
- >
- > 1. **Technical** — has code blocks, tables, formulas, diagrams (e.g. programming books, academic papers, architecture guides)
- > 2. **Text-heavy** — mostly prose, few or no tables/code (e.g. management, productivity, narrative non-fiction)
- > 3. **Not sure** — I'll use the fast method and warn you if quality seems limited"
- Store the answer as `BOOK_TYPE`:
- - Option 1 → `BOOK_TYPE=technical`
- - Option 2 → `BOOK_TYPE=text`
- - Option 3 → `BOOK_TYPE=text`
- **If `BOOK_TYPE=technical`**, inform the user before proceeding:
- > "📐 Technical mode selected — PDFs attempt Docling for layout-aware extraction. EPUBs and other formats use their existing text extractors; code, tables, and formulas may lose structure. Check each source's `extraction_method` and `fallback_reason` in metadata.json for the actual extraction path."
- **If `BOOK_TYPE=text`**, inform:
- > "📄 Text mode selected — using the fastest suitable extractor for each file type. Plain text/Markdown/HTML are usually ready in seconds; PDFs use pdftotext when available."
- ---
- ## Step 2 — Extract text from the source documents
- Run the extraction script, passing the input paths:
- ```bash
- SCRIPT_PATH=""
- HERMES_HOME_RESOLVED="${HERMES_HOME:-$HOME/.hermes}"
- OPENCLAW_STATE_DIR_RESOLVED="${OPENCLAW_STATE_DIR:-$HOME/.openclaw}"
- PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)"
- HERMES_PROJECT_TRUSTED=false
- if [ -n "$PROJECT_ROOT" ] && [ "${HERMES_AGENT:-}" = true ] && \
- command -v hermes >/dev/null 2>&1 && \
- command -v python3 >/dev/null 2>&1 && \
- hermes config get skills.trusted_project_dirs --json 2>/dev/null | PROJECT_ROOT="$PROJECT_ROOT" python3 -c 'import json, os, pathlib, sys; root=pathlib.Path(os.environ["PROJECT_ROOT"]).resolve(); sys.exit(not any(pathlib.Path(p).expanduser().resolve() == root for p in json.load(sys.stdin)))' 2>/dev/null
- then
- HERMES_PROJECT_TRUSTED=true
- fi
- CANDIDATES=(
- "$HOME/.copilot/skills/book-to-skill/scripts/extract.py"
- "$HOME/.agents/skills/book-to-skill/scripts/extract.py"
- "$HOME/.claude/skills/book-to-skill/scripts/extract.py"
- "${OPENCLAW_STATE_DIR_RESOLVED}/skills/book-to-skill/scripts/extract.py"
- "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/book-to-skill/scripts/extract.py
- "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/book-to-skill/scripts/extract.py
- "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/*/book-to-skill/scripts/extract.py
- "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/*/*/book-to-skill/scripts/extract.py
- "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/*/*/*/book-to-skill/scripts/extract.py
- "${OPENCLAW_STATE_DIR_RESOLVED}/skills"/*/*/*/*/*/*/book-to-skill/scripts/extract.py
- "$HOME/.config/opencode/skills/book-to-skill/scripts/extract.py"
- "$HERMES_HOME_RESOLVED/skills/book-to-skill/scripts/extract.py"
- "$HERMES_HOME_RESOLVED"/skills/*/book-to-skill/scripts/extract.py
- )
- if [ "${HERMES_AGENT:-}" != true ]; then
- # Project-local roots are resolved against the git worktree, not the current
- # directory: an agent can be invoked from anywhere inside the project (e.g.
- # `src/nested`) and these roots still have to be found — that is how OpenCode
- # and the other hosts discover project skills. The CWD-relative forms are kept
- # so the probe also works outside a git repository.
- if [ -n "$PROJECT_ROOT" ]; then
- CANDIDATES+=(
- "$PROJECT_ROOT/.github/skills/book-to-skill/scripts/extract.py"
- "$PROJECT_ROOT/.claude/skills/book-to-skill/scripts/extract.py"
- "$PROJECT_ROOT/.agents/skills/book-to-skill/scripts/extract.py"
- "$PROJECT_ROOT/.opencode/skills/book-to-skill/scripts/extract.py"
- )
- fi
- CANDIDATES+=(
- ".github/skills/book-to-skill/scripts/extract.py"
- ".claude/skills/book-to-skill/scripts/extract.py"
- ".agents/skills/book-to-skill/scripts/extract.py"
- "skills/book-to-skill/scripts/extract.py"
- "skills"/*/book-to-skill/scripts/extract.py
- "skills"/*/*/book-to-skill/scripts/extract.py
- "skills"/*/*/*/book-to-skill/scripts/extract.py
- "skills"/*/*/*/*/book-to-skill/scripts/extract.py
- "skills"/*/*/*/*/*/book-to-skill/scripts/extract.py
- "skills"/*/*/*/*/*/*/book-to-skill/scripts/extract.py
- ".opencode/skills/book-to-skill/scripts/extract.py"
- )
- if [ -n "$PROJECT_ROOT" ]; then
- CANDIDATES+=(
- "$PROJECT_ROOT/skills/book-to-skill/scripts/extract.py"
- "$PROJECT_ROOT/skills"/*/book-to-skill/scripts/extract.py
- "$PROJECT_ROOT/skills"/*/*/book-to-skill/scripts/extract.py
- "$PROJECT_ROOT/skills"/*/*/*/book-to-skill/scripts/extract.py
- "$PROJECT_ROOT/skills"/*/*/*/*/book-to-skill/scripts/extract.py
- "$PROJECT_ROOT/skills"/*/*/*/*/*/book-to-skill/scripts/extract.py
- "$PROJECT_ROOT/skills"/*/*/*/*/*/*/book-to-skill/scripts/extract.py
- )
- fi
- fi
- CANDIDATES+=(
- "$HOME/.config/agents/skills/book-to-skill/scripts/extract.py"
- "$HOME/.config/amp/skills/book-to-skill/scripts/extract.py"
- )
- if [ "$HERMES_PROJECT_TRUSTED" = true ]; then
- CANDIDATES=(
- "$PROJECT_ROOT/.hermes/skills/book-to-skill/scripts/extract.py"
- "$PROJECT_ROOT/.hermes/skills"/*/book-to-skill/scripts/extract.py
- "$PROJECT_ROOT/.agents/skills/book-to-skill/scripts/extract.py"
- "$PROJECT_ROOT/.agents/skills"/*/book-to-skill/scripts/extract.py
- "${CANDIDATES[@]}"
- )
- fi
- for candidate in "${CANDIDATES[@]}"
- do
- if [ -f "$candidate" ]; then
- SCRIPT_PATH="$candidate"
- break
- fi
- done
- if [ -z "$SCRIPT_PATH" ]; then
- echo "Could not find scripts/extract.py for book-to-skill" >&2
- exit 1
- fi
- PYTHON_BIN="${PYTHON_BIN:-python3}"
- if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
- PYTHON_BIN="python"
- fi
- "$PYTHON_BIN" "$SCRIPT_PATH" $INPUT_PATHS --mode <BOOK_TYPE> --install-missing ask
- ```
- Before extraction, the script checks optional Python packages needed for the detected format. If a better extractor is missing, it prompts the user with the available fallback. Non-interactive sessions default to fallback unless install mode is explicitly `yes`.
- **Tip — preflight the environment:** run `"$PYTHON_BIN" "$SCRIPT_PATH" --check` to print a per-format report of which extractors are installed and the exact command to install whatever is missing, without processing any file. Useful when a user reports a setup or quality problem.
- This creates a **per-run** work directory — `<tempdir>/book_skill_work-<pid>/` by default, or exactly the path you set in `BOOK_SKILL_WORKDIR` — containing:
- - `full_text.txt` — combined extracted text of all sources with clear visually demarcated boundaries.
- - `metadata.json` — overall combined size, words, pages, token counts, dropped EPUB image counts, the resolved `workdir`, and a detailed list of individual processed `sources`.
- The run prints all three paths on completion (`Workdir ->`, `Text ->`, `Meta ->`). **Take the paths from that output (or from `metadata.json`'s own `workdir` field) rather than assuming a fixed location** — the directory name differs per run so that concurrent extractions on one machine cannot overwrite each other's results.
- Read that run's `metadata.json` to inspect the results.
- **Always confirm the extraction is the document you asked for** before generating anything: check `filename` / `source_file` in `metadata.json`, or the `SOURCE:` header on the first line of `full_text.txt`. If you are waiting on a background run, wait on *its* specific workdir — polling a shared path can surface a different run's output.
- ---
- ## Step 2.5 — Pre-flight cost estimate
- Read this run's `metadata.json` (the `Meta ->` path from the extraction output) and present the user with an estimate **before doing any generation**:
- ```
- 📖 Sources detected: <total_sources> source(s)
- <list each source filename and format from the sources metadata list>
- <if images_dropped > 5: warn that N source images were not read>
- 📄 Combined Pages/Sections: ~<N> | Words: ~<N> | Total tokens: ~<N>K
- 💰 Estimated token cost (Full Conversion / Update):
- Input (reading + prompts): ~<N>K tokens
- Output (skill files generated/updated): ~<N>K tokens
- Total: ~<N>K tokens
- Cost: multiply the token counts above by your model's current
- input/output per-1M-token rates (prices and model names change often —
- do not hardcode them; quote today's rate and label it as an estimate).
- ⏱ Estimated time: ~<N> minutes
- 📁 Files to be generated/updated:
- SKILL.md + chapter files + glossary + patterns + cheatsheet
- ➡ Proceed with Full Conversion / Update? (or type "analyze only" to preview first)
- ```
- **How to estimate:**
- - Input tokens ≈ `estimated_tokens` from metadata × 1.3 (prompts overhead per chapter pass)
- - Output tokens ≈ chapters × per-chapter budget + 4,000 (SKILL.md) + 4,500 (glossary + patterns + cheatsheet)
- - Per-chapter budget midpoint by `BOOK_TYPE` (DEPTH is decided later in Step 4 and can raise it): `text` ≈ 1,000, `technical` ≈ 1,800. If the user has already indicated reference-only vs deep study, use the matching row of the Step 7 matrix.
- - Cost: report the token counts and multiply by the user's current per-1M-token input/output rates. Do NOT hardcode dollar figures — model names and prices change; if you show one, label it an estimate and date it.
- Wait for the user to confirm before proceeding. If they say "analyze only", switch to Mode 2.
- ---
- ## Step 2.6 — REPL-style access for large books (> 50k tokens)
- Inspired by the Recursive Language Model (RLM) paradigm: treat `full_text.txt` as a queryable corpus, not a single read. Loading the whole file into context burns budget you will need later for generation.
- For books over ~50k tokens, prefer programmatic probes over `Read(full_text.txt)` without bounds:
- ```bash
- # Size check before any Read
- wc -w "$FULL_TEXT_PATH"
- # Find chapter offsets without loading the whole file
- grep -n -E "^\s*(Chapter|CHAPTER)\s+[0-9]+" "$FULL_TEXT_PATH" | head -40
- # Pull only the chapter you need (lines start..end inclusive)
- sed -n '<start>,<end>p' "$FULL_TEXT_PATH"
- # Verify a framework is actually mentioned before claiming it in SKILL.md
- grep -c -i "westrum\|dora" "$FULL_TEXT_PATH"
- # Targeted Read with offset/limit avoids dumping the full file
- # Read(file_path=full_text.txt, offset=<line>, limit=<lines>)
- ```
- Use this approach for Step 3 (structure analysis), Step 7 (per-chapter summaries), and Step 8 (glossary / patterns extraction). On books under 50k tokens, a single `Read` is fine.
- Why this matters: a 200-page book is ~75k tokens. Re-reading it once per chapter (28 passes) costs ~2M input tokens; using grep + sed to pull only relevant slices keeps generation cost proportional to the output, not the source.
- ---
- ## Step 3 — Analyze book structure
- Read the first 8,000 characters of the extracted `full_text.txt` to identify:
- - Book **title** and **author(s)**
- - **Chapter structure** (look for "Chapter N", "PART I", numbered headings, table of contents)
- - **Core themes** and subject domain
- - Approximate number of chapters
- Then read the Table of Contents section if present to map all chapters.
- **If mode is "Analyze Only":** produce the extraction report now and stop. Structure:
- ```
- ## Extraction Report — <Title>
- ### Author's Core Frameworks
- - **<Framework Name>**: <what it is and when to apply>
- ### Key Principles
- - <Principle>: <actionable rule>
- ### Techniques & Methods
- - <Technique>: <step-by-step or how-to>
- ### Anti-patterns
- - <What to avoid>: <why>
- ### Suggested Skill Name
- `{author-lastname}-{core-concept}` — e.g. `cialdini-influence`
- ### Chapters Detected
- | # | Title | Main Frameworks |
- ```
- ---
- ## Step 4 — Ask purpose (Full Conversion only)
- Before generating, ask the user:
- > "What should this skill help you do? (Pick one or more)
- > 1. Apply the author's frameworks while working
- > 2. Think with the author's mental models
- > 3. Reference specific chapters and concepts
- > 4. All of the above"
- Use the answer to weight what gets highlighted in the SKILL.md Core section.
- **Derive `DEPTH` from the answer (no extra prompt):**
- - Answer is **only** option 3 (reference) → `DEPTH=reference` — lean, fast-lookup chapters.
- - Answer includes option 1, 2, or 4 → `DEPTH=study` — deeper chapters with more worked detail, examples, and reasoning.
- `DEPTH` and `BOOK_TYPE` together set the per-chapter token budget in Step 7. Do **not** ask a separate "study vs reference" question — it is inferred here. (In Modes 2/3, where Step 4 is skipped, default `DEPTH=study`.)
- ---
- ## Step 5 — Determine skill name
- If `SKILL_NAME` was provided, use it as the skill slug.
- Otherwise, propose two options and let the user choose:
- - **By author-concept**: `{author-lastname}-{core-concept}` (e.g. `cialdini-influence`, `meadows-systems`)
- - **By title**: lowercase hyphens from book title (e.g. `designing-data-intensive-apps`)
- Default to author-concept format if the book has a strong methodological identity.
- Choose the destination skill root (`SKILLS_HOME`). First resolve **scope** from an explicit user request or `BOOK_TO_SKILL_SCOPE`, then probe **host**. A request for project-local/project output selects the project-local row; a request for personal/global output selects the personal row. If neither scope is requested, preserve the established personal default (`~/.agents/skills` for non-Hermes hosts). Do not ask a mandatory scope question solely because project-local roots exist. The selected root may still require host approval before writing.
- | Host agent | Personal skill root | Project-local root |
- |---|---|---|
- | **GitHub Copilot CLI** | `~/.agents/skills` (discovered natively) | `.github/skills` → `.claude/skills` → `.agents/skills` |
- | **Amp** | `~/.agents/skills` (discovered natively) | `.agents/skills` |
- | **OpenAI Codex** | `~/.agents/skills` (discovered natively; follows symlinks) | `.agents/skills` |
- | **Hermes Agent** | `$HERMES_HOME/skills/<category>` (defaults to `~/.hermes/skills/<category>`) | `.hermes/skills/<category>` → `.agents/skills` |
- | **Claude Code** | `~/.agents/skills` + symlink from `~/.claude/skills/<skill_name>` | `.claude/skills` |
- | **OpenClaw** | `${OPENCLAW_STATE_DIR:-~/.openclaw}/skills` (active state; `~/.agents/skills` only with default state) | `.agents/skills` → `skills/` |
- Hermes Agent is the one host that keeps its own personal root: it partitions personal skills by category and does not scan the cross-agent root. Use the active profile's `HERMES_HOME` and choose a category that matches the generated skill's subject. Do not construct profile paths manually. If the user selects a project-local Hermes root, run `hermes skills trust <project-root>` after generation and verify discovery with `hermes skills list`; project skills remain unavailable until the project is trusted.
- For OpenClaw, use the active state directory's `skills/` root: `${OPENCLAW_STATE_DIR:-~/.openclaw}/skills/`. The shared `~/.agents/skills` compatibility root is discoverable only when `OPENCLAW_STATE_DIR` is unset or the default `~/.openclaw`; with a non-default state, do not claim that OpenClaw will see a shared-root install. Verify discovery with `openclaw skills list` after generation.
- | **OpenCode** | `~/.agents/skills` (discovered natively; `~/.config/opencode/skills` and `~/.claude/skills` are read too) | `.opencode/skills` → `.agents/skills` → `.claude/skills` |
- Hermes Agent is the one host that keeps its own personal root: it partitions personal skills by category and does not scan the cross-agent root. Use the active profile's `HERMES_HOME` and choose a category that matches the generated skill's subject. Do not construct profile paths manually. If the user selects a project-local Hermes root, run `hermes skills trust <project-root>` after generation and verify discovery with `hermes skills list`; project skills remain unavailable until the project is trusted.
- OpenCode scans the cross-agent `~/.agents/skills` root natively, so the generated-skill default above already serves it — no symlink and no trust step. It also reads its own managed `~/.config/opencode/skills` root and the Claude compatibility root `~/.claude/skills`; use the OpenCode-managed root only when the user asks for it. Project skills are discovered by walking up from the working directory to the git worktree, so start a new session after generation if the skill does not appear. (`~/.cache/opencode/skills` is the remote-skill download cache from `skills.urls`, not an authoring root — never write a generated skill there.)
- Selection rules:
- 1. Personal install: set `SKILLS_HOME` to `~/.agents/skills` (create the directory if missing). One exception, so the default does not invent a convention in someone else's house: if `~/.agents/skills` does not exist **and** the host's private root already contains skills, use the private root instead and say why in the report.
- 2. **Claude Code does not scan `~/.agents/skills`** — after generation completes, Step 10 links the skill in with `ln -sfn "$HOME/.agents/skills/<skill_name>" "$HOME/.claude/skills/<skill_name>"`.
- 3. **Hermes Agent personal installs use the Hermes row above**, not the cross-agent root, and take no symlink.
- 4. If the user explicitly asks for a host-private root (`~/.copilot/skills`, `~/.claude/skills`, `~/.config/agents/skills`, `~/.config/amp/skills`), honor it and skip the symlink.
- 5. If the user explicitly asked for project-local output, use the project-local row for their host.
- 6. If the choice requires knowing the host (project-local output, the Hermes personal root, the OpenClaw state root, or the Claude Code symlink) and you cannot identify it, ask: "Which agent are you running in — OpenCode, OpenClaw, Hermes Agent, GitHub Copilot CLI, Amp, Codex, or Claude Code?"
- 7. For OpenClaw personal output, use `${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/skills`. The shared `~/.agents/skills` root is a valid OpenClaw destination only when `OPENCLAW_STATE_DIR` is unset or equals the default `$HOME/.openclaw`; otherwise use the active state root or a project/extra directory.
- 8. If the user explicitly asks for an OpenClaw-managed personal root, use the active state root and verify discovery with `openclaw skills list`.
- Set `SKILLS_HOME` to the selected root and check if `$SKILLS_HOME/<skill_name>/` already exists. On Claude Code, also check whether `~/.claude/skills/<skill_name>` exists as a **real directory** (not a symlink) — a previous install may live there; if so, offer to migrate it (move the directory into `~/.agents/skills/` and replace the original path with the symlink) before continuing.
- If the skill already exists, prompt the user to choose:
- 1. **Update / Fold-in** (Mode 4) — integrate new files/content into the existing skill components.
- 2. **Overwrite** — delete and regenerate the skill from scratch.
- 3. **Rename** — append `-2` or use a different custom slug.
- If the user selects **Update / Fold-in**, proceed immediately to the **Update / Fold-in Workflow** section after Step 2.5 (skipping Steps 3, 4, 6, 7, 8, 9).
- ---
- ## Step 6 — Create skill directory structure
- ```bash
- mkdir -p "$SKILLS_HOME/<skill_name>/chapters"
- ```
- ---
- ## Step 7 — Generate chapter summaries
- **TOKEN BUDGET RULE — CRITICAL (adaptive):**
- The per-chapter budget scales with `BOOK_TYPE` and `DEPTH`. Technical chapters need room for code and tables; study depth needs room for worked reasoning. Pick the budget from this matrix:
- | | `DEPTH=reference` | `DEPTH=study` |
- |---|---|---|
- | `BOOK_TYPE=text` | 800–1,200 tokens | 1,000–1,800 tokens |
- | `BOOK_TYPE=technical` | 1,200–1,800 tokens | 2,000–3,000 tokens |
- - These are per-file targets, not hard caps — a dense chapter may run over, a thin one under. Density still beats length (Quality Rule #3): never pad to hit a number.
- - Files are loaded on-demand, so a larger chapter only costs tokens when that chapter is actually read.
- - When in doubt between two cells (e.g. mixed-content book), use the lower budget and let depth come from precision, not volume.
- **`DEPTH=study` is earned with content, not a bigger number.** The standard section template (Core Idea → Connects To) naturally lands a dense prose chapter around 700–900 tokens. To reach the study budget *honestly* — not by padding — a study-depth chapter must add concrete material:
- - **Reproduce one worked example or artifact** from the chapter (e.g. the example press release, a sample dialogue, a filled-in template, a decision the author walks through) under a `## Worked Example` section. This is the single biggest lever and the main thing a learner returns for.
- - **Expand the "How" of each framework** into explicit steps or criteria, not a one-liner.
- - **Add a short "Why it works / failure mode" note** to the top 1–2 frameworks.
- If a chapter genuinely has no worked example and resists expansion, let it land below the study floor rather than padding — and note that the chapter is thin in its Core Idea. A `reference`-depth chapter, by contrast, deliberately omits worked examples and keeps only the decision-ready essentials.
- For EACH chapter/major section identified in Step 3:
- Read the corresponding section of the extracted `full_text.txt` using top-level `chapter_ranges` from `metadata.json` when present (`start_line`/`end_line` are 1-based inclusive, for `sed`/`Read`); otherwise grep for chapter headings. (`sources[].chapter_ranges` use the same fields but line numbers in that source's text only, not the consolidated `full_text.txt`.)
- Create `$SKILLS_HOME/<skill_name>/chapters/ch<NN>-<slug>.md` using the structure below.
- **Adapt emphasis based on `BOOK_TYPE`:**
- - `technical` → prioritize "Code Examples", "Reference Tables", and "Commands & APIs" sections; preserve exact syntax
- - `text` → prioritize "Frameworks Introduced", "Mental Models", and "Key Takeaways"; skip empty technical sections
- ```markdown
- # Chapter N: <Full Title>
- ## Core Idea
- <1–2 sentences: the single most important thing this chapter teaches>
- ## Frameworks Introduced
- - **<Framework Name>**: <exact formulation — preserve the author's naming>
- - When to use: <specific situation>
- - How: <steps or criteria>
- ## Key Concepts
- - **<Term>**: <precise definition in 1 sentence>
- (5–10 most important terms from this chapter)
- ## Mental Models
- <2–4 frameworks or thinking tools. Write as "Use X when Y" or "Think of X as Y">
- ## Anti-patterns
- - **<What to avoid>**: <why it fails>
- ## Code Examples *(technical books only — omit if BOOK_TYPE=text)*
- <!-- Copy the most instructive snippet from the chapter. Preserve indentation exactly. -->
- ```<language>
- <key code example from this chapter>
- ```
- - **What it demonstrates**: <one line>
- ## Reference Tables *(technical books only — omit if BOOK_TYPE=text)*
- <!-- Reproduce any comparison matrix, parameter table, or decision table from the chapter in markdown. -->
- ## Worked Example *(DEPTH=study only — omit for DEPTH=reference)*
- <!-- Reproduce or reconstruct one concrete example the author works through: a
- sample document, a dialogue, a filled-in template, a before/after, or a
- decision walked end-to-end. This is what makes a study chapter worth its
- budget. Keep it faithful to the source; never copy long raw passages —
- reconstruct the example compactly. -->
- ## Key Takeaways
- 1. <Actionable insight>
- 2. <Actionable insight>
- 3. <Actionable insight>
- (3–7 takeaways a practitioner must remember)
- ## Connects To
- - **Ch N**: <why this chapter relates>
- - **<Concept>**: <external concept or standard it connects with>
- ```
- ---
- ## Step 8 — Generate supporting files
- ### glossary.md
- Create `$SKILLS_HOME/<skill_name>/glossary.md`:
- - Every significant term from the book, alphabetically sorted
- - Format: `**Term** — definition (Ch N)`
- - Max 1,500 tokens
- ### patterns.md
- Create `$SKILLS_HOME/<skill_name>/patterns.md`:
- - All concrete techniques, design patterns, algorithms from the book
- - Format: `## Pattern Name\n**When to use**: ...\n**How**: ...\n**Trade-offs**: ...`
- - Max 2,000 tokens
- ### cheatsheet.md
- Create `$SKILLS_HOME/<skill_name>/cheatsheet.md`:
- **This is the most differentiated layer of the skill — treat it as a reasoning aid, not a keyword list.** Anyone can grep the glossary for a term. The cheatsheet captures the author's *judgment*: the decisions they'd make and why. It's the file that turns "I know the words" into "I'd act the way the author would".
- Prioritize, in order:
- 1. **Decision rules** — "When X, do Y, because Z." The if/then logic the author applies, stated so the reader can apply it without re-reading the book.
- 2. **Decision trees / flowcharts** (as nested bullets or a small table) — for choices with more than two branches.
- 3. **Trade-off matrices** — competing options scored on the dimensions the author cares about, so the reader can pick under their own constraints.
- 4. **Thresholds & defaults** — the specific numbers, ratios, or rules of thumb the author commits to (e.g. "keep functions under ~20 lines", "alert when error budget < 10%").
- 5. **Tells & smells** — fast heuristics for recognizing a situation ("if you see X, you're probably in trouble Y").
- Avoid: bare term→definition rows (that's the glossary), and prose paragraphs (that's the chapters). Every line should help the reader *decide* something.
- - Format mostly as compact tables and decision rules; the content you'd want on a single printed page kept beside you while working.
- - Max 1,200 tokens.
- ---
- ## Step 9 — Generate the master SKILL.md
- **CRITICAL TOKEN BUDGET: Keep SKILL.md body under 4,000 tokens.**
- Compaction truncates from the END — put the most important content FIRST.
- Create `$SKILLS_HOME/<skill_name>/SKILL.md`:
- ```markdown
- ---
- name: <skill_name>
- description: "Knowledge base from \"<Full Title>\" by <Author(s)>. Use when applying <author>'s frameworks for <key topics, 3–6 terms>, studying the book, or referencing its concepts."
- ---
- <!-- argument-hint: [topic, framework name, or chapter number] -->
- # <Full Title>
- **Author**: <Author(s)> | **Pages**: ~<N> | **Chapters**: <N> | **Generated**: <YYYY-MM-DD>
- ## How to Use This Skill
- - **Without arguments** — load core frameworks for reference
- - **With a topic** — ask about `replication`, `pricing`, or another indexed topic; I find and read the relevant chapter
- - **With chapter** — ask for `ch05`; I load that specific chapter
- - **Browse** — ask "what chapters do you have?" to see the full index
- When you ask about a topic not covered in Core Frameworks below, I will read
- the relevant chapter file before answering.
- ---
- ## Core Frameworks & Mental Models
- <!-- ~2,000 tokens: the author's most important named frameworks and principles.
- Preserve exact names. Write as "Use X when Y", "Prefer X over Y because Z".
- This is a toolkit, not a summary. -->
- <generate 2,000 tokens of the most critical frameworks and insights here>
- ---
- ## Chapter Index
- | # | Title | Key Frameworks |
- |---|-------|----------------|
- | [ch01](chapters/ch01-<slug>.md) | <Title> | <framework1>, <framework2> |
- | [ch02](chapters/ch02-<slug>.md) | <Title> | <framework1>, <framework2> |
- ...
- ## Topic Index
- <!-- Alphabetical. Major terms/frameworks → chapter(s) that cover them. -->
- - **<Term>** → ch<N>[, ch<N>]
- - **<Term>** → ch<N>
- ## Supporting Files
- - [glossary.md](glossary.md) — all key terms with definitions
- - [patterns.md](patterns.md) — all techniques and design patterns
- - [cheatsheet.md](cheatsheet.md) — quick reference tables and decision guides
- - Full text: `<output_text from metadata.json>` — grep it for depth the chapter files omit (local extraction, not shipped)
- ---
- ## Scope & Limits
- This skill covers the book content only. For hands-on implementation in your codebase,
- combine with project-specific tools. For topics beyond this book, check related skills
- or ask the agent directly.
- <if images_dropped > 5: state that N source images were not read>
- ```
- ---
- ## Step 9.5 — Scan the generated skill
- Before reporting success, loading the skill in another session, or publishing it, run the advisory security scan:
- ```bash
- SKILL_CONVERTER_ROOT="$(cd "$(dirname "$SCRIPT_PATH")/.." && pwd)"
- "$PYTHON_BIN" "$SKILL_CONVERTER_ROOT/tools/scan_generated_skill.py" "$SKILLS_HOME/<skill_name>"
- ```
- If the scanner exits non-zero, stop and ask a human to review its file/line findings. Do not silently rewrite the generated files, and do not load or publish the skill until the findings are resolved or explicitly accepted.
- ---
- ## Step 10 — Cleanup and report
- If the host is Claude Code and `SKILLS_HOME` is `~/.agents/skills` (the default personal install), expose the skill to Claude Code with a symlink — Claude Code only scans `~/.claude/skills`:
- ```bash
- mkdir -p "$HOME/.claude/skills"
- LINK="$HOME/.claude/skills/<skill_name>"
- TARGET="$HOME/.agents/skills/<skill_name>"
- if [ -d "$LINK" ] && [ ! -L "$LINK" ]; then
- CLAUDE_STATUS="skipped-realdir" # Step 5 migration declined; leave the old dir
- else
- ln -sfn "$TARGET" "$LINK" 2>/dev/null || true
- # Read the link back — do NOT trust that `ln` did what was asked. On Windows/MSYS
- # `ln -s` may COPY instead of link (or need Developer Mode / an elevated shell), and
- # PowerShell/cmd have no `ln` at all. The report must reflect what is on disk, not the
- # fact that the command ran.
- if [ -L "$LINK" ] && [ "$(readlink "$LINK")" = "$TARGET" ]; then
- CLAUDE_STATUS="linked"
- elif [ -e "$LINK" ]; then
- CLAUDE_STATUS="copy" # a real file/dir landed instead of a link
- else
- CLAUDE_STATUS="absent" # ln unavailable or refused
- fi
- fi
- ```
- The real-directory guard is required: `ln -sfn` into an existing real directory would nest the link *inside* it (`~/.claude/skills/<skill_name>/<skill_name>`), leaving Claude Code loading the stale copy. If the user declined the Step 5 migration, skip the symlink and say so in the report — Claude Code keeps using the old directory until it is migrated.
- **Read the link back before you report anything about it.** The symlink is a claim, not a fact: fill the "Discoverable by" line from `CLAUDE_STATUS` (what is actually on disk), never from "the command was issued". **Do not hard-fail when the link is missing or is a copy** — the skill exists at the hub and every other host still finds it; the honest report is "written to `~/.agents/skills/<skill_name>`; Claude Code will not see it until the link is created", not an abort. (Windows lead, unverified: a directory junction — `mklink /J` in an elevated `cmd`, or `New-Item -ItemType Junction` in PowerShell — needs neither Developer Mode nor a symlink privilege; if you attempt it, it does not change the read-back-then-report rule.)
- Skip this when the user chose a host-private or project-local root (Step 5, rules 3-4).
- Then clean up the extraction workdir:
- ```bash
- PYTHON_BIN="${PYTHON_BIN:-python3}"
- if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
- PYTHON_BIN="python"
- fi
- Remove **the work directory this run actually used** — the `Workdir ->` path from the
- extraction output, which is also stored as `workdir` in `metadata.json`. Never delete a
- directory you did not create: another extraction may be running beside yours.
- ```bash
- # WORKDIR is the path this run reported; quote it in case of spaces.
- rm -rf "$WORKDIR"
- ```
- Equivalently, if you still have the metadata file:
- ```bash
- "$PYTHON_BIN" - "$WORKDIR_METADATA_JSON" <<'PY'
- import json
- import shutil
- import sys
- from pathlib import Path
- meta_path = Path(sys.argv[1])
- workdir = json.loads(meta_path.read_text(encoding="utf-8")).get("workdir")
- if workdir:
- shutil.rmtree(workdir, ignore_errors=True)
- PY
- ```
- Older copies of this file removed a single fixed `book_skill_work` directory. That path is
- no longer used, so such a cleanup is now a harmless no-op rather than something that could
- delete a concurrent run's output.
- Then report to the user:
- ```
- ✅ Skill created: $SKILLS_HOME/<skill_name>/
- 📚 Book: <Full Title> — <Author>
- 📄 Pages: ~<N> | Chapters: <N>
- Files generated:
- SKILL.md — core frameworks + index (~X tokens)
- chapters/ — <N> chapter summaries (~X tokens each, ~X total)
- glossary.md — key terms (~X tokens)
- patterns.md — techniques & patterns (~X tokens)
- cheatsheet.md — quick reference (~X tokens)
- ─────────────────────────────────────────────────────
- Total skill size: ~X tokens (loaded on-demand, not all at once)
- 💡 Tip: check your agent's session cost/usage command to see actual token usage.
- Usage:
- Ask for <skill_name> → load core frameworks
- Ask <skill_name> about <topic> → find and explain a topic
- Ask <skill_name> for ch<N> → dive into a specific chapter
- Discoverable by: <only what is true for the chosen destination — see below>
- Somewhere else? mv ~/.agents/skills/<skill_name> <dest_root>/<skill_name> \
- && ln -sfn <dest_root>/<skill_name> ~/.claude/skills/<skill_name>
- Prompted for permission on every file? That is your host gating writes outside the
- working directory. Say "save it in this project" and re-run to write inside it.
- Reload (if your agent doesn't auto-detect new skills):
- GitHub Copilot CLI: /skills reload
- Claude Code: restart the session
- Amp: restart the session
- Hermes Agent: start a new session
- OpenClaw: openclaw skills list (new session if watcher disabled)
- OpenCode: start a new session
- Share this skill (optional):
- GitHub repo, installable on any host (Step 11): say "publish"
- Copilot ecosystem: gh skill publish $SKILLS_HOME/<skill_name>
- ```
- Fill the "Discoverable by" line from `CLAUDE_STATUS` (the read-back result), never from the fact that `ln` ran — for `~/.agents/skills` installs:
- - `linked` → "Copilot CLI, Amp, Codex (natively); Claude Code via symlink ~/.claude/skills/<skill_name>"
- - `skipped-realdir` → "Copilot CLI, Amp, Codex (natively); **NOT** Claude Code — migrate the real directory at ~/.claude/skills/<skill_name> first"
- - `copy` or `absent` → "Copilot CLI, Amp, Codex (natively); **NOT** Claude Code — the host could not create the symlink (a plain copy drifts on the next Update/Fold-in). Enable Developer Mode / create the link manually, or run the skill from ~/.agents/skills"
- - Hermes Agent personal root → "Hermes Agent (from `$HERMES_HOME/skills/<category>`)"; no symlink claim, and no cross-agent claim, because the other hosts do not scan the Hermes root
- - other host-private or project-local root → name only the host(s) that scan that root; no symlink claim
- The "Somewhere else?" relocation line must be correct for the path actually taken, so it never breaks the symlink the run just created. **`mv` always targets the final skill directory, `<dest_root>/<skill_name>`, never `<dest_root>` itself.** `mv ~/.agents/skills/mybook ~/.copilot/skills && ln -sfn ~/.copilot/skills ~/.claude/skills/mybook` reads as valid and is not: the skill lands at `~/.copilot/skills/mybook` while the link points one level up at the root, so Claude Code resolves to a directory with no `SKILL.md`, which is the exact breakage this line exists to avoid. Substitute the destination the user actually named, so the printed command carries real paths and there is nothing left to interpret:
- - `~/.agents/skills` + symlink → `mv ~/.agents/skills/<skill_name> <dest_root>/<skill_name> && ln -sfn <dest_root>/<skill_name> ~/.claude/skills/<skill_name>`
- - host-private root, Hermes Agent included → `mv <src_root>/<skill_name> <dest_root>/<skill_name>`
- - project-local root → `mv <project_root>/<skill_name> <dest_root>/<skill_name>`
- The "Prompted for permission on every file?" line is the answer to a host that gates writes outside the working directory (any personal-scope root is out-of-cwd): the destination was announced above, and the one-line fix — re-run asking for the project-local root — sits next to it. Keep it only for personal-scope installs; drop it when the user already chose project-local.
- ---
- ## Step 11 — Publish the generated skill to GitHub (optional)
- After the Step 10 report, offer once — and only if the Step 9.5 scan passed:
- > "Want me to publish this skill to GitHub so any Agent Skills host can install it with `npx skills add`? (yes / skip)"
- If the user declines, stop here. Requirements: the `gh` CLI, authenticated (check `gh auth status`). If `gh` is missing or unauthenticated, offer to set it up (`brew install gh` or https://cli.github.com, then `gh auth login`) — or use the no-`gh` path: the user creates an empty repo of the chosen visibility in the GitHub web UI, then you run the `git init`/`add`/`commit` commands below followed by `git remote add origin <repo-url> && git push -u origin main`. The visibility rule below applies to the web-created repo exactly the same.
- **Visibility is a separate closed question — never inferred, never read out of an earlier answer.** Once the user accepts, ask it on its own and require a one-word reply:
- > "Private or public repository? Reply with one word: `private` or `public`."
- **The reply must *be* `public`, not merely contain it — a hard rule, not a suggestion.** Run `gh repo create` with `--private` in every case except one: the answer to the visibility question is the bare word `public`. Substring matching is forbidden, because a sentence about **the source's licence is not a visibility answer** — "it's public domain", "the book is public domain", "it's publicly available" all describe the material, not the repository, and all resolve to `--private`. A paraphrase, a sentence, an ambiguous answer, silence, or your own inference is NOT consent: re-ask once, and if the reply is still not the bare word, use `--private` and say so in the report. A private repo can be flipped public later; a public push of book-derived content cannot be un-published.
- **Copyright gate — always apply before creating the repo:** chapter files are synthesized summaries, not raw text, but they still derive from the source material. Per the README's Copyright & fair use policy, skills generated from **third-party copyrighted books must stay private**; offer public only when the source is the user's own writing, openly licensed content, or material the user explicitly confirms they are authorized to redistribute publicly — and state which case applies. Having access to internal company material is not permission to disclose it: skills from internal docs stay **private** unless the user states they hold publication rights.
- If accepted:
- 1. Add a repo `README.md` inside `$SKILLS_HOME/<skill_name>/` (never overwrite an existing file) — the skill title, a one-paragraph description ("Agent skill generated from *<Title>* by <Author> with [book-to-skill](https://github.com/virgiliojr94/book-to-skill)"), the install command from step 3 below, the file inventory, and a note that the content is synthesized summaries, not the book text.
- 2. Initialize the skill folder as a git repository and create the remote (default repo name `<skill_name>`; let the user override — some prefer a `<skill_name>-skill` suffix). **Nested-repo guard:** first check whether the skill folder already sits inside a git repository (`git -C "$SKILLS_HOME/<skill_name>" rev-parse --show-toplevel` — always the case for project-local roots like `.claude/skills/`). If it does, do NOT `git init` in place: the outer repository would record the folder as an embedded repo (gitlink, mode 160000) without `.gitmodules`, and fresh clones of the outer project would silently omit the skill. Instead, copy the skill folder to a scratch directory, run the commands below from the copy, and tell the user the published repo — not the project-local folder — is the remote's working copy.
- ```bash
- cd "$SKILLS_HOME/<skill_name>"
- git init -b main
- git add -A
- git commit -m "Add <skill_name> skill"
- gh repo create <repo_name> --private --source . --push
- # --private is the default; substitute --public ONLY under the visibility rule above
- # (the visibility answer WAS the bare word "public" AND the copyright gate allows it)
- ```
- 3. Report the repo URL and the cross-host install command:
- ```
- ✅ Published: https://github.com/<owner>/<repo_name> (<private|public>)
- Install on any Agent Skills host:
- npx skills add https://github.com/<owner>/<repo_name> --skill <skill_name>
- ```
- When the nested-repo guard fired and the repo was published from a scratch copy, add one line — that local folder never gains a remote, so the Update/Fold-in push offer will never appear for it:
- ```
- ⚠️ Published from a copy: <skill folder> sits inside another git repository, so it has
- no remote of its own. To publish a later update, re-run Step 11, or clone
- https://github.com/<owner>/<repo_name> and fold new material into the clone.
- ```
- The root-level `SKILL.md` layout is exactly what the `skills` CLI detects, so the repo is installable as-is — no restructuring needed. Outside the nested-repo case the local folder stays the live install for this machine and is the remote's working copy, so later Update/Fold-in runs can commit and push their changes to the same remote.
- ---
- ## Update / Fold-in Workflow
- When performing an Update/Fold-in operation on an existing skill at `$SKILLS_HOME/<skill_name>/`:
- ### 1. Read Existing Skill Structure
- Read and parse the existing skill's files:
- - Read `$SKILLS_HOME/<skill_name>/SKILL.md` to parse the existing **Chapter Index**, **Topic Index**, metadata (author, total chapters), and **Core Frameworks**.
- - List all files in `$SKILLS_HOME/<skill_name>/chapters/` to find the highest chapter number (e.g. `ch12`).
- - Read `$SKILLS_HOME/<skill_name>/glossary.md`, `$SKILLS_HOME/<skill_name>/patterns.md`, and `$SKILLS_HOME/<skill_name>/cheatsheet.md` to see what terms and frameworks are already indexed.
- ### 2. Match Content & Identify Revisions vs. Additions
- Analyze the new extracted text in this run's `full_text.txt` (the `Text ->` path from the extraction output) to identify if the new content represents:
- - **Updates/Revisions to existing chapters**: If a section of the new content directly updates or expands an existing chapter's topic, read the existing chapter file, merge the new details into it, and rewrite the file.
- - **New additions**: If the content introduces new chapters, papers, or separate sections, create **new chapter summary files** under `chapters/`. Start numbering these files after the highest existing chapter number (e.g. if the existing chapters stop at `ch12`, create `ch13-*.md`, `ch14-*.md`, etc.).
- ### 3. Generate or Update Chapter Summary Files
- For each new or revised chapter:
- - Read the corresponding section of the extracted new text.
- - Follow the formatting guidelines in **Step 7** to build the summary.
- - Write/update the file in `$SKILLS_HOME/<skill_name>/chapters/`.
- ### 4. Merge Supporting Files
- - **Merge glossary.md**:
- - Read the existing `$SKILLS_HOME/<skill_name>/glossary.md`.
- - Extract all new terms and definitions from the new content (Step 8 glossary guidelines).
- - Combine and alphabetize the list of existing and new terms.
- - If a term already exists, append the new chapter/source references to it (e.g. `**Term** — definition (Ch 4, Ch 13)`).
- - Rewrite `$SKILLS_HOME/<skill_name>/glossary.md` with the fully merged, alphabetized list.
- - **Merge patterns.md**:
- - Read existing `$SKILLS_HOME/<skill_name>/patterns.md`.
- - Extract any new techniques, algorithms, or patterns from the new content.
- - Append the new patterns, ensuring consistent formatting, and keeping the total length concise (under 2,500 tokens).
- - **Merge cheatsheet.md**:
- - Read existing `$SKILLS_HOME/<skill_name>/cheatsheet.md`.
- - Extract new comparison rules, decision tables, or parameter guides.
- - Integrate them cleanly into the cheatsheet structure.
- ### 5. Re-generate the Master SKILL.md
- Update the master skill file `$SKILLS_HOME/<skill_name>/SKILL.md`:
- - **Metadata**: Increment the chapter count, update the estimated page count, and add the new source names if appropriate. Update the `Generated` date to the current date.
- - **Core Frameworks**: Fold in the most high-impact mental models or principles from the new content (ensuring the overall file remains under 4,000 tokens).
- - **Chapter Index**: Append the new chapters to the index table, linking to the newly created files.
- - **Topic Index**: Merge the new topics alphabetically. If an existing topic is also covered in the new chapters, append the new chapter links to its line (e.g. `- **Topic** → ch05, ch13`).
- ### 6. Scan, Cleanup, and Report
- Once the files are successfully written and merged, run **Step 9.5**, then proceed to **Step 10** to perform cleanup and print a custom update report summarizing the newly added chapters, merged glossary terms, and updated indices. If the skill folder is a git repository with a remote (published via **Step 11**), offer to commit the update and push it.
- ---
- ## Quality Rules
- 1. **Extract structure, not summaries** — capture named frameworks, exact formulations, anti-patterns; not chapter recaps
- 2. **Preserve the author's precision** — "The 5 Whys" ≠ "ask why multiple times"; keep exact naming
- 3. **Density over completeness** — a 1,000-token summary beats a 10,000-token excerpt
- 4. **Practitioner voice** — write "Use X when Y", not "The book explains X"
- 5. **Front-load SKILL.md** — compaction keeps the first 5,000 tokens; most important content comes first
- 6. **Chapter files are on-demand** — they don't count against skill budget until loaded
- 7. **Never copy raw book text** — always synthesize, summarize, extract signal
- 8. **Topic index is critical** — it's how the agent navigates to the right chapter file
README.md · README · 265 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <p align="center">
- <img src="docs/assets/banner.webp" alt="Booklin, the book-to-skill wizard, holding an open book whose pages scatter into sparkles that settle into an ordered grid" width="100%">
- </p>
- <h1 align="center">book-to-skill</h1>
- <p align="center">
- <a href="README.md"><strong>English</strong></a> ·
- <a href="README.ru.md">Русский</a> ·
- <a href="README.zh-CN.md">简体中文</a>
- </p>
- <p align="center">
- <strong>Turn any technical book, document folder, or collection of sources into a unified agent skill — ready to study, reference, and use while you work in GitHub Copilot CLI, Amp, Claude Code, Codex, Hermes Agent, OpenCode, or OpenClaw.</strong></p>
- <p align="center">
- <a href="https://github.com/virgiliojr94/book-to-skill/releases"><img src="https://img.shields.io/github/v/release/virgiliojr94/book-to-skill?style=for-the-badge&color=blueviolet" alt="Latest release"></a>
- <img src="https://img.shields.io/badge/Agent_Skills-Open_Standard-blueviolet?style=for-the-badge" alt="Agent Skills standard">
- <img src="https://img.shields.io/badge/PDF%20%E2%80%A2%20EPUB%20%E2%80%A2%20DOCX%20%E2%80%A2%20MD%20%E2%80%A2%20HTML%20%E2%80%A2%20RTF%20%E2%80%A2%20MOBI-supported-green?style=for-the-badge" alt="Formats supported">
- <img src="https://img.shields.io/badge/License-MIT-blue?style=for-the-badge" alt="MIT License">
- <a href="https://github.com/sponsors/virgiliojr94"><img src="https://img.shields.io/github/sponsors/virgiliojr94?style=for-the-badge&color=ea4aaa&logo=githubsponsors&logoColor=white" alt="Sponsor"></a>
- </p>
- <p align="center">
- <a href="https://trendshift.io/repositories/27038?utm_source=repository-badge&utm_medium=badge&utm_campaign=badge-repository-27038" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/27038" alt="virgiliojr94%2Fbook-to-skill | Trendshift" width="250" height="55"/></a>
- <a href="https://trendshift.io/repositories/27038?utm_source=trendshift-badge&utm_medium=badge&utm_campaign=badge-trendshift-27038" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/27038/daily?language=Python" alt="virgiliojr94%2Fbook-to-skill | Trendshift (daily, Python)" width="250" height="55"/></a>
- <a href="https://www.star-history.com/virgiliojr94/book-to-skill"><img src="https://api.star-history.com/badge?repo=virgiliojr94/book-to-skill&type=rank" alt="Star History Rank" /></a>
- </p>
- <p align="center">
- <a href="#-why">Why</a> ·
- <a href="#-what-it-generates">What it generates</a> ·
- <a href="#-beyond-books">Beyond books</a> ·
- <a href="docs/how-it-works.md">How it works</a> ·
- <a href="docs/usage.md">Usage</a> ·
- <a href="docs/install.md">Install</a> ·
- <a href="docs/faq.md">FAQ</a> ·
- <a href="docs/performance.md">Performance</a> ·
- <a href="docs/architecture.md">Architecture</a> ·
- <a href="CHANGELOG.md">Changelog</a>
- </p>
- <p align="center">
- <strong>24×–51× fewer tokens than dumping the book into context</strong> to answer one question, measured on real books (<a href="docs/performance.md#the-discovery-loop-tax">how it's measured</a>).
- </p>
- **How it works, in 3 steps:**
- 1. **Point** it at a file, folder, or glob — `/book-to-skill ./my-book.pdf`
- 2. **It distills** the book into a skill — frameworks, decision rules, anti-patterns, and per-chapter files. Structure, not a summary.
- 3. **Your agent loads it on demand** — ask `/my-book replication` and it reads the right chapter and answers from the real content, no hallucination.
- ---
- ## 🤔 Why
- <img align="right" width="200" src="docs/assets/booklin.png" alt="Booklin — the book-to-skill mascot, a purple wizard holding a book">
- You buy a great technical book. You read it once. Three months later you can't remember chapter 7 existed.
- The usual workarounds don't help:
- - 📄 "Let me just search the PDF" → you get a list of pages, not answers
- - 🧠 "I'll ask the agent about this book" → it either hallucinates or says it doesn't have the content
- - 📝 "I'll take notes as I read" → you end up with a 200-line doc you never open again
- **book-to-skill solves this by turning the book into a structured skill your agent loads on demand.**
- Once installed, you just type `/your-book-slug replication` and the agent reads the right chapter and answers from the actual content. No hallucination. No digging through PDFs. The book becomes part of your workflow.
- Works with any host that supports the open [Agent Skills](https://github.com/agentskills/agentskills) standard — GitHub Copilot CLI, Amp, Claude Code, Codex, Hermes Agent, OpenCode, and OpenClaw all read the same `SKILL.md` format.
- ---
- ## 📦 What it generates
- Running `/book-to-skill your-book.pdf` (or a folder, glob, or list of files) creates a full skill in the user-level cross-agent skills directory `~/.agents/skills/<slug>/` — one copy that Copilot CLI, Amp, Codex, OpenCode, and OpenClaw discover natively (OpenCode also reads its own managed root `~/.config/opencode/skills/<slug>/`; OpenClaw also discovers it when using its default state directory, with a non-default `OPENCLAW_STATE_DIR` use the active state's `skills/` root instead). When run under Claude Code, the converter also attempts a verified symlink at `~/.claude/skills/<slug>/` — Claude Code sees the skill when the link read-back confirms it; otherwise the run report says so. Hermes Agent partitions its personal skills by category and does not scan the cross-agent root, so a Hermes install lands in `/skills/<category>/<slug>/` instead. OpenClaw-managed, OpenCode-managed, and project-local destinations remain available when explicitly requested:
- | File | Purpose | Size |
- |------|---------|------|
- | `SKILL.md` | Core mental models + chapter index | ~4,000 tokens |
- | `chapters/ch01-*.md` … | One file per chapter, loaded on-demand | ~1,000 tokens each |
- | `glossary.md` | Every key term, alphabetically sorted with chapter refs | ~1,500 tokens |
- | `patterns.md` | All techniques, algorithms, and design patterns | ~2,000 tokens |
- | `cheatsheet.md` | Decision tables and quick-reference rules | ~1,000 tokens |
- **Chapter files are loaded on-demand** — they don't count against the skill budget until you ask about that topic.
- ---
- ## 🏢 Beyond books
- The name says "book", but the input is any structured prose. The same extraction works on knowledge you own and re-read constantly:
- - **Internal documentation** — architecture decision records, runbooks, onboarding guides. Fold a whole `docs/` folder into one skill and ask it while you code.
- - **Brand & design systems** — voice guidelines, tone-of-voice docs, component principles. Turn a brand book into a skill your team queries instead of skimming a 60-page PDF.
- - **Research clusters** — a stack of papers plus your own notes, merged into a single unified skill and updated as new material lands (see [Update / fold-in](#-usage)).
- - **Specs & standards** — RFCs, API contracts, compliance docs you reference but never memorize.
- If you re-open a document often enough to wish you'd memorized it, it's a candidate.
- ---
- ## 🧾 The Discovery Loop Tax
- A PDF-reading agent doesn't just read — it *navigates*: it re-fetches the ToC, backtracks, and re-processes all of it on every turn. book-to-skill pays that structuring cost **once**, at conversion, so queries stay proportional to the answer — **24×–51× fewer tokens** than dumping the book into context, measured on real books.
- 📊 **Full methodology, numbers, and per-book tables → [docs/performance.md](docs/performance.md#the-discovery-loop-tax)**
- ---
- ## ⚙️ How it works
- Two halves: a deterministic Python **extractor** (document → clean text + metadata) and a spec-driven **generator** (your agent follows `SKILL.md` to turn that into a structured skill). On-demand chapter files keep the loaded skill small.
- 🔧 **Full walkthrough (Steps 0–10, extraction modes, token budgets) → [docs/how-it-works.md](docs/how-it-works.md)**
- ---
- ## 🚀 Usage
- `/book-to-skill <path|folder|glob> [skill-name]` — plus analyze-only, generate-from-analysis, and update/fold-in modes. After a conversion, the converter can publish the skill to GitHub (private by default) so any host installs it with `npx skills add`.
- ▶️ **All modes and examples → [docs/usage.md](docs/usage.md)**
- 💬 **In practice → [use cases](https://github.com/virgiliojr94/book-to-skill-use-cases)** — a DevEx book became a survey of 300+ engineers; a scanned PDF that stalled became [#130](https://github.com/virgiliojr94/book-to-skill/pull/130). Add yours: the account lives in your own Gist, the index takes a one-line PR.
- ---
- ## 📥 Install
- ```bash
- # One command, any host — via the cross-agent skills CLI:
- npx skills add virgiliojr94/book-to-skill
- # Or manually — clone into your skills folder (registers /book-to-skill):
- git clone https://github.com/virgiliojr94/book-to-skill.git ~/.claude/skills/book-to-skill
- # (Copilot CLI: ~/.copilot/skills/ · Amp/cross-agent: ~/.agents/skills/)
- # (Hermes Agent: ${HERMES_HOME:-$HOME/.hermes}/skills/<category>/)
- # (OpenClaw: ${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/skills/; ~/.agents/skills/ only with default state)
- # (OpenCode: ~/.agents/skills/ — it also reads ~/.config/opencode/skills/)
- ```
- 📥 **All hosts, optional extractors, and the standalone CLI → [docs/install.md](docs/install.md)**
- ---
- ## ❓ FAQ
- Common questions — "why not just dump the PDF?", cost, privacy, non-book inputs, multi-file books.
- ❓ **Answers → [docs/faq.md](docs/faq.md)**
- ---
- <details>
- <summary>🔧 <strong>Requirements</strong></summary>
- The extractor tries tools in order per format and uses the first available. If nothing is installed, it tells you which command to run. Plain text, Markdown, reStructuredText and AsciiDoc need no extra deps.
- > **Check your setup in one command:** `python3 scripts/extract.py --check` prints which extractors are installed for every format and the exact command to install anything missing — no file needed.
- **PDF — choose by book type:**
- | Book type | Tool | Install | Speed |
- |-----------|------|---------|-------|
- | Text-heavy (prose, few tables) | `pdftotext` (poppler) | `sudo apt install poppler-utils` | ⚡ instant |
- | Text-heavy fallback | `pypdf` | `pip3 install pypdf` | ⚡ instant |
- | Text-heavy fallback | `pdfminer.six` | `pip3 install pdfminer.six` | ⚡ instant |
- | **Technical (code, tables, formulas)** | **`docling`** | `pip3 install docling` | ~1.5s/page |
- > Before extraction begins, the skill asks you whether the book is **technical** or **text-heavy** and picks the right tool automatically. Docling preserves markdown tables and code blocks; pdftotext is faster for prose-only books.
- > **Scanned PDFs need OCR first.** A PDF that is page images with no text layer — a photographed or scanned book — has nothing for these tools to extract. The extractor checks the first pages and stops immediately with an explanation, rather than working through the whole book to produce an empty skill. Run OCR yourself, then convert the result:
- >
- > ```bash
- > ocrmypdf input.pdf output.pdf
- > ```
- **EPUB:**
- | Tool | Install | Quality |
- |------|---------|---------|
- | `ebooklib` + `beautifulsoup4` | `pip3 install ebooklib beautifulsoup4` | ⭐⭐⭐ Best |
- | stdlib `zipfile` | built-in — no install needed | ⭐⭐ Always available |
- **Other formats:**
- | Format | Tool | Install |
- |--------|------|---------|
- | DOCX | `python-docx` (fallback: stdlib ZIP/XML) | `pip3 install python-docx` |
- | HTML | `beautifulsoup4` (fallback: stdlib `html.parser`) | `pip3 install beautifulsoup4` |
- | RTF | `striprtf` (fallback: regex) | `pip3 install striprtf` |
- | MOBI / AZW / AZW3 | Calibre `ebook-convert` (external app, not pip) | https://calibre-ebook.com/download |
- | TXT / Markdown / reStructuredText / AsciiDoc | built-in | — |
- ---
- </details>
- <details>
- <summary>📁 <strong>Repository structure</strong></summary>
- ```
- book-to-skill/
- ├── SKILL.md # Skill definition + step-by-step instructions (the generator spec)
- ├── scripts/
- │ ├── extract.py # Thin entrypoint wrapper
- │ └── extractor/ # Modular extraction package
- │ ├── config.py # Extensions, paths, dependency constants
- │ ├── dependencies.py # optional-dep probing + --check
- │ ├── exceptions.py # ExtractionError (per-source failures, batch-safe)
- │ ├── utils.py # CLI parsing, multi-source resolution, chapter detection, runner
- │ └── parsers/ # Format-specific parsers (pdf, epub, docx, html, rtf, calibre, text)
- ├── tools/
- │ ├── discovery_tax.py # measures token cost vs context-dump / discovery loop
- │ └── validate_skill.py # checks a generated SKILL.md against host rules (--lens claude|copilot|amp|hermes|openclaw|opencode)
- ├── tests/ # pytest suite (extraction, detection, discovery tax)
- ├── docs/
- │ ├── performance.md # measured benchmarks, discovery tax, cost
- │ └── architecture.md # pipeline + component map
- ├── CHANGELOG.md # release history (semver)
- ├── CONTRIBUTING.md # dev setup, PR conventions, release process
- ├── SECURITY.md # vulnerability reporting
- └── README.md # This file
- ```
- ---
- </details>
- ---
- ## ⚖️ Copyright & fair use
- book-to-skill ships **no book content** — not a single page. It's a converter you point at files you already own.
- - **Processing is local.** Extraction and analysis run on your machine. Your files are never uploaded by this tool. (If your agent's model runs in the cloud, the text you feed it follows that provider's normal data terms — same as any prompt.)
- - **You use your own copy.** Bring a book you bought, docs your company owns, or papers you have the right to read.
- - **The output is your notes.** A generated skill is a structured, synthesized derivative — framework names, definitions, takeaways — not a reproduction of the text. The skill explicitly never copies raw passages (see Quality Rule #7). Treat it like handwritten study notes: yours, for personal use.
- - **Don't redistribute.** Publishing or sharing a generated skill of a copyrighted work can infringe the rights holder. Keep skills of third-party books private. Internal docs, your own writing, and openly-licensed material are fine to share within the bounds of their license.
- When in doubt, follow the license or terms of the source document. This project is a tool; how you use it is on you.
- ---
- ## 💖 Sponsors
- <img align="right" width="150" src="docs/assets/booklin-celebrating.png" alt="Booklin celebrating">
- book-to-skill is free and MIT-licensed, maintained on personal time. If it saves you tokens or study hours, consider sponsoring its upkeep: PR reviews, multilingual fixes, releases, and docs.
- **[Become a sponsor → github.com/sponsors/virgiliojr94](https://github.com/sponsors/virgiliojr94)**
- Every sponsor is listed in [BACKERS.md](BACKERS.md). Thank you for keeping open, privacy-first tooling alive. ✨
- ## License
- MIT — applies to the converter (code + skill definition) in this repository, **not** to any book or document you process with it.
book_to_skill/utils.py · Bundled script · 2,041 lines
- from __future__ import annotations
- import glob
- import hashlib
- import json
- import os
- import re
- import statistics
- import sys
- import tempfile
- from collections import Counter
- import shutil
- import zipfile
- from pathlib import Path
- from book_to_skill.exceptions import ExtractionError
- from book_to_skill.config import (
- OUTPUT_DIR,
- OUTPUT_TEXT,
- OUTPUT_META,
- WORDS_PER_TOKEN,
- CJK_CHARS_PER_TOKEN,
- SUPPORTED_EXTENSIONS,
- TEXT_EXTENSIONS,
- HTML_EXTENSIONS,
- CALIBRE_EBOOK_EXTENSIONS,
- supported_formats_message,
- )
- from book_to_skill.dependencies import (
- normalize_install_mode,
- prepare_dependencies,
- run_dependency_check,
- )
- from book_to_skill.parsers.text import read_text_file
- from book_to_skill.parsers.html import extract_html_file
- from book_to_skill.parsers.docx import extract_docx
- from book_to_skill.parsers.rtf import extract_rtf
- from book_to_skill.parsers.calibre import extract_with_ebook_convert
- from book_to_skill.parsers.pdf import (
- _PART_HEADER,
- extract_with_docling,
- extract_with_pdftotext,
- extract_with_pypdf,
- extract_with_pdfminer,
- extract_pypdf_page_records,
- looks_image_only,
- count_pages,
- pdf_outline_titles,
- pdf_outline_all_titles,
- )
- from book_to_skill.parsers.epub import (
- extract_with_ebooklib,
- extract_with_zipfile,
- count_epub_chapters,
- count_epub_images,
- )
- from book_to_skill.sanitize import sanitize_extracted_text
- # Covers and decorative assets are common in prose EPUBs, so only surface the
- # omission when the archive contains more than five images.
- _EPUB_IMAGE_NOTICE_THRESHOLD = 5
- # CJK codepoints: ideographs + extensions, kana, hangul, CJK punctuation, and
- # fullwidth forms. These are not whitespace-delimited, so counting "words" on a
- # Chinese/Japanese book collapses it to a handful of tokens; count them directly.
- #
- # The last range is Planes 2 and 3 (U+20000-U+3FFFF), the ideographic
- # supplementary planes, taken end to end rather than enumerated block by block
- # so a future extension does not silently fall through the way Extension H
- # (U+31350-U+323AF) did. Nothing non-ideographic lives up here: emoji,
- # mathematical alphanumerics and regional indicators are all in Plane 1, which
- # this range does not touch. Classical Chinese, Cantonese, Hong Kong and
- # Taiwan place/personal names, and Japanese 人名用漢字 all draw on it. Without it
- # those characters fell through to the whitespace-word branch, where a
- # space-less run of them counts as a single "word": the same ~1000x undercount
- # #103 fixed for the BMP, one plane up.
- # The Kangxi-radical range (U+2F00-U+2FDF) is included because some Chinese
- # ebooks render ordinary Han characters — 网 as ⽹ (U+2F79), 大 as ⼤
- # (U+2F24), 一 as ⼀ (U+2F00) — as radical forms throughout the whole text;
- # without it such a book still falls through to the whitespace-word branch.
- _CJK_RE = re.compile(
- r"[⼀- -〿-ヿ㐀-䶿一-鿿"
- r"가-힣豈--"
- r"\U00020000-\U0003FFFF]"
- )
- def estimate_tokens(text: str) -> int:
- """Estimate the token count of ``text`` with a deterministic heuristic.
- Latin / whitespace-delimited text is counted by words (``words /
- WORDS_PER_TOKEN`` — the project's long-standing ratio). CJK characters are
- counted directly against ``CJK_CHARS_PER_TOKEN`` because they carry little
- or no whitespace; without this a space-less Chinese/Japanese book estimates
- at a few tokens and the cost pre-flight under-reports by ~1000x. Kept
- dependency-free on purpose so the same book always yields the same number.
- """
- if not text:
- return 0
- cjk = len(_CJK_RE.findall(text))
- if not cjk:
- return int(len(text.split()) / WORDS_PER_TOKEN)
- latin_words = len(_CJK_RE.sub(" ", text).split())
- return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)
- # Explicit chapter heading: "Chapter 5", "Capítulo 5: ...", "Chapter 1. Intro".
- # Also French/German/Italian/Dutch/Vietnamese chapter words (chapitre/kapitel/
- # capitolo/hoofdstuk/chương), matching the ToC languages added alongside. "ch.?"
- # stays last so the longer words match in full. Captures the number (bounded to
- # 1..99 — drops years like "2025.") and whatever follows it on the line, so we
- # can reject prose.
- _EXPLICIT_CHAPTER = re.compile(
- r"^\s*(?:chapter|unit|lesson|module|lecture|part|chapitre|kapitel|cap[ií]tulo|capitolo|hoofdstuk|chương|ch\.?)\s*(?:(\d{1,2})|(?P<roman>[IVXLCDMivxlcdm]{1,7}))\b(?P<rest>.*)$",
- re.IGNORECASE,
- )
- # A heading's number is followed by end-of-line, punctuation (“. : - —“), or a
- # Capitalized title word. A lowercase continuation (“Chapter 6 explores...”,
- # “Chapter 8 are relevant...”) is prose / a cross-reference, not a heading.
- # The uppercase class is À-Þ so titles starting with Ü/Û (common in German, e.g. “Überblick”) are recognized.
- _HEADING_TAIL = re.compile(r"^\s*$|^\s*[.:\-—–]|^\s+(?![a-z])")
- # A tail that opens with a sentence period — "Chapter 4. References are
- # indicated by the `&` symbol…", "Chapter 17.) We have seen…" — is prose wrapped
- # at the column limit, unless it reads like a title. Two shapes, measured on the
- # Rust Book sources (rust-lang/book, MIT OR Apache-2.0, 112 files) where the
- # numeric scan accepted 13 lines and every one of them was prose or code:
- # * a clause of 10-14 words ("Chapter 4. References are indicated by the `&`
- # symbol and borrow the value they"), and
- # * a bare "Chapter 6." ending a sentence that began on the line above.
- # The longest period-tail heading in this repository's fixtures and tests is 5
- # words ("Chapter 1. Introduction to Building AI"). (Issue #237)
- _PERIOD_TAIL = re.compile(r"^\s*\.[)\"'”’\]]*(\s|$)")
- _TAIL_WORD = re.compile(r"[^\s]*[0-9A-Za-z][^\s]*")
- _TITLE_TAIL_MAX_WORDS = 8
- # A line that ends a sentence / paragraph. When the line above does NOT end like
- # this, the next line continues it instead of starting a heading.
- _SENTENCE_END = re.compile(r"[.!?…:;\"'”’)\]]\s*$")
- # Roman-numeral chapter heading: "I: Loomings", "II. The Carpet-Bag".
- # Uppercase alone at line start is safe — no common English word is a valid
- # uppercase Roman numeral. Lowercase ("i: Loomings") is only accepted inside
- # a markdown heading ("## i. introduction") to avoid false positives from
- # words that happen to be valid Roman numerals ("vi: the editor" → 6).
- _ROMAN_HEAD = re.compile(r"^\s*([IVXLCDM]+)\s*[:.]\s+[A-ZÀ-Þ0-9\"“(]")
- _LC_MD_ROMAN = re.compile(r"^\s*#{1,6}\s+([ivxlcdm]+)\s*[:.]\s+[A-Za-zÀ-Þ\"“(]")
- _ROMAN_VALUES = {"I": 1, "V": 5, "X": 10, "L": 50, "C": 100, "D": 500, "M": 1000}
- # Optional Markdown / AsciiDoc heading prefix ("## Chapter 1", "== Section").
- # Stripped in _chapter_number() as a second pass so the CJK/Thai/Korean
- # matchers (which already tolerate the prefix inline) are untouched. (Issue #91)
- _MD_HEADING_PREFIX = re.compile(r"^(#{1,6}|={1,6})\s+")
- # Chinese chapter headings. Two common styles:
- # 1. explicit "第N章" / "第 3 回" / "第十二节" / "第一讲" — 第 + numeral + a
- # chapter classifier (章回卷节篇讲);
- # 2. a Markdown heading led by a CJK ordinal and a separator, e.g.
- # "## 一 · 缘起" or "## 第一讲" — common in CJK ebooks and lecture notes.
- # Scoped to CJK numerals, so Latin/Roman detection above is completely unaffected
- # (e.g. "## 5 Setup" is still not treated as a heading here). detect_structure()
- # dedupes by number, so a "##" heading and a repeated "###" sub-ordinal collapse
- # to a single chapter.
- _CN_NUM_VALUES = {
- "〇": 0, "零": 0, "一": 1, "二": 2, "两": 2, "三": 3, "四": 4, "五": 5,
- "六": 6, "七": 7, "八": 8, "九": 9,
- }
- _CN_NUM_UNITS = {"十": 10, "百": 100, "千": 1000}
- _CN_NUM_CLASS = "〇零一二两三四五六七八九十百千"
- # Kangxi-radical numerals → CJK ideograph numerals. Some Chinese ebooks
- # (e.g. certain e-reader platforms) encode numerals as Kangxi radicals from
- # the U+2F00 block instead of CJK unified ideographs — "第⼀章" with
- # U+2F00 (⼀) rather than U+4E00 (一). NFKC does not map these, so normalize
- # them explicitly before chapter detection. Only numerals that exist as
- # Kangxi radicals are listed (三/四/五/六/七/九 have no radical form).
- _KANGXI_NUMERAL_TRANS = {
- 0x2F00: ord("一"), # ⼀ KANGXI RADICAL ONE
- 0x2F06: ord("二"), # ⼆ KANGXI RADICAL TWO
- 0x2F0B: ord("八"), # ⼋ KANGXI RADICAL EIGHT
- 0x2F17: ord("十"), # ⼗ KANGXI RADICAL TEN
- }
- # Full-width Arabic digits (U+FF10–U+FF19) are common in Japanese typesetting,
- # e.g. "第1章". int() already parses them (str.isdigit() is True), so only the
- # regex character classes need to accept them.
- _FW_DIGITS = "0-9"
- _CN_CHAPTER = re.compile(rf"^\s*第\s*([0-9{_FW_DIGITS}{_CN_NUM_CLASS}]+)\s*[章回卷节篇讲]")
- _MD_CN_HEADING = re.compile(rf"^#{{1,6}}\s+第?\s*([{_FW_DIGITS}{_CN_NUM_CLASS}]+)\s*[·、.::章回卷节篇讲]")
- # Thai chapter headings: "บทที่ 3", "บทที่ ๑๒", "ตอนที่ ๘๗", "ภาคที่ 2".
- # Thai digits (U+0E50-U+0E59) are positional like Arabic — unlike the Chinese
- # numerals above they need no unit composition, only a digit remap. Optional
- # Markdown "#" prefix so "## บทที่ ๑" is recognized in converted ebooks.
- _TH_DIGITS = "๐-๙"
- _TH_DIGIT_MAP = str.maketrans("๐๑๒๓๔๕๖๗๘๙", "0123456789")
- _TH_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?(?:บทที่|ตอนที่|ภาคที่|บท|ตอน|ภาค)\s*([0-9{_TH_DIGITS}]+)\b"
- )
- # Hindi (Devanagari) chapter headings: "अध्याय 1", "अध्याय १", "## अध्याय 2".
- # अध्याय ("chapter") + a number. Devanagari digits (U+0966-U+096F) are positional
- # like Arabic, so — as with Thai — only a digit remap is needed, no composition.
- # Optional Markdown "#" prefix so "## अध्याय १" is recognized in converted ebooks.
- # Scoped to the digit form (not word ordinals like "पहला अध्याय") and requiring a
- # number keeps prose that merely uses the word अध्याय from matching.
- _HI_DIGITS = "०-९"
- _HI_DIGIT_MAP = str.maketrans("०१२३४५६७८९", "0123456789")
- _HI_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?अध्याय\s*([0-9{_HI_DIGITS}]+)\b"
- )
- # Bengali chapter headings: "অধ্যায় 1", "অধ্যায় ১", "## অধ্যায় 2".
- # অধ্যায় ("chapter") + a number. Bengali digits (U+09E6-U+09EF) are positional
- # like the Hindi block above, so only a digit remap is needed. Optional Markdown
- # "#" prefix so "## অধ্যায় ১" is recognized in converted ebooks. Requiring a
- # number keeps prose that merely uses the word অধ্যায় from matching.
- _BN_DIGITS = "০-৯"
- _BN_DIGIT_MAP = str.maketrans("০১২৩৪৫৬৭৮৯", "0123456789")
- _BN_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?অধ্যায়\s*([0-9{_BN_DIGITS}]+)\b"
- )
- # Odia chapter headings: "ଅଧ୍ୟାୟ 1", "ଅଧ୍ୟାୟ ୧", "## ଅଧ୍ୟାୟ 2".
- # ଅଧ୍ୟାୟ ("chapter") + a number. Grouped next to Bengali because both are
- # Eastern Brahmic and the words look alike, but the scripts occupy separate
- # blocks (Odia U+0B00-U+0B7F, Bengali U+0980-U+09FF), so the two patterns
- # cannot see each other's text. Odia digits (U+0B66-U+0B6F) are positional, so
- # only a digit remap is needed. Requiring a number keeps prose that merely uses
- # an inflected form ("ଅଧ୍ୟାୟରେ", a locative suffix) from matching.
- _OR_DIGITS = "୦-୯"
- _OR_DIGIT_MAP = str.maketrans("୦୧୨୩୪୫୬୭୮୯", "0123456789")
- _OR_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?ଅଧ୍ୟାୟ\s*([0-9{_OR_DIGITS}]+)\b"
- )
- # Tamil chapter headings: "அத்தியாயம் 1", "அத்தியாயம் ௧", "## அத்தியாயம் 2".
- # அத்தியாயம் ("chapter") + a number. Tamil digits (U+0BE6-U+0BEF) are positional
- # like the Devanagari/Bengali blocks above, so only a digit remap is needed.
- # Optional Markdown "#" prefix. Requiring a number keeps prose that merely uses
- # the word (or an inflected form like "அத்தியாயத்தில்", which is a different stem)
- # from matching.
- _TA_DIGITS = "௦-௯"
- _TA_DIGIT_MAP = str.maketrans("௦௧௨௩௪௫௬௭௮௯", "0123456789")
- _TA_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?அத்தியாயம்\s*([0-9{_TA_DIGITS}]+)\b"
- )
- # Sinhala chapter headings: "පරිච්ඡේදය 1", "අධ්යාය ෨", "## පරිච්ඡේදය 2".
- # Two words are used: පරිච්ඡේදය and the Sanskrit loan අධ්යාය. Grouped next to
- # Tamil as the other Sri Lankan script. Modern Sinhala prints numbers with ASCII
- # digits; the Lith digits (U+0DE6-U+0DEF) are included for historical texts,
- # remapped like the other Indic blocks.
- #
- # The joiner in අධ්යාය is matched optionally, and that is deliberate: the word
- # carries a ZERO WIDTH JOINER inside its ධ්ය conjunct, but
- # sanitize_extracted_text() strips U+200D before detect_structure() ever runs, so
- # the spelling that reaches this pattern has no joiner even though every real
- # source file has one. "\u200d?" matches both, which keeps the pattern working
- # for callers that reach _chapter_number() without sanitizing first.
- #
- # අධ්යාය shares its transliteration with Devanagari अध्याय, Gujarati અધ્યાય and
- # Gurmukhi ਅਧਿਆਇ, but Sinhala occupies U+0D80-U+0DFF and none of those blocks
- # overlap it, so the matchers cannot see each other's text. Requiring a number
- # keeps prose that merely inflects a word ("පරිච්ඡේදයේ", "අධ්යායේ") from matching.
- _SI_DIGITS = "෦-෯"
- _SI_DIGIT_MAP = str.maketrans("෦෧෨෩෪෫෬෭෮෯", "0123456789")
- _SI_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?(?:පරිච්ඡේදය|අධ්\u200d?යාය)\s*([0-9{_SI_DIGITS}]+)\b"
- )
- # Telugu chapter headings: "అధ్యాయము 1", "అధ్యాయం ౧", "## అధ్యాయం 2".
- # The chapter word has two common spellings — అధ్యాయము (formal) and అధ్యాయం
- # (modern), sharing the stem అధ్యాయ — so both endings are matched. Telugu digits
- # (U+0C66-U+0C6F) are positional like the other Indic blocks above, so only a
- # digit remap is needed. Requiring the full word then a number keeps prose (or an
- # inflected form like "అధ్యాయంలో", a different suffix) from matching.
- _TE_DIGITS = "౦-౯"
- _TE_DIGIT_MAP = str.maketrans("౦౧౨౩౪౫౬౭౮౯", "0123456789")
- _TE_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?అధ్యాయ(?:ము|ం)\s*([0-9{_TE_DIGITS}]+)\b"
- )
- # Kannada chapter headings: "ಅಧ್ಯಾಯ 1", "ಅಧ್ಯಾಯ ೧", "## ಅಧ್ಯಾಯ 2".
- # ಅಧ್ಯಾಯ ("chapter") + a number. Kannada digits (U+0CE6-U+0CEF) are positional
- # like the other Indic blocks above, so only a digit remap is needed. Requiring
- # the word then a number keeps prose that merely uses an inflected form
- # ("ಅಧ್ಯಾಯದಲ್ಲಿ", a case suffix) from matching.
- _KN_DIGITS = "೦-೯"
- _KN_DIGIT_MAP = str.maketrans("೦೧೨೩೪೫೬೭೮೯", "0123456789")
- _KN_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?ಅಧ್ಯಾಯ\s*([0-9{_KN_DIGITS}]+)\b"
- )
- # Malayalam chapter headings: "അധ്യായം 1", "അധ്യായം ൧", "## അദ്ധ്യായം 2".
- # The chapter word has two common spellings — അധ്യായം (modern) and അദ്ധ്യായം
- # (older ദ്ധ orthography), differing only in that conjunct — so both stems are
- # matched. Malayalam digits (U+0D66-U+0D6F) are positional like the other Indic
- # blocks above, so only a digit remap is needed. Requiring the full word
- # (including the final ം) then a number keeps prose that merely uses an
- # inflected form like "അധ്യായത്തിൽ" from matching.
- _ML_DIGITS = "൦-൯"
- _ML_DIGIT_MAP = str.maketrans("൦൧൨൩൪൫൬൭൮൯", "0123456789")
- _ML_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?അ(?:ധ്|ദ്ധ്)യായം\s*([0-9{_ML_DIGITS}]+)\b"
- )
- # Gujarati chapter headings: "પ્રકરણ 1", "અધ્યાય ૧", "## પ્રકરણ 2".
- # Gujarati uses two distinct words for "chapter" — પ્રકરણ (common in modern
- # books) and અધ્યાય (classical/religious) — so both are matched. અધ્યાય shares
- # its transliteration with Devanagari अध्याय (the Hindi block above) but not its
- # codepoints (U+0A85.. vs U+0905..), so the two scripts cannot collide. Gujarati
- # digits (U+0AE6-U+0AEF) are positional like the other Indic blocks above, so
- # only a digit remap is needed. Unlike Malayalam, an inflected form keeps the
- # whole word ("પ્રકરણમાં" is the word + માં), so requiring a number right after
- # it is what keeps prose from matching.
- _GU_DIGITS = "૦-૯"
- _GU_DIGIT_MAP = str.maketrans("૦૧૨૩૪૫૬૭૮૯", "0123456789")
- _GU_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?(?:પ્રકરણ|અધ્યાય)\s*([0-9{_GU_DIGITS}]+)\b"
- )
- # Punjabi (Gurmukhi) chapter headings: "ਕਾਂਡ 1", "ਅਧਿਆਇ ੧", "## ਕਾਂਡ 2".
- # Like Gujarati above, Punjabi uses two distinct words for "chapter" — ਕਾਂਡ
- # (usual in books) and ਅਧਿਆਇ — so both are matched. ਅਧਿਆਇ resembles Gujarati
- # અધ્યાય and Devanagari अध्याय but the scripts are disjoint (Gurmukhi
- # U+0A00-U+0A7F, Gujarati U+0A80-U+0AFF, Devanagari U+0900-U+097F), so none of
- # the three patterns can see another's text. Gurmukhi digits (U+0A66-U+0A6F) are
- # positional, so only a digit remap is needed. Requiring a number right after
- # the word keeps prose from matching, whether it follows the word with a
- # postposition ("ਕਾਂਡ ਵਿੱਚ") or inflects it ("ਕਾਂਡਾਂ").
- _PA_DIGITS = "੦-੯"
- _PA_DIGIT_MAP = str.maketrans("੦੧੨੩੪੫੬੭੮੯", "0123456789")
- _PA_CHAPTER = re.compile(
- rf"^\s*(?:#{{1,6}}\s+)?(?:ਕਾਂਡ|ਅਧਿਆਇ)\s*([0-9{_PA_DIGITS}]+)\b"
- )
- # Russian (Cyrillic) chapter headings: "Глава 1", "ГЛАВА 12", "## Глава 2".
- # "Глава" ("chapter") + a number. Cyrillic uses ordinary Arabic digits, so —
- # unlike the Devanagari/Bengali blocks above — no digit remap is needed. A
- # dedicated matcher (rather than adding the word to _EXPLICIT_CHAPTER) is used
- # because that alternation is Latin-only and its number would still be read
- # there. Requiring whitespace then a number keeps prose that merely uses an
- # inflected form ("В этой главе…", "Главная страница") from matching.
- _RU_CHAPTER = re.compile(r"^\s*(?:#{1,6}\s+)?глава\s+([0-9]+)\b", re.IGNORECASE)
- # Greek chapter headings: "Κεφάλαιο 1", "ΚΕΦΑΛΑΙΟ 12", "## Κεφάλαιο 2".
- # "Κεφάλαιο" ("chapter") + a number. Greek uses ordinary Arabic digits, so — like
- # the Russian block above — no digit remap is needed. The 4th letter is written
- # [άα] because all-caps Greek drops the accent ("ΚΕΦΑΛΑΙΟ" has plain Α, not Ά),
- # and case-folding alone would not bridge Ά↔Α. Requiring whitespace then a number
- # keeps prose that merely uses the word (also "capital", "Το κεφάλαιο αυτό…") from
- # matching.
- _EL_CHAPTER = re.compile(r"^\s*(?:#{1,6}\s+)?κεφ[άα]λαιο\s+([0-9]+)\b", re.IGNORECASE)
- # Hebrew chapter headings: "פרק 1", "פרק 12", "## פרק 2".
- # פרק ("chapter") + a number. Hebrew is unicameral and uses ordinary Arabic
- # digits, so neither case folding nor a digit remap is needed. Being right-to-left
- # does not affect the pattern: the regex reads the logical order stored in the
- # file, which is פ-ר-ק then the number, and sanitize_extracted_text() removes any
- # bidi wrapper (RLM/RLE/PDF) before detect_structure() sees the line. Requiring
- # whitespace then a number keeps prose from matching — the word also means
- # "joint", and the prefixed and plural forms ("בפרק", "פרקים") are not followed by
- # a number. Gematria numbering ("פרק א"), common in religious texts, is a separate
- # letter-to-number mapping and is deliberately not handled here.
- _HE_CHAPTER = re.compile(r"^\s*(?:#{1,6}\s+)?פרק\s+([0-9]+)\b")
- # Armenian chapter headings: "Գլուխ 1", "ԳԼՈՒԽ 12", "## Գլուխ 2".
- # "Գլուխ" ("chapter") + a number, with ordinary Arabic digits so no remap is
- # needed. Armenian is bicameral and — unlike Greek above — its case mapping
- # round-trips exactly ("Գլուխ".upper() == "ԳԼՈՒԽ" letter for letter), so plain
- # re.IGNORECASE is enough and no letter class is required. The word also means
- # "head", so requiring *whitespace* then a number does the guarding: an
- # inflected form elides the ու stem vowel ("գլխում") and so shares no prefix,
- # and the plural keeps the word intact, which is why \s+ matters — it stops
- # "Գլուխներ 5" ("chapters 5") from reading as chapter 5.
- _HY_CHAPTER = re.compile(r"^\s*(?:#{1,6}\s+)?գլուխ\s+([0-9]+)\b", re.IGNORECASE)
- # Korean chapter headings: "제1장 총칙", "## 제4장 근로시간과 휴식", "제6장의2 …".
- # 제 + Arabic numeral + a classifier (장 chapter / 편 part / 절 section / 관
- # subsection), with an optional "의N" branch suffix that Korean statutes use for
- # inserted chapters (제6장의2). Modern Korean numbers chapters with Arabic digits,
- # so unlike the Chinese branch no numeral composition is needed. Optional Markdown
- # "#" prefix so "## 제1장" is recognized in converted ebooks.
- #
- # The trailing group is the Korean analogue of _HEADING_TAIL: Korean has no letter
- # case, so the existing "capitalized title word" test does not transfer.
- # Requiring end-of-line, punctuation, or whitespace-then-content is what separates
- # a heading from a prose cross-reference, because Korean particles attach directly
- # to the noun ("제5장에서", "제2장의") with no intervening space.
- _KO_CHAPTER = re.compile(
- r"^\s*(?:#{1,6}\s+)?제\s*([0-9]+)\s*[장편절관](?:\s*의\s*[0-9]+)?(?:\s*$|[.:\-]|\s+\S)"
- )
- # Persian chapter headings: "فصل ۱", "فصل اول", "بخش ۲: مفاهیم",
- # "فصل بیست و یکم", "فصل سی و چهارمخداحافظ…" (PDF glue on long forms).
- # Labels are فصل / بخش. Digits may be ASCII, Persian (U+06F0–U+06F9), or
- # Arabic-Indic (U+0660–U+0669); int() parses all three. Word numerals use a
- # small ordinal map (1–34) with longest-prefix matching so compounds
- # ("بیست و یکم") and teens ("یازدهم") stay maintainable. Markdown "#" prefixes
- # are handled by `_chapter_number`'s second pass (Issue #91).
- #
- # Trailing rules (Persian has no letter case for a Latin-style `_HEADING_TAIL`):
- # - digits: EOL / punctuation / spaced title (Korean-style);
- # - short word ordinals 1–10: require a separator (space, punct, ZWNJ, or EOL)
- # so "فصل اولویتها" / "فصل اولیه" are not read as chapter 1;
- # - teens and compounds: also allow a glued title letter — PDF extractors
- # often drop the space, and a long ordinal is not a plausible word prefix.
- _FA_DIGITS = "۰-۹٠-٩" # Persian then Arabic-Indic
- _FA_ONES = (
- "اول", "دوم", "سوم", "چهارم", "پنجم", "ششم", "هفتم", "هشتم", "نهم", "دهم",
- )
- # Ones used after "بیست و" / "سی و" (یکم, not اول).
- _FA_COMPOUND_ONES = (
- "یکم", "دوم", "سوم", "چهارم", "پنجم", "ششم", "هفتم", "هشتم", "نهم",
- )
- _FA_TEENS = (
- "یازدهم", "دوازدهم", "سیزدهم", "چهاردهم", "پانزدهم",
- "شانزدهم", "هفدهم", "هجدهم", "نوزدهم",
- )
- _FA_ONES_SET = frozenset(_FA_ONES)
- # After a short (1–10) word ordinal: end, whitespace, punctuation, or ZWNJ.
- _FA_SHORT_ORDINAL_TAIL = re.compile(r"^(?:$|\s|[.:\-—–:]|\u200c)")
- def _fa_ordinal_map() -> dict[str, int]:
- """Persian chapter ordinals 1–34, including common spelling variants."""
- m: dict[str, int] = {}
- for i, w in enumerate(_FA_ONES, 1):
- m[w] = i
- for i, w in enumerate(_FA_TEENS, 11):
- m[w] = i
- m["هیجدهم"] = 18 # common alternate spelling of هجدهم
- # Fused "بیستم" only — "بیست ام" / "بیستام" are not common spellings
- # (unlike "سی ام" / "سیام" for 30), so they stay unmapped on purpose.
- m["بیستم"] = 20
- m["سی ام"] = 30
- m["سیام"] = 30 # ZWNJ spelling common in Persian typography
- for i, w in enumerate(_FA_COMPOUND_ONES, 1):
- m[f"بیست و {w}"] = 20 + i
- m[f"سی و {w}"] = 30 + i
- # sanitize_extracted_text() strips U+200C (it is a prompt-injection spacer)
- # before detect_structure() runs, so a key spelled with a ZWNJ can never be
- # reached through the pipeline. Register the stripped spelling of every such
- # key as well, so both the typographic form and what detection actually
- # receives resolve to the same chapter number.
- for key, value in list(m.items()):
- m.setdefault(key.replace("\u200c", ""), value)
- return m
- _FA_ORDINALS = _fa_ordinal_map()
- # Longest first so "چهاردهم" wins over "چهارم", "بیست و یکم" over nothing shorter.
- _FA_ORDINAL_KEYS = sorted(_FA_ORDINALS, key=len, reverse=True)
- _FA_LABEL_REST = re.compile(r"^\s*(?:فصل|بخش)\s+(.*)$")
- _FA_DIGIT_HEAD = re.compile(rf"^([0-9{_FA_DIGITS}]+)(.*)$")
- # Digit form: same idea as the Korean trailing guard (no Latin case to lean on).
- _FA_DIGIT_TAIL = re.compile(r"^(?:\s*$|[.:\-—–:]|\s+\S)")
- def _fa_chapter_number(s: str) -> int | None:
- """Return a Persian chapter number (1–99 digits / 1–34 words) or None."""
- m = _FA_LABEL_REST.match(s)
- if not m:
- return None
- rest = m.group(1)
- dm = _FA_DIGIT_HEAD.match(rest)
- if dm:
- n = int(dm.group(1))
- if 1 <= n <= 99 and _FA_DIGIT_TAIL.match(dm.group(2)) is not None:
- return n
- return None
- for key in _FA_ORDINAL_KEYS:
- if not rest.startswith(key):
- continue
- tail = rest[len(key):]
- # Short 1–10 ordinals need a separator; teens/compounds may be PDF-glued.
- if key in _FA_ONES_SET and _FA_SHORT_ORDINAL_TAIL.match(tail) is None:
- return None
- return _FA_ORDINALS[key]
- return None
- # Table-of-contents header lines across common languages. Anchored to a whole
- # line (^\s*X\s*$) so an inline "the contents of this chapter" never matches.
- _TOC_HEADERS = (
- "table of contents", "contents", "índice", "sumário", # EN / ES / PT
- "sumario", # PT (no accent — OCR / accent-stripped, like indice below)
- "table des matières", # French
- "inhaltsverzeichnis", # German
- "indice", "sommario", # Italian (no accent — distinct from índice above)
- "inhoudsopgave", # Dutch
- )
- _TOC_CJK_PATTERN = r"目[ \t\u3000]*(?:录|錄|次)"
- _TOC_PATTERN = re.compile(
- r"^\s*(?:#{1,6}\s*)?(?:"
- + "|".join([*(re.escape(h) for h in _TOC_HEADERS), _TOC_CJK_PATTERN])
- + r")\s*$",
- re.IGNORECASE | re.MULTILINE,
- )
- # Structural extraction sees every heading at the selected depth, including
- # the book's framing sections. These whole-title guards remove only conventional
- # front/back matter; instructional titles such as "Indexing Strategies" and
- # "Glossary of Terms" must remain chapters.
- _FRONT_MATTER_HEADERS = (
- "cover",
- "preface",
- "foreword",
- "acknowledgements",
- "acknowledgments",
- "dedication",
- "copyright",
- "epigraph",
- "prologue",
- )
- _BACK_MATTER_HEADERS = (
- "bibliography",
- "references",
- "works cited",
- "further reading",
- "index",
- "glossary",
- "notes",
- "endnotes",
- "footnotes",
- "colophon",
- "about the author",
- "about the authors",
- )
- _NON_CHAPTER_HEADERS = frozenset((*_FRONT_MATTER_HEADERS, *_BACK_MATTER_HEADERS))
- _NON_CHAPTER_PREFIX = re.compile(r"^(?:appendix|appendices)\b", re.IGNORECASE)
- def _is_non_chapter_section(title: str) -> bool:
- """Return whether *title* is conventional book framing, not a chapter."""
- normalized = title.strip()
- return (
- normalized.casefold() in _NON_CHAPTER_HEADERS
- or bool(_TOC_PATTERN.fullmatch(normalized))
- or bool(_NON_CHAPTER_PREFIX.match(normalized))
- or bool(_PART_HEADER.match(normalized))
- )
- # ATX-style heading: "# Title", "## Section", AsciiDoc "= Title", "== Section".
- # The required space after the marker distinguishes an AsciiDoc "== X" from a
- # reStructuredText underline "=====" (no space) — the latter is intentionally
- # ignored (RST underline headings are out of scope).
- _ATX_HEADING = re.compile(r"^(#{1,6}|={1,6})\s+(.+?)\s*#*$")
- # Setext/RST underline: a full line of "=" (level 1) or "-" (level 2), length
- # >= 2. Marks the line directly above it as a heading title.
- _SETEXT_UNDERLINE = re.compile(r"^(={2,}|-{2,})$")
- # Opening or closing line of a fenced code block: three or more backticks or
- # tildes. The captured marker lets the closer be matched to its opener.
- _CODE_FENCE = re.compile(r"^(`{3,}|~{3,})")
- def _closed_fence_line_numbers(lines: list[str]) -> set[int]:
- """Line indices inside a fenced code block that is actually CLOSED.
- A fence that never closes is treated as ordinary text rather than swallowing
- everything after it. Extraction routinely loses a closing fence, and a book
- about Markdown can simply contain a stray one — and the old live-toggling
- scan then dropped every heading from that point to the end of the document.
- Counting a handful of code lines as prose is a far cheaper mistake than
- losing most of a book's structure.
- Per CommonMark, a closer must use the same character, be at least as long
- as its opener, and have no trailing info string. A longer fence can then
- safely contain examples of shorter fences without exposing sample headings.
- """
- inside: set[int] = set()
- opener: tuple[str, int] | None = None
- for index, line in enumerate(lines):
- stripped = line.strip()
- match = _CODE_FENCE.match(stripped)
- if not match:
- continue
- marker = match.group(1)
- if opener is None:
- opener = (marker, index)
- elif (
- marker[0] == opener[0][0]
- and len(marker) >= len(opener[0])
- and not stripped[match.end():]
- ):
- # Include both fence marker lines themselves.
- inside.update(range(opener[1], index + 1))
- opener = None
- return inside
- # A numbered heading is a chapter when the numbering is systematic AND the
- # sections carry a chapter's worth of text. Both are required, because neither
- # separates the two shapes alone: a three-step tutorial is also systematic and
- # also ascends from 1, while a single long section is not a numbering scheme.
- # Measured medians of body text per section: tutorial steps ~20 chars, doc
- # sections ~500, paper sections ~2,000, real book chapters ~5,000. The floor
- # sits an order of magnitude below the smallest real chapter seen and an order
- # above the largest tutorial step.
- _MIN_NUMBERED_TITLES = 3
- _MIN_NUMBERED_BODY_CHARS = 200
- def _numbered_titles_are_structural(
- entries: list[tuple[str, int, str]], heading_lines: list[int], lines: list[str]
- ) -> bool:
- """Decide whether digit-led titles at one depth are chapters or list items.
- Deliberately not based on the numbers themselves. An ascending run starting
- at 1 describes "Step 1 / Step 2 / Step 3" as accurately as it describes a
- paper's sections, and requiring the run to be unbroken would throw away a
- whole book when extraction drops one heading, a chapter list that starts at
- 0, or a multi-source corpus where the numbering restarts.
- """
- if len(entries) < _MIN_NUMBERED_TITLES:
- return False
- ordered = sorted(heading_lines)
- bodies = []
- for _, index, _ in entries:
- after = [ln for ln in ordered if ln > index]
- end = after[0] if after else len(lines)
- bodies.append(sum(len(ln) for ln in lines[index + 1:end]))
- return statistics.median(bodies) >= _MIN_NUMBERED_BODY_CHARS
- def _structural_chapter_headings(text: str) -> list[str]:
- """Return chapter-like structural headings in Markdown/AsciiDoc/RST sources.
- Recognizes ATX headings ("# Title", "== Section") and setext/RST underline
- headings (a title line directly above a row of "=" or "-"). Groups distinct
- (case-normalized) titles by depth and returns the count at the shallowest
- depth with >= 2 distinct titles — this selects the real chapter level in the
- common "# Book Title / ## Chapter" layout where the top level appears once.
- Guards against false positives: headings inside fenced code blocks are
- skipped; an ATX title starting with a bare digit ("## 5 Setup") or made only
- of punctuation ("=====" table borders) is rejected; a setext underline counts
- only when it sits directly under a non-blank title line at least as long as
- the underline (so thematic breaks, table borders, and front-matter "---" do
- not match).
- """
- lines = text.splitlines()
- # Map normalized titles to their original representation. Dict insertion
- # order makes the returned sample match the source order while preserving
- # the previous case-insensitive de-duplication behavior.
- levels: dict[int, dict[str, str]] = {}
- # Digit-led titles are held back and judged per depth at the end (see
- # _numbered_titles_are_structural): "## 1. Introduction" and "## 5 Setup"
- # are the same string shape, so the line alone cannot decide.
- numbered: dict[int, list[tuple[str, int, str]]] = {}
- heading_lines: list[int] = []
- fenced = _closed_fence_line_numbers(lines)
- prev = "" # previous non-fence line (stripped); a setext title candidate
- for index, line in enumerate(lines):
- if index in fenced:
- prev = ""
- continue
- s = line.strip()
- # Setext/RST underline: "=" (level 1) or "-" (level 2) directly under a
- # title line at least as long as the underline.
- if (
- _SETEXT_UNDERLINE.match(s)
- and prev
- and not _SETEXT_UNDERLINE.match(prev)
- and len(s) >= len(prev)
- # A title made only of punctuation is never a chapter. Two thematic
- # breaks in a row ("***" over "---"), an ASCII box rule, a row of
- # dots, or a table border sitting above an underline all reach this
- # point. The ATX branch below already rejects them with the same
- # test; the setext branch had no equivalent, so the identical string
- # counted as a heading here and not there.
- and re.search(r"\w", prev)
- ):
- depth = 1 if s[0] == "=" else 2
- heading_lines.append(index)
- if not _is_non_chapter_section(prev):
- levels.setdefault(depth, {}).setdefault(prev.casefold(), prev)
- prev = ""
- continue
- # ATX heading ("# Title", "== Section").
- m = _ATX_HEADING.match(s)
- if m:
- title = m.group(2).strip().lower()
- depth = len(m.group(1))
- # Reject empty and all-punctuation ("=====" table-border) titles.
- if title and re.search(r"\w", title):
- heading_lines.append(index)
- if not _is_non_chapter_section(title):
- if title[0].isdigit():
- numbered.setdefault(depth, []).append((title, index, s))
- else:
- levels.setdefault(depth, {}).setdefault(title, s)
- # An ATX heading line is not a setext title for the next line.
- prev = ""
- continue
- prev = s
- for depth, entries in numbered.items():
- if _numbered_titles_are_structural(entries, heading_lines, lines):
- level = levels.setdefault(depth, {})
- for title, _, sample in entries:
- level.setdefault(title, sample)
- if not levels:
- return []
- for depth in sorted(levels):
- if len(levels[depth]) >= 2:
- return list(levels[depth].values())
- # No level has >= 2 distinct headings: a thin doc (e.g. one heading per
- # level). Count them all — this path runs only as a fallback when numeric
- # chapter detection already found zero, so it cannot inflate real books.
- return [
- title
- for depth in sorted(levels)
- for title in levels[depth].values()
- ]
- def _structural_chapter_count(text: str) -> int:
- """Count chapter-like structural headings in Markdown/AsciiDoc/RST sources."""
- return len(_structural_chapter_headings(text))
- def _cn_numeral_to_int(s: str) -> int | None:
- """Parse a Chinese (or ASCII-digit) chapter numeral into an int (1..999)."""
- if s.isdigit():
- n = int(s)
- return n if 1 <= n <= 999 else None
- section = current = 0
- for ch in s:
- if ch in _CN_NUM_VALUES:
- current = _CN_NUM_VALUES[ch]
- elif ch in _CN_NUM_UNITS:
- section += (current or 1) * _CN_NUM_UNITS[ch]
- current = 0
- else:
- return None
- total = section + current
- return total if 1 <= total <= 999 else None
- def _int_to_roman(n: int) -> str:
- table = [(1000, "M"), (900, "CM"), (500, "D"), (400, "CD"), (100, "C"),
- (90, "XC"), (50, "L"), (40, "XL"), (10, "X"), (9, "IX"),
- (5, "V"), (4, "IV"), (1, "I")]
- out = []
- for val, sym in table:
- while n >= val:
- out.append(sym)
- n -= val
- return "".join(out)
- def _roman_to_int(s: str) -> int | None:
- """Convert a Roman numeral to int, returning None if it isn't canonical."""
- s = s.upper()
- total = prev = 0
- for ch in reversed(s):
- v = _ROMAN_VALUES.get(ch)
- if v is None:
- return None
- total += -v if v < prev else v
- prev = max(prev, v)
- if total == 0 or total > 200:
- return None
- # Reject non-canonical forms ("IIII", "VV") by round-tripping.
- return total if _int_to_roman(total) == s else None
- def _is_prose_period_tail(tail: str, prev_line: str | None) -> bool:
- """True when a period tail is a wrapped sentence, not a title.
- The context decides first: a line that stands on its own (nothing above it,
- or the line above ends a sentence) keeps its heading. When the paragraph
- continues into the line, the tail must read like a title — a bare
- "Chapter 6." and a clause of more than `_TITLE_TAIL_MAX_WORDS` words are the
- sentence that wrapped. (Issue #237)
- """
- if not _PERIOD_TAIL.match(tail):
- return False
- if not prev_line or _SENTENCE_END.search(prev_line):
- return False
- words = _TAIL_WORD.findall(tail)
- return not words or len(words) > _TITLE_TAIL_MAX_WORDS
- def _match_chapter_number(
- line: str, prev_line: str | None = None, heading_marked: bool = False
- ) -> int | None:
- """Return the chapter number if the line is a genuine chapter heading,
- with no Markdown/AsciiDoc heading prefix (the caller strips it first).
- `prev_line` is the line directly above (None when it is blank or inside a
- fenced block); it is consulted only for a bare period tail. `heading_marked`
- says the line carried an explicit "#"/"==" prefix before it was stripped —
- an explicit heading is trusted as it stands, so "## Chapter 4. References
- are indicated by the `&` symbol…" keeps counting. (Issue #237)
- """
- # Normalize Kangxi-radical numerals (⼀⼆⼋⼗) to ideographs so Chinese
- # ebooks that encode chapter numbers in the U+2F00 block are detected.
- s = line.strip().translate(_KANGXI_NUMERAL_TRANS)
- if len(s) > 80:
- return None
- # Plain numbered chapter headings used by many technical books,
- # e.g. "1 Introduction" or "12 Advanced Topics".
- #
- # Require at least two spaces after the chapter number. This avoids
- # treating ordinary numbered list items such as "1. Item" as chapters.
- #
- # A unified-diff hunk line starts with exactly that shape
- # ("1 + use crate::trpl::StreamExt;", "12 - use std::io::prelude::*;").
- # Its first non-space character is the diff marker, which no chapter title
- # starts with, so the marker decides. Fenced blocks are skipped by the
- # caller; this covers code that arrives without fences. (Issue #237)
- plain = re.match(r"^([1-9]\d{0,2})\s{2,}(\S)", s)
- if plain and plain.group(2) not in "+-":
- return int(plain.group(1))
- m = _EXPLICIT_CHAPTER.match(s)
- if m and _HEADING_TAIL.match(m.group("rest")):
- if not heading_marked and _is_prose_period_tail(m.group("rest"), prev_line):
- return None
- if m.group(1):
- return int(m.group(1))
- return _roman_to_int(m.group("roman").upper())
- rm = _ROMAN_HEAD.match(s) or _LC_MD_ROMAN.match(s)
- if rm:
- return _roman_to_int(rm.group(1))
- cm = _CN_CHAPTER.match(s) or _MD_CN_HEADING.match(s)
- if cm:
- return _cn_numeral_to_int(cm.group(1))
- tm = _TH_CHAPTER.match(s)
- if tm:
- return int(tm.group(1).translate(_TH_DIGIT_MAP))
- hm = _HI_CHAPTER.match(s)
- if hm:
- return int(hm.group(1).translate(_HI_DIGIT_MAP))
- bm = _BN_CHAPTER.match(s)
- if bm:
- return int(bm.group(1).translate(_BN_DIGIT_MAP))
- orm = _OR_CHAPTER.match(s)
- if orm:
- return int(orm.group(1).translate(_OR_DIGIT_MAP))
- tam = _TA_CHAPTER.match(s)
- if tam:
- return int(tam.group(1).translate(_TA_DIGIT_MAP))
- sim = _SI_CHAPTER.match(s)
- if sim:
- return int(sim.group(1).translate(_SI_DIGIT_MAP))
- tem = _TE_CHAPTER.match(s)
- if tem:
- return int(tem.group(1).translate(_TE_DIGIT_MAP))
- knm = _KN_CHAPTER.match(s)
- if knm:
- return int(knm.group(1).translate(_KN_DIGIT_MAP))
- mlm = _ML_CHAPTER.match(s)
- if mlm:
- return int(mlm.group(1).translate(_ML_DIGIT_MAP))
- gum = _GU_CHAPTER.match(s)
- if gum:
- return int(gum.group(1).translate(_GU_DIGIT_MAP))
- pam = _PA_CHAPTER.match(s)
- if pam:
- return int(pam.group(1).translate(_PA_DIGIT_MAP))
- rum = _RU_CHAPTER.match(s)
- if rum:
- return int(rum.group(1))
- elm = _EL_CHAPTER.match(s)
- if elm:
- return int(elm.group(1))
- hem = _HE_CHAPTER.match(s)
- if hem:
- return int(hem.group(1))
- hym = _HY_CHAPTER.match(s)
- if hym:
- return int(hym.group(1))
- km = _KO_CHAPTER.match(s)
- if km:
- return int(km.group(1))
- fa = _fa_chapter_number(s)
- if fa is not None:
- return fa
- return None
- def _chapter_number(line: str, prev_line: str | None = None) -> int | None:
- """Return the chapter number if the line is a genuine chapter heading.
- `prev_line` is the line directly above (None when it is blank or inside a
- fenced block); see `_match_chapter_number` for how it is used.
- Handles Arabic ("Chapter 5", "Capítulo 5: ..."), Roman-numeral
- ("I: Loomings", "## i. introduction", "II. The Carpet-Bag"),
- Chinese ("第三章 …", "## 一 · …", "## 第一讲"), Thai ("บทที่ 3",
- "## บทที่ ๑"), Hindi ("अध्याय 1", "अध्याय १", "## अध्याय 2"),
- Bengali ("অধ্যায় 1", "অধ্যায় ১", "## অধ্যায় 2"),
- Odia ("ଅଧ୍ୟାୟ 1", "ଅଧ୍ୟାୟ ୧", "## ଅଧ୍ୟାୟ 2"),
- Tamil ("அத்தியாயம் 1", "அத்தியாயம் ௧", "## அத்தியாயம் 2"),
- Sinhala ("පරිච්ඡේදය 1", "අධ්යාය ෨", "## පරිච්ඡේදය 2"),
- Telugu ("అధ్యాయము 1", "అధ్యాయం ౧", "## అధ్యాయం 2"),
- Kannada ("ಅಧ್ಯಾಯ 1", "ಅಧ್ಯಾಯ ೧", "## ಅಧ್ಯಾಯ 2"),
- Malayalam ("അധ്യായം 1", "അധ്യായം ൧", "## അദ്ധ്യായം 2"),
- Gujarati ("પ્રકરણ 1", "અધ્યાય ૧", "## પ્રકરણ 2"),
- Punjabi ("ਕਾਂਡ 1", "ਅਧਿਆਇ ੧", "## ਕਾਂਡ 2"),
- Russian ("Глава 1", "ГЛАВА 12", "## Глава 2"),
- Greek ("Κεφάλαιο 1", "ΚΕΦΑΛΑΙΟ 12", "## Κεφάλαιο 2"),
- Hebrew ("פרק 1", "פרק 12", "## פרק 2"),
- Armenian ("Գլուխ 1", "ԳԼՈՒԽ 12", "## Գլուխ 2"),
- Korean ("제1장 총칙", "## 제4장 근로시간과 휴식"), and
- Persian ("فصل ۱", "فصل اول", "فصل بیست و یکم", "بخش ۲: مفاهیم",
- "## فصل ۱: مقدمه", PDF-glued "فصل سی و چهارمخداحافظ…") heading styles — each
- optionally preceded by a Markdown/AsciiDoc heading marker
- ("## Chapter 1" is a chapter heading just like "Chapter 1").
- """
- match = _match_chapter_number(line, prev_line)
- if match is not None:
- return match
- # Second pass: a Markdown/AsciiDoc heading prefix ("## Chapter 1",
- # "== Section") hides the heading from the matchers above — the CJK
- # matchers tolerate the prefix inline but the Latin/Thai/Korean ones anchor
- # on the line start. Strip the prefix and retry so --mode technical
- # (Docling emits headings as Markdown) detects the same chapters as
- # plain-text extraction. (Issue #91)
- #
- # No previous line is passed and the line is marked as heading-marked: an
- # explicit "#"/"==" prefix is a structural claim, so a period tail there
- # ("## Chapter 6. Enums", "## Chapter 4. References are indicated by…") is
- # trusted as written.
- s = line.strip()
- md = _MD_HEADING_PREFIX.match(s)
- if md:
- return _match_chapter_number(s[md.end():], heading_marked=True)
- return None
- def _line_title_text(line: str) -> str:
- """Normalize a source line to its heading title, if any."""
- stripped = line.strip()
- match = _ATX_HEADING.match(stripped)
- return match.group(2).strip() if match else stripped
- _TOC_LEADER = re.compile(r"\.{3,}")
- _PAGE_CHAPTER_HEADER = re.compile(r"^\d+\s*\|\s*Chapter\s+\d", re.IGNORECASE)
- _CHAPTER_MARKER = re.compile(r"^\s*CHAPTER\s+\d+\s*$", re.IGNORECASE)
- _OUTLINE_CHAPTER_PREFIX = re.compile(r"^chapter\s+\d+[.:]?\s*", re.IGNORECASE)
- def _outline_bare_title(outline_title: str) -> str:
- return _OUTLINE_CHAPTER_PREFIX.sub("", outline_title.strip(), count=1).strip()
- def _normalized_heading(line: str) -> str:
- return _line_title_text(line).strip().rstrip(".").casefold()
- def _is_disqualified_chapter_line(line: str) -> bool:
- stripped = line.strip()
- if not stripped:
- return True
- if _PAGE_CHAPTER_HEADER.match(stripped):
- return True
- if _TOC_LEADER.search(stripped):
- return True
- return False
- def _chapter_start_at(lines: list[str], title_line: int) -> int:
- if title_line > 0 and _CHAPTER_MARKER.match(lines[title_line - 1].strip()):
- return title_line - 1
- return title_line
- def _line_matches_outline_title(lines: list[str], index: int, outline_title: str) -> int | None:
- """Return the 0-based start line if *outline_title* begins at *index*, else None."""
- if _is_disqualified_chapter_line(lines[index]):
- return None
- bare = _outline_bare_title(outline_title)
- full_cf = outline_title.strip().casefold()
- bare_cf = bare.casefold()
- line = lines[index].strip()
- heading_cf = _normalized_heading(lines[index])
- if heading_cf == bare_cf or heading_cf == full_cf or line.casefold() == bare_cf:
- return _chapter_start_at(lines, index)
- if index + 1 < len(lines) and not _is_disqualified_chapter_line(lines[index + 1]):
- combined = f"{line} {lines[index + 1].strip()}".casefold()
- if combined == bare_cf or combined == full_cf:
- return _chapter_start_at(lines, index)
- if _CHAPTER_MARKER.match(line):
- for look in range(index + 1, min(index + 4, len(lines))):
- if _is_disqualified_chapter_line(lines[look]):
- continue
- if _normalized_heading(lines[look]) == bare_cf:
- return index
- if look + 1 < len(lines) and not _is_disqualified_chapter_line(lines[look + 1]):
- pair = f"{lines[look].strip()} {lines[look + 1].strip()}".casefold()
- if pair == bare_cf:
- return index
- return None
- def _find_outline_chapter_start(lines: list[str], title: str, start: int) -> int | None:
- """0-based line index where the outline chapter begins in extracted text."""
- for index in range(start, len(lines)):
- found = _line_matches_outline_title(lines, index, title)
- if found is not None:
- return found
- return None
- def _is_isolated_heading_line(lines: list[str], index: int) -> bool:
- if not lines[index].strip():
- return False
- prev_blank = index == 0 or not lines[index - 1].strip()
- next_blank = index + 1 >= len(lines) or not lines[index + 1].strip()
- return prev_blank and next_blank
- def _is_chapter_content_boundary(
- lines: list[str], index: int, range_end_titles: frozenset[str]
- ) -> bool:
- """True when a book- or part-level framing heading ends the chapter body."""
- stripped = lines[index].strip()
- if not stripped or _PAGE_CHAPTER_HEADER.match(stripped):
- return False
- heading_cf = _normalized_heading(lines[index])
- if heading_cf in range_end_titles:
- return True
- title = _line_title_text(lines[index])
- if _PART_HEADER.match(title) or _NON_CHAPTER_PREFIX.match(title):
- return _is_isolated_heading_line(lines, index)
- return False
- def _first_chapter_boundary(
- lines: list[str], start: int, limit: int, range_end_titles: frozenset[str]
- ) -> int:
- """Earliest line in [start, limit) that ends the preceding chapter body."""
- for index in range(start, limit):
- if _is_chapter_content_boundary(lines, index, range_end_titles):
- return index
- return limit
- def _range_end_title_keys(titles: list[str]) -> frozenset[str]:
- return frozenset(_normalized_heading(title) for title in titles if title.strip())
- def chapter_line_ranges(
- text: str,
- chapter_titles: list[str],
- *,
- range_end_titles: list[str] | None = None,
- ) -> list[dict[str, int | str]]:
- """1-based inclusive line ranges for outline chapters in *text*."""
- lines = text.splitlines()
- end_keys = _range_end_title_keys(range_end_titles or [])
- starts: list[tuple[str, int]] = []
- cursor = 0
- for title in chapter_titles:
- line_index = _find_outline_chapter_start(lines, title, cursor)
- if line_index is None:
- print(
- f"WARNING: outline chapter not found in extracted text: {title!r}",
- file=sys.stderr,
- )
- continue
- starts.append((title, line_index))
- cursor = line_index + 1
- if not starts:
- return []
- ranges: list[dict[str, int | str]] = []
- for index, (title, start_line) in enumerate(starts):
- next_start = starts[index + 1][1] if index + 1 < len(starts) else len(lines)
- end_exclusive = _first_chapter_boundary(
- lines, start_line + 1, next_start, end_keys
- )
- ranges.append(
- {
- "title": title,
- "start_line": start_line + 1,
- "end_line": end_exclusive,
- }
- )
- return ranges
- def detect_structure(text: str) -> dict:
- """Detect chapter count and table of contents presence.
- Scans the whole text (not just the head) and counts DISTINCT chapter numbers
- from explicit "Chapter N"/"Capítulo N" headings, rejecting prose
- cross-references and numbered list items. Counting distinct numbers means a
- ToC entry and its body heading are not double-counted.
- Lines inside closed code fences are skipped, the same guard
- `_structural_chapter_count()` uses: a sample holding a diff
- ("1 + use crate::trpl::StreamExt;") otherwise matched the numeric scan and
- the count built from code lines won over the structural count. A period
- tail is judged against the line above it so a wrapped cross-reference
- ("...as we cover in / Chapter 6. Closures create types...") is not counted
- while a heading on its own line still is. (Issue #237)
- """
- lines = text.splitlines()
- fenced = _closed_fence_line_numbers(lines)
- headings = []
- numbers = set()
- part_headings = []
- part_numbers = set()
- prev = "" # previous non-blank line; cleared by blanks and fences
- for index, line in enumerate(lines):
- if index in fenced:
- prev = ""
- continue
- s = line.strip()
- if not s:
- prev = ""
- continue
- num = _chapter_number(line, prev)
- if num is not None:
- title = _MD_HEADING_PREFIX.sub("", s)
- if _PART_HEADER.match(title):
- part_numbers.add(num)
- part_headings.append(s)
- else:
- numbers.add(num)
- headings.append(s)
- prev = s
- numeric_count = len(numbers)
- # Fall back to structural (Markdown/AsciiDoc) headings only when no numeric
- # "Chapter N" headings were found, so books with real chapters are unaffected.
- #
- # Which branch answered is reported alongside the count. The two disagree
- # often, and a wrong count is not visible in the output it produces: it
- # becomes the plan in Step 3 and the chapter files of the generated skill.
- # Every parser in this project already announces which method it used
- # ("Trying python-docx... OK"); this decision had the same shape and was
- # the only silent one.
- if numeric_count >= 2:
- chapters_detected = numeric_count
- chapters_method = "numeric"
- chapter_headings_sample = headings[:10]
- else:
- # A single stray number (e.g. a Roman numeral inside an example paper
- # reproduced in the book, or a lone "Part 1") is not enough to suppress
- # the structural (Markdown/AsciiDoc) heading count, so course-style
- # books with "### Unit N" headings still get counted via max().
- structural_headings = _structural_chapter_headings(text)
- structural_count = len(structural_headings)
- # Parts group chapters and must not win the numeric branch ahead of
- # the structural chapters inside them. Retain the old Part-only
- # fallback when the extraction contains no other chapter evidence.
- if not numeric_count and not structural_count:
- numeric_count = len(part_numbers)
- headings = part_headings
- chapters_detected = max(numeric_count, structural_count)
- chapters_method = (
- "structural" if structural_count > numeric_count
- else "numeric" if numeric_count
- else "none"
- )
- chapter_headings_sample = (
- structural_headings[:10]
- if chapters_method == "structural"
- else headings[:10]
- )
- # Look for ToC indicators in the first ~30k chars (multilingual; see _TOC_PATTERN)
- has_toc = bool(_TOC_PATTERN.search(text[:30000]))
- return {
- "chapters_detected": chapters_detected,
- "chapters_method": chapters_method,
- "chapter_headings_sample": chapter_headings_sample,
- "has_toc": has_toc,
- }
- def apply_pdf_outline(structure: dict, pdf_path: str) -> dict:
- """Prefer the PDF's embedded outline over the text heuristics.
- The text scan treats any short line starting with a number and two spaces
- ("1 Introduction") as a chapter heading, so page-layout lines such as
- numbered figure/listing captions and ToC rows inflate a technical book to
- hundreds of "chapters". A publisher outline is authoritative; the
- heuristics remain the fallback when the PDF has none.
- """
- titles = [t for t in pdf_outline_titles(pdf_path) if not _is_non_chapter_section(t)]
- if len(titles) < 2:
- return structure
- all_titles = pdf_outline_all_titles(pdf_path)
- range_end_titles = [t for t in all_titles if _is_non_chapter_section(t)]
- return {
- **structure,
- "chapters_detected": len(titles),
- "chapters_method": "outline",
- "chapter_headings_sample": titles[:10],
- "chapter_titles": titles,
- "chapter_range_end_titles": range_end_titles,
- }
- def parse_arguments(argv: list[str]) -> tuple[list[str], str, str]:
- """Parse argv into (input_paths, extraction_mode, install_mode)."""
- input_paths = []
- extraction_mode = "text"
- args = argv[1:]
- i = 0
- while i < len(args):
- arg = args[i]
- if arg == "--mode":
- if i + 1 < len(args):
- extraction_mode = args[i+1].lower()
- i += 2
- else:
- i += 1
- elif arg == "--install-missing":
- if i + 1 < len(args) and not args[i+1].startswith("--"):
- i += 2
- else:
- i += 1
- elif arg == "--no-install-missing":
- i += 1
- elif arg == "--pdf-page-sidecar":
- i += 1
- elif arg.startswith("-"):
- print(f"WARNING: Unknown flag '{arg}' — ignoring it.", file=sys.stderr)
- i += 1
- else:
- input_paths.append(arg)
- i += 1
- install_mode = normalize_install_mode(argv)
- if extraction_mode not in ("technical", "text"):
- extraction_mode = "text"
- return input_paths, extraction_mode, install_mode
- def resolve_input_files(paths: list[str]) -> list[Path]:
- """Resolve paths including files, directories, and glob patterns to Path objects.
- User-given order is preserved for explicit file arguments. Expanded
- results (directories, globs) are sorted deterministically so repeated
- runs produce the same output.
- A leading "~" is expanded here rather than relying on the shell: a glob has
- to be quoted to reach us unexpanded ("~/books/*.epub"), and quoting stops
- the shell expanding the tilde too. `glob.glob` and `Path` both treat "~" as
- a literal directory name, so without this the pattern silently matches
- nothing.
- """
- resolved = []
- for raw_path in paths:
- # Normalise "~" once, at the entry point, so both the glob branch and
- # the file/directory branch below see a real path.
- path_str = os.path.expanduser(raw_path)
- # Check if it has glob wildcards
- if not Path(path_str).exists() and any(
- char in path_str for char in ("*", "?", "[")
- ):
- glob_matches = glob.glob(path_str, recursive=True)
- # Sort expanded glob results deterministically
- expanded = []
- for match in glob_matches:
- p = Path(match)
- if p.is_file() and p.suffix.lower() in SUPPORTED_EXTENSIONS:
- expanded.append(p.resolve())
- expanded.sort(key=lambda x: str(x).lower())
- resolved.extend(expanded)
- else:
- p = Path(path_str)
- if p.is_dir():
- # Sort expanded directory results deterministically
- dir_files = []
- for root, _, files in os.walk(p):
- for file in files:
- file_path = Path(root) / file
- if file_path.suffix.lower() in SUPPORTED_EXTENSIONS:
- dir_files.append(file_path.resolve())
- dir_files.sort(key=lambda x: str(x).lower())
- resolved.extend(dir_files)
- else:
- # Keep even if it doesn't exist so the error check can report it
- resolved.append(p.resolve())
- # Deduplicate while preserving insertion order (user order for explicit files)
- seen = set()
- unique_paths = []
- for path in resolved:
- resolved_path = path.resolve() if path.exists() else path
- if resolved_path not in seen:
- seen.add(resolved_path)
- unique_paths.append(resolved_path)
- return unique_paths
- def _sha256_file(path: str) -> str:
- """Streaming sha256 — constant memory for multi-GB sources."""
- h = hashlib.sha256()
- with open(path, "rb") as f:
- for chunk in iter(lambda: f.read(1 << 20), b""):
- h.update(chunk)
- return h.hexdigest()
- def reuse_is_safe(current_inputs, metadata, current_mode):
- """Decide whether an interrupted run's extraction workdir is safe to resume.
- current_inputs: list of (path, fingerprint) — fingerprint is the sha256
- hexdigest of each file as it exists NOW, recomputed by the caller
- (hashlib.sha256(path.read_bytes()).hexdigest(); _sha256_file is available
- for streaming on large files). Size is deliberately NOT part of matching:
- metadata.json only records file_size_mb rounded to 2 decimals, so an
- exact-size comparison is unimplementable — the fingerprint subsumes size
- entirely.
- metadata: the recorded run's metadata dict (parsed metadata.json).
- current_mode: the extraction mode THIS run intends ("technical" or "text").
- Returns (ok, reason). ok=True only when ALL hold:
- - metadata["workdir"] exists and is a directory
- - metadata["extraction_mode"] == current_mode
- - the recorded sources match the current inputs one-to-one on
- filename AND sha256 — the recomputed fingerprints vs the recorded ones
- Anything else: (False, reason) — the caller must start a fresh
- extraction (or ask the user when unsure), never resume.
- """
- if not isinstance(metadata, dict):
- return False, "metadata.json unreadable or not an object"
- workdir = metadata.get("workdir")
- if not workdir or not os.path.isdir(workdir):
- return False, "workdir missing or unreadable"
- if metadata.get("extraction_mode") != current_mode:
- return False, (
- f"mode changed (recorded {metadata.get('extraction_mode')!r}, "
- f"current is {current_mode!r})"
- )
- recorded = metadata.get("sources") or []
- if len(recorded) != len(current_inputs):
- return False, "sources changed"
- current_pairs = [(Path(path).name, fingerprint) for path, fingerprint in current_inputs]
- current_by_name = {}
- for filename, fingerprint in current_pairs:
- current_by_name.setdefault(filename, []).append(fingerprint)
- recorded_pairs = []
- for src in recorded:
- filename = src.get("filename")
- if filename not in current_by_name:
- return False, f"sources changed (recorded {filename!r} is not in this run)"
- recorded_hash = src.get("sha256")
- if not recorded_hash:
- return False, (
- f"no recorded fingerprint for {filename} — legacy metadata "
- "(recorded before fingerprints existed)"
- )
- recorded_pairs.append((filename, recorded_hash))
- if recorded_pairs != current_pairs:
- if Counter(recorded_pairs) == Counter(current_pairs):
- return False, "sources reordered"
- recorded_name_counts = Counter(filename for filename, _ in recorded_pairs)
- current_name_counts = Counter(filename for filename, _ in current_pairs)
- if all(count == 1 for count in recorded_name_counts.values()) and all(
- count == 1 for count in current_name_counts.values()
- ):
- for filename, recorded_hash in recorded_pairs:
- if recorded_hash != current_by_name[filename][0]:
- return False, f"content changed for {filename}"
- return False, "sources changed"
- return True, "workdir intact and all sources match the current inputs"
- def extract_single_file(input_path: Path, extraction_mode: str, install_mode: str) -> dict:
- """Extract text and metadata from a single file path."""
- input_str = str(input_path)
- if not input_path.exists():
- raise ExtractionError(f"File not found: {input_str}")
- ext = input_path.suffix.lower()
- document_format = ext.lstrip(".")
- # Sniff magic bytes if suffix is not supported.
- #
- # Every failure in this function has to surface as ExtractionError: the
- # batch loop in main() catches only that, and anything else aborts the whole
- # run — including the sources that would have extracted fine. An unreadable
- # or unopenable file is a per-source problem, so translate it here. (The
- # ZipFile branch below already does this for OSError.)
- if ext not in SUPPORTED_EXTENSIONS:
- try:
- with open(input_str, "rb") as f:
- header = f.read(8)
- except OSError as exc:
- raise ExtractionError(
- f"Could not read {input_path.name}: {exc.strerror or exc}"
- ) from exc
- if header[:4] == b"%PDF":
- ext = ".pdf"
- document_format = "pdf"
- elif header[:2] == b"PK":
- try:
- with zipfile.ZipFile(input_str) as zf:
- names = set(zf.namelist())
- if "mimetype" in names and zf.read("mimetype").startswith(b"application/epub"):
- ext = ".epub"
- document_format = "epub"
- elif "word/document.xml" in names:
- ext = ".docx"
- document_format = "docx"
- else:
- raise ExtractionError(
- f"Unsupported ZIP-based format '{input_path.name}'. Supported: {supported_formats_message()}"
- )
- except (zipfile.BadZipFile, KeyError, OSError):
- raise ExtractionError(
- f"Unsupported ZIP-based format '{input_path.name}'. Supported: {supported_formats_message()}"
- )
- else:
- raise ExtractionError(
- f"Unsupported format '{ext or '<none>'}'. Supported: {supported_formats_message()}"
- )
- prepare_dependencies(ext, extraction_mode, install_mode)
- if ext in CALIBRE_EBOOK_EXTENSIONS and not shutil.which("ebook-convert"):
- raise ExtractionError(
- "MOBI/AZW/AZW3 extraction requires Calibre's ebook-convert command. "
- "Install Calibre and ensure ebook-convert is on PATH, then rerun this command."
- )
- text = ""
- method = ""
- fallback_reason = None
- pages = 0
- pages_label = "sections"
- images_dropped = None
- if ext == ".epub":
- print(f"Extracting EPUB: {input_str}")
- if extraction_mode == "technical":
- fallback_reason = "format_not_supported_in_technical_mode"
- print(
- "WARNING: Technical mode is not supported for EPUB; "
- "using the EPUB text extraction chain. "
- "Layout-aware extraction was not used.",
- file=sys.stderr,
- )
- text = extract_with_ebooklib(input_str)
- if text and text.strip():
- method = "ebooklib"
- else:
- print("ebooklib not available")
- print("Trying stdlib zipfile parser...", end=" ", flush=True)
- text = extract_with_zipfile(input_str)
- if text and text.strip():
- print("OK")
- method = "zipfile"
- else:
- print("FAILED")
- raise ExtractionError(
- "Could not extract text from EPUB.\n"
- "Install ebooklib + beautifulsoup4 for best results:\n"
- " pip3 install ebooklib beautifulsoup4"
- )
- pages = count_epub_chapters(input_str)
- pages_label = "spine_items"
- images_dropped = count_epub_images(input_str)
- if images_dropped > _EPUB_IMAGE_NOTICE_THRESHOLD:
- print(
- f" [warn] {input_path.name} contains {images_dropped} image(s); "
- "their content is not extracted",
- file=sys.stderr,
- )
- elif ext == ".pdf":
- print(f"Extracting PDF: {input_str}")
- if looks_image_only(input_str):
- raise ExtractionError(
- f"{input_path.name} looks like a scanned (image-only) PDF: its first pages "
- "contain no extractable text, only images.\n"
- "Run OCR on it first, then retry:\n"
- " ocrmypdf input.pdf output.pdf"
- )
- if extraction_mode == "technical":
- print("Mode: technical — using Docling (layout-aware)...")
- try:
- text = extract_with_docling(input_str)
- except Exception as exc:
- fallback_reason = "docling_failed"
- print(
- f"WARNING: Docling failed ({type(exc).__name__}: {exc}); "
- "using the PDF text fallback chain.",
- file=sys.stderr,
- )
- text = None
- if text and text.strip():
- method = "docling"
- print("OK")
- else:
- if fallback_reason is None:
- if text is None:
- fallback_reason = "docling_unavailable"
- detail = "Docling is unavailable"
- else:
- fallback_reason = "docling_empty"
- detail = "Docling produced no text"
- print(
- f"WARNING: {detail}; using the PDF text fallback chain.",
- file=sys.stderr,
- )
- extraction_mode = "text"
- if extraction_mode == "text" or not text:
- print("Mode: text — using pdftotext...")
- print("Trying pdftotext...", end=" ", flush=True)
- text = extract_with_pdftotext(input_str)
- if text and text.strip():
- method = "pdftotext"
- print("OK")
- else:
- print("not available")
- print("Trying pypdf...", end=" ", flush=True)
- text = extract_with_pypdf(input_str)
- if text and text.strip():
- method = "pypdf"
- print("OK")
- else:
- print("not available")
- print("Trying pdfminer.six...", end=" ", flush=True)
- text = extract_with_pdfminer(input_str)
- if text and text.strip():
- method = "pdfminer"
- print("OK")
- else:
- print("FAILED")
- raise ExtractionError(
- "Could not extract text from PDF.\n"
- "Install one of: poppler-utils (pdftotext), pypdf, or pdfminer.six\n"
- " sudo apt install poppler-utils\n"
- " pip3 install pypdf\n"
- " pip3 install pdfminer.six"
- )
- pages = count_pages(input_str)
- pages_label = "pages"
- elif ext in TEXT_EXTENSIONS:
- print(f"Extracting text document: {input_str}")
- text = read_text_file(input_str)
- if text is None or not text.strip():
- raise ExtractionError(f"Could not read text document: {input_path.name}")
- method = "plain-text"
- pages = 0
- pages_label = "sections"
- elif ext in HTML_EXTENSIONS:
- print(f"Extracting HTML: {input_str}")
- text = extract_html_file(input_str)
- if text is None or not text.strip():
- raise ExtractionError(f"Could not extract text from HTML: {input_path.name}")
- method = "html-parser"
- pages = 0
- pages_label = "sections"
- elif ext == ".docx":
- print(f"Extracting DOCX: {input_str}")
- text, method = extract_docx(input_str)
- pages = 0
- pages_label = "sections"
- elif ext == ".rtf":
- print(f"Extracting RTF: {input_str}")
- text, method = extract_rtf(input_str)
- pages = 0
- pages_label = "sections"
- elif ext in CALIBRE_EBOOK_EXTENSIONS:
- print(f"Extracting ebook with Calibre: {input_str}")
- text = extract_with_ebook_convert(input_str)
- if text is None or not text.strip():
- raise ExtractionError(
- f"Could not extract text from {ext}. Install Calibre and ensure ebook-convert is on PATH."
- )
- method = "ebook-convert"
- pages = 0
- pages_label = "sections"
- text, removed_invisible = sanitize_extracted_text(text)
- if removed_invisible:
- print(
- f" [security] removed {removed_invisible} invisible Unicode "
- f"code point(s) from {input_path.name}",
- file=sys.stderr,
- )
- if not text.strip():
- raise ExtractionError(
- f"Extracted text from {input_path.name} contained no visible content "
- "after Unicode sanitization."
- )
- tokens = estimate_tokens(text)
- structure = detect_structure(text)
- if ext == ".pdf":
- structure = apply_pdf_outline(structure, input_str)
- print(
- f" chapters: {structure['chapters_detected']} "
- f"({structure['chapters_method']})"
- )
- try:
- file_size_mb = os.path.getsize(input_str) / (1024 * 1024)
- file_sha256 = _sha256_file(input_str)
- except OSError as exc:
- raise ExtractionError(
- f"Could not read file size for {input_path.name}: {exc}"
- ) from exc
- return {
- "source_file": str(input_path.resolve()),
- "filename": input_path.name,
- "format": document_format,
- "extraction_method": method,
- "fallback_reason": fallback_reason,
- "file_size_mb": round(file_size_mb, 2),
- "sha256": file_sha256,
- pages_label: pages,
- "pages_label": pages_label,
- "pages": pages,
- "chars": len(text),
- "words": len(text.split()),
- "estimated_tokens": tokens,
- "images_dropped": images_dropped,
- "text": text,
- **structure,
- }
- def prepare_output_dir(path: Path) -> None:
- """Create the work directory, guarding against two shared-tmp risks:
- a pre-planted symlink at a predictable path, and reusing a directory
- another user already owns (either could expose or tamper with the
- extracted document text, which may be sensitive).
- """
- if path.is_symlink():
- raise ExtractionError(
- f"Refusing to use {path}: it is a symbolic link, not a real "
- "directory. Remove it or set BOOK_SKILL_WORKDIR to a private path."
- )
- if path.exists():
- if not path.is_dir():
- raise ExtractionError(f"Refusing to use {path}: it exists and is not a directory.")
- if hasattr(os, "getuid"):
- owner_uid = path.stat().st_uid
- if owner_uid != os.getuid():
- raise ExtractionError(
- f"Refusing to use {path}: it is owned by a different user "
- f"(uid {owner_uid}). Set BOOK_SKILL_WORKDIR to a private directory."
- )
- os.chmod(path, 0o700)
- else:
- path.mkdir(parents=True, mode=0o700)
- def write_pdf_page_sidecar(sources: list[dict], output_dir: Path) -> Path | None:
- """Write opt-in physical-page evidence without changing legacy outputs.
- pypdf's page text is a separate text-layer view, not a character-offset map
- into ``full_text.txt``. When the main extractor differs, mark that explicitly.
- A source whose hash changes between extraction and this pass gets no page
- evidence, so citations cannot silently refer to a different PDF revision.
- """
- entries = []
- for source in sources:
- if source["format"] != "pdf":
- continue
- entry = {
- "source_file": source["source_file"],
- "source_sha256": source["sha256"],
- "full_text_extraction_method": source["extraction_method"],
- "page_text_extraction_method": "pypdf",
- "status": "unavailable",
- "pages": [],
- }
- try:
- if _sha256_file(source["source_file"]) != source["sha256"]:
- entry["reason"] = "source_changed_since_extraction"
- else:
- records = extract_pypdf_page_records(source["source_file"])
- if _sha256_file(source["source_file"]) != source["sha256"]:
- entry["reason"] = "source_changed_during_page_extraction"
- elif records is None:
- entry["reason"] = "pypdf_unavailable_or_failed"
- else:
- entry["status"] = "ok"
- if source["pages"] and len(records) != source["pages"]:
- entry["low_confidence_reasons"] = ["page_count_mismatch"]
- for record in records:
- record["source_sha256"] = source["sha256"]
- if source["extraction_method"] != "pypdf":
- record["low_confidence_reasons"].append(
- "different_extractor_from_full_text"
- )
- record["requires_manual_review"] = bool(
- record["low_confidence_reasons"]
- )
- entry["pages"] = records
- except OSError as exc:
- entry["reason"] = f"source_unreadable: {type(exc).__name__}"
- entries.append(entry)
- payload = {"schema_version": 1, "sources": entries}
- destination = output_dir / "pdf_pages.json"
- temporary = None
- try:
- with tempfile.NamedTemporaryFile(
- mode="w", encoding="utf-8", newline="\n", dir=output_dir,
- prefix=".pdf_pages-", suffix=".tmp", delete=False,
- ) as stream:
- temporary = Path(stream.name)
- json.dump(payload, stream, indent=2, ensure_ascii=False)
- stream.write("\n")
- os.replace(temporary, destination)
- except OSError as exc:
- print(f" [warn] could not write PDF page sidecar: {exc}", file=sys.stderr)
- if temporary is not None:
- temporary.unlink(missing_ok=True)
- return None
- return destination
- def print_intro() -> None:
- """Two lines of attribution at the start of every run.
- Printed here rather than only in SKILL.md so it shows however the agent
- invokes extraction. States who maintains the project without asking for
- anything — the ask belongs at the end, after the work is delivered.
- """
- sys.stderr.write(
- "book-to-skill · turns a document into a structured agent skill\n"
- "free and MIT-licensed · maintained in personal time · "
- "github.com/virgiliojr94/book-to-skill\n\n"
- )
- def print_support_note() -> None:
- """One closing line about funding, printed only after a successful run.
- Deliberately at the end and deliberately conditional: the reader has just
- received something that worked, and the sentence says what the money is
- for rather than asking for it. Never printed when extraction failed —
- nobody should be asked to fund what just wasted their time.
- Written to stdout, with the rest of the closing report: stderr is
- unbuffered and stdout is not when the run is piped (which is how an agent
- captures it), so mixing the two puts the closing line at the top.
- """
- print(
- "\n book-to-skill is free, and maintained in personal time."
- "\n If it saves you work, you can fund its upkeep: "
- "github.com/sponsors/virgiliojr94"
- )
- def print_usage() -> None:
- """Print standalone CLI usage."""
- print(
- "Usage: book-to-skill <path-to-document-folder-or-glob>... "
- "[--mode technical|text] [--install-missing ask|yes|no] "
- "[--pdf-page-sidecar]",
- file=sys.stderr,
- )
- print(
- " book-to-skill --check # report which extractors are installed",
- file=sys.stderr,
- )
- print(f"Supported formats: {supported_formats_message()}", file=sys.stderr)
- def main():
- print_intro()
- if any(arg in {"-h", "--help"} for arg in sys.argv[1:]):
- print_usage()
- sys.exit(0)
- if "--check" in sys.argv[1:]:
- sys.exit(run_dependency_check())
- if len(sys.argv) < 2:
- print_usage()
- sys.exit(1)
- raw_input_paths, extraction_mode, install_mode = parse_arguments(sys.argv)
- if not raw_input_paths:
- print("ERROR: No input document, folder, or glob pattern specified.", file=sys.stderr)
- sys.exit(1)
- input_files = resolve_input_files(raw_input_paths)
- if not input_files:
- print(f"ERROR: No supported files found matching: {', '.join(raw_input_paths)}", file=sys.stderr)
- sys.exit(1)
- prepare_output_dir(OUTPUT_DIR)
- extracted_sources = []
- combined_texts = []
- errors = []
- for file_path in input_files:
- try:
- res = extract_single_file(file_path, extraction_mode, install_mode)
- except ExtractionError as exc:
- print(f"WARNING: Skipping {file_path.name}: {exc}", file=sys.stderr)
- errors.append((file_path, str(exc)))
- continue
- extracted_sources.append(res)
- # Format the text with a clear boundary
- separator = f"\n\n{'=' * 80}\nSOURCE: {res['filename']} (Path: {res['source_file']})\n{'=' * 80}\n\n"
- combined_texts.append(separator + res["text"])
- if not extracted_sources:
- print(f"\nERROR: All {len(errors)} source(s) failed extraction:", file=sys.stderr)
- for path, err in errors:
- print(f" - {path.name}: {err}", file=sys.stderr)
- sys.exit(1)
- # Combine texts
- consolidated_text = "".join(combined_texts).strip()
- # Preserve existing source line endings. On Windows, the default newline
- # translation turns an extracted CRLF into CRCRLF and breaks setext headings
- # when the persisted corpus is read back.
- with OUTPUT_TEXT.open("w", encoding="utf-8", newline="\n") as output_file:
- output_file.write(consolidated_text)
- # Consolidate metadata
- total_file_size_mb = sum(src["file_size_mb"] for src in extracted_sources)
- total_pages = sum(src["pages"] for src in extracted_sources)
- total_chars = len(consolidated_text)
- total_words = len(consolidated_text.split())
- total_tokens = estimate_tokens(consolidated_text)
- total_images_dropped = sum(
- src["images_dropped"] or 0 for src in extracted_sources
- )
- # Detect structure from source content only. The generated SOURCE banners in
- # full_text.txt use rows of "=", which can otherwise become phantom setext
- # headings and make the result depend on the source-path length.
- structure_text = "\n\n".join(src["text"] for src in extracted_sources)
- consolidated_structure = detect_structure(structure_text)
- if len(extracted_sources) == 1 and extracted_sources[0]["chapters_method"] == "outline":
- # The outline came from the file, not the text; a re-scan would lose it.
- src = extracted_sources[0]
- consolidated_structure.update(
- {k: src[k] for k in (
- "chapters_detected",
- "chapters_method",
- "chapter_headings_sample",
- "chapter_titles",
- ) if k in src}
- )
- chapter_titles = src.get("chapter_titles")
- if chapter_titles:
- end_titles = src.get("chapter_range_end_titles")
- consolidated_structure["chapter_ranges"] = chapter_line_ranges(
- consolidated_text,
- chapter_titles,
- range_end_titles=end_titles,
- )
- structural_sources = [
- src for src in extracted_sources
- if src["chapters_method"] == "structural" and src["chapters_detected"] >= 2
- ]
- if (
- len(extracted_sources) > 1
- and structural_sources
- and all(src["chapters_method"] != "numeric" for src in extracted_sources)
- ):
- # Each source has already selected its own chapter depth. Re-scanning
- # the joined corpus selects the H1 source titles instead when there are
- # multiple books with H2 chapters (issue #272). A lone heading in a
- # source may only be its title, so it is not added to the chapter count.
- consolidated_structure["chapters_detected"] = sum(
- src["chapters_detected"] for src in structural_sources
- )
- consolidated_structure["chapter_headings_sample"] = [
- heading
- for src in structural_sources
- for heading in src["chapter_headings_sample"]
- ][:10]
- consolidated_structure["chapters_method"] = "structural"
- # has_toc is a per-source property, so it has to be combined per source
- # rather than re-derived from the corpus. detect_structure only scans the
- # first ~30k chars, because a table of contents sits in a book's front
- # matter -- but on a consolidated corpus that window covers only the FIRST
- # source, so a ToC in any later book was invisible and the answer flipped on
- # input order alone. Each per-source result already scanned its own front
- # matter, so OR them.
- consolidated_structure["has_toc"] = any(
- src["has_toc"] for src in extracted_sources
- )
- metadata = {
- "source_file": "Consolidated from multiple sources" if len(extracted_sources) > 1 else extracted_sources[0]["source_file"],
- "filename": "multi-source" if len(extracted_sources) > 1 else extracted_sources[0]["filename"],
- "format": "mixed" if len(extracted_sources) > 1 else extracted_sources[0]["format"],
- "extraction_method": "multi-method" if len(extracted_sources) > 1 else extracted_sources[0]["extraction_method"],
- "extraction_mode": extraction_mode,
- "file_size_mb": round(total_file_size_mb, 2),
- "pages": total_pages,
- "chars": total_chars,
- "words": total_words,
- "estimated_tokens": total_tokens,
- "estimated_tokens_human": f"~{total_tokens // 1000}K",
- "images_dropped": total_images_dropped,
- # Self-describing so a consumer can clean up exactly the directory this
- # run created, without having to reconstruct the per-run default path.
- "workdir": str(OUTPUT_DIR),
- "output_text": str(OUTPUT_TEXT),
- "total_sources": len(extracted_sources),
- "sources": [
- {
- "source_file": src["source_file"],
- "filename": src["filename"],
- "format": src["format"],
- "extraction_method": src["extraction_method"],
- "fallback_reason": src["fallback_reason"],
- "file_size_mb": src["file_size_mb"],
- "sha256": src["sha256"],
- "pages": src["pages"],
- "pages_label": src["pages_label"],
- "chars": src["chars"],
- "words": src["words"],
- "estimated_tokens": src["estimated_tokens"],
- "images_dropped": src["images_dropped"],
- "chapters_detected": src["chapters_detected"],
- "chapters_method": src["chapters_method"],
- "chapter_headings_sample": src["chapter_headings_sample"],
- **(
- {"chapter_titles": src["chapter_titles"]}
- if "chapter_titles" in src
- else {}
- ),
- **(
- {
- "chapter_ranges": chapter_line_ranges(
- src["text"],
- src["chapter_titles"],
- range_end_titles=src.get("chapter_range_end_titles"),
- )
- }
- if "chapter_titles" in src
- else {}
- ),
- "has_toc": src["has_toc"]
- }
- for src in extracted_sources
- ],
- **consolidated_structure,
- }
- # encoding="utf-8" is required, not cosmetic: the payload is dumped with
- # ensure_ascii=False, so any non-ASCII chapter heading, filename or path
- # reaches the encoder verbatim. Without it, write_text() falls back to the
- # locale encoding and raises UnicodeEncodeError on a Windows cp1252 host or
- # under LC_ALL=C — after every source has already been extracted.
- OUTPUT_META.write_text(
- json.dumps(metadata, indent=2, ensure_ascii=False), encoding="utf-8"
- )
- page_sidecar = None
- if "--pdf-page-sidecar" in sys.argv[1:]:
- page_sidecar = write_pdf_page_sidecar(extracted_sources, OUTPUT_DIR)
- page_line = f" Total Pages: {total_pages}"
- print("\nExtraction complete:")
- print(f" Sources : {len(extracted_sources)} processed")
- print(f" Size : {total_file_size_mb:.2f} MB")
- print(page_line)
- print(f" Words : {total_words:,}")
- print(f" Tokens : ~{total_tokens // 1000}K")
- print(
- f" Chapters: {consolidated_structure['chapters_detected']} detected overall "
- f"({consolidated_structure['chapters_method']})"
- )
- if consolidated_structure["chapters_method"] == "structural" and (
- consolidated_structure["chapters_detected"] <= 1 and total_words > 5000
- ):
- # Numeric "Chapter N" headings found nothing and the structural fallback
- # came back with one section for a document of real length. That pairing
- # is a detection failure far more often than it is a one-chapter book,
- # and it is invisible in the output it produces.
- print(
- " WARN : only one section found in a document this long — chapter "
- "detection likely failed; check the headings before generating."
- )
- print(f" ToC : {'yes' if consolidated_structure['has_toc'] else 'not detected'}")
- if not consolidated_structure["has_toc"]:
- print(
- " WARN : No table of contents detected — chapter mapping in Step 3 "
- "will rely on heading scan only, which may miss or duplicate sections."
- )
- print(f"\n Workdir -> {OUTPUT_DIR}")
- print(f" Text -> {OUTPUT_TEXT}")
- print(f" Meta -> {OUTPUT_META}")
- if page_sidecar is not None:
- print(f" PDF pages -> {page_sidecar}")
- if errors:
- print(f"\n WARNING: {len(errors)} source(s) skipped due to errors:")
- for path, err in errors:
- print(f" - {path.name}: {err}")
- else:
- print_support_note()
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.