Skip to content
Back to search

refactored-spoon

Reliable offline-first document corpus + MCP server for local LLMs.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ljrseals-dev--refactored-spoon/badge.svg)](/listings/ljrseals-dev--refactored-spoon)

HTML

<a href="/listings/ljrseals-dev--refactored-spoon"><img src="/api/listings/ljrseals-dev--refactored-spoon/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# Install
pip install -e .

# (Optional) Download legal corpora
python -m offline_doc_mcp.cli acquire --service project_gutenberg
python -m offline_doc_mcp.cli acquire --service wikipedia_dumps
python -m offline_doc_mcp.cli acquire --service internet_archive_pd

# Process + index
python -m offline_doc_mcp.cli process
python -m offline_doc_mcp.cli index

# Run MCP server (stdio)
python -m offline_doc_mcp.mcp_server

# Optional local HTML UI
python -m offline_doc_mcp.html_server --port 8765

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 112 lines
  1. # Offline Document MCP Server
  2. Reliable offline-first document corpus + MCP server for local LLMs.
  3. ## Core Guarantees (from mode-isolation patch)
  4. - **CLASSICAL / ARCHIVE / AUTO** modes **never** touch an LLM provider, never perform network calls after corpus is built, and never discover API keys.
  5. - Only modes explicitly marked `allows_provider=True` may use a provider explicitly injected by the caller; there is no environment-key discovery.
  6. - All document acquisition is limited to clearly public-domain / open-license sources.
  7. - Once indexed, the system runs fully offline.
  8. ## Supported Sources (modular)
  9. Configured in `config/services.yaml`. Add new legal sources by appending a service block.
  10. Current defaults:
  11. - Project Gutenberg (public domain)
  12. - Wikipedia REST page extracts (CC BY-SA; large dumps are not automatically mirrored)
  13. - Internet Archive public-domain / CC0 items
  14. ## Quick Start
  15. ```bash
  16. # Install
  17. pip install -e .
  18. # (Optional) Download legal corpora
  19. python -m offline_doc_mcp.cli acquire --service project_gutenberg
  20. python -m offline_doc_mcp.cli acquire --service wikipedia_dumps
  21. python -m offline_doc_mcp.cli acquire --service internet_archive_pd
  22. # Process + index
  23. python -m offline_doc_mcp.cli process
  24. python -m offline_doc_mcp.cli index
  25. # Run MCP server (stdio)
  26. python -m offline_doc_mcp.mcp_server
  27. # Optional local HTML UI
  28. python -m offline_doc_mcp.html_server --port 8765
  29. ```
  30. ## Real acquisition, robots policy and attribution
  31. The acquisition services now download source material rather than creating metadata
  32. stubs. All source requests use HTTPS, a descriptive `RefactoredSpoonBot` User-Agent,
  33. a per-host delay, bounded retries, redirect limits, download-size caps, and the
  34. origin's `robots.txt`. A disallowed path is skipped; if robots policy cannot be
  35. read (network error or server error), the downloader fails closed. It does not
  36. spoof browser identities, bypass robots rules, or defeat authentication.
  37. Set a contact address so source operators can identify the crawler:
  38. ```bash
  39. export OFFLINE_DOC_CONTACT="crawler-admin@example.org"
  40. ```
  41. This is contact metadata, not a password or API credential. Public Gutenberg,
  42. Wikipedia REST and public Internet Archive files do not require login credentials.
  43. The downloader does not send stored cookies or API keys. It will not fetch private,
  44. restricted, or login-gated items.
  45. ### Download and index
  46. ```bash
  47. # Project Gutenberg: downloads UTF-8 plain text for the configured US
  48. # public-domain jurisdiction; confirm legal status outside the US.
  49. offline-doc acquire --service project_gutenberg
  50. # Internet Archive: searches the configured query, verifies a Public Domain
  51. # or CC0 license in item metadata, and downloads a .txt file when available.
  52. offline-doc acquire --service internet_archive_pd
  53. # Wikipedia: provide a search phrase; saves attributed CC BY-SA extracts.
  54. offline-doc acquire --service wikipedia_dumps --query "open source software"
  55. # Build / refresh the FTS index, then search it.
  56. offline-doc index
  57. offline-doc search "document search"
  58. python -m offline_doc_mcp.mcp_server
  59. ```
  60. Official source links:
  61. - [Project Gutenberg catalog](https://www.gutenberg.org/cache/epub/feeds/pg_catalog.csv)
  62. - [Project Gutenberg terms](https://www.gutenberg.org/policy/terms_of_use.html)
  63. - [Internet Archive advanced search](https://archive.org/advancedsearch.php)
  64. - [Internet Archive metadata API](https://archive.org/developers/md-read.html)
  65. - [Wikimedia dump portal](https://dumps.wikimedia.org/)
  66. - [Wikipedia REST API](https://en.wikipedia.org/api/rest_v1/)
  67. Project Gutenberg's public-domain status is jurisdiction-dependent; this
  68. configuration is explicitly limited to US public-domain use and is not a legal
  69. determination for other countries. Internet Archive files are accepted only
  70. when item metadata identifies a Public Domain Mark / CC0 license or explicit
  71. public-domain rights. Wikipedia extracts retain page URLs and CC BY-SA
  72. attribution. Sidecar JSON files preserve source/license metadata.
  73. Only `.txt`, `.md`, and `.text` files are indexed. PDF/EPUB conversion and
  74. multi-gigabyte Wikimedia dump mirroring are not silently attempted. A zero
  75. download count can mean robots.txt disallowed the source, no suitable public
  76. text was found, or the remote source was unavailable; inspect the source's
  77. robots policy and item rights rather than bypassing them.
  78. ## Mode Isolation
  79. See `src/offline_doc_mcp/engine/modes.py` and `engine.py`.
  80. Offline modes short-circuit before any provider lookup.
  81. ## License
  82. MIT (code).
  83. Documents retain their original licenses (public domain / CC BY-SA / CC0 etc.).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.