Skip to content
Back to search

content-core

Extract what matters from any media source

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/lfnovo--content-core/badge.svg)](/listings/lfnovo--content-core)

HTML

<a href="/listings/lfnovo--content-core"><img src="/api/listings/lfnovo--content-core/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pip install content-core
import content_core

result = await content_core.extract_content(url="https://example.com")
print(result.content)

Or with zero install:

uvx content-core extract "https://example.com"

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 18 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "content-core",
  3. "owner": {
  4. "name": "Luis Novo",
  5. "email": "lfnovo@gmail.com"
  6. },
  7. "description": "Content Core — content extraction toolkit for AI agents",
  8. "version": "1.0.0",
  9. "plugins": [
  10. {
  11. "name": "content-core",
  12. "source": "./",
  13. "description": "Extract text content from URLs, documents, YouTube videos, Reddit posts, and audio/video files — via CLI (uvx) or MCP.",
  14. "category": "development",
  15. "keywords": ["extraction", "content", "pdf", "youtube", "transcription", "mcp"]
  16. }
  17. ]
  18. }
README.md · README · 311 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Content Core
  2. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
  3. [![PyPI version](https://badge.fury.io/py/content-core.svg)](https://badge.fury.io/py/content-core)
  4. [![Downloads](https://pepy.tech/badge/content-core)](https://pepy.tech/project/content-core)
  5. [![Downloads](https://pepy.tech/badge/content-core/month)](https://pepy.tech/project/content-core)
  6. [![GitHub stars](https://img.shields.io/github/stars/lfnovo/content-core?style=social)](https://github.com/lfnovo/content-core)
  7. [![GitHub forks](https://img.shields.io/github/forks/lfnovo/content-core?style=social)](https://github.com/lfnovo/content-core)
  8. [![GitHub issues](https://img.shields.io/github/issues/lfnovo/content-core)](https://github.com/lfnovo/content-core/issues)
  9. [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
  10. Extract, process, and summarize content from URLs, files, and text through a unified async Python API, CLI, or MCP server.
  11. ## Supported Formats
  12. | Category | Formats |
  13. |----------|---------|
  14. | Web | URLs, HTML pages, YouTube videos, Reddit posts |
  15. | Documents | PDF, DOCX, PPTX, XLSX, ODT, ODS, ODP, EPUB, HTML, Markdown, plain text |
  16. | Media | MP3, WAV, M4A, FLAC, OGG (audio); MP4, AVI, MOV, MKV (video) |
  17. ## Quick Start
  18. ```bash
  19. pip install content-core
  20. ```
  21. ```python
  22. import content_core
  23. result = await content_core.extract_content(url="https://example.com")
  24. print(result.content)
  25. ```
  26. Or with zero install:
  27. ```bash
  28. uvx content-core extract "https://example.com"
  29. ```
  30. ## CLI Usage
  31. Content Core provides a unified `content-core` command with subcommands for extraction, summarization, and MCP server.
  32. ### Extract
  33. ```bash
  34. # From a URL
  35. content-core extract "https://example.com"
  36. # From a file
  37. content-core extract document.pdf
  38. # With JSON output
  39. content-core extract document.pdf --format json
  40. # With a specific engine
  41. content-core extract "https://example.com" --engine firecrawl
  42. # From stdin
  43. echo "some text" | content-core extract
  44. ```
  45. ### Summarize
  46. ```bash
  47. # Summarize text
  48. content-core summarize "Long article text here..."
  49. # With context
  50. content-core summarize "Long text" --context "bullet points"
  51. # From stdin
  52. cat article.txt | content-core summarize --context "explain to a child"
  53. ```
  54. ### MCP Server
  55. ```bash
  56. content-core mcp
  57. ```
  58. ### Configuration
  59. ```bash
  60. # Set persistent config
  61. content-core config set llm_provider anthropic
  62. content-core config set llm_model claude-sonnet-5
  63. # List current config
  64. content-core config list
  65. # Delete a config value
  66. content-core config delete llm_provider
  67. ```
  68. Config is stored in `~/.content-core/config.toml`. Priority: command flags > env vars > config file > defaults.
  69. ### Zero-Install with uvx
  70. All commands work without installation using `uvx`:
  71. ```bash
  72. uvx content-core extract "https://example.com"
  73. uvx content-core summarize "text" --context "one sentence"
  74. uvx content-core mcp
  75. ```
  76. ## Python API
  77. ### Extraction
  78. ```python
  79. import content_core
  80. # From a URL
  81. result = await content_core.extract_content(url="https://example.com")
  82. # From a file
  83. result = await content_core.extract_content(file_path="document.pdf")
  84. # From text
  85. result = await content_core.extract_content(content="some text")
  86. # With engine override
  87. from content_core import ContentCoreConfig
  88. config = ContentCoreConfig(url_engine="firecrawl")
  89. result = await content_core.extract_content(url="https://example.com", config=config)
  90. ```
  91. ### Summarization
  92. ```python
  93. import content_core
  94. summary = await content_core.summarize("long article text", context="bullet points")
  95. ```
  96. ### Configuration
  97. ```python
  98. from content_core import ContentCoreConfig
  99. config = ContentCoreConfig(
  100. url_engine="firecrawl",
  101. document_engine="docling",
  102. audio_concurrency=5,
  103. )
  104. result = await content_core.extract_content(url="https://example.com", config=config)
  105. ```
  106. ## MCP Integration
  107. Content Core includes a Model Context Protocol (MCP) server for use with Claude Desktop and other MCP-compatible applications.
  108. <a href="https://glama.ai/mcp/servers/@lfnovo/content-core">
  109. <img width="380" height="200" src="https://glama.ai/mcp/servers/@lfnovo/content-core/badge" />
  110. </a>
  111. Add to your `claude_desktop_config.json`:
  112. ```json
  113. {
  114. "mcpServers": {
  115. "content-core": {
  116. "command": "uvx",
  117. "args": ["content-core", "mcp"],
  118. "env": {
  119. "OPENAI_API_KEY": "sk-..."
  120. }
  121. }
  122. }
  123. }
  124. ```
  125. The MCP server exposes two tools: `extract_content` and `summarize_content`. Both return plain text.
  126. For detailed setup, see the [MCP documentation](docs/mcp.md).
  127. ## Agent Skill (Claude Code & Codex)
  128. Content Core ships an [Agent Skill](skills/content-core/SKILL.md) that teaches AI agents how to use it for extracting content from external sources. This repository is also a plugin marketplace, so the skill installs natively in both harnesses.
  129. **Claude Code** — add the marketplace and install the plugin:
  130. ```
  131. /plugin marketplace add lfnovo/content-core
  132. /plugin install content-core@content-core
  133. ```
  134. **Codex** — the repository carries a Codex plugin manifest (`.codex-plugin/plugin.json`) and marketplace catalog (`.agents/plugins/marketplace.json`) pointing at the same skill.
  135. **Manual fallback** — copy the skill file directly into your project:
  136. ```bash
  137. curl -o .claude/skills/content-core/SKILL.md --create-dirs \
  138. https://raw.githubusercontent.com/lfnovo/content-core/main/skills/content-core/SKILL.md
  139. ```
  140. Once installed, the agent can use content-core to extract content from URLs, documents, and media files — either via CLI (`uvx content-core`) or MCP if configured.
  141. ## AI Providers
  142. Content Core uses [Esperanto](https://github.com/lfnovo/esperanto) to support multiple LLM and STT providers. Switch providers by changing the config — no code changes needed:
  143. ```bash
  144. # Use Anthropic for summarization
  145. content-core config set llm_provider anthropic
  146. content-core config set llm_model claude-sonnet-5
  147. # Use Groq for transcription
  148. content-core config set stt_provider groq
  149. content-core config set stt_model whisper-large-v3
  150. ```
  151. Supported providers include OpenAI, Anthropic, Google, Groq, DeepSeek, Ollama, and more. See the [Esperanto documentation](https://github.com/lfnovo/esperanto) for the full list.
  152. ## Configuration
  153. Content Core uses `ContentCoreConfig` powered by pydantic-settings. Settings are resolved in priority order: constructor args > env vars (`CCORE_*`) > config file (`~/.content-core/config.toml`) > defaults.
  154. ### Environment Variables
  155. | Variable | Description | Default |
  156. |----------|-------------|---------|
  157. | `CCORE_URL_ENGINE` | URL extraction engine (`auto`, `simple`, `firecrawl`, `jina`, `crawl4ai`) | `auto` |
  158. | `CCORE_DOCUMENT_ENGINE` | Document extraction engine (`auto`, `simple`, `docling`) — `docling` raises `ConfigurationError` if the extra is not installed; `auto` falls back silently | `auto` |
  159. | `CCORE_AUDIO_CONCURRENCY` | Concurrent audio transcriptions (1-10) | `3` |
  160. | `CCORE_AUDIO_SEGMENT_MINUTES` | Length of the segments long audio is split into; `0` sends the file whole | `10` |
  161. | `CRAWL4AI_API_URL` | Crawl4AI Docker API URL (omit for local browser mode) | - |
  162. | `CRAWL4AI_API_TOKEN` | Bearer token for the Crawl4AI Docker API (required by Crawl4AI >= 0.9.0) | - |
  163. | `FIRECRAWL_API_URL` | Custom Firecrawl API URL for self-hosted instances or Firecrawl-compatible backends (e.g. fastCRW) | - |
  164. | `CCORE_FIRECRAWL_PROXY` | Firecrawl proxy mode (`auto`, `basic`, `stealth`) | `auto` |
  165. | `CCORE_FIRECRAWL_WAIT_FOR` | Wait time in ms before extraction | `3000` |
  166. | `CCORE_LLM_PROVIDER` | LLM provider for summarization | - |
  167. | `CCORE_LLM_MODEL` | LLM model for summarization | - |
  168. | `CCORE_STT_PROVIDER` | Speech-to-text provider | - |
  169. | `CCORE_STT_MODEL` | Speech-to-text model | - |
  170. | `CCORE_STT_TIMEOUT` | Speech-to-text timeout in seconds | - |
  171. | `CCORE_YOUTUBE_LANGUAGES` | Preferred YouTube transcript languages | - |
  172. | `CCORE_YOUTUBE_COOKIES_FILE` | Path to a Netscape `cookies.txt` for YouTube transcripts (missing file raises `ConfigurationError`) | - |
  173. | `CCORE_YOUTUBE_PROXY` | Proxy URL for YouTube transcripts, e.g. `http://user:pass@host:port` | - |
  174. | `CCORE_DOCLING_VISION_URL` | OpenAI-compatible endpoint (vLLM, Ollama, LM Studio...) for Docling image descriptions; unset keeps Docling's inline model. Related: `CCORE_DOCLING_VISION_MODEL`, `_API_KEY`, `_PROMPT`, `_MAX_TOKENS`, `_TIMEOUT`, `_TOKEN_PARAM` ([details](docs/usage.md#image-descriptions-via-an-openai-compatible-endpoint)) | - |
  175. API keys for external services are set via their standard environment variables (e.g., `OPENAI_API_KEY`, `FIRECRAWL_API_KEY`, `JINA_API_KEY`).
  176. ### Proxy Configuration
  177. Content Core reads standard `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY` environment variables automatically. No additional configuration is needed.
  178. ### YouTube on Blocked Networks
  179. If YouTube extraction fails with `IpBlocked`/`RequestBlocked` on a flagged IP (cloud hosts, VPNs), either point `CCORE_YOUTUBE_COOKIES_FILE` at a `cookies.txt` exported from a logged-in browser (re-export it when the cookies expire), or set `CCORE_YOUTUBE_PROXY` to a **residential** proxy — datacenter proxies get blocked or CAPTCHA'd. See [docs/usage.md](docs/usage.md#youtube-on-blocked-networks).
  180. ## Optional Dependencies
  181. ```bash
  182. # Docling for advanced document parsing (PDF, DOCX, PPTX, XLSX)
  183. # Required by document_engine="docling", which raises ConfigurationError without it.
  184. # Use document_engine="auto" (default) or "simple" to proceed without Docling.
  185. pip install content-core[docling]
  186. # Crawl4AI for local browser-based URL extraction
  187. pip install content-core[crawl4ai]
  188. python -m playwright install --with-deps
  189. # LangChain tool wrappers
  190. pip install content-core[langchain]
  191. # All optional features
  192. pip install content-core[docling,crawl4ai,langchain]
  193. ```
  194. ### Using with LangChain
  195. When installed with the `langchain` extra, Content Core provides LangChain-compatible tool wrappers:
  196. ```python
  197. from content_core.tools import extract_content_tool, summarize_content_tool
  198. tools = [extract_content_tool, summarize_content_tool]
  199. ```
  200. ## Documentation
  201. - [Usage Guide](docs/usage.md) -- Python API details, configuration, and examples
  202. - [Processors](docs/processors.md) -- How content extraction works for each format
  203. - [MCP Server](docs/mcp.md) -- Claude Desktop and MCP integration
  204. ## Development
  205. ```bash
  206. git clone https://github.com/lfnovo/content-core
  207. cd content-core
  208. uv sync --group dev
  209. # Run tests
  210. make test
  211. # Lint
  212. make ruff
  213. ```
  214. ## License
  215. This project is licensed under the [MIT License](LICENSE).
  216. ## Contributing
  217. Contributions are welcome! Please see our [Contributing Guide](CONTRIBUTING.md) for details.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.