content-core
Extract what matters from any media source
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/lfnovo--content-core)HTML
<a href="/listings/lfnovo--content-core"><img src="/api/listings/lfnovo--content-core/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
pip install content-core
import content_core result = await content_core.extract_content(url="https://example.com") print(result.content)
Or with zero install:
uvx content-core extract "https://example.com"
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 18 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- {
- "name": "content-core",
- "owner": {
- "name": "Luis Novo",
- "email": "lfnovo@gmail.com"
- },
- "description": "Content Core — content extraction toolkit for AI agents",
- "version": "1.0.0",
- "plugins": [
- {
- "name": "content-core",
- "source": "./",
- "description": "Extract text content from URLs, documents, YouTube videos, Reddit posts, and audio/video files — via CLI (uvx) or MCP.",
- "category": "development",
- "keywords": ["extraction", "content", "pdf", "youtube", "transcription", "mcp"]
- }
- ]
- }
README.md · README · 311 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # Content Core
- [](https://opensource.org/licenses/MIT)
- [](https://badge.fury.io/py/content-core)
- [](https://pepy.tech/project/content-core)
- [](https://pepy.tech/project/content-core)
- [](https://github.com/lfnovo/content-core)
- [](https://github.com/lfnovo/content-core)
- [](https://github.com/lfnovo/content-core/issues)
- [](https://github.com/astral-sh/ruff)
- Extract, process, and summarize content from URLs, files, and text through a unified async Python API, CLI, or MCP server.
- ## Supported Formats
- | Category | Formats |
- |----------|---------|
- | Web | URLs, HTML pages, YouTube videos, Reddit posts |
- | Documents | PDF, DOCX, PPTX, XLSX, ODT, ODS, ODP, EPUB, HTML, Markdown, plain text |
- | Media | MP3, WAV, M4A, FLAC, OGG (audio); MP4, AVI, MOV, MKV (video) |
- ## Quick Start
- ```bash
- pip install content-core
- ```
- ```python
- import content_core
- result = await content_core.extract_content(url="https://example.com")
- print(result.content)
- ```
- Or with zero install:
- ```bash
- uvx content-core extract "https://example.com"
- ```
- ## CLI Usage
- Content Core provides a unified `content-core` command with subcommands for extraction, summarization, and MCP server.
- ### Extract
- ```bash
- # From a URL
- content-core extract "https://example.com"
- # From a file
- content-core extract document.pdf
- # With JSON output
- content-core extract document.pdf --format json
- # With a specific engine
- content-core extract "https://example.com" --engine firecrawl
- # From stdin
- echo "some text" | content-core extract
- ```
- ### Summarize
- ```bash
- # Summarize text
- content-core summarize "Long article text here..."
- # With context
- content-core summarize "Long text" --context "bullet points"
- # From stdin
- cat article.txt | content-core summarize --context "explain to a child"
- ```
- ### MCP Server
- ```bash
- content-core mcp
- ```
- ### Configuration
- ```bash
- # Set persistent config
- content-core config set llm_provider anthropic
- content-core config set llm_model claude-sonnet-5
- # List current config
- content-core config list
- # Delete a config value
- content-core config delete llm_provider
- ```
- Config is stored in `~/.content-core/config.toml`. Priority: command flags > env vars > config file > defaults.
- ### Zero-Install with uvx
- All commands work without installation using `uvx`:
- ```bash
- uvx content-core extract "https://example.com"
- uvx content-core summarize "text" --context "one sentence"
- uvx content-core mcp
- ```
- ## Python API
- ### Extraction
- ```python
- import content_core
- # From a URL
- result = await content_core.extract_content(url="https://example.com")
- # From a file
- result = await content_core.extract_content(file_path="document.pdf")
- # From text
- result = await content_core.extract_content(content="some text")
- # With engine override
- from content_core import ContentCoreConfig
- config = ContentCoreConfig(url_engine="firecrawl")
- result = await content_core.extract_content(url="https://example.com", config=config)
- ```
- ### Summarization
- ```python
- import content_core
- summary = await content_core.summarize("long article text", context="bullet points")
- ```
- ### Configuration
- ```python
- from content_core import ContentCoreConfig
- config = ContentCoreConfig(
- url_engine="firecrawl",
- document_engine="docling",
- audio_concurrency=5,
- )
- result = await content_core.extract_content(url="https://example.com", config=config)
- ```
- ## MCP Integration
- Content Core includes a Model Context Protocol (MCP) server for use with Claude Desktop and other MCP-compatible applications.
- <a href="https://glama.ai/mcp/servers/@lfnovo/content-core">
- <img width="380" height="200" src="https://glama.ai/mcp/servers/@lfnovo/content-core/badge" />
- </a>
- Add to your `claude_desktop_config.json`:
- ```json
- {
- "mcpServers": {
- "content-core": {
- "command": "uvx",
- "args": ["content-core", "mcp"],
- "env": {
- "OPENAI_API_KEY": "sk-..."
- }
- }
- }
- }
- ```
- The MCP server exposes two tools: `extract_content` and `summarize_content`. Both return plain text.
- For detailed setup, see the [MCP documentation](docs/mcp.md).
- ## Agent Skill (Claude Code & Codex)
- Content Core ships an [Agent Skill](skills/content-core/SKILL.md) that teaches AI agents how to use it for extracting content from external sources. This repository is also a plugin marketplace, so the skill installs natively in both harnesses.
- **Claude Code** — add the marketplace and install the plugin:
- ```
- /plugin marketplace add lfnovo/content-core
- /plugin install content-core@content-core
- ```
- **Codex** — the repository carries a Codex plugin manifest (`.codex-plugin/plugin.json`) and marketplace catalog (`.agents/plugins/marketplace.json`) pointing at the same skill.
- **Manual fallback** — copy the skill file directly into your project:
- ```bash
- curl -o .claude/skills/content-core/SKILL.md --create-dirs \
- https://raw.githubusercontent.com/lfnovo/content-core/main/skills/content-core/SKILL.md
- ```
- Once installed, the agent can use content-core to extract content from URLs, documents, and media files — either via CLI (`uvx content-core`) or MCP if configured.
- ## AI Providers
- Content Core uses [Esperanto](https://github.com/lfnovo/esperanto) to support multiple LLM and STT providers. Switch providers by changing the config — no code changes needed:
- ```bash
- # Use Anthropic for summarization
- content-core config set llm_provider anthropic
- content-core config set llm_model claude-sonnet-5
- # Use Groq for transcription
- content-core config set stt_provider groq
- content-core config set stt_model whisper-large-v3
- ```
- Supported providers include OpenAI, Anthropic, Google, Groq, DeepSeek, Ollama, and more. See the [Esperanto documentation](https://github.com/lfnovo/esperanto) for the full list.
- ## Configuration
- Content Core uses `ContentCoreConfig` powered by pydantic-settings. Settings are resolved in priority order: constructor args > env vars (`CCORE_*`) > config file (`~/.content-core/config.toml`) > defaults.
- ### Environment Variables
- | Variable | Description | Default |
- |----------|-------------|---------|
- | `CCORE_URL_ENGINE` | URL extraction engine (`auto`, `simple`, `firecrawl`, `jina`, `crawl4ai`) | `auto` |
- | `CCORE_DOCUMENT_ENGINE` | Document extraction engine (`auto`, `simple`, `docling`) — `docling` raises `ConfigurationError` if the extra is not installed; `auto` falls back silently | `auto` |
- | `CCORE_AUDIO_CONCURRENCY` | Concurrent audio transcriptions (1-10) | `3` |
- | `CCORE_AUDIO_SEGMENT_MINUTES` | Length of the segments long audio is split into; `0` sends the file whole | `10` |
- | `CRAWL4AI_API_URL` | Crawl4AI Docker API URL (omit for local browser mode) | - |
- | `CRAWL4AI_API_TOKEN` | Bearer token for the Crawl4AI Docker API (required by Crawl4AI >= 0.9.0) | - |
- | `FIRECRAWL_API_URL` | Custom Firecrawl API URL for self-hosted instances or Firecrawl-compatible backends (e.g. fastCRW) | - |
- | `CCORE_FIRECRAWL_PROXY` | Firecrawl proxy mode (`auto`, `basic`, `stealth`) | `auto` |
- | `CCORE_FIRECRAWL_WAIT_FOR` | Wait time in ms before extraction | `3000` |
- | `CCORE_LLM_PROVIDER` | LLM provider for summarization | - |
- | `CCORE_LLM_MODEL` | LLM model for summarization | - |
- | `CCORE_STT_PROVIDER` | Speech-to-text provider | - |
- | `CCORE_STT_MODEL` | Speech-to-text model | - |
- | `CCORE_STT_TIMEOUT` | Speech-to-text timeout in seconds | - |
- | `CCORE_YOUTUBE_LANGUAGES` | Preferred YouTube transcript languages | - |
- | `CCORE_YOUTUBE_COOKIES_FILE` | Path to a Netscape `cookies.txt` for YouTube transcripts (missing file raises `ConfigurationError`) | - |
- | `CCORE_YOUTUBE_PROXY` | Proxy URL for YouTube transcripts, e.g. `http://user:pass@host:port` | - |
- | `CCORE_DOCLING_VISION_URL` | OpenAI-compatible endpoint (vLLM, Ollama, LM Studio...) for Docling image descriptions; unset keeps Docling's inline model. Related: `CCORE_DOCLING_VISION_MODEL`, `_API_KEY`, `_PROMPT`, `_MAX_TOKENS`, `_TIMEOUT`, `_TOKEN_PARAM` ([details](docs/usage.md#image-descriptions-via-an-openai-compatible-endpoint)) | - |
- API keys for external services are set via their standard environment variables (e.g., `OPENAI_API_KEY`, `FIRECRAWL_API_KEY`, `JINA_API_KEY`).
- ### Proxy Configuration
- Content Core reads standard `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY` environment variables automatically. No additional configuration is needed.
- ### YouTube on Blocked Networks
- If YouTube extraction fails with `IpBlocked`/`RequestBlocked` on a flagged IP (cloud hosts, VPNs), either point `CCORE_YOUTUBE_COOKIES_FILE` at a `cookies.txt` exported from a logged-in browser (re-export it when the cookies expire), or set `CCORE_YOUTUBE_PROXY` to a **residential** proxy — datacenter proxies get blocked or CAPTCHA'd. See [docs/usage.md](docs/usage.md#youtube-on-blocked-networks).
- ## Optional Dependencies
- ```bash
- # Docling for advanced document parsing (PDF, DOCX, PPTX, XLSX)
- # Required by document_engine="docling", which raises ConfigurationError without it.
- # Use document_engine="auto" (default) or "simple" to proceed without Docling.
- pip install content-core[docling]
- # Crawl4AI for local browser-based URL extraction
- pip install content-core[crawl4ai]
- python -m playwright install --with-deps
- # LangChain tool wrappers
- pip install content-core[langchain]
- # All optional features
- pip install content-core[docling,crawl4ai,langchain]
- ```
- ### Using with LangChain
- When installed with the `langchain` extra, Content Core provides LangChain-compatible tool wrappers:
- ```python
- from content_core.tools import extract_content_tool, summarize_content_tool
- tools = [extract_content_tool, summarize_content_tool]
- ```
- ## Documentation
- - [Usage Guide](docs/usage.md) -- Python API details, configuration, and examples
- - [Processors](docs/processors.md) -- How content extraction works for each format
- - [MCP Server](docs/mcp.md) -- Claude Desktop and MCP integration
- ## Development
- ```bash
- git clone https://github.com/lfnovo/content-core
- cd content-core
- uv sync --group dev
- # Run tests
- make test
- # Lint
- make ruff
- ```
- ## License
- This project is licensed under the [MIT License](LICENSE).
- ## Contributing
- Contributions are welcome! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.