doctor
Doctor is a tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/sisig-ai--doctor)HTML
<a href="/listings/sisig-ai--doctor"><img src="/api/listings/sisig-ai--doctor/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
⚙️ Prerequisites
- Docker and Docker Compose - Python 3.10+ - uv (Python package manager) - OpenAI API key
📦 Installation
1. Clone this repository 2. Set up environment variables:
export OPENAI_API_KEY=your-openai-key
3. Run the stack:
docker compose up
---
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 214 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <div align="center">
- <picture>
- <img alt="Doctor Logo" src="doctor.png" height="30%" width="30%">
- </picture>
- <br>
- <h2>🩺 Doctor</h2>
- [](https://github.com/sisig-ai/doctor)
- [](LICENSE.md)
- [](https://github.com/sisig-ai/doctor/actions/workflows/pytest.yml)
- [](https://codecov.io/gh/sisig-ai/doctor)
- A tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents for better and more up-to-date reasoning and code generation.
- </div>
- ---
- ### 🔍 Overview
- Doctor provides a complete stack for:
- - Crawling web pages using crawl4ai with hierarchy tracking
- - Chunking text with LangChain
- - Creating embeddings with OpenAI via litellm
- - Storing data in DuckDB with vector search support
- - Exposing search functionality via a FastAPI web service
- - Making these capabilities available to LLMs through an MCP server
- - Navigating crawled sites with hierarchical site maps
- ---
- ### 🏗️ Core Infrastructure
- #### 🗄️ DuckDB
- - Database for storing document data and embeddings with vector search capabilities
- - Managed by unified Database class
- #### 📨 Redis
- - Message broker for asynchronous task processing
- #### 🕸️ Crawl Worker
- - Processes crawl jobs
- - Chunks text
- - Creates embeddings
- #### 🌐 Web Server
- - FastAPI service exposing endpoints
- - Fetching, searching, and viewing data
- - Exposing the MCP server
- ---
- ### 💻 Setup
- #### ⚙️ Prerequisites
- - Docker and Docker Compose
- - Python 3.10+
- - uv (Python package manager)
- - OpenAI API key
- #### 📦 Installation
- 1. Clone this repository
- 2. Set up environment variables:
- ```
- export OPENAI_API_KEY=your-openai-key
- ```
- 3. Run the stack:
- ```
- docker compose up
- ```
- ---
- ### 👁 Usage
- 1. Go to http://localhost:9111/docs to see the OpenAPI docs
- 2. Look for the `/fetch_url` endpoint and start a crawl job by providing a URL
- 3. Use `/job_progress` to see the current job status
- 4. Configure your editor to use `http://localhost:9111/mcp` as an MCP server
- ---
- ### ☁️ Web API
- #### Core Endpoints
- - `POST /fetch_url`: Start crawling a URL
- - `GET /search_docs`: Search indexed documents
- - `GET /job_progress`: Check crawl job progress
- - `GET /list_doc_pages`: List indexed pages
- - `GET /get_doc_page`: Get full text of a page
- #### Site Map Feature
- The Maps feature provides a hierarchical view of crawled websites, making it easy to navigate and explore the structure of indexed sites.
- **Endpoints:**
- - `GET /map`: View an index of all crawled sites
- - `GET /map/site/{root_page_id}`: View the hierarchical tree structure of a specific site
- - `GET /map/page/{page_id}`: View a specific page with navigation (parent, siblings, children)
- - `GET /map/page/{page_id}/raw`: Get the raw markdown content of a page
- **Features:**
- - **Hierarchical Navigation**: Pages maintain parent-child relationships, allowing you to navigate through the site structure
- - **Domain Grouping**: Pages from the same domain crawled individually are automatically grouped together
- - **Automatic Title Extraction**: Page titles are extracted from HTML or markdown content
- - **Breadcrumb Navigation**: Easy navigation with breadcrumbs showing the path from root to current page
- - **Sibling Navigation**: Quick access to pages at the same level in the hierarchy
- - **Legacy Page Support**: Pages crawled before hierarchy tracking are grouped by domain for easy access
- - **No JavaScript Required**: All navigation works with pure HTML and CSS for maximum compatibility
- **Usage Example:**
- 1. Crawl a website using the `/fetch_url` endpoint
- 2. Visit `/map` to see all crawled sites
- 3. Click on a site to view its hierarchical structure
- 4. Navigate through pages using the provided links
- ---
- ### 🔧 MCP Integration
- Ensure that your Docker Compose stack is up, and then add to your Cursor or VSCode MCP Servers configuration:
- ```json
- "doctor": {
- "type": "sse",
- "url": "http://localhost:9111/mcp"
- }
- ```
- ---
- ### 🧪 Testing
- #### Running Tests
- To run all tests:
- ```bash
- # Run all tests with coverage report
- pytest
- ```
- To run specific test categories:
- ```bash
- # Run only unit tests
- pytest -m unit
- # Run only async tests
- pytest -m async_test
- # Run tests for a specific component
- pytest tests/lib/test_crawler.py
- ```
- #### Test Coverage
- The project is configured to generate coverage reports automatically:
- ```bash
- # Run tests with detailed coverage report
- pytest --cov=src --cov-report=term-missing
- ```
- #### Test Structure
- - `tests/conftest.py`: Common fixtures for all tests
- - `tests/lib/`: Tests for library components
- - `test_crawler.py`: Tests for the crawler module
- - `test_crawler_enhanced.py`: Tests for enhanced crawler with hierarchy tracking
- - `test_chunker.py`: Tests for the chunker module
- - `test_embedder.py`: Tests for the embedder module
- - `test_database.py`: Tests for the unified Database class
- - `test_database_hierarchy.py`: Tests for database hierarchy operations
- - `tests/common/`: Tests for common modules
- - `tests/services/`: Tests for service layer
- - `test_map_service.py`: Tests for the map service
- - `tests/api/`: Tests for API endpoints
- - `test_map_api.py`: Tests for map API endpoints
- - `tests/integration/`: Integration tests
- - `test_processor_enhanced.py`: Tests for enhanced processor with hierarchy
- ---
- ### 🐞 Code Quality
- #### Pre-commit Hooks
- The project is configured with pre-commit hooks that run automatically before each commit:
- - `ruff check --fix`: Lints code and automatically fixes issues
- - `ruff format`: Formats code according to project style
- - Trailing whitespace removal
- - End-of-file fixing
- - YAML validation
- - Large file checks
- #### Setup Pre-commit
- To set up pre-commit hooks:
- ```bash
- # Install pre-commit
- uv pip install pre-commit
- # Install the git hooks
- pre-commit install
- ```
- #### Running Pre-commit Manually
- You can run the pre-commit hooks manually on all files:
- ```bash
- # Run all pre-commit hooks
- pre-commit run --all-files
- ```
- Or on staged files only:
- ```bash
- # Run on staged files
- pre-commit run
- ```
- ---
- ### ⚖️ License
- This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.