Skip to content
Back to search

doctor

Doctor is a tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/sisig-ai--doctor/badge.svg)](/listings/sisig-ai--doctor)

HTML

<a href="/listings/sisig-ai--doctor"><img src="/api/listings/sisig-ai--doctor/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

⚙️ Prerequisites

- Docker and Docker Compose - Python 3.10+ - uv (Python package manager) - OpenAI API key

📦 Installation

1. Clone this repository 2. Set up environment variables:

   export OPENAI_API_KEY=your-openai-key

3. Run the stack:

   docker compose up

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 214 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. <picture>
  3. <img alt="Doctor Logo" src="doctor.png" height="30%" width="30%">
  4. </picture>
  5. <br>
  6. <h2>🩺 Doctor</h2>
  7. [![Python Version](https://img.shields.io/badge/python-%3E=3.12-3776ab?style=flat&labelColor=333333&logo=python&logoColor=white)](https://github.com/sisig-ai/doctor)
  8. [![License](https://img.shields.io/badge/license-MIT-00acc1?style=flat&labelColor=333333&logo=open-source-initiative&logoColor=white)](LICENSE.md)
  9. [![Python Tests](https://github.com/sisig-ai/doctor/actions/workflows/pytest.yml/badge.svg)](https://github.com/sisig-ai/doctor/actions/workflows/pytest.yml)
  10. [![codecov](https://codecov.io/gh/sisig-ai/doctor/branch/main/graph/badge.svg)](https://codecov.io/gh/sisig-ai/doctor)
  11. A tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents for better and more up-to-date reasoning and code generation.
  12. </div>
  13. ---
  14. ### 🔍 Overview
  15. Doctor provides a complete stack for:
  16. - Crawling web pages using crawl4ai with hierarchy tracking
  17. - Chunking text with LangChain
  18. - Creating embeddings with OpenAI via litellm
  19. - Storing data in DuckDB with vector search support
  20. - Exposing search functionality via a FastAPI web service
  21. - Making these capabilities available to LLMs through an MCP server
  22. - Navigating crawled sites with hierarchical site maps
  23. ---
  24. ### 🏗️ Core Infrastructure
  25. #### 🗄️ DuckDB
  26. - Database for storing document data and embeddings with vector search capabilities
  27. - Managed by unified Database class
  28. #### 📨 Redis
  29. - Message broker for asynchronous task processing
  30. #### 🕸️ Crawl Worker
  31. - Processes crawl jobs
  32. - Chunks text
  33. - Creates embeddings
  34. #### 🌐 Web Server
  35. - FastAPI service exposing endpoints
  36. - Fetching, searching, and viewing data
  37. - Exposing the MCP server
  38. ---
  39. ### 💻 Setup
  40. #### ⚙️ Prerequisites
  41. - Docker and Docker Compose
  42. - Python 3.10+
  43. - uv (Python package manager)
  44. - OpenAI API key
  45. #### 📦 Installation
  46. 1. Clone this repository
  47. 2. Set up environment variables:
  48. ```
  49. export OPENAI_API_KEY=your-openai-key
  50. ```
  51. 3. Run the stack:
  52. ```
  53. docker compose up
  54. ```
  55. ---
  56. ### 👁 Usage
  57. 1. Go to http://localhost:9111/docs to see the OpenAPI docs
  58. 2. Look for the `/fetch_url` endpoint and start a crawl job by providing a URL
  59. 3. Use `/job_progress` to see the current job status
  60. 4. Configure your editor to use `http://localhost:9111/mcp` as an MCP server
  61. ---
  62. ### ☁️ Web API
  63. #### Core Endpoints
  64. - `POST /fetch_url`: Start crawling a URL
  65. - `GET /search_docs`: Search indexed documents
  66. - `GET /job_progress`: Check crawl job progress
  67. - `GET /list_doc_pages`: List indexed pages
  68. - `GET /get_doc_page`: Get full text of a page
  69. #### Site Map Feature
  70. The Maps feature provides a hierarchical view of crawled websites, making it easy to navigate and explore the structure of indexed sites.
  71. **Endpoints:**
  72. - `GET /map`: View an index of all crawled sites
  73. - `GET /map/site/{root_page_id}`: View the hierarchical tree structure of a specific site
  74. - `GET /map/page/{page_id}`: View a specific page with navigation (parent, siblings, children)
  75. - `GET /map/page/{page_id}/raw`: Get the raw markdown content of a page
  76. **Features:**
  77. - **Hierarchical Navigation**: Pages maintain parent-child relationships, allowing you to navigate through the site structure
  78. - **Domain Grouping**: Pages from the same domain crawled individually are automatically grouped together
  79. - **Automatic Title Extraction**: Page titles are extracted from HTML or markdown content
  80. - **Breadcrumb Navigation**: Easy navigation with breadcrumbs showing the path from root to current page
  81. - **Sibling Navigation**: Quick access to pages at the same level in the hierarchy
  82. - **Legacy Page Support**: Pages crawled before hierarchy tracking are grouped by domain for easy access
  83. - **No JavaScript Required**: All navigation works with pure HTML and CSS for maximum compatibility
  84. **Usage Example:**
  85. 1. Crawl a website using the `/fetch_url` endpoint
  86. 2. Visit `/map` to see all crawled sites
  87. 3. Click on a site to view its hierarchical structure
  88. 4. Navigate through pages using the provided links
  89. ---
  90. ### 🔧 MCP Integration
  91. Ensure that your Docker Compose stack is up, and then add to your Cursor or VSCode MCP Servers configuration:
  92. ```json
  93. "doctor": {
  94. "type": "sse",
  95. "url": "http://localhost:9111/mcp"
  96. }
  97. ```
  98. ---
  99. ### 🧪 Testing
  100. #### Running Tests
  101. To run all tests:
  102. ```bash
  103. # Run all tests with coverage report
  104. pytest
  105. ```
  106. To run specific test categories:
  107. ```bash
  108. # Run only unit tests
  109. pytest -m unit
  110. # Run only async tests
  111. pytest -m async_test
  112. # Run tests for a specific component
  113. pytest tests/lib/test_crawler.py
  114. ```
  115. #### Test Coverage
  116. The project is configured to generate coverage reports automatically:
  117. ```bash
  118. # Run tests with detailed coverage report
  119. pytest --cov=src --cov-report=term-missing
  120. ```
  121. #### Test Structure
  122. - `tests/conftest.py`: Common fixtures for all tests
  123. - `tests/lib/`: Tests for library components
  124. - `test_crawler.py`: Tests for the crawler module
  125. - `test_crawler_enhanced.py`: Tests for enhanced crawler with hierarchy tracking
  126. - `test_chunker.py`: Tests for the chunker module
  127. - `test_embedder.py`: Tests for the embedder module
  128. - `test_database.py`: Tests for the unified Database class
  129. - `test_database_hierarchy.py`: Tests for database hierarchy operations
  130. - `tests/common/`: Tests for common modules
  131. - `tests/services/`: Tests for service layer
  132. - `test_map_service.py`: Tests for the map service
  133. - `tests/api/`: Tests for API endpoints
  134. - `test_map_api.py`: Tests for map API endpoints
  135. - `tests/integration/`: Integration tests
  136. - `test_processor_enhanced.py`: Tests for enhanced processor with hierarchy
  137. ---
  138. ### 🐞 Code Quality
  139. #### Pre-commit Hooks
  140. The project is configured with pre-commit hooks that run automatically before each commit:
  141. - `ruff check --fix`: Lints code and automatically fixes issues
  142. - `ruff format`: Formats code according to project style
  143. - Trailing whitespace removal
  144. - End-of-file fixing
  145. - YAML validation
  146. - Large file checks
  147. #### Setup Pre-commit
  148. To set up pre-commit hooks:
  149. ```bash
  150. # Install pre-commit
  151. uv pip install pre-commit
  152. # Install the git hooks
  153. pre-commit install
  154. ```
  155. #### Running Pre-commit Manually
  156. You can run the pre-commit hooks manually on all files:
  157. ```bash
  158. # Run all pre-commit hooks
  159. pre-commit run --all-files
  160. ```
  161. Or on staged files only:
  162. ```bash
  163. # Run on staged files
  164. pre-commit run
  165. ```
  166. ---
  167. ### ⚖️ License
  168. This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.