Skip to content
Back to search

flexible-graphrag

Python, LlamaIndex, LangChain, 15 Property Graph, 4 RDF , 10 Vector, OpenSearch, Elasticsearch DBs. 14 data sources, 10 auto sync: Alfresco, Nuxeo, etc. KG auto-building, Ontologies, LLMs, Docling, LlamaParse, LiteParse, GraphRAG, RAG, Hybrid Search, AI Chat. TypeScript React, Vue, Angular frontends, REST, MCP Server. Options: Langflow, CocoIndex

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/stevereiner/flexible-graphrag with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/stevereiner/flexible-graphrag'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/stevereiner--flexible-graphrag/badge.svg)](/listings/stevereiner--flexible-graphrag)

HTML

<a href="/listings/stevereiner--flexible-graphrag"><img src="/api/listings/stevereiner--flexible-graphrag/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

cd flexible-graphrag
uv pip install -e .

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 1,569 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Flexible GraphRAG
  2. [![PyPI - flexible-graphrag](https://img.shields.io/pypi/v/flexible-graphrag?label=flexible-graphrag&color=blue)](https://pypi.org/project/flexible-graphrag/)
  3. [![Downloads - flexible-graphrag](https://img.shields.io/pepy/dt/flexible-graphrag)](https://pepy.tech/project/flexible-graphrag)
  4. [![PyPI - flexible-graphrag-mcp](https://img.shields.io/pypi/v/flexible-graphrag-mcp?label=flexible-graphrag-mcp&color=blue)](https://pypi.org/project/flexible-graphrag-mcp/)
  5. [![Downloads - flexible-graphrag-mcp](https://img.shields.io/pepy/dt/flexible-graphrag-mcp)](https://pepy.tech/project/flexible-graphrag-mcp)
  6. [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)
  7. [![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13%20%7C%203.14-blue)](https://www.python.org/)
  8. [![React](https://img.shields.io/badge/React-18-61DAFB?logo=react&logoColor=white)](https://react.dev/)
  9. [![Angular](https://img.shields.io/badge/Angular-20-DD0031?logo=angular&logoColor=white)](https://angular.dev/)
  10. [![Vue](https://img.shields.io/badge/Vue-3-4FC08D?logo=vue.js&logoColor=white)](https://vuejs.org/)
  11. [![Docker Hub](https://img.shields.io/badge/Docker%20Hub-integratedsemantics-2496ED?logo=docker&logoColor=white)](https://hub.docker.com/u/integratedsemantics)
  12. [![Ask DeepWiki](https://img.shields.io/badge/Ask-DeepWiki-20B2AA)](https://deepwiki.com/stevereiner/flexible-graphrag)
  13. [![Documentation](https://img.shields.io/badge/Documentation-GitHub%20Pages-blue)](https://stevereiner.github.io/flexible-graphrag/)
  14. [![Blog](https://img.shields.io/badge/Blog-integratedsemantics.org-orange)](https://integratedsemantics.org)
  15. [![Supports Hyland OpenArch](https://img.shields.io/badge/Supports-Hyland%20OpenArch-0b5394.svg)](https://github.com/Hyland/OpenArch)
  16. <p align="center">
  17. <a href="./screen-shots/langflow/langflow-sharepoint-graphdb-neo4j.png">
  18. <img src="./screen-shots/langflow/langflow-sharepoint-graphdb-neo4j.png" alt="Flexible GraphRAG in Langflow pipeline mode: a SharePoint auto-sync source using Microsoft Graph delta queries, a PDF parsed by LiteParse, and one ingest populating Qdrant vector, Elasticsearch search, Neo4j property graph, and Graphwise GraphDB RDF at the same time, with the processing tab showing initial sync status and AI chat answering questions from an ontology-mode knowledge graph" width="900">
  19. </a>
  20. </p>
  21. <p align="center"><em>Langflow pipeline mode — a SharePoint auto-sync data source (MS Graph delta query) ingesting and targeting Qdrant vector, Elasticsearch, Neo4j, and Graphwise Ontotext GraphDB, with ontology-mode, AI chat drawing context from all four.</em></p>
  22. <p align="center">
  23. <a href="./images/flexible-graphrag-v0.8.0-architecture.png">
  24. <img src="./images/flexible-graphrag-v0.8.0-architecture.png" alt="Flexible GraphRAG v0.8.0 architecture: default, CocoIndex, and Langflow ingest pipelines sharing the same data sources, database targets, UI, and REST / MCP APIs" width="900">
  25. </a>
  26. </p>
  27. <p align="center"><em>Can use 1 of 3 ingest pipelines with same configured data sources and database targets, with same UI and same REST / MCP APIs</em></p>
  28. **Flexible GraphRAG** is an open source AI context platform supporting a document processing pipeline (Docling, LlamaParse, or LiteParse), knowledge graph auto-building, ontologies, schemas, many LLM providers, GraphRAG and RAG, hybrid semantic search (fulltext, vector, property graph, RDF/SPARQL), AI query, and AI chat. The backend is **Python** with **LlamaIndex** and **LangChain** as peer frameworks. **LlamaIndex** is the default for each pipeline stage; **LangChain** can be selected per stage in environment configuration. The API is a REST **FastAPI** service. **Angular**, **React**, and **Vue** TypeScript frontends and an **MCP** server are included. The stack supports 14 data sources (10 with incremental auto-sync), 15 property graph databases, 4 RDF triple stores (Apache Jena Fuseki, Ontotext GraphDB, Oxigraph, Amazon Neptune RDF), 10 vector databases, OpenSearch / Elasticsearch / BM25 search, Alfresco, and Nuxeo. Databases and dashboards can be enabled with the provided Docker Compose layout. Optionally, the ingest pipeline, hybrid search, and AI query can run through customizable **Langflow** visual flows (12 custom Langflow components). As a further option, ingest can run on a **CocoIndex** (Rust engine) pipeline that reuses the same sources, targets, parsers and KG extractors, adding step-level memoization and automatic delete reconciliation. And optionally, **[Hyland OpenArch](https://github.com/Hyland/OpenArch)**, an AI-ready content repository for documents with their extracted text and vector embeddings, can be the search + vector store and the store for data sources and document state.
  29. **New 11/16/26 — v0.11.0 release:** an early version of a **Desktop App** (Tauri/Rust) around the React, Vue or Angular UI with the 5 Flexible GraphRAG tabs, that adds browsing, ingesting, searching and graph viewing (5th tab) of **local files and folders (can auto-sync)** along with all the other data sources ([desktop app](flexible-graphrag-ui/desktop/README.md)); in the desktop and web UIs, **Browse...** shows a file/folder browser in the **Alfresco, Nuxeo, CMIS, SharePoint, OneDrive, Box, S3, Azure Blob, GCS, Google Drive** data source forms; an optional lighter Docker-free config (e.g. for the desktop app): **SQLite** as the auto-sync incremental update state store (`INCREMENTAL_STATE_STORE=sqlite`) along with embedded stores such as LanceDB, BM25, LadybugDB; **Federated Flexible GraphRAG Backends**: search and AI query/chat that fuse results e.g. from a personal backend, plus a team one, plus a company one (`REMOTE_FG_BACKENDS`).
  30. **New 11/10/26 — v0.10.0 release:** a **Graph tab** in the React, Vue and Angular UIs showing the entities and relationships extracted from an ingest, search results, or AI answer sources, and from a document or folder in KG Spaces. Also an Ollama fix that keeps models on the GPU (new `OLLAMA_CONTEXT_WINDOW`, default 8192) (see [Ollama configuration](docs/LLM/OLLAMA-CONFIGURATION.md)).
  31. **New 11/3/26 — v0.9.1 release:** **OpenArch in the Flexible GraphRAG Docker Compose stack** (`docker/includes/openarch.yaml`, not just in KG Spaces Deployment); **faster KG extraction** by raising `KG_EXTRACTION_WORKERS` (default 4; 8-16 with cloud LLMs); **security fixes** (see [SECURITY.md](SECURITY.md) for reporting and the deployment model): REST / MCP filesystem ingest only reads paths under `uploads/`, `SOURCE_PATHS` or the new `INGEST_ALLOWED_PATHS`; query injection fixed in the Memgraph, Apache AGE, ArcadeDB, Ladybug, Gremlin / Cosmos DB and NebulaGraph adapters and the RDF converter; and graph Q&A now runs only read queries, so a question can't get the LLM to change or delete graph data. Also preparatory work for **Python 3.15** and free-threaded 3.15t (see [Python 3.15](docs/ADVANCED/PYTHON-3.15.md)).
  32. **New 10/23/26 — v0.9.0 release:** Optional **[Hyland OpenArch](https://github.com/Hyland/OpenArch)** as the search + vector store (`ENABLE_OPENARCH`) and as the store for data sources and document state, with no PostgreSQL (`INCREMENTAL_STATE_STORE=openarch`) (see [usage](docs/DATABASES/OPENARCH/OPENARCH.md)); separate **Graph** and **RDF** choices per row and ingest; a per-user Jobs list; a title bar with Dark / Light theme in the Angular UI library; and **[KG Spaces for Nuxeo](https://github.com/stevereiner/kg-spaces-nuxeo)** (Add to KG Spaces and Ask from Nuxeo Web UI).
  33. **New 10/19/26 — v0.8.6 release:** **Sign-in and permission-aware search for every repository and cloud source**: Alfresco, Nuxeo, CMIS, SharePoint / OneDrive, Box, Google Drive / Cloud Storage, Azure Blob and Amazon S3, with one sign-in pill in the header whose drop-down signs you in and out of each data source and server, and a Sources picker, in the React, Vue and Angular UIs and in KG Spaces; S3 and Azure Blob results open the file through short-lived signed links; per-row Processing columns for every data source and a Deep (subfolders) checkbox; MCP permission levels and browser sign-in. See the [Sign-In Guide](docs/DATA-SOURCES/SIGN-IN-GUIDE.md).
  34. **New 10/17/26 — v0.8.5 release:** **Sign in to Alfresco** in the React, Vue and Angular UIs: search and AI chat answer only from Alfresco documents you may read (with a server picker in the sign-in dialog when several Alfresco servers are in the stores), and the Alfresco source form can ingest using the sign-in. The MCP server's search and query tools filter by Alfresco permissions too (pass the user's ticket or credentials). Data-source forms no longer pre-fill credentials or get autofilled by the browser, and KG Spaces can ingest from another Alfresco server (in Other Sources, after adding `alfresco` to its `enabledSources` config; KG Spaces search still covers only the Alfresco server ACA is signed in to).
  35. **New 10/10/26 — v0.8.4 release:** Hybrid search results (graph and RDF too) and AI query / AI chat answers now **name the documents they came from, as links**: Alfresco documents open in the **Alfresco Content App viewer** over their folder (or Share), Nuxeo in its Web UI, web pages and Wikipedia at their URL, and YouTube at the transcript segment's time. In KG Spaces, closing ACA's viewer returns to your search results and chat. KG Spaces hybrid search returns full results again, and ArcadeDB moves to 26.10.1.
  36. **New 10/8/26 — v0.8.3 release:** The Processing tab now has per-row Search+Vector, Graphs and Auto Sync checkbox columns (Alfresco, Nuxeo and file uploads initially), so in **addition to ingest, can toggle having vector+search, graphs, and auto sync, row by row**, **Jobs** sub-tab and **Run in background**. KG Spaces adds **Ask KG Spaces about this document / folder** (then ask in AI chat). Hybrid search, AI query, and AI chat, return only from documents the signed-in Alfresco user may read.
  37. **New 10/2/26 — v0.8.2 release:** The Flexible GraphRAG Angular frontend's tab components are now a shared library project, published on npm as [`@flexible-graphrag/angular-ui`](https://www.npmjs.com/package/@flexible-graphrag/angular-ui), and also used by the new [KG Spaces for Alfresco Content App](https://github.com/stevereiner/kg-spaces-aca) (kg-spaces-aca), redone as a real Apache 2.0 ACA extension. Besides using multi-select of Alfresco files and folders (any mix) from ACA's regular file list views, wherever they are, it now has the data sources tab (renamed there **OTHER SOURCES**) for choosing additional data sources supported by Flexible GraphRAG. AI context can be kept up to date with **auto change sync** from Alfresco, Nuxeo, SharePoint, Box, Amazon S3, Azure Blob Storage, Google Cloud Storage, Google Drive and OneDrive, and ingested without auto sync from File Upload, CMIS, Web Pages, Wikipedia and YouTube.
  38. **New 9/16/26 — v0.8.1 release:** Alfresco Docker stack upgraded to Community 26.2.0, support for Alfresco OpenSearch and Elasticsearch (shared with the Flexible GraphRAG app or dedicated), and a file upload data source path-traversal fix.
  39. **New 8/18/26 — v0.8.0 release:** Optional **CocoIndex integration** — Rust-backed [CocoIndex](https://github.com/cocoindex-io/cocoindex) pipeline mixed with Flexible GraphRAG sources (incl. detectors), functions, and targets (more PG/vector/RDF/search); same UI/REST/MCP. Standalone `app.py` also supported. Now with **custom KG extractors** (bring your own, or fall back to the built-in one per document) and **entity resolution**. New **meeting-notes example** ([`examples/cocoindex/meeting_notes_graph_any/`](examples/cocoindex/meeting_notes_graph_any/README.md)) — a CocoIndex example ported to run against any configured graph store and source. See [CocoIndex Integration](#cocoindex-integration).
  40. **New 8/8/26 — v0.7.2 release:** **Nuxeo** added as a data source — all 3 UIs (React/Vue/Angular) plus REST/MCP, with basic / token / OAuth2 auth and real-time incremental sync via the Nuxeo audit event stream (Kafka). **Alfresco OAuth2 and ticket** authentication added across the source, all 3 UIs, initial ingest, and real-time sync. **Alfresco Community 26.1** Docker upgrade. The **MCP server** gained optional OAuth2 on its transport (bearer token via your IdP) and moved to **FastMCP 3**. Requires `python-alfresco-api >= 1.2.1`.
  41. **New 7/20/26 — v0.7.1 release:** Document processing now supports **LiteParse** in addition to the previous **Docling** and **LlamaParse** support. Langflow integration ships with fixes and an optional **Langflow Docker** image bundling the 12 "Flexible" components. **MS Graph delta query** support was added for more efficient incremental updating with SharePoint and OneDrive data sources.
  42. **New 7/5/26:** Optional **Langflow visual flows** — the app can run its ingest pipeline, hybrid search, and AI query through customizable [Langflow](https://www.langflow.org/) flows (12 custom Flexible GraphRAG components), using your existing `.env` config. See [Langflow Integration](docs/GETTING-STARTED/LANGFLOW-INTEGRATION.md).
  43. **New 5/6/26:** 15 property graph databases total: 8 supported on both LlamaIndex and LangChain, 1 LI-only (Google Cloud Spanner Graph), 6 LC-only (ArangoDB, Apache AGE, Azure Cosmos DB for Gremlin, Apache HugeGraph, SurrealDB, TigerGraph). AWS Neptune RDF/SPARQL added. All 10 vector databases, all 3 search engines, and all LLM/embedding providers work with both LlamaIndex and LangChain. Every pipeline stage (chunking, KG extraction, graph write, vector write, search write, and retrieval fusion) can be configured independently. (Data source reading is LlamaIndex only; RDF stores use framework-independent adapters with LangChain Text-to-SPARQL retrieval.)
  44. **New:** Flexible GraphRAG now supports RDF-based ontologies for both property graph databases and RDF triple store databases (Graphwise Ontotext GraphDB, Fuseki, and Oxigraph). Document ingestion with KG extraction, auto incremental data source change detection, and UI search (hybrid search, AI query, and AI chat) are all supported with both database types.
  45. **New:** Flexible GraphRAG supports **automatic incremental updates** (Optional) from most data sources, keeping your Vector, Search and Graph databases synchronized in real-time or near real-time.
  46. **New:** [KG Spaces Integration of Flexible GraphRAG in Alfresco ACA Client](https://github.com/stevereiner/kg-spaces-aca)
  47. **New in v0.6.0:** Version 0.6.0 broadened framework and database choice: **LangChain** is a full peer to **LlamaIndex** (per-stage env pickers for chunking, vector, search, property graph, KG extraction, fusion). **15** property graph backends: **8** on both frameworks, **Google Cloud Spanner** (LlamaIndex-only), **6** LangChain-only (ArangoDB, Apache AGE, Azure Cosmos DB for Gremlin, HugeGraph, SurrealDB, TigerGraph). **RDF** includes **Apache Jena Fuseki**, **Ontotext GraphDB**, **Oxigraph**, and **Amazon Neptune RDF**. Incremental delete, LangChain adapters, and cleanup paths were extended across stores.
  48. ## Features
  49. - **Hybrid Search**: Configurable hybrid search combining vector search, full-text search, property-graph GraphRAG, and SPARQL against RDF stores.
  50. - **Knowledge Graph GraphRAG**: Extracts entities and relationships from documents to build graphs in property graph databases and RDF stores. Optional schemas and ontologies guide extraction or act as a starting point for the LLM to extend.
  51. - **RDF/Ontology Support**: Load OWL/RDFS ontologies to guide KG extraction into any property graph or RDF store; SPARQL 1.1 queries; RDF 1.2 triple annotations; full UI pipeline (ingest, hybrid search, AI query/chat, incremental auto-sync). See [Ontology and RDF Support](#ontology-and-rdf-support) below.
  52. - **15 Property Graph Databases**: 9 on both LI+LC (Neo4j, ArcadeDB, FalkorDB, Ladybug, Memgraph, NebulaGraph, Amazon Neptune, Neptune Analytics, Google Cloud Spanner), 6 LC-only (ArangoDB, Apache AGE, Cosmos Gremlin, HugeGraph, SurrealDB, TigerGraph) — with KG extraction, hybrid search, and AI query/chat
  53. - **4 RDF Triple Stores**: Apache Jena Fuseki, Ontotext GraphDB, Oxigraph, Amazon Neptune RDF.
  54. - **10 Vector Databases**: Qdrant, Elasticsearch, OpenSearch, Neo4j, Chroma, Milvus, Weaviate, Pinecone, PostgreSQL pgvector, LanceDB — for semantic similarity search
  55. - **3 Search Databases**: Elasticsearch, OpenSearch, BM25 (built-in) — for full-text search and hybrid ranking
  56. - **Hyland OpenArch (optional)**: [Hyland OpenArch](https://github.com/Hyland/OpenArch), an AI-ready content repository for documents with their extracted text and vector embeddings, as a combined search + vector store (`ENABLE_OPENARCH=true`) and/or as the auto-sync state store in place of PostgreSQL (`INCREMENTAL_STATE_STORE=openarch`), with per-folder / per-document scope rules — see [docs/DATABASES/OPENARCH/OPENARCH.md](docs/DATABASES/OPENARCH/OPENARCH.md)
  57. - **LLM providers (KG extraction & chat)**: Ollama, OpenAI, Azure OpenAI, Google Gemini, Anthropic Claude, Google Vertex AI, Amazon Bedrock, Groq, Fireworks AI, OpenAI-compatible endpoints (`openai_like`), OpenRouter, LiteLLM proxy, and vLLM — configurable via `LLM_PROVIDER`; see [Supported LLM Providers](#supported-llm-providers)
  58. - **Embedding providers**: OpenAI, Ollama, Azure OpenAI, Google GenAI, Vertex AI, Bedrock, Fireworks, OpenAI-like (`EMBEDDING_KIND=openai_like`), and LiteLLM — see [LLM Configuration](#llm-configuration)
  59. - **Dual-framework pipeline**: **LlamaIndex** and **LangChain** are first-class choices for chunking, vector and search adapters, property graphs, KG extraction, RDF text-to-SPARQL retrieval, and hybrid fusion—each stage can be set independently (**LlamaIndex** defaults). See [Framework Configuration](#framework-configuration).
  60. - **Document Processing**: Docling (default), LlamaParse (cloud API), or LiteParse (local, lightweight) document parsing.
  61. - **14 Data Sources**: 10 with incremental auto sync (Alfresco, Nuxeo, SharePoint, OneDrive, Box, Google Drive, Amazon S3, Google Cloud Storage, Azure Blob Storage, Filesystem) and 4 without (CMIS, Web Pages, Wikipedia, YouTube).
  62. - **Sign-In and Permission-Aware Search**: One sign-in pill in the header whose drop-down signs users in and out of each repository and cloud service; hybrid search, AI query and AI chat then use only the documents that user may read, and a **Sources** picker limits a question to chosen sources.
  63. - **Observability**: Built-in OpenTelemetry instrumentation with automatic LlamaIndex tracing, Prometheus metrics, Jaeger traces, and Grafana dashboards for production monitoring
  64. - **FastAPI Server with REST API**: Python based FastAPI server with REST APIs for document ingesting, hybrid search, AI query, and AI chat.
  65. - **MCP Server**: MCP server providing Claude Desktop and other MCP clients with tools for document/text ingesting (all 14 data sources with 10 supporting incremental auto sync), hybrid search, and AI query. Uses FastAPI backend REST APIs.
  66. - **UI Clients**: Angular, React, and Vue UI clients support choosing the data source (filesystem, Alfresco, CMIS, etc.), ingesting documents, performing hybrid searches, AI queries, and AI chat. The UI clients use the REST APIs of the FastAPI backend.
  67. - **Docker Deployment Flexibility**: Supports both standalone and Docker deployment modes. Docker infrastructure provides modular database selection via docker-compose includes - vector, graph, search engines, and Alfresco can be included or excluded with a single comment. Choose between hybrid deployment (databases in Docker, backend and UIs standalone) or full containerization.
  68. - **Langflow Visual Flows (optional)**: Run the ingest pipeline, hybrid search, and AI query through customizable [Langflow](https://www.langflow.org/) flows built from 12 custom Flexible GraphRAG components — the same backend machinery (all database, LLM, and framework `.env` config applies), orchestrated visually. See [Langflow Integration](docs/GETTING-STARTED/LANGFLOW-INTEGRATION.md).
  69. - **CocoIndex Integration (optional)**: Optional Rust-backed [CocoIndex](https://github.com/cocoindex-io/cocoindex) ingest (`PIPELINE_BACKEND=cocoindex`) mixing CocoIndex connectors with Flexible GraphRAG sources (incl. event detectors), parsers, chunkers, embeddings, LI/LC KG extractors, and broader targets (all 15 PG + 10 vector + RDF + search) — same UI / REST / MCP. Mutually exclusive with `ENABLE_INCREMENTAL_UPDATES=true` and `ENABLE_LANGFLOW_FLOWS=true`. Standalone `app.py` also supported. See [CocoIndex Integration](#cocoindex-integration).
  70. ## Frontend Screenshots
  71. ### Angular Frontend - Tabbed Interface
  72. <details>
  73. <summary>Click to view Angular UI screenshots (Light Theme)</summary>
  74. | Sources Tab | Processing Tab | Search Tab | Chat Tab |
  75. |-------------|----------------|------------|----------|
  76. | [![Angular Sources](./screen-shots/angular/angular-sources.png)](./screen-shots/angular/angular-sources.png) | [![Angular Processing](./screen-shots/angular/angular-processing.png)](./screen-shots/angular/angular-processing.png) | [![Angular Search](./screen-shots/angular/angular-search.png)](./screen-shots/angular/angular-search.png) | [![Angular Chat](./screen-shots/angular/angular-chat.png)](./screen-shots/angular/angular-chat.png) |
  77. </details>
  78. ### React Frontend - Tabbed Interface
  79. <details open>
  80. <summary>Click to view React UI screenshots (Dark Theme)</summary>
  81. | Sources Tab | Processing Tab | Search Tab | Chat Tab |
  82. |-------------|----------------|------------|----------|
  83. | [![React Sources](./screen-shots/react/react-sources.png)](./screen-shots/react/react-sources.png) | [![React Processing](./screen-shots/react/react-processing.png)](./screen-shots/react/react-processing.png) | [![React Search](./screen-shots/react/react-search-hybrid-search.png)](./screen-shots/react/react-search-hybrid-search.png) | [![React Chat](./screen-shots/react/react-chat-using.png)](./screen-shots/react/react-chat-using.png) |
  84. </details>
  85. <details>
  86. <summary>Click to view React UI screenshots (Light Theme)</summary>
  87. | Sources Tab | Processing Tab | Search Tab | Chat Tab |
  88. |-------------|----------------|------------|----------|
  89. | [![React Sources Light](./screen-shots/react/react-sources-light.png)](./screen-shots/react/react-sources-light.png) | [![React Processing Light](./screen-shots/react/react-processing-light.png)](./screen-shots/react/react-processing-light.png) | [![React Search Light](./screen-shots/react/react-search-hybrid-search-light.png)](./screen-shots/react/react-search-hybrid-search-light.png) | [![React Chat Light](./screen-shots/react/react-chat-using-light.png)](./screen-shots/react/react-chat-using-light.png) |
  90. </details>
  91. ### Vue Frontend - Tabbed Interface
  92. <details>
  93. <summary>Click to view Vue UI screenshots (Light Theme)</summary>
  94. | Sources Tab | Processing Tab | Search Tab | Chat Tab |
  95. |-------------|----------------|------------|----------|
  96. | [![Vue Sources](./screen-shots/vue/vue-sources.png)](./screen-shots/vue/vue-sources.png) | [![Vue Processing](./screen-shots/vue/vue-processing.png)](./screen-shots/vue/vue-processing.png) | [![Vue Search](./screen-shots/vue/vue-search.png)](./screen-shots/vue/vue-search.png) | [![Vue Chat](./screen-shots/vue/vue-chat.png)](./screen-shots/vue/vue-chat.png) |
  97. </details>
  98. ## System Components
  99. ### FastAPI Backend (`/flexible-graphrag`)
  100. - **REST API Server**: Provides endpoints for document ingestion, search, and AI query/chat
  101. - **Hybrid Search Engine**: Combines vector similarity (RAG), fulltext (BM25), and graph traversal (GraphRAG)
  102. - **Document Processing**: Advanced document conversion with Docling, LlamaParse, and LiteParse integration
  103. - **Configurable Architecture**: Environment-based configuration for all components
  104. - **Async Processing**: Background task processing with real-time progress updates
  105. ### MCP Server (`/flexible-graphrag-mcp`)
  106. - **MCP Client support**: Model Context Protocol server for Claude Desktop and other MCP clients
  107. - **Full API Parity**: Tools like `ingest_documents()` support all 14 data sources with source-specific configs: filesystem, repositories (Alfresco, SharePoint, Box, CMIS, Nuxeo), cloud storage, web; `skip_graph` flag for all data sources; `paths` parameter for filesystem/Alfresco/CMIS; Alfresco also supports `nodeDetails` list (multi-select for KG Spaces)
  108. - **Additional Tools**: `search_documents()`, `query_documents()`, `ingest_text()`, system diagnostics, and health checks
  109. - **Dual Transport**: HTTP mode for debugging, stdio mode for production
  110. - **Tool Suite**: 12 tools for document processing, search, sign-in and system management (14 with `MCP_BROWSER_SIGN_IN=true`); search and query can be permission-filtered per user (three levels)
  111. - **Multiple Installation**: pipx system installation or uvx no-install execution
  112. ### UI Clients (`/flexible-graphrag-ui`)
  113. - **Angular Frontend**: Material Design with TypeScript
  114. - **React Frontend**: Modern React with Vite and TypeScript
  115. - **Vue Frontend**: Vue 3 Composition API with Vuetify and TypeScript
  116. - **Unified Features**: All clients support the 4 tab views, async processing, progress tracking, and cancellation
  117. ### Docker Infrastructure (`/docker`)
  118. - **Modular Database Selection**: Include/exclude vector, graph, and search engines, and Alfresco with single-line comments
  119. - **Flexible Deployment**: Hybrid mode (databases in Docker, apps standalone) or full containerization
  120. - **NGINX Reverse Proxy**: Unified access to all services with proper routing
  121. - **Built-in Database Dashboards**: Most server dockers also provide built-in web interface dashboards (Neo4j browser, ArcadeDB, FalkorDB, OpenSearch, etc.)
  122. - **Separate Dashboards**: Additional dashboard dockers are provided: including Kibana for Elasticsearch and optional Ladybug Explorer (see `docker/includes/ladybug-explorer.yaml`).
  123. ## Data Sources
  124. Flexible GraphRAG supports **14 different data sources** for ingesting documents into your knowledge base (Nuxeo missing from older screenshots):
  125. <p align="center">
  126. <a href="./screen-shots/react/data-sources-1.jpeg">
  127. <img src="./screen-shots/react/data-sources-1.jpeg" alt="Data Sources" width="700">
  128. </a>
  129. </p>
  130. <p align="center">
  131. <a href="./screen-shots/auto-sync/auto-sync.png">
  132. <img src="./screen-shots/auto-sync/auto-sync.png" alt="Flexible GraphRAG data sources, processing tab, auto-sync document states in Postgres, Neo4j" width="700">
  133. </a>
  134. </p>
  135. <p align="center"><em>Flexible GraphRAG data sources, processing tab, auto-sync document states in Postgres (default pipeline, with auto incremental update system enabled), Neo4j shown (Qdrant, Elasticsearch not shown)</em></p>
  136. ### File & Upload Sources
  137. 1. **File Upload** - Direct file upload through web interface with drag & drop support
  138. ### Cloud Storage Sources
  139. 2. **Amazon S3** - AWS S3 bucket integration
  140. 3. **Google Cloud Storage (GCS)** - Google Cloud storage buckets
  141. 4. **Azure Blob Storage** - Microsoft Azure blob containers
  142. 5. **OneDrive** - Microsoft OneDrive personal/business storage
  143. 6. **Google Drive** - Google Drive file storage
  144. ### Enterprise Repository Sources
  145. 7. **Alfresco** - Alfresco ECM/content repository with two integration options:
  146. - **[KG Spaces ACA Extension](https://github.com/stevereiner/kg-spaces-aca)** - Integrates the Flexible GraphRAG Angular UI as an extension plugin within the Alfresco Content Application (ACA), enabling multi-select document/folder ingestion with nodeIds directly from the Alfresco interface. Its title bar has a sign-in pill for the other sources and servers (Angular lib option `searchSignIn`), so KG Spaces search and chat can also use other sources' documents the user may read
  147. - **Flexible GraphRAG Alfresco Data Source** - Direct integration using Alfresco paths (e.g., /Shared/GraphRAG, /Company Home/Shared/GraphRAG, or /Shared/GraphRAG/cmispress.txt)
  148. 8. **Nuxeo** - Nuxeo content repository (File and Note documents); basic / token (X-Authentication-Token) / OAuth2 auth, path or node selection, and real-time incremental sync via the Nuxeo audit event stream (Kafka)
  149. - **[KG Spaces for Nuxeo](https://github.com/stevereiner/kg-spaces-nuxeo)** - A Nuxeo Web UI add-on (a KG Spaces menu with "Add to KG Spaces" and "Ask KG Spaces about this document / folder") plus an Angular app on the same shared Angular UI library, sharing the Web UI session; ingest and answers use the Web UI user's Nuxeo token, so only documents that user may read
  150. 9. **SharePoint** - Microsoft SharePoint document libraries
  151. 10. **Box** - Box.com cloud storage
  152. 11. **CMIS (Content Management Interoperability Services)** - Industry-standard content repository interface
  153. ### Web Sources
  154. 12. **Web Pages** - Extract content from web URLs
  155. 13. **Wikipedia** - Ingest Wikipedia articles by title or URL
  156. 14. **YouTube** - Process YouTube video transcripts
  157. Each data source includes:
  158. - **Configuration Forms**: Easy-to-use interfaces for credentials and settings
  159. - **Progress Tracking**: Real-time per-file progress indicators
  160. - **Flexible Authentication**: Support for various auth methods (API keys, OAuth, service accounts)
  161. - **Per-user sign-in** (Alfresco, Nuxeo, CMIS, SharePoint, OneDrive, Box, Google Drive, GCS, Azure Blob, S3): search and chat answer only from documents the signed-in user may read. See [Sign-In and Permission-Aware Search](docs/DATA-SOURCES/SIGN-IN-GUIDE.md)
  162. ### Incremental Updates & Auto-Sync
  163. **NEW!** Flexible GraphRAG supports **automatic incremental updates** (Optional) from most data sources, keeping your Vector, Search and Graph databases synchronized in real-time or near real-time:
  164. | Data Source | Auto-Sync Support | Detection Method | Status | Notes |
  165. |-------------|-------------------|------------------|--------|-------|
  166. | **Alfresco** | ✅ Real-time | Apache ActiveMQ | Ready | |
  167. | **Nuxeo** | ✅ Real-time | Nuxeo audit stream (Kafka) | Ready | |
  168. | **Amazon S3** | ✅ Real-time | SQS event notifications | Ready | |
  169. | **Azure Blob Storage** | ✅ Real-time | Change feed | Ready | |
  170. | **Google Cloud Storage** | ✅ Real-time | Pub/Sub notifications | Ready | |
  171. | **Google Drive** | ✅ Near real-time | Changes API (polling) | Ready | |
  172. | **OneDrive** | ✅ Near real-time | MS Graph delta query | Ready | |
  173. | **SharePoint** | ✅ Near real-time | MS Graph delta query | Ready | |
  174. | **Box** | ✅ Near real-time | Events API (polling) | Ready | |
  175. | **Local Filesystem** | ✅ Real-time | OS events (watchdog) | Ready | REST API and MCP Server only |
  176. | **File Upload UI, CMIS, Web Pages, Wikipedia, YouTube** | ➖ Not supported | - | - | No support for incremental updates |
  177. **Features**:
  178. - **Modification Date Tracking**: Uses file modification timestamps (ordinal) to detect changes
  179. - **Content Hash Optimization**: Skips reprocessing when file modification date changed but content hasn't
  180. - **Dual Mechanism**: Event-driven streams (real-time) + periodic polling fallback
  181. - **LlamaIndex Integration**: Uses proper abstractions for all databases
  182. - **UI, REST API, MCP Server**: Setting up an auto update data source location can be done thru the 3 UIs, with the REST API, or with the MCP server
  183. **Setup Requirements**:
  184. Enable incremental updates in your `.env` file:
  185. ```bash
  186. ENABLE_INCREMENTAL_UPDATES=true
  187. # PostgreSQL database for state management
  188. # By default, uses the pgvector database from docker-compose.yaml
  189. POSTGRES_INCREMENTAL_URL=postgresql://postgres:password@localhost:5433/postgres
  190. ```
  191. **Note**: The incremental updates system uses PostgreSQL to track document state. The `docker-compose.yaml` includes a pgvector container that can be used both as a vector database option and for incremental updates state management. The database connection creates the necessary tables automatically on first use. Or keep the state in OpenArch instead, with no PostgreSQL (`INCREMENTAL_STATE_STORE=openarch`, v0.9.0+): see [`docs/DATABASES/OPENARCH/OPENARCH-STATE-STORE.md`](docs/DATABASES/OPENARCH/OPENARCH-STATE-STORE.md).
  192. **Usage**:
  193. - Tick the **Auto Sync** column for the row(s) in the Processing tab (with the incremental registry on, every data source has per-row Search+Vector / Graph / RDF / Auto Sync columns (one Graphs column before v0.9.0); the older **"Enable auto change sync"** checkbox appears only when the registry is off)
  194. - **Deep** on the Alfresco, Nuxeo and Box forms includes subfolders (recursive) in the ingest and its auto sync; it starts at the backend's `DEFAULT_RECURSIVE` (true unless set)
  195. - A new row's **Graph** and **RDF** checkboxes start at the backend's `DEFAULT_ENABLED_GRAPH` and `DEFAULT_ENABLED_RDF` (true unless set; `env-sample.txt` sets both to false: search + vector first, a row's graphs added when wanted). A REST or MCP ingest that doesn't say `enabled_graph` / `enabled_rdf` uses them too
  196. - For **S3**: Also provide the "SQS Queue URL" for event notifications
  197. - For **GCS**: Also provide the "Pub/Sub Subscription Name" for real-time updates
  198. **PostgreSQL for State Management**:
  199. The `docker/includes/postgres-pgvector.yaml` sets up two databases automatically on first start: `flexible_graphrag` (for optional pgvector vector storage) and `flexible_graphrag_incremental` (for incremental update state management, with its schema created automatically). pgAdmin is also configured at http://localhost:5050 with both databases pre-registered (no master password) — use `password` for the server connection and save it. See [docs/DATABASES/POSTGRES-SETUP.md](docs/DATABASES/POSTGRES-SETUP.md) for details.
  200. **Documentation**:
  201. - System overview: [`docs/DATA-SOURCES/INCREMENTAL-UPDATE-AUTO-SYNC/README.md`](docs/DATA-SOURCES/INCREMENTAL-UPDATE-AUTO-SYNC/README.md)
  202. - Quick start: [`docs/DATA-SOURCES/INCREMENTAL-UPDATE-AUTO-SYNC/QUICKSTART.md`](docs/DATA-SOURCES/INCREMENTAL-UPDATE-AUTO-SYNC/QUICKSTART.md)
  203. - Detailed setup: [`docs/DATA-SOURCES/INCREMENTAL-UPDATE-AUTO-SYNC/SETUP-GUIDE.md`](docs/DATA-SOURCES/INCREMENTAL-UPDATE-AUTO-SYNC/SETUP-GUIDE.md)
  204. - API reference: [`docs/DATA-SOURCES/INCREMENTAL-UPDATE-AUTO-SYNC/API-REFERENCE.md`](docs/DATA-SOURCES/INCREMENTAL-UPDATE-AUTO-SYNC/API-REFERENCE.md)
  205. - PostgreSQL setup: [`docs/DATABASES/POSTGRES-SETUP.md`](docs/DATABASES/POSTGRES-SETUP.md)
  206. - OpenArch as the state store: [`docs/DATABASES/OPENARCH/OPENARCH-STATE-STORE.md`](docs/DATABASES/OPENARCH/OPENARCH-STATE-STORE.md)
  207. **Scripts**:
  208. - `scripts/incremental/sync-now.sh|.ps1|.bat` - Trigger immediate synchronization
  209. - `scripts/incremental/set-refresh-interval.sh|.ps1|.bat` - Configure polling interval
  210. - `scripts/incremental/TIMING-CONFIGURATION.md` - Timing configuration details
  211. - `scripts/incremental/README.md` - Script usage documentation
  212. ### Document Processing Options
  213. All data sources support three document parser options (full per-parser format matrix in [Supported File Formats](docs/DATA-SOURCES/DOC-PROCESSING/SUPPORTED-FILE-FORMATS.md)):
  214. **Docling (Default)**:
  215. - Open-source, local processing
  216. - Free with no API costs
  217. - **GPU acceleration** supported (CUDA/Apple Silicon) for 5-10x faster processing
  218. - Built-in OCR for scanned documents and images — `DOCLING_OCR=true` + `DOCLING_OCR_ENGINE=auto|rapidocr|easyocr|tesseract_cli|tesserocr|ocrmac`
  219. - Multi-language support (English, German, French, Spanish, Czech, Russian, Chinese, Japanese, etc.)
  220. - Configured via: `DOCUMENT_PARSER=docling`
  221. - `DOCLING_DEVICE=auto|cpu|cuda|mps` — control GPU vs CPU processing
  222. - `SAVE_PARSING_OUTPUT=true` — save intermediate parsing results for inspection (works for all three parsers)
  223. - `PARSER_FORMAT_FOR_EXTRACTION=auto|markdown|plaintext` — control format used for knowledge graph extraction
  224. - See [Docling GPU + OCR Configuration Guide](docs/DATA-SOURCES/DOC-PROCESSING/DOCLING-GPU-CONFIGURATION.md) for setup details | [Quick Reference](docs/DATA-SOURCES/DOC-PROCESSING/DOCLING-GPU-CONFIGURATION.md#quick-reference-installation-commands)
  225. **LlamaParse**:
  226. - Cloud-based API service with advanced AI
  227. - Multimodal parsing with Claude Sonnet 3.5
  228. - Three modes available:
  229. - `parse_page_without_llm` - 1 credit/page
  230. - `parse_page_with_llm` - 3 credits/page (default)
  231. - `parse_page_with_agent` - 10-90 credits/page
  232. - Configured via: `DOCUMENT_PARSER=llamaparse` + `LLAMAPARSE_API_KEY`
  233. - Get your API key from [LlamaCloud](https://cloud.llamaindex.ai/)
  234. - **New**: `SAVE_PARSING_OUTPUT=true` - Save parsed output and metadata for inspection
  235. - **New**: `PARSER_FORMAT_FOR_EXTRACTION=auto|markdown|plaintext` - Control format used for knowledge graph extraction
  236. **LiteParse**:
  237. - Open-source, local processing (Rust/PyO3) — free, no API key
  238. - Natively parses PDFs with bundled Tesseract OCR; `.txt`/`.md` read directly
  239. - Office formats need **LibreOffice**, images need **ImageMagick** — see [Supported File Formats](docs/DATA-SOURCES/DOC-PROCESSING/SUPPORTED-FILE-FORMATS.md#liteparse-free-local-lightweight)
  240. - Optional complexity-based routing of scanned/complex docs to Docling or LlamaParse via `LITEPARSE_COMPLEX_ROUTING`
  241. - Configured via: `DOCUMENT_PARSER=liteparse`; install `uv pip install liteparse`
  242. - Supports `SAVE_PARSING_OUTPUT` and `PARSER_FORMAT_FOR_EXTRACTION` like the other parsers
  243. ## Supported File Formats
  244. Flexible GraphRAG processes **documents, images, and audio** across its three parsers:
  245. - **Documents:** PDF (`.pdf`); Microsoft Office + legacy (`.docx`/`.xlsx`/`.pptx`, `.doc`/`.xls`/`.ppt`); web (`.html`/`.htm`/`.xhtml`); data (`.csv`/`.tsv`/`.json`/`.xml`); documentation (`.md`/`.markdown`/`.asciidoc`/`.rtf`/`.txt`/`.epub`)
  246. - **Images:** `.png`/`.jpg`/`.jpeg`/`.gif`/`.bmp`/`.webp`/`.tiff`/`.tif`
  247. - **Audio:** `.wav`/`.mp3`/`.mp4`/`.m4a` (speech recognition / transcription)
  248. Capabilities vary by parser — **Docling** and **LlamaParse** handle all of the above natively (advanced layout/table/formula analysis, configurable OCR, VLM/multimodal, ASR); **LiteParse** parses PDFs natively and needs LibreOffice for Office formats / ImageMagick for images. Both markdown and plaintext are saved, and the best is auto-selected for knowledge-graph extraction, embeddings, and search (markdown for tables, plaintext for text-heavy docs; override with `PARSER_FORMAT_FOR_EXTRACTION`).
  249. **Full per-parser matrix, OCR/VLM options, and output formats: [Supported File Formats](docs/DATA-SOURCES/DOC-PROCESSING/SUPPORTED-FILE-FORMATS.md).**
  250. ## Database Configuration
  251. Flexible GraphRAG uses three types of databases for its hybrid search capabilities. Each can be configured independently via environment variables.
  252. ### Search Databases (Full-Text Search)
  253. Set `SEARCH_DB` to select the store and `SEARCH_BACKEND=llamaindex` or `langchain` for the framework.
  254. - **BM25 (Built-in)**: Local in-memory BM25 full-text search with TF-IDF ranking
  255. - Dashboard: None (file-based)
  256. - Configuration:
  257. ```bash
  258. SEARCH_DB=bm25
  259. BM25_SEARCH_DB_CONFIG={"persist_dir": "./bm25_index"}
  260. ```
  261. - **Elasticsearch**: Enterprise search engine with advanced analyzers, faceted search, and real-time analytics
  262. - Dashboard: Kibana (http://localhost:5601)
  263. - Configuration:
  264. ```bash
  265. SEARCH_DB=elasticsearch
  266. ELASTICSEARCH_SEARCH_DB_CONFIG={"hosts": ["http://localhost:9200"], "index_name": "hybrid_search"}
  267. ```
  268. - **OpenSearch**: AWS-led open-source fork with native hybrid scoring (vector + BM25) and k-NN algorithms
  269. - Dashboard: OpenSearch Dashboards (http://localhost:5601)
  270. - Configuration:
  271. ```bash
  272. SEARCH_DB=opensearch
  273. OPENSEARCH_SEARCH_DB_CONFIG={"hosts": ["http://localhost:9201"], "index_name": "hybrid_search"}
  274. ```
  275. - **None**: Disable full-text search (vector search only)
  276. - Configuration:
  277. ```bash
  278. SEARCH_DB=none
  279. ```
  280. ### Vector Databases (Semantic Search)
  281. Set `VECTOR_DB` to select the store and `VECTOR_BACKEND=llamaindex` or `langchain` for the framework.
  282. When switching embedding models, delete existing vector indexes — dimensions differ by provider. See [docs/DATABASES/VECTOR-DATABASES/VECTOR-DIMENSIONS.md](docs/DATABASES/VECTOR-DATABASES/VECTOR-DIMENSIONS.md) for cleanup instructions.
  283. #### Supported Vector Databases
  284. - **Neo4j**: Can be used as vector database with separate vector configuration
  285. - Dashboard: Neo4j Browser (http://localhost:7474)
  286. - Configuration:
  287. ```bash
  288. VECTOR_DB=neo4j
  289. NEO4J_VECTOR_DB_CONFIG={"uri": "bolt://localhost:7687", "username": "neo4j", "password": "your_password", "index_name": "hybrid_search_vector"}
  290. ```
  291. - **Qdrant**: Dedicated vector database with advanced filtering
  292. - Dashboard: Qdrant Web UI (http://localhost:6333/dashboard)
  293. - Configuration:
  294. ```bash
  295. VECTOR_DB=qdrant
  296. QDRANT_VECTOR_DB_CONFIG={"host": "localhost", "port": 6333, "collection_name": "hybrid_search"}
  297. ```
  298. - **Elasticsearch**: Can be used as vector database with separate vector configuration
  299. - Dashboard: Kibana (http://localhost:5601)
  300. - Configuration:
  301. ```bash
  302. VECTOR_DB=elasticsearch
  303. ELASTICSEARCH_VECTOR_DB_CONFIG={"hosts": ["http://localhost:9200"], "index_name": "hybrid_search_vectors"}
  304. ```
  305. - **OpenSearch**: Can be used as vector database with separate vector configuration
  306. - Dashboard: OpenSearch Dashboards (http://localhost:5601)
  307. - Configuration:
  308. ```bash
  309. VECTOR_DB=opensearch
  310. OPENSEARCH_VECTOR_DB_CONFIG={"hosts": ["http://localhost:9201"], "index_name": "hybrid_search_vectors"}
  311. ```
  312. - **Chroma**: Open-source vector database with dual deployment modes
  313. - Dashboard: Swagger UI (http://localhost:8001/docs/) (HTTP mode)
  314. - Configuration (Local Mode):
  315. ```bash
  316. VECTOR_DB=chroma
  317. CHROMA_VECTOR_DB_CONFIG={"persist_directory": "./chroma_db", "collection_name": "hybrid_search"}
  318. ```
  319. - Configuration (HTTP Mode):
  320. ```bash
  321. VECTOR_DB=chroma
  322. CHROMA_VECTOR_DB_CONFIG={"host": "localhost", "port": 8001, "collection_name": "hybrid_search"}
  323. ```
  324. - **Milvus**: Cloud-native, scalable vector database for similarity search
  325. - Dashboard: Attu (http://localhost:3003)
  326. - Configuration:
  327. ```bash
  328. VECTOR_DB=milvus
  329. MILVUS_VECTOR_DB_CONFIG={"host": "localhost", "port": 19530, "collection_name": "hybrid_search"}
  330. ```
  331. - **Weaviate**: Vector search engine with semantic capabilities and data enrichment
  332. - Dashboard: Weaviate Console (http://localhost:8086/console)
  333. - Configuration:
  334. ```bash
  335. VECTOR_DB=weaviate
  336. WEAVIATE_VECTOR_DB_CONFIG={"url": "http://localhost:8086", "index_name": "HybridSearch"}
  337. ```
  338. - **Pinecone**: Managed vector database service optimized for real-time applications
  339. - Dashboard: Pinecone Console (web-based)
  340. - Configuration:
  341. ```bash
  342. VECTOR_DB=pinecone
  343. PINECONE_VECTOR_DB_CONFIG={"api_key": "your_api_key", "region": "us-east-1", "cloud": "aws", "index_name": "hybrid-search"}
  344. ```
  345. - **PostgreSQL**: Traditional database with pgvector extension for vector similarity search
  346. - Dashboard: pgAdmin (http://localhost:5050)
  347. - Configuration:
  348. ```bash
  349. VECTOR_DB=postgres
  350. POSTGRES_VECTOR_DB_CONFIG={"host": "localhost", "port": 5433, "database": "postgres", "username": "postgres", "password": "your_password"}
  351. ```
  352. - **LanceDB**: Modern, lightweight vector database designed for high-performance ML applications
  353. - Dashboard: LanceDB Viewer (http://localhost:3005)
  354. - Configuration:
  355. ```bash
  356. VECTOR_DB=lancedb
  357. LANCEDB_VECTOR_DB_CONFIG={"uri": "./lancedb", "table_name": "hybrid_search"}
  358. ```
  359. #### RAG without GraphRAG
  360. For faster document ingest processing (no graph extraction), and hybrid search with only full text + vector, configure:
  361. ```bash
  362. VECTOR_DB=qdrant # Any vector store
  363. SEARCH_DB=elasticsearch # Any search engine
  364. PG_GRAPH_DB=none
  365. ```
  366. ### Property Graph Databases (Knowledge Graph / GraphRAG)
  367. Set `PG_GRAPH_DB` to select the store and `GRAPH_BACKEND=llamaindex` or `langchain` for the framework where both are supported. **LangChain-only** stores (ArangoDB, Apache AGE, HugeGraph, SurrealDB, TigerGraph, Cosmos Gremlin) route property-graph ingestion and retrieval through LangChain adapters regardless of other env defaults.
  368. - **Neo4j Property Graph**: Primary knowledge graph storage with Cypher querying
  369. - Dashboard: Neo4j Browser (http://localhost:7474)
  370. - Configuration:
  371. ```bash
  372. PG_GRAPH_DB=neo4j
  373. NEO4J_GRAPH_DB_CONFIG={"uri": "bolt://localhost:7687", "username": "neo4j", "password": "your_password"}
  374. ```
  375. - **ArcadeDB**: Multi-model database supporting graph, document, key-value, and search with SQL and Cypher
  376. - Dashboard: ArcadeDB Studio (http://localhost:2480)
  377. - Configuration:
  378. ```bash
  379. PG_GRAPH_DB=arcadedb
  380. ARCADEDB_GRAPH_DB_CONFIG={"host": "localhost", "port": 2480, "username": "root", "password": "password", "database": "flexible_graphrag", "query_language": "sql"}
  381. ```
  382. - **FalkorDB**: High-performance graph database using GraphBLAS; purpose-built for LLM / GraphRAG
  383. - Dashboard: FalkorDB Browser (http://localhost:3001)
  384. - Configuration:
  385. ```bash
  386. PG_GRAPH_DB=falkordb
  387. FALKORDB_GRAPH_DB_CONFIG={"url": "falkor://localhost:6379", "database": "falkor"}
  388. ```
  389. - **Ladybug**: Embedded property graph database (Cypher, single `.lbug` file) with optional structured schema and HNSW vector index on chunks; Explorer UI via Docker (port 7003)
  390. - Configuration:
  391. ```bash
  392. PG_GRAPH_DB=ladybug
  393. LADYBUG_GRAPH_DB_CONFIG={"db_dir": "./ladybug", "db_file": "database.lbug", "use_vector_index": true, "has_structured_schema": false, "strict_schema": false}
  394. ```
  395. - **MemGraph**: Real-time graph database with streaming support and advanced graph algorithms
  396. - Dashboard: MemGraph Lab (http://localhost:3002)
  397. - Configuration:
  398. ```bash
  399. PG_GRAPH_DB=memgraph
  400. MEMGRAPH_GRAPH_DB_CONFIG={"url": "bolt://localhost:7687", "username": "", "password": ""}
  401. ```
  402. - **NebulaGraph**: Distributed graph database for large-scale data with horizontal scalability
  403. - Dashboard: NebulaGraph Studio (http://localhost:7001)
  404. - Configuration:
  405. ```bash
  406. PG_GRAPH_DB=nebula
  407. NEBULA_GRAPH_DB_CONFIG={"space": "flexible_graphrag", "host": "localhost", "port": 9669, "username": "root", "password": "nebula"}
  408. ```
  409. - **Amazon Neptune**: Fully managed graph database service supporting property graph and RDF models
  410. - Dashboard: Graph-Explorer (http://localhost:3007) or Neptune Workbench (AWS Console)
  411. - Configuration:
  412. ```bash
  413. PG_GRAPH_DB=neptune
  414. NEPTUNE_GRAPH_DB_CONFIG={"host": "your-cluster.region.neptune.amazonaws.com", "port": 8182}
  415. ```
  416. - **Amazon Neptune Analytics**: Serverless graph analytics with openCypher support
  417. - Dashboard: Graph-Explorer (http://localhost:3007) or Neptune Workbench (AWS Console)
  418. - Configuration:
  419. ```bash
  420. PG_GRAPH_DB=neptune_analytics
  421. NEPTUNE_ANALYTICS_GRAPH_DB_CONFIG={"graph_identifier": "g-xxxxx", "region": "us-east-1"}
  422. ```
  423. - **Google Cloud Spanner Graph**: Managed relational + property graph (GQL). Uses `llama-index-spanner` (LlamaIndex) or `langchain-google-spanner` 0.10.0+ (LangChain) — install with `uv pip install -e ".[spanner-extras]"` then `uv pip uninstall llama-index` (see [Optional](#optional) under Prerequisites).
  424. - Setup: [docs/DATABASES/GRAPH-DATABASES/SPANNER-SETUP.md](docs/DATABASES/GRAPH-DATABASES/SPANNER-SETUP.md)
  425. - Configuration:
  426. ```bash
  427. PG_GRAPH_DB=spanner
  428. # GRAPH_BACKEND=llamaindex or langchain (give each its own graph_name in one database)
  429. SPANNER_GRAPH_DB_CONFIG={"project_id": "my-gcp-project", "instance_id": "my-spanner-instance", "database_id": "my-database", "graph_name": "knowledge_graph", "credentials_file": "./gcs.json"}
  430. ```
  431. - **ArangoDB** *(LangChain only)*: Multi-model database with AQL graph queries
  432. - Dashboard: ArangoDB Web UI (http://localhost:8529)
  433. - Configuration:
  434. ```bash
  435. PG_GRAPH_DB=arangodb
  436. ARANGODB_GRAPH_DB_CONFIG={"url": "http://localhost:8529", "database": "flexible_graphrag", "username": "root", "password": "password"}
  437. ```
  438. - **Apache AGE** *(LangChain only)*: PostgreSQL extension for graph data via Cypher
  439. - Dashboard: pgAdmin (http://localhost:5050)
  440. - Configuration:
  441. ```bash
  442. PG_GRAPH_DB=apache_age
  443. APACHE_AGE_GRAPH_DB_CONFIG={"host": "localhost", "port": 5434, "database": "flexible_graphrag_age", "username": "postgres", "password": "password", "graph_name": "knowledge_graph"}
  444. ```
  445. - **HugeGraph** *(LangChain only)*: Distributed graph database with Gremlin and openCypher
  446. - Dashboard: HugeGraph Hubble (http://localhost:8085)
  447. - Configuration:
  448. ```bash
  449. PG_GRAPH_DB=hugegraph
  450. HUGEGRAPH_GRAPH_DB_CONFIG={"host": "localhost", "port": 8082, "database": "hugegraph"}
  451. ```
  452. - **SurrealDB** *(LangChain only)*: Multi-model database with SurrealQL graph queries
  453. - Dashboard: Surrealist (http://localhost:8011)
  454. - Configuration:
  455. ```bash
  456. PG_GRAPH_DB=surrealdb
  457. SURREALDB_GRAPH_DB_CONFIG={"url": "ws://localhost:8010/rpc", "namespace": "test", "database": "flexible_graphrag", "username": "root", "password": "root"}
  458. ```
  459. - **TigerGraph** *(LangChain only)*: Distributed graph database with GSQL
  460. - Dashboard: GraphStudio (http://localhost:14240)
  461. - Configuration:
  462. ```bash
  463. PG_GRAPH_DB=tigergraph
  464. TIGERGRAPH_GRAPH_DB_CONFIG={"host": "http://localhost", "port": 14240, "restpp_port": 9002, "database": "MyGraph", "username": "tigergraph", "password": "tigergraph"}
  465. ```
  466. - **Cosmos Gremlin** *(LangChain only)*: Azure Cosmos DB for Gremlin API
  467. - Configuration:
  468. ```bash
  469. PG_GRAPH_DB=cosmos_gremlin
  470. COSMOS_GREMLIN_GRAPH_DB_CONFIG={"url": "ws://localhost:8182/gremlin"}
  471. ```
  472. - **None**: Disable knowledge graph extraction for RAG-only mode
  473. - Configuration:
  474. ```bash
  475. PG_GRAPH_DB=none
  476. ```
  477. ## Ontology and RDF Support
  478. Flexible GraphRAG supports RDF/RDFS/OWL ontologies to guide knowledge graph extraction, with optional RDF graph store backends. Ontology-guided extraction works with **any** configured store — property graph, RDF graph store, or both.
  479. - Load OWL/RDFS ontologies (`owl:Class`, `owl:ObjectProperty`, `owl:DatatypeProperty`, `rdfs:domain`, `rdfs:range`) to constrain entity/relation extraction; OWL is supported but not required
  480. - Works with all 15 property graph databases — no RDF store required to use ontology-guided extraction
  481. - Full pipeline for all 4 RDF graph stores: UI document ingest → KG extraction → RDF storage; auto incremental sync; Hybrid Search and AI Query/Chat fuse RDF store results alongside vector, BM25, and property graph results
  482. - SPARQL 1.1 queries; RDF 1.2 triple terms and relation annotations (`{| |}` syntax); XSD-typed literals from OWL `DatatypeProperty` ranges
  483. **RDF Graph Store Configuration** — set `RDF_GRAPH_DB` to select the store (all four support RDF 1.2 triple terms; Neptune is AWS-managed—no local compose include):
  484. - **Apache Jena Fuseki** — SPARQL 1.1 server; dashboard: http://localhost:3030
  485. ```bash
  486. RDF_GRAPH_DB=fuseki
  487. FUSEKI_BASE_URL=http://localhost:3030
  488. FUSEKI_DATASET=flexible-graphrag
  489. ```
  490. - **Ontotext GraphDB** — enterprise RDF store with OWL reasoning; dashboard: http://localhost:7200
  491. ```bash
  492. RDF_GRAPH_DB=graphdb
  493. GRAPHDB_BASE_URL=http://localhost:7200
  494. GRAPHDB_REPOSITORY=flexible-graphrag
  495. GRAPHDB_USERNAME=admin
  496. GRAPHDB_PASSWORD=admin
  497. ```
  498. - **Oxigraph** — lightweight local store, native RDF 1.2; dashboard: http://localhost:7878
  499. ```bash
  500. RDF_GRAPH_DB=oxigraph
  501. OXIGRAPH_URL=http://localhost:7878
  502. ```
  503. - **Amazon Neptune RDF** — managed SPARQL 1.1 on Neptune (same cluster can host property graph and RDF; IAM SigV4 auth). See [Neptune RDF setup](docs/DATABASES/GRAPH-DATABASES/NEPTUNE-SETUP.md).
  504. ```bash
  505. RDF_GRAPH_DB=neptune_rdf
  506. NEPTUNE_RDF_HOST=db-neptune-1.cluster-xxxxxxxxxxxx.us-east-1.neptune.amazonaws.com
  507. NEPTUNE_RDF_PORT=8182
  508. NEPTUNE_RDF_REGION=us-east-1
  509. NEPTUNE_RDF_USE_IAM_AUTH=true
  510. NEPTUNE_RDF_USE_HTTPS=true
  511. # Optional explicit keys (else default AWS credential chain):
  512. # NEPTUNE_RDF_AWS_ACCESS_KEY_ID=
  513. # NEPTUNE_RDF_AWS_SECRET_ACCESS_KEY=
  514. ```
  515. - **None** — disable RDF graph store:
  516. ```bash
  517. RDF_GRAPH_DB=none
  518. ```
  519. **Docker Setup:** Uncomment local RDF store includes in `docker-compose.yaml` (Fuseki, GraphDB, Oxigraph):
  520. ```yaml
  521. includes:
  522. # - includes/jena-fuseki.yaml
  523. # - includes/ontotext-graphdb.yaml
  524. # - includes/oxigraph.yaml
  525. ```
  526. **Complete Documentation:** [docs/DATABASES/RDF/RDF-ONTOLOGY-SUPPORT.md](docs/DATABASES/RDF/RDF-ONTOLOGY-SUPPORT.md) | [docs/DATABASES/RDF/RDF-STORE-USER-GUIDE.md](docs/DATABASES/RDF/RDF-STORE-USER-GUIDE.md)
  527. ## Framework Configuration
  528. Every pipeline stage can independently run on LlamaIndex or LangChain via env var pickers:
  529. | Variable | Options | Description |
  530. |---|---|---|
  531. | `GRAPH_BACKEND` | `llamaindex` \| `langchain` | Property graph store and KG retrieval |
  532. | `VECTOR_BACKEND` | `llamaindex` \| `langchain` | Vector store adapter |
  533. | `SEARCH_BACKEND` | `llamaindex` \| `langchain` | Full-text search adapter |
  534. | `CHUNKER_BACKEND` | `llamaindex` \| `langchain` | Document chunking / splitting |
  535. | `KG_EXTRACTOR_BACKEND` | `llamaindex` \| `langchain` | KG extraction from chunks |
  536. | `RETRIEVAL_FUSION` | `llamaindex` \| `langchain` | Result fusion across retrievers |
  537. **KG extraction speed:** `KG_EXTRACTION_WORKERS` (default `4`) sets how many LLM calls the LlamaIndex KG extractors make at once. Extraction is most of an ingest's time and is mostly waiting on the LLM, so with a cloud LLM (OpenAI, Gemini, Anthropic, etc.) raising it to 8-16 makes KG extraction much faster, up to the provider's rate limits. With a local LLM (Ollama etc.) keep it at `4`. See [Knowledge Graph Extractors - Extraction Concurrency](docs/DATABASES/GRAPH-DATABASES/KNOWLEDGE-GRAPH-EXTRACTORS.md#extraction-concurrency-kg_extraction_workers).
  538. LangChain-only graph stores (ArangoDB, Apache AGE, HugeGraph, SurrealDB, TigerGraph, Cosmos Gremlin) auto-select `GRAPH_BACKEND=langchain`.
  539. **Complete Documentation:** [docs/ADVANCED/LANGCHAIN/LANGCHAIN-GRAPH-INTEGRATION.md](docs/ADVANCED/LANGCHAIN/LANGCHAIN-GRAPH-INTEGRATION.md)
  540. ## LLM and Embedding Configuration
  541. Set via `LLM_PROVIDER` and provider-specific environment variables.
  542. ### Supported LLM Providers
  543. 1. **OpenAI** - gpt-4o-mini (default), gpt-4o, gpt-4.1-mini, gpt-5-mini, etc.
  544. 2. **Ollama** - Local deployment (llama3.2, llama3.1, qwen2.5, gpt-oss, etc.)
  545. 3. **Azure OpenAI** - Azure-hosted OpenAI models
  546. 4. **Google Gemini** - gemini-2.5-flash, gemini-3-flash-preview, gemini-3.1-pro-preview, etc.
  547. 5. **Anthropic Claude** - claude-sonnet-4-5, claude-haiku-4-5, etc.
  548. 6. **Google Vertex AI** - Google Cloud-hosted Vertex AI Platform Gemini models
  549. 7. **Amazon Bedrock** - Amazon Nova, Titan, Anthropic Claude, Meta Llama, Mistral AI, etc.
  550. 8. **Groq** - Fast low-cost LPU inference, low latency: OpenAI GPT-OSS, Meta Llama (4, 3.3, 3.1), Qwen3, Kimi, etc.
  551. 9. **Fireworks AI** - More choices, fine-tuning: Meta, Qwen, Mistral AI, DeepSeek, OpenAI GPT-OSS, Kimi, GLM, MiniMax, etc.
  552. 10. **OpenAI-Compatible** (`openai_like`) - Any OpenAI-compatible endpoint (LM Studio, LocalAI, Llamafile, vLLM, etc.)
  553. 11. **OpenRouter** - 200+ models via unified API (openai/gpt-4o-mini, anthropic/claude, meta-llama, etc.)
  554. 12. **LiteLLM Proxy** - 100+ providers via LiteLLM proxy; sample config in `scripts/litellm_config.yaml`
  555. 13. **vLLM** - High-performance local inference server (Linux/macOS; use `openai_like` on Windows)
  556. ### LLM Provider Configuration
  557. See [docs/LLM/LLM-EMBEDDING-CONFIG.md](docs/LLM/LLM-EMBEDDING-CONFIG.md) for all 13 providers with detailed configuration examples.
  558. **OpenAI** (recommended):
  559. ```bash
  560. LLM_PROVIDER=openai
  561. OPENAI_API_KEY=your_api_key
  562. OPENAI_MODEL=gpt-4o-mini
  563. ```
  564. **Ollama** (local):
  565. ```bash
  566. LLM_PROVIDER=ollama
  567. OLLAMA_BASE_URL=http://localhost:11434
  568. OLLAMA_MODEL=gpt-oss:20b
  569. ```
  570. After the first AI query or graph ingest (once the LLM is loaded, not just the embedding model), `ollama ps` should show the LLM at `100% GPU` with context 8192; see [Ollama Configuration](docs/LLM/OLLAMA-CONFIGURATION.md).
  571. **Azure OpenAI**:
  572. ```bash
  573. LLM_PROVIDER=azure_openai
  574. AZURE_OPENAI_API_KEY=your_key
  575. AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
  576. AZURE_OPENAI_ENGINE=gpt-4o-mini
  577. ```
  578. ### Embedding Configuration
  579. Set `EMBEDDING_KIND` to choose the embedding provider — independent of the LLM provider. All 13 LLM providers are also supported as embedding providers. See [docs/LLM/LLM-EMBEDDING-CONFIG.md](docs/LLM/LLM-EMBEDDING-CONFIG.md) for all providers and options.
  580. **OpenAI**:
  581. ```bash
  582. EMBEDDING_KIND=openai
  583. OPENAI_EMBEDDING_MODEL=text-embedding-3-small
  584. OPENAI_API_KEY=your_api_key
  585. ```
  586. **Ollama** (local):
  587. ```bash
  588. EMBEDDING_KIND=ollama
  589. OLLAMA_EMBEDDING_MODEL=nomic-embed-text
  590. OLLAMA_BASE_URL=http://localhost:11434
  591. ```
  592. **Azure OpenAI**:
  593. ```bash
  594. EMBEDDING_KIND=azure_openai
  595. AZURE_EMBEDDING_MODEL=text-embedding-3-small
  596. AZURE_EMBEDDING_DEPLOYMENT=your_deployment_name
  597. AZURE_OPENAI_API_KEY=your_key
  598. AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
  599. ```
  600. **Common embedding dimensions:**
  601. - OpenAI: 1536 (text-embedding-3-small), 3072 (text-embedding-3-large)
  602. - Ollama: 384 (all-minilm), 768 (nomic-embed-text), 1024 (mxbai-embed-large)
  603. - Google: 768 (gemini-embedding-2-preview)
  604. - Bedrock: 1024 (amazon.titan-embed-text-v2:0)
  605. When switching embedding models, delete existing vector indexes. See [docs/DATABASES/VECTOR-DATABASES/VECTOR-DIMENSIONS.md](docs/DATABASES/VECTOR-DATABASES/VECTOR-DIMENSIONS.md) for cleanup instructions.
  606. ### Ollama Configuration
  607. When using Ollama, configure system-wide environment variables before starting the Ollama service:
  608. **Key requirements**:
  609. - Configure environment variables **system-wide** (not in Flexible GraphRAG `.env` file)
  610. - `OLLAMA_NUM_PARALLEL=4` for optimal performance (or 1-2 if resource constrained)
  611. - Leave `KG_EXTRACTION_WORKERS` at its default `4` in the Flexible GraphRAG `.env` (the 8-16 suggested for cloud LLMs would just queue up in Ollama)
  612. - Always restart Ollama service after changing environment variables
  613. See [docs/LLM/OLLAMA-CONFIGURATION.md](docs/LLM/OLLAMA-CONFIGURATION.md) for complete setup instructions including platform-specific steps and performance optimization.
  614. ## Prerequisites
  615. ### Required
  616. - Python 3.12, 3.13, or 3.14 (as specified in `pyproject.toml`)
  617. - UV package manager (for dependency management)
  618. - Node.js 22.x (for UI clients)
  619. - npm (package manager)
  620. - Search database: Elasticsearch or OpenSearch
  621. - Vector database: Qdrant (or other supported vector databases)
  622. - Property graph database: Neo4j (or other supported property graph databases) - unless using vector-only RAG
  623. - OpenAI with API key (recommended) or Ollama (for LLM processing)
  624. **Note**: The `docker/docker-compose.yaml` file can provide all these databases via Docker containers.
  625. ### Install
  626. ```bash
  627. cd flexible-graphrag
  628. uv pip install -e .
  629. ```
  630. ### Optional (see flexible-graphrag/pyproject.toml for all options)
  631. - **LangChain 1.x integration** — Optional peer stack alongside LlamaIndex (extras pin **`langchain>=1.0`** and the LangChain **1.x** line, not legacy 0.3):
  632. - `uv pip install -e ".[langchain]"` — core LC extras: property graph stores via `langchain-community` where supported, 10 vector stores, 3 search stores, RDF SPARQL retrieval, native LC LLM/embedding clients for all 13 providers, KG extraction via `langchain-experimental`, retrieval fusion
  633. - `uv pip install --override extras-overrides.txt -e ".[langchain,langchain-extras]"` — adds Neo4j (LC), PostgreSQL pgvector, ArcadeDB, ArangoDB, Cosmos Gremlin, HugeGraph, TigerGraph, and related dependencies (see `pyproject.toml` group `langchain-extras`)
  634. - **Apache AGE** — property graph via LangChain needs the separate **`age-extras`** group (BAEM1N `langchain-age` driver):
  635. ```bash
  636. uv pip install --override extras-overrides.txt -e ".[langchain,langchain-extras,age-extras]"
  637. python scripts/patch_langchain_age.py
  638. ```
  639. Run `patch_langchain_age.py` on **Python 3.14+** (required); on 3.12/3.13 it is harmless.
  640. - `uv pip install -e ".[spanner-extras]"` — adds Spanner support via `llama-index-spanner` and `langchain-google-spanner`. **Note:** `llama-index-spanner` declares `llama-index` (the meta-package) as a dependency, which `uv` will install. Uninstall it immediately after: `uv pip uninstall llama-index` — having both `llama-index` and `llama-index-core` installed simultaneously can cause version conflicts, as the meta-package pins versions of `llama-index-*` component packages that can clash with the versions already required by this project
  641. - SurrealDB — two-step install required (resolver conflict):
  642. ```bash
  643. uv pip install -e ".[surrealdb-extras]"
  644. uv pip install "surrealdb>=2.0" "langchain-core>=1.3"
  645. ```
  646. - **ArcadeDB embedded mode** (`uv pip install arcadedb-embedded>=26.3.2`) — runs ArcadeDB in-process; includes a bundled JVM, no separate Java install needed; latest release: 26.3.2
  647. - **Enterprise Repositories**:
  648. - Alfresco repository - only if using Alfresco data source
  649. - SharePoint - requires SharePoint access
  650. - Box - requires Box Business account (3 users minimum), API keys
  651. - CMIS-compliant repository (e.g., Alfresco) - only if using CMIS data source
  652. - **Cloud Storage** (requires accounts and API keys/credentials):
  653. - Amazon S3 - requires AWS account and access keys
  654. - Google Cloud Storage - requires GCP account and service account credentials
  655. - Google Drive - requires Google Cloud account and OAuth credentials or service account
  656. - Azure Blob Storage - requires Azure account and connection string or account keys
  657. - Microsoft OneDrive - requires OneDrive for Business (not personal OneDrive)
  658. - **Note**: SharePoint and OneDrive for Business are also available with a M365 Developer Program sandbox (with full Visual Studio annual subscription, not monthly).
  659. - **File Upload** (no account required):
  660. - Web interface with file dialog (drag & drop or click to select)
  661. - **Web Sources** (no account required):
  662. - Web pages, Wikipedia, YouTube - no accounts needed
  663. ## Setup
  664. ### 🐳 Docker Deployment
  665. Docker deployment offers multiple scenarios. Before deploying any scenario, set up your environment files:
  666. **Environment File Setup (Required for All Scenarios):**
  667. 1. **Backend Configuration** (`.env`):
  668. ```bash
  669. # Navigate to backend directory
  670. cd flexible-graphrag
  671. # Linux/macOS
  672. cp env-sample.txt .env
  673. # Windows Command Prompt
  674. copy env-sample.txt .env
  675. # Edit .env with your database credentials, API keys, and settings
  676. # Then return to project root
  677. cd ..
  678. ```
  679. 2. **Docker Configuration** (`docker/.env`):
  680. ```bash
  681. # Navigate to docker directory
  682. cd docker
  683. # Linux/macOS
  684. cp docker-env-sample.txt .env
  685. # Windows Command Prompt
  686. copy docker-env-sample.txt .env
  687. # Edit docker/.env for Docker-specific overrides (network addresses, service names)
  688. # Stay in docker directory for next steps
  689. ```
  690. ---
  691. #### Scenario A: Databases in Docker, App Standalone (Hybrid)
  692. **Configuration Setup:**
  693. ```bash
  694. # If not already in docker directory from previous step:
  695. # cd docker
  696. # Edit docker-compose.yaml to uncomment/comment services as needed
  697. # Scenario A setup in docker-compose.yaml:
  698. # Keep these services uncommented (default setup):
  699. - includes/neo4j.yaml
  700. - includes/qdrant.yaml
  701. - includes/elasticsearch-dev.yaml
  702. - includes/kibana-simple.yaml
  703. # Keep these services commented out:
  704. # - includes/app-stack.yaml # Must be commented out for Scenario A
  705. # - includes/proxy.yaml # Must be commented out for Scenario A
  706. # - All other services remain commented unless you want a different vector database,
  707. # graph database, OpenSearch for search, or Alfresco included
  708. ```
  709. **Deploy Services:**
  710. ```bash
  711. # From the docker directory
  712. docker-compose -f docker-compose.yaml -p flexible-graphrag up -d
  713. ```
  714. #### Scenario B: Full Stack in Docker (Complete)
  715. **Configuration Setup:**
  716. ```bash
  717. # If not already in docker directory from previous step:
  718. # cd docker
  719. # Edit docker-compose.yaml to uncomment/comment services as needed
  720. # Scenario B setup in docker-compose.yaml:
  721. # Keep these services uncommented:
  722. - includes/neo4j.yaml
  723. - includes/qdrant.yaml
  724. - includes/elasticsearch-dev.yaml
  725. - includes/kibana-simple.yaml
  726. - includes/app-stack.yaml # Backend and UI in Docker
  727. - includes/proxy.yaml # NGINX reverse proxy
  728. # Keep other services commented out unless you want a different vector database,
  729. # graph database, OpenSearch for search, or Alfresco included
  730. ```
  731. **Deploy Services:**
  732. ```bash
  733. # From the docker directory
  734. docker-compose -f docker-compose.yaml -p flexible-graphrag up -d
  735. ```
  736. **Scenario B Service URLs:**
  737. - **Angular UI**: http://localhost:8070/ui/angular/
  738. - **React UI**: http://localhost:8070/ui/react/
  739. - **Vue UI**: http://localhost:8070/ui/vue/
  740. - **Backend API**: http://localhost:8070/api/
  741. #### Other Deployment Scenarios
  742. **Scenario C: Fully Standalone** - Not using docker-compose at all
  743. - Standalone backend, standalone UIs, all databases running separately
  744. - Configure all database connections in `flexible-graphrag/.env`
  745. **Scenario D: Backend/UIs in Docker, Databases External**
  746. - Using docker-compose for backend and UIs (app-stack + proxy)
  747. - Some or all databases running separately (same docker-compose, other local Docker, cloud/remote servers)
  748. - Configure database connections in `docker/.env`: Backend in Docker reads this file
  749. - For databases in same docker-compose: Use service names (e.g., `neo4j:7687`, `qdrant:6333`)
  750. - For databases in other local Docker containers: Use `host.docker.internal:PORT`
  751. - For remote/cloud databases: Use actual hostnames/IPs
  752. **Scenario E: Mixed Docker/Standalone**
  753. - Standalone backend and UIs
  754. - Running some databases in Docker (local) and some outside (cloud, external servers)
  755. - Configure all database connections in `flexible-graphrag/.env`: Use `host.docker.internal:PORT` for locally-running Docker databases, use actual hostnames/IPs for remote Docker or non-Docker databases
  756. #### Docker Control and Configuration
  757. **Managing Docker services:**
  758. ```bash
  759. # Navigate to docker directory (if not already there)
  760. cd docker
  761. # Create and start services (recreates if configuration changed)
  762. docker-compose -f docker-compose.yaml -p flexible-graphrag up -d
  763. # Stop services (keeps containers)
  764. docker-compose -f docker-compose.yaml -p flexible-graphrag stop
  765. # Start stopped services
  766. docker-compose -f docker-compose.yaml -p flexible-graphrag start
  767. # Stop and remove services
  768. docker-compose -f docker-compose.yaml -p flexible-graphrag down
  769. # View logs
  770. docker-compose -f docker-compose.yaml -p flexible-graphrag logs -f
  771. # Restart after configuration changes
  772. docker-compose -f docker-compose.yaml -p flexible-graphrag down
  773. # Edit docker-compose.yaml, docker/.env, or includes/app-stack.yaml as needed
  774. docker-compose -f docker-compose.yaml -p flexible-graphrag up -d
  775. ```
  776. **Configuration:**
  777. - **Modular deployment**: Comment/uncomment services in `docker/docker-compose.yaml`
  778. - **Backend configuration** (Scenario B): Backend uses `flexible-graphrag/.env` with `docker/.env` for Docker-specific overrides (like using service names instead of localhost). No configuration needed in `app-stack.yaml`
  779. See [docker/README.md](./docker/README.md) for detailed Docker configuration.
  780. ### 🔧 Local Development Setup (Scenario A)
  781. **Note**: Skip this entire section if using Scenario B (Full Stack in Docker).
  782. #### Environment Configuration
  783. **Create environment file** (cross-platform):
  784. ```bash
  785. # Linux/macOS
  786. cp flexible-graphrag/env-sample.txt flexible-graphrag/.env
  787. # Windows Command Prompt
  788. copy flexible-graphrag\env-sample.txt flexible-graphrag\.env
  789. ```
  790. Edit `.env` with your database credentials and API keys.
  791. ### Python Backend Setup (Standalone)
  792. #### Option A — Install from PyPI package (Quickstart)
  793. ```bash
  794. # 1. Create and activate a virtual environment
  795. uv venv venv-3.13 --python 3.13
  796. venv-3.13\Scripts\Activate # Windows
  797. source venv-3.13/bin/activate # Linux/macOS
  798. # 2. Install flexible-graphrag
  799. uv pip install flexible-graphrag
  800. # 3. Optionally install ArcadeDB embedded mode support (includes bundled JVM, no Java install needed)
  801. uv pip install arcadedb-embedded>=26.3.2
  802. # 3a. Optional dependency groups, for example:
  803. uv pip install "flexible-graphrag[langchain]"
  804. # Other extras ([langchain-extras], [age-extras], overrides): see source README, Prerequisites > Optional.
  805. # 4. Create .env from the sample (copy from the source repo or download env-sample.txt)
  806. copy env-sample.txt .env # Windows
  807. cp env-sample.txt .env # Linux/macOS
  808. # Edit .env with your LLM API keys and database settings
  809. # 5. Start your databases (docker compose or standalone)
  810. docker compose -f docker/docker-compose.yml up -d
  811. # 6. Run the backend
  812. flexible-graphrag
  813. # or: uv run start.py
  814. ```
  815. #### Option B — Install from source (editable)
  816. 1. Navigate to the backend directory:
  817. ```bash
  818. cd flexible-graphrag
  819. ```
  820. 2. Create and activate a virtual environment, then install in editable mode:
  821. ```bash
  822. uv venv venv-3.13 --python 3.13
  823. venv-3.13\Scripts\Activate # Windows
  824. source venv-3.13/bin/activate # Linux/macOS
  825. uv pip install -e .
  826. # see flexible-graphrag/pyproject.toml for all options
  827. # --- Optional: dependency groups from pyproject.toml [project.optional-dependencies] ---
  828. # LangChain (peer framework; use overrides when combining with langchain-extras)
  829. uv pip install -e ".[langchain]"
  830. uv pip install --override extras-overrides.txt -e ".[langchain,langchain-extras]"
  831. uv pip install --override extras-overrides.txt -e ".[langchain,langchain-extras,age-extras]"
  832. python scripts/patch_langchain_age.py
  833. uv pip install --override extras-overrides.txt -e ".[surrealdb-extras]"
  834. uv pip install "surrealdb>=2.0" "langchain-core>=1.3"
  835. uv pip install --override extras-overrides.txt -e ".[spanner-extras]"
  836. uv pip uninstall llama-index
  837. # RDF extras (base install already includes rdflib/pyoxigraph; use these if you need the named groups)
  838. uv pip install -e ".[rdf]"
  839. uv pip install -e ".[rdf-full]"
  840. # Observability
  841. uv pip install -e ".[observability]"
  842. uv pip install -e ".[observability-openlit]"
  843. uv pip install -e ".[observability-dual]"
  844. # Development tests / tooling
  845. uv pip install -e ".[dev]"
  846. # Docling OCR backends (see DOCLING_OCR in env-sample)
  847. uv pip install -e ".[docling-ocr-easyocr]"
  848. uv pip install -e ".[docling-ocr-tesserocr]"
  849. uv pip install -e ".[docling-ocr-ocrmac]" # macOS only
  850. # Embedded ArcadeDB (not a bracket extra; bundled JVM)
  851. uv pip install arcadedb-embedded>=26.3.2
  852. ```
  853. **uv-managed venv** (alternative): change `managed = false` to `managed = true` in `pyproject.toml` `[tool.uv]` section, then just run `uv pip install -e .`.
  854. Notes: run only the optional lines you need. For **`age-extras`**, run **`patch_langchain_age.py`** on **Python 3.14+** (safe on 3.12/3.13). For **`surrealdb-extras`**, keep the follow-up **`surrealdb` / `langchain-core`** upgrades. For **`spanner-extras`**, **`uv pip uninstall llama-index`** removes the meta-package pulled in by **`llama-index-spanner`**. See **### Optional** under **Prerequisites** for context.
  855. **Windows Note**: If installation fails with "Microsoft Visual C++ 14.0 or greater is required" error, install [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) (required for compiling Docling dependencies). Select "Desktop development with C++" during installation.
  856. 3. Create a `.env` file by copying the sample and customizing:
  857. ```bash
  858. cp env-sample.txt .env # Linux/macOS
  859. copy env-sample.txt .env # Windows
  860. ```
  861. Edit `.env` with your specific configuration. See [docs/GETTING-STARTED/ENVIRONMENT-CONFIGURATION.md](docs/GETTING-STARTED/ENVIRONMENT-CONFIGURATION.md) for detailed setup guide.
  862. **Note**: The system requires Python 3.12, 3.13, or 3.14 as specified in `pyproject.toml` (requires-python = ">=3.12,<3.15"). Python 3.12 and 3.13 are fully tested. Python 3.14 works with the patches applied automatically in `main.py` at startup. Virtual environment management is controlled by `managed = false` in `pyproject.toml` `[tool.uv]` section (you control venv creation and naming).
  863. 4. Start the backend:
  864. ```bash
  865. flexible-graphrag # after uv pip install flexible-graphrag
  866. # or: uv run start.py # with source
  867. ```
  868. The backend will be available at `http://localhost:8000`.
  869. ### Frontend Setup (Standalone)
  870. **Standalone backend and frontend URLs:**
  871. - **Backend API**: http://localhost:8000 (FastAPI server)
  872. - **Angular**: http://localhost:4200 (npm start)
  873. - **React**: http://localhost:5174 (npm run dev)
  874. - **Vue**: http://localhost:3000 (npm run dev)
  875. Choose one of the following frontend options to work with:
  876. #### React Frontend
  877. 1. Navigate to the React frontend directory:
  878. ```bash
  879. cd flexible-graphrag-ui/frontend-react
  880. ```
  881. 2. Install Node.js dependencies (first time only):
  882. ```bash
  883. npm install
  884. ```
  885. 3. Start the development server (uses Vite):
  886. ```bash
  887. npm run dev
  888. ```
  889. The React frontend will be available at `http://localhost:5174`.
  890. #### Angular Frontend
  891. 1. Navigate to the Angular frontend directory:
  892. ```bash
  893. cd flexible-graphrag-ui/frontend-angular
  894. ```
  895. 2. Install Node.js dependencies (first time only):
  896. ```bash
  897. npm install
  898. ```
  899. 3. Start the development server (uses Angular CLI):
  900. ```bash
  901. npm start
  902. ```
  903. The Angular frontend will be available at `http://localhost:4200`.
  904. #### Vue Frontend
  905. 1. Navigate to the Vue frontend directory:
  906. ```bash
  907. cd flexible-graphrag-ui/frontend-vue
  908. ```
  909. 2. Install Node.js dependencies (first time only):
  910. ```bash
  911. npm install
  912. ```
  913. 3. Start the development server (uses Vite):
  914. ```bash
  915. npm run dev
  916. ```
  917. The Vue frontend will be available at `http://localhost:3000`.
  918. ## UI Usage
  919. The system provides a tabbed interface for document processing and querying. Follow these steps in order. See [docs/UI-GUIDE/UI-GUIDE.md](docs/UI-GUIDE/UI-GUIDE.md) for full details.
  920. ### 1. Sources Tab
  921. Configure your data source and select files for processing. The system supports **14 data sources**:
  922. **Detailed Configuration:**
  923. #### File Upload Data Source
  924. - **Select**: "File Upload" from the data source dropdown
  925. - **Add Files**:
  926. - **Drag & Drop**: Drag files directly onto the upload area
  927. - **Click to Select**: Click the upload area to open file selection dialog (supports multi-select)
  928. - **Note**: If you drag & drop new files after selecting via dialog, only the dragged files will be used
  929. - **Supported Formats**: PDF, DOCX, XLSX, PPTX, TXT, MD, HTML, CSV, PNG, JPG, and more
  930. - **Next Step**: Click "CONFIGURE PROCESSING →" to proceed to Processing tab
  931. #### Alfresco Repository
  932. - **Select**: "Alfresco Repository" from the data source dropdown
  933. - **Configure**:
  934. - Alfresco Base URL (e.g., `http://localhost:8080/alfresco`)
  935. - Username and password
  936. - Path (e.g., `/Sites/example/documentLibrary`)
  937. - **Next Step**: Click "CONFIGURE PROCESSING →" to proceed to Processing tab
  938. #### CMIS Repository
  939. - **Select**: "CMIS Repository" from the data source dropdown
  940. - **Configure**:
  941. - CMIS Repository URL (e.g., `http://localhost:8080/alfresco/api/-default-/public/cmis/versions/1.1/atom`)
  942. - Username and password
  943. - Folder path (e.g., `/Sites/example/documentLibrary`)
  944. - **Next Step**: Click "CONFIGURE PROCESSING →" to proceed to Processing tab
  945. **All Data Sources** (13 available):
  946. - **Web Sources**: Web Page, Wikipedia, YouTube
  947. - **Cloud Storage**: Amazon S3, Google Cloud Storage, Azure Blob Storage, Google Drive, Microsoft OneDrive
  948. - **Enterprise Repositories**: Alfresco, Microsoft SharePoint, Box, CMIS
  949. See the [Data Sources](#data-sources) section for complete details on all 14 sources.
  950. ### 2. Processing Tab
  951. Process your selected documents and monitor progress:
  952. - **Start Processing**: Click "START PROCESSING" to begin document ingestion
  953. - **Monitor Progress**: View real-time progress bars for each file
  954. - **Per-row columns**: With the incremental registry on, every data source gets **Search+Vector**, **Graph**, **RDF** (one **Graphs** column before v0.9.0) and **Auto Sync** checkbox columns (single-folder / bucket sources get one row for the whole datasource); unticking removes from those stores, ticking re-ingests. A v0.9.0+ backend splits **Graphs** into **Graph** and **RDF** columns. See [Tab 2 — Processing](docs/UI-GUIDE/UI-TAB-PROCESSING.md)
  955. - **File Management**: Click a row's **✕** to take it off the list (not from your system or the stores)
  956. - **Processing Pipeline**: Documents are processed through Docling conversion, vector indexing, and knowledge graph creation
  957. ### 3. Search Tab
  958. Perform searches on your processed documents. The header's **sign-in pill** signs you in to repositories and cloud services so results include only documents you may read, and the **Sources** chips limit a search to chosen sources (see [Sign-In and Permission-Aware Search](docs/DATA-SOURCES/SIGN-IN-GUIDE.md)):
  959. #### Hybrid Search
  960. - **Purpose**: Find and rank the most relevant document excerpts
  961. - **Usage**: Enter search terms or phrases (e.g., "machine learning algorithms", "financial projections")
  962. - **Action**: Click "SEARCH" button
  963. - **Results**: Ranked list of document excerpts with relevance scores and source information
  964. - **Best for**: Research, fact-checking, finding specific information across documents
  965. #### Q&A Query
  966. - **Purpose**: Get AI-generated answers to natural language questions
  967. - **Usage**: Enter natural language questions (e.g., "What are the main findings in the research papers?")
  968. - **Action**: Click "ASK" button
  969. - **Results**: AI-generated narrative answers that synthesize information from multiple documents
  970. - **Best for**: Summarization, analysis, getting overviews of complex topics
  971. ### 4. Chat Tab
  972. Interactive conversational interface for document Q&A:
  973. - **Chat Interface**:
  974. - **Your Questions**: Displayed on the right side vertically
  975. - **AI Answers**: Displayed on the left side vertically
  976. - **Usage**: Type questions and press Enter or click send
  977. - **Conversation History**: All questions and answers are preserved in the chat history
  978. - **Clear History**: Click "CLEAR HISTORY" button to start a new conversation
  979. - **Best for**: Iterative questioning, follow-up queries, conversational document exploration
  980. ### Testing Cleanup
  981. Between tests you can clean up data:
  982. - **Run `cleanup.py`**: Clears vector, graph, and search indexes in one step — run from the `flexible-graphrag` directory
  983. - **Vector Indexes**: See [docs/DATABASES/VECTOR-DATABASES/VECTOR-DIMENSIONS.md](docs/DATABASES/VECTOR-DATABASES/VECTOR-DIMENSIONS.md) for vector database cleanup instructions
  984. - **Graph Data**: See [docs/DATABASES/GRAPH-DATABASES/README-neo4j.md](docs/DATABASES/GRAPH-DATABASES/README-neo4j.md) for graph-related cleanup commands
  985. ## Langflow Visual Flows (Optional)
  986. Flexible GraphRAG ships **12 custom Langflow components** (implemented in **Python**) and four ready-made flows (ingest, search, AI query, and a combined query flow for the Langflow Playground). With `ENABLE_LANGFLOW_FLOWS=true`, the app runs its **ingest pipeline, hybrid search, and AI query through those visual flows** instead of calling the system directly — the flows execute the **same backend machinery** driven by your existing `.env` (all database, LLM/embedding, chunking, KG, RDF, and LlamaIndex/LangChain settings apply unchanged), so the pipeline becomes a flow you can customize. You can also just drag the components onto the Langflow canvas to build your own flows.
  987. The main requirement is a **separate venv for Langflow** — Langflow runs the flows' Python component code in its own process, and the backend app calls the **Langflow REST API** to run the two flows.
  988. **1. Langflow venv** — create it on **any Python 3.14 except 3.14.0** (3.14.4 and 3.14.5 both work), or 3.13. Use an explicit patch version: `uv venv --python 3.14.4` (or greater) — plain `uv venv --python 3.14` resolves to **3.14.0**, which has an OpenSSL bug that makes Langflow abort on startup. Then install Langflow and the backend (LlamaIndex + fuller LangChain), and run Langflow **from the `flexible-graphrag` backend dir**:
  989. ```powershell
  990. # --system-certs handles an SSL-inspecting corporate proxy.
  991. # On uv < 0.11 the flag is --native-tls (newer uv deprecates that name).
  992. uv pip install --system-certs langflow==1.11.2
  993. uv pip install --system-certs --override extras-overrides.txt -e ".[langchain,langchain-extras]"
  994. # (No PyJWT restore step needed: flexible-graphrag installs plain `nuxeo` +
  995. # `authlib` + `pyjwt[crypto]`, never `nuxeo[oauth2]`, so nothing evicts PyJWT.)
  996. # Run from the flexible-graphrag backend dir. PYTHONPATH must point at it so this repo's bundled
  997. # `langchain` package wins over the real one (search / AI query need it):
  998. $env:PYTHONPATH = (Get-Location).Path
  999. langflow run --port 7860 --log-level WARNING --log-file langflow.log
  1000. ```
  1001. Wait for Langflow to **fully start** — after the purple "Welcome to Langflow" box it prints `Launching Langflow...`; it's ready once that finishes. Then create an API key in the Langflow UI: **Settings → Langflow API Keys → Add New** (copy it — shown once).
  1002. **2. Backend venv** — set `ENABLE_LANGFLOW_FLOWS=true`, `LANGFLOW_URL=http://localhost:7860`, and `LANGFLOW_API_KEY=<the key from the UI>` in `.env`, then start the app as usual. (A no-API-key local shortcut and the underlying `LANGFLOW_SECRET_KEY` are covered in the docs' Authentication section.)
  1003. **Docker Compose alternative** — instead of the two-venv setup above, the compose stack runs the **backend in Docker** alongside an optional **Langflow container** (with the 12 components already bundled), which you enable by flipping `ENABLE_LANGFLOW_FLOWS: "true"` and uncommenting the Langflow include. Covered in detail in the doc link below.
  1004. For the full setup (both the two-venv and Docker Compose paths), configuration reference, flow customization, and the 12-component developer reference, see **[Langflow Integration](docs/GETTING-STARTED/LANGFLOW-INTEGRATION.md)** and **[Langflow Components](docs/DEVELOPER/DEVELOPER-LANGFLOW-COMPONENTS.md)**.
  1005. ## MCP Server Setup (Quickstart)
  1006. The MCP server (`flexible-graphrag-mcp`) is a lightweight standalone package that connects MCP clients (Claude Desktop, Cursor, etc.) to the Flexible GraphRAG backend via its REST API.
  1007. For full details see [`flexible-graphrag-mcp/README.md`](flexible-graphrag-mcp/README.md) and [`flexible-graphrag-mcp/QUICK-USAGE-GUIDE.md`](flexible-graphrag-mcp/QUICK-USAGE-GUIDE.md). For the full list of available MCP tools see [MCP Tools for Claude Desktop and Other MCP Clients](#mcp-tools-for-claude-desktop-and-other-mcp-clients) below.
  1008. ### Steps
  1009. 1. **First terminal — install and run the flexible-graphrag backend** (see [Python Backend Setup](#python-backend-setup-standalone) above) — it must be running on `http://localhost:8000`.
  1010. 2. **Second terminal — install and start the MCP server** in HTTP mode:
  1011. ```bash
  1012. uv venv venv-mcp --python 3.13
  1013. venv-mcp\Scripts\Activate # Windows
  1014. source venv-mcp/bin/activate # Linux/macOS
  1015. uv pip install flexible-graphrag-mcp
  1016. flexible-graphrag-mcp --http --port 3001
  1017. ```
  1018. 3. **Third terminal — test with MCP Inspector**:
  1019. ```bash
  1020. npx @modelcontextprotocol/inspector
  1021. ```
  1022. Open the URL printed in the console (token pre-filled), set transport to **Streamable HTTP**, URL to `http://localhost:3001/mcp`, then click **Connect**.
  1023. 4. **Use with Claude Desktop and other MCP clients** — see [`flexible-graphrag-mcp/README.md`](flexible-graphrag-mcp/README.md) for stdio transport config and client-specific setup.
  1024. ## MCP Tools for Claude Desktop and Other MCP Clients
  1025. The MCP server provides 12 tools for document intelligence workflows (14 with `MCP_BROWSER_SIGN_IN=true`). Search and query take `source_auth` (per-user permission filtering) and `sources`; see [Permission filtering: three levels](flexible-graphrag-mcp/README.md#permission-filtering-three-levels).
  1026. | Tool | Purpose | Usage |
  1027. |------|---------|-------|
  1028. | `get_system_status()` | System health and configuration | Verify setup and database connections |
  1029. | `ingest_documents()` | Bulk document processing | All sources support `skip_graph`; filesystem/Alfresco/CMIS use `paths`; Alfresco also supports `nodeDetails` list (14 sources have their own config: filesystem, repositories (Alfresco, SharePoint, Box, CMIS, Nuxeo), cloud storage, web) |
  1030. | `ingest_text(content, source_name)` | Custom text analysis | Analyze specific text content |
  1031. | `search_documents(query, top_k)` | Hybrid document retrieval | Find relevant document excerpts |
  1032. | `query_documents(query, top_k)` | AI-powered Q&A | Generate answers from document corpus |
  1033. | `list_sources()` | Source picker | Sources with documents in the stores, for search / query `sources` |
  1034. | `get_auth_requirements()` | Sign-in check | Which sign-ins the stored documents need |
  1035. | `sign_in(source, username, password)` | Repository sign-in | `alfresco`, `nuxeo`, `cmis` or `s3`; returns the credential for `source_auth` |
  1036. | `start_sign_in(source)` / `finish_sign_in(handoff_id)` | Browser sign-in | Microsoft 365, Box, Google, Azure Storage; only with `MCP_BROWSER_SIGN_IN=true` |
  1037. | `test_with_sample()` | System verification | Quick test with sample content |
  1038. | `check_processing_status(id)` | Async operation monitoring | Track long-running ingestion tasks |
  1039. | `get_python_info()` | Environment diagnostics | Debug Python environment issues |
  1040. | `health_check()` | Backend connectivity | Verify API server connection |
  1041. ### Client Support
  1042. - **Claude Desktop and other MCP clients**: Native MCP integration with stdio transport
  1043. - **MCP Inspector**: HTTP transport for debugging and development
  1044. - **Multiple Installation**: pipx (system-wide) or uvx (no-install) options
  1045. ## CocoIndex Integration
  1046. Flexible GraphRAG optionally uses a [CocoIndex](https://github.com/cocoindex-io/cocoindex)
  1047. pipeline (Rust engine) that **mixes CocoIndex and Flexible GraphRAG (LlamaIndex /
  1048. LangChain) sources, processing functions, and database targets** — while keeping the
  1049. same FastAPI REST/MCP/UI surface. Langflow "Flexible" components are separate from this.
  1050. ### Two incremental modes
  1051. Do not enable more than one of these orchestrators at once (startup skips / force-disables conflicts).
  1052. | | Default pipeline + auto-incremental | CocoIndex pipeline |
  1053. |---|---|---|
  1054. | **`ENABLE_INCREMENTAL_UPDATES`** | `true` | `false` (ignored / skipped if set — FG incremental uses `hybrid_system`, not CocoIndex) |
  1055. | **`PIPELINE_BACKEND`** | default (not `cocoindex`) | `cocoindex` |
  1056. | **`ENABLE_LANGFLOW_FLOWS`** | optional (`true` for Langflow flow mode on the default pipeline) | `false` (ignored / force-disabled — CocoIndex not supported in Langflow flows) |
  1057. | **Multi-source configs** | `datasource_config` rows (UI / MCP / REST) | Same — `datasource_config` rows (UI / MCP / REST) |
  1058. | **Change events** | Event detectors emit ADD / MODIFY / DELETE | Same detectors for flexible data sources |
  1059. | **File / doc tracking** | Postgres `document_state` rows | No `document_state` — CocoIndex LMDB + reconciler |
  1060. | **Who drives the pipeline** | FG incremental engine re-ingests via `hybrid_system` (default LI/LC pipeline) | CocoIndex bridge owns ingest + change processing (LMDB step memoization) |
  1061. **What you can mix in a CocoIndex pipeline:**
  1062. | Category | CocoIndex | Flexible GraphRAG |
  1063. |---|---|---|
  1064. | **Sources** | Native: `localfs`, S3, Azure Blob, Google Drive | 13 sources (9 with event detectors / auto-sync): filesystem, Alfresco, S3, Azure Blob, GCS, OneDrive, SharePoint, Google Drive, Box, CMIS, web, Wikipedia, YouTube |
  1065. | **Document-processor function** | — | One `parse_document` function: Docling, LlamaParse, or LiteParse |
  1066. | **Chunker / splitter functions** | `split_with_cocoindex` (`CHUNKER_BACKEND=cocoindex`) | `split_with_llamaindex` / `split_with_langchain` (`CHUNKER_BACKEND=llamaindex` / `langchain`) |
  1067. | **Embedding providers** | `COCOINDEX_EMBEDDING_KIND=sentence_transformer` or `litellm` (in-process; no LiteLLM proxy) | All FG kinds via `EMBEDDING_KIND` (or other `COCOINDEX_EMBEDDING_KIND` values): OpenAI, Ollama, Azure, Google, Vertex, Bedrock, Fireworks, OpenAI-like, LiteLLM |
  1068. | **LLM providers** | — | Same `LLM_PROVIDER` list for **KG extraction** and **AI query / chat**. Hybrid search is retrieval (vector / BM25 / graph); an LLM is used only when a text-to-query graph retriever is in the fusion mix. Framework for KG: `KG_EXTRACTOR_BACKEND=llamaindex` → LI LLM + extractors; `langchain` → LC LLM + `LLMGraphTransformer` |
  1069. | **KG extraction functions** | Not used — CocoIndex’s own extractors do not produce multi-label Neo4j-style entity graphs | LlamaIndex Schema / Dynamic / Simple path extractors; LangChain `LLMGraphTransformer` — ontology-guided multi-label graphs for both CocoIndex native and Flexible (LI/LC) property-graph backends |
  1070. | **Vector targets** | Native: Qdrant, LanceDB, Postgres (pgvector) | All 10 vector stores via adapters |
  1071. | **Property graph targets** | Native: Neo4j, FalkorDB, SurrealDB | All 15 property graph databases via adapters |
  1072. | **RDF targets** | — | 4 RDF triple stores (Fuseki, GraphDB, Oxigraph, Neptune RDF) |
  1073. | **Search targets** | — | 3 search stores (Elasticsearch, OpenSearch, BM25) |
  1074. **Why use it (mixed pipeline):**
  1075. - **Broader Flexible GraphRAG stack in a CocoIndex flow** — more sources (incl. event detectors), more property-graph and vector databases, plus RDF, search, and the existing UI / REST / MCP — not only CocoIndex’s native connectors.
  1076. - **Rust engine** — CocoIndex’s core is Rust; orchestration and reconciler are built for high-throughput incremental transforms.
  1077. - **Incremental cost control (both modes)** — with the **default** pipeline + `ENABLE_INCREMENTAL_UPDATES` / event detectors, unchanged files never re-enter ingest. With **CocoIndex**, detectors still gate file-level work, and LMDB also memoizes parse / embed / KG by step input so unchanged chunks skip those calls if content is re-seen.
  1078. - **Field-specific sources and targets (custom code)** — CocoIndex can map individual source fields and write field-level target rows with custom transforms; the default FG mixed pipeline stays document-oriented.
  1079. - **Granularity** — FG detectors are **file / document** level (ADD / MODIFY / DELETE). CocoIndex step memoization is **chunk / content** level inside the pipeline. Page / section / field units need custom CocoIndex coding — not the default FG shape.
  1080. - **Standalone** — same `cocoindex_integration/pipeline/app.py` can run outside the FastAPI server for custom mixed apps.
  1081. **To enable:** add `PIPELINE_BACKEND=cocoindex` to `.env` and install with
  1082. `uv pip install -e ".[cocoindex]"`. All REST/MCP/UI endpoints remain unchanged.
  1083. **Worked example:** [`examples/cocoindex/meeting_notes_graph_any/`](examples/cocoindex/meeting_notes_graph_any/README.md)
  1084. ports CocoIndex's own `meeting_notes_graph_neo4j` example to run against **any** of the 15
  1085. property graph databases and **any** of the 10 auto-sync data sources. It shows a custom
  1086. `KGExtractor` (with untagged documents delegated back to the built-in extractor), entity
  1087. resolution, and three ways to run the same extractor — a short app, the standard pipeline, and
  1088. the server for UI-driven use.
  1089. See [CocoIndex Integration](docs/GETTING-STARTED/COCOINDEX-INTEGRATION.md) and
  1090. [CocoIndex Configuration](docs/CONFIGURATION/CONFIG-COCOINDEX.md) for details, or the
  1091. [CocoIndex Developer Guide](docs/DEVELOPER/DEVELOPER-COCOINDEX.md) to build on it.
  1092. ---
  1093. ## Backend REST API
  1094. The FastAPI backend provides the following REST API endpoints:
  1095. **Base URL**: `http://localhost:8000/api/`
  1096. **System**
  1097. | Endpoint | Method | Purpose |
  1098. |---|---|---|
  1099. | `/api/health` | GET | Health check — verify backend is running |
  1100. | `/api/status` | GET | System status and configuration (databases, LLM, feature flags) |
  1101. | `/api/info` | GET | System information and package versions |
  1102. | `/api/python-info` | GET | Python environment diagnostics |
  1103. | `/api/ui-defaults` | GET | Where the UIs' Deep, Graph and RDF checkboxes start, and which graph columns to show (v0.9.0+) |
  1104. **Ingestion**
  1105. | Endpoint | Method | Purpose |
  1106. |---|---|---|
  1107. | `/api/ingest` | POST | Ingest documents from a data source (`filesystem`, `s3`, `web`, `cmis`, ...) |
  1108. | `/api/upload` | POST | Upload files directly for processing |
  1109. | `/api/ingest-text` | POST | Ingest raw text content |
  1110. | `/api/test-sample` | POST | Test the system with built-in sample content |
  1111. | `/api/cleanup-uploads` | POST | Remove temporarily uploaded files |
  1112. **Async Processing**
  1113. | Endpoint | Method | Purpose |
  1114. |---|---|---|
  1115. | `/api/processing-status/{id}` | GET | Poll status of an async ingestion operation |
  1116. | `/api/processing-events/{id}` | GET | Server-Sent Events stream for real-time progress |
  1117. | `/api/cancel-processing/{id}` | POST | Cancel an ongoing processing operation |
  1118. **Search & Query**
  1119. | Endpoint | Method | Purpose |
  1120. |---|---|---|
  1121. | `/api/search` | POST | Hybrid search — returns ranked document excerpts |
  1122. | `/api/query` | POST | AI-powered Q&A — generates an answer from the document corpus |
  1123. **Graph**
  1124. | Endpoint | Method | Purpose |
  1125. |---|---|---|
  1126. | `/api/graph` | GET | Graph database status and node/relationship counts (Neo4j: live Cypher counts; other LC-backed stores: counts via `lc_graph.query()` where supported; remaining stores: status + dashboard URL) |
  1127. | `/api/graph/query` | POST | Execute a native graph query against the configured store — Cypher (Neo4j, Memgraph, FalkorDB, ArcadeDB, Ladybug, Apache AGE), AQL (ArangoDB), SurrealQL (SurrealDB), Gremlin (Cosmos), GSQL (TigerGraph), openCypher (Neptune/Analytics), GQL (Spanner), SPARQL fallback for RDF-only |
  1128. **RDF / Ontology** *(when `RDF_GRAPH_DB` is configured)*
  1129. | Endpoint | Method | Purpose |
  1130. |---|---|---|
  1131. | `/api/rdf/query/sparql` | POST | Execute a SPARQL query against the configured RDF store |
  1132. | `/api/rdf/ontology/info` | GET | Return loaded ontology entity and relation type lists |
  1133. | `/api/rdf/ontology/upload` | POST | Upload a new ontology file at runtime |
  1134. | `/api/rdf/rdf-store/list` | GET | List registered RDF stores |
  1135. | `/api/rdf/rdf-store/connect` | POST | Register an additional RDF store at runtime |
  1136. | `/api/rdf/rdf-store/{name}` | DELETE | Deregister an RDF store |
  1137. | `/api/rdf/export/rdf` | POST | Export knowledge graph as RDF *(501 stub — not yet implemented)* |
  1138. **Interactive API Documentation** (requires running backend):
  1139. | UI | URL | Notes |
  1140. |---|---|---|
  1141. | **Swagger UI** | http://localhost:8000/docs | Try endpoints, inspect schemas, submit requests |
  1142. | **ReDoc** | http://localhost:8000/redoc | Cleaner read-only reference view |
  1143. See [docs/DEVELOPER/REST-API.md](docs/DEVELOPER/REST-API.md) for the full endpoint reference with request/response examples.
  1144. ## Full-Stack Debugging (Standalone Mode)
  1145. VS Code launch configurations, backend/frontend debugging, log levels, and MCP Inspector setup — see [docs/DEVELOPER/DEVELOPER-FULL-STACK-DEBUGGING.md](docs/DEVELOPER/DEVELOPER-FULL-STACK-DEBUGGING.md).
  1146. ## Observability and Monitoring
  1147. Flexible GraphRAG includes comprehensive observability features for production monitoring:
  1148. - **OpenTelemetry Integration**: Industry-standard instrumentation with automatic LlamaIndex tracing
  1149. - **Distributed Tracing**: Jaeger UI for visualizing complete request flows
  1150. - **Metrics Collection**: Prometheus for RAG-specific metrics (retrieval/LLM latency, token usage, entity/relation counts)
  1151. - **Visualization**: Grafana dashboards with pre-configured RAG metrics panels
  1152. - **Dual Mode Support**: OpenInference (LlamaIndex) + OpenLIT (optional) as dual OTLP producers
  1153. - **Custom Instrumentation**: Decorators for adding tracing to custom code
  1154. ### Quick Start
  1155. 1. Install observability dependencies (optional):
  1156. ```bash
  1157. cd flexible-graphrag
  1158. uv pip install -e ".[observability-dual]" # OpenInference (LlamaIndex + LangChain) + OpenLIT (recommended)
  1159. # Or combine with dev tools: uv pip install -e ".[observability-dual,dev]"
  1160. ```
  1161. 2. Enable in `.env`:
  1162. ```bash
  1163. ENABLE_OBSERVABILITY=true
  1164. OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
  1165. OBSERVABILITY_BACKEND=both # openinference, openlit, or both (recommended)
  1166. ```
  1167. 3. Start observability stack:
  1168. ```bash
  1169. cd docker
  1170. # Uncomment observability.yaml in docker-compose.yaml first
  1171. docker-compose -f docker-compose.yaml -p flexible-graphrag up -d
  1172. ```
  1173. 4. Access dashboards:
  1174. - **Grafana**: http://localhost:3009 (admin/admin) - RAG metrics dashboards
  1175. - **Jaeger**: http://localhost:16686 - Distributed tracing
  1176. - **Prometheus**: http://localhost:9090 - Raw metrics
  1177. <p align="center">
  1178. <a href="./screen-shots/observability/observability-grafana-prometheus-jaeger-ui.png">
  1179. <img src="./screen-shots/observability/observability-grafana-prometheus-jaeger-ui.png" alt="Observability Dashboard" width="700">
  1180. </a>
  1181. </p>
  1182. See [docs/DEVELOPER/OBSERVABILITY/OBSERVABILITY.md](docs/DEVELOPER/OBSERVABILITY/OBSERVABILITY.md) for complete setup, custom instrumentation, and production best practices.
  1183. ## Project Structure
  1184. - `/flexible-graphrag`: Python FastAPI backend (`pyproject.toml`; `start.py` is the `flexible-graphrag` command)
  1185. - Top-level modules: `main.py` (REST API), `backend.py` (shared by the API and MCP), `config.py` (settings),
  1186. `hybrid_system.py` (ingest + hybrid search), `retriever_setup.py` and `query_engine.py` (retrieval and answers),
  1187. `incremental_system.py` (auto-sync startup), `remote_backends.py` (federated backends), `api_security.py`
  1188. (API token, desktop app ingest folders), `permission_filter.py` and `scope_filter.py` (per-user and scoped
  1189. search), `graph_view.py` (Graph tab), `flow_service.py` (Langflow flow mode), and smaller helpers
  1190. - `adapters/`: framework-neutral interfaces and factories (graph, RDF, vector, search, LLM, chunking, KG extraction)
  1191. - `llamaindex/`, `langchain/`: the two peer frameworks' implementations: property graph, RDF, vector and search
  1192. store adapters, LLM and embedding providers, chunkers, KG extractors, retrievers
  1193. - `sources/`: data source connectors; `browse/`: the **Browse...** source browsers (`POST /api/browse`)
  1194. - `ingest/`, `process/`, `stores/`: ingest steps, document parsing (Docling / LlamaParse / LiteParse) and KG
  1195. extraction, index managers
  1196. - `incremental_updates/`: auto-sync engine (change detectors, orchestrator, state stores: PostgreSQL, OpenArch, SQLite)
  1197. - `rdf/`: ontologies, KG-to-RDF conversion, SPARQL, RDF stores
  1198. - `openarch/`: Hyland OpenArch as the search + vector store and as the auto-sync state store
  1199. - `cocoindex_integration/`: optional CocoIndex pipeline backend (`PIPELINE_BACKEND=cocoindex`)
  1200. - `langflow_components/`: the 12 custom Langflow components
  1201. - `observability/`: OpenTelemetry tracing and Prometheus metrics
  1202. - `/flows`: Langflow flow JSONs the app uploads in flow mode
  1203. - `/flexible-graphrag-mcp`: standalone MCP server (`main.py`; calls the REST API, 3 dependencies)
  1204. - `/flexible-graphrag-ui`: frontends
  1205. - `frontend-react/`, `frontend-vue/`, `frontend-angular/`: the 5-tab UIs (Vite, Vite, Angular CLI)
  1206. - `frontend-angular/projects/flexible-graphrag-angular-lib/`: the shared Angular UI library, on npm as
  1207. `@flexible-graphrag/angular-ui` (used by this app, KG Spaces ACA and KG Spaces for Nuxeo)
  1208. - `graph-view/`: the Graph tab's graph viewer, on npm as `@flexible-graphrag/graph-view` (used by all three UIs)
  1209. - `desktop/`: the Tauri desktop app around the React, Vue or Angular UI (see its `README.md`)
  1210. - `/docker`: Docker Compose stack (`docker-compose.yaml` with modular `includes/`, nginx config)
  1211. - `/docs`: documentation site ([Zensical](https://zensical.org/), nav in `zensical.toml`)
  1212. - `/schemas`: ontology and schema examples (`.env` paths use `../schemas/...`)
  1213. - `/sample-docs`: sample documents to ingest
  1214. - `/scripts`: `cleanup.py` (clears the configured stores), `rdf_cleanup.py`, OpenSearch pipeline setup, `incremental/`
  1215. (sync-now and refresh-interval scripts)
  1216. - `/tests`: pytest unit tests; `integration/` (tests against running databases and sources, `run_matrix.py`) and
  1217. `integration/autosync/` (auto-sync harness, `run_autosync.py`)
  1218. - `/examples`: standalone examples: observability, RDF / ontology, CocoIndex (meeting notes on any graph store)
  1219. ## License
  1220. This project is licensed under the terms of the Apache License 2.0. See the [LICENSE](LICENSE) file for details.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.