doc-intel-assistant
Cited Q&A over scanned Telugu and English welfare-scheme documents: OCR, hybrid BM25 + vector RAG, a LangGraph eligibility agent, an MCP server, evaluation in CI and a QLoRA fine-tuning comparison.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/shyam2119--doc-intel-assistant)HTML
<a href="/listings/shyam2119--doc-intel-assistant"><img src="/api/listings/shyam2119--doc-intel-assistant/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
python -m venv .venv .venv\Scripts\activate # Windows (macOS/Linux: source .venv/bin/activate) pip install -r requirements-dev.txt python scripts/make_demo_corpus.py # only if data/demo/docs is empty python -m docintel ingest data/demo/docs python -m docintel ask "What is the hostel allowance for degree students?" python -m docintel serve # open http://localhost:8000 (API docs at /docs) pytest -q
OCR needs Tesseract: - **Windows:** install the UB-Mannheim build. Tick **Telugu** under "Additional language data", then add `C:\Program Files\Tesseract-OCR` to PATH. - **Linux:** run `sudo apt install tesseract-ocr tesseract-ocr-tel poppler-utils`.
Scanned PDFs on Windows also need Poppler on PATH.
Better models
Copy `.env.example` to `.env` and choose:
| Goal | Setting | |---|---| | Telugu and English questions over both languages | `DOCINTEL_EMBEDDER=st:intfloat/multilingual-e5-small` (needs `pip install -r requirements-gpu.txt`) | | Free local LLM on your GPU | Install [Ollama](https://ollama.com), run `ollama pull qwen2.5:3b`, then set `DOCINTEL_LLM_PROVIDER=ollama` | | Hosted LLM | `DOCINTEL_LLM_PROVIDER=gemini` plus `GEMINI_API_KEY`, or `openai` with any OpenAI-compatible endpoint |
After changing the embedder, re-run `ingest`, then `python -m docintel calibrate` to pick the "not found" thresholds.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 155 lines
- # Document Intelligence Assistant
- Ask questions about government welfare-scheme documents, in Telugu or English, and get answers that cite the exact page. Scanned PDFs are read with OCR. A second mode checks whether a person qualifies for each scheme and explains why, rule by rule.
- 
- ## What it does
- - **Reads messy documents.** It reads text PDFs, scanned PDFs and photos (Tesseract OCR, English and Telugu) and plain text. Repeated headers and stamps are removed before indexing.
- - **Finds the right page.** Hybrid retrieval combines BM25 for exact words, amounts and scheme names with vector search for meaning, merged with Reciprocal Rank Fusion.
- - **Answers with citations or says "not found".** The model may only use the retrieved passages. Every fact carries a `[n]` citation that points at a file and page. Citations that point nowhere are removed. If retrieval is too weak, the system refuses before calling the model.
- - **Checks eligibility with an agent** (stage 2). A LangGraph workflow reads a free-text description of a person and checks each scheme's rules. It answers *eligible*, *not eligible* or *need info*, and asks follow-up questions. The rules are JSON that cites its source page. Verdicts come from code, not from the model, so they are repeatable.
- - **Measures itself.** It includes an evaluation set, a threshold calibration command, 100 agent test cases, and quality gates in CI that fail the build if accuracy drops.
- - **Compares models** (stage 3). One script compares an extractive baseline, a QLoRA fine-tuned 1.5B–3B open model and hosted API models on the same questions.
- - **Ships as a service.** It has a FastAPI backend, a web UI, SQL query logging with feedback and stats, a Dockerfile, docker-compose with Postgres, GitHub Actions CI, and an MCP server so other AI agents can use its tools.
- 
- ## Results so far
- The numbers below are on the **fictional demo corpus** in `data/demo`. That corpus has four made-up schemes: two text PDFs, one scanned PDF and one Telugu document. There are 39 questions, including 8 that the documents cannot answer. The defaults were used: TF-IDF embeddings and the extractive answerer, with no API key. Reproduce them with `python -m docintel eval`.
- | Metric | Extractive baseline |
- |---|---|
- | Retrieval hit@5 (gold page in top 5) | 93.5% (100% for same-language questions) |
- | Answer accuracy (all required facts present) | 67.7% |
- | Citation accuracy (answer cites the gold page) | 86.7% |
- | Correct refusals on unanswerable questions | 50.0% |
- | Wrong refusals on answerable questions | 3.2% |
- | Eligibility agent, 100 cases | 100% verdicts, 100% field extraction |
- The baseline is weakest at refusing near-miss questions, such as "interest rate on a Kisan Sahaya crop loan". An LLM that may answer `NOT_FOUND` is expected to fix that, and stage 3 measures it. The two English-question/Telugu-document questions fail with TF-IDF by design. They need the multilingual embedder (`st:intfloat/multilingual-e5-small`).
- The 100 agent cases were written together with the parser, so they guard against regressions. They are not a measure of how well it generalises.
- **Still to do:** run the evaluation on real scheme documents (see [docs/REAL_DATA.md](docs/REAL_DATA.md)), and run the stage 3 fine-tune on a GPU and fill in the comparison table. No numbers are claimed for either yet.
- ## Quick start
- ```bash
- python -m venv .venv
- .venv\Scripts\activate # Windows (macOS/Linux: source .venv/bin/activate)
- pip install -r requirements-dev.txt
- python scripts/make_demo_corpus.py # only if data/demo/docs is empty
- python -m docintel ingest data/demo/docs
- python -m docintel ask "What is the hostel allowance for degree students?"
- python -m docintel serve # open http://localhost:8000 (API docs at /docs)
- pytest -q
- ```
- OCR needs Tesseract:
- - **Windows:** install the UB-Mannheim build. Tick **Telugu** under "Additional language data", then add `C:\Program Files\Tesseract-OCR` to PATH.
- - **Linux:** run `sudo apt install tesseract-ocr tesseract-ocr-tel poppler-utils`.
- Scanned PDFs on Windows also need Poppler on PATH.
- ### Better models
- Copy `.env.example` to `.env` and choose:
- | Goal | Setting |
- |---|---|
- | Telugu and English questions over both languages | `DOCINTEL_EMBEDDER=st:intfloat/multilingual-e5-small` (needs `pip install -r requirements-gpu.txt`) |
- | Free local LLM on your GPU | Install [Ollama](https://ollama.com), run `ollama pull qwen2.5:3b`, then set `DOCINTEL_LLM_PROVIDER=ollama` |
- | Hosted LLM | `DOCINTEL_LLM_PROVIDER=gemini` plus `GEMINI_API_KEY`, or `openai` with any OpenAI-compatible endpoint |
- After changing the embedder, re-run `ingest`, then `python -m docintel calibrate` to pick the "not found" thresholds.
- ## Commands
- | Command | What it does |
- |---|---|
- | `python -m docintel ingest <folder>` | Index every PDF, image and text file in a folder |
- | `python -m docintel ask "<question>"` | Answer one question with citations |
- | `python -m docintel eval [file]` | Retrieval and answer metrics; writes `eval/results/<model>.md` |
- | `python -m docintel calibrate` | Choose the refusal thresholds from the evaluation set |
- | `python -m docintel agent --text "..."` | Run the eligibility agent |
- | `python -m docintel agent-eval` | Score the agent on 100 test cases |
- | `python -m docintel serve` | API and web UI on port 8000 |
- | `python -m docintel mcp` | MCP server (stdio) exposing `search_documents`, `ask_documents`, `check_eligibility`, `list_schemes` |
- ## API
- | Method | Path | Purpose |
- |---|---|---|
- | GET | `/health` | Status, number of passages, models in use |
- | GET / POST | `/documents` | List documents, or upload one (≤ 25 MB) |
- | POST | `/ask` | `{"question": "..."}` returns an answer, citations, latency and retrieval details |
- | POST | `/feedback` | `{"query_id": 1, "helpful": true}` |
- | GET | `/stats` | Per-model question count, latency, refusal rate and feedback (SQL `GROUP BY`) |
- | POST | `/agent/eligibility` | `{"text": "...", "profile": {...}}` returns per-scheme verdicts, reasons and follow-up questions |
- ## How it works
- ```
- file ─▶ loader ─(no text layer?)─▶ OCR ─▶ boilerplate removal ─▶ page-aware chunks
- │
- BM25 index ◀─────────────┼────────▶ vector index
- └──────── RRF fusion ┘
- question ─▶ hybrid search ─▶ "enough evidence?" ─no─▶ "not found"
- │ yes
- ▼
- numbered passages ─▶ LLM ─▶ keep valid [n] citations ─▶ answer + pages
- ```
- Eligibility agent (LangGraph):
- ```
- parse_profile ─▶ select_schemes ─▶ check_eligibility ─┬─▶ explain (some verdicts known)
- └─▶ ask_follow_up (every scheme lacks facts)
- ```
- [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md) explains each design decision and the trade-offs.
- ## Stage 3: fine-tuning on an 8 GB laptop GPU
- ```bash
- pip install torch --index-url https://download.pytorch.org/whl/cu128 # RTX 50-series needs CUDA 12.8 builds
- pip install -r finetune/requirements.txt
- # 1. a teacher model writes Q&A pairs for each passage (questions close to eval questions are dropped)
- set DOCINTEL_LLM_PROVIDER=gemini & set GEMINI_API_KEY=...
- python finetune/build_dataset.py --per-chunk 4
- # 2. QLoRA: 4-bit base model, LoRA r=16 on all projections, loss on answer tokens only
- python finetune/train_qlora.py --model Qwen/Qwen2.5-1.5B-Instruct --epochs 2
- # 3. same questions, same retrieval, different generators
- set DOCINTEL_HF_ADAPTER=finetune/out/adapter
- python finetune/compare.py --providers extractive hf_local ollama gemini --price gemini=0.35
- ```
- ## Project layout
- ```
- docintel/
- ingest/ loader (PDF, OCR, text), sentence splitting, chunking, boilerplate removal
- index/ BM25 + embeddings + RRF hybrid index
- generate/ prompt, LLM providers, citation checking, refusal gate
- agent/ rules engine, free-text profile parser, tools, LangGraph graph
- eval/ answer evaluation, threshold calibration, agent evaluation
- api.py FastAPI app db.py SQLAlchemy models and stats
- mcp_server.py static/index.html web UI
- finetune/ dataset builder, QLoRA trainer, model comparison
- data/demo/ fictional documents and their rules
- eval/ question sets and agent cases
- tests/ 42 tests, including evaluation quality gates
- ```
- ## Limits
- - The demo documents are fictional. Real government PDFs have tables, two columns and stamps that this pipeline will handle less well. That is why the real-data evaluation matters.
- - Eligibility rules are written by hand from the documents, one JSON file per scheme. Extracting them with an LLM is possible, but every rule would still need a human check.
- - The agent's verdict is guidance only. The department makes the final decision.
- ## License
- MIT
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.