Skip to content
Back to search

doc-intel-assistant

Cited Q&A over scanned Telugu and English welfare-scheme documents: OCR, hybrid BM25 + vector RAG, a LangGraph eligibility agent, an MCP server, evaluation in CI and a QLoRA fine-tuning comparison.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/shyam2119--doc-intel-assistant/badge.svg)](/listings/shyam2119--doc-intel-assistant)

HTML

<a href="/listings/shyam2119--doc-intel-assistant"><img src="/api/listings/shyam2119--doc-intel-assistant/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

python -m venv .venv
.venv\Scripts\activate            # Windows  (macOS/Linux: source .venv/bin/activate)
pip install -r requirements-dev.txt
python scripts/make_demo_corpus.py   # only if data/demo/docs is empty
python -m docintel ingest data/demo/docs
python -m docintel ask "What is the hostel allowance for degree students?"
python -m docintel serve             # open http://localhost:8000  (API docs at /docs)
pytest -q

OCR needs Tesseract: - **Windows:** install the UB-Mannheim build. Tick **Telugu** under "Additional language data", then add `C:\Program Files\Tesseract-OCR` to PATH. - **Linux:** run `sudo apt install tesseract-ocr tesseract-ocr-tel poppler-utils`.

Scanned PDFs on Windows also need Poppler on PATH.

Better models

Copy `.env.example` to `.env` and choose:

| Goal | Setting | |---|---| | Telugu and English questions over both languages | `DOCINTEL_EMBEDDER=st:intfloat/multilingual-e5-small` (needs `pip install -r requirements-gpu.txt`) | | Free local LLM on your GPU | Install [Ollama](https://ollama.com), run `ollama pull qwen2.5:3b`, then set `DOCINTEL_LLM_PROVIDER=ollama` | | Hosted LLM | `DOCINTEL_LLM_PROVIDER=gemini` plus `GEMINI_API_KEY`, or `openai` with any OpenAI-compatible endpoint |

After changing the embedder, re-run `ingest`, then `python -m docintel calibrate` to pick the "not found" thresholds.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 155 lines
  1. # Document Intelligence Assistant
  2. Ask questions about government welfare-scheme documents, in Telugu or English, and get answers that cite the exact page. Scanned PDFs are read with OCR. A second mode checks whether a person qualifies for each scheme and explains why, rule by rule.
  3. ![Asking a question](docs/img/ask.png)
  4. ## What it does
  5. - **Reads messy documents.** It reads text PDFs, scanned PDFs and photos (Tesseract OCR, English and Telugu) and plain text. Repeated headers and stamps are removed before indexing.
  6. - **Finds the right page.** Hybrid retrieval combines BM25 for exact words, amounts and scheme names with vector search for meaning, merged with Reciprocal Rank Fusion.
  7. - **Answers with citations or says "not found".** The model may only use the retrieved passages. Every fact carries a `[n]` citation that points at a file and page. Citations that point nowhere are removed. If retrieval is too weak, the system refuses before calling the model.
  8. - **Checks eligibility with an agent** (stage 2). A LangGraph workflow reads a free-text description of a person and checks each scheme's rules. It answers *eligible*, *not eligible* or *need info*, and asks follow-up questions. The rules are JSON that cites its source page. Verdicts come from code, not from the model, so they are repeatable.
  9. - **Measures itself.** It includes an evaluation set, a threshold calibration command, 100 agent test cases, and quality gates in CI that fail the build if accuracy drops.
  10. - **Compares models** (stage 3). One script compares an extractive baseline, a QLoRA fine-tuned 1.5B–3B open model and hosted API models on the same questions.
  11. - **Ships as a service.** It has a FastAPI backend, a web UI, SQL query logging with feedback and stats, a Dockerfile, docker-compose with Postgres, GitHub Actions CI, and an MCP server so other AI agents can use its tools.
  12. ![Eligibility check](docs/img/eligibility.png)
  13. ## Results so far
  14. The numbers below are on the **fictional demo corpus** in `data/demo`. That corpus has four made-up schemes: two text PDFs, one scanned PDF and one Telugu document. There are 39 questions, including 8 that the documents cannot answer. The defaults were used: TF-IDF embeddings and the extractive answerer, with no API key. Reproduce them with `python -m docintel eval`.
  15. | Metric | Extractive baseline |
  16. |---|---|
  17. | Retrieval hit@5 (gold page in top 5) | 93.5% (100% for same-language questions) |
  18. | Answer accuracy (all required facts present) | 67.7% |
  19. | Citation accuracy (answer cites the gold page) | 86.7% |
  20. | Correct refusals on unanswerable questions | 50.0% |
  21. | Wrong refusals on answerable questions | 3.2% |
  22. | Eligibility agent, 100 cases | 100% verdicts, 100% field extraction |
  23. The baseline is weakest at refusing near-miss questions, such as "interest rate on a Kisan Sahaya crop loan". An LLM that may answer `NOT_FOUND` is expected to fix that, and stage 3 measures it. The two English-question/Telugu-document questions fail with TF-IDF by design. They need the multilingual embedder (`st:intfloat/multilingual-e5-small`).
  24. The 100 agent cases were written together with the parser, so they guard against regressions. They are not a measure of how well it generalises.
  25. **Still to do:** run the evaluation on real scheme documents (see [docs/REAL_DATA.md](docs/REAL_DATA.md)), and run the stage 3 fine-tune on a GPU and fill in the comparison table. No numbers are claimed for either yet.
  26. ## Quick start
  27. ```bash
  28. python -m venv .venv
  29. .venv\Scripts\activate # Windows (macOS/Linux: source .venv/bin/activate)
  30. pip install -r requirements-dev.txt
  31. python scripts/make_demo_corpus.py # only if data/demo/docs is empty
  32. python -m docintel ingest data/demo/docs
  33. python -m docintel ask "What is the hostel allowance for degree students?"
  34. python -m docintel serve # open http://localhost:8000 (API docs at /docs)
  35. pytest -q
  36. ```
  37. OCR needs Tesseract:
  38. - **Windows:** install the UB-Mannheim build. Tick **Telugu** under "Additional language data", then add `C:\Program Files\Tesseract-OCR` to PATH.
  39. - **Linux:** run `sudo apt install tesseract-ocr tesseract-ocr-tel poppler-utils`.
  40. Scanned PDFs on Windows also need Poppler on PATH.
  41. ### Better models
  42. Copy `.env.example` to `.env` and choose:
  43. | Goal | Setting |
  44. |---|---|
  45. | Telugu and English questions over both languages | `DOCINTEL_EMBEDDER=st:intfloat/multilingual-e5-small` (needs `pip install -r requirements-gpu.txt`) |
  46. | Free local LLM on your GPU | Install [Ollama](https://ollama.com), run `ollama pull qwen2.5:3b`, then set `DOCINTEL_LLM_PROVIDER=ollama` |
  47. | Hosted LLM | `DOCINTEL_LLM_PROVIDER=gemini` plus `GEMINI_API_KEY`, or `openai` with any OpenAI-compatible endpoint |
  48. After changing the embedder, re-run `ingest`, then `python -m docintel calibrate` to pick the "not found" thresholds.
  49. ## Commands
  50. | Command | What it does |
  51. |---|---|
  52. | `python -m docintel ingest <folder>` | Index every PDF, image and text file in a folder |
  53. | `python -m docintel ask "<question>"` | Answer one question with citations |
  54. | `python -m docintel eval [file]` | Retrieval and answer metrics; writes `eval/results/<model>.md` |
  55. | `python -m docintel calibrate` | Choose the refusal thresholds from the evaluation set |
  56. | `python -m docintel agent --text "..."` | Run the eligibility agent |
  57. | `python -m docintel agent-eval` | Score the agent on 100 test cases |
  58. | `python -m docintel serve` | API and web UI on port 8000 |
  59. | `python -m docintel mcp` | MCP server (stdio) exposing `search_documents`, `ask_documents`, `check_eligibility`, `list_schemes` |
  60. ## API
  61. | Method | Path | Purpose |
  62. |---|---|---|
  63. | GET | `/health` | Status, number of passages, models in use |
  64. | GET / POST | `/documents` | List documents, or upload one (≤ 25 MB) |
  65. | POST | `/ask` | `{"question": "..."}` returns an answer, citations, latency and retrieval details |
  66. | POST | `/feedback` | `{"query_id": 1, "helpful": true}` |
  67. | GET | `/stats` | Per-model question count, latency, refusal rate and feedback (SQL `GROUP BY`) |
  68. | POST | `/agent/eligibility` | `{"text": "...", "profile": {...}}` returns per-scheme verdicts, reasons and follow-up questions |
  69. ## How it works
  70. ```
  71. file ─▶ loader ─(no text layer?)─▶ OCR ─▶ boilerplate removal ─▶ page-aware chunks
  72. │
  73. BM25 index ◀─────────────┼────────▶ vector index
  74. └──────── RRF fusion ┘
  75. question ─▶ hybrid search ─▶ "enough evidence?" ─no─▶ "not found"
  76. │ yes
  77. ▼
  78. numbered passages ─▶ LLM ─▶ keep valid [n] citations ─▶ answer + pages
  79. ```
  80. Eligibility agent (LangGraph):
  81. ```
  82. parse_profile ─▶ select_schemes ─▶ check_eligibility ─┬─▶ explain (some verdicts known)
  83. └─▶ ask_follow_up (every scheme lacks facts)
  84. ```
  85. [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md) explains each design decision and the trade-offs.
  86. ## Stage 3: fine-tuning on an 8 GB laptop GPU
  87. ```bash
  88. pip install torch --index-url https://download.pytorch.org/whl/cu128 # RTX 50-series needs CUDA 12.8 builds
  89. pip install -r finetune/requirements.txt
  90. # 1. a teacher model writes Q&A pairs for each passage (questions close to eval questions are dropped)
  91. set DOCINTEL_LLM_PROVIDER=gemini & set GEMINI_API_KEY=...
  92. python finetune/build_dataset.py --per-chunk 4
  93. # 2. QLoRA: 4-bit base model, LoRA r=16 on all projections, loss on answer tokens only
  94. python finetune/train_qlora.py --model Qwen/Qwen2.5-1.5B-Instruct --epochs 2
  95. # 3. same questions, same retrieval, different generators
  96. set DOCINTEL_HF_ADAPTER=finetune/out/adapter
  97. python finetune/compare.py --providers extractive hf_local ollama gemini --price gemini=0.35
  98. ```
  99. ## Project layout
  100. ```
  101. docintel/
  102. ingest/ loader (PDF, OCR, text), sentence splitting, chunking, boilerplate removal
  103. index/ BM25 + embeddings + RRF hybrid index
  104. generate/ prompt, LLM providers, citation checking, refusal gate
  105. agent/ rules engine, free-text profile parser, tools, LangGraph graph
  106. eval/ answer evaluation, threshold calibration, agent evaluation
  107. api.py FastAPI app db.py SQLAlchemy models and stats
  108. mcp_server.py static/index.html web UI
  109. finetune/ dataset builder, QLoRA trainer, model comparison
  110. data/demo/ fictional documents and their rules
  111. eval/ question sets and agent cases
  112. tests/ 42 tests, including evaluation quality gates
  113. ```
  114. ## Limits
  115. - The demo documents are fictional. Real government PDFs have tables, two columns and stamps that this pipeline will handle less well. That is why the real-data evaluation matters.
  116. - Eligibility rules are written by hand from the documents, one JSON file per scheme. Extracting them with an LLM is possible, but every rule would still need a human check.
  117. - The agent's verdict is guidance only. The department makes the final decision.
  118. ## License
  119. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.