Skip to content
Back to search

MCP

An MCP server that gives Claude a local, searchable library of research papers (arXiv + PDFs). Built on the MCP Python SDK 2.x (MCPServer, the 2.x successor to FastMCP).

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/neelghoshal--mcp/badge.svg)](/listings/neelghoshal--mcp)

HTML

<a href="/listings/neelghoshal--mcp"><img src="/api/listings/neelghoshal--mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

cd paper-librarian
python -m venv .venv
.venv\Scripts\activate      # Windows  (source .venv/bin/activate on macOS/Linux)
pip install -e ".[dev]"
pytest -q                   # 90 tests

The first run downloads the `all-MiniLM-L6-v2` embedding model (~90 MB).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 368 lines
  1. # Paper Librarian
  2. An MCP server that gives Claude a local, searchable library of research papers (arXiv + PDFs).
  3. Built on the **MCP Python SDK 2.x** (`MCPServer`, the 2.x successor to `FastMCP`).
  4. Add papers by arXiv id or PDF path, then ask questions about them. Answers cite the paper, section
  5. and page, and references between papers in the library are linked in both directions.
  6. ---
  7. ## Demo
  8. A session in Claude Code with the server connected.
  9. **1. Add a paper by arXiv id.** `add_paper` downloads *Attention Is All You Need*, splits it into
  10. 44 searchable chunks and parses its 40 references. None are linked yet because the library was empty.
  11. ![Adding the first paper](paper-librarian/docs/screenshots/01-add-paper.png)
  12. **2. Claude calls the MCP tool.** Asked to add BERT (`1810.04805`), Claude Code routes the request to
  13. the `paper-librarian` server.
  14. ![Claude Code calling paper-librarian](paper-librarian/docs/screenshots/02-calling-tool.png)
  15. **3. Add a second paper; citations link automatically.** BERT is added with 71 chunks and 56 references.
  16. One of them matches the Transformer paper already in the library, so the two papers are now linked.
  17. ![Second paper added with a citation link](paper-librarian/docs/screenshots/03-second-paper-linked.png)
  18. **4. Ask a question; answers cite section and page.** Hybrid search finds the relevant passages in both
  19. papers, and Claude cites each claim, e.g. *(Attention Is All You Need, Positional Encoding, p.6)*.
  20. ![Search answer with section and page citations](paper-librarian/docs/screenshots/04-search-with-citations.png)
  21. **5. Save a note to a paper.** `add_note` attaches a summary to the paper; it is returned whenever the
  22. paper is looked up with `get_paper`.
  23. ![Adding a note to a paper](paper-librarian/docs/screenshots/05-add-note.png)
  24. ---
  25. ## Problem statement
  26. Researchers and students keep dozens of papers as PDFs, but an AI assistant can't use them well:
  27. - **No memory of your library.** Each chat starts from nothing; pasting whole PDFs runs into context
  28. limits and has to be repeated every time.
  29. - **Answers from memory, not sources.** Without access to the actual text, a model summarizes papers
  30. from training data, which may be outdated, vague or wrong, and can't point to where a claim comes from.
  31. - **PDFs are hard to read programmatically.** Two-column layouts, section headings, hyphenation,
  32. ligatures and reference lists in several citation styles all break naive text extraction.
  33. - **Papers are connected, but the connections are invisible.** Which of my papers cite each other?
  34. That information is buried in reference lists.
  35. **Paper Librarian** solves this with an MCP server that runs locally and:
  36. 1. ingests papers from arXiv or local PDFs into a structured library (sections, pages, references);
  37. 2. indexes them for **hybrid search** (meaning-based + keyword), returning passages with
  38. section and page so every answer can be cited;
  39. 3. parses reference lists and **links citations** between papers in the library;
  40. 4. exposes all of this to Claude (or any MCP client) as tools, a resource and a prompt.
  41. Everything — database, PDFs, embedding model — stays on your machine.
  42. ---
  43. ## Features
  44. - **Add papers** by arXiv id (`1706.03762`, `arXiv:1706.03762v5`), arXiv URL, or local PDF path, with
  45. duplicate detection by arXiv id and file hash.
  46. - **Layout-aware PDF extraction**: two-column reading order, section detection using font weight and
  47. size, page tracking, ligature and hyphenation repair.
  48. - **Hybrid search**: vector similarity (sqlite-vec) + BM25 keyword search (SQLite FTS5), merged with
  49. reciprocal rank fusion; filters by year and paper.
  50. - **Citation graph**: reference lists split in `[n]`, `n.` and author-year styles; matched to library
  51. papers by arXiv id, exact title or fuzzy title; linked forwards and backwards as papers are added.
  52. - **Section reader**: fetch the full text of any section ("Method", "Positional Encoding", …).
  53. - **Comparison helper**: gathers the most relevant passages per paper per aspect for comparison tables.
  54. - **Notes** attached to papers.
  55. - **Local and private**: one SQLite file plus a PDF folder; no external services except arXiv
  56. downloads and the one-time model download.
  57. ---
  58. ## Tech stack
  59. | Layer | Technology | Why |
  60. |---|---|---|
  61. | Language | Python 3.11+ (developed on 3.13) | Ecosystem for PDFs, ML and MCP |
  62. | MCP server | [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) **2.3** (`MCPServer`) | Tools, resources, prompts, progress reporting, stdio transport |
  63. | PDF parsing | [PyMuPDF](https://pymupdf.readthedocs.io/) 2.0 | Fast text extraction with font, size and position data per span |
  64. | arXiv | [`arxiv`](https://pypi.org/project/arxiv/) 4.0 + `requests` | Metadata lookup (title, authors, abstract) and PDF download |
  65. | Embeddings | [sentence-transformers](https://www.sbert.net/) 6.2, model `all-MiniLM-L6-v2` (384-dim) | Small (~90 MB), CPU-friendly, good quality for retrieval |
  66. | Vector index | [sqlite-vec](https://github.com/asg017/sqlite-vec) 0.1.9 (`vec0` table) | KNN search inside the same SQLite file |
  67. | Keyword index | SQLite FTS5 (BM25 ranking) | Built into Python's `sqlite3`, no extra service |
  68. | Storage | SQLite | Single file, transactional, zero setup |
  69. | Testing | pytest 9, anyio, MCP in-process `Client` | Unit tests plus end-to-end tests through a real MCP client |
  70. | Build | hatchling (`pyproject.toml`) | Standard packaging; `paper-librarian` console script |
  71. ---
  72. ## Architecture
  73. ```mermaid
  74. flowchart LR
  75. Claude["Claude / MCP client"] <-- "stdio (JSON-RPC)" --> Server["server.py<br/>MCPServer: 8 tools,<br/>1 resource, 1 prompt"]
  76. Server --> Ingest["ingest.py<br/>fetch · extract · sections · chunks"]
  77. Server --> Search["search.py<br/>vector + BM25 + RRF"]
  78. Server --> Store
  79. Ingest --> ArXiv[("arXiv API")]
  80. Ingest --> Embed["embed.py<br/>MiniLM embeddings"]
  81. Ingest --> Cit["citations.py<br/>split · match · link"]
  82. Search --> Embed
  83. Ingest --> Store["store.py<br/>SQLite + FTS5 + sqlite-vec"]
  84. Cit --> Store
  85. Search --> Store
  86. Store --> DB[("~/.librarian/library.db")]
  87. Ingest --> PDFs[("~/.librarian/pdfs/")]
  88. ```
  89. ### How a paper is added (`add_paper`)
  90. ```mermaid
  91. flowchart TD
  92. A["source: arXiv id / URL / PDF path"] --> B{"arXiv id?"}
  93. B -- yes --> C["duplicate check by arXiv id"] --> D["fetch metadata + download PDF"]
  94. B -- no --> E["duplicate check by sha256"]
  95. D --> F["extract_pdf: column-aware page text"]
  96. E --> F
  97. F --> G["split_sections: headings → segments (section, page)"]
  98. G --> H["chunk_segments: 180-word windows, 30-word overlap"]
  99. H --> I["embed chunks (MiniLM, normalized)"]
  100. G --> J["References section → split_references"]
  101. I --> K["one transaction: paper + chunks + vectors + FTS + citations + backlinks"]
  102. J --> K
  103. ```
  104. ### How a question is answered (`search`)
  105. ```mermaid
  106. flowchart LR
  107. Q["query"] --> V["embed → KNN on chunk_vectors"]
  108. Q --> K["sanitize → FTS5 MATCH, BM25"]
  109. V --> R["reciprocal rank fusion<br/>score = Σ 1/(60 + rank)"]
  110. K --> R
  111. R --> F["filters: year, paper ids<br/>cap: N hits per paper"]
  112. F --> S["hits: title · section · page · snippet · matched_by"]
  113. ```
  114. ---
  115. ## Project structure
  116. ```
  117. paper-librarian/
  118. ├── pyproject.toml # dependencies, console script, pytest config
  119. ├── README.md
  120. ├── docs/screenshots/ # demo screenshots used in this README
  121. ├── src/librarian/
  122. │ ├── __init__.py # package version
  123. │ ├── server.py # MCP server: tools, resource, prompt, lifespan (333 lines)
  124. │ ├── ingest.py # arXiv fetch, PDF extraction, sections, chunking (457 lines)
  125. │ ├── search.py # hybrid search, RRF, snippets (198 lines)
  126. │ ├── citations.py # reference splitting, matching, linking (161 lines)
  127. │ ├── store.py # SQLite schema and data access (325 lines)
  128. │ └── embed.py # embedding model wrapper ( 53 lines)
  129. └── tests/ # 90 tests
  130. ├── test_server.py # end-to-end through an in-process MCP client (9)
  131. ├── test_ingest.py # arXiv ids, headings, sections, chunking, ingestion (39)
  132. ├── test_citations.py # splitting styles, matching, linking (20)
  133. ├── test_search.py # keyword / semantic / hybrid, filters, snippets (10)
  134. ├── test_store.py # schema, constraints, cascades (9)
  135. └── test_smoke.py # dependencies load (3)
  136. ```
  137. ---
  138. ## Code details
  139. ### `store.py` — data model and storage
  140. All data lives in one SQLite file. The schema:
  141. | Table | Columns | Notes |
  142. |---|---|---|
  143. | `papers` | `id`, `title`, `authors`, `year`, `abstract`, `source`, `arxiv_id`, `sha256`, `path`, `added_at` | `id` is `arxiv:<id>` or `pdf:<sha256[:16]>`; `arxiv_id` and `sha256` are `UNIQUE` (duplicate protection); `source` is `CHECK`ed to `arxiv`/`pdf` |
  144. | `chunks` | `id`, `paper_id`, `seq`, `section`, `page`, `text` | Searchable passages; `UNIQUE(paper_id, seq)` |
  145. | `chunks_fts` | FTS5 over `text`, `section` | External-content index kept in sync by `AFTER INSERT/DELETE` triggers |
  146. | `chunk_vectors` | `vec0(chunk_id, embedding float[384])` | sqlite-vec KNN index; same id as `chunks.id` so results can be fused |
  147. | `citations` | `id`, `paper_id`, `seq`, `raw_reference`, `matched_paper_id` | `matched_paper_id` → `papers.id`, `ON DELETE SET NULL` |
  148. | `notes` | `id`, `paper_id`, `text`, `created_at` | |
  149. | `meta` | `key`, `value` | Stores `embedding_dim`; reopening with a different model raises instead of corrupting the index |
  150. Deleting a paper cascades to its chunks, FTS entries, vectors (via trigger), citations and notes.
  151. `Store` provides typed access (`Paper`, `Chunk`, `Citation`, `Note` dataclasses):
  152. `add_paper`, `get_paper`, `find_duplicate`, `list_papers`, `delete_paper`, `add_chunks` (with vectors),
  153. `get_chunks(section=…)`, `add_citations`, `get_citations`, `get_cited_by`, `unmatched_citations`,
  154. `set_citation_match`, `delete_citations`, `add_note`, `get_notes`.
  155. `store.transaction(commit=True)` wraps writes; passing `commit=False` to individual methods lets a
  156. caller group several writes into one atomic transaction (used by ingestion).
  157. The connection is opened with `check_same_thread=False` because MCP tools run in worker threads; the
  158. server serializes access with a lock.
  159. ### `ingest.py` — getting papers in
  160. | Function | What it does |
  161. |---|---|
  162. | `parse_arxiv_id(source)` | Recognizes new-style (`2310.06825v2`), old-style (`hep-th/9901001`), `arXiv:` prefixes and `abs/`/`pdf/` URLs; returns the version-less id |
  163. | `fetch_arxiv(id, pdf_dir)` | Metadata via the `arxiv` API, PDF via `requests` (arxiv 4.x dropped `download_pdf`); validates the `%PDF` header |
  164. | `extract_pdf(path)` | Title (PDF metadata, else largest horizontal text on page 1), authors, year, section segments, raw References text |
  165. | `page_text(page)` | **Column-aware reading order**: blocks are classified left / right / full-width around the page centre; within bands separated by full-width blocks, the left column is read before the right. Marks indented lines (for reference splitting) and emphasized lines (bold or ≥1.1× body font size, for heading detection). Merges spans on the same baseline and applies NFKC normalization (`fi` → `fi`) |
  166. | `heading_name(line, appendix)` | Decides whether a line is a section heading. Known names (Introduction, Methods, …) map to canonical names; other numbered headings (`3 Model Architecture`) must be emphasized. Rejects years and decimals as section numbers, lines with column gaps, and sentence fragments. Letter numbering (`A`, `B.2`) only counts after References/Appendix |
  167. | `split_sections(pages)` | Walks lines, producing `Segment(section, page, text, raw)`; a new segment starts on each page so page numbers stay accurate |
  168. | `chunk_segments(...)` | 180-word windows with 30-word overlap, never crossing a section boundary. 180 words fits MiniLM's 256-token input limit, so nothing is silently truncated |
  169. | `ingest(source, store, embedder, progress)` | The whole pipeline (diagram above); writes paper, chunks, vectors and citations in **one transaction**, so a failure never leaves half a paper |
  170. | `rebuild_citations(store)` | Re-parses every stored PDF's references, e.g. after improving the parser |
  171. ### `embed.py` — embeddings
  172. `Embedder` is a `Protocol` (`dim`, `embed(texts)`), so tests use small fake embedders.
  173. `SentenceTransformerEmbedder` loads the model lazily (thread-safe), normalizes vectors, and offers
  174. `warm_up()` to load in the background at server start so the first search doesn't wait.
  175. ### `search.py` — hybrid retrieval
  176. | Function | What it does |
  177. |---|---|
  178. | `fts_query(text)` | Turns free text into a safe FTS5 query (each word quoted, OR-ed), so input like `"x"? AND NEAR*` can't cause syntax errors |
  179. | `keyword_search(...)` | FTS5 `MATCH` ranked by BM25, joined to papers for filtering |
  180. | `vector_search(...)` | sqlite-vec KNN; with filters it over-fetches (10×, min 200) and filters after, since `vec0` can't filter on paper columns during the scan |
  181. | `rrf(rankings)` | Reciprocal rank fusion, `score = Σ 1/(60 + rank)`; records which methods found each chunk |
  182. | `search(...)` | Runs one or both retrievers, fuses, applies a per-paper cap, and builds `SearchHit`s |
  183. | `make_snippet(text, terms)` | Picks the window containing the most distinct meaningful query words (short words like "of" ignored) |
  184. On a 4-paper, 10-query check (Transformer, BERT, ResNet, word2vec), hybrid search ranked the
  185. correct paper first 10/10 times.
  186. ### `citations.py` — the citation graph
  187. | Function / class | What it does |
  188. |---|---|
  189. | `split_references(raw)` | Tries `[n]` markers (≥2), then sequential `n.` markers (≥3, mostly counting up), then **hanging indents** for author-year styles (first line at the column edge, continuation lines indented). Rejoins words hyphenated across lines |
  190. | `reference_arxiv_id(ref)` | Finds `arXiv:1607.06450`, `abs/1409.0473`, arxiv.org URLs |
  191. | `Matcher` | Matches a reference to a library paper: **arXiv id** → **exact title** (compared as lowercase alphanumerics, so `Pre-\ntraining` = `Pretraining`) → **fuzzy title** (≥92% of title characters found in order, after a cheap word-overlap pre-filter). Titles under 12 characters ("BERT") never match on title alone; the longest matching title wins |
  192. | `link_paper(store, paper, refs)` | Stores the paper's references matched against the library, **and** re-checks older papers' unmatched references against the new paper (backlinks). Self-citations are excluded. Re-running replaces the paper's previous citations |
  193. ### `server.py` — the MCP interface
  194. `create_server(db_path=None, embedder=None)` builds an `MCPServer` whose **lifespan** opens the
  195. `Store`, creates the embedder (warming it up), and yields an `AppState(store, embedder, lock)`.
  196. Tools get it from `ctx.request_context.lifespan_context`. The factory lets tests inject a temporary
  197. database and a fake embedder; `main()` runs the default server over stdio and quiets
  198. model-hub HTTP logging (`httpx`, `httpx2`, `huggingface_hub`).
  199. **Tools**
  200. | Tool | Arguments | Returns | Annotations |
  201. |---|---|---|---|
  202. | `add_paper` | `source` | paper summary, `already_in_library`, chunk and reference counts; reports progress | open-world, idempotent |
  203. | `search` | `query`, `limit=8` (max 30), `mode` (`hybrid`/`semantic`/`keyword`), `year_from`, `year_to`, `paper_ids` | `results[]`: paper id, title, year, section, page, snippet, matched_by | read-only |
  204. | `get_paper` | `paper_id`, `section?` | metadata, abstract, sections (with start page), notes — or one section's full text (chunk overlaps removed, partial names accepted) | read-only |
  205. | `get_citations` | `paper_id` | `references[]` with `in_library` link, `cited_by[]` | read-only |
  206. | `compare_papers` | `paper_ids` (2–8), `aspects` (1–8), `passages_per_aspect=2` | passages per paper per aspect | read-only |
  207. | `add_note` | `paper_id`, `text` | note id, timestamp | |
  208. | `list_papers` | — | count, paper summaries | read-only |
  209. | `remove_paper` | `paper_id` | removed id and title (PDF file kept) | destructive |
  210. **Resource:** `library://papers` — the library as a Markdown list.
  211. **Prompt:** `literature_review(topic)` — guides Claude through search → compare → citations → a cited review.
  212. Paper ids are accepted in full (`arxiv:1706.03762`) or bare (`1706.03762`). Expected failures (unknown
  213. paper, missing file, download error, unknown section) are raised as `ToolError`, so Claude gets an
  214. actionable message such as *"No paper '…' in the library. Use list_papers to see paper ids."*
  215. ---
  216. ## Setup
  217. ```bash
  218. cd paper-librarian
  219. python -m venv .venv
  220. .venv\Scripts\activate # Windows (source .venv/bin/activate on macOS/Linux)
  221. pip install -e ".[dev]"
  222. pytest -q # 90 tests
  223. ```
  224. The first run downloads the `all-MiniLM-L6-v2` embedding model (~90 MB).
  225. ## Connect to Claude
  226. **Claude Code**
  227. ```bash
  228. claude mcp add paper-librarian --scope user -- <path-to-repo>/.venv/Scripts/python.exe -m librarian.server
  229. claude mcp get paper-librarian # should show: Status: ✔ Connected
  230. ```
  231. Start a new Claude Code session afterwards; servers are loaded when a session starts.
  232. Remove with `claude mcp remove paper-librarian -s user`.
  233. **Claude Desktop** — add to `claude_desktop_config.json` (Settings → Developer → Edit Config), then restart:
  234. ```json
  235. {
  236. "mcpServers": {
  237. "paper-librarian": {
  238. "command": "C:/path/to/paper-librarian/.venv/Scripts/python.exe",
  239. "args": ["-m", "librarian.server"]
  240. }
  241. }
  242. }
  243. ```
  244. ## Try it
  245. - "Add arXiv 1706.03762 and 1810.04805 to my paper library"
  246. - "What do my papers say about positional encoding? Cite sections."
  247. - "Compare BERT and the Transformer on pre-training corpus, model size and results"
  248. - "Which papers in my library cite Attention Is All You Need?"
  249. - Use the `literature_review` prompt with a topic
  250. ## Configuration
  251. | Variable | Default | Meaning |
  252. |---|---|---|
  253. | `LIBRARIAN_HOME` | `~/.librarian` | Library folder (`library.db`, downloaded `pdfs/`) |
  254. | `LIBRARIAN_EMBED_MODEL` | `sentence-transformers/all-MiniLM-L6-v2` | Embedding model (must produce 384-dim vectors) |
  255. | `HF_HUB_OFFLINE` | unset | Set to `1` once the model is downloaded to skip Hugging Face checks at startup |
  256. ## Testing and debugging
  257. ```bash
  258. pytest -q # unit + end-to-end MCP tests
  259. npx @modelcontextprotocol/inspector .venv/Scripts/python.exe -m librarian.server
  260. ```
  261. `tests/test_server.py` connects the SDK's `Client` to the server in-process and exercises every
  262. tool, the resource and the prompt, including progress notifications and error cases.
  263. ---
  264. ## Known limitations
  265. - **Scanned PDFs** (no text layer) are rejected; there is no OCR.
  266. - **Unusual layouts**: three-column pages, or reference lists that fit none of the three styles,
  267. may extract or split imperfectly.
  268. - **Subsections replace parent names**: text under 3.2.1 is labelled `Scaled Dot-Product Attention`,
  269. not `Model Architecture`.
  270. - **Filtered semantic search** over-fetches and filters afterwards; with many thousands of papers, a
  271. very narrow filter could miss results.
  272. - **Ingestion holds the database lock**, so other tool calls wait while a paper is being added
  273. (typically 5–30 s).
  274. - **Hyphen rejoining** also merges genuine compounds broken at a line end (`sequence-aligned` →
  275. `sequencealigned`).
  276. - References to papers **outside** the library are stored as text only (no Semantic Scholar lookup yet).
  277. ## Roadmap
  278. - Semantic Scholar lookup for references not in the library
  279. - OCR fallback for scanned PDFs
  280. - Parent/child section hierarchy
  281. - Packaging for `uvx paper-librarian`

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.