Skip to content
Back to search

filings-agent

Citation-grounded research agent over SEC 10-K filings: hybrid retrieval on pgvector, LangGraph agent, MCP server, calibrated LLM-judge eval

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ll333111--filings-agent/badge.svg)](/listings/ll333111--filings-agent)

HTML

<a href="/listings/ll333111--filings-agent"><img src="/api/listings/ll333111--filings-agent/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires Python 3.11+. The index needs about 20 minutes to build on a CPU.

python -m venv .venv && . .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
filings build                                        # download 10 filings, chunk, embed into pgvector

One-command demo (needs a GLM API key in `GLM_API_KEY`, or in a file named by `GLM_KEY_FILE`):

filings ask "How much did NVIDIA's headcount grow from the end of fiscal 2025 to the end of fiscal 2026?"

It prints the answer, the cited chunk ids with their source sections, the agent's plan and tool calls, and the token cost of the question. Other commands:

filings ask "..." --rag                              # single-shot RAG instead of the agent
filings search "H20 export license charge" --ticker NVDA --year 2026
python -m filings_agent.mcp_server                   # MCP server over stdio
pytest -m "not integration"                          # unit tests (integration tests need the index)

Postgres: by default `pgserver` starts an embedded PostgreSQL 16 with pgvector under `~/.filings_agent/pgdata`, so nothing needs to be installed. To use Docker instead, run `docker compose up -d` and set `DATABASE_URL=postgresql://postgres:postgres@localhost:5433/postgres`.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 246 lines · 1 finding marked
  1. # filings-agent
  2. A research assistant for SEC 10-K filings that cites its sources. It answers analyst questions such as
  3. *"Did Microsoft's headcount go up or down between June 2025 and June 2026, and by how much?"*.
  4. Every claim in an answer is tied to the exact chunk id it came from, and the assistant refuses
  5. when the indexed filings don't contain the answer.
  6. The bulk of this repo is the **evaluation harness**:
  7. - 43 hand-checked questions with gold evidence quotes;
  8. - retrieval metrics;
  9. - answer correctness and citation checks;
  10. - an LLM judge calibrated against 86 hand labels;
  11. - every prompt change tracked as a separate, saved experiment.
  12. ```
  13. EDGAR 10-K HTML ─► parse (Items, tables) ─► section-aware chunks ─► Postgres + pgvector
  14. │
  15. question ─► BM25 ─┐ │
  16. dense ┴► reciprocal-rank fusion ─► cross-encoder rerank ┘
  17. │
  18. single-shot RAG ◄────────────┴────────────► LangGraph agent
  19. (retrieve once, answer) plan ─► research ⇄ tools ─► answer ─► verify
  20. tools: search_filings, get_section_outline,
  21. read_chunks, compare_numbers, list_filings
  22. (the same tools are exposed over MCP)
  23. ```
  24. ## Quick start
  25. Requires Python 3.11+. The index needs about 20 minutes to build on a CPU.
  26. ```bash
  27. python -m venv .venv && . .venv/bin/activate # Windows: .venv\Scripts\activate
  28. pip install -e ".[dev]"
  29. filings build # download 10 filings, chunk, embed into pgvector
  30. ```
  31. One-command demo (needs a GLM API key in `GLM_API_KEY`, or in a file named by `GLM_KEY_FILE`):
  32. ```bash
  33. filings ask "How much did NVIDIA's headcount grow from the end of fiscal 2025 to the end of fiscal 2026?"
  34. ```
  35. It prints the answer, the cited chunk ids with their source sections, the agent's plan and tool
  36. calls, and the token cost of the question. Other commands:
  37. ```bash
  38. filings ask "..." --rag # single-shot RAG instead of the agent
  39. filings search "H20 export license charge" --ticker NVDA --year 2026
  40. python -m filings_agent.mcp_server # MCP server over stdio
  41. pytest -m "not integration" # unit tests (integration tests need the index)
  42. ```
  43. Postgres: by default `pgserver` starts an embedded PostgreSQL 16 with pgvector under
  44. `~/.filings_agent/pgdata`, so nothing needs to be installed. To use Docker instead, run
  45. `docker compose up -d` and set `DATABASE_URL=postgresql://postgres:postgres@localhost:5433/postgres`.
  46. ## Corpus and question set
  47. - **Corpus:** the two latest 10-Ks of Apple, Microsoft, NVIDIA, Coca-Cola and Visa (10 filings).
  48. They are split into 3,426 chunks of about 220 words with a 40-word overlap. Each chunk records
  49. its Item section, and its id is stable, e.g. `NVDA-FY2026-1A-0041`.
  50. - **Questions** ([eval/questions.jsonl](eval/questions.jsonl)):
  51. - 23 single-filing questions;
  52. - 12 multi-hop questions, which need two filings or a comparison;
  53. - 8 unanswerable questions (company or year not indexed, or a figure the filings don't disclose).
  54. - **Gold evidence:** stored as verbatim quotes rather than chunk ids, so the gold set survives
  55. re-chunking. A chunk counts as relevant if it contains a gold quote.
  56. ## Results
  57. All numbers below come from saved runs in [results/](results/). To regenerate the tables, run
  58. `python -m filings_agent.evaluate table`.
  59. ### Retrieval (35 answerable questions)
  60. | retriever | recall@5 | recall@10 | nDCG@10 | MRR@10 | s/query (CPU) |
  61. |---|---|---|---|---|---|
  62. | BM25 | 0.486 | 0.571 | 0.434 | 0.396 | 0.02 |
  63. | dense (bge-small-en-v1.5) | 0.429 | 0.600 | 0.389 | 0.331 | 0.99 |
  64. | hybrid (RRF, k=60) | 0.500 | 0.657 | 0.456 | 0.407 | 0.15 |
  65. | **hybrid + cross-encoder rerank** | **0.557** | **0.686** | **0.503** | **0.453** | 4.41 |
  66. | hybrid + rerank + parsed ticker/year filters | 0.557 | 0.657 | 0.504 | 0.461 | 5.14 |
  67. Findings:
  68. - Hybrid retrieval plus reranking improves nDCG@10 by 0.114 over dense alone.
  69. - Multi-hop questions are the weak spot. Recall@10 for hybrid + rerank is 0.826 on single-filing
  70. questions but only 0.417 on multi-hop questions, because one query has to surface evidence from
  71. two filings.
  72. - Parsing ticker and year filters out of the question did not help, so it is not the default.
  73. ### Answers: single-shot RAG vs agent
  74. All systems answer with `glm-4.7-flashx` at temperature 0. RAG reads the top 8 chunks.
  75. | system | correct (judge) | correct, multi-hop | supported (judge) | key-fact recall | gold evidence cited | refusal recall | false refusal | LLM calls/q | CNY/q | s/q |
  76. |---|---|---|---|---|---|---|---|---|---|---|
  77. | RAG, dense, prompt v3 | 0.674 | 0.417 | 0.791 | 0.648 | 0.529 | 0.875 | 0.114 | 1.0 | 0.0020 | 2.3 |
  78. | RAG, hybrid+rerank, prompt v1 | 0.721 | 0.417 | 0.814 | 0.700 | 0.586 | 0.750 | 0.086 | 1.0 | 0.0016 | 11.0 |
  79. | RAG, hybrid+rerank, prompt v2 | 0.767 | 0.417 | 0.860 | 0.686 | 0.557 | 1.000 | 0.114 | 1.0 | 0.0017 | 4.4 |
  80. | RAG, hybrid+rerank, prompt v3 | 0.744 | 0.417 | 0.860 | 0.671 | 0.586 | 1.000 | 0.114 | 1.0 | 0.0019 | 5.4 |
  81. | **agent, research prompt v1** | **0.837** | 0.667 | 0.860 | **0.729** | **0.657** | 1.000 | **0.029** | 6.0 | 0.0127 | 132.4 |
  82. | agent, research prompt v2 | 0.814 | **0.750** | 0.814 | 0.629 | 0.529 | 0.875 | 0.029 | 6.1 | 0.0137 | 126.2 |
  83. What the columns mean:
  84. - **correct / supported:** verdicts from the LLM judge (judge prompt v2, below).
  85. - **key-fact recall:** string match against the key facts of the reference answer.
  86. - **gold evidence cited:** the share of gold evidence groups that the answer actually cites.
  87. - **refusal recall:** the share of the 8 unanswerable questions that the system flags.
  88. - **false refusal:** the share of answerable questions that the system wrongly flags.
  89. - **CNY/q and s/q:** from the first, uncached run, in [results/costs/](results/costs/).
  90. - For RAG, s/q is the LLM call alone; retrieval adds about 4.4 s.
  91. - The prompt v1 figure includes rate-limit backoff waits.
  92. - The agent figure includes CPU reranking on every search call.
  93. Hand labels give the same picture on the two systems that were labeled:
  94. | system | correct (hand) | correct, multi-hop (hand) | supported (hand) |
  95. |---|---|---|---|
  96. | RAG, hybrid+rerank, v3 | 0.744 | 0.417 | 0.837 |
  97. | agent v1 | 0.860 | 0.750 | 0.837 |
  98. Takeaways:
  99. - **The agent beats single-shot RAG** on correctness: 0.837 vs 0.744 by the judge, and 0.860 vs
  100. 0.744 by hand.
  101. - **The gain is concentrated in multi-hop questions:** 0.750 vs 0.417 by hand. The planner splits a
  102. question into one sub-question per filing and searches each with ticker and year filters.
  103. - **The agent also refuses less often** when the answer exists: false refusals fall from 4 of 35 to 1.
  104. - **The cost** is about 6 LLM calls and roughly 7× the tokens per question, plus much higher latency
  105. on a CPU-only machine.
  106. - **Caveat:** with 43 questions, one question moves a rate by 2.3 points, so differences of one or
  107. two questions (RAG v2 vs v3, for example) are within noise.
  108. ### Prompt iterations (tracked experiments)
  109. Each prompt version is a saved run with its own outputs (`results/answers_<system>_<prompt>.jsonl`).
    • Warning: Contains a <system> block, a common wrapper for instructions hidden in a tool description
    • Line 136 on GitHub
  110. - **RAG answer prompt** ([prompts.py](src/filings_agent/prompts.py)):
  111. - **v1 → v2:** added grounding and refusal rules. Refusal recall rose from 0.75 to 1.00; v1 had
  112. invented a "Visa FY2022 revenue" figure and a Germany-only Microsoft search revenue figure.
  113. Judge correctness rose from 0.721 to 0.767.
  114. - **v2 → v3:** added inline `[chunk_id]` citations and a company/year match check. Gold evidence
  115. cited rose from 0.557 to 0.586, while correctness moved by one question (0.767 → 0.744).
  116. v3 is kept because its citations can be checked claim by claim.
  117. - **Agent research prompt** ([agent.py](src/filings_agent/agent.py)):
  118. - **v1 → v2:** asked the agent to write search queries as full questions, because the
  119. cross-encoder ranked keyword queries poorly on one headcount question.
  120. - The result was a regression: correctness fell from 0.837 to 0.814, key-fact recall from 0.729
  121. to 0.629, and refusal recall from 1.00 to 0.875 (v2 answered "Microsoft search revenue in
  122. Germany" with an unrelated figure).
  123. - v2 is not adopted; v1 remains the default.
  124. - **Judge prompt:** see the next section.
  125. ### LLM judge calibration
  126. - **Judge model:** `glm-5.3-flash`, a different and newer model than the `glm-4.7-flashx` being
  127. graded, so the system is not grading its own answers.
  128. - **What it grades:** each answer is checked against the reference answer for *correct* and
  129. against the cited chunk text for *supported*.
  130. - **Hand labels:** 86 in [eval/hand_labels.jsonl](eval/hand_labels.jsonl), covering every answer
  131. from RAG v3 and agent v1, each with a note explaining it.
  132. | judge prompt | label | agreement with hand labels | Cohen's κ | hand rate | judge rate |
  133. |---|---|---|---|---|---|
  134. | v1 (bare rubric) | correct | 0.988 | 0.964 | 0.802 | 0.791 |
  135. | v1 (bare rubric) | supported | 0.907 | 0.677 | 0.837 | 0.814 |
  136. | **v2 (explicit rubric)** | correct | 0.988 | 0.964 | 0.802 | 0.791 |
  137. | **v2 (explicit rubric)** | supported | **0.953** | **0.819** | 0.837 | 0.860 |
  138. - Both prompts judge correctness almost exactly like a human.
  139. - The explicit rubric was needed for faithfulness: claims cited to a chunk that doesn't state them,
  140. and how refusals should be scored. It raised κ from 0.68 to 0.82 on that label.
  141. - In [results/judge_calibration.json](results/judge_calibration.json), the judge's lean toward
  142. "supported" (0.860 vs 0.837) is its main remaining bias.
  143. ### Bugs the evaluation caught
  144. - **The model sometimes ignores JSON mode.** `glm-4.7-flashx` occasionally replies in prose with
  145. inline `[chunk_id]` citations. At first those answers were parsed as empty, and two correct RAG
  146. answers were scored as wrong. Prose replies are now parsed by a deterministic fallback
  147. (`rag.parse_answer`).
  148. - **Citations came back in mixed formats**, such as `"[ID]"` or `"ID, ID"`. They are now normalized
  149. to bare ids before validation.
  150. - **The judge ran out of tokens.** `glm-5.3-flash` always thinks, and thinking tokens count against
  151. `max_tokens`. At 400 tokens, about 30% of verdicts were cut off before any JSON was written; the
  152. limit is now 3,000. Verdicts that still fail to parse are counted as wrong rather than dropped:
  153. 1 of 258 under judge v2 (an agent v1 answer) and 4 of 258 under judge v1.
  154. ### Cost
  155. All experiments in this README cost about **¥2.5 (about US$0.35)** in GLM API calls:
  156. - answers: ¥1.44 for 6 systems × 43 questions;
  157. - judge passes: ¥1.01 for 2 judge prompts × 6 systems.
  158. Every request is cached on disk by a hash of its body, so re-scoring is free; the cache also serves
  159. as an audit log. The client enforces a hard spend cap (`MAX_SPEND_CNY`, default 20). Prices are in
  160. [config.py](src/filings_agent/config.py). The free `glm-4.7-flash` also works but is rate-limited to
  161. about 1 call a minute on a basic account.
  162. ## How it works
  163. - **Ingestion** ([edgar.py](src/filings_agent/edgar.py), [parse.py](src/filings_agent/parse.py), [chunk.py](src/filings_agent/chunk.py)):
  164. - Downloads each 10-K from the EDGAR submissions API.
  165. - Flattens HTML tables into `a | b | c` rows and drops hidden iXBRL and page furniture
  166. (page numbers, running "Item 7" headers).
  167. - Detects Item sections, keeping the longest occurrence of each Item so the table of contents is skipped.
  168. - Chunks within sections on paragraph boundaries, with overlap.
  169. - **Index** ([store.py](src/filings_agent/store.py)): chunks and embeddings live in Postgres. Each
  170. embedding model gets its own `emb_<model>` table with an HNSW cosine index, filled by binary COPY.
  171. - **Retrieval** ([retrieve.py](src/filings_agent/retrieve.py)):
  172. - BM25 and pgvector dense search, 50 candidates each;
  173. - reciprocal-rank fusion;
  174. - `ms-marco-MiniLM-L-6-v2` cross-encoder rerank of the top 30;
  175. - optional filters on ticker, fiscal year and Item.
  176. - **Agent** ([agent.py](src/filings_agent/agent.py)):
  177. - The plan step binds each sub-question to a ticker and fiscal year, and lists any requested
  178. filings that are not indexed.
  179. - Research is a tool loop capped at 8 steps.
  180. - Answers are written only from chunks the agent actually read.
  181. - Verification is deterministic: citations to unseen chunks are dropped, and an uncited answer
  182. gets one retry before it is flagged as insufficient evidence.
  183. - **MCP** ([mcp_server.py](src/filings_agent/mcp_server.py)): the five tools are exposed to any
  184. MCP client over stdio. An integration test drives the server through a real MCP client session.
  185. ## Limitations
  186. - **The question set is small:** 43 questions over 5 companies. Treat differences of one or two
  187. questions as noise.
  188. - **The hand labels were drafted with AI assistance.** Each one was checked against the cited chunk
  189. text, and the reasoning is recorded in its `note` field, but they are one annotator's judgments,
  190. not a multi-rater set.
  191. - **Latency is high on a CPU machine.** The agent averages about 130 s per question, mostly CPU
  192. reranking plus API rate limits; a GPU reranker or a smaller rerank pool would cut most of it.
  193. - **Retrieval is the bottleneck for multi-hop questions** (recall@10 of 0.417). The agent works
  194. around it by searching more than once, rather than fixing retrieval itself.
  195. ## Layout
  196. ```
  197. src/filings_agent/ config, edgar, parse, chunk, ingest, index, store, retrieve,
  198. llm (GLM client, cache, spend cap), prompts, rag, tools, agent,
  199. mcp_server, evaluate, cli
  200. eval/ questions.jsonl (gold quotes), hand_labels.jsonl
  201. results/ every run's per-question outputs, summaries, calibration, costs
  202. tests/ unit tests (parsing, chunking, fusion, metrics, LLM adapter) + integration
  203. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.