filings-agent
Citation-grounded research agent over SEC 10-K filings: hybrid retrieval on pgvector, LangGraph agent, MCP server, calibrated LLM-judge eval
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/ll333111--filings-agent)HTML
<a href="/listings/ll333111--filings-agent"><img src="/api/listings/ll333111--filings-agent/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Requires Python 3.11+. The index needs about 20 minutes to build on a CPU.
python -m venv .venv && . .venv/bin/activate # Windows: .venv\Scripts\activate pip install -e ".[dev]" filings build # download 10 filings, chunk, embed into pgvector
One-command demo (needs a GLM API key in `GLM_API_KEY`, or in a file named by `GLM_KEY_FILE`):
filings ask "How much did NVIDIA's headcount grow from the end of fiscal 2025 to the end of fiscal 2026?"
It prints the answer, the cited chunk ids with their source sections, the agent's plan and tool calls, and the token cost of the question. Other commands:
filings ask "..." --rag # single-shot RAG instead of the agent filings search "H20 export license charge" --ticker NVDA --year 2026 python -m filings_agent.mcp_server # MCP server over stdio pytest -m "not integration" # unit tests (integration tests need the index)
Postgres: by default `pgserver` starts an embedded PostgreSQL 16 with pgvector under `~/.filings_agent/pgdata`, so nothing needs to be installed. To use Docker instead, run `docker compose up -d` and set `DATABASE_URL=postgresql://postgres:postgres@localhost:5433/postgres`.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 246 lines · 1 finding marked
- # filings-agent
- A research assistant for SEC 10-K filings that cites its sources. It answers analyst questions such as
- *"Did Microsoft's headcount go up or down between June 2025 and June 2026, and by how much?"*.
- Every claim in an answer is tied to the exact chunk id it came from, and the assistant refuses
- when the indexed filings don't contain the answer.
- The bulk of this repo is the **evaluation harness**:
- - 43 hand-checked questions with gold evidence quotes;
- - retrieval metrics;
- - answer correctness and citation checks;
- - an LLM judge calibrated against 86 hand labels;
- - every prompt change tracked as a separate, saved experiment.
- ```
- EDGAR 10-K HTML ─► parse (Items, tables) ─► section-aware chunks ─► Postgres + pgvector
- │
- question ─► BM25 ─┐ │
- dense ┴► reciprocal-rank fusion ─► cross-encoder rerank ┘
- │
- single-shot RAG ◄────────────┴────────────► LangGraph agent
- (retrieve once, answer) plan ─► research ⇄ tools ─► answer ─► verify
- tools: search_filings, get_section_outline,
- read_chunks, compare_numbers, list_filings
- (the same tools are exposed over MCP)
- ```
- ## Quick start
- Requires Python 3.11+. The index needs about 20 minutes to build on a CPU.
- ```bash
- python -m venv .venv && . .venv/bin/activate # Windows: .venv\Scripts\activate
- pip install -e ".[dev]"
- filings build # download 10 filings, chunk, embed into pgvector
- ```
- One-command demo (needs a GLM API key in `GLM_API_KEY`, or in a file named by `GLM_KEY_FILE`):
- ```bash
- filings ask "How much did NVIDIA's headcount grow from the end of fiscal 2025 to the end of fiscal 2026?"
- ```
- It prints the answer, the cited chunk ids with their source sections, the agent's plan and tool
- calls, and the token cost of the question. Other commands:
- ```bash
- filings ask "..." --rag # single-shot RAG instead of the agent
- filings search "H20 export license charge" --ticker NVDA --year 2026
- python -m filings_agent.mcp_server # MCP server over stdio
- pytest -m "not integration" # unit tests (integration tests need the index)
- ```
- Postgres: by default `pgserver` starts an embedded PostgreSQL 16 with pgvector under
- `~/.filings_agent/pgdata`, so nothing needs to be installed. To use Docker instead, run
- `docker compose up -d` and set `DATABASE_URL=postgresql://postgres:postgres@localhost:5433/postgres`.
- ## Corpus and question set
- - **Corpus:** the two latest 10-Ks of Apple, Microsoft, NVIDIA, Coca-Cola and Visa (10 filings).
- They are split into 3,426 chunks of about 220 words with a 40-word overlap. Each chunk records
- its Item section, and its id is stable, e.g. `NVDA-FY2026-1A-0041`.
- - **Questions** ([eval/questions.jsonl](eval/questions.jsonl)):
- - 23 single-filing questions;
- - 12 multi-hop questions, which need two filings or a comparison;
- - 8 unanswerable questions (company or year not indexed, or a figure the filings don't disclose).
- - **Gold evidence:** stored as verbatim quotes rather than chunk ids, so the gold set survives
- re-chunking. A chunk counts as relevant if it contains a gold quote.
- ## Results
- All numbers below come from saved runs in [results/](results/). To regenerate the tables, run
- `python -m filings_agent.evaluate table`.
- ### Retrieval (35 answerable questions)
- | retriever | recall@5 | recall@10 | nDCG@10 | MRR@10 | s/query (CPU) |
- |---|---|---|---|---|---|
- | BM25 | 0.486 | 0.571 | 0.434 | 0.396 | 0.02 |
- | dense (bge-small-en-v1.5) | 0.429 | 0.600 | 0.389 | 0.331 | 0.99 |
- | hybrid (RRF, k=60) | 0.500 | 0.657 | 0.456 | 0.407 | 0.15 |
- | **hybrid + cross-encoder rerank** | **0.557** | **0.686** | **0.503** | **0.453** | 4.41 |
- | hybrid + rerank + parsed ticker/year filters | 0.557 | 0.657 | 0.504 | 0.461 | 5.14 |
- Findings:
- - Hybrid retrieval plus reranking improves nDCG@10 by 0.114 over dense alone.
- - Multi-hop questions are the weak spot. Recall@10 for hybrid + rerank is 0.826 on single-filing
- questions but only 0.417 on multi-hop questions, because one query has to surface evidence from
- two filings.
- - Parsing ticker and year filters out of the question did not help, so it is not the default.
- ### Answers: single-shot RAG vs agent
- All systems answer with `glm-4.7-flashx` at temperature 0. RAG reads the top 8 chunks.
- | system | correct (judge) | correct, multi-hop | supported (judge) | key-fact recall | gold evidence cited | refusal recall | false refusal | LLM calls/q | CNY/q | s/q |
- |---|---|---|---|---|---|---|---|---|---|---|
- | RAG, dense, prompt v3 | 0.674 | 0.417 | 0.791 | 0.648 | 0.529 | 0.875 | 0.114 | 1.0 | 0.0020 | 2.3 |
- | RAG, hybrid+rerank, prompt v1 | 0.721 | 0.417 | 0.814 | 0.700 | 0.586 | 0.750 | 0.086 | 1.0 | 0.0016 | 11.0 |
- | RAG, hybrid+rerank, prompt v2 | 0.767 | 0.417 | 0.860 | 0.686 | 0.557 | 1.000 | 0.114 | 1.0 | 0.0017 | 4.4 |
- | RAG, hybrid+rerank, prompt v3 | 0.744 | 0.417 | 0.860 | 0.671 | 0.586 | 1.000 | 0.114 | 1.0 | 0.0019 | 5.4 |
- | **agent, research prompt v1** | **0.837** | 0.667 | 0.860 | **0.729** | **0.657** | 1.000 | **0.029** | 6.0 | 0.0127 | 132.4 |
- | agent, research prompt v2 | 0.814 | **0.750** | 0.814 | 0.629 | 0.529 | 0.875 | 0.029 | 6.1 | 0.0137 | 126.2 |
- What the columns mean:
- - **correct / supported:** verdicts from the LLM judge (judge prompt v2, below).
- - **key-fact recall:** string match against the key facts of the reference answer.
- - **gold evidence cited:** the share of gold evidence groups that the answer actually cites.
- - **refusal recall:** the share of the 8 unanswerable questions that the system flags.
- - **false refusal:** the share of answerable questions that the system wrongly flags.
- - **CNY/q and s/q:** from the first, uncached run, in [results/costs/](results/costs/).
- - For RAG, s/q is the LLM call alone; retrieval adds about 4.4 s.
- - The prompt v1 figure includes rate-limit backoff waits.
- - The agent figure includes CPU reranking on every search call.
- Hand labels give the same picture on the two systems that were labeled:
- | system | correct (hand) | correct, multi-hop (hand) | supported (hand) |
- |---|---|---|---|
- | RAG, hybrid+rerank, v3 | 0.744 | 0.417 | 0.837 |
- | agent v1 | 0.860 | 0.750 | 0.837 |
- Takeaways:
- - **The agent beats single-shot RAG** on correctness: 0.837 vs 0.744 by the judge, and 0.860 vs
- 0.744 by hand.
- - **The gain is concentrated in multi-hop questions:** 0.750 vs 0.417 by hand. The planner splits a
- question into one sub-question per filing and searches each with ticker and year filters.
- - **The agent also refuses less often** when the answer exists: false refusals fall from 4 of 35 to 1.
- - **The cost** is about 6 LLM calls and roughly 7× the tokens per question, plus much higher latency
- on a CPU-only machine.
- - **Caveat:** with 43 questions, one question moves a rate by 2.3 points, so differences of one or
- two questions (RAG v2 vs v3, for example) are within noise.
- ### Prompt iterations (tracked experiments)
- Each prompt version is a saved run with its own outputs (`results/answers_<system>_<prompt>.jsonl`).
- Warning: Contains a <system> block, a common wrapper for instructions hidden in a tool description
- Line 136 on GitHub
- - **RAG answer prompt** ([prompts.py](src/filings_agent/prompts.py)):
- - **v1 → v2:** added grounding and refusal rules. Refusal recall rose from 0.75 to 1.00; v1 had
- invented a "Visa FY2022 revenue" figure and a Germany-only Microsoft search revenue figure.
- Judge correctness rose from 0.721 to 0.767.
- - **v2 → v3:** added inline `[chunk_id]` citations and a company/year match check. Gold evidence
- cited rose from 0.557 to 0.586, while correctness moved by one question (0.767 → 0.744).
- v3 is kept because its citations can be checked claim by claim.
- - **Agent research prompt** ([agent.py](src/filings_agent/agent.py)):
- - **v1 → v2:** asked the agent to write search queries as full questions, because the
- cross-encoder ranked keyword queries poorly on one headcount question.
- - The result was a regression: correctness fell from 0.837 to 0.814, key-fact recall from 0.729
- to 0.629, and refusal recall from 1.00 to 0.875 (v2 answered "Microsoft search revenue in
- Germany" with an unrelated figure).
- - v2 is not adopted; v1 remains the default.
- - **Judge prompt:** see the next section.
- ### LLM judge calibration
- - **Judge model:** `glm-5.3-flash`, a different and newer model than the `glm-4.7-flashx` being
- graded, so the system is not grading its own answers.
- - **What it grades:** each answer is checked against the reference answer for *correct* and
- against the cited chunk text for *supported*.
- - **Hand labels:** 86 in [eval/hand_labels.jsonl](eval/hand_labels.jsonl), covering every answer
- from RAG v3 and agent v1, each with a note explaining it.
- | judge prompt | label | agreement with hand labels | Cohen's κ | hand rate | judge rate |
- |---|---|---|---|---|---|
- | v1 (bare rubric) | correct | 0.988 | 0.964 | 0.802 | 0.791 |
- | v1 (bare rubric) | supported | 0.907 | 0.677 | 0.837 | 0.814 |
- | **v2 (explicit rubric)** | correct | 0.988 | 0.964 | 0.802 | 0.791 |
- | **v2 (explicit rubric)** | supported | **0.953** | **0.819** | 0.837 | 0.860 |
- - Both prompts judge correctness almost exactly like a human.
- - The explicit rubric was needed for faithfulness: claims cited to a chunk that doesn't state them,
- and how refusals should be scored. It raised κ from 0.68 to 0.82 on that label.
- - In [results/judge_calibration.json](results/judge_calibration.json), the judge's lean toward
- "supported" (0.860 vs 0.837) is its main remaining bias.
- ### Bugs the evaluation caught
- - **The model sometimes ignores JSON mode.** `glm-4.7-flashx` occasionally replies in prose with
- inline `[chunk_id]` citations. At first those answers were parsed as empty, and two correct RAG
- answers were scored as wrong. Prose replies are now parsed by a deterministic fallback
- (`rag.parse_answer`).
- - **Citations came back in mixed formats**, such as `"[ID]"` or `"ID, ID"`. They are now normalized
- to bare ids before validation.
- - **The judge ran out of tokens.** `glm-5.3-flash` always thinks, and thinking tokens count against
- `max_tokens`. At 400 tokens, about 30% of verdicts were cut off before any JSON was written; the
- limit is now 3,000. Verdicts that still fail to parse are counted as wrong rather than dropped:
- 1 of 258 under judge v2 (an agent v1 answer) and 4 of 258 under judge v1.
- ### Cost
- All experiments in this README cost about **¥2.5 (about US$0.35)** in GLM API calls:
- - answers: ¥1.44 for 6 systems × 43 questions;
- - judge passes: ¥1.01 for 2 judge prompts × 6 systems.
- Every request is cached on disk by a hash of its body, so re-scoring is free; the cache also serves
- as an audit log. The client enforces a hard spend cap (`MAX_SPEND_CNY`, default 20). Prices are in
- [config.py](src/filings_agent/config.py). The free `glm-4.7-flash` also works but is rate-limited to
- about 1 call a minute on a basic account.
- ## How it works
- - **Ingestion** ([edgar.py](src/filings_agent/edgar.py), [parse.py](src/filings_agent/parse.py), [chunk.py](src/filings_agent/chunk.py)):
- - Downloads each 10-K from the EDGAR submissions API.
- - Flattens HTML tables into `a | b | c` rows and drops hidden iXBRL and page furniture
- (page numbers, running "Item 7" headers).
- - Detects Item sections, keeping the longest occurrence of each Item so the table of contents is skipped.
- - Chunks within sections on paragraph boundaries, with overlap.
- - **Index** ([store.py](src/filings_agent/store.py)): chunks and embeddings live in Postgres. Each
- embedding model gets its own `emb_<model>` table with an HNSW cosine index, filled by binary COPY.
- - **Retrieval** ([retrieve.py](src/filings_agent/retrieve.py)):
- - BM25 and pgvector dense search, 50 candidates each;
- - reciprocal-rank fusion;
- - `ms-marco-MiniLM-L-6-v2` cross-encoder rerank of the top 30;
- - optional filters on ticker, fiscal year and Item.
- - **Agent** ([agent.py](src/filings_agent/agent.py)):
- - The plan step binds each sub-question to a ticker and fiscal year, and lists any requested
- filings that are not indexed.
- - Research is a tool loop capped at 8 steps.
- - Answers are written only from chunks the agent actually read.
- - Verification is deterministic: citations to unseen chunks are dropped, and an uncited answer
- gets one retry before it is flagged as insufficient evidence.
- - **MCP** ([mcp_server.py](src/filings_agent/mcp_server.py)): the five tools are exposed to any
- MCP client over stdio. An integration test drives the server through a real MCP client session.
- ## Limitations
- - **The question set is small:** 43 questions over 5 companies. Treat differences of one or two
- questions as noise.
- - **The hand labels were drafted with AI assistance.** Each one was checked against the cited chunk
- text, and the reasoning is recorded in its `note` field, but they are one annotator's judgments,
- not a multi-rater set.
- - **Latency is high on a CPU machine.** The agent averages about 130 s per question, mostly CPU
- reranking plus API rate limits; a GPU reranker or a smaller rerank pool would cut most of it.
- - **Retrieval is the bottleneck for multi-hop questions** (recall@10 of 0.417). The agent works
- around it by searching more than once, rather than fixing retrieval itself.
- ## Layout
- ```
- src/filings_agent/ config, edgar, parse, chunk, ingest, index, store, retrieve,
- llm (GLM client, cache, spend cap), prompts, rag, tools, agent,
- mcp_server, evaluate, cli
- eval/ questions.jsonl (gold quotes), hand_labels.jsonl
- results/ every run's per-question outputs, summaries, calibration, costs
- tests/ unit tests (parsing, chunking, fusion, metrics, LLM adapter) + integration
- ```
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.