Skip to content
Back to search

federal-courts-analyst

Federal Courts Analyst: an edition-aware MCP server and watsonx Orchestrate agent for federal district court workload statistics

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/gsa-mcp-hackathon--federal-courts-analyst/badge.svg)](/listings/gsa-mcp-hackathon--federal-courts-analyst)

HTML

<a href="/listings/gsa-mcp-hackathon--federal-courts-analyst"><img src="/api/listings/gsa-mcp-hackathon--federal-courts-analyst/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 347 lines
  1. # Federal Courts Analyst
  2. **An MCP server and a watsonx Orchestrate agent that give the official
  3. federal court caseload statistics, with the edition each number was
  4. published in.**
  5. Ask "how busy were the federal courts in FY2020?" and you can get four
  6. official answers. The judiciary publishes the same statistic in several
  7. editions of its tables and revises it between them. National weighted civil
  8. filings per judgeship for FY2020 were **first published as 554** (Judicial
  9. Business 2020, repeated through 2022) and **revised to 322** (Judicial Facts
  10. and Figures 2023 onward). N.D. Fla. accounts for about 211 of the 232-point
  11. drop: in FY2020 over 200,000 cases of the 3M earplug multidistrict
  12. litigation were filed directly there, which suggests the revision excluded or
  13. down-weighted them. No published note says so, and the server labels that
  14. explanation *inferred* until the AO confirms it. A search engine
  15. or a general-purpose assistant mixes
  16. these figures from different editions without noticing. This server never
  17. does that. Every figure comes with its edition and table, the
  18. first-published value and why the editions differ.
  19. Built for GSA's [MCP Server and SI Agent Government Hackathon](https://www.gsa.gov/super-intelligence/si-community-of-practice/events-and-training/mcp-server-and-si-agent-government-hackathon),
  20. on the [GSA MCP hackathon template](https://github.com/GSA-TTS/mcp-hackathon-template),
  21. in IBM's environment: the IBM Bob IDE on a TechZone VM, with the agent in
  22. **watsonx Orchestrate**.
  23. | Hackathon application | What this repository does |
  24. | --- | --- |
  25. | **Dataset access** | Read tools over an edition-aware catalog of the official uscourts.gov tables (FY2016–FY2025, 60 table editions) and the FJC civil Integrated Database |
  26. | **Service "write" integration (sandbox)** | A request desk: a user can keep a checked answer, watch its figures, or ask the statistics office why a figure was revised. Nothing is saved without the user's explicit yes, and nothing reaches the AO or FJC |
  27. ## Start here
  28. | | |
  29. | --- | --- |
  30. | **Evaluation summary** | [`eval/SUMMARY.md`](eval/SUMMARY.md): method, results, findings, limits |
  31. | **The watsonx Orchestrate agent** | [Running on watsonx Orchestrate](#running-on-watsonx-orchestrate), instructions [`eval/wxo/instructions/v8.txt`](eval/wxo/instructions/v8.txt) |
  32. | **Check the evaluation yourself** | [`eval/VERIFY.md`](eval/VERIFY.md): re-derive every number from the saved replies in minutes, with no keys or accounts |
  33. | **Reproduce the evaluation** | [`eval/wxo/README.md`](eval/wxo/README.md), `eval/wxo/run_eval.sh` |
  34. | **The request desk** | [`docs/desk.md`](docs/desk.md), [ADR 0013](docs/adr/0013-request-desk.md) |
  35. | **Security** | [`SECURITY.md`](SECURITY.md), [threat model](docs/security/threat-model.md) |
  36. | **Design decisions** | [`docs/adr/`](docs/adr/README.md) |
  37. | **Data sources and methodology** | [`docs/data-sources.md`](docs/data-sources.md), [`docs/methodology/`](docs/methodology/) |
  38. | **Every revised figure** | [`docs/revision-log.md`](docs/revision-log.md) |
  39. ## Results
  40. The deployed agent: `courts-analyst-syed` on watsonx Orchestrate, model
  41. `groq/openai/gpt-oss-120b`, instructions v8, 8 of this server's tools. Every
  42. item is asked 3 times and scored **deterministically** against the official
  43. tables (no LLM judge). Full detail in [`eval/SUMMARY.md`](eval/SUMMARY.md).
  44. | Run | Items × repeats | Mean (95% CI) | pass^3 |
  45. | --- | --- | --- | ---: |
  46. | **Held-out `test2`**, written before any agent run, scored once | 45 × 3 | **80.0% (72.5–85.9%)** | 73.3% |
  47. | Dev split, on the deployed server | 117 × 3 | 88.3% (84.5–91.3%) | 80.3% |
  48. | Dev split, draft agent with the 4 case-level tools added (12 tools) | 117 × 3 | 92.9% (89.7–95.1%) | 88.9% |
  49. - **The server is what makes the answers possible.** The same model (Claude
  50. Sonnet) on the same 117 dev items scored **93.5% with the server and
  51. 20.6% without it**. With the server it got all 38 single-figure answers
  52. right, each with its edition; without it, none.
  53. - **Figures are reliable.** On held-out, every single-figure answer had the
  54. official value with the right edition, 3 times out of 3.
  55. - **Saving is safe.** In 36 desk test runs since the server-side check, no
  56. answer was saved without the user's approval. Before the server-side check, 2 of
  57. 10 runs did.
  58. - **Every tool is evaluated, and the evidence picks the agent's 8.** Claude
  59. runs on the full server cover the 4 case-level tools. They now run on wxO
  60. too, from a compact copy of the IDB shipped inside a second toolkit: on
  61. all 12 tools, dev on the same 117 items was 92.9% (89.7–95.1), pass^3
  62. 88.9%, with the engine items rebuilt from the IDB; on Live 1.2.0, with
  63. the case-type fix, 88.6% (84.9–91.5) in a second run (EVALUATION.md 3.1). Three step-by-step save tools trust
  64. the model's `confirm`: in scripted desk conversations Claude Sonnet saved
  65. before asking in 6 of 6 save conversations with them, and 0 of 15 with
  66. `keep_answer`'s server-checked preview. Table:
  67. [`eval/EVALUATION.md`](eval/EVALUATION.md), section 3.12.
  68. - **Main weakness: clarifying.** Asked a vague question ("which court was
  69. busiest?"), the agent usually picks a measure instead of asking which one
  70. is meant (1 of 12 times on held-out).
  71. The held-out run used the server build from before the desk fixes. Those
  72. fixes touch only the desk tools, and dev on the deployed build shows no
  73. regression (88.3% against 88.0% before). Dev scores are re-scored after a
  74. scorer fix for thousands separators written as narrow spaces ("297 877");
  75. as first scored they were 85.2% and 86.3%. Held-out was unaffected. A
  76. later fix, found by checking the scorer against an LLM judge, would raise
  77. held-out to 81.5%; we keep 80.0% as the headline because the fix was found
  78. by reading held-out replies.
  79. ## How it works
  80. ```mermaid
  81. flowchart LR
  82. U[User] --> A["watsonx Orchestrate agent<br/>gpt-oss-120b, instructions v8"]
  83. A -- "MCP over stdio" --> S["courts-analyst server<br/>(local MCP toolkit, runs inside wxO)"]
  84. S --> C[("Official tables catalog<br/>DuckDB, read-only")]
  85. S --> D[("Request desk<br/>SQLite sandbox")]
  86. ```
  87. - **Tools are built around what the user wants to know**, not around tables:
  88. "why did this change", "what is the official figure", "how does this court
  89. compare". Each returns a Pydantic model, published as the tool's
  90. `outputSchema`, with provenance (edition, table, first-published value) on
  91. every figure.
  92. - **The domain logic lives in `core/`**, with no MCP or HTTP imports. The MCP
  93. tools, the REST API and the web app are thin adapters over the same
  94. functions ([ADR 0002](docs/adr/0002-core-adapter-architecture.md)).
  95. - **One guardrail layer covers every tool.** No mixing of editions in a
  96. series, small-cell suppression, factual answers for years not held,
  97. structured declines for judge-level and party-name requests, an output
  98. scanner for party names, and neutral wording on judgeship policy.
  99. - **The official data is read-only.** The catalog and the IDB are attached
  100. `READ_ONLY`; only the desk's separate sandbox store is writable.
  101. ## Tools
  102. **Read tools.** ✓ marks the tools attached to the watsonx Orchestrate agent.
  103. | Tool | wxO | Answers |
  104. | --- | :---: | --- |
  105. | `get_official_figure` | ✓ | One published statistic in the latest edition, with the first-published value, every edition and why they differ |
  106. | `explain_change` | ✓ | Why a measure changed between two fiscal years: breakdowns by district and case type, drivers labelled official, IDB-derived or inferred |
  107. | `compare_courts` | ✓ | Courts ranked on one measure, all from one edition |
  108. | `judgeship_context` | ✓ | Weighted filings against the Judicial Conference's 430/600 thresholds and its recommendations, never its own |
  109. | `explain_term` | ✓ | Definitions of terms, measures and nature-of-suit codes |
  110. | `find_cases` / `analyze_cohort` | | A cohort of IDB civil cases; medians, outcome rates and trends with n and 95% CIs |
  111. | `reconcile` | | Whether the IDB reproduces a published count, and why not |
  112. | `sample_dockets` | | Example docket references for CourtListener, labelled "match may be inexact" |
  113. | `server_info` | | Version, protocols and which datasets are loaded |
  114. The case-level tools need the IDB. The full build (440 MB) is larger than
  115. watsonx Orchestrate SaaS accepts in a toolkit upload (413). A compact copy
  116. fits: `make data-idb-slim` keeps every record and only the columns the server
  117. reads, stores the record table as a view over a narrower table, and gives
  118. identical answers (37 MiB zipped with the code and catalog). It is
  119. registered as a second toolkit, `courts_analyst_syed_idb`. Since 2026-11-17
  120. **Live is version 1.2.0** with 12 tools: the 8 ✓ plus these 4, from a third
  121. toolkit, `courts_analyst_syed_v2`. It adds a fix found on Live: asked for a
  122. case type in words ("patent cases"), version 1.1.0 guessed the wrong
  123. nature-of-suit code. Now `explain_term` maps case types to codes, and on 7
  124. new dev items worded that way the agent never used a wrong code (21
  125. attempts).
  126. **Request desk tools** (sandbox, [docs/desk.md](docs/desk.md)). Asking a
  127. question never writes anything.
  128. | Tool | wxO | Does |
  129. | --- | :---: | --- |
  130. | `keep_answer` | ✓ | Checks the answer, then saves it as a verified copy, a watch, or a question to the statistics office |
  131. | `my_requests` | ✓ | The asker's inbox: status, verified answer, the office's response, updates |
  132. | `check_inquiry_eligibility` | ✓ | Whether a figure's revision can go to the statistics office (read-only) |
  133. | `find_published_answers` | | Search approved answers in the public library (read-only) |
  134. | `open_request`, `save_verified_answer`, `file_revision_inquiry` | | The same steps one at a time |
  135. | `inquiry_queue`, `record_official_response`, `publish_answer` | | The simulated statistics office; registered only with `DESK_STAFF_MODE=true` |
  136. Plus 7 resources (`courts://catalog`, `courts://revisions`,
  137. `courts://glossary`, `courts://editions/{edition}/{table}`,
  138. `courts://codebook/{field}`, `courts://methodology/{topic}`,
  139. `desk://library`) and 5 prompts (`district_caseload_profile`,
  140. `what_changed_and_why`, `judgeship_briefing`, `compare_editions`,
  141. `keep_this_answer`). An ambiguous question (raw or weighted filings?) is
  142. asked back through elicitation on clients that support it, and returned as
  143. `needs_clarification` otherwise.
  144. ### How the desk keeps a save safe
  145. 1. **The user decides.** The agent describes what will be saved and asks
  146. "Shall I go ahead?". It never calls `keep_answer` on its own initiative.
  147. 2. **The server checks the preview.** A `confirm=false` call returns a
  148. single-use `preview_id` tied to a digest of exactly what would be saved
  149. (text, option, asker, citations). The `confirm=true` call must pass it
  150. back, so a changed answer or a retry after a refusal needs a new preview
  151. and a new yes. The step-by-step tools don't have this check, which is one
  152. reason they aren't attached to the agent.
  153. 3. **The verification gate.** Every number in the answer must be a cited
  154. figure that matches what its edition printed, a number the server's own
  155. read tools produce, a published threshold, or a difference between those.
  156. The figures for each measure must come from consistent editions. A refusal
  157. names each offending number and what to write instead.
  158. 4. **The transcript checker.** The server can't prove the user saw the
  159. preview, so `courts-eval preview-check` checks every evaluation
  160. transcript for a confirm without a "Shall I go ahead?" and a yes. It
  161. caught violations that IBM's own evaluator marked as successful.
  162. ## Running on watsonx Orchestrate
  163. The server goes into watsonx Orchestrate as a **local MCP toolkit**:
  164. Orchestrate installs it and runs it over stdio, with no HTTP endpoint or
  165. container. The full runbook is
  166. [`deploy/ibm/local-mcp-toolkit/README.md`](deploy/ibm/local-mcp-toolkit/README.md).
  167. 1. Build the catalog: `make data-tables`. For the case-level tools, also
  168. `make data-idb data-idb-slim`.
  169. 2. Configure `deploy/ibm/local-mcp-toolkit/.env` (instance URL, environment
  170. and toolkit names; no secrets) and put the API key in `WXO_API_KEY`.
  171. 3. Register: `deploy/ibm/local-mcp-toolkit/register-toolkit.sh`
  172. (`INCLUDE_IDB=1` ships the compact IDB). It won't replace an existing
  173. toolkit unless `REPLACE_EXISTING=1`, because replacing takes its tools off
  174. every agent. Snapshot, deploy and rollback steps:
  175. [`ROLLBACK.md`](deploy/ibm/local-mcp-toolkit/ROLLBACK.md).
  176. 4. In Orchestrate, create an agent with the model `groq/openai/gpt-oss-120b`,
  177. attach the 8 tools marked ✓ above, and paste
  178. [`eval/wxo/instructions/v8.txt`](eval/wxo/instructions/v8.txt) as its
  179. instructions.
  180. `eval/wxo/agent_check.py` checks that a live agent matches an instruction
  181. file and tool count before an evaluation run.
  182. **The agent** is deployed to Live on our instance and available in
  183. Orchestrate Chat: version 1.2.0 (v8, 12 tools, the case-level tools
  184. included, with the case-type fix) since 2026-11-17. Version 1.0.0, "v8 evaluated, submission" (8
  185. tools), is the configuration held-out test2 measured, and stays in the
  186. version history. Its
  187. **Evaluate** page holds six showcase tests run by IBM's built-in evaluator
  188. (6 of 6 pass): see [`eval/EVALUATION.md`](eval/EVALUATION.md), section 3.7.
  189. ## Running locally
  190. Needs [uv](https://docs.astral.sh/uv/) and `make`. Docker is optional.
  191. ```bash
  192. make setup # install dev + pipeline deps, pre-commit hooks, create .env
  193. make test # pytest with coverage (builds both datasets from committed fixtures)
  194. make data-tables # download the official tables (≈4 min) and build courts.duckdb
  195. make data-idb # optional: download the FJC civil IDB (330 MB) and build idb.duckdb (≈1.5 min)
  196. make serve # MCP over stdio
  197. make serve-http # MCP over streamable HTTP on :8000 (/mcp, /health, /version)
  198. make help # every target
  199. ```
  200. Any MCP client can connect. For a client that launches stdio servers (Claude
  201. Desktop, VS Code, IBM Bob), add:
  202. ```json
  203. {
  204. "mcpServers": {
  205. "courts-analyst": {
  206. "command": "uv",
  207. "args": ["run", "--quiet", "--directory", "/absolute/path/to/this/repo", "courts-analyst"]
  208. }
  209. }
  210. }
  211. ```
  212. The repo ships this as `.mcp.json` and `.bob/mcp.json`. `make inspector`
  213. opens the MCP Inspector. More in [QUICKSTART.md](QUICKSTART.md).
  214. Over HTTP the same process also serves a REST API with the same result
  215. envelopes as the tools (`/api`, schema at `/api/openapi.json`), a streaming
  216. agent (`POST /api/ask`, [ADR 0011](docs/adr/0011-agent-and-api.md)) and a web
  217. app at `/` ([ADR 0012](docs/adr/0012-web-app.md),
  218. [web/README.md](web/README.md)).
  219. ## Evaluation
  220. The harness lives in `src/courts_analyst/evaluation/` and runs as
  221. `courts-eval`. The question bank (`eval/questions/`, 231 items) has a known
  222. answer and the expected behaviour (answer, clarify or decline) for each item,
  223. across five suites: single figures, explanations, tool selection, the
  224. analytics engine, and adversarial items (out of scope, advocacy, judge and
  225. party names, edition-mixing traps, prompt injection, over-refusal controls).
  226. - **Scoring is deterministic.** Numbers are matched against the official
  227. tables at printed precision, the edition label must match, claims are
  228. checked by phrase, and behaviour is inferred from the reply and the tool
  229. calls. "554 (JFF-2025)" fails, because JFF-2025 prints 322.
  230. - **Checked against an LLM judge.** `courts-eval judge` re-grades saved
  231. replies with an LLM judge in the GSA template's style. It agrees with our
  232. scorer on 96.7% of the agent's replies, but it passed all 155 replies we
  233. stripped of their edition labels, and it changes its verdict on 3% of
  234. replies when asked again. Reading the disagreements found one scorer bug,
  235. now fixed ([`eval/EVALUATION.md`](eval/EVALUATION.md), section 3.10).
  236. - **Two layers against the real agent.** Layer 2 sends the question bank to
  237. the watsonx Orchestrate agent over its runs API (`--answerer wxo`). Layer 1
  238. runs IBM's `orchestrate evaluations` on multi-turn desk conversations, and
  239. `courts-eval preview-check` checks their transcripts.
  240. - **Tuned on dev only.** Instructions v0–v13 and why each was kept or not are
  241. in [`eval/wxo/README.md`](eval/wxo/README.md). The held-out `test2` split
  242. was scored once.
  243. ```bash
  244. eval/wxo/run_eval.sh dev # dev split, 3 repeats, plus the preview check
  245. DESK=1 eval/wxo/run_eval.sh dev # also the three Layer 1 desk tests
  246. make eval # harness self-check: gold answers 100%, null 0%
  247. make eval-write-safety # the desk's write-safety suite
  248. ```
  249. `run_eval.sh` needs `WXO_INSTANCE_URL`, `WXO_API_KEY` and `WXO_AGENT_ID`;
  250. read the key with `read -rs` so it is never echoed. Every committed run is in
  251. [`eval/results/`](eval/results/README.md).
  252. ## Repository layout
  253. ```
  254. src/courts_analyst/
  255. ├── app.py # entry point: build_server() + transport choice
  256. ├── config.py # typed settings (pydantic-settings) from env / .env
  257. ├── models.py # shared Pydantic models (tool outputs, provenance)
  258. ├── tools/ # one file per MCP tool, each with register(mcp)
  259. ├── resources/ # courts:// and desk:// resources
  260. ├── prompts/ # analyst prompts
  261. ├── core/ # domain logic, no MCP/HTTP imports (ADR 0002)
  262. │ ├── guardrails/ # editions (vintage), small cells, periods, intent, policy, PII
  263. │ └── desk/ # request desk: gate, previews, store, service
  264. ├── pipeline/ # data ingestion: official tables and the IDB
  265. ├── evaluation/ # courts-eval: question bank, scorers, answerers (incl. wxo)
  266. ├── api/ # FastAPI REST + agent SSE at /api
  267. └── agent/ # the web app's agent loop
  268. tests/ # unit, golden, MCP protocol, transport, container tests
  269. eval/ # question bank, results, SUMMARY.md, wxo/ (instructions, Layer 1 tests)
  270. deploy/ibm/ # watsonx Orchestrate kits (local MCP toolkit, Code Engine)
  271. docs/ # ADRs, desk, security, methodology, revision log
  272. web/ # React web app served at /
  273. plan/ # the phased build plan
  274. ```
  275. ## Configuration
  276. Every setting is optional; see [`.env.example`](.env.example) and
  277. [`config.py`](src/courts_analyst/config.py).
  278. | Variable | Default | Purpose |
  279. | --- | --- | --- |
  280. | `MCP_TRANSPORT` | `stdio` | `stdio` or `streamable-http` (a host-injected `PORT` forces HTTP) |
  281. | `MCP_HOST` / `MCP_PORT` | `0.0.0.0` / `8000` | HTTP bind address |
  282. | `DATA_DIR` / `DUCKDB_PATH` | `data` / `$DATA_DIR/build/courts.duckdb` | Downloads, build outputs, the catalog |
  283. | `DESK_STORE_PATH` | unset (in memory) | Request desk sandbox store |
  284. | `DESK_STAFF_TOKEN` / `DESK_STAFF_MODE` | unset / `false` | The simulated statistics office's screen and MCP tools |
  285. | `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `AGENT_MODEL` | unset | Only for the web app's agent and paid eval answerers |
  286. | `LOG_LEVEL` / `LOG_FORMAT` | `INFO` / `auto` | `auto` = JSON over HTTP, text on stdio |
  287. ## Adding a tool
  288. 1. Put the logic in `core/` with unit tests.
  289. 2. Create `tools/<name>.py` with `register(mcp)`, returning a Pydantic model.
  290. 3. Call it from `tools/__init__.py` and add the name to `EXPECTED_TOOLS` in
  291. `tests/test_registration.py`.
  292. Checks before every commit: `uv run ruff check . && uv run ruff format --check . && uv run mypy && uv run pytest -q`.
  293. Built with [FastMCP 4](https://gofastmcp.com) (MCP spec 2026-07-28, backward
  294. compatible with 2025-11-25 clients) and [uv](https://docs.astral.sh/uv/).
  295. ## License
  296. [MIT](LICENSE). See [SECURITY.md](SECURITY.md) for the vulnerability disclosure policy.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.