Skip to content
Back to search

firmix

Tool-grounded multi-agent bug finder for STM32F4 firmware: 3 MCP servers, 4 agents, evaluated on a 50-file benchmark with pre-registered predictions.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/rushik2612--firmix/badge.svg)](/listings/rushik2612--firmix)

HTML

<a href="/listings/rushik2612--firmix"><img src="/api/listings/rushik2612--firmix/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 170 lines
  1. # Firmix 2.0
  2. **GCC compiles it. ChatGPT explains it. Firmix catches it before it crashes your device.**
  3. Firmix 2.0 is a tool-grounded multi-agent bug finder for **STM32F4 firmware**. It finds ten kinds of embedded bugs that compile
  4. cleanly and pass basic tests (ISR/main data races, missing `volatile`, FreeRTOS misuse, wrong GPIO or clock setup, buffer overflows),
  5. backs every finding with tool evidence, and proposes a fix that is checked to compile.
  6. > **Live demo: [firmix.onrender.com](https://firmix.onrender.com)** (free tier: the first visit after 15 idle minutes takes about a minute to wake up).
  7. > The *Analyze* tab runs the deterministic rules and the ARM compile check on pasted code (no LLM). The *Benchmark replay* tab shows the
  8. > recorded agent runs: pinned findings and fixes for the 50 test files, full reasoning/verifier traces for the 20 dev files.
  9. ![Firmix demo: rules + compile check on a dev file, then the recorded agent trace and fix for d05](docs/firmix_demo.gif)
  10. ## The problem
  11. These bugs compile, pass a quick test, and then crash devices in the field; finding them takes days with a debugger.
  12. General static analyzers do not understand embedded semantics (interrupt context, RTOS rules, peripheral registers), and general
  13. LLMs explain code well but invent bugs and line numbers. On the benchmark below, GCC + cppcheck found **2 of 40** seeded bugs and a
  14. single local LLM with the same rules found **0 of 40** (best prompt variant: 9 of 40, with 42 false alarms).
  15. ## Results
  16. Frozen test set: **40 buggy + 10 clean near-miss files**, ground truth fixed before any detector existed (git tag `benchmark-frozen`).
  17. One local model for every LLM row: `qwen3:8b` via Ollama, temperature 0, seed 42. Run once, at git tag `pre-test-eval`, with the
  18. predictions written down first ([CHANGELOG entry 25](CHANGELOG.md)). Recall counts a bug only if the right type is reported on the
  19. right line. Some bugs span lines, so the ground truth lists every accepted line; a second report on another accepted line of a bug
  20. already found counts as neither an extra TP nor an FP. That is why full Firmix shows 33/33 although it made 35 reports (f035 and f048
  21. were each reported on two accepted lines), and rules-only shows 38/45 from 49 reports.
  22. | Setup | Precision | Recall (40 bugs) | 95% CI for recall | LLM calls | Tokens |
  23. |---|---|---|---|---|---|
  24. | GCC `-Wall -Wextra` + cppcheck (lenient: type not checked) | n/a | 5% (2/40) | 1-17% | 0 | 0 |
  25. | Single LLM, no tools, v1 prompt (pre-registered) | 0% (0/17) | 0% (0/40) | 0-9% | 50 | 133k |
  26. | Single LLM, no tools, one call per bug type *(post-hoc, tuned on dev)* | 18% (9/51) | 22% (9/40) | 12-38% | 500 | 698k |
  27. | Rules only (code-facts, no LLM) | 84% (38/45) | 95% (38/40) | 83-99% | 0 | 0 |
  28. | Agents, no verifier | 90% (37/41) | 92% (37/40) | 80-97% | 98 | 106k |
  29. | **Full Firmix** (planner, reasoning, verifier) | **100% (33/33)** | **82% (33/40)** | 68-91% | 142 | 145k |
  30. Full Firmix with temperature 0.3 over three seeds: precision 99% +- 2, recall 78% +- 4 (31, 30, 33 of 40). The single LLM stays at
  31. 0 of 40 in all three sampled runs.
  32. **What this shows, and what it does not.**
  33. - **The tools are what works.** Tool-grounded setups find 33-38 bugs; the best tool-free LLM design finds 9, at 4.8x the tokens.
  34. Even with line anchoring ignored (right bug type anywhere in the file) the best tool-free variant reaches only 38%.
  35. - **The LLM stages trade recall for precision; they do not improve both.** Rules alone: 84% precision, 95% recall. Full Firmix:
  36. 100% precision, 82% recall. On F1 the three are about equal (0.89, 0.91, 0.90), and their recall confidence intervals overlap, so
  37. at 40 bugs the recall differences among them are **not statistically significant**. What the verifier does buy is **zero false
  38. alarms on the 10 clean files** and a written, line-cited reason for every finding.
  39. - **Where the bugs are lost** (test set): 2 never became candidates (f029, f036), the reasoning agent dropped 1 (f004), the
  40. verifier refuted 4 real bugs (f003, f024, f037, f041) and correctly removed all 4 remaining false positives.
  41. ### Per bug group (exact-line recall; test set, pinned run)
  42. | Group (bugs) | Rules only | Agents, no verifier | Full Firmix recall | Full Firmix recall, sampled | Full Firmix precision | Single LLM (one call per type) |
  43. |---|---|---|---|---|---|---|
  44. | Interrupts (12) | 92% | 92% | 83% | 81% +- 13 | 100% | 33% |
  45. | RTOS (12) | 100% | 100% | 100% | 89% +- 5 | 100% | 25% |
  46. | Peripherals / HAL (8) | 100% | 88% | 62% | 58% +- 14 | 100% | 0% |
  47. | Memory (8) | 88% | 88% | 75% | 79% +- 14 | 100% | 25% |
  48. Peripherals/HAL is the weakest group for the full system: the verifier refuted 2 of its 7 candidates that were true bugs.
  49. ### Fixes
  50. The fixer proposes line-range edits for each correctly found bug. **Code, not the model, decides success**: the patch must be well
  51. formed, compile with no new warnings, make the detector rule stop flagging the bug, and stay small.
  52. | Measure (35 attempts on true positives) | Result |
  53. |---|---|
  54. | Patch compiles | 25 / 35 |
  55. | Compiles + rule no longer flags the bug + small diff | 20 / 35 |
  56. | **Fully correct on author review of the code** | **16 / 35 (46%)**, plus 1 partial |
  57. Three of the 20 rule-successful fixes were **wrong although they pass every automated check**: f048 moved a label so both validation
  58. failures now fall through to the save (and left the mutex release in dead code); f006 turned an ADC input pin into an output to
  59. silence the rule; f025 deleted a busy-wait and left a comment asking someone to fix it. By bug type, INT-02 was fixed 0 of 4 times,
  60. INT-03 1 of 4, RTOS-01 1 of 4. The lesson: "compiles and the rule is satisfied" can be gamed, so fix correctness needs a human read.
  61. Per-fix verdicts: `results/test_main/fix_review.json`.
  62. ## How it works
  63. ```
  64. STM32 C file
  65. |
  66. [ code-facts MCP server ] tree-sitter AST, macro table, ISR context, shared variables,
  67. | NVIC priorities, GPIO/clock configs -> deterministic CANDIDATES (file, line, type, evidence)
  68. v
  69. Planner (code) groups candidates; an LLM triage variant was tested and rejected (see below)
  70. v
  71. Reasoning agent one candidate at a time; rule text from [ knowledge MCP server ] (BM25 over STM32 docs + the SPEC)
  72. | schema-forced verdict + exact code quote; the LINE always comes from the tool, never the model
  73. v
  74. Verifier stage 1 (code): does the quote match the line, does the suspect reproduce?
  75. | stage 2 (LLM): try to refute with a cited guard; the refutation only counts if code confirms
  76. v the quote, and code rejects hedged reasoning or a "guard" that is the suspect line itself
  77. Fixer minimal edits -> [ build MCP server ] compile + warnings check -> re-run the rule -> retry (max 2)
  78. ```
  79. - **Three MCP servers:** code facts (14 tools), knowledge (4 tools, 1,460 chunks), build (5 tools: arm-none-eabi-gcc, cppcheck).
  80. - **Four agents:** planner, reasoning, verifier, fixer.
  81. - **Hallucination control:** every finding carries tool evidence; code validates line numbers and quotes; an LLM refutation or fix
  82. is never trusted without a code check. Candidate recall is the ceiling for everything downstream (38/40 on test).
  83. - **Retrieval check:** 17 of 25 hand-written queries return a correct chunk in the top 3 (BM25, exact API names).
  84. ### The ten bug types (`SPEC.md`)
  85. Interrupts: INT-01 missing `volatile` on an ISR-shared variable, INT-02 non-atomic read-modify-write shared with an ISR,
  86. INT-03 blocking call in an ISR. RTOS: RTOS-01 non-ISR-safe API from an ISR, RTOS-02 mutex not released on every path,
  87. RTOS-03 `FromISR` call at priority < 5. Peripherals: PER-01 peripheral used without its clock enable, PER-02 GPIO mode contradicts
  88. use. Memory: MEM-01 fixed-size buffer overflow, MEM-02 stack buffer passed to an async HAL transfer.
  89. ## Evaluation integrity
  90. - **Benchmark frozen** before any detector ran (`benchmark-frozen`). Prompts, rules and thresholds were developed on a separate
  91. **dev set only** (20 files: d01-d12, then d13-d20 written later to measure out-of-sample behaviour); the test split is refused by
  92. every script unless `--allow-test` is passed.
  93. - **Pre-registered**: the test run was logged with predictions first. They held: recall 60-85% (82%), precision 85-100% (100%),
  94. single LLM < 15% (0%), fixer rule success < 90% (57%).
  95. - **Every change is in [CHANGELOG.md](CHANGELOG.md)** (30 entries), including mistakes: a verifier rule I over-generalized from one
  96. file (caught by fresh dev files), a fixer prompt that showed two stacked line labels, a baseline harness bug.
  97. - **Post-hoc rows are labelled.** The single-LLM v1 result (0/40) is pre-registered and weak evidence, because that prompt made the
  98. model answer with an empty list for every dev file. A fairer tool-free baseline was tuned on dev afterwards and run once on test.
  99. - **Out-of-sample check:** the first run on d13-d20 found 11/13 bugs with 1 false positive before any change; that false positive
  100. exposed the over-general verifier rule above.
  101. ## Limitations
  102. - **One 8B local model for every LLM row.** A stronger model could narrow the gap to the tool-grounded setups; this was not tested.
  103. - **Small benchmark.** 40 bugs: confidence intervals are wide (recall 68-91%), and the three tool-grounded setups cannot be told
  104. apart on recall.
  105. - **Synthetic benchmark by the same author as the detector.** Files were frozen before the detector existed and audited, but the
  106. same person's mental model shaped both. Files are 40-150 lines with one seeded bug; real firmware is larger and messier.
  107. - **Scope:** STM32F4 only, ten bug types, HAL + FreeRTOS style. The pin-to-alternate-function table covers one chip.
  108. - **Known detector gaps:** f029 and f036 (test), and d14 (dev: a `memcpy` length taken from a received byte). The MEM-01 rule
  109. covers `i <= N` and constant sizes, not lengths from received data. Candidate rules are frozen as v1.
  110. - **Recall cost of the verifier:** it removed 4 real bugs to remove 4 false alarms. Dev numbers (92% recall) overstated test recall (82%).
  111. - **Fix correctness is the weakest part** (46% fully correct), and "bug removed" is judged by the same rule that found the bug.
  112. - **"Hallucination" for tool-grounded setups (16-17%)** counts findings whose model-written quote is not on the line; the line itself
  113. is tool-proposed, so this is a sloppy quote, not an invented location. In Full Firmix the displayed quote is replaced by the real line.
  114. - **Planner:** candidates per file average about 1 on this benchmark (49 over 50 test files), so an LLM planner could save at most
  115. about 7 of 49 reasoning calls while risking recall. The planner therefore stays in code; the LLM triage variant is kept as an experiment.
  116. ## Reproduce
  117. ```
  118. python tools/setup_deps.py # fetch HAL / CMSIS / FreeRTOS headers
  119. python tools/check_compile.py benchmark # every benchmark file compiles
  120. python tools/validate_gt.py # ground truth matches the files
  121. python tools/build_knowledge.py # build the BM25 index
  122. python tools/run_baseline.py # GCC + cppcheck baseline
  123. python tools/run_eval.py --split dev # dev evaluation (test split needs --allow-test)
  124. python tools/run_fixer.py --split dev # fixer on dev
  125. ```
  126. Requires Ollama with `qwen3:8b`, `arm-none-eabi-gcc` and `cppcheck`. Plumbing tests (no model needed): `tools/test_*_plumbing.py`.
  127. ## Repository map
  128. | Path | What |
  129. |---|---|
  130. | `SPEC.md` | scope, the ten bug definitions, benchmark and evaluation rules |
  131. | `benchmark/` | 50 frozen test files, 20 dev files, `ground_truth.json` |
  132. | `src/codefacts/` | tree-sitter facts, candidate rules, pin map, API (14 tools) |
  133. | `src/knowledge/` | chunker, BM25 store, API (4 tools) |
  134. | `src/buildtools/` | compile and cppcheck runner, fix check (5 tools) |
  135. | `src/servers/` | the three MCP servers |
  136. | `src/agents/` | planner, reasoning, verifier, fixer, single-LLM baselines, pipeline |
  137. | `tools/` | evaluation harness, baseline, plumbing tests |
  138. | `results/` | baseline, dev, test (main, robustness, post-hoc) tables and raw JSON |

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.