Skip to content
Back to search

mcp-behavioural-diff

Differential runtime analysis for MCP servers. Runs two versions under identical scenarios, captures HTTP/filesystem/process activity, filters nondeterministic noise, and traces outbound values to tool arguments or env vars. Surfaces real on-the-wire behavioural changes for human review

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mcboomer--mcp-behavioural-diff/badge.svg)](/listings/mcboomer--mcp-behavioural-diff)

HTML

<a href="/listings/mcboomer--mcp-behavioural-diff"><img src="/api/listings/mcboomer--mcp-behavioural-diff/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

RUNS=10 ./run.sh          # installs SDK + pinned servers, runs probe + evaluations

Single comparison:

npm install && node engine/mock.cjs > engine/.mockport &   # start mock
RUNS=10 node engine/cli.mjs <serverA/index.js> <serverB/index.js> scenarios/postmark.json

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 65 lines
  1. # MCP Behavioural Diff
  2. Differential **runtime** analysis for Model Context Protocol (MCP) servers.
  3. Given two versions of an MCP server, it executes both under **identical tool-call scenarios**,
  4. captures their outbound HTTP / filesystem / process activity and tool results, learns which
  5. fields are nondeterministic from repeated runs, and reports **behavioural changes** with each
  6. outbound value traced to its **observed source** (a tool argument or an environment variable).
  7. **Scope of claims (read this first).** Three separate things, kept separate:
  8. 1. **Behavioural-change detection** — did on-the-wire / result behaviour change between versions? *(this tool does this)*
  9. 2. **Suspicious-change identification** — does a change have a risk-relevant shape (new destination, unexplained value, a secret appearing in a field)? *(partially)*
  10. 3. **Maliciousness classification** — is it actually an attack? ***(this tool does NOT do this.)*** A flagged change can be a legitimate release change; a human adjudicates.
  11. It is **not** a malware detector and makes no such claim.
  12. ## Why runtime diff (vs source/AI code review)
  13. Postmark `@activecampaign/postmark-mcp` 1.0.0 → 2.1.1 is a near-total rewrite
  14. (**1,539 lines of source diff**, 4 → 24 tools, Postmark SDK → `fetch`). Its `sendEmail`
  15. outbound request is behaviourally **identical** — 0 field changes. Source review drowns in
  16. 1,539 lines; behavioural diff surfaces only what actually changed on the wire.
  17. ## Install & run (isolated environment only — see docs/SAFETY.md)
  18. ```bash
  19. RUNS=10 ./run.sh # installs SDK + pinned servers, runs probe + evaluations
  20. ```
  21. Single comparison:
  22. ```bash
  23. npm install && node engine/mock.cjs > engine/.mockport & # start mock
  24. RUNS=10 node engine/cli.mjs <serverA/index.js> <serverB/index.js> scenarios/postmark.json
  25. ```
  26. ## What a report contains
  27. - `primary_alerts` — stable-in-both-versions fields whose value changed (the signal).
  28. - `evidence_intermittent` — fields that varied in *some* runs (2..<80%); **retained for review, never dropped** (an attacker may exfiltrate intermittently).
  29. - `suppressed_noise` — high-confidence nondeterministic fields (≥80% of runs differ: nonces, per-call timestamps).
  30. - `observed_source` on each value: `tool_argument` | `environment_variable` | `unexplained` | `null`.
  31. `observed_source` is an **observed in→out relationship** via unique canary markers, **not** full
  32. taint lineage. It cannot prove data flow through arbitrary transforms, subprocesses or native code.
  33. ## Verified results (in `reports/`, reproducible via `run.sh`)
  34. - **Benign** postmark v1→v2, N=10: **1 primary alert** — a real `getDeliveryStats` output-format change (correctly detected behavioural change, not a false positive).
  35. - **Synthetic edits** (my own one-line edits to a benign package, `.example` hosts — **not malware**): new destination, secret-into-existing-field, secret-in-new-header, new webhook, conditional leak, subprocess request, result injection. See `atk/` method and `reports/example-synthetic-*`.
  36. - secret-into-existing-field: 1 primary alert, `observed_source=environment_variable` (the subtle case).
  37. - new destination: detected (exfil host + token in query).
  38. - subprocess request: **partial** — command captured, child's body not (see docs/INSTRUMENTATION.md).
  39. - conditional leak: detected **only when the triggering input is in the scenario** (dynamic-analysis coverage limit).
  40. - **Filesystem** server, N=10: `get_file_info` timestamp varied 4/10 → classified **intermittent**, retained as evidence, kept out of primary alerts.
  41. ## Network-active coverage (3 independent implementations)
  42. Postmark (SDK `https.request` **and** `fetch`), Brave Search (`fetch` GET), `@tokenizin/mcp-npx-fetch` (`fetch` GET). See `servers.manifest.json` for pinned versions + sha512.
  43. ## Layout
  44. ```
  45. engine/ shim.cjs (capture+redirect) runner.mjs (MCP driver) mock.cjs diff.mjs cli.mjs probe.mjs canaries.json
  46. scenarios/ tool-call scenarios (canary-tagged)
  47. docs/ INSTRUMENTATION.md SAFETY.md
  48. servers.manifest.json run.sh reports/
  49. ```
  50. ## Status / limitations
  51. Node.js + stdio servers only. Positional request alignment (a new request shifts indices).
  52. `strace`/eBPF syscall backstop **not yet implemented** (planned). N=10 volatility learning is a
  53. small sample. See docs for the full list.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.