Skip to content
Back to search

multi-agent-workflow-lab

Testing and observability for multi-agent delegation, MCP tools, permissions, sandboxed actions, prompts, and workflow replay.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/christiangrey922--multi-agent-workflow-lab/badge.svg)](/listings/christiangrey922--multi-agent-workflow-lab)

HTML

<a href="/listings/christiangrey922--multi-agent-workflow-lab"><img src="/api/listings/christiangrey922--multi-agent-workflow-lab/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requirements: Node.js 22 or newer and pnpm 11.

From a clone or an extracted release archive:

cd multi-agent-workflow-lab
pnpm install --frozen-lockfile
pnpm demo

The demo requires no API key or paid service. A public clone URL can be added after the maintainer chooses and publishes the repository location.

What the demo shows

`pnpm demo` runs a deterministic review chain through the CLI:

status: completed

orchestrator
  → researcher
    → analyst
      → reviewer
        → evaluator

delegation score: 88.75/100
workflow completed

The actual CLI output is JSON and includes the run ID, every task/agent/status, all eight evaluation dimensions, derived metrics, and the JSONL trace path.

Delegation observability

A trace records behavior rather than only the final answer:

orchestrator
  delegation.requested → researcher
  delegation.created   context + permissions + budget + depth

researcher
  tool.requested       → mcp.mock.search
  policy.allowed       explicit decision evidence
  tool.completed       size + redaction metadata

orchestrator
  delegation.result.accepted → child task integrated

Use these commands after a run:

pnpm mawl trace show <run-id>
pnpm mawl evaluate <run-id>
pnpm mawl graph <run-id>
pnpm mawl replay <run-id> dry-run

Evaluation scores are normalized diagnostics, not mathematical truth or a security guarantee.

Example workflows

| Workflow | Purpose | Expected result | | ---------------------------------- | ---------------------------------------------------------- | ---------------------------------------------- | | `01-basic-delegation.yaml` | Orchestrator delegates one bounded research task | Completes | | `02-parallel-research.yaml` | Two research tasks fan out and an analyst joins them | Completes | | `03-review-chain.yaml` | Researcher → analyst → reviewer → evaluator | Completes | | `04-permission-denied.yaml` | Direct delegation to a non-allowlisted target | Fails safely | | `05-mcp-untrusted-output.yaml` | Hostile MCP content is treated as data in the test harness | Completes safely | | `06-delegation-loop.yaml` | Agent A → B → A recursion | Fails safely | | `07-human-approval.yaml` | Approval-provider checkpoint behavior | Completes; approval logic is tested separately | | `08-budget-exceeded.yaml` | Workflow-wide token budget exhaustion | Fails safely | | `09-bad-vs-good-orchestrator.yaml` | Good baseline for comparative evaluation | Completes |

Run any positive example with:

pnpm mawl run workflows/02-parallel-research.yaml

The negative examples intentionally return a non-zero exit code when the requested action is blocked.

Good vs. bad orchestrator

The lab compares a bounded, least-privilege orchestration trace with a synthetic bad trace containing duplicate work, leaked context, escalated permission, and an unauthorized tool request. `RunComparator` and `DelegationEvaluator` report changes in:

- agent selection and delegation edges; - task decomposition and depth; - context and permission minimization; - tool use, retries, tokens, runtime, and failures; - result integration and overall diagnostic score.

The executable coverage is in [`tests/lab.test.ts`](tests/lab.test.ts), with scenario inputs under [`scenarios/`](scenarios/).

CLI

In this workspace, prefix CLI arguments with `pnpm mawl`:

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 267 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Multi-Agent Workflow Lab
  2. An open-source testing and observability framework for multi-agent delegation, tool execution, MCP workflows, permissions, sandboxed actions, prompts, and runtime behavior.
  3. > **Status:** Experimental — v0.1.0 release candidate. Suitable for local development, evaluation, and policy testing; not production-hardened infrastructure.
  4. ## Why this exists
  5. Most model evaluation stops at `input → model → output`. Multi-agent systems add behavior that a final answer cannot explain:
  6. - Which agent delegated a task, to whom, and why?
  7. - What context, permissions, tools, MCP servers, and budget did the child receive?
  8. - Did an agent attempt privilege escalation, repeat work, or enter a loop?
  9. - Was the delegation efficient, and was the child result actually integrated?
  10. - Can the run be inspected, compared, or replayed without repeating side effects?
  11. MAWL makes those decisions explicit, policy-controlled, and traceable. Prompts are versioned assets; delegation is a first-class runtime event; deterministic rules evaluate behavior independently from optional model judges.
  12. ## Key capabilities
  13. | Area | What is implemented |
  14. | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
  15. | Agent runtime | Typed model actions, task limits, cancellation, retries, and deterministic mock execution |
  16. | Delegation | Parent/child task graph, target and capability checks, depth/fan-out limits, loop detection, and result integration events |
  17. | Tool execution | Registry-based invocation with schema validation, allowlists, permissions, policy, optional approval, redaction, timeout, and audit |
  18. | MCP | Official stdio and Streamable HTTP client adapters plus an in-memory mock connector; MCP data remains untrusted |
  19. | Sandbox | Restricted local-process adapter and optional Docker provider interface with auditable lifecycle events |
  20. | Identity and permissions | Runtime-issued agent identity, local development auth, deny-by-default permission and contextual policy engines |
  21. | Prompt system | Semantic versions, SHA-256 hashes, strict variables, trusted/untrusted provenance, and deterministic prompt-injection signals |
  22. | Input parsing | JSON, YAML, Markdown, text, and structured-task normalization |
  23. | Observability | Append-only events, SQLite/JSONL storage, structured logs, task traces, Mermaid delegation graphs, metrics, budgets, and monitors |
  24. | Evaluation | Nine deterministic rules, eight diagnostic dimensions, assertions, YAML specs, run comparison, prompt regression, and optional model judge |
  25. | Replay | Exact reconstruction, dry run, model rerun, and guarded tool rerun |
  26. | Security testing | Prompt injection, escalation, malicious MCP/tool output, secret leakage, traversal, sandbox, recursion, and resource-exhaustion cases |
  27. ## Architecture
  28. ```mermaid
  29. flowchart TD
  30. Input["User or test input"] --> Parser["Input parsers"]
  31. Parser --> Runtime["Agent runtime + task graph"]
  32. Runtime --> Delegation["Delegation engine"]
  33. Delegation --> Guards["Identity + permission + policy"]
  34. Guards --> Agents["Child agents"]
  35. Guards --> Tools["Tool executor"]
  36. Tools --> MCP["MCP connectors"]
  37. Tools --> Sandbox["Sandbox providers"]
  38. Agents --> Events["Runtime events"]
  39. MCP --> Events
  40. Sandbox --> Events
  41. Events --> Observe["Trace + metrics + monitors"]
  42. Events --> Evaluate["Rules + optional judge + replay"]
  43. ```
  44. # Quick Start
  45. Requirements: Node.js 22 or newer and pnpm 11.
  46. From a clone or an extracted release archive:
  47. ```bash
  48. cd multi-agent-workflow-lab
  49. pnpm install --frozen-lockfile
  50. pnpm demo
  51. ```
  52. The demo requires no API key or paid service. A public clone URL can be added after the maintainer chooses and publishes the repository location.
  53. ## What the demo shows
  54. `pnpm demo` runs a deterministic review chain through the CLI:
  55. ```text
  56. status: completed
  57. orchestrator
  58. → researcher
  59. → analyst
  60. → reviewer
  61. → evaluator
  62. delegation score: 88.75/100
  63. workflow completed
  64. ```
  65. The actual CLI output is JSON and includes the run ID, every task/agent/status, all eight evaluation dimensions, derived metrics, and the JSONL trace path.
  66. ## Delegation observability
  67. A trace records behavior rather than only the final answer:
  68. ```text
  69. orchestrator
  70. delegation.requested → researcher
  71. delegation.created context + permissions + budget + depth
  72. researcher
  73. tool.requested → mcp.mock.search
  74. policy.allowed explicit decision evidence
  75. tool.completed size + redaction metadata
  76. orchestrator
  77. delegation.result.accepted → child task integrated
  78. ```
  79. Use these commands after a run:
  80. ```bash
  81. pnpm mawl trace show <run-id>
  82. pnpm mawl evaluate <run-id>
  83. pnpm mawl graph <run-id>
  84. pnpm mawl replay <run-id> dry-run
  85. ```
  86. Evaluation scores are normalized diagnostics, not mathematical truth or a security guarantee.
  87. ## Example workflows
  88. | Workflow | Purpose | Expected result |
  89. | ---------------------------------- | ---------------------------------------------------------- | ---------------------------------------------- |
  90. | `01-basic-delegation.yaml` | Orchestrator delegates one bounded research task | Completes |
  91. | `02-parallel-research.yaml` | Two research tasks fan out and an analyst joins them | Completes |
  92. | `03-review-chain.yaml` | Researcher → analyst → reviewer → evaluator | Completes |
  93. | `04-permission-denied.yaml` | Direct delegation to a non-allowlisted target | Fails safely |
  94. | `05-mcp-untrusted-output.yaml` | Hostile MCP content is treated as data in the test harness | Completes safely |
  95. | `06-delegation-loop.yaml` | Agent A → B → A recursion | Fails safely |
  96. | `07-human-approval.yaml` | Approval-provider checkpoint behavior | Completes; approval logic is tested separately |
  97. | `08-budget-exceeded.yaml` | Workflow-wide token budget exhaustion | Fails safely |
  98. | `09-bad-vs-good-orchestrator.yaml` | Good baseline for comparative evaluation | Completes |
  99. Run any positive example with:
  100. ```bash
  101. pnpm mawl run workflows/02-parallel-research.yaml
  102. ```
  103. The negative examples intentionally return a non-zero exit code when the requested action is blocked.
  104. ### Good vs. bad orchestrator
  105. The lab compares a bounded, least-privilege orchestration trace with a synthetic bad trace containing duplicate work, leaked context, escalated permission, and an unauthorized tool request. `RunComparator` and `DelegationEvaluator` report changes in:
  106. - agent selection and delegation edges;
  107. - task decomposition and depth;
  108. - context and permission minimization;
  109. - tool use, retries, tokens, runtime, and failures;
  110. - result integration and overall diagnostic score.
  111. The executable coverage is in [`tests/lab.test.ts`](tests/lab.test.ts), with scenario inputs under [`scenarios/`](scenarios/).
  112. ## CLI
  113. In this workspace, prefix CLI arguments with `pnpm mawl`:
  114. ```text
  115. pnpm mawl run <workflow.yaml> [json-input]
  116. pnpm mawl agents list
  117. pnpm mawl workflow inspect <run-id>
  118. pnpm mawl task inspect <task-id>
  119. pnpm mawl inspect agent permissions <agent-id>
  120. pnpm mawl inspect mcp server [server-id]
  121. pnpm mawl trace show <run-id>
  122. pnpm mawl evaluate <run-id>
  123. pnpm mawl graph <run-id> [output.mmd]
  124. pnpm mawl replay <run-id> [exact|model-rerun|tool-rerun|dry-run]
  125. pnpm mawl compare <run-a> <run-b>
  126. pnpm mawl test <workflow-test.yaml>
  127. pnpm mawl prompts list
  128. pnpm mawl prompts inspect <prompt-id> [version]
  129. pnpm mawl permissions inspect <agent-id>
  130. pnpm mawl doctor
  131. ```
  132. `tool-rerun` blocks tool names or manifests classified as external side effects unless the embedding application supplies explicit permission for that exact tool. The CLI supplies no side-effect permissions.
  133. ## Prompts are first-class assets
  134. This is not a directory of free-form prompt snippets. YAML assets under [`prompts/`](prompts/) include:
  135. - stable ID and semantic version;
  136. - owner, type, purpose, and known risks;
  137. - declared input schema and expected output;
  138. - allowed actions and recommended capabilities;
  139. - content hash verified by `PromptRegistry`;
  140. - provenance and trust classification when assembled at runtime.
  141. Runtime and security policy layers remain separate from untrusted user, tool, MCP, and child-agent content. Prompt quality tests detect duplicate IDs, invalid versions, missing metadata, variable/placeholder errors, unexpected hashes, and undeclared references.
  142. ## Security model
  143. - Permissions and contextual policies deny unmatched requests by default.
  144. - Agent execution identities are issued by the runtime and linked to workflow/task/session IDs.
  145. - Child context, authority, and budget are reduced from the parent.
  146. - MCP responses and model/tool/child output are untrusted data, not authority.
  147. - Secret references and output redaction limit accidental exposure.
  148. - Tool requests pass manifest resolution, validation, permission, policy, optional approval, output checks, and audit.
  149. - Delegation depth, ancestry, task count, retries, calls, tokens, output, and runtime are bounded.
  150. - Replay refuses external side effects without an explicit per-tool permission.
  151. The restricted local sandbox is intended for development and policy testing. It is **not** equivalent to hardened container, VM, microVM, or kernel isolation, and network denial is best-effort. MAWL does not claim to be prompt-injection-proof, sandbox-escape-proof, or fully zero-trust. Read the [threat model](docs/threat-model.md) and [security policy](SECURITY.md).
  152. ## Using a real model provider
  153. The default path always uses `MockModelProvider`. An optional OpenAI-compatible chat-completions adapter is available in `@mawl/providers`:
  154. ```bash
  155. cp .env.example .env
  156. ```
  157. Populate `MODEL_PROVIDER`, `MODEL_ENDPOINT`, and `MODEL_API_KEY` from a secret manager, then inject `OpenAICompatibleProvider` into the runtime. The CLI intentionally does not auto-load credentials. See [provider integration](docs/provider-integration.md).
  158. ## Repository structure
  159. ```text
  160. multi-agent-workflow-lab/
  161. ├── apps/ CLI and runnable examples
  162. ├── packages/
  163. │ ├── core/ Schemas and provider contracts
  164. │ ├── runtime/ Agent loop, tasks, delegation, scheduler, replay
  165. │ ├── evaluation/ Rules, assertions, scores, specs, comparison
  166. │ ├── observability/ Events, traces, graphs, metrics, budgets, monitors
  167. │ ├── tools/ mcp/ sandbox/ Controlled execution surfaces
  168. │ ├── auth/ permissions/ policy/ security/ secrets/
  169. │ └── prompts/ parsers/ providers/ storage/ testing/
  170. ├── agents/ Agent definitions
  171. ├── prompts/ Versioned prompt assets
  172. ├── workflows/ Executable workflow examples
  173. ├── examples/ Compatibility examples and guide
  174. ├── scenarios/ redteam/ Deterministic failure and attack inputs
  175. ├── tests/ Unit, security, lab, and end-to-end tests
  176. ├── config/ Example pricing configuration
  177. └── docs/ Architecture, operations, security, and release docs
  178. ```
  179. ## Testing and development
  180. All default tests use local mock providers and require no external API access:
  181. ```bash
  182. pnpm install --frozen-lockfile
  183. pnpm typecheck
  184. pnpm lint
  185. pnpm build
  186. pnpm test
  187. pnpm test:e2e
  188. pnpm test:security
  189. pnpm demo
  190. pnpm mawl doctor
  191. ```
  192. To extend the framework, add YAML agents under `agents/`, versioned prompts under `prompts/`, workflows under `workflows/`, tool manifests through `ToolRegistry`, MCP adapters through `McpConnection`/`McpConnector`, rules through `RuleEvaluator`, and scenario coverage under `tests/` or `scenarios/`. See the [documentation index](docs/README.md) and [contributing guide](CONTRIBUTING.md).
  193. ## Roadmap
  194. - distributed execution and atomic shared budgets;
  195. - hardened remote container or microVM sandbox providers;
  196. - OpenTelemetry exporters and remote observability storage;
  197. - additional model, identity, and MCP adapters;
  198. - workflow visualization UI and benchmark datasets.
  199. ## Contributing, security, and license
  200. Contributions to agents, prompts, tools, MCP connectors, workflow scenarios, evaluators, and security tests are welcome. Read [`CONTRIBUTING.md`](CONTRIBUTING.md) and [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md).
  201. Report vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md). Do not place sensitive vulnerability details in public issues.
  202. Licensed under the [`MIT License`](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.