Skip to content
Back to search

trustabl

MarketplaceWorks with Claude CodeGitHub

Agent Reliability Analyzer: deterministic scanner that finds and fixes AI agent reliability failures

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/trustabl/agent-reliability-analyzer with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/trustabl/agent-reliability-analyzer'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/trustabl--agent-reliability-analyzer/badge.svg)](/listings/trustabl--agent-reliability-analyzer)

HTML

<a href="/listings/trustabl--agent-reliability-analyzer"><img src="/api/listings/trustabl--agent-reliability-analyzer/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install Agent Reliability Analyzer (the `trustabl` CLI) with Homebrew, Scoop, Docker or a direct download.

Homebrew (macOS, Linux)

brew install trustabl/tap/trustabl

Scoop (Windows)

scoop bucket add trustabl https://github.com/trustabl/scoop-bucket
scoop install trustabl

Docker

docker run --rm -v "$PWD:/repo" ghcr.io/trustabl/trustabl:latest scan /repo

Direct download

Grab a prebuilt archive for your platform from the [Releases page](https://github.com/trustabl/agent-reliability-analyzer/releases). Each release includes a `checksums.txt` and a build-provenance attestation; verify with:

gh attestation verify <archive> --repo trustabl/agent-reliability-analyzer

> Archives released **before** the repository was renamed verify only under the > old slug (`--repo trustabl/trustabl`): the Sigstore certificate binds the > repository path as it was at signing time, so the recorded identity cannot > follow a rename.

Claude Code plugin

The plugin installs two skills (`trustabl-scan` and `trustabl-enrich`), a `SessionStart` hook, and a bundled MCP server — and auto-downloads the pinned CLI binary on first session (no `sudo`, nothing outside Claude Code's plugin data directory touched).

Install from within a Claude Code session using the `/plugin` slash command:

/plugin install trustabl@claude-plugins-official

Then reload to activate:

/reload-plugins

After installation, the `mcp__trustabl__scan` tool is available for direct calls, the `trustabl-scan` and `trustabl-enrich` skills appear in the skill picker, and the hook confirms the binary is ready at session start. Uninstall with `/plugin uninstall trustabl`.

Claude Code agent

A standalone subagent (`trustabl`) runs the full scan → enrich → review → apply pipeline interactively. It does not require the plugin — just the `trustabl` binary already on your PATH. Copy the agent file into your project or home config:

# Project-level (scoped to this repo)
curl -fsSL https://raw.githubusercontent.com/trustabl/agent-reliability-analyzer/main/agents/trustabl.md \
  -o .claude/agents/trustabl.md

# Global (available in every project)
curl -fsSL https://raw.githubusercontent.com/trustabl/agent-reliability-analyzer/main/agents/trustabl.md \
  -o ~/.claude/agents/trustabl.md

Once installed, invoke it via `@trustabl` in Claude Code or select it from the subagent picker. It will ask which directory to scan, run `trustabl scan` and `trustabl enrich`, then walk you through each proposed fix as a diff.

The enrich step requires an Anthropic API key. Set it before invoking the agent:

export ANTHROPIC_API_KEY=sk-ant-api03-...   # or: trustabl llm key set

From source

Requires `CGO_ENABLED=1` because the AST parsers use tree-sitter (Python + TypeScript + TSX bindings), which is a C library:

# macOS / Linux
CGO_ENABLED=1 go build -o trustabl ./cmd/trustabl

# Cross-compile: pick a C toolchain for the target. zig is the easiest.
CGO_ENABLED=1 CC="zig cc -target x86_64-linux-gnu" \
  GOOS=linux GOARCH=amd64 go build -o trustabl-linux ./cmd/trustabl

This is the cost of using tree-sitter for accurate AST parsing. If a single-binary, no-CGO distribution becomes a hard requirement later, the swap target is `github.com/go-python/gpython` for Python (with lower fidelity on modern Python); TypeScript would need a separate replacement.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 29 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "trustabl",
  3. "owner": {
  4. "name": "Trustabl",
  5. "url": "https://github.com/trustabl"
  6. },
  7. "metadata": {
  8. "description": "Trustabl's Claude Code plugin marketplace."
  9. },
  10. "plugins": [
  11. {
  12. "name": "trustabl",
  13. "source": "./",
  14. "description": "Self-audit AI agent, tool, and MCP-server code for security and reliability misconfigurations with Trustabl, the agent reliability scanner for the OpenAI Agents SDK, Claude Agent SDK, Google ADK, and MCP.",
  15. "homepage": "https://github.com/trustabl/agent-reliability-analyzer",
  16. "license": "Apache-2.0",
  17. "category": "security",
  18. "keywords": [
  19. "security",
  20. "static-analysis",
  21. "agents",
  22. "mcp",
  23. "openai-agents-sdk",
  24. "claude-agent-sdk",
  25. "google-adk"
  26. ]
  27. }
  28. ]
  29. }
README.md · README · 1,557 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <p align="center">
  2. <img src="assets/github_banner.jpg" alt="Agent Reliability Analyzer by Trustabl — open-source AI agent reliability scanner" width="100%">
  3. </p>
  4. <p align="center">
  5. <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="License: Apache-2.0"></a>
  6. <a href="https://github.com/trustabl/agent-reliability-analyzer/releases"><img src="https://img.shields.io/github/v/release/trustabl/agent-reliability-analyzer" alt="Latest release"></a>
  7. <a href="https://github.com/trustabl/agent-reliability-analyzer/releases"><img src="https://img.shields.io/github/downloads/trustabl/agent-reliability-analyzer/total?label=downloads&color=2ea043" alt="Total downloads"></a>
  8. <a href="https://github.com/trustabl/agent-reliability-analyzer/actions/workflows/test.yml"><img src="https://img.shields.io/github/actions/workflow/status/trustabl/agent-reliability-analyzer/test.yml?branch=main&amp;label=tests" alt="Tests"></a>
  9. <a href="go.mod"><img src="https://img.shields.io/github/go-mod/go-version/trustabl/agent-reliability-analyzer" alt="Go version"></a>
  10. <br>
  11. <a href="https://github.com/trustabl/agent-reliability-rules"><img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftrustabl%2Fagent-reliability-rules%2Fmain%2Fbadges%2Frules.json" alt="Detection rule count"></a>
  12. <a href="COVERAGE.md"><img src="https://img.shields.io/badge/SDKs-9-blue" alt="9 SDKs supported"></a>
  13. <a href="COVERAGE.md"><img src="https://img.shields.io/badge/languages-7-blue" alt="7 languages supported"></a>
  14. <a href="COVERAGE.md"><img src="https://img.shields.io/badge/scopes-5-blue" alt="5 detection scopes"></a>
  15. <a href="COVERAGE.md"><img src="https://img.shields.io/badge/surfaces-tools%20%C2%B7%20agents%20%C2%B7%20subagents%20%C2%B7%20skills%20%C2%B7%20plugins-blue" alt="Analyzed surfaces"></a>
  16. <a href="README.md#output-modes"><img src="https://img.shields.io/badge/output-human%20%7C%20JSON%20%7C%20SARIF-blue" alt="Output formats"></a>
  17. </p>
  18. # Agent Reliability Analyzer — find and fix AI agent reliability gaps
  19. **Agent Reliability Analyzer** is Trustabl's open-source, deterministic scanner that finds and fixes agent reliability failures in AI agent code before it reaches production. Find what will make your AI agent fail — then fix it with one command.
  20. Deterministic static analysis for agent code, across nine SDKs and seven languages.
  21. It runs entirely on your machine: no cloud scanner, no account, no code upload, no LLM.
  22. Agent reliability coverage as of Oct 2026: 340+ rules · 9 SDKs · 7 languages · 5 scopes
  23. If Agent Reliability Analyzer saves you a production incident, please ⭐ the repo — it helps other teams find it.
  24. ---
  25. <a id="trustabl-partners"></a>
  26. <h2 align="center">Trustabl partners — install from the catalogue you already use</h2>
  27. <p align="center">
  28. <sub>One agent reliability engine, ten front doors.</sub>
  29. </p>
  30. <table align="center">
  31. <tr>
  32. <td align="center" width="200">
  33. <a href="https://registry.modelcontextprotocol.io/?q=trustabl"><b>MCP Registry</b></a> &middot;
  34. <a href="https://mcprush.com/trustabl"><b>mcprush</b></a><br>
  35. <sub>Scan from any<br>MCP-aware client</sub>
  36. </td>
  37. <td align="center" width="200">
  38. <a href="docs/claude-plugin.md"><b>Claude Directory</b></a><br>
  39. <sub>Skills, agent and<br>scanner inside Claude</sub>
  40. </td>
  41. <td align="center" width="200">
  42. <a href="https://www.npmjs.com/package/@trustabl/ai-sdk"><b>npm</b></a> &middot;
  43. <a href="https://pypi.org/project/pydantic-ai-trustabl/"><b>PyPI</b></a><br>
  44. <sub>A tool your AI SDK or<br>Pydantic AI agent calls</sub>
  45. </td>
  46. </tr>
  47. <tr>
  48. <td align="center">
  49. <a href="https://marketplace.visualstudio.com/items?itemName=trustabl.trustabl"><b>VS Code</b></a><br>
  50. <sub>Findings as you<br>write, in the editor</sub>
  51. </td>
  52. <td align="center">
  53. <a href="https://cursor.directory/plugins/trustabl"><b>Cursor</b></a><br>
  54. <sub>Same scan, wired<br>in as an MCP server</sub>
  55. </td>
  56. <td align="center">
  57. <a href="https://github.com/trustabl/agent-reliability-analyzer/releases"><b>CLI</b></a><br>
  58. <sub>Homebrew, Scoop,<br>Docker or a binary</sub>
  59. </td>
  60. </tr>
  61. <tr>
  62. <td align="center">
  63. <a href="https://github.com/marketplace/actions/trustabl-fix-agent-reliability-issues"><b>GitHub Actions</b></a><br>
  64. <sub>Gate the build<br>on a severity threshold</sub>
  65. </td>
  66. <td align="center">
  67. <a href="https://gitlab.com/explore/catalog/trustabl-ai/components"><b>GitLab CI/CD</b></a><br>
  68. <sub>Published component<br>in the catalogue</sub>
  69. </td>
  70. <td align="center">
  71. <a href="https://bitbucket.org/hoolisoftware/trustabl-pipe"><b>Bitbucket</b></a><br>
  72. <sub>Official pipe, no<br>install step needed</sub>
  73. </td>
  74. </tr>
  75. </table>
  76. <p align="center">
  77. <sub>
  78. Every one of these runs the same deterministic scan on your own machine.<br>
  79. See <a href="docs/integrations.md">ecosystem integrations</a> for which agent SDKs are covered.
  80. </sub>
  81. </p>
  82. ---
  83. <p align="center">
  84. <a href="https://www.star-history.com/#trustabl/agent-reliability-analyzer&Date">
  85. <img src="https://api.star-history.com/svg?repos=trustabl/agent-reliability-analyzer&type=Date" alt="GitHub stars over time for trustabl/agent-reliability-analyzer" width="800">
  86. </a>
  87. </p>
  88. Trustabl scans an agent repository for the gaps that break agents in production:
  89. tool descriptions too vague for a model to know when to use them, missing retry
  90. and timeout handling, untyped parameters, absent guardrails, and tool grants that
  91. exceed what the agent claims to do. Then it applies the fix directly to your
  92. source.
  93. Every other agent scanner hands you a report. Trustabl hands you a patch.
  94. Reliability is engineered before deployment, not observed after it.
  95. **Scanning runs entirely on your machine.** No cloud scanner, no account, no
  96. code upload, no LLM — deterministic static analysis.
  97. ```bash
  98. docker run --rm -v "$PWD:/repo" ghcr.io/trustabl/trustabl:latest scan /repo # try it, nothing installed
  99. brew install trustabl/tap/trustabl # macOS / Linux
  100. scoop bucket add trustabl https://github.com/trustabl/scoop-bucket # Windows
  101. scoop install trustabl
  102. trustabl scan . # find issues (fully local)
  103. trustabl scan . --format json > scan.json
  104. trustabl enrich --input scan.json --repo . --diff --apply # preview, then fix
  105. ```
  106. 9 SDKs · 7 languages · human, JSON, or SARIF 2.1.0 output ·
  107. CI-friendly exit codes · also runs as a local stdio MCP server (`trustabl mcp`).
  108. Deepest coverage for **Claude Agent SDK**, **OpenAI Agents SDK**, **Google ADK**,
  109. and **MCP servers**. Also scans LangChain / LangGraph, CrewAI, AutoGen / AG2,
  110. Pydantic AI, and the Vercel AI SDK — see [COVERAGE.md](COVERAGE.md) for the full
  111. matrix.
  112. Scanning needs no key and no network. Applying fixes uses your own Anthropic,
  113. OpenAI, or Google key.
  114. <a id="what-a-scan-looks-like"></a>
  115. ## What an agent reliability scan looks like
  116. Real output from an agent reliability scan of a repo that ships a Claude Agent SDK
  117. client, an MCP server, a subagent, and two skills:
  118. ```console
  119. $ trustabl scan .
  120. Scan summary
  121. Languages: typescript, javascript
  122. SDKs: claude_agent_sdk
  123. Tool definitions: 2 (custom tools with function bodies)
  124. Agent tool grants: 14 (tool names the agent may call)
  125. MCP servers: 1 (createSdkMcpServer)
  126. Subagents: 1 (inbox-searcher)
  127. Skills: 2 (action-creator, listener-creator)
  128. Findings: 14
  129. Overall score: 66%
  130. Surface readiness
  131. skill:action-creator 45% (5 findings)
  132. skill:listener-creator 54% (4 findings)
  133. agent:AIClient.queryStream 64% (2 findings)
  134. subagent:inbox-searcher 79% (1 finding)
  135. tool:search_inbox 98% (1 finding)
  136. tool:read_emails 100% (0 findings)
  137. Findings
  138. inbox-searcher
  139. [CSDK-201] HIGH Subagent is granted Bash (agent/.claude/agents/inbox-searcher.md:1-5)
  140. A subagent with shell access can run arbitrary commands if it is
  141. compromised or misdirected.
  142. fix: Remove `Bash` from the subagent's `tools:` list unless shell access
  143. is essential. Prefer specific tools (Read, Grep, Glob) for read-only roles.
  144. ```
  145. Three things worth noticing:
  146. - **It scores each surface separately.** A repo-wide number hides the problem;
  147. `skill:action-creator` at 45% tells you where to look first.
  148. - **Every finding carries a fix**, not just a complaint — and
  149. `trustabl enrich --apply` writes those fixes to source.
  150. - **The exit code is the CI gate.** `0` when nothing reaches medium severity,
  151. `1` when something does, so a pipeline can stop a bad agent from shipping.
  152. ## AI agents pass their demo and fail in production
  153. Most agent reliability failures are not exotic. They are the same handful of gaps, over and over:
  154. - An agent holding shell tools with **no input guardrails**
  155. - A tool making an HTTP call with **no timeout**, hanging the whole run
  156. - **Untyped tool parameters**, so the model guesses and guesses wrong
  157. - **No observability wired at all**, so the first three have no trace to read
  158. - A tool description so vague the model calls the wrong tool — 56% of MCP tool
  159. descriptions fail to state their purpose clearly, and 97.1% carry at least one
  160. description defect (Hasan et al., *MCP Tool Descriptions Are Smelly!*,
  161. [arXiv 2602.14878](https://arxiv.org/abs/2602.14878) — 856 tools across 103
  162. servers)
  163. - A **subagent granted `Bash`** despite a read-only description
  164. - A **skill that auto-approves unrestricted shell access**
  165. - **No retry handling**, so one transient 500 fails the task
  166. - An agent loop with **no iteration bound**, burning tokens until it is killed
  167. - A **user-controlled URL** flowing into a fetch — SSRF, from your agent
  168. Every one of these is visible in the source before the agent ever runs. Trustabl
  169. finds them in seconds, offline, with no LLM — and fixes them.
  170. The rest of this document explains *what Trustabl reasons about* and *how
  171. the scan works*, then covers building and running it. For the full
  172. implementation reference see [ARCHITECTURE.md](ARCHITECTURE.md); for the
  173. at-a-glance SDK coverage matrix see [COVERAGE.md](COVERAGE.md).
  174. <a id="what-it-analyzes--the-five-scope-model"></a>
  175. ## What it analyzes — agent reliability across five scopes
  176. Agent Reliability Analyzer does not treat a repository as one undifferentiated
  177. blob. Every agent reliability rule is classified into exactly one of five scopes,
  178. and each scope receives a different typed input:
  179. - **`tool`** — fires once per tool definition. Input: a `ToolDef` (a
  180. `@function_tool` / `@tool` / `@claude_tool` function, a Claude TS
  181. `tool(name, description, schema, handler)` factory call, a
  182. `FunctionTool(fn)` ADK wrapper, an `@server.tool` MCP registration, or a
  183. bare shell-invoking function) plus its parsed file. Catches a missing
  184. docstring, an HTTP call with no timeout, untyped parameters, or an
  185. unnormalized path flowing into `open()`. (Hosted tools like
  186. `WebSearchTool()` are agent-scope edge data, captured as `HostedToolDef`,
  187. not `ToolDef`.)
  188. - **`agent`** — fires once per agent declaration. Input: an `AgentDef` —
  189. a Python `Agent(...)` / `SandboxAgent(...)` / `AgentDefinition(...)`
  190. call, a Claude TS typed-const `AgentDefinition`, a Claude TS sub-agent
  191. inline in `options.agents`, or the Claude TS `query(...)` main-thread
  192. agent (`QueryMainAgent`) — with every constructor kwarg captured and its
  193. edges to tools, handoffs, and guardrails resolved. Catches an agent with
  194. shell tools and no `input_guardrails`,
  195. `tool_use_behavior="stop_on_first_tool"` paired with filesystem-touching
  196. tools, or a main-thread agent with unrestricted `allowedTools`.
  197. - **`subagent`** — fires once per Claude Code subagent markdown
  198. declaration. Discovery is **hybrid**: canonical `.claude/agents/*.md`
  199. (any path depth, monorepo-safe) PLUS a frontmatter-shape fallback over
  200. all markdown files (gated on `name` + `tools`/`model`) that catches
  201. flat-collection repos which ship subagents under `categories/*.md`,
  202. `plugins/<x>/agents/*.md`, or similar layouts. Input: a `SubagentDef`
  203. parsed from frontmatter — `name`, `description`, `tools[]` (verbatim) +
  204. `ToolGrants[]` (parsed permission grammar), `disallowedTools`, `model`,
  205. `permissionMode` (incl. `bypassPermissions`), `mcpServers`, `skills`,
  206. `isolation`, `hasHooks`. Catches a subagent granted the built-in `Bash`
  207. tool despite a read-only description (CSDK-110). Subagent presence alone
  208. contributes `claude_agent_sdk` to `SDKsDetected`, so the Claude pack
  209. loads and CSDK-110 fires on pure-markdown subagent collections.
  210. - **`skill`** — fires once per Claude Code skill (`SKILL.md`, any path
  211. depth). Input: a `SkillDef` parsed from frontmatter — `name`,
  212. `description`, `allowed-tools` → `ToolGrants[]`,
  213. `disable-model-invocation` — plus body facts (dynamic-context exec
  214. commands, external URLs, prompt-injection markers) and a bundled-file
  215. inventory. Catches a skill that auto-approves unrestricted `Bash`
  216. (CSKILL-001), runs a dynamic-context command that performs network egress
  217. or reads secrets before the model sees it (CSKILL-003), or is
  218. model-invocable while granting side-effecting tools (CSKILL-050). Skills
  219. are markdown, so skill rules carry no `language:`; the `claude_skill`
  220. pack loads whenever a `SKILL.md` is present.
  221. - **`repo`** — fires once per scan against the whole inventory. Catches
  222. project-wide gaps such as the OpenAI Agents SDK being present with no
  223. custom trace processor configured.
  224. ### The agent is the unit of analysis, not the repo
  225. A repo can declare zero, one, or many agents, across one or more SDKs.
  226. **Two agents in the same repo can be in completely different security
  227. postures** — one wired with input/output guardrails, the other not.
  228. Agent-scoped findings therefore attribute to a *specific* agent at its
  229. constructor call site; flattening them to a single repo-level verdict
  230. would lose that attribution and be wrong. Discovery builds a small
  231. per-repo graph (tools, agents, subagents, and the edges between them) so
  232. agent-scope and subagent-scope rules can query it.
  233. ### Rules are scoped to one SDK *and* one language
  234. A Claude-SDK rule and an OpenAI-Agents-SDK rule that detect the same
  235. conceptual problem (a missing timeout, say) are two separate rules with
  236. SDK-specific explanation and fix text — there is no cross-SDK casting.
  237. When a repo declares agents from multiple SDKs side by side, each agent is
  238. checked only against the rules for the SDK that declared it. The same
  239. holds across languages: a `language: python` rule will not fire on a
  240. TypeScript agent.
  241. ## How it reasons — the scanning pipeline
  242. Agent Reliability Analyzer (the `trustabl` CLI) scans in four steps. Each step's output is the typed input to the
  243. next, with no shared state between runs — and the inventory the early
  244. steps build is what makes policy selection *data-driven* rather than
  245. statically configured.
  246. The binary ships with **no embedded rules**. Before the pipeline runs,
  247. Trustabl resolves its detection rules from a separate git repository
  248. ([`agent-reliability-rules`](https://github.com/trustabl/agent-reliability-rules)) —
  249. fetching the latest, caching the clone locally, and falling back to the
  250. cache when the network is unreachable. This decouples rule updates from
  251. binary releases: rules can be added or changed without rebuilding the
  252. scanner. The resolved rules commit is recorded in the result and folded
  253. into the `ScanID`, so a scan is honest about *which* rules produced it.
  254. If no rules can be fetched and none are cached, the scan exits `2` and
  255. tells you to run `trustabl rules pull` — Trustabl never runs rule-less.
  256. ```mermaid
  257. flowchart LR
  258. target[("Agent repo<br/>(local path or GitHub URL)")]
  259. recon["Recon<br/>files · SDK deps"]
  260. inv["Inventory<br/>Python + TS AST:<br/>tools · agents ·<br/>subagents · MCP servers"]
  261. pol["Policy selection<br/>load rules per<br/>detected SDK ·<br/>META findings"]
  262. ana["Analysis<br/>tool · agent · subagent ·<br/>repo detectors"]
  263. score["Scoring<br/>per-surface score ·<br/>overall readiness"]
  264. out[("ScanResult<br/>findings · scores<br/>(human / JSON / SARIF)")]
  265. target --> recon --> inv --> pol --> ana --> score --> out
  266. ```
  267. 1. **Recon** — walk the repo and answer "what's in here" cheaply, without
  268. parsing any source language: languages present (by extension), SDK
  269. dependencies declared in manifests (`pyproject.toml` / `requirements.txt`
  270. / `Pipfile` / `poetry.lock` / `package.json` for the
  271. `claude-agent-sdk` / `@anthropic-ai/claude-agent-sdk` / `openai-agents` /
  272. `@openai/agents` / `google-adk` / `@google/adk` needles), the file inventory, and
  273. discovered agent components (MCP configs, hook scripts, `CLAUDE.md` and
  274. `AGENTS.md` guidance docs,
  275. `.claude/agents/*.md` subagents at any depth, `SKILL.md` skills,
  276. slash commands at both `.claude/commands/*.md` and
  277. `<plugin-root>/commands/*.md`, `.claude-plugin/{plugin,marketplace}.json`
  278. manifests, sandbox policies, and the repo-root `.github/workflows/*.yml`
  279. GitHub Actions workflows). No
  280. tree-sitter parses happen here — this step decides whether the
  281. expensive AST work is even worth attempting.
  282. 2. **Inventory** — for each language Recon cleared, do the AST work and
  283. extract a typed inventory: `ToolDef`s with their config and body facts,
  284. `AgentDef`s with all kwargs captured, `SubagentDef`s / `SkillDef`s /
  285. `SlashCommandDef`s / `PluginManifest`s parsed from markdown and JSON
  286. frontmatter, `MCPServerDef`s, guardrails, sessions, and the resolved
  287. edges between agents and the tools/guardrails they reference. Detectors
  288. read fields off these structs — they never re-parse raw source.
  289. 3. **Policy selection** — load **only** the rule packs for SDKs actually
  290. *observed in code*. An SDK seen in code with no shipped pack emits a
  291. `META-001` info finding ("Trustabl does not currently audit this SDK")
  292. — silence on an unknown SDK is wrong. A dep declared but never used in
  293. code emits a different info finding flagging the drift.
  294. 4. **Analysis** — run the selected scope-aware detectors against the
  295. inventory. Findings carry the scope they fired at and attribute to the
  296. right location: tool file/line, agent call site, subagent markdown
  297. file, or the manifest.
  298. Three properties fall out of this staging, by design:
  299. - **Performance.** A repo with no Python skips Python AST work; a repo
  300. with only Claude TS code skips Python AST work AND OpenAI policy
  301. loading.
  302. - **Honest coverage.** An "unaudited SDK" info finding is louder than a
  303. zero-findings clean bill of health on an SDK Trustabl doesn't know. A
  304. `META-004` finding further distinguishes "audited and clean" from
  305. "could not audit — discovery extracted nothing a rule targets."
  306. - **Determinism is a contract.** Same inputs → same `ScanID`, and the
  307. report is byte-stable across runs (findings sorted by
  308. `(RuleID, FilePath, Line)`, inventory slices sorted deterministically).
  309. CI consumers can diff scans without spurious churn.
  310. See [ARCHITECTURE.md § 2](ARCHITECTURE.md#2-pipeline) for the full
  311. diagram with typed inputs at each step.
  312. ### What's wired today
  313. Tool/agent AST discovery is wired for:
  314. - **Python** — Claude Agent SDK (decorators), OpenAI Agents SDK, Google
  315. ADK, LangChain / LangGraph, CrewAI, AutoGen / AG2, and Pydantic AI.
  316. Discovery extracts tool definitions, agent constructors, hosted
  317. tools, MCP servers, guardrails, sessions. The bare `Agent(...)`
  318. constructor shared by OpenAI / ADK / CrewAI / Pydantic AI is
  319. import-gated per SDK so the classes never cross-match, and the
  320. shared `@tool` decorator is routed to the owning SDK by its import
  321. binding.
  322. - **TypeScript** — Claude Agent SDK (the `tool()` factory, the
  323. `query()` main-thread `QueryMainAgent`, inline-in-`query()` sub-agents,
  324. typed-const `AgentDefinition`s, `createSdkMcpServer` and the four
  325. `options.mcpServers` config literals), OpenAI Agents SDK (the
  326. `tool({...})` factory, `new Agent({...})` and `Agent.create({...})`,
  327. 9 hosted-tool factories, MCP server classes across 3 transports plus
  328. the `MCPServers` wrapper, 4 `defineX` guardrail factories, and the
  329. `MemorySession` / `OpenAIConversationsSession` /
  330. `OpenAIResponsesCompactionSession` session classes — gated on imports
  331. from `@openai/agents`, `@openai/agents-core`, or
  332. `@openai/agents-openai`), and Google ADK (the
  333. `new FunctionTool({...})` constructor, 5 agent constructors —
  334. `new LlmAgent({...})` / `SequentialAgent` / `ParallelAgent` /
  335. `LoopAgent` / `RoutedAgent` — 13 hosted-tool classes, and `subAgents`
  336. edges — gated on imports from `@google/adk`), LangChain / LangGraph
  337. (the `tool(fn, {...})` factory, `DynamicStructuredTool` / `DynamicTool`,
  338. and `createReactAgent` / `createAgent` / `new AgentExecutor` — gated on
  339. the `@langchain/*` / `langchain` / `langgraph` ecosystem), and the
  340. Vercel AI SDK (the `tool({...})` / `dynamicTool({...})` single-object
  341. factory, the call-based `generateText` / `streamText` / `generateObject`
  342. / `streamObject` agents and the class `ToolLoopAgent` /
  343. `Experimental_Agent`, with `tools` walked as an object/record, plus the
  344. `<provider>.tools.*()` hosted tools — gated on the bare `ai` import).
  345. Handles `.ts` / `.tsx` / `.mts` / `.cts` plus JavaScript
  346. `.js` / `.jsx` / `.mjs` / `.cjs` with the `tree-sitter-typescript` and
  347. `tree-sitter-tsx` grammars (JavaScript routes to the tsx grammar — a JS
  348. superset — and is audited by the same `language: typescript` rule packs).
  349. TypeScript rule packs ship for the Claude Agent SDK
  350. (CSDK-010/011/012/013/014/016 tool rules; CSDK-120/130/131 agent rules),
  351. OpenAI Agents SDK (OAI-016/017/019/022/024 tool rules; OAI-105/116 agent
  352. rules), Google ADK (ADK-013/015/016 tool rules; ADK-109 agent rule), MCP
  353. (MCP-011/012/013/014 tool rules), LangChain (LC-010/011/012/013/014 tool
  354. rules; LC-111 agent rule), and the Vercel AI SDK (VAI-001..008 tool/agent
  355. rules; VAI-012 repo rule). A TS repo for any of these no longer produces a
  356. blanket `META-004`; see `COVERAGE.md` for the full matrix.
  357. JavaScript (`.js` / `.jsx` / `.mjs` / `.cjs`) is AST-parsed through the shared
  358. TypeScript-family pipeline: its tools and agents are discovered, tagged
  359. `javascript`, and audited by the `language: typescript` rule packs (both ES
  360. `import` and CommonJS `require()` bindings are recognized). Go has
  361. tree-sitter-go discovery for MCP tools (mark3labs/mcp-go and the official
  362. modelcontextprotocol/go-sdk), audited by the `language: go` rules in the MCP
  363. pack. C# has tree-sitter-c-sharp discovery for the official ModelContextProtocol
  364. SDK's `[McpServerTool]` methods, audited by the `language: csharp` rules. PHP has
  365. tree-sitter-php discovery for `#[McpTool]`-attributed methods (official mcp/sdk
  366. and community php-mcp/server), audited by the `language: php` rules. Rust has
  367. tree-sitter-rust discovery for the official rmcp crate's `#[tool]`-attributed
  368. methods (descriptions read from the `description = "..."` arg or the `///` doc
  369. comment), audited by the `language: rust` rules; other Go, .NET, PHP, and Rust
  370. SDKs are recognized as files by Recon but not yet AST-parsed.
  371. The rule schema's `language:` field gates per-language rule sets.
  372. ### Scope boundaries
  373. - **LLM enrichment is a separate post-scan step (`trustabl enrich`).** Rule-based
  374. detection (`trustabl scan`) makes no network call — there is no LLM involved in
  375. the scan itself. `trustabl enrich` reads the scan output and calls the configured
  376. LLM provider (Anthropic, OpenAI, or Google Gemini) with BYOK (key stored via
  377. `trustabl llm key set` at `~/.config/trustabl/keys.json`, mode 0600). Each call
  378. carries a request timeout, and `--apply` rewrites a file only when its current
  379. contents still match what the model reviewed (writing a `.trustabl.bak` backup
  380. first) — a stale scan is skipped, never mis-applied. With `--langsmith`
  381. (opt-in, requires `LANGSMITH_API_KEY`), tool-scope findings are additionally
  382. grounded in runtime trace evidence sampled from a LangSmith project: error
  383. rate, latency, and recent error messages for each flagged tool, carried on
  384. the output as `trace_evidence` and fed to the LLM alongside the static code
  385. snippet; tools with no trace history degrade per finding to plain static
  386. enrichment.
  387. - **Confidence scores are heuristic**, not LLM-judged, and not yet
  388. calibrated against a labelled real-agent corpus — treat findings as
  389. signal to investigate.
  390. - **The CLI is the surface.** No web app, API server, or hosted service:
  391. pipe `--format json` or `--format sarif` into your own automation. On GitHub
  392. Actions, [`trustabl/trustabl-action`](https://github.com/trustabl/trustabl-action)
  393. wraps the scan and uploads SARIF to the Security tab for you; for any other
  394. CI, `--format sarif --output <file>` produces a SARIF 2.1.0 report that feeds
  395. `github/codeql-action/upload-sarif` or any SARIF-aware step.
  396. - **Trustabl is open core.** Everything documented here is Apache-2.0 and
  397. complete for its job: the scan needs no key and no network, and autofix uses
  398. your own LLM key. Policy generation, compliance mapping and signed governance
  399. snapshots are Trustabl Console, a separate commercial product. See
  400. [docs/open-source-vs-console.md](docs/open-source-vs-console.md) for the full
  401. capability split.
  402. ## What it produces
  403. `trustabl scan` is detect-and-report: it does **not** write or modify any
  404. files in the scanned repo. Fixes are applied only when you opt in with
  405. `trustabl enrich --apply`. Each run produces a `ScanResult` containing:
  406. - **Findings** — one per rule hit, each with `severity`, `confidence`,
  407. an `explanation`, a `suggested_fix`, and the location it fired at
  408. (tool file/line, agent call site, subagent file, or the manifest). A finding
  409. in a test-path file carries `"origin": "test"` and is excluded from
  410. scoring and the exit code by default — see "Test-path findings" below.
  411. - **Per-surface readiness scores** (one per discovered tool, agent, subagent,
  412. or the repo as a whole) and an **overall score** (a breadth-aware,
  413. badness-weighted mean — weak surfaces pull it down harder, but a single
  414. poor surface does not zero it; the score is a triage signal, not the CI gate).
  415. - **The discovered inventory** — tools, agents, hosted tools, MCP
  416. servers, subagents, skills, slash commands, plugin manifests, and
  417. Claude settings — surfaced at the top level for CI consumers.
  418. - **Observability signals** (`observability`) — the instrumentation found in
  419. code, one record per fact: the vendor (OpenTelemetry, Langfuse, Logfire,
  420. Phoenix/OpenInference, OpenLLMetry, Braintrust, Weave, AgentOps, MLflow,
  421. Datadog LLM Observability, LangSmith) and whether it is merely **imported** or
  422. actually **initialized**, with the file and line. That distinction is the
  423. point: a repo that imports a tracing SDK and never starts it looks
  424. instrumented in its manifest and produces no traces at all. Detected
  425. statically in Python and TypeScript/JavaScript only — Trustabl does not ingest
  426. runtime traces.
  427. - **Process labels** (`otel_labels`) — the `agent-reliability-otel-labels`
  428. calls found in code (import, `Run` construction, run start, tool-span label,
  429. run end) and any `trustabl.*` label keys written by hand, with the file and
  430. line and the facts the OBS-006..012 rules read: whether a `Run` is shared,
  431. whether a label runs when the tool fails, whether `attempt` is a fixed
  432. number, and the literal `side_effect` / `exit_reason` values. Python and
  433. TypeScript/JavaScript only. Not folded into the scan ID.
  434. - **Server entrypoints** (`entrypoints`, omitted when none) — Python functions a
  435. server framework invokes for an external request or job: FastAPI and Flask
  436. route handlers (`kind: http`, with the literal `route` when present) and Celery
  437. tasks / Dramatiq actors (`kind: worker`). A fact only: no rule consumes it yet.
  438. Entrypoints under `examples/`, `demo/`, `docs/` and similar directories are not
  439. recorded.
  440. ### The summary's tool surface, broken out
  441. The human format honestly separates the three things people commonly
  442. conflate:
  443. ```text
  444. Tool definitions: 2 (custom tools with function bodies — scored below)
  445. Agent tool grants: 14 (tool names the agent may call — audited by agent-scope rules)
  446. Hosted tools: 1 (...)
  447. ```
  448. Only the "Tool definitions" category flows through tool-scope rules
  449. (they have function bodies a rule can read). Agent grants and hosted
  450. instances are inputs to *agent-scope* rules, not unanalyzed — they just
  451. don't appear in the per-surface readiness table.
  452. ### Output modes
  453. `--format human` (default) renders a human summary to stdout and live
  454. progress to stderr — an animated spinner and progress bar on an
  455. interactive terminal, or plain `[phase] summary` lines when piped
  456. (CI-friendly).
  457. `--format json` marshals the full `ScanResult` for piping into your
  458. own automation.
  459. `--format sarif` emits a SARIF 2.1.0 document, suitable for
  460. `github/codeql-action/upload-sarif` and other SARIF-aware tools. The suggested
  461. fix is carried at the rule level (`help.text`); Trustabl emits no per-result
  462. `fixes[]`, so the document passes GitHub Code Scanning's schema validator (which
  463. rejects a `fix` that lacks `artifactChanges`).
  464. `--json-out <file>` and `--sarif-out <file>` write the JSON / SARIF document to a
  465. file independent of `--format` — one scan can print the human summary to stdout
  466. while persisting both machine artifacts. The file bytes are identical to the
  467. matching `--format` stdout output.
  468. `--bom-out <file>` additionally writes a byte-stable CycloneDX 1.5 BOM of the
  469. dependencies the repo declares across every supported language — `requirements.txt`
  470. / `pyproject.toml` / `Pipfile` (pip), `package.json` (npm), `go.mod` (Go),
  471. `composer.json` (Composer), `*.csproj` (NuGet), `Cargo.toml` (Cargo). It is pure
  472. inventory of DECLARED direct deps and makes no network call. Where the manifest
  473. carries a `license` field (`package.json`, `Cargo.toml`, `composer.json`,
  474. `pyproject.toml`), the SPDX identifier is included in each component's
  475. `licenses[]` array in the BOM.
  476. `--vuln-scan` turns that BOM into a vulnerability verdict: it matches the repo's
  477. concretely-pinned dependencies against a pinned [OSV](https://osv.dev) snapshot
  478. and reports each affected package as a finding carrying the advisory ID
  479. (CVE / GHSA / PYSEC / …), a CVSS-derived severity, and the first fixed version —
  480. so a vulnerable dependency fails the scan through the normal severity gate and
  481. exit codes and lands in the JSON / SARIF output alongside the rule findings, on
  482. `ScanResult.vulnerabilities`. Unlike the rest of a scan it is **opt-in and
  483. online**: the OSV snapshot is fetched from osv.dev on first use, cached under
  484. your user cache directory, and then **cache-first** — a later `--vuln-scan`
  485. reuses the cached database (no re-download) until it is older than 24h, so
  486. repeated scans are fast and offline-capable. `trustabl vulndb pull` refreshes the
  487. cache on demand; `--no-rules-update` pins to the cache at any age (fully offline).
  488. Only concretely-pinned versions are matched
  489. — a declared range (`^1.0`, `>=2`) can't be resolved to one version without a
  490. lockfile, so it is left unmatched rather than guessed. The snapshot version is
  491. folded into the `ScanID` only when `--vuln-scan` is on, so the result is honest
  492. about which vulnerability data produced it while a default scan stays
  493. byte-identical to before.
  494. Combining `--vuln-scan` with `--bom-out` upgrades the CycloneDX document from a
  495. plain inventory into a BOM **plus VEX**: the matched advisories are emitted as a
  496. CycloneDX 1.5 `vulnerabilities[]` array — each with the advisory ID, an OSV
  497. `source`, a severity `rating`, an upgrade `recommendation`, and an `affects[]`
  498. reference linking it to the affected component's `bom-ref` — so a single
  499. `trustabl scan ./repo --vuln-scan --bom-out bom.json` produces a standards-based
  500. artifact that any CycloneDX-aware tool can ingest. Without `--vuln-scan` the
  501. `vulnerabilities[]` array is omitted and the BOM stays pure inventory.
  502. `--format json` and `--format sarif` are progress-silent and byte-stable
  503. across identical-input runs (pure functions of the `ScanResult`). The human
  504. format is not byte-stable by design: its ANSI color is auto-detected from the
  505. terminal (TTY vs pipe, `NO_COLOR`), so the same scan can render with or without
  506. color. Use `--no-color`, or diff the JSON/SARIF output, when byte-stability
  507. matters.
  508. ### Test-path findings
  509. A finding whose file lives under a test path (`tests/`, `__tests__/`,
  510. `testdata/`, `test_*.py`, `*_test.go`, `*.spec.ts`, …) is still reported in
  511. every format — nothing is silently hidden — but by default it is **excluded
  512. from the readiness score and does not fail the build**: a `WebSearchTool`
  513. declared in `tests/test_adapter.py` is not the same finding as one shipped in
  514. a production agent, and scoring them identically made the report
  515. non-actionable. Each such finding carries `"origin": "test"` in JSON, an
  516. `origin` property plus a `result.suppressions` entry in SARIF (excluded from
  517. GitHub code scanning's default alert view, still present in the raw log), and
  518. renders under a trailing "Test-path findings (not scored)" section in the
  519. human output instead of mixing into the scored groups above it.
  520. Pass `--include-test-paths` to score and fail on these findings exactly like
  521. any other — useful if your project ships code that legitimately lives under a
  522. `tests/`-shaped path.
  523. ### Diagnostics (`--verbose` / `--debug`)
  524. `--verbose` (`-v`) narrates the scan on **stderr**: rule provenance (repo, ref,
  525. resolved SHA, and any cache fallback), per-phase discovery counts (languages,
  526. tools, agents, detected SDKs, loaded detectors, unaudited SDKs), output
  527. destinations, and a final result summary (scan ID, score, findings by severity,
  528. exit code). `--debug` adds everything `--verbose` shows plus per-phase timing and
  529. capped per-entity / per-finding detail (each discovered tool/agent and each
  530. finding with its `file:line`).
  531. Both are **global** flags — they work on `scan`, `mcp`, and `rules pull`, and may
  532. appear before or after the subcommand (`trustabl -v scan …` or `trustabl scan -v
  533. …`). `--debug` implies `--verbose`. Both write **only to stderr**, so they never
  534. perturb the report on stdout or the JSON/SARIF byte-stability contract:
  535. `--format json --debug` still emits a clean document on stdout while the
  536. diagnostics stream to stderr. Diagnostic color follows the same rules as the
  537. report (off under `--no-color`, `NO_COLOR`, or when stderr is not a terminal).
  538. Because an animated progress panel and interleaved log lines would corrupt each
  539. other on the same stderr, `--verbose`/`--debug` automatically render progress as
  540. plain `[phase]` lines instead of the live spinner.
  541. **Saving diagnostics to a file.** There is no dedicated `--log-file` flag —
  542. because diagnostics are a separate stream (stderr), redirecting stderr is the
  543. intended mechanism:
  544. ```bash
  545. # Report and diagnostics to separate files (stdout vs stderr)
  546. trustabl scan ./repo --debug --format json >report.json 2>diagnostics.log
  547. # Human report on screen, diagnostics to a file
  548. trustabl scan ./repo --debug 2>diagnostics.log
  549. # Everything (report + diagnostics) in one file
  550. trustabl scan ./repo --debug &>everything.log
  551. ```
  552. With `--format json`/`sarif` progress is off, so the stderr file is
  553. diagnostics-only; with `--format human` it also carries the plain `[phase]`
  554. progress lines.
  555. Exit codes (a test-path finding is excluded from this gate unless
  556. `--include-test-paths` is set — see "Test-path findings" above):
  557. - `0` — no findings ≥ medium severity (or no findings at all).
  558. - `1` — at least one finding ≥ medium severity, OR `--strict` with any
  559. finding present.
  560. - `2` — scanner / I/O error, OR no usable rules found and none fetchable
  561. (run `trustabl rules pull`), OR a signed channel (`--rules-source`) that failed
  562. verification (bad signature, untrusted/expired key, channel confusion, an
  563. expired or rolled-back statement, or a digest mismatch) — Trustabl refuses to
  564. run unverified rules.
  565. OpenShell surfaces are still discovered (shell-invocation functions,
  566. `openshell/*.yaml` policies) and reported on a `Risk surfaces: openshell`
  567. block in the human format: the count of shell-invoking functions, the first
  568. three file:line locations (deterministically sorted), a `why:` line stating
  569. the threat model (a prompt-injected agent that exposes one of these as a
  570. callable tool can run arbitrary commands), and a `fix:` line with concrete
  571. remediations (sandbox, allowlist, drop `shell=True`, keep shell logic out
  572. of agent-callable code). The OSH-* detection rules that audited these
  573. surfaces have moved to a closed-source companion project; with no OSH rules
  574. shipped, such repos fire no rule and no `META` finding — the block makes
  575. the unaudited risk legible without claiming an audit happened. OpenShell is
  576. a risk surface, not an SDK, so it is not flagged as "unaudited" the way an
  577. unknown SDK would be.
  578. ## Install
  579. Install Agent Reliability Analyzer (the `trustabl` CLI) with Homebrew, Scoop, Docker or a direct download.
  580. ### Homebrew (macOS, Linux)
  581. ```sh
  582. brew install trustabl/tap/trustabl
  583. ```
  584. ### Scoop (Windows)
  585. ```sh
  586. scoop bucket add trustabl https://github.com/trustabl/scoop-bucket
  587. scoop install trustabl
  588. ```
  589. ### Docker
  590. ```sh
  591. docker run --rm -v "$PWD:/repo" ghcr.io/trustabl/trustabl:latest scan /repo
  592. ```
  593. ### Direct download
  594. Grab a prebuilt archive for your platform from the
  595. [Releases page](https://github.com/trustabl/agent-reliability-analyzer/releases). Each release
  596. includes a `checksums.txt` and a build-provenance attestation; verify with:
  597. ```sh
  598. gh attestation verify <archive> --repo trustabl/agent-reliability-analyzer
  599. ```
  600. > Archives released **before** the repository was renamed verify only under the
  601. > old slug (`--repo trustabl/trustabl`): the Sigstore certificate binds the
  602. > repository path as it was at signing time, so the recorded identity cannot
  603. > follow a rename.
  604. ### Claude Code plugin
  605. The plugin installs two skills (`trustabl-scan` and `trustabl-enrich`), a
  606. `SessionStart` hook, and a bundled MCP server — and auto-downloads the pinned
  607. CLI binary on first session (no `sudo`, nothing outside Claude Code's plugin
  608. data directory touched).
  609. Install from within a Claude Code session using the `/plugin` slash command:
  610. ```text
  611. /plugin install trustabl@claude-plugins-official
  612. ```
  613. Then reload to activate:
  614. ```text
  615. /reload-plugins
  616. ```
  617. After installation, the `mcp__trustabl__scan` tool is available for direct
  618. calls, the `trustabl-scan` and `trustabl-enrich` skills appear in the skill
  619. picker, and the hook confirms the binary is ready at session start. Uninstall
  620. with `/plugin uninstall trustabl`.
  621. ### Claude Code agent
  622. A standalone subagent (`trustabl`) runs the full scan → enrich → review →
  623. apply pipeline interactively. It does not require the plugin — just the
  624. `trustabl` binary already on your PATH. Copy the agent file into your project
  625. or home config:
  626. ```sh
  627. # Project-level (scoped to this repo)
  628. curl -fsSL https://raw.githubusercontent.com/trustabl/agent-reliability-analyzer/main/agents/trustabl.md \
  629. -o .claude/agents/trustabl.md
  630. # Global (available in every project)
  631. curl -fsSL https://raw.githubusercontent.com/trustabl/agent-reliability-analyzer/main/agents/trustabl.md \
  632. -o ~/.claude/agents/trustabl.md
  633. ```
  634. Once installed, invoke it via `@trustabl` in Claude Code or select it from
  635. the subagent picker. It will ask which directory to scan, run `trustabl scan`
  636. and `trustabl enrich`, then walk you through each proposed fix as a diff.
  637. The enrich step requires an Anthropic API key. Set it before invoking the agent:
  638. ```sh
  639. export ANTHROPIC_API_KEY=sk-ant-api03-... # or: trustabl llm key set
  640. ```
  641. ### From source
  642. Requires `CGO_ENABLED=1` because the AST parsers use tree-sitter
  643. (Python + TypeScript + TSX bindings), which is a C library:
  644. ```bash
  645. # macOS / Linux
  646. CGO_ENABLED=1 go build -o trustabl ./cmd/trustabl
  647. # Cross-compile: pick a C toolchain for the target. zig is the easiest.
  648. CGO_ENABLED=1 CC="zig cc -target x86_64-linux-gnu" \
  649. GOOS=linux GOARCH=amd64 go build -o trustabl-linux ./cmd/trustabl
  650. ```
  651. This is the cost of using tree-sitter for accurate AST parsing. If a
  652. single-binary, no-CGO distribution becomes a hard requirement later, the
  653. swap target is `github.com/go-python/gpython` for Python (with lower
  654. fidelity on modern Python); TypeScript would need a separate replacement.
  655. ## Attestation
  656. ### Test Usage Example
  657. #### **Prerequisite and setup:**
  658. Install cosign (any **2.x or 3.x** — Trustabl adapts to the installed version)
  659. ##### Add to target folder path:
  660. `export PATH="<folder_path>:$PATH"`
  661. ##### Export and Generate Key Pair:
  662. `export COSIGN_PASSWORD="" `
  663. skip the passphrase prompt (omit for a real key)
  664. `cosign generate-key-pair`
  665. writes cosign.key (private) + cosign.pub (public)
  666. ##### Trustabl Scan and Attest:
  667. `./trustabl.exe scan https://github.com/google/adk-samples --json-out report.json --attest --attest-key cosign.key --attest-bundle att.bundle.json --attest-no-tlog`
  668. ##### VERIFICATION
  669. `./trustabl.exe verify report.json --key cosign.pub --bundle att.bundle.json --no-tlog`
  670. #### Keyless (CI / no key to manage)
  671. Keyless signs with your ambient **OIDC identity** instead of a key pair — nothing
  672. to generate or store. In CI (GitHub Actions) the runner's identity is used
  673. automatically; **on a laptop it opens a browser to log in**. Either way the
  674. signing event is recorded in the **public Rekor transparency log** — so use key
  675. mode above if that visibility is unacceptable. No `--no-tlog` here (keyless *is*
  676. the transparency-log path), so the cosign v2/v3 difference does not apply.
  677. ##### Scan and Attest (keyless):
  678. `./trustabl.exe scan https://github.com/google/adk-python --json-out report.json --attest --attest-bundle att.bundle.json`
  679. ##### VERIFICATION (keyless — you must pin who signed and the issuer):
  680. `./trustabl.exe verify report.json --bundle att.bundle.json --certificate-identity "<signer-identity>" --certificate-oidc-issuer "https://token.actions.githubusercontent.com"`
  681. In GitHub Actions the identity is the workflow ref, e.g.
  682. `https://github.com/OWNER/REPO/.github/workflows/scan.yml@refs/heads/main`.
  683. ### cosign (optional — only for scan attestation)
  684. Trustabl's `attest` / `verify` commands and `scan --attest` shell out to the
  685. [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) CLI —
  686. cosign does the signing and verification, and Trustabl ships no keys of its own.
  687. It is needed **only** if you use attestation; a plain `scan` never touches it.
  688. ```sh
  689. # macOS / Linux
  690. brew install cosign
  691. # Windows
  692. scoop install cosign
  693. ```
  694. In CI you do not install it by hand — the Trustabl GitHub Action and GitLab
  695. component add `sigstore/cosign-installer` for you.
  696. #### Scan with attestation — step by step (key mode, offline)
  697. Key mode signs with a local key pair and skips the public transparency log — no
  698. browser, no network, ideal for a laptop or air-gapped run.
  699. > **cosign version:** works with cosign **2.x and 3.x** — Trustabl selects the
  700. > right no-transparency-log mechanism per version automatically
  701. > (`--tlog-upload=false` on v2; a no-Rekor `--signing-config` on v3+, which
  702. > removed that flag). On a PATH install the command is `trustabl`; for a local
  703. > Windows build it is `.\trustabl.exe`.
  704. > **Attestation tiers:** this is the **free-tier basic attestation** — it signs
  705. > scan *provenance* (scan ID, rules version, reliability score, verdict, severity
  706. > counts). Behavioral and policy-aware attestations are on the paid roadmap.
  707. **Step 0 — install cosign** (see the commands above for your OS).
  708. **Step 1 — generate a signing key pair:**
  709. ```sh
  710. export COSIGN_PASSWORD="" # skip the passphrase prompt (omit for a real key)
  711. cosign generate-key-pair # writes cosign.key (private) + cosign.pub (public)
  712. ```
  713. **Step 2 — scan and sign in one step:**
  714. ```sh
  715. trustabl scan https://github.com/google/adk-samples \
  716. --json-out report.json \
  717. --attest --attest-key cosign.key --attest-bundle att.bundle.json --attest-no-tlog
  718. ```
  719. Writes `report.json` (the scan result), `trustabl-predicate.json`, and the signed
  720. `att.bundle.json`. Exit 1 only means findings were present — the bundle is still
  721. written (it signs the verdict, pass or fail).
  722. **Step 3 — verify (consumer side):**
  723. ```sh
  724. trustabl verify report.json --key cosign.pub --bundle att.bundle.json --no-tlog
  725. ```
  726. `Verified OK`, exit 0 — the report is authentic and unmodified.
  727. **Step 4 — (optional) prove tamper detection:**
  728. ```sh
  729. echo '{"tampered":true}' >> report.json
  730. trustabl verify report.json --key cosign.pub --bundle att.bundle.json --no-tlog
  731. ```
  732. Verification now FAILS (exit 1): the signature is bound to the exact report bytes,
  733. so any edit is caught.
  734. Keyless signing (no private key to manage) is the default in CI, where an ambient
  735. OIDC identity exists; it also records the signature in the public Rekor
  736. transparency log. See [Use](#use) for the full command surface.
  737. <a id="use"></a>
  738. ## Use — run your first agent reliability scan
  739. ```bash
  740. # Local repo
  741. trustabl scan ./path/to/agent-repo
  742. # GitHub repo (shallow clone to temp dir, removed on exit)
  743. trustabl scan https://github.com/org/repo
  744. # Restrict detectors
  745. trustabl scan ./repo --detectors claude_sdk
  746. trustabl scan ./repo --detectors openai_sdk
  747. trustabl scan ./repo --detectors google_adk
  748. trustabl scan ./repo --detectors claude_sdk,openai_sdk,google_adk
  749. # --detectors openshell is accepted but selects zero rules (pack is closed-source now)
  750. # Agent Skill security (SKILL.md) — flags unrestricted allowed-tools (a bare
  751. # `Bash` grant), pre-model dynamic-context exec, bundled-script network egress /
  752. # secret reads, committed secrets, hidden-Unicode prompt injection, and a
  753. # description that claims read-only while granting side-effecting tools (the
  754. # CSKILL-* rules). Skills are discovered and scanned automatically.
  755. trustabl scan ./repo # scans skills alongside tools/agents/MCP
  756. trustabl scan ./repo --detectors claude_skill # only the Agent Skill (CSKILL-*) rules
  757. trustabl scan ./path/to/my-skill # point straight at one skill's directory
  758. # Dependency BOM (supply chain): export the repo's DECLARED deps across all
  759. # supported languages (pip / npm / Go / Composer / NuGet / Cargo manifests) as a
  760. # CycloneDX SBOM, to hand to OSV-Scanner / Dependabot / syft. Pure inventory —
  761. # the scan itself does no CVE lookup.
  762. trustabl scan ./repo --bom-out sbom.json
  763. # Vulnerability scan (opt-in, online): match the repo's pinned deps against the
  764. # OSV database and FAIL on known CVEs — advisory id, CVSS severity, fixed version.
  765. trustabl vulndb pull # pre-download OSV (optional; --vuln-scan auto-fetches)
  766. trustabl scan ./repo --vuln-scan # BOM inventory + CVE verdict in one pass
  767. trustabl scan ./repo --vuln-scan --bom-out bom.json # CycloneDX BOM + VEX (vulnerabilities[]) in one file
  768. # License scan (opt-in): flag dependencies with copyleft licenses (GPL-2.0,
  769. # GPL-3.0, AGPL-3.0, LGPL-2.1, SSPL-1.0) as medium-severity findings.
  770. # License data is read from the manifest at parse time — no network call.
  771. trustabl scan ./repo --license-scan
  772. trustabl scan ./repo --license-scan --bom-out bom.json # findings + BOM with licenses[]
  773. # Secret scan (opt-in): walk all text files in the repo for hardcoded
  774. # credential literals (SECRET-LIT-001, high) and scripts that read
  775. # credential environment variables (SECRET-ENV-001, medium).
  776. trustabl scan ./repo --secret-scan
  777. # Test-path findings (a tests/ agent, test_*.py, *.spec.ts, …) are always
  778. # reported but excluded from scoring and the exit code by default — see
  779. # "Test-path findings" below. --include-test-paths scores and fails on them
  780. # like any other finding.
  781. trustabl scan ./repo --include-test-paths
  782. # JSON output for CI piping
  783. trustabl scan ./repo --format json
  784. # SARIF output for GitHub Code Scanning / SARIF-aware tools
  785. trustabl scan ./repo --format sarif > trustabl.sarif
  786. # Write the report to a file instead of stdout (any format). --output writes
  787. # the file even when the scan exits 1 on findings, so a CI step can upload it.
  788. trustabl scan ./repo --format sarif --output trustabl.sarif
  789. # One scan, both machine artifacts written to files (human summary to stdout)
  790. trustabl scan ./repo --json-out trustabl.json --sarif-out trustabl.sarif
  791. # Exit 1 on any finding regardless of severity
  792. trustabl scan ./repo --strict
  793. # --- Attestation (opt-in; requires the cosign CLI on PATH) -------------------
  794. # Sign the JSON report into a cosign attestation of the SCANNED repo's result.
  795. # Keyless by default: in CI it uses the runner's ambient OIDC identity (no keys
  796. # to manage) and logs the signing event to the PUBLIC Rekor transparency log.
  797. trustabl scan ./repo --json-out trustabl.json --attest
  798. trustabl attest trustabl.json # same, as a separate step
  799. # Offline / private signing with a key (no public transparency log):
  800. trustabl attest trustabl.json --key cosign.key --no-tlog
  801. # Verify (run where you CONSUME the attestation). Keyless pins who signed + issuer:
  802. trustabl verify trustabl.json \
  803. --certificate-identity https://github.com/OWNER/REPO/.github/workflows/scan.yml@refs/heads/main \
  804. --certificate-oidc-issuer https://token.actions.githubusercontent.com
  805. trustabl verify trustabl.json --key cosign.pub --no-tlog # key-mode, offline
  806. # Download / refresh the detection rule packs into the local cache
  807. trustabl rules pull
  808. # Validate a local rule-pack directory against this build's schema (CI gate
  809. # for the agent-reliability-rules repo — strict-loads every pack, fails on the first error)
  810. trustabl rules validate ./agent-reliability-rules
  811. # Use a custom rules repo, or pin a specific released ruleset (env: TRUSTABL_RULES_REPO).
  812. # Default pulls the latest reviewed rules from agent-reliability-rules main; pin a tag for stability.
  813. trustabl scan ./repo --rules-repo https://github.com/org/my-rules
  814. trustabl scan ./repo --rules-ref v0.1.0
  815. # Air-gapped / offline: skip the network fetch, use the cached rules only
  816. trustabl scan ./repo --no-rules-update
  817. # Progress output (human format): animated on a terminal, plain lines when piped
  818. trustabl scan ./repo # spinner + bars on a TTY; "[phase] summary" lines when piped
  819. trustabl scan ./repo --no-progress # disable progress entirely
  820. # Diagnostics on stderr (global flags; stdout/report unaffected)
  821. trustabl scan ./repo --verbose # -v: rule provenance, discovery counts, result summary
  822. trustabl scan ./repo --debug # + per-phase timing and per-entity/per-finding detail
  823. trustabl scan ./repo --debug --format json > out.json # clean JSON on stdout, diagnostics on stderr
  824. # Run as a stdio MCP server so an MCP client (Claude Code, Cursor, Claude
  825. # Desktop) can scan code an agent just wrote (see "Run as an MCP server" below)
  826. trustabl mcp
  827. # Configure LLM provider, then enrich a scan result with AI explanations and fixes
  828. export ANTHROPIC_API_KEY=sk-ant-api03-... # preferred: env var (works in CI, no setup)
  829. trustabl llm list # show configured providers with masked keys
  830. trustabl llm key set # prompt securely for an API key (persistent)
  831. trustabl llm key set sk-ant-api03-... # set key non-interactively
  832. trustabl llm key get # show masked key for active provider
  833. trustabl llm key delete # delete key with confirmation prompt
  834. trustabl llm model set claude-sonnet-4-6 # change model for active provider
  835. trustabl llm base-url set http://host:4000 # set API base URL (only meaningful for "custom")
  836. trustabl llm provider set openai # switch active provider (auto-creates entry)
  837. trustabl llm provider list # list configured providers
  838. # Self-hosted / local model server (any OpenAI-compatible endpoint, e.g. a
  839. # LiteLLM or vLLM gateway). Two ways to configure it:
  840. #
  841. # 1. Env vars — no persistent config, works in CI:
  842. # OPENAI_BASE_URL alone activates "custom", ahead of a plain OPENAI_API_KEY.
  843. # No default model, so TRUSTABL_LLM_MODEL is required. OPENAI_API_KEY is
  844. # optional: only set it if the endpoint actually checks it.
  845. export OPENAI_BASE_URL=http://localhost:4000
  846. export TRUSTABL_LLM_MODEL=qwen-32b
  847. #
  848. # 2. Persistent config — same three fields, saved to ~/.config/trustabl/keys.json:
  849. trustabl llm provider set custom
  850. trustabl llm base-url set http://localhost:4000
  851. trustabl llm model set qwen-32b
  852. # trustabl llm key set is optional — skip it if the endpoint needs no key
  853. # Enrich a scan result (requires any configured LLM provider — a custom
  854. # endpoint may not need a key, every other provider does)
  855. trustabl scan ./myrepo --format json | trustabl enrich --repo ./myrepo # pipe scan into enrich (stdout)
  856. trustabl enrich --input scan.json --repo ./myrepo --output enriched.json # file in, file out
  857. trustabl enrich --input scan.json --repo ./myrepo --diff # preview proposed fixes as a unified diff (stderr)
  858. trustabl enrich --input scan.json --repo ./myrepo --diff --apply # preview and apply fixes
  859. trustabl enrich --input scan.json --repo ./myrepo --apply # apply fixes without previewing
  860. trustabl enrich --input scan.json --repo ./myrepo --rule CSDK-010 # focus on one rule
  861. trustabl enrich --input scan.json --repo ./myrepo --only-enriched # CI: only enriched findings
  862. # Ground tool findings in LangSmith runtime traces (opt-in; error rates, latency,
  863. # recent errors are fetched per flagged tool and fed to the LLM as evidence)
  864. export LANGSMITH_API_KEY=lsv2_... # BYOK; LANGSMITH_ENDPOINT overrides for self-hosted
  865. trustabl enrich --input scan.json --repo ./myrepo --langsmith # project: $LANGSMITH_PROJECT or "default"
  866. trustabl enrich --input scan.json --repo ./myrepo --langsmith --langsmith-project prod
  867. ```
  868. Rules are cached under your OS cache dir (`os.UserCacheDir()`, e.g.
  869. `%LocalAppData%\trustabl\rules\` on Windows, `~/.cache/trustabl/rules/`
  870. on Linux). The first scan (or an explicit `trustabl rules pull`)
  871. populates it; each subsequent scan checks for an update first (unless
  872. `--no-rules-update`), falling back to the cached rules if the fetch
  873. fails.
  874. **Signed rules channels (default).** A plain `trustabl scan` resolves rules from
  875. the signature-verified **production** release channel: Trustabl verifies a
  876. signed channel statement against an embedded trust keyring, fetches the bundle
  877. it commits to, and refuses (exit `2`) on any verification failure rather than
  878. running unverified rules. `--rules-source git` selects the unsigned git path
  879. explicitly, and `--rules-repo` / `--rules-ref` imply it; a non-default channel
  880. can still be pointed at a fork with `--rules-source <name> --rules-repo <fork>`
  881. to test a signed fork. (`--channel <name>` is a deprecated alias for
  882. `--rules-source <name>`.) The **default scan now uses the signed production
  883. channel**; set `TRUSTABL_REQUIRE_SIGNED=1` (or `--require-signed`) to forbid the
  884. unsigned git fallback entirely in CI. Already-shipped pre-cutover binaries are
  885. unaffected (the default and keyring are compiled in); upgrading shifts `ScanID`
  886. once as the origin tag and git-SHA become the channel's bundle digest. A scan
  887. that did not use blessed
  888. production rules (a pre-release channel, or a custom `--rules-repo` source) is
  889. watermarked in the report and in the JSON `rules_origin` field, and its
  890. provenance is folded into `ScanID`. This build embeds the rule-signing public
  891. key, so a signed `--rules-source` runs the verification pipeline and refuses
  892. (exit `2`) only on a genuine verification failure (bad signature, untrusted or
  893. expired key, channel mismatch, expired or rolled-back statement, or digest
  894. mismatch).
  895. ### Pre-coding constraints (trustabl forge)
  896. `trustabl forge` is the counterpart to `trustabl scan`: instead of auditing
  897. code *after* it is written, it generates a combined pre-coding `SKILL.md` that
  898. tells an agent *before* it writes any code which patterns to avoid. The
  899. generated skill organizes rules by SDK and scope (tool, agent, subagent, repo,
  900. skill) and is ordered by severity so the highest-risk constraints appear first.
  901. **How SDK detection works.** Forge scans the target's dependency manifests —
  902. `pyproject.toml`, `requirements.txt`, `Pipfile`, `poetry.lock`, `package.json`,
  903. and `go.mod` — to determine which SDK policy packs to include. This is the
  904. same lightweight dep-file recon that `trustabl scan` performs in its recon
  905. step: no AST parsing, fast, and safe on any repo. When no manifests are found
  906. (or none declare a recognized SDK), forge exits `1` and tells you what to pass
  907. via `--policy`.
  908. **Language detection.** The same manifests tell forge which language each SDK
  909. is declared in: `pyproject.toml`, `requirements.txt`, `Pipfile` and
  910. `poetry.lock` mean Python, `package.json` means TypeScript, `go.mod` means Go.
  911. That selects the language blocks of the Runtime Tracing section. Pass
  912. `--lang python,typescript,go` to add a language. `--lang` does not select a
  913. rule pack, so a brand-new repo with no manifest yet needs `--policy` as well
  914. (see the last example below). The same languages filter
  915. the rule sections: an SDK declared only in `go.mod` gets only its Go rules
  916. and the rules that apply in every language, so a Go-only Google ADK repo gets
  917. the Go tracing recipe, the language-independent ADK rules, and a note that no
  918. Go-specific ADK rules exist yet. A category given only through `--policy`, or
  919. also declared in a manifest forge cannot place (such as `composer.json`), keeps
  920. all of its rules.
  921. ```bash
  922. # Auto-detect SDKs from the current directory
  923. trustabl forge
  924. # Auto-detect from a specific repo path
  925. trustabl forge /path/to/repo
  926. # Add a policy on top of auto-detected SDKs — useful when introducing a new
  927. # SDK that is not yet in the dependency manifest
  928. trustabl forge --policy openai_sdk
  929. trustabl forge /path/to/repo --policy claude_sdk,mcp
  930. # Write the generated SKILL.md to a file instead of stdout
  931. trustabl forge -o .claude/skills/trustabl-pre-coding/SKILL.md
  932. # Pin a specific rules version (same as trustabl scan --rules-ref)
  933. trustabl forge --rules-ref v0.3.0
  934. # Backward-compatible: generate only the Agent Skill (CSKILL-*) rules
  935. trustabl forge --policy claude_skill
  936. # New Go repo with no go.mod yet
  937. trustabl forge --policy google_adk --lang go
  938. ```
  939. **Available policy categories** (pass to `--policy`):
  940. | Category | SDK / framework |
  941. |---|---|
  942. | `claude_sdk` | Claude Agent SDK (Python + TypeScript) |
  943. | `openai_sdk` | OpenAI Agents SDK (Python + TypeScript) |
  944. | `google_adk` | Google Agent Development Kit (Python + TypeScript) |
  945. | `mcp` | Model Context Protocol servers |
  946. | `langchain` | LangChain / LangGraph (Python + TypeScript) |
  947. | `crewai` | CrewAI (Python) |
  948. | `pydantic_ai` | Pydantic AI (Python) |
  949. | `vercel_ai` | Vercel AI SDK (TypeScript) |
  950. | `autogen` | AutoGen / AG2 (Python) |
  951. | `claude_skill` | Claude Code skills (`SKILL.md`) |
  952. | `raw_llm_sdk` | Bare `anthropic` / `openai` client tool loops (not an agent framework) |
  953. | `agent_config` | Harness config committed to a repo that runs code or grants authority before the folder is trusted (`.claude/settings.json`, `.mcp.json`, `.gemini/.env`, `.vscode/settings.json`, `.codex/config.toml`) |
  954. | `ci` | Coding agents run from GitHub Actions workflows (`.github/workflows/*.yml`) on events any GitHub user can cause |
  955. | `openshell` | Shell-invocation tools (no rules in the current pack) |
  956. Multiple categories are comma-separated: `--policy openai_sdk,mcp`.
  957. **What the output contains.** The generated file is a valid `SKILL.md` with:
  958. - A frontmatter block (`name: trustabl-pre-coding`, `allowed-tools: Read`,
  959. `disable-model-invocation: false`).
  960. - A passive stamp comment embedding the rules SHA, date, schema version,
  961. template (layout) version, and detected SDK list — so you know exactly
  962. which ruleset and layout produced the file and when to regenerate.
  963. - A `## How to Apply These Constraints` apply-loop section, immediately after
  964. the stamp: a static four-step procedure telling the model to check each
  965. definition it writes against the matching constraints, name any violation
  966. by rule ID, apply that rule's directive, and log the repair. This is a
  967. self-check the model performs — it does not run `trustabl scan` itself;
  968. scanning the resulting code remains a separate step.
  969. - A `## Tool-Call Claims` section: every tool call with an effect carries a small typed
  970. claim (`ToolInvocationExplanation`: intent class, selected tool, expected
  971. effect, risk tier, uncertainty, untrusted-context dependencies, evidence
  972. digests), and a mediation step outside the model checks it against
  973. server-held facts before the call has any effect. Conflicts deny, unknowns
  974. go to review, and the most restrictive outcome wins; the claim never grants
  975. authority. It follows Explanation-Bound Tool Execution (EBTE) and names
  976. where the step goes in each detected SDK. Left out when `claude_skill` is
  977. the only category.
  978. - A `## Runtime Tracing` section: how to make the agent emit OpenTelemetry
  979. spans, then how to label them with `agent-reliability-otel-labels` (run
  980. identity, step order, tool-call fingerprints, retry count, side-effect class,
  981. exit reason). It has a block per selected language (Python, TypeScript, Go)
  982. and a verified setup for Google ADK (Python, Go), the Vercel AI SDK (TypeScript),
  983. Pydantic AI (Python), the OpenAI Agents SDK (Python, TypeScript),
  984. LangChain / LangGraph (Python), CrewAI (Python), AutoGen (Python), and MCP
  985. client sessions (Python). Other frameworks get the language block only.
  986. Non-OpenTelemetry tracing stacks are not covered. The section is left out
  987. when no supported language is found, and when `claude_skill` is the only
  988. category (Agent Skills are `SKILL.md` files, not agent code to trace).
  989. - One `## SDK Name` section per detected SDK, each containing `### Tool Rules`,
  990. `### Agent Rules`, `### Subagent Rules`, `### Repo Rules`, and
  991. `### Skill Rules` subsections (empty subsections are omitted).
  992. - Per-rule blocks: **Directive** (the fix, first sentence), **Why** (the
  993. explanation, first sentence), **When this applies**, severity, and confidence.
  994. Drop the output into `.claude/skills/trustabl-pre-coding/SKILL.md` in the
  995. agent repo and Claude Code will invoke it automatically before any agent code
  996. is written. Re-run `trustabl forge` after adding a new SDK dependency, after
  997. pulling a rules update, or after upgrading to a forge version with a newer
  998. template — `trustabl forge check` reports staleness from either cause.
  999. ### Continuous integration
  1000. Two CI patterns are supported, and they compose:
  1001. - **Gate the build.** The exit code is the gate: `0` clean, `1` on a finding
  1002. of medium severity or higher (`--strict` lowers the bar to any finding),
  1003. `2` on an operational error. A bare `trustabl scan ./repo` in a job step
  1004. fails the job when it should.
  1005. - **Publish to GitHub Code Scanning.** On GitHub Actions, the
  1006. [`trustabl/trustabl-action`](https://github.com/trustabl/trustabl-action)
  1007. runs the scan and uploads the SARIF to the repository's Security tab in a
  1008. single step (`upload-sarif` defaults to `true`), with inline PR alerts and
  1009. optional threshold gating — the recommended path, and the single source of
  1010. truth for the workflow. Outside GitHub Actions, `--format sarif --output
  1011. <file>` writes a SARIF 2.1.0 report that any
  1012. `github/codeql-action/upload-sarif` or SARIF-aware step can publish. Because
  1013. `--output` writes the file before the findings-based exit code is applied,
  1014. the scan step can run with `continue-on-error: true` and the upload with
  1015. `if: always()`, so a scan that finds issues still surfaces them instead of
  1016. aborting the run with nothing uploaded.
  1017. The SARIF document is a pure function of the scan result: byte-stable across
  1018. identical-input runs, repo-relative file URIs, and a stable
  1019. `partialFingerprints` per finding so Code Scanning deduplicates alerts across
  1020. runs rather than re-opening them.
  1021. ### Run as an MCP server
  1022. `trustabl mcp` runs a Model Context Protocol (MCP) server over stdio, so an MCP
  1023. client (Claude Code, Cursor, Claude Desktop) can scan a directory an agent just
  1024. edited and read the findings back. It is the same scan as `trustabl scan`,
  1025. exposed as an MCP tool — it opens no network port. The server speaks JSON-RPC on
  1026. stdout, so it writes nothing else there; status lines and diagnostics go to
  1027. stderr.
  1028. It exposes two tools:
  1029. - `scan` — input `{ "path": "<dir>", "rules_ref": "<branch-or-tag>"?, "vuln_scan": true? }`.
  1030. Scans `path` and returns the full scan result (findings, scores, discovered
  1031. inventory) as JSON — the same shape as `--format json`. Set `vuln_scan: true`
  1032. to also match declared dependencies against a pinned OSV snapshot and report
  1033. known CVEs (mirrors `--vuln-scan`; off by default).
  1034. - `version` — reports the build version, commit, and date.
  1035. Register it with an MCP client by pointing the client at the binary with the
  1036. `mcp` argument over stdio. For Claude Code:
  1037. ```bash
  1038. claude mcp add trustabl -- trustabl mcp
  1039. ```
  1040. For Gemini CLI, add to `~/.gemini/settings.json`:
  1041. ```json
  1042. {
  1043. "mcpServers": {
  1044. "trustabl": {
  1045. "command": "trustabl",
  1046. "args": ["mcp"]
  1047. }
  1048. }
  1049. }
  1050. ```
  1051. For OpenAI Codex, add to `~/.codex/config.toml`:
  1052. ```toml
  1053. [mcp_servers.trustabl]
  1054. command = "trustabl"
  1055. args = ["mcp"]
  1056. ```
  1057. Or configure it directly in a client's MCP config (the `mcpServers` stdio
  1058. shape used by Claude Desktop / Cursor):
  1059. ```json
  1060. {
  1061. "mcpServers": {
  1062. "trustabl": {
  1063. "command": "trustabl",
  1064. "args": ["mcp"]
  1065. }
  1066. }
  1067. }
  1068. ```
  1069. The rules-source flags (`--rules-repo`, `--rules-ref`, `--no-rules-update`) work
  1070. on `trustabl mcp` exactly as on `trustabl scan`; a client may also pass a
  1071. per-call `rules_ref` in the `scan` tool arguments, which overrides the
  1072. command-level `--rules-ref` for that call.
  1073. ### Claude Code plugin (self-audit at generation time)
  1074. Trustabl ships a Claude Code plugin under [`.claude-plugin/`](.claude-plugin/)
  1075. with two skills that form a scan-and-fix loop:
  1076. - **[`trustabl-scan`](skills/trustabl-scan/)** — triggers right after agent,
  1077. tool, subagent, or MCP-server code is written or changed and calls Trustabl's
  1078. `scan` tool (the bundled MCP server, `mcp__trustabl__scan`) to self-audit it
  1079. before committing, upstream of CI.
  1080. - **[`trustabl-enrich`](skills/trustabl-enrich/)** — takes the output of a
  1081. `trustabl scan` run (JSON, SARIF, or pasted terminal text) and applies each
  1082. finding as a targeted code edit, guided entirely by the scan's own
  1083. `explanation` and `suggested_fix` fields. It does not re-run the scanner; use
  1084. `trustabl-scan` first, then invoke `trustabl-enrich` with the results.
  1085. Scanning runs through a **bundled MCP server**: [`.mcp.json`](.mcp.json)
  1086. registers a `trustabl` server whose command is a launcher
  1087. ([`scripts/trustabl-mcp.sh`](scripts/trustabl-mcp.sh)) exposing the
  1088. `mcp__trustabl__scan` tool. The launcher and a `SessionStart` hook
  1089. ([`hooks/hooks.json`](hooks/hooks.json) →
  1090. [`scripts/check-trustabl.sh`](scripts/check-trustabl.sh)) share install logic
  1091. ([`scripts/lib-trustabl.sh`](scripts/lib-trustabl.sh)) that downloads the pinned
  1092. CLI version from GitHub Releases, verifies it against the release
  1093. `checksums.txt`, and installs it into the plugin's private data directory
  1094. (`$CLAUDE_PLUGIN_DATA` — no `sudo`, nothing outside that dir touched). The
  1095. install is idempotent (re-runs only when the pin changes or the copy is
  1096. missing/corrupt), and the launcher installs synchronously before starting the
  1097. server, so there is no first-session race. The same binary is exposed as
  1098. `$TRUSTABL_BIN` for the enrich skill's direct-CLI path. When auto-install cannot
  1099. run (offline first session, an unsupported platform, or missing `curl`/`tar`) it
  1100. falls back to whatever `trustabl` is on `PATH`; the system-wide install stays a
  1101. consented step inside `trustabl-scan`. The plugin runs no network service of its
  1102. own and modifies nothing outside the scan target.
  1103. ### Gemini CLI extension (self-audit at generation time)
  1104. Trustabl ships a Gemini CLI extension under [`gemini-extension.json`](gemini-extension.json)
  1105. with the same two skills available in the Claude Code plugin.
  1106. **Prerequisites.** Install the Trustabl binary first — the extension does not
  1107. auto-install it the way the Claude Code plugin does (Gemini CLI has no private
  1108. plugin data directory):
  1109. ```bash
  1110. # macOS / Linux
  1111. brew install trustabl/tap/trustabl
  1112. # Windows
  1113. scoop bucket add trustabl https://github.com/trustabl/scoop-bucket
  1114. scoop install trustabl
  1115. ```
  1116. **Install the extension:**
  1117. ```bash
  1118. gemini extensions install https://github.com/trustabl/agent-reliability-analyzer
  1119. ```
  1120. **Register the MCP server** so Gemini CLI can call `mcp__trustabl__scan`. Add
  1121. to `~/.gemini/settings.json`:
  1122. ```json
  1123. {
  1124. "mcpServers": {
  1125. "trustabl": {
  1126. "command": "trustabl",
  1127. "args": ["mcp"]
  1128. }
  1129. }
  1130. }
  1131. ```
  1132. **How it works.** When a Gemini CLI session starts, the extension injects
  1133. [`GEMINI.md`](GEMINI.md) as context, which pulls in the full `trustabl-scan`
  1134. and `trustabl-enrich` skill content via `@` imports. Gemini then treats those
  1135. skills exactly as it would any loaded context: it runs the scan after you write
  1136. or change agent, tool, or MCP-server code, and offers to apply findings via
  1137. `trustabl-enrich`. The MCP tool is named `mcp__trustabl__scan` — the same name
  1138. as in Claude Code.
  1139. **Update the extension later:**
  1140. ```bash
  1141. gemini extensions update trustabl
  1142. ```
  1143. ### OpenAI Codex plugin (self-audit at generation time)
  1144. Trustabl ships an OpenAI Codex plugin under [`.codex-plugin/`](.codex-plugin/)
  1145. with the same two skills.
  1146. **Prerequisites.** Install the Trustabl binary first:
  1147. ```bash
  1148. # macOS / Linux
  1149. brew install trustabl/tap/trustabl
  1150. # Windows
  1151. scoop bucket add trustabl https://github.com/trustabl/scoop-bucket
  1152. scoop install trustabl
  1153. ```
  1154. **Install the plugin.** In Codex CLI, open the plugin search interface and
  1155. search for `trustabl`:
  1156. ```bash
  1157. /plugins
  1158. ```
  1159. Search for `trustabl` and select **Install Plugin**. In the Codex App, go to
  1160. **Plugins** in the sidebar, find Trustabl under the Security category, and
  1161. click **+**.
  1162. **Register the MCP server** so Codex can call `mcp__trustabl__scan`. Add to
  1163. `~/.codex/config.toml`:
  1164. ```toml
  1165. [mcp_servers.trustabl]
  1166. command = "trustabl"
  1167. args = ["mcp"]
  1168. ```
  1169. **How it works.** Codex reads [`.codex-plugin/plugin.json`](.codex-plugin/plugin.json),
  1170. which points `"skills": "./skills/"`. Codex loads the `trustabl-scan` and
  1171. `trustabl-enrich` skills from that directory automatically — no explicit
  1172. invocation step required. Skills trigger when you write or modify agent, tool,
  1173. or MCP-server code, and Codex applies findings via `trustabl-enrich`. The MCP
  1174. tool is named `mcp__trustabl__scan` — the same name as in Claude Code and Gemini CLI.
  1175. To enable subagent delegation in Codex (used by `trustabl-enrich` for parallel
  1176. file edits), set `multi_agent = true` in `~/.codex/config.toml`.
  1177. ### "rules are newer than this Trustabl build"
  1178. Trustabl loads rules **forward-compatibly**. If the resolved pack targets a
  1179. newer rule-schema version than your binary understands, the scan still runs: it
  1180. evaluates every rule your build *can* understand and **skips** the rest, warning
  1181. on stderr (and recording the skipped rule IDs on `ScanResult.RulesSkipped`):
  1182. ```text
  1183. warning: the rules target schema version 13 but this Trustabl build supports up to 12; 2 rule(s) newer than this build were skipped. Upgrade Trustabl to evaluate them.
  1184. ```
  1185. The same per-rule skip happens for any *individual* rule that references a
  1186. `scope`, an `applies_to` value, or a predicate your build does not understand —
  1187. that rule is dropped while its siblings still run, so a newer rules release never
  1188. forces a lockstep binary upgrade. Every skip is also surfaced **in the report
  1189. itself** as a single `META-005` info finding ("N rules require a newer Trustabl
  1190. engine"), so a degraded scan is never mistaken for a clean one. A *malformed*
  1191. rule your build does understand (a missing field, a bad value) is **not**
  1192. forward-skipped — it still hard-fails the load, so real authoring errors are
  1193. never silently dropped.
  1194. To evaluate the skipped rules, **upgrade Trustabl** to a build whose
  1195. `SupportedSchemaVersion` (see `internal/rules/schema_version.go`) covers the
  1196. pack. No action is needed if you're comfortable running the subset.
  1197. The scan only **fails** (exit 2) when nothing usable remains:
  1198. - **"all rules require a newer engine schema"** — *every* rule is too new for
  1199. your build, so there is nothing to run. Upgrade Trustabl, or pin an older
  1200. rules branch/tag your build fully understands (`--rules-ref` resolves branches
  1201. and tags only, not raw commit SHAs, so a compatible ref must already exist):
  1202. ```bash
  1203. trustabl scan ./repo --rules-ref <branch-or-tag>
  1204. ```
  1205. - **"no usable rules manifest"** — the pack's `manifest.yaml` is missing,
  1206. unparseable, or declares a non-positive version (a corrupt/truncated pack).
  1207. Run `trustabl rules pull` to refresh.
  1208. - **"no usable rules found"** — nothing cached and nothing fetchable (offline
  1209. with a cold cache). Run `trustabl rules pull` while online.
  1210. ## Where the code lives
  1211. | Pipeline node | Code path |
  1212. | ------------------ | ---------------------------------------- |
  1213. | Importer | `internal/ingestion/importer.go` |
  1214. | Normalizer (recon) | `internal/ingestion/normalizer.go` |
  1215. | Discovery (Python AST + markdown/JSON) | `internal/analysis/discovery.go`, `agents.go`, `hosted_tools.go`, `mcp_servers.go`, `adk_agents.go` (Python AST); `subagents.go`, `markdown_agents.go`, `skills.go`, `slash_commands.go` (markdown frontmatter); `plugins.go`, `claude_settings.go` (JSON) |
  1216. | TypeScript discovery | `internal/analysis/ts_discovery.go`, `ts_agents.go`, `ts_mcp_servers.go`, `ts_handler_facts.go`, `ts_openai_tools.go`, `ts_openai_agents.go`, `ts_openai_hosted_tools.go`, `ts_openai_mcp_servers.go`, `ts_openai_guardrails.go`, `ts_openai_sessions.go`, `ts_adk_tools.go`, `ts_adk_agents.go`, `ts_adk_hosted_tools.go`, `astutil/ts.go` |
  1217. | Detector runtime | `internal/analysis/detectors/` |
  1218. | Rule source | `internal/rulesource/` (git fetch + cache + schema-version gate) |
  1219. | Detector rules | external `agent-reliability-rules` repo (tests: `testdata/rules-fixture/`) |
  1220. | Rule engine | `internal/rules/{schema,loader,evaluator,predicates,rule_detector}.go` |
  1221. | Scoring engine | `internal/analysis/scoring.go` |
  1222. | Report renderer | `internal/review/diff.go` (human), `internal/sarif/render.go` (SARIF), JSON marshal in `cmd/trustabl` |
  1223. | LLM config | `internal/llm/` (key storage · masking · validation) |
  1224. Rule packs live in the separate `agent-reliability-rules` git repository (grouped
  1225. `{claude_sdk,openai_sdk,google_adk,mcp}/`), resolved at scan time rather
  1226. than embedded in the binary. Naming convention: `CSDK-NNN` for Claude
  1227. Agent SDK rules (CSDK-0xx tool-scope, CSDK-1xx agent + subagent-scope),
  1228. `OAI-NNN` for OpenAI Agents SDK rules, `ADK-NNN` for Google ADK rules,
  1229. `MCP-NNN` for the dedicated MCP tool-scope pack.
  1230. See
  1231. [ARCHITECTURE.md § 2 — steps 3–4](ARCHITECTURE.md#2-pipeline) for the
  1232. shipped rule table and [COVERAGE.md](COVERAGE.md) for per-SDK
  1233. recognition detail.
  1234. ## Testing
  1235. `testdata/corpus/` holds real-world agent code (Claude SDK demos, OpenAI Agents
  1236. SDK demos, Google ADK demos, a TS Claude SDK fixture) — a corpus, not a
  1237. controlled fixture, so well-written agents won't trigger most rules and
  1238. that's correct. See [`testdata/corpus/PROVENANCE.md`](testdata/corpus/PROVENANCE.md)
  1239. for upstream sources and licenses of each example. Per-rule fire/silent
  1240. correctness lives in `internal/rules/policies_test.go`; the end-to-end
  1241. sweep in `internal/scanner/scanner_test.go` only asserts the scanner
  1242. doesn't crash on real-world inputs. A labelled 20–40 real-agent-repo
  1243. corpus is the detection-quality target (see
  1244. [ARCHITECTURE.md § 10](ARCHITECTURE.md#10-what-is-intentionally-out));
  1245. the current tests are regression coverage, not detection-quality
  1246. measurement.
  1247. ## Privacy & telemetry
  1248. Trustabl optionally collects anonymous usage data — which SDKs are scanned, scan duration, which rules fire — to guide product direction. **No source code, file paths, repo names, or finding details are ever sent.** Telemetry is **off by default**; on your first interactive scan, Trustabl asks you to choose a level.
  1249. On a panic, Trustabl writes a scrubbed crash report to `~/.config/trustabl/crash-<timestamp>.log` and, only in an interactive terminal, offers to send it anonymously or open a pre-filled GitHub issue — otherwise nothing is transmitted. There is no auto-send: the default is always silence, and sending is an explicit per-crash choice. The report contains the panic message (with common secret shapes redacted best-effort) and stack frames; no source code, file contents, or argument values are included.
  1250. Three levels are available:
  1251. - **Disabled** — no data sent (default)
  1252. - **Minimal** — CLI version and scan outcome only
  1253. - **Full** — anonymous usage stats
  1254. ```sh
  1255. trustabl telemetry off # disable
  1256. trustabl telemetry minimal # version and outcome only
  1257. trustabl telemetry full # full anonymous usage stats
  1258. export TRUSTABL_TELEMETRY=0 # or disable for the current shell session
  1259. trustabl telemetry status # show current level and its source
  1260. ```
  1261. The full event schema — every event name and every property, verbatim — is at [TELEMETRY.md](TELEMETRY.md) (and at [trustabl.ai/docs/telemetry](https://www.trustabl.ai/docs/telemetry/)). That page is kept in sync with the code; stale docs are treated as a defect.
  1262. ## FAQ
  1263. **What is Agent Reliability Analyzer?**
  1264. An open-source, deterministic scanner from Trustabl that finds and fixes agent reliability failures in AI agent code before production, across nine agent SDKs and seven languages.
  1265. **How do I test AI agent reliability before production?**
  1266. Run `trustabl scan ./your-agent-repo` locally or in CI. It checks tools, agents, subagents, skills and MCP servers against agent reliability rules, and exits non-zero on medium-or-higher findings so the CI job fails when it should.
  1267. **Does it replace LangSmith, Langfuse, Braintrust or Arize?**
  1268. No. Those tools observe and evaluate agents at runtime; Agent Reliability Analyzer catches defects in the code before it runs. Use them together.
  1269. **Does it send my code anywhere?**
  1270. No. Scanning is local and deterministic — no LLM, no code upload. By default it downloads the latest rules and vulnerability data; `--no-rules-update` runs fully offline from the local cache. `trustabl enrich` is opt-in and uses your own LLM key. Telemetry is off by default.
  1271. **Which agent frameworks does it support?**
  1272. Nine SDKs, including Claude Agent SDK, OpenAI Agents SDK, Google ADK, LangChain / LangGraph, CrewAI, AutoGen / AG2 and Pydantic AI. Full list: [COVERAGE.md](COVERAGE.md).
  1273. ## Part of Trustabl's open-source agent reliability tooling
  1274. Agent Reliability Analyzer is one piece of a set of open-source agent reliability tools from [Trustabl](https://www.trustabl.ai/):
  1275. - [agent-reliability-rules](https://github.com/trustabl/agent-reliability-rules) — the deterministic agent reliability detection rules the analyzer loads
  1276. - [trustabl-action](https://github.com/trustabl/trustabl-action) — run the analyzer as a GitHub Action
  1277. - [agent-opentelemetry-rules](https://github.com/trustabl/agent-opentelemetry-rules) — agent-specific OpenTelemetry rules for observability platforms
  1278. - [trustabl-aws](https://github.com/trustabl/trustabl-aws) — the analyzer for AWS CodePipeline and CodeCatalyst
  1279. The checks implement concepts from [Agent Reliability Engineering (ARE)](https://agentreliabilityengineering.org/), an open discipline for building reliable, governable AI agents. Trustabl is affiliated with the author of ARE.
  1280. ## Community
  1281. Join the [Trustabl Discord](https://discord.gg/G6gM8nKxPg) to ask questions, share feedback, and follow development.
  1282. ## License
  1283. Apache-2.0. See [`LICENSE`](LICENSE).
testdata/corpus/ToolBench/toolbench/inference/Downstream_tasks/rapidapi.py · Bundled script · 552 lines · 1 finding marked
  1. import re
  2. import os
  3. import json
  4. import time
  5. import requests
  6. from tqdm import tqdm
  7. from termcolor import colored
  8. import random
  9. from toolbench.inference.LLM.chatgpt_function_model import ChatGPTFunction
  10. from toolbench.inference.LLM.davinci_model import Davinci
  11. from toolbench.inference.LLM.tool_llama_lora_model import ToolLLaMALoRA
  12. from toolbench.inference.LLM.tool_llama_model import ToolLLaMA
  13. from toolbench.inference.LLM.retriever import ToolRetriever
  14. from toolbench.inference.Algorithms.single_chain import single_chain
  15. from toolbench.inference.Algorithms.DFS import DFS_tree_search
  16. from toolbench.inference.server import get_rapidapi_response
  17. from toolbench.utils import (
  18. standardize,
  19. change_name,
  20. replace_llama_with_condense
  21. )
  22. from toolbench.inference.Downstream_tasks.base_env import base_env
  23. # For pipeline environment preparation
  24. def get_white_list(tool_root_dir):
  25. # print(tool_root_dir)
  26. white_list_dir = os.path.join(tool_root_dir)
  27. white_list = {}
  28. for cate in tqdm(os.listdir(white_list_dir)):
  29. if not os.path.isdir(os.path.join(white_list_dir,cate)):
  30. continue
  31. for file in os.listdir(os.path.join(white_list_dir,cate)):
  32. if not file.endswith(".json"):
  33. continue
  34. standard_tool_name = file.split(".")[0]
  35. # print(standard_tool_name)
  36. with open(os.path.join(white_list_dir,cate,file)) as reader:
  37. js_data = json.load(reader)
  38. origin_tool_name = js_data["tool_name"]
  39. white_list[standardize(origin_tool_name)] = {"description": js_data["tool_description"], "standard_tool_name": standard_tool_name}
  40. return white_list
  41. def contain(candidate_list, white_list):
  42. output = []
  43. for cand in candidate_list:
  44. if cand not in white_list.keys():
  45. return False
  46. output.append(white_list[cand])
  47. return output
  48. # rapidapi env wrapper
  49. class rapidapi_wrapper(base_env):
  50. def __init__(self, query_json, tool_descriptions, retriever, args, process_id=0):
  51. super(rapidapi_wrapper).__init__()
  52. self.tool_root_dir = args.tool_root_dir
  53. self.toolbench_key = args.toolbench_key
  54. self.rapidapi_key = args.rapidapi_key
  55. self.use_rapidapi_key = args.use_rapidapi_key
  56. self.api_customization = args.api_customization
  57. self.service_url = "http://8.130.32.149:8080/rapidapi"
    • Note: Talks to a raw IP address or throwaway host, where malware is often staged
    • Line 64 on GitHub
  58. self.max_observation_length = args.max_observation_length
  59. self.observ_compress_method = args.observ_compress_method
  60. self.retriever = retriever
  61. self.process_id = process_id
  62. self.tool_names = []
  63. self.cate_names = []
  64. self.input_description = query_json["query"]
  65. self.functions = []
  66. self.api_name_reflect = {}
  67. if self.retriever is not None:
  68. query_json = self.retrieve_rapidapi_tools(self.input_description, args.retrieved_api_nums, args.tool_root_dir)
  69. data_dict = self.fetch_api_json(query_json)
  70. tool_descriptions = self.build_tool_description(data_dict)
  71. else:
  72. data_dict = self.fetch_api_json(query_json)
  73. for k,api_json in enumerate(data_dict["api_list"]):
  74. standard_tool_name = tool_descriptions[k][0]
  75. openai_function_json,cate_name, pure_api_name = self.api_json_to_openai_json(api_json,standard_tool_name)
  76. self.functions.append(openai_function_json)
  77. self.api_name_reflect[openai_function_json["name"]] = pure_api_name
  78. self.tool_names.append(standard_tool_name)
  79. self.cate_names.append(cate_name)
  80. finish_func = {
  81. "name": "Finish",
  82. "description": "If you believe that you have obtained a result that can answer the task, please call this function to provide the final answer. Alternatively, if you recognize that you are unable to proceed with the task in the current state, call this function to restart. Remember: you must ALWAYS call this function at the end of your attempt, and the only part that will be shown to the user is the final answer, so it should contain sufficient information.",
  83. "parameters": {
  84. "type": "object",
  85. "properties": {
  86. "return_type": {
  87. "type": "string",
  88. "enum": ["give_answer","give_up_and_restart"],
  89. },
  90. "final_answer": {
  91. "type": "string",
  92. "description": "The final answer you want to give the user. You should have this field if \"return_type\"==\"give_answer\"",
  93. }
  94. },
  95. "required": ["return_type"],
  96. }
  97. }
  98. self.functions.append(finish_func)
  99. self.CALL_MAX_TIME = 3
  100. self.task_description = f'''You should use functions to help handle the real time user querys. Remember:
  101. 1.ALWAYS call \"Finish\" function at the end of the task. And the final answer should contain enough information to show to the user,If you can't handle the task, or you find that function calls always fail(the function is not valid now), use function Finish->give_up_and_restart.
  102. 2.Do not use origin tool names, use only subfunctions' names.
  103. You have access of the following tools:\n'''
  104. unduplicated_reflection = {}
  105. for standardize_tool_name, tool_des in tool_descriptions:
  106. unduplicated_reflection[standardize_tool_name] = tool_des
  107. for k,(standardize_tool_name, tool_des) in enumerate(unduplicated_reflection.items()):
  108. striped = tool_des[:512].replace('\n','').strip()
  109. if striped == "":
  110. striped = "None"
  111. self.task_description += f"{k+1}.{standardize_tool_name}: {striped}\n"
  112. self.success = 0
  113. def build_tool_description(self, data_dict):
  114. white_list = get_white_list(self.tool_root_dir)
  115. origin_tool_names = [standardize(cont["tool_name"]) for cont in data_dict["api_list"]]
  116. tool_des = contain(origin_tool_names,white_list)
  117. tool_descriptions = [[cont["standard_tool_name"], cont["description"]] for cont in tool_des]
  118. return tool_descriptions
  119. def retrieve_rapidapi_tools(self, query, top_k, jsons_path):
  120. retrieved_tools = self.retriever.retrieving(query, top_k=top_k)
  121. query_json = {"api_list":[]}
  122. for tool_dict in retrieved_tools:
  123. if len(query_json["api_list"]) == top_k:
  124. break
  125. category = tool_dict["category"]
  126. tool_name = tool_dict["tool_name"]
  127. api_name = tool_dict["api_name"]
  128. if os.path.exists(jsons_path):
  129. if os.path.exists(os.path.join(jsons_path, category)):
  130. if os.path.exists(os.path.join(jsons_path, category, tool_name+".json")):
  131. query_json["api_list"].append({
  132. "category_name": category,
  133. "tool_name": tool_name,
  134. "api_name": api_name
  135. })
  136. return query_json
  137. def fetch_api_json(self, query_json):
  138. data_dict = {"api_list":[]}
  139. for item in query_json["api_list"]:
  140. cate_name = item["category_name"]
  141. tool_name = standardize(item["tool_name"])
  142. api_name = change_name(standardize(item["api_name"]))
  143. tool_json = json.load(open(os.path.join(self.tool_root_dir, cate_name, tool_name + ".json"), "r"))
  144. append_flag = False
  145. api_dict_names = []
  146. for api_dict in tool_json["api_list"]:
  147. api_dict_names.append(api_dict["name"])
  148. pure_api_name = change_name(standardize(api_dict["name"]))
  149. if pure_api_name != api_name:
  150. continue
  151. api_json = {}
  152. api_json["category_name"] = cate_name
  153. api_json["api_name"] = api_dict["name"]
  154. api_json["api_description"] = api_dict["description"]
  155. api_json["required_parameters"] = api_dict["required_parameters"]
  156. api_json["optional_parameters"] = api_dict["optional_parameters"]
  157. api_json["tool_name"] = tool_json["tool_name"]
  158. data_dict["api_list"].append(api_json)
  159. append_flag = True
  160. break
  161. if not append_flag:
  162. print(api_name, api_dict_names)
  163. return data_dict
  164. def api_json_to_openai_json(self, api_json,standard_tool_name):
  165. description_max_length=256
  166. templete = {
  167. "name": "",
  168. "description": "",
  169. "parameters": {
  170. "type": "object",
  171. "properties": {
  172. },
  173. "required": [],
  174. "optional": [],
  175. }
  176. }
  177. map_type = {
  178. "NUMBER": "integer",
  179. "STRING": "string",
  180. "BOOLEAN": "boolean"
  181. }
  182. pure_api_name = change_name(standardize(api_json["api_name"]))
  183. templete["name"] = pure_api_name+ f"_for_{standard_tool_name}"
  184. templete["name"] = templete["name"][-64:]
  185. templete["description"] = f"This is the subfunction for tool \"{standard_tool_name}\", you can use this tool."
  186. if api_json["api_description"].strip() != "":
  187. tuncated_description = api_json['api_description'].strip().replace(api_json['api_name'],templete['name'])[:description_max_length]
  188. templete["description"] = templete["description"] + f"The description of this function is: \"{tuncated_description}\""
  189. if "required_parameters" in api_json.keys() and len(api_json["required_parameters"]) > 0:
  190. for para in api_json["required_parameters"]:
  191. name = standardize(para["name"])
  192. name = change_name(name)
  193. if para["type"] in map_type:
  194. param_type = map_type[para["type"]]
  195. else:
  196. param_type = "string"
  197. prompt = {
  198. "type":param_type,
  199. "description":para["description"][:description_max_length],
  200. }
  201. default_value = para['default']
  202. if len(str(default_value)) != 0:
  203. prompt = {
  204. "type":param_type,
  205. "description":para["description"][:description_max_length],
  206. "example_value": default_value
  207. }
  208. else:
  209. prompt = {
  210. "type":param_type,
  211. "description":para["description"][:description_max_length]
  212. }
  213. templete["parameters"]["properties"][name] = prompt
  214. templete["parameters"]["required"].append(name)
  215. for para in api_json["optional_parameters"]:
  216. name = standardize(para["name"])
  217. name = change_name(name)
  218. if para["type"] in map_type:
  219. param_type = map_type[para["type"]]
  220. else:
  221. param_type = "string"
  222. default_value = para['default']
  223. if len(str(default_value)) != 0:
  224. prompt = {
  225. "type":param_type,
  226. "description":para["description"][:description_max_length],
  227. "example_value": default_value
  228. }
  229. else:
  230. prompt = {
  231. "type":param_type,
  232. "description":para["description"][:description_max_length]
  233. }
  234. templete["parameters"]["properties"][name] = prompt
  235. templete["parameters"]["optional"].append(name)
  236. return templete, api_json["category_name"], pure_api_name
  237. def check_success(self):
  238. return self.success
  239. def to_json(self):
  240. return {}
  241. def restart(self):
  242. pass
  243. def get_score(self):
  244. return 0.0
  245. def step(self,**args):
  246. obs, code = self._step(**args)
  247. if len(obs) > self.max_observation_length:
  248. obs = obs[:self.max_observation_length] + "..."
  249. return obs, code
  250. def _step(self, action_name="", action_input=""):
  251. """Need to return an observation string and status code:
  252. 0 means normal response
  253. 1 means there is no corresponding api name
  254. 2 means there is an error in the input
  255. 3 represents the end of the generation and the final answer appears
  256. 4 means that the model decides to pruning by itself
  257. 5 represents api call timeout
  258. 6 for 404
  259. 7 means not subscribed
  260. 8 represents unauthorized
  261. 9 represents too many requests
  262. 10 stands for rate limit
  263. 11 message contains "error" field
  264. 12 error sending request
  265. """
  266. if action_name == "Finish":
  267. try:
  268. json_data = json.loads(action_input,strict=False)
  269. except:
  270. json_data = {}
  271. if '"return_type": "' in action_input:
  272. if '"return_type": "give_answer"' in action_input:
  273. return_type = "give_answer"
  274. elif '"return_type": "give_up_and_restart"' in action_input:
  275. return_type = "give_up_and_restart"
  276. else:
  277. return_type = action_input[action_input.find('"return_type": "')+len('"return_type": "'):action_input.find('",')]
  278. json_data["return_type"] = return_type
  279. if '"final_answer": "' in action_input:
  280. final_answer = action_input[action_input.find('"final_answer": "')+len('"final_answer": "'):]
  281. json_data["final_answer"] = final_answer
  282. if "return_type" not in json_data.keys():
  283. return "{error:\"must have \"return_type\"\"}", 2
  284. if json_data["return_type"] == "give_up_and_restart":
  285. return "{\"response\":\"chose to give up and restart\"}",4
  286. elif json_data["return_type"] == "give_answer":
  287. if "final_answer" not in json_data.keys():
  288. return "{error:\"must have \"final_answer\"\"}", 2
  289. self.success = 1 # succesfully return final_answer
  290. return "{\"response\":\"successfully giving the final answer.\"}", 3
  291. else:
  292. return "{error:\"\"return_type\" is not a valid choice\"}", 2
  293. else:
  294. for k, function in enumerate(self.functions):
  295. if function["name"].endswith(action_name):
  296. pure_api_name = self.api_name_reflect[function["name"]]
  297. payload = {
  298. "category": self.cate_names[k],
  299. "tool_name": self.tool_names[k],
  300. "api_name": pure_api_name,
  301. "tool_input": action_input,
  302. "strip": self.observ_compress_method,
  303. "toolbench_key": self.toolbench_key
  304. }
  305. if self.process_id == 0:
  306. print(colored(f"query to {self.cate_names[k]}-->{self.tool_names[k]}-->{action_name}",color="yellow"))
  307. if self.use_rapidapi_key or self.api_customization:
  308. payload["rapidapi_key"] = self.rapidapi_key
  309. response = get_rapidapi_response(payload, api_customization=self.api_customization)
  310. else:
  311. time.sleep(2) # rate limit: 30 per minute
  312. headers = {"toolbench_key": self.toolbench_key}
  313. response = requests.post(self.service_url, json=payload, headers=headers, timeout=15)
  314. if response.status_code != 200:
  315. return json.dumps({"error": f"request invalid, data error. status_code={response.status_code}", "response": ""}), 12
  316. try:
  317. response = response.json()
  318. except:
  319. print(response)
  320. return json.dumps({"error": f"request invalid, data error", "response": ""}), 12
  321. # 1 Hallucinating function names
  322. # 4 means that the model decides to pruning by itself
  323. # 5 represents api call timeout
  324. # 6 for 404
  325. # 7 means not subscribed
  326. # 8 represents unauthorized
  327. # 9 represents too many requests
  328. # 10 stands for rate limit
  329. # 11 message contains "error" field
  330. # 12 error sending request
  331. if response["error"] == "API not working error...":
  332. status_code = 6
  333. elif response["error"] == "Unauthorized error...":
  334. status_code = 7
  335. elif response["error"] == "Unsubscribed error...":
  336. status_code = 8
  337. elif response["error"] == "Too many requests error...":
  338. status_code = 9
  339. elif response["error"] == "Rate limit per minute error...":
  340. print("Reach api calling limit per minute, sleeping...")
  341. time.sleep(10)
  342. status_code = 10
  343. elif response["error"] == "Message error...":
  344. status_code = 11
  345. else:
  346. status_code = 0
  347. return json.dumps(response), status_code
  348. # except Exception as e:
  349. # return json.dumps({"error": f"Timeout error...{e}", "response": ""}), 5
  350. return json.dumps({"error": f"No such function name: {action_name}", "response": ""}), 1
  351. class pipeline_runner:
  352. def __init__(self, args, add_retrieval=False, process_id=0, server=False):
  353. self.args = args
  354. self.add_retrieval = add_retrieval
  355. self.process_id = process_id
  356. self.server = server
  357. if not self.server: self.task_list = self.generate_task_list()
  358. else: self.task_list = []
  359. def get_backbone_model(self):
  360. args = self.args
  361. if args.backbone_model == "toolllama":
  362. # ratio = 4 means the sequence length is expanded by 4, remember to change the model_max_length to 8192 (2048 * ratio) for ratio = 4
  363. ratio = int(args.max_sequence_length/args.max_source_sequence_length)
  364. replace_llama_with_condense(ratio=ratio)
  365. if args.lora:
  366. backbone_model = ToolLLaMALoRA(base_name_or_path=args.model_path, model_name_or_path=args.lora_path, max_sequence_length=args.max_sequence_length)
  367. else:
  368. backbone_model = ToolLLaMA(model_name_or_path=args.model_path, max_sequence_length=args.max_sequence_length)
  369. else:
  370. backbone_model = args.backbone_model
  371. return backbone_model
  372. def get_retriever(self):
  373. return ToolRetriever(corpus_tsv_path=self.args.corpus_tsv_path, model_path=self.args.retrieval_model_path)
  374. def get_args(self):
  375. return self.args
  376. def generate_task_list(self):
  377. args = self.args
  378. query_dir = args.input_query_file
  379. answer_dir = args.output_answer_file
  380. if not os.path.exists(answer_dir):
  381. os.mkdir(answer_dir)
  382. method = args.method
  383. backbone_model = self.get_backbone_model()
  384. white_list = get_white_list(args.tool_root_dir)
  385. task_list = []
  386. querys = json.load(open(query_dir, "r"))
  387. for query_id, data_dict in enumerate(querys):
  388. if "query_id" in data_dict:
  389. query_id = data_dict["query_id"]
  390. if "api_list" in data_dict:
  391. origin_tool_names = [standardize(cont["tool_name"]) for cont in data_dict["api_list"]]
  392. tool_des = contain(origin_tool_names,white_list)
  393. if tool_des == False:
  394. continue
  395. tool_des = [[cont["standard_tool_name"], cont["description"]] for cont in tool_des]
  396. else:
  397. tool_des = None
  398. task_list.append((method, backbone_model, query_id, data_dict, args, answer_dir, tool_des))
  399. return task_list
  400. def method_converter(self, backbone_model, openai_key, method, env, process_id, single_chain_max_step=12, max_query_count=60, callbacks=None):
  401. if callbacks is None: callbacks = []
  402. if backbone_model == "chatgpt_function":
  403. model = "gpt-3.5-turbo-16k-0613"
  404. llm_forward = ChatGPTFunction(model=model, openai_key=openai_key)
  405. elif backbone_model == "davinci":
  406. model = "text-davinci-003"
  407. llm_forward = Davinci(model=model, openai_key=openai_key)
  408. else:
  409. model = backbone_model
  410. llm_forward = model
  411. if method.startswith("CoT"):
  412. passat = int(method.split("@")[-1])
  413. chain = single_chain(llm=llm_forward, io_func=env,process_id=process_id)
  414. result = chain.start(
  415. pass_at=passat,
  416. single_chain_max_step=single_chain_max_step,
  417. answer=1)
  418. elif method.startswith("DFS"):
  419. pattern = r".+_w(\d+)"
  420. re_result = re.match(pattern,method)
  421. assert re_result != None
  422. width = int(re_result.group(1))
  423. with_filter = True
  424. if "woFilter" in method:
  425. with_filter = False
  426. chain = DFS_tree_search(llm=llm_forward, io_func=env,process_id=process_id, callbacks=callbacks)
  427. result = chain.start(
  428. single_chain_max_step=single_chain_max_step,
  429. tree_beam_size = width,
  430. max_query_count = max_query_count,
  431. answer=1,
  432. with_filter=with_filter)
  433. else:
  434. print("invalid method")
  435. raise NotImplementedError
  436. return chain, result
  437. def run_single_task(self, method, backbone_model, query_id, data_dict, args, output_dir_path, tool_des, retriever=None, process_id=0, callbacks=None, server= None):
  438. if server is None:
  439. server = self.server
  440. if callbacks is None:
  441. if server: print("Warning: no callbacks are defined for server mode")
  442. callbacks = []
  443. splits = output_dir_path.split("/")
  444. os.makedirs("/".join(splits[:-1]),exist_ok=True)
  445. os.makedirs("/".join(splits),exist_ok=True)
  446. output_file_path = os.path.join(output_dir_path,f"{query_id}_{method}.json")
  447. if (not server) and os.path.exists(output_file_path):
  448. return
  449. [callback.on_tool_retrieval_start() for callback in callbacks]
  450. env = rapidapi_wrapper(data_dict, tool_des, retriever, args, process_id=process_id)
  451. [callback.on_tool_retrieval_end(
  452. tools=env.functions
  453. ) for callback in callbacks]
  454. query = data_dict["query"]
  455. if process_id == 0:
  456. print(colored(f"[process({process_id})]now playing {query}, with {len(env.functions)} APIs", "green"))
  457. [callback.on_request_start(
  458. user_input=query,
  459. method=method,
  460. ) for callback in callbacks]
  461. chain,result = self.method_converter(
  462. backbone_model=backbone_model,
  463. openai_key=args.openai_key,
  464. method=method,
  465. env=env,
  466. process_id=process_id,
  467. single_chain_max_step=12,
  468. max_query_count=200,
  469. callbacks=callbacks
  470. )
  471. [callback.on_request_end(
  472. chain=chain.terminal_node[0].messages,
  473. outputs=chain.terminal_node[0].description,
  474. ) for callback in callbacks]
  475. if output_dir_path is not None:
  476. with open(output_file_path,"w") as writer:
  477. data = chain.to_json(answer=True,process=True)
  478. data["answer_generation"]["query"] = query
  479. json.dump(data, writer, indent=2)
  480. success = data["answer_generation"]["valid_data"] and "give_answer" in data["answer_generation"]["final_answer"]
  481. print(colored(f"[process({process_id})]valid={success}", "green"))
  482. return result
  483. def run(self):
  484. task_list = self.task_list
  485. random.seed(42)
  486. random.shuffle(task_list)
  487. print(f"total tasks: {len(task_list)}")
  488. new_task_list = []
  489. for task in task_list:
  490. out_dir_path = task[-2]
  491. query_id = task[2]
  492. output_file_path = os.path.join(out_dir_path,f"{query_id}_{self.args.method}.json")
  493. if not os.path.exists(output_file_path):
  494. new_task_list.append(task)
  495. task_list = new_task_list
  496. print(f"undo tasks: {len(task_list)}")
  497. if self.add_retrieval:
  498. retriever = self.get_retriever()
  499. else:
  500. retriever = None
  501. for k, task in enumerate(task_list):
  502. print(f"process[{self.process_id}] doing task {k}/{len(task_list)}: real_task_id_{task[2]}")
  503. result = self.run_single_task(*task, retriever=retriever, process_id=self.process_id)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.