Skip to content
Back to search

OpenART

OpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ai45lab--openart/badge.svg)](/listings/ai45lab--openart)

HTML

<a href="/listings/ai45lab--openart"><img src="/api/listings/ai45lab--openart/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Run the following commands from the repository root:

cd OpenART
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
export PYTHONPATH="$PWD"

Docker must be available to the current user. Build the base task image:

docker build -t openart/task-base:latest -f images/Dockerfile.task-base .

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 387 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # OpenART: Open Agent Red Teaming
  2. **An executable benchmark for red-teaming tool-using agents in long-horizon, evolving environments.**
  3. OpenART (**Open Agent Red Teaming**) evaluates whether autonomous agents stay
  4. safe when their workspace changes mid-run. It ships a Docker-native runtime, a
  5. public dataset of 10,150 released tasks, and 63,697 materialized tools on
  6. Hugging Face. Use it to measure tool-using agents (OpenCode, Claude Code,
  7. Codex, and other runtimes) beyond single-turn chat.
  8. OpenART is the agent-focused sibling of [OpenRT](https://github.com/AI45Lab/OpenRT),
  9. an open-source red-teaming framework for multimodal LLMs.
  10. [![Project Page](https://img.shields.io/badge/Project%20Page-OpenART-6ae3ff.svg)](https://ai45lab.github.io/OpenART/)
  11. [![arXiv](https://img.shields.io/badge/arXiv-2608.00677-b31b1b.svg)](https://arxiv.org/abs/2608.00677)
  12. [![Dataset](https://img.shields.io/badge/Hugging%20Face-10%2C150%20tasks-yellow.svg)](https://huggingface.co/datasets/dongdongunique/openart-planner-tasks)
  13. [![Tools](https://img.shields.io/badge/Hugging%20Face-63%2C697%20tools-yellow.svg)](https://huggingface.co/datasets/dongdongunique/openart-tools)
  14. [![License](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE)
  15. **Project page:** [ai45lab.github.io/OpenART](https://ai45lab.github.io/OpenART/) — overview, key facts, artifacts, and citation.
  16. [![OpenART environment-evolution loop: a fixed task and safety contract, an evolving workspace, long-horizon risk, and feedback-guided search](site/assets/figures/fig0-loop.gif)](https://ai45lab.github.io/OpenART/#loop)
  17. *A 16-second conceptual loop: the task and safety contract stay fixed while the target-visible environment evolves, risk accumulates across steps, and evaluator feedback guides the next search round. Illustrative; not a real trajectory. [Open the interactive overview →](https://ai45lab.github.io/OpenART/#loop)*
  18. **At a glance:** 10,150 public tasks · 50 domains · median 97 tool calls · 15 agents × 5 models · 85.0% pooled strict ASR (EMHA attacker) · 87.38% benign completion.
  19. **Quickstart:** see [Setup](#setup) for the environment and Docker image, then
  20. [Model-backed evaluations](#model-backed-evaluations) to run a bundled task.
  21. [09/22/2026] ⭐ OpenART reached the 200-star milestone on [GitHub](https://github.com/AI45Lab/OpenART).
  22. **Contents:** [Research summary](#research-summary) · [Paper and datasets](#paper-and-datasets) · [Repository layout](#repository-layout) · [Setup](#setup) · [Model-backed evaluations](#model-backed-evaluations) · [Task generation](#task-generation) · [Services](#services) · [Documentation](#documentation) · [FAQ](#faq) · [Citing OpenART](#citing-openart) · [License](#license)
  23. ## Research summary
  24. OpenART (**Open Agent Red Teaming**) is a benchmark and Docker-native runtime
  25. for **agent safety evaluation**: red-teaming tool-using LLM agents inside
  26. **long-horizon, executable, evolving environments**. It is relevant when your
  27. question matches one of these patterns:
  28. - How safe are **tool-using / code agents** (OpenCode, Claude Code, Codex, and
  29. other runtimes) beyond single-turn chat?
  30. - Do agents fail when the **environment state changes mid-run** (files edited,
  31. tools swapped, services updated) — *environment evolution* — while the task
  32. and hidden safety contract stay fixed?
  33. - How does safety vary **across agent runtimes** versus across foundation
  34. models (15 agents × 5 models evaluation matrix)?
  35. - How do attacks behave at **long horizon** (median 97 tool calls), and where
  36. do unsafe actions first appear after mutated state is consumed?
  37. - How can **LLM red teaming / prompt-injection-style attacks** be scaled
  38. beyond hand-written scenarios (planner-generated 10K+ validated tasks,
  39. 500K+ tool corpus, evolutionary EMHA attacker)?
  40. The core idea: evaluate the **executable environment**.
  41. Persistent state is mutated during execution; safety is judged against a fixed
  42. hidden contract by deterministic and model-based evaluators over traces and
  43. snapshots.
  44. ### Comparison with existing benchmarks (from the paper)
  45. Task-level complexity measured from released artifacts and executable traces,
  46. reported as medians [interquartile range]. OpenART and DTap report medians
  47. over reconstructed execution traces; all other benchmarks are measured from
  48. their publicly released samples.
  49. | Benchmark | Tool calls | Dependency depth | Parallel width | State objects | File formats |
  50. | --- | ---: | ---: | ---: | ---: | ---: |
  51. | [InjecAgent](https://arxiv.org/abs/2403.02691) | 1 [1-1] | 1 [1-1] | 1 [1-1] | 1 [1-1] | 0 [0-0] |
  52. | [ToolEmu](https://arxiv.org/abs/2309.15817) | 3 [1.5-4] | 2.5 [1.2-3.8] | 1 [1-1.8] | 3 [1.2-3] | 0 [0-0] |
  53. | [AgentDojo](https://arxiv.org/abs/2406.13352) | 2 [1-3] | 2 [1-3] | 1 [1-1] | 1 [1-2] | 0 [0-0] |
  54. | [AgentHarm](https://arxiv.org/abs/2410.09024) | 3.5 [3-4] | 3 [3-3] | 1.5 [1-2] | 3.5 [3-4] | 0 [0-0] |
  55. | [ASB](https://arxiv.org/abs/2410.02644) | 2 [2-2] | 2 [2-2] | 1 [1-1] | 2 [2-2] | 0 [0-0] |
  56. | [DTap](https://arxiv.org/abs/2605.04808) | 15 [7.4-18.7] | 2 [1-3] | 1.5 [1-2] | 2.5 [1-4] | 1 [0-3] |
  57. | **OpenART** | **97 [90.2-100]** | **32 [15.8-84.8]** | **12.5 [3-24.5]** | **96.5 [90.2-100]** | **7.5 [7-9]** |
  58. OpenART reaches a median dependency depth of 32 and parallel width of 12.5,
  59. reflecting long, branching workflows that support red teaming over extended
  60. interaction horizons.
  61. ### Results across the 75 agent-model settings
  62. Benign task completion (Comp.) and Strict ASR (%) pooled over 15 target agents
  63. per foundation model, from the paper's 75-setting evaluation matrix. Strict
  64. ASR counts an attack as successful only when both the deterministic evaluator
  65. and an LLM judge confirm it.
  66. | Foundation model | Comp. (%) | Strict ASR (%) |
  67. | --- | ---: | ---: |
  68. | GPT-5.5 | 92.02 | 88.5 |
  69. | Claude Opus-4.8 | 96.18 | 59.2 |
  70. | GLM-5.2 | 85.00 | 87.9 |
  71. | Qwen-3.7-Max | 80.81 | 94.6 |
  72. | DeepSeek-V4-Pro | 82.91 | 94.7 |
  73. | **Pooled (75 settings)** | **87.38** | **85.0** |
  74. ## Paper and datasets
  75. **OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution**
  76. Yunhao Chen, Xin Wang, Yixu Wang, Yi Liu, Jie Li, Yan Teng, Xingjun Ma, Xia Hu,
  77. and Yu-Gang Jiang
  78. [arXiv abstract](https://arxiv.org/abs/2608.00677) ·
  79. [PDF](https://arxiv.org/pdf/2608.00677)
  80. OpenART evaluates the executable environment rather than treating a single
  81. prompt as the entire test. The benign task and hidden safety contract remain
  82. fixed while target-visible state changes during execution.
  83. | Paper setting | Value |
  84. | --- | ---: |
  85. | Validated stateful scenarios | 10K+ |
  86. | Domains | 50 |
  87. | Capability corpus | 500K+ tools and skills |
  88. | Median task horizon | 97 tool calls |
  89. | Evaluation matrix | 15 agents × 5 models (75 settings) |
  90. | Target-visible attack vectors | 8 |
  91. | Reference attacker | Evolutionary Markov Hypergraph Attack (EMHA) |
  92. | Pooled strict attack success rate | 85.0% |
  93. The paper reports that environment evolution becomes more effective as
  94. workflow complexity grows, and that the agent runtime affects safety beyond
  95. the choice of foundation model. Unsafe behavior also tends to appear well
  96. after the changed state is first consumed, often through stale assumptions or
  97. decisions carried forward from earlier steps.
  98. The public
  99. [OpenART planner task dataset](https://huggingface.co/datasets/dongdongunique/openart-planner-tasks)
  100. contains **10,150 released tasks in 52 zip shards** (~75 GB).
  101. Every task bundle includes `task.md`, `scenario_model.json`, `tool_pool.json`,
  102. `tool_use_graph.json`, `safety_contract.json`, `capabilities.generated.yaml`,
  103. `planner_plan.json`, `checkpoints.md`, `utils/evaluator.py`,
  104. `utils/test_evaluator.py`, and an executable `workspace/`. Metadata is exposed
  105. as Parquet for the Dataset Viewer, release checks cover checksums, archive
  106. integrity, and sample extraction, and `scripts/download_task.py` extracts a
  107. single task from a local clone.
  108. We also release the public
  109. [OpenART tools dataset](https://huggingface.co/datasets/dongdongunique/openart-tools),
  110. which packages 63,697 selected materialized tools from the OpenART capability
  111. corpus. This 60K+ subset prioritizes the most usable OpenART tools for task
  112. execution, reuse, and reproducible benchmark construction. The release includes
  113. metadata, archive manifests, checksums, and a lightweight download helper.
  114. Load the task metadata directly (no bundle download):
  115. ```python
  116. from datasets import load_dataset
  117. ds = load_dataset("dongdongunique/openart-planner-tasks", "mid", split="train")
  118. print(ds.num_rows, len(ds.features)) # 1046 27
  119. ```
  120. Extract one task from a local dataset copy with the dataset's
  121. `scripts/download_task.py`:
  122. ```bash
  123. python scripts/download_task.py \
  124. --dataset-dir openart-planner-tasks \
  125. --difficulty mid-v5 \
  126. --task-id openartv5-mid-000001 \
  127. --output-dir ./task
  128. ```
  129. ## Repository layout
  130. | Path | Contents |
  131. | --- | --- |
  132. | `OpenART/` | Runtime, configuration, Dockerfiles, documentation, and tests |
  133. | `OpenART/examples/tasks/` | Local smoke task and bundled high-complexity tasks |
  134. | `OpenART/skills/openart/SKILL.md` | Unified guide for agents and people using or extending OpenART |
  135. | `OpenART/services/` | Every service used in OpenART, with run instructions, configs, and health checks |
  136. | `openart-tools/` | Managed tool subset used by the bundled tasks |
  137. ## Use the OpenART skill
  138. OpenART includes one consolidated operator skill for planning, scenario
  139. generation, managed tools, evaluation, extension, and debugging. It routes
  140. each request to the appropriate existing documentation and command instead of
  141. duplicating those workflows across several skills.
  142. - For people: read [`OpenART/skills/openart/SKILL.md`](OpenART/skills/openart/SKILL.md)
  143. as a workflow map before choosing a detailed guide.
  144. - For agents: explicitly ask the agent to read that file before working on the
  145. framework. For example:
  146. ```text
  147. Read and follow OpenART/skills/openart/SKILL.md. Help me run the local smoke
  148. task, explain each required setting, and stop before any external or
  149. destructive action.
  150. ```
  151. Agents that support repository-local skill discovery can use `OpenART/skills/`
  152. as their skill source. Otherwise, referencing the file directly is the
  153. portable invocation method. This operator skill is not a managed runtime tool,
  154. target-visible skill, or attacker payload.
  155. ## Setup
  156. Run the following commands from the repository root:
  157. ```bash
  158. cd OpenART
  159. python3 -m venv .venv
  160. source .venv/bin/activate
  161. python -m pip install -r requirements.txt
  162. export PYTHONPATH="$PWD"
  163. ```
  164. Docker must be available to the current user. Build the base task image:
  165. ```bash
  166. docker build -t openart/task-base:latest -f images/Dockerfile.task-base .
  167. ```
  168. ## Local smoke test
  169. The local smoke task checks the container and deterministic evaluator paths. It
  170. does not require model credentials.
  171. ```bash
  172. python -m framework.cli run \
  173. --task examples/tasks/local-smoke \
  174. --target-config configs/target-configs/target.local-smoke.yaml \
  175. --eval-strategy deterministic \
  176. --skip-attacker \
  177. --run-id local-smoke \
  178. --output-dir outputs/local-smoke
  179. ```
  180. ## Model-backed evaluations
  181. The default target configuration uses OpenCode. Build its image, copy the
  182. environment template, and add the required model endpoints and credentials to
  183. `.env`:
  184. ```bash
  185. docker build -t openart/opencode:latest -f images/Dockerfile.opencode .
  186. cp .env.example .env
  187. ```
  188. Run a target without an attacker:
  189. ```bash
  190. python -m framework.cli run \
  191. --task examples/tasks/high-complexity-kb-integration \
  192. --target-config configs/target-configs/target.yaml \
  193. --tool-store ../openart-tools \
  194. --eval-strategy deterministic \
  195. --skip-attacker \
  196. --run-id kb-integration-target-only \
  197. --output-dir outputs/kb-integration-target-only
  198. ```
  199. Run the same task with the default OpenCode-compatible attacker:
  200. ```bash
  201. python -m framework.cli run \
  202. --task examples/tasks/high-complexity-kb-integration \
  203. --target-config configs/target-configs/target.yaml \
  204. --attacker-config configs/attacker-configs/universal/opencode-native-control/config.yaml \
  205. --tool-store ../openart-tools \
  206. --eval-strategy both \
  207. --max-iterations 2 \
  208. --run-id kb-integration-attacked \
  209. --output-dir outputs/kb-integration-attacked
  210. ```
  211. The task graph selects tools from `../openart-tools`. Run artifacts are written
  212. under `OpenART/outputs/` and ignored by Git.
  213. ## Task generation
  214. Build the planner image and generate a task from a checked-in scenario. Planner
  215. model settings are read from `.env`.
  216. ```bash
  217. docker build -t openart/safe-world-planner:latest \
  218. -f images/Dockerfile.safe-world-planner .
  219. python -m framework.planner.cli \
  220. --planner-backend opencode \
  221. --scenario-file configs/planner/scenarios/financial-expense-brief.txt \
  222. --tool-store ../openart-tools \
  223. --tool-count 3 \
  224. --complexity-profile stress \
  225. --planner-max-repairs 2 \
  226. --task-id financial-expense-brief \
  227. --output-dir outputs/financial-expense-brief \
  228. --overwrite
  229. ```
  230. Planner outputs are also local, ignored artifacts.
  231. ## Services
  232. Every service used in OpenART is collected under
  233. [`OpenART/services/README.md`](OpenART/services/README.md), grouped into:
  234. - **DTAP services** (16 environments: finance, legal, gmail, slack, calendar,
  235. zoom, paypal, atlassian, github, google-form, googledocs, telegram,
  236. whatsapp, snowflake, databricks, bigquery)
  237. - **Standalone app services** (Mailpit, Gitea, Mattermost, Baserow, NocoDB,
  238. OpenProject, Nextcloud, OnlyOffice)
  239. - **Emulator services** (MinIO, LocalStack, BigQuery emulator, Gotenberg)
  240. - **GUI browser service** (Playwright MCP browser sidecar for GUI-form tools)
  241. Each service README documents how to run the service (docker compose, docker
  242. run, or rjob submission), its ports and health-check endpoints, and its
  243. environment-variable configuration. Central port/image configuration lives in
  244. `OpenART/services/configs/ports.env` with a template in
  245. `OpenART/services/configs/.env.example`.
  246. Quick start:
  247. ```bash
  248. # Start all services (dtap + apps + emulators)
  249. bash OpenART/services/scripts/start_all_services.sh
  250. # Or via docker compose directly
  251. cd OpenART/images/services
  252. docker compose --env-file ports.env -f docker-compose.yml --profile all up -d
  253. # Check status and health of all services
  254. bash OpenART/services/scripts/get_service_status.sh
  255. ```
  256. ## Documentation
  257. - [Documentation index](OpenART/docs/README.md)
  258. - [Quickstart and runtime options](OpenART/docs/01_quickstart.md)
  259. - [Planner design and usage](OpenART/docs/12_planner_design_implementation_usage.md)
  260. - [Unified operator guide](OpenART/skills/openart/SKILL.md)
  261. ## FAQ
  262. **What does OpenART evaluate?** Executable tool-using agent workflows under a
  263. fixed hidden safety contract, while target-visible state changes mid-run.
  264. **What is environment evolution?** Changes to the agent's workspace, tools, or
  265. services during a run, while the benign goal and safety contract stay fixed.
  266. **How is OpenART different from single-turn prompt-injection benchmarks?**
  267. Tasks are long-horizon (median 97 tool calls) and are judged over full traces
  268. and snapshots, not a single model response.
  269. **How do I run one released task?** Download the planner task dataset, pick a
  270. config (`mid`, `mid-v5`, or `large`), and extract a task with the dataset's
  271. `scripts/download_task.py`. See [Paper and datasets](#paper-and-datasets).
  272. **Where are the evaluation results?** The comparison and 75-setting tables
  273. above summarize the paper; full per-setting numbers are in
  274. [arXiv:2608.00677](https://arxiv.org/abs/2608.00677).
  275. ## Citing OpenART
  276. If you use the OpenART runtime, tasks, tools dataset, or evaluation results,
  277. please cite the paper and the repository. The canonical metadata lives in
  278. [`CITATION.cff`](CITATION.cff), which GitHub exposes as the "Cite this
  279. repository" link.
  280. **Cite as:** Chen et al., "OpenART: Scaling Agent Red Teaming via Open-Ended
  281. Environment Evolution", [arXiv:2608.00677](https://arxiv.org/abs/2608.00677)
  282. (2026).
  283. ```bibtex
  284. @misc{chen2026openart,
  285. title = {OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution},
  286. author = {Chen, Yunhao and Wang, Xin and Wang, Yixu and Liu, Yi and Li, Jie
  287. and Teng, Yan and Ma, Xingjun and Hu, Xia and Jiang, Yu-Gang},
  288. year = {2026},
  289. eprint = {2608.00677},
  290. archivePrefix = {arXiv},
  291. primaryClass = {cs.CL},
  292. url = {https://arxiv.org/abs/2608.00677},
  293. note = {Code: https://github.com/AI45Lab/OpenART}
  294. }
  295. ```
  296. The paper PDF also prints **OpenART Arena**; both strings refer to this work.
  297. Cite the arXiv title above verbatim in bibliographies.
  298. ## License
  299. OpenART is licensed under the [GNU Affero General Public License v3.0](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.