Skip to content
Back to search

agentic-harness-engineering

Official AHE code — Agentic Harness Engineering: observability-driven automatic evolution of coding-agent harnesses (concurrent w/ meta-harness). NexAU-AHE reaches 84.7% ± 2.1 pass@1 on Terminal-Bench 2 (GPT-5.5). Lifts GPT-5.4 69.7→77.0% over 10 iters, beats Codex/ACE/Training-Free GRPO; frozen harness transfers to SWE-bench-Verified.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/china-qijizhifeng--agentic-harness-engineering/badge.svg)](/listings/china-qijizhifeng--agentic-harness-engineering)

HTML

<a href="/listings/china-qijizhifeng--agentic-harness-engineering"><img src="/api/listings/china-qijizhifeng--agentic-harness-engineering/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

0. Prerequisites

- Python ≥ 3.13 - [uv](https://docs.astral.sh/uv/) - tmux

# macOS
brew install uv tmux

# Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
sudo apt install -y tmux

1. Clone + install dependencies

git clone https://github.com/Curry09/agentic-harness-engineering.git
cd agentic-harness-engineering
uv sync

> `uv sync` installs every dependency declared in `pyproject.toml`.

2. Configure environment variables

cp .env.example .env

Edit `.env`. At minimum, set:

| Variable | Purpose | |---|---| | `LLM_API_KEY` / `LLM_BASE_URL` | Main LLM endpoint (`code_agent` and `evolve_agent` both consume it) | | `E2B_API_KEY` | [E2B](https://e2b.dev/) sandbox — see the next subsection for SaaS vs. self-hosted | | `SERPER_API_KEY` | Web search used by `evolve_agent` |

`ADB_LLM_*` and `GPT54_LLM_*` are optional — leave them unset to fall back to `LLM_*`, or set them to point ADB / the gpt-5.4 experiment at a stronger model. `LANGFUSE_*`, `BP_HTML_PARSER_*`, and `FEISHU_WEBHOOK` are all optional observability / convenience hooks; see `.env.example` for the full list.

E2B sandbox: SaaS vs. self-hosted

AHE runs every rollout inside an E2B sandbox. Two deployment modes are supported:

- **SaaS E2B (default).** Set **only** `E2B_API_KEY` and leave `E2B_API_URL` / `E2B_DOMAIN` unset (or commented out). The SDK talks to `e2b.dev` automatically.

> ⚠️ **Concurrency cap.** SaaS E2B enforces a per-account **concurrent sandbox limit** tied to your tier. If harbor tries to spawn more sandboxes than the cap allows, the extra sandboxes fail to start and the iteration stalls. Before raising parallelism in your harbor / experiment config, check your tier's quota and stay safely under it.

- **Self-hosted E2B cluster.** Set `E2B_API_KEY` **and** point the SDK at your cluster:

  E2B_API_KEY="your_e2b_key"
  E2B_API_URL="https://your-e2b-host.example.com"
  E2B_DOMAIN="your-e2b-host.example.com"

No shared concurrency cap applies, but the cluster's hardware capacity still does.

3. Build E2B templates (one-time per dataset)

The dataset here is a pack from [`laude-institute/harbor-datasets`](https://github.com/laude-institute/harbor-datasets) — clone the subset you need and point `--dataset-dir` at its directory.

Every rollout runs inside an E2B sandbox spawned from a prebuilt template that already has `uv` and the NexAU/harbor venv at `/opt/nexau-venv`. Build those templates once before launching:

# Build every template declared by the dataset, 16 in parallel
uv run python scripts/build_templates.py --dataset-dir /path/to/dataset -j 16

# Resume after a failure: only retry tasks whose latest E2B build status is ERROR
uv run python scripts/build_templates.py --dataset-dir /path/to/dataset --retry-failed

# Build a specific subset of tasks
uv run python scripts/build_templates.py --dataset-dir /path/to/dataset task_a task_b

The dataset directory must contain one subdir per task with a `task.toml` declaring `[environment].docker_image` (or an `environment/Dockerfile` fallback). Each task's template alias is `<task_name>` with `.` replaced by `-`.

The default packages baked into each template come from `scripts/build_templates.py:DEFAULT_NEXAU_PACKAGES` (a public NexAU + the in-sandbox `NexAU-harbor` variant, intentionally distinct from the host-side `harbor-LJH` in `pyproject.toml`). Override with one or more `--nexau-package <git-or-pip-spec>` flags if you need a different revision in the sandbox.

If your tasks pull from a private Docker registry, also export `DOCKER_REGISTRY_USERNAME` and `DOCKER_REGISTRY_PASSWORD` before invoking the script.

4. Launch

# Run a single experiment in the background via tmux
./scripts/evolve.sh configs/experiments/exp-simple-code-gpt54.yaml

# Launch and auto-attach to the log stream
./scripts/evolve.sh --attach configs/experiments/exp-simple-code-gpt54.yaml

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 336 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
  2. <div align="left">
  3. <p align="left">
  4. <a href="https://arxiv.org/abs/2604.25850"><img alt="Paper" src="https://img.shields.io/badge/Paper-arXiv-b31b1b.svg?logo=arxiv&logoColor=white"></a>
  5. <a href="agentic_harness_engineering.pdf"><img alt="PDF" src="https://img.shields.io/badge/PDF-Download-ec1c24.svg?logo=adobeacrobatreader&logoColor=white"></a>
  6. <a href="https://dawning-road.github.io/blog/agentic-harness-engineering"><img alt="Blog" src="https://img.shields.io/badge/Blog-Dawning_Road-ff7e1b.svg?logo=readthedocs&logoColor=white"></a>
  7. <img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg">
  8. <img alt="Python" src="https://img.shields.io/badge/python-%E2%89%A53.13-blue.svg">
  9. <img alt="Managed with uv" src="https://img.shields.io/badge/managed_with-uv-261230?logo=python&logoColor=white">
  10. </p>
  11. </div>
  12. <p align="center">
  13. <img src="assets/figures/banner.jpg" alt="Agentic Harness Engineering" width="100%">
  14. </p>
  15. <p align="left">
  16. English | <a href="README_zh.md">简体中文</a>
  17. </p>
  18. ---
  19. ## 📰 News
  20. - **[2026-05-14]** 🏆 AHE (on GPT-5.5) ranked **#3** on the [Terminal-Bench 2.0 leaderboard](https://www.tbench.ai/leaderboard/terminal-bench/2.0) with **84.7%** — ranking as of 2026-05-15
  21. - **[2026-04-30]** ✍️ Blog post on Dawning Road (English & Chinese) — a more detailed account of the exploration behind AHE: [Agentic Harness Engineering](https://dawning-road.github.io/blog/agentic-harness-engineering)
  22. - **[2026-04-28]** 📄 Paper released on arXiv: [Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses](https://arxiv.org/abs/2604.25850)
  23. - **[2026-04]** 🎉 Framework released
  24. ---
  25. ## 🎯 Overview
  26. **AHE (Agentic Harness Engineering)** is an open **observability system** for automatically evolving the harness around a coding agent. The base model is held fixed; what evolves are the harness components — system prompts, tool descriptions, tool implementations, middleware, skills, sub-agents, and long-term memory.
  27. AHE rests on three observability layers:
  28. - **Component observability** — [**NexAU**](https://github.com/hzhua/nexau.git) decomposes the harness into seven orthogonal, file-level components, each git-tracked so every edit is auditable and revertible.
  29. - **Experience observability** — *Agent Debugger* distills ~10M-token raw traces into layered, sourced reports; the optimizer reads digests by default but can always drill back to any rollout's raw trace.
  30. - **Decision observability** — *Evolve Agent* proposes evidence-backed edits, predicts their impact, and is automatically falsified by the next iteration's flipped tasks.
  31. Across ten `evaluate → analyze → improve` iterations, **AHE (Agentic Harness Engineering)** lifts Terminal-Bench 2 pass@1 from **69.7% to 77.0%** on GPT-5.4, surpasses the hand-written Codex (71.9%) and the self-evolving ACE and TF-GRPO baselines, and produces a frozen harness that transfers without re-evolution to SWE-bench-verified and to four alternate base models, indicating that the evolved components encode general engineering experience rather than benchmark-specific tuning.
  32. <p align="center">
  33. <img src="assets/figures/transfer_model.png" alt="Cross-Model Transfer" width="28%">
  34. <img src="assets/figures/case_study.png" alt="Case Study" width="31%">
  35. <img src="assets/figures/training_curve.png" alt="Training Curve" width="39%">
  36. </p>
  37. ---
  38. ## 🚀 Quick Start
  39. ### 0. Prerequisites
  40. - Python ≥ 3.13
  41. - [uv](https://docs.astral.sh/uv/)
  42. - tmux
  43. ```bash
  44. # macOS
  45. brew install uv tmux
  46. # Linux
  47. curl -LsSf https://astral.sh/uv/install.sh | sh
  48. sudo apt install -y tmux
  49. ```
  50. ### 1. Clone + install dependencies
  51. ```bash
  52. git clone https://github.com/china-qijizhifeng/agentic-harness-engineering.git
  53. cd agentic-harness-engineering
  54. uv sync
  55. ```
  56. > `uv sync` installs every dependency declared in `pyproject.toml`.
  57. NexAU is pinned to the [0.4.1 source snapshot `28511d9`](https://github.com/hzhua/nexau/commit/28511d9f0362191a6c378fd441b2be55309a9e32). The host, debugger, sandbox templates, and source explorer use this same revision. After upgrading an existing checkout, run `uv sync` and rebuild your E2B templates with `scripts/build_templates.py`; existing templates retain their previously installed packages.
  58. Start a new experiment after upgrading. To resume an older experiment, first merge the updated context-compaction middleware into its `evolve_agent/` copy and update its saved `explore_agent.code_sources.nexau` URL and ref. Existing experiment code and config snapshots are preserved, so pulling this update does not migrate them automatically. The Harbor adapter also pins on-demand installs when `USE_BP_E2B=False`.
  59. ### 2. Configure environment variables
  60. ```bash
  61. cp .env.example .env
  62. ```
  63. Edit `.env`. At minimum, set:
  64. | Variable | Purpose |
  65. |---|---|
  66. | `LLM_API_KEY` / `LLM_BASE_URL` | Main LLM endpoint (`code_agent` and `evolve_agent` both consume it) |
  67. | `E2B_API_KEY` | [E2B](https://e2b.dev/) sandbox — see the next subsection for SaaS vs. self-hosted |
  68. | `SERPER_API_KEY` | Web search used by `evolve_agent` |
  69. `ADB_LLM_*` and `GPT54_LLM_*` are optional — leave them unset to fall back to `LLM_*`, or set them to point ADB / the gpt-5.4 experiment at a stronger model. `LANGFUSE_*`, `BP_HTML_PARSER_*`, and `FEISHU_WEBHOOK` are all optional observability / convenience hooks; see `.env.example` for the full list.
  70. #### E2B sandbox: SaaS vs. self-hosted
  71. AHE runs every rollout inside an E2B sandbox. Two deployment modes are supported:
  72. - **SaaS E2B (default).** Set **only** `E2B_API_KEY` and leave `E2B_API_URL` / `E2B_DOMAIN` unset (or commented out). The SDK talks to `e2b.dev` automatically.
  73. > ⚠️ **Concurrency cap.** SaaS E2B enforces a per-account **concurrent sandbox limit** tied to your tier. If harbor tries to spawn more sandboxes than the cap allows, the extra sandboxes fail to start and the iteration stalls. Before raising parallelism in your harbor / experiment config, check your tier's quota and stay safely under it.
  74. - **Self-hosted E2B cluster.** Set `E2B_API_KEY` **and** point the SDK at your cluster:
  75. ```dotenv
  76. E2B_API_KEY="your_e2b_key"
  77. E2B_API_URL="https://your-e2b-host.example.com"
  78. E2B_DOMAIN="your-e2b-host.example.com"
  79. ```
  80. No shared concurrency cap applies, but the cluster's hardware capacity still does.
  81. ### 3. Build E2B templates (one-time per dataset)
  82. The dataset here is a pack from [`laude-institute/harbor-datasets`](https://github.com/laude-institute/harbor-datasets) — clone the subset you need and point `--dataset-dir` at its directory.
  83. Every rollout runs inside an E2B sandbox spawned from a prebuilt template that already has `uv` and the NexAU/harbor venv at `/opt/nexau-venv`. Build those templates once before launching:
  84. ```bash
  85. # Build every template declared by the dataset, 16 in parallel
  86. uv run python scripts/build_templates.py --dataset-dir /path/to/dataset -j 16
  87. # Resume after a failure: only retry tasks whose latest E2B build status is ERROR
  88. uv run python scripts/build_templates.py --dataset-dir /path/to/dataset --retry-failed
  89. # Build a specific subset of tasks
  90. uv run python scripts/build_templates.py --dataset-dir /path/to/dataset task_a task_b
  91. ```
  92. The dataset directory must contain one subdir per task with a `task.toml` declaring `[environment].docker_image` (or an `environment/Dockerfile` fallback). Each task's template alias is `<task_name>` with `.` replaced by `-`.
  93. The default packages baked into each template come from `scripts/build_templates.py:DEFAULT_NEXAU_PACKAGES` (a public NexAU + the in-sandbox `NexAU-harbor` variant, intentionally distinct from the host-side `harbor-LJH` in `pyproject.toml`). Override with one or more `--nexau-package <git-or-pip-spec>` flags if you need a different revision in the sandbox.
  94. If your tasks pull from a private Docker registry, also export `DOCKER_REGISTRY_USERNAME` and `DOCKER_REGISTRY_PASSWORD` before invoking the script.
  95. ### 4. Launch
  96. ```bash
  97. # Run a single experiment in the background via tmux
  98. ./scripts/evolve.sh configs/experiments/exp-simple-code-gpt54.yaml
  99. # Launch and auto-attach to the log stream
  100. ./scripts/evolve.sh --attach configs/experiments/exp-simple-code-gpt54.yaml
  101. # Batch: launch every experiment under configs/experiments/
  102. ./scripts/evolve.sh --batch
  103. ```
  104. Common tmux operations after launch:
  105. ```bash
  106. tmux ls # list sessions
  107. tmux attach -t <session> # attach to a session
  108. # Ctrl-b d # detach (keeps running in background)
  109. tmux kill-session -t <session> # terminate
  110. ```
  111. ---
  112. ## 🔧 How It Works
  113. The base model is held fixed; what evolves is the **harness around it**. Each outer iteration is `evaluate → analyze → improve`, built on the three observability layers from the Overview.
  114. ### 1. Evaluate — emit traces, not just scores
  115. `harbor` runs the current `code_agent` over the dataset inside isolated E2B sandboxes. Per task it writes:
  116. - `agent/nexau_in_memory_tracer.cleaned.json` — full step-level trace (messages, tool calls, middleware events)
  117. - `agent/nexau.txt` — runtime log (middleware errors, crashes, warnings)
  118. - `verifier/reward.txt` — pass/fail outcome
  119. The **trace, not the pass rate**, is the unit every later step operates on.
  120. ### 2. Analyze — distill ~10M-token traces into sourced evidence
  121. *Agent Debugger* compresses each iteration's raw traces (routinely >10M tokens) into layered reports:
  122. - `analysis/overview.md` — cross-task root-cause summary
  123. - `analysis/detail/{task}.md` — per-task deep analysis
  124. The optimizer reads digests by default, but every claim links back to the originating raw trace, so it can drill down before committing to a change.
  125. > **Note on Agent Debugger licensing.** The current release ships a *partially* open-sourced Agent Debugger; due to company strategy, it cannot be fully open-sourced at this time.
  126. ### 3. Improve — evidence-backed, falsifiable edits
  127. *Evolve Agent* may only write inside `workspace/`, which exposes the seven NexAU components: `systemprompt.md`, `code_agent.yaml`, `tool_descriptions/`, `tools/`, `middleware/`, `skills/`, `sub_agents/` (plus `LongTermMEMORY.md`). For every edit it must commit four fields:
  128. 1. **Failure evidence** — the failing tasks and trace excerpts that motivate the change
  129. 2. **Root cause** — *why* it failed, not just *what* failed
  130. 3. **Targeted fix** — the change that directly addresses that cause
  131. 4. **Predicted impact** — which tasks should flip to pass, and which are at risk
  132. ### 4. Loop — staggered generations enable falsification
  133. Each `runs/iteration_NNN/` mixes two generations: `input/` holds the workspace produced by loop `NNN-1` (just evaluated), `evolve/` holds what loop `NNN` writes (evaluated next loop). Flips (pass↔fail) on the next eval are attributed back to this loop's edits in `change_evaluation.json` — predictions that don't hold get rolled back or revised. The loop terminates on `target_pass_rate` or `max_iterations`.
  134. ### Main components
  135. | Component | Role |
  136. |---|---|
  137. | `evolve.py` | Main-loop orchestrator |
  138. | `agents/code_agent_simple/` | The coding agent that is being evaluated and evolved |
  139. | `agents/evolve_agent/` | The meta-agent that performs the improvement step (built on the [NexAU](https://github.com/hzhua/nexau.git) framework) |
  140. | `agents/explore_agent/` | Upstream dataset / source-code exploration agent |
  141. | `configs/` | `base.yaml` (shared defaults) + `experiments/` (per-experiment overlays) |
  142. | `scripts/` | tmux launcher wrappers (`evolve.sh`, `evolve-resume.sh`) |
  143. ### Directory layout
  144. ```
  145. agentic-harness-engineering/
  146. ├── evolve.py # main loop
  147. ├── trace_converter.py # rollout trace → debugger-friendly JSON
  148. ├── agents/
  149. │ ├── code_agent_simple/ # the coding agent under evolution
  150. │ ├── evolve_agent/ # the evolution meta-agent
  151. │ │ ├── evolve_prompt.md
  152. │ │ ├── middleware/ # context compaction / failover / ralph loop …
  153. │ │ ├── skills/ # agent-debugger-cli / nexau-evolution-guide
  154. │ │ └── tools/ # file / shell / web / session tools
  155. │ └── explore_agent/ # exploration agent (sources + web)
  156. ├── configs/
  157. │ ├── base.yaml # shared defaults
  158. │ └── experiments/ # one overlay per experiment
  159. ├── scripts/
  160. │ ├── evolve.sh # tmux launcher
  161. │ └── evolve-resume.sh # resume helper
  162. └── .env.example
  163. ```
  164. ---
  165. ## Configuration (base + overlay)
  166. `configs/base.yaml` holds the shared defaults. Each `configs/experiments/exp-*.yaml` inherits it via a leading `_base: ../base.yaml` line and overrides only the fields that differ. Any `${ENV_NAME}` reference inside a YAML file is substituted from `.env`.
  167. **Key fields in `base.yaml`:**
  168. | Field | Description |
  169. |---|---|
  170. | `path` | Dataset path |
  171. | `target_pass_rate` | Stop once reached (default 0.95) |
  172. | `max_iterations` | Maximum number of iterations (default 100) |
  173. | `harbor_job_timeout_minutes` | Per-harbor-evaluation timeout (0 = unlimited) |
  174. | `experiment_timeout_minutes` | Total wall-clock budget for the experiment (0 = unlimited) |
  175. | `llm.api_key / base_url / model` | Main LLM config (usually left as `${LLM_*}`) |
  176. | `agent_debugger.llm` | Dedicated LLM for ADB (can use a stronger model for debugging) |
  177. | `notify.feishu_webhook` | Optional Feishu webhook for experiment milestones |
  178. ### Dataset configuration
  179. An experiment's data source is specified via `path` **or** `dataset` — pick one:
  180. | Form | Meaning | Example |
  181. |---|---|---|
  182. | `path: "./dataset/xxx"` | Local dataset directory (relative to the AHE root) | `./dataset/terminal-bench-2` |
  183. | `path: "/abs/path/xxx"` | Local dataset directory (absolute path) | `/root/dataset/terminal-bench-2` |
  184. | `dataset: "<name>@<ver>"` | Reference a harbor built-in dataset (no local files required) | `terminal-bench@2.0` |
  185. Public dataset packs (in the layout AHE expects under `path:`) are published at [`laude-institute/harbor-datasets`](https://github.com/laude-institute/harbor-datasets) — clone or download the subset you need and point `path` at its directory.
  186. The default `path` values in `base.yaml` and `configs/experiments/*.yaml` are **placeholders only** — adjust them for your environment, or comment out `path` and uncomment the `dataset` line to use a harbor built-in dataset instead.
  187. ---
  188. ## CLI reference
  189. ### `python evolve.py`
  190. | Flag | Description |
  191. |---|---|
  192. | `--config <file>` | Config file (overlay takes precedence) |
  193. | `--batch [dir\|files...]` | Batch mode; defaults to scanning `configs/experiments/` |
  194. | `--experiment <name>` | Resume an existing experiment (pass the directory name under `experiments/`) |
  195. | `--start-iteration N` | Start from iteration N (default 1) |
  196. | `--skip-eval` | Skip evaluation and reuse existing rollouts (for debugging) |
  197. ### `./scripts/evolve.sh`
  198. A thin wrapper around `uv run python evolve.py` + tmux.
  199. | Flag | Description |
  200. |---|---|
  201. | `<config_file>` | Positional argument: path to the config file |
  202. | `--experiment <name>` | Resume an existing experiment |
  203. | `--start-iteration N` | Starting iteration |
  204. | `--skip-eval` | Skip evaluation |
  205. | `--session <name>` | Custom tmux session name |
  206. | `--batch` | Launch every overlay in batch mode |
  207. | `--attach` | Auto-attach after launch |
  208. ---
  209. ## Common scenarios
  210. **Resume an interrupted experiment from iteration 16:**
  211. ```bash
  212. ./scripts/evolve.sh \
  213. --experiment 2026-04-10__23-20-14__gpt54 \
  214. --start-iteration 16 \
  215. configs/experiments/exp-simple-code-gpt54.yaml
  216. ```
  217. **Run only evolve_agent without re-running evaluation:**
  218. ```bash
  219. ./scripts/evolve.sh \
  220. --experiment <existing-exp-dir> \
  221. --skip-eval \
  222. configs/experiments/exp-simple-code-gpt54.yaml
  223. ```
  224. ---
  225. ## License
  226. MIT
  227. ---
  228. ## Star History
  229. <picture>
  230. <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=china-qijizhifeng/agentic-harness-engineering&type=Date&theme=dark" />
  231. <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=china-qijizhifeng/agentic-harness-engineering&type=Date" />
  232. <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=china-qijizhifeng/agentic-harness-engineering&type=Date" />
  233. </picture>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.