Skip to content
Back to search

agtx

๐Ÿ„๐Ÿผโ€โ™‚๏ธ The blackboard for coding agents - agentic development environment for claude code, codex, cursor, opencode, grok and more.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/fynnfluegge--agtx/badge.svg)](/listings/fynnfluegge--agtx)

HTML

<a href="/listings/fynnfluegge--agtx"><img src="/api/listings/fynnfluegge--agtx/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

# Install
curl -fsSL https://raw.githubusercontent.com/fynnfluegge/agtx/main/install.sh | bash
# Run in any git repository
cd your-project && agtx

> [!NOTE] > Add `.agtx/` to your project's `.gitignore` to avoid committing worktrees and local task data.

# Install from source โ€” `serve` adds the mobile board (`W`), and the
# published binaries are built with it
cargo build --release --features serve
cp target/release/agtx ~/.local/bin/

Updating

agtx checks GitHub once a day for a newer release and shows `โฌ† 0.2.8 [u]` in the board header when there is one. Press **`u`** for the details and one key to install it, or from a shell:

agtx update          # download, verify and replace this binary in place
agtx update --check  # report only; exits 1 when an update is available
agtx --version

Requirements

- **tmux** โ€” agent sessions run in a dedicated tmux server - **gh** (optional) โ€” GitHub CLI for PR operations

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json ยท Manifest ยท 14 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "agtx-marketplace",
  3. "description": "Marketplace for agtx plugins",
  4. "owner": {
  5. "name": "fynnfluegge"
  6. },
  7. "plugins": [
  8. {
  9. "name": "agtx",
  10. "description": "Manage and decompose work into feature-level tasks on the agtx kanban board",
  11. "source": "./"
  12. }
  13. ]
  14. }
README.md ยท README ยท 1,000 lines ยท 1 finding marked

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. [//]: <img src="https://github.com/user-attachments/assets/54ac039b-085e-490b-aacc-36c8e244e313" width="428" />
  3. # ๐Ÿ„๐Ÿผโ€โ™‚๏ธ agtx
  4. **The terminal-native agentic development environment for 10x productivity.**
  5. <div align="left">
  6. > **A blackboard for coding agents** - One shared board. A fleet of agents. Add tasks, delegate to multiple coding agents running in parallel and let
  7. > **different models collaborate** on the same task with automatic session switching and context awareness, e.g. Codex planning, Claude implementing, Grok review.
  8. </div>
  9. [![CI](https://github.com/fynnfluegge/agtx/actions/workflows/ci.yml/badge.svg)](https://github.com/fynnfluegge/agtx/actions/workflows/ci.yml)
  10. [![Release](https://img.shields.io/github/v/release/fynnfluegge/agtx)](https://github.com/fynnfluegge/agtx/releases)
  11. [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
  12. [![License: Apache-2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
  13. <a href="https://trendshift.io/repositories/23889" target="_blank"><img src="https://trendshift.io/api/badge/repositories/23889" alt="fynnfluegge%2Fagtx | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
  14. <p align="center">
  15. <a href="#quick-start">Quick Start</a> โ€ข
  16. <a href="#features">Features</a> โ€ข
  17. <a href="#usage">Usage</a> โ€ข
  18. <a href="#mobile">Mobile</a> โ€ข
  19. <a href="#skills">Skills</a> โ€ข
  20. <a href="#oneshot-skill">Oneshot</a> โ€ข
  21. <a href="#mcp-server">MCP Server</a> โ€ข
  22. <a href="#plugins">Plugins</a> โ€ข
  23. <a href="#orchestrator-agent-experimental">Orchestrator</a> โ€ข
  24. <a href="#configuration">Configuration</a>
  25. </p>
  26. ---
  27. <img width="840" src="https://github.com/user-attachments/assets/45858e09-ab61-422b-b708-db060c73a900" />
  28. [//]: <img width="840" src="https://github.com/user-attachments/assets/42f71a6c-424c-4cc4-80fc-dc9bb8ba1467" />
  29. <br/>
  30. [//]: <img width="1486" height="680" src="https://github.com/user-attachments/assets/45858e09-ab61-422b-b708-db060c73a900" />
  31. [//]: <![Xnapper-2026-02-14-09 36 33 (1)](https://github.com/user-attachments/assets/fce21a9c-2fe1-4b14-8f24-55e058531370)>
  32. </div>
  33. ## Features
  34. - **Supported Agents**:&nbsp; <a href="https://github.com/anthropics/claude-code"><kbd><img src="docs/logos/claude.svg" width="18" valign="middle" /> Claude Code</kbd></a>
  35. <a href="https://github.com/openai/codex"><kbd><img src="docs/logos/codex-dark.svg" width="18" valign="middle" /> Codex</kbd></a>
  36. <a href="https://docs.x.ai/build/overview"><kbd><img src="docs/logos/grok-dark.svg" width="18" valign="middle" /> Grok</kbd></a>
  37. <a href="https://cursor.com/docs/cli/overview"><kbd><img src="docs/logos/cursor-dark.svg" width="18" valign="middle" /> Cursor</kbd></a>
  38. <a href="https://github.com/sst/opencode"><kbd><img src="docs/logos/opencode-dark.svg" width="18" valign="middle" /> OpenCode</kbd></a>
  39. <a href="https://github.com/google-antigravity/antigravity-cli"><kbd><img src="docs/logos/antigravity.png" width="18" valign="middle" /> Antigravity</kbd></a>
  40. <a href="https://github.com/google-gemini/gemini-cli"><kbd><img src="docs/logos/gemini.svg" width="18" valign="middle" /> Gemini CLI</kbd></a>
  41. <a href="https://github.com/github/copilot-cli"><kbd><img src="docs/logos/copilot-dark.svg" width="18" valign="middle" /> Copilot</kbd></a>
  42. <a href="https://github.com/earendil-works/pi"><kbd><img src="docs/logos/pi-dark.svg" width="18" valign="middle" /> pi</kbd></a>
  43. - **Parallel Multi-agent task lifecycle**: Configure different agents per workflow phase โ€” e.g. Grok for research, Claude for implementation, Codex for review โ€” with automatic agent switching and context handover.
  44. - **Multi-project Kanban board**: Manage agent sessions across all projects via a single TUI without leaving your terminal.
  45. - **Vim-native Keybindings**: Control the board and agent sessions with your Vim-powered muscle memory.
  46. - **Mobile support**: Press `W` and scan the QR for an installable web app with the board, task details, diffs and the agent's live terminal โ€” including a keyboard, so you can answer a permission prompt without going back to your desk. Over your wifi, or from anywhere via your tailnet. See <a href="#mobile">Mobile</a>.
  47. - **Orchestrator agent (experimental)**: A dedicated agent that autonomously manages your kanban board via MCP โ€” delegates to coding agents, advances phases, checks for merge conflicts.
  48. - **Oneshot skill**: Give a coding agent session a goal with `/agtx:oneshot` and it runs the whole board unattended โ€” plans the work in milestone waves, starts tasks, unblocks the workers, judges Review, and merges each task. See <a href="#oneshot-skill">Oneshot Skill</a>.
  49. - **Brainstorm & Sweep skills**: Capture ideas and push them to the board from any coding agent session โ€” `/agtx:brainstorm` to explore freely, `/agtx:sweep` to decompose and create tasks with one confirmation step.
  50. - **Spec-driven plugins**: Plug in [GSD](https://github.com/fynnfluegge/get-shit-done-cc), [Spec-kit](https://github.com/github/spec-kit), [OpenSpec](https://github.com/Fission-AI/OpenSpec), [BMAD](https://github.com/bmad-code-org/BMAD-METHOD), [Superpowers](https://github.com/obra/superpowers) and more โ€” fully customizable. Ddefine your own workflow via a single TOML file. See <a href="#plugins">Plugins</a> how to create a plugin.
  51. > [!NOTE]
  52. > Just need a plain coding-agent session-manager with **full human-in-the-loop control** and **no spec-driven skill execution and orchestration** on advancing tasks?
  53. >
  54. > Choose the **`void` plugin** and enjoy agtx as a batteries included multi-agent session-manager.
  55. ## Quick Start
  56. ```bash
  57. # Install
  58. curl -fsSL https://raw.githubusercontent.com/fynnfluegge/agtx/main/install.sh | bash
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 78 on GitHub
  59. ```
  60. ```bash
  61. # Run in any git repository
  62. cd your-project && agtx
  63. ```
  64. > [!NOTE]
  65. > Add `.agtx/` to your project's `.gitignore` to avoid committing worktrees and local task data.
  66. ```bash
  67. # Install from source โ€” `serve` adds the mobile board (`W`), and the
  68. # published binaries are built with it
  69. cargo build --release --features serve
  70. cp target/release/agtx ~/.local/bin/
  71. ```
  72. ### Updating
  73. agtx checks GitHub once a day for a newer release and shows `โฌ† 0.2.8 [u]` in the
  74. board header when there is one. Press **`u`** for the details and one key to
  75. install it, or from a shell:
  76. ```bash
  77. agtx update # download, verify and replace this binary in place
  78. agtx update --check # report only; exits 1 when an update is available
  79. agtx --version
  80. ```
  81. ### Requirements
  82. - **tmux** โ€” agent sessions run in a dedicated tmux server
  83. - **gh** (optional) โ€” GitHub CLI for PR operations
  84. ## Usage
  85. <details>
  86. <summary><strong>Keyboard Shortcuts</strong></summary>
  87. | Key | Action |
  88. |-----|--------|
  89. | `h/l` or `โ†/โ†’` | Move between columns |
  90. | `j/k` or `โ†‘/โ†“` | Move between tasks |
  91. | `o` | Create new task |
  92. | `R` | Enter research mode |
  93. | `โ†ฉ` | Open task (view agent session) |
  94. | `Ctrl+f` | Open the task popup fullscreen; press again to return to windowed mode |
  95. | `m` | Move task forward in workflow |
  96. | `r` | Resume task (Review โ†’ Running) / Move back (Running โ†’ Planning) |
  97. | `p` | Next phase (Review โ†’ Planning, cyclic plugins only) |
  98. | `d` | Show git diff |
  99. | `x` | Delete task |
  100. | `/` | Search tasks |
  101. | `P` | Select spec-driven workflow plugin |
  102. | `,` | Open the config editor |
  103. | `?` | Show every keyboard shortcut |
  104. | `u` | Update agtx (only shown when a new release is available) |
  105. | `W` | Serve the board to a phone (QR pairing, device list) |
  106. | `O` | Toggle orchestrator agent (`--experimental`) |
  107. | `e` | Toggle project sidebar |
  108. | `q` | Quit |
  109. </details>
  110. <details>
  111. <summary><strong>Task Creation Wizard</strong></summary>
  112. Press `o` to create a new task. The wizard guides you through:
  113. 1. **Title** โ€” enter a short task name
  114. 2. **Plugin** โ€” select a workflow plugin (auto-skipped if only one option)
  115. 3. **Prompt** โ€” write a detailed task description with inline references
  116. The agent is configured at the project level via `config.toml` (not per-task).
  117. </details>
  118. <details>
  119. <summary><strong>Task Description Editor</strong></summary>
  120. When writing a task description, you can reference files, skills, and other tasks inline:
  121. | Key | Action |
  122. |-----|--------|
  123. | `#` or `@` | Fuzzy search and insert a file path |
  124. | `/` | Fuzzy search and insert an agent skill/command (at line start or after space) |
  125. | `!` | Fuzzy search and insert a task reference (at line start or after space) |
  126. </details>
  127. <details>
  128. <summary><strong>Agent Sessions</strong></summary>
  129. Each task runs in its own tmux window with a dedicated coding agent. The session persists across the entire task lifecycle โ€” you can open the task popup at any time to see live agent output, or press `Ctrl+f` to open it fullscreen inside agtx.
  130. - **Persistent context**: The agent's full conversation history is preserved across Planning โ†’ Running โ†’ Review
  131. - **Resume from Review**: Moving a task back to Running simply reconnects to the existing session โ€” no re-initialization needed
  132. - **Inline view**: Press `โ†ฉ` on any active task to open a scrollable tmux view inside the TUI
  133. - **Fullscreen**: Press `Ctrl+f` to expand the task popup inside agtx. Press `Ctrl+f` again for windowed mode or `Ctrl+q` to return to the board.
  134. - **Auto merge-conflict resolution**: When a Review task becomes idle, agtx checks for merge conflicts with the default branch using a non-destructive virtual merge (`git merge-tree`). If conflicts are detected, the agent is automatically sent the `/agtx:merge-conflicts` skill to resolve them and re-commit
  135. </details>
  136. ## Why agtx? - The blackboard model
  137. Most AI coding tools give you one agent, one task, one terminal. agtx is built on a different and much
  138. older idea: the [**blackboard system**](https://en.wikipedia.org/wiki/Blackboard_system).
  139. > A blackboard system is an approach where a common knowledge base โ€” the *blackboard* โ€” is iteratively
  140. > updated by a diverse group of specialist *knowledge sources*, starting from a problem specification
  141. > and ending with a solution. Each specialist writes a partial solution to the blackboard when the
  142. > state on the board matches what it can contribute.
  143. >
  144. > โ€” after [*Blackboard system*](https://en.wikipedia.org/wiki/Blackboard_system), Wikipedia (CC BY-SA)
  145. That architecture was designed for problems that are too ill-defined for a single solver and too
  146. interdependent to split cleanly up front. **Shipping software with coding agents is exactly that
  147. problem**, so agtx implements the model directly:
  148. The dependency graph gives the blackboard its structure. Tasks references they build on,
  149. forming a graph of partial solutions - agtx holds downstream tasks until their dependencies reach
  150. Review or Done, then carries the relevant diffs and artifacts into the dependent task's context.
  151. ```
  152. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  153. โ”‚ CONTROL orchestrator agent ยท phase gates ยท dep graph โ”‚
  154. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  155. โ”‚
  156. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  157. โ”‚ THE BLACKBOARD โ”‚
  158. โ”‚ backlog โ†’ planning โ†’ running โ†’ review โ†’ done โ”‚
  159. โ”‚ dependency graph ยท specs ยท plans ยท diffs ยท reviews โ”‚
  160. โ””โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”˜
  161. โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
  162. โ”Œโ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”
  163. โ”‚ Claude โ”‚ โ”‚ Codex โ”‚ โ”‚Gemini โ”‚ โ”‚Cursor โ”‚ โ”‚ Grok โ”‚ โ”‚ ... โ”‚
  164. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  165. KNOWLEDGE SOURCES โ€” one git worktree + tmux window each
  166. ```
  167. | Blackboard model | In agtx |
  168. |------------------|---------|
  169. | **The blackboard** โ€” a shared repository of the problem, partial solutions and contributed information | The kanban board, its dependency graph, and everything the phases leave behind: specs, plans, diffs, reviews, and phase artifacts. Every agent reads from and writes to the same board |
  170. | **Knowledge sources** โ€” independent specialists that never talk to each other, only to the board | Eight coding agent CLIs, each running in its **own git worktree and tmux window**. No agent can see another's context โ€” they exchange only what lands on the board |
  171. | **Control shell** โ€” decides opportunistically which specialist runs next | Plugin phase gates determine when a task can advance; the dependency graph determines which tasks are ready to start; and the [orchestrator agent](#orchestrator-agent-experimental) coordinates the board over MCP |
  172. ## Mobile
  173. [//]: <> (screenshot: the board on a phone โ€” take it on a real device, then drag it into a GitHub comment and paste the attachment URL here, as with the TUI shot above)
  174. Press `W` on the board to serve it to your phone. agtx prints a QR code; scanning
  175. it pairs the device and opens an installable web app with the board, task
  176. details, git diffs, and the agent's live terminal โ€” including a keyboard, so you
  177. can answer a permission prompt from wherever you are.
  178. ```bash
  179. # Or start it yourself, without the TUI
  180. agtx serve # this machine only
  181. agtx serve --tunnel # your tailnet โ€” anywhere, your devices only
  182. agtx serve --devices # list paired devices
  183. agtx serve --revoke <id> # revoke one; --revoke-all for the lot
  184. ```
  185. Two keys in the overlay, each a whole action: `s` serves to the local network,
  186. `t` serves via your tailnet.
  187. > [!IMPORTANT]
  188. > **agtx never opens a port on its own.** Serving lasts for as long as that agtx
  189. > runs: quit it and the server stops, so the next time you want the board on
  190. > your phone you press `W` then `s`/`t` again. Pairing does not change this โ€” it
  191. > is a credential, not a trigger, and a paired phone whose Mac is not serving
  192. > just times out.
  193. > [!NOTE]
  194. > This needs a binary built with `--features serve`. The released ones are; a
  195. > local `cargo build --release` without it reports "this build has no web
  196. > server" under both options.
  197. <details>
  198. <summary><strong>Setup โ€” on your wifi</strong></summary>
  199. Works in about thirty seconds, and needs nothing installed.
  200. 1. Run `agtx` in a project and press `W`
  201. 2. Press `s`
  202. 3. Point your phone's camera at the QR code โ€” the phone must be on the same wifi
  203. 4. In Safari or Chrome: **Share โ†’ Add to Home Screen**
  204. That last step is what turns it into an app: its own icon, no address bar, and
  205. the pairing remembered so you never scan again โ€” but you still press `W` then
  206. `s` to start serving each time you run agtx.
  207. The URL here is a private address like `192.168.1.20`, which exists only on your
  208. network. **It will not work on mobile data** โ€” if you leave the house, the app
  209. will sit there timing out. That is what the tailnet option below is for.
  210. </details>
  211. <details>
  212. <summary><strong>Setup โ€” from anywhere, via Tailscale</strong></summary>
  213. A few minutes once, then it is the same two taps forever. Your board becomes
  214. reachable from your phone on any network, while staying invisible to everyone
  215. else.
  216. **One-time, on your Mac and your phone:**
  217. 1. Install [Tailscale](https://tailscale.com/download) on both
  218. 2. Sign both into the **same** account โ€” they need to be on one tailnet
  219. 3. Enable Serve for your tailnet. It is off by default and is a separate switch
  220. from installing Tailscale โ€” the step most people miss. The simplest way is to
  221. just press `t` in agtx once: Tailscale refuses with a one-time link that
  222. enables it for this machine, and agtx prints that link rather than swallowing
  223. it. Follow it, then press `t` again.
  224. **Once more, to install the app:**
  225. 1. Run `agtx`, press `W`, press `t`
  226. 2. Scan the QR, and **Share โ†’ Add to Home Screen**
  227. **Then, every time you want the board on your phone:** run `agtx`, press `W`,
  228. press `t`. That is all โ€” the pairing is remembered, so there is no QR to scan;
  229. what you are starting is the server.
  230. Now the icon on your home screen works on 5G, on hotel wifi, anywhere โ€” the
  231. address is an HTTPS name on your tailnet, and only devices signed into it can
  232. resolve or reach it.
  233. </details>
  234. <details>
  235. <summary><strong>When it does not work</strong></summary>
  236. | What you see | What it means |
  237. |---|---|
  238. | `Unavailable: install Tailscale and sign this machine in` | `t` needs Tailscale on *this* machine. Install it, or use `s` for wifi. |
  239. | `Unavailable: Tailscale is installed but not signed in` | Run `tailscale up`. |
  240. | `could not start the tunnel: โ€ฆ Serve is not enabled on your tailnet` | The one-time switch above. agtx relays Tailscale's own message, which carries the URL that enables it. |
  241. | Scanned fine, then the app times out | You pressed `s` (wifi) and the phone is on mobile data. Press `s` again to stop, then `t`. |
  242. | The tailnet URL loads on the Mac but not the phone | The phone is not on the tailnet. Open the Tailscale app there and connect. |
  243. | `This device is not authorised` | The pairing code expired โ€” it lasts two minutes and is single-use. Press `s` or `t` again for a fresh QR. |
  244. | The app worked yesterday, today it just times out | Nothing is serving. agtx does not start the server on its own โ€” run it, press `W`, then `s`/`t`. The pairing is still good; the port is not open. |
  245. | `No agtx running` on the board | Expected with no TUI open. Reading works; actions queue until one is. |
  246. </details>
  247. > [!IMPORTANT]
  248. > Anything that can reach this server can read every task, every diff and every
  249. > agent's screen, and can type into a running agent โ€” which is arbitrary code
  250. > execution on your machine. So: loopback needs no credential because reaching it
  251. > already means being on the machine, and **everything wider requires a paired
  252. > device**. Tokens are per-device and stored hashed, so a lost phone is revoked
  253. > without disturbing the rest. `--tunnel public` publishes to the open internet
  254. > and is deliberately not offered behind a keypress.
  255. **Actions queue; they do not execute on their own.** Moving a task writes a
  256. request that only a running `agtx` picks up, so with no TUI open your taps are
  257. accepted and then wait โ€” the board says so rather than pretending. Creating,
  258. editing and deleting Backlog tasks take effect immediately, since they need no
  259. agent. Starting the server with `W` keeps the two together by construction.
  260. ## Skills
  261. Companion skills for any coding agent session: capture ideas, turn them into tasks on the agtx board, or hand a session the whole board.
  262. | Skill | Command | When to use |
  263. |-------|---------|-------------|
  264. | **Brainstorm** | `/agtx:brainstorm` | Explore a feature idea โ€” discussion only, no planning or implementation |
  265. | **Sweep** | `/agtx:sweep` | Push conversation outcomes to the agtx board as tasks |
  266. | **Oneshot** | `/agtx:oneshot` | Give a goal and let the session run the whole board unattended โ€” see [Oneshot Skill](#oneshot-skill) |
  267. **Typical flow:**
  268. ```
  269. /agtx:brainstorm โ† explore the idea freely
  270. โ†“
  271. /agtx:sweep โ† extract tasks, confirm, push to board
  272. โ†“
  273. agtx board โ† tasks appear in Backlog, ready to advance
  274. ```
  275. The brainstorm skill keeps the agent in discussion mode โ€” asking questions, surfacing trade-offs, no code or plans. When the conversation feels complete, run `/agtx:sweep` to decompose outcomes into feature-level tasks and push them to the board with a single confirmation step.
  276. ### Install
  277. The Claude Code plugin brings all three skills. Oneshot runs from Claude Code; the steps for the other agents install sweep.
  278. <details>
  279. <summary><strong>Claude Code</strong></summary>
  280. ```bash
  281. claude plugin marketplace add fynnfluegge/agtx
  282. claude plugin install agtx@agtx-marketplace
  283. claude mcp add --scope user agtx -- agtx mcp-serve
  284. ```
  285. </details>
  286. <details>
  287. <summary><strong>Codex</strong></summary>
  288. ```bash
  289. codex mcp add agtx -- agtx mcp-serve
  290. ```
  291. Add to your project's `.agents/plugins/marketplace.json`:
  292. ```json
  293. {
  294. "name": "local-repo",
  295. "plugins": [
  296. {
  297. "name": "agtx",
  298. "source": {
  299. "source": "local",
  300. "path": "./plugins/agtx"
  301. },
  302. "policy": {
  303. "installation": "AVAILABLE",
  304. "authentication": "ON_INSTALL"
  305. },
  306. "category": "Productivity"
  307. }
  308. ]
  309. }
  310. ```
  311. Then in any Codex session: `@agtx:sweep` / `@agtx:brainstorm`
  312. </details>
  313. <details>
  314. <summary><strong>Gemini CLI</strong></summary>
  315. ```bash
  316. gemini mcp add agtx -- agtx mcp-serve
  317. echo "@skills/sweep/SKILL.md" >> ~/GEMINI.md
  318. ```
  319. </details>
  320. <details>
  321. <summary><strong>Cursor</strong></summary>
  322. ```bash
  323. cursor mcp add agtx -- agtx mcp-serve
  324. cp skills/sweep/SKILL.md ~/.cursor/rules/agtx-sweep.md
  325. ```
  326. </details>
  327. <details>
  328. <summary><strong>Grok Build</strong></summary>
  329. ```bash
  330. grok mcp add agtx -- agtx mcp-serve
  331. mkdir -p ~/.grok/skills/agtx-sweep && cp skills/sweep/SKILL.md ~/.grok/skills/agtx-sweep/SKILL.md
  332. ```
  333. </details>
  334. <details>
  335. <summary><strong>Antigravity</strong></summary>
  336. ```bash
  337. agy mcp add agtx agtx mcp-serve
  338. mkdir -p ~/.gemini/antigravity-cli/skills/agtx-sweep
  339. cp skills/sweep/SKILL.md ~/.gemini/antigravity-cli/skills/agtx-sweep/SKILL.md
  340. ```
  341. </details>
  342. <details>
  343. <summary><strong>pi</strong></summary>
  344. pi has no MCP client of its own โ€” the `pi-mcp-adapter` package provides one and
  345. reads servers in the standard `mcpServers` shape:
  346. ```bash
  347. mkdir -p ~/.pi/skills/agtx-sweep && cp skills/sweep/SKILL.md ~/.pi/skills/agtx-sweep/SKILL.md
  348. ```
  349. Register `agtx mcp-serve` with the adapter, then in any pi session:
  350. `/skill:agtx-sweep` / `/skill:agtx-brainstorm`
  351. </details>
  352. <details>
  353. <summary><strong>Other</strong></summary>
  354. Register `agtx mcp-serve` as an MCP server, then copy `skills/sweep/SKILL.md` into your agent's context.
  355. </details>
  356. > [!NOTE]
  357. > The project must have been opened in agtx at least once to appear in `list_projects`. Run `agtx` in your project directory first.
  358. ## Oneshot Skill
  359. > Give it a goal, not a task list. It runs the board until the goal is built and merged.
  360. `/agtx:oneshot` turns a Claude Code session into the person sitting at the board. It breaks
  361. the goal into milestones, turns the current milestone into tasks, starts them, watches the
  362. workers, answers their questions, judges each Review, and merges the finished work into your
  363. base branch โ€” then plans the next wave from what was actually built. The workers write the
  364. code; the oneshot session only writes tasks and its own state file.
  365. ### Setup
  366. 1. **Install the plugin and MCP server for Claude Code** โ€” see [Install](#install).
  367. 2. **Let agtx answer trust prompts.** Nobody is at the board to answer an agent's trust or
  368. bypass-permissions dialog, so without this every task parks as blocked:
  369. ```toml
  370. # ~/.config/agtx/config.toml
  371. auto_trust = true
  372. ```
  373. 3. **Open your coding agent once in the project root** and accept its trust prompt. Task
  374. worktrees live inside the project, so they inherit that decision.
  375. 4. **Start from a clean base branch.** An empty repository works too โ€” `git init -b main` is
  376. enough, and agtx makes the first commit.
  377. ### Run
  378. ```bash
  379. # Terminal 1 โ€” the board. Nothing moves without it.
  380. cd your-project && agtx
  381. # Terminal 2 โ€” the session that runs the board
  382. cd your-project && claude
  383. > /agtx:oneshot Build a CLI todo app in Rust with SQLite storage and tests
  384. ```
  385. Then leave it. Follow along on the board, or on your phone with `W`.
  386. ### What it does
  387. - **Plans in waves.** A spine of 4โ€“8 milestones first, then tasks for the current milestone
  388. only, with dependencies wired so a task waits for the ones it builds on. The next wave is
  389. planned from what the workers built.
  390. - **Runs 3โ€“6 tasks at once**, starting the next as its dependencies reach Review or Done.
  391. - **Waits instead of polling.** `wait_for_board_change` blocks until a task needs attention
  392. and returns only what changed, so the session spends turns only when something happened.
  393. - **Unblocks the workers.** It answers a tool's yes/no prompt, or a question its milestone
  394. plan already settles. Anything that would change the product goes to you under *Open
  395. questions* in `oneshot-state.md`, while the rest of the board keeps moving.
  396. - **Judges Review.** A small fix goes to the reviewer in place with `send_to_task`; real
  397. rework sends the task back to Running.
  398. - **Merges each task** with `move_to_done_and_merge`. On a conflict the task stays in Review
  399. and its own agent resolves it with `/agtx:merge-conflicts`.
  400. - **Keeps a durable record** in `oneshot-state.md` โ€” milestones, board, decisions and open
  401. questions โ€” so a fresh session can pick up the run.
  402. > [!IMPORTANT]
  403. > Oneshot merges into your **local** base branch, in your own checkout. Leave that checkout
  404. > on the base branch with no uncommitted changes to tracked files while it runs. Otherwise the
  405. > merge is refused and nothing is touched โ€” agtx never switches your branch or stashes your
  406. > work. Nothing is pushed.
  407. > [!NOTE]
  408. > A full run takes hours and many turns of the session running the board. Start with a small
  409. > goal to see how it works on your project.
  410. ## Configuration
  411. Config file location: `~/.config/agtx/config.toml`
  412. ### General
  413. ```toml
  414. default_agent = "claude"
  415. fullscreen_on_enter = false # When true, Enter opens the tmux pane fullscreen inside agtx
  416. agent_hooks = true # Let agents report their own phase status via lifecycle hooks
  417. auto_trust = false # Answer agents' trust prompts on your behalf
  418. update_check = true # Check GitHub daily for a new release (see Updating)
  419. ```
  420. ### Worktree Base Branch
  421. agtx creates a new git worktree for each task. By default it auto-detects the base branch in this
  422. order: `main`, then `master`, then the current branch. You can override this to force a specific
  423. base branch (for example `dev` or `develop`).
  424. Global worktree defaults can be set here:
  425. ```toml
  426. # ~/.config/agtx/config.toml
  427. [worktree]
  428. base_branch = "dev"
  429. worktree_dir = ".worktrees" # default: ".agtx/worktrees"
  430. ```
  431. `worktree_dir` is the directory (relative to project root) where task worktrees are created. Defaults
  432. to `.agtx/worktrees` if not set.
  433. ### Project Configuration
  434. Per-project settings can be placed in `.agtx/config.toml` at the project root:
  435. ```toml
  436. # Base branch used when creating new task worktrees (optional)
  437. base_branch = "dev"
  438. # Directory where worktrees are created (optional, default: ".agtx/worktrees")
  439. worktree_dir = ".worktrees"
  440. # Files to copy from project root into each new worktree (comma-separated)
  441. # Paths are relative and preserve directory structure
  442. copy_files = ".env, .env.local, web/.env.local"
  443. # Shell command to run inside the worktree after creation and file copying
  444. init_script = "scripts/init_worktree.sh"
  445. # Shell command to run inside the worktree before removal
  446. cleanup_script = "scripts/cleanup_worktree.sh"
  447. ```
  448. `base_branch` controls which branch new task worktrees are created from. If omitted or empty, agtx
  449. auto-detects `main`, `master`, or falls back to the current branch.
  450. ### Per-Phase Agent Configuration
  451. By default, all phases use `default_agent`. You can override the agent for specific phases globally or per project:
  452. ```toml
  453. # ~/.config/agtx/config.toml
  454. default_agent = "claude"
  455. [agents]
  456. research = "gemini"
  457. planning = "claude"
  458. running = "claude"
  459. review = "codex"
  460. ```
  461. ```toml
  462. # .agtx/config.toml (project override โ€” takes precedence over global)
  463. [agents]
  464. running = "codex"
  465. ```
  466. ## Plugins
  467. Plug any spec-driven framework into the task lifecycle. Define commands, prompts, and artifacts โ€” agtx handles phase gating, artifact polling, worktree sync, agent switching, and autonomous execution.
  468. Press `P` to switch plugins. Ships with 10 built-in:
  469. | Plugin | Description |
  470. |--------|-------------|
  471. | **void** | Plain agent session - no prompting or skills, task description prefilled in input |
  472. | **agtx** (default) | Built-in workflow with skills and prompts for each phase |
  473. | **agtx-terse** | Token-efficient workflow - same workflow with compressed output and minimal tokens |
  474. | **gsd** | [Get Shit Done](https://github.com/fynnfluegge/get-shit-done-cc) - structured spec-driven development with interactive planning |
  475. | **spec-kit** | [Spec-Driven Development](https://github.com/github/spec-kit) by GitHub - specifications become executable artifacts |
  476. | **openspec** | [OpenSpec](https://github.com/Fission-AI/OpenSpec) - lightweight AI-guided specification framework |
  477. | **bmad** | [BMAD Method](https://github.com/bmad-code-org/BMAD-METHOD) - AI-driven agile development with structured phases |
  478. | **superpowers** | [Superpowers](https://github.com/obra/superpowers) - brainstorming, plans, TDD, subagent-driven development |
  479. | **oh-my-claudecode** | [oh-my-claudecode](https://github.com/Yeachan-Heo/oh-my-claudecode) - multi-agent orchestration with 37 skills and 22 specialized agents |
  480. | **agent-skills** | [Agent Skills](https://github.com/addyosmani/agent-skills) - production-grade engineering skills covering the full spec-to-ship lifecycle |
  481. ### Agent Compatibility
  482. Commands are written once in canonical format and automatically translated per agent:
  483. | Canonical (plugin.toml) | Claude / Gemini | Codex | OpenCode | Cursor | Grok | Antigravity | pi |
  484. |--------------------------|-----------------|-------|----------|--------|------|-------------|----|
  485. | `/agtx:plan` | `/agtx:plan` | `$agtx-plan` | `/agtx-plan` | `/agtx-plan` | `/agtx-plan` | `/agtx-plan` | `/skill:agtx-plan` |
  486. | | Claude | Codex | Gemini | OpenCode | Cursor | Copilot | Grok | Antigravity | pi |
  487. |--|:------:|:-----:|:------:|:--------:|:------:|:-------:|:----:|:-----------:|:--:|
  488. | **agtx** | โœ… | โœ… | โœ… | โœ… | โœ… | ๐ŸŸก | โœ… | โœ… | โœ… |
  489. | **gsd** | โœ… | โœ… | โœ… | โœ… | โœ… | โŒ | โœ… | โŒ | โŒ |
  490. | **spec-kit** | โœ… | โœ… | โœ… | โœ… | โœ… | ๐ŸŸก | โœ… | โœ… | ๐ŸŸก |
  491. | **openspec** | โœ… | โœ… | โœ… | โœ… | โœ… | ๐ŸŸก | โœ… | โœ… | ๐ŸŸก |
  492. | **bmad** | โœ… | โœ… | โœ… | โœ… | โœ… | ๐ŸŸก | โœ… | โœ… | ๐ŸŸก |
  493. | **superpowers** | โœ… | โŒ | โŒ | โŒ | โŒ | โŒ | โŒ | โŒ | โŒ |
  494. | **oh-my-claudecode** | โœ… | โŒ | โŒ | โŒ | โŒ | โŒ | โŒ | โŒ | โŒ |
  495. | **agent-skills** | โœ… | ๐ŸŸก | ๐ŸŸก | ๐ŸŸก | ๐ŸŸก | ๐ŸŸก | ๐ŸŸก | ๐ŸŸก | ๐ŸŸก |
  496. | **void** | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… |
  497. โœ… Skills, commands, and prompts fully supported ยท ๐ŸŸก Prompt only, no interactive skill support ยท โŒ Not supported
  498. <details>
  499. <summary><b>Creating a Plugin</b></summary>
  500. Place your plugin at `.agtx/plugins/<name>/plugin.toml` in your project root (or `~/.config/agtx/plugins/<name>/plugin.toml` for global use). It will appear in the plugin selector automatically.
  501. **Minimal example** โ€” a plugin that uses custom slash commands:
  502. ```toml
  503. name = "my-plugin"
  504. description = "My custom workflow"
  505. [commands]
  506. research = "/my-plugin:research {task}"
  507. planning = "/my-plugin:plan"
  508. running = "/my-plugin:execute"
  509. review = "/my-plugin:review"
  510. [prompts]
  511. planning = "Task: {task}"
  512. ```
  513. **Full reference** with all available fields:
  514. ```toml
  515. name = "my-plugin"
  516. description = "My custom workflow"
  517. # Shell command to run in the worktree after creation, before the agent starts.
  518. # {agent} is replaced with the agent name (claude, codex, gemini, etc.)
  519. init_script = "npm install --prefix .my-plugin --{agent}"
  520. # Restrict to specific agents (empty or omitted = all agents supported)
  521. supported_agents = ["claude", "codex", "gemini", "opencode"]
  522. # Extra directories to copy from project root into each worktree.
  523. # Agent config dirs (.claude, .gemini, .codex, .github/agents, .config/opencode)
  524. # are always copied automatically.
  525. copy_dirs = [".my-plugin"]
  526. # Individual files to copy from project root into each worktree.
  527. # Merged with project-level copy_files from .agtx/config.toml.
  528. copy_files = ["PROJECT.md", "REQUIREMENTS.md"]
  529. # When true, enables Review โ†’ Planning transition via the `p` key.
  530. # Each cycle increments the phase counter ({phase} placeholder).
  531. # Use this for multi-milestone workflows (e.g. plan โ†’ execute โ†’ review โ†’ next milestone).
  532. cyclic = false
  533. # Artifact files that signal phase completion.
  534. # When detected, the task shows a checkmark instead of the spinner.
  535. # Supports * wildcard for one directory level (e.g. "specs/*/plan.md").
  536. # Use {phase} for cycle-aware paths (replaced with the current cycle number).
  537. # Omitted phases show no completion indicator.
  538. [artifacts]
  539. research = ".my-plugin/research.md"
  540. planning = ".my-plugin/{phase}/plan.md"
  541. running = ".my-plugin/{phase}/summary.md"
  542. review = ".my-plugin/{phase}/review.md"
  543. # Slash commands sent to the agent via tmux for each phase.
  544. # Written in canonical format (Claude/Gemini style): /namespace:command
  545. # Automatically transformed per agent:
  546. # Claude/Gemini: /my-plugin:plan (unchanged)
  547. # OpenCode: /my-plugin-plan (colon -> hyphen)
  548. # Codex: $my-plugin-plan (slash -> dollar, colon -> hyphen)
  549. # Cursor/Grok/Antigravity: /my-plugin-plan (colon -> hyphen)
  550. # Omitted phases fall back to agent-native agtx skill invocation
  551. # (e.g. /agtx:plan for Claude, $agtx-plan for Codex).
  552. # Set to "" to skip sending a command for that phase.
  553. # Use {phase} for cycle-aware commands (replaced with the current cycle number).
  554. # Use {task} to inline the task description.
  555. [commands]
  556. preresearch = "/my-plugin:research {task}" # Used only when no research artifacts exist yet
  557. research = "/my-plugin:discuss {phase}"
  558. planning = "/my-plugin:plan {phase}"
  559. running = "/my-plugin:execute {phase}"
  560. review = "/my-plugin:review {phase}"
  561. # Prompt templates sent as task content after the command.
  562. # {task} = task title + description, {task_id} = unique task ID, {phase} = cycle number.
  563. # Omitted phases send no prompt (the skill/command handles instructions).
  564. [prompts]
  565. research = "Task: {task}"
  566. # Text patterns to wait for in the tmux pane before sending the prompt.
  567. # Useful when a command triggers an interactive prompt that must appear first.
  568. # Polls every 500ms, times out after 5 minutes.
  569. [prompt_triggers]
  570. research = "What do you want to build?"
  571. # Files/dirs to copy from worktree back to project root after a phase completes.
  572. # Triggered automatically when the phase artifact is detected (spinner โ†’ checkmark).
  573. # Useful for sharing research artifacts (specs, plans) across worktrees.
  574. [copy_back]
  575. research = ["PROJECT.md", "REQUIREMENTS.md", ".my-plugin"]
  576. # Auto-dismiss interactive prompts that appear before the prompt trigger.
  577. # Each rule fires when ALL detect patterns are present and the pane is stable.
  578. # Response is newline-separated keystrokes (e.g. "2\nEnter" sends "2" then Enter).
  579. [[auto_dismiss]]
  580. detect = ["Map codebase", "Skip mapping", "Enter to select"]
  581. response = "2\nEnter"
  582. ```
  583. **What happens at each phase transition:**
  584. 1. The **command** is sent to the agent via tmux (e.g., `/my-plugin:plan`)
  585. 2. If a **prompt_trigger** is set, agtx waits for that prompt trigger to appear in the tmux pane
  586. 3. The **prompt** is sent with `{task}`, `{task_id}`, and `{phase}` replaced
  587. 4. agtx polls for the **artifact** file โ€” when found, the spinner becomes a checkmark
  588. 5. If **copy_back** is configured, artifacts are copied from worktree to project root on completion
  589. 6. If the agent appears idle (no output for 15s), the spinner becomes a pause icon
  590. **Phase gating:** Whether a phase can be entered directly from Backlog is derived from the plugin config. If a phase's command or prompt contains `{task}`, it can receive task context and is accessible from Backlog. If neither has `{task}`, the phase depends on a prior phase and is blocked until that artifact exists. For example, OpenSpec's `/opsx:propose {task}` allows direct Backlog โ†’ Planning, but `/opsx:apply` (no `{task}`) blocks Backlog โ†’ Running until proposal artifacts exist.
  591. **Preresearch fallback:** When pressing `R` on a task, if `preresearch` is configured and no research artifacts from `copy_back` exist in the project root yet, the `preresearch` command is used instead of `research`. This lets plugins run a one-time project setup (e.g. `/gsd:new-project`) before switching to the regular research command for subsequent tasks. If the plugin has no research command at all (e.g. OpenSpec), pressing `R` shows a warning.
  592. **Cyclic workflows:** When `cyclic = true`, pressing `p` in Review moves the task back to Planning with an incremented phase counter. This enables multi-milestone workflows where each cycle (plan โ†’ execute โ†’ review) produces artifacts in a separate `{phase}` directory.
  593. **Custom skills:** If your plugin provides its own skill files, place them in the plugin directory:
  594. ```
  595. .agtx/plugins/my-plugin/
  596. โ”œโ”€โ”€ plugin.toml
  597. โ””โ”€โ”€ skills/
  598. โ”œโ”€โ”€ agtx-plan/SKILL.md
  599. โ”œโ”€โ”€ agtx-execute/SKILL.md
  600. โ””โ”€โ”€ agtx-review/SKILL.md
  601. ```
  602. These override the built-in agtx skills and are automatically deployed to each agent's native discovery path (`.claude/commands/`, `.codex/skills/`, `.gemini/commands/`, etc.) in every worktree.
  603. </details>
  604. ## How It Works
  605. ### Architecture
  606. ```
  607. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  608. โ”‚ agtx TUI โ”‚
  609. โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
  610. โ”‚ Backlog โ”‚ Planning โ”‚ Running โ”‚ Review โ”‚ Done โ”‚
  611. โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚
  612. โ”‚ โ”‚Task1โ”‚ โ”‚ โ”‚Task2โ”‚ โ”‚ โ”‚Task3โ”‚ โ”‚ โ”‚Task4โ”‚ โ”‚ โ”‚
  613. โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚
  614. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  615. โ”‚ โ”‚
  616. โ–ผ โ–ผ
  617. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  618. โ”‚ tmux server "agtx" โ”‚
  619. โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
  620. โ”‚ โ”‚ Session: "my-project" โ”‚ โ”‚
  621. โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚
  622. โ”‚ โ”‚ โ”‚Window: โ”‚ โ”‚Window: โ”‚ โ”‚Window: โ”‚ โ”‚ โ”‚
  623. โ”‚ โ”‚ โ”‚task2 โ”‚ โ”‚task3 โ”‚ โ”‚task4 โ”‚ โ”‚ โ”‚
  624. โ”‚ โ”‚ โ”‚(Claude)โ”‚ โ”‚(Claude)โ”‚ โ”‚(Claude)โ”‚ โ”‚ โ”‚
  625. โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚
  626. โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
  627. โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
  628. โ”‚ โ”‚ Session: "other-project" โ”‚ โ”‚
  629. โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚
  630. โ”‚ โ”‚ โ”‚ Window: โ”‚ โ”‚ โ”‚
  631. โ”‚ โ”‚ โ”‚ some_other_task โ”‚ โ”‚ โ”‚
  632. โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚
  633. โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
  634. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  635. โ”‚ โ”‚
  636. โ–ผ โ–ผ
  637. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
  638. โ”‚ Git Worktrees โ”‚
  639. โ”‚ .agtx/worktrees/task2/ โ”‚
  640. โ”‚ .agtx/worktrees/task3/ โ”‚
  641. โ”‚ .agtx/worktrees/task4/ โ”‚
  642. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  643. ```
  644. ### Tmux Structure
  645. - **Server**: All sessions run on a dedicated tmux server named `agtx`
  646. - **Sessions**: Each project gets its own tmux session (named after the project)
  647. - **Windows**: Each task gets its own window within the project's session
  648. ```bash
  649. # List all sessions
  650. tmux -L agtx list-sessions
  651. # List all windows across sessions
  652. tmux -L agtx list-windows -a
  653. # Attach to the agtx server
  654. tmux -L agtx attach
  655. ```
  656. ### Data Storage
  657. - **Database**: `~/Library/Application Support/agtx/` (macOS) or `~/.config/agtx/` (Linux)
  658. - Config: `~/.config/agtx/config.toml`
  659. - **Worktrees**: `.agtx/worktrees/` in each project
  660. - **Tmux**: Dedicated server `agtx` with per-project sessions
  661. ## Container Sandbox
  662. Run agtx in an isolated container (Docker or Podman, including rootless Podman) so agents can only touch the project you pass in โ€” no access to the rest of your home directory, credentials are read-only, and any files the agent creates in the project are owned by your host user.
  663. ```bash
  664. # Run agtx on a project
  665. ./docker/sandbox.sh path/to/your-project
  666. # Or from inside the project directory
  667. ./docker/sandbox.sh
  668. ```
  669. The sandbox:
  670. - Mounts only the target project as writable; everything else on the host is inaccessible
  671. - Copies `~/.claude` credentials read-only at startup so they are never written back to the host
  672. - Runs as a non-root user whose UID/GID matches your host user (files created inside the container appear correctly owned on the host)
  673. - Stores agtx state in named container volumes (persists across runs, isolated from your host's agtx data)
  674. - Pre-accepts the bypass permissions prompt, which is appropriate in an isolated container
  675. > [!NOTE]
  676. > Requires [Docker Engine](https://docs.docker.com/engine/install/) (Linux), [Docker Desktop](https://docs.docker.com/desktop/) (macOS/Windows), or [Podman](https://podman.io/docs/installation) (rootful or rootless). The image is built automatically on first run and cached for subsequent runs. Set `AGTX_CONTAINER_RUNTIME=podman` (or `=docker`) to pick a runtime explicitly when both are installed.
  677. ## MCP Server
  678. The agtx MCP server (`agtx mcp-serve`) exposes the board to any coding agent session via the [Model Context Protocol](https://modelcontextprotocol.io). Used by the orchestrator agent, the oneshot skill, and the brainstorm & sweep skills.
  679. ### Modes
  680. | Mode | Command | Used by |
  681. |------|---------|---------|
  682. | **Global** | `agtx mcp-serve` | Brainstorm, sweep and oneshot skills โ€” works across all projects |
  683. | **Project-scoped** | `agtx mcp-serve <path>` | Orchestrator โ€” bound to one project at startup |
  684. In global mode all tools require a `project_id` parameter. Call `list_projects` first to resolve it.
  685. ### Tools
  686. | Tool | Description |
  687. |------|-------------|
  688. | `list_projects` | List all projects indexed in agtx |
  689. | `get_config` | The project's effective config, global and project merged โ€” e.g. whether `auto_trust` is on |
  690. | `list_tasks` | List tasks with their phase status, optionally filtered by status (descriptions on request) |
  691. | `get_task` | Get task details + `allowed_actions` for valid transitions |
  692. | `wait_for_board_change` | Block until a task needs attention, then return only what changed and the outcome of queued transitions |
  693. | `create_task` | Create a single backlog task |
  694. | `create_tasks_batch` | Batch-create tasks with index-based dependencies |
  695. | `update_task` | Modify a backlog task (title, description, deps) |
  696. | `delete_task` | Delete a backlog task |
  697. | `move_task` | Queue a phase transition, including `move_to_done_and_merge` |
  698. | `get_transition_status` | Check if a queued transition completed or errored |
  699. | `check_conflicts` | Non-destructive merge conflict check against default branch |
  700. | `get_notifications` | Fetch pending orchestrator notifications |
  701. | `read_pane_content` | Read the last N lines of a task's tmux pane |
  702. | `send_to_task` | Send a message to a task's agent (Planning, Running or Review) |
  703. ## Orchestrator Agent (Experimental)
  704. > Press `O` and walk away. Come back to changes ready to merge.
  705. The orchestrator is an AI agent that **drives other AI agents to completion**. You triage tasks into Planning or Running โ€” the orchestrator takes over from there, advancing each task through its phases until it lands in Review, ready for you to merge.
  706. ```bash
  707. agtx --experimental # then press O
  708. ```
  709. **What it does:**
  710. - Monitors tasks in Planning and Running
  711. - Advances tasks automatically as phases complete (Planning โ†’ Running โ†’ Review)
  712. - Respects plugin phase rules โ€” checks `allowed_actions` before each transition
  713. - Detects stuck tasks (idle for 1+ minute without a phase artifact) and reads the agent pane to diagnose the cause
  714. - Nudges stuck agents, answers CLI prompts automatically, or escalates to you with a reason when human input is needed
  715. **You triage. It executes.** Move tasks from Backlog into Planning or Running โ€” the orchestrator handles the rest. Merging is your call. For a session that owns the whole board, from planning the work to merging it, see the [Oneshot Skill](#oneshot-skill).
  716. ### MCP Integration
  717. The orchestrator communicates with agtx through the [Model Context Protocol (MCP)](https://modelcontextprotocol.io). agtx ships with a built-in MCP server (`agtx serve`) that exposes the kanban board as a set of tools over JSON-RPC via stdio.
  718. ```
  719. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€-โ” MCP (stdio) โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” SQLite โ”Œโ”€โ”€โ”€โ”€โ”€โ”
  720. โ”‚ Orchestrator โ”‚ โ†โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ†’ โ”‚ MCP Server โ”‚ โ†โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ†’ โ”‚ DB โ”‚
  721. โ”‚ (Claude Code)โ”‚ โ”‚ (agtx serve) โ”‚ โ””โ”€โ”€โ”ฌโ”€โ”€โ”˜
  722. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
  723. โ”‚ push-when-idle notifications โ”‚
  724. โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
  725. โ”‚ TUI (agtx) โ”‚ โ†โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€--โ”€โ”˜
  726. โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  727. ```
  728. **How it works:**
  729. 1. When you press `O`, the TUI registers the MCP server with the orchestrator agent via `claude mcp add-json --scope local`
  730. 2. The orchestrator receives phase completion notifications pushed to its tmux pane when idle
  731. 3. It reacts by calling `get_task` to check `allowed_actions`, then `move_task` to advance the task
  732. 4. The TUI processes the transition request, executes all side effects (agent switching, skill deployment, prompt sending), and updates the database
  733. 5. If a task has been idle for 1+ minute without a phase artifact, the orchestrator is notified โ€” it reads the pane with `read_pane_content`, then either nudges the agent with `send_to_task` or calls `move_task` with `escalate_to_user` to flag it for your attention
  734. 6. Escalated tasks show a `โš ` badge on the kanban board; opening the task popup shows the reason and dismisses the flag
  735. 7. MCP registration is cleaned up when the orchestrator is stopped
  736. ## Benchmark
  737. agtx includes a [SWE-bench Lite](https://www.swebench.com) benchmark runner that uses agtx itself as the agent orchestration layer โ€” driving coding agent workflows against 300 real GitHub bug-fix tasks via the MCP server.
  738. See **[benchmark/README.md](benchmark/README.md)** for setup, usage, bundled configs, and evaluation instructions.
  739. ## Contributing
  740. Contributions are welcome! Whether it's a bug fix, new plugin, agent integration, or documentation improvement.
  741. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full guide. Here's the short version:
  742. ```bash
  743. # Fork & clone
  744. git clone https://github.com/<you>/agtx && cd agtx
  745. # Build & test
  746. cargo build && cargo test --features test-mocks
  747. ```
  748. ### Good First Contributions
  749. Not sure where to start? Here are some ideas:
  750. - **Write a plugin** โ€” A single `plugin.toml` is all you need. See [Creating a Plugin](#plugins) for the full reference
  751. - **Add a new agent** โ€” Integrate your favorite AI coding CLI. See the [architecture docs](CLAUDE.md) for how agents are structured
  752. - **Improve documentation** โ€” Found something unclear? Help others by improving it
  753. - **Report bugs** โ€” Open an [issue](https://github.com/fynnfluegge/agtx/issues). Reproduction steps are always appreciated
  754. - **Browse open issues** โ€” Check the [`good first issue`](https://github.com/fynnfluegge/agtx/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) label for beginner-friendly tasks
  755. ## Development
  756. See [CLAUDE.md](CLAUDE.md) for full architecture docs and development patterns.
  757. ```bash
  758. # Build
  759. cargo build
  760. # Run tests (includes mock-based tests)
  761. cargo test --features test-mocks
  762. # Build release
  763. cargo build --release
  764. ```
benchmark/swebench/benchmark.py ยท Bundled script ยท 2,152 lines ยท 1 finding marked
  1. #!/usr/bin/env python3
  2. """
  3. SWE-bench Lite benchmark runner for agtx.
  4. Drives agtx via its MCP server (JSON-RPC 2.0 over stdio) to run coding agent
  5. workflows against SWE-bench Lite (300 tasks), collects git diff patches when
  6. tasks reach Review, uses tokscale for token/cost metrics (all agents), and
  7. writes SWE-bench-compatible predictions.jsonl + results.json.
  8. A config.toml is required. It is written to .agtx/config.toml in each repo
  9. before the TUI starts โ€” this is how agent, plugin, worktree_dir, etc. are
  10. configured. The script reads plugin and default_agent back from the config to
  11. drive artifact polling.
  12. Token/cost metrics use tokscale (https://github.com/junhoyeo/tokscale) if
  13. available. Supports claude, codex, gemini, and 20+ other agents. Install with:
  14. npm install -g tokscale
  15. If tokscale is not installed, cost fields are null.
  16. Usage:
  17. python benchmark.py --config my_config.toml --instances 1
  18. python benchmark.py --config my_config.toml --concurrency 2 --instances 10
  19. python benchmark.py --config my_config.toml --instance-ids sympy__sympy-20590
  20. python benchmark.py --config my_config.toml --instance-ids astropy__astropy-12907 --sandbox --verbose --agtx ../../target/release/agtx
  21. Example config.toml:
  22. default_agent = "claude"
  23. workflow_plugin = "agtx"
  24. worktree_dir = ".agtx/worktrees"
  25. """
  26. from __future__ import annotations
  27. import argparse
  28. import json
  29. import os
  30. import re
  31. import shlex
  32. import subprocess
  33. import sys
  34. import tempfile
  35. import threading
  36. import time
  37. from concurrent.futures import ThreadPoolExecutor, as_completed
  38. from datetime import datetime
  39. from pathlib import Path
  40. from typing import Any
  41. try:
  42. from datasets import load_dataset
  43. from tqdm import tqdm
  44. except ImportError:
  45. print("Missing dependencies. Run: pip install -r requirements.txt", file=sys.stderr)
  46. sys.exit(1)
  47. try:
  48. import tomllib # Python 3.11+
  49. except ImportError:
  50. try:
  51. import tomli as tomllib # fallback
  52. except ImportError:
  53. # Minimal TOML parser for the subset we need (key = "value" lines only)
  54. tomllib = None # type: ignore
  55. # ---------------------------------------------------------------------------
  56. # Config loading
  57. # ---------------------------------------------------------------------------
  58. def load_config_toml(path: Path) -> dict:
  59. """Load a TOML config file, returning a dict of its keys."""
  60. content = path.read_text()
  61. if tomllib is not None:
  62. return tomllib.loads(content)
  63. # Fallback: parse simple key = "value" lines and [section] tables
  64. result: dict = {}
  65. current_section: dict | None = None
  66. current_section_name: str | None = None
  67. for line in content.splitlines():
  68. line = line.strip()
  69. if not line or line.startswith("#"):
  70. continue
  71. if line.startswith("[") and line.endswith("]"):
  72. current_section_name = line[1:-1].strip()
  73. current_section = {}
  74. result[current_section_name] = current_section
  75. continue
  76. if "=" in line:
  77. k, _, v = line.partition("=")
  78. k = k.strip()
  79. v = v.strip().strip('"').strip("'")
  80. if current_section is not None:
  81. current_section[k] = v
  82. else:
  83. result[k] = v
  84. return result
  85. def running_agent(config: dict) -> str:
  86. """Return the agent that will handle the running phase."""
  87. return (
  88. config.get("agents", {}).get("running")
  89. or config.get("default_agent")
  90. or "claude"
  91. )
  92. # ---------------------------------------------------------------------------
  93. # MCP Client
  94. # ---------------------------------------------------------------------------
  95. class McpError(Exception):
  96. pass
  97. class McpClient:
  98. """Raw JSON-RPC 2.0 MCP client over subprocess stdin/stdout."""
  99. def __init__(self, agtx_bin: str, repo_path: str, container_id: str | None = None):
  100. self._seq = 0
  101. self._lock = threading.Lock()
  102. if container_id:
  103. cmd = [
  104. "docker", "exec", "-i", container_id,
  105. "/bin/bash", "-c",
  106. "export HOME=/home/bench"
  107. " && source /opt/miniconda3/bin/activate && conda activate testbed"
  108. " && agtx mcp-serve /testbed",
  109. ]
  110. else:
  111. cmd = [agtx_bin, "mcp-serve", repo_path]
  112. self._proc = subprocess.Popen(
  113. cmd,
  114. stdin=subprocess.PIPE,
  115. stdout=subprocess.PIPE,
  116. stderr=subprocess.DEVNULL,
  117. text=True,
  118. bufsize=1,
  119. )
  120. self._handshake()
  121. def _next_id(self) -> int:
  122. with self._lock:
  123. self._seq += 1
  124. return self._seq
  125. def _send(self, msg: dict) -> None:
  126. line = json.dumps(msg) + "\n"
  127. self._proc.stdin.write(line)
  128. self._proc.stdin.flush()
  129. # A reply should take milliseconds. This is not a latency budget, it is a
  130. # deadlock guard: `readline()` on a live-but-silent server blocks forever, and
  131. # no --phase-timeout can rescue it because the block is inside the MCP call
  132. # rather than inside the artifact poll that the timeout governs.
  133. RECV_TIMEOUT_SECONDS = 120
  134. def _recv(self) -> dict:
  135. import selectors
  136. with selectors.DefaultSelector() as sel:
  137. sel.register(self._proc.stdout, selectors.EVENT_READ)
  138. if not sel.select(timeout=self.RECV_TIMEOUT_SECONDS):
  139. raise McpError(
  140. f"MCP server did not respond within {self.RECV_TIMEOUT_SECONDS}s "
  141. "(agent may be parked on an interactive prompt or a usage limit)"
  142. )
  143. line = self._proc.stdout.readline()
  144. if not line:
  145. raise McpError("MCP server closed connection")
  146. return json.loads(line)
  147. def _request(self, method: str, params: dict) -> Any:
  148. req_id = self._next_id()
  149. self._send({"jsonrpc": "2.0", "id": req_id, "method": method, "params": params})
  150. while True:
  151. msg = self._recv()
  152. if msg.get("id") == req_id:
  153. if "error" in msg:
  154. raise McpError(f"MCP error: {msg['error']}")
  155. result = msg.get("result", {})
  156. content = result.get("content", [])
  157. if content and content[0].get("type") == "text":
  158. return json.loads(content[0]["text"])
  159. if result.get("isError"):
  160. raise McpError(f"Tool returned error: {content}")
  161. return result
  162. # ignore notifications
  163. def _notify(self, method: str, params: dict) -> None:
  164. self._send({"jsonrpc": "2.0", "method": method, "params": params})
  165. def _handshake(self) -> None:
  166. req_id = self._next_id()
  167. self._send({
  168. "jsonrpc": "2.0",
  169. "id": req_id,
  170. "method": "initialize",
  171. "params": {
  172. "protocolVersion": "2024-11-05",
  173. "capabilities": {},
  174. "clientInfo": {"name": "swebench-runner", "version": "1.0"},
  175. },
  176. })
  177. while True:
  178. msg = self._recv()
  179. if msg.get("id") == req_id:
  180. break
  181. self._notify("notifications/initialized", {})
  182. def call(self, tool: str, **kwargs) -> Any:
  183. params = {k: v for k, v in kwargs.items() if v is not None}
  184. return self._request("tools/call", {"name": tool, "arguments": params})
  185. def close(self) -> None:
  186. try:
  187. self._proc.stdin.close()
  188. self._proc.wait(timeout=5)
  189. except Exception:
  190. self._proc.kill()
  191. # ---------------------------------------------------------------------------
  192. # Repo Setup
  193. # ---------------------------------------------------------------------------
  194. def setup_repo(instance: dict, workdir: str, config_path: Path, verbose: bool = False, smoke_test: bool = False) -> Path:
  195. """
  196. Clone the repo at base_commit and write .agtx/config.toml.
  197. Returns repo path. Safe to call again on an existing clone (resumable).
  198. In smoke test mode, initializes an empty git repo instead of cloning.
  199. """
  200. instance_id = instance["instance_id"]
  201. if smoke_test:
  202. repo_path = Path(workdir) / instance_id
  203. if not repo_path.exists():
  204. repo_path.mkdir(parents=True)
  205. subprocess.run(["git", "init"], cwd=repo_path, capture_output=True, check=True)
  206. subprocess.run(
  207. ["git", "commit", "--allow-empty", "-m", "init"],
  208. cwd=repo_path,
  209. capture_output=True,
  210. check=True,
  211. env={**__import__("os").environ, "GIT_AUTHOR_NAME": "smoke", "GIT_AUTHOR_EMAIL": "smoke@test", "GIT_COMMITTER_NAME": "smoke", "GIT_COMMITTER_EMAIL": "smoke@test"},
  212. )
  213. if verbose:
  214. print(f" [setup] Initialized empty git repo at {repo_path}", file=sys.stderr)
  215. _write_agtx_config(repo_path, config_path, "HEAD")
  216. return repo_path
  217. repo_url = f"https://github.com/{instance['repo']}.git"
  218. base_commit = instance["base_commit"]
  219. repo_path = Path(workdir) / instance_id
  220. if repo_path.exists():
  221. # Check if the repo is clean: HEAD must be at base_commit and no task/* branches.
  222. head_result = subprocess.run(
  223. ["git", "rev-parse", "HEAD"],
  224. cwd=repo_path,
  225. capture_output=True,
  226. text=True,
  227. )
  228. branch_result = subprocess.run(
  229. ["git", "branch", "--list", "task/*"],
  230. cwd=repo_path,
  231. capture_output=True,
  232. text=True,
  233. )
  234. head_ok = head_result.returncode == 0 and head_result.stdout.strip() == base_commit
  235. no_task_branches = branch_result.returncode != 0 or not branch_result.stdout.strip()
  236. if not head_ok:
  237. # Wrong commit โ€” need a clean clone
  238. if verbose:
  239. print(
  240. f" [setup] Stale clone (wrong commit), removing {repo_path}",
  241. file=sys.stderr,
  242. )
  243. subprocess.run(["rm", "-rf", str(repo_path)], check=True)
  244. else:
  245. if not no_task_branches:
  246. # Just delete leftover task branches, no need to re-clone
  247. for branch in branch_result.stdout.strip().splitlines():
  248. branch = branch.strip()
  249. if branch:
  250. subprocess.run(
  251. ["git", "branch", "-D", branch],
  252. cwd=repo_path,
  253. capture_output=True,
  254. )
  255. if verbose:
  256. print(f" [setup] Cleaned stale task branches, reusing {repo_path}", file=sys.stderr)
  257. else:
  258. if verbose:
  259. print(f" [setup] Using cached repo at {repo_path}", file=sys.stderr)
  260. _write_agtx_config(repo_path, config_path, base_commit)
  261. return repo_path
  262. if verbose:
  263. print(f" [setup] Cloning {repo_url} โ†’ {repo_path}", file=sys.stderr)
  264. repo_path.mkdir(parents=True, exist_ok=True)
  265. subprocess.run(
  266. ["git", "clone", "--quiet", "--depth=1", repo_url, str(repo_path)],
  267. check=True,
  268. capture_output=True,
  269. timeout=300,
  270. )
  271. if verbose:
  272. print(f" [setup] Checking out {base_commit}", file=sys.stderr)
  273. fetch_result = subprocess.run(
  274. ["git", "fetch", "--depth=1", "origin", base_commit],
  275. cwd=repo_path,
  276. capture_output=True,
  277. text=True,
  278. timeout=120,
  279. )
  280. if fetch_result.returncode != 0:
  281. subprocess.run(
  282. ["git", "fetch", "--unshallow"],
  283. cwd=repo_path,
  284. capture_output=True,
  285. timeout=600,
  286. )
  287. subprocess.run(
  288. ["git", "checkout", base_commit],
  289. cwd=repo_path,
  290. check=True,
  291. capture_output=True,
  292. timeout=60,
  293. )
  294. _write_agtx_config(repo_path, config_path, base_commit)
  295. return repo_path
  296. def _write_agtx_config(repo_path: Path, config_path: Path, base_commit: str) -> None:
  297. """
  298. Write config.toml and wipe any stale agtx state (old worktrees, task DB, old artifacts).
  299. This ensures each benchmark run starts clean even if the repo clone is reused.
  300. """
  301. agtx_dir = repo_path / ".agtx"
  302. # Remove stale worktrees and the project DB so the TUI starts fresh
  303. for stale in ["worktrees", "db"]:
  304. stale_path = agtx_dir / stale
  305. if stale_path.exists():
  306. subprocess.run(["rm", "-rf", str(stale_path)], check=True)
  307. agtx_dir.mkdir(exist_ok=True)
  308. dest = agtx_dir / "config.toml"
  309. # Insert base_branch = "<commit>" before any [section] headers so it stays
  310. # a top-level key. Appending at the end would put it inside the last section
  311. # (e.g. [agents]) and TOML would parse it as agents.base_branch instead.
  312. content = config_path.read_text()
  313. base_branch_line = f'base_branch = "{base_commit}"\n'
  314. # Remove any existing base_branch line first (idempotent)
  315. lines = [l for l in content.splitlines(keepends=True) if not l.startswith("base_branch =")]
  316. # Find the first [section] line and insert before it; if none, append at end
  317. insert_at = next((i for i, l in enumerate(lines) if l.startswith("[")), len(lines))
  318. lines.insert(insert_at, base_branch_line)
  319. dest.write_text("".join(lines))
  320. # Prune stale git worktree registrations and delete old task branches.
  321. # git worktree prune won't remove registrations whose gitdir is gone until the
  322. # grace period expires (default 3 months). Delete .git/worktrees/ directly so
  323. # the task/* branches become deletable immediately.
  324. git_worktrees_meta = repo_path / ".git" / "worktrees"
  325. if git_worktrees_meta.exists():
  326. subprocess.run(["rm", "-rf", str(git_worktrees_meta)], check=True)
  327. subprocess.run(["git", "worktree", "prune"], cwd=repo_path, capture_output=True)
  328. result = subprocess.run(
  329. ["git", "branch", "--list", "task/*"],
  330. cwd=repo_path, capture_output=True, text=True,
  331. )
  332. for branch in result.stdout.splitlines():
  333. branch = branch.strip().lstrip("*").strip()
  334. if branch:
  335. subprocess.run(
  336. ["git", "branch", "-D", branch],
  337. cwd=repo_path, capture_output=True,
  338. )
  339. # Ensure .agtx/ is ignored locally without touching .gitignore (which is a tracked file).
  340. # .git/info/exclude is a local-only ignore file โ€” changes never appear in git diff.
  341. exclude_file = repo_path / ".git" / "info" / "exclude"
  342. marker = ".agtx/"
  343. existing = exclude_file.read_text() if exclude_file.exists() else ""
  344. if marker not in existing:
  345. with exclude_file.open("a") as f:
  346. f.write(f"\n# agtx benchmark artifacts\n{marker}\n")
  347. def start_tui_in_tmux(slug: str, repo_path: Path, agtx_bin: str, verbose: bool = False) -> None:
  348. """Start an agtx TUI instance in a detached tmux session on the swebench server."""
  349. # Kill any stale session with the same slug on the swebench server
  350. subprocess.run(
  351. ["tmux", "-L", "swebench", "kill-session", "-t", slug],
  352. capture_output=True,
  353. )
  354. # Also kill the agtx-server session for this repo (named after the repo dir)
  355. # so the new TUI doesn't inherit stale task windows from a previous run
  356. repo_session = repo_path.name # e.g. "astropy__astropy-12907"
  357. subprocess.run(
  358. ["tmux", "-L", "agtx", "kill-session", "-t", repo_session],
  359. capture_output=True,
  360. )
  361. if verbose:
  362. print(f" [tmux] Starting TUI session '{slug}' for {repo_path}", file=sys.stderr)
  363. result = subprocess.run(
  364. [
  365. "tmux", "-L", "swebench",
  366. "new-session", "-d", "-s", slug,
  367. f"{agtx_bin} {repo_path}",
  368. ],
  369. capture_output=True,
  370. text=True,
  371. )
  372. if result.returncode != 0:
  373. raise RuntimeError(f"tmux new-session failed: {result.stderr.strip()}")
  374. def kill_tmux_session(slug: str, repo_path: Path | None = None) -> None:
  375. subprocess.run(
  376. ["tmux", "-L", "swebench", "kill-session", "-t", slug],
  377. capture_output=True,
  378. )
  379. if repo_path is not None:
  380. subprocess.run(
  381. ["tmux", "-L", "agtx", "kill-session", "-t", repo_path.name],
  382. capture_output=True,
  383. )
  384. # ---------------------------------------------------------------------------
  385. # Docker helpers
  386. # ---------------------------------------------------------------------------
  387. def docker_image_name(instance_id: str) -> str:
  388. """Convert an instance_id to its SWE-bench Docker image name.
  389. e.g. astropy__astropy-12907 โ†’ swebench/sweb.eval.x86_64.astropy_1776_astropy-12907:latest
  390. """
  391. image_slug = instance_id.replace("__", "_1776_")
  392. return f"swebench/sweb.eval.x86_64.{image_slug}:latest"
  393. TOOLS_IMAGE = "agtx/swebench-tools:latest"
  394. TOOLS_CONTAINER = "agtx-swebench-tools"
  395. TOOLS_VOLUME = "agtx-swebench-tools"
  396. def _tools_image_exists() -> bool:
  397. """Return True if the shared tools image has been built by prebake_images.py."""
  398. result = subprocess.run(
  399. ["docker", "image", "inspect", TOOLS_IMAGE],
  400. capture_output=True,
  401. )
  402. return result.returncode == 0
  403. def _tools_volume_exists() -> bool:
  404. result = subprocess.run(
  405. ["docker", "volume", "inspect", TOOLS_VOLUME],
  406. capture_output=True,
  407. )
  408. return result.returncode == 0
  409. def ensure_tools_volume(verbose: bool = False) -> None:
  410. """Create and populate the tools volume from the tools image if it doesn't exist.
  411. The volume contains tmux + libutempter (so SWE-bench containers don't need apt),
  412. node, node_modules, and rtk. Mounting read-only at /tools avoids copying the large
  413. Node stack into instance containers' writable layers.
  414. """
  415. if _tools_volume_exists():
  416. if verbose:
  417. print(f" [docker] Tools volume '{TOOLS_VOLUME}' already exists.", file=sys.stderr)
  418. return
  419. if verbose:
  420. print(f" [docker] Creating tools volume '{TOOLS_VOLUME}'...", file=sys.stderr)
  421. subprocess.run(["docker", "volume", "create", TOOLS_VOLUME], check=True, capture_output=True)
  422. # Populate volume by running a one-shot container that copies from the image into the volume.
  423. # libutempter is copied alongside tmux because SWE-bench containers don't have it.
  424. subprocess.run(
  425. [
  426. "docker", "run", "--rm", "--platform", "linux/amd64",
  427. "-v", f"{TOOLS_VOLUME}:/tools",
  428. TOOLS_IMAGE,
  429. "/bin/bash", "-c",
  430. "cp -a /usr/bin/tmux /usr/bin/node /usr/lib/node_modules /tools/"
  431. " && mkdir -p /tools/lib/x86_64-linux-gnu"
  432. " && cp /lib/x86_64-linux-gnu/libutempter.so.* /tools/lib/x86_64-linux-gnu/"
  433. " && cp /lib/x86_64-linux-gnu/libevent_core-2.1.so.* /tools/lib/x86_64-linux-gnu/",
  434. ],
  435. check=True, capture_output=not verbose,
  436. )
  437. if verbose:
  438. print(f" [docker] Tools volume ready.", file=sys.stderr)
  439. def start_tools_container(verbose: bool = False) -> str:
  440. """Start the shared tools container and return its container ID.
  441. The tools container is a long-running instance of agtx/swebench-tools:latest
  442. used as a source to copy tmux/Node/Claude Code binaries into benchmark containers.
  443. Starting it once and sharing it across all concurrent runs avoids redundant installs.
  444. """
  445. # Kill any stale tools container
  446. subprocess.run(["docker", "rm", "-f", TOOLS_CONTAINER], capture_output=True)
  447. result = subprocess.run(
  448. ["docker", "run", "-d", "--platform", "linux/amd64",
  449. "--name", TOOLS_CONTAINER, TOOLS_IMAGE, "sleep", "infinity"],
  450. check=True, capture_output=True, text=True,
  451. )
  452. container_id = result.stdout.strip()
  453. if verbose:
  454. print(f" [docker] Started tools container {container_id[:12]}", file=sys.stderr)
  455. return container_id
  456. def stop_tools_container(verbose: bool = False) -> None:
  457. subprocess.run(["docker", "rm", "-f", TOOLS_CONTAINER], capture_output=True)
  458. def _agtx_db_dir() -> str:
  459. """Return the host path to the agtx database directory."""
  460. if sys.platform == "darwin":
  461. return str(Path.home() / "Library" / "Application Support" / "agtx")
  462. return str(Path.home() / ".config" / "agtx")
  463. def _docker_env_overrides() -> list[str]:
  464. """Build -e flags for docker run to fix localhost URLs in Claude env settings.
  465. Reads ~/.claude/settings.json env overrides (read-only, never modified) and
  466. replaces http://localhost: with http://host.docker.internal: so proxies running
  467. on the macOS host are reachable from inside the container.
  468. Returns a flat list of ["-e", "KEY=VALUE", ...] suitable for subprocess args.
  469. """
  470. settings_path = Path.home() / ".claude" / "settings.json"
  471. if not settings_path.exists():
  472. return []
  473. try:
  474. data = json.loads(settings_path.read_text())
  475. env_overrides = data.get("env", {})
  476. except Exception:
  477. return []
  478. result = []
  479. for key, val in env_overrides.items():
  480. if isinstance(val, str) and "localhost" in val:
  481. val = val.replace("http://localhost:", "http://host.docker.internal:")
  482. val = val.replace("https://localhost:", "https://host.docker.internal:")
  483. result += ["-e", f"{key}={val}"]
  484. return result
  485. def start_docker_container(
  486. instance_id: str, slug: str, agtx_bin: str, verbose: bool = False
  487. ) -> str:
  488. """Pull the SWE-bench image and start a long-running container. Returns container_id."""
  489. container_name = f"swebench-{slug}"
  490. image = docker_image_name(instance_id)
  491. # Kill any stale container with the same name
  492. subprocess.run(["docker", "rm", "-f", container_name], capture_output=True)
  493. if verbose:
  494. print(f" [docker] Pulling {image}...", file=sys.stderr)
  495. subprocess.run(
  496. ["docker", "pull", "--platform", "linux/amd64", image],
  497. check=True,
  498. capture_output=not verbose,
  499. )
  500. if verbose:
  501. print(f" [docker] Starting container {container_name}...", file=sys.stderr)
  502. # Mount the tools volume if available โ€” provides tmux, node, node_modules, rtk
  503. # without consuming writable layer space in the instance container.
  504. extra_mounts = []
  505. if _tools_volume_exists():
  506. extra_mounts = ["-v", f"{TOOLS_VOLUME}:/tools:ro"]
  507. if verbose:
  508. print(f" [docker] Mounting tools volume.", file=sys.stderr)
  509. run_result = subprocess.run(
  510. [
  511. "docker", "run", "-d",
  512. "--platform", "linux/amd64",
  513. "--name", container_name,
  514. "-v", f"{agtx_bin}:/usr/local/bin/agtx",
  515. ] + extra_mounts + [
  516. image,
  517. "sleep", "infinity",
  518. ],
  519. capture_output=True,
  520. text=True,
  521. check=True,
  522. )
  523. container_id = run_result.stdout.strip()
  524. if verbose:
  525. print(f" [docker] Container id: {container_id[:12]}", file=sys.stderr)
  526. return container_id
  527. def seed_agent_trust(container_id: str, agent: str, *, verbose: bool = False) -> None:
  528. """Record /testbed as trusted for `agent`, the way the user would by hand.
  529. Trust records are **path-keyed**, and the repo lands at /testbed inside the
  530. container. Copying the host's ~/.claude.json (below) therefore does *not*
  531. prevent the trust dialog: no entry matches that path. Until now the run
  532. depended on agtx clearing the dialog by reading the pane, which is no longer
  533. the default.
  534. Deterministic and done up front, because here a stuck dialog costs a whole
  535. instance run rather than one task.
  536. """
  537. py = {
  538. "claude": (
  539. "import json,os;p='/home/bench/.claude.json';"
  540. "d=json.load(open(p)) if os.path.exists(p) else {};"
  541. "d.setdefault('projects',{}).setdefault('/testbed',{})"
  542. "['hasTrustDialogAccepted']=True;"
  543. "json.dump(d,open(p,'w'))"
  544. ),
  545. "codex": (
  546. "import os;p='/home/bench/.codex/config.toml';"
  547. "os.makedirs(os.path.dirname(p),exist_ok=True);"
  548. "s=open(p).read() if os.path.exists(p) else '';"
  549. "s+='\\n[projects.\"/testbed\"]\\ntrust_level = \"trusted\"\\n' "
  550. "if '[projects.\"/testbed\"]' not in s else '';"
  551. "open(p,'w').write(s)"
  552. ),
  553. "gemini": (
  554. "import json,os;p='/home/bench/.gemini/trustedFolders.json';"
  555. "os.makedirs(os.path.dirname(p),exist_ok=True);"
  556. "d=json.load(open(p)) if os.path.exists(p) else {};"
  557. "d['/testbed']='TRUST_FOLDER';json.dump(d,open(p,'w'))"
  558. ),
  559. "antigravity": (
  560. "import json,os;p='/home/bench/.gemini/antigravity-cli/settings.json';"
  561. "os.makedirs(os.path.dirname(p),exist_ok=True);"
  562. "d=json.load(open(p)) if os.path.exists(p) else {};"
  563. "w=d.setdefault('trustedWorkspaces',[]);"
  564. "w.append('/testbed') if '/testbed' not in w else None;"
  565. "json.dump(d,open(p,'w'))"
  566. ),
  567. }.get(agent)
  568. # cursor and grok pass --trust at launch; opencode has no trust gate; copilot
  569. # is unmeasured. Nothing to seed, and nothing to guess at.
  570. if py is None:
  571. return
  572. if verbose:
  573. print(f" [docker] Seeding {agent} trust for /testbed...", file=sys.stderr)
  574. subprocess.run(
  575. ["docker", "exec", container_id, "python3", "-c", py],
  576. check=False, capture_output=not verbose,
  577. )
  578. def setup_container(
  579. container_id: str, config_path: Path, base_commit: str, verbose: bool = False,
  580. tools_container_id: str | None = None,
  581. extra_dirs: list[tuple[Path, str]] | None = None,
  582. sandbox_init: list[str] | None = None,
  583. ) -> None:
  584. """Write agtx config into the container and install tmux + Node + Claude Code + Grok.
  585. If tools_container_id is provided, binaries are copied from the shared tools
  586. container instead of being installed from the network (~3-5 min saved).
  587. extra_dirs is an optional list of (host_path, container_dest) pairs for copying
  588. additional files into the container (e.g. third-party plugin skill directories).
  589. sandbox_init is an optional list of shell commands to run inside the container
  590. after setup (e.g. ["rtk init -g"] to activate rtk hooks).
  591. """
  592. # Parse default_agent from the benchmark config so we can write a global agtx config
  593. # inside the container. Without it agtx shows the agent-selection wizard on startup.
  594. bench_config = load_config_toml(config_path)
  595. default_agent = bench_config.get("default_agent", "claude")
  596. # Copy Claude credentials into the container as snapshots (not mounts).
  597. # This avoids writing back to host files and lets us patch localhost URLs.
  598. # We only copy the credential/config files โ€” NOT ~/.claude/projects/ which can be
  599. # gigabytes of conversation history and would exhaust the container's disk space.
  600. home = Path.home()
  601. subprocess.run(
  602. ["docker", "exec", container_id, "/bin/bash", "-c",
  603. "mkdir -p /home/bench/.config/claude /home/bench/.claude"],
  604. check=True, capture_output=not verbose,
  605. )
  606. # ~/.config/claude/ โ€” API key and auth tokens
  607. config_claude = home / ".config" / "claude"
  608. if config_claude.exists():
  609. subprocess.run(
  610. ["docker", "cp", f"{config_claude}/.", f"{container_id}:/home/bench/.config/claude"],
  611. check=True, capture_output=not verbose,
  612. )
  613. # ~/.claude/ โ€” copy only top-level files (settings.json, credentials), skip subdirs
  614. # (projects/, worktrees/, etc.) which contain large conversation history
  615. claude_dir = home / ".claude"
  616. if claude_dir.exists():
  617. for item in claude_dir.iterdir():
  618. if item.is_file():
  619. subprocess.run(
  620. ["docker", "cp", str(item), f"{container_id}:/home/bench/.claude/{item.name}"],
  621. check=True, capture_output=not verbose,
  622. )
  623. # macOS keeps the Claude OAuth credentials in the login Keychain, not in
  624. # ~/.claude/.credentials.json, so a subscription login does not travel with the
  625. # file copy above and the container lands on "Not logged in ยท Please run /login".
  626. # Materialise the same JSON that Claude Code writes on Linux.
  627. #
  628. # The secret is piped through stdin: never written to a host temp file, and
  629. # never placed in argv where `ps` would expose it.
  630. if sys.platform == "darwin" and not (claude_dir / ".credentials.json").exists():
  631. creds = ""
  632. try:
  633. creds = subprocess.run(
  634. ["security", "find-generic-password", "-s", "Claude Code-credentials", "-w"],
  635. capture_output=True, text=True, check=True,
  636. ).stdout.strip()
  637. except (subprocess.CalledProcessError, FileNotFoundError):
  638. pass
  639. if creds:
  640. subprocess.run(
  641. ["docker", "exec", "-i", container_id, "/bin/bash", "-c",
  642. "umask 077 && cat > /home/bench/.claude/.credentials.json"],
  643. input=creds, text=True, check=True, capture_output=not verbose,
  644. )
  645. if verbose:
  646. print(" [docker] Claude credentials taken from the macOS Keychain.",
  647. file=sys.stderr)
  648. else:
  649. print(
  650. " [docker] WARNING: no Claude credentials found. On macOS the OAuth token "
  651. "lives in the Keychain; `security` could not read it (you may need to allow "
  652. "access when prompted). Set ANTHROPIC_API_KEY instead, or the agent will "
  653. "report 'Not logged in'.",
  654. file=sys.stderr,
  655. )
  656. # Copy ~/.claude.json as a snapshot so Claude skips the first-launch welcome screen.
  657. claude_json = home / ".claude.json"
  658. if claude_json.exists():
  659. subprocess.run(
  660. ["docker", "cp", str(claude_json), f"{container_id}:/home/bench/.claude.json"],
  661. check=True, capture_output=not verbose,
  662. )
  663. # Copy Grok credentials as snapshots so `grok` can authenticate inside the
  664. # container without a browser login. Only the small auth/config files are copied โ€”
  665. # ~/.grok/sessions/ and friends are skipped. No-op unless the host has ~/.grok.
  666. grok_home = home / ".grok"
  667. if grok_home.exists():
  668. subprocess.run(
  669. ["docker", "exec", container_id, "/bin/bash", "-c", "mkdir -p /home/bench/.grok"],
  670. check=True, capture_output=not verbose,
  671. )
  672. for name in ("auth.json", "config.toml"):
  673. item = grok_home / name
  674. if item.exists():
  675. subprocess.run(
  676. ["docker", "cp", str(item), f"{container_id}:/home/bench/.grok/{name}"],
  677. check=True, capture_output=not verbose,
  678. )
  679. seed_agent_trust(container_id, default_agent, verbose=verbose)
  680. # Copy extra directories into the container (e.g. third-party plugin skill dirs).
  681. for src, dst in (extra_dirs or []):
  682. if not src.exists():
  683. print(f" [docker] Warning: extra_dir source not found, skipping: {src}", file=sys.stderr)
  684. continue
  685. if verbose:
  686. print(f" [docker] Copying {src} โ†’ container:{dst}", file=sys.stderr)
  687. subprocess.run(
  688. ["docker", "exec", container_id, "/bin/bash", "-c", f"mkdir -p {dst}"],
  689. capture_output=True,
  690. )
  691. subprocess.run(
  692. ["docker", "cp", f"{src}/.", f"{container_id}:{dst}"],
  693. check=True, capture_output=not verbose,
  694. )
  695. # Patch localhost โ†’ host.docker.internal in settings.json inside the container.
  696. # Claude reads ANTHROPIC_BASE_URL from settings.json["env"] which overrides process env,
  697. # so we must fix it in the file itself (container-local copy only, host file untouched).
  698. subprocess.run(
  699. ["docker", "exec", container_id, "/bin/bash", "-c",
  700. r"sed -i 's|http://localhost:|http://host.docker.internal:|g'"
  701. r" /home/bench/.claude/settings.json 2>/dev/null || true"],
  702. capture_output=True,
  703. )
  704. # Drop empty auth keys from the container's settings.json. Because that `env`
  705. # block outranks process env, an `ANTHROPIC_AUTH_TOKEN: ""` left over from a
  706. # proxy setup can shadow both the Keychain credentials and a real
  707. # ANTHROPIC_API_KEY, and the agent reports "Not logged in" with valid auth
  708. # available. Host file untouched.
  709. subprocess.run(
  710. ["docker", "exec", container_id, "/bin/bash", "-c",
  711. "python3 - <<'ENDOFPATCH'\n"
  712. "import json\n"
  713. "p='/home/bench/.claude/settings.json'\n"
  714. "try:\n"
  715. " d=json.load(open(p))\n"
  716. "except Exception:\n"
  717. " raise SystemExit(0)\n"
  718. "env=d.get('env')\n"
  719. "if isinstance(env, dict):\n"
  720. " drop=[k for k,v in env.items() if v == '']\n"
  721. " for k in drop: env.pop(k)\n"
  722. " if drop:\n"
  723. " json.dump(d, open(p,'w'), indent=2)\n"
  724. " print('dropped empty env keys: ' + ', '.join(drop))\n"
  725. "ENDOFPATCH"],
  726. capture_output=not verbose,
  727. )
  728. # Write global agtx config for bench user so the TUI skips the agent-selection wizard.
  729. #
  730. # `auto_trust = true` because a benchmark container has no human at the board.
  731. # agtx's default is off โ€” a trust prompt surfaces as a Blocked card and waits
  732. # for a person โ€” which here would stall the instance until its timeout. Same
  733. # reasoning as docker/entrypoint.sh: disposable container, nothing in it but
  734. # the repo under test.
  735. global_config_content = f'default_agent = "{default_agent}"\nauto_trust = true\n'
  736. if verbose:
  737. print(f" [docker] Writing global agtx config (default_agent={default_agent})...", file=sys.stderr)
  738. subprocess.run(
  739. ["docker", "exec", container_id,
  740. "/bin/bash", "-c",
  741. f"mkdir -p /home/bench/.config/agtx && cat > /home/bench/.config/agtx/config.toml << 'ENDOFCONFIG'\n{global_config_content}\nENDOFCONFIG"],
  742. check=True,
  743. capture_output=not verbose,
  744. )
  745. # Write .agtx/config.toml into /testbed inside the container
  746. # Build the config content (with base_branch injected) in memory
  747. content = config_path.read_text()
  748. base_branch_line = f'base_branch = "{base_commit}"\n'
  749. lines = [l for l in content.splitlines(keepends=True) if not l.startswith("base_branch =")]
  750. insert_at = next((i for i, l in enumerate(lines) if l.startswith("[")), len(lines))
  751. lines.insert(insert_at, base_branch_line)
  752. config_content = "".join(lines)
  753. if verbose:
  754. print(f" [docker] Writing .agtx/config.toml into container...", file=sys.stderr)
  755. subprocess.run(
  756. ["docker", "exec", container_id,
  757. "/bin/bash", "-c",
  758. f"mkdir -p /testbed/.agtx && cat > /testbed/.agtx/config.toml << 'ENDOFCONFIG'\n{config_content}\nENDOFCONFIG"],
  759. check=True,
  760. capture_output=not verbose,
  761. )
  762. # Clean stale agtx state (worktrees, project DB) โ€” same as _write_agtx_config
  763. subprocess.run(
  764. ["docker", "exec", container_id,
  765. "/bin/bash", "-c",
  766. "rm -rf /testbed/.agtx/worktrees /testbed/.agtx/db"],
  767. capture_output=True,
  768. )
  769. # Activate the conda 'testbed' environment for all new shells.
  770. # SWE-bench images ship with repo dependencies pre-installed in conda env 'testbed'.
  771. # Without this, Claude uses the system python/pip and tries to reinstall everything.
  772. subprocess.run(
  773. ["docker", "exec", container_id, "/bin/bash", "-c",
  774. "echo 'source /opt/miniconda3/etc/profile.d/conda.sh && conda activate testbed'"
  775. " >> /root/.bashrc"],
  776. check=True, capture_output=not verbose,
  777. )
  778. if tools_container_id:
  779. # Tools volume is mounted at /tools (read-only). Symlink everything from there โ€”
  780. # zero writable-layer usage. libutempter is included in the volume because
  781. # SWE-bench containers don't have it (tmux needs it).
  782. if verbose:
  783. print(f" [docker] Wiring tools from volume...", file=sys.stderr)
  784. subprocess.run(
  785. ["docker", "exec", container_id, "/bin/bash", "-c",
  786. "ln -sf /tools/tmux /usr/bin/tmux"
  787. " && ln -sf /tools/node /usr/bin/node"
  788. " && ln -sf /tools/node_modules/npm/bin/npm-cli.js /usr/bin/npm"
  789. " && ln -sf /tools/node_modules/npm/bin/npx-cli.js /usr/bin/npx"
  790. " && ln -sf /tools/node_modules/@anthropic-ai/claude-code/bin/claude.exe /usr/bin/claude"
  791. " && mkdir -p /usr/lib/node_modules"
  792. " && ln -sf /tools/node_modules/@anthropic-ai /usr/lib/node_modules/@anthropic-ai"
  793. " && ln -sf /tools/node_modules/npm /usr/lib/node_modules/npm"
  794. " && test -f /tools/node_modules/.bin/caveman && ln -sf /tools/node_modules/.bin/caveman /usr/bin/caveman || true"
  795. " && test -e /tools/node_modules/@xai-official/grok/bin/grok"
  796. " && ln -sf /tools/node_modules/@xai-official/grok/bin/grok /usr/bin/grok"
  797. " && ln -sf /tools/node_modules/@xai-official /usr/lib/node_modules/@xai-official || true"
  798. " && mkdir -p /usr/lib/x86_64-linux-gnu"
  799. " && ln -sf /tools/lib/x86_64-linux-gnu/libutempter.so.0 /usr/lib/x86_64-linux-gnu/libutempter.so.0"
  800. " && ln -sf /tools/lib/x86_64-linux-gnu/libutempter.so.1.2.1 /usr/lib/x86_64-linux-gnu/libutempter.so.1.2.1"
  801. " && ln -sf /tools/lib/x86_64-linux-gnu/libevent_core-2.1.so.7 /usr/lib/x86_64-linux-gnu/libevent_core-2.1.so.7"],
  802. check=True, capture_output=not verbose,
  803. )
  804. else:
  805. if verbose:
  806. print(f" [docker] Installing tmux, Node.js 22, Claude Code, Grok...", file=sys.stderr)
  807. subprocess.run(
  808. ["docker", "exec", container_id,
  809. "/bin/bash", "-c",
  810. "apt-get update -qq && apt-get install -y -qq tmux curl ca-certificates "
  811. "&& curl -fsSL https://deb.nodesource.com/setup_22.x | bash - "
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 910 on GitHub
  812. "&& apt-get install -y -qq nodejs "
  813. "&& npm install -g @anthropic-ai/claude-code @xai-official/grok"],
  814. check=True,
  815. capture_output=not verbose,
  816. )
  817. # Run optional sandbox init commands (e.g. "rtk init -g").
  818. # These run after credentials are in place so tools can write into ~/.claude/.
  819. # Prepend ~/.local/bin to PATH so tools installed there (e.g. rtk) are found.
  820. for cmd in (sandbox_init or []):
  821. if verbose:
  822. print(f" [docker] sandbox_init: {cmd}", file=sys.stderr)
  823. subprocess.run(
  824. ["docker", "exec", container_id, "/bin/bash", "-c",
  825. f"HOME=/home/bench && export HOME PATH=$HOME/.local/bin:$PATH && {cmd}"],
  826. check=True, capture_output=not verbose,
  827. )
  828. def start_tui_in_container(slug: str, container_id: str, verbose: bool = False) -> None:
  829. """Start the agtx TUI inside the container's tmux session."""
  830. if verbose:
  831. print(f" [docker] Starting agtx TUI in container tmux session '{slug}'...", file=sys.stderr)
  832. # Launch agtx with all required env vars baked into the command so every tmux
  833. # window it spawns (agent sessions) inherits them from the start โ€” no race.
  834. # HOME=/home/bench โ€” agtx reads ~/.config/agtx/config.toml
  835. # IS_SANDBOX=1 โ€” bypasses Claude's root check (--dangerously-skip-permissions)
  836. # PATH/CONDA vars โ€” activate conda 'testbed' env so pre-installed deps are used
  837. testbed_bin = "/opt/miniconda3/envs/testbed/bin"
  838. env_prefix = (
  839. f"HOME=/home/bench "
  840. f"IS_SANDBOX=1 "
  841. f"CONDA_DEFAULT_ENV=testbed "
  842. f"CONDA_PREFIX=/opt/miniconda3/envs/testbed "
  843. f"PATH={testbed_bin}:/opt/miniconda3/bin:/home/bench/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  844. )
  845. # Pass through API keys that agents read from the environment (only when set
  846. # on the host). Grok uses XAI_API_KEY when ~/.grok/auth.json is absent.
  847. passthrough_env = {
  848. k: v for k in ("XAI_API_KEY", "ANTHROPIC_API_KEY") if (v := os.environ.get(k))
  849. }
  850. for key, val in passthrough_env.items():
  851. env_prefix += f" {key}={shlex.quote(val)}"
  852. subprocess.run(
  853. ["docker", "exec", "-d", container_id,
  854. "tmux", "new-session", "-d", "-s", slug,
  855. f"env {env_prefix} agtx /testbed"],
  856. check=True,
  857. capture_output=not verbose,
  858. )
  859. # Also set them in the agtx tmux server global env so windows created later
  860. # (e.g. after a phase transition) also inherit the correct PATH.
  861. time.sleep(3)
  862. for key, val in [
  863. ("HOME", "/home/bench"),
  864. ("IS_SANDBOX", "1"),
  865. ("CONDA_DEFAULT_ENV", "testbed"),
  866. ("CONDA_PREFIX", "/opt/miniconda3/envs/testbed"),
  867. ("PATH", f"{testbed_bin}:/opt/miniconda3/bin:/home/bench/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"),
  868. ] + list(passthrough_env.items()):
  869. subprocess.run(
  870. ["docker", "exec", container_id,
  871. "tmux", "-L", "agtx", "set-environment", "-g", key, val],
  872. capture_output=True,
  873. )
  874. # Give the TUI time to fully start before the caller proceeds
  875. time.sleep(5)
  876. def stop_docker_container(container_id: str, verbose: bool = False) -> None:
  877. """Stop and remove the container."""
  878. if verbose:
  879. print(f" [docker] Stopping container {container_id[:12]}...", file=sys.stderr)
  880. subprocess.run(["docker", "stop", container_id], capture_output=True)
  881. subprocess.run(["docker", "rm", container_id], capture_output=True)
  882. # ---------------------------------------------------------------------------
  883. # Token/cost tracking via tokscale
  884. # ---------------------------------------------------------------------------
  885. # CLI flag that filters tokscale output to a specific agent
  886. _TOKSCALE_FLAG: dict[str, str] = {
  887. "claude": "--claude",
  888. "codex": "--codex",
  889. "gemini": "--gemini",
  890. "copilot": "--copilot",
  891. "opencode": "--opencode",
  892. }
  893. # Cached result of tokscale availability check
  894. _tokscale_bin: str | None | bool = False # False = not yet checked
  895. def _find_tokscale() -> str | None:
  896. """Return path to tokscale binary, or None if not installed."""
  897. global _tokscale_bin
  898. if _tokscale_bin is not False:
  899. return _tokscale_bin # type: ignore
  900. result = subprocess.run(["which", "tokscale"], capture_output=True, text=True)
  901. _tokscale_bin = result.stdout.strip() or None
  902. return _tokscale_bin # type: ignore
  903. def _tokscale_snapshot(agent: str) -> dict:
  904. """
  905. Run `tokscale --json --today --{agent}` and return aggregated totals across all models:
  906. {input, output, cache_read, cache_write, reasoning, cost_usd}
  907. Returns zeros if tokscale is unavailable or the agent has no records today.
  908. """
  909. tokscale = _find_tokscale()
  910. if not tokscale:
  911. return {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "cost_usd": 0.0}
  912. flag = _TOKSCALE_FLAG.get(agent)
  913. cmd = [tokscale, "--json", "--today"]
  914. if flag:
  915. cmd.append(flag)
  916. try:
  917. result = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
  918. if result.returncode != 0 or not result.stdout.strip():
  919. return {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "cost_usd": 0.0}
  920. data = json.loads(result.stdout)
  921. entries = data.get("entries", [])
  922. totals = {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "cost_usd": 0.0}
  923. for entry in entries:
  924. totals["input"] += entry.get("input", 0)
  925. totals["output"] += entry.get("output", 0)
  926. totals["cache_read"] += entry.get("cacheRead", 0)
  927. totals["cache_write"] += entry.get("cacheWrite", 0)
  928. totals["reasoning"] += entry.get("reasoning", 0)
  929. totals["cost_usd"] += entry.get("cost", 0.0)
  930. return totals
  931. except Exception:
  932. return {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "cost_usd": 0.0}
  933. def tokscale_diff(before: dict, after: dict) -> dict:
  934. """
  935. Subtract before-snapshot from after-snapshot to get this task's usage.
  936. Returns the standard cost_data dict used throughout the script.
  937. """
  938. if not _find_tokscale():
  939. return {"cost_usd": None, "cost_tokens": None}
  940. inp = max(0, after["input"] - before["input"])
  941. out = max(0, after["output"] - before["output"])
  942. cost = max(0.0, after["cost_usd"] - before["cost_usd"])
  943. total = inp + out + max(0, after["cache_read"] - before["cache_read"]) + \
  944. max(0, after["cache_write"] - before["cache_write"]) + \
  945. max(0, after["reasoning"] - before["reasoning"])
  946. return {
  947. "cost_usd": round(cost, 6) if cost > 0 else None,
  948. "cost_tokens": total if total > 0 else None,
  949. }
  950. def tokscale_from_container(container_id: str, agent: str) -> dict:
  951. """
  952. Read token/cost usage from a sandbox container by copying /home/bench/.claude
  953. to a temp dir and running tokscale --home against it.
  954. Returns the standard cost_data dict {cost_usd, cost_tokens}, or nulls on failure.
  955. """
  956. tokscale = _find_tokscale()
  957. if not tokscale:
  958. return {"cost_usd": None, "cost_tokens": None}
  959. flag = _TOKSCALE_FLAG.get(agent)
  960. if not flag:
  961. return {"cost_usd": None, "cost_tokens": None}
  962. try:
  963. with tempfile.TemporaryDirectory() as tmpdir:
  964. # Copy /home/bench/.claude out of the container
  965. result = subprocess.run(
  966. ["docker", "cp", f"{container_id}:/home/bench/.claude", tmpdir],
  967. capture_output=True,
  968. )
  969. if result.returncode != 0:
  970. return {"cost_usd": None, "cost_tokens": None}
  971. # tokscale --home expects the home dir (parent of .claude)
  972. result = subprocess.run(
  973. [tokscale, "--home", tmpdir, "--json", flag],
  974. capture_output=True,
  975. text=True,
  976. timeout=15,
  977. )
  978. if result.returncode != 0 or not result.stdout.strip():
  979. return {"cost_usd": None, "cost_tokens": None}
  980. data = json.loads(result.stdout)
  981. entries = data.get("entries", [])
  982. inp = out = cr = cw = reasoning = 0
  983. cost = 0.0
  984. for entry in entries:
  985. inp += entry.get("input", 0)
  986. out += entry.get("output", 0)
  987. cr += entry.get("cacheRead", 0)
  988. cw += entry.get("cacheWrite", 0)
  989. reasoning += entry.get("reasoning", 0)
  990. cost += entry.get("cost", 0.0)
  991. total = inp + out + cr + cw + reasoning
  992. return {
  993. "cost_usd": round(cost, 6) if cost > 0 else None,
  994. "cost_tokens": total if total > 0 else None,
  995. }
  996. except Exception:
  997. return {"cost_usd": None, "cost_tokens": None}
  998. class ResultsStore:
  999. """Thread-safe, resumable persistence for benchmark results."""
  1000. def __init__(self, output_dir: Path):
  1001. self.output_dir = output_dir
  1002. output_dir.mkdir(parents=True, exist_ok=True)
  1003. self.predictions_path = output_dir / "predictions.jsonl"
  1004. self.results_path = output_dir / "results.json"
  1005. self._lock = threading.Lock()
  1006. self._results: list[dict] = []
  1007. self._done_ids: set[str] = set()
  1008. self._load_existing()
  1009. def _load_existing(self) -> None:
  1010. if self.results_path.exists():
  1011. try:
  1012. data = json.loads(self.results_path.read_text())
  1013. self._results = data if isinstance(data, list) else []
  1014. self._done_ids = {r["instance_id"] for r in self._results}
  1015. print(f"Resuming: {len(self._done_ids)} tasks already completed.")
  1016. except Exception:
  1017. pass
  1018. def is_done(self, instance_id: str) -> bool:
  1019. return instance_id in self._done_ids
  1020. def save_result(
  1021. self,
  1022. instance_id: str,
  1023. status: str,
  1024. duration_seconds: float,
  1025. model_name: str,
  1026. model_patch: str = "",
  1027. cost_usd: float | None = None,
  1028. cost_tokens: int | None = None,
  1029. error: str | None = None,
  1030. ) -> None:
  1031. with self._lock:
  1032. result = {
  1033. "instance_id": instance_id,
  1034. "status": status,
  1035. "duration_seconds": round(duration_seconds, 1),
  1036. "cost_usd": cost_usd,
  1037. "cost_tokens": cost_tokens,
  1038. "model_patch": model_patch,
  1039. "error": error,
  1040. }
  1041. self._results.append(result)
  1042. self._done_ids.add(instance_id)
  1043. # Append prediction (SWE-bench format)
  1044. with self.predictions_path.open("a") as f:
  1045. pred = {
  1046. "instance_id": instance_id,
  1047. "model_name_or_path": model_name,
  1048. "model_patch": model_patch,
  1049. }
  1050. f.write(json.dumps(pred) + "\n")
  1051. # Rewrite results atomically
  1052. tmp = self.results_path.with_suffix(".json.tmp")
  1053. tmp.write_text(json.dumps(self._results, indent=2))
  1054. tmp.replace(self.results_path)
  1055. # ---------------------------------------------------------------------------
  1056. # Phase artifact paths per bundled plugin
  1057. # ---------------------------------------------------------------------------
  1058. # Artifact path for the "planning" phase (relative to worktree root).
  1059. # None means the plugin has no planning artifact โ†’ fall back to pane stability.
  1060. PLUGIN_PLANNING_ARTIFACTS: dict[str, str | None] = {
  1061. "agtx": ".agtx/plan.md",
  1062. "agtx-terse": ".agtx/plan.md",
  1063. "gsd": ".planning/phases/*/{phase}-CONTEXT.md",
  1064. "spec-kit": None,
  1065. "bmad": None,
  1066. "openspec": None,
  1067. "agent-skills": None,
  1068. "superpowers": "docs/superpowers/plans/*.md",
  1069. "oh-my-claudecode": None,
  1070. "void": None,
  1071. }
  1072. # Artifact path for the "running" phase (relative to worktree root).
  1073. # None means the plugin has no running artifact โ†’ fall back to pane stability.
  1074. PLUGIN_RUNNING_ARTIFACTS: dict[str, str | None] = {
  1075. "agtx": ".agtx/execute.md",
  1076. "agtx-terse": ".agtx/execute.md",
  1077. "gsd": ".planning/phases/*/{phase}-SUMMARY.md",
  1078. "spec-kit": None,
  1079. "bmad": "_bmad-output/implementation-artifacts/*.md",
  1080. "openspec": None,
  1081. "agent-skills": None,
  1082. "superpowers": None,
  1083. "oh-my-claudecode": None,
  1084. "void": None,
  1085. }
  1086. # Artifact path for the "review" phase (relative to worktree root).
  1087. # None means no review artifact โ†’ collect patch immediately after entering Review.
  1088. PLUGIN_REVIEW_ARTIFACTS: dict[str, str | None] = {
  1089. "agtx": ".agtx/review.md",
  1090. "agtx-terse": ".agtx/review.md",
  1091. "gsd": None,
  1092. "spec-kit": None,
  1093. "bmad": None,
  1094. "openspec": None,
  1095. "agent-skills": None,
  1096. "superpowers": None,
  1097. "oh-my-claudecode": None,
  1098. "void": None,
  1099. }
  1100. def _artifact_exists(worktree: Path, pattern: str, container_id: str | None = None) -> bool:
  1101. """Return True if the artifact path (possibly with * or {phase} placeholders) exists.
  1102. In Docker mode (container_id provided), checks inside the container via docker exec
  1103. instead of the local filesystem (the worktree lives inside the container).
  1104. """
  1105. if "{phase}" in pattern:
  1106. for n in range(1, 21):
  1107. for fmt in (f"{n:02d}", str(n)):
  1108. if _artifact_exists(worktree, pattern.replace("{phase}", fmt), container_id):
  1109. return True
  1110. return False
  1111. if container_id:
  1112. # In Docker: use shell glob via docker exec so * patterns work
  1113. path = str(worktree / pattern)
  1114. result = subprocess.run(
  1115. ["docker", "exec", container_id, "/bin/bash", "-c", f"ls {path} 2>/dev/null | head -1"],
  1116. capture_output=True, text=True,
  1117. )
  1118. return bool(result.stdout.strip())
  1119. if "*" not in pattern:
  1120. return (worktree / pattern).exists()
  1121. parts = Path(pattern).parts
  1122. base = worktree
  1123. for i, part in enumerate(parts):
  1124. if "*" in part:
  1125. remaining = str(Path(*parts[i:]))
  1126. return any(True for _ in base.glob(remaining))
  1127. base = base / part
  1128. return False
  1129. # ---------------------------------------------------------------------------
  1130. # Task runner
  1131. # ---------------------------------------------------------------------------
  1132. class TaskRunner:
  1133. """Runs a single SWE-bench instance through the full agtx lifecycle."""
  1134. SMOKE_TEST_DESCRIPTION = (
  1135. "This is a smoke test run. Do not modify any source files.\n\n"
  1136. "Planning phase: write the following content to `.agtx/plan.md` in the current working directory:\n"
  1137. "```\n"
  1138. "# Smoke Test Plan\n"
  1139. "This is a smoke test. In the running phase: write an empty `.agtx/execute.md`. "
  1140. "In the review phase: write an empty `.agtx/review.md`. Do not modify any source files.\n"
  1141. "```\n"
  1142. "Then stop and wait."
  1143. )
  1144. NOTE = (
  1145. "IMPORTANT CONSTRAINTS โ€” these apply to all phases including review:\n"
  1146. "- NEVER install packages, dependencies, or build the project.\n"
  1147. "- NEVER run tests under any circumstances.\n"
  1148. "- Do not run any git commands (no fetch, pull, merge, or commit).\n"
  1149. "- The repo may not be installable in this environment โ€” do not attempt it.\n"
  1150. "- Read the source code, understand the bug, and fix it by editing the relevant files directly."
  1151. )
  1152. @staticmethod
  1153. def _strip_code(problem_statement: str) -> str:
  1154. """Remove fenced code blocks and inline code, keep prose."""
  1155. # Remove fenced code blocks (``` ... ```)
  1156. text = re.sub(r"```.*?```", "", problem_statement, flags=re.DOTALL)
  1157. # Collapse runs of blank lines left behind
  1158. text = re.sub(r"\n{3,}", "\n\n", text)
  1159. return text.strip()
  1160. def _build_description(self, problem_statement: str) -> str:
  1161. if self.hard:
  1162. return self._strip_code(problem_statement) + "\n\n---\n" + self.NOTE
  1163. return problem_statement + "\n\n---\n" + self.NOTE
  1164. def __init__(
  1165. self,
  1166. instance: dict,
  1167. repo_path: Path,
  1168. agtx_bin: str,
  1169. plugin: str,
  1170. agent: str,
  1171. running_agent: str,
  1172. model_name: str,
  1173. phase_timeout: int,
  1174. verbose: bool = False,
  1175. smoke_test: bool = False,
  1176. hard: bool = False,
  1177. container_id: str | None = None,
  1178. ):
  1179. self.instance = instance
  1180. self.instance_id = instance["instance_id"]
  1181. self.repo_path = repo_path
  1182. self.agtx_bin = agtx_bin
  1183. self.plugin = plugin
  1184. self.agent = agent
  1185. self.running_agent = running_agent
  1186. self.model_name = model_name
  1187. self.phase_timeout = phase_timeout
  1188. self.verbose = verbose
  1189. self.smoke_test = smoke_test
  1190. self.hard = hard
  1191. self.container_id = container_id
  1192. self.mcp: McpClient | None = None
  1193. def _poll_transition(self, request_id: str, timeout: int = 120) -> None:
  1194. deadline = time.monotonic() + timeout
  1195. while time.monotonic() < deadline:
  1196. result = self.mcp.call("get_transition_status", request_id=request_id)
  1197. status = result.get("status", "pending")
  1198. if status == "completed":
  1199. return
  1200. if status == "error":
  1201. raise McpError(f"Transition failed: {result.get('error')}")
  1202. time.sleep(2)
  1203. raise TimeoutError(f"Transition {request_id} timed out after {timeout}s")
  1204. def _move_task(self, task_id: str, action: str) -> None:
  1205. result = self.mcp.call("move_task", task_id=task_id, action=action)
  1206. self._poll_transition(result["request_id"])
  1207. def _wait_for_status(self, task_id: str, target_status: str, timeout: int = 120) -> dict:
  1208. deadline = time.monotonic() + timeout
  1209. while time.monotonic() < deadline:
  1210. task = self.mcp.call("get_task", task_id=task_id)
  1211. if task.get("status") == target_status:
  1212. return task
  1213. time.sleep(3)
  1214. raise TimeoutError(f"Task never reached {target_status} within {timeout}s")
  1215. def _wait_for_worktree(self, task_id: str, timeout: int = 180) -> dict:
  1216. """
  1217. Wait until the task has a worktree_path set in the DB.
  1218. This is populated asynchronously by the TUI's background setup thread
  1219. after transition_to_planning/running completes โ€” the transition request
  1220. being marked 'completed' only means the TUI accepted it, not that
  1221. worktree setup is done.
  1222. """
  1223. deadline = time.monotonic() + timeout
  1224. while time.monotonic() < deadline:
  1225. task = self.mcp.call("get_task", task_id=task_id)
  1226. if task.get("worktree_path"):
  1227. return task
  1228. time.sleep(3)
  1229. raise TimeoutError(f"Worktree never populated for task {task_id} within {timeout}s")
  1230. def _wait_for_artifact(self, task_id: str, worktree_path: str, artifact_pattern: str, timeout: int) -> bool:
  1231. """Poll for an artifact file in the worktree. Returns True if found, False on timeout."""
  1232. worktree = Path(worktree_path)
  1233. deadline = time.monotonic() + timeout
  1234. while time.monotonic() < deadline:
  1235. if _artifact_exists(worktree, artifact_pattern, self.container_id):
  1236. return True
  1237. time.sleep(5)
  1238. return False
  1239. # Patterns that indicate a shell command run by the agent is waiting for input,
  1240. # blocking Claude from finishing. Only used in the stability fallback path.
  1241. _INTERACTIVE_PROMPT_PATTERNS = [
  1242. r"\[y/N\]",
  1243. r"\[Y/n\]",
  1244. r"\[y/n\]",
  1245. r"\[Y/N\]",
  1246. r"Press Enter to continue",
  1247. r"Press any key",
  1248. # Claude's usage-limit pause. It ends the turn and waits on a keypress
  1249. # that nothing sends, so the task parks indefinitely โ€” observed sitting
  1250. # ~50 minutes, past both the stated reset time and --phase-timeout.
  1251. # Matching it turns a silent stall into a named failure.
  1252. r"You've hit your session limit",
  1253. r"Press .* to continue after reset",
  1254. ]
  1255. def _check_interactive_prompt(self, content: str) -> str | None:
  1256. """Return the matching line if pane content looks like an interactive prompt, else None."""
  1257. for line in content.splitlines()[-20:]:
  1258. for pattern in self._INTERACTIVE_PROMPT_PATTERNS:
  1259. if re.search(pattern, line, re.IGNORECASE):
  1260. return line.strip()
  1261. return None
  1262. # Claude always prints one of these lines when it finishes a response.
  1263. # Detecting this (plus the โฏ prompt) is more reliable than pure pane stability.
  1264. _CLAUDE_FINISHED_RE = re.compile(r"โœป\s+[A-Z][a-z]+ for \d+s")
  1265. def _is_claude_idle(self, content: str) -> bool:
  1266. """Return True when Claude has finished its last response and shows the โฏ prompt."""
  1267. lines = content.splitlines()
  1268. has_finish_marker = any(self._CLAUDE_FINISHED_RE.search(l) for l in lines)
  1269. has_prompt = any(l.strip() == "โฏ" or l.strip().startswith("โฏ ") for l in lines[-10:])
  1270. return has_finish_marker and has_prompt
  1271. def _wait_for_pane_stable(self, task_id: str, timeout: int) -> None:
  1272. """Wait until Claude has finished its last response (no artifact available).
  1273. Primary signal: Claude prints 'โœป Cooked/Worked/Crunched for Xs' + โฏ prompt.
  1274. Fallback: pane content identical for 2 consecutive 10s checks.
  1275. If an interactive prompt is detected while stable, prints a warning so the user
  1276. can attach to the session and answer it.
  1277. """
  1278. prev_content: str | None = None
  1279. stable_count = 0
  1280. prompt_warned = False
  1281. deadline = time.monotonic() + timeout
  1282. while time.monotonic() < deadline:
  1283. time.sleep(5)
  1284. try:
  1285. result = self.mcp.call("read_pane_content", task_id=task_id, lines=80)
  1286. content = result.get("content", "")
  1287. except McpError:
  1288. content = ""
  1289. # Primary: Claude idle signal
  1290. if self._is_claude_idle(content):
  1291. return
  1292. # Interactive prompt check (even when not yet stable)
  1293. if content == prev_content:
  1294. stable_count += 1
  1295. prompt_line = self._check_interactive_prompt(content)
  1296. if prompt_line:
  1297. if not prompt_warned:
  1298. print(f"\n[{self.instance_id}] โš  Agent is waiting for input: \"{prompt_line}\"", file=sys.stderr)
  1299. slug = self.instance_id.replace("__", "-").replace("_", "-")
  1300. print(f" Attach to answer: docker exec -it swebench-{slug} tmux -L agtx attach -t testbed:1", file=sys.stderr)
  1301. prompt_warned = True
  1302. stable_count = 0 # Reset โ€” not actually done
  1303. elif stable_count >= 2:
  1304. # Pane stable but no Claude finish marker โ€” needs attention.
  1305. # Possible causes: agent error, waiting for user approval (e.g. superpowers brainstorm),
  1306. # or genuinely stuck. Warn once and keep polling; do NOT advance.
  1307. if not prompt_warned:
  1308. slug = self.instance_id.replace("__", "-").replace("_", "-")
  1309. print(f"\n[{self.instance_id}] โš  Pane stable but no finish marker โ€” agent may be stuck, hit an error, or requires manual approval.", file=sys.stderr)
  1310. print(f" Attach to inspect or interact: docker exec -it swebench-{slug} tmux -L agtx attach -t testbed:1", file=sys.stderr)
  1311. prompt_warned = True
  1312. stable_count = 0 # Keep polling, don't advance
  1313. else:
  1314. stable_count = 0
  1315. prompt_warned = False
  1316. prev_content = content
  1317. def run(self) -> dict:
  1318. start = time.monotonic()
  1319. # What `git diff <base_commit>` already reports before the agent has done
  1320. # anything. SWE-bench images are not pristine checkouts of base_commit:
  1321. # for astropy-12907 the baseline is 1861 files (3722 lines of
  1322. # `old mode 100644` / `new mode 100755`) plus the image's own
  1323. # `setuptools==68.0.0` pin in pyproject.toml. Subtracted at collection so
  1324. # a real fix is not buried in ~224KB of noise.
  1325. self.baseline_patch = self._raw_diff()
  1326. if self.verbose and self.baseline_patch:
  1327. n = self.baseline_patch.count("diff --git ")
  1328. print(
  1329. f" [{self.instance_id}] Baseline diff: {n} file(s) already differ from "
  1330. f"base_commit before the agent runs; these are excluded from the prediction.",
  1331. file=sys.stderr,
  1332. )
  1333. if self.verbose:
  1334. print(f" [{self.instance_id}] Starting MCP handshake...", file=sys.stderr)
  1335. self.mcp = McpClient(self.agtx_bin, str(self.repo_path), container_id=self.container_id)
  1336. task_id = None
  1337. try:
  1338. if self.verbose:
  1339. print(f" [{self.instance_id}] Creating task...", file=sys.stderr)
  1340. problem = self.instance.get("problem_statement", "")
  1341. if self.smoke_test:
  1342. description = self.SMOKE_TEST_DESCRIPTION
  1343. else:
  1344. description = self._build_description(problem)
  1345. task_resp = self.mcp.call(
  1346. "create_task",
  1347. title=self.instance_id,
  1348. description=description,
  1349. plugin=self.plugin,
  1350. )
  1351. task_id = task_resp["id"]
  1352. if self.verbose:
  1353. print(f" [{self.instance_id}] task_id={task_id}, moving to Planning...", file=sys.stderr)
  1354. # Backlog โ†’ Planning
  1355. self._move_task(task_id, "move_forward")
  1356. self._wait_for_status(task_id, "planning", timeout=120)
  1357. # Wait for worktree to be created (set by TUI background thread)
  1358. if self.verbose:
  1359. print(f" [{self.instance_id}] Waiting for worktree...", file=sys.stderr)
  1360. planning_task = self._wait_for_worktree(task_id, timeout=180)
  1361. worktree_path = planning_task.get("worktree_path", "")
  1362. # Snapshot tokscale before agent starts working (non-sandbox only)
  1363. cost_before = _tokscale_snapshot(self.running_agent) if not self.container_id else None
  1364. # Wait for planning phase artifact (.agtx/plan.md for agtx/agtx-terse plugins)
  1365. planning_artifact = PLUGIN_PLANNING_ARTIFACTS.get(self.plugin)
  1366. if planning_artifact and worktree_path:
  1367. if self.verbose:
  1368. print(f" [{self.instance_id}] Waiting for planning artifact ({planning_artifact})...", file=sys.stderr)
  1369. found = self._wait_for_artifact(task_id, worktree_path, planning_artifact, self.phase_timeout)
  1370. if not found:
  1371. raise TimeoutError(f"Planning artifact not found within {self.phase_timeout}s")
  1372. else:
  1373. if self.verbose:
  1374. print(f" [{self.instance_id}] Waiting for planning phase (pane stability)...", file=sys.stderr)
  1375. self._wait_for_pane_stable(task_id, self.phase_timeout)
  1376. if self.verbose:
  1377. print(f" [{self.instance_id}] Planning done, moving to Running...", file=sys.stderr)
  1378. # Small delay: let the agent finish any final output after writing the artifact
  1379. # before the TUI sends /exit (clear_context_on_advance) and restarts it.
  1380. time.sleep(5)
  1381. # Planning โ†’ Running
  1382. self._move_task(task_id, "move_forward")
  1383. running_task = self._wait_for_status(task_id, "running", timeout=60)
  1384. worktree_path = running_task.get("worktree_path") or worktree_path
  1385. # Wait for running phase artifact (.agtx/execute.md for agtx/agtx-terse plugins)
  1386. running_artifact = PLUGIN_RUNNING_ARTIFACTS.get(self.plugin)
  1387. if running_artifact and worktree_path:
  1388. if self.verbose:
  1389. print(f" [{self.instance_id}] Waiting for running artifact ({running_artifact})...", file=sys.stderr)
  1390. found = self._wait_for_artifact(task_id, worktree_path, running_artifact, self.phase_timeout)
  1391. if not found:
  1392. raise TimeoutError(f"Running artifact not found within {self.phase_timeout}s")
  1393. else:
  1394. if self.verbose:
  1395. print(f" [{self.instance_id}] Waiting for running phase (pane stability)...", file=sys.stderr)
  1396. self._wait_for_pane_stable(task_id, self.phase_timeout)
  1397. # Collect token/cost usage:
  1398. # - sandbox: read directly from container's /home/bench/.claude via tokscale --home
  1399. # - non-sandbox: diff host tokscale snapshots taken before/after running phase
  1400. if self.container_id:
  1401. cost_data = tokscale_from_container(self.container_id, self.running_agent)
  1402. else:
  1403. cost_data = tokscale_diff(cost_before, _tokscale_snapshot(self.running_agent))
  1404. if self.verbose:
  1405. print(f" [{self.instance_id}] Running done, moving to Review...", file=sys.stderr)
  1406. # Running โ†’ Review
  1407. self._move_task(task_id, "move_forward")
  1408. review_task = self._wait_for_status(task_id, "review", timeout=60)
  1409. worktree_path = review_task.get("worktree_path") or worktree_path
  1410. # Wait for review phase artifact (.agtx/review.md for agtx/agtx-terse plugins)
  1411. review_artifact = PLUGIN_REVIEW_ARTIFACTS.get(self.plugin)
  1412. if review_artifact and worktree_path:
  1413. if self.verbose:
  1414. print(f" [{self.instance_id}] Waiting for review artifact ({review_artifact})...", file=sys.stderr)
  1415. found = self._wait_for_artifact(task_id, worktree_path, review_artifact, self.phase_timeout)
  1416. if not found:
  1417. raise TimeoutError(f"Review artifact not found within {self.phase_timeout}s")
  1418. else:
  1419. if self.verbose:
  1420. print(f" [{self.instance_id}] Waiting for review phase (pane stability)...", file=sys.stderr)
  1421. self._wait_for_pane_stable(task_id, self.phase_timeout)
  1422. model_patch = self._collect_patch(review_task)
  1423. if self.verbose:
  1424. print(f" [{self.instance_id}] Done. patch_len={len(model_patch)}", file=sys.stderr)
  1425. # Review โ†’ Done (cleanup worktree)
  1426. try:
  1427. self._move_task(task_id, "move_to_done")
  1428. except Exception:
  1429. pass
  1430. return {
  1431. "status": "success",
  1432. "duration_seconds": time.monotonic() - start,
  1433. "model_patch": model_patch,
  1434. **cost_data,
  1435. "error": None,
  1436. }
  1437. except TimeoutError as e:
  1438. self._cleanup(task_id)
  1439. return self._error_result("timeout", time.monotonic() - start, str(e))
  1440. except Exception as e:
  1441. self._cleanup(task_id)
  1442. return self._error_result("error", time.monotonic() - start, str(e))
  1443. finally:
  1444. if self.mcp:
  1445. self.mcp.close()
  1446. @staticmethod
  1447. def _split_file_sections(patch: str) -> dict[str, str]:
  1448. """Split a unified diff into {path_header: section_text}."""
  1449. sections: dict[str, str] = {}
  1450. current: str | None = None
  1451. buf: list[str] = []
  1452. for line in patch.splitlines(keepends=True):
  1453. if line.startswith("diff --git "):
  1454. if current is not None:
  1455. sections[current] = "".join(buf)
  1456. current = line.strip()
  1457. buf = [line]
  1458. elif current is not None:
  1459. buf.append(line)
  1460. if current is not None:
  1461. sections[current] = "".join(buf)
  1462. return sections
  1463. def _strip_baseline(self, patch: str) -> str:
  1464. """Drop file sections identical to the pre-agent baseline.
  1465. Per file rather than per line: a file the agent genuinely edited has a
  1466. different section and survives whole, so an agent that legitimately
  1467. touches pyproject.toml is not silently dropped along with the image's
  1468. own pin of it.
  1469. """
  1470. baseline = getattr(self, "baseline_patch", "") or ""
  1471. if not baseline:
  1472. return patch
  1473. base_sections = self._split_file_sections(baseline)
  1474. kept = [
  1475. text
  1476. for header, text in self._split_file_sections(patch).items()
  1477. if base_sections.get(header) != text
  1478. ]
  1479. return "".join(kept)
  1480. def _collect_patch(self, task: dict) -> str:
  1481. return self._strip_baseline(self._raw_diff(task))
  1482. def _raw_diff(self, task: dict | None = None) -> str:
  1483. base_commit = self.instance["base_commit"]
  1484. if self.container_id:
  1485. # In Docker mode, diff /testbed directly inside the container
  1486. result = subprocess.run(
  1487. [
  1488. "docker", "exec", self.container_id,
  1489. "git", "-C", "/testbed", "diff", base_commit,
  1490. "--", ".", ":!.agtx/",
  1491. ],
  1492. capture_output=True,
  1493. )
  1494. if result.returncode != 0:
  1495. return ""
  1496. return result.stdout.decode("utf-8", errors="replace")
  1497. worktree_path = (task or {}).get("worktree_path") or str(self.repo_path)
  1498. if not worktree_path:
  1499. return ""
  1500. # Diff the worktree's actual files against the base commit.
  1501. # Using the worktree path directly avoids picking up any extra commits
  1502. # the agent may have merged/fetched onto the branch โ€” we only care about
  1503. # file-level changes, not git history.
  1504. result = subprocess.run(
  1505. ["git", "diff", base_commit,
  1506. "--", ".", ":!.agtx/"],
  1507. cwd=worktree_path,
  1508. capture_output=True,
  1509. )
  1510. if result.returncode != 0:
  1511. return ""
  1512. # Decode as UTF-8, replacing undecodable bytes so binary files don't crash the run
  1513. return result.stdout.decode("utf-8", errors="replace")
  1514. def _cleanup(self, task_id: str | None) -> None:
  1515. if task_id:
  1516. try:
  1517. self.mcp.call("move_task", task_id=task_id, action="move_to_done")
  1518. except Exception:
  1519. pass
  1520. def _error_result(self, status: str, duration: float, error: str) -> dict:
  1521. return {
  1522. "status": status,
  1523. "duration_seconds": duration,
  1524. "model_patch": "",
  1525. "cost_usd": None,
  1526. "cost_tokens": None,
  1527. "error": error,
  1528. }
  1529. # ---------------------------------------------------------------------------
  1530. # Orchestrator
  1531. # ---------------------------------------------------------------------------
  1532. class BenchmarkOrchestrator:
  1533. """Drives N tasks, manages concurrency."""
  1534. def __init__(
  1535. self,
  1536. instances: list[dict],
  1537. agtx_bin: str,
  1538. config_path: Path,
  1539. plugin: str,
  1540. agent: str,
  1541. running_agent: str,
  1542. model_name: str,
  1543. phase_timeout: int,
  1544. workdir: str,
  1545. output_dir: Path,
  1546. concurrency: int,
  1547. verbose: bool = False,
  1548. smoke_test: bool = False,
  1549. hard: bool = False,
  1550. docker: bool = False,
  1551. extra_dirs: list[tuple[Path, str]] | None = None,
  1552. sandbox_init: list[str] | None = None,
  1553. ):
  1554. self.instances = instances
  1555. self.agtx_bin = agtx_bin
  1556. self.config_path = config_path
  1557. self.plugin = plugin
  1558. self.agent = agent
  1559. self.running_agent = running_agent
  1560. self.model_name = model_name
  1561. self.phase_timeout = phase_timeout
  1562. self.workdir = workdir
  1563. self.store = ResultsStore(output_dir)
  1564. self.concurrency = concurrency
  1565. self.verbose = verbose
  1566. self.smoke_test = smoke_test
  1567. self.hard = hard
  1568. self.docker = docker
  1569. self.extra_dirs = extra_dirs or []
  1570. self.sandbox_init = sandbox_init or []
  1571. self.tools_container_id: str | None = None # set in run() when --sandbox is active
  1572. def _run_one(self, instance: dict, progress: tqdm) -> None:
  1573. instance_id = instance["instance_id"]
  1574. if self.store.is_done(instance_id):
  1575. progress.update(1)
  1576. return
  1577. slug = re.sub(r"[^a-z0-9]+", "-", instance_id.lower()).strip("-")[:50]
  1578. start = time.monotonic()
  1579. container_id: str | None = None
  1580. repo_path: Path | None = None
  1581. if self.docker:
  1582. if self.verbose:
  1583. print(f"\n[{instance_id}] Starting Docker container...", file=sys.stderr)
  1584. try:
  1585. container_id = start_docker_container(
  1586. instance_id, slug, self.agtx_bin, verbose=self.verbose
  1587. )
  1588. setup_container(
  1589. container_id, self.config_path, instance["base_commit"],
  1590. verbose=self.verbose, tools_container_id=self.tools_container_id,
  1591. extra_dirs=self.extra_dirs or None,
  1592. sandbox_init=self.sandbox_init or None,
  1593. )
  1594. except Exception as e:
  1595. if container_id:
  1596. stop_docker_container(container_id, verbose=self.verbose)
  1597. self.store.save_result(
  1598. instance_id=instance_id,
  1599. status="setup_error",
  1600. duration_seconds=time.monotonic() - start,
  1601. model_name=self.model_name,
  1602. error=str(e),
  1603. )
  1604. progress.update(1)
  1605. return
  1606. if self.verbose:
  1607. print(f"[{instance_id}] Starting TUI inside container...", file=sys.stderr)
  1608. print(f" [docker] To watch the agent:", file=sys.stderr)
  1609. print(f" docker exec -it swebench-{slug} tmux -L agtx attach -t testbed:1", file=sys.stderr)
  1610. try:
  1611. start_tui_in_container(slug, container_id, verbose=self.verbose)
  1612. # start_tui_in_container already waits for TUI startup internally
  1613. except Exception as e:
  1614. stop_docker_container(container_id, verbose=self.verbose)
  1615. self.store.save_result(
  1616. instance_id=instance_id,
  1617. status="setup_error",
  1618. duration_seconds=time.monotonic() - start,
  1619. model_name=self.model_name,
  1620. error=f"TUI startup failed: {e}",
  1621. )
  1622. progress.update(1)
  1623. return
  1624. # In Docker mode, repo_path is unused by McpClient/patch collection
  1625. # but TaskRunner still needs a Path object (used as identifier).
  1626. repo_path = Path("/testbed")
  1627. else:
  1628. if self.verbose:
  1629. print(f"\n[{instance_id}] Setting up repo...", file=sys.stderr)
  1630. try:
  1631. repo_path = setup_repo(
  1632. instance, self.workdir, self.config_path,
  1633. verbose=self.verbose, smoke_test=self.smoke_test,
  1634. )
  1635. except Exception as e:
  1636. self.store.save_result(
  1637. instance_id=instance_id,
  1638. status="setup_error",
  1639. duration_seconds=time.monotonic() - start,
  1640. model_name=self.model_name,
  1641. error=str(e),
  1642. )
  1643. progress.update(1)
  1644. return
  1645. if self.verbose:
  1646. print(f"[{instance_id}] Starting TUI...", file=sys.stderr)
  1647. try:
  1648. start_tui_in_tmux(slug, repo_path, self.agtx_bin, verbose=self.verbose)
  1649. time.sleep(8) # Wait for TUI startup + project registration
  1650. except Exception as e:
  1651. self.store.save_result(
  1652. instance_id=instance_id,
  1653. status="setup_error",
  1654. duration_seconds=time.monotonic() - start,
  1655. model_name=self.model_name,
  1656. error=f"TUI startup failed: {e}",
  1657. )
  1658. progress.update(1)
  1659. return
  1660. if self.verbose:
  1661. print(f"[{instance_id}] Connecting MCP client...", file=sys.stderr)
  1662. runner = TaskRunner(
  1663. instance=instance,
  1664. repo_path=repo_path,
  1665. agtx_bin=self.agtx_bin,
  1666. plugin=self.plugin,
  1667. agent=self.agent,
  1668. running_agent=self.running_agent,
  1669. model_name=self.model_name,
  1670. phase_timeout=self.phase_timeout,
  1671. verbose=self.verbose,
  1672. smoke_test=self.smoke_test,
  1673. hard=self.hard,
  1674. container_id=container_id,
  1675. )
  1676. result = runner.run()
  1677. if self.docker:
  1678. stop_docker_container(container_id, verbose=self.verbose)
  1679. else:
  1680. kill_tmux_session(slug, repo_path)
  1681. self.store.save_result(
  1682. instance_id=instance_id,
  1683. model_name=self.model_name,
  1684. **result,
  1685. )
  1686. progress.update(1)
  1687. progress.set_postfix_str(f"{result['status']} {instance_id}")
  1688. def run(self) -> None:
  1689. pending = [i for i in self.instances if not self.store.is_done(i["instance_id"])]
  1690. total = len(self.instances)
  1691. already_done = total - len(pending)
  1692. print(f"Agent: {self.agent} | Plugin: {self.plugin} | Tasks: {len(pending)} | Concurrency: {self.concurrency}")
  1693. print(f"Output: {self.store.output_dir}")
  1694. if self.docker and _tools_image_exists():
  1695. if self.verbose:
  1696. print("[docker] Ensuring tools volume is ready...", file=sys.stderr)
  1697. ensure_tools_volume(verbose=self.verbose)
  1698. self.tools_container_id = "volume" # sentinel: volume is mounted, no container needed
  1699. try:
  1700. with tqdm(total=total, initial=already_done, unit="task") as progress:
  1701. if self.concurrency == 1:
  1702. for instance in pending:
  1703. self._run_one(instance, progress)
  1704. else:
  1705. with ThreadPoolExecutor(max_workers=self.concurrency) as pool:
  1706. futures = {
  1707. pool.submit(self._run_one, instance, progress): instance
  1708. for instance in pending
  1709. }
  1710. for future in as_completed(futures):
  1711. exc = future.exception()
  1712. if exc:
  1713. inst = futures[future]
  1714. print(f"\nUnhandled error for {inst['instance_id']}: {exc}", file=sys.stderr)
  1715. finally:
  1716. if self.tools_container_id and self.tools_container_id != "volume":
  1717. stop_tools_container(verbose=self.verbose)
  1718. statuses: dict[str, int] = {}
  1719. for r in self.store._results:
  1720. statuses[r["status"]] = statuses.get(r["status"], 0) + 1
  1721. print(f"\nDone. {total} tasks total.")
  1722. for status, count in sorted(statuses.items()):
  1723. print(f" {status}: {count}")
  1724. # ---------------------------------------------------------------------------
  1725. # Dataset loader
  1726. # ---------------------------------------------------------------------------
  1727. def load_swebench(split: str = "test", dataset_name: str = "princeton-nlp/SWE-bench_Lite") -> list[dict]:
  1728. print(f"Loading {dataset_name} ({split} split)...")
  1729. dataset = load_dataset(dataset_name, split=split)
  1730. return list(dataset)
  1731. # ---------------------------------------------------------------------------
  1732. # CLI
  1733. # ---------------------------------------------------------------------------
  1734. def main() -> None:
  1735. parser = argparse.ArgumentParser(
  1736. description="Run SWE-bench Lite benchmark using agtx as the agent runner.",
  1737. formatter_class=argparse.RawDescriptionHelpFormatter,
  1738. epilog="""
  1739. The --config file is an agtx ProjectConfig TOML written to .agtx/config.toml
  1740. in each cloned repo before the TUI starts. It controls agent, plugin, and all
  1741. other agtx project settings. Example:
  1742. default_agent = "claude"
  1743. workflow_plugin = "agtx"
  1744. worktree_dir = ".agtx/worktrees"
  1745. """,
  1746. )
  1747. parser.add_argument(
  1748. "--config",
  1749. required=True,
  1750. metavar="PATH",
  1751. help="Path to agtx config.toml (written to .agtx/config.toml in each repo)",
  1752. )
  1753. parser.add_argument(
  1754. "--instances",
  1755. type=int,
  1756. default=None,
  1757. metavar="N",
  1758. help="Run first N tasks (default: all 300)",
  1759. )
  1760. parser.add_argument(
  1761. "--instance-ids",
  1762. nargs="+",
  1763. metavar="ID",
  1764. dest="instance_ids",
  1765. help="Run specific instance IDs",
  1766. )
  1767. parser.add_argument(
  1768. "--concurrency",
  1769. type=int,
  1770. default=1,
  1771. help="Parallel tasks (default: 1)",
  1772. )
  1773. parser.add_argument(
  1774. "--output-dir",
  1775. default=None,
  1776. dest="output_dir",
  1777. help="Output directory (default: ./swebench_output/{plugin}_{agent}_{timestamp}/)",
  1778. )
  1779. parser.add_argument(
  1780. "--workdir",
  1781. default="/tmp/swebench_repos",
  1782. help="Repo clone directory (default: /tmp/swebench_repos)",
  1783. )
  1784. parser.add_argument(
  1785. "--agtx",
  1786. default="./target/release/agtx",
  1787. help="Path to agtx binary (default: ../target/release/agtx relative to benchmark/)",
  1788. )
  1789. parser.add_argument(
  1790. "--phase-timeout",
  1791. type=int,
  1792. default=1200,
  1793. dest="phase_timeout",
  1794. help="Per-phase max seconds (default: 1200)",
  1795. )
  1796. parser.add_argument(
  1797. "--model-name",
  1798. default=None,
  1799. dest="model_name",
  1800. help="Label in predictions.jsonl (default: agtx-{plugin}-{agent})",
  1801. )
  1802. parser.add_argument(
  1803. "--split",
  1804. default="test",
  1805. help="HuggingFace dataset split (default: test)",
  1806. )
  1807. parser.add_argument(
  1808. "--dataset",
  1809. default="princeton-nlp/SWE-bench_Lite",
  1810. dest="dataset",
  1811. help="HuggingFace dataset name (default: princeton-nlp/SWE-bench_Lite). "
  1812. "Use princeton-nlp/SWE-bench_Verified for multi-file/long-exploration tasks.",
  1813. )
  1814. parser.add_argument(
  1815. "--verbose", "-v",
  1816. action="store_true",
  1817. help="Print step-by-step progress to stderr",
  1818. )
  1819. parser.add_argument(
  1820. "--smoke-test",
  1821. action="store_true",
  1822. dest="smoke_test",
  1823. help="Replace task description with a trivial prompt (create artifact files and stop). "
  1824. "Use to verify the full pipeline works without spending tokens on real coding work.",
  1825. )
  1826. parser.add_argument(
  1827. "--hard",
  1828. action="store_true",
  1829. dest="hard",
  1830. help="Strip code blocks and stack traces from the problem statement, keeping only the prose. "
  1831. "The agent must find and fix the bug from first principles.",
  1832. )
  1833. parser.add_argument(
  1834. "--sandbox",
  1835. action="store_true",
  1836. dest="docker",
  1837. help=(
  1838. "Run each task inside its SWE-bench Docker image "
  1839. "(swebench/sweb.eval.x86_64.*). Gives agents a working test "
  1840. "environment with pre-installed dependencies. "
  1841. "Requires Docker with Rosetta on Apple Silicon."
  1842. ),
  1843. )
  1844. args = parser.parse_args()
  1845. # Load and validate config
  1846. config_path = Path(args.config).resolve()
  1847. if not config_path.exists():
  1848. print(f"Config not found: {config_path}", file=sys.stderr)
  1849. sys.exit(1)
  1850. config = load_config_toml(config_path)
  1851. plugin = config.get("workflow_plugin", "void")
  1852. agent = config.get("default_agent", "claude")
  1853. run_agent = running_agent(config)
  1854. # Resolve sandbox_copy_dirs โ€” paths relative to the config file's directory
  1855. extra_dirs: list[tuple[Path, str]] = []
  1856. for entry in config.get("sandbox_copy_dirs", []):
  1857. src = (config_path.parent / entry["src"]).resolve()
  1858. dst = entry["dst"]
  1859. extra_dirs.append((src, dst))
  1860. # Resolve sandbox_init โ€” shell commands to run inside the container after setup
  1861. sandbox_init: list[str] = config.get("sandbox_init", [])
  1862. # Resolve agtx binary
  1863. agtx_bin = str(Path(args.agtx).resolve())
  1864. if not Path(agtx_bin).exists():
  1865. print(f"agtx binary not found: {agtx_bin}", file=sys.stderr)
  1866. print("Build with: cargo build --release", file=sys.stderr)
  1867. sys.exit(1)
  1868. model_name = args.model_name or config_path.stem
  1869. if args.output_dir is None:
  1870. ts = datetime.now().strftime("%Y%m%d_%H%M%S")
  1871. config_stem = config_path.stem # e.g. "claude-rtk-caveman-ponytail-agtx"
  1872. output_dir = Path(f"./swebench_output/{config_stem}_{ts}").resolve()
  1873. else:
  1874. output_dir = Path(args.output_dir).resolve()
  1875. # In smoke test mode with no explicit instance selection, skip loading the dataset
  1876. # entirely and use a single synthetic instance.
  1877. if args.smoke_test and args.instance_ids is None and args.instances is None:
  1878. instances = [{"instance_id": "smoke-test", "repo": "", "base_commit": "HEAD", "problem_statement": ""}]
  1879. else:
  1880. instances = load_swebench(args.split, args.dataset)
  1881. if args.instance_ids:
  1882. id_set = set(args.instance_ids)
  1883. instances = [i for i in instances if i["instance_id"] in id_set]
  1884. if not instances:
  1885. print("No matching instance IDs found.", file=sys.stderr)
  1886. sys.exit(1)
  1887. elif args.instances is not None:
  1888. instances = instances[: args.instances]
  1889. orchestrator = BenchmarkOrchestrator(
  1890. instances=instances,
  1891. agtx_bin=agtx_bin,
  1892. config_path=config_path,
  1893. plugin=plugin,
  1894. agent=agent,
  1895. running_agent=run_agent,
  1896. model_name=model_name,
  1897. phase_timeout=args.phase_timeout,
  1898. workdir=args.workdir,
  1899. output_dir=output_dir,
  1900. concurrency=args.concurrency,
  1901. verbose=args.verbose,
  1902. smoke_test=args.smoke_test,
  1903. hard=args.hard,
  1904. docker=args.docker,
  1905. extra_dirs=extra_dirs or None,
  1906. sandbox_init=sandbox_init or None,
  1907. )
  1908. orchestrator.run()
  1909. print(f"\nPredictions: {orchestrator.store.predictions_path}")
  1910. print(f"Results: {orchestrator.store.results_path}")
  1911. run_id = f"{config_path.stem}-{int(time.time())}"
  1912. print("\nTo evaluate:")
  1913. print(f" uv run python -m swebench.harness.run_evaluation \\")
  1914. print(f" --dataset_name {args.dataset} \\")
  1915. print(f" --predictions_path {orchestrator.store.predictions_path} \\")
  1916. print(f" --run_id {run_id}")
  1917. print("\nTo report (after evaluation):")
  1918. print(f" uv run python swebench/report.py \\")
  1919. print(f" --results {orchestrator.store.results_path} \\")
  1920. print(f" --logs logs/run_evaluation/{run_id}/")
  1921. if __name__ == "__main__":
  1922. main()
benchmark/swebench/prebake_images.py ยท Bundled script ยท 91 lines ยท 1 finding marked
  1. #!/usr/bin/env python3
  2. """
  3. Build the shared agtx/swebench-tools:latest image with tmux, Node.js, Claude Code, and Grok.
  4. This single image is used by all benchmark sandbox runs โ€” the benchmark runner copies
  5. the installed binaries from a running tools container into each SWE-bench instance
  6. container at startup, saving ~3-5 min per instance compared to installing from scratch.
  7. The tools image is based on Ubuntu 22.04 (matching SWE-bench images) and contains only
  8. the agent tooling layer. SWE-bench instance images are used as-is from Docker Hub.
  9. Usage:
  10. # Build the tools image (run once, or after updating Claude Code / Grok)
  11. python prebake_images.py
  12. # Force rebuild even if image already exists
  13. python prebake_images.py --force
  14. # Show docker build output
  15. python prebake_images.py --verbose
  16. """
  17. from __future__ import annotations
  18. import argparse
  19. import subprocess
  20. import sys
  21. TOOLS_IMAGE = "agtx/swebench-tools:latest"
  22. DOCKERFILE = """\
  23. FROM --platform=linux/amd64 ubuntu:22.04
  24. ENV DEBIAN_FRONTEND=noninteractive
  25. RUN apt-get update -qq \\
  26. && apt-get install -y -qq tmux curl ca-certificates \\
  27. && curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \\
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 36 on GitHub
  28. && apt-get install -y -qq nodejs \\
  29. && npm install -g @anthropic-ai/claude-code @xai-official/grok \\
  30. && apt-get clean \\
  31. && rm -rf /var/lib/apt/lists/*
  32. """
  33. def tools_image_exists() -> bool:
  34. result = subprocess.run(
  35. ["docker", "image", "inspect", TOOLS_IMAGE],
  36. capture_output=True,
  37. )
  38. return result.returncode == 0
  39. def build_tools_image(force: bool = False, verbose: bool = False) -> None:
  40. if not force and tools_image_exists():
  41. print(f"Tools image already exists: {TOOLS_IMAGE}")
  42. print("Use --force to rebuild.")
  43. return
  44. print(f"Building {TOOLS_IMAGE}...")
  45. capture = not verbose
  46. subprocess.run(
  47. ["docker", "build", "--platform", "linux/amd64", "-t", TOOLS_IMAGE, "-"],
  48. input=DOCKERFILE,
  49. text=True,
  50. capture_output=capture,
  51. check=True,
  52. )
  53. print(f"Done: {TOOLS_IMAGE}")
  54. def main() -> None:
  55. parser = argparse.ArgumentParser(
  56. description="Build the shared agtx/swebench-tools image for sandbox benchmark runs.",
  57. formatter_class=argparse.RawDescriptionHelpFormatter,
  58. epilog=__doc__,
  59. )
  60. parser.add_argument(
  61. "--force",
  62. action="store_true",
  63. help="Rebuild even if the tools image already exists",
  64. )
  65. parser.add_argument(
  66. "--verbose", "-v",
  67. action="store_true",
  68. help="Show docker build output",
  69. )
  70. args = parser.parse_args()
  71. build_tools_image(force=args.force, verbose=args.verbose)
  72. if __name__ == "__main__":
  73. main()
install.sh ยท Bundled script ยท 200 lines ยท 1 finding marked
  1. #!/bin/bash
  2. set -e
  3. # agtx installer
  4. # Usage: curl -fsSL https://raw.githubusercontent.com/fynnfluegge/agtx/main/install.sh | bash
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 5 on GitHub
  5. REPO="fynnfluegge/agtx"
  6. BINARY_NAME="agtx"
  7. INSTALL_DIR="${AGTX_INSTALL_DIR:-$HOME/.local/bin}"
  8. # Colors
  9. RED='\033[0;31m'
  10. GREEN='\033[0;32m'
  11. YELLOW='\033[0;33m'
  12. BLUE='\033[0;34m'
  13. NC='\033[0m' # No Color
  14. info() {
  15. echo -e "${BLUE}==>${NC} $1"
  16. }
  17. success() {
  18. echo -e "${GREEN}==>${NC} $1"
  19. }
  20. warn() {
  21. echo -e "${YELLOW}==>${NC} $1"
  22. }
  23. error() {
  24. echo -e "${RED}==>${NC} $1"
  25. exit 1
  26. }
  27. # Detect OS
  28. detect_os() {
  29. case "$(uname -s)" in
  30. Linux*) echo "linux" ;;
  31. Darwin*) echo "darwin" ;;
  32. *) error "Unsupported OS: $(uname -s)" ;;
  33. esac
  34. }
  35. # Detect architecture
  36. detect_arch() {
  37. case "$(uname -m)" in
  38. x86_64) echo "x86_64" ;;
  39. amd64) echo "x86_64" ;;
  40. arm64) echo "aarch64" ;;
  41. aarch64) echo "aarch64" ;;
  42. *) error "Unsupported architecture: $(uname -m)" ;;
  43. esac
  44. }
  45. # Get latest release version
  46. get_latest_version() {
  47. curl -fsSL "https://api.github.com/repos/${REPO}/releases/latest" | \
  48. grep '"tag_name"' | \
  49. sed -E 's/.*"([^"]+)".*/\1/'
  50. }
  51. # Main installation
  52. main() {
  53. echo ""
  54. echo " โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ"
  55. echo " โ”‚ agtx installer โ”‚"
  56. echo " โ”‚ Terminal Kanban โ”‚"
  57. echo " โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ"
  58. echo ""
  59. OS=$(detect_os)
  60. ARCH=$(detect_arch)
  61. info "Detected: ${OS}/${ARCH}"
  62. # Check for required tools
  63. if ! command -v curl &> /dev/null; then
  64. error "curl is required but not installed"
  65. fi
  66. # Version: AGTX_VERSION pins one (e.g. AGTX_VERSION=v0.2.6), which is the
  67. # downgrade path when a release goes wrong. Otherwise take the latest.
  68. if [ -n "${AGTX_VERSION:-}" ]; then
  69. VERSION="${AGTX_VERSION}"
  70. info "Pinned version: ${VERSION}"
  71. else
  72. info "Fetching latest version..."
  73. VERSION=$(get_latest_version)
  74. if [ -z "$VERSION" ]; then
  75. error "Could not determine latest version. Check https://github.com/${REPO}/releases"
  76. fi
  77. info "Latest version: ${VERSION}"
  78. fi
  79. # Construct download URL
  80. ARCHIVE_NAME="${BINARY_NAME}-${VERSION}-${ARCH}-${OS}.tar.gz"
  81. DOWNLOAD_URL="https://github.com/${REPO}/releases/download/${VERSION}/${ARCHIVE_NAME}"
  82. # Create temp directory
  83. TMP_DIR=$(mktemp -d)
  84. trap "rm -rf ${TMP_DIR}" EXIT
  85. # Download
  86. info "Downloading ${ARCHIVE_NAME}..."
  87. if ! curl -fsSL "${DOWNLOAD_URL}" -o "${TMP_DIR}/${ARCHIVE_NAME}"; then
  88. error "Download failed. Check if release exists: ${DOWNLOAD_URL}"
  89. fi
  90. # Verify checksum (if available)
  91. CHECKSUM_URL="https://github.com/${REPO}/releases/download/${VERSION}/${ARCHIVE_NAME}.sha256"
  92. info "Verifying checksum..."
  93. if curl -fsSL "${CHECKSUM_URL}" -o "${TMP_DIR}/${ARCHIVE_NAME}.sha256" 2>/dev/null; then
  94. PREV_DIR=$(pwd)
  95. cd "${TMP_DIR}"
  96. if command -v sha256sum &> /dev/null; then
  97. sha256sum -c "${ARCHIVE_NAME}.sha256" || error "Checksum verification failed"
  98. elif command -v shasum &> /dev/null; then
  99. shasum -a 256 -c "${ARCHIVE_NAME}.sha256" || error "Checksum verification failed"
  100. else
  101. warn "No sha256sum or shasum found, skipping checksum verification"
  102. fi
  103. cd "${PREV_DIR}"
  104. success "Checksum verified"
  105. else
  106. warn "Checksum file not found, skipping verification"
  107. fi
  108. # Extract
  109. info "Extracting..."
  110. tar -xzf "${TMP_DIR}/${ARCHIVE_NAME}" -C "${TMP_DIR}"
  111. # Create install directory if needed
  112. if [ ! -d "${INSTALL_DIR}" ]; then
  113. info "Creating ${INSTALL_DIR}..."
  114. mkdir -p "${INSTALL_DIR}"
  115. fi
  116. # Install binary
  117. info "Installing to ${INSTALL_DIR}/${BINARY_NAME}..."
  118. mv "${TMP_DIR}/${BINARY_NAME}" "${INSTALL_DIR}/${BINARY_NAME}"
  119. chmod +x "${INSTALL_DIR}/${BINARY_NAME}"
  120. # Check if install dir is in PATH
  121. if [[ ":$PATH:" != *":${INSTALL_DIR}:"* ]]; then
  122. warn "${INSTALL_DIR} is not in your PATH"
  123. echo ""
  124. echo "Add it to your shell config:"
  125. echo ""
  126. echo " # For bash (~/.bashrc)"
  127. echo " export PATH=\"\$HOME/.local/bin:\$PATH\""
  128. echo ""
  129. echo " # For zsh (~/.zshrc)"
  130. echo " export PATH=\"\$HOME/.local/bin:\$PATH\""
  131. echo ""
  132. fi
  133. echo ""
  134. success "agtx ${VERSION} installed successfully!"
  135. echo ""
  136. echo " Get started:"
  137. echo " cd your-project"
  138. echo " agtx"
  139. echo ""
  140. echo " Or run in dashboard mode:"
  141. echo " agtx -g"
  142. echo ""
  143. # Check dependencies
  144. echo " Checking dependencies..."
  145. if command -v tmux &> /dev/null; then
  146. echo -e " ${GREEN}โœ“${NC} tmux"
  147. else
  148. echo -e " ${RED}โœ—${NC} tmux (required)"
  149. fi
  150. if command -v git &> /dev/null; then
  151. echo -e " ${GREEN}โœ“${NC} git"
  152. else
  153. echo -e " ${RED}โœ—${NC} git (required)"
  154. fi
  155. if command -v gh &> /dev/null; then
  156. echo -e " ${GREEN}โœ“${NC} gh"
  157. else
  158. echo -e " ${YELLOW}โ—‹${NC} gh (optional, for PR operations)"
  159. fi
  160. if command -v claude &> /dev/null; then
  161. echo -e " ${GREEN}โœ“${NC} claude"
  162. else
  163. echo -e " ${YELLOW}โ—‹${NC} claude (optional, for Claude Code integration)"
  164. fi
  165. echo ""
  166. }
  167. main "$@"

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If itโ€™s helped, pass it on.