agtx
๐๐ผโโ๏ธ The blackboard for coding agents - agentic development environment for claude code, codex, cursor, opencode, grok and more.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/fynnfluegge--agtx)HTML
<a href="/listings/fynnfluegge--agtx"><img src="/api/listings/fynnfluegge--agtx/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
These steps run a script from the network that findsafeskills does not scan.
# Install curl -fsSL https://raw.githubusercontent.com/fynnfluegge/agtx/main/install.sh | bash
# Run in any git repository cd your-project && agtx
> [!NOTE] > Add `.agtx/` to your project's `.gitignore` to avoid committing worktrees and local task data.
# Install from source โ `serve` adds the mobile board (`W`), and the # published binaries are built with it cargo build --release --features serve cp target/release/agtx ~/.local/bin/
Updating
agtx checks GitHub once a day for a newer release and shows `โฌ 0.2.8 [u]` in the board header when there is one. Press **`u`** for the details and one key to install it, or from a shell:
agtx update # download, verify and replace this binary in place agtx update --check # report only; exits 1 when an update is available agtx --version
Requirements
- **tmux** โ agent sessions run in a dedicated tmux server - **gh** (optional) โ GitHub CLI for PR operations
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json ยท Manifest ยท 14 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- {
- "name": "agtx-marketplace",
- "description": "Marketplace for agtx plugins",
- "owner": {
- "name": "fynnfluegge"
- },
- "plugins": [
- {
- "name": "agtx",
- "description": "Manage and decompose work into feature-level tasks on the agtx kanban board",
- "source": "./"
- }
- ]
- }
README.md ยท README ยท 1,000 lines ยท 1 finding marked
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <div align="center">
- [//]: <img src="https://github.com/user-attachments/assets/54ac039b-085e-490b-aacc-36c8e244e313" width="428" />
- # ๐๐ผโโ๏ธ agtx
- **The terminal-native agentic development environment for 10x productivity.**
- <div align="left">
- > **A blackboard for coding agents** - One shared board. A fleet of agents. Add tasks, delegate to multiple coding agents running in parallel and let
- > **different models collaborate** on the same task with automatic session switching and context awareness, e.g. Codex planning, Claude implementing, Grok review.
- </div>
- [](https://github.com/fynnfluegge/agtx/actions/workflows/ci.yml)
- [](https://github.com/fynnfluegge/agtx/releases)
- [](CONTRIBUTING.md)
- [](https://opensource.org/licenses/Apache-2.0)
- <a href="https://trendshift.io/repositories/23889" target="_blank"><img src="https://trendshift.io/api/badge/repositories/23889" alt="fynnfluegge%2Fagtx | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
- <p align="center">
- <a href="#quick-start">Quick Start</a> โข
- <a href="#features">Features</a> โข
- <a href="#usage">Usage</a> โข
- <a href="#mobile">Mobile</a> โข
- <a href="#skills">Skills</a> โข
- <a href="#oneshot-skill">Oneshot</a> โข
- <a href="#mcp-server">MCP Server</a> โข
- <a href="#plugins">Plugins</a> โข
- <a href="#orchestrator-agent-experimental">Orchestrator</a> โข
- <a href="#configuration">Configuration</a>
- </p>
- ---
- <img width="840" src="https://github.com/user-attachments/assets/45858e09-ab61-422b-b708-db060c73a900" />
- [//]: <img width="840" src="https://github.com/user-attachments/assets/42f71a6c-424c-4cc4-80fc-dc9bb8ba1467" />
- <br/>
- [//]: <img width="1486" height="680" src="https://github.com/user-attachments/assets/45858e09-ab61-422b-b708-db060c73a900" />
- [//]: <>
- </div>
- ## Features
- - **Supported Agents**: <a href="https://github.com/anthropics/claude-code"><kbd><img src="docs/logos/claude.svg" width="18" valign="middle" /> Claude Code</kbd></a>
- <a href="https://github.com/openai/codex"><kbd><img src="docs/logos/codex-dark.svg" width="18" valign="middle" /> Codex</kbd></a>
- <a href="https://docs.x.ai/build/overview"><kbd><img src="docs/logos/grok-dark.svg" width="18" valign="middle" /> Grok</kbd></a>
- <a href="https://cursor.com/docs/cli/overview"><kbd><img src="docs/logos/cursor-dark.svg" width="18" valign="middle" /> Cursor</kbd></a>
- <a href="https://github.com/sst/opencode"><kbd><img src="docs/logos/opencode-dark.svg" width="18" valign="middle" /> OpenCode</kbd></a>
- <a href="https://github.com/google-antigravity/antigravity-cli"><kbd><img src="docs/logos/antigravity.png" width="18" valign="middle" /> Antigravity</kbd></a>
- <a href="https://github.com/google-gemini/gemini-cli"><kbd><img src="docs/logos/gemini.svg" width="18" valign="middle" /> Gemini CLI</kbd></a>
- <a href="https://github.com/github/copilot-cli"><kbd><img src="docs/logos/copilot-dark.svg" width="18" valign="middle" /> Copilot</kbd></a>
- <a href="https://github.com/earendil-works/pi"><kbd><img src="docs/logos/pi-dark.svg" width="18" valign="middle" /> pi</kbd></a>
- - **Parallel Multi-agent task lifecycle**: Configure different agents per workflow phase โ e.g. Grok for research, Claude for implementation, Codex for review โ with automatic agent switching and context handover.
- - **Multi-project Kanban board**: Manage agent sessions across all projects via a single TUI without leaving your terminal.
- - **Vim-native Keybindings**: Control the board and agent sessions with your Vim-powered muscle memory.
- - **Mobile support**: Press `W` and scan the QR for an installable web app with the board, task details, diffs and the agent's live terminal โ including a keyboard, so you can answer a permission prompt without going back to your desk. Over your wifi, or from anywhere via your tailnet. See <a href="#mobile">Mobile</a>.
- - **Orchestrator agent (experimental)**: A dedicated agent that autonomously manages your kanban board via MCP โ delegates to coding agents, advances phases, checks for merge conflicts.
- - **Oneshot skill**: Give a coding agent session a goal with `/agtx:oneshot` and it runs the whole board unattended โ plans the work in milestone waves, starts tasks, unblocks the workers, judges Review, and merges each task. See <a href="#oneshot-skill">Oneshot Skill</a>.
- - **Brainstorm & Sweep skills**: Capture ideas and push them to the board from any coding agent session โ `/agtx:brainstorm` to explore freely, `/agtx:sweep` to decompose and create tasks with one confirmation step.
- - **Spec-driven plugins**: Plug in [GSD](https://github.com/fynnfluegge/get-shit-done-cc), [Spec-kit](https://github.com/github/spec-kit), [OpenSpec](https://github.com/Fission-AI/OpenSpec), [BMAD](https://github.com/bmad-code-org/BMAD-METHOD), [Superpowers](https://github.com/obra/superpowers) and more โ fully customizable. Ddefine your own workflow via a single TOML file. See <a href="#plugins">Plugins</a> how to create a plugin.
- > [!NOTE]
- > Just need a plain coding-agent session-manager with **full human-in-the-loop control** and **no spec-driven skill execution and orchestration** on advancing tasks?
- >
- > Choose the **`void` plugin** and enjoy agtx as a batteries included multi-agent session-manager.
- ## Quick Start
- ```bash
- # Install
- curl -fsSL https://raw.githubusercontent.com/fynnfluegge/agtx/main/install.sh | bash
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 78 on GitHub
- ```
- ```bash
- # Run in any git repository
- cd your-project && agtx
- ```
- > [!NOTE]
- > Add `.agtx/` to your project's `.gitignore` to avoid committing worktrees and local task data.
- ```bash
- # Install from source โ `serve` adds the mobile board (`W`), and the
- # published binaries are built with it
- cargo build --release --features serve
- cp target/release/agtx ~/.local/bin/
- ```
- ### Updating
- agtx checks GitHub once a day for a newer release and shows `โฌ 0.2.8 [u]` in the
- board header when there is one. Press **`u`** for the details and one key to
- install it, or from a shell:
- ```bash
- agtx update # download, verify and replace this binary in place
- agtx update --check # report only; exits 1 when an update is available
- agtx --version
- ```
- ### Requirements
- - **tmux** โ agent sessions run in a dedicated tmux server
- - **gh** (optional) โ GitHub CLI for PR operations
- ## Usage
- <details>
- <summary><strong>Keyboard Shortcuts</strong></summary>
- | Key | Action |
- |-----|--------|
- | `h/l` or `โ/โ` | Move between columns |
- | `j/k` or `โ/โ` | Move between tasks |
- | `o` | Create new task |
- | `R` | Enter research mode |
- | `โฉ` | Open task (view agent session) |
- | `Ctrl+f` | Open the task popup fullscreen; press again to return to windowed mode |
- | `m` | Move task forward in workflow |
- | `r` | Resume task (Review โ Running) / Move back (Running โ Planning) |
- | `p` | Next phase (Review โ Planning, cyclic plugins only) |
- | `d` | Show git diff |
- | `x` | Delete task |
- | `/` | Search tasks |
- | `P` | Select spec-driven workflow plugin |
- | `,` | Open the config editor |
- | `?` | Show every keyboard shortcut |
- | `u` | Update agtx (only shown when a new release is available) |
- | `W` | Serve the board to a phone (QR pairing, device list) |
- | `O` | Toggle orchestrator agent (`--experimental`) |
- | `e` | Toggle project sidebar |
- | `q` | Quit |
- </details>
- <details>
- <summary><strong>Task Creation Wizard</strong></summary>
- Press `o` to create a new task. The wizard guides you through:
- 1. **Title** โ enter a short task name
- 2. **Plugin** โ select a workflow plugin (auto-skipped if only one option)
- 3. **Prompt** โ write a detailed task description with inline references
- The agent is configured at the project level via `config.toml` (not per-task).
- </details>
- <details>
- <summary><strong>Task Description Editor</strong></summary>
- When writing a task description, you can reference files, skills, and other tasks inline:
- | Key | Action |
- |-----|--------|
- | `#` or `@` | Fuzzy search and insert a file path |
- | `/` | Fuzzy search and insert an agent skill/command (at line start or after space) |
- | `!` | Fuzzy search and insert a task reference (at line start or after space) |
- </details>
- <details>
- <summary><strong>Agent Sessions</strong></summary>
- Each task runs in its own tmux window with a dedicated coding agent. The session persists across the entire task lifecycle โ you can open the task popup at any time to see live agent output, or press `Ctrl+f` to open it fullscreen inside agtx.
- - **Persistent context**: The agent's full conversation history is preserved across Planning โ Running โ Review
- - **Resume from Review**: Moving a task back to Running simply reconnects to the existing session โ no re-initialization needed
- - **Inline view**: Press `โฉ` on any active task to open a scrollable tmux view inside the TUI
- - **Fullscreen**: Press `Ctrl+f` to expand the task popup inside agtx. Press `Ctrl+f` again for windowed mode or `Ctrl+q` to return to the board.
- - **Auto merge-conflict resolution**: When a Review task becomes idle, agtx checks for merge conflicts with the default branch using a non-destructive virtual merge (`git merge-tree`). If conflicts are detected, the agent is automatically sent the `/agtx:merge-conflicts` skill to resolve them and re-commit
- </details>
- ## Why agtx? - The blackboard model
- Most AI coding tools give you one agent, one task, one terminal. agtx is built on a different and much
- older idea: the [**blackboard system**](https://en.wikipedia.org/wiki/Blackboard_system).
- > A blackboard system is an approach where a common knowledge base โ the *blackboard* โ is iteratively
- > updated by a diverse group of specialist *knowledge sources*, starting from a problem specification
- > and ending with a solution. Each specialist writes a partial solution to the blackboard when the
- > state on the board matches what it can contribute.
- >
- > โ after [*Blackboard system*](https://en.wikipedia.org/wiki/Blackboard_system), Wikipedia (CC BY-SA)
- That architecture was designed for problems that are too ill-defined for a single solver and too
- interdependent to split cleanly up front. **Shipping software with coding agents is exactly that
- problem**, so agtx implements the model directly:
- The dependency graph gives the blackboard its structure. Tasks references they build on,
- forming a graph of partial solutions - agtx holds downstream tasks until their dependencies reach
- Review or Done, then carries the relevant diffs and artifacts into the dependent task's context.
- ```
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- โ CONTROL orchestrator agent ยท phase gates ยท dep graph โ
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- โ
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- โ THE BLACKBOARD โ
- โ backlog โ planning โ running โ review โ done โ
- โ dependency graph ยท specs ยท plans ยท diffs ยท reviews โ
- โโโโโโฒโโโโโโโโโโฒโโโโโโโโโโฒโโโโโโโโโโฒโโโโโโโโโโฒโโโโโโโโโโฒโโโโโ
- โ โ โ โ โ โ
- โโโโโโดโโโโ โโโโโดโโโโ โโโโโดโโโโ โโโโโดโโโโ โโโโโดโโโโ โโโโโดโโโโ
- โ Claude โ โ Codex โ โGemini โ โCursor โ โ Grok โ โ ... โ
- โโโโโโโโโโ โโโโโโโโโ โโโโโโโโโ โโโโโโโโโ โโโโโโโโโ โโโโโโโโโ
- KNOWLEDGE SOURCES โ one git worktree + tmux window each
- ```
- | Blackboard model | In agtx |
- |------------------|---------|
- | **The blackboard** โ a shared repository of the problem, partial solutions and contributed information | The kanban board, its dependency graph, and everything the phases leave behind: specs, plans, diffs, reviews, and phase artifacts. Every agent reads from and writes to the same board |
- | **Knowledge sources** โ independent specialists that never talk to each other, only to the board | Eight coding agent CLIs, each running in its **own git worktree and tmux window**. No agent can see another's context โ they exchange only what lands on the board |
- | **Control shell** โ decides opportunistically which specialist runs next | Plugin phase gates determine when a task can advance; the dependency graph determines which tasks are ready to start; and the [orchestrator agent](#orchestrator-agent-experimental) coordinates the board over MCP |
- ## Mobile
- [//]: <> (screenshot: the board on a phone โ take it on a real device, then drag it into a GitHub comment and paste the attachment URL here, as with the TUI shot above)
- Press `W` on the board to serve it to your phone. agtx prints a QR code; scanning
- it pairs the device and opens an installable web app with the board, task
- details, git diffs, and the agent's live terminal โ including a keyboard, so you
- can answer a permission prompt from wherever you are.
- ```bash
- # Or start it yourself, without the TUI
- agtx serve # this machine only
- agtx serve --tunnel # your tailnet โ anywhere, your devices only
- agtx serve --devices # list paired devices
- agtx serve --revoke <id> # revoke one; --revoke-all for the lot
- ```
- Two keys in the overlay, each a whole action: `s` serves to the local network,
- `t` serves via your tailnet.
- > [!IMPORTANT]
- > **agtx never opens a port on its own.** Serving lasts for as long as that agtx
- > runs: quit it and the server stops, so the next time you want the board on
- > your phone you press `W` then `s`/`t` again. Pairing does not change this โ it
- > is a credential, not a trigger, and a paired phone whose Mac is not serving
- > just times out.
- > [!NOTE]
- > This needs a binary built with `--features serve`. The released ones are; a
- > local `cargo build --release` without it reports "this build has no web
- > server" under both options.
- <details>
- <summary><strong>Setup โ on your wifi</strong></summary>
- Works in about thirty seconds, and needs nothing installed.
- 1. Run `agtx` in a project and press `W`
- 2. Press `s`
- 3. Point your phone's camera at the QR code โ the phone must be on the same wifi
- 4. In Safari or Chrome: **Share โ Add to Home Screen**
- That last step is what turns it into an app: its own icon, no address bar, and
- the pairing remembered so you never scan again โ but you still press `W` then
- `s` to start serving each time you run agtx.
- The URL here is a private address like `192.168.1.20`, which exists only on your
- network. **It will not work on mobile data** โ if you leave the house, the app
- will sit there timing out. That is what the tailnet option below is for.
- </details>
- <details>
- <summary><strong>Setup โ from anywhere, via Tailscale</strong></summary>
- A few minutes once, then it is the same two taps forever. Your board becomes
- reachable from your phone on any network, while staying invisible to everyone
- else.
- **One-time, on your Mac and your phone:**
- 1. Install [Tailscale](https://tailscale.com/download) on both
- 2. Sign both into the **same** account โ they need to be on one tailnet
- 3. Enable Serve for your tailnet. It is off by default and is a separate switch
- from installing Tailscale โ the step most people miss. The simplest way is to
- just press `t` in agtx once: Tailscale refuses with a one-time link that
- enables it for this machine, and agtx prints that link rather than swallowing
- it. Follow it, then press `t` again.
- **Once more, to install the app:**
- 1. Run `agtx`, press `W`, press `t`
- 2. Scan the QR, and **Share โ Add to Home Screen**
- **Then, every time you want the board on your phone:** run `agtx`, press `W`,
- press `t`. That is all โ the pairing is remembered, so there is no QR to scan;
- what you are starting is the server.
- Now the icon on your home screen works on 5G, on hotel wifi, anywhere โ the
- address is an HTTPS name on your tailnet, and only devices signed into it can
- resolve or reach it.
- </details>
- <details>
- <summary><strong>When it does not work</strong></summary>
- | What you see | What it means |
- |---|---|
- | `Unavailable: install Tailscale and sign this machine in` | `t` needs Tailscale on *this* machine. Install it, or use `s` for wifi. |
- | `Unavailable: Tailscale is installed but not signed in` | Run `tailscale up`. |
- | `could not start the tunnel: โฆ Serve is not enabled on your tailnet` | The one-time switch above. agtx relays Tailscale's own message, which carries the URL that enables it. |
- | Scanned fine, then the app times out | You pressed `s` (wifi) and the phone is on mobile data. Press `s` again to stop, then `t`. |
- | The tailnet URL loads on the Mac but not the phone | The phone is not on the tailnet. Open the Tailscale app there and connect. |
- | `This device is not authorised` | The pairing code expired โ it lasts two minutes and is single-use. Press `s` or `t` again for a fresh QR. |
- | The app worked yesterday, today it just times out | Nothing is serving. agtx does not start the server on its own โ run it, press `W`, then `s`/`t`. The pairing is still good; the port is not open. |
- | `No agtx running` on the board | Expected with no TUI open. Reading works; actions queue until one is. |
- </details>
- > [!IMPORTANT]
- > Anything that can reach this server can read every task, every diff and every
- > agent's screen, and can type into a running agent โ which is arbitrary code
- > execution on your machine. So: loopback needs no credential because reaching it
- > already means being on the machine, and **everything wider requires a paired
- > device**. Tokens are per-device and stored hashed, so a lost phone is revoked
- > without disturbing the rest. `--tunnel public` publishes to the open internet
- > and is deliberately not offered behind a keypress.
- **Actions queue; they do not execute on their own.** Moving a task writes a
- request that only a running `agtx` picks up, so with no TUI open your taps are
- accepted and then wait โ the board says so rather than pretending. Creating,
- editing and deleting Backlog tasks take effect immediately, since they need no
- agent. Starting the server with `W` keeps the two together by construction.
- ## Skills
- Companion skills for any coding agent session: capture ideas, turn them into tasks on the agtx board, or hand a session the whole board.
- | Skill | Command | When to use |
- |-------|---------|-------------|
- | **Brainstorm** | `/agtx:brainstorm` | Explore a feature idea โ discussion only, no planning or implementation |
- | **Sweep** | `/agtx:sweep` | Push conversation outcomes to the agtx board as tasks |
- | **Oneshot** | `/agtx:oneshot` | Give a goal and let the session run the whole board unattended โ see [Oneshot Skill](#oneshot-skill) |
- **Typical flow:**
- ```
- /agtx:brainstorm โ explore the idea freely
- โ
- /agtx:sweep โ extract tasks, confirm, push to board
- โ
- agtx board โ tasks appear in Backlog, ready to advance
- ```
- The brainstorm skill keeps the agent in discussion mode โ asking questions, surfacing trade-offs, no code or plans. When the conversation feels complete, run `/agtx:sweep` to decompose outcomes into feature-level tasks and push them to the board with a single confirmation step.
- ### Install
- The Claude Code plugin brings all three skills. Oneshot runs from Claude Code; the steps for the other agents install sweep.
- <details>
- <summary><strong>Claude Code</strong></summary>
- ```bash
- claude plugin marketplace add fynnfluegge/agtx
- claude plugin install agtx@agtx-marketplace
- claude mcp add --scope user agtx -- agtx mcp-serve
- ```
- </details>
- <details>
- <summary><strong>Codex</strong></summary>
- ```bash
- codex mcp add agtx -- agtx mcp-serve
- ```
- Add to your project's `.agents/plugins/marketplace.json`:
- ```json
- {
- "name": "local-repo",
- "plugins": [
- {
- "name": "agtx",
- "source": {
- "source": "local",
- "path": "./plugins/agtx"
- },
- "policy": {
- "installation": "AVAILABLE",
- "authentication": "ON_INSTALL"
- },
- "category": "Productivity"
- }
- ]
- }
- ```
- Then in any Codex session: `@agtx:sweep` / `@agtx:brainstorm`
- </details>
- <details>
- <summary><strong>Gemini CLI</strong></summary>
- ```bash
- gemini mcp add agtx -- agtx mcp-serve
- echo "@skills/sweep/SKILL.md" >> ~/GEMINI.md
- ```
- </details>
- <details>
- <summary><strong>Cursor</strong></summary>
- ```bash
- cursor mcp add agtx -- agtx mcp-serve
- cp skills/sweep/SKILL.md ~/.cursor/rules/agtx-sweep.md
- ```
- </details>
- <details>
- <summary><strong>Grok Build</strong></summary>
- ```bash
- grok mcp add agtx -- agtx mcp-serve
- mkdir -p ~/.grok/skills/agtx-sweep && cp skills/sweep/SKILL.md ~/.grok/skills/agtx-sweep/SKILL.md
- ```
- </details>
- <details>
- <summary><strong>Antigravity</strong></summary>
- ```bash
- agy mcp add agtx agtx mcp-serve
- mkdir -p ~/.gemini/antigravity-cli/skills/agtx-sweep
- cp skills/sweep/SKILL.md ~/.gemini/antigravity-cli/skills/agtx-sweep/SKILL.md
- ```
- </details>
- <details>
- <summary><strong>pi</strong></summary>
- pi has no MCP client of its own โ the `pi-mcp-adapter` package provides one and
- reads servers in the standard `mcpServers` shape:
- ```bash
- mkdir -p ~/.pi/skills/agtx-sweep && cp skills/sweep/SKILL.md ~/.pi/skills/agtx-sweep/SKILL.md
- ```
- Register `agtx mcp-serve` with the adapter, then in any pi session:
- `/skill:agtx-sweep` / `/skill:agtx-brainstorm`
- </details>
- <details>
- <summary><strong>Other</strong></summary>
- Register `agtx mcp-serve` as an MCP server, then copy `skills/sweep/SKILL.md` into your agent's context.
- </details>
- > [!NOTE]
- > The project must have been opened in agtx at least once to appear in `list_projects`. Run `agtx` in your project directory first.
- ## Oneshot Skill
- > Give it a goal, not a task list. It runs the board until the goal is built and merged.
- `/agtx:oneshot` turns a Claude Code session into the person sitting at the board. It breaks
- the goal into milestones, turns the current milestone into tasks, starts them, watches the
- workers, answers their questions, judges each Review, and merges the finished work into your
- base branch โ then plans the next wave from what was actually built. The workers write the
- code; the oneshot session only writes tasks and its own state file.
- ### Setup
- 1. **Install the plugin and MCP server for Claude Code** โ see [Install](#install).
- 2. **Let agtx answer trust prompts.** Nobody is at the board to answer an agent's trust or
- bypass-permissions dialog, so without this every task parks as blocked:
- ```toml
- # ~/.config/agtx/config.toml
- auto_trust = true
- ```
- 3. **Open your coding agent once in the project root** and accept its trust prompt. Task
- worktrees live inside the project, so they inherit that decision.
- 4. **Start from a clean base branch.** An empty repository works too โ `git init -b main` is
- enough, and agtx makes the first commit.
- ### Run
- ```bash
- # Terminal 1 โ the board. Nothing moves without it.
- cd your-project && agtx
- # Terminal 2 โ the session that runs the board
- cd your-project && claude
- > /agtx:oneshot Build a CLI todo app in Rust with SQLite storage and tests
- ```
- Then leave it. Follow along on the board, or on your phone with `W`.
- ### What it does
- - **Plans in waves.** A spine of 4โ8 milestones first, then tasks for the current milestone
- only, with dependencies wired so a task waits for the ones it builds on. The next wave is
- planned from what the workers built.
- - **Runs 3โ6 tasks at once**, starting the next as its dependencies reach Review or Done.
- - **Waits instead of polling.** `wait_for_board_change` blocks until a task needs attention
- and returns only what changed, so the session spends turns only when something happened.
- - **Unblocks the workers.** It answers a tool's yes/no prompt, or a question its milestone
- plan already settles. Anything that would change the product goes to you under *Open
- questions* in `oneshot-state.md`, while the rest of the board keeps moving.
- - **Judges Review.** A small fix goes to the reviewer in place with `send_to_task`; real
- rework sends the task back to Running.
- - **Merges each task** with `move_to_done_and_merge`. On a conflict the task stays in Review
- and its own agent resolves it with `/agtx:merge-conflicts`.
- - **Keeps a durable record** in `oneshot-state.md` โ milestones, board, decisions and open
- questions โ so a fresh session can pick up the run.
- > [!IMPORTANT]
- > Oneshot merges into your **local** base branch, in your own checkout. Leave that checkout
- > on the base branch with no uncommitted changes to tracked files while it runs. Otherwise the
- > merge is refused and nothing is touched โ agtx never switches your branch or stashes your
- > work. Nothing is pushed.
- > [!NOTE]
- > A full run takes hours and many turns of the session running the board. Start with a small
- > goal to see how it works on your project.
- ## Configuration
- Config file location: `~/.config/agtx/config.toml`
- ### General
- ```toml
- default_agent = "claude"
- fullscreen_on_enter = false # When true, Enter opens the tmux pane fullscreen inside agtx
- agent_hooks = true # Let agents report their own phase status via lifecycle hooks
- auto_trust = false # Answer agents' trust prompts on your behalf
- update_check = true # Check GitHub daily for a new release (see Updating)
- ```
- ### Worktree Base Branch
- agtx creates a new git worktree for each task. By default it auto-detects the base branch in this
- order: `main`, then `master`, then the current branch. You can override this to force a specific
- base branch (for example `dev` or `develop`).
- Global worktree defaults can be set here:
- ```toml
- # ~/.config/agtx/config.toml
- [worktree]
- base_branch = "dev"
- worktree_dir = ".worktrees" # default: ".agtx/worktrees"
- ```
- `worktree_dir` is the directory (relative to project root) where task worktrees are created. Defaults
- to `.agtx/worktrees` if not set.
- ### Project Configuration
- Per-project settings can be placed in `.agtx/config.toml` at the project root:
- ```toml
- # Base branch used when creating new task worktrees (optional)
- base_branch = "dev"
- # Directory where worktrees are created (optional, default: ".agtx/worktrees")
- worktree_dir = ".worktrees"
- # Files to copy from project root into each new worktree (comma-separated)
- # Paths are relative and preserve directory structure
- copy_files = ".env, .env.local, web/.env.local"
- # Shell command to run inside the worktree after creation and file copying
- init_script = "scripts/init_worktree.sh"
- # Shell command to run inside the worktree before removal
- cleanup_script = "scripts/cleanup_worktree.sh"
- ```
- `base_branch` controls which branch new task worktrees are created from. If omitted or empty, agtx
- auto-detects `main`, `master`, or falls back to the current branch.
- ### Per-Phase Agent Configuration
- By default, all phases use `default_agent`. You can override the agent for specific phases globally or per project:
- ```toml
- # ~/.config/agtx/config.toml
- default_agent = "claude"
- [agents]
- research = "gemini"
- planning = "claude"
- running = "claude"
- review = "codex"
- ```
- ```toml
- # .agtx/config.toml (project override โ takes precedence over global)
- [agents]
- running = "codex"
- ```
- ## Plugins
- Plug any spec-driven framework into the task lifecycle. Define commands, prompts, and artifacts โ agtx handles phase gating, artifact polling, worktree sync, agent switching, and autonomous execution.
- Press `P` to switch plugins. Ships with 10 built-in:
- | Plugin | Description |
- |--------|-------------|
- | **void** | Plain agent session - no prompting or skills, task description prefilled in input |
- | **agtx** (default) | Built-in workflow with skills and prompts for each phase |
- | **agtx-terse** | Token-efficient workflow - same workflow with compressed output and minimal tokens |
- | **gsd** | [Get Shit Done](https://github.com/fynnfluegge/get-shit-done-cc) - structured spec-driven development with interactive planning |
- | **spec-kit** | [Spec-Driven Development](https://github.com/github/spec-kit) by GitHub - specifications become executable artifacts |
- | **openspec** | [OpenSpec](https://github.com/Fission-AI/OpenSpec) - lightweight AI-guided specification framework |
- | **bmad** | [BMAD Method](https://github.com/bmad-code-org/BMAD-METHOD) - AI-driven agile development with structured phases |
- | **superpowers** | [Superpowers](https://github.com/obra/superpowers) - brainstorming, plans, TDD, subagent-driven development |
- | **oh-my-claudecode** | [oh-my-claudecode](https://github.com/Yeachan-Heo/oh-my-claudecode) - multi-agent orchestration with 37 skills and 22 specialized agents |
- | **agent-skills** | [Agent Skills](https://github.com/addyosmani/agent-skills) - production-grade engineering skills covering the full spec-to-ship lifecycle |
- ### Agent Compatibility
- Commands are written once in canonical format and automatically translated per agent:
- | Canonical (plugin.toml) | Claude / Gemini | Codex | OpenCode | Cursor | Grok | Antigravity | pi |
- |--------------------------|-----------------|-------|----------|--------|------|-------------|----|
- | `/agtx:plan` | `/agtx:plan` | `$agtx-plan` | `/agtx-plan` | `/agtx-plan` | `/agtx-plan` | `/agtx-plan` | `/skill:agtx-plan` |
- | | Claude | Codex | Gemini | OpenCode | Cursor | Copilot | Grok | Antigravity | pi |
- |--|:------:|:-----:|:------:|:--------:|:------:|:-------:|:----:|:-----------:|:--:|
- | **agtx** | โ | โ | โ | โ | โ | ๐ก | โ | โ | โ |
- | **gsd** | โ | โ | โ | โ | โ | โ | โ | โ | โ |
- | **spec-kit** | โ | โ | โ | โ | โ | ๐ก | โ | โ | ๐ก |
- | **openspec** | โ | โ | โ | โ | โ | ๐ก | โ | โ | ๐ก |
- | **bmad** | โ | โ | โ | โ | โ | ๐ก | โ | โ | ๐ก |
- | **superpowers** | โ | โ | โ | โ | โ | โ | โ | โ | โ |
- | **oh-my-claudecode** | โ | โ | โ | โ | โ | โ | โ | โ | โ |
- | **agent-skills** | โ | ๐ก | ๐ก | ๐ก | ๐ก | ๐ก | ๐ก | ๐ก | ๐ก |
- | **void** | โ | โ | โ | โ | โ | โ | โ | โ | โ |
- โ Skills, commands, and prompts fully supported ยท ๐ก Prompt only, no interactive skill support ยท โ Not supported
- <details>
- <summary><b>Creating a Plugin</b></summary>
- Place your plugin at `.agtx/plugins/<name>/plugin.toml` in your project root (or `~/.config/agtx/plugins/<name>/plugin.toml` for global use). It will appear in the plugin selector automatically.
- **Minimal example** โ a plugin that uses custom slash commands:
- ```toml
- name = "my-plugin"
- description = "My custom workflow"
- [commands]
- research = "/my-plugin:research {task}"
- planning = "/my-plugin:plan"
- running = "/my-plugin:execute"
- review = "/my-plugin:review"
- [prompts]
- planning = "Task: {task}"
- ```
- **Full reference** with all available fields:
- ```toml
- name = "my-plugin"
- description = "My custom workflow"
- # Shell command to run in the worktree after creation, before the agent starts.
- # {agent} is replaced with the agent name (claude, codex, gemini, etc.)
- init_script = "npm install --prefix .my-plugin --{agent}"
- # Restrict to specific agents (empty or omitted = all agents supported)
- supported_agents = ["claude", "codex", "gemini", "opencode"]
- # Extra directories to copy from project root into each worktree.
- # Agent config dirs (.claude, .gemini, .codex, .github/agents, .config/opencode)
- # are always copied automatically.
- copy_dirs = [".my-plugin"]
- # Individual files to copy from project root into each worktree.
- # Merged with project-level copy_files from .agtx/config.toml.
- copy_files = ["PROJECT.md", "REQUIREMENTS.md"]
- # When true, enables Review โ Planning transition via the `p` key.
- # Each cycle increments the phase counter ({phase} placeholder).
- # Use this for multi-milestone workflows (e.g. plan โ execute โ review โ next milestone).
- cyclic = false
- # Artifact files that signal phase completion.
- # When detected, the task shows a checkmark instead of the spinner.
- # Supports * wildcard for one directory level (e.g. "specs/*/plan.md").
- # Use {phase} for cycle-aware paths (replaced with the current cycle number).
- # Omitted phases show no completion indicator.
- [artifacts]
- research = ".my-plugin/research.md"
- planning = ".my-plugin/{phase}/plan.md"
- running = ".my-plugin/{phase}/summary.md"
- review = ".my-plugin/{phase}/review.md"
- # Slash commands sent to the agent via tmux for each phase.
- # Written in canonical format (Claude/Gemini style): /namespace:command
- # Automatically transformed per agent:
- # Claude/Gemini: /my-plugin:plan (unchanged)
- # OpenCode: /my-plugin-plan (colon -> hyphen)
- # Codex: $my-plugin-plan (slash -> dollar, colon -> hyphen)
- # Cursor/Grok/Antigravity: /my-plugin-plan (colon -> hyphen)
- # Omitted phases fall back to agent-native agtx skill invocation
- # (e.g. /agtx:plan for Claude, $agtx-plan for Codex).
- # Set to "" to skip sending a command for that phase.
- # Use {phase} for cycle-aware commands (replaced with the current cycle number).
- # Use {task} to inline the task description.
- [commands]
- preresearch = "/my-plugin:research {task}" # Used only when no research artifacts exist yet
- research = "/my-plugin:discuss {phase}"
- planning = "/my-plugin:plan {phase}"
- running = "/my-plugin:execute {phase}"
- review = "/my-plugin:review {phase}"
- # Prompt templates sent as task content after the command.
- # {task} = task title + description, {task_id} = unique task ID, {phase} = cycle number.
- # Omitted phases send no prompt (the skill/command handles instructions).
- [prompts]
- research = "Task: {task}"
- # Text patterns to wait for in the tmux pane before sending the prompt.
- # Useful when a command triggers an interactive prompt that must appear first.
- # Polls every 500ms, times out after 5 minutes.
- [prompt_triggers]
- research = "What do you want to build?"
- # Files/dirs to copy from worktree back to project root after a phase completes.
- # Triggered automatically when the phase artifact is detected (spinner โ checkmark).
- # Useful for sharing research artifacts (specs, plans) across worktrees.
- [copy_back]
- research = ["PROJECT.md", "REQUIREMENTS.md", ".my-plugin"]
- # Auto-dismiss interactive prompts that appear before the prompt trigger.
- # Each rule fires when ALL detect patterns are present and the pane is stable.
- # Response is newline-separated keystrokes (e.g. "2\nEnter" sends "2" then Enter).
- [[auto_dismiss]]
- detect = ["Map codebase", "Skip mapping", "Enter to select"]
- response = "2\nEnter"
- ```
- **What happens at each phase transition:**
- 1. The **command** is sent to the agent via tmux (e.g., `/my-plugin:plan`)
- 2. If a **prompt_trigger** is set, agtx waits for that prompt trigger to appear in the tmux pane
- 3. The **prompt** is sent with `{task}`, `{task_id}`, and `{phase}` replaced
- 4. agtx polls for the **artifact** file โ when found, the spinner becomes a checkmark
- 5. If **copy_back** is configured, artifacts are copied from worktree to project root on completion
- 6. If the agent appears idle (no output for 15s), the spinner becomes a pause icon
- **Phase gating:** Whether a phase can be entered directly from Backlog is derived from the plugin config. If a phase's command or prompt contains `{task}`, it can receive task context and is accessible from Backlog. If neither has `{task}`, the phase depends on a prior phase and is blocked until that artifact exists. For example, OpenSpec's `/opsx:propose {task}` allows direct Backlog โ Planning, but `/opsx:apply` (no `{task}`) blocks Backlog โ Running until proposal artifacts exist.
- **Preresearch fallback:** When pressing `R` on a task, if `preresearch` is configured and no research artifacts from `copy_back` exist in the project root yet, the `preresearch` command is used instead of `research`. This lets plugins run a one-time project setup (e.g. `/gsd:new-project`) before switching to the regular research command for subsequent tasks. If the plugin has no research command at all (e.g. OpenSpec), pressing `R` shows a warning.
- **Cyclic workflows:** When `cyclic = true`, pressing `p` in Review moves the task back to Planning with an incremented phase counter. This enables multi-milestone workflows where each cycle (plan โ execute โ review) produces artifacts in a separate `{phase}` directory.
- **Custom skills:** If your plugin provides its own skill files, place them in the plugin directory:
- ```
- .agtx/plugins/my-plugin/
- โโโ plugin.toml
- โโโ skills/
- โโโ agtx-plan/SKILL.md
- โโโ agtx-execute/SKILL.md
- โโโ agtx-review/SKILL.md
- ```
- These override the built-in agtx skills and are automatically deployed to each agent's native discovery path (`.claude/commands/`, `.codex/skills/`, `.gemini/commands/`, etc.) in every worktree.
- </details>
- ## How It Works
- ### Architecture
- ```
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- โ agtx TUI โ
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
- โ Backlog โ Planning โ Running โ Review โ Done โ
- โ โโโโโโโ โ โโโโโโโ โ โโโโโโโ โ โโโโโโโ โ โ
- โ โTask1โ โ โTask2โ โ โTask3โ โ โTask4โ โ โ
- โ โโโโโโโ โ โโโโโโโ โ โโโโโโโ โ โโโโโโโ โ โ
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- โ โ
- โผ โผ
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- โ tmux server "agtx" โ
- โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
- โ โ Session: "my-project" โ โ
- โ โ โโโโโโโโโโ โโโโโโโโโโ โโโโโโโโโโ โ โ
- โ โ โWindow: โ โWindow: โ โWindow: โ โ โ
- โ โ โtask2 โ โtask3 โ โtask4 โ โ โ
- โ โ โ(Claude)โ โ(Claude)โ โ(Claude)โ โ โ
- โ โ โโโโโโโโโโ โโโโโโโโโโ โโโโโโโโโโ โ โ
- โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
- โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
- โ โ Session: "other-project" โ โ
- โ โ โโโโโโโโโโโโโโโโโโโโโ โ โ
- โ โ โ Window: โ โ โ
- โ โ โ some_other_task โ โ โ
- โ โ โโโโโโโโโโโโโโโโโโโโโ โ โ
- โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- โ โ
- โผ โผ
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- โ Git Worktrees โ
- โ .agtx/worktrees/task2/ โ
- โ .agtx/worktrees/task3/ โ
- โ .agtx/worktrees/task4/ โ
- โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
- ```
- ### Tmux Structure
- - **Server**: All sessions run on a dedicated tmux server named `agtx`
- - **Sessions**: Each project gets its own tmux session (named after the project)
- - **Windows**: Each task gets its own window within the project's session
- ```bash
- # List all sessions
- tmux -L agtx list-sessions
- # List all windows across sessions
- tmux -L agtx list-windows -a
- # Attach to the agtx server
- tmux -L agtx attach
- ```
- ### Data Storage
- - **Database**: `~/Library/Application Support/agtx/` (macOS) or `~/.config/agtx/` (Linux)
- - Config: `~/.config/agtx/config.toml`
- - **Worktrees**: `.agtx/worktrees/` in each project
- - **Tmux**: Dedicated server `agtx` with per-project sessions
- ## Container Sandbox
- Run agtx in an isolated container (Docker or Podman, including rootless Podman) so agents can only touch the project you pass in โ no access to the rest of your home directory, credentials are read-only, and any files the agent creates in the project are owned by your host user.
- ```bash
- # Run agtx on a project
- ./docker/sandbox.sh path/to/your-project
- # Or from inside the project directory
- ./docker/sandbox.sh
- ```
- The sandbox:
- - Mounts only the target project as writable; everything else on the host is inaccessible
- - Copies `~/.claude` credentials read-only at startup so they are never written back to the host
- - Runs as a non-root user whose UID/GID matches your host user (files created inside the container appear correctly owned on the host)
- - Stores agtx state in named container volumes (persists across runs, isolated from your host's agtx data)
- - Pre-accepts the bypass permissions prompt, which is appropriate in an isolated container
- > [!NOTE]
- > Requires [Docker Engine](https://docs.docker.com/engine/install/) (Linux), [Docker Desktop](https://docs.docker.com/desktop/) (macOS/Windows), or [Podman](https://podman.io/docs/installation) (rootful or rootless). The image is built automatically on first run and cached for subsequent runs. Set `AGTX_CONTAINER_RUNTIME=podman` (or `=docker`) to pick a runtime explicitly when both are installed.
- ## MCP Server
- The agtx MCP server (`agtx mcp-serve`) exposes the board to any coding agent session via the [Model Context Protocol](https://modelcontextprotocol.io). Used by the orchestrator agent, the oneshot skill, and the brainstorm & sweep skills.
- ### Modes
- | Mode | Command | Used by |
- |------|---------|---------|
- | **Global** | `agtx mcp-serve` | Brainstorm, sweep and oneshot skills โ works across all projects |
- | **Project-scoped** | `agtx mcp-serve <path>` | Orchestrator โ bound to one project at startup |
- In global mode all tools require a `project_id` parameter. Call `list_projects` first to resolve it.
- ### Tools
- | Tool | Description |
- |------|-------------|
- | `list_projects` | List all projects indexed in agtx |
- | `get_config` | The project's effective config, global and project merged โ e.g. whether `auto_trust` is on |
- | `list_tasks` | List tasks with their phase status, optionally filtered by status (descriptions on request) |
- | `get_task` | Get task details + `allowed_actions` for valid transitions |
- | `wait_for_board_change` | Block until a task needs attention, then return only what changed and the outcome of queued transitions |
- | `create_task` | Create a single backlog task |
- | `create_tasks_batch` | Batch-create tasks with index-based dependencies |
- | `update_task` | Modify a backlog task (title, description, deps) |
- | `delete_task` | Delete a backlog task |
- | `move_task` | Queue a phase transition, including `move_to_done_and_merge` |
- | `get_transition_status` | Check if a queued transition completed or errored |
- | `check_conflicts` | Non-destructive merge conflict check against default branch |
- | `get_notifications` | Fetch pending orchestrator notifications |
- | `read_pane_content` | Read the last N lines of a task's tmux pane |
- | `send_to_task` | Send a message to a task's agent (Planning, Running or Review) |
- ## Orchestrator Agent (Experimental)
- > Press `O` and walk away. Come back to changes ready to merge.
- The orchestrator is an AI agent that **drives other AI agents to completion**. You triage tasks into Planning or Running โ the orchestrator takes over from there, advancing each task through its phases until it lands in Review, ready for you to merge.
- ```bash
- agtx --experimental # then press O
- ```
- **What it does:**
- - Monitors tasks in Planning and Running
- - Advances tasks automatically as phases complete (Planning โ Running โ Review)
- - Respects plugin phase rules โ checks `allowed_actions` before each transition
- - Detects stuck tasks (idle for 1+ minute without a phase artifact) and reads the agent pane to diagnose the cause
- - Nudges stuck agents, answers CLI prompts automatically, or escalates to you with a reason when human input is needed
- **You triage. It executes.** Move tasks from Backlog into Planning or Running โ the orchestrator handles the rest. Merging is your call. For a session that owns the whole board, from planning the work to merging it, see the [Oneshot Skill](#oneshot-skill).
- ### MCP Integration
- The orchestrator communicates with agtx through the [Model Context Protocol (MCP)](https://modelcontextprotocol.io). agtx ships with a built-in MCP server (`agtx serve`) that exposes the kanban board as a set of tools over JSON-RPC via stdio.
- ```
- โโโโโโโโโโโโโโ-โ MCP (stdio) โโโโโโโโโโโโโโโโ SQLite โโโโโโโ
- โ Orchestrator โ โโโโโโโโโโโโโโโโโโโ โ MCP Server โ โโโโโโโโโโโโโโ โ DB โ
- โ (Claude Code)โ โ (agtx serve) โ โโโโฌโโโ
- โโโโโโโโฌโโโโโโโโ โโโโโโโโโโโโโโโโ โ
- โ push-when-idle notifications โ
- โโโโโโโโดโโโโโโโโ โ
- โ TUI (agtx) โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ--โโ
- โโโโโโโโโโโโโโโโ
- ```
- **How it works:**
- 1. When you press `O`, the TUI registers the MCP server with the orchestrator agent via `claude mcp add-json --scope local`
- 2. The orchestrator receives phase completion notifications pushed to its tmux pane when idle
- 3. It reacts by calling `get_task` to check `allowed_actions`, then `move_task` to advance the task
- 4. The TUI processes the transition request, executes all side effects (agent switching, skill deployment, prompt sending), and updates the database
- 5. If a task has been idle for 1+ minute without a phase artifact, the orchestrator is notified โ it reads the pane with `read_pane_content`, then either nudges the agent with `send_to_task` or calls `move_task` with `escalate_to_user` to flag it for your attention
- 6. Escalated tasks show a `โ ` badge on the kanban board; opening the task popup shows the reason and dismisses the flag
- 7. MCP registration is cleaned up when the orchestrator is stopped
- ## Benchmark
- agtx includes a [SWE-bench Lite](https://www.swebench.com) benchmark runner that uses agtx itself as the agent orchestration layer โ driving coding agent workflows against 300 real GitHub bug-fix tasks via the MCP server.
- See **[benchmark/README.md](benchmark/README.md)** for setup, usage, bundled configs, and evaluation instructions.
- ## Contributing
- Contributions are welcome! Whether it's a bug fix, new plugin, agent integration, or documentation improvement.
- See [CONTRIBUTING.md](CONTRIBUTING.md) for the full guide. Here's the short version:
- ```bash
- # Fork & clone
- git clone https://github.com/<you>/agtx && cd agtx
- # Build & test
- cargo build && cargo test --features test-mocks
- ```
- ### Good First Contributions
- Not sure where to start? Here are some ideas:
- - **Write a plugin** โ A single `plugin.toml` is all you need. See [Creating a Plugin](#plugins) for the full reference
- - **Add a new agent** โ Integrate your favorite AI coding CLI. See the [architecture docs](CLAUDE.md) for how agents are structured
- - **Improve documentation** โ Found something unclear? Help others by improving it
- - **Report bugs** โ Open an [issue](https://github.com/fynnfluegge/agtx/issues). Reproduction steps are always appreciated
- - **Browse open issues** โ Check the [`good first issue`](https://github.com/fynnfluegge/agtx/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) label for beginner-friendly tasks
- ## Development
- See [CLAUDE.md](CLAUDE.md) for full architecture docs and development patterns.
- ```bash
- # Build
- cargo build
- # Run tests (includes mock-based tests)
- cargo test --features test-mocks
- # Build release
- cargo build --release
- ```
benchmark/swebench/benchmark.py ยท Bundled script ยท 2,152 lines ยท 1 finding marked
- #!/usr/bin/env python3
- """
- SWE-bench Lite benchmark runner for agtx.
- Drives agtx via its MCP server (JSON-RPC 2.0 over stdio) to run coding agent
- workflows against SWE-bench Lite (300 tasks), collects git diff patches when
- tasks reach Review, uses tokscale for token/cost metrics (all agents), and
- writes SWE-bench-compatible predictions.jsonl + results.json.
- A config.toml is required. It is written to .agtx/config.toml in each repo
- before the TUI starts โ this is how agent, plugin, worktree_dir, etc. are
- configured. The script reads plugin and default_agent back from the config to
- drive artifact polling.
- Token/cost metrics use tokscale (https://github.com/junhoyeo/tokscale) if
- available. Supports claude, codex, gemini, and 20+ other agents. Install with:
- npm install -g tokscale
- If tokscale is not installed, cost fields are null.
- Usage:
- python benchmark.py --config my_config.toml --instances 1
- python benchmark.py --config my_config.toml --concurrency 2 --instances 10
- python benchmark.py --config my_config.toml --instance-ids sympy__sympy-20590
- python benchmark.py --config my_config.toml --instance-ids astropy__astropy-12907 --sandbox --verbose --agtx ../../target/release/agtx
- Example config.toml:
- default_agent = "claude"
- workflow_plugin = "agtx"
- worktree_dir = ".agtx/worktrees"
- """
- from __future__ import annotations
- import argparse
- import json
- import os
- import re
- import shlex
- import subprocess
- import sys
- import tempfile
- import threading
- import time
- from concurrent.futures import ThreadPoolExecutor, as_completed
- from datetime import datetime
- from pathlib import Path
- from typing import Any
- try:
- from datasets import load_dataset
- from tqdm import tqdm
- except ImportError:
- print("Missing dependencies. Run: pip install -r requirements.txt", file=sys.stderr)
- sys.exit(1)
- try:
- import tomllib # Python 3.11+
- except ImportError:
- try:
- import tomli as tomllib # fallback
- except ImportError:
- # Minimal TOML parser for the subset we need (key = "value" lines only)
- tomllib = None # type: ignore
- # ---------------------------------------------------------------------------
- # Config loading
- # ---------------------------------------------------------------------------
- def load_config_toml(path: Path) -> dict:
- """Load a TOML config file, returning a dict of its keys."""
- content = path.read_text()
- if tomllib is not None:
- return tomllib.loads(content)
- # Fallback: parse simple key = "value" lines and [section] tables
- result: dict = {}
- current_section: dict | None = None
- current_section_name: str | None = None
- for line in content.splitlines():
- line = line.strip()
- if not line or line.startswith("#"):
- continue
- if line.startswith("[") and line.endswith("]"):
- current_section_name = line[1:-1].strip()
- current_section = {}
- result[current_section_name] = current_section
- continue
- if "=" in line:
- k, _, v = line.partition("=")
- k = k.strip()
- v = v.strip().strip('"').strip("'")
- if current_section is not None:
- current_section[k] = v
- else:
- result[k] = v
- return result
- def running_agent(config: dict) -> str:
- """Return the agent that will handle the running phase."""
- return (
- config.get("agents", {}).get("running")
- or config.get("default_agent")
- or "claude"
- )
- # ---------------------------------------------------------------------------
- # MCP Client
- # ---------------------------------------------------------------------------
- class McpError(Exception):
- pass
- class McpClient:
- """Raw JSON-RPC 2.0 MCP client over subprocess stdin/stdout."""
- def __init__(self, agtx_bin: str, repo_path: str, container_id: str | None = None):
- self._seq = 0
- self._lock = threading.Lock()
- if container_id:
- cmd = [
- "docker", "exec", "-i", container_id,
- "/bin/bash", "-c",
- "export HOME=/home/bench"
- " && source /opt/miniconda3/bin/activate && conda activate testbed"
- " && agtx mcp-serve /testbed",
- ]
- else:
- cmd = [agtx_bin, "mcp-serve", repo_path]
- self._proc = subprocess.Popen(
- cmd,
- stdin=subprocess.PIPE,
- stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL,
- text=True,
- bufsize=1,
- )
- self._handshake()
- def _next_id(self) -> int:
- with self._lock:
- self._seq += 1
- return self._seq
- def _send(self, msg: dict) -> None:
- line = json.dumps(msg) + "\n"
- self._proc.stdin.write(line)
- self._proc.stdin.flush()
- # A reply should take milliseconds. This is not a latency budget, it is a
- # deadlock guard: `readline()` on a live-but-silent server blocks forever, and
- # no --phase-timeout can rescue it because the block is inside the MCP call
- # rather than inside the artifact poll that the timeout governs.
- RECV_TIMEOUT_SECONDS = 120
- def _recv(self) -> dict:
- import selectors
- with selectors.DefaultSelector() as sel:
- sel.register(self._proc.stdout, selectors.EVENT_READ)
- if not sel.select(timeout=self.RECV_TIMEOUT_SECONDS):
- raise McpError(
- f"MCP server did not respond within {self.RECV_TIMEOUT_SECONDS}s "
- "(agent may be parked on an interactive prompt or a usage limit)"
- )
- line = self._proc.stdout.readline()
- if not line:
- raise McpError("MCP server closed connection")
- return json.loads(line)
- def _request(self, method: str, params: dict) -> Any:
- req_id = self._next_id()
- self._send({"jsonrpc": "2.0", "id": req_id, "method": method, "params": params})
- while True:
- msg = self._recv()
- if msg.get("id") == req_id:
- if "error" in msg:
- raise McpError(f"MCP error: {msg['error']}")
- result = msg.get("result", {})
- content = result.get("content", [])
- if content and content[0].get("type") == "text":
- return json.loads(content[0]["text"])
- if result.get("isError"):
- raise McpError(f"Tool returned error: {content}")
- return result
- # ignore notifications
- def _notify(self, method: str, params: dict) -> None:
- self._send({"jsonrpc": "2.0", "method": method, "params": params})
- def _handshake(self) -> None:
- req_id = self._next_id()
- self._send({
- "jsonrpc": "2.0",
- "id": req_id,
- "method": "initialize",
- "params": {
- "protocolVersion": "2024-11-05",
- "capabilities": {},
- "clientInfo": {"name": "swebench-runner", "version": "1.0"},
- },
- })
- while True:
- msg = self._recv()
- if msg.get("id") == req_id:
- break
- self._notify("notifications/initialized", {})
- def call(self, tool: str, **kwargs) -> Any:
- params = {k: v for k, v in kwargs.items() if v is not None}
- return self._request("tools/call", {"name": tool, "arguments": params})
- def close(self) -> None:
- try:
- self._proc.stdin.close()
- self._proc.wait(timeout=5)
- except Exception:
- self._proc.kill()
- # ---------------------------------------------------------------------------
- # Repo Setup
- # ---------------------------------------------------------------------------
- def setup_repo(instance: dict, workdir: str, config_path: Path, verbose: bool = False, smoke_test: bool = False) -> Path:
- """
- Clone the repo at base_commit and write .agtx/config.toml.
- Returns repo path. Safe to call again on an existing clone (resumable).
- In smoke test mode, initializes an empty git repo instead of cloning.
- """
- instance_id = instance["instance_id"]
- if smoke_test:
- repo_path = Path(workdir) / instance_id
- if not repo_path.exists():
- repo_path.mkdir(parents=True)
- subprocess.run(["git", "init"], cwd=repo_path, capture_output=True, check=True)
- subprocess.run(
- ["git", "commit", "--allow-empty", "-m", "init"],
- cwd=repo_path,
- capture_output=True,
- check=True,
- env={**__import__("os").environ, "GIT_AUTHOR_NAME": "smoke", "GIT_AUTHOR_EMAIL": "smoke@test", "GIT_COMMITTER_NAME": "smoke", "GIT_COMMITTER_EMAIL": "smoke@test"},
- )
- if verbose:
- print(f" [setup] Initialized empty git repo at {repo_path}", file=sys.stderr)
- _write_agtx_config(repo_path, config_path, "HEAD")
- return repo_path
- repo_url = f"https://github.com/{instance['repo']}.git"
- base_commit = instance["base_commit"]
- repo_path = Path(workdir) / instance_id
- if repo_path.exists():
- # Check if the repo is clean: HEAD must be at base_commit and no task/* branches.
- head_result = subprocess.run(
- ["git", "rev-parse", "HEAD"],
- cwd=repo_path,
- capture_output=True,
- text=True,
- )
- branch_result = subprocess.run(
- ["git", "branch", "--list", "task/*"],
- cwd=repo_path,
- capture_output=True,
- text=True,
- )
- head_ok = head_result.returncode == 0 and head_result.stdout.strip() == base_commit
- no_task_branches = branch_result.returncode != 0 or not branch_result.stdout.strip()
- if not head_ok:
- # Wrong commit โ need a clean clone
- if verbose:
- print(
- f" [setup] Stale clone (wrong commit), removing {repo_path}",
- file=sys.stderr,
- )
- subprocess.run(["rm", "-rf", str(repo_path)], check=True)
- else:
- if not no_task_branches:
- # Just delete leftover task branches, no need to re-clone
- for branch in branch_result.stdout.strip().splitlines():
- branch = branch.strip()
- if branch:
- subprocess.run(
- ["git", "branch", "-D", branch],
- cwd=repo_path,
- capture_output=True,
- )
- if verbose:
- print(f" [setup] Cleaned stale task branches, reusing {repo_path}", file=sys.stderr)
- else:
- if verbose:
- print(f" [setup] Using cached repo at {repo_path}", file=sys.stderr)
- _write_agtx_config(repo_path, config_path, base_commit)
- return repo_path
- if verbose:
- print(f" [setup] Cloning {repo_url} โ {repo_path}", file=sys.stderr)
- repo_path.mkdir(parents=True, exist_ok=True)
- subprocess.run(
- ["git", "clone", "--quiet", "--depth=1", repo_url, str(repo_path)],
- check=True,
- capture_output=True,
- timeout=300,
- )
- if verbose:
- print(f" [setup] Checking out {base_commit}", file=sys.stderr)
- fetch_result = subprocess.run(
- ["git", "fetch", "--depth=1", "origin", base_commit],
- cwd=repo_path,
- capture_output=True,
- text=True,
- timeout=120,
- )
- if fetch_result.returncode != 0:
- subprocess.run(
- ["git", "fetch", "--unshallow"],
- cwd=repo_path,
- capture_output=True,
- timeout=600,
- )
- subprocess.run(
- ["git", "checkout", base_commit],
- cwd=repo_path,
- check=True,
- capture_output=True,
- timeout=60,
- )
- _write_agtx_config(repo_path, config_path, base_commit)
- return repo_path
- def _write_agtx_config(repo_path: Path, config_path: Path, base_commit: str) -> None:
- """
- Write config.toml and wipe any stale agtx state (old worktrees, task DB, old artifacts).
- This ensures each benchmark run starts clean even if the repo clone is reused.
- """
- agtx_dir = repo_path / ".agtx"
- # Remove stale worktrees and the project DB so the TUI starts fresh
- for stale in ["worktrees", "db"]:
- stale_path = agtx_dir / stale
- if stale_path.exists():
- subprocess.run(["rm", "-rf", str(stale_path)], check=True)
- agtx_dir.mkdir(exist_ok=True)
- dest = agtx_dir / "config.toml"
- # Insert base_branch = "<commit>" before any [section] headers so it stays
- # a top-level key. Appending at the end would put it inside the last section
- # (e.g. [agents]) and TOML would parse it as agents.base_branch instead.
- content = config_path.read_text()
- base_branch_line = f'base_branch = "{base_commit}"\n'
- # Remove any existing base_branch line first (idempotent)
- lines = [l for l in content.splitlines(keepends=True) if not l.startswith("base_branch =")]
- # Find the first [section] line and insert before it; if none, append at end
- insert_at = next((i for i, l in enumerate(lines) if l.startswith("[")), len(lines))
- lines.insert(insert_at, base_branch_line)
- dest.write_text("".join(lines))
- # Prune stale git worktree registrations and delete old task branches.
- # git worktree prune won't remove registrations whose gitdir is gone until the
- # grace period expires (default 3 months). Delete .git/worktrees/ directly so
- # the task/* branches become deletable immediately.
- git_worktrees_meta = repo_path / ".git" / "worktrees"
- if git_worktrees_meta.exists():
- subprocess.run(["rm", "-rf", str(git_worktrees_meta)], check=True)
- subprocess.run(["git", "worktree", "prune"], cwd=repo_path, capture_output=True)
- result = subprocess.run(
- ["git", "branch", "--list", "task/*"],
- cwd=repo_path, capture_output=True, text=True,
- )
- for branch in result.stdout.splitlines():
- branch = branch.strip().lstrip("*").strip()
- if branch:
- subprocess.run(
- ["git", "branch", "-D", branch],
- cwd=repo_path, capture_output=True,
- )
- # Ensure .agtx/ is ignored locally without touching .gitignore (which is a tracked file).
- # .git/info/exclude is a local-only ignore file โ changes never appear in git diff.
- exclude_file = repo_path / ".git" / "info" / "exclude"
- marker = ".agtx/"
- existing = exclude_file.read_text() if exclude_file.exists() else ""
- if marker not in existing:
- with exclude_file.open("a") as f:
- f.write(f"\n# agtx benchmark artifacts\n{marker}\n")
- def start_tui_in_tmux(slug: str, repo_path: Path, agtx_bin: str, verbose: bool = False) -> None:
- """Start an agtx TUI instance in a detached tmux session on the swebench server."""
- # Kill any stale session with the same slug on the swebench server
- subprocess.run(
- ["tmux", "-L", "swebench", "kill-session", "-t", slug],
- capture_output=True,
- )
- # Also kill the agtx-server session for this repo (named after the repo dir)
- # so the new TUI doesn't inherit stale task windows from a previous run
- repo_session = repo_path.name # e.g. "astropy__astropy-12907"
- subprocess.run(
- ["tmux", "-L", "agtx", "kill-session", "-t", repo_session],
- capture_output=True,
- )
- if verbose:
- print(f" [tmux] Starting TUI session '{slug}' for {repo_path}", file=sys.stderr)
- result = subprocess.run(
- [
- "tmux", "-L", "swebench",
- "new-session", "-d", "-s", slug,
- f"{agtx_bin} {repo_path}",
- ],
- capture_output=True,
- text=True,
- )
- if result.returncode != 0:
- raise RuntimeError(f"tmux new-session failed: {result.stderr.strip()}")
- def kill_tmux_session(slug: str, repo_path: Path | None = None) -> None:
- subprocess.run(
- ["tmux", "-L", "swebench", "kill-session", "-t", slug],
- capture_output=True,
- )
- if repo_path is not None:
- subprocess.run(
- ["tmux", "-L", "agtx", "kill-session", "-t", repo_path.name],
- capture_output=True,
- )
- # ---------------------------------------------------------------------------
- # Docker helpers
- # ---------------------------------------------------------------------------
- def docker_image_name(instance_id: str) -> str:
- """Convert an instance_id to its SWE-bench Docker image name.
- e.g. astropy__astropy-12907 โ swebench/sweb.eval.x86_64.astropy_1776_astropy-12907:latest
- """
- image_slug = instance_id.replace("__", "_1776_")
- return f"swebench/sweb.eval.x86_64.{image_slug}:latest"
- TOOLS_IMAGE = "agtx/swebench-tools:latest"
- TOOLS_CONTAINER = "agtx-swebench-tools"
- TOOLS_VOLUME = "agtx-swebench-tools"
- def _tools_image_exists() -> bool:
- """Return True if the shared tools image has been built by prebake_images.py."""
- result = subprocess.run(
- ["docker", "image", "inspect", TOOLS_IMAGE],
- capture_output=True,
- )
- return result.returncode == 0
- def _tools_volume_exists() -> bool:
- result = subprocess.run(
- ["docker", "volume", "inspect", TOOLS_VOLUME],
- capture_output=True,
- )
- return result.returncode == 0
- def ensure_tools_volume(verbose: bool = False) -> None:
- """Create and populate the tools volume from the tools image if it doesn't exist.
- The volume contains tmux + libutempter (so SWE-bench containers don't need apt),
- node, node_modules, and rtk. Mounting read-only at /tools avoids copying the large
- Node stack into instance containers' writable layers.
- """
- if _tools_volume_exists():
- if verbose:
- print(f" [docker] Tools volume '{TOOLS_VOLUME}' already exists.", file=sys.stderr)
- return
- if verbose:
- print(f" [docker] Creating tools volume '{TOOLS_VOLUME}'...", file=sys.stderr)
- subprocess.run(["docker", "volume", "create", TOOLS_VOLUME], check=True, capture_output=True)
- # Populate volume by running a one-shot container that copies from the image into the volume.
- # libutempter is copied alongside tmux because SWE-bench containers don't have it.
- subprocess.run(
- [
- "docker", "run", "--rm", "--platform", "linux/amd64",
- "-v", f"{TOOLS_VOLUME}:/tools",
- TOOLS_IMAGE,
- "/bin/bash", "-c",
- "cp -a /usr/bin/tmux /usr/bin/node /usr/lib/node_modules /tools/"
- " && mkdir -p /tools/lib/x86_64-linux-gnu"
- " && cp /lib/x86_64-linux-gnu/libutempter.so.* /tools/lib/x86_64-linux-gnu/"
- " && cp /lib/x86_64-linux-gnu/libevent_core-2.1.so.* /tools/lib/x86_64-linux-gnu/",
- ],
- check=True, capture_output=not verbose,
- )
- if verbose:
- print(f" [docker] Tools volume ready.", file=sys.stderr)
- def start_tools_container(verbose: bool = False) -> str:
- """Start the shared tools container and return its container ID.
- The tools container is a long-running instance of agtx/swebench-tools:latest
- used as a source to copy tmux/Node/Claude Code binaries into benchmark containers.
- Starting it once and sharing it across all concurrent runs avoids redundant installs.
- """
- # Kill any stale tools container
- subprocess.run(["docker", "rm", "-f", TOOLS_CONTAINER], capture_output=True)
- result = subprocess.run(
- ["docker", "run", "-d", "--platform", "linux/amd64",
- "--name", TOOLS_CONTAINER, TOOLS_IMAGE, "sleep", "infinity"],
- check=True, capture_output=True, text=True,
- )
- container_id = result.stdout.strip()
- if verbose:
- print(f" [docker] Started tools container {container_id[:12]}", file=sys.stderr)
- return container_id
- def stop_tools_container(verbose: bool = False) -> None:
- subprocess.run(["docker", "rm", "-f", TOOLS_CONTAINER], capture_output=True)
- def _agtx_db_dir() -> str:
- """Return the host path to the agtx database directory."""
- if sys.platform == "darwin":
- return str(Path.home() / "Library" / "Application Support" / "agtx")
- return str(Path.home() / ".config" / "agtx")
- def _docker_env_overrides() -> list[str]:
- """Build -e flags for docker run to fix localhost URLs in Claude env settings.
- Reads ~/.claude/settings.json env overrides (read-only, never modified) and
- replaces http://localhost: with http://host.docker.internal: so proxies running
- on the macOS host are reachable from inside the container.
- Returns a flat list of ["-e", "KEY=VALUE", ...] suitable for subprocess args.
- """
- settings_path = Path.home() / ".claude" / "settings.json"
- if not settings_path.exists():
- return []
- try:
- data = json.loads(settings_path.read_text())
- env_overrides = data.get("env", {})
- except Exception:
- return []
- result = []
- for key, val in env_overrides.items():
- if isinstance(val, str) and "localhost" in val:
- val = val.replace("http://localhost:", "http://host.docker.internal:")
- val = val.replace("https://localhost:", "https://host.docker.internal:")
- result += ["-e", f"{key}={val}"]
- return result
- def start_docker_container(
- instance_id: str, slug: str, agtx_bin: str, verbose: bool = False
- ) -> str:
- """Pull the SWE-bench image and start a long-running container. Returns container_id."""
- container_name = f"swebench-{slug}"
- image = docker_image_name(instance_id)
- # Kill any stale container with the same name
- subprocess.run(["docker", "rm", "-f", container_name], capture_output=True)
- if verbose:
- print(f" [docker] Pulling {image}...", file=sys.stderr)
- subprocess.run(
- ["docker", "pull", "--platform", "linux/amd64", image],
- check=True,
- capture_output=not verbose,
- )
- if verbose:
- print(f" [docker] Starting container {container_name}...", file=sys.stderr)
- # Mount the tools volume if available โ provides tmux, node, node_modules, rtk
- # without consuming writable layer space in the instance container.
- extra_mounts = []
- if _tools_volume_exists():
- extra_mounts = ["-v", f"{TOOLS_VOLUME}:/tools:ro"]
- if verbose:
- print(f" [docker] Mounting tools volume.", file=sys.stderr)
- run_result = subprocess.run(
- [
- "docker", "run", "-d",
- "--platform", "linux/amd64",
- "--name", container_name,
- "-v", f"{agtx_bin}:/usr/local/bin/agtx",
- ] + extra_mounts + [
- image,
- "sleep", "infinity",
- ],
- capture_output=True,
- text=True,
- check=True,
- )
- container_id = run_result.stdout.strip()
- if verbose:
- print(f" [docker] Container id: {container_id[:12]}", file=sys.stderr)
- return container_id
- def seed_agent_trust(container_id: str, agent: str, *, verbose: bool = False) -> None:
- """Record /testbed as trusted for `agent`, the way the user would by hand.
- Trust records are **path-keyed**, and the repo lands at /testbed inside the
- container. Copying the host's ~/.claude.json (below) therefore does *not*
- prevent the trust dialog: no entry matches that path. Until now the run
- depended on agtx clearing the dialog by reading the pane, which is no longer
- the default.
- Deterministic and done up front, because here a stuck dialog costs a whole
- instance run rather than one task.
- """
- py = {
- "claude": (
- "import json,os;p='/home/bench/.claude.json';"
- "d=json.load(open(p)) if os.path.exists(p) else {};"
- "d.setdefault('projects',{}).setdefault('/testbed',{})"
- "['hasTrustDialogAccepted']=True;"
- "json.dump(d,open(p,'w'))"
- ),
- "codex": (
- "import os;p='/home/bench/.codex/config.toml';"
- "os.makedirs(os.path.dirname(p),exist_ok=True);"
- "s=open(p).read() if os.path.exists(p) else '';"
- "s+='\\n[projects.\"/testbed\"]\\ntrust_level = \"trusted\"\\n' "
- "if '[projects.\"/testbed\"]' not in s else '';"
- "open(p,'w').write(s)"
- ),
- "gemini": (
- "import json,os;p='/home/bench/.gemini/trustedFolders.json';"
- "os.makedirs(os.path.dirname(p),exist_ok=True);"
- "d=json.load(open(p)) if os.path.exists(p) else {};"
- "d['/testbed']='TRUST_FOLDER';json.dump(d,open(p,'w'))"
- ),
- "antigravity": (
- "import json,os;p='/home/bench/.gemini/antigravity-cli/settings.json';"
- "os.makedirs(os.path.dirname(p),exist_ok=True);"
- "d=json.load(open(p)) if os.path.exists(p) else {};"
- "w=d.setdefault('trustedWorkspaces',[]);"
- "w.append('/testbed') if '/testbed' not in w else None;"
- "json.dump(d,open(p,'w'))"
- ),
- }.get(agent)
- # cursor and grok pass --trust at launch; opencode has no trust gate; copilot
- # is unmeasured. Nothing to seed, and nothing to guess at.
- if py is None:
- return
- if verbose:
- print(f" [docker] Seeding {agent} trust for /testbed...", file=sys.stderr)
- subprocess.run(
- ["docker", "exec", container_id, "python3", "-c", py],
- check=False, capture_output=not verbose,
- )
- def setup_container(
- container_id: str, config_path: Path, base_commit: str, verbose: bool = False,
- tools_container_id: str | None = None,
- extra_dirs: list[tuple[Path, str]] | None = None,
- sandbox_init: list[str] | None = None,
- ) -> None:
- """Write agtx config into the container and install tmux + Node + Claude Code + Grok.
- If tools_container_id is provided, binaries are copied from the shared tools
- container instead of being installed from the network (~3-5 min saved).
- extra_dirs is an optional list of (host_path, container_dest) pairs for copying
- additional files into the container (e.g. third-party plugin skill directories).
- sandbox_init is an optional list of shell commands to run inside the container
- after setup (e.g. ["rtk init -g"] to activate rtk hooks).
- """
- # Parse default_agent from the benchmark config so we can write a global agtx config
- # inside the container. Without it agtx shows the agent-selection wizard on startup.
- bench_config = load_config_toml(config_path)
- default_agent = bench_config.get("default_agent", "claude")
- # Copy Claude credentials into the container as snapshots (not mounts).
- # This avoids writing back to host files and lets us patch localhost URLs.
- # We only copy the credential/config files โ NOT ~/.claude/projects/ which can be
- # gigabytes of conversation history and would exhaust the container's disk space.
- home = Path.home()
- subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c",
- "mkdir -p /home/bench/.config/claude /home/bench/.claude"],
- check=True, capture_output=not verbose,
- )
- # ~/.config/claude/ โ API key and auth tokens
- config_claude = home / ".config" / "claude"
- if config_claude.exists():
- subprocess.run(
- ["docker", "cp", f"{config_claude}/.", f"{container_id}:/home/bench/.config/claude"],
- check=True, capture_output=not verbose,
- )
- # ~/.claude/ โ copy only top-level files (settings.json, credentials), skip subdirs
- # (projects/, worktrees/, etc.) which contain large conversation history
- claude_dir = home / ".claude"
- if claude_dir.exists():
- for item in claude_dir.iterdir():
- if item.is_file():
- subprocess.run(
- ["docker", "cp", str(item), f"{container_id}:/home/bench/.claude/{item.name}"],
- check=True, capture_output=not verbose,
- )
- # macOS keeps the Claude OAuth credentials in the login Keychain, not in
- # ~/.claude/.credentials.json, so a subscription login does not travel with the
- # file copy above and the container lands on "Not logged in ยท Please run /login".
- # Materialise the same JSON that Claude Code writes on Linux.
- #
- # The secret is piped through stdin: never written to a host temp file, and
- # never placed in argv where `ps` would expose it.
- if sys.platform == "darwin" and not (claude_dir / ".credentials.json").exists():
- creds = ""
- try:
- creds = subprocess.run(
- ["security", "find-generic-password", "-s", "Claude Code-credentials", "-w"],
- capture_output=True, text=True, check=True,
- ).stdout.strip()
- except (subprocess.CalledProcessError, FileNotFoundError):
- pass
- if creds:
- subprocess.run(
- ["docker", "exec", "-i", container_id, "/bin/bash", "-c",
- "umask 077 && cat > /home/bench/.claude/.credentials.json"],
- input=creds, text=True, check=True, capture_output=not verbose,
- )
- if verbose:
- print(" [docker] Claude credentials taken from the macOS Keychain.",
- file=sys.stderr)
- else:
- print(
- " [docker] WARNING: no Claude credentials found. On macOS the OAuth token "
- "lives in the Keychain; `security` could not read it (you may need to allow "
- "access when prompted). Set ANTHROPIC_API_KEY instead, or the agent will "
- "report 'Not logged in'.",
- file=sys.stderr,
- )
- # Copy ~/.claude.json as a snapshot so Claude skips the first-launch welcome screen.
- claude_json = home / ".claude.json"
- if claude_json.exists():
- subprocess.run(
- ["docker", "cp", str(claude_json), f"{container_id}:/home/bench/.claude.json"],
- check=True, capture_output=not verbose,
- )
- # Copy Grok credentials as snapshots so `grok` can authenticate inside the
- # container without a browser login. Only the small auth/config files are copied โ
- # ~/.grok/sessions/ and friends are skipped. No-op unless the host has ~/.grok.
- grok_home = home / ".grok"
- if grok_home.exists():
- subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c", "mkdir -p /home/bench/.grok"],
- check=True, capture_output=not verbose,
- )
- for name in ("auth.json", "config.toml"):
- item = grok_home / name
- if item.exists():
- subprocess.run(
- ["docker", "cp", str(item), f"{container_id}:/home/bench/.grok/{name}"],
- check=True, capture_output=not verbose,
- )
- seed_agent_trust(container_id, default_agent, verbose=verbose)
- # Copy extra directories into the container (e.g. third-party plugin skill dirs).
- for src, dst in (extra_dirs or []):
- if not src.exists():
- print(f" [docker] Warning: extra_dir source not found, skipping: {src}", file=sys.stderr)
- continue
- if verbose:
- print(f" [docker] Copying {src} โ container:{dst}", file=sys.stderr)
- subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c", f"mkdir -p {dst}"],
- capture_output=True,
- )
- subprocess.run(
- ["docker", "cp", f"{src}/.", f"{container_id}:{dst}"],
- check=True, capture_output=not verbose,
- )
- # Patch localhost โ host.docker.internal in settings.json inside the container.
- # Claude reads ANTHROPIC_BASE_URL from settings.json["env"] which overrides process env,
- # so we must fix it in the file itself (container-local copy only, host file untouched).
- subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c",
- r"sed -i 's|http://localhost:|http://host.docker.internal:|g'"
- r" /home/bench/.claude/settings.json 2>/dev/null || true"],
- capture_output=True,
- )
- # Drop empty auth keys from the container's settings.json. Because that `env`
- # block outranks process env, an `ANTHROPIC_AUTH_TOKEN: ""` left over from a
- # proxy setup can shadow both the Keychain credentials and a real
- # ANTHROPIC_API_KEY, and the agent reports "Not logged in" with valid auth
- # available. Host file untouched.
- subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c",
- "python3 - <<'ENDOFPATCH'\n"
- "import json\n"
- "p='/home/bench/.claude/settings.json'\n"
- "try:\n"
- " d=json.load(open(p))\n"
- "except Exception:\n"
- " raise SystemExit(0)\n"
- "env=d.get('env')\n"
- "if isinstance(env, dict):\n"
- " drop=[k for k,v in env.items() if v == '']\n"
- " for k in drop: env.pop(k)\n"
- " if drop:\n"
- " json.dump(d, open(p,'w'), indent=2)\n"
- " print('dropped empty env keys: ' + ', '.join(drop))\n"
- "ENDOFPATCH"],
- capture_output=not verbose,
- )
- # Write global agtx config for bench user so the TUI skips the agent-selection wizard.
- #
- # `auto_trust = true` because a benchmark container has no human at the board.
- # agtx's default is off โ a trust prompt surfaces as a Blocked card and waits
- # for a person โ which here would stall the instance until its timeout. Same
- # reasoning as docker/entrypoint.sh: disposable container, nothing in it but
- # the repo under test.
- global_config_content = f'default_agent = "{default_agent}"\nauto_trust = true\n'
- if verbose:
- print(f" [docker] Writing global agtx config (default_agent={default_agent})...", file=sys.stderr)
- subprocess.run(
- ["docker", "exec", container_id,
- "/bin/bash", "-c",
- f"mkdir -p /home/bench/.config/agtx && cat > /home/bench/.config/agtx/config.toml << 'ENDOFCONFIG'\n{global_config_content}\nENDOFCONFIG"],
- check=True,
- capture_output=not verbose,
- )
- # Write .agtx/config.toml into /testbed inside the container
- # Build the config content (with base_branch injected) in memory
- content = config_path.read_text()
- base_branch_line = f'base_branch = "{base_commit}"\n'
- lines = [l for l in content.splitlines(keepends=True) if not l.startswith("base_branch =")]
- insert_at = next((i for i, l in enumerate(lines) if l.startswith("[")), len(lines))
- lines.insert(insert_at, base_branch_line)
- config_content = "".join(lines)
- if verbose:
- print(f" [docker] Writing .agtx/config.toml into container...", file=sys.stderr)
- subprocess.run(
- ["docker", "exec", container_id,
- "/bin/bash", "-c",
- f"mkdir -p /testbed/.agtx && cat > /testbed/.agtx/config.toml << 'ENDOFCONFIG'\n{config_content}\nENDOFCONFIG"],
- check=True,
- capture_output=not verbose,
- )
- # Clean stale agtx state (worktrees, project DB) โ same as _write_agtx_config
- subprocess.run(
- ["docker", "exec", container_id,
- "/bin/bash", "-c",
- "rm -rf /testbed/.agtx/worktrees /testbed/.agtx/db"],
- capture_output=True,
- )
- # Activate the conda 'testbed' environment for all new shells.
- # SWE-bench images ship with repo dependencies pre-installed in conda env 'testbed'.
- # Without this, Claude uses the system python/pip and tries to reinstall everything.
- subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c",
- "echo 'source /opt/miniconda3/etc/profile.d/conda.sh && conda activate testbed'"
- " >> /root/.bashrc"],
- check=True, capture_output=not verbose,
- )
- if tools_container_id:
- # Tools volume is mounted at /tools (read-only). Symlink everything from there โ
- # zero writable-layer usage. libutempter is included in the volume because
- # SWE-bench containers don't have it (tmux needs it).
- if verbose:
- print(f" [docker] Wiring tools from volume...", file=sys.stderr)
- subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c",
- "ln -sf /tools/tmux /usr/bin/tmux"
- " && ln -sf /tools/node /usr/bin/node"
- " && ln -sf /tools/node_modules/npm/bin/npm-cli.js /usr/bin/npm"
- " && ln -sf /tools/node_modules/npm/bin/npx-cli.js /usr/bin/npx"
- " && ln -sf /tools/node_modules/@anthropic-ai/claude-code/bin/claude.exe /usr/bin/claude"
- " && mkdir -p /usr/lib/node_modules"
- " && ln -sf /tools/node_modules/@anthropic-ai /usr/lib/node_modules/@anthropic-ai"
- " && ln -sf /tools/node_modules/npm /usr/lib/node_modules/npm"
- " && test -f /tools/node_modules/.bin/caveman && ln -sf /tools/node_modules/.bin/caveman /usr/bin/caveman || true"
- " && test -e /tools/node_modules/@xai-official/grok/bin/grok"
- " && ln -sf /tools/node_modules/@xai-official/grok/bin/grok /usr/bin/grok"
- " && ln -sf /tools/node_modules/@xai-official /usr/lib/node_modules/@xai-official || true"
- " && mkdir -p /usr/lib/x86_64-linux-gnu"
- " && ln -sf /tools/lib/x86_64-linux-gnu/libutempter.so.0 /usr/lib/x86_64-linux-gnu/libutempter.so.0"
- " && ln -sf /tools/lib/x86_64-linux-gnu/libutempter.so.1.2.1 /usr/lib/x86_64-linux-gnu/libutempter.so.1.2.1"
- " && ln -sf /tools/lib/x86_64-linux-gnu/libevent_core-2.1.so.7 /usr/lib/x86_64-linux-gnu/libevent_core-2.1.so.7"],
- check=True, capture_output=not verbose,
- )
- else:
- if verbose:
- print(f" [docker] Installing tmux, Node.js 22, Claude Code, Grok...", file=sys.stderr)
- subprocess.run(
- ["docker", "exec", container_id,
- "/bin/bash", "-c",
- "apt-get update -qq && apt-get install -y -qq tmux curl ca-certificates "
- "&& curl -fsSL https://deb.nodesource.com/setup_22.x | bash - "
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 910 on GitHub
- "&& apt-get install -y -qq nodejs "
- "&& npm install -g @anthropic-ai/claude-code @xai-official/grok"],
- check=True,
- capture_output=not verbose,
- )
- # Run optional sandbox init commands (e.g. "rtk init -g").
- # These run after credentials are in place so tools can write into ~/.claude/.
- # Prepend ~/.local/bin to PATH so tools installed there (e.g. rtk) are found.
- for cmd in (sandbox_init or []):
- if verbose:
- print(f" [docker] sandbox_init: {cmd}", file=sys.stderr)
- subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c",
- f"HOME=/home/bench && export HOME PATH=$HOME/.local/bin:$PATH && {cmd}"],
- check=True, capture_output=not verbose,
- )
- def start_tui_in_container(slug: str, container_id: str, verbose: bool = False) -> None:
- """Start the agtx TUI inside the container's tmux session."""
- if verbose:
- print(f" [docker] Starting agtx TUI in container tmux session '{slug}'...", file=sys.stderr)
- # Launch agtx with all required env vars baked into the command so every tmux
- # window it spawns (agent sessions) inherits them from the start โ no race.
- # HOME=/home/bench โ agtx reads ~/.config/agtx/config.toml
- # IS_SANDBOX=1 โ bypasses Claude's root check (--dangerously-skip-permissions)
- # PATH/CONDA vars โ activate conda 'testbed' env so pre-installed deps are used
- testbed_bin = "/opt/miniconda3/envs/testbed/bin"
- env_prefix = (
- f"HOME=/home/bench "
- f"IS_SANDBOX=1 "
- f"CONDA_DEFAULT_ENV=testbed "
- f"CONDA_PREFIX=/opt/miniconda3/envs/testbed "
- f"PATH={testbed_bin}:/opt/miniconda3/bin:/home/bench/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
- )
- # Pass through API keys that agents read from the environment (only when set
- # on the host). Grok uses XAI_API_KEY when ~/.grok/auth.json is absent.
- passthrough_env = {
- k: v for k in ("XAI_API_KEY", "ANTHROPIC_API_KEY") if (v := os.environ.get(k))
- }
- for key, val in passthrough_env.items():
- env_prefix += f" {key}={shlex.quote(val)}"
- subprocess.run(
- ["docker", "exec", "-d", container_id,
- "tmux", "new-session", "-d", "-s", slug,
- f"env {env_prefix} agtx /testbed"],
- check=True,
- capture_output=not verbose,
- )
- # Also set them in the agtx tmux server global env so windows created later
- # (e.g. after a phase transition) also inherit the correct PATH.
- time.sleep(3)
- for key, val in [
- ("HOME", "/home/bench"),
- ("IS_SANDBOX", "1"),
- ("CONDA_DEFAULT_ENV", "testbed"),
- ("CONDA_PREFIX", "/opt/miniconda3/envs/testbed"),
- ("PATH", f"{testbed_bin}:/opt/miniconda3/bin:/home/bench/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"),
- ] + list(passthrough_env.items()):
- subprocess.run(
- ["docker", "exec", container_id,
- "tmux", "-L", "agtx", "set-environment", "-g", key, val],
- capture_output=True,
- )
- # Give the TUI time to fully start before the caller proceeds
- time.sleep(5)
- def stop_docker_container(container_id: str, verbose: bool = False) -> None:
- """Stop and remove the container."""
- if verbose:
- print(f" [docker] Stopping container {container_id[:12]}...", file=sys.stderr)
- subprocess.run(["docker", "stop", container_id], capture_output=True)
- subprocess.run(["docker", "rm", container_id], capture_output=True)
- # ---------------------------------------------------------------------------
- # Token/cost tracking via tokscale
- # ---------------------------------------------------------------------------
- # CLI flag that filters tokscale output to a specific agent
- _TOKSCALE_FLAG: dict[str, str] = {
- "claude": "--claude",
- "codex": "--codex",
- "gemini": "--gemini",
- "copilot": "--copilot",
- "opencode": "--opencode",
- }
- # Cached result of tokscale availability check
- _tokscale_bin: str | None | bool = False # False = not yet checked
- def _find_tokscale() -> str | None:
- """Return path to tokscale binary, or None if not installed."""
- global _tokscale_bin
- if _tokscale_bin is not False:
- return _tokscale_bin # type: ignore
- result = subprocess.run(["which", "tokscale"], capture_output=True, text=True)
- _tokscale_bin = result.stdout.strip() or None
- return _tokscale_bin # type: ignore
- def _tokscale_snapshot(agent: str) -> dict:
- """
- Run `tokscale --json --today --{agent}` and return aggregated totals across all models:
- {input, output, cache_read, cache_write, reasoning, cost_usd}
- Returns zeros if tokscale is unavailable or the agent has no records today.
- """
- tokscale = _find_tokscale()
- if not tokscale:
- return {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "cost_usd": 0.0}
- flag = _TOKSCALE_FLAG.get(agent)
- cmd = [tokscale, "--json", "--today"]
- if flag:
- cmd.append(flag)
- try:
- result = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
- if result.returncode != 0 or not result.stdout.strip():
- return {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "cost_usd": 0.0}
- data = json.loads(result.stdout)
- entries = data.get("entries", [])
- totals = {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "cost_usd": 0.0}
- for entry in entries:
- totals["input"] += entry.get("input", 0)
- totals["output"] += entry.get("output", 0)
- totals["cache_read"] += entry.get("cacheRead", 0)
- totals["cache_write"] += entry.get("cacheWrite", 0)
- totals["reasoning"] += entry.get("reasoning", 0)
- totals["cost_usd"] += entry.get("cost", 0.0)
- return totals
- except Exception:
- return {"input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "reasoning": 0, "cost_usd": 0.0}
- def tokscale_diff(before: dict, after: dict) -> dict:
- """
- Subtract before-snapshot from after-snapshot to get this task's usage.
- Returns the standard cost_data dict used throughout the script.
- """
- if not _find_tokscale():
- return {"cost_usd": None, "cost_tokens": None}
- inp = max(0, after["input"] - before["input"])
- out = max(0, after["output"] - before["output"])
- cost = max(0.0, after["cost_usd"] - before["cost_usd"])
- total = inp + out + max(0, after["cache_read"] - before["cache_read"]) + \
- max(0, after["cache_write"] - before["cache_write"]) + \
- max(0, after["reasoning"] - before["reasoning"])
- return {
- "cost_usd": round(cost, 6) if cost > 0 else None,
- "cost_tokens": total if total > 0 else None,
- }
- def tokscale_from_container(container_id: str, agent: str) -> dict:
- """
- Read token/cost usage from a sandbox container by copying /home/bench/.claude
- to a temp dir and running tokscale --home against it.
- Returns the standard cost_data dict {cost_usd, cost_tokens}, or nulls on failure.
- """
- tokscale = _find_tokscale()
- if not tokscale:
- return {"cost_usd": None, "cost_tokens": None}
- flag = _TOKSCALE_FLAG.get(agent)
- if not flag:
- return {"cost_usd": None, "cost_tokens": None}
- try:
- with tempfile.TemporaryDirectory() as tmpdir:
- # Copy /home/bench/.claude out of the container
- result = subprocess.run(
- ["docker", "cp", f"{container_id}:/home/bench/.claude", tmpdir],
- capture_output=True,
- )
- if result.returncode != 0:
- return {"cost_usd": None, "cost_tokens": None}
- # tokscale --home expects the home dir (parent of .claude)
- result = subprocess.run(
- [tokscale, "--home", tmpdir, "--json", flag],
- capture_output=True,
- text=True,
- timeout=15,
- )
- if result.returncode != 0 or not result.stdout.strip():
- return {"cost_usd": None, "cost_tokens": None}
- data = json.loads(result.stdout)
- entries = data.get("entries", [])
- inp = out = cr = cw = reasoning = 0
- cost = 0.0
- for entry in entries:
- inp += entry.get("input", 0)
- out += entry.get("output", 0)
- cr += entry.get("cacheRead", 0)
- cw += entry.get("cacheWrite", 0)
- reasoning += entry.get("reasoning", 0)
- cost += entry.get("cost", 0.0)
- total = inp + out + cr + cw + reasoning
- return {
- "cost_usd": round(cost, 6) if cost > 0 else None,
- "cost_tokens": total if total > 0 else None,
- }
- except Exception:
- return {"cost_usd": None, "cost_tokens": None}
- class ResultsStore:
- """Thread-safe, resumable persistence for benchmark results."""
- def __init__(self, output_dir: Path):
- self.output_dir = output_dir
- output_dir.mkdir(parents=True, exist_ok=True)
- self.predictions_path = output_dir / "predictions.jsonl"
- self.results_path = output_dir / "results.json"
- self._lock = threading.Lock()
- self._results: list[dict] = []
- self._done_ids: set[str] = set()
- self._load_existing()
- def _load_existing(self) -> None:
- if self.results_path.exists():
- try:
- data = json.loads(self.results_path.read_text())
- self._results = data if isinstance(data, list) else []
- self._done_ids = {r["instance_id"] for r in self._results}
- print(f"Resuming: {len(self._done_ids)} tasks already completed.")
- except Exception:
- pass
- def is_done(self, instance_id: str) -> bool:
- return instance_id in self._done_ids
- def save_result(
- self,
- instance_id: str,
- status: str,
- duration_seconds: float,
- model_name: str,
- model_patch: str = "",
- cost_usd: float | None = None,
- cost_tokens: int | None = None,
- error: str | None = None,
- ) -> None:
- with self._lock:
- result = {
- "instance_id": instance_id,
- "status": status,
- "duration_seconds": round(duration_seconds, 1),
- "cost_usd": cost_usd,
- "cost_tokens": cost_tokens,
- "model_patch": model_patch,
- "error": error,
- }
- self._results.append(result)
- self._done_ids.add(instance_id)
- # Append prediction (SWE-bench format)
- with self.predictions_path.open("a") as f:
- pred = {
- "instance_id": instance_id,
- "model_name_or_path": model_name,
- "model_patch": model_patch,
- }
- f.write(json.dumps(pred) + "\n")
- # Rewrite results atomically
- tmp = self.results_path.with_suffix(".json.tmp")
- tmp.write_text(json.dumps(self._results, indent=2))
- tmp.replace(self.results_path)
- # ---------------------------------------------------------------------------
- # Phase artifact paths per bundled plugin
- # ---------------------------------------------------------------------------
- # Artifact path for the "planning" phase (relative to worktree root).
- # None means the plugin has no planning artifact โ fall back to pane stability.
- PLUGIN_PLANNING_ARTIFACTS: dict[str, str | None] = {
- "agtx": ".agtx/plan.md",
- "agtx-terse": ".agtx/plan.md",
- "gsd": ".planning/phases/*/{phase}-CONTEXT.md",
- "spec-kit": None,
- "bmad": None,
- "openspec": None,
- "agent-skills": None,
- "superpowers": "docs/superpowers/plans/*.md",
- "oh-my-claudecode": None,
- "void": None,
- }
- # Artifact path for the "running" phase (relative to worktree root).
- # None means the plugin has no running artifact โ fall back to pane stability.
- PLUGIN_RUNNING_ARTIFACTS: dict[str, str | None] = {
- "agtx": ".agtx/execute.md",
- "agtx-terse": ".agtx/execute.md",
- "gsd": ".planning/phases/*/{phase}-SUMMARY.md",
- "spec-kit": None,
- "bmad": "_bmad-output/implementation-artifacts/*.md",
- "openspec": None,
- "agent-skills": None,
- "superpowers": None,
- "oh-my-claudecode": None,
- "void": None,
- }
- # Artifact path for the "review" phase (relative to worktree root).
- # None means no review artifact โ collect patch immediately after entering Review.
- PLUGIN_REVIEW_ARTIFACTS: dict[str, str | None] = {
- "agtx": ".agtx/review.md",
- "agtx-terse": ".agtx/review.md",
- "gsd": None,
- "spec-kit": None,
- "bmad": None,
- "openspec": None,
- "agent-skills": None,
- "superpowers": None,
- "oh-my-claudecode": None,
- "void": None,
- }
- def _artifact_exists(worktree: Path, pattern: str, container_id: str | None = None) -> bool:
- """Return True if the artifact path (possibly with * or {phase} placeholders) exists.
- In Docker mode (container_id provided), checks inside the container via docker exec
- instead of the local filesystem (the worktree lives inside the container).
- """
- if "{phase}" in pattern:
- for n in range(1, 21):
- for fmt in (f"{n:02d}", str(n)):
- if _artifact_exists(worktree, pattern.replace("{phase}", fmt), container_id):
- return True
- return False
- if container_id:
- # In Docker: use shell glob via docker exec so * patterns work
- path = str(worktree / pattern)
- result = subprocess.run(
- ["docker", "exec", container_id, "/bin/bash", "-c", f"ls {path} 2>/dev/null | head -1"],
- capture_output=True, text=True,
- )
- return bool(result.stdout.strip())
- if "*" not in pattern:
- return (worktree / pattern).exists()
- parts = Path(pattern).parts
- base = worktree
- for i, part in enumerate(parts):
- if "*" in part:
- remaining = str(Path(*parts[i:]))
- return any(True for _ in base.glob(remaining))
- base = base / part
- return False
- # ---------------------------------------------------------------------------
- # Task runner
- # ---------------------------------------------------------------------------
- class TaskRunner:
- """Runs a single SWE-bench instance through the full agtx lifecycle."""
- SMOKE_TEST_DESCRIPTION = (
- "This is a smoke test run. Do not modify any source files.\n\n"
- "Planning phase: write the following content to `.agtx/plan.md` in the current working directory:\n"
- "```\n"
- "# Smoke Test Plan\n"
- "This is a smoke test. In the running phase: write an empty `.agtx/execute.md`. "
- "In the review phase: write an empty `.agtx/review.md`. Do not modify any source files.\n"
- "```\n"
- "Then stop and wait."
- )
- NOTE = (
- "IMPORTANT CONSTRAINTS โ these apply to all phases including review:\n"
- "- NEVER install packages, dependencies, or build the project.\n"
- "- NEVER run tests under any circumstances.\n"
- "- Do not run any git commands (no fetch, pull, merge, or commit).\n"
- "- The repo may not be installable in this environment โ do not attempt it.\n"
- "- Read the source code, understand the bug, and fix it by editing the relevant files directly."
- )
- @staticmethod
- def _strip_code(problem_statement: str) -> str:
- """Remove fenced code blocks and inline code, keep prose."""
- # Remove fenced code blocks (``` ... ```)
- text = re.sub(r"```.*?```", "", problem_statement, flags=re.DOTALL)
- # Collapse runs of blank lines left behind
- text = re.sub(r"\n{3,}", "\n\n", text)
- return text.strip()
- def _build_description(self, problem_statement: str) -> str:
- if self.hard:
- return self._strip_code(problem_statement) + "\n\n---\n" + self.NOTE
- return problem_statement + "\n\n---\n" + self.NOTE
- def __init__(
- self,
- instance: dict,
- repo_path: Path,
- agtx_bin: str,
- plugin: str,
- agent: str,
- running_agent: str,
- model_name: str,
- phase_timeout: int,
- verbose: bool = False,
- smoke_test: bool = False,
- hard: bool = False,
- container_id: str | None = None,
- ):
- self.instance = instance
- self.instance_id = instance["instance_id"]
- self.repo_path = repo_path
- self.agtx_bin = agtx_bin
- self.plugin = plugin
- self.agent = agent
- self.running_agent = running_agent
- self.model_name = model_name
- self.phase_timeout = phase_timeout
- self.verbose = verbose
- self.smoke_test = smoke_test
- self.hard = hard
- self.container_id = container_id
- self.mcp: McpClient | None = None
- def _poll_transition(self, request_id: str, timeout: int = 120) -> None:
- deadline = time.monotonic() + timeout
- while time.monotonic() < deadline:
- result = self.mcp.call("get_transition_status", request_id=request_id)
- status = result.get("status", "pending")
- if status == "completed":
- return
- if status == "error":
- raise McpError(f"Transition failed: {result.get('error')}")
- time.sleep(2)
- raise TimeoutError(f"Transition {request_id} timed out after {timeout}s")
- def _move_task(self, task_id: str, action: str) -> None:
- result = self.mcp.call("move_task", task_id=task_id, action=action)
- self._poll_transition(result["request_id"])
- def _wait_for_status(self, task_id: str, target_status: str, timeout: int = 120) -> dict:
- deadline = time.monotonic() + timeout
- while time.monotonic() < deadline:
- task = self.mcp.call("get_task", task_id=task_id)
- if task.get("status") == target_status:
- return task
- time.sleep(3)
- raise TimeoutError(f"Task never reached {target_status} within {timeout}s")
- def _wait_for_worktree(self, task_id: str, timeout: int = 180) -> dict:
- """
- Wait until the task has a worktree_path set in the DB.
- This is populated asynchronously by the TUI's background setup thread
- after transition_to_planning/running completes โ the transition request
- being marked 'completed' only means the TUI accepted it, not that
- worktree setup is done.
- """
- deadline = time.monotonic() + timeout
- while time.monotonic() < deadline:
- task = self.mcp.call("get_task", task_id=task_id)
- if task.get("worktree_path"):
- return task
- time.sleep(3)
- raise TimeoutError(f"Worktree never populated for task {task_id} within {timeout}s")
- def _wait_for_artifact(self, task_id: str, worktree_path: str, artifact_pattern: str, timeout: int) -> bool:
- """Poll for an artifact file in the worktree. Returns True if found, False on timeout."""
- worktree = Path(worktree_path)
- deadline = time.monotonic() + timeout
- while time.monotonic() < deadline:
- if _artifact_exists(worktree, artifact_pattern, self.container_id):
- return True
- time.sleep(5)
- return False
- # Patterns that indicate a shell command run by the agent is waiting for input,
- # blocking Claude from finishing. Only used in the stability fallback path.
- _INTERACTIVE_PROMPT_PATTERNS = [
- r"\[y/N\]",
- r"\[Y/n\]",
- r"\[y/n\]",
- r"\[Y/N\]",
- r"Press Enter to continue",
- r"Press any key",
- # Claude's usage-limit pause. It ends the turn and waits on a keypress
- # that nothing sends, so the task parks indefinitely โ observed sitting
- # ~50 minutes, past both the stated reset time and --phase-timeout.
- # Matching it turns a silent stall into a named failure.
- r"You've hit your session limit",
- r"Press .* to continue after reset",
- ]
- def _check_interactive_prompt(self, content: str) -> str | None:
- """Return the matching line if pane content looks like an interactive prompt, else None."""
- for line in content.splitlines()[-20:]:
- for pattern in self._INTERACTIVE_PROMPT_PATTERNS:
- if re.search(pattern, line, re.IGNORECASE):
- return line.strip()
- return None
- # Claude always prints one of these lines when it finishes a response.
- # Detecting this (plus the โฏ prompt) is more reliable than pure pane stability.
- _CLAUDE_FINISHED_RE = re.compile(r"โป\s+[A-Z][a-z]+ for \d+s")
- def _is_claude_idle(self, content: str) -> bool:
- """Return True when Claude has finished its last response and shows the โฏ prompt."""
- lines = content.splitlines()
- has_finish_marker = any(self._CLAUDE_FINISHED_RE.search(l) for l in lines)
- has_prompt = any(l.strip() == "โฏ" or l.strip().startswith("โฏ ") for l in lines[-10:])
- return has_finish_marker and has_prompt
- def _wait_for_pane_stable(self, task_id: str, timeout: int) -> None:
- """Wait until Claude has finished its last response (no artifact available).
- Primary signal: Claude prints 'โป Cooked/Worked/Crunched for Xs' + โฏ prompt.
- Fallback: pane content identical for 2 consecutive 10s checks.
- If an interactive prompt is detected while stable, prints a warning so the user
- can attach to the session and answer it.
- """
- prev_content: str | None = None
- stable_count = 0
- prompt_warned = False
- deadline = time.monotonic() + timeout
- while time.monotonic() < deadline:
- time.sleep(5)
- try:
- result = self.mcp.call("read_pane_content", task_id=task_id, lines=80)
- content = result.get("content", "")
- except McpError:
- content = ""
- # Primary: Claude idle signal
- if self._is_claude_idle(content):
- return
- # Interactive prompt check (even when not yet stable)
- if content == prev_content:
- stable_count += 1
- prompt_line = self._check_interactive_prompt(content)
- if prompt_line:
- if not prompt_warned:
- print(f"\n[{self.instance_id}] โ Agent is waiting for input: \"{prompt_line}\"", file=sys.stderr)
- slug = self.instance_id.replace("__", "-").replace("_", "-")
- print(f" Attach to answer: docker exec -it swebench-{slug} tmux -L agtx attach -t testbed:1", file=sys.stderr)
- prompt_warned = True
- stable_count = 0 # Reset โ not actually done
- elif stable_count >= 2:
- # Pane stable but no Claude finish marker โ needs attention.
- # Possible causes: agent error, waiting for user approval (e.g. superpowers brainstorm),
- # or genuinely stuck. Warn once and keep polling; do NOT advance.
- if not prompt_warned:
- slug = self.instance_id.replace("__", "-").replace("_", "-")
- print(f"\n[{self.instance_id}] โ Pane stable but no finish marker โ agent may be stuck, hit an error, or requires manual approval.", file=sys.stderr)
- print(f" Attach to inspect or interact: docker exec -it swebench-{slug} tmux -L agtx attach -t testbed:1", file=sys.stderr)
- prompt_warned = True
- stable_count = 0 # Keep polling, don't advance
- else:
- stable_count = 0
- prompt_warned = False
- prev_content = content
- def run(self) -> dict:
- start = time.monotonic()
- # What `git diff <base_commit>` already reports before the agent has done
- # anything. SWE-bench images are not pristine checkouts of base_commit:
- # for astropy-12907 the baseline is 1861 files (3722 lines of
- # `old mode 100644` / `new mode 100755`) plus the image's own
- # `setuptools==68.0.0` pin in pyproject.toml. Subtracted at collection so
- # a real fix is not buried in ~224KB of noise.
- self.baseline_patch = self._raw_diff()
- if self.verbose and self.baseline_patch:
- n = self.baseline_patch.count("diff --git ")
- print(
- f" [{self.instance_id}] Baseline diff: {n} file(s) already differ from "
- f"base_commit before the agent runs; these are excluded from the prediction.",
- file=sys.stderr,
- )
- if self.verbose:
- print(f" [{self.instance_id}] Starting MCP handshake...", file=sys.stderr)
- self.mcp = McpClient(self.agtx_bin, str(self.repo_path), container_id=self.container_id)
- task_id = None
- try:
- if self.verbose:
- print(f" [{self.instance_id}] Creating task...", file=sys.stderr)
- problem = self.instance.get("problem_statement", "")
- if self.smoke_test:
- description = self.SMOKE_TEST_DESCRIPTION
- else:
- description = self._build_description(problem)
- task_resp = self.mcp.call(
- "create_task",
- title=self.instance_id,
- description=description,
- plugin=self.plugin,
- )
- task_id = task_resp["id"]
- if self.verbose:
- print(f" [{self.instance_id}] task_id={task_id}, moving to Planning...", file=sys.stderr)
- # Backlog โ Planning
- self._move_task(task_id, "move_forward")
- self._wait_for_status(task_id, "planning", timeout=120)
- # Wait for worktree to be created (set by TUI background thread)
- if self.verbose:
- print(f" [{self.instance_id}] Waiting for worktree...", file=sys.stderr)
- planning_task = self._wait_for_worktree(task_id, timeout=180)
- worktree_path = planning_task.get("worktree_path", "")
- # Snapshot tokscale before agent starts working (non-sandbox only)
- cost_before = _tokscale_snapshot(self.running_agent) if not self.container_id else None
- # Wait for planning phase artifact (.agtx/plan.md for agtx/agtx-terse plugins)
- planning_artifact = PLUGIN_PLANNING_ARTIFACTS.get(self.plugin)
- if planning_artifact and worktree_path:
- if self.verbose:
- print(f" [{self.instance_id}] Waiting for planning artifact ({planning_artifact})...", file=sys.stderr)
- found = self._wait_for_artifact(task_id, worktree_path, planning_artifact, self.phase_timeout)
- if not found:
- raise TimeoutError(f"Planning artifact not found within {self.phase_timeout}s")
- else:
- if self.verbose:
- print(f" [{self.instance_id}] Waiting for planning phase (pane stability)...", file=sys.stderr)
- self._wait_for_pane_stable(task_id, self.phase_timeout)
- if self.verbose:
- print(f" [{self.instance_id}] Planning done, moving to Running...", file=sys.stderr)
- # Small delay: let the agent finish any final output after writing the artifact
- # before the TUI sends /exit (clear_context_on_advance) and restarts it.
- time.sleep(5)
- # Planning โ Running
- self._move_task(task_id, "move_forward")
- running_task = self._wait_for_status(task_id, "running", timeout=60)
- worktree_path = running_task.get("worktree_path") or worktree_path
- # Wait for running phase artifact (.agtx/execute.md for agtx/agtx-terse plugins)
- running_artifact = PLUGIN_RUNNING_ARTIFACTS.get(self.plugin)
- if running_artifact and worktree_path:
- if self.verbose:
- print(f" [{self.instance_id}] Waiting for running artifact ({running_artifact})...", file=sys.stderr)
- found = self._wait_for_artifact(task_id, worktree_path, running_artifact, self.phase_timeout)
- if not found:
- raise TimeoutError(f"Running artifact not found within {self.phase_timeout}s")
- else:
- if self.verbose:
- print(f" [{self.instance_id}] Waiting for running phase (pane stability)...", file=sys.stderr)
- self._wait_for_pane_stable(task_id, self.phase_timeout)
- # Collect token/cost usage:
- # - sandbox: read directly from container's /home/bench/.claude via tokscale --home
- # - non-sandbox: diff host tokscale snapshots taken before/after running phase
- if self.container_id:
- cost_data = tokscale_from_container(self.container_id, self.running_agent)
- else:
- cost_data = tokscale_diff(cost_before, _tokscale_snapshot(self.running_agent))
- if self.verbose:
- print(f" [{self.instance_id}] Running done, moving to Review...", file=sys.stderr)
- # Running โ Review
- self._move_task(task_id, "move_forward")
- review_task = self._wait_for_status(task_id, "review", timeout=60)
- worktree_path = review_task.get("worktree_path") or worktree_path
- # Wait for review phase artifact (.agtx/review.md for agtx/agtx-terse plugins)
- review_artifact = PLUGIN_REVIEW_ARTIFACTS.get(self.plugin)
- if review_artifact and worktree_path:
- if self.verbose:
- print(f" [{self.instance_id}] Waiting for review artifact ({review_artifact})...", file=sys.stderr)
- found = self._wait_for_artifact(task_id, worktree_path, review_artifact, self.phase_timeout)
- if not found:
- raise TimeoutError(f"Review artifact not found within {self.phase_timeout}s")
- else:
- if self.verbose:
- print(f" [{self.instance_id}] Waiting for review phase (pane stability)...", file=sys.stderr)
- self._wait_for_pane_stable(task_id, self.phase_timeout)
- model_patch = self._collect_patch(review_task)
- if self.verbose:
- print(f" [{self.instance_id}] Done. patch_len={len(model_patch)}", file=sys.stderr)
- # Review โ Done (cleanup worktree)
- try:
- self._move_task(task_id, "move_to_done")
- except Exception:
- pass
- return {
- "status": "success",
- "duration_seconds": time.monotonic() - start,
- "model_patch": model_patch,
- **cost_data,
- "error": None,
- }
- except TimeoutError as e:
- self._cleanup(task_id)
- return self._error_result("timeout", time.monotonic() - start, str(e))
- except Exception as e:
- self._cleanup(task_id)
- return self._error_result("error", time.monotonic() - start, str(e))
- finally:
- if self.mcp:
- self.mcp.close()
- @staticmethod
- def _split_file_sections(patch: str) -> dict[str, str]:
- """Split a unified diff into {path_header: section_text}."""
- sections: dict[str, str] = {}
- current: str | None = None
- buf: list[str] = []
- for line in patch.splitlines(keepends=True):
- if line.startswith("diff --git "):
- if current is not None:
- sections[current] = "".join(buf)
- current = line.strip()
- buf = [line]
- elif current is not None:
- buf.append(line)
- if current is not None:
- sections[current] = "".join(buf)
- return sections
- def _strip_baseline(self, patch: str) -> str:
- """Drop file sections identical to the pre-agent baseline.
- Per file rather than per line: a file the agent genuinely edited has a
- different section and survives whole, so an agent that legitimately
- touches pyproject.toml is not silently dropped along with the image's
- own pin of it.
- """
- baseline = getattr(self, "baseline_patch", "") or ""
- if not baseline:
- return patch
- base_sections = self._split_file_sections(baseline)
- kept = [
- text
- for header, text in self._split_file_sections(patch).items()
- if base_sections.get(header) != text
- ]
- return "".join(kept)
- def _collect_patch(self, task: dict) -> str:
- return self._strip_baseline(self._raw_diff(task))
- def _raw_diff(self, task: dict | None = None) -> str:
- base_commit = self.instance["base_commit"]
- if self.container_id:
- # In Docker mode, diff /testbed directly inside the container
- result = subprocess.run(
- [
- "docker", "exec", self.container_id,
- "git", "-C", "/testbed", "diff", base_commit,
- "--", ".", ":!.agtx/",
- ],
- capture_output=True,
- )
- if result.returncode != 0:
- return ""
- return result.stdout.decode("utf-8", errors="replace")
- worktree_path = (task or {}).get("worktree_path") or str(self.repo_path)
- if not worktree_path:
- return ""
- # Diff the worktree's actual files against the base commit.
- # Using the worktree path directly avoids picking up any extra commits
- # the agent may have merged/fetched onto the branch โ we only care about
- # file-level changes, not git history.
- result = subprocess.run(
- ["git", "diff", base_commit,
- "--", ".", ":!.agtx/"],
- cwd=worktree_path,
- capture_output=True,
- )
- if result.returncode != 0:
- return ""
- # Decode as UTF-8, replacing undecodable bytes so binary files don't crash the run
- return result.stdout.decode("utf-8", errors="replace")
- def _cleanup(self, task_id: str | None) -> None:
- if task_id:
- try:
- self.mcp.call("move_task", task_id=task_id, action="move_to_done")
- except Exception:
- pass
- def _error_result(self, status: str, duration: float, error: str) -> dict:
- return {
- "status": status,
- "duration_seconds": duration,
- "model_patch": "",
- "cost_usd": None,
- "cost_tokens": None,
- "error": error,
- }
- # ---------------------------------------------------------------------------
- # Orchestrator
- # ---------------------------------------------------------------------------
- class BenchmarkOrchestrator:
- """Drives N tasks, manages concurrency."""
- def __init__(
- self,
- instances: list[dict],
- agtx_bin: str,
- config_path: Path,
- plugin: str,
- agent: str,
- running_agent: str,
- model_name: str,
- phase_timeout: int,
- workdir: str,
- output_dir: Path,
- concurrency: int,
- verbose: bool = False,
- smoke_test: bool = False,
- hard: bool = False,
- docker: bool = False,
- extra_dirs: list[tuple[Path, str]] | None = None,
- sandbox_init: list[str] | None = None,
- ):
- self.instances = instances
- self.agtx_bin = agtx_bin
- self.config_path = config_path
- self.plugin = plugin
- self.agent = agent
- self.running_agent = running_agent
- self.model_name = model_name
- self.phase_timeout = phase_timeout
- self.workdir = workdir
- self.store = ResultsStore(output_dir)
- self.concurrency = concurrency
- self.verbose = verbose
- self.smoke_test = smoke_test
- self.hard = hard
- self.docker = docker
- self.extra_dirs = extra_dirs or []
- self.sandbox_init = sandbox_init or []
- self.tools_container_id: str | None = None # set in run() when --sandbox is active
- def _run_one(self, instance: dict, progress: tqdm) -> None:
- instance_id = instance["instance_id"]
- if self.store.is_done(instance_id):
- progress.update(1)
- return
- slug = re.sub(r"[^a-z0-9]+", "-", instance_id.lower()).strip("-")[:50]
- start = time.monotonic()
- container_id: str | None = None
- repo_path: Path | None = None
- if self.docker:
- if self.verbose:
- print(f"\n[{instance_id}] Starting Docker container...", file=sys.stderr)
- try:
- container_id = start_docker_container(
- instance_id, slug, self.agtx_bin, verbose=self.verbose
- )
- setup_container(
- container_id, self.config_path, instance["base_commit"],
- verbose=self.verbose, tools_container_id=self.tools_container_id,
- extra_dirs=self.extra_dirs or None,
- sandbox_init=self.sandbox_init or None,
- )
- except Exception as e:
- if container_id:
- stop_docker_container(container_id, verbose=self.verbose)
- self.store.save_result(
- instance_id=instance_id,
- status="setup_error",
- duration_seconds=time.monotonic() - start,
- model_name=self.model_name,
- error=str(e),
- )
- progress.update(1)
- return
- if self.verbose:
- print(f"[{instance_id}] Starting TUI inside container...", file=sys.stderr)
- print(f" [docker] To watch the agent:", file=sys.stderr)
- print(f" docker exec -it swebench-{slug} tmux -L agtx attach -t testbed:1", file=sys.stderr)
- try:
- start_tui_in_container(slug, container_id, verbose=self.verbose)
- # start_tui_in_container already waits for TUI startup internally
- except Exception as e:
- stop_docker_container(container_id, verbose=self.verbose)
- self.store.save_result(
- instance_id=instance_id,
- status="setup_error",
- duration_seconds=time.monotonic() - start,
- model_name=self.model_name,
- error=f"TUI startup failed: {e}",
- )
- progress.update(1)
- return
- # In Docker mode, repo_path is unused by McpClient/patch collection
- # but TaskRunner still needs a Path object (used as identifier).
- repo_path = Path("/testbed")
- else:
- if self.verbose:
- print(f"\n[{instance_id}] Setting up repo...", file=sys.stderr)
- try:
- repo_path = setup_repo(
- instance, self.workdir, self.config_path,
- verbose=self.verbose, smoke_test=self.smoke_test,
- )
- except Exception as e:
- self.store.save_result(
- instance_id=instance_id,
- status="setup_error",
- duration_seconds=time.monotonic() - start,
- model_name=self.model_name,
- error=str(e),
- )
- progress.update(1)
- return
- if self.verbose:
- print(f"[{instance_id}] Starting TUI...", file=sys.stderr)
- try:
- start_tui_in_tmux(slug, repo_path, self.agtx_bin, verbose=self.verbose)
- time.sleep(8) # Wait for TUI startup + project registration
- except Exception as e:
- self.store.save_result(
- instance_id=instance_id,
- status="setup_error",
- duration_seconds=time.monotonic() - start,
- model_name=self.model_name,
- error=f"TUI startup failed: {e}",
- )
- progress.update(1)
- return
- if self.verbose:
- print(f"[{instance_id}] Connecting MCP client...", file=sys.stderr)
- runner = TaskRunner(
- instance=instance,
- repo_path=repo_path,
- agtx_bin=self.agtx_bin,
- plugin=self.plugin,
- agent=self.agent,
- running_agent=self.running_agent,
- model_name=self.model_name,
- phase_timeout=self.phase_timeout,
- verbose=self.verbose,
- smoke_test=self.smoke_test,
- hard=self.hard,
- container_id=container_id,
- )
- result = runner.run()
- if self.docker:
- stop_docker_container(container_id, verbose=self.verbose)
- else:
- kill_tmux_session(slug, repo_path)
- self.store.save_result(
- instance_id=instance_id,
- model_name=self.model_name,
- **result,
- )
- progress.update(1)
- progress.set_postfix_str(f"{result['status']} {instance_id}")
- def run(self) -> None:
- pending = [i for i in self.instances if not self.store.is_done(i["instance_id"])]
- total = len(self.instances)
- already_done = total - len(pending)
- print(f"Agent: {self.agent} | Plugin: {self.plugin} | Tasks: {len(pending)} | Concurrency: {self.concurrency}")
- print(f"Output: {self.store.output_dir}")
- if self.docker and _tools_image_exists():
- if self.verbose:
- print("[docker] Ensuring tools volume is ready...", file=sys.stderr)
- ensure_tools_volume(verbose=self.verbose)
- self.tools_container_id = "volume" # sentinel: volume is mounted, no container needed
- try:
- with tqdm(total=total, initial=already_done, unit="task") as progress:
- if self.concurrency == 1:
- for instance in pending:
- self._run_one(instance, progress)
- else:
- with ThreadPoolExecutor(max_workers=self.concurrency) as pool:
- futures = {
- pool.submit(self._run_one, instance, progress): instance
- for instance in pending
- }
- for future in as_completed(futures):
- exc = future.exception()
- if exc:
- inst = futures[future]
- print(f"\nUnhandled error for {inst['instance_id']}: {exc}", file=sys.stderr)
- finally:
- if self.tools_container_id and self.tools_container_id != "volume":
- stop_tools_container(verbose=self.verbose)
- statuses: dict[str, int] = {}
- for r in self.store._results:
- statuses[r["status"]] = statuses.get(r["status"], 0) + 1
- print(f"\nDone. {total} tasks total.")
- for status, count in sorted(statuses.items()):
- print(f" {status}: {count}")
- # ---------------------------------------------------------------------------
- # Dataset loader
- # ---------------------------------------------------------------------------
- def load_swebench(split: str = "test", dataset_name: str = "princeton-nlp/SWE-bench_Lite") -> list[dict]:
- print(f"Loading {dataset_name} ({split} split)...")
- dataset = load_dataset(dataset_name, split=split)
- return list(dataset)
- # ---------------------------------------------------------------------------
- # CLI
- # ---------------------------------------------------------------------------
- def main() -> None:
- parser = argparse.ArgumentParser(
- description="Run SWE-bench Lite benchmark using agtx as the agent runner.",
- formatter_class=argparse.RawDescriptionHelpFormatter,
- epilog="""
- The --config file is an agtx ProjectConfig TOML written to .agtx/config.toml
- in each cloned repo before the TUI starts. It controls agent, plugin, and all
- other agtx project settings. Example:
- default_agent = "claude"
- workflow_plugin = "agtx"
- worktree_dir = ".agtx/worktrees"
- """,
- )
- parser.add_argument(
- "--config",
- required=True,
- metavar="PATH",
- help="Path to agtx config.toml (written to .agtx/config.toml in each repo)",
- )
- parser.add_argument(
- "--instances",
- type=int,
- default=None,
- metavar="N",
- help="Run first N tasks (default: all 300)",
- )
- parser.add_argument(
- "--instance-ids",
- nargs="+",
- metavar="ID",
- dest="instance_ids",
- help="Run specific instance IDs",
- )
- parser.add_argument(
- "--concurrency",
- type=int,
- default=1,
- help="Parallel tasks (default: 1)",
- )
- parser.add_argument(
- "--output-dir",
- default=None,
- dest="output_dir",
- help="Output directory (default: ./swebench_output/{plugin}_{agent}_{timestamp}/)",
- )
- parser.add_argument(
- "--workdir",
- default="/tmp/swebench_repos",
- help="Repo clone directory (default: /tmp/swebench_repos)",
- )
- parser.add_argument(
- "--agtx",
- default="./target/release/agtx",
- help="Path to agtx binary (default: ../target/release/agtx relative to benchmark/)",
- )
- parser.add_argument(
- "--phase-timeout",
- type=int,
- default=1200,
- dest="phase_timeout",
- help="Per-phase max seconds (default: 1200)",
- )
- parser.add_argument(
- "--model-name",
- default=None,
- dest="model_name",
- help="Label in predictions.jsonl (default: agtx-{plugin}-{agent})",
- )
- parser.add_argument(
- "--split",
- default="test",
- help="HuggingFace dataset split (default: test)",
- )
- parser.add_argument(
- "--dataset",
- default="princeton-nlp/SWE-bench_Lite",
- dest="dataset",
- help="HuggingFace dataset name (default: princeton-nlp/SWE-bench_Lite). "
- "Use princeton-nlp/SWE-bench_Verified for multi-file/long-exploration tasks.",
- )
- parser.add_argument(
- "--verbose", "-v",
- action="store_true",
- help="Print step-by-step progress to stderr",
- )
- parser.add_argument(
- "--smoke-test",
- action="store_true",
- dest="smoke_test",
- help="Replace task description with a trivial prompt (create artifact files and stop). "
- "Use to verify the full pipeline works without spending tokens on real coding work.",
- )
- parser.add_argument(
- "--hard",
- action="store_true",
- dest="hard",
- help="Strip code blocks and stack traces from the problem statement, keeping only the prose. "
- "The agent must find and fix the bug from first principles.",
- )
- parser.add_argument(
- "--sandbox",
- action="store_true",
- dest="docker",
- help=(
- "Run each task inside its SWE-bench Docker image "
- "(swebench/sweb.eval.x86_64.*). Gives agents a working test "
- "environment with pre-installed dependencies. "
- "Requires Docker with Rosetta on Apple Silicon."
- ),
- )
- args = parser.parse_args()
- # Load and validate config
- config_path = Path(args.config).resolve()
- if not config_path.exists():
- print(f"Config not found: {config_path}", file=sys.stderr)
- sys.exit(1)
- config = load_config_toml(config_path)
- plugin = config.get("workflow_plugin", "void")
- agent = config.get("default_agent", "claude")
- run_agent = running_agent(config)
- # Resolve sandbox_copy_dirs โ paths relative to the config file's directory
- extra_dirs: list[tuple[Path, str]] = []
- for entry in config.get("sandbox_copy_dirs", []):
- src = (config_path.parent / entry["src"]).resolve()
- dst = entry["dst"]
- extra_dirs.append((src, dst))
- # Resolve sandbox_init โ shell commands to run inside the container after setup
- sandbox_init: list[str] = config.get("sandbox_init", [])
- # Resolve agtx binary
- agtx_bin = str(Path(args.agtx).resolve())
- if not Path(agtx_bin).exists():
- print(f"agtx binary not found: {agtx_bin}", file=sys.stderr)
- print("Build with: cargo build --release", file=sys.stderr)
- sys.exit(1)
- model_name = args.model_name or config_path.stem
- if args.output_dir is None:
- ts = datetime.now().strftime("%Y%m%d_%H%M%S")
- config_stem = config_path.stem # e.g. "claude-rtk-caveman-ponytail-agtx"
- output_dir = Path(f"./swebench_output/{config_stem}_{ts}").resolve()
- else:
- output_dir = Path(args.output_dir).resolve()
- # In smoke test mode with no explicit instance selection, skip loading the dataset
- # entirely and use a single synthetic instance.
- if args.smoke_test and args.instance_ids is None and args.instances is None:
- instances = [{"instance_id": "smoke-test", "repo": "", "base_commit": "HEAD", "problem_statement": ""}]
- else:
- instances = load_swebench(args.split, args.dataset)
- if args.instance_ids:
- id_set = set(args.instance_ids)
- instances = [i for i in instances if i["instance_id"] in id_set]
- if not instances:
- print("No matching instance IDs found.", file=sys.stderr)
- sys.exit(1)
- elif args.instances is not None:
- instances = instances[: args.instances]
- orchestrator = BenchmarkOrchestrator(
- instances=instances,
- agtx_bin=agtx_bin,
- config_path=config_path,
- plugin=plugin,
- agent=agent,
- running_agent=run_agent,
- model_name=model_name,
- phase_timeout=args.phase_timeout,
- workdir=args.workdir,
- output_dir=output_dir,
- concurrency=args.concurrency,
- verbose=args.verbose,
- smoke_test=args.smoke_test,
- hard=args.hard,
- docker=args.docker,
- extra_dirs=extra_dirs or None,
- sandbox_init=sandbox_init or None,
- )
- orchestrator.run()
- print(f"\nPredictions: {orchestrator.store.predictions_path}")
- print(f"Results: {orchestrator.store.results_path}")
- run_id = f"{config_path.stem}-{int(time.time())}"
- print("\nTo evaluate:")
- print(f" uv run python -m swebench.harness.run_evaluation \\")
- print(f" --dataset_name {args.dataset} \\")
- print(f" --predictions_path {orchestrator.store.predictions_path} \\")
- print(f" --run_id {run_id}")
- print("\nTo report (after evaluation):")
- print(f" uv run python swebench/report.py \\")
- print(f" --results {orchestrator.store.results_path} \\")
- print(f" --logs logs/run_evaluation/{run_id}/")
- if __name__ == "__main__":
- main()
benchmark/swebench/prebake_images.py ยท Bundled script ยท 91 lines ยท 1 finding marked
- #!/usr/bin/env python3
- """
- Build the shared agtx/swebench-tools:latest image with tmux, Node.js, Claude Code, and Grok.
- This single image is used by all benchmark sandbox runs โ the benchmark runner copies
- the installed binaries from a running tools container into each SWE-bench instance
- container at startup, saving ~3-5 min per instance compared to installing from scratch.
- The tools image is based on Ubuntu 22.04 (matching SWE-bench images) and contains only
- the agent tooling layer. SWE-bench instance images are used as-is from Docker Hub.
- Usage:
- # Build the tools image (run once, or after updating Claude Code / Grok)
- python prebake_images.py
- # Force rebuild even if image already exists
- python prebake_images.py --force
- # Show docker build output
- python prebake_images.py --verbose
- """
- from __future__ import annotations
- import argparse
- import subprocess
- import sys
- TOOLS_IMAGE = "agtx/swebench-tools:latest"
- DOCKERFILE = """\
- FROM --platform=linux/amd64 ubuntu:22.04
- ENV DEBIAN_FRONTEND=noninteractive
- RUN apt-get update -qq \\
- && apt-get install -y -qq tmux curl ca-certificates \\
- && curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \\
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 36 on GitHub
- && apt-get install -y -qq nodejs \\
- && npm install -g @anthropic-ai/claude-code @xai-official/grok \\
- && apt-get clean \\
- && rm -rf /var/lib/apt/lists/*
- """
- def tools_image_exists() -> bool:
- result = subprocess.run(
- ["docker", "image", "inspect", TOOLS_IMAGE],
- capture_output=True,
- )
- return result.returncode == 0
- def build_tools_image(force: bool = False, verbose: bool = False) -> None:
- if not force and tools_image_exists():
- print(f"Tools image already exists: {TOOLS_IMAGE}")
- print("Use --force to rebuild.")
- return
- print(f"Building {TOOLS_IMAGE}...")
- capture = not verbose
- subprocess.run(
- ["docker", "build", "--platform", "linux/amd64", "-t", TOOLS_IMAGE, "-"],
- input=DOCKERFILE,
- text=True,
- capture_output=capture,
- check=True,
- )
- print(f"Done: {TOOLS_IMAGE}")
- def main() -> None:
- parser = argparse.ArgumentParser(
- description="Build the shared agtx/swebench-tools image for sandbox benchmark runs.",
- formatter_class=argparse.RawDescriptionHelpFormatter,
- epilog=__doc__,
- )
- parser.add_argument(
- "--force",
- action="store_true",
- help="Rebuild even if the tools image already exists",
- )
- parser.add_argument(
- "--verbose", "-v",
- action="store_true",
- help="Show docker build output",
- )
- args = parser.parse_args()
- build_tools_image(force=args.force, verbose=args.verbose)
- if __name__ == "__main__":
- main()
install.sh ยท Bundled script ยท 200 lines ยท 1 finding marked
- #!/bin/bash
- set -e
- # agtx installer
- # Usage: curl -fsSL https://raw.githubusercontent.com/fynnfluegge/agtx/main/install.sh | bash
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 5 on GitHub
- REPO="fynnfluegge/agtx"
- BINARY_NAME="agtx"
- INSTALL_DIR="${AGTX_INSTALL_DIR:-$HOME/.local/bin}"
- # Colors
- RED='\033[0;31m'
- GREEN='\033[0;32m'
- YELLOW='\033[0;33m'
- BLUE='\033[0;34m'
- NC='\033[0m' # No Color
- info() {
- echo -e "${BLUE}==>${NC} $1"
- }
- success() {
- echo -e "${GREEN}==>${NC} $1"
- }
- warn() {
- echo -e "${YELLOW}==>${NC} $1"
- }
- error() {
- echo -e "${RED}==>${NC} $1"
- exit 1
- }
- # Detect OS
- detect_os() {
- case "$(uname -s)" in
- Linux*) echo "linux" ;;
- Darwin*) echo "darwin" ;;
- *) error "Unsupported OS: $(uname -s)" ;;
- esac
- }
- # Detect architecture
- detect_arch() {
- case "$(uname -m)" in
- x86_64) echo "x86_64" ;;
- amd64) echo "x86_64" ;;
- arm64) echo "aarch64" ;;
- aarch64) echo "aarch64" ;;
- *) error "Unsupported architecture: $(uname -m)" ;;
- esac
- }
- # Get latest release version
- get_latest_version() {
- curl -fsSL "https://api.github.com/repos/${REPO}/releases/latest" | \
- grep '"tag_name"' | \
- sed -E 's/.*"([^"]+)".*/\1/'
- }
- # Main installation
- main() {
- echo ""
- echo " โญโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ"
- echo " โ agtx installer โ"
- echo " โ Terminal Kanban โ"
- echo " โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ"
- echo ""
- OS=$(detect_os)
- ARCH=$(detect_arch)
- info "Detected: ${OS}/${ARCH}"
- # Check for required tools
- if ! command -v curl &> /dev/null; then
- error "curl is required but not installed"
- fi
- # Version: AGTX_VERSION pins one (e.g. AGTX_VERSION=v0.2.6), which is the
- # downgrade path when a release goes wrong. Otherwise take the latest.
- if [ -n "${AGTX_VERSION:-}" ]; then
- VERSION="${AGTX_VERSION}"
- info "Pinned version: ${VERSION}"
- else
- info "Fetching latest version..."
- VERSION=$(get_latest_version)
- if [ -z "$VERSION" ]; then
- error "Could not determine latest version. Check https://github.com/${REPO}/releases"
- fi
- info "Latest version: ${VERSION}"
- fi
- # Construct download URL
- ARCHIVE_NAME="${BINARY_NAME}-${VERSION}-${ARCH}-${OS}.tar.gz"
- DOWNLOAD_URL="https://github.com/${REPO}/releases/download/${VERSION}/${ARCHIVE_NAME}"
- # Create temp directory
- TMP_DIR=$(mktemp -d)
- trap "rm -rf ${TMP_DIR}" EXIT
- # Download
- info "Downloading ${ARCHIVE_NAME}..."
- if ! curl -fsSL "${DOWNLOAD_URL}" -o "${TMP_DIR}/${ARCHIVE_NAME}"; then
- error "Download failed. Check if release exists: ${DOWNLOAD_URL}"
- fi
- # Verify checksum (if available)
- CHECKSUM_URL="https://github.com/${REPO}/releases/download/${VERSION}/${ARCHIVE_NAME}.sha256"
- info "Verifying checksum..."
- if curl -fsSL "${CHECKSUM_URL}" -o "${TMP_DIR}/${ARCHIVE_NAME}.sha256" 2>/dev/null; then
- PREV_DIR=$(pwd)
- cd "${TMP_DIR}"
- if command -v sha256sum &> /dev/null; then
- sha256sum -c "${ARCHIVE_NAME}.sha256" || error "Checksum verification failed"
- elif command -v shasum &> /dev/null; then
- shasum -a 256 -c "${ARCHIVE_NAME}.sha256" || error "Checksum verification failed"
- else
- warn "No sha256sum or shasum found, skipping checksum verification"
- fi
- cd "${PREV_DIR}"
- success "Checksum verified"
- else
- warn "Checksum file not found, skipping verification"
- fi
- # Extract
- info "Extracting..."
- tar -xzf "${TMP_DIR}/${ARCHIVE_NAME}" -C "${TMP_DIR}"
- # Create install directory if needed
- if [ ! -d "${INSTALL_DIR}" ]; then
- info "Creating ${INSTALL_DIR}..."
- mkdir -p "${INSTALL_DIR}"
- fi
- # Install binary
- info "Installing to ${INSTALL_DIR}/${BINARY_NAME}..."
- mv "${TMP_DIR}/${BINARY_NAME}" "${INSTALL_DIR}/${BINARY_NAME}"
- chmod +x "${INSTALL_DIR}/${BINARY_NAME}"
- # Check if install dir is in PATH
- if [[ ":$PATH:" != *":${INSTALL_DIR}:"* ]]; then
- warn "${INSTALL_DIR} is not in your PATH"
- echo ""
- echo "Add it to your shell config:"
- echo ""
- echo " # For bash (~/.bashrc)"
- echo " export PATH=\"\$HOME/.local/bin:\$PATH\""
- echo ""
- echo " # For zsh (~/.zshrc)"
- echo " export PATH=\"\$HOME/.local/bin:\$PATH\""
- echo ""
- fi
- echo ""
- success "agtx ${VERSION} installed successfully!"
- echo ""
- echo " Get started:"
- echo " cd your-project"
- echo " agtx"
- echo ""
- echo " Or run in dashboard mode:"
- echo " agtx -g"
- echo ""
- # Check dependencies
- echo " Checking dependencies..."
- if command -v tmux &> /dev/null; then
- echo -e " ${GREEN}โ${NC} tmux"
- else
- echo -e " ${RED}โ${NC} tmux (required)"
- fi
- if command -v git &> /dev/null; then
- echo -e " ${GREEN}โ${NC} git"
- else
- echo -e " ${RED}โ${NC} git (required)"
- fi
- if command -v gh &> /dev/null; then
- echo -e " ${GREEN}โ${NC} gh"
- else
- echo -e " ${YELLOW}โ${NC} gh (optional, for PR operations)"
- fi
- if command -v claude &> /dev/null; then
- echo -e " ${GREEN}โ${NC} claude"
- else
- echo -e " ${YELLOW}โ${NC} claude (optional, for Claude Code integration)"
- fi
- echo ""
- }
- main "$@"
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If itโs helped, pass it on.